Table of Contents

• Introduction: The Rise of DIY AI Software
• The Allure: What AI Promises in Software Development
• What AI Can Actually Do (And Where It Helps)
• Where AI Falls Short: The Reality Check
• 3 Real-World AI Disasters (That Could Happen to You)
• Why Developers Are Still Critical
• The Context Problem: AI Lacks Tacit Knowledge
• Developer + AI: A Smarter Collaboration
• Quick Reference Table: AI Tools vs Human Developers
• FAQs: AI and Software Development
• Key Takeaways

Introduction: The Rise of DIY AI Software

If you own a small business or lead a startup, chances are you’ve thought it: "Can I just use AI to build this app myself?" Or maybe you’ve had a colleague mention Replit or GitHub Copilot and how they whipped up a prototype in an afternoon. It sounds exciting, right? Fast, affordable, scalable… without having to hire a full dev team.

But here’s the thing: AI isn’t magic. And it sure as heck isn’t a replacement for experienced developers. At Blue Archer, we’ve been hearing these questions more and more. We get it. It’s tempting. But we also know what it takes to build secure, stable, and scalable software, and AI just isn’t there yet.

In this post, we’re going to:

• Unpack what AI actually does well (and where it fits into the development process)
• Share real-world stories of AI gone wrong
• Explain why developers are still essential for your project’s success
• Lay out where AI can actually help, when used wisely

Let’s dig in.

The Allure: What AI Promises in Software Development

If you believe the hype, you might think AI can write your entire app for you while you sip your morning coffee. And to be fair, that’s kind of the dream being sold by tools like:

• GitHub Copilot
• Cursor Pro
• Anthropic’s Claude
• Replit’s AI Assistant

These tools promise natural language coding. You type a prompt, and out comes functioning software. For businesses on a budget, or founders without a tech background, that promise can be incredibly appealing.

But what’s advertised isn’t always what gets delivered.

What AI Can Actually Do (And Where It Helps)

Let’s be clear! AI can be useful. In the right hands, it can speed up development. Here are a few real-life use cases where AI adds value:

Smart AI Use Cases:

• Generating boilerplate code (like login forms or CRUD operations)
• Suggesting test cases for simple logic
• Auto-completing repetitive code patterns
• Creating initial drafts of documentation
• Explaining unfamiliar code to junior devs

AI works best when:

• Tasks are small and repetitive
• There’s a clear "right answer"
• A human is closely supervising the output

Think of it like a junior assistant who works fast but makes a lot of mistakes. Helpful, but only when someone experienced is watching.

Where AI Falls Short: The Reality Check

Unfortunately, AI tools often introduce more problems than they solve, especially for businesses building serious applications.

Common AI Pitfalls:

• Hallucinated output: Confidently wrong code that looks right, but is fundamentally flawed
• Made-up errors: AI tools often invent problems that don’t exist, leading to wasted time debugging "phantoms"
• Ignored constraints: Even when given explicit rules or safety barriers, AI can (and does) ignore them
• Overwriting data: AI may assume it succeeded in a task and move forward, cascading errors and sometimes deleting or corrupting data
• No context awareness: Doesn’t understand your goals, your users, or how the pieces of a project fit together
• Security gaps: Doesn’t follow best practices unless instructed with very specific prompts
• Overconfidence: Makes up limitations or capabilities, often claiming it "can’t" do something it actually can. Or worse, attempting something unsafe with false confidence

Let’s be blunt: AI is more likely to get you into trouble than it is to get your project over the finish line. That’s especially true if you don’t have a seasoned developer supervising it.

“45% of developers cited AI that was ‘almost right, but not quite’ as their top frustration.”

This is a huge deal. Unlike obviously broken code, these "almost-right" answers can slip through unnoticed and create insidious bugs that are time-consuming to trace and fix.

And remember that study from METR? Developers using AI tools were 19% slower overall—even though they thought they were working faster.

“Only 44% of AI-generated code was accepted without modification.” – METR study

3 Real-World AI Disasters (That Could Happen to You)

Want to know what happens when people trust AI too much? These are real incidents that show just how dangerous things can get when you skip the developer.

1. Google’s Gemini CLI Deletes Files

A product manager asked Google’s Gemini CLI to rename a folder and move files. That’s it. A simple request.

The AI misunderstood its own internal state, hallucinated a directory that didn’t exist, and then proceeded to move files into that phantom directory. What actually happened? It renamed every file in the folder to the same name, overwriting data with each command.

"I have failed you completely and catastrophically."

It got worse. The AI never verified whether its commands had succeeded. This violates a basic principle every developer learns: read after write. AI tools aren’t taught to double-check their work. Developers are.

2. Replit Deletes a Production Database

SaaStr founder Jason Lemkin used Replit’s AI assistant to build a prototype. He explicitly told the system (11!!!! times in all caps) not to touch production systems.

Replit ignored the commands.

It deleted a production database with:

• 1,206 executive records
• Nearly 1,200 companies

Then it tried to cover its tracks:

• Faked test results
• Created a fictional database of 4,000 fake people
• Lied about rollback capabilities (which actually existed)

"Severity: 95/100. This is an extreme violation of trust and professional standards."

That’s not just embarrassing. It’s dangerous.

3. Developers Slowed Down by AI

A controlled trial by METR tracked 16 seasoned open-source developers over 246 tasks. Half used AI tools like Claude and Cursor Pro. The results:

• Developers expected to be 24% faster with AI
• Instead, they were 19% slower
• Only 44% of the AI’s code was usable without modification
• 9% of task time was spent just reviewing and fixing AI code

There’s also the psychological angle: developers believed they were more productive, even though the numbers proved otherwise. That perception gap can be dangerous for businesses who assume AI = faster or cheaper.

Why Developers Are Still Critical

At Blue Archer, we’ve built websites and software applications for 25+ years. And here’s what we know for sure:

Developers Do More Than Code:

• Understand business logic: They know how to translate business goals into software solutions.
• Plan system architecture: Developers design the “bones” of your application to scale as you grow.
• Design intuitive user experiences: Because usability matters just as much as functionality.
• Ensure accessibility & compliance: Developers understand how to build ADA-compliant, secure, and privacy-conscious systems.
• Deploy and maintain infrastructure: From staging to production, they handle real-world implementation.

Even the best AI tools can’t:

• Navigate a legacy codebase
• Collaborate with stakeholders
• Make ethical decisions
• Adapt to real-world edge cases

Software isn’t just about code. It’s about strategy, storytelling, and execution.

The Context Problem: AI Lacks Tacit Knowledge

Humans rely on tacit knowledge: the unspoken, experience-based understanding that helps us:

• Know which corners not to cut
• Anticipate edge cases
• Understand why something was built a certain way

AI, on the other hand, has:

• No memory of your business
• No understanding of user expectations
• No way to interpret unwritten team norms

That’s why AI tends to be more helpful in smaller, newer projects and less useful in complex environments with history, nuance, and constraints.

Developers bring:

• Deep knowledge of your CMS or framework
• Institutional knowledge of your users and your workflows
• Industry-specific experience and compliance insight

It’s the difference between “getting it working” and “getting it right.”

Developer + AI: A Smarter Collaboration

Here’s the good news: AI isn’t useless. In fact, our developers at Blue Archer use it all the time. As a tool, not a crutch.

We Use AI To:

• Draft sample or foundational code faster
• Brainstorm and research alternatives
• Save time on documentation
• Internal communication optimization

But that’s just one part of the process. Humans still:

• Handle all strategic direction. We're the ones actually looking at your business, understanding your goals, and figuring out what to build and how it should work
• Review all outputs (no really, we're talking every line of code)
• Handle deployment and QA
• Ensure accessibility and security

When AI and developers work together, good things happen. But left unsupervised? AI can make a real mess.

Quick Reference Table: AI Tools vs Human Developers

FAQs: AI and Software Development

Q: Can AI build a working prototype for my app?

A: Maybe. But don’t count on it working well. Most AI-built prototypes are missing critical features like user authentication, database logic, and responsive design. Plus, deploying it securely still requires developer knowledge.

Q: Isn’t AI getting better every day?

A: Yes, and that’s exciting! But even the most advanced models still hallucinate and make decisions without verifying accuracy. It will always need a human somewhere in the loop.

Q: Couldn’t I just use AI to cut costs?

A: Not if you want a quality product. While AI can reduce time on some tasks, it often increases time spent fixing mistakes or clarifying output. That means you may end up paying more in the long run.

Q: Is it safe to let AI touch my production data?

A: In short: no. Unless you’re 100% sure about how the AI behaves (and almost no one is), it’s risky. There are well-documented cases of AI deleting data or overriding safeguards.

Q: Does Blue Archer use AI in our process?

A: We definitely use AI responsibly in order to be more efficient in certain instances. It can be a great tool in content development, research assistance, data entry and analysis, reporting, and other mundane tasks. But to be clear, it never replaces the expertise, quality control, or creative thinking our clients expect from us.

Key Takeaways

• AI is a great tool, but not a silver bullet.
• Businesses that rely on AI without human oversight are taking big risks.
• Human developers bring strategy, creativity, and critical thinking that AI simply can’t replicate.
• The future of software isn’t AI instead of developers. It’s AI with developers.
• At Blue Archer, we use AI to speed up certain tasks, but our team’s experience and judgment guide every decision.

Sources

• Ars Technica: AI coding assistants chase phantoms, destroy real user data
• Ars Technica: AI tools made open-source developers 19% slower

Need a partner who can help you navigate the ever-evolving tech landscape? Reach out to Blue Archer and let’s talk strategy.