Turns out that AI may not be the replacement to mankind that we’ve been told it is:
The buzz about AI coding tools is unrelenting. To listen to the reports, startups are launching with tiny engineering teams, non-programmers are “vibe-coding” entire apps, and the job market for entry-level programmers is crashing. But according to a METR experiment conducted in the spring of 2025, there’s at least one cohort that AI tools still aren’t serving.
METR performed a rigorous study (blog post, full paper) to measure the productivity gain provided by AI tools for experienced developers working on mature projects. The results are surprising everyone: a 19 percent decrease in productivity. Even the study participants themselves were surprised: they estimated that AI had increased their productivity by 20 percent. If you take away just one thing from this study, it should probably be this: when people report that AI has accelerated their work, they might be wrong!
I’ve been using AI in my job for several months now and this is what I have discovered:
1. Depending on the task, it can incrementally improve performance.
2. It’s predictive powers are useful for “boilerplate” tasks, but not original work.
3. It can’t tell the difference between different versions of the same library leading to erroneous code generation.
4. Training it to make 3-4 line suggestions is more productive than having it make 20 line suggestions.
There is a future for AI, but I can’t see it taking over anyone’s job anytime soon. The only exception I can see are jobs that require pattern recognition, like fruit pickers identifying and picking ripe fruit.
BTW, I’d love to hear other people’s opinion on the matter. We have an ongoing discussion at work about this.
Agreed. Treat its outputs like a 5th week co-op student’s work and give the AI the drudge tasks. Don’t think it’s an instant master of anything.
I’ve mostly trained our AI to only suggest a few lines at a time. This gives me a chance to code review the lines before accepting them. My workflow for boilerplate stuff is:
– start typing
– scan suggestion
– tab and type some more or tab and correct
– repeat as needed
So far the AI has been slowly changing its behaviour to adopt to what I want. If it follows that flow it increases my productivity. If it suggests anymore than that my productivity is slowed.
I’ve been building software professionally for 35 years. I was *very* reluctant to use AI but two tech associates kind of shamed me into it. So I started using Grok AI about 6 months ago. I’ve mostly used it to help me write complex Postgres SQL queries. Through Vibe Coding (telling it in English what I’m looking for), it has successfully helped me write several queries that would have taken me hours/days using Google and Stack Overflow to figure out. I’ve also found that it often needs to be used in an iterative way; don’t ever assume it’ll do things perfectly the first time.
I’m also very careful to not get too reliant upon it. If I were to do that, it would be akin to using a calculator for everything instead of math in my head or asking a [human] domain expert every single question instead of trying to figure out things on my own. That makes one lazy and doesn’t keep one’s skills sharp.
Finally, I have a good friend with a Ph.D. in Computer Science who works at a very high level at Google. He insists that that AI will forever be incapable of original thought. For now, I tend to believe that this is true.
Interesting that you were able to use Grok to write SQL. I’ve been using chatGPT and haven’t had much luck with SQL.
I think you’re friend is correct, I haven’t had any insights reviewing AI generated code yet.
Incapable of original thought…so AI is a liberal.
Maybe you shouldn’t be so hard on AI. It’s ignorant, not malevolent.
I responded to a linked in article about “still using Excel? Here’s 10 reasons why you should use AI tools instead” to note that I mostly use AI ax a transparent calculator to verify that the numeric (geological) models are calculating the way I want to. Other professionals agreed, the programmers doubled-down on “you don’t need to check its work.
I guess these programmers do check (and then debug) that AI’s work.
As human productivity declines, AI is sure to catch up. /sarc
“Has AI Been Overhyped?”
Yes. And may I say, I am profoundly sick of it.
It can’t do what they say it will, it’s never going to, and most particularly it will not -ever- turn into Skynet and take over the world.
Dear AI hypsters, please shut up.
Oh, and by the way. -IF- it is true that Microsoft just laid off 10K employees because it plans to use LLMs to do all that work, in about a year they will be hiring 11K new employees to clean up the mess. (Personally I think the management realized they can’t afford to keep those people and blamed AI to keep their hands looking clean. That’s how they roll, right?)
Everything is overhyped when it first comes out. EVs, site builders, Justin Trudeau. Early adopters and evangelists always oversell everything. Eventually the truth comes out and the conversation shifts to the next new thing. The previous thing finds its niche and everyone uses it in a way that makes sense. At that point it “proves” itself.
BTW, I think you are correct that a number of companies over-hired in the early 2020’s and are now using AI as a smokescreen to align headcount with revenue.
Yes, it’s overhyped, and a bunch of companies using it are going to run into trouble because they aren’t running their own LLM for their tasks, probably get sued.
There are three very different streams of AI development.
The first is the result of what I call the “dot-com” effect. AI was, and to a certain extent still is the one area of tech where it’s possible to easily obtain venture funding. Investors remember the heady days following the dot-com crash at the turn of the century, when new companies like Facebook appeared and made their investors billions – even without a clear plan for monetization. This results in companies based on an idea without a clear monetization plan, and often results in technology in search of an application. This results in silly ideas like the dozens of websites that allow you to create cartoon avatars from photographs, or dangerous ideas like using AI to generate engineering plans. The developers compete for hype and users, in the hopes that money will somehow manifest in the future. Most of these companies will quietly implode over the next couple of years.
The second stream of AI development is where the real money is, and it’s devoted to controlling how people access information, and making it as fungible as possible. Instead of searching through dozens of possible answers to a query to find the information you’re looking for the idea is that you will simply ask one authoritative source your question, and accept whatever response you’re given. This creates a single destination for search queries that can be sold to advertisers at a premium, (since most users will search no further). It also means that you can shape what people think on any issue by getting them accustomed to accept whatever they are told as authoritative.
In addition to gaining control over the transmission of ideas, it gives the creators access to corporate information, (through scheduling and “workflow productivity” applications), since all of this data passes through the servers of the AI companies that offer it. A particularly disturbing trend is the use of AI to select and onboard employees via HR automation applications, since the client companies are unaware of the biases built into the tools. This certainly lends itself to DEI initiatives.
Quite often, these AI services are functionally massive data harvesting operations. Users have know way of knowing the extent to which this information is shared or acted upon. Microsoft, for example, has added AI tools to their Office productivity suite, which means that if these tools are left enabled, every line of text in a document, and every cell in a spreadsheet is data that they can legally access, store, and evaluate.
The third stream of AI development is the most distressing of all, and that is its evolution as an engine of deception. How many “ain’t it cool?” articles have we seen about the death of Hollywood, as AI inches closer and closer to being able to counterfeit reality in still images, video, and audio? Right now, all we have to worry about is an uncanny valley full of dead-eyed Jordan Peterson and Wayne Gretzky clones hawking cryptocurrency scams and health supplements, but that’s changing. Soon social media will be awash in shock videos intended to sway public opinion.
Before the year is out, we won’t be able to watch a single video without asking ourselves “did that really happen?” Did Jordan Peterson just call that college girl a whore? Did the IDF really beat that child to death? Did Donald Trump really call Zohran Mamdami a “Commie Jeet”?
Authoritarian governments and globalist bureaucrats are rubbing their hands at this prospect. When we reach the point where public trust in all social and citizen media breaks down, they’ll be able to appear as rescuers by insisting on protecting us from “misinformation”. People will be more willing to accept corporate media as authoritative when no alternative can be trusted. And there’ll be no way to tell whether or not governments are indirectly using these tools to manufacture their own false narratives.
The true danger from AI isn’t that of computers somehow becoming self-aware and hostile to humanity. We’re not in danger of a Skynet emerging to create killer robots: humans are building those on their own. The true danger of AI is its use as a tool to deceive people and hide the truth.
Chris
Great post, thanks.
My experience with AI coding tools so far has been that they are very good at generating boilerplate-style code that, if you were an intermediate+ level software engineer, you should have already created a template or library for.
Due to the way they’re trained, they’re utterly useless on any non-mainstream language or framework, especially if that language/framework has never had a set of published best practices and patterns that most people adhere to. It’s great at python. It’s horrible at C# or Elixir.
A template? Not the mysterious Simon template?
I spent 38 years as a software engineer.
AI is not as smart as hyped (at least ChatGPT). I have asked it to write gCode for my CNC mill. Some relatively obscure problems such as cutting arcs, it has been effective. Even though I told it i was running the gCode on linuxCNC, it still produced gCode that does not work there. When informed of this, it immediately knew why and how to fix it. So why not code it right the first time?
I was trying to get Windows OpenSSH working for doing sftp transfers from Linux. After many hours of trial by error (repeatedly), I gave up.
For simple C programs it seems to provide a decent first draft. Trying to articulate what you want is, as it is when writing a programming specification for a junior coder, a lot of work. I do not doubt that it can replace a lot of not terribly competent BSCS graduates. I think even a two years of experience software engineer still has job prospects.
In a previous role I had to help recruit developers. The applications we got were overwhelmingly East Indians, and most lied cheerfully and fulsomely on their resumes. I read resumes from “full stack developers” who clearly had no clue what the “stack” was.
Naturally, they were applying to write C++ shader code for Vulcan.
Just imagine the lovely chaos that will result from thousands of software engineers like this ,(recruited with AI-enabled HR software), using AI tools to complete their work, (which will be checked in without testing it beyond getting it to compile).
I don’t envy software development managers right now.