The eLLMephant in the room

Sorry about the dumb title.

So I feel like I should put something to paper about how AI is going at the moment. Maybe that's the last thing anyone needs, god knows we hear enough about it.

If I have a purpose in doing so, it's to try and create a snapshot of how I perceive AI in May of 2026. Things change pretty fast, so it might be interesting to look back in a few months or years and see how much things have changed.

As it relates to software engineering

I'm always wary of adopting centrist positions for fear that I'm just trying not to offend either side. But that's pretty much how I feel about AI in software at the moment.

There's one very vocal camp that seems to want to convince you that AI is literally useless for serious software engineering. Everything it does is riddled with insurmountable design failures and bugs, to the point that no self-respecting software engineer should even look sideways at it. If you even let AI get a whiff of your code, your system will collapse in a flaming heap, ejecting sensitive user data and credentials across the Internet for all to see.

I'm somewhat sympathetic to this point of view. Not because I think it's a good model of reality, but because it does capture how I often subjectively feel about AI. Spoilers for later, I don't really like it, mostly for reasons unrelated to software engineering. I'd love an excuse to believe this kind of vitriol. But I just can't convince myself this is true. I've used AI to help build some pretty sophisticated stuff, I've reviewed the code, and yeah - it's not perfect by any stretch and there's sometimes some wild stuff in there - but to claim it's worse than useless just doesn't reflect the experience I've actually had.

I know, I know, there are studies showing that people think they're being more productive with AI when they're actually less productive. I don't have a good answer to that. I really don't think it's true from what I've done with it so far, but I haven't got any data to back that up.

The opposite, equally vocal, camp would try to convince you that software engineering is already dead as a career and there's nothing AI can't do. Just tell it it's a genius programmer and not to make mistakes, and you can one-shot a billion-dollar web app.

This is equally in contradiction with reality, at least for the moment. Software just doesn't work that way. Any mature product reflects years of subtle decisions hard-wrought from bug reports, user complaints, security breaches, hours spent in front of a whiteboard sketching out flows that make sense to users and can survive the wild and irrational behaviours they often exhibit (sorry users).

Maybe a good way to think about AI is that it's really good at speed-running a waterfall project management approach. You can get it to flesh out a hugely detailed spec up-front, break that down into implementation stages, and write the code. But you still have all the problems of the waterfall approach, just faster. You haven't learned anything from users. You haven't done any experiments, A/B testing, user interviews, persona modelling, or any of the other stuff that helps slowly iterate a software product into something decent.

There's also so much human context that goes into designing software that it's just intractable to get into an AI system. Who's going to be operating it? How stable is the organisation? Does it need to be something that a consultancy can be dropped into blind to work on later, or will it be lovingly maintained by a team of craftspeople? Does it need to be easy to change, or have the most features, or incredibly stable? What business pressures will affect how the software will evolve? Do we need to encode heaps of guardrails into the software so it's hard to accidentally break, or should we keep it light and easy to modify?

All this stuff affects how we design systems, whether we realise it or not. If you know one particular dev team are loose units, you might try and build more defensive checks into parts of the system they touch so it's less likely they'll break something. Maybe you insist on a really tight DB schema so it's harder for them to accidentally violate data integrity rules. Maybe you know there's going to be a sales push to add some particular crazy feature later, so you build in some defensive stuff now so that it's not as stressful when that happens.

We absorb all that by being humans existing alongside other humans in our organisations. We chat, and gossip, and read between the lines in Slack messages. I just find it hard to believe that AI will be able to do this anytime soon. It's just too much context.

The counterargument from the AI maximalist camp might be that none of that matters. You get the AI to build the software, then when there are problems, you get the AI to fix them. The design doesn't matter, you don't even need to look at the code.

But I think that tends to lead to systems that suck as a user. I don't care if I can submit a bug report and AI will ship a fix in three minutes. By the time I've hit my fifth issue while I'm just trying to get something done, I'm not submitting another bug report, I'm just rage-quitting that software.

The other thing is that - at the moment at least - AI isn't really good at evolving codebases in a principled way. It's heavily biased towards solving problems by adding more code. It takes a lot of pressure to get AI to refactor things, to reduce duplicated code, to simplify. I don't think the AI maximalist position of "just use more AI" gets you out of this one. As the amount of code in a system increases, possible interactions between parts of the system explode in number, and you tend to get emergent complexity that's very hard to reason about. Once systems get too complex, fixing them is like fighting a hydra: cut off one bug, and two more grow in its place. At that point, each fix is a net increase in bugs, and AI just lets you do that at a superhuman pace.

Then, once you've reached that point, your app is likely so massive and sprawling that no AI system can fit even a simplified version of it into context. So you can't even get the AI to fix the fundamentals anymore. What do you do in this situation? The only approach I can think of would be to start again. And because humans weren't designing the app, no-one learned anything, so I'm not sure you can even hope that the next version will be better due to experience.

Well that was all very long-winded. I guess my point is: I do think AI is good enough to be a useful tool at the moment, I don't think it can be trusted to design complex systems unsupervised. I'm fully prepared for that to change. Today, you can't one-shot a complex app with AI and expect it to work well, but having an experienced software engineer steering the AI can lead to a good result. Maybe soon will come a day when that's no longer true, and the AI can produce as good a result unsupervised as the software engineer would be able to get out of it. We're not there yet, but I'm trying to be mentally prepared for when - or if - we arrive.

As it relates to the Internet

It makes me pretty mad to be honest. This really struck me the other day; AI may well ruin the Internet in a complete, final way that would have been unimaginable a few years ago.

It's not hard to imagine a future where the vast, vast majority of content on the Internet is AI generated. The long-form articles, the tutorials, the comments, the videos, the memes - everything. Trying to find anything created by humans will be like trying to find a needle in a haystack, but where the hay is also made of imperceptibly differently-shaped needles. And every single piece of that content may be trying to mislead you, manipulate you, or gaslight you, likely in coordinated ways more sophisticated than could ever have been done by humans.

That's why I think AI can comprehensively destroy the Internet in a way that no other technology could. Even if you were to tear down all the physical infrastructure, people can rebuild that. Folks are already building their own networks with things like MeshCore and Meshtastic. But if you have a technology that can swamp any system with information that's machine-generated but indistinguishable from human communication, then it doesn't matter what infrastructure you build, none of the information it carries can be trusted.

The only way out, that I can think of, is awful dystopian ideas like Peter Thiel's Worldcoin. Using shockingly invasive biometrics to prove that content on the Internet originated from a human. Here's not the place for a Thiel rant, but I hope you can understand why I'm not exactly excited by that as a solution.

What really makes me angry is - why are we allowing this to happen?! These companies are coming out and saying "we're building the technology that will quite likely permanently destroy the Internet for everyone, and we're pouring trillions of dollars into doing it as fast as we can". And instead of treating that as a sinister threat to one of the most important technological developments in human history, people seem to be cheering it on. I just don't get it.

As it relates to society

I don't think I have much to say here that other people haven't already put more eloquently.

We're racing to create technology that outcompetes human labour, in a world where the default is that you're not permitted to live unless you can sell your labour.

It fills me with despair that a technology which should be a wondrous liberator from drudgery will instead become a tool used to strip even more power and dignity from the 99% of humanity that don't own it.