AI Coding Agents' Limitations
I have been using the AI agents in coding for nearly 1 year in work and personal projects. The AI models/agent systems are admittedly very impressive and it’s still evolving with a rapid speed. In some peripheral projects, I have come to a point that I nearly leave all the dev works for the AI to Complete without extended code review. All I do is test the outcome/result of the code generated by the agent.
Still I have some problems in mind that the agent system cannot reliably solve. The implication is that I am leaning towards the opinion that human engineers are still very much needed/relevant in the loop of software engineering at least in the near future.
An agent cannot know reliably when a complicated problem is impossible.
Or it needs your decision, pushing back the problem at you again. You still need to make a decision: Let it keep grinding or give up and find another way out. Your expertise is still very much relevant in this case. Or you will end up with hundreds of millions of tokens and a worse codebase. Picture a very simple problem: To predict the nth digit of Pi. You don’t understand anything about calculus, you ask the agent about this thing. Your agent work his way in a hard way for half an hour. Come up with a choice for you: Give up or try to run through the first 1 billion digits and build up a Markov model to observe any pattern. At this stage, you have no idea, but you have the belief your agent can do anything for you. You tell it to go for the Markov thing. 10 minutes goes by, and the agent come up with a vague result, claiming some pattern it finds and would like to do more verification further. You approved it and It came up disapproving it’s own conclusion 10 minutes ago. You are back to the launch. You tell your agent to try another way… This loop can go on and on until one day you are broke and end up realizing it may not be possible.
An agent won’t tell you if you are going to make it upfront
An agent is capable of a lot of things. But maybe this fact can bite you. Because you may have the belief that your cutting edge Fable model can do anything for you. You are also lucky that your company have your back on the budget side: unlimited token usage per month. You trust your cli buddy in your gut. Whenever you have a problem you ask it to go ahead. But here comes the danger: From the agents’ point of view, The best way from an engineering point of view is to go for route A. Whilst it does not have the full context you have: You may be under very tight constraint – Maybe you can’t afford a recompilation because it’s an urgent production issue. You need to fix it asap and every second matters. You have no idea at this time and tell your agent to go. It took 1 hour and still going, you can’t wait and stopped it. But then it tells you it need another 20 minutes to roll back what it has already done and go for a hot fix. Meanwhile, your company lost 10 million dollars during that period. An engineer’s context and judgement is very much relevant in this case.
You don’t know the right problem to solve
You can ask your agent for a big favor, something like “Create me an absolutely gorgeous website”. And it really does yield a pretty one for your in 10 minutes. You think case closed. But if you are not an experienced engineer, you won’t know that there are quite some more stuff to test/consider about before your site go live. e.g., Does it work equally nicely on mobile devices ? Maybe the website is dynamic and you are not going to be able to host it on github sites. Or maybe your agent has used some framework that’s going to charge you for commercial usage and you haven’t considered about it. The list goes on and on. It’s not that the agent can’t help you solve these problems, it’s that you don’t even realize you have so many problems to solve in association with your real goal.
Security
AI agents are becoming more and more intelligent and clever and it has come to a point that an innocent agent, given enough permission can cost you a lot. And securities are becoming more and more of a concern for engineers. I can foresee that in the near future that the whole software engineer will lean more towards safety/security. And permission control will be much of a topic than it is now.