AI has made it easier to build impressive software quickly. It has also made it harder to tell who actually knows what they are doing.
A company looking for a software team used to be able to learn quite a lot from a prototype. If a small group could turn a rough brief into a functioning product in a week, that told you something about its technical ability. The software itself was evidence: somebody had understood the problem well enough to translate it into architecture, interfaces, data models and working code.
That inference is becoming much less reliable. Give a reasonably capable developer a modern coding agent and a clear brief, and a surprising amount of software can appear in a day. The interface may be polished, the database sensible, the tests green and the deployment automated. None of it has to be fake. The important change is that output and understanding are no longer as tightly coupled as they once were.
For companies hiring external engineering teams, this creates an awkward problem. The things that are easiest to inspect during a sales process (prototypes, portfolios, velocity and sheer volume of code) are precisely the things AI has made cheaper. The qualities that determine whether a product survives contact with reality are harder to see.
So the question is no longer simply “Can this team build what we need?” Almost certainly, somebody can. The better question is: what should you look for when building itself is no longer the scarce part?
1. Treat the demo as evidence of speed, not engineering quality
A good demo still matters. It tells you whether a team can understand a brief, move quickly and turn an abstraction into something concrete. What it no longer tells you is whether the same team can make good decisions once the neat assumptions behind the demo disappear.
Production software has a habit of becoming difficult only after people start using it. Customers behave in ways nobody predicted. Permissions multiply. An integration changes. Data arrives in the wrong format. A feature designed for one department is suddenly needed by six. The elegant shortcut that worked for 500 users becomes an expensive constraint at 50,000.
AI is remarkably useful at generating solutions to well-framed problems. Engineering is largely the work of discovering that the problem was not as well framed as everyone thought.
This distinction is already visible in developer research. In Stack Overflow’s 2026 survey, the most commonly reported costly AI mistake was not obviously broken code, but “plausible but wrong code,” cited by 69.3% of respondents. More than half also reported losing time to AI-generated code that was difficult to review.
That is a useful phrase to remember during procurement: plausible but wrong. The most dangerous team in the room may not be the one producing visibly bad work. It may be the one capable of producing convincing work without recognizing its hidden assumptions.
A prototype should therefore start the technical evaluation, not conclude it.
2. Ask what they decided not to build
One of the clearest signs of experienced engineering is surprisingly difficult to display in a portfolio: restraint.
Senior engineers tend to leave behind a trail of things that never became software. A service that did not need to be separated. A database that was not introduced. An AI feature replaced by a deterministic workflow. Six weeks of custom development avoided because an existing integration could do the job.
This matters more as implementation gets cheaper. When writing software is expensive, bad ideas encounter natural resistance because somebody has to spend weeks building them. When AI makes the first version inexpensive, unnecessary software becomes dangerously affordable.
So ask a prospective team to explain an important decision in terms of the alternatives it rejected. Why this architecture rather than another? Why build rather than buy? Which part of the brief would they challenge? What would they remove if the budget were cut by a third?
Then change one of the assumptions. Tell them customer data must remain in Europe. Multiply expected traffic. Introduce an ugly legacy system that cannot be replaced. Ask what happens if your internal team needs to take over in twelve months.
You are not testing whether the original answer was correct. You are testing whether the team understands why it was correct.
Weak technical proposals often become oddly rigid when challenged. Strong ones change shape because architecture is not an ideology. It is a response to constraints.
3. Don’t ask whether they use AI. Ask what happens when AI is wrong
Whether a software team uses AI is becoming a poor screening question. In Stack Overflow’s 2026 survey, code generation, debugging and test creation were already among the most common uses of AI in development. A team refusing to use these tools is not necessarily more rigorous; it may simply be slower.
The more revealing question is what happens between an AI agent producing code and that code reaching your customers.
Good teams have built friction into precisely the places where machines create dangerous confidence. They keep changes small enough for humans to understand. They run automated tests, but do not confuse passing tests with correctness. Important releases can be staged, observed and reversed. Generated code is evaluated in the context of the existing system rather than treated as an isolated answer to a prompt.
Google’s DORA research makes this point particularly well. It describes AI as an amplifier whose value depends heavily on the engineering system around it. DORA now highlights small-batch development as especially important because AI makes it tempting to generate large changes faster than humans can safely review them. Smaller changes help convert additional velocity into useful output rather than instability.
This suggests a better line of questioning. Ask how large their pull requests tend to be. Ask who reviews AI-generated changes. Ask what can prevent a release. Ask whether they use feature flags, staged deployments and rollbacks. Ask how they know a change has damaged something no test covered.
AI fluency is easy to demonstrate now. Verification discipline is much more interesting.
4. Find out who is doing the thinking
There is an old outsourcing trick that AI makes easier to hide. An impressive senior engineer appears in the sales meeting, asks excellent questions and proposes a thoughtful architecture. Once the contract is signed, that person largely disappears and the work moves to a more junior team.
This has always been risky. AI can now keep the illusion alive for longer.
A junior developer with good tools can produce a tremendous amount of competent-looking work. That is not an argument against junior engineers. With strong technical leadership, they can be excellent and AI can accelerate their development considerably. The dangerous setup is junior execution without enough senior judgment around it.
You therefore need to understand the actual decision structure of the team you are hiring. Who owns the architecture? Who reviews consequential changes? Who talks to your product team when a requirement is ambiguous? Who decides whether technical debt is acceptable? How much time will those people really spend on the project?
Do not accept job titles as the answer. Ask for the mechanism.
A well-designed engineering organization does not depend on a senior person staring at every line of code. It creates standards, review practices and technical boundaries that allow less experienced engineers to move quickly without making every local decision a permanent architectural decision.
The distinction matters because AI has increased the leverage of both good and bad judgment.
5. Ask them about something that broke
Most software pitches are museums of successful projects. You can often learn more from one good failure story.
Ask the team to describe a production problem whose cause it initially misunderstood. What did users see? What did the engineers first suspect? Which evidence proved them wrong? Could they reproduce the problem? What did they change afterward?
The interesting part is not whether they fixed the bug. Most teams eventually do. You are looking for how they move from uncertainty to knowledge.
Good debugging exposes several capabilities at once: understanding of the wider system, useful observability, the ability to form and discard hypotheses, communication under pressure and enough humility to notice when reality disagrees with the engineer.
There is a useful parallel in recent research on coding agents. METR asked maintainers of real open-source projects to evaluate AI-generated pull requests that had already passed the automated SWE-bench grader. Roughly half of those test-passing patches would still not have been merged by the maintainers. Some failed on core functionality, some broke surrounding code, and others were simply not of sufficient quality. METR does not present this as a hard limit on what coding agents can do; the more interesting conclusion is that passing an automated evaluation and satisfying someone who understands the system are different standards.
The same distinction applies when choosing an engineering partner. A working feature is evidence. It is not yet proof.
6. Make the breakup part of the buying decision
There is one final test that has little to do with AI but becomes more important because of it: ask what happens when the relationship ends.
Rapid AI-assisted development can create a remarkable amount of software in a short period. If the architecture, reasoning and operational knowledge remain largely inside the vendor’s head, that velocity can create dependency just as quickly.
Before work begins, establish who owns the repositories, cloud accounts, deployment pipelines and credentials. Ask where important architectural decisions are recorded and whether another competent team could operate the product. Ask what documentation and knowledge transfer look like, not as a final-week deliverable but as part of normal development.
Most importantly, ask what the system would look like if your own engineers had to take it over next year.
A good software partner should become replaceable over time. That may sound commercially strange, but technical captivity is not evidence of a successful partnership. Coyotiv’s engineering services, for example, keep technical leadership, design and engineering in one team from the first build through launch and maintenance. The documentation, team knowledge and handover a product needs are agreed as part of the engagement, so clients are not locked in.
The principle is broader than any one company: if a vendor’s value depends on nobody else being able to understand its work, you are buying a dependency rather than an engineering capability.
A better software team interview
Much of this can be condensed into six questions. They are more useful than asking how many developers a company employs, which AI tools it uses or how quickly it can produce an MVP.
- What is one thing in our brief you would probably not build, and why?
- Show us a technical decision where the obvious solution turned out not to be the right one.
- What happens between an AI agent generating code and that code reaching production?
- Who will make the difficult technical decisions on our project, in practice rather than on the org chart?
- Tell us about a production failure you initially misunderstood. How did you find the real cause?
- If another team took over our product next year, what would make that transition easy or difficult?
None of these questions has a perfect answer. That is the point. They force a team away from prepared demonstrations and into the territory where engineering judgment becomes visible.
Code is getting cheaper. Consequences are not.
The rise of coding agents is often discussed as a question of whether AI will replace software engineers. For companies buying software, there is a more immediate consequence: AI is changing what they should be paying engineers for.
Syntax was never the most valuable part of software engineering. Nor was the ability to memorize APIs or produce boilerplate quickly. Those activities simply consumed enough time that they became highly visible.
As their cost falls, the less visible parts of the profession become more important: deciding which problem to solve, understanding how one change affects the rest of a system, choosing where complexity is justified, recognizing when an apparently correct answer is misleading, designing for failure and taking responsibility after release.
This makes selecting a software team both harder and, potentially, more rational. Companies can stop being overly impressed by output that is becoming abundant and start looking for the thing that remains scarce: sound technical judgment exercised under real constraints.
The demo is cheap now. What happens after the demo is what you are buying.
