The starting point
The company had an AI product in production that writes notes, quizzes and mock exams from students’ material. In our tests, the exams often didn’t reflect the notes students uploaded, and some questions came from unrelated subjects.
The causes went beyond the code. Team responsibilities overlapped, the agents were built to answer in one step, and there was no shared definition of a good exam.
How the teams worked
Responsibilities for the AI product were spread across more than one team, and it wasn’t always clear who owned what. Some work duplicated what the core platform already provided.
We recommended small teams of two or three engineers and a product manager. Each team owns one AI product from start to finish on a shared platform, with its own goals and measures of success.
How the agents worked
We mentored the AI engineers on agent architecture. The agents wrote whole exams from one prompt, and their search sometimes returned material that sounded similar but didn’t fit the course.
We showed why one prompt can’t produce a reliable exam, and laid out the stages it needs: reading the document, understanding the course, writing questions, writing answers, and checking each one with a limited number of retries. We shared our research on structured reasoning, and recommended choosing models through evaluation and fine-tuning only narrow tasks.
How to think about AI products
We recommended defining what a good exam is together with educators, starting with standardized tests that publish their guidelines and learning objectives.
We suggested building on checked content and on what the company already knows about how students study. To keep cost and quality in view, we proposed a short preview before a full exam, which also gives the product a natural place for a paid tier.
We also looked at smaller issues in the interface. One problem that looked like a data architecture question turned out to be a loading bug, which saved the team a much larger project.
How the engineers worked
We recommended a simpler setup so new engineers can start working in hours, a budget for AI coding tools with shared ways of using them, and written standups in place of large meetings.
The outcome
At the meeting that closed the review, leadership chose to focus on core quality before new features. They adopted five recommendations: focus on the core product, give students’ own notes priority, show a short preview before a full exam, fix the basics before building a full test setup, and put image generation on hold. The CTO had a written plan for the organization and the next two quarters.