--- title: "Reviewing how a company builds its AI products." description: "A study platform had an AI product live that writes notes, quizzes and mock exams from students' material, and the results weren't good enough. Over two months, we reviewed the teams and the agent architecture, mentored engineers and product teams, and gave the CTO a written assessment and plan." url: https://coyotiv.com/work/edtech-ai-transformation/ --- 1. [Work](https://coyotiv.com/work/) 2. [Education technology company](https://coyotiv.com/work/edtech-ai-transformation/) # Reviewing how a company builds its AI products. A study platform had an AI product live that writes notes, quizzes and mock exams from students' material, and the results weren't good enough. Over two months, we reviewed the teams and the agent architecture, mentored engineers and product teams, and gave the CTO a written assessment and plan. [Book a call](https://coyotiv.com/work/edtech-ai-transformation/#book) [See all work](https://coyotiv.com/work/) Client Education technology company Sector Education technology Our role AI organization review, architecture mentoring and technical leadership advisory Today Five recommendations adopted, with a plan for the next two quarters The story 1. [The starting point](https://coyotiv.com/work/edtech-ai-transformation/#the-starting-point) 2. [How the teams worked](https://coyotiv.com/work/edtech-ai-transformation/#how-the-teams-worked) 3. [How the agents worked](https://coyotiv.com/work/edtech-ai-transformation/#how-the-agents-worked) 4. [How to think about AI products](https://coyotiv.com/work/edtech-ai-transformation/#how-to-think-about-ai-products) 5. [How the engineers worked](https://coyotiv.com/work/edtech-ai-transformation/#how-the-engineers-worked) 6. [The outcome](https://coyotiv.com/work/edtech-ai-transformation/#the-outcome) ## The starting point The company had an AI product in production that writes notes, quizzes and mock exams from students’ material. In our tests, the exams often didn’t reflect the notes students uploaded, and some questions came from unrelated subjects. The causes went beyond the code. Team responsibilities overlapped, the agents were built to answer in one step, and there was no shared definition of a good exam. ## How the teams worked Responsibilities for the AI product were spread across more than one team, and it wasn’t always clear who owned what. Some work duplicated what the core platform already provided. We recommended small teams of two or three engineers and a product manager. Each team owns one AI product from start to finish on a shared platform, with its own goals and measures of success. ## How the agents worked We mentored the AI engineers on agent architecture. The agents wrote whole exams from one prompt, and their search sometimes returned material that sounded similar but didn’t fit the course. We showed why one prompt can’t produce a reliable exam, and laid out the stages it needs: reading the document, understanding the course, writing questions, writing answers, and checking each one with a limited number of retries. We shared our research on structured reasoning, and recommended choosing models through evaluation and fine-tuning only narrow tasks. ## How to think about AI products We recommended defining what a good exam is together with educators, starting with standardized tests that publish their guidelines and learning objectives. We suggested building on checked content and on what the company already knows about how students study. To keep cost and quality in view, we proposed a short preview before a full exam, which also gives the product a natural place for a paid tier. We also looked at smaller issues in the interface. One problem that looked like a data architecture question turned out to be a loading bug, which saved the team a much larger project. ## How the engineers worked We recommended a simpler setup so new engineers can start working in hours, a budget for AI coding tools with shared ways of using them, and written standups in place of large meetings. ## The outcome At the meeting that closed the review, leadership chose to focus on core quality before new features. They adopted five recommendations: focus on the core product, give students’ own notes priority, show a short preview before a full exam, fix the basics before building a full test setup, and put image generation on hold. The CTO had a written plan for the organization and the next two quarters. How it went ## Four areas we reviewed. 1. Organization ### How the teams worked We interviewed people across product, AI, infrastructure and leadership. We recommended small teams that each own one AI product from start to finish, on a shared platform. 2. Architecture ### How the agents worked We mentored the engineers on agent design, and traced weak exams to how the agents found their material. We laid out a staged workflow that writes and checks each question on its own. 3. Product ### How to think about AI products We helped the product teams define what a good exam is, involve educators, and test small versions before building more. We proposed a short preview before a full exam. 4. Assessment ### The written plan We delivered a written assessment to the CTO and leadership, with actions for 30 days, 90 days and two quarters, a list of risks, and measures of success. The result ## The result A written plan for the CTO, covering the next two quarters. Assessment Interviewed across product, AI, infrastructure and leadership. 15 people Recommendations leadership adopted, starting with the quality of mock exams. 5 adopted More work ## More work. [See all work](https://coyotiv.com/work/) Personalized products and fulfillment ### [A critical application, ready to run the business.](https://coyotiv.com/work/thefreshener/) Months of continued development added features and integrations, allowing the company to expand its operations to new product types. TheFreshener HR technology and hiring ### [A hiring workflow that had outgrown a single agent.](https://coyotiv.com/work/klareda/) Klareda could take on larger hiring use cases, with the work divided across agents and their steps available for review. Contact ## Book a call. Free, 30 minutes, with an engineer. Tell us what you are working on. - Startups Free architecture review - Enterprises AI review - School Apply for the next cohort [hello@coyotiv.com](mailto:hello@coyotiv.com)