A personal project I ran to get hands-on with AI-assisted coding and to actually learn how simulated user testing works
I designed the study, personas, methodology, and scoring rubric myself. I used Claude to write and debug the implementation code (API calls, data pipelines, scoring scripts) based on my specifications.
This was as much a learning exercise in working with AI as it was a project. I wanted to know what it actually feels like to build something like this end to end.
A lot of AI products and companies are starting to test by simulating users. So, instead of hiring real people to try an app, you have an AI pretend to be a user and see how the product responds.
I thought it would be interesting to test how well AI can reconstruct a user’s chat from a small slice of what a real person actually said. How much of the persona is preserved and what gets lost?
<aside> 💡
Fidelity, in the context of AI, is the measure of how accurately an AI model can replicate human user behavior, style of talking and decision-making patterns when simulating a human user.
</aside>
I was also personally interested in learning more about AI-generated user generations. Turns out, user simulation has a whole history before LLMs and I had studied Markovian chains and decision trees in class but never made the connection of this work to neural networks and LLMs.
Along the way, my research question and method shifted a bit. A few unexpected findings came up that I hadn't planned for, and I decided they were worth following instead of sticking rigidly to the original plan.
I made up six fictional students, grade 3 through college sophomore, each with their own math level and way of talking. I had Claude write example homework-help conversations for each one. Then I only showed part of each conversation (60%) to a different model, Gemini, and asked it to figure out who this student was and generate predicted conversation, purely from that partial slice.
Then I ran two tests on that reconstruction:
A note on the data: I didn't have real interview subjects to work with, nor did I think I needed to have real participants at this stage of the process, so the tool simulates interviews from a persona I fed in. Every transcript in here is clearly labeled synthetic.