4 free questions from a 20-question segment. Get the full 100 →

Q21. How do you collaborate with a data science team when scoping an AI feature?

• Type: Behavioral | Difficulty: Medium
• What they're testing: Working with DS beyond "give me a model." Bringing problem + constraint, not solution.
• Framework hint:
  • Bring problem + business metric
  • Define acceptance criteria jointly
  • Agree eval set upfront
  • Iterate weekly on outputs
  • Clear PM vs DS responsibility split

Q22. Walk me through how you'd handle model drift in production — you shipped 3 months ago and users are complaining more now.

• Type: Case | Difficulty: Hard
• What they're testing: Post-launch AI ops. Most PMs think shipping is done; drift is where real AI PM work lives.
• Framework hint:
  • Detect: monitor + complaint tracking
  • Diagnose: input shift vs output degradation
  • Root cause: new segments, new uses, vendor updates
  • Fix: retrain, adjust prompts, update index
  • Prevent: continuous eval + alerting

Q28. What's the difference between training data, fine-tuning data, and eval data — and how would you source each for a resume-scoring product?

• Type: Case | Difficulty: Medium
• What they're testing: Basic ML fluency. Can you talk to engineers without cringing.
• Framework hint:
  • Definitions of each
  • Resume-scoring: labeled pairs for fine-tune, diverse held-out for eval
  • Sourcing with user consent + synthetic edge cases
  • Bias audit
  • Labeling cost and inter-rater agreement

Q30. Your model is 92% accurate in eval but users hate it. What went wrong?

• Type: Case | Difficulty: Medium
• What they're testing: Accuracy vs user value. Distribution mismatch, cost of errors, latency, presentation.
• Framework hint:
  • Eval vs production distribution gap
  • Error asymmetry — which 8% and cost
  • Non-accuracy factors: latency, tone, presentation
  • UI overselling mismatch
  • Rebuild eval + cost-weighted metrics

🔓 16 more questions in this segment alone — including grounding architecture, cold-start data strategy, and third-party data evaluation.