Qualitative-Research RAG

Built and deployed a production GenAI/RAG system on AWS for a major German market-research institute, transforming thousands of confidential, multi-format qualitative research documents into a searchable knowledge base powered by semantic retrieval.

Problem

A market-research institute had years of qualitative discussion guides and quantitative questionnaires locked in Word, PowerPoint, Text and Excel files. Researchers couldn't search across past studies to reuse questions, find precedents, or answer "have we asked this before?". Institutional knowledge was effectively unsearchable.

My role

End-to-end: technical discovery with the client, architecture, implementation, and the production AWS deployment across PROD and STAGING environments. I translated a vague "we can't find anything" into a working retrieval system, then took it from prototype to a documented, operable production architecture.

Stack

Python · FastAPI · Streamlit · Qdrant (vector DB) · PostgreSQL (AWS RDS) · Azure OpenAI (embeddings) · AWS (App Runner, EC2, ECR, S3) · Docker · Heroku (auth-flow testing)

Architecture

Three decoupled services, each in its own repo:

• Backend API (FastAPI): owns the DB access layer, Qdrant integration, S3 integration, and Azure OpenAI calls. Containerized, pushed to ECR, deployed on AWS App Runner, with separate PROD and STAGING services.
• Frontend (Streamlit): researcher-facing query UI, embeds queries via Azure OpenAI, talks to the backend over HTTPS. Dockerized, deployed on AWS EC2 rather than App Runner: Streamlit keeps a persistent websocket connection per session, which EC2 handles natively, while App Runner's timeout/health-check behavior on long-lived connections is less predictable. EC2 also keeps a clean path open for a planned migration to a React/Node frontend with proper load balancing as traffic grows, instead of being boxed into a PaaS's constraints. A separate Heroku deployment served one specific purpose: validating the app's Microsoft Entra ID authentication flow against a dedicated test S3 bucket. We didn't have access to the client's Microsoft tenant, so we proved out the auth flow on data IQ's own tenant first.
• Import/batch pipeline: a separate ingestion service, decoupled from the request path. A daily cron job on EC2 spins up a Docker container, pulls a zipped export from S3, parses the Word/Excel files, and calls the backend's internal API (insert / delete / vector upsert) to keep the index in sync.

Data layer: PostgreSQL (RDS) for document metadata, Qdrant Cloud for vector similarity search (cosine, multi-filter on study type / qual-quant / date), S3 for raw file storage. PostgreSQL and Qdrant both run as genuine PROD and STAGING instances.

Architecture diagram

Infrastructure & environments

Designed and documented the AWS setup as part of handing the system off cleanly:

• Genuine PROD and STAGING environments for AWS App Runner, RDS and Qdrant Cloud, each with its own instance
• Deployment flow: build Docker image, push to ECR, App Runner auto-deploys the corresponding environment