• positioning of a creator might change, thus regenerate creator embeddings every 30-90 days
• metrics like engagement rate, followers/following etc are already recorded and regenerated regularly

Pipeline lifecycle

1. Seed creator

• creator record is created (or upserted)
• Basic engagement metrics + metadata are fetched (separate background job) and stored

2. Enumerate similar creators

• get similar creators via instagram creator discovery job
• fetched creators are stored in the mongo with application status external

3. Creator embedding pipeline

• includes two different jobs
  • niche categorization — what kind of content does the creator post (fitness, lifestyle, finance etc.)
  • style categorization — what style of content does the creator post (cinematic, voice over, documentary, humour)
• how?
  • sample top N posts (latest, most performing etc.) → run through video analysis service
  • compute a summary for niche and category → generate and store as raw text in mongo, embedding in vector db

Search orchestration

1. User query → intent parsing

example query:

cinematic fitness creators for beginners