<aside> 🌌

GITHUB UNIVERSE is an interactive GitHub data-visualization experiment that turns a large preprocessed dataset into a multi-level spatial universe: galaxy → geography → developers → repositories.

</aside>

1. Project overview

GitHub Universe explores how a large public-data experience can feel interactive and immediate without requiring a runtime application backend for every visitor.

The project uses a build-time/static-data pipeline: data is collected and transformed ahead of deployment, partitioned into JSON payloads, and consumed directly by the visualization frontend.

The experience is organized as a four-level drill-down:

1. Galaxy view — geographic/country-level entry points;
2. Country / city view — regional exploration;
3. Developer population — user nodes;
4. Repository field — repositories associated with the selected context.

2. Problem

A naive GitHub visualization can become expensive and slow in two places:

• collecting enough useful data from public APIs;
• rendering hundreds or thousands of visual nodes without overwhelming the browser.

The project therefore separates data acquisition from runtime visualization and treats rendering density as a performance problem rather than assuming every node/label should always be rendered at full fidelity.

3. Static-data pipeline

flowchart LR
    API["GitHub data sources"] --> SCRAPER["Python 3.12 scraper / builder"]
    SCRAPER --> BUCKETS["Query-bucket collection"]
    BUCKETS --> NORMALIZE["Normalize + deduplicate"]
    NORMALIZE --> INDEX["Global index JSON"]
    NORMALIZE --> PARTITION["Partitioned country payloads"]
    INDEX --> DEPLOY["Static deployment"]
    PARTITION --> DEPLOY
    DEPLOY --> BROWSER["Visualization client"]

The browser does not need to reproduce the entire collection process. It reads prebuilt payloads that are already shaped for visualization.

Why build-time processing

This provides several benefits: