🧵https://filecoinproject.slack.com/archives/C08TVNKJV7C/p1778671570686079

Prior Art

FilBeam for IPFS data on Filecoin

Requirements

• Trusted Gateway
  • FB as trusted gateway
  • Render IPFS data
  • Copy code from Boxo trusted gateway
  • Maybe base on https://specs.ipfs.tech/http-gateways/path-gateway/, but don’t implement everything (eg IPNS resolution)
• Personas
  • IPFS Gateway Operator
    • Example: PinMe
      • ~471 GB/mo estimated
  • IPFS Storage User
    • Example: NFT2sMe
      • 1 TiB/mo estimated
  • Suggestion: Target Storage Users first
    • Root CID retrieval (GET /ipfs/{rootCid}) covers the majority of their use cases — single files, CAR exports, bulk retrieval
    • ❌ This proposal doesn’t work, as NFTs are usually stored in multiple blocks, so you still need to do the full traversal
      • Instead: go with subpath retrieval, as is in the FB WIP PR
    • Outdated thinking
• Customers
  • PinMe
    • Would host their own deployment
      • That would mean their own contracts
      • Complex, gets into PDP Boss land
      • Doesn’t help with ARR for us
      • Unless PinMe becomes a partner
    • [ ] Reach out to learn about their needs
• Metadata
  • withCDN
    • Enables CDN, but requires auth token
  • public
    • Together with withCDN enables unauthorized retrievals
    • [ ] Write spec
• Authorized retrievals
  • Jenni suggests we distinguish authorized retrieval withCDN from public retrieval withCDN+public
  • Authorized retrieval requires passing an authentication token with the request
  • Problem: If you want to expose a FilBeam authorized URL to the public, you need to leak the authorization token
  • Without a mechanism for rotating the token, or creating time-locked tokens, this renders it useless for that use case
  • Only works for when customer backend fetches from SP, and not public client.
  • How important is this use case?
    • If not important enough: Remove authorized retrievals
    • If important: Specify more sophisticated token system
  • Jenni also suggests a policy layer that might offer more fine grained control
    • Example: Rate limit, to prevent egress quota drain
• CAR fetching and rendering
  • Can @ipld/car + ipfs-unixfs-exporter be fast enough?
  • Options
    • Similar to Boxo: Block-by-block on-demand fetching as we walk using ipfs-unixfs-exporter
    • As in Miro’s POC (@ipld/car + ipfs-unixfs-exporter). Hugo was concerned that’s too slow. We need to fetch the full CAR file to extract a single block from it.
      • Caching is harder, especially in CF workers

Suggestion

• Follow the IPFS subdomain spec and render subpaths of CARs addressed by their root CID + subpath (as already is in the FB WIP PR)
• This should both support the IPFS Gateway Operator persona (PinMe) and the IPFS Storage User persona (NFT2sMe)
• Performance is still questionable, as FB always needs to fetch the full CAR to extract one or more blocks
• The majority of remaining work is productionizing what is already there in the PR
• Open questions
  • Authorized retrievals: This is an entirely new feature for FilBeam, which is independent of IPFS. I'm not sure we should add that to the scope
    • Jenni requested this, to protect egress quotas
    • If we go too simple, it will have limited use. E.g. if there's one fixed token per data set, you will leak it and expose the data set when pointing public clients at it. This only works well for backends fetching from SPs.
    • If we allow to rotate tokens, or time-lock them, things get more complex and need to be specified
    • Suggestion: Is fixed-token-per-data-set good enough for v1 users? If so, go that way. If not, don't include auth at all.
    • Still need to design fixed token architecture: How to share the secret between Client, SP and FilBeam, but no-one else
  • PinMe deploying their own infra + contracts will not help with ARR (for now), do they have to?
  • Cloudflare: When do we move away from Cloudflare?
• Action items
  • Ask PinMe about their requirements