Product Description

The product is a distributed storage system but for datasets. Each dataset can be public or private. Storing a public dataset is free but a private one is paid. There is the feature of advanced searched where users can search for datasets by anything related.

Non functional requirements

First we defined our non functional requirements to help us decide in future design choices:

• this project is for AI training and dataset storage purposes.
• we are going to store multi GB files.
• thus, we are going to chunk files to 64 or 128 MB chunks
• reads will be more frequent than writes.
• random writes are not supported we are going to support appends only (for now).
• we are going to use a centrilized architecture for simplicity. A master with all the metadata and many sotrage servers.
• we will have a fixed replication factor of 3.
• we are going to assume that reads are mostly sequential for these datasets.

Core functions

These are the basics features for our projects and this is what we envisioned:

• writing a chunk.
• reading a chunk.
• appending to a file.
• having authentication for all interactions.
• having a heartbeat between master and storage servers.
• history of actions.