Motivation
It’s easy to host a static file server for daily development. I usually use it for sharing files between my devices or with my workmates.
But the maintenance cost is high. I have to decide what to keep when space runs out, which is time-consuming. This is especially true on my personal VPS server, which only has 20GB of disk space.
One solution is to use a cloud platform service like AWS S3, but it’s too pricey for me, and I can’t upload any proprietary files to the cloud.
use a Git server as object storage
I started designing my own object storage service that uses GitHub repositories to hold data.
The maximum size of a GitHub repository is 100GB, and the maximum file size is 100MB. So, I have to split bigger files into smaller pieces, and for each file, I use a meta file to store its information. Each repository is a “bucket”. When one is full, I just make a new one. (But I’d rather redesign it so a single “bucket” can use multiple repositories, so I don’t have to switch buckets when one runs out of space.). All operations can be handled by the GitHub API, so there’s no need to run any Git commands locally. I haven’t implemented an S3-compatible API for my object storage service yet, because it’s just not necessary for me.
I also deployed this service on a personal server provided by my organization, where the Git service is replaced by a self-hosted GitLab instance.
An image registry is inherently an object store
Git servers can be pretty limiting, and I don’t want to adapt to every possible Git server like Gitea or Azure DevOps. One day, I realized that an image registry is a great option for storing objects. It’s simple, easy to use, and performs really well because it’s specifically designed for large binary files. As far as I know, there’s no size limit for images or layers. I haven’t run into any limits personally. So I redesigned my object storage service to use an image registry. An “image” can serve as a “bucket,” and the digest of the “key” can be the “tag” of the “image”.
While there’s a limit on the number of layers, I only need a single layer to store file. The layer blob structure turned out to be more complex than I expected, but I discovered that the registry server doesn’t actually validate the uploaded blob content. So I just went ahead and uploaded compressed files directly to the registry server . and hey, it works great!