Shaser

Shaser is a SHA256-indexed file cache and HTTP server. It keeps a local database that maps a SHA256 hash to an absolute file path, can download missing files from peer servers, and can register uploaded files into the server cache.

The index layout is compatible with iaito's samples database.

Build

make

The default cache root is platform-specific and can be overridden with -root or SHASER_ROOT:

macOS:   ~/Library/Application Support/shaser/samples
Linux:   ${XDG_DATA_HOME:-~/.local/share}/shaser/samples
Windows: %LOCALAPPDATA%\shaser\samples

Install system-wide or for the current user:

make install PREFIX=/usr/local
make user-install
make uninstall PREFIX=/usr/local
make user-uninstall

user-install and user-uninstall are intended for r2pm. They install the binary into $(r2pm -H R2PM_BINDIR) and the library/header under $(r2pm -H R2PM_PREFIX).

DESTDIR is supported for package managers:

make install PREFIX=/usr DESTDIR="$pkgdir"
make user-install DESTDIR="$pkgdir"

Cache Layout

Index bucket files use the iaito layout:

<root>/<first2>/<next2>/<first4>

Each bucket line is:

<sha256> <absolute-path>

This gives 65,536 buckets and keeps lookups cheap: resolving one hash reads one small bucket file. Downloaded or uploaded blob content is stored beside the bucket using the full hash:

<root>/<first2>/<next2>/<full-sha256>

Those names cannot collide because bucket files are 4 hex chars and blobs are 64 hex chars. Access metadata for quota cleanup lives under <root>/meta/.

CLI

Inspect defaults:

./shaser config
./shaser config -json

Compute a SHA256:

./shaser hash /path/to/file

Register files:

./shaser cache register -root ./samples /path/to/file
./shaser cache scan -root ./samples /path/to/tree

Resolve and list:

./shaser cache lookup -root ./samples <sha256>
./shaser cache list -root ./samples
./shaser cache stats -root ./samples -large 104857600 -older 720h

Remove entries:

./shaser cache remove -root ./samples <sha256>
./shaser cache remove-path -root ./samples /path/prefix

Configure peers:

./shaser peer add http://127.0.0.1:8080
./shaser peer add -token READ_TOKEN https://peer.example
./shaser peer list

Download from local cache or peers. Successful downloads are registered into the local cache and the command prints the local absolute path:

./shaser cache get -root ./samples <sha256>

Server

Run the server:

./shaser server -root ./samples -listen 127.0.0.1:8080

Endpoints:

GET  /
GET  /?sha256=<sha256>
GET  /sha256/<sha256>
HEAD /sha256/<sha256>
GET  /<first2>/<next2>/<full-sha256>
POST /sha256
PUT  /sha256/<sha256>

The root page serves a small HTML form for pasting a SHA256. Submitting it checks the local cache and configured peers, redirects to the download when the hash is available, or shows an error page when it is not.

Downloads are always sent with Content-Encoding: gzip. Direct hash paths make it possible to expose the cache root with a generic static HTTP server too.

Useful server flags:

-peers FILE          JSON peer list
-auth-file FILE      JSON auth config
-auth                require read bearer tokens
-read-token TOKEN    comma-separated read tokens
-upload-token TOKEN  comma-separated upload tokens
-bandwidth BYTES     per-response bandwidth limit
-tls-cert FILE       enable HTTPS with this cert
-tls-key FILE        HTTPS key

Reads are public unless read auth is enabled. Uploads always require an upload token.

Auth

Generate and manage bearer tokens:

./shaser auth gen -kind upload -name alice
./shaser auth add -kind read -name bob READ_TOKEN
./shaser auth add -kind upload -name ci UPLOAD_TOKEN
./shaser auth require-read on
./shaser auth list

Named tokens are stored with their creation time, so auth list shows who each token was issued for and when it was added.

Upload to a peer:

./shaser upload -peer http://127.0.0.1:8080 -token UPLOAD_TOKEN /path/to/file

C API

The C API is opt-in and uses radare2's RMuta for local SHA256 computation. Normal Go builds do not require radare2 headers or libraries.

Build a shared library:

go build -tags capi -buildmode=c-shared -o libshaser.dylib ./capi

On Linux, use libshaser.so as the output name. The build expects pkg-config --cflags --libs r_muta to work.

Public functions are declared in capi/shaser.h:

char *shaser_sha256_file(const char *path);
char *shaser_register_file(const char *root, const char *path);
char *shaser_lookup(const char *root, const char *sha256);
char *shaser_get(const char *root, const char *peer_file, const char *sha256);
char *shaser_upload_file(const char *base_url, const char *token, const char *path);
char *shaser_last_error(void);
void shaser_free(char *s);

All returned strings must be released with shaser_free(). NULL means failure; call shaser_last_error() for a diagnostic string.

Environment

SHASER_ROOT    default cache root
SHASER_CONFIG  default config JSON file
SHASER_PEERS   default peer JSON file
SHASER_AUTH    default auth JSON file

Verification

go test ./...
go vet ./...
go test -tags capi ./capi
go build -tags capi -buildmode=c-shared -o /tmp/libshaser.dylib ./capi