README
¶
cas
A content-addressable file cache with download verification.
Files are downloaded from pluggable remote sources, stored locally by their SHA-256 content hash, and verified before being committed to the cache. The cache is safe for concurrent use — simultaneous requests for the same checksum share a single download, and callers can cancel via context without affecting in-progress downloads.
Requires Go 1.25 or later.
Installation
go get github.com/acycl/cas
Source packages are installed separately to avoid pulling in unnecessary dependencies:
go get github.com/acycl/cas/gcs
go get github.com/acycl/cas/s3
The HTTPS source lives in the root module and requires no additional dependencies:
import "github.com/acycl/cas/https"
Usage
GCS
client, _ := storage.NewClient(ctx)
d, _ := transfermanager.NewDownloader(client)
src := gcs.NewSource(d)
cache := cas.New("/var/cache/files", src)
m, _ := cas.NewManifest(
cas.File("file.txt", "gs://bucket/file.txt", "ab12cd34..."),
)
f, _ := cache.Open(ctx, m, "file.txt")
defer f.Close()
data, _ := io.ReadAll(f)
S3
cfg, _ := config.LoadDefaultConfig(ctx)
client := awss3.NewFromConfig(cfg)
d := manager.NewDownloader(client)
src := s3.NewSource(d)
cache := cas.New("/var/cache/files", src)
m, _ := cas.NewManifest(
cas.File("file.txt", "s3://bucket/file.txt", "ab12cd34..."),
)
f, _ := cache.Open(ctx, m, "file.txt")
defer f.Close()
data, _ := io.ReadAll(f)
HTTPS
src := https.NewSource()
cache := cas.New("/var/cache/files", src)
m, _ := cas.NewManifest(
cas.File("file.txt", "https://example.com/file.txt", "ab12cd34..."),
)
f, _ := cache.Open(ctx, m, "file.txt")
defer f.Close()
data, _ := io.ReadAll(f)
Manifest
A Manifest is a standalone type that maps names to remote files. It has no
dependency on a Cache, so it can be defined at package scope, loaded from
configuration, or shared across cache instances:
m, _ := cas.NewManifest(
cas.File("model.bin", "gs://bucket/model.bin", "ab12cd34..."),
cas.File("config.json", "s3://bucket/config.json", "ef56ab78..."),
)
f, _ := cache.Open(ctx, m, "model.bin")
defer f.Close()
Use Validate to check at startup that all manifest URIs have registered
sources, without downloading anything:
if err := cache.Validate(m); err != nil {
log.Fatal(err)
}
Custom sources
Implement the Source interface to add support for any protocol:
type Source interface {
Scheme() string
Download(ctx context.Context, dst *os.File, u *url.URL) error
}
Pass custom sources directly to New:
cache := cas.New("/var/cache/files", mySource)
Documentation
¶
Overview ¶
Package cas provides a content-addressable cache for remote files.
Files are downloaded from remote sources, stored locally by their content hash, and verified during download. The cache ensures data integrity by validating SHA-256 checksums before committing files to the cache.
Example usage:
client, _ := storage.NewClient(ctx)
d, _ := transfermanager.NewDownloader(client)
src := gcs.NewSource(d)
cache := cas.New("/var/cache/files", src)
m, _ := cas.NewManifest(
cas.File("file.txt", "gs://bucket/file.txt", "sha256hash..."),
)
f, _ := cache.Open(ctx, m, "file.txt")
defer f.Close()
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrUnsupportedScheme = errors.New("unsupported URI scheme")
ErrUnsupportedScheme is returned when a URI's scheme has no registered source.
Functions ¶
This section is empty.
Types ¶
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache manages a local cache of remote files, indexed by content hash. Use Cache.Open to retrieve files by manifest name, and Cache.Validate to check that all manifest URIs have registered sources. It is safe for concurrent use. Concurrent requests for the same checksum share a single download, and callers can cancel their wait via context without affecting the in-progress download.
func New ¶
New creates a new Cache that stores files in the specified directory. Cache subdirectories are created as needed when files are downloaded. Each source is registered under the URI scheme returned by its Scheme method. If multiple sources share a scheme, the last one wins.
type Entry ¶
type Entry struct {
// contains filtered or unexported fields
}
Entry describes a remote file by its name, URI, and expected content hash. The checksum is validated when the entry is added to a Manifest.
type ErrInvalidChecksum ¶
type ErrInvalidChecksum struct {
Expected, Actual string
}
ErrInvalidChecksum is returned when a file's content does not match its expected checksum.
func (*ErrInvalidChecksum) Error ¶
func (e *ErrInvalidChecksum) Error() string
type Manifest ¶
type Manifest struct {
// contains filtered or unexported fields
}
Manifest maps logical file names to remote entries. It is a pure data type with no dependency on a Cache, so it can be defined at package scope, loaded from configuration, or shared across cache instances.
type Source ¶
type Source interface {
// Scheme returns the URI scheme this source handles (e.g., "gs", "s3", "https").
Scheme() string
// Download writes the contents of the remote file at u to dst.
Download(ctx context.Context, dst *os.File, u *url.URL) error
}
Source downloads files from a remote location.
Source Files
Directories