Support the VFS API

[zombiezen]

A few folks on the Gophers Slack requested support for the SQLite VFS API.

Use cases:

• Fetching a database lazily over HTTP
• more...?

I somehow missed that modernc.org/sqlite/vfs is now a thing.

[zombiezen]

Wanted to give an update: I am slowly making some progress on this feature. I've made some decent progress on sketching out the API and a proof-of-concept implementation, but I don't have something that's ready for testing yet.

The challenge I've identified with this API is that unlike other SQLite extension APIs, there (understandably) seems to be a strong use case for wrapping other VFSes, including those that are implemented in (transpiled) C. This has led me to a somewhat split API where a Go implementation gets converted into a VFS object during registration.

// VFSImplementation is the set of parameters to [NewVFS]
// that enables creation of custom [VFS] objects.
type VFSImplementation struct {
	Name          string
	MaxPathLength int

	Open         func(name string, open VFSOpenFlags) (File, VFSOpenFlags, error)
	FullPathname func(name string) (string, error)
	ReadRand     func(p []byte) (n int, err error)
	Sleep        func(d time.Duration) error
	Now          func() (time.Time, error)
        // etc.
}

// File is the interface for files in [VFSImplementation].
type File interface {
	io.ReaderAt
	Size() (int64, error)
	io.Closer
}

// A VFS is a handle to an [operating system interface].
//
// [operating system interface]: https://www.sqlite.org/vfs.html
type VFS struct { /* ... */ }

// NewVFS creates and registers a [VFS] from a Go implementation.
func NewVFS(impl *VFSImplementation) (*VFS, error)

// FindVFS returns the VFS with the given name
// or the default VFS if the name is the empty string.
// If no such VFS exists, FindVFS returns nil.
func FindVFS(name string) *VFS

// Name returns the name of the VFS.
func (vfs *VFS) Name() string

// RegisterDefault marks the VFS as the one used by default in [OpenConn].
func (vfs *VFS) RegisterDefault() error

// Unregister removes the VFS from the global registry.
// Once a VFS has been unregistered, it must no longer be used.
func (vfs *VFS) Unregister() error

// Open opens the file with the given name.
// TODO: Add details about calling FullPathname etc.
func (vfs *VFS) Open(name string, open VFSOpenFlags) (*VFSFile, VFSOpenFlags, error)

func (vfs *VFS) FullPathname(name string) (string, error)

// and so on for VFS methods...

// VFSFile is a file returned by [VFS.Open].
type VFSFile struct { /* ... */ }

// VFSFile will have various io-related methods.

[glerchundi]

Cannot wait to see what you come up with!

Another use case is databases stored in cloud storages like s3, gcs, azure.

[atombender]

Very excited to hear about this. How's the work going?

I somehow missed that modernc.org/sqlite/vfs is now a thing.

That VFS implementation is read-only, sadly!

[zombiezen]

Not much progress since last update. It's still on my radar, but lower priority than client work at the moment, I'm afriad. If there's a business can benefit from this, I'm happy to set up a short-term contract to finish it up.

[delaneyj]

It's been a year, not sure if this helps motivate at all... https://fly.io/blog/litestream-vfs/

[alvarolm]

It would be nice to do it like https://github.com/ncruces/go-sqlite3/tree/main/vfs does
that way, encryption will be easy to implement