Exporting SQLite to Git

Theoretically it's as simple as:

fossil export --git | git fast-import

Unfortunately it doesn't work that way and that's what this project is all about.

Quick Start

1. Run the script build to fetch and build a suitable fossil tool and a git-export-filter tool.

2. Run the script import to create an sqlite.git Git clone of the http://sqlite.org/src fossil sources. (May take up to 30 minutes.)

3. See the "Building" entry at the bottom of this README.

Fossil Issues

There are two problems with fossil export:

1. fossil versions after 1.18 produce a Git fast-import data stream that causes git fast-import to fail with a fatal error.

2. fossil versions starting with 1.18 mangle export branch and tag names to avoid including characters git does not allow. The problem is that many more characters are mangled than needed so that a tag like version-1.18 is converted to version_1_18 unnecessarily.

The Export Problem

The fossil change that introduces the export problem is here:

http://www.fossil-scm.org/index.html/info/bc8d368b66

There is even a ticket about this export issue here:

http://www.fossil-scm.org/index.html/info/4013b0a81a

This issue affects the sqlite, sqlite_docsrc and fossil repositories making it impossible to export them from fossil and import them into Git with a current version of fossil.

The Tag Problem

The fossil change that introduces tag mangling is here:

http://www.fossil-scm.org/index.html/info/b707622f29

It was a well-intentioned change as previously invalid Git names would be exported, but it went way, way too far. In fact, the actual Git rules about allowable characters in names are:

1. Characters with an ASCII value less than or equal to 32 are not allowed
2. The character with an ASCII value of 0x7F is not allowed
3. The characters '~', '^', ':', '', '*', '?', and '[' are not allowed
4. The character '/' is a separator and is not allowed within a name
5. The name may not start with '.' or end with '.'
6. The name may not end with '.lock'
7. The name may not contain the '..' sequence
8. The name may not contain the '@{' sequence
9. If multiple components are used (separated by '/'), no empty '' components

A patch is included in the file patches/export_c_patch_diff.txt that allows the full diversity of git names to be used and should be applied to the fossil src/export.c file of fossil version 1.18 before building fossil. It also adds an optional --notes option to the fossil export --git command that if given will add a note in the refs/notes/fossil namespace to each commit giving the original fossil check-in hash for the commit.

A .tar.gz archive of the fossil 1.18 sources may be fetched from:

http://fossil-scm.org/index.html/tarball/fossil-v1.18.tar.gz?uuid=df9da91ba8

The downloaded .tar.gz file should have these hash values:

md5: 8f4eca8f7c0fc4a3d7566c54462fb725
sha1: 449414cb6e6cff15ce670b32cddc34bb8857d160

or these hash values:

md5: e6fa34c8803f1a1ce22b8a89224495bf
sha1: a5c8ce53c8d4c5d0188b095e97fee8e8bded0777

The difference being that at some point archives generated by the above link started adding an extra line to the manifest file included in the archive:

--- fossil-v1.18-old/manifest   2011-07-13 16:03:41.000000000 -0700
+++ fossil-v1.18-new/manifest   2011-07-13 16:03:41.000000000 -0700
@@ -325,3 +325,4 @@
 R 675d9f02faa4c8ad97474da91bcc8b39
 U drh
 Z 04f0f6f457863c33e57b89a526d699e9
+# Remove this line to create a well-formed manifest.

The original version (without the extra line) is included in the archives subdirectory for convenience and will be automatically used by the build script.

Git Fast Import Issues

The Git fast-import facility does not provide a means to filter the incoming data stream to adjust user names (fossil export data only includes the user login name as the email address) nor a means to adjust branch/tag names (fossil exports a 'trunk' branch where Git expects a 'master' branch and fossil also exports what are essentially lightweight tags as annotated tags).

To deal with these issues, the git-export-filter utility is used.

It can be found at:

http://repo.or.cz/w/git-export-filter.git

The included sqlite_authors file is used with the git-export-filter tool to supply real user names and email addresses. Also note that the sqlite_authors file also works for the http://sqlite.org/docsrc fossil repository as well.

The Final Solution

After building a patched version of fossil 1.18 as described above and the git-export-filter utility, a Git repository of the SQLite sources can be created like so (which is what the import script does):

fossil clone http://sqlite.org/src sqlite.fsl
git --git-dir=sqlite.git init --bare
fossil export --git sqlite.fsl | \
git-export-filter --authors-file sqlite_authors --require-authors \
  --trunk-is-master --convert-tagger tagger | \
git --git-dir=sqlite.git fast-import

The above will create the sqlite.git Git repository that is a clone of the SQLite sources from the SQLite fossil respository http://sqlite.org/src (note that only sources are cloned, not tickets or wiki pages or events).

The provided build script will attempt to download the necessary sources, patch them and build suitable fossil and git-export-filter executable files.

The provided import script will then attempt to clone the SQLite sources and convert them into an sqlite.git repository. It may be run again to update the sqlite.git repository with new changes.

The initial run of the import script may take up to 30 minutes on a fast machine, but subsequent runs of import on a fast machine will take more like 5 minutes but the CPU will be pounded in either case.

Building

Ideally, simply cloning from the new sqlite.git repository would allow one to then build SQLite by simply using make (or configure and make).

Unfortunately, this is not the case, the make will fail with a message about no rule to make the files manifest and/or manifest.uuid.

Both the SQLite sources and the Fossil sources require two fossil vcs specific files to be created (manifest and manifest.uuid) in order for make to be successful.

The manifest.uuid file simply contains the SHA-1 hash of the current checkout and while a real manifest file contains a bunch of information, the only thing that need be present is a line containing the UTC ISO date preceded by 'D '.

The create-fossil-manifest script takes care of creating these files and should be run with the current working directory set to the top-level of the git clone's working directory.

Any time the HEAD commit changes, the create-fossil-manifest script should be run to update the manifest and manifest.uuid files before next running make or the output of the sqlite_source_id() function will be incorrect.