Share the test data as CSV, one directory per dataset

[phdoerfler]

What

This PR rewrites the test data .sql files into .csv and adds build-time code which recreates the previous .sql files customised for all three flavours of SQL currently supported by Grackle. It also groups these data files into named datasets. Previously, the dialect directories contained a flat list of .sql files. They still end up getting loaded into the same database. But now they are organised in a way that should make it easier to find them.

Why

Ever since Grackle started supporting more than one database, there were three almost identical copies of .sql files in the repository. With more tests and especially even more backends on the horizon, these have become unwieldy and redundant.

Layout

A dataset is now one directory under testdata:

testdata/world/
├── pg.sql               schema only
├── oracle.sql
├── mssql.sql
├── city.csv             the rows, shared by all three
├── country.csv
└── countrylanguage.csv

The remaining .sql files

For now, the actual CREATE TABLE parts are left as is. They are specific to the dialect. The data, however, not so much.

The directory structure

Grouping by dataset rather than by dialect puts both halves of one fixture in one place.
At container-up time the schema and the rows are written together into target/testdata/<dialect>/<dataset>.sql, which is what docker compose mounts into the container's init directory.

The CSV format

It is inspired by postgres' COPY ... FROM STDIN blocks. Hence \N for NULL, for instance, and | as the separator. Almost all the .sql files have been converted into equivalent .csv. Two were not: mutation, which only creates a sequence, and qualified-names, which is Postgres only.

Values the dialects spell differently

Most values are written as a string literal and handed to the database verbatim. The exceptions are arrays, dates, times, timestamps and booleans. For those, a column names its kind in the CSV header and each dialect renders it accordingly:

kind	in the CSV	pg	oracle	mssql
array	drama,comedy	'{"drama","comedy"}'	string_array2('drama', 'comedy')	'["drama", "comedy"]'
date	1974-10-07	'1974-10-07'	DATE '1974-10-07'	'1974-10-07'
time	19:35:00	'19:35:00'	INTERVAL '0 19:35:00' DAY TO SECOND (0)	'19:35:00'
timestamptz	2020-05-22T19:35:00Z	as written	TIMESTAMP '2020-05-22 19:35:00 +00:00'	'2020-05-22 19:35:00 +00:00'
boolean	true	'TRUE'	'TRUE'	1

Oracle is a bit extra and builds an array by calling its collection type, so the constructor name is read out of the column's type in Oracle's own schema. Note: The CSV is completely dialect-agnostic.

Two data changes

Resolved in favour of what two of the three dialects already did:

• a city district that was empty on Postgres is now '-', as in the other two, because Oracle reads the empty string as NULL and the column is NOT NULL
• countrylanguage.isofficial is written 1/0, which all three accept

What did not change

No test code was changed, and no data except for the two values above.

The statements are shaped differently, since they are generated now: Postgres gets INSERTs where it had COPY ... FROM STDIN blocks, and Oracle gets one multi-row VALUES where some scripts had a statement per row. The values are the same, 500 rows to a statement.

Also here

• sbt "newDataset foo" writes the three schema skeletons so a new dataset starts in the right shape
• sbt checkTestData renders every dataset in memory and reports what does not line up: a CSV whose table no schema creates, a header naming an unknown kind, a ragged row, a dataset with no scripts. It needs no database, so CI runs it
• CI now builds the scripts before starting the containers. It runs docker compose directly, and a bind mount whose source is missing gets created by the daemon as root, which sbt then cannot write into

Checking it

sbt genTestData writes the scripts without starting anything, so target/testdata can be diffed against a previous run. Every commit here was checked that way.

[milessabin]

This is great stuff! LGTM!