Add a gpx_tracks table and convert trace points into linestrings

[Rub21]

Ref: #7339

This PR adds a gpx_tracks table that stores the geometry of each trace. We use geometry(GeometryZM, 4326), where Z is the altitude and M is the timestamp.

The table is filled in two ways:

• A rake task (rake db:gpx_tracks) converts the traces that already exist. It runs on its own and has to be started by hand. It skips the traces that already have rows, and a trace is written completely or not at all, so the task can stop and start again without leaving half traces.

• When a user uploads a new trace during the migration, a job converts it and saves it in gpx_tracks too, so new traces do not need the migration.

Nothing reads this table yet. The job only writes to it, so the API, the trace pages and the tiles work the same as before. The goal is to have all gps_points as linestrings in gpx_tracks, so the migration away from gps_points can later be done on top of this table.

What this PR does:

• New gpx_tracks table with a primary key of (gpx_id, trackid, segment), a geometry(GeometryZM, 4326) column, a GiST index on it, and a foreign key to gpx_files.
• A check constraint only allows ST_LineString and ST_Point, so nothing else can end up in the table.
• TraceLinestringJob runs after a successful import and does the work in a single INSERT ... SELECT, on the traces queue.
• Segments are cut every 500 points (max_points_per_track_segment in settings), so a long track becomes several rows instead of one very large geometry. 500 is a starting value, easy to change if the team prefers another number.
• Segments do not repeat the border point. To draw a full track, join them in order by trackid and segment.
• A track with a single point is saved as a point, because a line needs two.
• Points without a timestamp are skipped, because M cannot be null.
• Traces of any visibility are converted.
• The job is safe to run again for the same trace (a re-import or a retry): it replaces the rows of that trace in one transaction, so the result is always clean.
• Altitude uses 0 when the point has none.

[Rub21]

I wrote a bash script to convert the old traces: it runs rake db:gpx_tracks in parallel, passing a range of trace ids to each process.
https://gist.github.com/Rub21/5376e14fc7597af471945d7d6b8ddd6f

It splits the traces into ranges of ids and runs a few rake processes at the same time. It shows the plan and asks for a yes before it starts. status reads the logs and shows the percentage, the segments written and the errors. Traces that already have rows are skipped.

Note: the bash script is only a proposal to run the conversion, if there is another way you normally do migrations, I can take a look at it.

[pablobm]

Are we ok with losing the points without timestamp? Has this been discussed?

Incidentally, I realised that the Gpx module doesn't have tests, and we don't have any fixtures for files with more than one <trkpt>. Not for this PR, but we should add something.

[tomhughes]

Are we ok with losing the points without timestamp? Has this been discussed?

We don't allow points without timestamps, and never have done. They fail the validity check at:

openstreetmap-website/lib/gpx.rb

Lines 175 to 179 in 26ba84e

	def valid?
	latitude && longitude && timestamp &&
	latitude >= -90 && latitude <= 90 &&
	longitude >= -180 && longitude <= 180
	end

and also the model validations:

openstreetmap-website/app/models/trace.rb

Line 54 in 26ba84e

validates :timestamp, :presence => true

[pablobm]

We don't allow points without timestamps, and never have done.

Is that the case even with primeval points? The ancient scriptures quoth the addition of validation (parser and model), so I wonder if there were exceptions before that.

[tomhughes]

I believe so because it was always a point of contention with some people wanting to remove them before uploading for "privacy" reasons.