Question about importing/exporting timeline data

[h-wb]

Hi there,

First off, really appreciate all the work you’ve been putting into this great software! I’ve been using Arc Timeline for a while now (and more recently, the new version currently in testflight, Arc Editor) and I’m looking to move to a more self-hosted setup (ideally using Overland for tracking since it also uses iOS CoreLocation) and I wanted to get your thoughts or advice on how best to approach that.

I’ve liked the way Arc represents its timeline and from what I can tell, Reitti offers a similar kind of experience, which is great.

IIRC Arc Timeline used to be able to export GPX, or GeoJSON but Arc Editor right now only dumps its data as JSON. There's items, samples and places.

I don't think it's really important to detail the structure but it might potentially interest you. An item is a timeline object and it can either be:

A trip

  {
    "trip" : {
      "classifiedActivityType" : 20,
      "distance" : 257301.93218154195,
      "confirmedActivityType" : 20,
      "speed" : 0,
      "lastSaved" : 773240026.037,
      "uncertainActivityType" : false,
      "itemId" : "EE157C78-EFFB-47C5-A1E3-CADEA5349827"
    },
    "base" : {
      "disabled" : false,
      "activeEnergyBurned" : 79.38600000000002,
      "averageHeartRate" : 66.44661701086795,
      "deleted" : false,
      "stepCount" : 1265,
      "endDate" : 727592241.943,
      "maxHeartRate" : 81,
      "source" : "LocoKit",
      "lastSaved" : 773240038,
      "nextItemId" : "11B912CE-4DFD-4751-835C-033DEDB904F6",
      "isVisit" : false,
      "startDate" : 727582202,
      "id" : "EE157C78-EFFB-47C5-A1E3-CADEA5349827",
      "sourceVersion" : "9.0.0",
      "samplesChanged" : true
    }
  }

A visit

  {
    "visit" : {
      "streetAddress" : "[REDACTED]",
      "radiusMean" : 50,
      "itemId" : "11B912CE-4DFD-4751-835C-033DEDB904F6",
      "confirmedPlace" : true,
      "lastSaved" : 773240026.038,
      "placeId" : "323FCBA3-D9D1-4437-92DB-587BF25D7E24",
      "radiusSD" : 10,
      "latitude" : 0,
      "longitude" : 0,
      "uncertainPlace" : false
    },
    "base" : {
      "disabled" : false,
      "nextItemId" : "FFC5A602-F37B-4781-9940-A81AF959888E",
      "id" : "11B912CE-4DFD-4751-835C-033DEDB904F6",
      "lastSaved" : 773240038,
      "deleted" : false,
      "samplesChanged" : true,
      "source" : "LocoKit",
      "activeEnergyBurned" : 50.747000000000014,
      "stepCount" : 897,
      "sourceVersion" : "9.0.0",
      "startDate" : 727592248.436,
      "averageHeartRate" : 101,
      "maxHeartRate" : 101,
      "previousItemId" : "EE157C78-EFFB-47C5-A1E3-CADEA5349827",
      "endDate" : 727592657.55,
      "isVisit" : true
    }
  }

And as you can see a visit can link to a place which contains address and geolocation (and reverse geocoding) details.

A place

  {
    "mapboxMakiIcon" : "cafe",
    "visitCount" : 9,
    "countryCode" : "[REDACTED]",
    "visitDays" : 9,
    "streetAddress" : "[REDACTED]",
    "isStale" : false,
    "mapboxPlaceId" : "[REDACTED]",
    "radiusSD" : [REDACTED]
    "mapboxCategory" : "cafe, coffee, tea, tea house",
    "name" : "[REDACTED]",
    "latitude" : [REDACTED],
    "locality" : "[REDACTED]",
    "longitude" : [REDACTED]
    "rtreeId" : 5,
    "id" : "0BCE8A06-8889-46C4-9CEB-1434DD6470F1",
    "radiusMean" : [REDACTED],
    "secondsFromGMT" : 0,
    "source" : "LocoKit",
    "lastSaved" : 773258264
  }

What would be the best way to bring it into Reitti without having it redo reverse geocoding or re-infer the trip structure? TBH i’m not super familiar with all the data formats yet, so I’m wondering what you’d recommend:

• Should I convert the Arc JSON to GeoJSON or a Google Timeline format?
• Or would it make more sense to write a small import script directly against Reitti’s database or API?
• Just actually code the import into the codebase (it's quite a niche app thought)

I understand your philosophy for Reitti is something "that requires as little manual intervention as possible" so maybe adding data manually is not a recommended approach at all but I'd like to hear your thoughts.

On a related note, I saw in this discussion that GeoJSON export is not gonna happen. Out of curiosity, what’s the best way to get data out of Reitti: a simple database dump ? I was wondering if you’d ever consider supporting JSON export, or if it’s something users are better off doing directly from their own DB instance. I’m mostly thinking out loud here. As I experiment with different tools, I’m wondering If I can have some flexibility to move data between them and keep the timeline I’ve built over the years if that makes sense.

Thanks again for all your work on this project!

[dgraf-gh]

Hello @h-wb,

Thanks a lot for the detailed discussion, that will help us find the right solution. I hadn’t heard of Arc Timeline before, but it looks very interesting. I’ll address your points one by one.

What would be the best way to bring it into Reitti without having it redo reverse geocoding or re-infer the trip structure? TBH I’m not super familiar with all the data formats yet, so I’m wondering what you’d recommend:

I don’t think there’s a feasible way to avoid redoing much of the processing without manual intervention. The easiest approach is to export GeoJSON or GPX from Arc and import that into Reitti. Reitti will then analyze the data and will probably infer Trips and Visits similarly to Arc. The results may differ slightly, but should be in the same range.

Geocoding must be done by Reitti, there’s currently no supported way to import already-geocoded places. If you use the provided Photon instance, geocoding is quite fast, but you will lose manual adjustments unless you preserve them.

Since #412, there is a way to preserve your Arc adjustments:

• If you can export your places (take all of them), you can insert them into the significant_places_overrides table. When Reitti needs to geocode a new significant place, it will override geocoder results with data from that table. Reitti searches the geom column within a 5 m radius. Example insert:

INSERT INTO significant_places_overrides(user_id, geom, name, category, timezone)
VALUES (1, 'POINT(10.69801481 53.86143331)', 'TEST NAME', 'OTHER', 'Europe/Berlin');

• If you have categorized trip transport modes in Arc, you can add overrides to the transport_mode_overrides table. Put an entry with the chosen transport mode and a timestamp roughly in the middle of the trip. Reitti will consult this table when creating trips and apply the override if it finds one.

After inserting your manual adjustments, you’ll need to convert the Arc export to GeoJSON or GPX. We mainly need a list of $(lat, lon, timestamp)$ points. Arc seems to behave similarly to Google Timeline, storing visit start/end and trip paths. In our BaseGoogleTimelineImporter we handle this by generating synthetic geopoints during visits every $x$ seconds and keeping trip paths as-is; a similar approach should work for Arc exports.

TL;DR

• Import all your manual edits into Reitti (significant_places_overrides, transport_mode_overrides).
• Convert the Arc export to GeoJSON or GPX by expanding the Visits with synthetic points (we mainly need $(lat, lon, timestamp)$ points).
• Import the converted file into Reitti.
• Reitti will then calculate Visits and Trips on the imported data using your overrides

On a related note, I saw in #422 that GeoJSON export is not gonna happen. Out of curiosity, what’s the best way to get data out of Reitti: a simple database dump ? I was wondering if you’d ever consider supporting JSON export, or if it’s something users are better off doing directly from their own DB instance. I’m mostly thinking out loud here. As I experiment with different tools, I’m wondering if I can have some flexibility to move data between them and keep the timeline I’ve built over the years if that makes sense.

We currently don’t have a full GeoJSON exporter because Reitti doesn’t retain all raw points and their data (we only recently started storing altitude). I didn’t add a GeoJSON exporter because I don’t want to encourage treating Reitti as permanent storage for raw geo data. Reitti can reconstruct the significant paths needed for calculations, but it drops a lot of raw data.

GPX is limited but consistent in what it stores; GeoJSON is more flexible. If you want all of Reitti’s geo data, it’s available in the database tables:

• raw_location_points - stores incoming raw location points
• processed_visits - calculated visits for the user
• trips - trips between visits

Right now, a simple DB dump or custom queries against those tables is the best way to get everything out. I’m open to adding API endpoints or exporters, but I prefer to implement them only when there’s a concrete use case.

If you want, I can help sketch a converter for Arc exports to GeoJSON/GPX or help map Arc fields to the override tables.

[h-wb]

• Import all your manual edits into Reitti (significant_places_overrides, transport_mode_overrides).
• Convert the Arc export to GeoJSON or GPX by expanding the Visits with synthetic points (we mainly need ( l a t , l o n , t i m e s t a m p ) points).
• Import the converted file into Reitti.
• Reitti will then calculate Visits and Trips on the imported data using your overrides

That's very helpful thank you! I'm gonna try with a month worth of data and compare the 2 timelines, see what it looks like!
Do you think there's any performance concerns with having a ton lot of places overrides or should it work just fine ?

Right now, a simple DB dump or custom queries against those tables is the best way to get everything out. I’m open to adding API endpoints or exporters, but I prefer to implement them only when there’s a concrete use case.

That's totally fair, I can't tell you a concrete use case either expect it being a nice QoL thing I guess lol. Custom queries will work good enough for what I want to do!

If you want, I can help sketch a converter for Arc exports to GeoJSON/GPX or help map Arc fields to the override tables.

That's very nice of you thank you. I wouldn't mind if I struggle to do it but no need to bother yet thought aha, I'll play around a bit first!

I'll come back and publish the results here when I get the chance!

[dgraf-gh]

Your welcome, and good luck with it. Let me know if you have any further questions.

Do you think there's any performance concerns with having a ton lot of places overrides or should it work just fine ?

Let's see ;) At the moment there is no index on the user, geom columns. I did not think there will be thousand of places in there. If it is getting slow, we will sort this out when the time comes.