Synchronize Aria glasses with timecode generator

[jyuntins]

Hi! Thank you for building this wonderful tool!

I’m trying to synchronize the Aria glasses with other cameras using a timecode generator (Tentacle Sync).
For regular cameras, I can record LTC audio directly since they have audio input ports, but the Aria glasses do not.
I understand that two Aria glasses can be synchronized using the TICSync method over a network with sub-millisecond accuracy.
I’m wondering if it’s possible to connect the timecode generator to a computer and somehow embed that timecode into the master Aria device?
Alternatively, would it be feasible to record the LTC audio separately on my computer and align the Aria recordings with it afterward?

Thanks for reading my questions!

[jyuntins]

https://facebookresearch.github.io/projectaria_tools/docs/data_formats/aria_vrs/timestamps_in_aria_vrs

I think I will need to leverage the TIME_CODE domain, but not sure what is the correct way to embed it.

[Tarunstk]

Hi @jyuntins I believe yes TIME_CODE would be the best option for your use case.

I think a easier method for registering a start of the recording could be a sharp audio event at the start and end of the recording, something like a clap or loud click when (or after) you start the recording for all the devices. This way you will have a "event" with which you can align all the audio files and the corresponding video files as well.

Once this has been registered you can pull all the audio and video files and start aligning it. The clap method is analog and simple but we have found it to be effective in time alignment. This does however need some post processing and analysis after the collection is complete.

Please let me know if this helps! Thank you.

[jyuntins]

Hi @Tarunstk, thank you so much for your reply!

Yes, I’ve previously tried to synchronize aria with other cameras using audio signal. I have also tried to use visual events to synchronize them. While both audio and visual event can provide me with pretty good alignment (frame-level accuracy), it’s still insufficient for my use case. I wish to achieve sub-frame accuracy across each camera. Furthermore, I observed noticeable drift across cameras after a few minutes of capture, which comes from each camera's independent internal clock, and I think this is impossible to solve with event-based synchronization.

I think embedding the external clock timestamp into aria's recording is the best option for me. Would you mind providing some instructions on how to do this? Below I also attach what I plan to do to solve this problem, please let me know whether you think this is feasible or not:

1. Forming a mapping between my laptop's local clock and the tentacle sync's LTC timecode.
2. Having my laptop act as a server, and let Aria store my laptop's local clock into its TIMECODE domain.
3. After recording, extracting the TIMECODE domain from vrs file and use the mapping to convert each aria device timestamp -> local server timestamp -> tentacle sync timecode.

Thank you for your time!

[nicholasstrickland]

Hello @jyuntins our team is continuing to investigate your request.

[jyuntins]

@nicholasstrickland

That sounds good! Thank you for letting me know!

[Tarunstk]

Hi @jyuntins your approach does work however this may require a lot of groundwork to setup and the possible point where you may have issues is at the second step as there is no straightforward way to embed this into the Aria recordings at the moment.

I researching internally and am getting some clarity on whether there is an easy or straightforward way (or if at all) to achieve this and if so a potential solution for it. Thank you for your patience while I am working to get some more clarity on this.

[jyuntins]

Hi @Tarunstk,

Got it! Thanks for working on this!
Please let me know If there is anything I can do/try on my end.

Also, If directly ingesting TIMECODE into Aria recordings is troublesome, maybe I can try the same strategy mentioned in the documentation here? Instead of embedding the timecode into the recording, it forms a mapping between the server's device time and aria's device time. In this way, the timecode can be stored on the server. I can then utilize this mapping to sync each aria image with the timecode generator.

[reggevlivney]

Hi, joining in on this issue. I am trying to get the unix timestamp for each frame (in the wifi network time) in the vrs file. The vrs file has a timestamp vector, with values relative to (apparently) the uptime of the aria glasses. I want to translate this time to the network unix time. I saw that the metadata json has a "start_time" field, but it is only in second level precision, and I am looking for at least frame-level precision.
Another workaround I tried is querying the Aria glasses uptime and unix time using adb shell commands, but this was not always accurate.

I looked through other solved issues here and I couldn't find something suitable for my application.

Thank you very much

[Tarunstk]

Hi @jyuntins @reggevlivney after some discussions and research on my end I found a few methods that might help you out. If you haven't tried the QR code timestamp method you could probably try using that. The method in itself is somewhat straightforward.

You can synchronise streams by displaying a QR code that encodes the current time which updates in real time on a screen which is visible to all the cameras.

Each device records the QR code during the session and this encodes the timestamp within the recording. You can develop a script in post processing that detects the QR code which helps compute the time offset for each stream and could be used for accurate alignment.

For QR code timestamp generation you could use an online tool like - XOS Digital TIme Generator or any other tool.

One thing to be vary off is the refresh rate sync between the camera shutter and the screen displaying the QR code. If you are capturing long recordings then you can just show the QR code after an interval of say ~30/45 minutes for better post processing alignment.

Let me know if this solution is feasible for your use case.

[reggevlivney]

This is an interesting workaround, though a bit hacky. I'll give it a try when I get the chance.
If there are any simpler solutions it would be better - even the ability to just query the internal clock would be great, since I can match it to a timestamp at the moment of querying.
Thank you for your assistance

[Tarunstk]

@reggevlivney yes this definitely is a hacky workaround. I am checking for a simpler solution. Let me get back to you.

[jyuntins]

Hi @Tarunstk,

I just want to follow up on this and describe my attempts so far.

The QR code approach does not work well for my case, since it still relies on visual cues to align the data. Even at the highest frame rate of 30 fps, the time resolution is limited to 33.3 ms, which leads to an average timing error of about 16.7 ms. My application will have some fast motion, so it might not be ideal to have that offset.

Currently, I think the two possible resolutions are:

1. Access to the Aria internal clock, which would allow us to establish a reliable mapping between the device’s time domain and the host system’s local time.
2. Directly embedding external timecode into aria’s timecode domain.

I have checked the aria sdk module, and it seems that it’s impossible to query the device time for the current version (please correct me if I am wrong). I have also tried to create a client and subscribe to the aria, but it seems that the public sdk block all the streaming.

[Tarunstk]

Hey @jyuntins thanks for that summary of your attempt. The observations you have showcased make sense and yeah seems like the offset will cause non-linear time drift (I believe).

I will need to look deeper into methods to extract the aria internal clock timestamps. Will need sometime and I hope to do this this week. I will have a solution or clarification for you in a couple of days. Thanks for understanding.

[jyuntins]

Hi @Tarunstk

Got it. Thanks for checking into this.
No worries at all. I will keep exploring on my side as well.