Dataset conversion code

[bhack]

Can you share you dataset conversion code?

I want to test train 1 stage but I want to recover some of the original Kubric-MOVIF metadata that is lost in annot.npy conversion.

[aharley]

What do you want from the original metadata?

[bhack]

I need the geometry bundle to convert every track in the current npy to a 3D world point.
I've not checked if it is available for all the dataset you have used but for the train stage 1 kubric-MOVi-f it available is for sure,
The derived world coords x point annotation is enough but I suppose it is going to require more work on your side. The basic idea is to test #2 on stage 1 or 2.

[aharley]

OK I think I can dig up the script you are asking for, but from your side it seems like it will be necessary to recompute the data, since I believe the Kubric simulator outputs the data randomly each time. (So, you cannot just "recover" the metadata; you need to generate new data.)

[bhack]

But at least for the Kubric/Movi-f case that you have used in stage 1 this data is already in the official TDFS storage on GCS. Or do you have regenerated your own run with Kubric?

[bhack]

Just for the phase 1 working with the original Kubric-Movif TDFS I've tried to remap your points in npy to the object I'd using the segmentation map in the TFDS but I have some subpixel/alignment issue over the classes boundaries to correctly assign 100% of the point to the right object/class over the edges.
As I want to maintain the same sampling strategy you have used and without the sampling code it would be nice if you can at least export the object/instance id (or exact 3d world coords without introducing reverse engineering back projection error of your point over the quantized depth map in TFDS)

[bhack]

Have you used a variant of https://github.com/google-research/kubric/tree/main/challenges/point_tracking for Kubric-Movif? What parameters you have used?

[aharley]

Thanks for your patience here. I confirmed that our S24 kubric data is indeed an export of the google-research data, referencing tfrecord files in a folder called movi_f/512x512/1.0.0/.

Here is the script we used to export our npy data from the tfrecords:
https://gist.github.com/aharley/8d3e1dd0435fcefa55c724136fa3093f

The S64 data was generated fresh, but it's similar.

[bhack]

Yes I've already generated S24 but not exactly with your params (as I see instead now in your gist).

So for S64, do you have generated MOVI-F 64?
It would be nice if you could share the TFDS/tfrecords of this so that I can run your gist or my similar version with your parameters.

[bhack]

Regenerating S64 at 512 is quite heavy. It would be great if you still have the TFDS to share

[aharley]

Turns out the S64 tfrecords are a bit messy: they were exported in about 100 batches of different sizes, partly because our environment was a bit unstable. I tarred the first batch for you here:

https://huggingface.co/datasets/aharley/kubric_s64_tfr/tree/main

Maybe you can try it out and let me know if it useful?

[bhack]

I've checked it and it is ok but do you still have the full set up to 05736(right?) ?

[bhack]

The alternative if you still have all the point datasets scripts working but a bit messy it is if you can add trajs_3d key (as in the original pointsodissey) in your new annot.npy schema.

[aharley]

That's a pretty good idea...

[bhack]

That's a pretty good idea...

It would be great as I've started also adding 3d track info to the the pointodissey S64 you have used but then I need to ping you forth and back about details not in the paper, like your 64 chunk policy over the video full length etc.., to be correctly aligned with you data, other then the slow KUB S64 full re-rendering.
So honestly having the extra 3d tracks key for the datasets you have used in your annot.npy version it is the fastest and the most aligned solution.
Let me know..