Set query index in the middle of the window yet still utilize the previous frames before the query index

[William-WANG2]

Dear Authors,

Thank you for your excellent work.

I’m working on a project that requires estimating optical flow for each frame. Because the motions in my data are extremely large, your method—which combines tracking and flow—seems like a great fit.

To obtain per-frame flow, my current approach is to iteratively truncate the video from the beginning and process frames 1…t for each target frame t, while keeping the flow from frame 1 to frame 2. However, this appears to make the model lose context prior to the target frame, which leads to poor tracking for parts of the moving object. Interestingly, adding just one frame before the target frame already resolves the issue.

Do you have any suggestions on how I could approach this problem using your project? Any guidance is appreciated : )

Thank you for your time.

[William-WANG2]

I notice that in the demo.py, to obtain a specific “query” index, it actually divides the video into two parts, reverse the first part and individually run the algorithm on the two parts. I also tried to modify the fmap_anchor in the code, making it not hardcoded to the first frame. But unfortunately it does not help too much. I wonder if this suggests that there is no way to utilize the full context information both before and after the query index for this model.

[aharley]

Hi, thanks for the comment/question. Sorry for the slow reply.

I'm not sure I totally follow your use-case yet. "my current approach is to iteratively truncate the video from the beginning and process frames 1…t for each target frame t, while keeping the flow from frame 1 to frame 2." -- so this means frames 1 and 2 are always there, and then you put the rest of the video after, like 1,2,501,502,503,... ?

[William-WANG2]

Thank you so much for your help and I'm sorry for my overlook. It should be "my current approach is to iteratively truncate the video from the beginning and process frames 1…t for each target frame t." Not sure why I add the last sentence initially : (

[aharley]

OK thanks. I almost understand now. So, you want the long-range flow from frame t to every other frame?

I want to clarify because you also said the project "requires estimating optical flow for each frame", which to me sounds like getting the flow for 1-2, then 2-3, then 3-4, ...

[William-WANG2]

I actually wish to have the optical flow for every frame, i.e., getting the flow for 1-2, then 2-3, then 3-4, ... I'm not sure if this project can achieve this by considering all the context before a specific frame t for more accurate optical flow.