DFlow mocap module time stamp is unreliable

[moorepants]

I just discovered that using the TimeStamp column in the mocap module was a bad idea. I new that the TimeStamp column had some anomalies (see http://gait-analysis-toolkit.readthedocs.org/en/latest/motek.html#data-column-descriptions) but none of my test data showed the additional anomalies I've just uncovered. Once I used a new differentiation method in PR #128 these anomalies were exposed, where as the previous differentiation method quietly but incorrectly smoothed them over. This issue is that DFlow receives streaming data from Cortex and it is accessible in the from the mocap module. Cortex likely handles the data stream correctly, i.e. within the 0.01 s sampling period it collects data from all cameras and computes the marker coordinates providing a value that was collected at sometime during the sampling period. Each cortex value is delivered along with the sequential frame number and it must have a mechanism to flag a frame number that has missing marker values. DFlow receives this data over ethernet and time stamps it using the DFlow CPU clock. This issue arises because the data flow comes into the DFlow computer serially and likely asynchronously wrt to time. A FIFO flow is likely used to so data is dropped, but this means that data can "stack up" with respect to the DFlow cpu time and if the TimeStamp column is relied upon, there can be issues.

The following plot shows the same graph from (http://gait-analysis-toolkit.readthedocs.org/en/latest/motek.html#data-column-descriptions) except with a data set (trial 31) that doesn't have as clean a time stamp. This plot shows the difference in the current and subsequent time value:

Note that most of the differences are around 0.01 s, but there are some large outliers and notice the values that are close to zero. These correspond to the data stack up wrt to time.

The next plot shows three plots where I've flagged all the values associated with the time stack ups in red. The first plot is TimeStamp vs FrameNumber and the following two show the RGTRO marker coordinate against time and frame number.

Notice that there are a fair amount of time stack ups in the data. We can zoom in to one instance to see what is going on:

This graph plots the same things but we see how the frames get stacked up wrt to time.

The motek.py module parses the data and uses the TimeStamp column as a reliable value. You can see how this will cause issues when trying to differentiate marker positions with respect to time. The real time differentiator does can differentiate these instance without too much error so it wasn't noticed before. But the values are still wrong. The traditional central differencing methods have major issues with this incorrect data.

Note that this issue is in addition to missing marker values. The DFlowData class needs to be reworked to make use of the FrameNumber instead of the TimeStamp column.

The other issue with this is that synchronizing the mocap module and record module data relies on the time stamp values. I assume that the TimeStamp column from each module is generated from the DFlow cpu time. We'll need to look into the synchronization after the time stamp issues are taken care of to ensure that synchronization still works as expected.

[moorepants]

@tvdbogert You may be interested in this. It's something that DFlow users will need to be aware of. I thought I investigated it thoroughly over a year ago, but missed this. It didn't show up till I tried the traditional differentiation methods on all of the data I have.

This is also a good example of finding an error at the beginning of a really long data processing pipe line. I noticed this when my gain identification plots weren't coming out well any more and started digging for the source. The source ended up being in the first processing step. Fortunately, I should be able to fix this at the beginning of the pipe line and just run make at the end to get the correct answers out.

[tvdbogert]

We looked into this a long time ago (the "hiccups" issue) and I thought it was resolved. Does this only happen in earlier trials?

@spinningplates do you remember the conclusion of the hiccups issue?

Clearly the RGTRO data shows that the timestamp can't be trusted and you are better off using the frame number. I always thought the timestamp would be more reliable because it would just advance by 20 ms if Cortex decided to skip a frame. I thought that's what the time stamp was for... If we start trusting the frame number, we could not know if frames are dropped but maybe that never happens.

This is a nasty mess but it looks like you have a working fix.

And yes, the synchronization between the files created by the mocap and record modules is now also an issue. Is the sync even recoverable? For the indirect identification we will need belt speed data synchronized with the mocap data. If needed (but let's try not to go there..) we can recover belt speed from the mocap data because there is always one foot on the ground that has marker on it.

[moorepants]

Does this only happen in earlier trials?

It happens in all of the trials that I collected, some more so than others. I think I didn't notice it because I based my software off of the data you provided from you HBM paper and it may not have had this issue. Not sure, but it got missed and is in all of the trials.

Clearly the RGTRO data shows that the timestamp can't be trusted and you are better off using the frame number.

Yes, the TimeStamp in the mocap module should not be trusted.

I always thought the timestamp would be more reliable because it would just advance by 20 ms if Cortex decided to skip a frame.

I thought it was a "smart" TimeStamp too but it isn't. Motek probably needs to rethink how they are handling this. When I built the custom system for my instrumented bicycle I had to pay careful attention to buffering to ensure correctly sampled data. It isn't trivial and I can imagine D-Flow's setup is a nightmare because they read in data from multiple USB sources. USB is a serial buffered protocol and you can't just slap on CPU time stamps to the data that comes from it.

If we start trusting the frame number, we could not know if frames are dropped but maybe that never happens.

There is really no way to tell if a "frame" was dropped. We'd have to get clarification from Motek on that. But they somehow do recognize the missing markers.

This is a nasty mess but it looks like you have a working fix.

Yes, I've worked around it.

Is the sync even recoverable?

I have a hack that gets this working. This is what I do:

1. Create a time vector based on frame number from the mocap module starting at t0=0.0 that is sampled at 100 hz.
2. Get the first value of the TimeStamp column in the mocap data (i.e. the D-Flow computer time that the first frame arrived).
3. Add the value from 2 to the vector in 1 to get the D-Flow CPU clock time that actually matches to the Cortex Frames.
4. Interpolate the data from the record module based on the Time column and the time vector from 3.

That should give a reasonably close synchronization, but it isn't guaranteed. That's the best I could think of.

[tvdbogert]

Your sync fix will depend on how well the Cortex clock (which is the camera frame rate, set at 100 Hz in our experiments) stays synchronized with the D-Flow clock.

Is there any way we can check if there is any drift between them, at the end of the trial?

Or better, compensate for any drift by looking at the final time stamp and frame number in the mocap data. This would take care of drift, unless at the final frame (or the initial frame, for that matter) we happen to be in the midst of a buffering delay.

[skhnat]

@tvdbogert @moorepants The hiccup issue was a complete freezing of all data, including the frame number.

Here's an example of the data:

Here's an example from the .txt file from the Mocap, where column 1 is the time stamp, column 2 is the time difference between the two timestamps, and column 3 is the frame number:

From my understanding, this particular issue was completely resolved. I ran several tests after they replaced the network card and it never seemed to happen again.

[tvdbogert]

OK thanks for clearing that up! And sorry for my confusion. It's a very different problem that we are seeing now. D-Flow is a nice system but I am beginning to see that it was not designed to be a scientific instrument... Also they have limited experience with Cortex, their usual mocap system is Vicon.