RBT Various additions and improvements #1778

SamFlt · 2025-09-24T15:59:02Z

This PR introduces new functionalities to the Render-Based Tracker:

General:

The render data now contains more information, such as the object's diameter (bounding box extent) and 3D corners in object space.
The tracker can now be reset and the model reloaded more easily, without bugs related to Panda3D appearing. The reset function should be called when changing object or reinitalizing tracking. Some of the tracker's parts are stateful so this is important.
Rendering is now done at a lower precision. High FP32 precision is not really needed, and GPU to CPU transfer is extremely slow. For the Geometry renderer, The amount of transmitted bytes has been divided by 4. These changes had no significant impact when evaluating on several benchmarking datasets. In my tests, they improved render speed by around ~30%. For instance, a render that took 7ms now takes 5, which is not negligible when considering that rendering can happen twice per frame (when odometry is activated).
Reworked the convergence metric (early stopping for the optimizer and rerendering criterion). Added the reprojection error as a criterion, which is easier to reason with.
Fixed a bug where the object contours that were too close to the computed object's 2D bounding box were not detected.
Update documentation with new features and odometry.

Masking:

a Rework of the color masking process to use logistic regression instead of the previous solution that was based on the object/background probability ratio along with an hyperparameter threshold to be tweaked by the user. The masks are now generally far less noisy. An another solution instead of logistic regression would be to use the Bayes'rule, but this solution fails when we consider a small object that has a color similar to that of the background.
Some computational speedup in color masking.
Introduced a new Depth mask strategy
Introduced a Combined color/depth masking strategy. The strategy is to take the minimum of the probabilities estimated by the color or depth mask.

Features:

Introduced an experimental trackable feature, based on photometric Visual servoing.
In Python, added XFeat keypoints as trackable features.
Some of the M-estimators now use adaptive values that depend on the object's size in the image or in 3D space
Improved speed for the depth based tracker

Odometry:

Integrated the XFeat Visual Odometry strategy.

Pose estimation:

Developed two pose estimation approaches based on leveraging XFeat to solve the PnP problem.
Usage tutorial and script to use those methods

Others:

Integrate Python JSON parsing for XFeat tracker and odometry

Known issues:

As there is now a more significant part of the code that is pure Python, it has become apparent that there is an issue with the stubs and documentation generation: The Python part is not parsed and Sphinx (used for the doc) does not correctly generate the pages for each class. This will be the subject of a later PR.

…houette sampling

…ow management so that other windows are correctly disabled

…g for RBT

… strategies

…perly extracted with the shader

…ng autodoc and autocomplete

… requirement of having the source git in XFEAT_PATH. This solution can still be used if torchhub fails.

fspindle

In general,

Check if copyright headers are present
Protect openmp usage with `#if defined(VISP_HAVE_OPENMP) in *.cpp files. Avoid openmp usage in *.h
Remove useless empty lines

fspindle · 2025-09-29T15:28:23Z

modules/ar/include/visp3/ar/vpPanda3DPostProcessFilter.h


  virtual FrameBufferProperties getBufferProperties() const = 0;
+  virtual PointerTo<Texture> setupTexture(const FrameBufferProperties &fbp) const;
+
