trying to run prediction on BézierLaneNet trained on CULane with ResNet18

[a2mar]

Hello,
As the title states, I'm trying to get the prediction for an image on a pretrained BézierLaneNet.
I installed all dependecies, cloned the repository and downloaded the correct .pt file for the weights (resnet18_bezierlanenet_culane_aug1b_20211109.pt).

I ran the command accordning to the shell script:

$ python main_landet.py --test --config=configs/lane_detection/bezierlanenet/resnet18_culane_aug1b.py --mixed-precision --checkpoint=resnet18_bezierlanenet_culane_aug1b_20211109.pt

However, I get the following Error:

Build from dict error in function or class: CULaneAsBezier
In Python: <class 'utils.datasets.lane_as_bezier.CULaneAsBezier'>
Traceback (most recent call last):
  File "/media/cpvrlab/Ubuntu/BezierLaneNet/pytorch-auto-drive/main_landet.py", line 75, in <module>
    runner = Runner(cfg=cfg)
  File "/media/cpvrlab/Ubuntu/BezierLaneNet/pytorch-auto-drive/utils/runners/lane_det_tester.py", line 22, in __init__
    super().__init__(cfg)
  File "/media/cpvrlab/Ubuntu/BezierLaneNet/pytorch-auto-drive/utils/runners/base.py", line 222, in __init__
    dataset = DATASETS.from_dict(cfg['test_dataset'] if 'test_dataset' in cfg.keys() else cfg['dataset'],
  File "/media/cpvrlab/Ubuntu/BezierLaneNet/pytorch-auto-drive/utils/registry.py", line 41, in from_dict
    raise e
  File "/media/cpvrlab/Ubuntu/BezierLaneNet/pytorch-auto-drive/utils/registry.py", line 38, in from_dict
    return function_or_class(**dict_params_)
  File "/media/cpvrlab/Ubuntu/BezierLaneNet/pytorch-auto-drive/utils/datasets/lane_as_bezier.py", line 39, in __init__
    self._init_all()
  File "/media/cpvrlab/Ubuntu/BezierLaneNet/pytorch-auto-drive/utils/datasets/lane_as_bezier.py", line 89, in _init_all
    with open(split_f, "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '../../dataset/culane/lists/test.txt'

Also, how am i supposed to pass an image to the inferenced model? I can't find a reference to any image path in the invoked code.
I don't want to train the model, I want to inference a fully trained one, proof of concept.

I would be very happy about your feedback.

[voldemortX]

@a2mar Hi, you want to just input an image?

[voldemortX]

What you did seem to be testing on the culane dataset, for this you need to follow the dataset setup instructions first.

[a2mar]

Hi @voldemortX, thanks for replying! Yes, I just want to input an image and get the model's prediction on it.

[voldemortX]

@a2mar I think VISUALIZATION.md is more suited for your needs.

[a2mar]

@a2mar I think VISUALIZATION.md is more suited for your needs.

You are right. I ran the visualization with the proposed command:

python tools/vis/lane_img_dir.py --image-path=test_images/ --image-suffix=.jpg --save-path=PAD_test_images/lane_test_images/culane_pred --pred --config=configs/lane_detection/bezierlanenet/resnet18_culane_aug1b.py --checkpoint=model.pt --style=bezier

It worked with a sample image from the CULane dataset, but I just get nothing from my custom image (same image resolution but from a simulated scene). It's a very clear road image, i'm surprised it's not working:

Maybe the missing features on the side of the road that create a vanishing point by linear perspective???

I might have to create a my own dataset to train with your model architecture.

I know, that working visualization with inferencing (with CULane sample image) states the issue as closed, but I have two questions towards the calculation of the Bézier control points:

1. inferenced control points in /utils/runners/lane_det_visualier.py:
   on line 140:
   

   pytorch-auto-drive/utils/runners/lane_det_visualizer.py

   Line 140 in 137e63a

   cps, keypoints = self.lane_inference(imgs, original_imgs.shape[2:])

For debugging, i printed out the cps of the CULane sample image:
print("control points from inference", cps)
gives this:

[array([[[ 521.5233 ,  588.79816],
        [ 588.9344 ,  526.7899 ],
        [ 660.4163 ,  466.8253 ],
        [ 727.26135,  405.08362]],

       [[1168.8438 ,  588.53705],
        [1044.2479 ,  526.2218 ],
        [ 916.8609 ,  465.54346],
        [ 791.47943,  403.98245]],

       [[1636.2465 ,  568.43604],
        [1376.845  ,  513.2661 ],
        [1115.457  ,  460.1371 ],
        [ 856.03094,  405.64594]],

       [[  46.17844,  586.171  ],
        [ 258.55942,  526.2587 ],
        [ 475.62637,  466.76587],
        [ 683.7774 ,  406.7499 ]]], dtype=float32)]

I interpreted this as the 4 sets of 4 control points for 4 Bézier curves. But I how are the coordinates outside the image boundary (288x800)?
Are these the coordinates for a "stretched" bird-eye view? If so, what are the transformation parameters?

2. generating Bézier control points for cubic Bézier curve from CULane image labels (key points).
   In your paper (bravo on that) you described that the ground truth for the Bézier control points are created by solving the equation system with the Bernstein matrix, the unknown control points and the given key points from the label file. Unlike ABCNet, you did not set the the first key point to the first Béz control point and the last key point to the last Béz control point.
   How exactly are you calculating the Bézier control points?

Can this calculation be found somewhere in the code? I might have overlooked it,

[a2mar]

BTW, turns out TuSimple was the better choice for me images.

However, on the curves I get more probability for the lines of the pavement (poor modeling choice from me).
I will still need to train, or at least transfer learn.

I'd be very happy if you could answer my Bézier control point related questions above.

[voldemortX]

@a2mar

1. It is possible that Bezier control points are out of the image border, by the curve's definition. Even the endpoints are not defined on image in out paper.

2. The calculation of GT control points can be found at https://www.github.com/voldemortX/pytorch-auto-drive/tree/master/docs%2FCURVE.md

3. About the generalization of BezierLaneNet, I believe these public datasets are far from sufficient to make a top-down method generalize. However, you might try out some bottom-up methods (e.g., SCNN) to see if it can generalize to local structures.

[a2mar]

@a2mar

1. It is possible that Bezier control points are out of the image border, by the curve's definition. Even the endpoints are not defined on image in out paper.
2. The calculation of GT control points can be found at https://www.github.com/voldemortX/pytorch-auto-drive/tree/master/docs%2FCURVE.md

Thank you!

@a2mar
3. About the generalization of BezierLaneNet, I believe these public datasets are far from sufficient to make a top-down method generalize. However, you might try out some bottom-up methods (e.g., SCNN) to see if it can generalize to local structures.

This is makes a lot of sense. Great advice! I'm going to try out bottom-up methods before deciding on further procedures.

[a2mar]

I also wanted to ask you, how exactly you calculated the sampling loss for BézierLaneNet. From your paper:

Do you make that calculation in the image coordinate system or in the world coordinate system?
If you use the image coordinate system, how do you account for the much higher importance of errors near the vanishing point?
They would be underrepresented compared to errors near the camera.

[voldemortX]

@a2mar In the image coordinates. Because these are still 2D datasets. It could be easily adapted to fit curves in 3D, since there are many 3D datasets these days.

[a2mar]

I want to try tuning BézierLaneNet with my own dataset.

From what I read in the CURVES.md documentation, the data preparation aims to create a json file with the Bézier labels of the desired degree:

   {"raw_file":filename1, "Bezier_control_points": [[...],[...], ..., [...]}
   {"raw_file":filename2, "Bezier_control_points": [[...],[...], ..., [...]}
   ...
   {"raw_file":filenamen, "Bezier_control_points": [[...],[...], ..., [...]}

These are Bézier control knots in the frame (image) coordinate system. I can generate this without the script you provide.

If I wanted to use the TuSimple dataset-config for training BézierLaneNet with my own dataset, what else would I need?
I wouldn't need any other labels in the style of TuSimple, or would I?

[voldemortX]

I want to try tuning BézierLaneNet with my own dataset.

From what I read in the CURVES.md documentation, the data preparation aims to create a json file with the Bézier labels of the desired degree:

   {"raw_file":filename1, "Bezier_control_points": [[...],[...], ..., [...]}
   {"raw_file":filename2, "Bezier_control_points": [[...],[...], ..., [...]}
   ...
   {"raw_file":filenamen, "Bezier_control_points": [[...],[...], ..., [...]}

These are Bézier control knots in the frame (image) coordinate system. I can generate this without the script you provide.

If I wanted to use the TuSimple dataset-config for training BézierLaneNet with my own dataset, what else would I need?
I wouldn't need any other labels in the style of TuSimple, or would I?

@a2mar The ideal approach is to label and format your dataset exactly like TuSimple. Otherwise you could write new dataset class & preprocess scripts to adapt your own format.

[a2mar]

Just to

I want to try tuning BézierLaneNet with my own dataset.
From what I read in the CURVES.md documentation, the data preparation aims to create a json file with the Bézier labels of the desired degree:

   {"raw_file":filename1, "Bezier_control_points": [[...],[...], ..., [...]}
   {"raw_file":filename2, "Bezier_control_points": [[...],[...], ..., [...]}
   ...
   {"raw_file":filenamen, "Bezier_control_points": [[...],[...], ..., [...]}

These are Bézier control knots in the frame (image) coordinate system. I can generate this without the script you provide.
If I wanted to use the TuSimple dataset-config for training BézierLaneNet with my own dataset, what else would I need?
I wouldn't need any other labels in the style of TuSimple, or would I?

@a2mar The ideal approach is to label and format your dataset exactly like TuSimple. Otherwise you could write new dataset class & preprocess scripts to adapt your own format.

So I will still need the original TuSimple labels json (with "lanes" and "h_samples"), even though I don't need the script to generate the Bézier label json?

[voldemortX]

@a2mar It would be the safest way.

Theretically you only need control points. For the segmentation auxillary loss you will also need seg masks (but you can erase that part).