Semantic Point Cloud Segmentation

A reproducible and extensible repository for semantic segmentation of LiDAR point clouds, combining theory, code, and literature into a single resource. This project is based on assignments from RWTH Aachen University’s Institute for Automotive Engineering (ika) and the Automated and Connected Driving Challenges (ACDC) MOOC on edX.

---

🚀 Roadmap for Semantic Segmentation

This repository is intended not only as working code but also as a learning and experimentation platform. It covers the essential steps for semantic segmentation of point clouds:

1. Understanding the Task

   • What semantic segmentation is and why it matters for automated driving.
   • Difference between image segmentation and point cloud segmentation.
   • Challenges: sparsity, occlusion, annotation limitations.

2. Data Handling

   • Loading point cloud datasets (.npy format).
   • Labeling via cross-modal label transfer (camera-based segmentation transferred to LiDAR).
   • Preprocessing: normalization, masking, depth calculation.

3. Modeling

   • Building TensorFlow data pipelines.
   • Input format: [height, width, channels] = [32, 240, 6] projection of LiDAR scans.
   • Semantic class definitions (Road, Car, Vegetation, etc.).

4. Training

   • Base model training on automatically annotated datasets.
   • Dealing with noisy/missing labels.

5. Boosting Performance

   • Applying data augmentation (random flips, random shifts).
   • Improving robustness to real-world variations.

6. Evaluation & Visualization

   • 2D segmentation maps.
   • 3D visualization with Plotly for interactive inspection.

7. Beyond This Repository

   • Integration of custom datasets.
   • Experimentation with advanced models (e.g., KPConv, RangeNet++, MinkowskiNet).
   • Deployment on real-world driving datasets.

---

📂 Repository Structure

semantic-point-cloud-segmentation/
│
├── docker/
│   ├── Dockerfile
│   ├── requirements.txt
│   └── run.sh
│
├── notebooks/
│   ├── 1_Semantic Point Cloud Segmentation.ipynb
│   └── 2_Boosting Semantic Point Cloud Segmentation.ipynb
│   ├── assets/
│   ├── datasets/
│   ├── grid_mapping/
│   ├── ipm_assets/
│   └── localization/
│   ├── object_detection/
│   ├── segmentation_utils/
│   ├── tensorflow_datasets/
├── experiments/
│   ├── runs/
│   └── configs/
├── literature/
│   ├── papers/
│   └── summaries.md
├── .gitignore
├── LICENSE
└── README.md

Suggested future expansion

• models/ → different architectures (baseline CNN, RangeNet++, etc.)
• datasets/ → links/scripts for public datasets (SemanticKITTI, nuScenes)
• training/ → reusable training scripts
• evaluation/ → metrics, benchmark comparisons

---

🐳 Running with Docker

Ensure Docker is installed. Then:

# 1. Clone repository
git clone https://github.com/your-username/semantic-point-cloud-segmentation.git
cd semantic-point-cloud-segmentation

# 2. Build Docker image
docker build -t pcl-segmentation ./docker

# 3. Run container with Jupyter
./docker/run.sh

The script run.sh maps the current repository into the container and launches Jupyter Lab. Open http://localhost:8888 in your browser.

---

📓 Notebooks Overview

1. Semantic Point Cloud Segmentation

• Loads LiDAR point cloud datasets (.npy).

• Explains cross-modal label transfer (labels from images to LiDAR).

• Defines semantic classes (Road, Car, Vegetation, etc.).

• Explores the dataset:

  • Plotting per-channel features (X, Y, Z, Intensity, Depth).
  • 2D segmentation maps.
  • 3D interactive visualizations with Plotly.

• Provides a baseline setup for segmentation.

2. Boosting Semantic Point Cloud Segmentation

• Builds on the first notebook by introducing data augmentation:

  • Random horizontal flips.
  • Random horizontal shifts (to simulate LiDAR rotations).

• Implements robust TensorFlow input pipelines with augmentation.

• Prepares the data for more generalizable training.

---

📖 Literature and References

This repository draws inspiration from:

• Automated and Connected Driving Challenges (ACDC) MOOC on edX.

  • Enroll for free: here
  • Taught by Institute for Automotive Engineering (ika) at RWTH Aachen University.

---

📝 Acknowledgments

• This repository builds upon coursework from RWTH Aachen University / ika.
• Special thanks to the creators of the ACDC MOOC for making the material freely available.

---

✨ Visual Example