Robotic arms using proximity sensors are notoriously inefficient at the pick and place task. Proximity sensors do not enable the user to select the category of objects to displace. Moreover, they fail to estimate the exact position of the object precisely in one pass without resorting to a long scanning procedure, equivalent to a blind person asking his dog to tell him where the car keys are by covering the whole field by foot.
Instead, contemporary progress in machine vision allows the machine to locate and seize the complex object directly after one look, in real time, without resorting to one-dimensional, close proximity sensors.
💬 Ask a question 📖 Read questions
A preconfigured development computer (auto-install scripts and documentation included) connects by SSH to a preconfigured Nvidia Jetson Nano (auto-install scripts and documentation included) with a Raspberry Pi camera v2.1 or a Arducam Camarray Hat (auto-install scripts and documentation included). Only one is attached by a 1 meter MIPI ribbon cable with a repeater extension to the tip of the uArm. The USB-controlled Jetson becomes the central controlling unit in this topology. The x86_64 (but could be other architectures) flashes the firmware on the ESP32 microcontroller (soldered on the Altium-designed Printed Circuit Board) through another USB port. This firmware (without going into the details or extras just yet) listens to the UART for GCODE which it then executes and effects using 3 Pulse Width Modulation outputs to 4 proprietary servomotors (the only uArm part which has not been (re)defined in this project). 4 analog inputs provide angle feedback to the firmware. The valve is strapped to VCC to save pins and the pump is driven with a GPIO by flipping the logical levels. The uARM (the name of the robotic arm used in this project) initializes to a position in the middle of its servomotor angle range. It can be controlled using GCODE from the UART to pick up objects using absolute, relative or polar coordinates (polar coordinates simplify the X and Y axis PIDs and are normalized to grads to represent the whole uArm range by all axes scaled by a +/- 100 range). It can then pick up a single selected class of common objects labeled from the COCO dataset (80 classes) and a bunch of machine learning goodies see software/jetson/fastmot/, using neural network detection feedback from a camera. It then places and drops the object to a predefined location and loops...
TODO if you want to use it with your own microcontroller (review presets before implementing your own!)
- Add an #ifdef to identify your microcontroller pin PWM, ADC and GPIO in uArmPin.h in UArmForArduino (only necessary if your application does not use the same pinout)
- If your microcontroller does not define timers, add it to the modified version of the Servo Arduino library with the slowmove extension by comparing it with the provided AVR (AtMega) implementation which was added (not necessary but recommended)
- Use the grab() and drop() methods in pyuarm to complete the pick and place in the set_servos() process (only the fetch, the hardest part, is set in the main loop)
- If you want the uArm to have perspective and grab objects, converting back from Polar(length, angle, height) back to Cartesian(x, y, z) (reuse functions in pyuarm) is recommended to match camera and uArm range coordinates exactly (the whole ranges were enabled, but not matched back with the initial camera position in cartesian coordinates)
- Tune the initial position height to scale the camera view and register the range coordinates of the uArm and of the camera equally
The project timeline was obtained with the folowing command
$ git log --pretty=format:"%h %ad | %s %d [%an]" --date=short >log.txt
Install documentation as website
This has not been attempted as I would lack the time to assemble the parts. It is available though.
see instructions in software/jetson/jetson-containers/README.md
A standard ESP32-WROOM-32 module is the microcontroller. A CP2109 USB to serial chip is used to program it and transfer data from a Micro-USB connector. An unused SD card slot is available if more external memory is required. A MCP16311T-E/M step-down regulator taps 12 volts from the jack barrel connector to provide the 5 volts servomotors need to function. A fixed output Complementary Metal Oxide Semiconductor Low-Dropout regulator (TC1264) taps 5 volts from the USB connector to provide 3.3 volts for the other circuits. One Light Emitting Diode confirms that regulator provides the 3.3 volts. There is one GPIO-controllable LED and one push button plugged on another GPIO. Another push button is plugged to the EN pin for enabling the ESP32-WROOM-32 module.
The main code was tested on PC and Jetson for easier modular tests while programming. Code quality and reproducibility are hallmarks of this project.
Some presets have been defined for AVR, ESP32 and ESP32-S2. The script below automates the flashing process. software/jetson/install/flash_firmware_custom.sh
See software/arduino-1.8.13/portable/sketchbook/libraries/UArmForArduino/README.md for more explanations.
ArduCAM Camarray or Raspberry Pi camera v2.1 (your choice, but the RPi cam is less expansive and requires less installs)
CLICK ME - Hardware prerequisites
- Micro SD card with at least 16Gb of storage
- Ubuntu host PC* with SD card slot or USB SD card reader/writer
- Jetson Nano Dev Kit with micro USB power supply (at least 5V, 2A) or DC power supply (5V 4A)
- Full HD display with HDMI
- Keyboard and mouse
- Ethernet cable connected to the Internet, or USB Wi-Fi adapter
- Either an Arducam camera array hat, or the Raspberry Pi camera v2.1 (recommended)
- You can use a Windows host PC to flash the microSD card instead, however this tutorial uses Ubuntu as it’s a simpler process. See NVIDIA’s guide for a Windows option.
CLICK ME - Software prerequisites (all installed automatically)
- CUDA>=10
- cuDNN>=7
- TensorRT>=7
- OpenCV>=3.3
- PyCuda
- Numpy>=1.15
- Scipy>=1.5
- TensorFlow<2.0 (for SSD support)
- Numba==0.48
- cython-bbox
- pyserial
CLICK ME - Hardware instructions for the Jetson Nano Devkit B01 (previous board revision also shown)
- Insert the microSD card into the appropriate slot
- Connect the display and USB keyboard /mouse and Ethernet cable.
- Depending on the power supply you want to use, you may have to add or remove the jumper for power selection:
- If using a jack, the jumper must be set.
- If using USB, the jumper must be off.
- Plug in the power supply. The green LED (D53) close to the micro USB port should turn green, and the display should show the NVIDIA logo before booting begins.
- You will want to install a jumper on J48 to power with the jack barrel
CLICK ME - Installation From Scratch Instructions for Linux (skip down to "Download and install the live *.iso" if installing Linux is needed and get back here just after the install)
None which aren't covered by this guide.
$ sudo apt update$ sudo apt install -y git$ mkdir -p ~/workspace$ cd ~/workspace$ git clone https://github.com/abstractguy/TSO_project.git$ cd TSO_project/software/jetsonCLICK ME - Download and install the live *.iso of Ubuntu 18.04.5 LTS for x86_64 (from here https://unetbootin.github.io)
$ sudo -H bash install/install_unetbootin.shOtherwise if running Windows, download and install the live *.iso of Ubuntu 18.04.5 LTS for x86_64 from here: https://unetbootin.github.io
Start with a fresh install of Ubuntu 18.04.5 LTS with automatic updates and proprietary drivers activated (do not activate secure boot through the USB install method as this can be done later)
Download Windows 10 Enterprise Edition for VirtualBox (https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/)
$ cd ~/workspace/TSO_project/software/jetson && bash install/install_virtualbox.shYou can now use the Windows 10 Enterprise Edition to run Windows 10 applications from a Linux host, like Altium for PCB developement
$ sudo -H bash ~/workspace/TSO_project/software/jetson/install/install_jetpack_prerequisites.shAfter the prescribed reboot, review software/jetson/install/INSTALL_DOCKER.md and go through the entire procedure
$ cd && bash ~/workspace/TSO_project/software/jetson/install/install_conda.sh$ cd ~/workspace/TSO_project/software/jetson && bash install/install_conda_environment.shCLICK ME - Flash uArm firmware
$ cd ~/workspace/jetson && bash install/flash_uarm_custom.sh$ cd ~/workspace/jetson && bash install/flash_uarm.sh
When awaiting a D7 button press, place the pump directly on the table, the uARM facing the center of its X axis
When awaiting a D4 button press, place the pump directly on the table, the uARM aligned by 45 degrees (pi/4 rad) from the clockwise extremum
$ sudo /opt/conda/envs/school/bin/python3 -m pyuarm.tools.calibration.calibrate --port /dev/ttyUSB0CLICK ME - Jetson Nano or AGX Xavier Devkit install instructions (JetPack-4.4)
If using the Nvidia Jetson AGX Xavier Devkit, sign in to install Nvidia's sdkmanager from https://developer.nvidia.com/nvsdk-manager
Else, if using the Nvidia Jetson Nano Devkit, flash the Micro-SD card and then plug the Micro-SD card in the device's slot
$ cd software/jetson && bash install/install_jetson_nano_sd_card.shEither follow the instructions on the monitor or type this below, after plugging in a mouse, a keyboard, a HDMI screen and the network connector type you have chosen
$ screen /dev/ttyACM0If it can't SSH into the Jetson, you have skipped the above step; redo it and the install will resume after
$ ssh sam@192.168.55.1 'mkdir -p ~/workspace'$ scp -r ~/workspace/TSO_project/software/jetson sam@192.168.55.1:/home/sam/workspace$ ssh -t sam@192.168.55.1 'cd ~/workspace/jetson && bash install/install_jetson.sh <JETSON_PASSWORD>'CLICK ME - Docker install for Ubuntu 18.04 on Jetson (not necessary)
Make sure to have nvidia-docker installed. The image requires an NVIDIA Driver version >= 450. Build and run the docker image:
$ cd ~/workspace/jetson
$ docker build -t project:latest .
$ docker run --rm --gpus all -it -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY project:latestCLICK ME - Inference
The use of FastMOT as a custom multiple object tracker (here post-processed for single objects) implements:
- YOLO detector
- SSD detector
- Deep SORT + OSNet ReID
- KLT optical flow tracking
- Camera motion compensation
- Support Scaled-YOLOv4 models
- DIoU-NMS for YOLO (+1% MOTA)
- Docker container provided on Ubuntu 18.04
Deep learning models are usually the bottleneck in Deep SORT, making Deep SORT unusable for real-time applications. FastMOT significantly speeds up the entire system to run in real-time even on Jetson. It also provides enough flexibility to tune the speed-accuracy tradeoff without a lightweight model.
To achieve faster processing, FastMOT only runs the detector and feature extractor every N frames. Optical flow is used to fill in the gaps. YOLOv4 was trained on CrowdHuman (82% mAP@0.5) while SSD's are pretrained COCO models from TensorFlow. OSNet outperforms the original feature extractor in Deep SORT. FastMOT also re-identifies targets that moved out of frame and will keep the same IDs.
Both detector and feature extractor use the TensorRT backend and perform asynchronous inference. In addition, most algorithms, including Kalman filter, optical flow, and data association, are optimized using Numba.
| Detector Skip | MOTA | MOTP | IDF1 | IDS | MT | ML |
|---|---|---|---|---|---|---|
| N = 1 | 63.3% | 72.8% | 54.2% | 5821 | 867 | 261 |
| N = 5 | 61.4% | 72.2% | 55.7% | 4517 | 778 | 302 |
| Sequence | Density | FPS |
|---|---|---|
| MOT17-13 | 5 - 30 | 38 |
| MOT17-04 | 30 - 50 | 22 |
| MOT17-03 | 50 - 80 | 15 |
Performance is evaluated with YOLOv4 using py-motmetrics. Note that neither YOLOv4 nor OSNet was trained or finetuned on the MOT20 dataset, so train set results should generalize well. FPS results are obtained on Jetson Xavier NX.
FastMOT has MOTA scores close to state-of-the-art trackers from the MOT Challenge. Tracking speed can reach up to 38 FPS depending on the number of objects. On a desktop CPU/GPU, FPS is expected to be much higher. More lightweight models can be used to achieve better tradeoff.
Uses vanilla COCO-pretrained weights to make predictions on images, but you can train your own YOLOv4. The table below displays the inference times when using images scaled to 608x608 as inputs. The taken YOLOv4 measurements show the inference time of this implementation on Nvidia Jetson AGX Xavier.
| Backbone | GPU | FPS (max smoothed) | mAP@0.5 |
|---|---|---|---|
| YOLOv4-608 | AGX Xavier | 34 | 65.7 |
| YOLOv4-mish-640 | AGX Xavier | 25 | 67.9 |
-
IoU (intersect over union) - average intersect over union of objects and detections for a certain threshold = 0.24
-
mAP (mean average precision) - mean value of
average precisionsfor each class, whereaverage precisionis average value of 11 points on PR-curve for each possible threshold (each probability of detection) for the same class (Precision-Recall in terms of PascalVOC, where Precision=TP/(TP+FP) and Recall=TP/(TP+FN) ), page-11: http://homepages.inf.ed.ac.uk/ckiw/postscript/ijcv_voc09.pdf
mAP is default metric of precision in the PascalVOC competition, this is the same as AP50 metric in the MS COCO competition. In terms of Wiki, indicators Precision and Recall have a slightly different meaning than in the PascalVOC competition, but IoU always has the same meaning.
YOLOv4x-mish-640
YOLOv4-608
$ tensorboard --logdir=runs
CLICK ME - Pre-trained models
There are weights-file for different cfg-files (trained for MS COCO dataset):
FPS on RTX 2070 (R) and Tesla V100 (V):
-
yolov4x-mish.cfg - 640x640 - 67.9% mAP@0.5 (49.4% AP@0.5:0.95) - 23(R) FPS / 50(V) FPS - 221 BFlops (110 FMA) - 381 MB: yolov4x-mish.weights
- pre-trained weights for training: https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v4_pre/yolov4x-mish.conv.166
-
yolov4-csp.cfg - 202 MB: yolov4-csp.weights paper Scaled Yolo v4
just change
width=andheight=parameters inyolov4-csp.cfgfile and use the sameyolov4-csp.weightsfile for all cases:width=640 height=640in cfg: 66.2% mAP@0.5 (47.5% AP@0.5:0.95) - 70(V) FPS - 120 (60 FMA) BFlopswidth=512 height=512in cfg: 64.8% mAP@0.5 (46.2% AP@0.5:0.95) - 93(V) FPS - 77 (39 FMA) BFlops- pre-trained weights for training: https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v4_pre/yolov4-csp.conv.142
-
yolov4.cfg - 245 MB: yolov4.weights (Google-drive mirror yolov4.weights ) paper Yolo v4 just change
width=andheight=parameters inyolov4.cfgfile and use the sameyolov4.weightsfile for all cases:width=608 height=608in cfg: 65.7% mAP@0.5 (43.5% AP@0.5:0.95) - 34(R) FPS / 62(V) FPS - 128.5 BFlopswidth=512 height=512in cfg: 64.9% mAP@0.5 (43.0% AP@0.5:0.95) - 45(R) FPS / 83(V) FPS - 91.1 BFlopswidth=416 height=416in cfg: 62.8% mAP@0.5 (41.2% AP@0.5:0.95) - 55(R) FPS / 96(V) FPS - 60.1 BFlopswidth=320 height=320in cfg: 60% mAP@0.5 ( 38% AP@0.5:0.95) - 63(R) FPS / 123(V) FPS - 35.5 BFlops
-
yolov4-tiny.cfg - 40.2% mAP@0.5 - 371(1080Ti) FPS / 330(RTX2070) FPS - 6.9 BFlops - 23.1 MB: yolov4-tiny.weights
-
enet-coco.cfg (EfficientNetB0-Yolov3) - 45.5% mAP@0.5 - 55(R) FPS - 3.7 BFlops - 18.3 MB: enetb0-coco_final.weights
-
yolov3-openimages.cfg - 247 MB - 18(R) FPS - OpenImages dataset: yolov3-openimages.weights
-
csresnext50-panet-spp-original-optimal.cfg - 65.4% mAP@0.5 (43.2% AP@0.5:0.95) - 32(R) FPS - 100.5 BFlops - 217 MB: csresnext50-panet-spp-original-optimal_final.weights
-
yolov3-spp.cfg - 60.6% mAP@0.5 - 38(R) FPS - 141.5 BFlops - 240 MB: yolov3-spp.weights
-
csresnext50-panet-spp.cfg - 60.0% mAP@0.5 - 44 FPS - 71.3 BFlops - 217 MB: csresnext50-panet-spp_final.weights
-
yolov3.cfg - 55.3% mAP@0.5 - 66(R) FPS - 65.9 BFlops - 236 MB: yolov3.weights
-
yolov3-tiny.cfg - 33.1% mAP@0.5 - 345(R) FPS - 5.6 BFlops - 33.7 MB: yolov3-tiny.weights
-
yolov3-tiny-prn.cfg - 33.1% mAP@0.5 - 370(R) FPS - 3.5 BFlops - 18.8 MB: yolov3-tiny-prn.weights
yolov2.cfg(194 MB COCO Yolo v2) - requires 4 GB GPU-RAM: https://pjreddie.com/media/files/yolov2.weightsyolo-voc.cfg(194 MB VOC Yolo v2) - requires 4 GB GPU-RAM: http://pjreddie.com/media/files/yolo-voc.weightsyolov2-tiny.cfg(43 MB COCO Yolo v2) - requires 1 GB GPU-RAM: https://pjreddie.com/media/files/yolov2-tiny.weightsyolov2-tiny-voc.cfg(60 MB VOC Yolo v2) - requires 1 GB GPU-RAM: http://pjreddie.com/media/files/yolov2-tiny-voc.weightsyolo9000.cfg(186 MB Yolo9000-model) - requires 4 GB GPU-RAM: http://pjreddie.com/media/files/yolo9000.weights
- Set
resolutionandframe_ratethat corresponds to the source data or camera configuration (optional). They are required for image sequence, camera sources, and MOT Challenge evaluation. List all configurations for your USB/CSI camera:$ v4l2-ctl -d /dev/video0 --list-formats-ext
- To change detector, modify
detector_type. This can be eitherYOLOorSSD - To change classes, set
class_idsunder the correct detector. Default class is1, which corresponds to person - To swap model, modify
modelunder a detector. For SSD, you can choose fromSSDInceptionV2,SSDMobileNetV1, orSSDMobileNetV2 - Note that with SSD, the detector splits a frame into tiles and processes them in batches for the best accuracy. Change
tiling_gridto[2, 2],[2, 1], or[1, 1]if a smaller batch size is preferred - If more accuracy is desired and processing power is not an issue, reduce
detector_frame_skip. Similarly, increasedetector_frame_skipto speed up tracking at the cost of accuracy. You may also want to changemax_agesuch thatmax_age × detector_frame_skip ≈ 30
CLICK ME - Track custom classes
FastMOT supports multi-class tracking and can be easily extended to custom classes (e.g. vehicle). You need to train both YOLO and a ReID model on your object classes. Check Darknet for training YOLO and fast-reid for training ReID. After training, convert the model to ONNX format and place it in fastmot/models. To convert YOLO to ONNX, use tensorrt_demos to be compatible with the TensorRT YOLO plugins.
- Subclass
YOLOlike here: https://github.com/GeekAlexis/FastMOT/blob/4e946b85381ad807d5456f2ad57d1274d0e72f3d/fastmot/models/yolo.py#L94Note that anchors may not follow the same order in the Darknet cfg file. You need to mask out the anchors for each yolo layer using the indices inENGINE_PATH: path to TensorRT engine (converted at runtime) MODEL_PATH: path to ONNX model NUM_CLASSES: total number of classes LETTERBOX: keep aspect ratio when resizing For YOLOv4-csp/YOLOv4x-mish, set to True NEW_COORDS: new_coords parameter for each yolo layer For YOLOv4-csp/YOLOv4x-mish, set to True INPUT_SHAPE: input size in the format "(channel, height, width)" LAYER_FACTORS: scale factors with respect to the input size for each yolo layer For YOLOv4/YOLOv4-csp/YOLOv4x-mish, set to [8, 16, 32] For YOLOv3, set to [32, 16, 8] For YOLOv4-tiny/YOLOv3-tiny, set to [32, 16] SCALES: scale_x_y parameter for each yolo layer For YOLOv4-csp/YOLOv4x-mish, set to [2.0, 2.0, 2.0] For YOLOv4, set to [1.2, 1.1, 1.05] For YOLOv4-tiny, set to [1.05, 1.05] For YOLOv3, set to [1., 1., 1.] For YOLOv3-tiny, set to [1., 1.] ANCHORS: anchors grouped by each yolo layermaskin Darknet cfg. Unlike YOLOv4, the anchors are usually in reverse for YOLOv3 and tiny - Change class labels here to your object classes
- Modify cfg/mot.json: set
modelinyolo_detectorto the added Python class and setclass_idsyou want to detect. You may want to play withconf_threshbased on the accuracy of your model
- Subclass
ReIDlike here: https://github.com/GeekAlexis/FastMOT/blob/aa707888e39d59540bb70799c7b97c58851662ee/fastmot/models/reid.py#L51ENGINE_PATH: path to TensorRT engine (converted at runtime) MODEL_PATH: path to ONNX model INPUT_SHAPE: input size in the format "(channel, height, width)" OUTPUT_LAYOUT: feature dimension output by the model (e.g. 512) METRIC: distance metric used to match features ('euclidean' or 'cosine') - Modify cfg/mot.json: set
modelinfeature_extractorto the added Python class. You may want to play withmax_feat_costandmax_reid_cost- float values from0to2, based on the accuracy of your model
CLICK ME - Last steps
Only required if you want to use SSD (not in this case)
$ install/download_data.shThis includes both pretrained OSNet, SSD, and custom YOLOv4 WEIGHTS/ONNX models
$ cd ~/workspace/jetson && bash install/download_models.sh$ #cd ~/workspace/jetson && python3 utils/convert_DarkNet_to_ONNX.py --darknet-weights ./fastmot/models/yolov4.weights --onnx-weights ./fastmot/models/yolov4.onnx --cfg ./utils/cfg/yolov4.cfg --image-shape 608 608 --names ./utils/cfg/coco.names --batch-size 1 --add-plugins
$ cd ~/workspace/jetson && python3 utils/yolo_to_onnx.py --model fastmot/models/yolov4x-mish-640 --category_num 80On your TV, open a terminal and run everything to convert yolov4x-mish-640 from ONNX *.onnx to TensorRT *.trt and run inference
$ cd ~/workspace/jetson && sudo python3 main.py --test-type nano
$ cd ~/workspace/jetson && sudo python3 main.py --test-type xavier
$ cd ~/workspace/TSO_project/software/jetson && sudo /opt/conda/envs/school/bin/python3 main.py --test-type x86_64- USB webcam:
$ sudo python3 main.py --input_uri /dev/video0
- MIPI CSI camera:
$ sudo python3 main.py --input_uri csi://0
- RTSP stream:
On host:
On device:
$ #ffmpeg -re -loop -1 -i /dev/video0 -c copy -f rtsp -rtsp_transport tcp rtsp://localhost:1337/live.sdp $ vlc v4l2:///dev/video0 --sout '#transcode {vcodec=h264,acodec=mp3,samplerate=44100}:std{access=http,mux=ffmpeg{mux=h264},dst=127.0.0.1:1337/live.sdp}'
$ #sudo python3 main.py --input_uri rtsp://<user>:<password>@<ip>:<port>/<path> $ sudo python3 main.py --input_uri rtsp://samuel@192.168.55.100:1337/live.sdp - HTTP stream:
$ sudo python3 main.py --input_uri http://<user>:<password>@<ip>:<port>/<path>
- Image sequence:
$ sudo python3 main.py --input_uri img_%06d.jpg
- Video file:
$ sudo python3 main.py --input_uri video.mp4
- Use
--helpfor help and--output_urito save output - To disable the GStreamer backend, set
WITH_GSTREAMER = Falsehere - Note that the first run will be slow due to Numba compilation
- software/arduino-1.8.13/README.md
- software/jetson/jetson-containers/README.md
- software/arduino-1.8.13/portable/sketchbook/libraries/UArmForArduino/README.md
- software/arduino-1.8.13/portable/sketchbook/libraries/Servo/docs/readme.md
- software/arduino-1.8.13/portable/sketchbook/libraries/ESP32Servo/README.md
[Deep SORT with Deep Association paper]
[Omni-Scale Feature Learning for Person Re-Identification]
[Learning Generalisable Omni-Scale Representations for Person Re-Identification]