[HW Accel Support]: Most Optimized ONNX Model for Nvidia GPU on Frigate 16 #19333
Replies: 4 comments 31 replies
-
|
The YOLO-NAS model will be considerably more accurate than the yolov7 model that the tensorrt detector used to run. Also, generally the inference time you see there is plenty fast especially since 2 detectors are being run at the same time. As long as you don't see any skipped fps then there is nothing lost with that inference speed, and the increased accuracy is a benefit. |
Beta Was this translation helpful? Give feedback.
-
|
I am using Yolo-NAS right now but it bogs down when lots of movement get detected, I've see some skipped fps at times unfortunately .. and GPU spiking 
|
Beta Was this translation helpful? Give feedback.
-
|
These speeds look quite fast and I don't see any skipped fps. This is similar to what I see on my 3070 running on 10 cameras. |
Beta Was this translation helpful? Give feedback.
-
|
In my case moving from tensorrt yolo-v7 to onnx yolo-nas increased the inferencing time from a mean of 20ms to 105ms 
|
Beta Was this translation helpful? Give feedback.
-
|
With a pure network approach possibly, I agree, but since all would be running on the same machine, sockets or some other IPC could provide close to 0 latency. |
Beta Was this translation helpful? Give feedback.
-
|
Potentially, but we're talking about 10 milliseconds. And in most cases of nvidia hardware the performance is comparable between those models and YOLO-NAS You might be able to use #19111 Personally, I don't see Frigate implementing / using anything other than ONNX directly. There are plenty of model options for all Nvidia hardware levels. |
Beta Was this translation helpful? Give feedback.
-
|
That's pretty cool, I also came across https://github.com/AnderssonPeter/YoloRest P.S Are you saying ~ 100ms is an acceptable inferencing time? Aside from the massively increased inferencing time, the CPU usage is also too high. When I run 0.16 with 2 cameras side by side with my 2x 0.15 instances with 12 cameras total, load average (on the right side) skyrockets 
P.S 2 Switched from yolov9 to yolo-nas, inferencing time doesn't change much |
Beta Was this translation helpful? Give feedback.
-
|
The tensorrt detector is currently still available for jetson (jp6) targets, is there a possibility this is going away at some point? For Jetson devices, the decreased performance with onnx is more problematic. Is it possible to make yolov9 or yolox work with the current tensorrt detector or would that require a significant rework? Frigate+ models will probably never be supported on tensorrt? |
Beta Was this translation helpful? Give feedback.
-
|
Jetson is community supported, so that's up to those that contribute to that, but the way Jetson is packaged it doesn't have the same difficulties as the Nvidia GPU detector. Some users have run yolo-nas via ONNX with success and I imagine yolov9 will run even better |
Beta Was this translation helpful? Give feedback.
-
|
I've switched to Yolox and have seen improvements over Yolo-Nas, not necessarily inference times but during multiple detections my GPU doesn't bog down as much, it still gets hit hard in comparison to direct tensorrt. That said, CPU definitely still has increased, I may be wrong on this but I believe in Tensorrt all the image prep to pass on tensorrt was offloaded to the GPU, now looks like CPU does resize to pass onto GPU. 
|
Beta Was this translation helpful? Give feedback.
-
|
In all cases the cropping and resizing (tensor preprocessing) is done on CPU beforehand the detector receives the input. |
Beta Was this translation helpful? Give feedback.
-
|
Just an update on moving to Yolox on the 16.0 release, I've found it's working without overwhelming my a4000, I don't find any challenges with detection confidence, inference is still slower over the previous tensorrt optimized detector, it's still more resource heavy I hope we get to see a tensorrt detector in the future, even if it's a monetized edition ;) |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the problem you are having
As some already have, I've moved from Tensorrt supported models to ONNX going into the latest 16.4 Beta and now Release candidate 1.
One thing that's been keeping me up at nigh is trying to get an optimized model that works well with ONNX and NVIDIA GPU.
Running an A4000 is no slouch but I'm starting to observe performance hit moving away from directly supported Tensorrt detectors. 3ms~ to 10ms~ on ONNX(depending on the models)
I'm not a developer or a coder and I truly want to acknowledge the amazing support that's been provided to this project.
So a big thank you for keeping up with all the questions!
I do have a big ask..., does anyone know a way to get me back closer to faster inference speeds?
I'm sure I'm not the only one invested into Nvidia GPU's that is now struggling to adapt to the new ONNX detector.
I like things fast and light, is there a way to optimize the yolo7-tiny or other model to run on ONNX detector that is optimized for my card?
Version
0.16.0-0b7a33d
Frigate config file
docker-compose file or Docker CLI command
Relevant Frigate log output
Relevant go2rtc log output
FFprobe output from your camera
root@4cf93339850f:/opt/frigate# ffprobe rtsp://127.0.0.1:8554/Front-Door bash: ffprobe: command not found root@4cf93339850f:/opt/frigate#Install method
Docker Compose
Object Detector
TensorRT
Network connection
Wired
Camera make and model
hivision
Screenshots of the Frigate UI's System metrics pages
Any other information that may be helpful
Need for Speed :
Beta Was this translation helpful? Give feedback.
All reactions