Different models for Coral

[tom0010]

Is anyone using a different model for their Coral, and if so, which are you using?

I've seen some discussions around people using different models but interested to see if there's anything better than the default model that frigate ships with for the Coral.

I'm interested in better accuracy and I've seen that some yolo models are slightly better, but the discussions seem around Frigate 13.
Example here: #11548

[hawkeye217]

Coral devices are specifically designed to accelerate TensorFlow Lite models that have been quantized to 8-bit integers and compiled using Google's Edge TPU compiler.

YOLO models are indeed more accurate because they are usually implemented in frameworks like PyTorch or Darknet and use 32-bit floating-point operations.

To run a YOLO model on Coral, you would need to convert it to TensorFlow/TensorFlow Lite format, quantize it to INT8, and ensure all operations are supported by the Edge TPU - a process that often requires significant model modifications and may not preserve the original model's accuracy. Additionally, many YOLO architectures contain operations or layer types that aren't supported by the Edge TPU's specialized instruction set, making direct deployment impossible without substantial reworking of the model architecture.

And finally, many of the freely available TensorFlow and YOLO models are all based on the same COCO dataset, which has not been trained on images specific to security cameras.

If you're looking for more accuracy, you might consider new hardware - like a Hailo or MemryX. Support for the Hailo 8/8L already exists in Frigate 0.16 while support for the MemryX MX3 is coming in Frigate 0.17. And while it's not a separate piece of dedicated hardware, an Intel iGPU can also run many YOLO variants very well.

[AmirHosseinOmidi0]

what is the default model for coral ?
ssd moiblenet?

[NickM-27]

In Frigate the default coral model is ssdlite mobiledet

[dbro]

Yes you can try running a YOLO v9 "s" 320x320 model on Coral using the model file, detector plug-in, and labels from this repo:
https://github.com/dbro/frigate-detector-edgetpu-yolo9
As @hawkeye217 said, it took some effort to get it to run well on the Coral, although the model structure was not changed, just the activation function was changed to ReLU6 instead of SiLU.

The YOLO v9 model's detection latency is about 11ms (vs 7ms for mobiledet). I see fewer false positives than I did when running mobiledet.

[tom0010]

@dbro I’ve been following your post on Reddit, and I was waiting for a 320 model. I’ve just deployed this on version 16.1, and getting around 8-9ms interference speed. Thanks for the work, would much rather run the coral than the iGPU.

[dbro]

FYI @tom0010 with yesterday's release there is a 512x512 version of YOLO v9 "s" for Coral. Detection takes about 21ms on my system.
https://github.com/dbro/frigate-detector-edgetpu-yolo9/releases/tag/v1.5

[tue-kyndal]

Hi @dbro
I have a question about models for for frigate,manly regarding LPR and the object ['license_plate']

Using the Secondary LPR Pipeline (Without Frigate+) works fine on my system, but there are no recording or snapshots available.

I don't have much of a use-case, but it would be fun to get a notification with a snapshot of the car and license plate number.
Or store it in a database for safe keeping. Neighborhood watch stuff.

When using the primary setup for LPR, to enable recording and snapshots etc, I get the error:

Lpr-Primary-Camera is configured to track ['license_plate'] objects, which are not supported by the current model.

Nothing bad said about frigate + and all the work done here!

I just don't what to subscribe for stuff like that.
I'm only doing this for fun and to learn about AI, and would rather make my own local native model work.

I would like to run a local model for Native license_plate Detection.

I have a PCI based google coral edge TPU, the system is running on a proxmox lxc as a docker.

According to the documentation here, all you need is a native model, but how do you install that?

Using a Frigate+ (or Native license_plate Detecting) Model

Users running a Frigate+ model (or any model that natively detects license_plate) can take advantage of license_plate detection. This allows license plates to be treated as standard objects in dedicated LPR mode, meaning that alerts, detections, snapshots, and other Frigate features work as usual, and plates are detected efficiently through your configured object detector.

[tom0010]

You do not need to have license_plate in the object list.
In settings you need to go to enrichments and enable LPR.
Then in the main config file:

lpr:
  enabled: true 

LPR only adds a sub label to the car label. Best way to check it’s working is to do a search for cars and then you should see some with the captured number plate. Bear in mind that it doesn’t always capture the plate.

@dbro just FYI this runs a new yolov9 model just for LPR, would be good to integrate the two.

[dbro]

Thanks for correcting me, sorry to have added incorrect information to the thread!

If the LPR model ONNX file and source code are available (with a permissive license), I could look into getting it running on Coral. Let me know where it can be found if that's possible.

[tue-kyndal]

Licence plate recognition software is used everywhere, and even included in cheap dedicated LPR cameras,

How can a LPR model not be available and open source? I don't get why there is not a ton on git-hub already?

This issue feels more seems like a license problem.

[NickM-27]

There are multiple open source license plate models, including the one that Frigate uses in its secondary pipeline.

However, there aren't any models that I know of that are open source and provide COCO labels along with license_plate in the same model. You can of course train your own if you want, or you can pay for and use Frigate+ which provides this.

[tom0010]

After a quick look, it looks like they are hosted here:

• https://github.com/hawkeye217/paddleocr-onnx

Frigate auto downloads the model size depending on the users config.

However, it looks like your new model could potentially be utilised. Assuming it would need to be trained? At the very least we’d have to change the object list to allow license_plate as per the docs: https://docs.frigate.video/configuration/license_plate_recognition#model-requirements

It all boils down to running one model or two yolov9 models. One on the coral and one for LPR:

Screenshot of the LPR CPU metrics:

[dbro]

Well I'd be ready to offer advice to someone who wants to take on porting any LPR models to run on Coral, but I don't want to put in the time to do it myself because I don't use it.

Here is a starting direction if someone wants to give it a go. Gather these things:

• PyTorch version of model (usually a .pt file that is about 50-100MB)
• Training data with lots of labelled images of license plates in various settings
• Source code for the model. The YOLO v9 repo I used is here: https://github.com/WongKinYiu/yolov9

And then the process is in this Colab notebook: https://colab.research.google.com/drive/1RAZqCHuDWFdFxqFpL78OVfZQvnGoM6rH

[tue-kyndal]

Training data with lots of labelled images of license plates in various settings

There are an abundance of plate examples to harvested here, especially the odd ones not following the strict formats:
This also shows that a tight format may not be such a good idea after all.
Today you can by none standard plates if you want :O)

Danish number plate FB page
Sjove nummerplader

More international number plate FB page
Private Number Plates Spotted, For Sale, Wanted & General Advice

Professional archive, sorted based on country - but making the model to national may not be a good idea- foreign cars are known to drive across borders :O)
Photos of vehicles and license plates

Perhaps if you asked nicely, they could provide a set of training data with labelled images of license plates, all ready to go!
Or access to the site API for AI training!

I hope some would take on this challenge.
A coral edge license plates model for frigate would be awesome ;O)