Looking for help testing Intel OpenVino support for NCS2

[NateMeyer]

I am working on a build that includes OpenVino support on x86 platforms ( #3768 ). I currently have it working with the iGPU on my i5 1135G7 laptop, more details are in the PR.

I was wondering if there was anyone who had a Neural Compute Stick 2 who wouldn't mind helping me test the VPU device integration?

I have an image pushed to github that should be ready to test. For now, this will only work on an x86 host.

NOW WITH ARM 64/32 BUILDS

docker pull ghcr.io/natemeyer/frigate:0.11.0-openvino-a3afb6f

With this image, you can add the VPU as a detector with the following config

detectors:
  ncs2:
    type: openvino
    device: MYRIAD

Additionally, you will have to add the following configuration to use the included OpenVino model.

model:
  path: /openvino-model/ssdlite_mobilenet_v2.xml
  width: 300
  height: 300
  input_tensor: [B, H, W, C]
  input_pixel_format: "bgr"
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

WIth this dev image, you need to add the following configuration to docker:

--device-cgroup-rule='c 189:\* rmw' -v /dev/bus/usb:/dev/bus/usb

or in your compose file:

device_cgroup_rules:
  - 'c 189:* rmw'
volumes:
  - /dev/bus/usb:/dev/bus/usb

Thanks!

[NickM-27]

I put this around on a few different existing discussions / issues, hopefully some users will be able to help.

[RLutsch]

happy to help but will have to be arm64 for me though

[RLutsch]

running detectors with ovArm works but the inference is slower than cpu getting 1200

[RLutsch]

lsusb returns

Bus 005 Device 023: ID 03e7:2485 Movidius Ltd. Movidius MyriadX
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00

[NateMeyer]

Ok, looks like you have the right NCS2. It is plugged into a USB2 port? Is that correct? Have you tried removing the devices section of your docker-compose?

I'm not surprised the ovArm detector is slow, I was mostly curious if I had the build correct and everything there to support it. Thanks!

[alucryd]

@RLutsch I second that you should not mount it as a device, rather a volume. Device does not work for some reason, intially the NCS2 is loaded as a high-speed device, the openvino driver actually unloads it and reloads it as a super-speed device to use it, and I guess this doesn't play nice with regular device mounting.

[RLutsch]

I tried all the usb's and all give usb 2.0

I'm using kubernetes and it's mounted as volume so should work, it's the same implimentation as what the coral uses.

I'll have to try compose again....

[MostHated]

Nice, I should be able to help out with this later today as well.

[NateMeyer]

Great! Let me know if you have any questions or problems

[alucryd]

Awesome, will give it a try over the weekend!

[NateMeyer]

Thanks, looks like the files are present in the build artifacts I copied over, just not in the right folder. I'll try getting them to the right place.

[NateMeyer]

@alucryd New image is up.

Since I'm building the openvino python wheel in this image, I'll probably switch to using that directly instead of the one from pypi. Then we should be able to have Arm support as well.

ghcr.io/natemeyer/frigate:0.11.0-openvino-f0d0893

[alucryd]

Apologies for the late reply, been bedridden for a few days. Unfortunately I get exactly the same error as #3797 (reply in thread).

Not sure how I can debug this, won't have time over the weekend but probably next week.

[NateMeyer]

No worries, hope you're feeling better. I should have an image by this weekend that includes the self-built openvino library. I'll take a look at what else could be causing that error.

Aside, building this library for arm takes forever on my laptop, but I finally got it to complete without error yesterday. Now we'll see if it works 😆 Might need to find a better way to manage this build.

[NateMeyer]

@alucryd Do you have the Myriad udev rules installed on your host? Have you successfully run other models on the NCS2 stick before?

[titilambert]

Hello, I just tested with ghcr.io/natemeyer/frigate:0.11.0-openvino-f0d0893 with core I7 3770S

And I add got:

E: [ncAPI] [    189878] [frigate.detecto] ncDeviceOpen:1013     Failed to find booted device after boot
Process detector:ncs2:
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/object_detection.py", line 121, in run_detector
    object_detector = LocalObjectDetector(
  File "/opt/frigate/frigate/object_detection.py", line 66, in __init__
    self.detect_api = OvDetector(
  File "/opt/frigate/frigate/detectors/openvino.py", line 16, in __init__
    self.interpreter = self.ov_core.compile_model(
  File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 266, in compile_model
    super().compile_model(model, device_name, {} if config is None else config)
RuntimeError: Failed to allocate graph: MYRIAD device is not opened.
[2022-09-24 12:49:51] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting frigate...

[alucryd]

@NateMeyer Thanks for the new image, still getting the same issue though. I do have the udev rules installed on the host, and the device is detected by the query_device sample program.

I'm a complete rookie when it comes to neural stuff and I can't find an easy way to test the device, tutorials I find all seem outdated, the first that pops up is ncappzoo, the documentation mentions a make run command, except run isn't even a valid target in the Makefile... This neural stuff looks like it's being made hard on purpose :/

I'll install the newest OpenVINO release, see if I can go further at least on the host side.

[NateMeyer]

I found this note regarding the issues with using the NCS2 inside Docker:
Use a Docker Image for Intel Movidius Neural Compute Stick and Intel Neural Compute Stick 2

Looks like I may have to rebuild libusb without udev support to make it work? The other workaround (not preferred, but faster to test) is to launch the docker image with --privileged -v /dev:/dev --network=host (gross)

[NateMeyer]

Another version of that page:
Running the Image on Intel® Neural Compute Stick 2

This page provides a sample command to test running the MYRIAD device in a prebuilt docker image. Could try this to see if it is working?

docker run -itu root:root --rm --device-cgroup-rule='c 189:\* rmw' -v /dev/bus/usb:/dev/bus/usb <image_name> \
/bin/bash -c "omz_downloader --name googlenet-v1 --precisions FP16 && omz_converter --name googlenet-v1 --precision FP16 && curl -O https://storage.openvinotoolkit.org/data/test_data/images/car_1.bmp && python3 samples/python/hello_classification/hello_classification.py public/googlenet-v1/FP16/googlenet-v1.xml car_1.bmp MYRIAD"

Also:

Only one NCS2 device connected to the host can be used when running inference in a container

[NateMeyer]

@alucryd, @titilambert, I updated the image once more, testing out building libusb without udev per the Intel documentation above. I also noted down what I expect you need to add to your docker-compose.yml file to pass through the device? See the updated instructions at the top of this thread and let me know if that makes sense. Thanks!

[alucryd]

@NateMeyer Thanks ! Still no luck though, I also tried intel's docker image and it doesn't even work either... So that's not on you.

I opened a bug report upstream, hopefully someone can help: openvinotoolkit/openvino#13288

[indieOR]

I was able to run this image with my Neural Compute Stick 2, and Frigate is reporting an inference speed of 10.

One word of caution if you are running the container inside of a VM. The Neural Compute Stick 2 does some odd things when in a VM. When the driver instantiates the device the device ID seems to change in /dev ont he host, making the mapping into the VM no longer relevant and the device disappears. This could explain some issues.

I have the container running directly on bare metal with the NCS2 and it's working without a hitch. I actually gave up Frigate earlier this year due to Coral unavailability, but have had this NCS2 sitting in my closet for almost 2 years. Glad to finally put it to use and be using Frigate again.

[NickM-27]

I fixed the formatting for you 👍

[NickM-27]

also 10 is the default value of inference speed, otherwise it is a calculated rolling average so odds are it may have just been showing the default before.

[NateMeyer]

@indieaz Is the NCS2 plugged into a USB 2 or 3 port? How many cameras are you running detection on? Curious as a reference for the latency number.

[NickM-27]

Yeah, now that we have multiple detectors will probably want to add inference numbers for each one to help users decide as multiple options may exist.

[indieOR]

@indieaz Is the NCS2 plugged into a USB 2 or 3 port? How many cameras are you running detection on? Curious as a reference for the latency number.

@NateMeyer I am running against 7 cameras right now.

NCS2 is plugged into a USB 3 port, and lsusb reports USB 3.2 link rate.

Bus 002 Device 003: ID 03e7:f63b Intel Myriad VPU [Movidius Neural Compute Stick                                                                                                                                                             ]
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.20

[n0rthking]

I don't have NCS2, but I have got it working on my server with Intel J4105 using the GPU device as detector. Interface speed is now 25 ms which is a nice improvement from the default CPU detector, which had about 150 ms interface speed. Additionally the CPU usage is now very low (about 15% in idle and 30% during detection, much better than full load with the default CPU detector).

[alucryd]

@NateMeyer I finally got it working! I tried with privileged, and with the cgroup rule, both worked, so cgroup rule it is.

I believe the issue all this time is that I was mounting /dev/bus/usb as a device instead of a volume. I've always passed devices that way so didn't think twice about it, but it seems that this doesn't play well with the NCS2.

Getting an inference speed of about 62, which is in line with @indieaz's results.

[NateMeyer]

@alucryd, for reference how many cameras were you running detection on?

[alucryd]

I only have 2 cameras.

[Jachimo]

Just wondering, is there an 0.11.1 version of the OpenVINO Docker image? Or should I still use 0.11.0? I have an NCS2 kicking around that I'd be happy to plug in and be able to use.

[NateMeyer]

Hey @Jachimo, this has now been merged into the dev branch and is available in the 0.12 Beta. Take a look at the release notes and documentation in #5156

[clamel77]

Great job, thank you very much. It was exactly what I was looking for.. I installed it on raspberry pi3 with Intel ncs2 usb stick.. :)

[amzaldua]

Can someone explain to me how to do this with an Intel HDU like I'm a baby?
My hardware is a NUC with a Celeron N5105 (10th generation)

[NickM-27]

it is all covered in the docs https://deploy-preview-4055--frigate-docs.netlify.app/configuration/detectors#openvino-detector

all you need to do is add that part to your config (assuming your GPU is visible in the container, if you are using hwaccel then it is already)

[amzaldua]

Thanks NickM-27.
It's just what I needed. I reduced my inference speed from 130-100ms to 32ms with this. Awesome.

[FrigateEnthusiast]

Is that 32ms with the Compute Stick? I had one sitting a drawer but thought but thought it would not compete with the Coral since it was USB2. Both are supposed to be 4 trillion ops once the bits arrive.

[amzaldua]

In my case, this value is only for the CPU.

[alucryd]

Has anyone used the stick on an aarch64 SBC recently ? I can't get it to work on a Pi 4 or Vim 3, both running a fully up to date Debian Bookworm. It's working fine on an Intel server with all parameters strictly identical.

[Stepyon]

Hey, is this still working and supported in 2025?
I have Intel ncs2 and Intel N100 mini PC with Frigate running in Docker wrapped with Proxmox LXC container.

[charredchar]

We did not selectively remove NCs2 support from Frigate, that was done via Intel removing support from their OpenVINO python package

Oh, I understand that. Sorry if my rant came off as sounding like it was directed at the Frigate team. 😅

You can of course still use Frigate 0.13 to run the NCS2

That's what I'm currently (struggling) to do now. Since I'm using older hardware I won't really benefit from the newer updates to OpenVINO anyway.
Is 0.13 the latest to support NCS2?

[NickM-27]

Is 0.13 the latest to support NCS2?

I believe so, you can find out by looking up the version of openvino that dropped support and what version of openvino is in each release

And thanks for the note, I put up a PR to remove all mentions of the NCS2 from the docs

[charredchar]

Thanks a ton for that information! I was able to (finally figure out how to) switch between a few different versions and test, for sure 0.13.2 is the last that worked with the NCS2. I found another discussion that confirmed it as well,
Considering the comment in that discussion... would it not be possible to build my own container with the older version of OpenVINO? I understand I'd lose some other benefits the latest version might have but if I'm not using those anyway it's worth experimenting with, right? And this could help anyone else reuse their NCS2 if they are in the same boat.
I tried to fork the repo and revert the change but obviously more has been changed since then and it's way over my head. I would hope there is an easier way, like just changing where Frigate pulls it's OpenVINO build from during the image build... Making it an ENV variable would be awesome.

[NickM-27]

Other packages require newer version. If you reduced the openvino version and made it so onnxruntime instead of onnxruntime-openvino is installed, it should work

[charredchar]

Thanks for the tip! I will keep trying to work on this later.
I've been wanting to utilize some of the newer features (along with ones yet to be released) in the newest versions but keep the older OpenVINO to utilize the hardware I have on-hand more effectively.

[Vlad1mir-D]

Here is rebuild of 0.16.2 with support for NCS2 which I'm using: https://github.com/Vlad1mir-D/frigate/pkgs/container/frigate/552639061?tag=0.16.2-ncs2

[FrigateEnthusiast]

Just a reminder, a Coral or NCS2 stick have to get popped into a USB 3 port or else you'll be disappointed.

[Stepyon]

Just a reminder, a Coral or NCS2 stick have to get popped into a USB 3 port or else you'll be disappointed.

I have all USB ports 3.0 (blue) on my MiniPC, but when I checked the speed of the bus and it really looks like 2.0:

I tried to insert it in all other ports but with the same result, they all sit on 001 or 003 bus. And other 2 high speed buses (001 and 004) are probably used for internal devices... disappointed now.

[FrigateEnthusiast]

Well just keep digging a little bit more... maybe a settings or driver change... "Grok: Why is my MiniPC usb port not running at USB 3 speeds?" :)

[FrigateEnthusiast]

See if this applies to your hw/sw but it is typical of some tweaks that have be done:

https://forums.linuxmint.com/viewtopic.php?t=448860

"I tried to apply only the following change:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt splash".
It worked. When the PC was restarted, the USB3 ports were detected and are currently working. Thanks for the precious help. I didn't think I could solve the problem so "easily".

[Stepyon]

I re-plugged it several times and it was recognized in 3.0 port now, also the device has changed the name from:
Intel Movidius MyriadX
to
Intel Myriad VPU [Movidius Neural Compute Stick]
Result is the same, 60ms.

See if this applies to your hw/sw but it is typical of some tweaks that have be done:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt splash".

Added this as well:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt" (Changed to Intel)

Still no difference, it is probably maximum what NCS2 can give :) Thanks for your help anyway.