Does it support running dinov3 distillation on NPU?

[elfpattern-code]

🤔 What’s your question?

I wanna run a dinov3 distillation yolo11 in NPU

🧠 What did you try?

Not supported now

📌 Related code or context (if any)

Optional, but helpful for understanding your context.

# Related snippet if relevant

[liopeer]

Hi @elfpattern-code,

Distillation requires significant compute resources and runs in PyTorch, it is therefore recommended to use GPUs from Nvidia for the best support. However, it is totally possible to

1. distill DINOv3 into your YOLO backbone;
2. fine-tune YOLO with the Ultralytics library (or similar);
3. export your model to ONNX;
4. compile the ONNX model to a format compatible with your NPU.

In order to better help you, you can maybe answer me the following questions:

• What is the manufacturer/type of NPU that you were planning to use?
• Which exact YOLO11 model are you trying to use? (size of the model, and the task e.g. object detection, instance segmentation etc.)
• What is your motivation for using distillation? Is your data distribution very different from COCO/ImageNet etc.? How much unlabeled images/labeled images do you have?

You can btw. also check out the LightlyTrain line-up of models that are ready for fine-tuning: https://docs.lightly.ai/train/stable/quick_start_object_detection.html

[205418367]

Hi @elfpattern-code,

Distillation requires significant compute resources and runs in PyTorch, it is therefore recommended to use GPUs from Nvidia for the best support. However, it is totally possible to

1. distill DINOv3 into your YOLO backbone;
2. fine-tune YOLO with the Ultralytics library (or similar);
3. export your model to ONNX;
4. compile the ONNX model to a format compatible with your NPU.

In order to better help you, you can maybe answer me the following questions:

• What is the manufacturer/type of NPU that you were planning to use?
• Which exact YOLO11 model are you trying to use? (size of the model, and the task e.g. object detection, instance segmentation etc.)
• What is your motivation for using distillation? Is your data distribution very different from COCO/ImageNet etc.? How much unlabeled images/labeled images do you have?

You can btw. also check out the LightlyTrain line-up of models that are ready for fine-tuning: https://docs.lightly.ai/train/stable/quick_start_object_detection.html

Hello, I also intend to distill DINOv3 into my YOLO backbone.
1.I need to deploy the YOLO model on an RK3588 NPU.
2.I have a large amount of both unlabeled and labeled grayscale image datasets (single-channel), which differ significantly from the COCO dataset.
May I ask, how much better would the distillation results be compared to those obtained by training with the Ultralytics library?

[liopeer]

May I ask, how much better would the distillation results be compared to those obtained by training with the Ultralytics library?

This is not easy to answer in full generality, however the performance gains are usually not very large for natural images, i.e. COCO-styles images, and noticeable for image distributions that are very far from COCO. Additionally, larger models generally seem to be benefitting a bit more from distillation. We currently also have strong signals, that task-specific training of the teacher model with subsequent distillation can yield better features in the student for the task at hand, i.e. it can further boost the improvement if you first fine-tune your DINOv2/DINOv3 model on detection/segmentation etc., and only then distill it into your YOLO model.

[205418367]

fine-tune the DINOv2/DINOv3 model on tasks such as detection and segmentation.

How can this process be implemented? My understanding is as follows:

1.First, fine-tune the DINOv3 model on your own labeled dataset to obtain exported_last.pt.
lightly_train.train_object_detection(
out="out",
model="dinov3/vitt16-ltdetr-coco"
)

2.Then, distill exported_last.pt into the YOLO backbone.
model = YOLO(myaml)
lightly_train.pretrain(
out="out/my_experiment",
data="my_data_dir",
model=model,
method="distillation",
method_args={
"teacher": "dinov2/vitl14",
"teacher_weights": "out/exported_last.pt"
}
)

Will this approach be effective?
You directly use your own unlabeled dataset to perform self-supervised training on YOLO, and then fine-tune the YOLO model.

model = YOLO(myaml)
lightly_train.pretrain(
out="out/",
data="images",
model=model,
method="dinov2"
)

[liopeer]

Since we're also only exploring this it's not 100% convenient just yet, so I don't think that your approach will directly work. I think you will need to manually extract and save the DINOv3 backbone:

lightly_train.train_object_detection(
    out="out",
    model="dinov3/vitt16-ltdetr-coco",
    ...
)

objdet_model = lightly_train.load_model("out/exported_last.pt")
dinov3_vit_or_convnext = od_model.backbone._model_wrapper.get_model()
torch.save(dinov3_vit_or_convnext.state_dict(), "out/objdet_backbone.pt")

model = YOLO(myaml)
lightly_train.pretrain(
    out="out/my_experiment",
    data="my_data_dir",
    model=model,
    method="distillation",
    method_args={
        "teacher": "dinov2/vitl14",
        "teacher_weights": "out/objdet_backbone.pt"
    }
)

# and then your YOLO fine-tuning code here

Let me know if you run into any issues!