[QUESTION] Clarification on model weights initialization (DINOv3 vs. COCO vs. Scratch)

[higo522]

🤔 What’s your question?

I'm trying to run a pretraining ablation study and want to confirm my understanding of the DINOv3 pretrained object detection models [ex. dinov3/convnext-small-ltdetr-coco]. Is my below understanding correct?:

• Backbone (ex. ConvNeXt-Small): DINOv3 self-supervised pretraining (LVD-1689M) → COCO fine-tuning
• Detector head (encoder + decoder): COCO fine-tuning only

For a clean ablation I'd like to compare three levels:

1. From scratch – no pretraining at all
2. DINOv3 backbone only without COCO fine-tuning
3. Full pretraining – DINOv3 and COCO fine-tuning (current -coco setup)

How exactly would I be able to experiment with these configurations? Any guidance will be appreciated.

🧠 What did you try?

I traced through the source code and found that the model weights are downloaded from _models/dinov3/hub/backbones.py

📌 Related code or context (if any)

# Related snippet if relevant

[yutong-xiang-97]

Hi @higo522,

Thank you for your interest in LightlyTrain!

Your understanding is mostly correct. dinov3/convnext-small-ltdetr-coco's backbone is from DINOv3 LVD-1689M with COCO fine-tuning. Specifically, the backbone was not frozen when it is fine-tuned.

For the configs you should use for your experiments:

1. From scratch – no pretraining at all

For this we don't expose any parameter in the train_object_detection command or provide any model checkpoints. Alternatively you can do it by specifying backbone_args:

lightly_train.train_object_detection(
    out="out/scratch",
    data=...,
    model="dinov3/convnext-small-ltdetr",
    model_args={
        "backbone_args": {"pretrained": False, "weights": None},
    },
)

this will call model.init_weights() when building the model instance.

2. DINOv3 backbone only without COCO fine-tuning

For this you could use "dinov3/convnext-small-ltdetr" which starts from a DINOv3 LVD-1689M backbone + random head, i.e. no training was done on Lightly's side.

3. Full pretraining – DINOv3 and COCO fine-tuning (current -coco setup)

For this you could use "dinov3/convnext-small-ltdetr-coco" which starts from a DINOv3 LVD-1689M backbone + random head, and was then trained supervised on COCO without freezing the backbone as explained above.

Let me know if this clarifies things!

Best,
Yutong

[higo522]

Thank you very much for the clarification, it makes sense now.

Following up on ablation studies — I was wondering if there was a way to isolate the contribution of DINOv3 by comparing it against a COCO-only baseline (LT-DETR pretrained on COCO but initialized from random weights rather than DINOv3). The motivation behind this is to evaluate how well DINOv3 transfers to detection tasks in addition to the default bounding-box pretraining (COCO supervision).
From the three levels you have explained I understand there is no checkpoint for this configuration, since DINOv3 initialization was the starting point for COCO fine-tuning. However, is there a supported way to reproduce this (without training LT-DETR from scratch on the full COCO dataset myself)?

Again I really appreciate the support and guidance.

[liopeer]

Hi @higo522,
I understand your motivation, and unfortunately you are right in your understanding that we don't have a checkpoint for this. I don't see an obvious way on how to do this without training LT-DETR from scratch only on COCO.

[higo522]

Thank you for your response. I made the following table to summarize the three levels w.r.t the backbone and detection head (encoder+decoder); can you please confirm my understanding is correct?

<style> </style>

	Backbone (ConvNeXT-Small)	Detection head (RT-DETRv2)
Level 1	—	—
Level 2	DINOv3	—
Level 3	DINOv3 → COCO	COCO

Moreover, I ran an ablation study across these 3 levels and evaluated their performance on a unique task: detecting animals in thermal drone imagery (a held-out test set was used). As expected, there was a significant jump from Level 2 to 3 (+ COCO fine-tuning). However, I found that the Level 1 and 2 models yielded identical performance, despite the DINOv3 pretrained backbone for Level 2.

I would highly appreciate your insights on why this might be happening. Is this due to the significant domain gap between the DINOv3 pretraining and thermal drone imagery, or is detection-specific pretraining (COCO) required to fully utilize the benefits of the DINOv3 backbone?

Thank you again for your time and expertise!

[liopeer]

Yes, those are exactly the 3 levels!

However, I found that the Level 1 and 2 models yielded identical performance, despite the DINOv3 pretrained backbone for Level 2.

Oh wow, that is also surprising to me! Can you give me some more information on the dataset? Is it single-channel monochrome and you broadcast it to 3 channels? How many labeled images does it have? Is it open source so that we could have a look at it?

In our experience DINOv3 helps in pretty much any domain you throw at it. It definitely performs best on natural image tasks, but its training corpus likely included pretty much anything in existence, so this is definitely surprising and I'd love some more information! What could be an issue though is our current training recipe: It might be that for your task we should use even lower learning rates for the backbone – or entirely switch to some kind of progressive unfreezing, where the backbone's learning rate essentially starts at 0 and is slowly and continuously raised.

Feel free to also reach out on LinkedIn or Discord, would be really interested in more details!

[higo522]

Thank you for the clarification! A lower backbone learning rate definitely makes sense for this domain gap.

Also, I am using this model for research and wanted to finally confirm if my description of LT-DETR (dinov3/convnext-small-ltdetr-coco ) is correct. Please let me know if anything is inaccurate:
"We adopted the LT-DETR architecture from the LightlyTrain framework, which implements and extends RT-DETRv2 with DINO pretrained backbones (Lightly Team, n.d.). Specifically, the pipeline consists of a DINOv3 pretrained backbone (ConvNeXt-Small) coupled with an RT-DETRv2 detection head comprised of its transformer encoder and decoder. The model was initialized with a ConvNeXt-Small backbone pretrained on the LVD-1689M dataset using DINOv3 self-supervised learning, which was then fully fine-tuned (end-to-end) with supervised learning on the COCO dataset, without freezing the backbone alongside the detection head."

Regarding the dataset: yes, it was originally single-channel thermal infrared broadcasted to 3 channels (identical values) and I have around 5000 labeled images across multiple survey periods. The dataset is tied to an unpublished manuscript so it is not publicly available yet; can I send you a message on linkedin/discord so we can discuss it further?

Thanks again for your help!

[liopeer]

The description matches! And feel free to send me a message.