Keeps Searching node when using nncf.IgnoredScope

[qiuzhewei]

Hi,
I am trying to quantize my model using PTQ and apply nncf.IgnoredScope to gain some performance refering to yolov11-object-detection.ipynb. The problem is, for some reason, when I specify the input and output names in nncf.IgnoredScope, the scripts keeps running for a long time. When I specify a relatively small subgraph, it runs fast. Any clue to fix that?
If you need other infomation, please let me know.
Thanks

[zulkifli-halim]

The nncf.IgnoredScope does not directly improve performance. It provides an option to specify portions of the model to be excluded from compression while maintaining model accuracy during quantization.

The reason behind this issue could be that the NNCF is processing a significantly larger model or a more complex computational graph compared to when you explicitly select a small subgraph.

Can you share the code? And what model are you using?

[qiuzhewei]

Thanks for reply.
I am using transformer based video classification model. For security consideration, uploading model is strictly prohibited. But I can provide more information:
github: https://github.com/OpenGVLab/VideoMAEv2
covert code:

import onnx
import torch
from torchvision import datasets
from torchvision import transforms

import nncf


val_dataset = datasets.ImageFolder(
    root = "./model_compression/calib_data/template_feature",
    transform = transforms.Compose([
    transforms.Resize([512,512]),
    transforms.ToTensor(),
    transforms.Normalize(mean = (0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
    ])
)
val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1, shuffle=False)

onnx_path = "./model_compression/models/template_feature/onnx/isc_ft_v107.onnx"
model = onnx_model = onnx.load(onnx_path)
input_name = onnx_model.graph.input[0].name

def transform_fn(data_item):
    images, _ = data_item
    return {input_name: images.numpy()}

ignored_scope = nncf.IgnoredScope(
        subgraphs=[
            nncf.Subgraph(inputs=["/backbone/blocks.5/blocks.5.8/Add",
                                  ],
                        outputs=["/backbone/blocks.5/blocks.5.15/Add"]),
            
            nncf.Subgraph(inputs=["/GlobalAveragePool", #/backbone/blocks.5/blocks.5.10/Add
                                  "/Shape"],
                        outputs=["/Div"]),
            ]
        )

calibration_dataset = nncf.Dataset(val_loader, transform_fn)
onnx_quantized_model = nncf.quantize(model,
                                     calibration_dataset,
                                     subset_size=200,
                                     preset=nncf.QuantizationPreset.MIXED,
                                     ignored_scope=ignored_scope,
                                     )


int8_model_path = "models/template_feature/onnx/model_int8_200_ignore_508_515.onnx"
onnx.save(onnx_quantized_model, int8_model_path)

In subgraphs, if I specify relatively "adjacent" inputs and outputs, say /backbone/blocks.5/blocks.5.8/Add and /backbone/blocks.5/blocks.5.10/Add, the searching procudure is fast, whereas for inputs and outputs far apart, /backbone/blocks.5/blocks.5.8/Add and /backbone/blocks.5/blocks.5.17/Add, it keeps searching for days.

[zulkifli-halim]

Hi @qiuzhewei

The adjacent nodes are fast because the input and output nodes are close. Based on your code above, we can see that the nodes are separated by many layers and far apart in the graph. The NNCF needs to find the possible path between the input and output. Making the subgraph search takes more time to complete.

I'm checking with the dev team on what can be done for this particular concern.

[qiuzhewei]

Thanks a lot.

[zulkifli-halim]

Hi @qiuzhewei

Since this is an NNCF-related issue, I recommend that you open a new issue at NNCF repository, which has a better target audience for this type of problem.

I recommend enabling debug mode to identify where the NNCF quantization is hanging, and encourage you to share those debug logs when posting in the NNCF repository.

import logging
from nncf.common.logging.logger import set_log_level
 
set_log_level(logging.DEBUG)

[qiuzhewei]

Got you, thanks you!