Fix mask visualization to exclude holes from polygon drawing

[Mr-Neutr0n]

Summary

Fixes #1771

When drawing segmentation masks with holes, the GenericMask.mask_to_polygons() method was returning all contours including internal hole contours. When these polygons were drawn, the holes were incorrectly filled in.

Problem

As reported in #1771:

1. Model predicts segmentation masks with holes (donut shapes, etc.)
2. Visualizer.draw_instance_predictions() creates GenericMask and calls overlay_instances
3. mask_to_polygons uses cv2.findContours with cv2.RETR_CCOMP which returns both external and internal (hole) contours
4. All contours are returned as polygons without filtering
5. When drawing, hole contours are drawn as filled polygons, incorrectly filling in the holes

Before fix:

Solution

Filter out hole contours by checking the hierarchy returned from cv2.findContours. With cv2.RETR_CCOMP:

• External contours have hierarchy[i][3] == -1 (no parent)
• Hole contours have hierarchy[i][3] >= 0 (have a parent)

Now only external contours (boundaries) are returned as polygons, while holes are correctly left empty.

Changes

Modified GenericMask.mask_to_polygons() to filter contours based on hierarchy:

# Filter out hole contours (those with a parent, i.e., hierarchy[i][3] >= 0)
# Only keep external contours (hierarchy[i][3] == -1)
res = [contours[i].flatten() for i in range(len(contours)) if hierarchy[i][3] < 0]

Test plan

import numpy as np
from detectron2.utils.visualizer import GenericMask

# Create a mask with a hole (donut shape)
mask = np.zeros((100, 100), dtype=np.uint8)
mask[20:80, 20:80] = 1  # outer square
mask[35:65, 35:65] = 0  # inner hole

gm = GenericMask(mask, 100, 100)
print(f"Has holes: {gm.has_holes}")  # Should be True
print(f"Number of polygons: {len(gm.polygons)}")  # Should be 1 (only outer boundary)

[Mr-Neutr0n]

hi, wanted to follow up on this. the polygon drawing bug causes holes to incorrectly show up as filled regions in mask visualizations. the fix is minimal — just filters out inner contours. let me know!