Kornia backend rejects 12 transform documented in `aug_configs`

[adhavan18]

Search before asking

• I have searched the RF-DETR issues and found no similar bug report.

Bug

Twelve of the transform names listed under Transform Categories in aug_configs.py raise ValueError on the Kornia backend while working on Albumentations. Because "cpu"/"auto" resolves to Kornia when it is installed and CUDA is available, the same aug_config can train fine on a CPU box and hard-fail on a GPU box.

I built every documented name on both backends on a700efc:

documented name	kornia	albumentations
HorizontalFlip	OK	OK
VerticalFlip	OK	OK
Rotate	OK	OK
Affine	OK	OK
ShiftScaleRotate	ValueError	OK
RandomCrop	ValueError	OK
CenterCrop	ValueError	OK
RandomResizedCrop	ValueError	OK
Perspective	ValueError	OK
ElasticTransform	ValueError	OK
GridDistortion	ValueError	OK
ColorJitter	OK	OK
HueSaturationValue	ValueError	OK
RandomBrightnessContrast	OK	OK
GaussianBlur	OK	OK
GaussNoise	OK	OK
Blur	ValueError	OK
CLAHE	ValueError	OK
Sharpen	ValueError	OK
Equalize	ValueError	OK

To be fair to the docstring: the Kornia GPU Backend section right below lists exactly the eight that do work, and it is accurate. The mismatch is that the categories list above it reads as the menu of available transforms with no backend caveat, and the two lists disagree. All four shipped presets stay inside the supported eight, so nothing is broken out of the box.

The reason it bites is the failure mode. AlbumentationsWrapper resolves names dynamically (getattr(alb, name, None)), so it picks up anything Albumentations ships. kornia_transforms._REGISTRY is a fixed eight-entry dict, and an unlisted key is a hard ValueError at pipeline-build time, not a warning or a fallback.

#1227 is one instance of this (ToGray, PR #1249).

Environment

• rfdetr at a700efc (1.9.0), installed with -e .
• kornia 0.8.3, albumentations 2.0.8, torch 2.13.0+cpu
• Windows, CPU only. Kornia builds pipelines identically on CPU, so no GPU is needed to reproduce.

Minimal reproducible example

from rfdetr.datasets.kornia_transforms import build_kornia_pipeline
from rfdetr.datasets.transforms import AlbumentationsWrapper

cfg = {"HueSaturationValue": {"p": 0.5}}

AlbumentationsWrapper.from_config(cfg)   # fine
build_kornia_pipeline(cfg, 560)          # ValueError: Unknown augmentation key 'HueSaturationValue'

Possible directions

Roughly in increasing order of effort, and entirely your call which is wanted:

1. Caveat the categories list, so it says which names are Albumentations-only and points at the Kornia table.
2. Map the ones with a direct Kornia equivalent: Perspective -> K.RandomPerspective, Blur -> K.RandomBoxBlur, Sharpen -> K.RandomSharpness, Equalize -> K.RandomEqualize, HueSaturationValue -> K.ColorJiggle (hue/saturation only).
3. Leave the crops and ElasticTransform/GridDistortion alone, or treat them separately: they interact with boxes and masks, so the semantics need a decision rather than a mapping.

I am happy to send PRs for any of these, sequenced rather than all at once. I would rather not guess at the box-interaction semantics for group 3 without a steer from you.

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[Borda]

Thinking if this project would help with the limitation - https://github.com/Borda/fuse-augmentations

[adhavan18]

Interesting, thanks. I installed it (0.9.0.dev0) and ran the twelve through AlbumentationsAdapter.category to see what it actually knows about, rather than guess:

transform	category
HueSaturationValue	POINTWISE
ShiftScaleRotate	GEOMETRIC_INTERP
Perspective	PROJECTIVE
RandomResizedCrop	CROP_RESIZE_FIXED
RandomCrop, CenterCrop	SPATIAL_KERNEL
ElasticTransform, GridDistortion	SPATIAL_KERNEL
Blur, CLAHE, Sharpen, Equalize	SPATIAL_KERNEL

So it recognises all twelve, which is more than I expected. For reference, the eight rf-detr already supports land as GEOMETRIC_EXACT (flips), GEOMETRIC_INTERP (Rotate, Affine), POINTWISE_LINEAR (RandomBrightnessContrast), and SPATIAL_KERNEL (ColorJitter, GaussianBlur, GaussNoise).

That last point is the one I would flag before treating this as the answer. SPATIAL_KERNEL is doing double duty: it covers pixel-only ops that cannot move a box (GaussianBlur, Blur, CLAHE, Sharpen, Equalize) and genuinely spatial ones that can (RandomCrop, CenterCrop, ElasticTransform, GridDistortion). rf-detr always carries boxes, and often masks and keypoints, so the split between those two matters here in a way the single category does not express. The README says as much:

Use auxiliary targets only with explicitly supported spatial transforms. An unknown crop, resize, or other spatial passthrough can modify the image while leaving a mask, box, or keypoint tensor stale.

Which lines up with how I split the list in the issue: the pixel-level group (HueSaturationValue, Blur, CLAHE, Sharpen, Equalize) is the low-risk half either way, and the crops plus ElasticTransform/GridDistortion are the half that needs a decision about target handling regardless of which engine executes them.

Given it is Alpha and explicitly not a drop-in for Compose, my instinct is that it is a good fit for the fusion/perf axis but a large dependency to take on purely to close a name-coverage gap. Happy to be wrong on that, you know the roadmap.

Either way, the two directions are independent, so I am glad to send the pixel-level five as direct Kornia mappings now (K.ColorJiggle for hue/saturation, K.RandomBoxBlur, K.RandomSharpness, K.RandomEqualize) and leave the geometric half alone until the target-handling question is settled, whichever engine it ends up going through. Say the word and I will open it.

[adhavan18]

Opened #1277 for the pixel-level group.

One change from what I said above: I offered five and have sent four. HueSaturationValue is left out because the mapping is not faithful. Albumentations shifts hue, saturation and value additively (hue_shift_limit in OpenCV hue units, sat_shift_limit on a 0-255 scale), while Kornia's ColorJiggle scales them multiplicatively. Mapping one to the other would change what an existing config means on the GPU path rather than port it, so it still raises a clear ValueError and the docs now record why.

The four that went in are Blur to K.RandomBoxBlur, Sharpen to K.RandomSharpness, Equalize to K.RandomEqualize and CLAHE to K.RandomClahe. None of them can move a box, mask or keypoint, so the auxiliary-target question does not arise for any of them.

That leaves the geometric half of the issue open, which is where the target-handling decision actually matters. Happy to take that on once you have a view, or to leave it if the fuse-augmentations direction covers it.

[adhavan18]

Thanks for merging #1277, and for reopening it after I closed it by mistake.

That leaves the geometric half of this issue, which is blocked on a decision rather than on code, so I would rather ask than guess.

The eight still raising on Kornia are ShiftScaleRotate, RandomCrop, CenterCrop, RandomResizedCrop, Perspective, ElasticTransform and GridDistortion, plus HueSaturationValue for the separate reason in #1277.

Kornia has a direct equivalent for most of them (K.RandomAffine, K.RandomCrop, K.CenterCrop, K.RandomResizedCrop, K.RandomPerspective, K.RandomElasticTransform), so the mapping itself is not the hard part. What matters is that all of them move boxes, and often masks and keypoints, which is exactly the case the README warns about:

An unknown crop, resize, or other spatial passthrough can modify the image while leaving a mask, box, or keypoint tensor stale.

Kornia's AugmentationSequential does transform auxiliary targets when the data keys are declared, and build_kornia_pipeline already passes with_masks and include_keypoints, so in principle these could go in with targets handled properly rather than as passthroughs. What I do not want to do is add seven names that quietly desynchronise a mask on someone's segmentation run.

Two questions, and I am happy either way:

1. Do you want these on the Kornia backend at all, or is the intent to leave the geometric group to the Albumentations path and just document that clearly? The latter is a much smaller change and I could send it as a docs-only PR.
2. If you do want them, would you rather have them in one PR or split by risk, for example the crops separately from ElasticTransform/GridDistortion, which are the two whose box behaviour I would want tests around before trusting?

If it is easier to decide against a concrete diff, I can put up the two lowest-risk ones (Perspective and RandomResizedCrop) with box round-trip tests, and we can use that to settle the pattern for the rest.

[adhavan18]

@Borda picking this back up. #1277 covered the pixel-level group and is merged; the geometric half is still open and it's waiting on one decision rather than on code, so I'll state it as concretely as I can.

eight names still raise on Kornia: ShiftScaleRotate, RandomCrop, CenterCrop, RandomResizedCrop, Perspective, ElasticTransform, GridDistortion, plus HueSaturationValue.

they split into two groups, and only the second needs you:

faithful mappings. RandomCrop, CenterCrop, RandomResizedCrop and Perspective have Kornia equivalents with matching semantics. I can send these the same way as #1277.

not faithful. ShiftScaleRotate, ElasticTransform, GridDistortion and HueSaturationValue differ in more than naming. HueSaturationValue is the clearest: Albumentations shifts hue/saturation/value additively (hue in OpenCV units, saturation on 0-255), while Kornia's ColorJiggle scales them multiplicatively. Same config would mean different augmentation depending on which backend resolved, which is worse than the current hard failure.

so the question: for the second group, would you rather

1. remove them from the documented list for Kornia, so the docs match reality, or
2. keep them and raise a clear backend-specific error naming the nearest equivalent, or
3. map them approximately and document the difference?

my preference is (1) plus (2): drop them from the Kornia column and make the error say which backend supports what, since a silent semantic difference across backends is the failure mode this issue is about in the first place.

happy to send the four faithful ones now regardless, if you'd rather not block those on the decision.

[adhavan18]

correcting myself on the geometric half, because I got it wrong in the comment above and only found out by tracing where the pipeline output actually goes.

I said the four crop/perspective names "have Kornia equivalents with matching semantics" and offered to send all four. that is true transform-for-transform, and wrong at the pipeline level. only Perspective is shippable as a registry entry, and it is up as #1330.

the reason is resolution. measured on 8444ce3:

RandomCrop(32,32)      in=(64,64) out=(32,32)   size changed
CenterCrop(32,32)      in=(64,64) out=(32,32)   size changed
RandomResizedCrop(48)  in=(64,64) out=(48,48)   size changed
RandomPerspective      in=(64,64) out=(64,64)   size preserved

and on_after_batch_transfer rebuilds the batch as NestedTensor(img_aug, samples.mask), reusing the pre-augmentation padding mask. a transform that resizes the image leaves the mask describing a different shape, and NestedTensor does not check:

NestedTensor(img 32x32, mask 64x64) -> constructed, no error

so the three crops need the mask cropped in step with the image, not just a registry entry. that is a real change with its own design question (what the mask should mean after a random crop), so I have not smuggled it into #1330. there is a test in that PR pinning the three stay rejected, so the gap cannot get quietly marked done later.

that leaves the issue in three parts rather than two:

1. pixel-level (4) — done, feat(augment): support Blur, Sharpen, Equalize and CLAHE on the Kornia backend #1277
2. Perspective — feat(augment): support Perspective on the Kornia backend #1330
3. the three crops + the four non-faithful names (ShiftScaleRotate, ElasticTransform, GridDistortion, HueSaturationValue) — still open

for group 3 the question from my last comment still stands, and now applies to the crops too: remove them from the documented list for Kornia, raise a clear backend-specific error naming the nearest equivalent, or map them approximately and document the difference. my preference is still the first two together.

happy to do the mask work for the crops if you want it, just would rather agree what the mask means after a crop before writing it.