ROCm7.0.0rc with AOTriton

[rrunner77]

Hi,

I have a linux and Windows installation done on
CPU: 9900X
GPU:7900XTX
RAM: 64GB

Yoday, I installed the Windows nightly build, which contains AOTriton.

Compare shows like 2.5x faster with the AOTriton.

The only one difference that I have is on WAN2.2.
Seems that this is still 3 times slower than on linux.
On linux the default WAN t2v takes 15 min, on Windows it takes 45min.

Does anyone have a idea why ?

M.

[ScottTodd]

Yoday, I installed the Windows nightly build, which contains AOTriton.

The aotriton flag flip was landed this morning on Windows, after the release build was triggered. Should be there in the next nightly.

[rrunner77]

On Linux the same 5B video takes like 6 minutes. On Linux there is almost never shared VRAM.
I assume that this is the reason why the WAN generation is taking too much time.

[jammm]

So the naive convs that cause the TDR can be reproduced by:

MIOpenDriver.exe convfp16 -n 1 -c 1024 --in_d 6 -H 178 -W 322 -k 512 --fil_d 3 -y 3 -x 3 --pad_d 0 -p 0 -q 0 --conv_stride_d 1 -u 1 -v 1 --dilation_d 1 -l 1 -j 1 --spatial_dim 3 -m conv -g 1 -F 1 -t 1

I was able to repro this on my gfx1201 as well. In this past, the also caused TDR's on gfx1151.

[rrunner77]

I found a workaround.
The best way is to connect via RDP to Windows. This will decrease the usage of VRAM to 0.2GB.

Run the ConfyUI with --listen and connect to the web ui from a different device(e.g., mobile).

Seems to solve the slowness issue. No shared GPU memory.

[jammm]

@rrunner77 that's good to hear, thanks for sharing the workaround! I'm sharing your findings internally.
BTW, did that also fix the MIOpen related crash?

[rrunner77]

No, this does not solve the issue with the error. It occurs only on VAEDecode. In case of VAEDecode tiled, it works fine.
This happens only on the 5B fp16 wan2.2 model. Everything else is working as expected.

Now, what I did is that I installed openssh on Windows and run ComfyUI in background. Seems to be even faster than on linux (may be I should free the VRAM usage also on linux). Or use a second card for OS and the 7900xtx only for AI.

[jammm]

Is this slowdown from the VAE decode part I wonder?

…

On Saturday, September 13, 2025, Scott Todd ***@***.***> wrote: Oh wait, yes. That is a "dev" release I manually triggered for testing. The rocm.nightlies URL is for "nightly" releases. So yes, what you installed from there should be similar to what appears in the next nightly build, and that build has aotriton, from the logs: 2025-09-12T00:19:41.8410373Z -- USE_ROCM : ON 2025-09-12T00:19:41.8410547Z -- ROCM_VERSION : 2025-09-12T00:19:41.8410731Z -- USE_FLASH_ATTENTION : 1 2025-09-12T00:19:41.8410913Z -- USE_MEM_EFF_ATTENTION : 1 2025-09-12T00:19:41.8411107Z -- USE_ROCM_CK_SDPA : OFF 2025-09-12T00:19:41.8411308Z -- USE_ROCM_CK_GEMM : OFF Compare shows like 2.5x faster with the AOTriton. The only one difference that I have is on WAN2.2. Seems that this is still 3 times slower than on linux. On linux the default WAN t2v takes 15 min, on Windows it takes 45min. Huh... well, that's good to know. cc @jammm <https://github.com/jammm> @xinyazhang <https://github.com/xinyazhang> . Maybe the code paths used for WAN t2v are not as optimized via aotriton on the 7900XTX, specifically for the prebuilt kernels used for Windows...? — Reply to this email directly, view it on GitHub <#1477 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AATCSOEJ2BCEVMFE2WMSO5D3SMQN3AVCNFSM6AAAAACGLYVM42VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIMZYG44DCMY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rrunner77]

The Slow down is on KSampler - I use the default WAN2.2 T2I in Comfy
On Linux:
WAN (length 81 - 16fps)-> low 59.67s/it(1m59s) - high 59.77(1m59s) - VAEDecode takes very long 10m-> -VRAM 23.5 - RAM -30/44 - in total 15m:03s

On Windows:( This was with fp8)
WAN (length 81 - 16fps)-> low 138s/it(4m:37s) - high 234(7m48s) - VAEDecode takes very long +30m-> -VRAM 23.5+2GB of shared - RAM is between 30-44GB - in total 42m:14s

[jammm]

VRAB 23.5GB ?
The 2GB extra shared RAM usage could explain the showdown. What I would recommend is to close all background processes, disable HW acceleration on your browser and try again after a fresh restart of comfyui. Also try adding one of those nodes that does something like torch.cuda.empty_cache()

[vitpekarek]

I am testing img generation with 7900XTX right now. How long does it take to generate image in your case? Is it weird that first generation took me 6-10 minutes, but afterwards it was only 10-15 seconds?

I installed torch-2.10.0a0+rocmsdk20250911-cp312-cp312-win_amd64.whl and I'm not sure if AOTriton is working correctly, because I got this warning:

UserWarning: 1Torch was not compiled with memory efficient attention. (Triggered internally at B:\src\torch\aten\src\ATen\native\transformers\hip\sdp_utils.cpp:769.)
return torch.nn.functional.scaled_dot_product_attention(q, k, v, *args, **kwargs)

And I got some MIOpen(HIP) errors too.

[jammm]

Can you share those errors?
The initial generation taking longer is expected, because things like MIOpen try to do some autotuning to figure out the best ops to use.

[vitpekarek]

I already restarted ComfyUI and I got that error only once this time, but it doesn't looks like something serious or it didn't cancel my image generation at least.

MIOpen(HIP): Error [EvaluateInvokers] C:/home/runner/_work/TheRock/TheRock/rocm-libraries/projects/miopen/src/conv/solver_finders.cpp:303: Invalid elapsed time detected in EvaluateInvokers, failed condition: elapsed <= 0

But I managed to crash gpu driver after few attempts and got this warning in cmd after driver automatically restarted:

RuntimeWarning: invalid value encountered in cast
img = Image.fromarray(np.clip(i, 0, 255).astype(np.uint8))

[jammm]

Thanks a lot for checking. Can you try to reproduct the GPU driver crash while running it again, but this time with the following environment variables set?

export MIOPEN_ENABLE_LOGGING=1
export MIOPEN_ENABLE_LOGGING_CMD=1
export MIOPEN_LOG_LEVEL=6

(windows equivelent would be set for cmd or $env:.. on powershell)

Try to reproduce it with the above env vars set and share the logs if possible. Thanks!

[vitpekarek]

I couldn't reproduce it, but I will keep logging on and if this happens again in future I'll create new issue here with logs attached.

I'm still pretty new to ComfyUI. Do you have any recommendations for arguments I should set?

[rrunner77]

This versiopn does not have the AOTriton compiled. You need to use this version:
https://d25kgig7rdsyks.cloudfront.net/v2/gfx110X-dgpu/

Install python 3.12 on Windows
git clone https://github.com/comfyanonymous/ComfyUI.git
python -m venv .venv
.venv/Scripts/activate
python -m pip install --upgrade pip wheel
python -m pip install --index-url https://d25kgig7rdsyks.cloudfront.net/v2/gfx110X-dgpu/ --pre torch==2.10.0a0+rocm7.0.0rc20250908 torchaudio==2.8.0a0+rocm7.0.0rc20250908 torchvision==0.25.0a0+rocm7.0.0rc20250908

pip3.12 install -r requirements.txt
python [main.py](http://main.py/) --use-pytorch-cross-attention

You need this env. variable: TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1

[stellaraccident]

@jammm on Monday, would you mind sending an email with this issue and summary to Adam Tran, and I will then include my docs team: this is a very important finding that we need to make sure is well documented on a FAQ.

[jammm]

Done!

[scottt]

Uploading logs for the same MIOpenDriver convfp16 operation under Linux, which triggers a kernel GPU reset.

• vae-slowdown.txt reproducer script
• miopen.log
• dmesg.log

[tcgu-amd]

Hi @scott, what is your set up (i.e. GPU, rocm version)? Also, were you using a display device (i.e. monitor), or running any other graphics workload when you were running the MIOpenDriver command? Thanks!