Skip to content

[Backend Tester] Clean up a few test issues #13258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: gh/GregoryComer/113/head
Choose a base branch
from
Open

[Backend Tester] Clean up a few test issues #13258

wants to merge 7 commits into from
+64 −47

Conversation

GregoryComer
Copy link
Member

@GregoryComer GregoryComer commented Aug 9, 2025 •
edited
Loading

There are a few broken tests that need cleaning up. Some are failing due to missing portable kernels. These tests are now skipped if any unsupported portable ops remain post-delegation. I also fixed a few other small issues and bumped the element-wise tolerance to reduce false positives. SNR should hopefully catch most blatant correctness issues. The fp16 and quantized tests can generate occasional high element-wise error but still have decent SNR (~60+).

@GregoryComer
Update
c99c41a 
[ghstack-poisoned]
@GregoryComer
Copy link
Member Author

GregoryComer commented Aug 9, 2025 •
edited
Loading

Stack from ghstack (oldest at bottom):

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Aug 9, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13258

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures

As of commit 5e92884 with merge base 3a02146 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 9, 2025
GregoryComer added a commit that referenced this pull request Aug 9, 2025
@GregoryComer
[Backend Tester] Clean up a few test issues
12d5de2 
ghstack-source-id: 0a3c4cc
ghstack-comment-id: 3172044664
Pull-Request: #13258
This was referenced Aug 9, 2025
Open
Open
Open
Open
Open
Open
Open
@GregoryComer
Update
bf57d6c 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Aug 9, 2025
@GregoryComer
[Backend Tester] Clean up a few test issues
b7324ca 
ghstack-source-id: f04ccde
ghstack-comment-id: 3172044664
Pull-Request: #13258
@GregoryComer
Update
0e162ab 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Aug 11, 2025
@GregoryComer
[Backend Tester] Clean up a few test issues
f4bc7b9 
ghstack-source-id: 5c06eb7
ghstack-comment-id: 3172044664
Pull-Request: #13258
@GregoryComer
Update
c6bd56b 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Aug 11, 2025
@GregoryComer
[Backend Tester] Clean up a few test issues
248d1bc 
ghstack-source-id: 8c0ec06
ghstack-comment-id: 3172044664
Pull-Request: #13258
@GregoryComer
Update
144a8ae 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Aug 12, 2025
@GregoryComer
[Backend Tester] Clean up a few test issues
d28c9c6 
ghstack-source-id: 9ac560d
ghstack-comment-id: 3172044664
Pull-Request: #13258
GregoryComer added a commit that referenced this pull request Aug 12, 2025
@GregoryComer
[Backend Tester] Clean up a few test issues
e485eff 
ghstack-source-id: 9ac560d
ghstack-comment-id: 3172044664
Pull-Request: #13258
This was referenced Aug 12, 2025
Open
Open
@GregoryComer
Update
ffaa1c3 
[ghstack-poisoned]
@GregoryComer GregoryComer mentioned this pull request Aug 12, 2025
Open
@GregoryComer GregoryComer marked this pull request as ready for review August 12, 2025 04:29
@GregoryComer GregoryComer requested a review from cccclai as a code owner August 12, 2025 04:29
This was referenced Aug 12, 2025
Open
Open
@GregoryComer
Update
5e92884 
[ghstack-poisoned]
@GregoryComer GregoryComer mentioned this pull request Aug 12, 2025
Open
digantdesai
digantdesai reviewed Aug 12, 2025
View reviewed changes
backends/test/suite/runner.py
@@ -9,6 +9,14 @@

import torch

# Set of unsupported ops that should cause tests to be skipped
UNSUPPORTED_PORTABLE_OPS = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are adding a Portable flow, how would these show up there? Some PTE_FAIL?

digantdesai
digantdesai reviewed Aug 12, 2025
View reviewed changes
backends/test/suite/runner.py
@@ -142,12 +159,15 @@ def build_result(
tester.run_method_and_compare_outputs(
inputs=None if generate_random_test_inputs else inputs,
statistics_callback=lambda stats: error_statistics.append(stats),
atol=1e-1,
Copy link
Contributor

@digantdesai digantdesai Aug 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

atol seems pretty high for a general default, no?

The fp16 and quantized tests can generate occasional high element-wise error but still have decent SNR (~60+).

Do you know how this is tested on PyTorch/PyTorch side? >60 SNR is good but outliers are not great esp for individual ops, if they are expected I would prefer if we set them per test basis which would allow us to reason about the math being done on that specific test warrenting high ATOL/RTOL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@digantdesai digantdesai digantdesai left review comments

@cccclai cccclai Awaiting requested review from cccclai cccclai is a code owner

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@GregoryComer @digantdesai