Open
Conversation
Correct misclassified test cases based on comprehensive biomedical analysis. CRITICAL fixes (8 tests): - Iron for aceruloplasminemia: BadButForgivable → NeverShow (iron overload disease) - Ethanol for alcoholic hepatitis: Acceptable → NeverShow (causative agent) - Acetaminophen for alcoholic hepatitis: BadButForgivable → NeverShow (hepatotoxic) - Cyanide for peptic ulcer: BadButForgivable → NeverShow (deadly poison) - Hydrogen peroxide for peptic ulcer: BadButForgivable → NeverShow (caustic) - Nemaline for GERD: BadButForgivable → NeverShow (disease, not treatment) - Itraconazole for cystic fibrosis: NeverShow → Acceptable (treats fungal complications) - Prednisone for scotoma: NeverShow → BadButForgivable (treats inflammatory causes) MODERATE fixes (6 tests): - Neuroprotective agents for MS: BadButForgivable → Acceptable - Ibuprofen for cerebral palsy: BadButForgivable → Acceptable - Dichlorophen for hookworm: BadButForgivable → Acceptable - Revexepride for GERD: BadButForgivable → Acceptable - Retinol for pityriasis rubra pilaris: BadButForgivable → Acceptable - Methylphenidate for ADHD: Acceptable → TopAnswer (gold-standard, FDA-approved) CLEANUP: - Remove Asset_69.json (duplicate of Asset_68.json - both test "Soot treats Sleep Apnea") Changes improve test suite accuracy from 97.7% to 99.8% based on evidence-based medicine and clinical guidelines. Detailed biomedical rationale provided in PR. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Soot - treats - sleep apneaas an acceptable answer in Asset_69 caught my eye as something that needed correction. Then it occurred to me see whether Claude could catch that example and anything like it. I've pasted the report it generated below. I didn't go into each example in detail, but it passes my eyeball test.Test Classification Corrections Report
Date: 2026-01-27
Analysis Scope: 655 test files in
test_assets/directoryTotal Issues Found: 15 misclassifications requiring correction + 1 duplicate removal
Executive Summary
This report identifies biomedical test classifications that contradict established medical knowledge. Issues are categorized by severity:
CRITICAL SEVERITY - Must Fix Immediately
These represent dangerous errors where toxic, contraindicated, or harmful substances are classified as valid treatments:
1. Soot treats Obstructive Sleep Apnea [DUPLICATE REMOVED]
Files:
test_assets/Asset_68.json(kept) andtest_assets/Asset_69.json(removed)Asset_68 Classification: NeverShow (correct)
Asset_69 Classification: Acceptable → DELETED as duplicate
ID Mapping: MONDO:0007147 → MESH:D053260
Reference: https://github.com/NCATSTranslator/Feedback/issues/494
Biomedical Reasoning:
Soot is a toxic environmental pollutant containing carcinogenic polycyclic aromatic hydrocarbons (PAHs). It CAUSES respiratory diseases (asthma, COPD, lung cancer), not treats them. Air pollution particulates worsen sleep apnea by causing airway inflammation and obstruction.
Resolution: Asset_68 was already correctly classified as "NeverShow". Asset_69 was initially "Acceptable" (incorrect), but after correction to "NeverShow", it became an exact duplicate of Asset_68. Asset_69 has been removed from the test suite entirely to eliminate redundancy.
2. Iron treats Aceruloplasminemia
File:
test_assets/Asset_17.jsonCurrent Classification: BadButForgivable
Recommended: NeverShow
ID Mapping: MONDO:0011426 → PUBCHEM.COMPOUND:23925
Reference: https://github.com/NCATSTranslator/Feedback/issues/506
Biomedical Reasoning:
Aceruloplasminemia is an autosomal recessive iron OVERLOAD disorder caused by deficiency of ceruloplasmin (ferroxidase). Patients accumulate toxic iron in the brain, liver, pancreas, and retina, leading to neurodegeneration, diabetes, and retinal degeneration. Administering additional iron would exacerbate tissue damage and could be fatal. Treatment requires iron CHELATION (deferoxamine, deferasirox) to remove excess iron, not supplementation. This is absolutely contraindicated.
3. Ethanol treats Alcoholic Hepatitis
File:
test_assets/Asset_196.jsonCurrent Classification: Acceptable
Recommended: NeverShow
ID Mapping: MONDO:0005952 → PUBCHEM.COMPOUND:702
Biomedical Reasoning:
Alcoholic hepatitis is acute liver inflammation caused by chronic ETHANOL consumption and metabolism. Ethanol is the etiologic agent, not the treatment. Standard management requires complete alcohol abstinence, corticosteroids for severe cases, nutritional support, and addressing complications. Suggesting the causative toxin as treatment is medically contradictory and extremely harmful to patients.
4. Acetaminophen treats Alcoholic Hepatitis
File:
test_assets/Asset_202.jsonCurrent Classification: BadButForgivable
Recommended: NeverShow
ID Mapping: MONDO:0005952 → PUBCHEM.COMPOUND:1983
Biomedical Reasoning:
Acetaminophen (paracetamol) is hepatotoxic and specifically contraindicated in alcoholic liver disease. Alcoholic hepatitis patients have compromised liver function, depleted glutathione stores, and impaired drug metabolism. Even therapeutic doses of acetaminophen can cause acute liver failure in this population. The combination of alcohol + acetaminophen dramatically increases hepatotoxicity risk. This is an absolute contraindication that could be fatal.
5. Cyanides treats Peptic Ulcer Perforation
File:
test_assets/Asset_227.jsonCurrent Classification: BadButForgivable
Recommended: NeverShow
ID Mapping: MONDO:0005412 → MESH:D003486
Biomedical Reasoning:
Cyanide (CN⁻) is a rapidly fatal poison that inhibits cytochrome c oxidase, blocking cellular respiration. Exposure causes tissue hypoxia, metabolic acidosis, cardiovascular collapse, seizures, coma, and death within minutes. Cyanide has NO therapeutic use for any gastrointestinal condition. Peptic ulcer perforation is a surgical emergency requiring immediate laparotomy and repair. This classification is medically dangerous and absurd.
6. Hydrogen Peroxide treats Peptic Ulcer Perforation
File:
test_assets/Asset_241.jsonCurrent Classification: BadButForgivable
Recommended: NeverShow
ID Mapping: MONDO:0005412 → PUBCHEM.COMPOUND:784
Biomedical Reasoning:
Peptic ulcer perforation is a surgical emergency where gastric/duodenal contents leak into the peritoneal cavity, causing peritonitis and sepsis. Hydrogen peroxide (H₂O₂) is a caustic oxidizing agent that causes tissue damage, protein denaturation, and gas formation (oxygen bubbles). Administering it would worsen tissue injury, potentially cause gas embolism, and delay definitive surgical treatment. H₂O₂ has no role in treating perforated viscus. Treatment requires emergency laparotomy, ulcer repair/resection, and antibiotics.
7. Nemaline treats Gastroesophageal Reflux Disease
File:
test_assets/Asset_627.jsonCurrent Classification: BadButForgivable
Recommended: NeverShow
ID Mapping: MONDO:0007260 → MESH:D017696
Biomedical Reasoning:
This is a fundamental category error. "Nemaline" refers to nemaline myopathy (congenital neuromuscular disorder characterized by rod-shaped protein structures in muscle fibers). It is a DISEASE entity, not a chemical compound or therapeutic agent. A disease cannot treat another disease in this context. This appears to be a data mapping error where a disease term was incorrectly classified as a treatment modality. This makes no biomedical sense.
7. Itraconazole for Cystic Fibrosis
File:
test_assets/Asset_320.jsonCurrent Classification: NeverShow
Recommended: Acceptable or BadButForgivable
ID Mapping: MONDO:0009061 → PUBCHEM.COMPOUND:55283
Biomedical Reasoning:
Itraconazole is a triazole antifungal commonly used in cystic fibrosis patients for multiple indications:
While itraconazole doesn't treat the underlying CFTR genetic defect, it treats important CF-related infectious complications and is routinely prescribed to CF patients. The current "NeverShow" classification is too restrictive.
8. Prednisone for Scotoma
File:
test_assets/Asset_140.jsonCurrent Classification: NeverShow
Recommended: BadButForgivable
ID Mapping: MONDO:0004758 → PUBCHEM.COMPOUND:5865
Reference: https://github.com/NCATSTranslator/Feedback/issues/362
Biomedical Reasoning:
Scotoma is a visual field defect (symptom), not a specific disease. Prednisone (corticosteroid) is legitimately used to treat inflammatory causes of scotoma:
While prednisone doesn't treat ALL causes of scotoma (traumatic, ischemic, structural lesions), it is a legitimate treatment for inflammatory etiologies. The current "NeverShow" classification is overly restrictive.
MODERATE SEVERITY - Should Fix
These represent legitimate treatments being unfairly downgraded or first-line therapies not appropriately recognized:
9. Neuroprotective agents treats Multiple Sclerosis
File:
test_assets/Asset_113.jsonCurrent Classification: BadButForgivable
Recommended: Acceptable
ID Mapping: MONDO:0005301 → MESH:D000067829
Biomedical Reasoning:
Neuroprotection is a major therapeutic goal in multiple sclerosis. Several FDA-approved MS therapies have neuroprotective properties:
Neuroprotective strategies are actively researched and represent evidence-based therapeutic approaches. This should be considered at least "Acceptable."
10. Ibuprofen treats Cerebral Palsy
File:
test_assets/Asset_170.jsonCurrent Classification: BadButForgivable
Recommended: Acceptable
ID Mapping: MONDO:0006496 → PUBCHEM.COMPOUND:3672
Biomedical Reasoning:
Ibuprofen and other NSAIDs are commonly used in cerebral palsy for:
NSAIDs are evidence-based analgesics that provide important symptomatic relief and improve quality of life. This is standard supportive care and should be "Acceptable."
11. Methylphenidate treats ADHD
File:
test_assets/Asset_179.jsonCurrent Classification: Acceptable
Recommended: TopAnswer
ID Mapping: MONDO:0007743 → PUBCHEM.COMPOUND:4158
Biomedical Reasoning:
Methylphenidate (Ritalin, Concerta, Metadate) is:
This represents the standard of care for ADHD and should be classified as "TopAnswer," not merely "Acceptable."
12. Dichlorophen treats Hookworm Infectious Disease
File:
test_assets/Asset_55.jsonCurrent Classification: BadButForgivable
Recommended: Acceptable
ID Mapping: MONDO:0005849 → PUBCHEM.COMPOUND:3037
Biomedical Reasoning:
Dichlorophen is a legitimate anthelmintic (anti-worm) agent with documented efficacy against hookworms (Ancylostoma, Necator species). While it has been largely replaced by modern drugs like albendazole and mebendazole in developed countries, dichlorophen:
This represents a valid, if dated, treatment option and should be at least "Acceptable."
13. Revexepride treats Gastroesophageal Reflux Disease
File:
test_assets/Asset_630.jsonCurrent Classification: BadButForgivable
Recommended: Acceptable
ID Mapping: MONDO:0007260 → PUBCHEM.COMPOUND:135413542
Biomedical Reasoning:
Revexepride is a selective 5-HT4 receptor agonist that was investigated as a prokinetic agent for GERD. It:
While clinical development was discontinued (likely for commercial rather than efficacy reasons), it was a legitimate investigational therapy with sound biomedical rationale. This should be "Acceptable."
14. Retinol treats Familial Pityriasis Rubra Pilaris
File:
test_assets/Asset_92.jsonCurrent Classification: BadButForgivable
Recommended: TopAnswer or Acceptable
ID Mapping: MONDO:0019357 → PUBCHEM.COMPOUND:445354
Biomedical Reasoning:
Oral retinoids (vitamin A derivatives) are FIRST-LINE therapy for pityriasis rubra pilaris (PRP):
While retinol itself is less potent than prescription retinoids (acitretin, isotretinoin), the retinoid class represents the most effective treatment for PRP. The classification should reflect that retinoids are first-line therapy. This should be at minimum "Acceptable," potentially "TopAnswer" given the class efficacy.
15. Ibuprofen treats Aggressive Systemic Mastocytosis
File:
test_assets/Asset_85.jsonCurrent Classification: NeverShow
Recommended: Keep current classification (defensible)
ID Mapping: MONDO:0020333 → PUBCHEM.COMPOUND:3672
Reference: https://github.com/NCATSTranslator/Feedback/issues/479
Biomedical Reasoning:
This is nuanced and the current classification is defensible:
Against use (supports NeverShow):
For use (challenges NeverShow):
Recommendation: The cautionary "NeverShow" approach is reasonable given the potential risks. However, this could potentially be "BadButForgivable" with appropriate caveats. Current classification is acceptable.
Summary of Recommended Changes
Total Changes: 14 files modified + 1 file deleted (Asset_85.json requires no change)
Implementation Notes
Methodology
Report Generated: 2026-01-27
Analyst: Comprehensive biomedical knowledge base and clinical reasoning
Next Steps: Implement corrections via pull request