Flagging some examples of some incorrect/not preferred NameRes outputs

[jxneli]

Hello NameResolver team!

At Every Cure, we've recently been working on curating a super high-quality drug indications dataset and as part of this process we have been manually correcting the mapping of drug indication text to Mondo IDs.

We thought it would be helpful to share examples of where NameResolved behaved either a) unexpectedly/incorrectly or b) not as preferred (for our use case) in order to help improve NameResolver as a tool in the long term!

Attaching a sheet of examples with the following fields:

• input_to_nameres
• output_curie
• output_label
• intended_curie
• intended_label
• comment: this column describes the issue seen for each example

Notably, we also found an example of non-determinism where the exact same string input yield two different output curies (we've labelled these examples with "non-deterministic output" in the comment column)

Thanks for your attention. Let us know if you have any follow up questions!

nameres_feedback.csv

[gaurav]

Thanks so much for submitting these test cases! I've incorporated them into my internal testing spreadsheet. I'm currently working on improving the scoring that we use (I just documented the current scoring if you're interested), but I don't know how much that will help with these particular cases. You could turn on the highlighting flag for information on what parts of the query are matching the search results, which might give you an idea of why Solr thinks a bad search result is better than a good one. We're also experimenting with using SAPBERT to do embedding-based searches that might work better for your cases. Ultimately, however, we'll probably need to combine concepts like UMLS:C0406317 "Chronic small plaque psoriasis" with MONDO:0005083 "psoriasis" so that they don't confuse NameRes (e.g. NCATSTranslator/Babel#267). I'll talk to the Every Cure folks at RENCI and figure out the best path forward. I don't think there is an easy fix there but I hope to get your results much improved in the future!

I'm more immediately concerned about the non-deterministic behavior you're seeing -- there are some cases when Solr can return non-deterministic behavior (e.g. a timeout that causes Solr to provide incomplete search results if it hasn't had time to finish its search), but it sounds like you're seeing cases where you get different results through different runs. I've written small tests to check for non-deterministic behavior in both NameRes and NodeNorm -- both of these tests work in the same way, which is that they send the same query to NameRes/NodeNorm 100 times and check to make sure it returns the same result every time. I haven't seen a failure in these tests yet, but I'll try running it again with large numbers of repeats and some time between different queries to see if I can catch it. In the meantime, if you could provide some more information on the non-deterministic behavior you're seeing, that might point me towards figuring out what's going on! Here's the information that would be most useful to me:

1. The queries you're running and the endpoint you're sending it to on NameRes -- in particular, making sure that NameRes is in autocomplete=false mode.
2. The actual results you're getting as JSON files (the scores will be useful to look at there -- I wonder if maybe part of the problem is two results that score so similarly that Solr switches them around when returning them).
3. You shouldn't need to normalize results from NameRes using NodeNorm, since they should already be normalized (and conflated with both GeneProtein and DrugChemical conflation). But if you're doing any additional filtering/transformation on the results coming from NameRes, please let me know!
4. The version number of NameRes (from https://name-resolution-sri.renci.org/openapi.json) and the Babel version it was loaded from (from https://name-resolution-sri.renci.org/status) -- it's possible that some glitch on our end is presenting you with different NameRes instances when you send it a query or something.
5. Any other clues you can think of would be very helpful.

Thanks again!