Certain tier 0 queries time out/socket closes #80
Description
If I send this query to tier 0, it takes forever (>10 mins) and eventually there's an error where the socket to DGraph has been closed:
curl -X POST http://localhost:8080/query \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"message": {
"query_graph": {
"nodes": {
"SN": {
"ids": null,
"categories": ["biolink:ChemicalEntity"],
"set_interpretation": "BATCH",
"constraints": [],
"member_ids": []
},
"ON": {
"ids": ["MONDO:0011705"],
"categories": ["biolink:DiseaseOrPhenotypicFeature"],
"set_interpretation": "BATCH",
"constraints": [],
"member_ids": []
},
"f": {
"ids": null,
"categories": ["biolink:Disease"],
"set_interpretation": "BATCH",
"constraints": [],
"member_ids": []
}
},
"edges": {
"edge_0": {
"subject": "SN",
"object": "f",
"knowledge_type": null,
"predicates": ["biolink:treats_or_applied_or_studied_to_treat"],
"attribute_constraints": [],
"qualifier_constraints": []
},
"edge_1": {
"subject": "f",
"object": "ON",
"knowledge_type": null,
"predicates": ["biolink:has_phenotype"],
"attribute_constraints": [],
"qualifier_constraints": []
},
"edge_2": {
"subject": "ON",
"object": "f",
"knowledge_type": null,
"predicates": ["biolink:has_phenotype"],
"attribute_constraints": [],
"qualifier_constraints": []
}
}
},
"knowledge_graph": { "nodes": {}, "edges": {} },
"results": [],
"auxiliary_graphs": {}
},
"parameters": { "tiers": [0], "timeout": 3000.0 }
}' > retriever_response.json
Here's the Retriever stack trace:
retriever | 2025-12-09T18:28:52.275+00:00 562 ERROR 3e25e92f Unhandled exception occurred while processing Tier 0. See logs for details. retriever.utils.logs:exception():145
retriever | Traceback (most recent call last):
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/pydgraph/txn.py", line 202, in handle_query_future
retriever | response = future.result()
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/grpc/_channel.py", line 886, in result
retriever | raise self
retriever | grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
retriever | status = StatusCode.UNAVAILABLE
retriever | details = "Socket closed"
retriever | debug_error_string = "UNKNOWN:Error received from peer {grpc_status:14, grpc_message:"Socket closed"}"
retriever | >
retriever |
retriever | The above exception was the direct cause of the following exception:
retriever |
retriever | Traceback (most recent call last):
retriever | File "<string>", line 1, in <module>
retriever | File "/usr/local/lib/python3.13/multiprocessing/spawn.py", line 122, in spawn_main
retriever | exitcode = _main(fd, parent_sentinel)
retriever | File "/usr/local/lib/python3.13/multiprocessing/spawn.py", line 135, in _main
retriever | return self._bootstrap(parent_sentinel)
retriever | File "/usr/local/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
retriever | self.run()
retriever | File "/usr/local/lib/python3.13/multiprocessing/process.py", line 108, in run
retriever | self._target(*self._args, **self._kwargs)
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
retriever | target(sockets=sockets)
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/uvicorn/supervisors/multiprocess.py", line 63, in target
retriever | return self.real_target(sockets)
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/uvicorn/server.py", line 65, in run
retriever | return asyncio.run(self.serve(sockets=sockets))
retriever | File "/usr/local/lib/python3.13/asyncio/runners.py", line 195, in run
retriever | return runner.run(main)
retriever | File "/usr/local/lib/python3.13/asyncio/runners.py", line 118, in run
retriever | return self._loop.run_until_complete(task)
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/opentelemetry/util/_decorator.py", line 71, in async_wrapper
retriever | return await func(*args, **kwargs) # type: ignore
retriever | > File "/usr/src/app/src/retriever/data_tiers/tier_0/base_query.py", line 52, in execute
retriever | backend_results = await self.get_results(self.qgraph)
retriever | File "/usr/src/app/src/retriever/data_tiers/tier_0/dgraph/query.py", line 27, in get_results
retriever | backend_record = await backend_driver.run_query(
retriever | File "/usr/src/app/src/retriever/data_tiers/tier_0/dgraph/driver.py", line 496, in run_query
retriever | result = await self._run_grpc_query(
retriever | File "/usr/src/app/src/retriever/data_tiers/tier_0/dgraph/driver.py", line 579, in _run_grpc_query
retriever | raise ConnectionError(
retriever | ConnectionError: Dgraph gRPC query failed: Socket closed
retriever | 2025-12-09T18:28:52.289+00:00 562 INFO 3e25e92f Collected 0 results from query tasks. retriever.lookup.lookup:lookup():108
retriever | 2025-12-09T18:28:52.289+00:00 562 DEBUG 3e25e92f KG Pruning: 0 (-0) nodes and 0 (-0) edges remain. retriever.utils.trapi:prune_kg():348
retriever | 2025-12-09T18:28:52.290+00:00 562 SUCCESS 3e25e92f Execution completed, obtained 0 results in 1083486ms. retriever.lookup.lookup:lookup():113
retriever | 2025-12-09T18:28:52.291+00:00 562 INFO 185.199.110.133:27047 - "POST /query HTTP/1.1" 200