OutOfMemory on large studies in Retrieve + Export

[anton-newlantern]

Hi team,

Context: We're running a dcm4chee-arc-light instance (v 5.34.0) that serves for prefetching of prior studies from external PACS / DICOM store. When a study arrives into our system (separate non-dcm4chee path) - we trigger priors prefetching - meaning query studies for a patient (C-FIND through dcm4chee DICOMWeb) and then for a subset that we determine as relevant those are put into "Retrieve" queue of dcm4chee for further retrieval (C-MOVE).

Scenario:
The issue we're facing is that for large studies (~10,000 images) the app dies with OurOfMemory.

Multiple studies are being retrieved in parallel in Retrieve queue (currently set parallel tasks to pragmatic 7 and the issue still manifests) - then when the large study comes in, it goes through retrieve queue but causes OOM when dcm4chee starts exporting it (Export queue). The large study serves as a poison pill - no matter how many times container restarts - it always dies with OOM.
I've also been changing the memory settings (setting Xmx to fairly large in my mind 56Gi) and it still wasn't helping - which made me think of some memory leak and ask for help here.

Investigation performed:
I lowered Xmx to ~16Gi and captured a heap dump upon OOM. The heap dump shows plenty of attributes stored in memory (screenshot below). This looks like filling up the heap and serve as a problem.

I started thinking whether this is some memory leak at the Retrieve or Export stage. Alternatively, related to attributes, we have attributes coercion at study and instance level and I recently set the Attribute Update Policy to MERGE - so another questions is could that be the reason?

Appreciate your response and opinions on whether there's a better way to have the retrieval pipeline set up.

Best,
Anton

[anton-newlantern]

Additional context: we have a separate instance of dcm4chee-arc-light that only handles C-STORE's from external PACS and that one seems to be handling large studies without OOM issues. The difference is of course that that instance is not doing retrieval - so maybe again points to what's been kept in the queue.

[anton-newlantern]

Solutions that worked for me:

1. Changed Xmx to 120g - that let the large study go through Export queue (this still looks like an insane amount of mem needed for a study to go through though - I attribute that to the fact that solid part of the JVM heap was also occupied by studies in Retrieve queue).
2. I changed the Export Entity from Study to Instance - this drastically improved throughput and now large studies are well being exported even when multiple large ones run in parallel. Apparently the study level and loading of the attributes in memory is the bottleneck.