Skip to content

Conversation

@parisni
Copy link
Contributor

@parisni parisni commented Feb 25, 2024

After few years of hibernation, we tried an install from scratch and noticed few issues, leading to very painful experience. This PR solve the issues, and also provide a no brain docker-compose script to build the mimic-omop database. The build duration is less than 1 hour on a laptop, and it is compatible with both mimic demo or latest mimic 3 export.

Some of the major issues:

  • postgresql > 11 handle CTE differently leading to hang the ETL for eternity
  • indexing and analyze lack on concepts tables made the ETL also last way to long

Painful aspects:

  • manual instruction is quite vague on some parts
  • the R script needs local installation with no instructions
  • some scripts were missing
  • the makefile has broken references

@parisni parisni marked this pull request as draft February 26, 2024 09:08
@tompollard tompollard marked this pull request as ready for review March 12, 2024 18:36
@briangow
Copy link

@parisni , thanks for this!

I've attempted to run the Docker MIMIC-III demo build. I added the files to the mimic/data-mimicdemo and extras/athena folders as described in the docker/README. I get numerous errors and ultimately docker-compose up fails with exited with code 2. Please see the attached docker.log for details.

When initially setting things up, I also noticed that a lot of the links in the primary README are broken.

@parisni
Copy link
Contributor Author

parisni commented Apr 1, 2024

Hi @briangow thanks for your attempt. From the logs most of the errors you have are no big deal, for example missing schemas that are then created.

Eventually the data is created, as this example shows:

etl-1  | INSERT 0 31010
etl-1  | Time: 10634.488 ms (00:10.634)

But then this one retains my attention:

sh: 1: cannot create etl/Result/attribute_definition.csv.gz: Directory nonexistent

You might have to create a Result folder in etl to fix this. Tell me if this help, I would then fix this PR accordingly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants