Skip to content

NestIO for large datasets #1001

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

morales-gregorio
Copy link
Contributor

Hi!

I was recently working together with @jasperalbers, who is using the NestIO to load simulated data from large scale multi-area models.

His problem was that he has hundreds of thousands of neurons (around 1GB in total), which when saved as neo.SpikeTrain objects it would take hours to load from disk (on all HDF5, pickle and nix). Incredible amounts of time were spent building the neo objects themselves. We found a rather unorthodox workaround to this problem, by saving the spikes directly as lists of lists, which brought down the load time to a few seconds.

We wrote a couple of extra functions to the NestIO to load the spike times as llists of lists, alongside the neuron IDs. This is obviously not ideal from a metadata perspective, but we thought it might still be a useful function to have, especially for agile analysis of large simulated data.

Let us know if this functions are any good, if you think they are worth including I can also write some tests.

Best,
Aitor

@JuliaSprenger
Copy link
Member

Ha @morales-gregorio Thanks for sharing your code. I think this problem might be improved quite a lot when #1000 is being merged as this allows to generate lists of spiketrains based on a gdf based data organization (one array of timestamps & one array of unit ids) and only do the conversion to spiketrains when required.
We should revisit your code once #1000 is merged.

@JuliaSprenger JuliaSprenger self-assigned this Jun 18, 2021
@morales-gregorio
Copy link
Contributor Author

Indeed! #1000 looks like the solution to this problem! Looking forward to it, happy to contribute to merging this with the neo.SpikeTrainList once it is ready

@JuliaSprenger JuliaSprenger added this to the 0.11.0 milestone Jul 1, 2021
@morales-gregorio
Copy link
Contributor Author

Hi! I see that #1000 was merged already, any updates on implementing it within the NestIO?

@apdavison apdavison modified the milestones: 0.10.3, 0.11.0 Aug 30, 2022
@apdavison apdavison modified the milestones: 0.11.0, 0.12.0 Sep 29, 2022
@JuliaSprenger JuliaSprenger modified the milestones: 0.12.0, 0.12.1, 0.13.0 Apr 2, 2023
@apdavison apdavison modified the milestones: 0.13.0, 0.13.1 Feb 2, 2024
@zm711 zm711 modified the milestones: 0.13.1, 0.14.0 May 3, 2024
@zm711 zm711 modified the milestones: 0.14.0, 0.15.0 Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants