NestIO for large datasets #1001
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Ha @morales-gregorio Thanks for sharing your code. I think this problem might be improved quite a lot when #1000 is being merged as this allows to generate lists of spiketrains based on a gdf based data organization (one array of timestamps & one array of unit ids) and only do the conversion to spiketrains when required. |
|
Indeed! #1000 looks like the solution to this problem! Looking forward to it, happy to contribute to merging this with the neo.SpikeTrainList once it is ready |
|
Hi! I see that #1000 was merged already, any updates on implementing it within the NestIO? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi!
I was recently working together with @jasperalbers, who is using the NestIO to load simulated data from large scale multi-area models.
His problem was that he has hundreds of thousands of neurons (around 1GB in total), which when saved as
neo.SpikeTrainobjects it would take hours to load from disk (on all HDF5, pickle and nix). Incredible amounts of time were spent building the neo objects themselves. We found a rather unorthodox workaround to this problem, by saving the spikes directly as lists of lists, which brought down the load time to a few seconds.We wrote a couple of extra functions to the
NestIOto load the spike times as llists of lists, alongside the neuron IDs. This is obviously not ideal from a metadata perspective, but we thought it might still be a useful function to have, especially for agile analysis of large simulated data.Let us know if this functions are any good, if you think they are worth including I can also write some tests.
Best,
Aitor