[WIP] Nest3io

[JuliaSprenger]

This PR adds support for the sion format, the new output format of NEST. The IO depends on the nestio-tools (https://github.com/apeyser/nestio-tools) and the sionlib library (https://www.fz-juelich.de/ias/jsc/EN/Expertise/Support/Software/SIONlib/sionlib-download_node.html).

Current status: Work in progress.

[pep8speaks]

Hello @JuliaSprenger! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file neo/rawio/nest3rawio.py:

Line 21:100: E501 line too long (113 > 99 characters)
Line 29:1: E303 too many blank lines (3)
Line 84:30: E261 at least two spaces before inline comment
Line 95:22: E201 whitespace after '{'
Line 96:52: E203 whitespace before ':'
Line 102:100: E501 line too long (101 > 99 characters)
Line 117:60: E231 missing whitespace after ','
Line 118:65: E231 missing whitespace after ','
Line 118:70: E261 at least two spaces before inline comment
Line 118:100: E501 line too long (120 > 99 characters)
Line 125:13: E303 too many blank lines (2)
Line 141:100: E501 line too long (108 > 99 characters)
Line 147:69: E261 at least two spaces before inline comment
Line 148:100: E501 line too long (119 > 99 characters)
Line 151:44: E262 inline comment should start with '# '
Line 153:100: E501 line too long (112 > 99 characters)
Line 158:100: E501 line too long (109 > 99 characters)
Line 165:9: E303 too many blank lines (2)
Line 167:13: E116 unexpected indentation (comment)
Line 167:13: E265 block comment should start with '# '
Line 167:100: E501 line too long (118 > 99 characters)
Line 168:13: E116 unexpected indentation (comment)
Line 183:9: E303 too many blank lines (2)
Line 211:9: E303 too many blank lines (2)
Line 230:9: E303 too many blank lines (3)
Line 254:100: E501 line too long (105 > 99 characters)
Line 263:5: E303 too many blank lines (2)
Line 274:15: E271 multiple spaces after keyword
Line 277:5: E303 too many blank lines (2)
Line 287:15: E271 multiple spaces after keyword
Line 290:5: E303 too many blank lines (2)
Line 302:5: E303 too many blank lines (2)
Line 323:5: E303 too many blank lines (2)
Line 324:100: E501 line too long (100 > 99 characters)
Line 344:9: E303 too many blank lines (3)
Line 348:31: E231 missing whitespace after ','
Line 348:34: E225 missing whitespace around operator
Line 348:47: E231 missing whitespace after ','
Line 348:100: E501 line too long (131 > 99 characters)
Line 349:36: E231 missing whitespace after ','
Line 365:26: E226 missing whitespace around arithmetic operator
Line 365:50: E226 missing whitespace around arithmetic operator
Line 369:46: E261 at least two spaces before inline comment
Line 369:46: E262 inline comment should start with '# '
Line 372:9: E303 too many blank lines (2)
Line 373:12: E225 missing whitespace around operator
Line 373:57: E261 at least two spaces before inline comment
Line 374:9: E741 ambiguous variable name 'l'
Line 375:25: E226 missing whitespace around arithmetic operator
Line 375:27: E226 missing whitespace around arithmetic operator
Line 381:9: E303 too many blank lines (2)
Line 391:100: E501 line too long (102 > 99 characters)
Line 394:100: E501 line too long (114 > 99 characters)
Line 402:9: E303 too many blank lines (7)
Line 409:9: E303 too many blank lines (2)
Line 425:13: E116 unexpected indentation (comment)
Line 426:13: E116 unexpected indentation (comment)
Line 427:13: E116 unexpected indentation (comment)
Line 428:13: E116 unexpected indentation (comment)
Line 435:9: E303 too many blank lines (2)
Line 446:5: E303 too many blank lines (2)
Line 459:5: E303 too many blank lines (3)
Line 483:100: E501 line too long (121 > 99 characters)
Line 484:56: E231 missing whitespace after ','
Line 485:100: E501 line too long (113 > 99 characters)
Line 498:5: E303 too many blank lines (2)
Line 504:5: E303 too many blank lines (2)
Line 508:5: E303 too many blank lines (2)
Line 512:5: E303 too many blank lines (2)
Line 516:5: E303 too many blank lines (2)
Line 525:5: E303 too many blank lines (2)
Line 532:100: E501 line too long (137 > 99 characters)
Line 538:100: E501 line too long (112 > 99 characters)
Line 541:29: E261 at least two spaces before inline comment
Line 544:15: E225 missing whitespace around operator
Line 551:5: E266 too many leading '#' for block comment
Line 551:5: E303 too many blank lines (2)
Line 553:5: E301 expected 1 blank line, found 0
Line 555:100: E501 line too long (110 > 99 characters)
Line 563:34: W292 no newline at end of file

• In the file neo/test/iotest/test_nest3io.py:

Line 35:100: E501 line too long (102 > 99 characters)
Line 41:5: E303 too many blank lines (2)

Comment last updated at 2019-09-27 14:38:17 UTC

[essink]

Hi @JuliaSprenger,

in order to load data generated with NEST 3 (no .gdfs anymore), some while ago I created a hotfix branch. It uses the existing nestio from the master branch and only makes small adjustments to allow loading the resulting .dat files.

Disclaimer: It is not properly tested, but seems to work at least for the spiking data. At the time of creating the hotfix, I was not aware of this pull request. The hotfix does not make use of any knowledge of nestio-tools or the sionlib library and thus does not generalize the nestio satisfyingly. Nevertheless, I wanted to let you know in the hope that it may help somehow.

Best,
Simon

[JuliaSprenger]

Hi @essink Thanks for linking your solution here. Unfortunately this topic is quite low in my queue, so if you want to make a full PR out of this feel free to go ahead.

[morales-gregorio]

Hi @rgutzen @jasperalbers and I just also stumbled upon the NestIO 3.0 problem. Any chance this is merged anytime soon?

@rgutzen created a simple hotfix that requires only minor changes in the __init__() of the class ColumnIO such that:

def __init__(self, filename):
    """
    filename: string, path to ASCII file to read.
    """

    self.filename = filename
    
    # read the first line to check the data type (int or float) of the data
    f = open(self.filename)
    line = f.readline()
    header_size = 0
    
    while line:
        if line[0].isdigit():
            break
        else:
            header_size += 1
            line = f.readline()

    # Warn user
    if header_size > 0:
        warnings.warn(f'Ignoring {str(header_size)} header lines.')
    
    additional_parameters = {}
    if '.' not in line:
        additional_parameters['dtype'] = np.int32
    
    self.data = np.loadtxt(self.filename, skiprows=header_size,
                           **additional_parameters)

In principle this can work for both nest 2.x and 3.x, since the only thing it does is ignoring the header lines in the file and proceeding as usual. We can create a different PR to include this in the nestio if you want.

Best,
Aitor, Robin and Jasper