Memory Error

[princerajsharma31]

When I am trying to extract the light curve with HENcurve with a binning of 2^-14 s the task crashes with the error:

File "<array_function internals>", line 6, in bincount
MemoryError: Unable to allocate 8.79 GiB for an array with shape (1179709440,) and data type int64

Then I changed the binning to 2^-10s and light curve can be extracted but then HENfspec crashes with the error:

contents[kcorr] = (integer_part + float_part) * 10.0 ** log10_part
MemoryError: Unable to allocate 563. MiB for an array with shape (73731840,) and data type float64

I am not sure what is going on. I have been getting these error for a few days now and I can not go past these errors.

[matteobachetti]

Hi @princerajsharma31, thanks for reporting the issue.
At the moment, HENDRICS needs to load all datasets into the RAM. MemoryError means that at a certain point of the processing you have no RAM left. We are working on a memory mapped version of Stingray that will allow larger-than-memory datasets, but at the moment this is not allowed.

I suggest to split the dataset by GTI (HENreadevents --split-gti (...)), calculate the light curve and the power spectra of these chunks, and then sum the power spectra with HENsumfspec.

Hope this helps.

[princerajsharma31]

Than you for the way out. Looking forward for the new and fixed version. Best regards Prince Prince Sharma

…

On 22-Nov-2020, at 12:37 AM, Matteo Bachetti ***@***.***> wrote: Hi @princerajsharma31, thanks for reporting the issue. At the moment, HENDRICS needs to load all datasets into the RAM. MemoryError means that at a certain point of the processing you have no RAM left. We are working on a memory mapped version of Stingray that will allow larger-than-memory datasets, but at the moment this is not allowed. I suggest to split the dataset by GTI (HENreadevents --split-gti (...)), calculate the light curve and the power spectra of these chunks, and then sum the power spectra with HENsumfspec. Hope this helps. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.