How does load work?

[wtoalabi]

I just met forerunnerdb and boy was I blown away!
Thanks for this project!
It's really a time saver!
OK, I have a very simple and perhaps dumb question here, please bear with me.

I am persisting my data into a JSON file. Cool.
And using a state manager (vuex on Vue) I felt maybe it would be wise to load my data incrementaly and only access the chunk that I need per time.
I even went ahead to use pagination.
All these went cool until I realise something obvious... When I called collection.load() all the data are actually loaded into memory... And that's why data manipulation runs so fast.
Which means, there really is no need to paginate the loads. I can just start consuming it directly.

Am I correct in this assumption?
If I am correct, would this not be too much for a large sized data store?
Like 50mb?
If all of that is loaded all at once, would that not be a bad thing?

Or could it be that I am just being naive?
That this is really the only obvious way of achieving this?

Thanks for your feedback...I earnestly await it.

[Irrelon]

Hi there, thanks for your question! The load() method does indeed load everything into memory so pagination is irrelevant for your use case.

50MB of data is a lot of data and it will take time to parse that and load it into memory. ForerunnerDB's initial use case was the manipulation and complex querying of JSON data in memory. Then people asked for the ability to save the data to the browser so I implemented a simple save and load system that just dumped the DB and loaded it all as required.

Then people asked for the load and save functionality to be more piece-meal so that it could be intelligent and load only what was required etc but I haven't implemented that yet. It's not on my priority list and is more a back-burner style item because I run 2 other businesses and am focussing on those almost entirely, plus loading data row by row as opposed to all at once was never something I needed so it's not a priority for any of my work either.

I have asked if someone would like to add their own load/save plugin that does things row by row but no takers unfortunately and without community help I'm unlikely to get around to it anytime soon.

I did start to re-write ForerunnerDB using ES6 and was intending to include a significantly better persistent storage module but that work is probably years away from completion.

[Irrelon]

Also, if all you need is a very simple query system and would prefer to add your own persistent storage layer in you can take a look at https://github.com/Irrelon/irrelon-path

That project is written in ES6 and is actually used by ForerunnerDB internally. Irrelon Path will probably become the new ForerunnerDB over time anyway and already has the match() function that mimics the very basic level of querying that ForerunnerDB does and the update() query that mimics very basic level updates. It doesn't support any $ operators so it really is very basic at the moment.

Worth a look if you want to "roll your own" and because it has named exports you can import what you need and use tree shaking to remove the rest :)