v7

[rowanwins]

Gday @tmcw @DenisCarriere @stebogit @stevage @Turbo87

As discussed in #1428 attached is a branch showing a potential way to rejig the code.

High Level Changes

• packages dir has become src
  • modules within src will now drop the turf- prefix
    • You'll see I've done this for a couple of modules eg src/along and src/helpers

At the module level

A module now contains the following files

• index.js which is an ES6 module
  • I used the existing index.ts as my starting point
    • I removed the ts syntax, this seemed simpler than using the output of tsc, although it does mean that some manual adjustments need to be made for initial validation of inputs.
  • where you need to use other turf modules it's as simple as doing ../whatever instead of @turf/whatever
  • I could possibly script a fair bit of this up in terms of changing the import references and removing the ts syntax.
• index.ts will be our ts integration which just uses an existing file which we used to have
  • Need to do some digging here as to how this will work - I'm guessing I need to set something in the root package.json or the tsconfig.json in the root
• test.js occasionally needs some tweaks if you call other turf modules, similar to above
• bench.js can be left exactly the same
• Lots of files have now been removed from the module level (eg license, readme's, package.json etc)

Tests

Because we now only have a single package.json at the root level I've had to rejig how the tests get called.

• There is a npm script called test-module which accepts an argument
  • eg running npm run test-module along will run the test.js in src/along
• Similar arrangement for benchmark files
  • eg npm run benchmark-module along will run the benchmark.js in src/along
• Because the tests are calling ES6 modules I'm using esm to load the modules
  • This is done by setting the -r esm flag

  "scripts": {
    "benchmark-module": "node -r esm scripts/benchmarkModule.js",

Summary

• So far the changes seems reasonably easy
• Haven't thought yet about the master turf package eg whats currently in src/turf but I'm not envisaging any tough changes here hopefully.
  • First thought is hopefully it's as simple as taking turf/index.mjs and moving it to src/index.js
• WIll also need to make a raft of changes to the package.json

Once we're getting close-ish I'd also like to have a list of issues to address before v7.0.0 is released. I'll get that organised at some stage.

And prior to publishing 7.0.0 I intend on publishing an alpha version to npm so that we can test various things (eg how you browserify folks can continue to work with turf).

Anyway getting late so that's enough for one night. Any major concerns at this stage anyone?

[bradjonesca]

Bloody hell! Nice work Rowan!

[DenisCarriere]

Love it 👍

It's a big change, but I think as we roll out the rest of the modules in v7.0 this will be a lot easier to review.

For now I think it's a great direction! 🚀

[stebogit]

Impressive @rowanwins! 👍

[rowanwins]

Just dropping a note here on various things I've been progressing

• I've been looking into the issue of projections and how they effect various operations. For example the buffer module has had projection issues for a while now.

  • I've looked into how other libs are tackling this and I propose we project our coordinates into the relevant UTM zone - from what I gather this is what PostGIS does
  • I've had a first crack at this based on this and I think I'll bundle it as a new helper eg wgsToUTM(someGeom) and utmToWGS(someGeom)
  • I've also done some comparisons of output results to ArcMap and our results are now much closer

• I've been investigating replacing the buffer dependency on jsts with polygon-offset. This isn't going to be entirely straight forward because that package needs a bit of love.

  • I also did some work on my own version of a turf native buffer algorithm at one stage which was looking fairly promising so I might revisit that

• The union/intersect/difference transition off jsts also needs some thought.

  • martinez was the original proposal for v6, it's the fastest but has the least flexible API and is a bit more bug prone.
  • A new contender is polygon-clipping which is a bit slower but has a more useful API for our use cases I think (eg I want to union a fc of polygons)
    • There are a few optimisation pieces still to be applied to the polygon-clipping repo, once these are done I think it'll consistently out-perform jsts at the minimum.

I'm going to give myself another week or so looking at some of those before giving up on them.

[rowanwins]

• Have one outstanding issue with mask due to some changes to the union module, shouldn't take too long to fix
• Also did a major overhaul of dissolve and that's now much more functional and is concave which depends on a the dissolve function.
• The only remaining jsts dependent module is buffer

[tmcw]

Awesome work thus far!

[rowanwins]

Thanks @tmcw - slowly getting there, unfortunantly time has been a rare commodity over the past few months.

My plan from here is

• get mask tidied
• look into the bundling
• attempt to publish a v7 alpha version
  • I propose we sit in alpha/beta for a few months given that it'll be a major breaking change, during this time
    • get the docs updated & ready for the release
    • do a bit a comms around the transition
    • work on some performance issues around polygon-clipping
    • look into reviving my buffer branch to see if we can kill off the remaining jsts dependency

In other positive news I've had a couple of past contributors looking at getting back into Turf so hopefully once v7 is reasonably stable we'll be able to get a few others contributing again. Once the v7 alpha is out I think I'll also write up a Medium post or something to try and attract a few more folks.

[tmcw]

👍 Let me know whenever you need a review on this branch.

[rowanwins]

Hi @tmcw,

• someone checked it out the other day and esm was the only issue they ran into - so need to add that as a dev dependency.

  • Scrap that - esm is already listed as a dependency

• i had a good go at the package.json the other day so i'd appreciate if you could take look at that

• And im on leave next week so i'll have a bit of time then to have another look through and make any other bug fixes, so if youve got me a list of suggestions im happy to get stuck in then

Cheers

[hutch120]

Hi @rowanwins I just had a chat with @simonokeefe (who you met at FOSS4G)... great work!! Looking forward to adding it into our project. Our major issue was multiple JSTS imports, but these changes clearly go much further toward ES6 compatibility. Again, thanks and keep up the great work.

[tmcw]

Marking as approved from my side - happy to help out landing this, as far as I can tell the checklist looks like

• Resolve conflicts
• Get dissolve passing again
• Release as alpha?

[rowanwins]

Hi @tmcw

Yeah for some reason dissolve is passing locally but not in travis, have even checked my node versions but there doesn't seem to be anything different, I think fallback is applying truncate on the results and seeing if that fixes it...

The conflicts are pretty minor I think so that shouldn't be hard.

There is already an alpha, I'd be happy to release this as the next version of the alpha :)

[rowanwins]

So dissolve is now passing with a bit of a refactor to the tests.

Outstanding issues

• unkink-polygon - has a problematic global for Number.prototype.modulo, also relies on rbush which is almost now treeshaking friendly
• polygonize - has an Object.defineProperties which needs some refactoring
• the polygon-clipping used for union, difference etc uses splay-tree which I think has the same problem as above, although splaytree has recently been updated and just needs to be updated in polygon-clipping for this to be resolved.
• isobands also needs some work, although I've not really dug in here.

[mourner]

relies on rbush which is almost now treeshaking friendly

BTW, for static processing use cases like unkink-polygon, I'd highly recommend looking into https://github.com/mourner/flatbush as an RBush replacement. It's much faster and already a ES6 module.

[rowanwins]

Down to isobands and the polygon-clipping treeshaking issue (for which I've generated a PR)

[dpmcmlxxvi]

@mourner @rowanwins I published a new package geojson-flatbush that may make it easier to roll flatbush into Turf. It's a wrapper like geojson-rbush but for flatbush to make it GeoJSON friendly.

[rowanwins]

Superceded