Can meteor-spk stop embedding nodejs?

[paulproteus]

Right now, with Meteor 1.4.x apps, you can use meteor-spk 0.3.0 and have great success. In the future, when Meteor switches to node 6, I believe the current strategy of meteor-spk embedding a nodejs binary is going to be bad news.

Context: Our current backwards-compatibility situation

If you try to use a meteor 1.4.x app with meteor-spk 0.1.8, you see an error message like this:

AssertionError: "undefined" === "function"
    at wrapPathFunction (/programs/server/mini-files.js:77:10)
    at Object.<anonymous> (/programs/server/mini-files.js:108:24)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/programs/server/boot.js:9:13)
    at Module._compile (module.js:456:26)

Proposal

Can meteor-spk extract the nodejs binaries out of the current Meteor app's Meteor bundle, rather than embedding a particular node version?

This has the upside that it will be automatically forwards-compatible when Meteor switches versions of node.

It also has the upside that such a meteor-spk version would be backwards-compatible -- I could use that meteor-spk with Meteor 1.3.x apps as well as 1.4.x apps. That would address one pain-point I'm seeing with regard to the vagrant-spk transition explained here: sandstorm-io/vagrant-spk#193

[zarvox]

Can meteor-spk extract the nodejs binaries out of the current Meteor app's Meteor bundle, rather than embedding a particular node version?

This will require that app-builders compile node-capnp from source against the node of their (or their Meteor app's) choice. This sounds like the "right thing", but would probably slow setup a bit, and I'm not sure exactly what it entails in terms of building capnproto, and testing node-capnp against additional versions of node - there may be some compatibility work to be done.

[paulproteus]

I think that in general, people have a huge amount of difficulty successfully using node-capnp. The fact that this change would expose that difficulty does not bother me necessarily, since it's a problem we ought to fix in general. We've taken so long to support Meteor 1.4 that if we make capnp easier to use from within node in general but delay Meteor 1.4 support by another month, that would be OK with me personally.

It seems to me that node-capnp ought to decide at package install time which headers to compile, rather than assuming that the only node version in the world is node4 (or that the only node version in the world is 0.10). I would expect that most compiled packages in the node ecosystem do this kind of dispatch; out of curiosity, do people here know if it really is common? https://github.com/laverdet/node-fibers seems to say it works against node 4, 5, and 6, but doesn't mention 0.10.x.

I agree that it would slow setup somewhat, but apparently we already need a C++ compiler to compile the fibers npm module, so adding capnproto to that doesn't bother me vastly. Hopefully it can be cached, but I don't know much about the caching-ness of the Meteor build system.

[zarvox]

Yeah, +1 on the idea from me in general. My comments were intended as a "these are likely to be things you hit along the way" rather than a "this is a bad idea; here's why". :)

I suppose if we already need the compiler for meteor-1.4 anyway, then it's not such a big jump to also build node-capnp and possibly capnproto from source.

I wonder which parts should live in meteor-spk vs in a vagrant-spk stack. In looking at this again, it feels like system deps/compilers/whatnot belong in a vagrant-spk setup.sh, rather than in meteor-spk. Maybe once we've pushed all the support down into the dependencies (vagrant-spk, node-capnp) and other packages (for e.g. node-capnp and the sandstorm-integration .capnp files), there's nothing left for meteor-spk to do, and it turns into just a vagrant-spk stack and a docs page for the brave souls spk deving directly?

[ocdtrekkie]

A solution that requires I build capnp from source to package an app means I won't bother trying to package that app. Unless you can automate the whole process for me.

This is why I haven't tried raw API packaging at all.

[paulproteus]

FWIW, my understanding is that this would be automated as normal as part of the meteor app build process.

[paulproteus]

For node-capnp, it also looks like we can use https://github.com/mapbox/node-pre-gyp/ to publish prebuild binaries to npm, if we want to make people's lives one step easier. THis is not 100% essential, but I do want to mention that we can retain this fast-setup-avoid-doing-builds advantage.

[ocdtrekkie]

Is there any idea where we are currently in the world of meteor-spk working correctly?