Issues with Meteor 1.3

[ndarilek]

Perhaps this is already present and I don't see it. There are lots of changes coming down the pipe in Meteor 1.3, and (I don't think) it is yet clear how to meteor build. However, one thing is certain, NPM integration works out of the box.

To that end, it'd be nice if meteor-spk shipped a recent version of NPM. We still have to hack our build scripts to insert npm installs and copy package.json around, but having NPM installed automatically would make figuring out the correct incantations easier. Having a release that shipped NPM would be very helpful.

Thanks!

[kentonv]

Hi Nolan,

Sorry, I'm a bit confused. Meteor's packaging system has had built-in support for declaring dependencies on NPM packages for a long time. If you do that, meteor-spk should take care of making sure those packages make their way into the SPK. Specifically, meteor-spk pack performs a meteor build and then an npm install as specified in the Meteor docs. It uses Meteor's bundled npm to do this.

I haven't kept up with the changes in Meteor 1.3, though. Is it breaking this somehow?

[ndarilek]

You can now put a package.json in your Meteor 1.3 apps and are expected to do an `npm install` as a step before `meteor build`. This is separate from, and will likely replace in many cases, putting NPM package dependencies in Meteor packages simply to use NPM packages in Meteor. So now where we have: ``` cd /opt/app meteor build --directory /home/vagrant/ ``` in build.sh, you'd now do: ``` cd /opt/app if [ -e package.json ]; then npm install fi meteor build --directory /home/vagrant/ ``` Anyhow, I've dug a bit deeper, and it looks like the vagrant-spk build scripts access npm from within the .meteor directory. So it's included, just getting at it isn't obvious, and I need to in order to invoke npm at the right points. Maybe this issue should be closed, or repurposed as a general "what's needed to get Meteor 1.3 working in Sandstorm" discussion. I'll be hacking on that, but finding npm was a bit much for my coffee-deprived brain this morning when I filed this. :) I was looking at the dependency-packing logic in meteor-spk and didn't see anywhere referencing the npm binary as was true with node and others.

[kentonv]

Well, npm is a build-time dependency, not a runtime dependency, so it shouldn't be included in the spk like the node binary is.

FWIW this line does npm install: https://github.com/sandstorm-io/meteor-spk/blob/master/meteor-spk#L72

From what you are saying it sounds like meteor-spk may need some tweaking for Meteor 1.3. Although, I'm a little confused by the suggestion that npm install would execute before meteor build. Normally (when not targeting Sandstorm) meteor build is expected to run on the developer's machine, but npm install really needs to run on the target machine in order to install the correct architecture for native-code components. If npm install now happens first, it seems like you'd get the wrong arch, or the build process now needs to run entirely on the target. I'd like to see what the docs have to say about this.

@zarvox I forget, does vagrant-spk invoke meteor-spk for the build step, or will it need to be updated separately?

[ndarilek]

So I added code similar to the above to my build.sh. I'm able to build the app, but am told that Meteor requires node 0.10.41 or greater to run.

The version of node shipped with Meteor 1.1.3 is 0.10.36. I'm wondering if the launcher script is using that, despite my Meteor app being set to 1.3 beta? I definitely have the correct Meteor build tools in place.

Where is METEOR_WAREHOUSE_DIR set initially? I see it modified in build.sh. It isn't immediately obvious to me where the scripts are pulling the version of Meteor they run binaries against.

Thanks.

[ndarilek]

See this issue for more about the npm install && meteor build workflow under 1.3.

[zarvox]

@kentonv It will need to be updated separately, but it should be reasonably straightforward to do so: https://github.com/sandstorm-io/vagrant-spk/blob/master/stacks/meteor/build.sh

@ndarilek usually METEOR_WAREHOUSE_DIR is unset; the script as written lets a value from the environment override the default value of $HOME/.meteor.

However, we are using the node from the meteor-spk bundle, which copies in a node (and a mongo) at meteor-spk's build time. What we should probably do instead is do something like what Sandstorm itself does with find-meteor-dev-bundle.sh and pull in the node from the Meteor dev_bundle that matches the current project's .meteor/release, and use that copy of node. Of course, since the default sandstorm-pkgdef.capnp hides /home, we'll probably need to copy it somewhere else, or adjust the pkgdef...

This does seem like a problem for meteor-spk in general though - the version of node will be whatever meteor-spk ships, not whatever your project specifies.

[kentonv]

I see, it looks like npm install is needed to install build tools invoked by meteor build, not runtime packages. Fair enough.

As for the node version error, yeah, we generally need to rev meteor-spk every time there is a new Meteor release. Happy to do that once 1.3 ships. In the meantime you may be able to get away with swapping out the binary manually.

[ndarilek]

No, NPM does ship runtime dependencies alongside MDG's own package
system. Meteor is ditching its own React package for the official
version from NPM, for instance. Also, lots of newer Meteor packages are
shipping as NPM packages. See https://github.com/mantrajs/mantra-core
for instance. It's meant only for use with Meteor and is shipped via NPM.

I'm a bit confused about the confusion here. :) My app's NPM-based
runtime dependencies aren't any different from its Meteor-based ones,
right? They're being installed in a Vagrant VM just like any other app
dependency, so are shipping with binaries compiled therein.

Might be worth retitling this issue to one about Meteor 1.3, which I'll
do tomorrow if no one beats me to it. Initially I went looking for npm
in /opt/meteor-spk and didn't find it, didn't think to poke around in
the .meteor directory as I'm used to using toolchains I install myself.
In the interim I'll go ahead and replace the older Meteor tools with the
newer and see what that gets me. Thanks for the tip.

[ndarilek]

I've done the following in my build.sh: ``` cd /opt/app cp /home/vagrant/.meteor/packages/meteor-tool/1.1.13-beta.12/mt-os.linux.x86_64/dev_bundle/bin/* "$METEOR_DEV_BUNDLE"/bin "$METEOR_DEV_BUNDLE/bin/npm" install ``` I've confirmed that `node -v` in $METEOR_DEV_BUNDLE/bin is 0.10.41, but I still get the error. Where is meteor-spk getting its node binary? I don't see a path or any other environment variable set in launcher.sh, just `exec node`. If it's getting it from $METEOR_DEV_BUNDLE/bin then I don't know what I'm missing. Looks like Meteor may be hiding a few additional node binaries around but I haven't investigated each `find . -name node` result. Thanks.

[kentonv]

meteor-spk gets its node binary from the download bundle linked in the readme. It will need to be rebuilt against 1.3.

[ndarilek]

So admittedly this won't work until 1.3 released Monday, but the new meteor node command can be used to run whatever version is intended for the app's Meteor release--0.10.43 for now, but 4.X/5.X very soon. However, unfortunately meteor node first does some nonsense or other attempting to open its package database, which I assume it does RW so it can upgrade itself if necessary, which of course fails on a read-only filesystem. I'm not entirely sure of that analysis but it seems likely, and Meteor annoys me enough sometimes that I don't care to understand its insanity further.

However you can use meteor node to introspect itself and symlink the expected node/npm binaries from the development bundle into /usr/local/bin. This in build.sh:

export BASEPATH=$(meteor node -e 'c=process.execPath.split("/"); console.log(c.slice(0, c.length-1).join("/"))')
sudo ln -sf $BASEPATH/node /usr/local/bin
sudo ln -sf $BASEPATH/npm /usr/local/bin

I have an app using the diy stack, community Mongo packages and a vanilla Meteor install, no more meteor-spk. If there's interest, I can work it into a meteor-new stack. You'll still need meteor-spk for older releases (unless you'd rather switch to official node packages.) Or maybe I can make the shell script not bomb out if meteor node doesn't work, meaning /usr/local/bin/node and /usr/local/bin/npm won't get created. If they are created they'd then override the 0.10.X version with whatever Meteor expects, since official versions would be in /usr/bin. It does take a bit longer to provision, but it's a lot less magical and (I think) easier for anyone looking to do anything vaguely more complicated than a vanilla Meteor app. Also, since meteor-spk's environment is very stripped-down, poking around in grains via nsenter is painful. I had to do echo $* instead of ls for instance, and my diy solution doesn't have that issue.

[kentonv]

Note that the meteor command itself is not included in the spk -- it's only available at build time.

But yeah, this could be a more-robust way to find the path of the node binary.

[ndarilek]

In my case it is included in the SPK because I've done away with meteor-spk and use the diy stack. It doesn't run at all however, failing silently. I've attempted to run it via the nsenter method but no luck. In any case I'm having pretty good results with this setup except, perhaps, for larger SPKs. My ebook reader is at 62M but it does have to pull in lots of Calibre and various Python standard libraries.

[kentonv]

Ah, yeah, the meteor tool is not going to like running inside a sandbox. It wants to check for updates, send tracking stats back to meteor, etc.