Better error handling in daemon + gateway

[victorb]

Today, if there is a error in a stream or any other error within the daemon or the gateway, the entire process crashes. While not a problem for testing, it's a problem when deploying and trying to run a long-term daemon (for example, for the js-ipfs daemons).

I raised this before in #1243

Basically, we should be able to gracefully handle errors down the stack without having the daemon and gateway crash.

Current issue is the following error that happens every ~10 minutes:

/usr/src/app/node_modules/pull-pair/index.js:11
      throw new Error('already piped')
      ^

Error: already piped
    at sink (/usr/src/app/node_modules/pull-pair/index.js:11:13)
    at consume (/usr/src/app/node_modules/pull-defer/sink.js:7:17)
    at consume (/usr/src/app/node_modules/pull-defer/sink.js:7:17)
    at Connection.consume (/usr/src/app/node_modules/pull-defer/sink.js:7:17)
    at pull (/usr/src/app/node_modules/pull-stream/pull.js:43:9)
    at Dialer.handle (/usr/src/app/node_modules/multistream-select/src/dialer/index.js:47:5)
    at attemptMuxerUpgrade (/usr/src/app/node_modules/libp2p-switch/src/dial.js:164:10)
    at gotWarmedUpConn (/usr/src/app/node_modules/libp2p-switch/src/dial.js:55:7)
    at Switch.dial (/usr/src/app/node_modules/libp2p-switch/src/dial.js:41:9)
    at _getPeerInfo (/usr/src/app/node_modules/libp2p/src/index.js:238:19)

I'm not saying that we should (or should not) fix this particular error, but rather when this error (and similar) happens, the daemon and gateway should not die but continue running.

[victorb]

@fsdiogo as you're the current OKR owner for this, I've assigned this to you

[victorb]

Another error that happens directly after starting daemon right now:

Initializing daemon...
Using wrtc for webrtc support
/usr/src/app/src/cli/commands/daemon.js:40
        throw err
        ^

Error: websocket error
    at WS.Transport.onError (/usr/src/app/node_modules/engine.io-client/lib/transport.js:64:13)
    at WebSocket.ws.onerror (/usr/src/app/node_modules/engine.io-client/lib/transports/websocket.js:150:10)
    at WebSocket.onError (/usr/src/app/node_modules/engine.io-client/node_modules/ws/lib/EventTarget.js:109:16)
    at WebSocket.emit (events.js:160:13)
    at WebSocket.finalize (/usr/src/app/node_modules/engine.io-client/node_modules/ws/lib/WebSocket.js:182:41)
    at ClientRequest._req.on (/usr/src/app/node_modules/engine.io-client/node_modules/ws/lib/WebSocket.js:653:12)
    at ClientRequest.emit (events.js:160:13)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:539:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)
    at TLSSocket.socketOnData (_http_client.js:439:20)

[fsdiogo]

@victorbjelkholm I can't seem to be able to reproduce the error in your last comment, can you share the steps or configs you have that produces the error?

[victorb]

@fsdiogo this issue (#1325) is not about a specific issue but rather having having a error handler for uncaught exceptions from within the daemon. Currently, any error that happens down the stack (in js-bitswap for example) crashes the daemon, which is why it's so unstable currently. We need to make it so the daemon won't fully be crashed and needing a restart just because a error in one stream or for one operation.

[fsdiogo]

Alright, so it's best to start off on the daemon itself, i.e. try to reproduce an error and make it so that the daemon doesn't crash?

[victorb]

@fsdiogo I think just adding if (Math.random() > 0.5) throw new Error('Test error') in various levels in the stack and testing manually will let you discover one or two places where we can catch the errors appropriately.

[fsdiogo]

I've been thinking about a better approach to this problem. Instead of a band aid approach, we should really understand how we're dealing with errors and have a robust solution to propagate them.

We are using async throughout the project, but check this:

/*
  With async the stack trace gets lost: it only tells the line
  where we throw the error, but it came from one of the calls
  in the series, we don't have a way to know from which one.
 */

const async = require('async')

async.series([
  (cb) => cb(null, 'async x'),
  (cb) => cb('async error'),
  (cb) => cb(null, 'async xxx'),
], (err, res) => {
  if (err) {
	  throw new Error(err)
	}
})

/*
> node async.js
  Users/diogo/async-promises/async.js:15
  	throw new Error(err);
  	^

  Error: async error
    at async.series (/Users/diogo/async-promises/async.js:15:10)
    at /Users/diogo/async-promises/node_modules/async/dist/async.js:3874:9
    at /Users/diogo/async-promises/node_modules/async/dist/async.js:473:16
    at iterateeCallback (/Users/diogo/async-promises/node_modules/async/dist/async.js:976:17)
    at /Users/diogo/async-promises/node_modules/async/dist/async.js:958:16
    at /Users/diogo/async-promises/node_modules/async/dist/async.js:3871:13
    at async.series (/Users/diogo/async-promises/async.js:11:11)
    at /Users/diogo/async-promises/node_modules/async/dist/async.js:3866:24
    at replenish (/Users/diogo/async-promises/node_modules/async/dist/async.js:998:17)
    at iterateeCallback (/Users/diogo/async-promises/node_modules/async/dist/async.js:983:17)
*/

A solution for this is Promises:

/*
  With promises the stack trace is intact, being possible
  to observe all the lines and modules that lead to the error.
 */

new Promise((resolve, reject) => {
  setTimeout(() => resolve('promise x'), 100)
})
.then((res) => res + 'x')
.then((res) => randomFn())
.then((res) => res + 'xxx')
.catch((err) => console.log(err))

const randomFn = () => {
  throw new Error('randomFn error')
}

/*
> node promises.js
  Error: randomFn error
    at randomFn (/Users/diogo/async-promises/promises.js:14:9)
    at Promise.then.then (/Users/diogo/async-promises/promises.js:9:16)
    at <anonymous>
*/

Promises will vastly improve our debug capability and therefore help us with the handling of future errors. Moreover, we standardise the way we handle errors and improve the codebase by handling errors that are being lost along the way right now.

PR #1335 by @hugomrdias is a good example of that.

I had already opened the issue #1311 to start using promises, but I think this KR is the perfect case of why we should really do that. It's not a matter of callbacks vs promises, but of what works vs what doesn't.

I've been thinking of possible solutions but this is the best way that I've found, I've talked with @vasco-santos and @hugomrdias and they agree with this approach.

What do you guys think? @victorbjelkholm @diasdavid

[fsdiogo]

@dryajov want to chime in as you've dealt with this too?

[dryajov]

@fsdiogo so the issue here is not really with async or Promise. The resason the stack is lost in your async example is because you are instantiating the Error object inside the series done callback:

async.series([
  (cb) => cb(null, 'async x'),
  (cb) => cb('async error'),
  (cb) => cb(null, 'async xxx'),
], (err, res) => {
  if (err) {
	  throw new Error(err) // --> error instantiated here
	}
})

The Error object will capture the stack at the current place of instantiation, so if we change you're example to:

const async = require('async')

async.series([
  (cb) => cb(null, 'async x'),
  (cb) => cb(new Error('async error')),  // --> error instantiated here
  (cb) => cb(null, 'async xxx'),
], (err, res) => {
  if (err) {
    throw err
  }
})

We get the correct stacktrace:

/Users/dryajov/personal/projects/ipfs/ipfs/js-ipfsd-ctl/test.js:11
    throw err
    ^

Error: async error
    at async.series (/Users/dryajov/personal/projects/ipfs/ipfs/js-ipfsd-ctl/test.js:7:14)
    at /Users/dryajov/personal/projects/ipfs/ipfs/js-ipfsd-ctl/node_modules/async/dist/async.js:3866:24
    at replenish (/Users/dryajov/personal/projects/ipfs/ipfs/js-ipfsd-ctl/node_modules/async/dist/async.js:998:17)
    at iterateeCallback (/Users/dryajov/personal/projects/ipfs/ipfs/js-ipfsd-ctl/node_modules/async/dist/async.js:983:17)
    at /Users/dryajov/personal/projects/ipfs/ipfs/js-ipfsd-ctl/node_modules/async/dist/async.js:958:16
    at /Users/dryajov/personal/projects/ipfs/ipfs/js-ipfsd-ctl/node_modules/async/dist/async.js:3871:13
    at async.series (/Users/dryajov/personal/projects/ipfs/ipfs/js-ipfsd-ctl/test.js:6:11)
    at /Users/dryajov/personal/projects/ipfs/ipfs/js-ipfsd-ctl/node_modules/async/dist/async.js:3866:24
    at replenish (/Users/dryajov/personal/projects/ipfs/ipfs/js-ipfsd-ctl/node_modules/async/dist/async.js:998:17)
    at /Users/dryajov/personal/projects/ipfs/ipfs/js-ipfsd-ctl/node_modules/async/dist/async.js:1002:9

Note that in you're promise example you do instantiate the error at the place of rejection, so the stack comes up correctly as well:

/*
  With promises the stack trace is intact, being possible
  to observe all the lines and modules that lead to the error.
 */

new Promise((resolve, reject) => {
  setTimeout(() => resolve('promise x'), 100)
})
.then((res) => res + 'x')
.then((res) => randomFn())
.then((res) => res + 'xxx')
.catch((err) => console.log(err))

const randomFn = () => {
  throw new Error('randomFn error')
}

/*
> node promises.js
  Error: randomFn error
    at randomFn (/Users/diogo/async-promises/promises.js:14:9)
    at Promise.then.then (/Users/diogo/async-promises/promises.js:9:16)
    at <anonymous>
*/

[hugomrdias]

@dryajov sure ur correct, but the problem isn't where you create the errors. It's propagation, with async, errors instances are just passed around (or teleported as i say it). With promises, errors are automatically re throwed and stack-traces follow the code path and with a lot less noise from async internals.

Not having full stack traces is really painful because we have to manually search the code base, commenting or whatever until we find the line we need. In contrast with complete stacks we can cmd+click BOOM editor jumped to the line we need perfect developer experience and full productivity.

[dryajov]

Aside from my comment above, this is still obviously a legitimate issue overall. Promises do help with that somewhat, but they are IMO not the final solution for this particular issue and I would argue that bringing them up here as an end all solution for this problem, might end up doing more harm than good.

I though about it for some time and I think some of the issues we have boil down to mixing error handling styles. We rely heavily on callback style async constructs in our code base, which occasionally get mixed with try/catch somewhere deep in the codebase or a third party module, add EventEmmiter error event types and Promises on top of that and we get a pretty wild compound of things to that can and do go wrong.

IMO, in order to handle errors correctly we need to:

• find all the places where we're not handling errors appropriately and fix them
• Standarise error handling across our code base
  • look for throws and return them as callback errors where possible
  • ditto for promises
• EventEmmiters errors should also be appropriately handled - non fatal (error) type errors should be emited as warnings/user errors, since error is a special case and throws if there are no listeners attached
• Finally, have a global error handler that catches unhandled errors and terminates the process gracefuly

[hugomrdias]

This stuff below also is tricky because callback might not be called, in some places we even have setImmediate(self.emit('error', err)) to solve that.

  const done = (err, res) => {
      if (err) {
        self.emit('error', err)
        return callback(err)
      }

      self.state.initialized()
      self.emit('init')
      callback(null, res)
    }

What do you think about removing event emitters ? moving stuff like boot() inside ipfs constructor to a instance method or a static factory method where we can return a promise or use callbacks.