When attempting to use RubyProf RPCs time out and this causes ripple on failures.

[josephglanville]

E, [2013-09-30T04:07:52.989930 #80319] ERROR -- : Actor crashed!
NoMethodError: undefined method `-' for nil:NilClass
    /Users/jpg/src/floss/lib/floss/log_replicator.rb:98:in `replicate'
    /Users/jpg/src/floss/lib/floss/log_replicator.rb:58:in `block (2 levels) in initialize'
    /Users/jpg/.rvm/gems/ruby-1.9.3-p448/bundler/gems/celluloid-11fe1990fd6e/lib/celluloid/actor.rb:398:in `block in task'
    /Users/jpg/.rvm/gems/ruby-1.9.3-p448/bundler/gems/celluloid-11fe1990fd6e/lib/celluloid/tasks.rb:55:in `block in initialize'
    /Users/jpg/.rvm/gems/ruby-1.9.3-p448/bundler/gems/celluloid-11fe1990fd6e/lib/celluloid/tasks/task_fiber.rb:13:in `block in create'

I am not sure what is causing this yet but I saw it during testing.

[aflatter]

This happens if the log is empty and the first append_entries rpc fails. In that case, @write_indices[peer] should not be decremented, but the real question is: Why does that peer fail?

[josephglanville]

I managed to produce a worse crash when I invoke it via the ruby-prof command line.

ruby-prof examples/distributed_hash.rb

Produces the following stacktrace.

/Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/bin/ruby-prof:302: [BUG] Segmentation fault
ruby 1.9.3p448 (2013-06-27 revision 41675) [x86_64-darwin12.5.0]

-- Control frame information -----------------------------------------------
c:0003 p:0029 s:0010 b:0009 l:001020 d:000008 BLOCK  /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/bin/ruby-prof:302
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:002178 d:002178 TOP

-- Ruby level backtrace information ----------------------------------------
/Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/bin/ruby-prof:302:in `block in <top (required)>'

-- C level backtrace information -------------------------------------------

   See Crash Report log file under ~/Library/Logs/CrashReporter or
   /Library/Logs/CrashReporter, for the more detail of.

-- Other runtime information -----------------------------------------------

* Loaded script: /Users/jpg/.rvm/gems/ruby-1.9.3-p448/bin/ruby-prof

* Loaded features:

    0 enumerator.so
    1 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/enc/encdb.bundle
    2 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/enc/trans/transdb.bundle
    3 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/rbconfig.rb
    4 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/compatibility.rb
    5 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/defaults.rb
    6 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/deprecate.rb
    7 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/errors.rb
    8 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/version.rb
    9 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/requirement.rb
   10 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/platform.rb
   11 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/basic_specification.rb
   12 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/stub_specification.rb
   13 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb
   14 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/exceptions.rb
   15 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_gem.rb
   16 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/thread.rb
   17 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/monitor.rb
   18 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb
   19 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems.rb
   20 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/path_support.rb
   21 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/dependency.rb
   22 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/executable-hooks-1.2.3/lib/executable-hooks/hooks.rb
   23 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/syntax_error.rb
   24 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/psych.bundle
   25 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/stringio.bundle
   26 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/nodes/node.rb
   27 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/nodes/stream.rb
   28 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/nodes/document.rb
   29 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/nodes/sequence.rb
   30 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/nodes/scalar.rb
   31 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/nodes/mapping.rb
   32 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/nodes/alias.rb
   33 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/nodes.rb
   34 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/streaming.rb
   35 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/visitors/visitor.rb
   36 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/strscan.bundle
   37 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/scalar_scanner.rb
   38 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/visitors/to_ruby.rb
   39 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/visitors/emitter.rb
   40 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/visitors/yaml_tree.rb
   41 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/json/ruby_events.rb
   42 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/visitors/json_tree.rb
   43 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/visitors/depth_first.rb
   44 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/visitors.rb
   45 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/handler.rb
   46 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/tree_builder.rb
   47 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/parser.rb
   48 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/omap.rb
   49 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/set.rb
   50 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/coder.rb
   51 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/core_ext.rb
   52 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/date_core.bundle
   53 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/date/format.rb
   54 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/date.rb
   55 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/deprecated.rb
   56 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/stream.rb
   57 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/json/yaml_events.rb
   58 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/json/tree_builder.rb
   59 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/json/stream.rb
   60 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych/handlers/document_stream.rb
   61 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/psych.rb
   62 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/yaml.rb
   63 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/etc.bundle
   64 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/fileutils.rb
   65 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/pathname.bundle
   66 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/pathname.rb
   67 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/gem_path_manipulation.rb
   68 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/gem_helpers.rb
   69 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/match_platform.rb
   70 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/rubygems_ext.rb
   71 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/user_interaction.rb
   72 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/config_file.rb
   73 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/rubygems_integration.rb
   74 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/version.rb
   75 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler.rb
   76 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-unload-1.0.2/lib/bundler-unload.rb
   77 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/digest.bundle
   78 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/digest.rb
   79 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/digest/sha1.bundle
   80 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/environment.rb
   81 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/shared_helpers.rb
   82 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/runtime.rb
   83 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/settings.rb
   84 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/set.rb
   85 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/definition.rb
   86 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/dependency.rb
   87 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/ruby_dsl.rb
   88 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/dsl.rb
   89 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/source.rb
   90 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri/common.rb
   91 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri/generic.rb
   92 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri/ftp.rb
   93 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri/http.rb
   94 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri/https.rb
   95 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri/ldap.rb
   96 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri/ldaps.rb
   97 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri/mailto.rb
   98 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri.rb
   99 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/socket.bundle
  100 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/socket.rb
  101 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/timeout.rb
  102 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/protocol.rb
  103 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/zlib.bundle
  104 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb
  105 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/time.rb
  106 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/request.rb
  107 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/uri_formatter.rb
  108 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/fcntl.bundle
  109 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/openssl.bundle
  110 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/openssl/bn.rb
  111 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/openssl/cipher.rb
  112 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/openssl/config.rb
  113 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/openssl/digest.rb
  114 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/openssl/buffering.rb
  115 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/openssl/ssl-internal.rb
  116 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/openssl/x509-internal.rb
  117 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/openssl.rb
  118 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/securerandom.rb
  119 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/resolv.rb
  120 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/remote_fetcher.rb
  121 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/text.rb
  122 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/name_tuple.rb
  123 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/spec_fetcher.rb
  124 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/source/rubygems.rb
  125 /Users/jpg/src/floss/lib/floss/version.rb
  126 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/source/path.rb
  127 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/source/git.rb
  128 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/lockfile_parser.rb
  129 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/lazy_specification.rb
  130 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/tsort.rb
  131 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/forwardable.rb
  132 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/spec_set.rb
  133 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/index.rb
  134 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/remote_specification.rb
  135 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/dep_proxy.rb
  136 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/source/git/git_proxy.rb
  137 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/resolver.rb
  138 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/ui.rb
  139 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/bundler-1.3.5/lib/bundler/endpoint_specification.rb
  140 /Users/jpg/.rvm/gems/ruby-1.9.3-p448@global/gems/rubygems-bundler-1.3.3/lib/rubygems-bundler/noexec.rb
  141 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby_prof.bundle
  142 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/aggregate_call_info.rb
  143 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/call_info.rb
  144 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/call_info_visitor.rb
  145 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/compatibility.rb
  146 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/method_info.rb
  147 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/profile.rb
  148 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/tmpdir.rb
  149 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/rack.rb
  150 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/thread.rb
  151 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/printers/abstract_printer.rb
  152 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/printers/call_info_printer.rb
  153 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/erb.rb
  154 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/printers/call_stack_printer.rb
  155 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/printers/call_tree_printer.rb
  156 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/printers/dot_printer.rb
  157 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/printers/flat_printer.rb
  158 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/printers/flat_printer_with_line_numbers.rb
  159 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/printers/graph_html_printer.rb
  160 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/printers/graph_printer.rb
  161 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof/printers/multi_printer.rb
  162 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ruby-prof-0.13.0/lib/ruby-prof.rb
  163 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/ostruct.rb
  164 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/optparse.rb
  165 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/logger.rb
  166 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/calls.rb
  167 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/call_chain.rb
  168 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/condition.rb
  169 /Users/jpg/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/x86_64-darwin12.5.0/fiber.bundle
  170 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/fiber.rb
  171 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/thread.rb
  172 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/core_ext.rb
  173 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/cpu_counter.rb
  174 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/fsm.rb
  175 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/internal_pool.rb
  176 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/links.rb
  177 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/logger.rb
  178 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/mailbox.rb
  179 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/evented_mailbox.rb
  180 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/method.rb
  181 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/properties.rb
  182 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/timers-1.1.0/lib/timers/version.rb
  183 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/timers-1.1.0/lib/timers.rb
  184 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/receivers.rb
  185 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/registry.rb
  186 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/responses.rb
  187 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/signals.rb
  188 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/stack_dump.rb
  189 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/system_events.rb
  190 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/tasks/task_fiber.rb
  191 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/tasks/task_thread.rb
  192 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/tasks.rb
  193 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/task_set.rb
  194 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/thread_handle.rb
  195 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/uuid.rb
  196 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/proxies/abstract_proxy.rb
  197 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/proxies/sync_proxy.rb
  198 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/proxies/actor_proxy.rb
  199 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/proxies/async_proxy.rb
  200 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/proxies/future_proxy.rb
  201 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/proxies/block_proxy.rb
  202 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/actor.rb
  203 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/future.rb
  204 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/pool_manager.rb
  205 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/supervision_group.rb
  206 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/supervisor.rb
  207 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/notifications.rb
  208 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/logging/log_event.rb
  209 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/logging/incident.rb
  210 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/logging/ring_buffer.rb
  211 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/logging/incident_logger.rb
  212 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/logging/incident_reporter.rb
  213 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/logging.rb
  214 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid/legacy.rb
  215 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-0.15.1/lib/celluloid.rb
  216 /Users/jpg/src/floss/lib/floss.rb
  217 /Users/jpg/src/floss/lib/floss/test_helper.rb
  218 /Users/jpg/src/floss/lib/floss/rpc.rb
  219 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-io-0.15.0/lib/celluloid/io/stream.rb
  220 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi_c.bundle
  221 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/platform.rb
  222 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/types.rb
  223 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/library.rb
  224 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/errno.rb
  225 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/pointer.rb
  226 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/memorypointer.rb
  227 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/struct_layout_builder.rb
  228 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/struct.rb
  229 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/union.rb
  230 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/managedstruct.rb
  231 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/callback.rb
  232 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/io.rb
  233 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/autopointer.rb
  234 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/variadic.rb
  235 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/enum.rb
  236 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi/ffi.rb
  237 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-1.9.0/lib/ffi.rb
  238 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq/libc.rb
  239 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq/libzmq.rb
  240 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq/constants.rb
  241 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq/util.rb
  242 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq/exceptions.rb
  243 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq/context.rb
  244 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq/message.rb
  245 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq/socket.rb
  246 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq/poll_items.rb
  247 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/io_extensions.rb
  248 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq/poll_item.rb
  249 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq/poll.rb
  250 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq/device.rb
  251 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/ffi-rzmq-1.0.2/lib/ffi-rzmq.rb
  252 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-zmq-0.15.0/lib/celluloid/zmq/mailbox.rb
  253 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-zmq-0.15.0/lib/celluloid/zmq/reactor.rb
  254 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-zmq-0.15.0/lib/celluloid/zmq/sockets.rb
  255 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-zmq-0.15.0/lib/celluloid/zmq/version.rb
  256 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-zmq-0.15.0/lib/celluloid/zmq/waker.rb
  257 /Users/jpg/.rvm/gems/ruby-1.9.3-p448/gems/celluloid-zmq-0.15.0/lib/celluloid/zmq.rb
  258 /Users/jpg/src/floss/lib/floss/rpc/zmq.rb
  259 /Users/jpg/src/floss/lib/floss/log.rb
  260 /Users/jpg/src/floss/lib/floss/peer.rb
  261 /Users/jpg/src/floss/lib/floss/one_off_latch.rb
  262 /Users/jpg/src/floss/lib/floss/count_down_latch.rb
  263 /Users/jpg/src/floss/lib/floss/log_replicator.rb
  264 /Users/jpg/src/floss/lib/floss/node.rb
  265 /Users/jpg/src/floss/lib/floss/proxy.rb

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

[1]    80817 abort      ruby-prof examples/distributed_hash.rb

[tarcieri]

@josephglanville that's likely an MRI bug. Can you try 2.0.0?

[aflatter]

You original issue should be solved by b20f384.