Why We Built It

The simple answer is that we needed to scale. Prior to the development of the stream-ruby infrastructure, we relied solely on cron jobs for asynchronous event processing. Work that could not or should not have been processed as part of a given web transaction, was delayed or queued for processing at a predefined interval by numerous cron-based tasks.

However, as our user base grew and new feature requirements emerged (i.e. social status feeds, Elastic Search Indexing, broadcast email notifications, etc.), it became apparent that we needed a framework for more timely processing of tasks that were either too expensive and/or unnecessary to process before returning a response from a web request. Queueing and distributed processing was the answer.