Setting aside the fact that we are yet to define what “NoSQL” actually is, some of the attributes that we commonly glob under this label are: document based, schema-free, distributed and “scalable”. The fact that being distributed and being scalable are not one and the same is a subject for another post, instead let’s take a closer look at what schema-free and document-based actually means. In fact, let me jump ahead: I am genuinely surprised that we are yet to see a schema-free engine built on top of MySQL. I know, I know, but suspend you disbelief for a second, because it is not as outrageous as it sounds.

Document Based: a Double Edged Sword

The original reason for and the benefit of the relational model is that by constraining the data schema (read, eliminating structural complexity of the data, or decomposing it into relations), you actually gain power and flexibility in the types of queries you can execute against your database. Said another way, normalized data design allows us to have a general-purpose query language, which allows for queries whose parameters we do not even know at design time, whereas denormalized designs do not. What we loose in flexibility of our data structures, we gain in our ability to interact with the data. Hence, in theory, if you have no way to anticipate the types of queries in the future, a relation model is your best bet. Lose some, win some, chose your poison.

At the same time, we all know that “no join is faster than no join”. The inherent disadvantage of decomposing your data is the required assembly. If you are looking for “speed” or “scalability”, then denormalizing your data is usually the first step. The disadvantage? Now you have introduced a number of potential anomalies into your data: updates, inserts, and deletes can cause data inconsistencies unless you keep careful accounting of all duplication. One-to-One, and One-to-Many relations are usually easy to manage, but Many-to-Many in denormalized schemas are nothing but a recipe for disaster. That is, if you care about consistency.

Finally, since you lose the power of a general purpose query language (SQL), you are now at a mercy of the DSL provided by your new database. Mongo, Couch and many others had to introduce their own query language constructs alongside "map-reduce" functionality to address the problem of querying arbitrarily deep records. Now, I am a fan of both, but frankly, none I have worked with so far are as clean, or as easy to understand as SQL (case in point) - with the downside of making me learn yet another query language.

Schema-free != Document Based

Document based and schema-free are often used interchangeably, but there is an important difference: schema-free does not necessarily imply nested data structures. Likewise, just because MySQL is “relational” does not mean that it must be fixed to a predefined schema - at create time, maybe, but not at runtime. Intersect the two statements, and it means that there is absolutely no reason why we cannot have a schema-free engine in MySQL:

> schema-free.sql

mysql> USE noschema;
mysql> CREATE TABLE widgets;  /* look ma, no schema! */
mysql> INSERT INTO widgets (id, name) VALUES("a", "apple");
mysql> INSERT INTO widgets (id, name, type) VALUES("b", "blackberry", "phone");
 
mysql> SELECT * FROM widgets WHERE id = "a";
+---------+---------------+
| id      | name          |
+---------+---------------+
| a       | apple         |
+---------+---------------+
 
mysql> SELECT * FROM widgets;
+---------+---------------+--------+
| id      | name          | type   |
+---------+---------------+--------+
| a       | apple         | NULL   |
| b       | blackberry    | phone  |
+---------+---------------+--------+
 

As long as we avoid nested data structures, then there is no reason why we should be limited by the columns defined in our tables because we can compose and decompose any relation at runtime. Not only would this mean no migrations or need to store null values, but you could also keep all the tools, drivers, and the SQL query language while adding the full flexibility of being schema-free.

Schema-free DB on top of MySQL

Not able to find any project that would give me this behavior, I ended up prototyping it myself over the weekend, and believe it or not, it works just fine. In fact, the output above is from a real console session with MySQL. All it took is an em-proxy server with a little low-level protocol and query rewriting, and all of the sudden, my MySQL forgot that it requires a schema. Take it for a test-drive yourself (you will need Ruby 1.9):

git clone git://github.com/igrigorik/em-proxy.git && cd em-proxy
ruby examples/schemaless-mysql/mysql_interceptor.rb
mysql -h localhost -P 3307 --protocol=tcp

> schema-free-mysql.rb

# snip ... 
# build the select statements, hide the tables behind each attribute
join = "select #{table}.id as id "
tables.each do |column|
  join += " , #{table}_#{column}.value as #{column} "
end
 
# add the joins to stich it all together
join += " FROM #{table} "
tables.each do |column|
  join += " LEFT OUTER JOIN #{table}_#{column} ON #{table}_#{column}.id = #{table}.id "
end
 
join += " WHERE #{table}.id = '#{key}' " if key
 

mysql_interceptor.rb (MySQL Proxy in Ruby)

Downloads: 122 File Size: 0.0 KB

Of course, this is nothing but a cute code example nor does it even cover all the different use cases, but let us look at the feature set: driver support for every language (you can point Rails + ActiveRecord, JDBC, etc. at it out the box, no problem), tool support (GUI and command line), replication that works, basically impossible to corrupt, transactions, and so on. Not bad for half a day of hacking with a simple data model in the background:

Instead of defining columns on a table, each attribute has its own table (new tables are created on the fly), which means that we can add and remove attributes at will. In turn, performing a select simply means joining all of the tables on that individual key. To the client this is completely transparent, and while the proxy server does the actual work, this functionality could be easily extracted into a proper MySQL engine - I’m just surprised that no one has done so already. For a closer look, check out the proxy code itself, there are plenty of comments, which explain how it is all pieced together.

The gray zone of SQL vs NoSQL

So what is the point of all this? Well, I hope someone actually writes such an engine, because I believe there is a market for it. There is a lot to be said for a drop in, SQL compatible, schema-free engine, and unlike what the NoSQL propaganda may say, there is absolutely no reason why we can’t have many of the benefits of “NoSQL” within MySQL itself. There is no one clear winner for a database engine or model, so put some thought into your decision up front. Just because Mongo, TC, or Couch are 'document-oriented' or 'schema-free' does not mean they are necessarily better for your application. In the meantime, don't get me wrong, I am still rooting for all the NoSQL projects, as well as have high expectations for Drizzle - they are all doing fantastic work.

Reviewing the latest benchmarks shows Avro as fully competitive in both speed and size of the output data to PB and Thrift. Though, neither speed nor size, while critical components, were the motivating reasons for Avro. Interestingly enough, Avro was designed by Doug Cutting with the goal of making it more friendly to dynamic environments (Python, Ruby, Pig, Hive, etc) where code generation is often an unnecessary and an unwanted step. Unlike PB, or Thrift, Avro stores its schema in plain JSON, as part of the output file, which makes it extremely easy to parse (JSON parsers are easy and abundant) and avoid the need for extra IDL definition stubs and compilers (though if you really want to, Avro can generate code stubs as well).

Embedding IDL with Avro

The decision to embed the Avro data schema alongside the binary packed data opens up a number of interesting use cases. First, dynamic frameworks such as Pig, Hive, or any other Hadoop infrastructure (the goal of Avro is to become the standard data exchange and RPC protocol for all of Hadoop), can load and process data on the fly, without looking for or invoking an IDL compiler. Additionally, having the original schema also allow us to do “data projection”: if the reader is only interested in a subset of the data, then it can selectively parse it out of the stream, allowing for faster processing and easy "versioning" support out of the box. Let’s take a look at a simple example:

> avro-writer.rb

SCHEMA = <<-JSON
{ "type": "record",
  "name": "User",
  "fields" : [
    {"name": "username", "type": "string"},
    {"name": "age", "type": "int"},
    {"name": "verified", "type": "boolean", "default": "false"}
  ]}
JSON
 
file = File.open('data.avr', 'wb')
schema = Avro::Schema.parse(SCHEMA)
writer = Avro::IO::DatumWriter.new(schema)
dw = Avro::DataFile::Writer.new(file, writer, schema)
dw << {"username" => "john", "age" => 25, "verified" => true}
dw << {"username" => "ryan", "age" => 23, "verified" => false}
dw.close
 

Avro specification provides all the primitives types you would expect (string, bool, double, etc.), and also a number of complex types such as records, enums, arrays, maps, unions, and fixed. Also, default values and sort order can be applied for some of the types. The schema itself is a JSON document, which you can peek at in the header of any serialized Avro file, which means that when it comes to reading the data, we don’t need to know anything about the data itself, or alternatively, only read an available subset:

> avro-reader.rb

# read all data from avro file 
file = File.open('data.avr', 'r+')
dr = Avro::DataFile::Reader.new(file, Avro::IO::DatumReader.new)
dr.each { |record| p record }
 
# extract the username only from the avro serialized file
READER_SCHEMA = <<-JSON
{ "type": "record",
  "name": "User",
  "fields" : [
    {"name": "username", "type": "string"}
 ]}
JSON
 
reader = Avro::IO::DatumReader.new(nil, Avro::Schema.parse(READER_SCHEMA))
dr = Avro::DataFile::Reader.new(file, reader)
dr.each { |record| p record }
 

RPC with Ruby and Avro

The RPC piece of Avro is also pretty straight forward: the protocol is defined as an Avro schema, where both the inputs and the methods (along side with the request / response input and output parameters) are provided inline. In principle, this also means that given the right framework, different clients could easily negotiate different data-formatting on the fly, without having to worry about versioning or conditional code paths. A simple Mail protocol with Avro:

> mail-protocol.js

{
  "namespace": "example.proto",
  "protocol": "Mail",
 
  "types": [{"name": "Message", "type": "record", "fields": [
        {"name": "to", "type": "string"},
        {"name": "from", "type": "string"},
        {"name": "body", "type": "string"}]
       }],
 
   "messages": {
      "replay": { "response": "string", "request": [] },
      "send": { "response": "string", "request": [{"name": "message", "type": "Message"}] }
  }
}
 

Here we defined a "Mail" protocol, which takes as input a record of type “Message”, which in turn, includes three strings: to, from, and body. Additionally, we defined two available methods (send and replay), which our server and client can use, as well as, their input and output parameters. Check out the full Ruby implementations of the server and client in the repo.

Avro, Ruby and Hadoop

The Ruby implementation of Avro still needs a lot of love and polish (it currently has a distinct Python smell to it), but given the growing adoption of Hadoop and the rising popularity of all the adjacent frameworks, Avro is definitely here to stay and for good reasons. So, before you invent another serialization framework, grab the source, build the gem, and give it a try.

Chances are, you will have to run a mix of all of the above to cover all the cases (ex: WikiMedia's Nagios & Ganglia) since there is no all-in-one solution and each has its tradeoffs. On that note, for performance monitoring, Ganglia is definitely an option to explore (it seems that I tried everything but Ganglia first, and I wish I did so much earlier). Originally developed at University of California, it is an open-source project designed from the ground up to be a distributed monitoring system for high-performance system such as clusters and grids - let's see what that means.

Ganglia's Distributed Architecture

Ganglia is powered by three independent components: gmond, gmetad and a PHP frontend. Due to how the system is architected, all three could either run on the same host, or more likely, be distributed between a number of different nodes. Gmond is the workhorse responsible for gathering user specified stats and sharing them over the network: a gmond daemon runs on every monitored node. It is designed to be fast, portable, with low memory footprint, and comes with a number of native monitoring modules (disk, memory, network, etc). Importantly, the gmond daemon never actually persists any data (memory only) to optimize for speed. But that's not all, because it can also receive data from other gmond's, allowing us to build arbitrary hierarchies of nodes - this is how and why Ganglia is capable of scaling to thousands of nodes.

Unlike other monitoring solutions such as Nagios or Cacti, all of the Ganglia metrics rely on data push - there is absolutely no polling involved. The gmond daemons are all responsible for periodically gathering and distributing their stats upstream. However, the gmond's do not persist data, and that is where the gmetad daemon comes in. This daemon is responsible for collecting data from an arbitrary number of gmond's, or even other gmetad daemons, persisting the metrics into correct RRD (round robin database) files, and then making this data available to the PHP frontend (or any other service that consumes RRD's).

Distributed Monitoring & Custom Metrics

Wikimedia's Ganglia setup is a good example to dissect: it consists of two "clusters" (Florida and Kennisnet), each most likely running their own gmetad node, which is then monitored by a central gmetad node which stitches them together. Each cluster, in turn, has its own hierarchy of gmond nodes: squids, apaches, databases, and so on. All together, it is monitoring over 350 nodes without breaking a sweat - not bad. Alas, one gotcha to be aware of upfront: run your gmetad's off a ram-disk to avoid the IO bottleneck associated with updating hundreds of RRD's.

With default configuration the gmond daemon will automatically monitor over 20 core metrics: load, network in and out, disk, memory and so on. However, it is also easily extensible via several mechanisms: custom monitoring modules, or straight up gmetric command line client. As of version 3.1.0, Ganglia now offers a simple Python API which allows us to create custom metrics modules which will be integrated directly into the gmond process. Hence the gmond process will call the code, gather the metrics and then distribute the data for us. A great working example is Gilad Raphaelli's MySQL extension, which gathers over 50 metrics about your database, and creating your own is also pretty straightforward.

However, if writing a python script is too heavy weight, Ganglia also provides the 'gmetric' executable which you can call right from the command line. Give the metric a name, type and a value, and you are off to the races. In other words, you can use bash, Ruby, or anything that will execute from the command line, in combination with gmetric, to funnel data into Ganglia:

# submit "variable_name" which is a string, with value of "hello", remove this var after 600s
gmetric -n variable_name -t string -v "hello" -d 600

# submit "int_var", which is an 8-bit int, with value of 20, remove this var after 60s
gmetric -n int_var -t uint8 -v 20 -d 60

Connecting Ganglia With Ruby

One of the key advantages of Ganglia over Cacti or similar services is that there is no per variable data source setup. Once a metric is pushed to a gmetad node, it automatically gets its own RRD file, and appears on your dashboard - no configuration required, just bring up a new service, push the data and you're rocking! Now, if you want to monitor a Ruby process, you have a couple of alternatives: write a python module, or shell out to gmetric. Neither is optimal because while the first method requires more (Python) code and extra polling, the system exec option can also be prohibitively expensive in terms of performance. Thankfully, Ganglia uses a simple UDP protocol with XDR data formatting which means that with a little reverse-engineering, we can talk to our gmond process directly from Ruby (by pretending that it is gmetric generating the packets):

> gmetric.rb

require 'gmetric'
 
# generate metric from Ruby and send it over UDP
Ganglia::GMetric.send("127.0.0.1", 8670, {
  :name => 'pageviews',
  :units => 'req/min',
  :type => 'uint16',     # unsigned 8-bit int 
  :value => 7000,       # value of metric
  :tmax => 60,          # maximum time in seconds between gmetric calls
  :dmax => 300          # lifetime in seconds of this metric
})
 

gmetric (GMetric Ruby Library)

Downloads: 174 File Size: 0.0 KB

Install the gmetric gem, specify the hostname and port number of your gmond daemon and fire off your metrics via UDP directly into the monitoring system. This way, you can track latency, request rates, throughput, or any other application metric directly from Ruby and push it into Ganglia without any performance penalties.

Finally, once you have pushed a dozen new metrics, you can then also generate your own cumulative reports, which aggregate data from multiple sources (key database metrics, etc). Ganglia is an incredibly flexible platform, and the 3.1.x release has done a lot to improve the ability to customize and extend it to fit your applications. If you haven't already, definitely a tool to look into for your cloud applications.

However, while the initial focus centered around the potential speed improvements offered by the VM, it is the persistence and distribution aspects of the runtime which make it stand out - if it happens to be faster, so much the better. Based on the Smalltalk VM, it offers integrated persistence (with ACID semantics) and distribution. In other words, you can treat MagLev as a distributed database that is capable of running Ruby code and storing native Ruby bytecode internally. Now that's a mouthful, let's see what it actually means.

MagLev VM: Features & Limitations

The goal of the GemStone team is to write as much of MagLev as possible in Ruby (the standard libraries, the parser, etc), which has already resulted in some good collaboration and synergies with the Rubinius project. As of the first public beta release, the project passes over 27,900 RubySpecs, features a pure ruby parser (slightly modified fork of ruby_parser), and runs RubyGems 1.3.5 out of the box. Popular gems such as rack, sinatra, and minitest all run unmodified, and there is even work on FFI support for C and Smalltalk extensions.

The end goal is full RubySpec compatibility, support for Ruby 1.9, and of course, running Rails - a stripped down version was demoed at RailsConf '09, but more work still needs to be done to make it fully compatible. The VM also ships with a MySQL driver, which means that you can use MagLev as any other Ruby runtime to power your applications, or, you could leverage the built-in persistence API's. MagLev has a distinctly different VM architecture which allows it to persist and share both code and data between multiple runtimes and execution cycles, all through a straight-forward Ruby API! Incidentally, this is also the reason for lack of support of several ObjectSpace methods (garbage_collect, each_object), as the enumeration could potentially mean retrieving gigabytes of persistent objects.

To get started, install MagLev via RVM, or follow the simple instructions on the wiki.

MagLev VM Architecture

The first thing you will notice about working with MagLev is that before you can run the interpreter, you will have to launch the MagLev service itself (maglev start). Turns out, unlike other Ruby VM's, all of the core Ruby classes, and all other persisted code and data actually lives in a separate "stone" process. The VM's ("gems"), connect to the stone and retrieve all of their data from this service. Ruby classes are stored as bytecode in the stone server, which is transported via shared memory for local connections, and via optimized binary protocol for remote connections, to the local interpreter and then compiled down to native machine code. This is how object persistence is made possible in MagLev: the stone server is a standalone process that acts as a database for your Ruby bytecode!

The added advantage is that the stone server supports full ACID semantics, which means that multiple processes can interact with the same repository and share state, objects, and code. A simple example of sharing data between multiple runs:

> maglev-data.rb

 # persist a string in the stone server
 Maglev::PERSISTENT_ROOT[:hello] = "world"
 Maglev.commit_transaction
 
 $ maglev-ruby -e 'p Maglev::MAGLEV_PERSISTENT_ROOT[:hello]'
 > "world"
 

That covers a simple key-value example, but Maglev is also capable of transparently persisting entire object graphs without any data-modeling impedance mismatch:

> maglev-persist.rb

graph_node =<<-EOS
  class Graph
    def initialize; @nodes = []; end
    def push(node); @nodes.push node; end
  end
  class Node; end
EOS
 
# commit Graph class's bytecode into stone server
# - can also load external file: load 'class.rb'
Maglev.persistent { eval graph_node }
 
# build a simple in memory graph
g = Graph.new
g.push Node.new
g.push Node.new
 
# commit in-memory graph to stone server
Maglev::PERSISTENT_ROOT[:data] = g
Maglev.commit_transaction
 
############################
# in different process / VM:
graph = Maglev::PERSISTENT_ROOT[:data]
puts graph.inspect
# > #<Graph:0xa205f01 @nodes=[#<Node:0xa202d01>, #<Node:0xa202c01>]>
 

maglev (GemStone Maglev Ruby Repository)

Downloads: 161 File Size: 0.0 KB

Instead of using an ORM to map Ruby classes to rows or documents in a database, you can simply store the objects directly in the stone server and interact with them through multiple processes, all without any extra conversions or additional infrastructure. The only caveat is that you would have to build your own indexing structures to power search and lookups beyond the key-value semantics. The KD-Tree example is a great showcase of the power and flexibility this can enable.

MagLev at Scale & In-Production

While the "stone" server persists all the core Ruby classes and any additional data, the VM's ("gems") are not free. According to the documentation, each VM takes ~30Mb of memory at boot time and starts growing from there. On the other hand, the shared memory communication is extremely efficient, which means that hundreds of VM's can be run in parallel on a single box. GemStone claims production deployments of their Smalltalk VM on 64-128 core machines with up to 512GB RAM, running hundreds of concurrent VM's, and achieving over 10K transactions per second (TPS) on their "stone" servers - impressive numbers!

With the new Smalltalk VM (3.0) on the horizon and years of production optimization and research, MagLev is definitely a project to watch. GemStone team has recently started a blog, opened up a Google group, and are starting to produce some great content to help Rubyists leverage their platform. What is missing now are the deployments, case studies, and new frameworks that can leverage all of these features - though, I'm sure, that will come.

Of course, the challenge is that it has to be practical - it needs to be "real-time" and be able to react to emerging trends in under a minute, all the while using a reasonable amount of CPU and memory. Now, we don't know how the actual system is implemented at Twitter, nor will we look at any specific solutions - I have some ideas, but I am more curious to hear how you would approach it. Instead, I want to revisit the concept of Bloom Filters, because as I am making my way through the literature, it is surprising how sparsely they are employed for these types of tasks. Specifically, a concept I have been thinking of prototyping for some time now: time-based, counting bloom filters!

Bloom Filters: What & Why

A Bloom Filter is a probabilistic data structure which can tell if an element is a member of a set. However, the reason it is interesting is because it accomplishes this task with an incredibly efficient use of memory: instead of storing a full hash map, it is simply a bit vector which guarantees that you may have some small fraction of false positives (the filter will report that a key is in the bloom filter when it is really not), but it will never report a false negative. File system and web caches frequently use bloom filters as the first query to avoid otherwise costly database or file system lookups. There is some math involved in determining the right parameters for your bloom filter, which you can read about in an earlier post.

Of course, as is, the Bloom Filter data structure is not very useful for analyzing continuous data streams - eventually we would fill up the filter and it would begin reporting false positives all the time. But, what if your bloom filter only remembered seen data for a fixed interval of time? Imagine adding time-to-live (TTL) timestamp on each record. All of the sudden, if you knew the approximate number of messages for the interval of time you wanted to analyze, then a bloom filter is once again an incredibly fast and space-efficient (fixed memory footprint) data structure!

Time-based Bloom Filters

Arguably the key feature of bloom filters is their compact representation as a bit vector. By associating a timestamp with each record, the size of the filter immediately expands by an order of magnitude, but even with that, depending on the size of the time window you are analyzing, you could store the TTL's in just a few additional bits. Conversely, if counting bits is not mission critical, you could even used a backend such as Redis or Memcached to drive the filter as well. The direct benefit of such approach is that the data can be shared by many distributed processes. On that note, I have added a prototype Redis backend to the bloomfilter gem which implements a time-based, counting Bloom Filter. Let's take a look at a simple example:

> chrono-bloom.rb

require 'bloomfilter'
 
options = {
  :size	=> 100,       # size of bit vector
  :hashes => 4,       # number of hash functions
  :seed	=> rand(100), # seed value for the filter
  :bucket => 3        # number of bits for the counting filter
}
 
# Regular, in-memory counting bloom filter	
bf = BloomFilter.new(options)
bf.insert("mykey")
bf.include?("mykey")  # => true
bf.include?("mykey1") # => false
 
#
# Redis-backed bloom filter, with optional time-based semantics
#
bf = BloomFilter.new(options.merge({:type => :redis, :ttl => 2, :server => {:host => 'localhost'}}))
bf.insert("mykey")
bf.include?("mykey")  # => true
sleep(3)
bf.include?("mykey")  # => false
 
# custom 5s TTL for a key
bf.insert("newkey", nil, 5)
 

bloomfilter.git (Ruby+Redis counting Bloom Filter)

Downloads: 982 File Size: 0.0 KB

Storing data in Redis or Memcached is roughly an order of magnitude less efficient, but it gives us an easy to use, distributed, and fixed memory filter for analyzing continuous data streams. In other words, a useful tool for applications such as duplicate detection, trends analysis, and many others.

Mechanics of Time-Based Bloom Filters

So how does it work? Given the settings above, we create a fixed memory vector of 100 buckets (or bits in raw C implementation). Then, for each key, we hash it 4 times with different key offsets and increment the counts in those buckets - a non-negative value indicates that one of the hash functions for some key has used that bucket. Then, for a lookup, we reverse the operation: generate the 4 different hash keys and look them up, if all of them are non-zero then either we have seen this key or there has been a collision (false positive). By optimizing the size of the bit vector we can control the false positive rate - you're always trading the of amount of allocated memory vs. collision rate. Finally, we also make use of the native expire functionality in Redis to guarantee that keys are only stored for a bounded amount of time.

Time-based bloom filters have seen a few rogue mentions in the academic literature, but to the best of my knowledge, have not seen wide applications in the real world. However, it is an incredibly powerful data structure, and one that could benefit many modern, big-data applications. Gem install the bloomfilter gem and give it a try, perhaps it will help you build a better trends analysis tool. Speaking of which, what other tools, algorithms, or data structures would you use to build a "Trending Topics" algorithm for a high-velocity stream?

WebSockets in HTML5 change all of that as they were designed from the ground up to be data agnostic (binary or text) with support for full-duplex communication. WebSockets are TCP for the web-browser. Unlike BOSH or equivalents, they require only a single connection, which translates into much better resource utilization for both the server and the client. Likewise, WebSockets are proxy and firewall aware, can operate over SSL and leverage the HTTP channel to accomplish all of the above - your existing load balancers, proxies and routers will work just fine.

WebSockets in the Browser: Chrome, Firefox & Safari

The WebSocket API is still a draft, but the developers of our favorite browsers have already implemented much of the functionality. Chrome’s developer build (4.0.249.0) now officially supports the API and has it enabled by default. Webkit nightly builds also support WebSockets, and Firefox has an outstanding patch under review. In other words, while mainstream adoption is still on the horizon, as developers we can start thinking about much improved architectures that WebSockets enable. A minimal example with the help of jQuery:

> websocket.html

<html>
  <head>
    <script src='http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js'></script>
    <script>
      $(document).ready(function(){
        function debug(str){ $("#debug").append("<p>"+str+"</p>"); };
 
        ws = new WebSocket("ws://yourservice.com/websocket");
        ws.onmessage = function(evt) { $("#msg").append("<p>"+evt.data+"</p>"); };
        ws.onclose = function() { debug("socket closed"); };
        ws.onopen = function() {
          debug("connected...");
          ws.send("hello server");
        };
      });
    </script>
  </head>
  <body>
    <div id="debug"></div>
    <div id="msg"></div>
  </body>
</html>
 

The above example showcases the bi-directional nature of WebSockets: send pushes data to the server, and onmessage callback is invoked anytime the server pushes data to the client. No need for long-polling, HTTP header overhead, or juggling multiple connections. In fact, you could even deploy the WebSocket API today without waiting for the browser adoption by using a Flash socket as an intermediate step: web-socket-js.

Streaming Data to WebSocket Clients

WebSockets are not the same as raw TCP sockets and for a good reason. While it may seem tempting to be able to open a raw TCP connections from within the browser, the security of the browser would be immediately compromised: any website could then access the network on behalf of the user, within the same security context as the user. For example, a website could open a connection to a remote SMTP server and start delivering spam - a scary thought. Instead, WebSockets extend the HTTP protocol by defining a special handshake in order for the browser to establish a connection. In other words, it is an opt-in protocol which requires a standalone server.

Nothing stops you from talking to an SMTP, AMQP, or any other server via the raw protocol, but you will have to introduce a WebSocket server in between to mediate the connection. Kaazing Gateway already provides adapters for STOMP and Apache ActiveMQ, and you could also implement your own JavaScript wrappers for others. And if a Java based WebSocket server is not for you, Ruby EventMachine also allows us to build a very simple event-driven WebSocket server in just a few lines of code:

> websocket.rb

require 'em-websocket'
 
EventMachine::WebSocket.start(:host => "0.0.0.0", :port => 8080) do |ws|
  ws.onopen    { ws.send "Hello Client!"}
  ws.onmessage { |msg| ws.send "Pong: #{msg}" }
  ws.onclose   { puts "WebSocket closed" }
end
 

em-websocket (Ruby EventMachine WebSocket Server)

Downloads: 564 File Size: 0.0 KB

Consuming WebSocket Services

Support for WebSockets in Chrome and Safari also means that our mobile devices will soon support bi-directional push, which is both easier on the battery, and much more efficient for bandwidth consumption. However, WebSockets can also be utilized outside of the browser (ex: real-time data firehose), which means that a regular Ruby HTTP client should be able to handle WebSockets as well:

> em-http-websocket.rb

require 'eventmachine'
 
EventMachine.run {
  http = EventMachine::HttpRequest.new("ws://yourservice.com/websocket").get :timeout => 0
 
  http.errback { puts "oops" }
  http.callback {
    puts "WebSocket connected!"
    http.send("Hello client")
  }
 
  http.stream { |msg|
    puts "Recieved: #{msg}"
    http.send "Pong: #{msg}"
  }
}
 

em-http-request (Asynchronous HTTP Client)

Downloads: 298 File Size: 0.0 KB

WebSocket support is still an experimental branch within em-http-request, but the aim is to provide a consistent and fully transparent API: simply specify a WebSocket resource and it will do the rest, just as if you were using a streaming HTTP connection! Best of all, HTTP & OAuth authentication, proxies and existing load balancers will all work and play nicely with this new delivery model.

WebHooks, PubSubHubbub, WebSockets, ...

Of course, WebSockets are not the panacea to every problem. WebHooks and PubSubHubbub are great protocols for intermittent push updates where a long-lived TCP connection may prove to be inefficient. Likewise, if you require non-trivial routing then AMQP is a powerful tool, and there is little reason to reinvent the powerful presence model built into XMPP. Right tool for the right job, but WebSockets are without a doubt a much-needed addition to every developers toolkit.

Performance is only one aspect that influences the choice of a database. Tree and graph structures are not easy to model within a relational structure, which in turn leads to complicated schemas and system overhead. For that reason alone, document-stores (Tokyo, CouchDB, MongoDB), graph stores (Neo4J), and other alternative data structure databases (Redis) are finding fertile ground for adoption. However, the end of "RDBMS as one size fits all" does not mean the end of relational systems all together. It is too early to bury RDBMS in favor of No (or Less) SQL. We just need to reset how we think about the RDBMS.

Disks are the New Tape

The evolution of disks has been extremely uneven over the last 25 years: disk capacity has increased 1000x, data transfer speeds increased 50x, while seek and rotational delays have only gone up by a factor of 2. Hence, if we only needed to transfer several hundred kilobytes of data in the mid 80's to achieve good disk utilization, then today we need to read at least 10MB of data to amortize the costs of seeking the data - refresh your memory on seek, rotational, and transfer times of our rusty hard drives.

When the best we can hope for is 100-200 IOPS out of a modern hard drive, the trend towards significantly larger block sizes begins to make a lot more sense. Whereas your local filesystem is likely to use 4 or 8kb blocks, systems such as Google's GFS and Hadoop's HDFS are opting out for 64MB+ blocks in order to amortize the cost of seeking for the data - by using much larger blocks, the cost of seeks and access time is once again brought down to single digit percent figures over the transfer time.

Hence, as we generate and store more and more data, the role of the disks must inevitably become more archival. Batch processing systems such as Map-Reduce are well suited for this world and are quickly replacing the old business intelligence (BI) systems for exactly these reasons. In the meantime, the limitations imposed by the random access to disk mean that we need to reconsider the role of disk in our database systems.

OLTP is Headed Into Main Memory & Flash

An average random seek will take 5-10ms when hitting the physical disk and hundreds of microseconds for accessing data from cache. Compare that to a fixed cost of 5-10 microseconds for accessing data in RAM and the benefits of a 100-1000x speed difference can be transformative. Instead of treating memory as a cache, why not treat it as a primary data store? John Ousterhout and his co-authors outline a compelling argument for "RAMCloud". After all, if Facebook keeps over 80% of their data in memcached, and Google stores entire indexes of the web in memory many times over, then your average database-backed application should easily fit and be able to take advantage of the pure memory model also.

The moment all of the data is available in memory, it is an entirely new game: access time and seek times become irrelevant (no disk seeks), the value of optimizing for locality and access patterns is diminished by orders of magnitude, and in fact, entirely new and much richer query models can enable a new class of data-intensive applications. In a world where the developer's time is orders of magnitude more expensive than the hardware (a recent phenomenon), this also means faster iterations and less data-optimization overhead.

The downside to the RAMCloud is the equivalent order of magnitude increase in costs - RAM prices are dropping, but dollar for dollar, RAMCloud systems are still significantly more expensive. Flash storage is an obvious compromise for both speed and price. Theoretical access time for solid-state devices is on the order of 50 microseconds for reads, and 200 microseconds for writes. However, in reality, wrapping solid-state storage in SATA-like hardware devices brings us back to ~200 microseconds for reads, or ~5000 IOPS. Though, of course, innovation continues and devices such as FusionIO’s PCI-E flash storage controller bring us back to 80 microsecond reads at a cost of ~$11 per Gigabyte.

However, even the significantly higher hardware price point is often quickly offset once you factor in the saved developer time and adjacent benefits such as guaranteed performance independent of access patterns or data locality. Database servers with 32GB and 64GB of RAM are no longer unusual, and when combined with SSDs, such as the system deployed at SmugMug, often offer a much easier upgrade path than switching your underlying database system to a NoSQL alternative.

Database Architecture for the RAMCloud

Migrating your data into RAM or Flash yields significant improvements via pure speedup in hardware, however, "it is time for a complete rewrite" argument still holds: majority of existing database systems are built with implicit assumptions for disk-backed storage. These architectures optimize for disk-based indexing structures, and have to rely on multithreading and locking-based concurrency to hide latency of the underlying storage.

When access time is measured in microseconds, optimistic and lock-free concurrency is fair game, which leads to much better multi-core performance and allows us to drop thousands of lines of code for multi-threaded data structures (concurrent B-Trees, etc). RethinkDB is a drop-in MySQL engine designed for SSD drives leveraging exactly these trends, and Drizzle is a larger fork of the entire MySQL codebase aimed at optimizing the relational model for "cloud and net applications": massively distributed, lightweight kernel and extensible.

Migrating Into Main Memory

Best of all, you can start leveraging the benefits of storing your data in main memory even with the existing MySQL databases - most of them are small enough to make the memory buffers nothing but a leaky abstraction. Enable periodic flush to disk for InnoDB (innodb_flush_log_at_trx_commit=2), and create covering indexes for your data (a covering index is an index which itself contains all the required data to answer the query). Issue a couple of warm-up requests to load the data into memory and you are off to the races.

Of course, the above strategy is at best an intermediate solution, so investigating SSD’s as a primary storage layer, and if you are adventurous, give RethinkDB a try. Also keep an eye on Drizzle as the first production release is aimed for summer of 2010. Alternative data storage engines such as Redis, MongoDB and others are also worth looking into, but let us not forget: laws of physics still apply to NoSQL. There is no magic there. Memory is fast, disks are slow. Nothing is stopping relational systems from taking advantage of main memory or SSD storage.

In a short span of just a couple of years, the Ruby VM space has evolved to more than just a handful of choices: MRI, JRuby, IronRuby, MacRuby, Rubinius, MagLev, REE and BlueRuby. In fact, keeping up with all of the most recent developments within each VM is now easily a full-time job. For that reason, and with RubyConf ‘09 in full swing, let’s take a quick survey of the space and where it’s taking us.

2010: Year of Ruby Renaissance

Observing the trends and acceleration of development amongst all the VM’s, it is clear that 2010 is going to be an exciting year for the language. While more than a few developers have proclaimed Ruby (and more often, Rails) as dead within the past year, likely due to it losing the initial novelty angle, in reality the language is at the cusp of becoming available to a much broader community. Within the next year, MacRuby, Rubinius, IronRuby, and MagLev should all hit the 1.0 status, effectively making all the things we love about Ruby available to entirely new communities of programmers. HotCocoa with MacRuby makes writing Mac apps a breeze, IronRuby will bring Ruby scripting to the .NET crowd, Rubinius will become a viable deployment platform, and MagLev will give us the distributed persistence model offered by their Smalltalk VM. All of this without even mentioning the growing adoption of JRuby, which marries the best of JVM with Ruby, or the rising popularity of REE fork of Matz’s Ruby which offers significant performance and memory improvements.

In other words, if the “Ruby revolution is over”, then the next year is likely to be the first year of its Renaissance. It won’t happen overnight, but slowly and surely we will see the same idioms, tools, and DSL’s we are all accustomed to in Ruby make their way to adjacent platforms. Many projects and companies are already using RSpec and WebRat to test their non-Ruby code. Likewise, why not use Ruby for DOM manipulation via Silverlight (Gestalt), or abstract Java or Cocoa API’s into concise DSL’s? It’s an exciting time to be a Rubyist.

MRI: Matz’s Ruby (1.8.x / 1.9.x)

MRI Ruby, which is the original and the default platform for the vast majority of Ruby users have been making steady progress throughout the past year. First, we saw the 1.9.1 release at the beginning of the year, which made Ruby 1.9 a viable deployment platform, although the overall pickup has remained relatively low. In mid July, Ruby 1.9.2 preview 1 hit the shelves, and the original schedule planned for the final 1.9.2 release on December 25th. However, Yuki Sonoda (release manager) recently indicated that the schedule will be canceled in favor of making Ruby 1.9.2 compatible with the RubySpec suite prior to the final release - this is great news for everyone and well worth the wait.

Sitting in the audience at RubyKaigi in Tokyo earlier this year, it was clear that the focus of the development team is on Ruby 1.9. Moving forward, there will be one more release within the 1.8.x branch (Ruby 1.8.8), and it will serve as a bridge between 1.8 and 1.9. If you haven’t already, you should investigate migrating your code to Ruby 1.9 - most of the critical gems, and all popular frameworks work out of the box, not to mention the numerous performance improvements.

In the coming year, we may even see a Ruby 2.0, and in all likelihood a continual improvement in speed and library support. Matz showed off a number of experimental branches at RubyKaigi, and Koichi Sasada indicated that many of the performance optimizations are yet to be turned on for Ruby 1.9 - to date the focus has been on compatibility and feature completeness.

JRuby: Ruby on the JVM

Out of all the “alternative” Ruby VM’s, JRuby is by far the most mature project both in terms of compatibility and community coverage. By combining the best of the JVM platform - generational GC, true concurrency (no GIL), and transparent interop with any Java library - with Ruby syntax, it is no surprise that JRuby has been quietly gaining market share in the community. It is fast, it runs Rails, and it will soon be compatible with Ruby 1.9.

Within the past year the JRuby team, which consists of 7 active committers (3 of whom migrated from Sun to Engine Yard earlier this year) , and dozens of intermittent contributors has fixed more user reported issues than in all previous releases combined. JRuby 1.4.0 is a good indicator of the health of the project: a large collection of new features, and over 307 bug fixes since JRuby 1.3.1.

Tickets for JRubyConf were sold out in a matter of hours following the announcement, and in all likelihood serve as a good indicator that JRuby is the platform to watch in the coming year. The combination of the JVM optimizations and its widescale deployment within the enterprise world will definitely make it an appealing Ruby VM.

MacRuby: Objective-C, LLVM and Ruby

Want to mix and match Cocoa API’s or access OSX system libraries all within a Ruby VM? Then MacRuby is the answer and the VM is picking up steam fast. On March 9th, MacRuby 0.4 shipped with a threaded GC, full 64-bit support, DTrace probes, and many improvements to the HotCocoa API’s. Since then, the project has switched from YARV to a completely new VM based on LLVM compiler infrastructure (shipped in 0.5 beta 1), and the benefits are numerous: machine code compilation, true concurrency (no GIL), a working JIT, and even ahead of time compilation (AOT)!

In other words, MacRuby is now a true Ruby compiler. You can write a HotCocoa app, leverage native POSIX threads, or even take advantage of Apple’s Grand Central Dispatch (GCD) and then compile your program and distribute it as a binary to any OSX user.

With 7 members on the team and a growing community (RubyOnOSX), MacRuby is quickly becoming one of the most promising open-source projects for Apple - it would be great to see them officially embrace it in the coming year. With the new VM and numerous performance improvements, MacRuby has the potential to bring Ruby to all the Objective-C developers and open up an entirely new market for Ruby. In theory, we could even see MacRuby on the IPhone, that is, if we overcome a few minor snags, like a missing GC. Definitely a project to watch in the coming year as it edges towards the 1.0 status.

MagLev: Smalltalk VM and Ruby

The GemStone team has been quietly making steady progress with MagLev over the past year. The development is being done in parallel with work on their upcoming GemStone 3.0 VM and is promising to bring Ruby to their 64-bit Smalltalk VM, which offers a JIT, years of VM optimizations, and most importantly, a built-in persistence and distribution layer. In other words, you could think of Maglev as a distributed database that runs Ruby code internally - thousands of concurrent VM’s can be spread across hundreds of nodes, all accessing the same data with ACID semantics.

At RailsConf this year the team showed off a working Sinatra app, and since then they have gone from passing 5000 to nearly ~27900 RubySpecs. Rack, Sinatra, MiniTest and a few others already run unmodified on the VM and the tentative plan is for the project to hit 1.0 sometime in the upcoming year. At the moment, there is a closed alpha test in progress, but soon it will be opened to the larger public. (Update: grab the public beta here).

The persistence layer offered by the VM is definitely one of the most interesting features, but the team has also indicated that MagLev will support other persistence models as well - you’ll be able to use ActiveRecord with MySQL, etc.

Ruby Enterprise Edition

Launched in mid 2008, REE is a fork of Ruby 1.8.7 optimized for server and production deployments of MRI Ruby. Combination of MBARI patches, improvements in thread and scheduling overhead by Joe Damato and Aman Gupta, a copy-on-write (COW) fork model, and a tunable GC all contribute to a measurable difference in the amount of used memory and overall performance. Earlier this year Twitter switched their infrastructure to REE and reported a 30% improvement in throughput!

Phusion Passenger, developed by the same team, has also seen a good pickup in the community within the past year, but it remains to be seen if REE adoption will continue to grow in light of all the progress by alternative VM’s.

IronRuby: Ruby on .NET

IronRuby is a .NET implementation of the Ruby VM which leverages Microsoft’s Dynamic Language Runtime (DLR), and hence enables an entire host of interesting use cases for Ruby. For example, not only is there seamless integration with all the .NET libraries and infrastructure, but running on top of the DLR also means that Ruby can be run within Silverlight (yep, right in your browser - check out Gestalt).

The 1.0 release is looming on the horizon and the team has been making great progress. IronRuby now passes over 92% of all the RubySpecs and the adaptive compiler, combined with the optimized DLR also means much lower VM startup times, as well as significant performance improvements over MRI. (Update: download IronRuby 1.0 RC1)

It remains to be seen if the .NET community will embrace IronRuby, but it sounds like tool support (Visual Studio, Intellisense, etc) might be the next big hurdle for the project.

Rubinius: Ruby written in (mostly) Ruby

Rubinius is an initiative to implement as much of Ruby as possible via Ruby code itself - turtles all the way down. In the works since 2006, the Rubinius team lost a few developers early this year, but the project is alive and healthy. The VM has been rewritten in C++, a JIT compiler has been added, and you can now run mongrel, thin, rack, amqp, and a number of other gems all unmodified. The roadmap shows 1.0RC in the works, and a focus on enabling real-world application deployments (packaging, distribution, etc).

It is too early to talk about performance, but the combination of the JIT and LLVM infrastructure means that Rubinius will have true concurrency (no GIL), and plenty of opportunities for introspection and optimization of your code - an inherent benefit of rewriting Ruby in Ruby itself.

Both FFI and RubySpecs, which are now used by virtually every other VM are a direct result of the project, so all things considered, Rubinius can already be called a resounding success. However, with 1.0 on the horizon, it remains to be seen how and if the community will react to the release. Getting a few production deployments under the belt, as well as building a larger user community, are likely to be the big challenges for the year ahead.

BlueRuby: Ruby on ABAP VM

An exploratory research project form SAP Labs, BlueRuby is an initiative to bring the Ruby runtime to the ABAP VM powering the AP NetWeaver and SAP ERP 6.0 products. It is already passing 75.5% of all the RubySpecs and is being pushed by SAP as a way to adopt TDD practices within their products. At this point, there is no known release schedule or public roadmap for BlueRuby, so don’t expect Rails apps on top of SAP NetWeaver anytime soon, but it does have the potential to bring Ruby and all the best practices enabled by it to thousands of enterprise developers.

Rubies everywhere!

Talking to the developers of all of the alternative VM's you can't help but to feel excited about the future of Ruby. There is a clear pattern emerging: using Ruby to go beyond Ruby, and as a bridge to other communities. Halfway through RubyConf, it is a clear theme here as well - several serialization talks, code generation, and an entire evening of presentations on all the Ruby VM's.

The PubSub specification within XMPP, as defined in XEP-0060, is definitely not as flexible as that of AMQP, but it is often times enough to cover the most popular use cases. However, technical merits aside, one of the key missing components, especially in Ruby, has been the historical lack of functioning libraries - xmpp4r claims to support it, but examples are lacking. Thankfully, after test driving the latest batch of gems, it looks like we're finally there.

Getting off the ground with XMPP

Without a good toolkit XMPP can be a gnarly protocol to get started with - Pidgin IM client has some great tools for spying on the exchange, but monitoring pages of XML scroll by can only get you so far. Thankfully, Seth Fitzsimmons has built switchboard ("curl for XMPP"), which offers a powerful command line tool to greatly simplify the process. Make sure to read the full tutorial, or jump right into it by testing it with the Wordpress XMPP stream:

# list available options, subscribe to a blog, list subscriptions and then open the stream
switchboard disco --target pubsub.im.wordpress.com info
switchboard pubsub --server pubsub.im.wordpress.com --node /blog/icanhazcheesburger.com subscribe
switchboard pubsub --server pubsub.im.wordpress.com subscriptions
switchboard pubsub --server pubsub.im.wordpress.com listen

Based on xmpp4r, switchboard is also a toolkit for assembling your own XMPP clients, which means that it can be easily customized to power a PubSub consumer. From start to finish, and since examples are still hard to come by:

> switchboard-pubsub.rb

require 'rubygems'
require 'switchboard'
 
class WordpressJack
  def self.connect(switchboard, settings)
    switchboard.plug!(PubSubJack)
    switchboard.hook(:post)
 
    switchboard.on_pubsub_event do |event|
      event.payload.each do |payload|
        payload.elements.each do |item|
          on(:post, item)
        end
      end
    end
  end
end
 
settings = Switchboard::Settings.new
settings['pubsub.server'] = 'pubsub.im.wordpress.com'
settings['jid'] = 'user@im.wordpress.com'
settings['password'] = 'password'
 
switchboard = Switchboard::Client.new(settings)
switchboard.plug!(WordpressJack)
 
switchboard.on_post do |post|
  puts "A new post was received:"
  puts post.methods.sort.uniq
  exit
end
 
switchboard.run!
 

XMPP with EventMachine and Nokogiri

If you have an EventMachine stack, or looking for a high performance library, Jeff Smick's blather is definitely a gem to investigate. The combination of the asynchronous nature of EventMachine, a SAX parser within Nokogiri, and a great DSL make it very fast and a pleasure to work with:

> blather-pubsub.rb

require 'rubygems'
require 'blather/client/client'
require 'blather/client/dsl/pubsub'
require 'blather'
 
EventMachine.run {
  host = 'pubsub.im.wordpress.com'
  node = 'blog/icanhazcheesburger.com'
  user = 'user@im.wordpress.com'
  pass = 'pass'
 
  jid = Blather::JID.new(user)
  client = Blather::Client.setup(jid, pass)
  client.register_handler(:ready) {
    puts "Connected. Send messages to #{client.jid.inspect}."
    pub = Blather::DSL::PubSub.new(client, host)
  }
 
  client.register_handler(:pubsub_event) { |event|
    puts event
  }
 
  client.connect
}

PubSub & Event-Driven Architecture

Having personally struggled in the past with XMPP PubSub and Ruby, it's been great to revisit the use case and find a new set of fully functional libraries. The event driven architecture which is enabled by technologies such as XMPP, AMQP, Comet, Webhooks and PubsubHubbub are increasingly becoming the staple of many web applications, and for a good reason. If you haven't already, grab switchboard or blather and take XMPP for a test drive.

Coined by Alex Russell in early 2006, the term Comet is an umbrella term for technologies which take advantage of persistent connections initiated by the client and kept open until data is available (long polling), or kept open indefinitely as the data is pushed to the client (streaming) in chunks. The immediate advantage of both techniques is that the client and server can communicate with minimal latency. For this reason, Comet is widely deployed in chat applications (Facebook, Google, Meebo, etc), and is also commonly used as a firehose delivery mechanism.

Converting Nginx into a Long Polling Comet Server

A large entry barrier to Comet adoption is the implicit requirement for specialized, event driven web servers capable of efficiently handling large numbers of long polling connections. Friendfeed's Tornado server is a good example of an app level server that meets the criteria. However, thanks to Leo Ponomarev's efforts, you can now also turn your Nginx server into a fully functional Comet server with the nginx_http_push_module plugin.

Instead of using a custom framework, Leo's plugin exposes two endpoints on your Nginx server: one for the subscribers, and one for the publisher. The clients open long-polling connections to a channel on the Nginx server and start waiting for data. Meanwhile, the publisher simply POST's the data to Nginx and the plugin then does all the heavy lifting for you by distributing the data to the waiting clients. This means that the publisher never actually serves the data directly, it is simply an event generator! It is hard to make it any simpler then that.

Best of all, it only gets better from here. Both the client and the publisher can create arbitrary channels, and the plugin is also capable of message queuing, which means that the Nginx server will store intermediate messages if the client is offline. Queued messages can be expired based on time, size of the waiting stack, or through a memory limit.

Configuring Nginx & Ruby Demo

To get started you will have to build Nginx from source. Unpack the source tree, grab the plugin repo from GitHub and then build the server with the push module (./configure --add-module=/path/to/plugin && make && make install). Next, consult the readme and the protocol files to learn about all the available options. A simple multi client broadcast configuration looks like the following:

> nginx-push.conf

# internal publish endpoint (keep it private / protected)
location /publish {
  set $push_channel_id $arg_id;      #/?id=239aff3 or somesuch
  push_publisher;
 
  push_store_messages on;            # enable message queueing 
  push_message_timeout 2h;           # expire buffered messages after 2 hours
  push_max_message_buffer_length 10; # store 10 messages
  push_min_message_recipients 0;     # minimum recipients before purge
}
 
# public long-polling endpoint
location /activity {
  push_subscriber;
 
  # how multiple listener requests to the same channel id are handled
  # - last: only the most recent listener request is kept, 409 for others.
  # - first: only the oldest listener request is kept, 409 for others.
  # - broadcast: any number of listener requests may be long-polling.
  push_subscriber_concurrency broadcast;
  set $push_channel_id $arg_id;
  default_type  text/plain;
}
 

Once you have the Nginx server up and running, we can setup a simple broadcast scenario with a single publisher and several subscribers to test-drive our new Comet server:

> comet-push-consume.rb

require 'rubygems'
require 'em-http'
 
def subscribe(opts)
  listener = EventMachine::HttpRequest.new('http://127.0.0.1/activity?id='+ opts[:channel]).get :head => opts[:head]
  listener.callback {
    # print recieved message, re-subscribe to channel with
    # the last-modified header to avoid duplicate messages 
    puts "Listener recieved: " + listener.response + "\\n"
 
    modified = listener.response_header['LAST_MODIFIED']
    subscribe({:channel => opts[:channel], :head => {'If-Modified-Since' => modified}})
  }
end
 
EventMachine.run {
  channel = "pub"
 
  # Publish new message every 5 seconds
  EM.add_periodic_timer(5) do
    time = Time.now
    publisher = EventMachine::HttpRequest.new('http://127.0.0.1/publish?id='+channel).post :body => "Hello @ #{time}"
    publisher.callback {
      puts "Published message @ #{time}"
      puts "Response code: " + publisher.response_header.status.to_s
      puts "Headers: " + publisher.response_header.inspect
      puts "Body: \\n" + publisher.response
      puts "\\n"
    }
  end
 
  # open two listeners (aka broadcast/pubsub distribution)
  subscribe(:channel => channel)
  subscribe(:channel => channel)
}
 

nginx-push.zip (Full Nginx Config + Ruby client)

Downloads: 415 File Size: 2.7 KB

In the script above, every five seconds a publisher emits a new event to our Nginx server, which in turn, pushes the data to two subscribers which have long-polling connections open and are waiting for data. Once the message is sent to each subscriber, Nginx closes their connections and the clients then immediately re-establish them to wait for the next available message. End result, a real-time message push between the publisher and the clients via Nginx!

Long Polling, Streaming, and Comet in Production

Leo's module is still very young and is under active development, but it is definitely one to keep an eye on. The upcoming release is focused on bug fixes, but looking ahead there are also plans to add a streaming protocol: instead of closing the connection every time (aka, long polling), Nginx would keep it open and stream the incoming events as chunks of data to the clients in real-time. Having such an option would make it ridiculously easy to deploy your own firehose API's (ex: Twitter streaming).

Last but not least, don't forget about the growing number of other available modules for Nginx, or if you are so inclined, get a head start on building your own by reading Evan Miller's great guide on the subject.