Building Data Services with XMPP PubSub

Many of the bloggers have been critical of twitter's instability and of it going down frequently against heavy traffic. This has been attributed by many to the use of RoR framework, rails architecture, ruby speed, database issues, twitter api interface, and so on. One thing to note here is that all the twitter clients keep flooding the twitter server every x seconds, traditionally known as polling even though there might not be update for the twitter stream.

At OSCON 2008, Evan Henshaw-Plath and Kellan Elliott-McCrea gave an interesting presentation on using XMPP based publish-subscribe model for online data services. The title of the presentation was "Beyond REST? Building data services with XMPP PubSub".

On monday friendfeed polled flickr nearly 3 million times for 45000 users, only 6K of whom were logged in. Architectural mismatch.

The above statement is from their presentation and explains how much of resources (cpu cycles, bandwidth, etc) are being wasted because of using polling techniques. They propose that XMPP based publish-subscribe model will greatly reduce the unnecessary transactions and overhead. Twitter is already providing XMPP streams for some of the other web services.

However not many services support publish-subscribe model. RSS based polling is very prevalent as of now. Time will tell how many of the current and upcoming services will support XMPP streams. But RSS is not going to fade away anytime soon.