17 9 / 2011

I’ve been using Redis lately and I really love the work they have done. In one of my recent projects I had a typical scenario of having users on different channels, and since the old front-end (Javascript) was doing a JSONP through jQuery, I had to take in account for people who were not able to get back to channel due to some disconnection. Yes, its a typical scenario of “timeout” operation. Within a particular time limit (30 seconds in my case) if a user is not coming back or responding back, you have to mark him as offline user (or may be different in your cases).

Timeout is a very typical operation that you usually come across all the time, In my case there were few options I had to get around this issue (I was using tornado for handling requests, and Redis for storage):

  • On every request before returning use add_timeout, keeping a Dictionary of ‘user_id’ -> add_timeout_returned_id; each time user comes back we remove the timeout event by looking up the user id in dictionary. If user doesn’t come back within time limit the callback fired will clean the things up.
  • Keep time stamps against user id in Redis, run a background process cleaning those who didn’t updated their time stamp and have their entry left in redis. The background worker will be a separate configurable instance running after intervals and cleaning up the dead users.
  • Use the lovely EXPIRE feature of Redis to remove the entries from DB and the worker process which looks for missing (expired) entries in channel and notifies other users with PUBLISH.

Now first entry, won’t be scalable since we are making the single python instance responsible for maintaining timers for same user, plus it complicates the code (yes I hate complicated code). You can although use cookies with NGINX to load-balance and assign same server again to same user, but this is what I am looking forward to avoid. Second approach is not real-time since we are using worker with update intervals to clean the mess I simply don’t like it. Third approach sounds fine from DB point of view, but again its not real-time. Exploring and digging deeper I found there was not way to have PUBLISH foo bar DELAY 30 (Publish after delay of 30 seconds). *GOSH* that would have killed the ugly duckling! Unfortunately for meeting deadline I took the 3rd approach since it was memory effective on DB side and keeping my code cleaner when compared to other approaches.

Last night I sat with this idea in my mind to have a PUBLISH event on a configured channel when a key expires (Cure for the itch). Walking around Redis community I found some really complicated discussions going on; with ideas of meta publish channels and some really brainy ideas stuff. I wanted to keep it simple, clean, configurable and that’s where opensource really helps; you have a different opinion go create your own flavor. I created my own fork and implemented what was really simple (and yummy chocolate flavored :P). So with new fork, Redis configuration can now contain an additional parameter named keys-expire-notify (so keys-expire-notify !KEYS will make Redis do a “PUBLISH !KEYS <expired_key>” every-time a key expires).

This feature will open up a whole new world of possibilities for the Redis users, including DB updates on cache invalidation, cascaded updates in Redis, time out events from Redis (Yes you can use keys to make it happen), Imagination is your limit! I’ve been playing around with the thing for a bit longer now and I am pretty much satisfied. It should be pretty much stable, since its built on top of existing infrastructure. You can grab your copy here and enjoy the new feature! I am really looking forward to see this idea as official part of Redis.

  1. maxpert posted this