Ecstortive

The browser script tags

Tue, 13 Dec 2011 20:58:03 +0500

I recently saw the announcement of Google Dart being part of Webkit as patch. Lots of people really started yelling. This post is just an opinion and a wish-list.

I would love to see dart as plug-in to browsers rather than a patch to Webkit!

If Google is really looking forward to have Dart as next language for webpages; I think its time to open the Pandora box and again start using the long forgotten language=”Dart” in script tags (other options may include language=”Lua”, language=”Coffescript”, language=”Brainfuck!” and who knows what). Remember when two giants were fighting we used to have “VBScript” in the language property? It’s long gone, but here is a chance for Microsoft again! Here is my wish list:

Scripting languages that ship as extension to browsers, just like Flash today! This is will open up true competition between developers (Google vs. Microsoft vs. Mozilla vs. Opera?) truly promoting the nature of open platform, where you get choice of choosing your language. Just think about it; if Javascript also becomes an extension to browsers, you would be able to plug v8 to Firefox (Ya I sound crazy)!
Script tags remain there with no changes, execution however depending upon the availability of language (plugin) mentioned in attribute:
<script type=”text/dart” language=”Dart”>/* My Dart Code */</script>
BAL, yes a Browser Abstraction Layer. Since all of the extensions would need a interface to interact with DOM, and various components in browser we would definitely need one!
Since we can switch between languages, we can also get options for compiled scripts! Which will take out the overhead of compression, minification and at times obfuscation crap. We would be able to ship byte-codes over the wire which means lesser size (compiled .pyc files ennnh) and more efficient VMs!

I have not yet calculated the Pros and Cons completely but as a web developer this is my wish list. What do you think?

The key value store everyone ignored (Postgresql)

Sun, 11 Dec 2011 17:43:00 +0500

Yes I know you are really happy with your “persistent” Key Value store. But did anybody notice hstore that comes along Postgresql. I find Postgresql to be a really great RDBMS that has been ignored all the time. It even has some great publisher/subscriber system as well (or LISTEN/NOTIFY in terms of Postgresql) that a lot of people may have implement using Redis, RabbitMQ etc. For people who have not lived anything other than MySQL. I would simply ask them to try out Postgres.

Instead of looking at benchmarks, I will be focusing on a key value store that is ACID compliant for real! Postgres takes advantage of its storage engine and has an extension on top for key value storage. So plan is to have a table can have a column that has a datatype of hstore; which in turn has a structure free storage. Thinking of this model multiple analogies throw themselves in. It can be a Column Family Store just like Cassandra where row key can be PK of the table, and each column of hstore type in table can be imagined like a super column, and each key in the hstore entry can be a column name. Similarly you can imagine it some what like Hash structures in Redis (HSET, HDEL), or 2 or 3 level MongoDB store (few modifications required). Despite being similar (when little tricks are applied) to your NoSQL store structures, this gives me an opportunity to demonstrate you some really trivial examples.

Lets setup our system first. For my experiment I will be using Postgres 9.1 and I will compile it from source. Once in source directory you can: ./configure && make install to install your Postgres. Don’t forget to install the extensions in the contrib directory: cd ./contrib && make install. Once you have setup the database you can create your own database and start the server (Hints: use initdb and pg_ctl). Then launch your psql and make sure you install your hstore extension:

CREATE EXTENSION hstore;
SELECT 'foo=>bar'::hstore;

If everything goes well you should be able to see table output. Now we are ready to do some DDL. I created a table my_store as schema definition below:

CREATE TABLE my_store
(
  id character varying(1024) NOT NULL,
  doc hstore,
  CONSTRAINT my_store_pkey PRIMARY KEY (id)
)
WITH (
  OIDS=FALSE
);

CREATE INDEX my_store_doc_idx_gist
  ON my_store
  USING gist
  (doc);

As you can see I’ve created a table with hstore column type and one GiST index (for operators ? ?& ?| etc.). You can checkout of documentation to have a look on different type of operators you have.

Now that we have database and tables setup I wrote a simple script to populate it with about 115K rows from twitter stream. Now keep in mind that its a real life data and I was interested in querying few basic things from collected data. For example, how many people are putting hash tags, or doing mentions, or were posting links in the tweets? For doing this I wrote a simple python script using tweepy and psycopg2 and ran it for about few hours. For each tweet in my store I added a key value pair of ‘has_hashtags=>:t’ if there were any hash tags in the tweet, similarly I introduced has_urls and has_mentions if they were present in tweet, I will be using these keys along with my GiST index to query my table later on.

So after populating my data with 115,142 tweets the database grew to a size of 239691780 bytes (Just 228MB). Now comes the fun part. I was totally blown away by what I can achieve by combining the power of relational and key value style under 1 store. So for example I want to query all the tweets tweeted at unix timestamp of 1323446095 (since I stored the timestamps as a string here is what my query looks like):

SELECT doc -> ‘text’ as tweet, doc -> ‘created_at’ as created_at
FROM my_store
WHERE doc @> ‘created_at=>00001323446095’;

I can add simple count or any other SQL famous aggregate function without going into any complications of my data store specific map reduce or new language to learn hustle. Do note that I padded my timestamp value with zeros since I am only storing strings as values. Also I am utilizing @> operator, thats gonna use the GiST to really do a quick bitmap index scan instead of sequential scan. That was pretty good for starter. Lets try to fetch out all the tweets that had hash tags in them:

SELECT doc -> ‘text’ as tweet, doc -> ‘created_at’ as created_at
FROM my_store
WHERE doc @> ‘has_hashtags=>:t’;

Yes querying complete database pulling out complete data (That you won’t probably do because you page the data :) ) gives me 14689 rows just under 360ms on average. Since we have SQL at hand lets make a condition little more complicated, and use a different operator for same stuff and also sort the data by created_at:

SELECT doc -> ‘text’ as tweet, doc -> ‘created_at’ as created_at
FROM my_store
WHERE doc @> ‘has_hashtags=>:t’ AND doc ? ‘has_urls’
ORDER BY doc -> ‘created_at’ DESC;

It already sounds tasty! This is not it Postgres has more operators, so pulling out hash tagged tweets with urls or mentions is also possible,

SELECT doc -> ‘text’ as tweet, doc -> ‘created_at’ as created_at
FROM my_store
WHERE doc @> ‘has_hashtags=>:t’ AND doc ?| ARRAY[‘has_urls’, ‘has_mentions’]

This is not it! hstore comes with all sort of operators and index systems that you can ask for hash store. Check them out here. Now, despite the NoSQL boom I think we have some great examples and reasons of why RDBMS still remains core part of many market giants (Facebook being something everyone knows). Postgres just gives me one more reason to not ignore RDBMS systems, So If you have been moving around on some document stores just because the reason that RDBMS don’t provide them; think again! You can get the same rock solid durability with structure free systems.

I will be pretty soon revisiting the FriendFeed use case with MySQL to store structure free data with Postgresql approach. Stay tuned, leave your comments and thoughts.

Oracle NoSQL simplicity and benchmarks

Mon, 14 Nov 2011 20:25:27 +0500

Today I will be doing the long awaited initial benchmarks of Oracle’s NoSQL. Before I jump in I would like to mention that I had a previous post on Oracle’s NoSQL that examines the philosophy and design of Oracle NoSQL store. It’s been few days I’ve been playing around with system. I found it pretty simple due to its major and minor key system; some great multi-get, iterator constructs for read, basic delete, and put system with versions. I did find some short comings, and we will see them in details at the end of this post. So lets get started.

For sake of benchmarking I am going to make things really simple. I will be benchmarking the insert and read speeds of a single node (1 machine no replication, no distribution) and look how a single node performs, based on that we can make rough estimates of performance gains by adding more machines.

Machine used for benchmarks is a pretty normal laptop machine with Dual Core 2.1GHz processor, 3 GB RAM, and commodity HDD. As an example I implemented a twitter like user stream of 100,000 tweets for 100 users randomly (that make approx 1000 tweets per user) each tweet with size of a little more than 100 bytes. So what we will be benchmarking is a speed test for inserting 100,000 tweets (approx 100+bytes) for inserting and reading them all. They should be pretty good experiments (close to a real world scenario) to give us an idea of what Oracle NoSQL can do.

Here is the piece of code that inserts and reads the 100,000 million entries and benchmarks the total time consumed. Compiling and running this I get output as following:

Write Time consumed 143156
Iteration Total time taken 223

Please note the time consumed in above benchmark is number of milliseconds. It turns out that inserting 100,000 entries in random order (since user ids generated are random) take 143156 ms and iterating over each entry of each user take 223 ms. I am pretty satisfied with read speed, and for write speed I found and average time of 1.4 ms (which is slower than reads but almost 700 inserts per second). Its important to note that each insert here is disk synchronized (with durability of Durability.SyncPolicy.SYNC on master, and since we have only 1 node means we are doing the most disk write with no buffering). If I lower the durability value (Durability.COMMIT_WRITE_NO_SYNC write with buffering), the average time drops to 0.5 ms; which is almost double the performance of previous version (or Durability.COMMIT_SYNC). It’s worth noting that you can also customize durability per transaction, which is great for letting a programmer choose what he wants.

I didn’t stop here and continued to implement a example twitter class, which proves the simplicity of what such this simple structure can do. I think the core power lies in not being a monolithic (JSON like) tree like object, and allowing us to pull data on partial paths as well as from full major key path. The class is used in this example code to create a user, authenticate, tweet, and query time-line of user.

Somebody may wonder how it compares with other data stores (document free, column based, key value). I can definitely achieve same effect by other key value stores (some what close as well), but I must say this style was much breeze and easy to imagine (just look at the code for reading tweets and you will know what I am saying), Oracle must rename store type from Key Value store to a new title. Because it’s pretty clear that its not a key value store; it overlaps between the document and key value style (Again I achieve this same effect in key sorted KV stores like LevelDB or TokyoCabinet).

In closing I must also give out some negatives I found, plus the some extras. Currently the number of partitions on Oracle NoSQL are fixed! What does this mean? It implies that at storage can’t be scaled horizontally yet since they can’t rebalance; once partitions have been made they remain fixed. Although Oracle in it’s documentation mentions about the next version having the re-balancing feature. Second (may be most for people) hurting part is dependence on Java platform. Now this will hurt lot of Ruby, and Python etc. people but again, as I said in my previous post, with help of a Java programmer you can write a REST API for yourself (I am planning a Protobuff server to eat this plague). Third, I found administration of system to be a little tricky (could not be understood without reading complete administration docs), but this is no hurdle for a programmers since it provides launch and go script to run the server. Right now I am planning to write a Scala wrapper (syntactic awesomeness) for Oracle NoSQL, so once I am done I will put it on GitHub. You folks in the mean while have few more calories of this NoSQL deliciousness.

Oracle NoSQL First Impression

Thu, 03 Nov 2011 23:23:01 +0500

Today I finally downloaded a community version copy of Oracle NoSQL, I am pretty happy to see a market giant like Oracle in the NoSQL era (Now waiting for DB2 guys to come in too). I’ve used Oracle 9i way back in my history and have been using MySQL for like decades (almost daily basis) and I know how feature rich these platforms are. Now since Oracle has landed with their NoSQL machinery, I just went through documentation from developers perspective to have a glimpse of feature sets. Lets go through highlights really quickly:

It’s a Key Value store that’s scalable, HA, predictable, and really simple.
Keys can behave like a composed structure of major, and minor keys. Major keys can be imagined just like path in your storage files (Hassan/Zohaib where there are two major keys Hassan and Zohaib) and minor keys can just be imagined like files in the (given major key) path holding your data in bytes (Hassan/Zohaib.birthdate, Hassan/Zohaib.phonenumber, Hassan/Zohaib.image, Hassan/Zohaib.userID holding different type of data). There can be multiple major and minor keys. I find this depth-limitation free structure to be extremely powerful, and really close to JSON structures (Yes it can act as document oriented database when crafted carefully).
Supports typical CRUD operations with high definition over consistency and durability, which lets you choose the amount of chocolate sauce you can afford on your taste buds.
A powerful Multi-Get, Get-Iterators, Set-Iterators, and depth controls that can set your retrieval on steroids.
One word that says it all. Transactions!
Last but not least. Well documented! Some of NoSQL projects have been suffering with this problem, so right from version 1.1 you are getting a really good documentation which is a big plus.

Despite the good features, I feel it has some set backs or I may say a wish list that can actually be fulfilled:

You need to talk Java, this will benefit the JVM dialects like Coljure, JRuby, Scala etc. But we can always write some REST API like HBase did in shape of Stargate. I won’t recommend that and I think Oracle should quickly figure out a good protocol (something like Memcache protocol will work) to do so.
Intelligent client library makes client end thick by maintaining state tables, calculating bounded network hops efficiency, and using major-minor keys to exploit the shards for efficient storage. I find it something like libmemcached or mongos (workings of mongos as library). I can’t call it a set back because some people prefer this way of sharding and distribution. But my favorite is Riak or Cassandra styled model of peers.

I can pretty much see how basic and really powerful this tool can turn out to be. I’ve only considered the Pros and Cons from programming point of view (API) completely ignoring administration and other parts (Will be doing a detailed benchmarks next). I must say I am pretty satisfied until now, and my first impressions are pretty good. Stay tuned for a detailed benchmarks, and stress test.

RVM+Ruby 1.9.2+Thrift 0.7.0 = RRT = Really Really Trashed!

Sun, 30 Oct 2011 22:15:25 +0500

OK, so I am writing this post after a whole day spent in effort to get Thrift working on Ruby 1.9.2, so that I can run the Cassandra gem. Now what I always expected out of Gem system is breezy a experience. But today it was really rough. Now some people and Google results suggest that its due to my Ubuntu 11.10 distribution, but if its due to Ubuntu guys why ain’t anybody reported on Ubuntu bug tracking! Some people say its due to Ruby, and there are lots of endless discussions with no conclusions.

Anyways I don’t want any of you to waste your time on the same exercise that I did today and waste a Sunday! Hopefully this post will help you fix issue. So here we go if you right now do gem install thrift on your system and you get error like

…(Some compile commands)
ruby/missing.h:157:20: note: previous declaration of ‘strlcpy’ was here
make: *** [struct.o] Error 1”

You are victim of RRT (Really Really Trash)! Here is how to fix it. First download the original Thrift 0.7.0 source code distribution from apache’s site. Extract, configure, and install it with your Ruby 1.9.2 activated (in case you use RVM). Here are commands to do so:

rvm 1.9.2
tar vxzf [your thrift archive].tar.gz
cd thrift-0.7.0
./configure
make && make install

Be sure you have ruby development library installed, and things will go fine. Next just to make sure you get the native thrift client for your activated Ruby, inside thrift source directory go to lib/rb (cd lib/rb) and run rvm 1.9.2 && ruby setup.rb. In case of Ruby 1.9.3, (Running ruby setup.rb) I encountered same as before error saying “struct.c:28:1: error: static declaration of ‘strlcpy’ follows non-static declaration”, to fix this error just open up the ext/struct.c (inside the same lib/rb directory) and remove the static keyword from line 26. Once you have done that you can now install native client of thrift. Lets install the gem now!

First download the thrift gem from here. Make sure its version is 0.7.0 (matches thrift protocol you compiled). Once done do gem unpack thrift-0.7.0 to unpack the gem contents into thrift-0.7.0 directory. Go to thrift-0.7.0 director and you will find similar structure as lib/rb in thrift source code (not exactly similar but I am mentioning ext directory specifically). Again open up the same file (struct.c) and remove the static keyword from line 26. Running ruby setup.rb inside gem directory should be error free now. Since I am using bundle to do package management a one liner in Gemfile should do the job for me.

gem “thrift”, :path => ‘thrift-0.7.0’ #path to modified gem directory

And thats about it. Launch your IRB an things should load up. Hope it saves you a day!

"I know you never wanted to die; neither did I want you to die. But I feel left alone with two..."

Thu, 06 Oct 2011 11:03:00 +0500

“I know you never wanted to die; neither did I want you to die. But I feel left alone with two realities; one I won’t be able to see you, and second I won’t be able to find a someone like you. You were right! It (Death) is Life’s change agent.”

- For Steve Jobs from Zohaib Sibt-e-Hassan

A journey to center of Redis

Sat, 17 Sep 2011 21:27:02 +0500

I’ve been using Redis lately and I really love the work they have done. In one of my recent projects I had a typical scenario of having users on different channels, and since the old front-end (Javascript) was doing a JSONP through jQuery, I had to take in account for people who were not able to get back to channel due to some disconnection. Yes, its a typical scenario of “timeout” operation. Within a particular time limit (30 seconds in my case) if a user is not coming back or responding back, you have to mark him as offline user (or may be different in your cases).

Timeout is a very typical operation that you usually come across all the time, In my case there were few options I had to get around this issue (I was using tornado for handling requests, and Redis for storage):

On every request before returning use add_timeout, keeping a Dictionary of ‘user_id’ -> add_timeout_returned_id; each time user comes back we remove the timeout event by looking up the user id in dictionary. If user doesn’t come back within time limit the callback fired will clean the things up.
Keep time stamps against user id in Redis, run a background process cleaning those who didn’t updated their time stamp and have their entry left in redis. The background worker will be a separate configurable instance running after intervals and cleaning up the dead users.
Use the lovely EXPIRE feature of Redis to remove the entries from DB and the worker process which looks for missing (expired) entries in channel and notifies other users with PUBLISH.

Now first entry, won’t be scalable since we are making the single python instance responsible for maintaining timers for same user, plus it complicates the code (yes I hate complicated code). You can although use cookies with NGINX to load-balance and assign same server again to same user, but this is what I am looking forward to avoid. Second approach is not real-time since we are using worker with update intervals to clean the mess I simply don’t like it. Third approach sounds fine from DB point of view, but again its not real-time. Exploring and digging deeper I found there was not way to have PUBLISH foo bar DELAY 30 (Publish after delay of 30 seconds). *GOSH* that would have killed the ugly duckling! Unfortunately for meeting deadline I took the 3rd approach since it was memory effective on DB side and keeping my code cleaner when compared to other approaches.

Last night I sat with this idea in my mind to have a PUBLISH event on a configured channel when a key expires (Cure for the itch). Walking around Redis community I found some really complicated discussions going on; with ideas of meta publish channels and some really brainy ideas stuff. I wanted to keep it simple, clean, configurable and that’s where opensource really helps; you have a different opinion go create your own flavor. I created my own fork and implemented what was really simple (and yummy chocolate flavored :P). So with new fork, Redis configuration can now contain an additional parameter named keys-expire-notify (so keys-expire-notify !KEYS will make Redis do a “PUBLISH !KEYS <expired_key>” every-time a key expires).

This feature will open up a whole new world of possibilities for the Redis users, including DB updates on cache invalidation, cascaded updates in Redis, time out events from Redis (Yes you can use keys to make it happen), Imagination is your limit! I’ve been playing around with the thing for a bit longer now and I am pretty much satisfied. It should be pretty much stable, since its built on top of existing infrastructure. You can grab your copy here and enjoy the new feature! I am really looking forward to see this idea as official part of Redis.

The Node Redumption!

Wed, 07 Sep 2011 23:40:00 +0500

In my previous post I did some benchmarks for NodeJS vs Java’s NIO. For benchmarking I wrote my custom event driven loop handling, and parsing HTTP connections; and what else you can expect! Hell broke loose when the NodeJS fans read that. They had sort of weird comments and objections. From those all the weirdness I learned a few things. Things that were pretty simple and straight forward. This post will be my last post on all I have to say about NodeJS, and Java (Yes, I won’t be comparing them again!), and other than just theory (as usual) I will present the experiments. While reading this post please bear in mind, I am not a technology maker or an Omega of any big platform; I am one of the developers among the race of curious ones, who when encounters a new technology readily accepts it, experiments with it and then places it in the right shelve (just like a weapon) for the right use case. The reasons behind my last blog post include clarifications (to my self and people around) about pros and cons, a solid idea of what’s going on inside, and what if I never had NodeJS on hand (more precisely how Facebook and Twitter guys handle so much load). I totally believe in greedy approach to problems, where I say “To optimize the bigger problem, you have to optimize the subpart of the problem”.

I know I annoyed alot of people just by comparing Apples and Bananas (Java and NodeJS). Both languages have a ridiculously different paradigm (OO vs Prototypal) and totally different maturity level (15 years of engineering on Java). So this post is to put both tools in “same situation” and experiment what are the results. Some people thought JIT, and JVM has nothing to do with IO based system, well I won’t say anything in reply to it yet. Let’s proceed with first benchmark.

This time I am taking the same NodeJS with Cluster (Thanks to suggestion in previous post), to add multi-core CPU utilization support (Like always you can suggest if there is a better option). Also I changed the code to do some really basic and simple processing, using a counter with timestamp system that just prints a simple message with number of hit counts, and timestamp. For Java I have a special arrangement this time; every body last time objected that my code was pretty unfair to the amount of processing NodeJS (http module) has to do, so I am making it fair this time. Netty NIO is the new contender in the ring for Java this time; with its fully implemented HTTP support, NodeJS people now cannot object on anything regarding the event loop and parsing system. The code gets divided into four classes now main launcher, pipeline factory, handler (NodeJS guys would be interested in this one), and a singleton. Pretty fair pieces of codes for both sides, both utilizing 1 worker per CPU core (I have a dual core), and having full fledged HTTP protocol support. Lets have look at benchmarks again! I used concurrency level 1000 and NodeJS managed to get only 5.2K req/sec [Raw Dump], a increase of 200 over last time (That’s pretty sweet you wasted my time again!). Java Netty NIO on other hand manages to handle 7.3K req/sec [Raw Dump], a decrease of 700 over the last time (now you know why I have decided not to write on both of these again); thats still over 1.7 times faster.

A basic application

Somebody suggested me to put them in a real scenario (do something with some storage and serve JSON and all typical stuff) and then do the benchmarks. Well taken, I picked Redis (Good enough for speed) for both of them and made an application where hit ‘counter’ now lies in Redis along with the hash ‘channel’ that contains a mapping of ‘timestamp’ => ‘counter value’. On each request I will check if the ‘counter’ exists; if no then I initialize it to 1 by set operation and set the hash value by hmset, if yes I increment the existing value and save that incremented value to map with hmset. Pretty basic and simple to understand (Like assigning a unique id to a user and making him join a chat channel). It can’t get any simpler and practical. So I am putting NodeJS redis (used npm install hiredis redis; its all steroids!) client’s “asynchronous coolness” with Java’s uncool asynchronous Jedis and Jackson for JSON serialization. Later on I discovered lettuce for Netty NIO as well, but I was too lazy to re-do benchmarks. Using lettuce will definitely improve the performance (or will it?). Let NodeJS have that advantage this time and see what results are we looking at. Here is the updated NodeJS code, and updated handler class and singleton class of Java.

Now back to benchmarks; running benchmarks with concurrency 1000 on NodeJS had a knockout, I started receiving connection reset by peer, super long delays (even with total 10000 requests it seemed like taking hours) and Redis running out of file numbers (Yes I’ve opened my limits and done everything possible but it was not working). So we have to lower the concurrency a little bit (since all this is running on same machine); with concurrency level 100 I was able to run benchmarks so I took it as standard point for Java too. With concurrency 100 NodeJS was able to make 1.1K req/sec [Raw Dump], and Java Netty NIO still managed 1.7K req/sec [Raw Dump]; after concurrency level 100 I tried shots on concurrency level 500 and results were pretty much same [NodeJS ~0.9K max][Java ~1.4 max]!

Update

After some fiddling around I was able to make Node work! Pretty much same results (Infact they went down ~0.7K [Raw Dump]), Netty keeps itself solid state [~1.3K Raw Dump].

So in conclusion:

NodeJS is good for long open connections (I’ve experienced that), but after this experiment I am pretty confident to roll something out on Netty NIO as well, the same design choice was made by plurk people (using NodeJS) but they reverted back to Java Netty NIO.
Speed of execution has alot to do with what you can push out, Ruby is suffering today due to its bad garbage collector and speed (That’s why Twitter abandoned it, that’s why Linked in is using JRuby with Jetty stuff to make things happen). Reason I always praise Java is when it provides you with so many options to tune your application according to your requirement (Yes its not easy but its fruitful).
Buzz words are dangerous! Stop looking at top trends and then adapting the technology.
If you are good at X you can achieve almost the same scale in X as that of Y (Not always true) but only if you are really good at X (Facebook took PHP to next level, MySQL still lives in their core, Heroku guys have been scaling alot of apps on notorious Ruby, there are many examples out there, just dig!).
If NodeJS was the best tool for scaling your requests it should have killed us all (even cockroaches).
Node will mature over the time; I have a strong believe over that, and I will be a happy coder using Javascript as server side too.
I am not asking you to leave NodeJS (who will help it grow then), things evolve out of experiments and we have to do it, but before holding the sword in your hand you should know the pros and cons.
Stop yelling and bragging about NodeJS being fast!
Thanks to everyone, I was pushed to do this experiment and make picture even clear! Lets move web forward.

Node on nails!

Fri, 02 Sep 2011 01:43:00 +0500

Before I start this post let me be very clear and precise “I am not advising you to leave NodeJS or shift to Java [PERIOD!!!]”.

I’ve been in this debate and there is a misunderstanding among my geek friends that NodeJS is the future and blah blah blah… I totally love Javascript (No bragging I have a history of being Javascript expert and I’ve been writing some love for JS too); there is no doubt about the beauty of closures, and the prototype styled programming. But when it comes to have Javascript as back-end, its totally a different story.

I find it pretty amusing when people are doing benchmarks of some solid technology stack to NodeJS, and declaring NodeJS to be the fastest thing on earth (just Google NodeJS vs *anything that comes to your mind* and you’ll find results like this, this, and what not). Beyond any doubt NodeJS model is worth attention, but will I use it in my production environment? I doubt that. I’ve worked on NodeJS on real serious basis; and experience was pretty bad. I had to write a complete HTTP client library with Multipart support (Today known as Reston) so that I can send files to Amazon S3 and few other REST services (at that time there was no existing library with HTTP Multipart support, HTTPS had issues and it was all really frustrating for me), to summarize it I have to say following things to my audience:

Not every web application is many connections application, you don’t code a chat system or a comet system everyday. NodeJS is good at few things and should be used for them. If you ask me to code a websocket based chat system using a IRC server, I would vote for NodeJS; but if you ask me to pull emails from all your email accounts and store them in some RDBMS or NOSQL store I would rather think twice.
Technology stack is important! Accept it! Live with it! I’ve seen people making wrong choices on technology stack (then bragging about it just because they used NodeJS) and then abandoning it, because later they discovered a better tool for doing the job.
If its about event based code and its readability I am pretty sure everybody agrees on the point that callback code is more spaghetti code than normal flowing code.
Static typed languages have a clear advantage over dynamic typed languages. You won’t have any trouble absorbing this fact if you know compiler internals.

Enough of my experience lets proceed to benchmarks. I have a typical commodity machine (3GB RAM, Dual Core processor) and as ground rules I will be doing straight forward benchmarks of Java NIO receiving a HTTP request, and responding hello world to it; same applies for NodeJS.

NodeJS code is pretty straight forward. I’ve used Node version 0.4.9. Notice that it relies on the typical ‘http’ module, which in turn relies on ‘net’, ‘stream’ etc. all of them are NodeJS libraries (nothing special from me) and they rely on JIT of V8 (accept the C parts) for speedy execution.

Now for Java I took NIO of Java and selector channels to achieve the same effect as NodeJS (single thread event dispatch). Code as expected is a little longer since I had to write the looping part too, but results are worth the experience. I’ve tried to put everything into a single file as much as I can, so code is not modular/production-ready at all! Here are the two files Runner.java and core.SocketSelectorCore.java. I’ve used HashMaps, String split, indexOf etc. and done basic HTTP header parsing of the request to simulate a normal request flow (read, parse, respond cycle). I’ve used pretty inefficient methods, but they are only there to make things worst for Java.

So launching NodeJS with “node test.js” and running Apache Benchmark (ab -c 1000 -n 100000) 3 times with concurrency level of 1000 yields [detailed dump] an average of almost 4 to 5 K requests per second.

Now before putting the custom written Java NIO event based system on trial; I would like to mention few things. Java is a beast, you get so many options on how to tune the JVM ranging from garbage collection to heap sizes (Checkout Java documentation in-case you are interested). So for my scenario, I used following flags to run my JVM “java -server -XX:+PrintCompilation -XX:+UseConcMarkSweepGC Runner”. Notice I’ve used verbose mode for JIT compilation, so that I can know when JVM is warmed up and I can take my benchmark samples; plus I changed the GC method (I tried different methods, but I liked this one the most). After the JVM warmed up (Google it n00bs) and running same Apache Benchmarks [detailed dump], Java was able to manage 8 to 8.5 K requests per second.

I tried different Heap Sizes and played around with other parameters; results were interesting. I was always above 6K requests per second on my machine. Lowering the concurrency (-c 100) takes figures as high as 11K. If you notice closely I was shipping more bytes per request (as compared to NodeJS), but that didn’t hurt Java and even with my really stupid approach for storing headers I was able to make it this far. Having these results at hand I also played around with JRuby for syntactic awesomeness and wrote a throw away code (Complete replica of Java code). And with a pretty rough code and a minor different flags passed to JRuby, I was again able to manage 4 - 4.5 K requests per second.

So question now remains is why did I do this, and whats the take away? I think answers are pretty clear. I personally love Javascript and NodeJS but I won’t take the argument NodeJS can do X that language Y can’t do. I believe comparing Java or PHP or some X to NodeJS is a stupid buzz that appeared recently. JIT of Java is pretty much state of art, and Google guys are putting in a great effort to bring V8 up-to mark. Frameworks like Netty NIO and Mina have been sitting out there for long time, but have been ignored due to Java’s weird syntax, memory hunger or steep learning curve reasons (I am just saying). Having said all that I’ve just busted the myth, “NodeJS can handle more connections due to its async style, and it enforces you to write code in callback(async) style; thus you write better code”. My answer is pretty simple “Write the core part in Java, wrap it in the JRuby or Scala beauty; and you have a much better way of expressing event driven system”.

EDIT: September 2nd 2011

Well I’ve been busy with work lately and seems like I had a nuclear melt down; and before I knew it the whole reactor exploded! In my previous post Node on Nails (I just realized I am good in making titles like these), just to purse the curiosity I tried to create a basic Java based NIO HTTP system and see the numbers with Node. Here is what I think reactions concluded for me (Will be helpful to you and me when writing a blog post):

Never ever use the word n00bs (although I tried to point the people who hear about node and switch to it instantly) it makes me sound like attacking you personally.
Avoid scandalish titles which I think is right; but I can’t avoid it (I will try to I promise).
Never criticize the technology or any tool I have a crush on (NodeJS in this case); initial roll was pretty encouraging towards Java but later comments really show the people about to bang their head in the walls and not being able to convince me with anything concrete.
Starting an article with such disclaimer never matters; you have to show the other site no matter if it is openly available.
Languages and tools evolve over time; one of the good factors for evolution is criticizing mistakes! Fanboys don’t take criticism constructively so there must be another way around.
People in today’s world believe that static typed can be beaten with dynamic typed languages.
I always used this blog as personal space for letting out such benchmarks, ideas, discussions. But since people take it too seriously I have to be more careful making it more formal (People commented on English spell mistakes at times as if it was an English exam).

Although I was busy at work, but the exploding hatred called me to address the issue; don’t worry I will still be doing more work on it as I get more time.

Edit: 6-Sep 2011

I’ve done some more benchmarks with a full fledged framework. If you have objections on this benchmark try new ones out.

LevelDB vs Kyoto Cabinet my findings

Mon, 01 Aug 2011 10:56:48 +0500

I’ve been pretty excited about Google’s LevelDB, not to mention there are some really old tanks already in the battle field like BDB, Tokyo Cabinet (Kyoto Cabinet as new one), HamsterDB etc. Fortunately I’ve already worked with Kyoto Cabinet and when I looked at the benchmarks I was totally blown away. I have a strong prejudice to name Google, but the benchmark results were way too good to believe. I also had an objection; LevelDB benchmarks compared itself only to TreeDB of Kyoto and not the HashDB (If ignoring key ordering and treating LevelDB to be true key value only). Some of you may start yelling right away that comparing Hash structure to an SSTable is not fair or something. Well by end of this post I will be comparing TreeDB with LevelDB as well and my results were different from officially posted benchmarks.

So to start off I decided to do a simple insertion, and iteration comparison (ignoring order) of simple HashDB and LevelDB without any compression or any sugar both stores come with, it will be a simple (no forced writes) benchmark of both comparing how much time each takes. I have a pretty normal machine (aka a commodity machine) with a normal HDD, and Intel Dual Core 3GB RAM. So without wasting anymore time; lets dive into code. Here is code for HashDB Kyoto and LevelDB. Pretty straight forward; I’ve carefully placed benchmarks point to only measure the “real time” consumed by both excluding additional key preparation and value preparation parts. The idea is to insert and iterate 100,000 key/value pairs. Running the benchmarks yield (Please note the first line indicates the time to insert 100,000 KV pairs, and second line indicates the time it took to iterate all those KV pairs, every time db was created from scratch i.e. previous copy was deleted. I took total 3 runs of each):

Kyoto Cabinet 100,000 entries:

Time consumed was 0.328945
Time consumed was 0.207315
Time consumed was 0.286653
Time consumed was 0.210296
Time consumed was 0.284954
Time consumed was 0.208595

LevelDB 100,000 entries:

Time consumed was 0.44742
Time consumed was 0.171842
Time consumed was 0.449287
Time consumed was 0.169108
Time consumed was 0.444432
Time consumed was 0.169381

Iterated reads are pretty fast in LevelDB, but writes are almost twice as slow as Kyoto Cabinet. I didn’t stop here because I’ve not hit the weak spot of LevelDB yet (yes larger values). So in my next test I fixed the length of value buffer to 512 bytes instead of the small string length, results took the expected form:

Kyoto Cabinet HashDB 512 bytes 100,000 entries:

Time consumed was 0.549354
Time consumed was 0.249182
Time consumed was 0.451567
Time consumed was 0.236295
Time consumed was 0.568072
Time consumed was 0.240299

LevelDB 512 bytes 100,000 entries:

Time consumed was 5.09164
Time consumed was 0.302542
Time consumed was 5.16017
Time consumed was 0.301258
Time consumed was 5.32948
Time consumed was 0.300175

At this point I became suspicious about the TreeDB benchmarks by LevelDB being true on my machine; and guess what, I was right! Here are benchmarks of TreeDB of Kyoto Cabinet:

TreeDB with Fixed 512 byte value length:

Time consumed was 0.393798
Time consumed was 0.445125
Time consumed was 0.389259
Time consumed was 0.439923
Time consumed was 0.392562
Time consumed was 0.438211

TreeDB with small length values (same as 1st HashDB benchmarks above):

Time consumed was 0.39401
Time consumed was 0.322735
Time consumed was 0.445422
Time consumed was 0.349465
Time consumed was 0.287259
Time consumed was 0.258459

Finally I took the total entries to 1,000,000 and ran a benchmark again with TreeDB, and LevelDB; results are as following:

LevelDB 1,000,000 entries (small values to give it an advantage):

Time consumed was 5.9836
Time consumed was 1.91519
Time consumed was 6.03402
Time consumed was 1.90402
Time consumed was 6.01079
Time consumed was 1.88251

TreeDB 1,000,000 entries (small values):

Time consumed was 3.9333
Time consumed was 2.56793
Time consumed was 3.87633
Time consumed was 2.56836
Time consumed was 3.93033
Time consumed was 2.57917

Surprised? Same here! Now the question is am I missing something? I am planning to put up some more benchmarks with some real life scenarios and values (may be a tweet sized stuff). Until then I would love to have feedback, and know if I’ve done any mistake while benchmarking.

JRuby vs Ruby the 2011 shodown

Sat, 09 Jul 2011 00:28:00 +0500

Well its been long time and I’ve been sitting quite in a corner waiting for this loud fight between the two Rubies to end. I once had a debate with one of ruby pro (when 1.9.1 was newly out) that JRuby can always stay ahead in speed (There are a-lot details available on how it does that for example this one). I had to start a project recently and choose a platform; ruby was obvious choice, after doing first few iterations when it came to select a database I realised Java has much better support for NoSQL stores than ruby (I can see people getting offended here; but accept it and live with it!) which lead me to the same old discussion in my mind which one is faster? I poked around on Google and found some mixed results. So I made up my mind and decided to do benchmarks myself! Here are the rules of game:

I will give each ruby complete isolation (ruby 1.9.2 and jruby 1.6.3 both latest stables) and 3 runs on same code just before starting off I will take the system time in milliseconds and at the end print out the total time elapsed. Since I don’t care about the start up time consumed I am pretty carelessly selecting jruby optimising “server” mode (perfect for web application scenario). My machine is a normal laptop with Intel Dual Core , 3GB RAM, and a pretty normal HDD.

Time to bust the bubble!

Run 1 focuses on calculation intensive thing; for that I took one of codes from Internet of generating random numbers you can view source code.

Ruby Runtime:

2.3871424198150635
2.309985637664795
2.3948707580566406

JRuby Runtime:

2.22599983215332
2.19400000572205
2.21500015258789

Ok fine (it was close) may be Java’s JIT or something else made.
So, Run 2 focuses on class instantiation and dynamic module extension, you can view source code here.

Ruby Runtime:

10.353898525238037
10.456978797912598
10.417702436447144

JRuby Runtime:

7.58999991416931
7.75600004196167
7.68299984931946

I am stopping right here for now; I mean why can’t a native language code beat a same code on top of JVM? It’s pretty obvious, I have to make some more benchmarks before I make a decision and put the nail in the coffin; So until I do that you people to enjoy this, give me your feedback and may be start thinking about porting your code to JRuby if its worth it!

To be continued…

Yet NOSQL dogma

Mon, 11 Apr 2011 23:42:42 +0500

Recently I posted some benchmarks on how can you tame MySQL to get the Key-Value behavior, (You can do almost same to get that column family behavior its all about your imagination *grinnn*) lets bullet down exactly what I meant when I used MySQL with HandlerSocket:

First and foremost whatever you write or read is on disk (persistent)! Its not living in volatile memory (until you really make your table in-memory).
Concurrency; MySQL speaks for itself and I don’t think we need any discussion on this.
MySQL was used because it is free, plus I found Handler-Socket which saved me time.
Not being antiquated I use NOSQL technologies (I’ve used Cassandra, Redis, MongoDB, TokyoTyrant, CouchDB, Redis and HBase nothing to brag about; but for proof of fact that am not technology allergic or afraid of something).
A relational database won’t be relational until you really define relationships yourself (and enforce them)! So don’t blame them for slowing down an insert just because you enforced a refrential integrity constraint, or unintentionally did a join. I totally hate this part, because people usually take so much pain in de-normalizing data for NOSQL databases, but when it comes to a SQL database they feel offended by denormalized structure.
We get a stable storage engine! I am mentioning it again because I don’t believe in stability of NOSQL storage systems, they are pretty young when compared to these RDBMS reptiles.
There is no deterministic way for guaranteeing the number of disk seeks or reads until you go for really raw disk level, and believe it or not some databases support that (MySQL in my case).

Given all this I did some benchmarks and as expected the NOSQL community was hurt and this is what one of them thinks:

with a similar setup, a NoSQL solution like Redis might give you even more operations per second

key-value only access might not be enough. Many NoSQL solutions are offering at least MapReduce support.

— Quoted from mynosql blog

And I think answers to what they say are pretty simple. Redis is not something to be compared to MySQL; why? Because, its not truly persistent. You have options to either do Append Only File fsync which can be timer based (not reliable) or on every write (try this option and your system will be dead doing writes all the time!), or save based on time interval in seconds (not reliable as quoted by Redis themselves), or the manual save command (a bomb in your head if you use save button frequently). In short Redis is truly memcached on steroids. Comparing MySQL to Redis is like comparing two different genres; where choosing one is matter of your requirements.

To answer MapReduce part; let me be very clear MapReduce was made by Google to meet its own needs, which is no where near the general business needs. Plus MapReduce is used as substitute to SQL; and thats how powerful SQL is ( Yep they have articles on it ); its like a runtime compiler + storage engine. So its not fair to compare a stripped down version of a (at times pre-compiled) routine to such a flexible thing (SQL).

Keeping things simple, here is what I want to say: “Keep NOSQL as Not Only SQL, rather than NoSQL due to some buzz word or marketing cliche”. I’ve not been paid by SUN to market MySQL, but I completely hate it when people make useless assumptions about the speeds, and put so much effort designing so much denormalized NOSQL structures which could have been achieved in an SQL database (without scattering data, avoiding additional headache different API, updates, bugs and maintainability issues).

Compiling TokyoCabinet with Java API under Ubuntu

Thu, 24 Feb 2011 00:07:00 +0500

Duhh… It may waste a little time (Java part only) so I thought better make it a blog post; so that everybody can save some time.

Get your copy of TokyoCabinet and Java API ( both are available on same page ) and extract them; First you will have to compile TokyoCabinet and that I think is pretty simple; I compiled my copy disabling bzip and enabling 64 bit offset like this

./configure —disable-bzip —enable-off64
make
sudo make install
sudo ldconfig

Make sure you don’t get any errors; if you do google will help you out here.

Time for Java API where things got messy for me to figure out; for this you have to make sure you have openjdk-6-jdk installed (I also had openjdk-6-source on my system).

Once done go to tokyocabinet Java API source directory and now if you do a ./configure you will get an error saying:

configure: error: jni.h is required

After doing some find commands on my /usr/lib/jvm I figured out the problem and here is how you can solve the niche

CPPFLAGS=”-I/usr/lib/jvm/java-6-openjdk/include/” ./configure

Note the path /usr/lib/jvm/java-6-openjdk/include/ is the path were your jni.h is present; this will vary distro to distro and you can easily google for it. Once done under ubuntu doing a simple make will once again give you a huge list of errors (yep you guessed it). Here is how you will fix that under ubuntu (solution will vary from distro to distro):

INCLUDEDIR=/usr/lib/jvm/java-6-openjdk/include make

Bummers I got some errors again; but this time I modified the Makefile (Again will vary from distro to distro)

JAVAC = /usr/bin/javac
JAVACFLAGS = -source 1.4 -d .
JAR = /usr/bin/jar
JAVAH = /usr/bin/javah
JAVADOC = /usr/bin/javadoc
JAVARUN = /usr/bin/java

this time

INCLUDEDIR=/usr/lib/jvm/java-6-openjdk/include make
sudo make install
sudo ldconfig

works fine. Once compiled don’t forget to copy the libjtokyocabinet.so to your jre lib binaries (sudo cp /usr/local/lib/libjtokyocabinet.so /usr/lib/jvm/java-6-openjdk/jre/lib/i386/.). You can go ahead enjoy the treat of adding the .jar files as dependencies and have fun with the cabinet!

The NoSQL Dogma

Wed, 16 Feb 2011 23:25:29 +0500

Every dogma has its day — Anthony Burgess

And yes I’ve been living the NoSQL dogma! The technologies like HBase, Voldermort, CouchDB, and a huge list of others (each designed for a particular scenario) made me believe; that my favorite RDBMS MySQL has grown old and it can’t handle my data anymore. When ever I encountered the API’s of these noSQL tools, I realized they are super trivial ( there are exceptions like MongoDB ), they are in no way as “rich” as SQL itself is. Secondly the rich “structure” and the “value” is gone (document stores are not always usable in real world scenarios, most of the time we are doing the key value look up).
Long story cut short; everybody just ignores the effort these beautiful creatures (RDBMSs) put into your daily problems; take care of issues like parsing, queuing and all the dirty stuff they are doing for you. So next question should be; if I get rid of SQL from MySQL (Just a specific case I am pretty sure we can do it on other engines as well) would it be ( only My ) any faster? Seems like yes. I would be demonstrating the benchmarks in a while. But before proceeding I would like to mention clearly that these benchmark values have pretty clearly proved me that a RDBMS when stripped off from the additional over-heads performs as good as any persistent key-value store (Yep its my personal opinion and I know noSQL community is hurt by this statement). In my demo I will be replicating a presistent Key/Value store using MySQL and HandlerSocket.

Tools of trade
I spent a whole day compiling these tools on my machine but by the end of the day it was a satisfaction (You should try as well)! You will require:

A MySQL 5.5.9 ( Latest Stable till date ) Source Code
A HandlerSocket ( Latest pull from Github ) Latest Commit
Lovely Python 2.6.5 ( Comes on my debian )
Newborn Python Socket Handler ( Latest ) Homepage
And lots of patience for compiling MySQL+HandlerSocket

Setting the stage
Once compiled and system is ready next thing would be creating a basic schema and table for storing the values; here is my basic schema named ‘foo’ and table named ‘kv’:

CREATE TABLE `foo`.`kv` (
`key` CHAR(255) NOT NULL,
`val` TEXT NOT NULL,
PRIMARY KEY (`key`)
) ENGINE = MyISAM;

Darn simple (I’ve used key as CHAR for now, INT will be definitely faster); realistically 255 is more than enough for a key to store its value (until you prove your self a Jimmy).
For my Python code here is the complete code that I used to benchmark my results.

Machine specs
My machine is just a normal laptop with following specs:

Results
Yep time for moment of truth, I inserted a hundered thousand (100,000) entries with a varying key sizes (see the code you will get what I am saying its pattern is like key0 - key99999 ), but a value of constant string with a size of 1K ( If I was a twitter it would have been much less ). Here are the statistics (Full console dump):

100,000 writes took 29.6542 seconds ( ~ 3372 writes / second )
100,000 reads in order took 18.4197 seconds ( ~ 5429 reads / second )
100,000 reads in random order 17.0343 seconds ( ~ 5870 reads / second ) (Socking!)

I would like to add I have used the default settings of MyISAM ( default ones with make install ), plus I repeated the experiment without truncating the tables (Nothing just happened it remain almost same every-time).

Whats the point?

MySQL is a proven technology, its engines are stable and well known, and scalable ( Yep modern forks for clouds are already rolling one example is drizzle); so I can’t find any point in shifting to a technology thats a new born, adds more calories to my code, and then brag about it just because its a buzz word! Untill there is no genuine reason to have a noSQL its just a trendy statement! I am pretty sure by tuning some more parameters, I can take these reads and writes to a higher number. Taking the same code on a server machine will off-course make these numbers even better. Despite all factors missing on a normal machine it was an impressive performance. For comparisons sake I would be making no excuses and posting benchmarks for other noSQL systems from my machine.

So it was my last day of noSQL dogma what about you?

Puny standalone JavaScript custom PubSub events library

Sun, 09 Jan 2011 04:23:00 +0500

I’ve been always fascinated by the power of JavaScript. Recently I had a tour to Ruby and the event machine. I’ve got to say Ruby stands no where near the power and flexibility of JS (Gosh I am in love again). There is a list of custom event system available including ones from famous giants like jQuery, Mootools, YUI. The whole idea of having an event published, and subscribers responding to them reminds me of the small-talk (again Ruby has a good attempt but not as good as JS). Just in curiosity of custom events, I went drilling down how to do it, and the fantastic Dean Edward’s article derived me nuts! In my hunt for DOM styled event equalizer in JS I derived a trick, but later came across a PubSubJS library already doing that. But nevertheless I still beat it on size at cost of readability, and few features. So if you are curious or doing plain JS on mobile devices this is supposed to be helpful and fully customizable.

You can checkout a demo here (open debug console to see the magic)

Code is pretty minimal and fits under 404 bytes when compressed, have a taste:

PubSub = (function(){
  
  var lst = {};
  
  return {
    // Subscribe for msg with function "fun" scoped with "obj" and additional parameters "ar"
    sub: function(msg, fun, obj, ar){
      lst[msg] = lst[msg] || [];
      ar = ar || [];
      obj = obj || window;
      lst[msg].push({args: ar, me: obj, f: fun});
    },
  
    // Publish an event "msg" with event event.data = dat and que makes the call itself async (call to pub)
    pub: function(msg, dat, que){
      var me = arguments.callee;
      if(que){
        setTimeout(function(){
          me.apply(me, arguments);
        }, 0);
      }
      var curr = lst[msg] || [];
      
      curr.filter(function(el){
        try{
          el.f.apply(el.me, [].concat({data: dat}, el.args));
        }catch(e){
          setTimeout(function(){throw e;}, 0);
        }
        return true;
      });                   
    },
    
    // Un-subscribe "fun" from event "msg"
    unsub: function(msg, fun){
      var cur = lst[msg] || [];
      lst[msg] = cur.filter(function(el){
        return el.f !== fun;
      });
    }
    
  }
  
})();

FITBJS template engine - Ultra thin

Sun, 19 Dec 2010 19:01:00 +0500

Theory:

Last few days I’ve been looking a template engines and came across some really fine and sleek solutions including PURE, EJS, Jhon Resig’s Micro Templates, jsTemplates, YAJET and a list of on going ones. I was constantly in pursuit of simplicity, minimality! To sum up engines claiming micro were just and overkill for my scenario. Templating at times is pretty trivial a simple case might be display set of links fetched from JSON in anchor tags, displaying list of emails with DIV elements, replacing items in a template URL etc. Having such a full fledged template engine on such puny never makes sense to me (yes I did this in some of my recent projects). I used String.prototype.replace at a lot of places and did some boilerplate work for saving extra lives at times (Yep I know there are loads of template engines out there doing this; but I required a precisely fine tuned engine). So in my labs I did another experiment and came up with my own version of really bare-bones yet flexible template engine. Checkout demo here.

Practical:

Well here is what I came up with (not meant for good readability but for experts it won’t be a problem). But full-fills the purpose its under almost 350 bytes and when YUI compressed goes under 330 bytes:

function fitb(tar, data, opt) {
    var reg = /(:\w+)/g,
		iht = tar.innerHTML,
        tpl = ((tar._$$f = tar._$$f || iht)) || tar;
    opt = opt || {};
    return (
		iht && 
		(tar.innerHTML = ((opt.append && tar.innerHTML) || "") + (opt.pre||"")+fitb(tpl, data, opt)+(opt.post||""))
	) || (
		data.length && 
		data.map(function(el) { return fitb(tpl, el, opt); }).join(opt.glue || "")
	) || (
		tpl.replace(reg, function(match) { return (data[match.substr(1)] || opt.none || "");})
	);
};

Now what about usage

// :name is replaced by Looping and :type is replaced by array
fitb('hello :name this was a simple :type. ', {'name': 'Looping', 'type': 'array'}); 
// outputs "hello A this was a simple array. hello Looping this was a simple array. 
fitb('hello :name this was a simple :type. ', [{'name': 'Again', 'type': 'array'}, {'name': 'Looping', 'type': 'array'}]);

Wait that’s not it! It can do more once you use third parameter:

// outputs "hello A this was a simple array. 
hello Looping this was a simple array. 
fitb('hello :name this was a simple :type. ', [{'name': 'Again', 'type': 'array'}, {'name': 'Looping', 'type': 'array'}], {glue: '
'});

That’s still not enough, you can even do direct operations on a DOM node:

// innerHTML of DOM node is used as template string and cached
fitb(document.getElementById('myNode'), [{'name': 'Again', 'type': 'array'}, {'name': 'Looping', 'type': 'array'}]);
// Calling multiple times is not effected by innerHTML of DOM node (it cache's original template string).
firtb(document.getElementById('myNode'), [{'name': 'Some other data', 'type': 'array'}, {'name': 'More data', 'type': 'array'}]); 
// Append the templated results in myNode excellent for adding new data
firtb(document.getElementById('myNode'), [{'name': 'Again', 'type': 'array'}, {'name': 'Looping', 'type': 'array'}], {append: true});

It’s pretty minimal yet usable; you can checkout demo here.

Flash or HTML5 the digibats!

Sat, 13 Nov 2010 20:34:00 +0500

Flash is no longer necessary to watch video or consume any kind of web content

Apple CEO - Steve Jobs

When you’re displaying content, any technology will use more power to display, versus not displaying content. If you used HTML5, for example, to display advertisements, that would use as much or more processing power than what Flash uses.

Adobe CTO - Kevin Lynch

After the dog fight there is still an ongoing argument for the future of Flash. I may sound little biased during the article, but standing in designer’s shoes and being a programmer as well gives me a different perspective. First of all lets make it clear, Apple was the one with its open letter to start Flash War for real. I’ve got to comment on the statements Apple made, recently they announced about dropping Java support but suddenly OpenJDK was announced, which really confused me for second time (Flash first time). I am great fan of Apple especially Steve Jobs, he inspires me at times. But looking at pattern of events from past, I think Apple is not calculating anymore (first they dropped Flash but then allowed it in App Store, then Java and now the OpenJDK story), and I think its enough for us (me at-least) to convince us that we have to make our own choice in technology. Since we ultimately are the technology consumers not Apple! It’s almost the same mistake that we did when IE6 became our favorite browser.

Now for second part, is Flash really the cursed technology? I recently came across the site HTML5 vs Flash, and honestly speaking I found the Flash animations smoother than HTML5 ones (Try it yourself). I did notice the processor usage while I was having the fun and I think I can sum it up in a single statement from Apple in 20XX “Disable JavaScript in your browser; it will increase your battery life to 1 Month”. HTML5 was almost all the time taking more CPU usage than Flash. Here is one of the images from one of the experiments I tried:

I tried chrome 8 too as well it varied from 28% to 35%. I have to agree with Adobe CTO if its processor intensive in Flash it’s (most probably more) processor intensive in HTML5. I am not giving any clear decisions about which technology is ultimate winner; I am only saying HTML5 is immature (I said it again) and there is no way it’s justified for me to drop out Flash (In fact its the only graceful degrading to supply rich media experience).

I only want the designers and programmers to realize the responsibility; we are just in phase of recovery from IE 6 (yes I consider it as viral infection). Over here we have to setup a framework for next decade. So be smart, don’t be foolish!

Playing around for javascript classes

Tue, 26 Oct 2010 23:22:00 +0500

Closures is a well known methodology to have public and private variables under JavaScript objects(in my opinion I think we exploit the closure technology). After spending some long time Mootools class module (I simply love it), PrototypeJS, JS.Class , and Dojo I asked myself; why can’t I have same JSON styled public, private structure? The exquisite way I used to do “this.privateVariableName” in Java or C++ was still not available by any of libraries (at-least the ones I mentioned); due the constraint that if we introduce anything in this structure, it will be accessible like any public property. As an example:

var Klass = function(){
    this.x = 'hello';
};

var v = new Klass();
v.x; // gives me hello

So I spent some time hacking around, and I finally came up with a sleek hack to get the job done.

By exploiting Function.prototype.apply one can easily bind any arbitrary object to a method. So for each public method I can bind an object that contains both public and private data. After spending time, and putting my head into problem I finally came up with my own solution (For the time being I’ve named it OOrJa abrivating Object Oriented Javascript).

Since code is pretty immature yet, I know there is a long journey ahead before it really becomes stable and robustly usable.

Below is a sample code demonstrating a proof of concept about how simple and elegant code can look:

//Animal base class
var animal = Class({
    Init: function(){
        console.log('animal');
    },
     
    Public: {
        eat: function(){
            console.log(this.name+' eats');
            this.name = 'animal';
        },
         
        name: 'animal',
         
        walk: function(){
            this.pwalk();
        }
    },
     
    Private: {
        pwalk: function(){
            console.log(this.name+' walks');
        }
    }
});
 
//Dog derived class from animal overriding eat
var dog = Class(animal, {
    Init: function(){
        console.log('dog');
    },
     
    Public: {
        eat: function(){
            console.log(this.name + ' eats');
            this.walk();
        },
         
        name: 'dog'
    }
});
 
//Hound inherited from dog provides a new method howl
var hound = Class(dog, {
    Init: function(){
        console.log('hound'); // log constructor
    },
     
    Public: {
        howl: function(){
            console.log(this.name+" howls"); // log name
            this.eat();
        }
    }
});
 
 
var dh = new hound(); // logs animal, dog, hound
dh.howl(); // Logs dog howls, dog eats, animal walks

Sweet. It follows the perfect C++ styled inheritance, since variable name is both in dog and animal class, each one of them creates it’s own context, hence the output is dog eats, and then animal walks (since dog class has no implementation of walk it inherits it from the parent class animal; and under animal class the name contains value “animal”).

Based on feedback and the taste of my own food (ya I always use my own code) I am looking forward to improve it.

Bluraity

Wed, 13 Oct 2010 20:15:24 +0500

Bluraity:

Library emulating the blur effect on your HTML elements. I consider it to be the step into future of HTML, that will eventually lead to a CSS property may be something like:

blur-radius: 3px;

Use cases can be many ranging from gallery scripts to HTML forms.

IE 9 b5 vs FF 4 b5 vs Chrome 6 vs Opera 10.61

Thu, 16 Sep 2010 15:18:00 +0500

Ok since last night I have been listening and watching that IE 9 beating Firefox and microsoft doing some really unfair comparison on there site. So I thought why not give it a shot my self and give Firefox a fair chance to compete with other browsers (excluding Chrome since its already fast chrome 7 is even more faster on GPU) and have the so called SunSpider benchmark quoted by engadget myself (BTW I was not able to replicate results engadget reports). Lets see the results first:

I would say Internet Explorer has hardly reached the point where Firefox is already positioned. I won’t lie I am a Firefox fan and even with bias factor there is no reason for to say Internet Explorer 9 is a bad browser. But it really makes me wonder when a company like microsoft is doing a comparison on their official site and doing and changing it so frequently as if they made some assumptions about the checklist and someone added correction later on ( I am not joking here is the clipping that I posted to my friend last night from this link; it totally changed within half an hour).

Before I close let me declare this “I am not saying IE 9 is bad; its probably the best IE release I’ve seen till today; I am only criticizing the false paradigm that microsoft may create around itself and stop the good development in browsers giving us an other IE 6 nightmare”.