Grails vs Rails Performance Benchmarking

Intro

Recently someone asked the question: "Where are the Grails vs Rails benchmarks??". Since we actually haven't got round to optimising Grails I thought it was silly that we should benchmark a 0.4.2 release against Rails which is now at 1.2.x. Nevertheless, to allay concerns about Grails I decided we should do it anyway. This page represents the results.

The reality of this benchmark is that it is actually more like Rails vs Groovy + Spring MVC + Hibernate + Sitemesh. Grails can take some credit of course, but the heavy lifting is being done by Spring & Hibernate in particular.

I am by no means a benchmarking expert, so if you have any changes or recommendations please shout

Update - After feedback from Jared Richardson who detailed how with Rails & Mongrel you actually need multiple Mongrel instances to be equivalent to tomcat I decided to conduct some further tests. This basically showed my naivety when configuring Rails, but there you go, so I apologise for the falseness of the original benchmarks.

To be clear what I have done is to configure Rails with a 10 Mongrel cluster and Pound as per Jared's tutorial here. To make things fair I also reduced Tomcat to use only 10 Java threads. The results can be seen in the updated benchmarks that follow.

If I have done something wrong in the Rails configuration please point it out, because if you read on, doing the clustering didn't help much.

Hardware Specs

We are trying to be entirely open about this benchmark. Personally I feel benchmarks are of limited value, but some live by them so there you go. To get some of the details out the way then. The test hardware is as follows:

Apple MacBook 1.83ghz Intel Core Duo 1GB 667 Mhz DDR2 SDRAM

Ok I know, I know it isn't a server, but I didn't have one lying around, so this will have to do. It has dual cores after all ;-)

Software Specs

Now onto the software platforms:

Grails

• OS: Mac OS X 10.4.9
• Server: Apache Tomcat 5.5.20
• Version: 0.5-SNAPSHOT from 20th of March
• Environment: Production
• Database: MySQL 5.0.27
• Java: Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_07-164)
• JDBC Driver: mysql-connector-java-3.1.10-bin.jar

I decided to benchmark with 0.5-SNAPSHOT because it implements the new URL mapping feature and since Rails has routes it evens the playing field out a bit. In other words it would be unfair on Rails if we benchmarked with Grails not having a routing feature, as it does in the upcoming 0.5

Configuration A: Standard Tomcat

The first test was with a standard Apache Tomcat 5.5.20 configuration, with the following settings:

<Connector port="8080" maxHttpHeaderSize="8192"
               maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
               enableLookups="false" redirectPort="8443" acceptCount="100"
               connectionTimeout="20000" disableUploadTimeout="true" />

Configuration B: Tomcat configured with a 10 thread pool

To even things up with Rails having 10 mongrel's I decided to configure Tomcat with only 10 threads with the following configuration:

<Connector port="8080" maxHttpHeaderSize="8192"
               maxThreads="10" minSpareThreads="1" maxSpareThreads="1"
               enableLookups="false" redirectPort="8443" acceptCount="100"
               connectionTimeout="20000" disableUploadTimeout="true" />

Rails

Set-up according to the article here: http://hivelogic.com/narrative/articles/ruby-rails-mongrel-mysql-osx

• OS: Mac OS X 10.4.9
• Server: Mongrel 1.0.1
• Version 1.2.3
• Environment: Production (Started with mongrel_rails start -e production)
• Database: MySQL 5.0.27
• Other notes: I installed the Ruby MySQL native bindings as per the aforementioned article

Also note that I have not bothered to connect Grails or Rails up to Apache using mod_jk or the equivalent Rails FastCGI connector.

Configuration A: A single mongrel instance

The first Rails configuration was using only a single mongrel instance and hitting it directly on port 3000.

Configuration B: 10 mongrels with load balancing by Pound

The second configuration I used a 10 mongrel cluster configured as per Jared Richardson's instructions here and configured it with this command:

sudo mongrel_rails cluster::configure -e production -p 8001 -N 10

sudo mongrel_rails cluster::start

ListenHTTP
   Address 0.0.0.0
   Port 3000
End
Service
  BackEnd
    Address 127.0.0.1
    Port 8001
  End
  BackEnd
    Address 127.0.0.1
    Port 8002
  End
  .. etc ..
  BackEnd
    Address 127.0.0.1
    Port 8011
  End
Session
Type BASIC
TTL 300
End
End

The Test Applications

Now onto the nature of the tests. So basically I didn't want to get into anything too fancy, so I'm testing these things:

1. Read operations
2. Create operations
3. Queries
4. Update operations
5. View rendering vs. writing directly to the response

class Book {
	String title
	String author
	String description
	Date  dateCreated = new Date()
	static constraints = {
		title(nullable:false, blank:false)
		author(nullable:false, blank:false)
	}
}

Then ran the following command to generate the base controllers and views:

grails generate-all

Similarly with Rails I created the following table in mysql:

!Picture 9.png|thumbnail!

To create this yourself you can do it in MySQL's tools or by running the DDL:

CREATE TABLE `books`.`books` (
  `id` INT NOT NULL AUTO_INCREMENT,
  `title` VARCHAR( 255 ) NOT NULL,
  `author` VARCHAR( 255 ) NOT NULL,
  `description` TEXT,
  `date_created` TIMESTAMP NOT NULL,
  PRIMARY KEY (`id`)
)
CHARACTER SET utf8;

I then created the following model in Rails:

class Book < ActiveRecord::Base
end

And then ran the command

ruby script/generate scaffold book

The Test Data

With that done, the way I setup some test data was to create an action to do so in each respective controller. In Grails I did this:

def data = {
     def i = 1000
     while( i > 0 ) {
         def book = new Book()
         book.title = "Book ${i}"
         book.author = "Author ${i}"
         book.description = "The description for book ${i}"
         book.save()
         i = i - 1
	 }
     render "Created data!"
	}

In Rails to achieve the same thing I did this:

def data
   i = 1000
   while i > 0
       @book = Book.new
       @book.title = "Book #{i}"
       @book.author = "Author #{i}"
       @book.description = "The description for book #{i}"
       @book.save
       i = i - 1
   end
   render :text => "Created data!"
end

Sending a request to each action created the data. Job done. Now we have some test data and the respective list views of each app look like this:

!Picture 2.png|thumbnail! !Picture 1.png|thumbnail!

If you want to download the full test applications they can be downloaded from here:

• Rails Benchmark Application
• Grails Benchmark Application

For the Grails application you should run "grails upgrade" before executing it

Test Tools

To perform the tests I used ApacheBench which is available with the "ab" command on Mac OS X. Just for clarity here is the version info

This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

The results have then been graphed up using Apple Pages. The full PDF can be download here

Test A: Read Performance

So after creating the test data we now have 1000 records in each applications respective database. Just to be clear the code that renders the above view is the standard scaffolded list view for both apps with the server side code in Grails being:

def list = {
    if(!params.max)params.max = 10
    [ bookList: Book.list( params ) ]
}

And in Rails:

def list
  @book_pages, @books = paginate :books, :per_page => 10
end

To run the read performance test I executed the commands:

// for rails
ab -c 50 -n 1000 -e "rails_list_test.csv" http://localhost:3000/books/list
// for grails
ab -c 50 -n 1000 -e "grails_list_test.csv" http://localhost:8080/books/book/list

Obviously not at the same time! ;-) So what this does is send 1000 requests to the respective servers with 50 concurrency. I made sure all other applications were closed to make sure the figures were as accurate as possible. The results of the read test are below:

Grails is able to handle 40 requests per second to Rails 32. Significantly you can notice Grails has a much poorer worst case response time, but a superior mean response time (The mean time is the average time taken per request over all 1000 requests). This can be put down the Java threading model and the Thread pool in Tomcat at some point running out of available threads and blocking for a moment. It just reflects the differences in architecture more than anything else. If you ran the same test against Grails with a concurrency of 1 you would see linear performance as expected:

ab -c 1 -n 1000 -e "grails_list_test.csv" http://localhost:8080/books/book/list

Interestingly, Grails with only 10 threads is even more performant and is able to cope with 54 requests per second. We also see that the spikes in the max response time are gone with a configuration for less threads. If you think about it this makes sense given the hardware I'm testing on. I only have dual cores on this Macbook, so the more threads there are the more processes have to share the two cores and hence the overall performance degrades with more threads. If I was running a machine with 8 processors then having more threads would obviously be more beneficial.

Rails on the other hand with 10 mongrels and Pound degrades in performance. I guess this is down to more processes having to share the two cores, but if someone can explain if I've done anything wrong in the config please say so

Test B: Creating Records

The next test was to test creating new records. In this case I used a simple action for both. The Rails code is:

def createTest
   i = 5
   while i > 0
       @book = Book.new
       @book.title = "Book #{i}"
       @book.author = "Author #{i}"
       @book.description = "The description for book #{i}"
       @book.save
       i = i - 1
   end
   render :text => "Created 5 records data!"
end

As you can see it just creates 5 records in a loop and then renders some text to the response. No view involved. The Grails code for this is:

def createTest = {
 def i = 5
 while( i > 0 ) {
     def book = new Book()
     book.title = "Book ${i}"
     book.author = "Author ${i}"
     book.description = "The description for book ${i}"
     book.save()
     i = i - 1
 }
 render "Created 5 records data!"
}

To run the create performance test I executed the following ab commands framework:

// for rails
ab -c 50 -n 1000 -e "rails_create_test.csv" http://localhost:3000/books/createTest
// for grails
ab -c 50 -n 1000 -e "grails_create_test.csv" http://localhost:8080/books/book/createTest

Again with a 1000 requests, and 50 concurrency. The results are as follows:

The differences here are quite astounding, Grails is able to cope with 140 requests per second and has a mean average of 345ms per request compared to Rails' 35 (and a half) and 1.3 seconds per request. I believe, without having done any real profiling, the reasons for this can largely put down to Hibernate's session-based model. Each call to save() in ActiveRecord immediately persists the data whilst Hibernate batches updates up until the end.

Again Grails with only 10 threads is faster, whilst Rails with 10 mongrels degrades performance.

Test C: Querying

The next test tackled querying using a like query. The bottleneck here is likely to be MySQL's query performance. The code for this test for Rails is as follows:

def queryTest
    @books = Book.find(:all, :conditions => ["title like ?", "Book 6%"])
end

The view used is the same as the default list view just cloned and renamed to queryTest.rhtml. For Grails the same thing is done:

def queryTest = {
   [bookList:Book.findAllByTitleLike("Book 6%")]
}

There is no equivalent to Rails' :conditions element in Grails so we use a finder expression. Again to execute this test we use similar commands:

// for rails
ab -c 50 -n 1000 -e "rails_query_test.csv" http://localhost:3000/books/queryTest
// for grails
ab -c 50 -n 1000 -e "grails_query_test.csv" http://localhost:8080/books/book/queryTest

The results for this test can be seen below:

As you can see the results here are much more equal as it is really reliant on database performance. Notably we can see Grails' worst case is poor down to blocking in the thread pool, but overall the mean time of 5.387ms is superior to Rails' 6.386, which illustrates why Grails' is able to deal with 8.92 requests per second as oppose to Rails' 7.65.

Neither framework comes off particularly well here, but in reality if you had this kind of load for queries you would need consider using some sort of caching and/or indexing solution (lucene/ferret).

This is the one test though, where Rails with 10 mongrels gets close to matching the default Grails tomcat configuration and where Grails with 10 threads is slower than the default tomcat config.

Test D: Updating record and rendering a view

For this test we're trying out updating and then rendering a view. To do so we're going to retrieve some random records from the database and update each and then store the value. To achieve this in Rails we do:

def updateTest
   count = Book.count
   ids = [rand(count),rand(count),rand(count)]
   @books = []
   ids.each { |id|
     book = Book.find(id)
     if book
        book.author = "Fred Flintstone #{Time.new}"
        book.save
        @books << book
     end
   }
end

To explain this logic we basically create 3 random ids from the total, retrieve them from the database, update the author field and then save. And before you ask I definitely checked that the databases of both were dropped and re-created with fresh test data with the ids on the tables indexing from 1-1000. The view for this action is again copied from the list view and renamed to updateTest.rhtml.

For Grails to achieve the same thing we do:

static final rand = new Random()
def updateTest = {
	def count = Book.count()
	def ids = [rand.nextInt(count), rand.nextInt(count),rand.nextInt(count)]
	def result = []
	ids.each { id ->
		def book = Book.get(id.toLong())
		if(book) {
			book.author = "Fred Flintstone ${new Date()}"
			book.save()
			result << book
		}
	}
	[bookList:result]
}

Again the same logic, instead of the C rand(num) function in Ruby we use Java's java.util.Random class and calculate a bunch of ids, iterate over each, saving each one to formulate the results.

Again to execute this test we use similar commands:

// for rails
ab -c 50 -n 1000 -e "rails_update_test.csv" http://localhost:3000/books/updateTest
// for grails
ab -c 50 -n 1000 -e "grails_update_test.csv" http://localhost:8080/books/book/updateTest

The outcome of this test can be seen below:

Once again we see a similar graph, Grails having the higher max, but a superior mean average response time. Grails is able to deal with 44.95 requests per second to Rails' 30.88. This can be put down to a number of things, again I believe it is the difference between Hibernate's session based approach and ActiveRecord.

Another thing to note is during the Grails test 2 Hibernate StaleObjectExceptions were thrown because Grails uses Hibernate's optimistic locking feature, which ActiveRecord doesn't seem to have any concept of (correct me if I'm wrong). In a real circumstance when dealing with high concurrency you would of course catch this exception in Grails, which would actually be a Spring HibernateOptimisticLockingFailureException and deal with it appropriately by getting fresh data or returning an error (the default case).

Again similar to the first two tests, we see Grails with only 10 threads being even more performant and Rails with 10 mongrels degrading.

Test E: Updating 3 random records and return idiomatic XML

So after the above result I wanted to see what the performance would be like when not using a view (RHTML or GSP). So instead we return some XML. In rails the code for this is:

def updateTest2
   count = Book.count
   ids = [rand(count),rand(count),rand(count)]
   @books = []
   ids.each { |id|
     book = Book.find(id)
     if book
        book.author = "Fred Flintstone #{Time.new}"
        book.save
        @books << book
     end
   }
   render_text @books.to_xml
end

Note here we call to to_xml method to auto-convert to XML which is a nice feature. The resulting XML looks something like:

<?xml version="1.0" encoding="UTF-8"?>
<books>
  <book>
    <author>Fred Flintstone Thu Mar 22 15:35:54 +0000 2007</author>
    <date-created type="timestamp">Thu Mar 22 12:20:21 +0000 2007</date-created>
    <description>The description for book 3</description>
    <id type="integer">5908</id>
    <title>Book 3</title>
  </book>
  …
</books>

To do the same thing in Grails requires a little more code as Grails doesn't have a toXML() yet (actually it does, but it is currently in the REST plug-in, see http://grails.org/Plugins). So instead of toXML() we use Groovy mark-up building:

def updateTest2 = {
	def count = Book.count()
	def ids = [rand.nextInt(count), rand.nextInt(count),rand.nextInt(count)]
	def result = []
	render(contentType:"text/xml") {
		books {
			for( ident in ids ) {
				def b = Book.get(ident.toLong())
				if(b) {
					b.author = "Fred Flintstone ${new Date()}"
					b.save()
                    book {
                    	author(b.author)
               			'date-created'(b.dateCreated)
               			'description'(b.description)
						id(b.id)
						title(b.title)
					}
				}
			}
		}
	}
}

The result of the two methods is exactly the same XML (changing data of course). Now for running the tests, again the same commands:

// for rails
ab -c 50 -n 1000 -e "rails_update_test2.csv" http://localhost:3000/books/updateTest2
// for grails
ab -c 50 -n 1000 -e "grails_update_test2.csv" http://localhost:8080/books/book/updateTest2

And the result is:

So that is pretty similar to the first update test. Again Grails is around 40-50% faster. Now there is a lot of dynamic stuff going on when producing the XML from these two actions so I thought I would have a go at building up a String and writing it to the response.

Again similar to the first two tests, we see Grails with only 10 threads being even more performant and Rails with 10 mongrels degrading.

Test F: Updating 3 random records and returning a String generated response

This test actually is more a like for like as we don't have to deal with the differences between to_xml in Rails and mark-up building in Grails. The Rails code for this is:

def updateTest3
  count = Book.count
   ids = [rand(count),rand(count),rand(count)]
   xml = "<books>"
   ids.each { |id|
     book = Book.find(id)
     if book
       xml << "<book>"
        book.author = "Fred Flintstone #{Time.new}"
        book.save
          xml << "<author>#{book.author}</author>"
          xml << "<date-created>#{book.date_created}</date-created>"
          xml << "<description>#{book.description}</description>"
          xml << "<id>#{book.id}</id>"
          xml << "<title>#{book.title}</title>"
       xml << "</book>"
     end
   }
   xml << "</books>"
   render_text xml
end

The XML that is produced is exactly the same as in the previous test, except constructed manually from a String. The equivalent Grails code looks like:

def updateTest3 = {
	def count = Book.count()
	def ids = [rand.nextInt(count), rand.nextInt(count),rand.nextInt(count)]
	def xml = new StringBuffer("<books>")
	ids.each { id ->
		def b = Book.get(id.toLong())
		if(b) {
			b.author = "Fred Flintstone ${new Date()}"
			b.save()
               xml << "<book>"
               	 xml << "<author>${b.author}</author>"
				 xml << "<date-created>${b.dateCreated}</date-created>"
             	 xml << "<description>${b.description}</description>"
             	 xml << "<id>${b.id}</id>"
             	 xml << "<title>${b.title}</title>"
			   xml << "</book>"
		}
	}
	xml << "</books>"
	render xml.toString()
}

Again it uses the static java.util.Random instance and constructs the String from a StringBuffer. Now we execute the commands:

// for rails
ab -c 50 -n 1000 -e "rails_update_test3.csv" http://localhost:3000/books/updateTest3
// for grails
ab -c 50 -n 1000 -e "grails_update_test3.csv" http://localhost:8080/books/book/updateTest3

Again with a 1000 requests, and 50 concurrency. The results are as follows:

Grails in this case is able to process nearly 3 times as many requests as Rails per second. So why are the first 2 update tests so much slower for Grails (although still faster than Rails?). Well what these tests have helped us identify is that with a large amount of load Groovy Server Pages, Grails' view technology, certainly needs some improvement.

And again similar to the first two tests, we see Grails with only 10 threads being even more performant and Rails with 10 mongrels degrading.

The way GSP works internally with tag libraries is that it uses try/catch blocks to intercept exceptions thrown by Groovy when it doesn't find a method. These are of course expensive as the JVM has to take a snapshot of the exception when it is thrown which causes all threads to freeze momentarily. So one area where we can improve Grails' performance is to look at tweaking GSP to not do this (there are other techniques for doing this).

So why does this effect the second example? Groovy's streaming markup builder is using the same technique!

Summary

Overall, as expected, given its base on solid Java technologies, Grails out performs Rails (100% Ruby) in pretty much every test. As I said at the start I'm no Rails performance tuning wizard, so if someone can suggest ways to get Rails to perform better please say so. We are conducting these tests purely to inform concerned Grails users who are asking for figures, and hope to review them as Grails progresses.

Interestingly, Grails, at least on this machine, sees performance increases when lowering the thread pool to 10, whilst Rails suffers when 10 mongrels are configured with Pound.

Grails hasn't undergone much optimisation at all yet and there are certainly areas we can improve on, GSP rendering being one. GSP is of course a much more complex view rendering technology in comparison to RHTML due to its support for dynamic tag libraries, plus Sitemesh decoration. However, there are definitely areas in GSP, such as the way it uses exceptions, that can be improved upon.

Another point to remember is performance != scalability. The way you would scale an application to deal with the kind of load we have here differs greatly between Grails & Rails I imagine. With Grails you could stick a Terracotta or Coherence distributed cache on the back of it to implement SSI (singe system image) or use HTTP session replication across a cluster whilst with Rails you could ??? (a Rails user please feel free to provide an answer to this question in the comments, I'm not qualified to answer).

Nevertheless, I am quite satisfied that Grails performs sufficiently well to be competitive and there is little point in optimising prematurely. Our main goals is to focus on getting 1.0 out towards the middle of this year.

Resources

• The Rails test app
• The Grails test app
• The test results in PDF