CF8 vs Railo: Speed Comparison
On CF-Talk, there's been an ongoing thread about Railo's blazing performance. Railo has always been known for its blazing speed and I'm glad people are finally taking notice. David Low posted an interesting speed comparison between CF8 & Railo. David's tests show Railo clearly outperforming CF8 and by a healthy margin.
This must've really set some things in motion because Adobe decided to run their own performance comparison. Adobe's Damon Cooper posted the following performance stats on CF-Talk today and I have to say I was impressed & surprised by CF8's results:
"OVERVIEW: In all load tests here, we see ColdFusion 8 outperform Railo everytime.
Testing Scenarios: All tests were run with 100 Virtual users on Windows 2003 Sp2 Server with dual 3.60GHz CPU's and 2GB RAM.
Because we do not support TomCat running on Beta virtualization software ( Apple Bootcamp ) the following tests were run on a fully supported server platform.
The following tests mirror some of the tests that were posted on your blog.
Test 1. Simple CFM page outputting a string.
- CF 8: Requests per Second: 800.94
- Railo: Requests per Second: 721.22
Test 2: Simple MySQL query returning 241 records.
- CF 8: Requests per Second: 279.15
- Railo: Requests per Second: 187.22
Test 3: Simple page with 5 cfincludes:
- CF 8: Requests per Second: 633.33
- Railo: Requests per Second: 631.43
I'm not sure why in your testing you are seeing CF 8 take one second longer for queries to MySQL. Railo ships with an older MySQL driver ( MySQL Connector/J 3.0.1 ). CF 8 ships with MySQL Connector 5.0.5. In our testing, CF 8 and the newer driver were substantially faster when running queries."
These results show that ColdFusion 8 outperforms Railo! That was a real stunner especially in light of David's results.
This looks like its going to turn into an interesting debate.
Speed testing is right up there with religion... Especially when talking about a Java servlet. I am surely not a Java expert, but the choice of servlet engine and configuration play a very significant role in the performance of the app.
I'm real curious about what engine Adobe used for their Railo testing. Was it just the Railix download with the included Jetty? They specified that Tomcat was not used, so what was it? I've seen some pretty qualified test showing that Jetty cannot handle near the load that Tomcat or Resin can, so the config of Railo would play a huge part in a load test comparison.They also don't specify what web server was used. There are so many variations on installing both products, that it's really hard to do a apples-to-apples comparison.
I'll keep watching the debate though! I always like the underdog, especially when they have a neat product. I'm not on the CF8-is-great-bandwagon, so it's nice to see a small company like Railo come along and stir things up a little.
T
just found this entry that's why I am just commenting it right now.
We have a saying in Switzerland that you should never trust a statistic you haven't faked yourself.
In my personal tests even under heavy load, with Resin or Tomcat, Railo outperforms CF8 by a high percentage. Not in every test, but in the majority of tests.
In my opinion a user has to experience these things for himself.
I would have been satisfied if Railo would have been as fast as CF8. And I was very impressed to find out that it outperforms CF8.
And there are reasons to think that even Adobe's result is a good one. Why?
For instance Railo RC3 uses the Java JRE 1.5 and MX8 uses JRE 1.6.
And MX8 uses the server JRE whereas Railo ships with the client JRE. So still things to improve.
There are in addition many leverages you can turn to improve the performance of Railo. I don't know if the folks of Adobe have turned all of them.
Anyhow. Performance comparison is allways a tricky business. Every engine claims to be the fastest one. Even Smith.
Greetings from Switzerland
Gert Franz
Customer Care
Railo Technologies GmbH
gert.franz@railo.ch
www.railo.ch
Join our Mailing List
english: http://groups.yahoo.com/group/railo_talk/