Alrond | The performance test of 6 leading frameworks

Thu 25 Jan 2007

The performance test of 6 leading frameworks

Posted by Alrond | Tags: Catalyst : CodeIgniter : Django : framework : Performance test : php : Ruby on Rails : Symfony : TurboGears :

As I already wrote in the previous posts, several months ago I chose framework for make a site. Most important for me were simple using of him, stability and performance.
For absence of the concrete developed information about speed comparison of frameworks in the Internet, (I have found only one testing, but it turned out naked enough. It was limited to three frameworks RubyOnRails, Django, Symfony and didn’t contain absolutely any detail), I conducted the detail analysis for decision of the 6 best frameworks in all these parameters.

Update 04.Feb.2007: Pylons, RoR 1.2.1, CakePHP + eAccelerator, Zend, Turbogears with Cheetah, Jinja, Genshi

So, the purpose of testing was the decision of work speed of frameworks and their comparisons with each other on speed of pages generating and on maximum quantity of inquiries by the given configuration.
I have chosen the test model, at which the controller rendered the appointed template and generated the answer. As a result turned out the HTML-file "Hello World!" (below there are codes for everyone framework). The Database in the test is not used, because it itself limits the speed important.

I did not plan to compare functionality of frameworks, community and work with database. It was important to me to determine work speed of MVC-framework, by as small as possible influence of external factors.
Therefore I used sheaf Nginx + FCGI through IP:Port

The following frameworks have been tested:

CodeIgniter on PHP
Catalyst on Perl
Django on Python
Django on Python with Psyco
RubyOnRails on Ruby
Symfony on PHP
TurboGears on Python

Conducted the test on following hardware and software:

CPU: AMD OpteronT Processor 146 (2 GHz)
Memory: 2 GB
OS: Debian 3.1 (Linux 2.6.14)
Web-Server: nginx/0.5.5

Versions of frameworks and programming languages:

CodeIgniter 1.5.1
Catalyst 5.7006 Rev.5996
Django Rev.4254 (28Dec.2006)
RoR 1.1.6
RoR 1.2.1
Symfony 1.0beta2 SVN-Rev 3122
Turbogears 1.0b3 Rev.2323
Python 2.4.4
Python Psyco 1.5.2
Flup Rev.2303
Ruby 1.8.5-p12
mongrel 0.3.13.4
PHP 5.2.0 (cgi-fcgi)
perl, v5.8.4
CPAN ver 1.8802

Programs for test:

Siege 2.65 www.joedog.org/JoeDog/Siege
Http_load 12.03.2006 www.acme.com/software/http_load/
ab 2.0.41-dev Rev: 1.141

Technique of testing:

Measurement of memory (ps aux: VSZ "virtual set size" and RSS "resident set size").
Testing Apache Benchmark (2 times in succession)

ab -c 5 -n 1000 project.com/

Measurement of memory and of used processor-time (ps aux: VSZ, RSS, %CPU, %MEM).
Restart of Framework.
Testing Apache Benchmark (2 times in succession)

ab -c 100 -n 10000 project.com/

Measurement of memory and of used processor-time.
Restart of Framework.
Measurement with http_load

http_load -rate 10 -seconds 5 project.com

Measurement of memory and of used processor-time.
Restart of Framework.
Test Siege with 50 concurrent users during one minute.

siege -d1 -t1M -c50 project.com

Test Siege with 200 concurrent users during one minute.

siege -d1 -t1M -c200 project.com

Test Siege with 300 concurrent users during one minute.

siege -d1 -t1M -c300 project.com

Measurement of memory and of used processor-time.

Comments to a technique:

For the beginning I have used only Nginx with the elementary config, at which on any inquiry the dot transparent gif-file in the 43b size comes, generated by a server.
This test determined an approximate maximal capacity of the server. In front I shall tell that a stock of its capacity is 10 times more then production capacity of the fastest framework.
All frameworks were started as FastCGI 127.0.0.1:PORT. If framework specified development- and production- modes, I worked with production.
In nginx the identical config is used for all frameworks.
New start of framework I conducted to clear statistics among tests different programs.

My subjective additions on every framework by installation and start:

CodeIgniter
It is easy and quick to organize it. I haven’t any problems with it. For start I used spawn-php.sh with five processes.
There is pretty bright framework for PHP.
Catalyst
For the beginning it is necessary to install CPAN and a couple of modules for FCGI. Installation looked a little involved.
Start isn’t simple too. For the beginning it is necessary to start Framework
```
./CatInABox/start.sh
```
Then the project itself
```
./script/world_fastcgi.pl -l 127.0.0.1:9500 -n 5
```
Django
It is easy to install from repository. Projects are created easy too. I started with two methods prefork and threaded. But the python doesn’t work so well with treads.
```
python manage.py runfcgi method=threaded host=127.0.0.1 port=8801
python manage.py runfcgi method=prefork host=127.0.0.1 port=8801
```
I also tested the framework with the acceleration module of psyco. In manage.py put:
```
from django.core.management import execute_manager
import psyco
psyco.full()
```
RubyOnRails
Following the instruction from a site it is easy to install, but it turned out difficult to start it. Though on the Internet I saw different configurations for sheaf Nginx + FastCGI, I failed to start it together with nginx.
However with lighttpd it all started normally. Therefore I have taken advantage of the "official" recommendation to start it through a server mongrel.
Owing to Alexey Kovyrin it is possible to estimate a multiplication factor.
Nginx+FastCGI is faster approximately in 1.29 times than Nginx + Mongrel.
I started 5 servers mongrel:
```
mongrel_rails start -d -e production --port 8501 --pid /tmp/rb1.pid
mongrel_rails start -d -e production --port 8502 --pid /tmp/rb2.pid
mongrel_rails start -d -e production --port 8503 --pid /tmp/rb3.pid
mongrel_rails start -d -e production --port 8504 --pid /tmp/rb4.pid
mongrel_rails start -d -e production --port 8505 --pid /tmp/rb5.pid
```
In a few days after test realization a new version RoR 1.2.1 came out.
Of course I has also tested it with all parameters, has only renewed framework, checked all specifications. Results are shocking a little: factor productivity halved in comparison with previous version in all tests!
Symfony
It wasn’t complicated to install it, but then was worn out with the project itself. There is difficult enough. For start I also used spawn-php.sh with five processes.

TurboGears
By installation were appearing some problems, which were solved with some efforts.
For start in threaded mode I used a script, I only changed

from fcgi import WSGIServer
	to
from flup.server.fcgi  import WSGIServer

WSGIServer(application=wsgiApp).run()                                                                                                                                                               
	to
WSGIServer(application=wsgiApp, bindAddress=

	('127.0.0.1', 8900)).run()

Configuration files of web-server:

„pure“ nginx
nginx + fcgi
nginx + mongrel

Project codes for every framework:

CodeIgniter
Catalyst
Django
RubyOnRails
Symfony
TurboGears

Start script of PHP-FCGI:

spawn-php.sh

Web-server results for comparison:

Nginx

Results for frameworks each taken separately:

CodeIgniter
Catalyst
Django prefork
Django threaded
Django prefork with Psyco
Django threaded with Psyco
RubyOnRails 1.1.6
RubyOnRails 1.2.1
Symfony
TurboGears
TurboGears with Psyco (error is taken out)

Results:

Test Apache Benchmark
- Table of inquiry working speed
  
  * 1) in second time framework is down
  * 2) a multiplication factor according to Kovyrin = 1.29. For conversion of performance from mongrel into fastcgi.
- Diagram of results. Test „ab c 5 n 1000“.
- Diagram of results. Test „ab c 100 n 10000“.
- Memory expense before and after tests
  * 3) The python is made so, that it restarts the processes at strong load in prefork-mode. Therefore it is impossible to determine real load of processor and maximal use of memory.
- VSZ "virtual set size"
- RSS "resident set size"
- CPU loads
Test http_load
- Table with results
- Diagram of time for connect
- Diagram with time of first-response
Siege test
- General table with results
- Transaction diagram
- Response time
- Longest transaction
- Memory expense after tests
- Processor load

Conclusions:

Results are visible according to tables and diagrams.
Some short additions:
Django has the least use of the processor.
Catalyst has surprised. By excessive test it overloads the processor. But losses of inquiries were not observed.
RoR 1.2.1 also loads the processor under big load.
It was interesting by TurboGears, which has shown low time use CPU in test "ab", but in the siege-test had the worst result.
Catalyst has appeared the greatest memory use. By RoR this more likely happens because of start through mongrel.
PHP-frameworks borrow many resources CPU even in a quiet condition.

Average connect time is approximate equal for all frameworks. The first response time differs strongly from each other. Django is exception, having the least “average time”, and the highest “maximal time”. Python-frameworks have proved good oneself, and RoR has disappointed.

During the siege-test by CodeIgniter, Symfony, RoR 1.2.1 and 1.1.6 losses were observed at a large number of "concurrent users".
RoR is fast by the small load, but it sharply loses productivity at a plenty of users.

Psyco module accelerates Django on 15-30 %, but it will be worth the grown memory use VSZ on 80 % in a prefork-mode and on 400 % (!!!) in threaded-mode. Use of RSS memory increases in 2-2.5 times.
The Prefork-mode takes away more memory, but for it the system comes out, which stably works on extreme loads, and the smaller use of the processor resources.
In a threaded-mode Django hanged under the big load and did not answer on inquiries.
The same happened with TurboGears too. This is due to the fact that python works badly in a treaded-mode.

Comments from Ivan Sagalaev:

"- FastCGI - server is started in a threaded-mode. It is known, that the Python works in treads essentially slow through GIL (Global Interpreter Lock), because of which all treads wait each other: the interpreter can be used of only one of these. Therefore in Unix at the opportunity servers with python-code should be started in a prefork-mode – it will be faster. By the way, in Windows this situation is reverse, because there creation of process is essentially slower than in Unix. And as far as I know, Python however hangs there less.
- About Psyco. Django-developers have tested it, and have found something interesting. On 32-bit systems it really gives an increase in productivity. And on 64 bit - on the contrary productivity sinks. It happens, because the processor switches under Psyco to a 32-bit mode and does not use all advantages of one’s own architecture. If it is so, than this fact, that Django worked more quickly under Psyco, very likely means, that Linux and Python don’t use 64 bits system (in spite of work on Opteron). That’s right?”

Yes, it is 32 bit system.

Distribution of places according to this test:

Django has won over the nearest competitors with the approximate triple superiority.
Second and third places have divided TurboGears and RoR 1.1.6, because they are equally fast, but behave differently at different loads, overtaking each other.
.
Catalyst. To be honest, I expected a good deal more from Perl-framework.
CodeIgniter. Just as expected PHP-frameworks have appeared the slowest. But CodeIgniter is to recommend those who wish to program only on PHP, and also to have handy system.
Results of RoR 1.2.1 shocked “a little”: falling of productivity in 2-4 times in comparison with 1.1.6 version. The response-time in http_load is also more in 2 times and excessively high CPU-load under high stream - all this attests to possible error in the new version.
Symfony has got last place. It is very complicated and slow framework. A difference with Django is up to 35 (!!!) times.

P.S. Because of the addition to Django of the module Psyco in test I have conducted at many requests a short test for PHP with eAccelerator-0.9.5.
I have used 2 times in succession the following commands:

ab -c 5 -n 1000 ci.test.com/
ab -c 100 -n 10000 ci.test.com/

As a result CodeIgniter works 6,5 times faster, gives out to 600 resquests / sek.
The difference with Django became to 2 times.
Here are the results

(140 votes, average: 4.5 out of 5)

Loading ...

76 Responses to "The performance test of 6 leading frameworks"

Pento Says:
January 29th, 2007 at 11:27 a.m.

1. А почему не учли тот факт, что php по большей части ставится и используется как модуль апача и в таком случае он работает быстрее?
2. Жаль, что в тестировании не участвовал CakePHP (cakephp.org).
BOLK Says:
January 29th, 2007 at 11:49 a.m.

Для Pento:

PHP в FastCGI-режиме + nginx работает производительнее, чем Apache + mod_php.

Вопрос к автору: PHP был с ускорителем/кешем или без?
zhulanov Says:
January 29th, 2007 at 11:57 a.m.

Отличный тест, спасибо за проделанную работу.

Хотел бы заметить, что в больших боевых проектах php редко используется без всяческих memcached'ов, zend optimizer'ов и прочих. Этот фактор сильно влияет на производительность. Странно, что про это в статье нет ни слова, а вот про питоновский psyco не забыли, ага.
thenexus6 Says:
January 29th, 2007 at 1:19 p.m.

У Вас в perl, простите, хрень какая-то в отчёте.
Запрашиваете один документ (/), а получаете - совсем, совсем другой (/hello/). Значит, для одного фреймворка тестировали ещё и редиректы в нагрузку.

Разброс min и max time для запросов косвенно указывает на то, что какие-то модули могли докомпиляться после рестарта фреймворка. Почему не делали "пустой" прогон теста, чтобы всё что нужно было уже откомпилированным в памяти перед основным запуском?.

Что тестировали, собственно, тоже не понятно: делаются ли проверки ACL при доступе к ресурсам? Как ведут себя встроенные template движки (вы же их используете) на больших страницах с малым количеством динамики, на больших страницах с больших количеством динамики?
Magic Egg Says:
January 29th, 2007 at 1:47 p.m.

Жаль, что Zope никак не учавстовал.
Alrond Says:
January 29th, 2007 at 2:02 p.m.

2 Pento: Про апач BOLK уже ответил, а на CakePHP можно взглянуть...
2 BOLK, zhulanov: PHP был без ускорителя и без кеша. я старался снизить посторонние влияния, то что psyco протестировал, так это для сравнения с самом django. Конечно и зенд и кеш сильно ускорят работу. Но это уже связки.

2 thenexus6: Спасибо, исправил. реально тестировал с прямым линком ct.test.com/hello/
Про подгрузку подулей в перле не подумал. Мой косяк. В других тестах прогонял по 2-3 раза, а http_load сделал один раз.
>>Как ведут себя встроенные template движки (вы же их
>>используете) на больших страницах с малым количеством
>>динамики, на больших страницах с больших количеством
>>динамики?
цель тестирования была одна маленькая страничка. и маленькая динамика.
Конечно можно тестировать все фреймворки со всеми ускорителями/без них, с проверкой ACL и без нее, с кучей динамики и не очень. с кучей темплейтов и с одним. еще кучу режимов для всех тестовых программ использовать.

Но этот тест дает только общее представление о производительности фреймворков. А вот конкретный проект каждый должен сам тестировать, и находить подходящий для него вариант.
Sokolov Yura Says:
January 29th, 2007 at 2:18 p.m.

Когда я у себя тестировал Ruby on Rails, то он действительно просаживался, будучи запущенным через mongrel.
Однако запущенный через lighttpd + fastcgi (модуль ruby-fcgi native, не pure ruby режиме) (команда запуска: script/server lighttpd) он не только показывал большую производительность (35 страниц/сек вместо максимального 29 через mongrel), но и не проседал - т.е. даже под 200 пользователями показывал те же 35 страниц в секунду.
Alexandr Kot Says:
January 29th, 2007 at 2:23 p.m.

Кретив - газетный, автор - метролог...

Писькомерка с погрешностями. Ведущие - это по чьему мнению? По мнению автора?
А где Zope питоновский?
Где CakePHP? Почему выбрали среди PHP фреймворков одну из самых медленных?

И если уже ROR юзали на Mongrel в родных условия, могли бы PHP в mod_php проверить. :(
Sokolov Yura Says:
January 29th, 2007 at 2:24 p.m.

И php обязательно нужно с ускорителем (тот же eAccelerator).
Ибо:
1. в python, ruby, perl компиляция кода фреймворка происходит лишь один раз, а в php без акселератора - каждый.
2. ни один нагруженный сайт на php не идет без акселератора
3. (как следствие) ни один хостер не запускает php без акселератора.

Т.е. сравнивать с php без акселератора не честно
savagex Says:
January 29th, 2007 at 6:06 p.m.

Бред. Еще один синтетический тест не имеющий отношения к реальности.
mordaha Says:
January 29th, 2007 at 6:15 p.m.

Еще бы с БД тесты увязаны были
Например отобразить 50 строк из одной и той же таблицы
А то есть подозрение что работа с БД в джанге любимой тормозид )
Mark Says:
January 29th, 2007 at 6:42 p.m.

Sorry, was ist mit Cake? Soviel ich weiß, codeigniter ist kein bestes unter PHP frameworks...oder?
Sam Says:
January 29th, 2007 at 6:53 p.m.

To thenexus6, Alexandr Kot, savagex Ну вы блин даете, нытики несчастные! Чел потратил время – работа не малая, тест познавательный. Обнародовал, так сказать для нужд всеобщих, умные возьмут что-то для себя.
Аффтар респект, не слушай никого, всегда найдутся, которые обсирают всех и все без разбора, и без реального понятия и объяснить то не могут: «Ведущие - это по чьему мнению?» ну напиши свое мнение. А его нет, они просто ходят по блогам и в коменты срут, где придется.
«Бред. Еще один синтетический тест не имеющий отношения к реальности.»
ты сам бред несешь, он о кирпиче, а ты о яйцах, ясно же написано – это только ориентиры, направление,а для каждого проекта все очень индивидуально.
Найми пару работничков, поработайте пару месяцев бесплатно и сделай себе «несинтетический тест, имеющий отношение к реальности»
Boo Says:
January 29th, 2007 at 7:27 p.m.

Почему не Python-2.5? И где CakePHP?
Муркт Says:
January 29th, 2007 at 9:45 p.m.

> Psyco модуль ускоряет django на 15-30%, но за это приходится расплачиваться возросшим потреблением памяти VSZ на 80% в prefork-режиме и на 400% (!!!) в threaded-режиме.

Дело всё в том, что использовался psyco.full(), а если делать хорошо, то по идее надо профайлером посмотреть, где именно тормозит, и отдавать на ускорение только части. Ну да это очень долго, и заниматься таким стоит только от безысходности :)
Jules Says:
January 29th, 2007 at 10:01 p.m.

Rails' development mode = slow. Doh. It reloads the entire application on every request.

Try again: `mongrel_rails start -e production`.
Masaaki Says:
January 29th, 2007 at 10:18 p.m.

Great work! Thanks for your efforts.
And the django-searchengine was very opportunely ;)
bUg Says:
January 29th, 2007 at 11:03 p.m.

Работаю с RoR давно, и связка mongrel + Apache mod_proxy_ballancer работает очень шустро.
Банзай Says:
January 29th, 2007 at 11:34 p.m.

Ответ от читателя:
PHP был с ускорителем/кешем или без? - это до лампочки в контексте разов.
framexpert Says:
January 29th, 2007 at 11:37 p.m.

Жаль, что не протестировали какую-нибудь отечественную разработку, например, Cairo (www.cairo.com.ua)
Alrond Says:
January 30th, 2007 at 12:31 a.m.

2 Mark: Du bist nicht der einzige, der fragt :-) Ich mach’s im nächst. Wollte heute schon, es hat aber nicht geklappt mit Cake schnell zur Recht zu kommen
Alrond Says:
January 30th, 2007 at 12:33 a.m.

2 Boo
Попробую этот фреймворк в ближайшее свободное время. Хотел сегодня еще сделать, установил, запустил, но не получилось быстро разобраться с проектом.
Alrond Says:
January 30th, 2007 at 12:35 a.m.

2 Jules:
Thanks, it’s a misprint, I have really used production, already corrected :-)
Alrond Says:
January 30th, 2007 at 1:55 a.m.

2 framexpert:
Каиро - это CMS, а я тестировал только CMF, причем только современные MVC
Mike Shock Says:
January 30th, 2007 at 9:26 a.m.

Работа большая и полезная. Хотелось бы, чтобы автор повторил тесты (через некоторое время) с учётом высказанных замечаний и предложений.
piro Says:
January 30th, 2007 at 2:41 p.m.

Hello,

this comparison is really a great idea and well done, thank you.

Just to increase the comparison completeness, above all if a Python developer wants to choose between one of the two choices on his side: could tou please add the Turbogears performance serving a page generated with CheetahTemplate instead of TAL?

Many would prefer Cheetah over TAL simply because they don't want to write XML template; an hint about the implication of this choice on the performance would be great.

Thank you again ;)
Mike Nolan Says:
January 30th, 2007 at 4:58 p.m.

This is completely unscientific! Running a PHP bytecode cache will improve the performance of PHP frameworks 10 fold! It's not surprising that they're slow with zero optimisation.
dogada Says:
January 30th, 2007 at 7:41 p.m.

It will be interested to see test of an java MVC framework also. For example struts.apache.org/ on tomcat.apache.org/

Thanks.
mike Says:
January 31st, 2007 at 12:59 p.m.

thanks
ihanick Says:
January 31st, 2007 at 6:32 p.m.

PHP specific: using byte code cache.

Server Software: Apache
Server Hostname: ******
Server Port: 80

Document Path: /myproject/web/mymodule/myAction
Document Length: 986 bytes

Concurrency Level: 100
Time taken for tests: 11.709703 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 1361360 bytes
HTML transferred: 986986 bytes
Requests per second: 85.40 [#/sec] (mean)
Time per request:

Alrond's technoblog

The performance test of 6 leading frameworks

Top Posts:

76 Responses to "The performance test of 6 leading frameworks"