Zend-expressive slow in ab tests


#1

Just moving the discussion from here: https://github.com/zendframework/zend-expressive/issues/500

After some tests today, a xdebug profiler shown that it took 33ms to process que code in zend-expressive, what is pretty good vs 261ms to process the Agavi code.

Even xdebug showing that zend-expressive was 8x faster, it was ~2x slower than agavi in an ab test (3020ms vs 6150ms). For the zend-expressive part, 6.1s was spent in “processing”, not “waiting” or “connecting”. The same difference was present in a request without database access.

Any idea what could be influencing the difference?


#2

All right, here it is some extra information:

Using the -k option in ab shows better results and it seems more consistent with the xdebug profile times.

Still, I don’t know why zend-expressive is twice slower without the keep-alive feature enabled in the ab tests.

But since all browsers use it, I guess the “right” way to test it is always use the -k option.


#3

First, xdebug should not be enabled when you are running benchmarks; it has known, severe impact on performance.

Second, we really need to know what you are benchmarking: is it a default skeleton? Are templates involved or not? which container was used? are you hitting the home page, a 404, an error page, or…? These sorts of details allow us to attempt to recreate the benchmark. Ideally, we should have a comparable setup in the other framework against which you are benchmarking, so we can iron out any other factors (e.g., is there any cache priming happening).


#4

Thanks. xdebug was disabled for the tests.

I was curious why it was slower when xdebug was showing it was actually faster.

The -k option solved the problems for me. I hope it helps others seeming the same results as I was.

I’m using the zend-expressive-skeleton.


#5

If you can, try adding a bash script that reproduces the benchmarks on both.

Agavi didn’t have a lot of the abstractions we have nowadays, which obviously means less static overhead, but also issues when it comes to lazy-loading large dependencies (older frameworks simply don’t have that). If we can reproduce the setup and test it ourselves, we can compare where the performance overhead is, and optimise the components :slight_smile:


#6

Since he is talking about the -k option I’m guessing that apache-bench is used and he encountered the same issue as reported here: https://github.com/zendframework/zend-expressive/issues/502

To reproduce it you need to setup a basic zend-expressive app with the installer, fire up apache and then run ab -c 1 -n 1 http://localhost:8080/. Without the -k option you will get a really slow request time. When using the -k option you will get the same fast results as with nginx or the php built in server.


#7

Yeap, I’m using the -k option and it solved the issue for me.


#8

Let me add one more thing. Using the fix described in the github thread I’m getting really good results (compared to the results of using the -k option): 54.11 vs 16.88 r/s. So, even if the -k option makes things more reliable, the fix is much better.

Since I started benchmarking I went from <1 r/s to >54 r/s. I’m really comfortable with the results now :slight_smile:

EDIT: the fix was generating an error for the requested URL that I benchmarked. Now I got the same results with -k and the referenced fix.