Skip to content

Poor performance with modsecurity enabled #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mimugmail opened this issue Feb 26, 2017 · 22 comments
Closed

Poor performance with modsecurity enabled #35

mimugmail opened this issue Feb 26, 2017 · 22 comments
Assignees
@zimmerle

Comments

@mimugmail
Copy link

mimugmail commented Feb 26, 2017 •
edited by defanator
Loading

As suggested within issues owasp-modsecurity/ModSecurity#1318 I'll open a new one here. With MS enabled with the current codebase the performance test with "ab" from apache2-utils is not going over 300req/s.

With a virtual machine on Virtualbox with 4 cores and 4gb ram:

root@nginx:~# ab -n 10000 -c 20 -k http://127.0.0.1:80/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.11.9
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /
Document Length:        612 bytes

Concurrency Level:      20
Time taken for tests:   73.444 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    9909
Total transferred:      8499545 bytes
HTML transferred:       6120000 bytes
Requests per second:    136.16 [#/sec] (mean)
Time per request:       146.889 [ms] (mean)
Time per request:       7.344 [ms] (mean, across all concurrent requests)
Transfer rate:          113.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    14  147  31.7    141    1308
Waiting:       14  147  31.7    141    1308
Total:         14  147  31.7    141    1308

Percentage of the requests served within a certain time (ms)
  50%    141
  66%    147
  75%    151
  80%    154
  90%    166
  95%    188
  98%    207
  99%    232
 100%   1308 (longest request)

And without:

root@nginx:~# /opt/nginx/sbin/nginx
root@nginx:~# ab -n 10000 -c 20 -k http://127.0.0.1:80/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.11.9
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /
Document Length:        612 bytes

Concurrency Level:      20
Time taken for tests:   0.197 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    9909
Total transferred:      8499545 bytes
HTML transferred:       6120000 bytes
Requests per second:    50755.75 [#/sec] (mean)
Time per request:       0.394 [ms] (mean)
Time per request:       0.020 [ms] (mean, across all concurrent requests)
Transfer rate:          42128.99 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     0    0   0.3      0       5
Waiting:        0    0   0.3      0       5
Total:          0    0   0.3      0       5

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      1
  98%      1
  99%      2
 100%      5 (longest request)

I tested this also with Nginx+ but then the tool stucks a 90%, no mather how many requests (20 cores and 64GB ram):

This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 999 requests completed

But with just 100 it's ok:

Server Software:        nginx
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /
Document Length:        162 bytes

Concurrency Level:      20
Time taken for tests:   0.243 seconds
Complete requests:      100
Failed requests:        0
Non-2xx responses:      100
Keep-Alive requests:    100
Total transferred:      31000 bytes
HTML transferred:       16200 bytes
Requests per second:    410.71 [#/sec] (mean)
Time per request:       48.696 [ms] (mean)
Time per request:       2.435 [ms] (mean, across all concurrent requests)
Transfer rate:          124.34 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.7      0       3
Processing:     3   42  34.6     58      99
Waiting:        3   42  34.6     58      99
Total:          3   42  34.6     60      99

Percentage of the requests served within a certain time (ms)
  50%     60
  66%     66
  75%     70
  80%     74
  90%     86
  95%     92
  98%     97
  99%     99
 100%     99 (longest request)

Perhaps others can post their results too.

Ah, I also tried to disable logging but it doens't affect the performance at all.
@mimugmail
Copy link
Author

mimugmail commented Feb 27, 2017 •
edited by defanator
Loading

Ok, I disabled modsec_audit.log and now it seems better the more requests I send :) (only tested with Nginx+)

root@waf-1-a-02:~# ab -n 100 -c 20 -k  http://127.0.0.1:80/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done


Server Software:        nginx
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /
Document Length:        162 bytes

Concurrency Level:      20
Time taken for tests:   0.175 seconds
Complete requests:      100
Failed requests:        0
Non-2xx responses:      100
Keep-Alive requests:    100
Total transferred:      31000 bytes
HTML transferred:       16200 bytes
Requests per second:    570.71 [#/sec] (mean)
Time per request:       35.044 [ms] (mean)
Time per request:       1.752 [ms] (mean, across all concurrent requests)
Transfer rate:          172.77 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.7      0       3
Processing:     4   29  20.5     25      76
Waiting:        3   29  20.5     25      76
Total:          4   30  20.6     26      77

Percentage of the requests served within a certain time (ms)
  50%     26
  66%     44
  75%     48
  80%     51
  90%     59
  95%     65
  98%     72
  99%     77
 100%     77 (longest request)
root@waf-1-a-02:~# ab -n 1000 -c 20 -k  http://127.0.0.1:80/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /
Document Length:        162 bytes

Concurrency Level:      20
Time taken for tests:   1.458 seconds
Complete requests:      1000
Failed requests:        0
Non-2xx responses:      1000
Keep-Alive requests:    996
Total transferred:      309980 bytes
HTML transferred:       162000 bytes
Requests per second:    685.68 [#/sec] (mean)
Time per request:       29.168 [ms] (mean)
Time per request:       1.458 [ms] (mean, across all concurrent requests)
Transfer rate:          207.57 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     2   29  18.6     31      84
Waiting:        2   29  18.6     31      84
Total:          2   29  18.6     33      85

Percentage of the requests served within a certain time (ms)
  50%     33
  66%     39
  75%     41
  80%     51
  90%     54
  95%     54
  98%     62
  99%     72
 100%     85 (longest request)
root@waf-1-a-02:~# ab -n 2000 -c 20 -k  http://127.0.0.1:80/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Completed 2000 requests
Finished 2000 requests


Server Software:        nginx
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /
Document Length:        162 bytes

Concurrency Level:      20
Time taken for tests:   2.007 seconds
Complete requests:      2000
Failed requests:        0
Non-2xx responses:      2000
Keep-Alive requests:    1988
Total transferred:      619940 bytes
HTML transferred:       324000 bytes
Requests per second:    996.74 [#/sec] (mean)
Time per request:       20.065 [ms] (mean)
Time per request:       1.003 [ms] (mean, across all concurrent requests)
Transfer rate:          301.72 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     2   20  23.0      3      94
Waiting:        2   20  23.0      3      94
Total:          3   20  23.0      3      94

Percentage of the requests served within a certain time (ms)
  50%      3
  66%     43
  75%     44
  80%     44
  90%     58
  95%     60
  98%     61
  99%     70
 100%     94 (longest request)
root@waf-1-a-02:~# ab -n 3000 -c 20 -k  http://127.0.0.1:80/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 300 requests
Completed 600 requests
Completed 900 requests
Completed 1200 requests
Completed 1500 requests
Completed 1800 requests
Completed 2100 requests
Completed 2400 requests
Completed 2700 requests
Completed 3000 requests
Finished 3000 requests


Server Software:        nginx
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /
Document Length:        162 bytes

Concurrency Level:      20
Time taken for tests:   2.702 seconds
Complete requests:      3000
Failed requests:        0
Non-2xx responses:      3000
Keep-Alive requests:    2981
Total transferred:      929905 bytes
HTML transferred:       486000 bytes
Requests per second:    1110.42 [#/sec] (mean)
Time per request:       18.011 [ms] (mean)
Time per request:       0.901 [ms] (mean, across all concurrent requests)
Transfer rate:          336.13 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       3
Processing:     2   18  20.5      3      90
Waiting:        2   18  20.5      3      90
Total:          2   18  20.5      3      91

Percentage of the requests served within a certain time (ms)
  50%      3
  66%     13
  75%     42
  80%     43
  90%     44
  95%     58
  98%     59
  99%     60
 100%     91 (longest request)
root@waf-1-a-02:~# ab -n 10000 -c 20 -k  http://127.0.0.1:80/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /
Document Length:        162 bytes

Concurrency Level:      20
Time taken for tests:   5.262 seconds
Complete requests:      10000
Failed requests:        0
Non-2xx responses:      10000
Keep-Alive requests:    9913
Total transferred:      3099565 bytes
HTML transferred:       1620000 bytes
Requests per second:    1900.24 [#/sec] (mean)
Time per request:       10.525 [ms] (mean)
Time per request:       0.526 [ms] (mean, across all concurrent requests)
Transfer rate:          575.19 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       3
Processing:     2   11  16.2      3      91
Waiting:        2   10  16.2      3      91
Total:          2   11  16.2      3      91

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      3
  75%      5
  80%      6
  90%     43
  95%     45
  98%     57
  99%     59
 100%     91 (longest request)

@defanator
Copy link
Collaborator

Hi @mimugmail, @zimmerle,

I have created a separate repository that can be used to easily spin up reproducible environments (thanks goes to Vagrant and SaltStack) suitable for running benchmarks against nginx with libmodsecurity connector module:

https://github.com/defanator/modsecurity-performance

It also can be extended to include more sophisticated things, if anyone would like to.

For now I'm seeing more than 4x performance drop when libmodsecurity is turned on without any actual rules configured, and more than 170x performance drop when libmodsecurity is turned on with OWASP CRS v3.0.0 loaded (I'm not including any absolute numbers here for obvious reasons; you can find my own results in the repo - I'll add a bit more details about the hardware used as well).

Main goal is to give an ability for anyone to easily reproduce the tests against particular combination of libmodsecurity, ModSecurity-nginx connector module, and nginx.

@defanator
Copy link
Collaborator

@mimugmail - can I ask you to share your modsecurity.conf you were using while testing?

(if it differs from the default one here - https://github.com/SpiderLabs/ModSecurity/blob/master/modsecurity.conf-recommended)

@mimugmail
Copy link
Author

mimugmail commented Mar 1, 2017 •
edited by defanator
Loading 

# -- Rule engine initialization ----------------------------------------------

# Enable ModSecurity, attaching it to every transaction. Use detection
# only to start with, because that minimises the chances of post-installation
# disruption.
#
SecRuleEngine On


# -- Request body handling ---------------------------------------------------

# Allow ModSecurity to access request bodies. If you don't, ModSecurity
# won't be able to see any POST parameters, which opens a large security
# hole for attackers to exploit.
#
SecRequestBodyAccess On


# Enable XML request body parser.
# Initiate XML Processor in case of xml content-type
#
SecRule REQUEST_HEADERS:Content-Type "(?:text|application)/xml" \
     "id:'200000',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"

# Enable JSON request body parser.
# Initiate JSON Processor in case of JSON content-type; change accordingly
# if your application does not use 'application/json'
#
SecRule REQUEST_HEADERS:Content-Type "application/json" \
     "id:'200001',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"

# Maximum request body size we will accept for buffering. If you support
# file uploads then the value given on the first line has to be as large
# as the largest file you are willing to accept. The second value refers
# to the size of data, with files excluded. You want to keep that value as
# low as practical.
#
SecRequestBodyLimit 536870912
SecRequestBodyNoFilesLimit 536870912

# Store up to 128 KB of request body data in memory. When the multipart
# parser reachers this limit, it will start using your hard disk for
# storage. That is slow, but unavoidable.
#
SecRequestBodyInMemoryLimit 134217728

# What do do if the request body size is above our configured limit.
# Keep in mind that this setting will automatically be set to ProcessPartial
# when SecRuleEngine is set to DetectionOnly mode in order to minimize
# disruptions when initially deploying ModSecurity.
#
SecRequestBodyLimitAction ProcessPartial

# Verify that we've correctly processed the request body.
# As a rule of thumb, when failing to process a request body
# you should reject the request (when deployed in blocking mode)
# or log a high-severity alert (when deployed in detection-only mode).
#
SecRule REQBODY_ERROR "!@eq 0" \
"id:'200002', phase:2,t:none,log,deny,status:400,msg:'Failed to parse request body.',logdata:'%{reqbody_error_msg}',severity:2"

# By default be strict with what we accept in the multipart/form-data
# request body. If the rule below proves to be too strict for your
# environment consider changing it to detection-only. You are encouraged
# _not_ to remove it altogether.
#
SecRule MULTIPART_STRICT_ERROR "!@eq 0" \
"id:'200003',phase:2,t:none,log,deny,status:400, \
msg:'Multipart request body failed strict validation: \
PE %{REQBODY_PROCESSOR_ERROR}, \
BQ %{MULTIPART_BOUNDARY_QUOTED}, \
BW %{MULTIPART_BOUNDARY_WHITESPACE}, \
DB %{MULTIPART_DATA_BEFORE}, \
DA %{MULTIPART_DATA_AFTER}, \
HF %{MULTIPART_HEADER_FOLDING}, \
LF %{MULTIPART_LF_LINE}, \
SM %{MULTIPART_MISSING_SEMICOLON}, \
IQ %{MULTIPART_INVALID_QUOTING}, \
IP %{MULTIPART_INVALID_PART}, \
IH %{MULTIPART_INVALID_HEADER_FOLDING}, \
FL %{MULTIPART_FILE_LIMIT_EXCEEDED}'"

# Did we see anything that might be a boundary?
#
SecRule MULTIPART_UNMATCHED_BOUNDARY "!@eq 0" \
"id:'200004',phase:2,t:none,log,deny,msg:'Multipart parser detected a possible unmatched boundary.'"

# PCRE Tuning
# We want to avoid a potential RegEx DoS condition
#
SecPcreMatchLimit 1000
SecPcreMatchLimitRecursion 1000

# Some internal errors will set flags in TX and we will need to look for these.
# All of these are prefixed with "MSC_".  The following flags currently exist:
#
# MSC_PCRE_LIMITS_EXCEEDED: PCRE match limits were exceeded.
#
SecRule TX:/^MSC_/ "!@streq 0" \
        "id:'200005',phase:2,t:none,deny,msg:'ModSecurity internal error flagged: %{MATCHED_VAR_NAME}'"


# -- Response body handling --------------------------------------------------

# Allow ModSecurity to access response bodies.
# You should have this directive enabled in order to identify errors
# and data leakage issues.
#
# Do keep in mind that enabling this directive does increases both
# memory consumption and response latency.
#
SecResponseBodyAccess On

# Which response MIME types do you want to inspect? You should adjust the
# configuration below to catch documents but avoid static files
# (e.g., images and archives).
#
SecResponseBodyMimeType text/plain text/html text/xml

# Buffer response bodies of up to 512 KB in length.
SecResponseBodyLimit 524288

# What happens when we encounter a response body larger than the configured
# limit? By default, we process what we have and let the rest through.
# That's somewhat less secure, but does not break any legitimate pages.
#
SecResponseBodyLimitAction ProcessPartial


# -- Filesystem configuration ------------------------------------------------

# The location where ModSecurity stores temporary files (for example, when
# it needs to handle a file upload that is larger than the configured limit).
#
# This default setting is chosen due to all systems have /tmp available however,
# this is less than ideal. It is recommended that you specify a location that's private.
#
SecTmpDir /tmp/

# The location where ModSecurity will keep its persistent data.  This default setting
# is chosen due to all systems have /tmp available however, it
# too should be updated to a place that other users can't access.
#
SecDataDir /tmp/


# -- File uploads handling configuration -------------------------------------

# The location where ModSecurity stores intercepted uploaded files. This
# location must be private to ModSecurity. You don't want other users on
# the server to access the files, do you?
#
#SecUploadDir /opt/modsecurity/var/upload/

# By default, only keep the files that were determined to be unusual
# in some way (by an external inspection script). For this to work you
# will also need at least one file inspection rule.
#
#SecUploadKeepFiles RelevantOnly

# Uploaded files are by default created with permissions that do not allow
# any other user to access them. You may need to relax that if you want to
# interface ModSecurity to an external program (e.g., an anti-virus).
#
#SecUploadFileMode 0600


# -- Debug log configuration -------------------------------------------------

# The default debug log configuration is to duplicate the error, warning
# and notice messages from the error log.
#
#SecDebugLog /var/log/modsec_debug.log
#SecDebugLogLevel 4


# -- Audit log configuration -------------------------------------------------

# Log the transactions that are marked by a rule, as well as those that
# trigger a server error (determined by a 5xx or 4xx, excluding 404,
# level response status codes).
#
#SecAuditEngine Off
SecAuditEngine RelevantOnly
#SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogRelevantStatus "403"

# Log everything we know about a transaction.
#SecAuditLogParts ABIJDEFHZ
SecAuditLogParts ABCDEFHIJKZ

# Use a single file for logging. This is much easier to look at, but
# assumes that you will use the audit log only ocassionally.
#
SecAuditLogType Serial
SecAuditLog /var/log/modsec_audit.log

# Specify the path for concurrent audit logging.
#SecAuditLogStorageDir /opt/modsecurity/var/audit/


# -- Miscellaneous -----------------------------------------------------------

# Use the most commonly used application/x-www-form-urlencoded parameter
# separator. There's probably only one application somewhere that uses
# something else so don't expect to change this value.
#
SecArgumentSeparator &

# Settle on version 0 (zero) cookies, as that is what most applications
# use. Using an incorrect cookie version may open your installation to
# evasion attacks (against the rules that examine named cookies).
#
SecCookieFormat 0

# Specify your Unicode Code Point.
# This mapping is used by the t:urlDecodeUni transformation function
# to properly map encoded data to your language. Properly setting
# these directives helps to reduce false positives and negatives.
#
SecUnicodeMapFile unicode.mapping 20127

# Improve the quality of ModSecurity by sharing information about your
# current ModSecurity version and dependencies versions.
# The following information will be shared: ModSecurity version,
# Web Server version, APR version, PCRE version, Lua version, Libxml2
# version, Anonymous unique id for host.
SecStatusEngine On

@defanator
Copy link
Collaborator

@mimugmail, @zimmerle, I've spent some time running simple benchmarks on multi-core VM, and found something interesting about SecAuditEngine:

https://github.com/defanator/modsecurity-performance/wiki#2017-03-05

@defanator
Copy link
Collaborator

@mimugmail, I've just updated your comments to get proper formatting for preformatted text (commands output) - JFYI.

@mimugmail
Copy link
Author

@defanator Thanks :) Just did a copy+paste.

This behavior fits perfectly with my measurements. Have you already tried concurrent logging?
I changed to:

SecAuditLogType Concurrent
SecAuditLog /var/log/modsec_audit.log
SecAuditLogStorageDir /opt/nginx/logs/audit

But performance is still around 144req/s

@zimmerle
Copy link
Contributor

zimmerle commented Mar 9, 2017

Hi @mimugmail and @defanator,

Do you mind to test the branch v3/dev/speedup?

I did some performance improvements on v3/dev/speedup. There are space for more, but it will be good to count in your tests to confirm that I am on the right track.

Oh btw, please give it a try with the SecAuditLog disabled, going to look into the audit log generation tomorrow.

@defanator
Copy link
Collaborator

@zimmerle, please check https://github.com/defanator/modsecurity-performance/wiki#2017-03-10

I ran a few cycles with SecAuditEngine RelevantOnly, and then switched to SecAuditEngine Off - numbers were nearly the same, in the margin of error.

However, in the worst case (full OWASP CRS v3.0.0 is loaded) average RPS is more than 10% greater, and latency is more than 10% lower than in previous run (v3/master as of 20170307).

SecAuditEngine RelevantOnly seems to affect only multi-worker configurations (probably some locking issues - I did not have time to test further).

@mimugmail
Copy link
Author

@zimmerle

With SecAuditEngine On

`root@nginx:/opt# ab -n 10000 -c 20 -k http://127.0.0.1:80/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software: nginx/1.11.9
Server Hostname: 127.0.0.1
Server Port: 80

Document Path: /
Document Length: 612 bytes

Concurrency Level: 20
Time taken for tests: 80.526 seconds
Complete requests: 10000
Failed requests: 0
Keep-Alive requests: 9909
Total transferred: 8499545 bytes
HTML transferred: 6120000 bytes
Requests per second: 124.18 [#/sec] (mean)
Time per request: 161.052 [ms] (mean)
Time per request: 8.053 [ms] (mean, across all concurrent requests)
Transfer rate: 103.08 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 15 161 32.7 159 1517
Waiting: 15 161 32.7 159 1517
Total: 15 161 32.7 159 1517

Percentage of the requests served within a certain time (ms)
50% 159
66% 161
75% 163
80% 165
90% 169
95% 175
98% 184
99% 215
100% 1517 (longest request)
root@nginx:/opt# ab -n 1000 -c 20 -k http://127.0.0.1:80/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software: nginx/1.11.9
Server Hostname: 127.0.0.1
Server Port: 80

Document Path: /
Document Length: 612 bytes

Concurrency Level: 20
Time taken for tests: 8.606 seconds
Complete requests: 1000
Failed requests: 0
Keep-Alive requests: 1000
Total transferred: 850000 bytes
HTML transferred: 612000 bytes
Requests per second: 116.19 [#/sec] (mean)
Time per request: 172.127 [ms] (mean)
Time per request: 8.606 [ms] (mean, across all concurrent requests)
Transfer rate: 96.45 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 7 171 18.3 169 243
Waiting: 7 171 18.3 169 243
Total: 8 171 18.2 169 243

Percentage of the requests served within a certain time (ms)
50% 169
66% 173
75% 176
80% 177
90% 183
95% 199
98% 211
99% 226
100% 243 (longest request)`

WIth SecAuditEngine Off results are the same ...

@zimmerle
Copy link
Contributor

@mimugmail @defanator thanks!

@mimugmail how many workers do you have?

@mimugmail
Copy link
Author

@zimmerle meeep you got me, now it's 100 (was 1) 👍

SecAuditEngine Off:

`root@nginx:/opt# ab -n 10000 -c 20 -k http://127.0.0.1:80/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software: nginx/1.11.9
Server Hostname: 127.0.0.1
Server Port: 80

Document Path: /
Document Length: 612 bytes

Concurrency Level: 20
Time taken for tests: 25.751 seconds
Complete requests: 10000
Failed requests: 0
Keep-Alive requests: 9909
Total transferred: 8499545 bytes
HTML transferred: 6120000 bytes
Requests per second: 388.34 [#/sec] (mean)
Time per request: 51.501 [ms] (mean)
Time per request: 2.575 [ms] (mean, across all concurrent requests)
Transfer rate: 322.34 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.8 0 28
Processing: 7 51 71.6 33 1140
Waiting: 0 50 71.5 32 1140
Total: 7 51 72.0 33 1151

Percentage of the requests served within a certain time (ms)
50% 33
66% 42
75% 49
80% 56
90% 92
95% 148
98% 235
99% 352
100% 1151 (longest request)

`SecAuditEngine On:

`root@nginx:/opt# ab -n 10000 -c 20 -k http://127.0.0.1:80/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 9981 requests completed`

Nginx 1.11.9, 4GB RAM, 4 CPUs

Also with just 1000:

`root@nginx:/opt# ab -n 1000 -c 20 -k http://127.0.0.1:80/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 981 requests completed`

@zimmerle
Copy link
Contributor

@mimugmail Great! Thanks ;)

@defanator
Copy link
Collaborator

@mimugmail, running 100 workers on 4-core machine serving CPU-heavy load (like ModSecurity) won't lead to any benefits. I don't see much sense in setting worker_processes to a number greater than total number of available cores (for this particular pattern).

Also, your latest results do not include any summary for SecAuditEngine On case due to timeouts detected by ab tool.

@mimugmail
Copy link
Author

@defanator Yep, I set it to 4, no difference. Funny thing is that between 1000 and 10000 checks it always stucks at the last 105.

With just 100 I get:
Requests per second: 206.24 [#/sec] (mean)

@zimmerle zimmerle self-assigned this Mar 10, 2017
@zimmerle
Copy link
Contributor

the branch v3/dev/speedup was merged into v3/master.

@AnoopAlias
Copy link

2G KVM VPS 1vcore

 ab -n 10000 -c 20 -k http://theo.in/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking theo.in (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        XtendWeb-nginx
Server Hostname:        theo.in
Server Port:            80

Document Path:          /
Document Length:        623 bytes

Concurrency Level:      20
Time taken for tests:   156.437 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    9908
Total transferred:      8149540 bytes
HTML transferred:       6230000 bytes
Requests per second:    63.92 [#/sec] (mean)
Time per request:       312.873 [ms] (mean)
Time per request:       15.644 [ms] (mean, across all concurrent requests)
Transfer rate:          50.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3  28.2      0     465
Processing:   190  310  74.8    297     688
Waiting:      190  310  74.8    297     688
Total:        190  313  80.7    298     959

Percentage of the requests served within a certain time (ms)
  50%    298
  66%    327
  75%    346
  80%    359
  90%    401
  95%    468
  98%    556
  99%    599
 100%    959 (longest request)


# nginx -V
nginx version: nginx/1.11.12
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) 
built with LibreSSL 2.5.1
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/etc/nginx/modules --with-pcre=./pcre-8.40 --with-pcre-jit --with-zlib=./zlib-1.2.11 --with-openssl=./libressl-2.5.1 --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error_log --http-log-path=/var/log/nginx/access_log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nobody --group=nobody --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --add-dynamic-module=naxsi-http2/naxsi_src --with-file-aio --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-compat --with-http_v2_module --with-http_geoip_module=dynamic --add-dynamic-module=ngx_pagespeed-release-1.11.33.4-beta --add-dynamic-module=/usr/local/rvm/gems/ruby-2.3.1/gems/passenger-5.1.2/src/nginx_module --add-dynamic-module=ngx_brotli --add-dynamic-module=echo-nginx-module-0.60 --add-dynamic-module=headers-more-nginx-module-0.32 --add-dynamic-module=ngx_http_redis-0.3.8 --add-dynamic-module=redis2-nginx-module --add-dynamic-module=srcache-nginx-module-0.31 --add-dynamic-module=ngx_devel_kit-0.3.0 --add-dynamic-module=set-misc-nginx-module-0.31 --add-dynamic-module=ModSecurity-nginx --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' --with-ld-opt=-Wl,-E

nginx_conf.txt

@defanator
Copy link
Collaborator

A couple of benchmarks with recent libmodsecurity sources: https://github.com/defanator/modsecurity-performance/wiki#2017-04-11-1 (single worker), https://github.com/defanator/modsecurity-performance/wiki#2017-04-11-2 (multi worker).

Noticeable improvement is that multi-worker setup does not lock all the processing inside single worker (as it was before [1]), so I was able to reach almost 1k RPS with the full OWASP CRS v3.0.0 loaded, with 6 workers.

[1] https://github.com/defanator/modsecurity-performance/wiki#2017-03-05

@zimmerle
Copy link
Contributor

zimmerle commented May 4, 2017

I am closing this issue in favor of @defanator wiki. I would encourage you guys to keep the work monitoring the performance. whenever you find a problem with it, please re-open this issue.

@zimmerle zimmerle closed this as completed May 4, 2017
@luengnat
Copy link

@defanator did you get a chance to re-run the benchmark again?

@defanator
Copy link
Collaborator

@luengnat I run it periodically, just updated the results page with most recent build here: https://github.com/defanator/modsecurity-performance/wiki#2017-12-11

@tatdatpham
Copy link

anyone have ideal for solving this problem. I'm using v3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zimmerle zimmerle

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@zimmerle @defanator @tatdatpham @AnoopAlias @luengnat @mimugmail