Missing transaction logs for blocked request ingress-nginx

[mkuchniak]

Describe the bug

We are missing transaction logs for some of the blocked requests.
Logs and dumps

188#188: *3490527 [client my.ip] ModSecurity: Access denied with code 403 (phase 2). Matched "Operator Ge' with parameter 5' against variable TX:BLOCKING_INBOUND_ANOMALY_SCORE' (Value: 20' ) [file "/etc/nginx/owasp-modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "222"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 20)"] [data ""] [severity "0"] [ver "OWASP_CRS/4.4.0"] [maturity "0"] [accuracy "0"] [tag "anomaly-evaluation"] [tag "OWASP_CRS"] [hostname "192.168.66.83"] [uri "/courses"] [unique_id "174185963544.478079"] [ref ""]

Output of:

1. DebugLogs (level 9)
2. AuditLogs
3. Error logs
4. If there is a crash, the core dump file.

Notice: Be careful to not leak any confidential information.

To Reproduce

Steps to reproduce the behavior:
curl -v "http://domain.com/?q=<script>alert('XSS')</script>"

trigger XSS rules
https://github.com/coreruleset/coreruleset/blob/main/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf

Expected behavior

Transaction logs for all requests

Server (please complete the following information):

• ModSecurity version (and connector): [e.g. ModSecurity v3.0.12 with nginx-connector v1.0.3]
• WebServer: [nginx-1.25.5]
• OS (and distro): [e.g. Linux, kubernetes]

Additional context

Add any other context about the problem here.

[airween]

Hi @mkuchniak,

Describe the bug

We are missing transaction logs for some of the blocked requests. Logs and dumps

Could you explain which blocked requests' log are missing from transaction log? (And do you mean audit.log?)

> 188#188: *3490527 [client my.ip] ModSecurity: Access denied with code 403 (phase 2). Matched "Operator `Ge' with parameter `5' against variable `TX:BLOCKING_INBOUND_ANOMALY_SCORE' (Value: `20' ) [file "/etc/nginx/owasp-modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "222"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 20)"] [data ""] [severity "0"] [ver "OWASP_CRS/4.4.0"] [maturity "0"] [accuracy "0"] [tag "anomaly-evaluation"] [tag "OWASP_CRS"] [hostname "192.168.66.83"] [uri "/courses"] [unique_id "174185963544.478079"] [ref ""]

I assume this is the content of error.log, right? Isn't there any other lines with unique_id "174185963544.478079"?

What is your settings of error.log parameter in Nginx's config?

[mkuchniak]

@airween
There are no other logs for this unique_id

We are using ingress-nginx with mod_security on Kubernetes, we send all our logs to stdout

So we only see the final log for the request blocked, but we don't see all the processing happening. Example from different request:

{
  "accuracy": 0,
  "data": "Matched Data: .env found within REQUEST_FILENAME: /.env.save",
  "file": "/etc/nginx/owasp-modsecurity-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf",
  "lineNumber": 124,
  "match": "Matched \"Operator `PmFromFile' with parameter `restricted-files.data' against variable `REQUEST_FILENAME' (Value: `/.env.save' )",
  "maturity": 0,
  "reference": "o1,4v4,10t:utf8toUnicode,t:urlDecodeUni,t:normalizePathWin",
  "rev": "",
  "ruleId": 930130,
  "severity": 2,
  "tags": [
    "application-multi",
    "language-multi",
    "platform-multi",
    "attack-lfi",
    "paranoia-level/1",
    "OWASP_CRS",
    "capec/1000/255/153/126",
    "PCI/6.5.4"
  ],
  "ver": "OWASP_CRS/4.4.0"
}

In previous output we see rule 949110 which is anomaly score, but we don't see the log on what inputed to that score.

[airween]

We are using ingress-nginx with mod_security on Kubernetes, we send all our logs to stdout

Sorry, I don't know Ingress nor Kubernetes.

So you don't have any error.log directive in your config? Is there any chance to try

    error_log /dev/stdout info;

?

[mkuchniak]

By default the nginx audit and error log is configured to pass to stdout

lrwxrwxrwx    1 www-data www-data      11 Dec 25 22:27 access.log -> /dev/stdout
lrwxrwxrwx    1 www-data www-data      11 Dec 25 22:27 error.log -> /dev/stderr
ingress-nginx-controller-94df9f9c5-f5hlh:/var/log/nginx$ 

And in the mod_sec config we have:

SecAuditLogType Serial
SecAuditLog /dev/stdout

[airween]

By default the nginx audit and error log is configured to pass to stdout

I see, but could you set up that with explicit way as I showed you above?

[mkuchniak]

Ok so after a lot of testing and comparing two deployments of nginx next to eachother, we concluded that this is caused by using "custom-http-errors" we have a seperate deployment for showing custom error pages with our desing, this is using:

extraArgs:
   default-backend-service: namespace/our-service

There we have custom error pages for 4xx and 5xx pages includes 404 429 403 422. In nginx config map we have configured:

custom-http-errors: "403,404,429,500,502,503"

When I've removed the 403 from custom pages, we can now see all transaction logs from mod_sec.

Is this a know issue?