Stop using CephFS for storing central services logs

[mapellidario]

Impact of the new feature

Central services, or whatever else we will ever want to run on k8s

Is your feature request related to a problem? Please describe.

At the moment, we mount the same CephFS volume into every wmcore container. Our services writes log files in that volume. However, rotating the logs requires maintenance effort and depending on CephFS means that our services can not run if Ceph has troubles.

Describe the solution you'd like

We should use tools that better integrate with our k8s infrastructure, similar to what CRAB did in the recent past [1]

Describe alternatives you've considered

We could implement better retention policies for logs in CephFS, however it does not make our central services less fragile.

Additional context

(I will provide details about the current CRAB implementation in the next comment)

---

[1] dmwm/CRABServer#7463

[mapellidario]

CRAB team has already implemented this, this comment summarizes how the achieved it.

references

• meta issue: Deprecating CephFS for writing crabserver logs CRABServer#7408
• issue: Make CRAB REST stream all logs to container's stdout CRABServer#7463
• ticket to cmsweb team https://its.cern.ch/jira/projects/CMSKUBERNETES/issues/CMSKUBERNETES-215?filter=allopenissues

implementation

• design: Make CRAB REST stream all logs to container's stdout CRABServer#7463 (comment)
  • Definitely read this comment! it's a must!
• PR: pipe crablog to stdout CRABServer#7496 2022-12
  • notice the use of
  • hdlr = logging.StreamHandler()
  • h = cherrypy.log._get_builtin_handler(cherrypy.log.access_log, 'screen'), h = cherrypy.log._get_builtin_handler(cherrypy.log.error_log, 'screen')
• issue CRAB REST cannot write logs to named pipe with PR #11431 #11459
  • PR Open REST-based file in append mode, update deactivated #11463
  • problem with opening fifo with a+, just use a
• PR Add start2() to run CRABREST with argument writing logs to named pipe deployment#1240
  • change manage: use start2 with different call to wmc-httpd, without rotatelogs
• PR: https://github.com/dmwm/CMSKubernetes/pull/1308/files 2023-02
  • change run.sh: introduces mkfifo
  • monitor.sh
  • logstash.conf

Then crab moved to using WMCore pypi packages, and refactored how they build their docker images. the current implementation is as follows:

• crabserver docker image calls run.sh at the end
  • https://github.com/dmwm/CRABServer/blob/master/cicd/crabserver_pypi/Dockerfile
• run.sh creates a fifo / named pipe and then calls manage.py start -c
  • https://github.com/dmwm/CRABServer/blob/master/cicd/crabserver_pypi/run.sh
  • pay attention to the need of cat /fifo & at the beginning
  • then manage.py will eventually call a double fork, hence the need for closing with a while-true loop of cat /fifo
• manage.py is just a wrapper, it calls manage.sh start (setting env variable MODE=current)
  • https://github.com/dmwm/CRABServer/blob/master/cicd/crabserver_pypi/manage.sh
• manage.sh start loads proper env then starts wmc-https
  • https://github.com/dmwm/CRABServer/blob/master/cicd/crabserver_pypi/manage.sh
  • wmc-httpd -r -d $STATEDIR -l "$STATEDIR/crabserver-fifo" $CFGFILE
  • keep in mind that this uses doublefork to create a daemon

  • WMCore/src/python/WMCore/REST/Main.py

    Line 421 in aafdebb

    # First fork. Discard the parent.

I invite anybody to edit this comment if it contains imprecise information.

And I want to close witho a great shout-out to Wa ( @novicecpp ) for the implementation. I always admired how you manged to convert frustration into deep technical investigations and robust implementations. thanks man :)

[mapellidario]

Valentin raised valid points in a private chat. I will try to address them with this comment, for public record and interest.

We eventually decided to use a named pipe / fifo file because we did not want to change how we handled operations. We all liked the freedom of stopping and restaring the crabserver from inside the container, from an interactive shell. if wmc-httpd writes directly to stdout, when you restart the process it goes to your shell, not to k8s machinery. See this comment from Wa [1].

---

[1] dmwm/CRABServer#7463 (comment)

The disadvantage of this solution is when we restart crabserver with /data/start.sh, i.e., to run in GH mode, it will use your shell stdout instead.

[mapellidario]

Then I will add some details about the implementation of the log streaming after the application, on the k8s infrastructure side.

• every crab pod logs is visible with kubectl logs crabserver-??-?? or with stern thanks to the while true; cat /fifo; end at the end of run.sh
  • https://github.com/dmwm/CRABServer/blob/master/cicd/crabserver_pypi/run.sh
• filebeat runs as a daemonset on every node of the production cluster
  • https://github.com/dmwm/CMSKubernetes/blob/master/helm/crabserver/templates/filebeat-daemonset.yaml
  • it mounts logs from the host at the paths /var/log/pods and /var/log/containers
  • for a crabserver container for example, you can see both
    • /var/log/pods/crab_crabserver-5db496f954-pqx2r_e369ef4e-48f0-4969-b065-10f2e9619727/crabserver/0.log
    • /var/log/container/crabserver-5db496f954-pqx2r_crab_crabserver-3756bb30f92f065abd75a3370302265d6b594291c1e2487474666fa8aacd8b83.log
  • filebeat reads from /var/log/pod and send data to a logstash instance
    • https://github.com/dmwm/CMSKubernetes/blob/master/helm/crabserver/templates/filebeat-configmap.yaml
  • filebeat sends data to a logstash pod running in the crab namespace
• logstash
  • runs as a deployment, single pod, https://github.com/dmwm/CMSKubernetes/blob/master/helm/crabserver/templates/logstash-deployment.yaml
  • with this configmap https://github.com/dmwm/CMSKubernetes/blob/master/helm/crabserver/templates/logstash-configmap.yaml
  • effectively loading this configuration https://github.com/dmwm/CMSKubernetes/tree/master/helm/crabserver/config/prod/logstash
  • logstash sends data to monit-logs.cern.ch, for adding data to opensearch and eventually grafana
• grafana
  • https://monit-grafana.cern.ch/goto/Kxpv79-HR?orgId=11 this dashboard contains data coming from the pod logs

there is another pipeline that reads crabserver logs from /var/log/pods on ever cluster node and sends them to s3.

• it is possible to read them from vocms0755, for example at /data/s3/logs/cmsweb/crab-logs/crabserver/2025/05/21/cmsweb-k8s-prodsrv2-c6ph6-startAt=00:00_0.txt
• I am not sure where to find the details for this pipeline inside the dmwm/CMSubernetes repository, i am only aware of the schema in this comment Make CRAB REST stream all logs to container's stdout CRABServer#7463 (comment)

[mapellidario]

I opened a ticket with CMSWEB team to followup with some small friction points that we identified when evaluating this solution: https://its.cern.ch/jira/projects/CMSKUBERNETES/issues/CMSKUBERNETES-292?filter=allissues

[mapellidario]

How to access live logs to our services - now

(you may need to specify the -n dmwm and -n crab to reproduce the following commands)

If you alredy know which pod you need to look into, then you can read the service logs as stdout of the filebeat sidecar container

kubectl logs --tail=100 -f reqmgr2-6b5c4bdd68-c2w4q reqmgr2-filebeat

Otherwise, use stern. It allows to stream logs from multiple containers. See also CMSWEB documentation about this https://cms-http-group.docs.cern.ch/k8s_cluster/k8s_logs/#why-use-stern

> stern "reqmgr.*" -c filebeat --since=10m
+ reqmgr2-6b5c4bdd68-c2w4q › reqmgr2-filebeat
+ reqmgr2-tasks-77fd97978b-5xnfl › reqmgr2-tasks-filebeat
+ reqmgr2-6b5c4bdd68-rjsjw › reqmgr2-filebeat
reqmgr2-6b5c4bdd68-rjsjw reqmgr2-filebeat [05/Jun/2025:14:26:55] reqmgr2-6b5c4bdd68-rjsjw 137.138.54.240 "GET /reqmgr2/data/wmagentconfig/vocms266.cern.ch HTTP/1.1" 200 OK [data: 1774 in 306 out 20420 us ] [auth: ok "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=cmst1/CN=718748/CN=Robot: cms t1" "" ] [ref: "https://cmsweb-preprod.cern.ch" "WMCore.Services.Requests/v002" ] - Podname=reqmgr2-6b5c4bdd68-rjsjw
reqmgr2-6b5c4bdd68-rjsjw reqmgr2-filebeat [05/Jun/2025:14:26:57] reqmgr2-6b5c4bdd68-rjsjw 188.185.29.242 "GET /reqmgr2/data/metrics HTTP/1.1" 200 OK [data: 1840 in 9214 out 1585 us ] [auth: ok "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=cmsmonit/CN=779320/CN=Robot: CMS Monitoring" "" ] [ref: "https://cmsweb-preprod.cern.ch" "Prometheus/3.1.0" ] - Podname=reqmgr2-6b5c4bdd68-rjsjw
[...]

the first three lines contain the list of pods that match the selection. every log line starts with the podname and container name.

or to grep across multiple containers:

> stern "reqmgr.*" -c filebeat --since=1h --tail=300 --no-follow | grep 'dmapelli_SC_ProdPsi_test_jun2025py312_v1_250526_142036_1442'
+ reqmgr2-tasks-77fd97978b-5xnfl › reqmgr2-tasks-filebeat
+ reqmgr2-6b5c4bdd68-c2w4q › reqmgr2-filebeat
+ reqmgr2-6b5c4bdd68-rjsjw › reqmgr2-filebeat
reqmgr2-6b5c4bdd68-rjsjw reqmgr2-filebeat [05/Jun/2025:14:28:35] reqmgr2-6b5c4bdd68-rjsjw 188.185.31.221 "GET /reqmgr2/data/request?name=dmapelli_SC_ProdPsi_test_jun2025py312_v1_250526_142036_1442 HTTP/1.1" 200 OK [data: 1795 in 2245 out 15019 us ] [auth: ok "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=cmst1/CN=718748/CN=Robot: cms t1" "" ] [ref: "https://cmsweb-preprod.cern.ch" "WMCore.Services.Requests/v002" ] - Podname=reqmgr2-6b5c4bdd68-rjsjw
reqmgr2-6b5c4bdd68-rjsjw reqmgr2-filebeat [05/Jun/2025:14:32:37] reqmgr2-6b5c4bdd68-rjsjw 137.138.54.240 "GET /reqmgr2/data/request?name=dmapelli_SC_ProdPsi_test_jun2025py312_v1_250526_142036_1442 HTTP/1.1" 200 OK [data: 1795 in 2245 out 22233 us ] [auth: ok "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=cmst1/CN=718748/CN=Robot: cms t1" "" ] [ref: "https://cmsweb-preprod.cern.ch" "WMCore.Services.Requests/v002" ] - Podname=reqmgr2-6b5c4bdd68-rjsjw
- reqmgr2-6b5c4bdd68-rjsjw › reqmgr2-filebeat
reqmgr2-6b5c4bdd68-c2w4q reqmgr2-filebeat [05/Jun/2025:14:30:36] reqmgr2-6b5c4bdd68-c2w4q 188.185.114.255 "GET /reqmgr2/data/request?name=dmapelli_SC_ProdPsi_test_jun2025py312_v1_250526_142036_1442 HTTP/1.1" 200 OK [data: 1797 in 2245 out 27834 us ] [auth: ok "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=cmst1/CN=718748/CN=Robot: cms t1" "" ] [ref: "https://cmsweb-preprod.cern.ch" "WMCore.Services.Requests/v002" ] - Podname=reqmgr2-6b5c4bdd68-c2w4q
- reqmgr2-tasks-77fd97978b-5xnfl › reqmgr2-tasks-filebeat
- reqmgr2-6b5c4bdd68-c2w4q › reqmgr2-filebeat

in this case, the line - reqmgr2-6b5c4bdd68-rjsjw › reqmgr2-filebeat tells which pod produced the lines that come before it.

how to access live logs after we implement changes described above

After we implement the changes described above, specifying the filebeat container will no longer be necessary as the our service will send its logs to stdout, for example to access crabserver logs you can use

> kubectl logs --tail=100 -f crabserver-55df5c4d79-nk45n

and

> stern "crabserver.*" --since=10m
+ crabserver-55df5c4d79-nk45n › crabserver
crabserver-55df5c4d79-nk45n crabserver [05/Jun/2025:16:34:55]  RESTSQL:pmVsELWZuAwi ENTER cmsweb_analysis_preprod@int2r CRABInterface.RESTBaseAPI.RESTBaseAPI (GET preprod workflowdb) inuse=0 idle=1 - Podname=crabserver-55df5c4d79-nk45n Type=cherrypylog
crabserver-55df5c4d79-nk45n crabserver [05/Jun/2025:16:34:55]  RESTSQL:pmVsELWZuAwi (previously yJFkaONRdUXA) connected, client: 21.5.0.0.0, server: 19.25.0.0.0, stmtcache: 50 - Podname=crabserver-55df5c4d79-nk45n Typ
[...]

[mapellidario]

how to access logs from long term storage

Logs will be stored in cern openstack object store, split per pod and in 15min time intervals.

These logs are copied into the vocms0755 VM every 5min and can be accessed at:

/data/s3/logs/cmsweb/$namespace-logs/$deployment/YYYY/MM/DD

for example

/data/s3/logs/cmsweb/crab-logs/crabserver/2025/06/04

since this files are copied to a local disk of the VM, all filesystem operations are fast. If you were to list the files or their content via http (for example using aws cli or a fuse mount such as https://github.com/s3fs-fuse/s3fs-fuse), you should expect worse performance.

if a file is missing, you should as cmsweb team to check if the file sync from s3 to vocms0755 is working.