SAP HANA & Oracle: RHEL 8.

10 In-Place Upgrade + Cluster Migration

Runbook

1. Purpose

This runbook provides a comprehensive, step-by-step guide for upgrading

from RHEL 7 to RHEL 8.10. It details the process of replacing Veritas Cluster
with Pacemaker while ensuring the integrity of SAP HANA System Replication
and preparing all application and database systems, including SAP, Oracle,
and HANA.

2. High-Level Sequence

1. Prepare Systems & Take Backup

2. Upgrade Node B (Secondary)

o OS: RHEL 7 ➔ RHEL 8.10

o Veritas ➔ Pacemaker

3. HSR Takeover to Node B

4. Upgrade Node A (now Secondary)

o OS Upgrade + Pacemaker

o Re-register in HSR

Verify Role States, Clusters, Synchronization

3. SAP Landscape System Types

 HANA DB Nodes

 Oracle DB Nodes

 Application Servers (SAP NetWeaver)

4. RHEL 8.10 Base Configurations (All Servers)

These steps have to be performed across all the Servers ( HANA, Oracle,
Application Servers)

Repositories

Ensure the following repositories are enabled:

 afkl-rhel-8-for-x86_64-baseos-e4s-rpms

 afkl-rhel-8-for-x86_64-appstream-e4s-rpms
  afkl-rhel-8-for-x86_64-sap-netweaver-e4s-rpms

 afkl-rhel-8-for-x86_64-sap-solutions-e4s-rpms

yum --enablerepo=afkl-rhel-8-for-x86_64-sap-solutions-e4s-rpms,afkl-rhel-8-
for-x86_64-sap-netweaver-e4s-rpms

Install Mandatory Packages

yum install uuidd libnsl tcsh psmisc nfs-utils bind-utils

Additional Recommended Packages

yum install tuned-profiles-sap bc binutils elfutils-libelf glibc glibc-devel \

ksh libaio libaio-devel libXrender libX11 libXau libXi libXtst libgcc \

libnsl libstdc++ libstdc++-devel libxcb libibverbs make policycoreutils \

policycoreutils-python-utils smartmontools sysstat

Note: “No match” errors for a few packages (e.g., libaio-devel,

smartmontools) can be ignored.

Disable Firewall

To protect your SAP servers from unauthorized access, you can configure the
built in firewall of your RHEL system to only allow access via the ports which
the SAP software uses for communication. You can use the firewall-config
graphical tool or the command line tool firewall-config-cmd which are part of
the firewall-config package.

systemctl stop firewalld

systemctl disable firewalld

Enable and Start uuidd

The UUID generation daemon needs to be started now and at system boot.
Run the following commands:

systemctl enable uuidd

systemctl start uuidd

Configure tuned to use profile "sap"

SAP Usage Databa Recommended Package

Command
Type se Profile Required
SAP HANA (DB tuned-adm tuned-profiles-
HANA sap-hana
only or mixed) profile sap-hana sap-hana

SAP NetWeaver / tuned-adm tuned-profiles-

— sap-netweaver
PAS only profile sap sap

PAS + HANA on tuned-adm tuned-profiles-

HANA sap-hana
same node profile sap-hana sap-hana

PAS + Oracle on tuned-adm tuned-profiles-

Oracle sap-netweaver
same node profile sap sap

Oracle DB only (Manual tuning

Oracle (None specific) —
(no PAS) required)

For the Servers, where HANA is installed use below :

systemctl start tuned

systemctl enable tuned

tuned-adm profile sap-hana

tuned-adm active

For the non-HANA Servers, use below:

systemctl start tuned

systemctl enable tuned

tuned-adm profile sap

tuned-adm active

Configure Process Limits

These settings only become effective for new processes, so you have to
logout and login all users belonging to these groups and to restart all
processes running under those users.

Use the command "cat /proc/self/limits" to verify the correct setting for "Max
open files" and "Max processes".

echo "@sapsys hard nofile 1048576" > /etc/security/limits.d/99-

sap.conf

echo "@sapsys soft nofile 1048576" >> /etc/security/limits.d/99-

sap.conf
echo "@sapsys hard nproc unlimited" >> /etc/security/limits.d/99-
sap.conf

echo "@sapsys soft nproc unlimited" >> /etc/security/limits.d/99-

sap.conf

Configuring systemd-tmpfiles

In order to ensure that important lock files and sockets in /tmp will not be
deleted by systemd-tmpfiles, please add the file /etc/tmpfiles.d/sap.conf with
the following contents to all RHEL 8 systems running SAP applications:

# systemd.tmpfiles exclude file for SAP

# SAP software stores some important files in /tmp which should not be
deleted automatically

echo "x /tmp/.sap\*" > /etc/tmpfiles.d/sap.conf

echo "x /tmp/.hdb\*lock" >> /etc/tmpfiles.d/sap.conf

echo "x /tmp/.trex\*lock" >> /etc/tmpfiles.d/sap.conf

5. SAP HANA Nodes Specific

These steps are exclusive for HANA servers ONLY

HANA-Specific Packages

yum --enablerepo=afkl-rhel-8-for-x86_64-sap-solutions-e4s-rpms,afkl-rhel-8-
for-x86_64-sap-netweaver-e4s-rpms install expect gtk2 krb5-workstation
krb5-libs libaio libatomic libcanberra-gtk2 libibverbs libicu libtool-ltdl numactl
openssl PackageKit-gtk3-module rsyslog sudo xfsprogs xorg-x11-xauth

HANA SPS Compatibility Package

For SPS05:

yum --enablerepo=afkl-rhel-8-for-x86_64-sap-solutions-e4s-rpms,afkl-rhel-8-
for-x86_64-sap-netweaver-e4s-rpms install compat-sap-c++-9

For SPS07:

yum --enablerepo=afkl-rhel-8-for-x86_64-sap-solutions-e4s-rpms,afkl-rhel-8-
for-x86_64-sap-netweaver-e4s-rpms install compat-sap-c++-11

SELinux configuration

Red Hat Enterprise Linux uses SELinux technology which is enabled by

default. Since there is no SELinux policy available for SAP HANA, leaving
SELinux enabled can lead to problems when running SAP HANA on RHEL. It is
therefore generally recommended to disable SELinux.

For checking the SELinux mode, use the getenforce command, as in the
following example:

# getenforce
Enforcing

Changing the SELinux mode from "Enforcing" to "Disabled" immediately on a

running system is not possible.

For setting the SELinux mode to "Disabled" permanently, run the following
command and reboot the server:

# sed -i 's/\(SELINUX=enforcing\|SELINUX=permissive\)/SELINUX=disabled/g'
/etc/selinux/config

Configure tuned to use profile "sap-hana"

The tuned profile "sap-hana", which is provided by Red Hat as part of the
RHEL for SAP Solutions subscription, contains many of the settings
mentioned below and configures some additional settings. Therefore the
"sap-hana" tuned profile must be activated on all systems running SAP
HANA.

Use the following commands to install and activate the tuned profile "sap-
hana" and check if it is active:

yum --enablerepo=afkl-rhel-8-for-x86_64-sap-solutions-e4s-rpms,afkl-rhel-8-
for-x86_64-sap-netweaver-e4s-rpms install tuned-profiles-sap-hana

systemctl start tuned

systemctl enable tuned

tuned-adm profile sap-hana

tuned-adm active

The output of "tuned-adm active" should look like:

Current active profile: sap-hana

Disable ABRT, Core Dumps, and kdump

All crashes of SAP HANA are typically analyzed by SAP support, so they do
not rely on operating system mechanisms for crash reporting. To avoid
delays when a Linux kernel crash or a core dump occurs, disable the
application crash and core file handling of the operating system.

The Automatic Bug Reporting Tool (ABRT), which handles application

crashes, is not installed by default when installing RHEL 8 with the Server
environment group. The corresponding packages are (for example) abrt and
abrt-addon-ccpp. If installed, they can be disabled with the following
commands:

systemctl stop abrtd abrt-ccpp kdump numad

systemctl disable abrtd abrt-ccpp kdump numad

Disable core file creation for all users

To disable core file creation for all users, open or create file
/etc/security/limits.d/99-sap.conf and add the following lines:

echo "soft core 0" >> /etc/security/limits.d/99-sap.conf

echo "hard core 0" >> /etc/security/limits.d/99-sap.conf

Disabling kernel crash dump facility (kdump)

The kernel crash dump facility (kdump) can be disabled with the following
commands:

systemctl stop kdump

systemctl disable kdump

Turn off auto-numa balancing

In RHEL 8, auto-numa balancing is switched off by default. To verify, use the

following command:

# sysctl kernel.numa_balancing

kernel.numa_balancing = 0

If package numad is installed on your server, please make sure that it is

switched off and disabled:

# Commands are executed even if numad is not enabled

systemctl stop numad

systemctl disable numad

Disable Transparent Hugepages (THP)

If you use the "sap-hana" tuned profile as described above, THP will be
disabled via the profile and you don’t need to change this setting manually.

To verify that THP have been disabled, use the following command:

cat /sys/kernel/mm/transparent_hugepage/enabled

The expected output will be as follows :

[temp-root@kl14583n:/home/b103717]# cat
/sys/kernel/mm/transparent_hugepage/enabled

always madvise [never]

[temp-root@kl14583n:/home/b103717]#

# If you use the "sap-hana" tuned profile as described above, THP will be
disabled via the profile, and you don’t need to change this setting manually.

# If you do not use the "sap-hana" tuned profile, to disable the usage of
transparent hugepages during runtime,

# set the kernel settings at runtime with the command:

# echo never > /sys/kernel/mm/transparent_hugepage/enabled

# There is no need to shut down the database to apply this setting. To make
this change persist across system reboots,

# it is necessary to edit the OS bootloader configuration as follows:

# Modify the file /etc/default/grub and append the following parameter to the
line starting with “GRUB_CMDLINE_LINUX”:

# transparent_hugepage=never

# Then rebuild the GRUB2 configuration as follows:

# On a system without UEFI, activate this option by issuing:

# grub2-mkconfig -o /boot/grub2/grub.cfg

# On a system with UEFI, activate this option by issuing:

#

# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

# To verify that THP has been disabled, use the following command:

# cat /sys/kernel/mm/transparent_hugepage/enabled

# always madvise [never]

cat /sys/kernel/mm/transparent_hugepage/enabled

Configure CPU Governor for performance (Intel 64 platform only)

No action required if the "sap-hana" tuned profile is used.

Configure Network Stack for HANA

echo "net.core.somaxconn=4096" > /etc/sysctl.d/sap-hana.conf

echo "net.ipv4.tcp_max_syn_backlog=8192" >> /etc/sysctl.d/sap-hana.conf

echo "net.ipv4.tcp_slow_start_after_idle=0" >> /etc/sysctl.d/sap-hana.conf

echo "net.core.wmem_max=4194304" >> /etc/sysctl.d/sap-hana.conf

echo "net.core.rmem_max=6291456" >> /etc/sysctl.d/sap-hana.conf

echo "net.ipv4.tcp_timestamps=1" >> /etc/sysctl.d/sap-hana.conf

sysctl --system

Configure Energy Performance Bias (EPB, Intel 64 platform only)

1. Activate the 'sap-hana' tuned profile as described above which will set
EPB to 'performance

# Verify:

cpupower info

If the command reports 'perf-bias: 0', EPB has been set to the correct value.

Disable Kernel samepage merging (KSM)

The KSM feature helps reduce physical memory overhead by detecting
memory pages with identical content. The feature is useful for VMs, but the
space-time tradeoff does not pay off for SAP HANA instances not running in
VMs.

The following command can be used to show the status of KSM:

# cat /sys/kernel/mm/ksm/run

The result of "0" confirms that KSM is deactivated.

If KSM is enabled, we recommend to disable it by adding the following

command to a script executed on system boot, such as /etc/init.d/boot.local:

# echo 0 > /sys/kernel/mm/ksm/run

Execute the command to turn off KSM immediately.

Oracle DB Nodes Specific

Configuring Process Resource Limits

echo "@dba hard nproc unlimited" >> /etc/security/limits.d/99-

sap.conf

echo "@dba soft nproc unlimited" >> /etc/security/limits.d/99-

sap.conf

echo "@dba hard nofile 1048576" >> /etc/security/limits.d/99-

sap.conf

echo "@dba soft nofile 1048576" >> /etc/security/limits.d/99-sap.conf

echo "@sdba hard nofile 1048576" >> /etc/security/limits.d/99-

sap.conf

echo "@sdba soft nofile 1048576" >> /etc/security/limits.d/99-

sap.conf

HugePages (Oracle)

echo "@dba hard memlock unlimited" >> /etc/security/limits.d/99-

sap.conf
echo "@dba soft memlock unlimited" >> /etc/security/limits.d/99-
sap.conf

Kernel Parameters

echo "fs.aio-max-nr = 1048576" > /etc/sysctl.d/97-oracle-database-

sysctl.conf

echo "fs.file-max = 6815744" >> /etc/sysctl.d/97-oracle-database-sysctl.conf

echo "kernel.shmmni = 4096" >> /etc/sysctl.d/97-oracle-database-

sysctl.conf

echo "kernel.sem = 1250 256000 100 1024" >> /etc/sysctl.d/97-oracle-

database-sysctl.conf

echo "net.ipv4.ip_local_port_range = 9000 65500" >> /etc/sysctl.d/97-

oracle-database-sysctl.conf

echo "net.core.rmem_default = 262144" >> /etc/sysctl.d/97-oracle-

database-sysctl.conf

echo "net.core.rmem_max = 4194304" >> /etc/sysctl.d/97-oracle-database-

sysctl.conf

echo "net.core.wmem_default = 262144" >> /etc/sysctl.d/97-oracle-

database-sysctl.conf

echo "net.core.wmem_max = 1048576" >> /etc/sysctl.d/97-oracle-database-

sysctl.conf

sysctl --system

sysctl --system (Loads and applies all the configuration files)

HSR Failover and OS Upgrade Procedure (SAP HANA)

Secondary Responsibl
Primary (Node A) Remarks
(Node B) e Team

Trigger full DB +
config backup
Backup HANA DB and
before any role
Configuration
switch or
unregistering HSR
python python Verify initial
SAPOPS
systemReplication systemReplication replication config

Freeze Cluster Core Infra

Disable AutoStart of
SAPOPS
HANA

Stops HANA on
secondary;
HDB stop SAPOPS
replication state is
preserved

Confirm replication
hdbnsutil -sr_state SAPOPS
stopped cleanly

Monitor log segment

SELECT * FROM
SAPOPS usage on Primary to
M_LOG_SEGMENTS;
prevent overflow

Upgrade Node B to
OS Upgrade Core Infra
RHEL 8.10

Clean up Veritas
Remove Veritas Core Infra
components

Platform team
Install Pacemaker Core Infra
action

Install required Validate

packages for SAPOPS prerequisites for
HANA SAP on RHEL 8.10

Adjust OS
Kernel parameters,
parameters for SAPOPS
limits, etc.
HANA

Restart VM /
Reboot for patching
hardware if Core Infra
or kernel
needed

Bring up HANA on
HDB start SAPOPS
Node B

hdbnsutil -sr_state SAPOPS Ensure HANA is

 healthy and not yet
in replication

Prepare Node A for

HDB stop SAPOPS
role switch

hdbnsutil - Promote Node B to

SAPOPS
sr_takeover Primary

Confirm Node B is
hdbnsutil -sr_state SAPOPS
now Primary

hdbnsutil -sr_register --
name=SiteA
--remoteHost=hana-
nodeB.prod.local Register Node A as
--remoteInstance=00 SAPOPS Secondary before
--replicationMode=sync upgrade
--
operationMode=logrepl
ay

Start HANA on Node

HDB start SAPOPS
A — now Secondary

Confirm replication
hdbnsutil -sr_state hdbnsutil -sr_state SAPOPS
is working B → A

Prepare Node A for

HDB stop SAPOPS
upgrade

Upgrade Node A to
OS Upgrade Core Infra
RHEL 8.10

Platform cleans up
Remove Veritas Core Infra
cluster stack

Platform installs
Install Pacemaker Core Infra
new cluster stack

Install required Match with Node B

SAPOPS
packages for HANA setup

Adjust OS parameters SAPOPS Kernel params, etc.

for HANA

Restart VM / hardware if
Core Infra Post-patching reboot
needed

Node A remains
HDB start SAPOPS
Secondary

Confirm full sync

hdbnsutil -sr_state hdbnsutil -sr_state SAPOPS between both
upgraded nodes

Use Pacemaker to
Core Infra /
test failover
SAPOPS
scenarios

7. Final Notes

 Ensure /etc/hosts, hostnamectl, and DNS are properly configured.

 Validate all OS-level parameters using sysctl -a, ulimit -a, and SAP
HANA checks.

 Utilize Ansible wherever indicated (GREEN in source), especially for

repeatable configurations.

 Confirm that SAP Host Agent and UUID services are operational.

References

 SAP Note 2772999

 SAP Note 2382421

 SAP Note 3057467

 SAP Note 2777782

 SAP Note 1672954