System Monitoring

The following information gives a brief idea about the daily monitoring tasks which are
handled by the teams. This document would give a generalized idea about the monitoring
transactions and the brief description of the check.

The Transactions may vary among different clients, and has to adapted as per their
requirements. The transaction which are given below are based on the 46C and 47

A Daily, Weekly, Monthly and a Quick checklist is included in this document to assist in
performing the system administration activities.

The specific areas covered in this document are described below.

SAP Administration.

- Spool management
- System logs
- Batch management
- User administration
- Lock entries
- Work process monitoring
- System alert monitoring
- Performance monitoring
- Update monitoring
- SAP Buffer Quality

SAP R/3 Database administration.

- Storage
- DB alerts
- Tablespace
- Reorganizations
- Redo logs
- Archive logs
- Critical objects
- Indexes

Comments:
 Try for the sample reports
 Root cause analysis
 SAP ADMINISTRATION

A) Quick checklist

Transaction / Menu
Path Description Check for

SM51
Tools >> Administration >> Overview of SAP servers, system
Monitoring >> work process 1. Missing Application Servers
System monitoring >> Servers and users login information

SM50
Tools >> Admin >> Monitoring 1. All work processes with a “running” or a “waiting”
>> Performance >> Process Overview status
Exceptions/Users >> Active 2. All unexplained process failures should be
Users >> Processes investigated.

1. Check for any job which is running for than 3000 ms,
SM66 Global Work Process Overview Copy and paste it in SM66
2. Also check out for jobs running more than 50,000ms
in Dialog and more than 1, 00,000 ms in Background

1. Check for Load Distribution, average Response Time,

SMLG Maintain Logon Groups and Quality Ratio.

RZ02
Tools >> Admin >> CCMS >> SAP System Monitor / Graphical 1. Graphics showing inactive work processes(bad
Control >> Monitor config) and red alerts
System monitoring

SM37
Tools >> Admin >> Jobs >>
Select Jobs Job Log Overview 1. Verify that all critical jobs were successful.
2. Check for jobs which have been released, but are
waiting on resources
2. Review any cancelled jobs.

DB02
Tools >> Admin >> Monitoring
>> Performance >> Storage Management
Database >> Tables/Indexes
>> Checks Select Table and Indexes. 1. High Extent number - runaways
….>> Tables/Indexes >>
Missing Indexes Missing Indexes. 2. High percentage > 95% used or any entries not online
….>> Tables/Indexes >> Any space critical objects demand immediate
Current Size Check Tablespace/Freespace. attention
….>> Tables/Indexes >>
Space Critical Objects Extents can be allocated to T/I.
B) Daily Checklist

Transaction / Menu
Path Description Check for

SM51
Overview of SAP servers,
Tools >> Administration >> system work process and
Monitoring >> users login information 1. Missing Application Servers
System monitoring >> Servers

Global Work Process 1. Check for any job which is running for than 3000 ms,
SM66 Overview Copy and paste it in SM66
2. Also check out for jobs running more than 50,000ms in
Dialog and more than 1, 00,000 ms in Background
SM21
Tools >> Administration >> System 1. Warnings, Errors, Messages, Abends, Database
Log R/3 System Log problems and any other different events

1. Check for Load Distribution, average Response Time,

SMLG Maintain Logon Groups and Quality Ratio.

1. Check for the users logged on each application

instances. Select only Active Instance and No. of Users
and paste it.
AL08 Current Active Users

RZ20
Tools >> Admin >> CCMS >>
Control >> Performance CCMS Monitor 1. Look for Red Alerts
Menu >> Alerts >> Global >> SAP
system 2. Acknowledge, Analyze, Correct and Document the alerts

RZ02
Tools >> Admin >> CCMS >> SAP System Monitor / 1. Graphics showing inactive work processes(bad config)
Control >> Graphical Monitor and red alerts
System monitoring

SM37
Tools >> Admin >> Jobs >> Select
Jobs Job Log Overview 1. Verify that all critical jobs were successful.
2. Check for jobs which have been released, but are
waiting on resources
2. Review any cancelled jobs.

SM13
Tools >> Admin >> Monitoring >> 1. Check for lines with "Err". Notify the user of the update
Update Update records status

SM50
Tools >> Admin >> Monitoring >>
Performance >> Process Overview 1. All work processes with a “running” or a “waiting” status
Exceptions/Users >> Active Users 2. All unexplained process failures should be investigated.
>> Processes

SP01
Tools >> CCMS >> Spool >>
Output Management Spool Request Overview 1. Look for spool jobs that have been “in process”
for over an hour.

ST03
Tools >> CCMS >> Performance
Menu >> Workload Analysis 1. Check overall performance.
Workload Analysis 2. Check transactions with highest response time.
3. Check batch programs eating cycles - thus effecting
dialogs

1.Check for SCOT errors, If the number exceeds send

SCOT SAP Connect: Administration a report to the client

1.Check for Transactional RFC whose time limit is

SM58 Transactional RFC’s exceeded, or with error status

SMQ1, SMQ2 QRFC Monitor: Inbound 1.Check for any Inbound queues
QRFC Monitor: Outbound 2. Check for Outbound queues
ST02
Tools >> Admin >> Monitoring >>
Performance >> Buffer Analysis/Statistics 1. Check for low hit ratios
Setup/Buffers >> Buffers 2. Check for free space
3. Check for free directory entries
4. Swaps should be monitored

SM35
Tools >> Admin >> Monitoring >> 1. Check for "Jobs to be processed" which are there for a
Batch Input Batch Input Log long time
2. Check jobs in error

SM12
Tools >> Admin >> Monitoring >> 1. Check long standing locks or locks mentioned in
Lock entries Lock Entries Exclusive lock wait
Enter an asterisk (*) for user ID.

ST22
Tools >> Utilities >> Test >> Dump
Analysis ABAP Dump Analysis 1. Look for dumps which are excessive in number
2. Look for dumps of an unusual nature.

SM04
Tools >> Admin >> Monitoring >> 1. Check users that are logged on. Look for unauthorized
Performance >> User Overview users.
Exception/Users >> Active Users
>> Users Local

Database
Tasks Transaction - Check for
Description
DB12
Tools >> Admin >> CCMS >> DB
Admin >> SAPDBA Logs 1. Check that daily backups are executed without errors
Backup logs 2. Check database backup runtime
3. Check free space in "saparch" log directory

AL02
Tools >> Admin >> CCMS >>
Control >> Performance Check for database alerts 1. Check for any alerts
Menu >> Alerts >> Global >> 2. Acknowledge, Analyze, Correct and Document the
Database System alerts

ST04
Tools >> Admin >> Monitoring >>
Performance >> Review log for problems 1. Look for Buffer Quality < 93%
Database >> Activity 2. DD Cache quality < 85%
…>> Activity >> Detail Analyses
menu >> DB msg log DB message logs 3. SQL get and pin ratio < 90%
…>> Activity >> Detail Analyses
menu >> FileSys req. File system 4. Look for current alerts in the log
5. Check I/O activity

RZ08/AL01 Alert Monitor Red graphic for alerts

Tasks Transaction - Check for

Description
DB01
Tools >> Admin >> Monitoring >> 1. Check for Exclusive lock waits. If existing, refresh
Performance >> Exclusive Locks and find the person
Database >> Exclusive Lock Waits Holding the lock and delete after confirmation.

DB02
Tools >> Admin >> Monitoring >>
Performance >> Storage Management
Database >> Tables/Indexes >>
Checks Select Table and Indexes. 1. High Extent number - runaways
….>> Tables/Indexes >> Missing
Indexes Missing Indexes. 2. High percentage > 95% used or any entries not online
….>> Tables/Indexes >> Current Check Tablespace
Size Freespace. Any space critical objects demand immediate attention
….>> Tables/Indexes >> Space Extents can be allocated to
Critical Objects T/I.

Operating System
Tasks Transaction - Check for
 Description

AL16
Tools >> Admin >> CCMS >>
Control >> Performance OS Alert Monitor 1. Look for Alerts
Menu >> Alerts >> Local >>
Operating System 2. Acknowledge, Analyze, Correct and Document the alerts

OS06
Tools >> Admin >> CCMS >>
Control >> Performance OS Monitor 1. Verify CPU utilization.
>> Operating System >> Local >>
Activity 2. Check for Load Average > 3
3. Check for System Utilization > 15%

SCHEDULE DAILY
Job Name Description Frequency
SAP_REORG_JOBS-RSBTCDEL Delete old Background Jobs 1
(Must create variant)
SAP_REORG_SPOOL-RSPO0041 Delete old Spool requests 1
(Must create variant)
Delete old Batch input
SAP_REORG_BATCHINPUT- sessions 1
RSBDCREO (Must create variant)
SAP_REORG_ABAPDUMPS- Delete old dumps 1
RSSNAPDNL (Must create variant)
SAP_COLLECTOR_FOR_JOBSTA Generate run-time statistics
TISTIC for 1
RSBPCOLL background jobs
SAP_COLLECTOR_FOR_PERFM
ONITOR Collect system performance 24
RSCOLL00 statistics
 Critical Task
 Task Transactio Procedure Check
n- off/
Description initial
Check that the R/3 Log on to the R/3 System
System is up.
Check that daily DB12 – Check database backup
backups executed Backup Logs: Database backup run time
Overview
without errors.
Check operating system level
backup
Operating system backup run time

R/3
Task  Transacti Procedure Check
on - off/
Description initial
Check that all application SM51 – SAP Check that all servers are up.
servers are up. Servers
Check the CCMS alert RZ20– CCMS Look for alerts.
monitor (4.0+). Monitor (4.0)
Check work processes SM50 – All work processes with a “running”
(started from SM51). Process or a “waiting” status
Overview
Look for any failed SM13 –  Set date to one year ago
updates (update Update
Records
 Enter * in the user ID
terminates).  Set to “all” updates
Check for lines with “Err.”
Task Transactio Procedure Check
n– off/
Descriptio initial
n
Check System Log SM21- Set date and time to before the last
System Log log review.
Check for:
 Errors
 Warnings
 Security messages
 Abends
 Database problems
 Any other different event
Review for cancelled SM37 – Enter * in User ID
and critical jobs Select Verify that all critical jobs were
Background successful.
jobs
Review any cancelled jobs.
RZ02 – Same as for SM37
Graphical
Job
Monitor
Check for “old” locks SM12 – Enter an asterisk (*) for user ID.
Lock Entry
List
Check for entries for prior days.
Check users on system SM04 – Review for an unknown or different
Users user ID and terminal. This task
AL08 – should be done several times a day.
Users
Check for spool SP01 – Spool: Look for spool jobs that have been
problems Request “in process” for over an hour.
Screen
Check job log SM35 – Check for:
Batch  Jobs to be processed
input:  Jobs in error
Initial
Screen
Check work processes SM50/51–
Processes
R/3 (contd.)
Task Transactio Procedure Check
n– off/ initial
Description
Review and resolve ST22 - Look for an excessive number of
dumps. ABAP dumps.
Dump Look for dumps of an unusual
Analysis nature.
Review workload ST03 –
statistics. Workload:
Analysis of
<sid>
Review buffer ST02 – Tune Look for swaps.
statistics. Summary

Database
Task Where Procedure Check
off/ initial
Review error log for AL02 –
problems. Database
alert
ST04 – DB
Performanc
e Analysis:
Oracle

Operating System
Task Where Procedure Check
off/ initial
Review UNIX system AL16 – OS
logs for problems. Alerts
OS06 – OS Review operating system log
Monitor
Review NT system logs for NT system Look for any errors or failures.
problems. log
NT security Check for failed logon attempts to
log the SAP servers.
Look for errors or failures.
NT Look for errors or failures.
application
log
 Other
Task Where Procedure  Check
off/ initial
Check the UPS Review for:
uninterruptible power program log  Events
source (UPS).  UPS self test
 Errors

C) Weekly Checklist

Transaction Description Check for

Application
ST07 Monitor 1. Look for abnormal statistical data

Performance 1. Check for any modifications the

STUN Monitoring profile
Tools >> Admin >> CCMS >>
Control >> Performance 2. Check for any recent date
Menu changes
>> Operating System >>
Local >> Parameter Changes
……>> Operating System >>
Remote >> Parameter
Changes
……>> Database >>
Parameter Changes

1. Check extended memory and

RZ03 Memory Utilization verify if any
Tools >> Admin >> CCMS >>
Control >> Control Panel >> restarts are occuring over or
Edit abnormally
>> Views >> Memory
Management >> Absolute
Values high memory utilization

Check
environment -
SM65 background 1. Look for any error messages
Tools >> Admin >> CCMS >> processing
Jobs >> Check Environment

DB02– DB Performance: Tables Check database for Record free space.

free space
D) Monthly Checklist

Task Transaction Procedure

Defragment the memory N/A Cycle the R/3 System

Transaction Description Check for

DB02—Database Plot database growth Record usage and plot
Performance: Tables
Check size of database High percentage used or not online

Consistency check of database Look for any entires in the report

tables and indexes.

Consistency check of database Look for any entires in the report

and SAP dictionary.

Consistency check of SAP kernel Look for any entires in the report
to dictionary.

Check reorg need for tables and Any entry with a number of extents
indexes nearing the max extents number.

Check reorg need for Tablespaces Any entry with a number of extents
nearing the max extents number.
Check space usage as relates to
extents Any entries with a high % used

 Task Where Procedure

Backup file server Perform a full server backup
Review file system usage Record file system usage. Plot usage.

Is additional storage space needed?

Is “house cleaning” needed?

Check for the free space in the file usr\sap\trans

system
 OtherTask Where Procedure
Check consumable supplies Spare tape cleaning cartridge available
for all tape drives.
 DAT
 DLT
Spare tape cartridges available for all
drive types.
 DAT
 DLT
Spare data cartridges available for
removable media devices:
 Zip®
 MO (Magneto-Optical)
 CD (Recordable)
Preprinted forms:
 Shipping documents
 Invoices
 Checks
Special supplies, such as magnetic
toner cartridge.
Normal supplies:
 Laser printer toner
 Paper (for printers)
 Batteries
 Diskette
 Pens, and so on
E) Quarterly Checklist

The R/3 System

Task Transacti Procedure
 Chec
on
k off/
initial
Archive quarterly Send quarter-end backup tapes to
backup long-term offsite storage.
Security review SU01— Review user ID for terminated users
User that should be locked or deleted.
Maintena
nce
SM31Tabl Review list of “prohibited”
e passwords (Table USR40).
Maintena
nce
RZ10— Review system profile parameters
Edit for password standards.
System
Profile
Review scheduled SM37— Review all scheduled jobs to
jobs Backgrou determine if they are still
nd Jobs appropriate.

Database
Task Where  Procedure Check
off/
initial
Archive quarterly Send quarter-end backup tape to
backup long-term offsite storage.
Review all scheduled Review all scheduled jobs to
jobs determine if they are still
appropriate.
Test database Restore database to a test server.
recovery process
Test the restored database.

Operating System
Task Where Procedure Check
off/
initial
Archive quarterly Send quarter-end backup tape
backup to long-term offsite storage.
Archive old transport Transport Archive the old transport files.
files directories; log,
 data, cofiles
Cleanup SAPDBA SAPDBA – Maintain init<SID>.dba
logs cleanup

Other
Task Where Procedure Check-
off/
initial
Check maintenance Check for expiration date.
contacts
Check for usage changes.
 DATABASE ADMINISTRATION

A) SUMMARY OF SAP MONITORING TASKS OF THE DATABASE

This provides a summary of requirements for monitoring the SAP database system on a regular
basis. This table could be updated with comments on a daily basis to record any actions taken to
resolve any issues encountered on monitoring the system.

Title Tran. Description Comments

Tablespace Overflow DB02 Check ORACLE Tablespace statistics.
Are there any which may overflow in
the next few days?
Maxetents on DB02 Check ORACLE Tablespace statistics.
Tables/Indexes Are any tables or indexes near the
maxextents limit?
Database Growth / DB02 Monitor ORACLE Database growth for
Capacity Planning capacity planning.
Is there sufficient free disk capacity?
DDIC / Oracle DB02 Consistency Check of database and
Inconsistencies ABAP Dictionary; check for missing
tables and indexes.
Oracle Exclusive DB01 Monitor for Exclusive Lockwaits at the
Lockwaits ORACLE level.

Oracle Logs ST04 Check ORACLE logs for Errors and

Alerts.

Database Backup DB12 Check your database backup. Was the

backup completed successfully? Is
todays backup scheduled? Also check at
UNIX and Omniback level.
Archive Directory DB12 Check for freespace in the ORACLE
Freespace Log directory /oracle/<SID>/saparch. If
this directory reaches 100%, the
ORACLE Database will stop.
Oracle parameters – DB03 Check for ORACLE Database
Changed by Parameter changes. Confirm that
authorised user profiles are adjusted only by authorized
users, and that correct values have been
change.
CCMS Alert Monitors AL02 Monitor the CCMS, Database Alert
Monitors. Alerts should be analyzed,
corrected, acknowledged and
documented.
 B) Database Monitoring

This section provides a summary of monitoring requirements on the ORACLE Database:

1) Database Growth Summary

 Use the performance monitor to check on database growth. Which tablespaces are growing
fastest? Which have the least free space? Which are getting the most I/O activity? When will
new disks have to be purchased and configured to the system?
 Use sapdba to extend a tablespace that needs extending. Backup that tablespace as suggested
by sapdba.

Extents Summary:
 Use the performance monitor and sapdba to check on table and index growth. Which tables and
indexes are growing fastest? Do these have appropriate next extent sizes?
 Use sapdba interactively to increase next extent sizes.
 In the worst case scenario, reorganize a table to reduce extents.

Index Summary:
 Use sapdba or the performance monitor to check on fragmentation in your indexes. Use the
performance monitor to see which tables are getting the most use (where an index recreate thus
has a chance of helping performance).
 Use CCMS to schedule regular analysis of indexes.
 Use sapdba to recreate an index.
 Check via DB02 if there are any missing Indexes.

2) TABLESPACE OVERFLOW
If the database system cannot assign any more extents in a tablespace (ORACLE error 1650,
1652-1655) due to lack of free space, you have to extend the tablespace, by adding a new data
file. Use SAP utility SAPDBA. The database does not have to be closed. By extending the
tablespace, additional freespace exists for further tasks.
Monitor Space Statistics for Tablespaces approaching 90% in size. They will have to be increased
as soon as possible using utility sapdba to prevent them becoming full. This will avoid numerous
Transaction Abends.

Use Transaction DB02 (Menu Path: Tools -> Administration -> Monitor -> Performance ->
Database -> Tables/Indexes). Select Current Sizes from the Tablespaces section. A list of the
tablespaces will be displayed on your screen. Position your cursor on the "Used" column, and then
click on the Sort button or keys (Sht-F1) to sort the list by % used.
You can also use the Database Alert Monitor to analyze freespace. Use Freespace Management to
check whether the Alert Monitor is displaying freespace problems. To start the Database Alert
Monitor, use Transaction AL02, (Menu Path: Tools -> Administration -> Monitor ->
Performance -> Alerts-> Global -> Database system).

On the initial SAP Performance Monitor screen, double-click on Freespace Management Checked
line.

Note 1: Also take into account that Tablespaces less than 90% utilized can fill up if a Table or
Index within the Tablespace has a Next Extent size that is larger than the available freespace on the
tablespace. The Maximum Next Extent Size for tables and indexes can be identified via DB02 ->
Space Statistics (in the Tablespaces section.) Confirm that the Maximum Next Extent Size is less
than the Maximum Free Space.

Note 2: When creating an Index in Production via a transport Change Request, ensure there is
sufficient freespace on the relevant Tablespace in production to enable creation of a new Index.

When identifying if a tablespace needs to be extended, and distinguish between two cases:
 The tablespace is highly fragmented.
If the tablespace has sufficient free storage space, it is probably a fragmentation of free
storage space; which causes the storage problems. This will occur only if DROP operations
are performed frequently. Database objects that are frequently deleted (dropped) and
created anew should be separated.
You can solve the storage problems by reorganizing the fragmented tablespace or the
fragmented tables and indexes which it contains. An extension of the tablespace would then
not be necessary.
Note: A reorganization is time-consuming and complicated, however. SAP therefore
recommends that you extend the tablespace rather than reorganizing it. You should
reorganize this tablespace only if one or more of the following conditions apply:
 The maximum number of data files for your system has been reached
 An extremely large number of extents has been assigned to very many objects in the
corresponding tablespace
 Accessing the data in the tablespace causes performance problems
 The tablespace has too little free storage space.
The full tablespace must always be extended.
If an extension of a tablespace causes your database to approach the limit value determined
by the operating system for the number of files, it can be useful to reorganize this tablespace
after an extension or to extend the tablespace during the reorganization. However, a
reorganization is not very likely necessary because the limit value for the data files is very
high (at least 254 files are available).
3) EXTENT OVERFLOW
Monitor Space Statistics for Tables and Indexes for extents over 100.
Use Transaction DB02 -> Tables and Indexes section -> Space Statistics -> Select All
Tables/Indices forAll Tablespaces. You can then sort on Total Extents.

If the database system cannot assign any more extents to a Table or Index (ORACLE error 1628-
1632) due to the Maximum number of extents on the Table or Index being reached, you have to
increase the storage parameter MAXEXTENTS. In addition, increase the value for NEXT such that
the number of extents does not increase too quickly, depending on the expected new data volume.
Use SAP utility SAPDBA. The database does not have to be closed. By increasing the
MAXEXTENTS and the NEXT size for the Extent, the table or Index is able to extend.

To prevent tables and indexes from approaching the maximum number of extents too fast, SAP
recommend that you run the command sapdba -next <tablespace/s> once a week. The tool
sapdba -next automatically adjusts the NEXT parameter of each table and index if necessary, ,
using an algorithm within SAP. This results in fewer but larger extents.

If Maxextents are reached, you must either adjust the maximum number of extents or reorganize
the table. Maxextents can be increased using sapdba.

4) DATABASE GROWTH

For Disk Capacity Planning purposes, monitor Space Statistics for Database System growth over a
period of time for increase in Space usage. This will enable a pro-active approach to try and gauge
when new disks are likely to be required for future expansion. This additional disk capacity can
then be purchased and installed in sufficient time, to enable ORACLE Tablespaces to be extended
as required.

To identify the Database size, use Transaction DB02 -> Database System section -> Space
Statistics.

From the Database History screen, you can monitor growth on a Monthly, Weekly or Daily scale.
The rule-of-thumb is that the %-Used for the database should not exceed 65% of the database size.
Also take into account the amount of free disk capacity available on the server for extending the
database.
5) DDIC AND ORACLE DISCREPANCIES

To prevent potential performance degradation by increasing Data Access time.

If the index is no longer required, and has been removed from the ORACLE Database, then
remove it from the SAP Data Dictionary to keep both environments in synch.
If the Index is required, it can be activated from transaction SE14.

5.1 CHECK WHETHER INDEXES ARE MISSING

From the Database Performance Analysis screen, from Transaction DB02, select Missing
Indexes from theTables and indexes section.
If you discover missing indexes, you should find out whether they are primary or secondary
indexes. To do this, use the path Goto->Table & Indices->DB <-> ABAP Dict. Click the
Display button.check.
 Primary indexes ensure that line keys (row keys) are unique. Missing primary indexes
are therefore critical.
 If a primary index is missing, you should consult SAP to get help with restoring it.
There is no simple procedure for restoring the index if a large number of duplicate
keys were created in a table.
 Secondary indexes are used for special scans and are only relevant for performance.
 Restore a secondary index via Transaction SE14 (ABAP/4 Dictionary Database
Utility).
5.2 MISSING DATABASE OBJECTS
You can check whether objects (tables or indexes), defined in the ABAP/4 Dictionary are also
defined in the database. These checks should not show inconsistencies.
Transaction DB02 shows the number of objects missing from the database (Objects missing in
the database).
If the checks show that some tables are missing, first find out whether they are test objects,
which should not be in the database. This is the most probable explanation for an inconsistency.
5.3 MISSING ABAP/4 DICTIONARY OBJECTS
You can check whether objects, such as tables and secondary indexes, defined in the database
system are also defined in the ABAP/4 Dictionary.
Transaction DB02 shows the number of objects missing from the ABAP/4 Dictionary
(Unknown objects in ABAP Dictionary). To do this, use the path Goto->Table & Indices->DB
<-> ABAP Dict. Click the Display button.check.
6) EXCLUSIVE LOCKWAITS
Avoid Exclusive Lockwaits as this can degrade performance severely!
To identify Exclusive Lockwaits on the Database, use Transaction DB01 (Menu Path: Tools ->
Administration -> Monitor -> Performance -> Database -> Exclusive Lock waits).
Exclusive Lockwaits are effectively a deadlock situation between processes accessing the same
table.
These situations are normally caused by long processing of ABAP/4 or SQL statements after
acquiring the database lock. The lock on the database will be released after the dialog step of the
session holding the lock is completed (which is equivalent to a COMMIT), or cancelled (which is
equivalent to a ROLLBACK).
For example, someone taking an order puts a lock on available material, gets distracted, goes off to
lunch without confirming the sale. No other person will be able to take an order for that item until
the lock is released.
Typical action: find the person holding the lock if possible so it can be released. If this is not
possible, delete the lock so that others can work.
Another example: A long running update program updating a table may put an Exclusive Lock on
the table. This can be resolved by ensuring that the program does regular commits of it’s work.

If there are any Exclusive Lockwaits, the screen will display the following information:
1. Lock-Holder (Host, Pid)
The process that is the root cause of the Exclusive Lockwait. Server that the process is running
on and Process ID are displayed.
2. Object-Name
The Table that is effected by the Exclusive Lockwait.
3. Lock-Waiters (Host, Pid)
The Processes that are being held up due to this Exclusive Lockwait.

Analyze what is running, and what impact the Exclusive Lockwait is having on the system.
Transaction SM66 would be used to get an overview of all processes to identify more information
on the relevant process id, and SM37 to identify more information (program, etc.) if it is a batch
job.
Under most circumstances, it may be necessary to cancel the process causing the Exclusive
Lockwait after some analysis to ensure that it is not near completion, etc.
After canceling the process, a Development programmer should analyze the program to identify if
it can be changed to include commits.
7) ORACLE LOG
A very import ORACLE log is /oracle/<SID>/saptrace/background/alert_<SID>_log.
<SID> is the relevant three character SAP System ID (e.g. D01).
This Log can also be monitored from within SAP using Transaction ST04 (Menu Path: Tools ->
Administration -> Monitor -> Performance -> Database -> Activity).
From Transaction ST04, select Detailed Analysis Menu -> Database Message Log ->Only Alerts.
This log should be checked on a daily basis for any errors.

An online utility exists to provide detailed information on Oracle-generated errors. The utility is
called oerr, and the format for using the command is as follows:

oerr ora 1631

This utility may be used from either the orasid or sidadm user accounts. In the above example,
1631 represents the actual Oracle error number you want to obtain information on, and ora
represents the utility you want to obtain information for, namely Oracle. A sample of the output
generated from the oerr utility is shown below:

01631, 00000, "Max # extents '%s' reached in Table '%s'"

// *Cause: A Table tried to extend past max extent’.
// *Action: If maxextents is less than the system minimum raise it, otherwise you must
recreate with larger initial next or pctincrease parameters.
8) CHECKING BACKUP LOGS

This is arguably the most important monitor function. The backup logs should be monitored daily
using the tools used to backup the database. The following is only valid if
BRBACKUP/BRARCHIVE was used to backup the database and redo logs.
This monitor will display a history of the database backups (SAP BRBACKUP) and redo log
archiving (BRARCHIVE). To get to the main Backup monitor screen use the following menu
paths:

Transaction DB12 (Menu Path: Tools -> CCMS ->DB Administration -> Backup logs).

Use any of the push buttons to get more information on any of the topics.

8.1 Overview of the Database Backup Logs Display

From the main screen, you can see the date and time of the last database backup.

Now you can check the return code of the last DB backup by clicking on the Last successful DB
backup button, on the main screen. The return code is displayed in the upper right section of the
screen. If a value is displayed that is greater than 0 (zero) this indicates that the backup was
completed. However, there where warning and errors that occurred during the backup process.
You should investigate these errors immediately by displaying the complete log.
The last successful backup needs to be as current as possible and should never exceed more than
24 hours. Large gaps in backups can result in an increase in recovery time. All the redo logs will
need to be re-imported since the last backup. The fewer redo logs the shorter the recovery time.
You can also get an overview of all the backup logs by clicking on the Overview of database logs
button.
You can select any log and click the Choose log (F2) button to display the selected log.

8.2 Overview of the Redo Log Backup Logs Display

The main display also gives you the status of the redo log files you will need to re-import from
archive tapes. The redo logs that have not been archived are on-line and are available to the
system. You will need to import again the redo logs during a database recovery. For more detail
on the redo logs click the Overview of redo log backup logs button on the main display.
The screen shows you the operation and the redo log number along with the time and date of the
operation as well as the status (zero -- success)

8.3 Freespace in REDO log Archive Directory

Providing that the database is operating in ARCHIVELOG mode (which it must be for a
production system) the SAP Archived REDO Logs (*.dbf, which are stored in directory
/oracle/<sid>/saparch) must be backed up using the SAPDBA BRARCHIVE Utility on a daily
basis. These files are crucial to the recoverability of the database should a database failure occur.
Because these files are so important, SAP recommends that two copies of the archive logs be
saved to tape. You should also monitor the file system where the archive logs are stored to ensure
there is adequate space available to capture database activity. If directory /oracle/<sid>/saparch
reaches 100%, the SAP R/3 application will go into a 'wait' state until the contents of this directory
are copied to tape and deleted. This directory holds the archived online ORACLE REDO logs for
use in recovery situations. To resolve this situation, run brarchive procedure.

The file system can be monitored using the UNIX df command, alternatively:
Transaction DB12 (Menu Path: Tools -> CCMS ->DB Administration -> Backup logs).
Under Redo Log Backups, check figure on Free space in kB: field. Ideally, this should be over
100,000 KB. If not, schedule a run of brarchive.

Brarchive must be scheduled at least once a day, and possibly more depending on the amount of
data being written to /oracle/<sid>/saparch.
One option is to schedule the following script from the UNIX crontab on an hourly basis to verify
that the freespace in /oracle/<sid>/saparch is adequate:

8.4 ORACLE INIT.ORA PARAMETER FILE

On a periodic basis, check for ORACLE Database Parameter changes. Confirm that profiles are
only adjusted by authorized users, and that correct values have been changed. To check for this,
use Transaction DB03.
 Backup Strategy

A) Introduction.

Backup and recovery is a crucial component of any system administration plan . This document is intended
as a guide to developing a backup and recovery solution based on Pratt and Whitney’s requirements and
standards. It outlines a list of areas to consider to develop a complete backup solution. It also briefly
describes the most common backup solutions used in a SAP/ORACLE environments. The goal of this
document is to aid in implementing a backup solution to allow point in time recovery of the entire system.

The following is a list of areas to evaluate to implement a backup strategy.

 Decide how often to perform complete database backups

 Decide how often to perform UNIX backups.

 Decide and evaluate the need for 3rd party tools to perform and manage backups (e.g. EMC’s BCV
with Legato, Omniback, etc. backup tools)

 Disaster recovery solution

 Decide how often and when to perform database redo log archives

 Provide ample disk space to hold database redo log in archive directory going back to the last
backup

 Create a volume labeling scheme to ensure smooth operations

 Decide on backup retention period

 Decide on backup hardware based on needs for fast backups

 Determine tape pool size

 Plan for growth

 Determine physical tape storage strategy both off-site and on-site.

 Document backup procedures in operations manual

 Train operators in backup procedures

B) Overview

Here is a brief overview of hardware and software solutions which are commonly used in
SAP/ ORACLE environments and their benefits:

 In development / configuration and test systems the common strategy is to do full off-line backups
every night and then do the archive logfiles either on demand or scheduled once or more often during
the day.
 In normal production environments where you are operating on a 24 hour availability basis during the
week and a reduced schedule over the weekend, the strategy is to do a full off-line backup every
weekend, do full on-line backups during the week. The archive logfiles are either saved on a scheduled
or on a on-demand basis.

 In heavy used production environments the strategy is following: doing on-line backups all the time
during the week as well as during the weekends. Like in the previous solutions the archive logfiles are
either backed up on demand or scheduled.

Hardware supported strategies:

As a security procurement you should have your SAP database on a disk-level mirrored. This will eliminate
hardware failures of a single volume. Common environments are having double to triple up to quadruple
mirrors active.

 If you are having three or more disks within this mirror, you can stop the SAP system and the database
software, then break up the mirrors and start the SAP system again. You can then take an off-line or
level 0 backup of your disks. After the backup just rejoin the volumes into the mirrors again.

 Some hardware vendors offer the capability to mirror the complete disk array/cabinet even over some
distance. This means you have your production system in town A and a backup disk array with another
machine attached in town B, where the distance between is depending on the solution of the vendor
chosen. Having this solution installed, you can run off-line backups anytime and continuous on the
shadow system, you are not even impacting the CPU utilization of the master system.
C) Backup strategy worksheet

Determine Cost of Downtime and Lost Data

Downtime in the production system would cost my company roughly ___________ per day.

Losing one day of productive data would cost my company roughly ____________.

Complete and catastrophic loss of productive data would cost my company roughly __________.

1) Establish Backup Requirements

Database “Landscape”

The following table shows my planned configuration (test, consolidation and productive systems and their
expected “full” size).

In the following table, “Purpose” could be one of "Test", "Development", "Consolidation" or "Production",
rough size in GB is sufficient. (The 3 character SID is only to help organize your answers to this and the
following questions.)

SID (e.g. Purpose Size (for Redo log volume Host Name
C11) (for example, Test) example, 14 per day (for example, host01)
GB) (default 1 GB)
2) Availability Requirements
In the following table, please note your availability requirements for your databases. Note the times you
have available for “hard” down time (no end users can do any work on the system) and so-called “soft”
down time (the system is available for end users but may have noticeably degraded response time, for
example, due to an online backup being run).

SID (for Allowed Hard Down Time (for Allowed Soft Down Time (for
example example, one Sunday per month, or example, one Sunday per month, or
C11) daily from 10 PM to 4 am) daily from 10 PM to 4 am)

3) Allowed Backup Run Times

Permissible durations of backup runs for your various systems, based on the preceding availability table.
The table is separated by “Offline” and “Online”, and also by “Daily”, “Weekly” and “Monthly” times, in
case these are different.

SID (for Allowed Daily Backup Allowed Weekly Backup Allowed Monthly Backup
example, Time Time Time
C11) Offline Online Offline Online Offline Online
4) Allowed Restore Times
Permissible duration of a full restore followed by recovery of two days’ worth of redo logs.

SID (for Allowed time for full restore and recovery of two days’ worth of redo logs
example, C11)
5) Backup and Archive Automation
Whether backups and archives should be able to run without direct operator intervention (for example, to
manually start the backup or to change tapes during the backup).

SID (for example, Automatic (unattended) backup Automatic (unattended) archive required
C11) required
Yes No Yes No
   

Yes No Yes No
   

Yes No Yes No
   

Yes No Yes No
   

Yes No Yes No
   
6) Backup Technology
Request for Offers from Hardware Vendors

Template for communication with your hardware partner or other backup solution vendors for
establishing the backup strategy:

Database Host hardware Anticipated Size in GB Anticipated daily redo log volume
Name in GB

For these databases, the backup solution must fulfill certain requirements. These are:
 Maximum run time for a complete backup of the database
 Maximum restore and recover time for the database and two GB of redo logs.
 Complete backup must be possible without operator intervention (for example, to change
tapes).
 A complete archive must run in addition to the complete backup without operator
intervention (that is, we must be able to schedule a backup followed immediately by an
archive of the redo logs without an operator needing to change tapes). For enhanced redo log
safety, two copies of the redo logs are saved.

Database Maximum time Maximum time Complete backup In addition, the archive of the
Name allowed for a allowed for full must be possible redo logs must be possible
full backup in restore with a without operator without operator intervention
hours recovery of two GBs intervention (for (scheduled with the backup)
of redo logs example, to change
tapes)