This blog contains experience gained over the years of implementing (and de-implementing) large scale IT applications/software.

Basic Performance Tuning Guide – SAP NetWeaver 7.0 – Part II – Reading Single Record Statistics

This is Part II. You may like to see Part I.

In Part I, we showed you how to trace an ABAP program using STAD to show the single record statistics of the response times for a transaction/program.
In this part, we will go through and decipher the statistics to help you decide where the problem may lie in your slow performing program.

Following Part I, we saw the STAD screen output.  Below is a snapshot of the right hand side of the screen showing the “standard” view fields:

STAD output fields

The “Response time (ms)” field shows the total response time for the dialog step (or summary of steps if you have chosen not to expand the left tree).

As a reminder, SAP says (https://help.sap.com/saphelp_nw70ehp1/helpdata/en/21/2c8f38c7215428e10000009b38f8cf/content.htm
):
Response time = wait time + execution time
where: execution time =
Generation time during the run
+ Load time for programs, screens, and graphical user interfaces
+ Roll times for rolling in the work data
+ ABAP processing time
+ Database time
+ Enqueue time for logical SAP lock processes
+ CPIC/RFC time
+ Roll wait time (excluding task types RFC/CPIC/ALE).

So if we take a look at our initial screen of values, the example below shows that the “DB req time (ms)” is one of the larger values (4929):

SAP STAD DB request time of response time

The total database request time (time taken for database work to be done for this step) is more than 92% of the total response time for the step.
If the database time IS NOT a significant percentage of total response time, then you should check the other parts that make up the total response time, for clues.

At this point, it’s better to change the screen columns so that you can see all the timings.
The best view is the “Time Analysis” view.  Switch views by clicking the “Sel. fields” button:

STAD select fields

Click “Time Analysis“:

STAD time analysis

You will now see more timing fields

STAD timing fields, response time
STAD timing fields, response time

The fields highlighted are responsible for the total response time calculation.

In our example, TOTAL RESPONSE TIME = Wait Time + DB Req Time + Processing Time + Load Time + Generating Time + Roll In Time + Roll Wait Time + Enqueue Time.
=> 5330 = 0 + 4929 + 122 + 17 + 0 + 262 + 0
(5330 / 1000 = 5.3 seconds total response time)

You should check each of the fields, looking for disproportionate response times.

The “Wait time (ms)” indicates the amount of time the dialog step was waiting to be processed by a dialog work process.  High values indicate that the overall system performance is degraded.  Maybe a lack of memory, CPU or I/O performance at the O/S layer could be causing issues.  Check the dispatcher queue.

The “Time in WPs (ms)” indicates the time that the dialog step was being actively serviced by a work process and not by any other type of process (BTC, UPD, UPD2 etc). Check “SM51 -> Goto -> Server Name -> Information -> Queue Information” and SM66 to see how busy the system is.

The “Processing time (ms)” indicates the amount of time the individual components spent processing (working) and not idle or sleeping or waiting.  Check SM50 and SM66 to see how busy the system is.
Use ST06 (O/S monitor) to check CPU usage and system load.

The “Load time (ms)” indicates the amount of time taken to load the program code into memory buffers.
A problem here could indicate the program buffer is not large enough (see ST02).  You should check the CPU time if the load time value is large.

Values for “Generating time (ms)” show the time spent compiling the ABAP source code.  You will see a value here if the code has not already been compiled through SGEN.  Any large values here for programs already generated could mean a problem in the database with the code storage tables.

Large values for “Roll (i+w) time (ms)” indicates that the Roll-In and Roll-Wait times are being affected.
Roll-In is the process of moving the users session context into the work process so that it can be used to execute the dialog step.  High values could indicate that the roll buffer is too small (see ST02) or that the user has a very large set of roles and profiles and is consuming too much memory.

Values for “Enqueue time (ms)” show the time waited for a logical database lock to be performed.
A high value here could indicate excessive contention (someone else doing the same work) in the SAP system or the underlying database. Maybe check SM13.

If we look at a CPU time example, a high value here would indicate a large amount of overall processing causing the WP to use more CPU.
Maybe a rogue loop, or a large internal table operation.
The example below shows a high CPU time for a background job performing a lot of RFC calls.
Notice how the processing time (just off the screen shot, sorry) has increased too (some component has to do the work):

STAD CPU time, response time

Could be that a lot of data/work was transfered by RFC call.

Summary:
If you see high database time in your analysis, Part III will show you how to perform an SQL trace to see what the database is doing for so long.

If you have high CPU/processing time, then in Part IV you will be shown how to find where exactly in the ABAP program is causing the most response time.

Detecting & Reducing Space Used For SAP Business Workplace Documents

Within the SAP Netweaver 7.0 system is a complete e-mail tool that allows users to send/receive electronic documents.
These documents are visible in the SAP Business Workplace (TX: SBWP) inbox/outbox and also in the SAP Connect transmission requests (TX: SOST).

As time goes by and users accumulate documents in their inbox and internal/external mails get sent through SAPConnect, space in the database will be used.

SAP Note: 966854 recommends running RSBCS_REORG to clear down these documents and free the space.
Unfortunately it doesn’t tell you how much space is taken up and how much you could get back.

Using the details in SAP Note: 706478 it is possible to check some of the tables at the database level.
Once again, the note doesn’t list all tables involved.
Instead, running the RSBCS_REORG report in “Test” mode reveals the full extent of the tables where records will be deleted:

So using this information, the following constructed SQL query will return the current size of the database segments (Oracle) for the selected tables (FOR ECC 6.0 based systems):


SELECT *
FROM (
select sum(s.bytes)/1024/1024 size_mb ,
s.segment_name,
s.segment_type,
s.segment_name table_name,
t.ddtext descr
from dba_segments s,
sapsr3.dd02t t
where s.segment_name in ('BCST_BOR',
'BCST_CAM',
'BCST_RE',
'BCST_SR',
'SOC3',
'SOCS',
'SOCX',
'SOER',
'SOES',
'SOFM',
'SOOD',
'SOOS',
'SOST')
and s.segment_name = t.tabname
and ddlanguage='E'
group by s.segment_name, s.segment_type, s.segment_name, t.ddtext
union all
select sum(s.bytes)/1024/1024 size_mb ,
s.segment_name,
s.segment_type,
i.table_name,
t.ddtext descr
from dba_segments s,
sapsr3.dd02t t,
dba_indexes i
where i.table_name in (
'BCST_BOR',
'BCST_CAM',
'BCST_RE',
'BCST_SR',
'SOC3',
'SOCS',
'SOCX',
'SOER',
'SOES',
'SOFM',
'SOOD',
'SOOS',
'SOST')
and i.table_name = t.tabname
and i.index_name = s.segment_name
and ddlanguage='E'
group by s.segment_name, s.segment_type, i.table_name, t.ddtext
) t1
ORDER BY t1.table_name ASC, t1.segment_type DESC;

For R/3 4.7 you can use the query below (removes the table and index descriptions):

SELECT *
FROM (
select sum(s.bytes)/1024/1024 size_mb ,
s.segment_name,
s.segment_type,
s.segment_name table_name
from dba_segments s
where s.segment_name in ('BCST_BOR',
'BCST_CAM',
'BCST_RE',
'BCST_SR',
'SOC3',
'SOCS',
'SOCX',
'SOER',
'SOES',
'SOFM',
'SOOD',
'SOOS',
'SOST')
group by s.segment_name, s.segment_type, s.segment_name
union all
select sum(s.bytes)/1024/1024 size_mb ,
s.segment_name,
s.segment_type,
i.table_name
from dba_segments s,
dba_indexes i
where i.table_name in (
'BCST_BOR',
'BCST_CAM',
'BCST_RE',
'BCST_SR',
'SOC3',
'SOCS',
'SOCX',
'SOER',
'SOES',
'SOFM',
'SOOD',
'SOOS',
'SOST')
and i.index_name = s.segment_name
group by s.segment_name, s.segment_type, i.table_name
) t1
ORDER BY t1.table_name ASC, t1.segment_type DESC;

You may just be able to see that the SOC3 table is using ~7GB of space with an index of 144MB and a lob object of 64KB.

Now if you run the RSBCS_REORG report using “Test” mode, with the settings to remove the documents that are >60 days old, then you can estimate what percentage of records will be removed from table SOC3 and therefore estimate the space saving.

As a rule of thumb, it may be wise to remove the documents from users who have left the company.
Generally the SAP account will be locked, so you can pull the account names using the following SQL query then add them into the RSBCS_REORG report “User” field:

SELECT DISTINCT bname FROM sapsr3.usr02 WHERE uflag =128;

NOTE: When running RSBCS_REORG, it will not remove assigned workflows from the “Workflow” folder (or sub-folders).

You may also consider running the report for the user who runs the SAPConnect background job step.  As this user will have the majority of occupied space.

Once you’ve deleted the records that are not required, the space in the database tables will be freed.
However, this will not release the space to the rest of the database, only the tables from where the records were deleted.
Stay tuned for my up-coming post on how to free the table segment space after you’ve removed thousands of records from the Oracle database.

Useful SAP Reports

It’s always good to have the exact report handy, just incase the transaction is not in your SAP role:

BTCTRNS1 – Suspend All Jobs For an Upgrade
BTCTRNS2 – Un-Suspend All Jobs For an Upgrade
RDBMIDOC – Create IDocs from change pointers and send.
RBDMOIND – Status conversion of successfull IDoc communication.
RSALDBRG – Reorganise CCMS Alert Database
RSALERTTEST – Test alerts
RSAVGL00 – Table adjustment across clients
RSINCL00 – Extended program list
RSBDCSUB – Release batch-input sessions automaticly
RSCPINST – Check language installation config.

RSCSAUTH – Authorisations assigned to ABAP reports.

RSTXSCRP – Transport SAPscript files across systems
RGUGBR00 – Substitution/Validation utility
RSBCSRE03 – Clean old SOST records (send deliveries)
RSBDCOS0 – Execute Shell command (non-interactive).
RSBTCDEL – Clean the old background job records
RSDBCREO – Clean batch input session log
RSEIDOCA – Active IDoc Monitoring
RSORAPATCHINFO – Oracle patch level.
RSPARAM – Display all instance parameters
RSPO0041 – Removing old spooling objects
RSSNAPDL – Clean the old ABAP error dumps
RSSLG200 – Remove expired application logs (SLG1).
RSSLGK90 – Delete application logs created before the specified date.
RSSOREST – Clean old SOST records (send deliveries)
RSPO1041 – Cross-Client Spool Request Cleanup.
RSPOR_SETUP – BW / BI Configuration Consistency Check
RSTXICON – List all icons in the system
RSTRFCEG – Consistency check of outbound queues (RFC/trfc/qrfc).
RSUSR002 – Roles by complex selection (new)
RSUSR003 – Check the passwords of users SAP* and DDIC in all clients
RSUSR006 – List users last login
RSUSR020 – Profiles by complex selection
RSUSR050 – Compare users
RSUSR070 – Roles by complex selection
RSUSR200 – List users by login date.
RSWUWFML2 – Delivers work items via users e-mail address.
RSWWWIDE – Remove work items and related sub-items.
RSXMB_CANCEL_MESSAGES – Cancel messages in error.
RSXMB_CHECK_MSG_QUEUE – Check messages in queue.
RSXMB_SHOW_STATUS – Show statuses of XI messages in the persistence layer.
SSF_ALERT_CERTEXPIRE – Certificate expiry check.

Useful SAP Transactions

Here are a collection of SAP transactions that I find useful:

AISUSER – Solution Manager maintain S-user for maint-opt.
AL08 – All Users Logged On
AL13 – Display Shared Memory
ALRTCATDEF – Alert category definitions
ALRTDISP – Alert display
ALTINBOX – Alert inbox
BDLSS – Shows all items associated with the logical system name.
BDM7 – ALE Audit Statistics
BD10 – Send Material Master
BD12 – Send Customer Master
BD14 – Send Vendor Master
BD22 – Reorganise (Delete) Change Pointers
BD87 – ALE Message Monitor
BF24 – Create Product (User exits)
BF34 – Register user exit event to function module.
CG3Z – Transfer file from frontend.
CMOD – List user exits
DB02 – Database Tables/Tablespace Statistics
DB13 – Database Planning Calendar
DB14 – Database Operations Log
DB26 – Database Parameters
DBCO – External Database Connections in SAP.
F.10 – GL Accounts
F110 – BACS Payments
FBL1N – List Invoice Documents
IDX1 – Port Maintenance In IDoc Adapter
IDX2 – Meta Data Overview in IDoc Adapter
IDX5 – XI IDoc search.
LISTSCHEMA – BW Lists underlying tables for InfoCubes.
MMRV – Show current posting period
OAAD – Archivelink Administration of Stored Documents
OAER – Business Document Navigator
OAM1 – ArchiveLink Monitor
OAC0 – Define content servers
OAWD – Archivelink Document Storage.
OB29 – Define Posting Periods (See also SPRO -> Fianncial Accounting -> Financial Accounting Global Settings -> Fiscal Year)
OMJ3 – Printer Determination by Plant.
OMJR – Printer Determination by Output Type.
OS01 – Ping servers
ORA_PERF – Analyse tables, delete statistics.
OY18 – Reports for table logging.
PA30 – Maintain HR Master
PFAL – HR ALE Transfer of Master Data.
PFCG – Role Maintenance
PFTC – Workflow task maintenance.
PFTC_CHG – Change a workflow task.
PFUD – Profile consistency (mass user comparison) check (schedule job).
RMPS_SET_SUBSTITUTE – Set Workflow substitute.
PPOME – HR Mini Master (Personnel Management)
PPOMW – HR Mini Master (Organisational Management)
RRMX – Launch BEX Analyser
RSA7 – BW Delta Queue Maintenance
RSCSAUTH  – Maintain authorisation groups.
RSM37 – Display background jobs with parameters.
RSRDA – Stop all daemons for real-time data acquisition in BW.
RZ01 – Job Scheduling Monitor.
RZ03 – Display/ Manully Switch Operation Mode (start/stop instance).
RZ04 – Maintain Operations Modes
RZ12 – Maintain RFC groups.
RZ21 – Monitoring Properties
RZ70 – SLD Administration
SA38 – ABAP Reporting
SAPL0ARC – Archiving Customising For Object.
SCCL – Local Client Copy.
SCEM – CATT initial screen.
SCU3 – Analyse table logging logs
SDCCN – Service Data Control Centre
SECR – AIS (Audit Information System)
SE03 – System Change Option
SE11 – Data Dictionary Display
SE21 – Package Builder.
SE61 – Change display screen text.
SE93 – Transaction lookup.
SE37 – ABAP Function Modules
SE54 – Generate Table Maintenance.
SBIW – Maintain Business Information Warehouse Data Sources
SCU0 – Customizing Cross System Viewer
SLDCHECK – Check status of the SLD
SLDAPICUST – SLD API Customizing
SLG1 – Analyse application log
SLG2 – Application Logs Delete
SMLG – Logon groups.
SMGW – Gateway monitoring
SMICM – ICM Monitor
SMQ1 – qRFC Monitor (Outbound Queue)
SMQ2 – qRFC Monitor (Inbound Queue)
SMQ3 – qRFC Monitor (Saved E-Queue)
SMQE – qRFC Administration
SMSY – SOLMAN – Landscape Directory
SMSY_SETUP – Solution Directory Setup
SMX – Own Jobs
SM18 – Security Audit Log Reorganisation
SM19 – Security Audit Log Configuration
SM20 – Security Audit Log Analysis
SM35 – Batch Input Log (LSMW etc)
SM58 – Asynchornous RFC Error Log
SM63 – Maintain Operation Modes Calendar
SM64 – Background Jobs Events.
SM65 – Background processing check.
SM66 – All processes across system.
SM69 – Maintain External Commands
SO36 – Automatic System Forwarding for E-Mails
SOA0 – ArchiveLink Document Types.
SOLMAN_WORKCENTER – Solution Manager Maintenance Optimizer.
SP12 – TemSe administration
SSAA – System Administrator Assistant
ST02 – SAP Buffer Tuning
ST03 – Global Workload Monitor
ST04 – Database Statistics
ST06 – Operating System Monitor
SUCU – Table / View Authorisations.
SUGR – User Group Maintenance
SWDD – Workflow Builder.
SWDM – Workflow Builder
SWI1 – Workflow Instance Log
SWE2 – Workflow linkages
SWEL – Display Event Trace
SWELS – Switch on/off event trace
SWEQADM – Event Queue Administration
SWEQBROWSER – Event queue browser
SWO1 – Business Object Builder
SWU0 – Workflow event simulation
SWU3 – Workflow customizing
SWUS – Test Workflow
SWWL – Delete workflow
SXMB_MONI – Integration Engine Monitoring
SXMB_MONI_BPE – Process Engine Monitoring
SXMB_IFR – Call Integration Builder
SXMB_ADM – Integration Engine Admin
SXI_CACHE – XI Directory Cache
SXI_MONITOR – XI Message Monitoring
TU02 – Change parameters
USMM – User Measurement Statistics.
WE02 – IDoc Display
WE07 – IDoc Monitoring
WE05 – IDoc List.
WE09 – IDoc Search.
WE11 – Delete IDocs
WE19 – IDoc Test Tool
WE20 – IDoc Partner Profiles
WE40 – Idoc administration setup
WE41 – Outbound Process codes.
WE42 – Inbound process codes.
WE46 – IDoc Admin
WE60 – IDoc Types Documentation
XK02 – Changer Vendor Details

Useful SAP Tables

During my time as a SAP BASIS administrator, I have accumulated a number of useful database tables.

ADR6 – User Address Data (cross with USR21)
AGR_1250 – Authorization data for the activity group
AGR_USERS – Assignment of roles to users
AGR_SELECT – Assignment of roles to Tcode
AGR_PROF – Profile name for role
AGR_DEFINE – Role definition
BDCPS – Change Pointers
DDLOG – Table buffer changes for sync between dialog instances.
DD02T – SAP Tables and descriptions.
DD09L – SAP Tables (with logging PROTOKOLL=X)
PA0105 – HR Communications InfoType (includes USERID to PERNO).
SH_PRIN – Printer list
TADIR – Repository object table.
TBRG – All Authorisation Groups.
TDDAT – Tables to Authorisation Groups.
TBTCP – Background Job Steps (see field AUTHCKNAM for job user)
TOLE – OLE Link table
USR02 – User passwords.
USR01 – Users.
USR04 – User Master Authorisations.
USR10 – Authorisation Profiles.
UST04 – Users to profiles.
TDEVC – Package table.
TSP03D – Printer Output Devices
TST03 – Spool requests
TSTC – Transactions
TVARV – Variant Variables
VARI – Variants