This blog contains experience gained over the years of implementing (and de-implementing) large scale IT applications/software.

Connecting SAP Netweaver ABAP Stack To SAP SLD

If you’ve ever quickly needed to know the points to check for customising settings connecting your SAP Netweaver *ABAP* system to an SLD (System Landscape Directory), then here they are:

Transaction: SLDAPICUST
Transacion: RZ70
Transaction: SM59 – TCP/IP connections: SAPSLDAPI (Use by ABAP API)
                                                                 LCRSAPRFC  (Use to read the exchange profile)

Here’s the SAP Help article: https://help.sap.com/saphelp_nw04s/helpdata/en/be/6e0f41218ff023e10000000a155106/content.htm

Enable SAP* in Netweaver 7.0

If you’ve lost the Administrator password or you want to log in as SAP* in your pure Java stack environment, this can be a little tricky as the way to set the password for the SAP* user in configtool is not as user friendly as you might think.

Log into configtool by opening an X-Windows session (on UNIX), and then execute: /usr/sap/<SID>/JC<SYS#>/j2ee/configtool/configtool.sh (.bat on Windows).

NOTE: Once you enable the SAP* user, you will not be able to log into the system as any other users e.g. Administrator or J2EE_ADMIN.

In the configtool screen, expand the “Global server configuration -> services” branches:

Click the “com.sap.security.core.ume.service” item:

On the right hand side, scroll down to the ume.superadmin.activated option:

Set the “Value:” field to “TRUE”:

Now single click the “ume.superadmin.password” item:

You can’t see the password and the “Value:” field looks like it doesn’t accept input, but it does.
Type the new password in to the “Value:” field at the bottom (even though the cursor doesn’t move):

Now click “Set” on the right:

You will be prompted to re-enter the password:

Click Save:

You should restart the J2EE stack before trying to log in as SAP*.

SAP on HP-UX

There are various methods of tuning an SAP system, but sometimes the operating system vendor will permit specific tuning so that the SAP system can take better advantage of the available resources.
Usually, you can find whitpapers on the O/S or hardware vendor’s web site, but also SAP will themselves develop a range of SAP notes that will offer avice and guidance when running on certain approved hardware partners.
If you’re using HP-UX 11iv3 (11.31) and you are running SAP, then here are some gems direct from SAP themselves:

Note:
172747 HP-UX OS Parameter Recommendations
798194 HP-UX async IO trace files
837670 HP-UX OS Patch Requirements
918279 HP-UX SAPOSCOL Versions
1077887 HP-UX filesystem options
1329878 Using non-default pagesize on HP-UX
1351051 Support of Oracle on HPVM
1457063 HP-UX Consolidating SAP Systems
1575609 Future HP-UX support for SAP on Oracle

I would highly recommend the OS Parameters note.
If you’re suffering I/O performance issues, make sure that you have set the correct block size as per 1077887.
A huge (>10%) reduction in memory usage can be obtained following 1329878, but only if you are using Oracle 11g.

Using the direct I/O option alone on a VxFS 5.0 environment decreased Data Protector backup times by over 1 quarter e.g. a 1.5 hour backup reduced to ~1 hour.

I haven’t investigated the HPUX_SCHED_NOAGE O/S parameter option yet (I need the audacity to recommend it to the server team!), but according to this excellent blog post by Christian Bilien, it should help your environment significantly if you’re running more than one Oracle database on a SMP system.
My reasoning is that there will be more than one CPU hungry Oracle thread and they could each be battling against each other (ageing each other out) even more so in a well tuned system with less I/O (large SGA/PGA and well tuned SQL) meaning less voluntary context switches and more forced ones. It’s possible that in this situation, without HPUX_SCHED_NOAGE, you could start to see CPU bottlenecks.

HP also produce their own performance recommendations for running Oracle.
Here’s one on the HP recommended filesystem IO options.
Notice that it says if you’re using VxFS v5.0, you don’t need to worry about 1KB block sizes on Redo and Archive Log partitions.

WARNING: Take heed in the warning about Progress databases on file system partitions with directio enabled on the mount options.  I have personally experienced issues where Progress applications have had a 20x decrease in performance!  It’s a fact, Progress is not very good without the file system cache.

This one is specific to tuning HP-UX TCP/IP.  Very useful if you’re seeing networking bandwidth problems in you environment.

Finally, the HP-UX Performance Cookbook provides an excellent source of information.
I have noticed that some of the SAP Oracle recommended parameters (830576 “Parameter Recommendations for Oracle 10g”) are specific to HP-UX, I have found that a good set of descriptions for some of these are covered in this HP Oracle paper: The Oracle Database on HP Integrity servers.

It’s really a trade off between what SAP say and what Oracle say.
Obviously Oracle know there own database system, so anything Oracle says can also be included.
I generally follow the top down rule, where SAP overrides anything that is contradictory from HP or Oracle, Oracle overrides anything contradictory from HP, and I only implement any parameters from HP if neither Oracle or SAP have anything to say on the matter:
– HP (OS)
 – Oracle (RDBMS)
  – SAP

Basic Performance Tuning Guide – SAP NetWeaver 7.0 – Part IV – ABAP Runtime Analysis

This is the final part of my SAP NetWeaver 7.0, Basic Performance Tuning Guide.
You may wish to review Part I, Part II or Part III.
In this part I will focus on the use of ABAP Runtime Analysis to check performance of ABAP programs/reports.  This will help you locate areas of ABAP that are eating up valuable response time.

Execute transaction SE30.
Enter the transaction (or program or function module) that you wish to analyse:

Edit the “Measurement Restrictions” variant to either create your own, or simply adjust the DEFAULT.
This will allow us to display additional information which can be very useful when trying to determine performance problems.

Tick the “DB-level ops” option on the Statements tab:

On the Duration/Type tab, select “None” for Aggregation and tick “With memory use”:

The transaction or program you entered will be run, but you should see a prompt at the bottom of the screen telling you that the measurement process has started:

Perform the actions you need in the transaction/program.
Once you have finished, simply click Back (exit from the transaction) to the SE30 main screen:

The SE30 screen will display a prompt showing the analysis has finished:

The trace data is stored in a file on the SAP instance server.
This can be seen by clicking the “File Info…” button:

You can see the file name and path:

If you need to load the trace data again for analysis, you can simply click the “Other File…” button on the main screen.
It’s also possible to transfer the analysis files via RFC to another SAP system from the “Other File…” sub-screens.
This is very useful if you want to compare performance between two systems.
NOTE: The performance data files are only retained on the server for eight days.  I haven’t found a way of extending this.

Let’s analyse the stats.
Click the Evaluate button:

You will see a basic breakdown of the transaction runtime performance analysis in histogram form.
The three different areas are shown (ABAP, Database and System) against time (as a percentage):

More details on this screen can be found here: https://help.sap.com/saphelp_nw70/helpdata/en/c6/617d27e68c11d2b2ab080009b43351/content.htm

The values shown correlate to the response times measured in the single record statistics plus the time required for analysis.
Note that DATABASE time above, also includes the time the system takes to open and close cursors etc, not just the SQL runtime.
For reference, the STAD output for the same operation (notice the slightly lower DB req time):

SAP STAD output DB request time

The most interesting buttons at the top of the SE30 screen are “Hit List”, “Table Hit List” and “Call hierarchy”.
 Click the Hit List button:

STAD output hit list

The Hit List will be displayed in descending order (gross time):

STAD output hit list output

The columns are defined as follows:

No.” – Shows the number of times a call was made.
If a “SELECT” is performed in a loop, then this will show the number of times the call was made in total in the loop.

Gross” – Total execution time in microseconds (running bottom to top of the hit list).
Net” – The time in microseconds for the specific call(s)
Call” – The program component that was being executed.
In program” – The main program of the component.
Type” – The type of call (DB, System, Non-system).

SAP says here https://help.sap.com/saphelp_nw70/helpdata/en/c6/617d27e68c11d2b2ab080009b43351/content.htm:
Gross time
The total time required for a call. This includes the runtime of all modularization units and ABAP statements called by the subject.

Net time
The net time is the gross time less the time required for modularization units (MODULE, PERFORM, CALL FUNCTION, CALL SCREEN, CALL TRANSACTION, CALL METHOD, CALL DIALOG, SUBMIT), and for the ABAP statements listed separately. For statements such as APPEND, the gross time is the same as the net time.

In this view, you are looking for records in the list with the larger “Net” value.
This indicates that a large amount of overall time was spent in the call/program listed.

If the Gross and Net time fields do not hold the same value for a record, then double clicking the record will display the sub-components and switch to the hierarchy view:

STAD output hit list sorting

At any time, you can single click a record, then jump to the ABAP source by clicking the “Display Source Code” button:

STAD output hit list call point in ABAP

STAD output hit list call point in ABAP

NOTE: I always recommend you enable the “New” ABAP editor.  It gives you visibility of line numbers and highlights the results of text “Find” operations so you can see what’s been found.

Back on the main analysis screen, select the “Table Hit List” button:

The list of tables used in the execution of the transaction/program are displayed in descending order of runtime:

STAD output hit list tables used

The #Acces column, displays the number of times the table was accessed (read/write) and corresponds to any loops you may have seen in the “Hit List” screen No. column.

The Class column shows TRANSP (Transparent Table), VIEW or POOL (cluster table).

Double click a table to display the call points, from which you can jump to the ABAP source.

Finally, from the main analysis screen, click the “Call hierarchy” button:

The Hit List that you have already seen, will now be displayed in a hierarchy with sub-calls indented under the parent call:

STAD output hit list call hierarchy

The only real difference between this view and the Hit List table view, is the indenting.
You can still double click to display the execution time overview for that sub-call, or use the “Display Source Code” button to jump to the ABAP source.

In my example screen shots, you will have seen that the PMSDO fetch is taking a large amount of the response time.
I was able to find that this operation was inside the GET_DATA function and check the ABAP code.

The main thing you are looking for in poor performing ABAP code, is where does the execution time get eaten up.  As a side note, you may also check the amount of memory used if you are looking to resolve TSV PAGE ALLOC errors.

Try sorting the Hit List by “Net Time” to find the individual components with the highest net execution time.
Check if the operation is a database related call or not.  You can then use Part III to trace this further with an SQL trace.
Check the number of loop iterations (The “No.” column) in the “Hit List”. Are you performing an expensive routine too many times in a loop?

Validate that the code is not doing too much work using iTabs, when some of the work could be done in the database.
The database is the ultimate place to do sorting, grouping and record elimination.
Don’t pull back more records than you need, only to loop through an iTab in ABAP and remove them, it’s more expensive.

That’s it for now.  Thanks for reading.

Basic Performance Tuning Guide – SAP NetWeaver 7.0 – Part III – Using SQL Trace

This is Part III, you may like to see Part I and Part II (or skip to part IV).

Once you have traced the poor performing program (see Part I of this guide), and you have identified that your problem is in the database response time (see Part II of this guide), then you can start digging deeper into the potential SQL problem by running an SQL trace using transaction ST05.
In ST05, click the “Activate Trace with Filter” button:

SAP ST05 Performance Analysis, activate trace with filter

NOTE: We are only concerned with an SQL statement that has directly accessed the database (https://help.sap.com/saphelp_nw04/helpdata/en/d1/801f89454211d189710000e8322d00/frameset.htm).
Unless you select the additional tick box “Table Buffer Trace” on the main ST05 screen, to show table buffer accesses, all the trace details in ST05 will be for direct database accesses (bypassing the table buffer).
Since direct database accesses will take longer than the buffer accesses, it is generally accepted that the longer runtimes will be associated with direct database access.

Enter your user id and the name of the report/transaction.  This is so that we can try to limit the records in the trace.  If you want to trace another user, simply enter their user id:

SAP ST05 trace by transaction name

Click Execute to start the trace:

ST05 trace activated

Now in a separate session (preferably) run the transaction/program with the performance problem.
Once you’re happy you have experienced the problem, you can go back into ST05 and click “Disable Trace”  (by default it will remember the trace details and disable the running trace, you won’t need to re-enter all the details):

ST05 trace deactivated

Still within ST05, click “Display Trace“:

ST05 trace display

The system remembers the last trace recorded and should automatically populate the details, but you can always enter them for date/time, user id etc:

ST05 trace display for user

The SQL trace records are displayed.
The poor performing SQL has it’s time (duration) in microseconds (millionths of a second) highlighted in red if it exceeds 10000 microseconds, 10 milliseconds or 0.01 seconds:

(1 second = 1000 milliseconds = 1000000 microseconds)

SAP ST05 SQL Trace output

At the top of the window we can see the transaction name (ZW39), the work process responsbile for executing the dialog step in which our statement was contained, the type of work process (DIA or BTC), the client and user and transaction id.
The OP column shows that this session has reused an existing cursor in the DB for this query.
This is because it has a “PREPARE” operation as the first OP before the cursor “OPEN” and not a “DECLARE” (see here https://help.sap.com/saphelp_nw04/helpdata/en/d1/801fa3454211d189710000e8322d00/content.htm).
If the session re-executed this query at another point in the trace, then it is likely (depending on code and available database resources), that the query would have re-used the existing cursor and even no “PREPARE” would have been visible, just OPEN and FETCH (https://help.sap.com/saphelp_nw04/helpdata/en/d8/a61d94e4b111d194cb00a0c94260a5/content.htm)
The deep yellow colour of each line will alternate between a dark and lighter colour when the table (obj name) changes in the list.

The RECS column shows the number of records retrieved in the “FETCH” operation.
The RC column shows the Oracle database return code 1403 (ORA-1403) which you get when you call “FETCH” in a loop and reach the end of the results (see SAP note 1207927).

The STATEMENT column shows the pre-parsed SQL including bind variables (:A1, :A2 etc).
The second line of the STATEMENT column shows the statement in the “OPEN” database call where all the bind variables have been normalised.
Double clicking on the STATEMENT column delves into the SQL statement a little more:

SAP ST05 SQL Trace output SQL statement with bind variables

Here you will see the SQL statement in full (above) and the bind variables, their data types (CH,3 = CHAR(3)) and respective values.
Back on the main trace screen, with the SQL statement selected (single click), you can click the “Explain” button:

SAP ST05 SQL Trace output SQL statement explain plan

Again, you will see the SQL statement with bind variables (remember you can get the values from the other screen) but more importantly, you will see the Oracle Explain Plan.
The Explain Plan shows how the Oracle optimizer *may* choose to execute the statement and the relative costs associated with performing the query (I’ll explain more on this in a later chapter of this guide).

SAP ST05 SQL Trace output SQL statement explain plan

In the above example, you can see that Oracle has chosen to perform an Index Range Scan on index “IHPA~Z1” before going to the table IHPA to get the matching data rows.
Single click the “Access Predicates” on the index and you can see what will be applied to the index scan:

SAP ST05 SQL Trace output SQL statement explain plan access predicates

SAP ST05 SQL Trace output SQL statement explain plan access predicates

Again, you will need to reference the very first SQL view screen to get the bind variables values.
Clicking the name of the index will show the statistics values for the index:

SAP ST05 SQL Trace output SQL statement explain plan index analysis

SAP ST05 SQL Trace output SQL statement explain plan index analysis

You can see the two columns that make up the index (objnr and parvw).
The Analyze button can be clicked to update the index database statistics real-time or as a background job.
WARNING: Be aware that SAP collects statistics in Oracle using it’s own set of predefined requirements as per SAP notes 588668, 1020260 and that you *must* disable the standard Oracle stats gathering jobs as per note 974781. Therefore, the stats may be out of date for a good reason.

The same detail can be seen when clicking the table name.

Back on the main screen, for those who have more knowledge of reading Oracle Explain Plans, you can choose the “EXPLAIN: Display as Text” button, which will display a more detailed Explain Plan that can be copied to a text editor also (I find this very useful):



Plan hash value: 1636536768

----------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
----------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 2 | 82 | 2 (50)| 00:00:01 |
| 1 | INLIST ITERATOR | | | | | |
|* 2 | TABLE ACCESS BY INDEX ROWID| IHPA | 2 | 82 | 1 (0)| 00:00:01 |
|* 3 | INDEX RANGE SCAN | IHPA~Z1 | 2 | | 1 (0)| 00:00:01 |
----------------------------------------------------------------------------------------

Query Block Name / Object Alias (identified by operation id):
-------------------------------------------------------------

1 - SEL$1
2 - SEL$1 / IHPA@SEL$1
3 - SEL$1 / IHPA@SEL$1

Predicate Information (identified by operation id):
---------------------------------------------------

2 - filter("MANDT"=:A0)
3 - access(("OBJNR"=:A1 OR "OBJNR"=:A2 OR "OBJNR"=:A3) AND "PARVW"=:A4)

Column Projection Information (identified by operation id):
-----------------------------------------------------------

1 - "OBJNR"[VARCHAR2,66], "PARVW"[VARCHAR2,6], "PARNR"[VARCHAR2,36],
"ERDAT"[VARCHAR2,24]
2 - "OBJNR"[VARCHAR2,66], "PARVW"[VARCHAR2,6], "PARNR"[VARCHAR2,36],
"ERDAT"[VARCHAR2,24]
3 - "IHPA".ROWID[ROWID,10], "OBJNR"[VARCHAR2,66], "PARVW"[VARCHAR2,6]

Finally, it’s possible to see the point in the ABAP program where the SQL statement was run.
Single click the statement column, then click the “Display Call Positions in ABAP Program” button or press F5:

The ABAP code will be displayed, and if you are using the new ABAP editor (I think you should), you will be positioned on the statement:

SAP ST05 SQL Trace output SQL statement call in ABAP

Looking at the ABAP statement, you can see how it has been converted from an OPEN SQL statement into an Oracle SQL statement.
The use of the “IN” keyword has been translated into multiple “OR” statements at the Oracle level.

Once at the ABAP statement level, it is sometimes required to know both Oracle SQL and ANSI SQL, since Oracle generally uses it’s own dialect and SAP uses ANSI SQL.
Therefore, you may see the original statement in a somewhat different form, especially when a table join is required on two or more tables.
Lastly, you will notice that the MANDT field is not present in the original ABAP, but it gets added to the predicate list by the ABAP OPEN SQL processor unless you use the “CLIENT SPECIFIED” keyword.

Armed with the SQL performance facts, the knowledge of how to perform an SQL trace and how to link this back to the piece of ABAP code; you should be able to use your database level investigative skills to work out why the statement is causing poor performance.

Make adjustments in the test system and see how the changes affect the performance.
I like to modify the Oracle SQL captured in the SQL trace, then enter it into ST05 (Execute SQL option) with the Explain button, to see if the explain plan is any better.

Some pointers:
– How much data is in the database tables being referenced?
– Is the Explain Plan showing that the query is making best use of available indexes?
– Can you replicate the problem in the test system?
– If this is a custom table, do you need to add additional indexes?
– Could the ABAP OPEN SQL statement be re-written to be more effective (use one query instead of two)?
– Are you SELECTing more fields than you actually need?

You may wish to read the master class in reading Explain Plans from Oracle.
I have blogged before about cardinality in Explain Plans here.

Find part IV of this blog post, here.