Netweaver Archives » Page 2 of 6 » Musings of an IT Implementor

SAP on ASE (Sybase) – 10 Lessons Learned

Continuing my SAP on ASE theme (see my other post on 10 positive points of SAP on ASE), I’m going to document the 10 best lessons I learned during the migration and implementation of SAP on SAP ASE (formerly Sybase).
Hopefully these will be useful to any implementation or migration project you may be undertaking.

1 Compatibility with SAP Business Suite
SAP provide SAP ASE for two groups of end-user; standalone use, or use with SAP Business Suite.
The SAP Business Suite is the name given to the suite of software packages such as ERP, BW, CRM, SRM etc.
Due to the 2 distinctions, there are always two delivery channels for the SAP ASE software binaries.

Certain software revisions of SAP ASE are certified (supported) for use with SAP Business Suite products, therefore you should always check if your use case is supported and that you download the relevant software version.

You should also ensure that you validate the information within the SAP notes, to ensure that it is relevant to your use case.
In some situations, there are SAP notes specifically only valid for SAP ASE in use with SAP Business Suite products.

2 Download the Latest
In my experience, the version of SAP ASE will make a big difference when considering the implementation project.
You should consult with your account manager to ensure that you plan to be on the latest release and patch level of SAP ASE possible for your deadlines.

The SAP ASE patches are released very frequently (I’ve seen 3 patches within 6 months); which usually contain some major fix for a possible data corruption scenario.

3 Stay Current
Revisions to the SAP ASE software are performed rapidly. The software stack is small and no minor fixes (binaries only or single binary fixes) are performed, so plan to patch the database at least once every 3 to 6 months minimum. If you fail to stay current, then you could face issues staying supported with other compatible software areas (such as SAP Replication Server).

A lot of the SAP notes will be updated to recommend the latest revision (usually for some major issue).
It has been known for a revision to be delayed dramatically (by months) only to be superceded almost immediately by a later revision.

So called “hot fixes” have also been seen, whereby the current revision receives an additional increment in between the previous and the latest. Usually this is only a stop-gap to fix an issue on a previous revision because the latest revision changes that functionality in a more intrusive manner (with respect to the code lines).

4 Ensure Server Parameters/Config
The configuration parameters of the SAP ASE database is performed against one core SAP note.
The SAP note has a rather messy layout and can be complex to digest manually.
Still, this is a required task and changing the parameters to those recommended by SAP is a must, since the out-of-the-box configuration is never really optimal or potentially even unstable (again refer to the SAP notes).
To ease this task, the method of checking these parameters (providing you’re SAP Netweaver revision is high enough) is through the DBACOCKPIT transaction.
Whilst this allows you to easily configure and apply the settings, it should never negate a manual review of the parameter settings. Plus, you may just learn something 😉 .

5 Confirm Statements with Your SAP account manager
Sometimes, I have come across erroneous support statement within the SAP notes or the help.sap.com website help pages.
This is potentially due to the two revisions of the SAP ASE software prevalent. Sometimes the SAP ASE standalone community forget that SAP ASE can run underneath SAP Business Suite.

Therefore, if you see a SAP note stating you’re not supported if you do “X” or have “Y” installed, query it with your account manager. It may not be relevant to your use case.

6 Set Relevant DB Parameters
Sometimes the parameters listed within the single SAP note (for that specific SAP ASE release), will be specific to SAP BW, or SAP ERP.

Since it’s entirely possible for you to be using the BW component heavily, inside your ERP system, you may wish to apply the BW related parameters (the old OLAP vs OLTP).
Therefore, you should double check the SAP note containing the SAP ASE recommended parameters, and not just blindly apply it.

7 Rigorously Test
It’s fairly simple and requires little effort to patch an SAP ASE database.
For this reason, you should expend the time saved, by testing.
As an example, I experienced an issue whereby I patched the SAP ASE software to a later revision, only to discover during a SAP system copy task, that a minor bug in the later software revision prevented the copy process from working.

In the end, after nearly a week of working on the problem (I immediately raised it with SAP also), I found (not SAP) that there was a workaround to get it working.
I also went on to note that the issue was later fixed in a later SAP ASE revision.

Therefore I decreed that, as part of upgrade testing, some of the standard internal department processes would be tested prior to upgrading the core systems.

8 Always Check the Bug List
As a follow on to the previous statement, you should check the bug listing of the next revision with eagle eyes.
You may not be experiencing a specific issue, but it could save you from potential corruption or getting into an unfixable situation.

Currently the SAP ASE bugs are not listed in individual SAP notes, but generally in the Release Information Note for the next revision (once it is solidified).
In addition to the RIN, there is one core SAP note for each SAP ASE release (e.g. 15.7 or 16.0) in which the up and coming revisions are noted.
Check this regularly.

9 Configure Your Own Housekeeping
One small shortfall in the SAP ASE software, is housekeeping.
The log files for the database, jobserver and backupserver do not rotate until the SAP ASE instance is restarted.
You may wish to keep these files tidy and compressed with your own housekeeping scripts.

There are also a couple of recommendations for retaining certain files such as the last configuration file, the dump history file and an export of the sysdevices table, on a separate file system (a bit like the old SystemV export of the partition table), just in case of a restore scenario.

10 Performance Tune Backups / Restores
Out-of-the-box the performance of backups and restores is adequate.
However on a 1.3TB database I saw backups (within 1 stripe only) taking in excess of 4 hours (to a DataDomain appliance).

By simply spending 2 days to performance tune, by adjusting just one SAP ASE parameter, I was able to reduce this by 1.5 hours (~30% improvement).

This will also be a future blog article as it was such a simple but effective task. Don’t accept the out-of-the-box settings.

Make sure that you test the restore capability and also ensure that you allocate adequate disk space for emergency backups (dumps) to disk (if you’re planning to backup to a third-party tool).

Make sure you watch out for future ASE articles for additional lessons-learned scenarios.

SAP NW 7.31 Java – HTTP Tracing

Scenario: You would like to trace a HTTP connection to a Netweaver 7.31 Java stack.

Prior to NW 7.31 you would have enabled tracing in either the old Java Administrative Tool, or newer Netweaver Administrator by setting the “enable” flag for tracing parameters of the HTTPProvider dispatcher service (see my previous blog article here).

Since Netweaver 7.31, the Java stack now includes an ICM (Internet Communication Manager) which is the main access point to which HTTP communications are routed to the Java server nodes.
Therefore, tracing in NW 7.31 Java is actually more like the ABAP stack tracing, because you simply increase the trace level in the ICM and then check the dev_icm trace file.

So the next question is, how do you access the ICM to increase the trace level in a Java stack?
Well this can be performed with the help of the SAPMC (only one “M” and not to be confused with SAP MMC for Windows).
The SAPMC was introduced in later NW 7.0 patches and is part of the sapstartsrv service.
The sapstartsrv actually listens on TCP port 5<##>13. So if you have access through your firewall, navigate to https://<app server>:5<##>13 and the the Java based applet will start up (note that you’ll need Java 1.4 or above on your PC).

Once loaded, you can see the SAP MC and expand the “AS Java” branch to locate the “Process Table” item:

SAP MC management console

From the “Process Table” you should be able to right click the ICM (on the right) and increment the trace level:

SAP MC increase ICM trace level

The trace level is dynamically increased.
To see the trace file, right click again and select “Show Developer Trace“:

SAP MC ICM display trace

You can always see the current trace level by right clicking the ICM and selecting “Parameters…”, or actually right clicking the ICM node further up the tree and selecting “Parameters…”:

SAP MC show ICM parameters

The current trace level is shown as the “rdisp/TRACE” parameter value on the Dispatcher tab:

SAP MC rdisp/TRACE ICM

Again, right clicking the actual ICM node in the tree gives you direct access to the ICM management web page:

MWSnap106 2014-07-23, 09_39_44

From the ICM management page, you can also see the current trace level:

ICM Manager Monitor

Good luck with your tracing.

DBCLONE Is Still Running, Running & Running Running…

Scenario: Your running through an upgrade of SAP on Oracle, either applying an EHP or a support package stack update. You’re using the “Standard” downtime minimized approach and you’ve got to the SUM stage MAIN_SHDCRE/SUBMOD_SHDDBCLONE/DBCLONE and it has just bailed out!

During the DBCLONE step, a number of background jobs are created that copy certain tables, programs sources etc, from the current SAP database schema, to the shadow instance schema (on Oracle SAPSR3SHD).
The copies of the tables, sources etc, are placed into the new tablespace that you were asked to create in the earlier steps.

During this copy process, the database will be generating a lot of redo information (it is performing a lot of INSERTs). This means that it will be generating a lot of archive logs also. Most systems are in archive log mode by default, as this is the safest way of upgrading a production system.

The DBCLONE step can take a long time depending on a few factors:

Size of your SAP system master data. Since the transactional data is not copied, most SAP systems will be roughly the same sort of size for the master data tables and sources etc (e.g tables D010TAB, D010INC, REPOSRC). Don’t forget, once tables are cloned, it needs to build the required indexes on those tables too! Then it will gather stats on the tables and indexes.
Quality of your database. If your Oracle database is highly fragmented, the indexes are not in good shape, or there is a lack of memory allocated to the database.

Redo disk write times. The faster the write times for redo, the quicker this is going to go.
Number of parallelised jobs. The SUM tool recommends 3 jobs in parallel. Prior to this SUM step, you would have been asked to configure the number of parallel jobs (and also your number of background work processes). If you configure less than 3, then it will take longer. I would personally recommend to have n+3, where n= your normal production number of background work processes. This means you will not be hampering day-to-day usage by blocking background jobs. The 3 jobs are created with high priority (Class A) so they get all the background processing they need.
Whether you elected to pre-size the new shadow tablespace data files.
Setting them to autoextend is fine, but by default, the SAP brspace commands create the files with only 200MB. By setting these files to be as big as they need to be (no autoextend) then you will save time.

During the DBCLONE step, the SUM tool monitors progress by RFC connection into the SAP system. It checks to see when the DBCLONE background jobs complete (and that they complete successfully).
If you have limited space available in your archive log area, and this fills up, then the RFC connection from SUM fails to work (archiver stuck issue).
This causes SUM to report that the step has aborted, but that DBCLONE was still running.

You will still see DBCLONE running in the background when you resolve the archiver stuck issue.
At this point, you could choose to manually cancel the jobs by “Cancel without Core” in SM50 for the busy background work processes where DBCLONE is running. However, they are still running, and simply waiting until they have finished, then restarting SUM, will continue from where it left off.

It knows where it got to, because it records the list of tables cloned in the SAPSR3.PUTTB_SHD table by setting the CLONSTATE column to ‘X’.
During the cloning process, the tables to be cloned are assigned to background jobs using the CLONSTATE column in the SAPSR3.PUTTB_SHD table.

You can monitor the cloning progress by using the following SQL:

SQL> select count(*) num, clonstate from SAPSR3.PUTTB_SHD group by clonstate;

You will notice that the CLONSTATE column will contain:
‘X’ – Tables cloned.
‘1’ – Tables on the work list of background job DBCLONE1.
‘2’ – Tables on the work list of background job DBCLONE2.
‘n’ – Tables on the work list of background job DBCLONEn.
‘ ‘ – Tables not being cloned.

As tables are cloned, the CLONSTATE changes from ‘n’ to ‘X’.
It seems that larger tables are performed first.

The method used to clone the tables is: “INSERT into <table> SELECT (<fields>) FROM <source>;”.
Then a “CREATE INDEX” is performed.

It’s also worth noting that you may need enough PSAPTEMP space to account for the index creation.
In a Solution Manager 7.1 SPS10 system, there are 13109 tables to be cloned.

As a final parting comment, if you have Oracle 11gR2, you should consider the database compression options available to you. Reducing the I/O requirements will massively help speed up the process.

Analysing High SAP Roll Wait Time – NW702

One of my clients identified an increase in the Response Time measurements they monitor. They use the EarlyWatch Report to monitor the SAP system SLAs against response time performance.
I initially thought this was going to a be a nice easy assignment, identify the database performance issues and propose some reorganisation work. After all, this does seem to be the case in ~75% of the cases I’ve seen.

Basic info: SAP NW 702 on Oracle 11gR2 on Unix.
SLAs are monitored against DIALOG response time from 9am to 6pm weekdays only.

I brought up the ST03 screen and checked the “Ranking Lists -> Top Response Time of Dialog Steps” (also known as the Hit List) for the past 3 month’s data, one month at a time.
All three months showed a similar pattern. The worst performing reports were customer developed reports. The database time looked reasonable (< 40% of Response Time – Wait Time), processing time was long-ish and load/gen time was tiny.
What didn’t look good was the “Roll Wait Time”.

I backtracked a step and looked again at the “Workload Overview”, and specifically tab “Parts of response time“:

MWSnap101 2014-01-02, 09_55_04

That’s right, 26% of the response time for Dialog, is Roll Wait time.
Time to dig out the definition of SAP Roll Wait Time…
I’ve blogged about the basic performance tuning steps before, but I’ve yet to deep dive into the Roll Wait Time metric.

What is “Roll Wait Time”.
SAP note 8963 says:

“The roll wait time is time that elapses on the client while the user context is in the roll area. Therefore, no resources are used on the client and there is no bottleneck here if the roll wait time is long. “

So we may not be looking at a problem with the SAP server, but a problem between the SAP application and the front-end.
The note goes on to state that for DIALOG processing, the roll out of the context can be caused by calls to other external applications from the R/3 system, or from a call to an external RFC.
More importantly, it says:

“As of Release 4.6, roll wait times also occur when the R/3 system communicates with the controls in the front end. While the data in the front end is processed, the R/3 context is rolled out, and the work process is released as a result. This may occur several times during a transaction step. If front-end office applications (such as Microsoft Word) are started and only closed after a long time (more than several minutes), a very long roll wait time also occurs.“

This means that the communication to the front-end (SAP GUI, Microsoft Excel, Word etc), can cause the DIALOG work process to roll out, subsequently increasing the “Roll Wait Time”.
Even further clarification is provided in SAP notes 364625 and 376148 which mention the “new” GUI controls introduced in R/3 4.7.
SAP note 606610 explains how an ABAP “WAIT” command causes a task to roll-out of the work process.

SAP note 203924 provides some more detailed information on high Roll Wait Time:

“As of Release 4.6 the roll wait time (and therefore the total response time) contains the time for installing “Enjoy Elements” (=controls) and therefore the essential part of the communication between the GUI and the R/3 instance. In this case, the response time displayed in Transaction ST03 is closer to (almost identical to) the time experienced by the user.“

Plus it also confirms what we’re thinking:

“As of Release 4.6, therefore, a high roll wait time is a first indication of a slow network between the GUI and the R/3 instance.“

Section 2a of the note provides some good pointers in diagnosing network performance issues and checking the size of the SAP user menus.

As per the note, I opened up transaction ST06 and via the “Detailed Analysis” button, I went to the “LAN Check by PING” screen, then clicked “Presentation Server“:

MWSnap104 2014-01-02, 10_57_13

Once here, I selected a random selection of “Presentation Server” clients and initiated a 10x PING.
What I found was that specific “Presentation Servers” (client PCs) were not responding within the expected 20ms:

I knew that we were operating in a WAN environment (multiple offices across different cities) so I should be expecting a WAN connection time of between 50ms and 250ms (according to SAP note 203924).
I was seeing ~60ms in some cases. So I can conclude that we have a moderately quick WAN setup.
The main point is that I was not seeing any packet loss. Which is a good sign.

Whilst the immediate data is always good to see, it’s worth mentioning that the speed could be circumstantial. It’s best to check the times in our ST03 statistics. Opening up ST03 again and navigating to the “Workload Overview” analysis view, I can see on the “Times” tab, an “Average GUI Time per Dialog Step” of 90.4ms:

round trip

GUI Time

GUI Time: Measured at the Application Server, this is the average time taken for all requests (of a Dialog step) to go from the Application Server to the front-end SAP GUI (it does not include the time taken for the SAP GUI to send data to the Application Server).
The time I see is not too bad and will vary depending on the amount of data sent.

We can verify the value by checking the “Average Front-end Network Time” (per Dialog step):

Front End Network Time

Front-end Network Time: Measured at the SAP GUI, we see that the Front-end has recorded an average of 91.4ms time taken for the first request (of a Dialog step) to go from the Application Server to the front-end SAP GUI, plus the time taken for the last request (of a Dialog step) to go from the Application Server to the SAP GUI plus the processing required on the front-end for screen formatting. This roughly agrees with our network finding of 60ms ping time (with no GUI processing at all), which means that on average, we’re probably seeing ~30ms of time for the SAP GUI to perform it’s rendering.

Based on the above findings I can rule out networking as being a probable cause to the high Roll Wait Time as it seems to be (on average) pretty normal. Although I could recommend to my client that they use the “Low Speed Connection” setting in the SAP GUI as this is recommended by SAP in WAN setups (see SAP note 203924 section 4). It’s also possible I could recommend reverting to the Classic Theme in the GUI. Also recommended in the mentioned note.

SAP note 62418 discusses the typical amount of data sent per Dialog step in SAP ECC 6.0. Around 4.6KB is to be expected (per Dialog step). It doesn’t state if that is a normal connection setup or using the “Low Speed Connection” setup.
We can also look at the relationship of GUI Time to Roll Wait Time. There’s something there that doesn’t look quite right.

If I go back to the Workload Overview, I can see that for DIALOG tasks, the “Average GUI Time” is 90.4ms, which should be almost equal to the “Roll-out” + “Roll Wait” + “Roll In” (these happen whilst the GUI is doing it’s stuff – during RFC).

Except, in my case, I can see that the GUI (plus the time to talk to it) is doing it’s stuff much quicker than the Application Server is rolling out and waiting (NOTE: We’ve had to drag the columns around in ST03 and we’re on the “All Data” tab):

0.4 + 147.7 + 15.8 = 163.9ms.

This is 163.9 – 90.4 = 73.5ms slower (on average, per Dialog step) than I would have expected!
This is ~12% of the response time (the Roll Wait Time is ~26% of the response time).

These are the possible reasons I considered to explain this:

Bad network performance. We’ve looked at this and we can see that GUI Time is actually pretty normal. Why would waiting around take longer than the time to do something. Network performance seems good.
Lack of dialog work processes. No it can’t be this, because this would not be attributable to Roll Wait Time, but instead would be measured as Dispatcher Queue Time (also known as “Wait Time” without the word “Roll”).
Longer time to Roll In and Roll Out. It’s possible that Roll In and Roll Out time could affect the calculations. I’m seeing average Roll In time (per dialog step) of 15.8ms and Roll Out time of 0.4ms. But this still doesn’t add up to 73.5ms.
Time taken to initialise and open the RFC connection to the SAP GUI. It’s possible that the network lookup/hostname buffer is slow to get going before we manage to open the connection to the SAP GUI, but 75ms is pretty slow.

I needed to get out of the aggregated data and into the real nitty gritty.
I needed transaction STAD and some real live data.

Opening STAD, I left the default options with a read time of 10 minutes ago.
Once the records were displayed, I changed the layout of the screen to remove fields I wasn’t interested in, I then added fields I was interested in:
– Started
– Server
– Transaction
– Program
– Type
– Screen
– Work Process
– User
– Response Time
– RFC+CPIC
– Roll Wait
– GUI Time

Once I had the screen setup, it was a simple case of scanning through the records looking for any that had “Roll Wait” > “GUI Time” with 0 “RFC+CPIC”.

I found some!
One of the records (shown below) has a huge “Roll Wait” time of around 5 seconds, yet “GUI Time” is zero:

It just so happens that at the same time as this STAD record was created, I was also running an Enqueue Trace, SQL Trace and Buffer Trace from transaction ST05 (yeah, it was a quiet day so I felt confident that dipping in and out of tracing wouldn’t hurt performance too bad).

So I had an opportunity to display these traces for the same period of time (and the same Work Process number).
I found that there were no long durations in the entire trace. In fact, the total duration of the trace period didn’t add up to even 1 second. What!

Sorting the trace by the “Time” column and manually scrolling through looking for the 5 seconds between one row and the next and sure enough, I found it. The missing 5 seconds:

I single clicked the first record after the time gap, and because it was an ABAP program, I was able to click the “Display Call Positions in ABAP Programs” button:

The source code position that I arrived at didn’t seem to be doing anything other than an OPENSQL statement, so I scrolled back up the program a little.
Then I saw it. The program was moving UNIX files around. Not only that, but there was an ABAP command “WAIT UP TO 5 SECONDS.”:

Here’s what the ABAP syntax help says:

“This statement interrupts the execution of the program by the number of seconds specified… … after the specified time has passed, the program continues with the statement following WAIT.“

It also states:

“Each time the statement WAIT is used, a database commit is performed. For this reason, WAIT must not be used between Open SQL statements that open or close a database cursor.“

SUMMARY:
We’ve seen how “GUI Time” is measured and checked the network performance stats to see how accurate it is.
We’ve also learned how “GUI Time” is actually related in some ways to the value of the “Roll Wait Time”.

It’s been a long hard slog to get to the bottom of why I have a high average “Roll Wait Time” shown in ST03, when the average “GUI Time” is much lower. A hardcoded ABAP statement was causing my work processes to WAIT for a fixed period of time, increasing the overall Response Time reported in ST03. We referenced SAP note 606610 at the beginning of our article, but it seems very difficult to actually find out (without going through ABAP source) if a WAIT statement is has been the cause of Roll Wait Time.

We have subsequently learned that the ST03 Dialog Response Time measurements should be taken lightly, and that you should always try to exclude “GUI Time” by using the “Response Time Distribution” analysis view and clicking the “Dialog Without GUI Time” button. This will exclude the “Roll Wait Time” as described in SAP note

ADDITIONAL NOTES FOR REFERENCE:
During my investigation, I also found a few other things.

We were actually suffering from the program error described in SAP note 1789729, so there is some potential to get better performance from the system by eliminating the additional database & buffer calls.

Some of the records in STAD contained HTTP records.
When I analysed these, I could see that certain document related transactions were calling out to the SAP HTTP Content Server to access documents.
I managed to calculate that the “Call Time” for the HTTP access, was recorded as “Processing Time” in the overall Response Time.
So, if you use a Content Server, be sure to check the response times, as this could also be a factor in slower response times, and this wasted time *is* recorded in the overall Response Time.
Obviously, using SSL will make this process slightly slower, so maybe some form of front-end cache would be better.

Thanks for reading.

SAP Secure Storage CL_HTTPSERVER HMAC

During a system copy, you are advised in SAP note 816861 to migrate the entries in the SAP Secure Store through transaction SECSTORE or report RSECADMIN.
When you run those transactions, you only see message SECSTORE031 for the CL_HTTP_SERVER_NET item which uses HMAC (Message Authentication Code).

If you are simply performing a system copy to an existing system (system refresh) then you do not need to migrate the system just for the CL_HTTP_SERVER_NET item.
As per SAP note 1908221, you can just delete the item CL_HTTP_SERVER_NET from SECSTORE and it will be re-created when the system next starts up.