Tag: SAP

SAP ASE does a bare minimum of rotating its text log files.
I’m talking about the logs that you can use to see what’s going on under the covers.
Each time the ASE instance is restarted, it pushes the previous instance log file to a date stamped file and starts fresh. This is not great if you have had issues with the database and the log has grown huge, because who restarts the database every day?

ASE also does not preserve the backupserver log or the job server agent log, which could be critical in solving problems.
Instead, it’s better to devise your own custom daily log rotation process.

In this post I describe the main SAP ASE log files, and my recommendation for a rotation configuration on Linux (yay!) and Windows (here be dragons!).

What Needs to Be Rotated?

The following are standard ASE 16.0 text log files that just keep on growing:

• ASE database instance log: /sybase/<SID>/ASE-16_0/install/<SID>.log
• ASE backup server log: /sybase/<SID>/ASE-16_0/install/<SID>_BS.log
• ASE Job Server log: /sybase/<SID>/ASE-16_0/install/<SID>_JSAGENT.log

I do not include the dumphist file, which (if dumphistory is enabled) gets backed up daily from /sybase/<SID>/ASE-16_0/dumphist to /sybase/<SID>/ASE-16_0/dumphist.nnnnnnnnn.
These can also accumulate, so it might be best to just clear them down, especially if (as is good practice) you copy them off with your database backups.

NOTE: For ASE instances with HADR installed (a.k.a SAP Replication Server) there are also an additional 4 log files which should be rotated, that I won’t go into in this post. (Hire me, and I’ll tell you).

What Happens When ASE Rotates Those Logs?

When the ASE instance is restarted, some of the above standard logs are rotated and some are cleared down. For the ones that are rotated, there is no end to the rotation, they keep building up. They are never removed.

As part of your log rotation solution, you should also rotate the ASE rotated logs, making sure that they don’t build up over the years of ASE instance restarts (especially if starting/stopping systems frequently in The Cloud).

This is what happens to the auto-rotated logs:

• <SID>.log is moved to <SID>.log.nnnnnnnn_nnnnnn
• <SID>_BS.log is cleared down.
• <SID>_JSAGENT.log is cleared down.

We can see that for those log files that are cleared down, a daily rotation should capture them, but it will be hit and miss. If the ASE instance is restarted part way through the day, then the morning logs will not have been preserved (unless you remember to manually run your log rotation). There’s not much we can do about this.

For the log that is auto-rotated by ASE (on restart), then we also need to capture that in our rotation schedule. We have a choice to maybe just compress it and only retain a specific number.

What Log Retention Works Best?

For production systems, I usually say 30 rotations is ideal (1 month).
It’s good to keep the logs so you can see when a problem may have started. Critically, this might help you decide how far back you may need to restore a database (in the dire event of corruption). If you keep 30 days of database backups, then having the <SID>_BS.log for 30 days might just help.

Rotation Schedule

With tools like logrotate, you can decide to rotate logs on a time basis, or even on a size basis.

Since SAP ASE automatically rotates the <SID>.log file on instance restart, it would get a bit messy if we also have a daily rotation on top of this.
Instead, it works a little bit better to rotate this file on a size basis. Keeping the <SID>.log within a defined file size means that you are able to prevent the log file from growing huge, but also reducing the number of rotations within rotations.

We’ve already mentioned that the ASE backup server log is quite important for helping us to understand when backups were taken and any issues during that backup. This is the reason we rotate this file daily, no matter what size it is.

The ASE job server is a little less critical, so we really don’t need it rotated every day. However it can grow quite big if the job server experiences a database space issue. So instead we can choose to rotate on size.

With the above in mind, this is what I would recommend for the ASE log files:

• <SID>.log rotated on size >5MB with 30 rotations.
• <SID>_BS.log rotated daily with 30 rotations.
• <SID>_JSAGENT.log rotated on size>5MB with 30 rotations.

For ASE rotated copies of <SID>.log (called <SID>.log.nnnnnnnn_nnnnnn), it’s best to retain something like 10 rotations for production systems.
This captures the logs from the last 10 restarts. If you are restarting the ASE instance every day, then you may wish to retain more than 10.

Log File Truncation

During the rotation process, you have an option to keep the existing log file or truncate it. My preference is to truncate the original log file, to prevent the log from growing.
If you don’t truncate the original, then you will have a cumulative collection of logs for 30 days, with each day’s copy being bigger than the last and including the previous day’s content.

The exception to this rule is obviously the ASE auto-rotated copies of the <SID>.log. We don’t want to be truncating that at all.

Log File Compression

After the copy of the logs has been taken, the copy can then be compressed. This makes complete sense, because text files (these are ASCII text logs) compress really, really well. You should expect ~90% compression ratios.

Just make sure that you chose a compression tool/utility that has an easy method to access the compressed contents.

For example, in Linux I can use gzip. Then if I need to scan the log for a particular error I can use zcat and pipe through grep to check the contents (like this: “zcat <theGZlog>|grep error”).
I would not want to first uncompress the log, then go through it. That would be painful.

On Windows, I would need to use PowerShell to stream the zipped contents and pattern match. So a script or a separate utility would be needed to help with this.

Log Copy Naming

Once the log file has been copied to a new file and compressed, the name of the copied log file should be appended with the date time stamp.
This allows you to easily identity from when a log file was copied.
Just bear in mind that if you run your log rotation at exactly midnight, the date time stamp will be for the current day, but the log contents will be for the previous day!

The exception to this rule is the ASE auto-rotated copies of <SID>.log.
These copies already have a date time stamp on the end, so we don’t need another one.

Sample Log Rotation Config

Here’s a sample logrotate configuration file that includes all that we have mentioned above.
As part of the logrotate setup we have a designated “status” file which is used by logrotate to record its current state of play, in preparation for its next execution.

The call to logrotate looks like this: logrotate –state <state-file> <config-file>

# logrotate configuration file. 
# This file defines the config for which logrotate uses.
################################################ 
# The global defaults are established first.
# compress all files using default gzip -9. 
compress
# Dont rotate it if its empty. 
notifempty
# Ignore if missing. 
missingok
# Add date as extension instead of just a number. 
dateext 
################################################
# Each section below is specific to a log file match. 
# The contents of the section can override the global defaults.
#
# The ASE dataserver log. 
/sybase/[A-Z][0-9][A-Z]/ASE-[0-9][0-9]*/install/[A-Z][0-9][A-Z].log { 
# Truncate the original log once rotated. 
copytruncate 
# Permit 30 rotations. 
rotate 30 
# Rotate the file based on size. 
size 5M 
}
#
# The auto-rotated ASE dataserver log after a restart of ASE. 
/sybase/[A-Z][0-9][A-Z]/ASE-[0-9][0-9]*/install/[A-Z][0-9][A-Z].log.[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9] { 
# We just zip this file. 
# We dont truncate it. 
nocopytruncate 
# We dont copy it. 
nocopy 
# We dont add the date on the end, its already there. 
nodateext 
# Check daily for these files. 
daily 
# Permit 10 rotations = 10 restarts of ASE. 
rotate 10 
}
#
# The ASE backup server log. 
/sybase/[A-Z][0-9][A-Z]/ASE-[0-9][0-9]*/install/*_BS.log { 
# Truncate the original log once rotated. 
copytruncate 
# Permit 30 rotations. 
rotate 30 
# Rotate the file daily. 
daily 
}
#
# The ASE jsagent server log. 
/sybase/[A-Z][0-9][A-Z]/ASE-[0-9][0-9]*/install/*_JSAGENT.log { 
# Truncate the original log once rotated. 
copytruncate 
# Permit 30 rotations. 
rotate 30 
# Rotate the file based on size. 
size 5M 
}

The above is just the start.
As I mentioned right at the beginning, if you have an ASE instance with HADR configured, then you should also include the 4 other log files (1 for RMA and 3 for RepServer).

There is also the issue of the <SID>.cfg configuration file which is copied to <SID>.nnnnnnnnn on each ASE instance configuration change. You can have a lot of these files build up.
It’s good practice to include this file in your backup regime, but you can also include it in a rotation schedule and remove the ASE created copies.

Executing File Removal

In the logrotate configuration file, in every log rotation file block, it is possible to include a script block that gets executed every time a successful file rotation is performed.
In the logrotate manual, look at the “lastaction” block and include the removal of old dumphist files in there using the “rm” command.
A word of warning, when using these script blocks, logrotate has the ability to pass the path and name of the rotated file as the 1st parameter (i.e. as “$1”). I have found that in my specific version it includes a space on the end. Rotating “somefile.ext.001” and then having a script block in “lastaction” with a pattern like “rm $1.[0-9][0-9][0-9]” would actually remove the file passed in as “$1” (somefile.ext) plus anything that it finds that matches pattern “.[0-9][0-9][0-9]”. Quite problematic and dangerous.
Instead I found it more reliable to code the file pattern manually just like the main part of the file block.

Rotation Fun with Windows

Contrary to the header, it’s not fun to try and rotate logs on Windows.
There are generally 2 issues I’ve found:

1. Windows locks files and can cause issues with log file truncation unless you use a utility able to handle this (LogRotateWin can).
2. There is no built-in log rotation utility on Windows.

If you have a heterogeneous landscape you generally do not want to be maintaining too many different tools and configurations.
I needed to rotate ASE logs on Linux and Windows, so I found that using the free LogRotateWin sort of worked well. It is mostly compatible with the same configuration options as the Linux version.
It does not support all the configuration settings of the Linux version. Example, “notifempty” is not supported and throws an error in the debug output.
It also does not like pattern matching properly. What it seems to do is traverse through the whole directory structure from the moment it sees a wildcard, looking for files at each branch, instead of following my specified pattern.
An example being pattern: F:\sybase\???\ASE-16_0\???.log.
The LogRotateWin tool seems to look through all directories under “F:\sybase”, trying to match *files* and folders with wildcard “???”, when it is clearly a folder that I have specified.
For this reason, I actually found it far more reliable to use PowerShell to transform an existing configuration file and find and replace those wildcards with the actual file names.
I used PowerShell to find the files I needed to be rotated and adjust a temporary version of the LogRotateWin configuration.
In fact this worked so well, that I went a step further! I created some PowerShell to download my Linux logrotate configuration file, adjust it to make it LogRotateWin compatible by swapping the slash directions, pre-finding the files (by traversing the system drives) and replacing the wildcards then also removing unsupported features like “notifempty”.
It also swapped my Linux “rm” syntax with a call to “pwsh” (PowerShell) and the short-hand call to the “remove-item” (rm) command-let.
The PowerShell script then calls the LogRotateWin with the temporary configuration file.

Not fun, but it works and it means I only have 1 configuration file I need to worry about.

Summary:

A good log rotation strategy for SAP ASE databases is important to maintain the ASE text log files effectively.
It’s not just about disk space management and housekeeping, but it is also about retaining vital information to help resolve issues and having easy and quick access to that information.

Maintaining a consistent approach across heterogeneous landscapes is not easy, and may require you to be creative in your method.
You should seek to harmonise configuration as much as possible and reduce maintenance effort.

As part of the ASE log rotation configuration, you should also look to include the dumphist file and also the HADR log files (from SRS and RMA).

A final comment: In some cases, I’ve witnessed an ASE system error which has caused thousands of retries to execute “sp_configure” which caused thousands of versions of the <SID>.cfg file to be created. So many, that it took multiple executions of “rm” to remove them.
You may wish to look at including these <SID>.cfg file copies in a rotation configuration as a preventative measure.