This blog contains experience gained over the years of implementing (and de-implementing) large scale IT applications/software.

Category: Azure

Azure Disk Cache Settings for an SAP Database on Linux

One of your go-live tasks once you have built a VM in Azure, should be to ensure that the Azure disk cache settings on the Linux VM data disks, are set correctly in accordance with the Microsoft recommended settings.
In this post I explain the disk cache options and how they apply to SAP and especially to SAP databases such as SAP ASE and SAP HANA, to ensure you get optimum performance.

What Are the Azure Disk Cache Settings?

In Microsoft Azure you can configure different disk cache settings on data disks that are attached to a VM.
NOTE: You do not need to consider changing the O/S root disk cache settings, as by default they are applied as per the Azure recommendations.

Only specific VMs and specific disks (Standard or Premium Storage) have the ability to use caching.
If you use Azure Standard storage, the cache is provided by local disks on the physical server hosting your Linux VM.
If you use Azure Premium storage, the cache is provided by a combination of RAM and local SSD on the physical server hosting your Linux VM.

There are 3 different Azure disk cache settings:

  • None
  • ReadOnly (or “read-only”)
  • ReadWrite (or “read/write”)

The cache settings can influence the performance and also the consistency of the data written to the Azure storage service where your data disks are stored.

Cache Setting: None

By specifying “None” as the cache setting, no caching is used and a write operation at the VM O/S level is confirmed as completed once the data is written to the storage service.
All read operations for data not already in the VM O/S file system cache, will be read from the storage service.

Cache Setting: ReadOnly

By specifying “ReadOnly” as the cache setting, a write operation at the VM O/S level is confirmed as completed once the data is written to the storage service.
All read operations for data not already in the VM O/S file system cache, will be read from the read cache on the underlying physical machine, before being read from the storage service.

Cache Setting: ReadWrite

By specifying “ReadWrite” as the cache setting, a write operation at the VM O/S level is confirmed as completed once the data is written to the cache on the underlying physical machine.
All read operations for data not already in the VM O/S file system cache, will be read from the read cache on the underlying physical machine, before being read from the storage service.

Where Do We Configure the Disk Cache Settings?

The disk cache settings are configured in Azure against the VM (in the Disks settings), since the disk cache is both physical host and VM series dependent. It is *not* configured against the disk resource itself, as explained in my previous blog post: Listing Azure VM DataDisks and Cache Settings Using Azure Portal JMESPATH & Bash

Any Recommendations for Disk Cache Settings?

There are specific recommendations for Azure disk cache settings, especially when running SAP and especially when running databases like SAP ASE or SAP HANA.

In general, the rules are:

Disk UsageAzure Disk Cache Setting
Root O/S disk (/)ReadWrite – ALWAYS!
HANA SharedReadOnly
ASE Home
(/sybase/<SID>)		ReadOnly
Database DataHANA=None, ASE=ReadOnly
Database LogNone

The above settings for SAP ASE have been obtained from SAP note 2367194 (SQL Server is same as ASE) and from the general deployment guide here: https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/dbms_guide_general
The use of write caching on the ASE home is optional, you could choose ReadOnly, it would help protect the ASE config file in a very specific scenario. It is envisaged that using ASE 16.0 with SRS/HADR you would have a separate data disk for the Replication Server data (I’ll talk about this in another post).

The above settings for HANA have been taken from the updated guide here: https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/hana-vm-operations-storage which is designed to meet the KPIs mentioned in SAP note 2762990.

The reason for not using a write cache every time, is because an issue at the physical host level, affecting the cache, could cause the application (e.g database) to think it has committed data, when it actually isn’t written to disk. This is not good for databases, especially if the issue affects the transaction/redo log area. Data loss could occur.

It’s worth noting that this cache “issue” has always been true of every caching technology ever created, on which databases run. Storage tech vendors try to mitigate this by putting batteries into the storage appliances, but since the write cache in Azure is at the physical host level, there’s just no guarantee that when the VM O/S thinks the write operation has committed to disk, that it has actually been written to disk.

How About Write Accelerator?

There are specific Azure VM series (M-series at current) that support something known as “Write Accelerator”.
This is an extra VM level setting for Premium Storage disks attached to M-series VMs.

Enabling the Write Accelerator setting is a requirement by Microsoft for production SAP HANA transaction log disks on M-Series VMs. This setting ebales the Azure VM to meet the SAP HANA key performance indicators in note 2762990. Azure Write Accelerator is designed to provide lower latency write times on Premium Storage.

You should ensure that the Write Accelerator setting is enabled where appropriate, for your HANA database transaction log disks. You can check if it is enabled following my previous blog post: Listing Azure VM DataDisks and Cache Settings Using Azure Portal JMESPATH & Bash

I’ve tried my best to find more detailed information on how the Write Accelerator feature is actually provided, but unfortunately it seems very elusive. Robert Boban (of Microsoft) commented on a LinkedIn post here: “It is special caching impl. for M-Series VM to fulfill SAP HANA req. for <1ms latency between VM and storage layer.“.

Check the IOPS

Once you have configured your disks and the cache settings, you should ensure that you test the IOPS achieved using the Microsoft recommended process.
You can follow similar steps as my previous post: Recreating SAP ASE Database I/O Workload using Fio on Azure

As mentioned in other places in the Microsoft documentation and SAP notes such as 2367194, you need to ensure that you choose the correct size and series of VM to ensure that you align the required VM maximum IOPS with the intended amount of data disks and their potential IOPS maximum. Otherwise you could hit the VM max IOPS before touching the disk IOPS maximum.

Enable Accelerated Networking

Since the storage is itself connected to your VM via the network, you should ensure that Accelerator Networking is enabled in your VMs Network Settings:

Checking Cache Settings Directly on the VM

As per my previous post Checking Azure Disk Cache Settings on a Linux VM in Shell, you can actually check the Azure disk cache settings on the VM itself. You can do it manually, or write a script (better option for whole landscape validation).

Summary:

I discussed the two types of storage (standard or premium) that offer disk caching, plus where in Azure you need to change the setting.
The table provided a list of cache settings for both SAP ASE and SAP HANA databases and their data disk areas, based on available best-practices.

I mentioned Write Accelerator for HANA transaction log disks and ensuring that you enable Accelerated Networking.
Also provided was a link to my previous post about running a check of IOPS for your data disks, as recommended by Microsoft as part of your go-live checks.

A final mention was made another post of mine, with a great way of checking the disk cache settings across the VMs in the landscape.

Useful Links:

Windows File Cache

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/premium-storage-performance

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/how-to-enable-write-accelerator

https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/hana-vm-operations-storage#production-storage-solution-with-azure-write-accelerator-for-azure-m-series-virtual-machines

https://petri.com/digging-into-azure-vm-disk-performance-features

https://techcommunity.microsoft.com/t5/running-sap-applications-on-the/sap-on-azure-general-update-march-2019/ba-p/377456

https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/dbms_guide_general

https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/hana-vm-operations-storage

SAP Note 2762990 – How to interpret the report of HWCCT File System Test

SAP Note 2367194 – Use of Azure Premium SSD Storage for SAP DBMS Instance

You may also be interested in:

Checking Azure Disk Cache Settings on a Linux VM in Shell

In a previous blog post, I ended the post by showing how you can use the Azure Enhanced Monitoring for Linux to obtain the disk cache settings.
Except, as we found, it doesn’t easily allow you to relate the Linux O/S disk device names and volume groups, to the Azure data disk names.

You can read the previous post here: Listing Azure VM DataDisks and Cache Settings Using Azure Portal JMESPATH & Bash

In this short post, I pick up where I left off and outline a method that will allow you to correlate the O/S volume group name, with the Linux O/S disk devices and correlate those Linux disk devices with the Azure data disk names, and finally, the Azure data disks with their disk cache settings.

Using the method I will show you, you will see how easily you can verify that the disk cache settings are consistent for all disks that make up a single volume group (very important), and also be able to easily associate those volume groups with the type of usage of the underlying Azure disks (e.g. is it for database data, logs or executable binaries).

1. Check If AEM Is Installed

Our first step is to check if the Azure Enhanced Monitoring for Linux (AEM) extension is installed on the Azure VM.
This extension is required, for your VM to be supported by SAP.

We use standard Linux command line to check for the extension on the VM:

ls -1 /var/lib/waagent/Microsoft.OSTCExtensions.AzureEnhancedMonitorForLinux-*/config/0.settings

The listing should return at least 1 file called “0.settings”.
If you don’t have this and you don’t have a directory starting with “Microsoft.OSTCExtensions.AzureEnhancedMonitorForLinux-“, then you don’t have AEM and you should get it installed following standard Microsoft documentation.

2. Get the Number of Disks Known to AEM

We need to know how many disks AEM knows about:

grep -c 'disk;Caching;' /var/lib/AzureEnhancedMonitor/PerfCounters

3. Get the Number of SCSI Disks Known to Linux

We need to know how many disks Linux knows about (we exclude the root disk /dev/sda):

lsscsi --size --size | grep -cv '/dev/sda'

4. Compare Disk Counts

Compare the disks quantity from AEM and from Linux.  They should be the same.  This is the number of data disks attached to the VM.

If you have a lower number from the AEM PerfCounters file, then you may be suffering the effects of an Azure bug in the AEM extension which is unable to handle more than 9 data disks.
Do you have more than 9 data disks?

At this point if you do not have matching numbers, then you will not be able to continue, as the AEM output is vital in the next steps.

Mapping Disks to the Cache Settings

Once we know our AEM PerfCounters file contains all our data disks, we are now ready to map the physical volumes (on our disk devices) to the cache settings. On the Linux VM:

pvs -o "pv_name,vg_name" --separator=' ' --noheadings

Your output should be a list of disks and their volume groups like so (based on our diagram earlier in the post):

/dev/sdc vg_data
/dev/sdd vg_data

Next we look for a line in the AEM PerfCounters file that contains that disk device name, to get the cache setting:

awk -F';' '/;disk;Caching;/ { sub(/\/dev\//,"",$4); printf "/dev/%s %s\n", tolower($4), tolower($6) }' /var/lib/AzureEnhancedMonitor/PerfCounters

The output will be the Linux disk device name and the Azure data disk cache setting:

/dev/sdc none
/dev/sdd none

For each line of disks from the cache setting, we can now see what volume group it belongs to.
Example: /dev/sdc is vg_data and the disk in Azure has a cache setting of “none”.

If there are multiple disks in the volume group, they all must have the same cache setting applied!

Finally, we look for the device name in the PerfCounters file again, to get the name of the Azure disk:

NOTE: Below is looking specifically for “sdc”.

awk -F';' '/;Phys. Disc to Storage Mapping;sdc;/ { print $6 }' /var/lib/AzureEnhancedMonitor/PerfCounters

The output will be like so:

None sapserver01-datadisk1
None sapserver01-datadisk2

We can ignore the first column output (“None”) in the above, it’s not needed.

Summary

If you package the AEM disk count check and the subsequent AEM PerfCounters AWK scripts into one neat script with the required loops, then you can get the output similar to this, in one call:

/dev/sdd none vg_data sapserver01-datadisk2
/dev/sdc none vg_data sapserver01-datadisk1
/dev/sda readwrite

Based on the above output, I can see that my vg_data volume group disks (sdc & sdd) all have the correct setting for Azure data disk caching in Azure for a HANA database data disk location.

Taking a step further, if you have intelligently named your volume group names, you then also check in your script, the cache setting based on the name of the volume group to determine if it is correct, or not.
You can then embed this validation script into a “custom validation” within SAP LaMa and it will alert you automatically if your VM disk cache settings are not correct.

You may be wondering, why not do all this from the Azure Portal?
Well, the answer to that is that you don’t know what Linux VM volume groups those Azure disks are used by, unless you have tagged them or named them intelligently in Azure.

You may also be interested in:

List Your Azure VMs in Excel – Part 3

The third and final part to the trilogy.
I show you how to list your Azure VMs in Excel O365 using Power Query, and this time we enhance the code even further using the Power Query function List.Accumulate, to accumulate the returned list of VMs from each subscription into one big list.

Like an episode of a tacky day-time television show, it’s always worth wasting some words on a recap.
So, in-case you’re not aware, here’s what has happened in this saga so far:

List Your VMs in Excel – part1
The first part of the trilogy, showed how to create a new Power Query in Excel O365, and how to enter the code, which generated a basic VM list from Azure.

List Your VMs in Excel – part2
The second part enhanced the code slightly, parameterising more of the text strings.
It also introduced the ability to get the powerState of each VM from Azure, allowing you to see in Excel, which VMs were running and which were deallocated.

By applying the code changes from this post, you will no longer need multiple Power Query queries, going from 1 per Azure subscription, to just 1 query for all subscriptions.

What’s New?

As mentioned, the code now includes the use of the Power Query “List.Accumulate” function to combine the lists of VMs from multiple subscriptions.

I’ve never really use Power Query before, so even I had no idea how to loop on a list of values and execute a function on each loop. After a bit of searching I found that the List.Accumulate function did exactly what I needed it to do, with minimal coding needed.

Here’s a pretty comprehensive description of how List.Accumulate works: https://datachant.com/2016/06/02/power-query-list-accumulate-unleashed/

What you will notice is that our code has the following properties:
subscriptions = Our defined list of Azure subscriptions.
{} = Our blank list as a seed.
List.Combine = Is executed for each entry in our subscriptions list.

List.Accumulate(subscriptions, 
                {},
                (state,current)=>
                              List.Combine({state,FnGeneratedVMList(current)}))

We define the subscriptions list right at the start of the code:

subscriptions = {"[your subscription1]","[your subscription2]"} as list,

To make our code work with the List.Accumulate, we have changed the “GeneratedVMList” variable contents, to be a function, instead of a string:

// FnGeneratedVMList pages through the subscription and gets the VM lists. 
FnGeneratedVMList = (subscription as text) as list => 
 List.Generate( ()=>[i=0, 
  res = FnGetOnePage(endPoint & "/subscriptions/" & subscription & "/providers/Microsoft.Compute/virtualMachines?api-version=" & apiVersion)], 
  each [i]null, 
  each [i=[i]+1, 
  res = FnGetOnePage([res][Next])], 
  each [res][Data]),

The main benefit of the function is that it can now be passed a parameter “subscription“, which is used to adjust the URI to the Azure API for the correct subscription.

The End Result

The end result of our changes is the following code:

let 
 iterations = 10 as number, 
 // Max Number of Pages of VMs. 
 endPoint = "https://management.azure.com" as text, 
 subscriptions = {"[your subscription1]","[your subscription2]"} as list, 
 apiVersion = "2019-07-01" as text,
 
// FnGetOnePage is the function that performs an import of single page. 
// The page consists of a record with the data and the URL in the 
// fields data and next. Other Web APIs hold the data and cursor in different formats 
// but the principle is the same. 
FnGetOnePage = (url) as record => 
 let Source = Json.Document(Web.Contents(url)), 
 data = try Source[value] otherwise null, 
 next = try Source[nextLink] otherwise null, 
 res = [Data=data, Next=next] 
in 
 res,
 
// FnGetVMdisplayStatus gets the instanceView object for the passed VM ID 
// then parses out the displayStatus from one of two possible locations. 
FnGetVMdisplayStatus = (idURI) as text => 
 let Source = Json.Document(Web.Contents(endPoint & idURI & "/instanceView?api-version=" & apiVersion)), 
 statuses = Source[statuses], 
 vmDisplayStatus1 = try statuses{1}[displayStatus] otherwise "", 
 vmDisplayStatus2 = try statuses{2}[displayStatus] otherwise "", 
 vmDisplayStatus = vmDisplayStatus1 & vmDisplayStatus2 
in
 vmDisplayStatus,

// FnGeneratedVMList pages through the subscription and gets the VM lists. 
FnGeneratedVMList = (subscription as text) as list => 
 List.Generate( ()=>[i=0, 
  res = FnGetOnePage(endPoint & "/subscriptions/" & subscription & "/providers/Microsoft.Compute/virtualMachines?api-version=" & apiVersion)], 
  each [i]null, 
  each [i=[i]+1, 
  res = FnGetOnePage([res][Next])], 
  each [res][Data]),

// SubscriptionsVMList combines the returned lists using the Accumulator. 
SubscriptionsVMList = 
 List.Accumulate(subscriptions, {},(state,current)=>List.Combine({state,FnGeneratedVMList(current)})),
 
#"VMListTable" = Table.FromList(SubscriptionsVMList, Splitter.SplitByNothing(), null, null, ExtraValues.Error), 
#"Expanded-VMListTable-Column1" = Table.ExpandListColumn(#"VMListTable", "Column1"), 
#"VMdetail-list" = Table.ExpandRecordColumn(#"Expanded-VMListTable-Column1", "Column1", {"name","id"}), 
#"VMdetail-list-with-displayStatus" = Table.AddColumn(#"VMdetail-list", "displayStatus", try each FnGetVMdisplayStatus([id]) otherwise "??") 
in 
#"VMdetail-list-with-displayStatus"

You will need to adjust [your subscription1] and [your subscription2] with your subscriptions, adding any additional subscriptions in the same format.

If you only have one subscription, you can still use this code, just remove one of the items from the subscriptions list, like so:

subscriptions = {"[your subscription1]"} as list,

Summary

Hopefully you can see how you can enhance the code slightly to include more VM details if you need them.
As an example, this line here can be adjusted to include additional columns as well as “name” and “id” (for example: “location“), as per the VM list API call (see the first part of this trilogy for more detail):

#"VMdetail-list" = Table.ExpandRecordColumn(#"Expanded-VMListTable-Column1", "Column1", {"name","id"}),

You can now enjoy your data in Excel.

Update 23-June: Due to popular demand, here is how the above code can be changed to include the “location” and “vmSize” fields, which will depict the location and series/size of the VM.
We make a small change to the last few lines like so:

#"VMListTable" = Table.FromList(SubscriptionsVMList, Splitter.SplitByNothing(), null, null, ExtraValues.Error), 
"Expanded-VMListTable-Column1" = Table.ExpandListColumn(#"VMListTable", "Column1"),
#"VMdetail-list" = Table.ExpandRecordColumn(#"Expanded-VMListTable-Column1", "Column1", {"name","id", "location", "properties"}),
#"VMdetail-list-properties" = Table.ExpandRecordColumn(#"VMdetail-list", "properties", {"hardwareProfile"}, {"hardwareProfile"}),
#"VMdetail-list-properties-hardwareProfile" = Table.ExpandRecordColumn(#"VMdetail-list-properties", "hardwareProfile", {"vmSize"}, {"vmSize"}),
#"VMdetail-list-with-displayStatus" = Table.AddColumn(#"VMdetail-list-properties-hardwareProfile", "displayStatus", try each FnGetVMdisplayStatus([id]) otherwise "??"),
in 
#"VMdetail-list-with-displayStatus"

Ultimate Active-Active SAP Web Dispatcher Architecture in Azure?

I have never been fully satisfied with the reference architecture on the Microsoft site for running active-active SAP Web Dispatchers in an Azure IaaS platform.

Don’t get me wrong, from a high-level Azure perspective they are representative of what you will be desiring. However, they just lack enough detail to make you really think about the solution more than I feel you should need to.

To re-cap, in an Active-Active SAP Web Dispatcher in Azure, you rely on the inherent capabilities of the Azure Internal Load Balancer (ILB) to provide availability routing to the active-working VMs that are running the Web Dispatcher instances.

To help you (and me) understand what needs to be configured, I’ve put together what I feel is a pretty good low-level architecture diagram.

It’s almost a version two to SAP on Azure Web Dispatcher High Availability.

Show Us the Picture or It Never Happened!

Below is the diagram that I have created.
There is quite a lot of detail in it, and also quite a lot of detail that is not included (required O/S params, instance params and config for the network layer etc). It is really not as simple as you first imagine to include the required detail in one picture.

It Happened, Now Explain Please

If we look at the diagram above, we can easily see that WD1 is the SAP System name, with 2 instances of WD1, both with an instance number of 98 but installed against 2 virtual hostnames of sapwd1 and sapwd2.

Could we have installed on the server hostname directly? Yes, we could have. But that is not inline with a SAP Adaptive Computing Design principal, which is separation of the SAP instance from the host.

Notice that we have a Highly Available NFS share that hosts our SAP system instance profile files and a single shared PSE (SAPSSLS.pse).
We don’t show that this could be from a HA fileshare or NetApp or some other technology, but please use your imagination here. For production the Azure Files service is not currently supported.

Our ILB is configured to accept HTTP and HTTPS connections on their native ports (80 and 443) and it routes these through to the 8098 and 44398 ports that the Web Dispatchers are configured to listen on. You can configure whatever ports you want, but ultimately, having separately addressable back-end ports allows you to re-use the SSL port for Web Dispatcher administration and tie-down the access to a DMZ hosted Jump Box (definitely not on the diagram).

The ILB is probing both back-end VM members on tcp/8098 to see if the Web Dispatcher is responding. It’s a basic TCP SYN check (can I open a connection – yes, OK). For a better check, you can use a HTTP health probe on tcp/8098, which would allow you to set the Web Dispatcher to “maintenance” mode, causing a HTTP “service unavailable” response to be returned to the ILB probe, which would remove that particular Web Dispatcher from the ILB routing. If you also followed the other suggestion of accessing the admin page from the 44398 port via the virtual hostname, then you will see that an administrator would still have admin control for maintenance purposes. Nice.

We have a SAN enabled SSL certificate inside our shared PSE, with 3 Common Names associated with that certificate, one for the ILB “host” name (sapwd), and 1 for each of the virtual hostnames against which we have installed the Web Dispatcher instances (sapwd1 and sapwd2).

Our “icm/host_name_full” parameter sets both Web Dispatchers to think that they are accessed through sapwd.corp.net. However, we have to be careful that we do not use EXTBIND in this particular case, because we do not have the IP address of the ILB bound onto the servers (although if you read my post on how to add a secondary IP address on the Loopback device I can show you how it’s possible to do this and why you may want to).

How Do We Cater for DR?

Because we do not have a high disk I/O throughput on a Web Dispatcher VM, it is perfect to be protected by Azure Site Recovery (ASR).

This means the VM is replicated across to the Azure DR region (the region of your choice).

Like this:

But wait, we’re only replicating 1 VM! Yes, we don’t need to pay for both, since a cost-optimised approach would be to just re-install the second Web Dispatcher after a DR failover.

We have a dependency on some sort of NFS share replication to exist in the DR region, but it doesn’t necessarily need to be fancy in the case of the SAP Web Dispatcher, because very little will be changing on the /sapmnt/<SID> area.

NOTE: The replicated VM is not accessible until a failover is instigated.

What Happens In a Failover to DR

In a DR scenario, the decision to failover to the DR region is manual.
The decision maker can choose to failover as soon as the primary region is unavailable, or they can choose to wait to see if the primary region is quickly recovered by Microsoft.

I’ve put together a diagram of how this could affect our simple HA Web Dispatcher setup:

The decision to failover should not be taken lightly, because it will take a lot of effort to failback (for databases).

Generally the recommendation from Microsoft is to use an Azure Automation Runbook to execute a pre-defined script of tasks.

In our case, the runbook will create the ILB above the Web Dispatcher VM and add the replicated VM to the ILB.
Our runbook will also then add secondary IP addresses to the VM and finally adjust DNS for all our hostnames and virtual host names, assigning the new IP addresses to the DNS records.

Once our Web Dispatcher is online and working, we could choose to then build out a further VM and add it into the ILB back-end pool, depending on how long we think we will be living in the DR region.

Summary

Did we successfully include more detail in the architecture diagram? Yes we sure did!
Was it all the detail? No. There’s a lot that I have not included still.
Will I be enhancing this diagram? Probably; I hate leaving holes.

I’ve shown above how an active-active SAP Web Dispatcher architecture can work in Azure and how that could be setup for a DR protection.

We also briefly touched on some good points about separation of administration traffic, using a HTTP health probe for an ILB aware Web Dispatcher maintenance capability, and how the SSL setup uses a SAN certificate.

Would this diagram be more complicated by having an active-active HA Web Dispatcher alongside an ASCS or SCS? Yes, it gets more complicated, but there are some great features in the ILB that allow simplification of the rules which allow you to use the ILB for more than one purpose, saving cost.

Update Jun-2020: This duplicate Web Dispatcher architecture is known in SAP as “Parallel Web Dispatcher” and a basic description is visible here: https://help.sap.com/viewer/683d6a1797a34730a6e005d1e8de6f22/1709%20002/en-US/489a9a6b48c673e8e10000000a42189b.html

Update Mar-2021: Some of you have asked about how the “Maintenance Mode” activation works with the ILB. This is siply that the WDisp returns a HTTP 503 when Wdisp maintenance mode is enabled.
By default the ILB health probe will be “http://<the-vm>:<your-port>/”, but if you don’t have a back-end service allocated to “/” then you will get a HTTP 404 constantly. You need to adjust the URL to an actual working URL location based on the config of your back-end systems.
If you don’t want the health probe to make a call to an actual back-end system (during the health probe ping) then use parameter “icm/HTTP/file_access_<xx>” to define a custom local directory and place a blank file called “health.htm”. Then just adjust the health problem URL with the path to the “health.htm” and the health probe pings will never call a back-end system URL. It also means that you can touch or remove the health.htm to permit the ILB to use or not use that specific WDisp.

SAP Instance Agent as Mini-Web Server

In this post, I will show you how a little known feature of the SAP Instance Agent (available on every “modern” SAP system) can be used to serve files via HTTP and HTTPS.

You may be thinking, “This is basic stuff, I know this already“, in which case, you may only be interested in the very last paragraph of this post 😉

Why Not Just Use a Real Web Server?

During SAP projects there always comes a point where software needs to be distributed throughout the landscape, where end-users need access to a predefined set of software, or where scripts need to be centralised and downloadable onto multiple target servers.
Essentially, you need a common file distribution point.

Some projects have large budgets and some have small budgets. This post is for those with small budgets. Those projects where using everything twice, if possible, becomes an artform.

Under What Circumstances Would I Possibly Want to Use Such a Method?

Let’s imagine a scenario where you have a set of Korn shell scripts that you would like centralised across the SAP landscape.
There is/was no budget for a common fileshare and there is no budget to have someone setting one up.
Instead you develop an offline deployment approach, where the scripts are pulled down from a central repository on a schedule.
You decide that the central repository needs to be a web server and the scripts will be downloaded by HTTP.

What Is the SAP Instance Agent?

The SAP Instance Agent is the agent that does the work, when you run sapcontrol functions.
It is a small set of executables that come as part of the SAP Kernel and generally you get one Instance Agent installation per SAP instance.

For example, if you have an ASCS instance, there will be an instance agent installed under /usr/sap/<SID>/ASCS<##>/.

You can see the Instance Agent running by querying the list of running processes with ps:

>ps-ef | grep sapstartsrv

as1adm     1969      1  0 06:31 ?        00:00:01 /usr/sap/AS1/ASCS00/exe/sapstartsrv pf=/usr/sap/AS1/SYS/profile/AS1_ASCS00_sapas1ase1 -D -u as1adm

You can see that the binary executable responsible is “sapstartsrv”.

You will also notice that the SAP Host Agent also has a “sapstartsrv”. This is because the Host Agent and the Instance Agent are siblings, sharing a similar code-set, just with different functions.

How Do You Access the SAP Instance Agent?

In the history of the SAP product range, SAP created a Java based GUI tool called SAP MMC. The SAP MMC can be used to administer SAP instances on the local server via the Instance Agent. To be able to start the SAP MMC, it was distributed from the Instance Agent by HTTP. I’m not going to go into the SAP MMC because it will be going away completely eventually.

Generally, whenever you run sapcontrol and call a function, you are accessing the SAP Instance Agent:

sapcontrol -nr 00 -function GetSystemInstanceList

07.05.2020 07:01:32
GetSystemInstanceList
OK
hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
sapas1ase1, 0, 50013, 50014, 1, MESSAGESERVER|ENQUE, GREEN

What Is the Document Root For the Instance Agent?

In Web Server lingo, the “document root” is the highest level directory that the web server can serve files from.

For the Instance Agent, this is /usr/sap/<SID>/<INST>/exe/servicehttp.
Example: /usr/sap/AS1/ASCS00/exe/servicehttp.

If we create a simple text file in the document root, we can access it via HTTP:

echo "Hello Darryl" > /usr/sap/AS1/ASCS00/exe/servicehttp/index.html

We can use wget to access the file like so:

wget -q -O - http://127.0.0.1:50013/index.html

Hello Darryl

Can We Use HTTPS?

You can use HTTPS, the TCP port is the secure port 5##14 (50014 in our example).
Because the certificate used by the SAP Instance Agent is self-signed, you will need to trust the certificate directly.

We can use wget again as follows, but with the secure port and telling wget to not check the SSL certificate:

wget --no-check-certificate -q -O - https://127.0.0.1:50014/index.html

Hello Darryl

Are the Files Persisted Forever?

The files that you create under the document root of the SAP Instance Agent, are not persisted indefinately.
They get removed when the SAP instance is started.
Do not confuse this with when the Instance Agent is started.

An example, the Instance Agent is started up automatically by sapinit on server boot. Only when the SAP instance that is served by the Instance Agent, is started (i.e. sapcontrol -nr 00 -function Start), do the files in the document root location get cleansed, just before the instance is started.

I have previously put together a nice diagram about the interactions of various components during SAP instance startup. See the post here:

How an Azure hosted SAP LaMa Controlled SAP System Starts Up

How Can We Make Our Files Persist?

You can make your files persist, by creating a filesystem soft link from the document root and re-creating this link as part of the SAP instance start process:

mkdir /home/as1adm/docroot
echo "Hello Darryl" > /home/as1adm/docroot/index.html
ln -s /home/as1adm/docroot /usr/sap/AS1/ASCS00/exe/servicehttp/

Then you need to add an entry to the SAP instance profile to re-create the link on startup.
FIrst we need to establish the current number of “Execute” tasks in the profile:

cdpro
grep Execute_ *

AS1_ASCS00_sapas1ase1:Execute_00 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG0)
AS1_ASCS00_sapas1ase1:Execute_01 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG1)
AS1_ASCS00_sapas1ase1:Execute_02 = local rm -f $(_MS)
AS1_ASCS00_sapas1ase1:Execute_03 = local ln -s -f $(DIR_EXECUTABLE)/msg_server$(FT_EXE) $(_MS)
AS1_ASCS00_sapas1ase1:Execute_04 = local rm -f $(_EN)
AS1_ASCS00_sapas1ase1:Execute_05 = local ln -s -f $(DIR_EXECUTABLE)/enserver$(FT_EXE) $(_EN)

We add a new entry as follows:

echo "Execute_05 = local ln -s /home/as1adm/docroot $(DIR_EXECUTABLE)/servicehttp/" >> AS1_ASCS00_sapas1ase1

Upon starting the ASCS instance, the profile is read and the link created.

You would then access the index.html as follows:

wget --no-check-certificate -q -O - https://127.0.0.1:50014/docroot/index.html

Voila!

Even if you don’t use the above method for serving basic file content over HTTP, there is another use if you are running your SAP system in Azure.
We can use the HTTP capability of the SAP Instance Agent as a method to dynamically control which back-end VMs are accessible through an Azure ILB. This is an interesting concept, especially when introduced with a couple of other posts I have written over the past year.
I will elaborate further on this over the next few months.