This blog contains experience gained over the years of implementing (and de-implementing) large scale IT applications/software.

HowTo: Find the Datacentre Region and Physical Host of your Azure VM

With VMs hosted in Azure you need a fine balance between protection from hardware failure on the underlying Azure platform, plus performance from having the tiers of your SAP application being physically close together.

For this very purpose, Microsoft introduced Proximity Placement Groups (PPGs) to allow an administrator to ensure that specific tiers (e.g. application and database) are located close. Potentially even in the same server rack.
The PPGs also affect the location of the storage assigned to the VMs, although the storage infrastructure is actually transparent to administrators.

The PPGs still allow Azure to honor the Availability Sets, Fault Domains and Update Domains.

In this post, I show a method of finding the physical hostname of your Linux VM which could be part of a check before/after implementing a PPG.
NOTE: PPGs should be created at the time a VM is created, and assigned to the “lead” system of the rarest size. Example, an M-series VM is rare, so this should be the lead system when creating the PPG. This will anchor the other VMs to this M-series VMs location.

A separate post shows how to do this for a Windows VM.
On a Linux VM in Azure, as any Linux user, you can use the following to see the name of the physical host on which your VM is running:

awk -F 'H' '{ sub(/ostName/,"",$2); print $2 }' /var/lib/hyperv/.kvp_pool_3

Example output: DUB012345678910

In this case, we take the first 3 chars to be “Dublin”, which is in the EU North Azure region.
The remaining characters consist of the rack and physical hostname.

If you have 2 VMs in the same rack on the same physical host, then you will have minimal latency for networking between them.

Conversely, if you have 2 VMs on the same physical host, you are open to HA issues.

Therefore, you need a good balance for SAP.
You should expect to see SAP S/4HANA application servers and HANA DBs in the same Proximity Placement Groups, within the same rack, even potentially on the same host (providing you have availability sets across the tiers you will be safe).

Update: 23-Apr-2020
To get the above script output into a bash variable, the output contains hidden characters, we can use the following:

awk '{ gsub(/[^[:print:]]/,""); split ($0,a,"H"); sub(/ostName/,"",a[2]); print a[2]}' /var/lib/hyperv/.kvp_pool_3

Update: 05-Oct-2020
I have since found that there is another location where the above information can also be found.
Depending on your Linux O/S, you may also find the physical server name in the network scripts as follows:

grep BOOTSERVERNAME /var/run/netconfig/eth0/netconfig0

The aboe will return something like:
BOOTSERVERNAME=’AMS072nnnnnnnnn’

Korn Shell vs Powershell and the New AZ Module

Do you know Korn and are thinking about learning Powershell?

Look at this:

function What-am-I {
   echo “Korn or powershell?”
}

what-am-i
echo $?

Looks like Korn, but it also looks like Powershell.
In actual fact, it executes in both Korn shell and Powershell.

There’s a slight difference in the output from “$?” because Powershell will output “True” and Korn will output “0”.
Not much in it really. That is just another reason Linux people are feeling the Microsoft love right now.

Plus, as recently highlighted by a Microsoft blog post, the Azure CLI known as “az” which allows you to interact with Azure APIs and functions, will now also be the name of the new Powershell module used to perform the same operations and replacing “AzureRM”.

It makes sense for Microsoft to harmonise the two names.
It could save them an awful lot of documentation because currently they have to write examples for both “az” CLI and Powershell cmdlets for each new Azure feature/function.

SUSE Linux 12 – Kernel 4.4.73 – Boot Hang – BTRFS Issue

I had a VMWare guest running SUSE Linux 12 SP3 64bit (kernel 4.4.73).
One day after a power outage, the VM failed to boot.
It would arrive at the SUSE Linux “lizard” splash screen and then just hang.

I noticed prior to this error that the SUSE 12 operating system creates it’s root partition inside a logical volume call “/dev/system/root” and it is then formatted as a BTRFS filesystem.

At this point I decided that I must have a corrupt disk block.
I launched the VM with the CDROM attached and pointing at the SUSE 12 installation ISO file.
While the VM starts you need to press F2 to get to the “BIOS” boot options to enable the CDROM to be bootable before the hard-disks.

Once the installation cdrom was booting, I selected “Recovery” from the SUSE menu.
This drops you into a recovery session with access to the BTRFS filesystem check tools.

Following a fair amount of Google action, I discovered I could run a “check” of the BTRFS file system (much like the old fsck on EXT file systems).

Since I already knew the device name for the root file system, things were pretty easy:

# btrfs check /dev/system/root
Checking filesystem on /dev/system/root

found 5274484736 bytes used err is 0

Looks like the command worked, but it is showing no errors.
So I tried to mount the partition:

# mkdir /old_root
# mount -t btrfs /dev/system/root /old_root

At this point the whole VM hung again!
I had to restart the whole process.
So there was definately an issue with the BTRFS filesystem on the root partition.

Starting the VM again and re-entering the recovery mode of SUSE, I decided to try and mount the partition in recovery mode:

# mkdir /old_root
# mount -t btrfs /dev/system/root /old_root -o ro,recovery

It worked!
No problems.  Weird.
So I unmounted and tried to re-mount in read-write mode again:

# umount /old_root
# mount -t btrfs /dev/system/root /old_root

BAM! The VM hung again.

Starting the VM again and re-entering the recovery mode of SUSE, I decided to just run the btrfs command with the “repair” command (although it says this should be a last resort).

# btrfs check –repair /dev/system/root
enabling repair mode
Checking filesystem on /dev/system/root
UUID: a09b7c3c-9d33-4195-af6e-9519fe550694
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don’t match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 5274484736 bytes used err is 0
total csum bytes: 4909484
total tree bytes: 236126208
total fs tree bytes: 215973888
total extent tree bytes: 13647872
btree space waste bytes: 38681887
file data blocks allocated: 5186543616

Maybe this cache problem that it fixed is the issue.

# mkdir /old_root
# mount -t btrfs /dev/system/root /old_root

Yay!
So, weird problem fixed.
Maybe this is a Kernel level issue and later Kernels have a patch, not sure.  It’s not my primary concern to fix this as I don’t plan on having many power outages, but if it was my production system then I might be more concerned and motivated.

When SLES for SAP is not SLES for SAP

I recently downloaded and installed “SUSE Enterprise Linux for SAP 12 SP3” into a local virtual machine.
It seemed to contain everything that I thought it would contain with regards to included SAP Linux packages.

Noteable were the following in my local VM:

# which saptune
/usr/sbin/saptune
# rpm -qa | grep sap
cyrus-sasl-gssapi-32bit-2.1.26-7.1.x86_64
sap-netscape-link-0.1-1.2.noarch
sap-installation-wizard-3.1.81-3.1.x86_64
yast2-sap-scp-1.0.3-11.2.noarch
saptune-1.1.3-1.1.x86_64
saprouter-systemd-0.2-1.1.noarch
cyrus-sasl-gssapi-2.1.26-7.1.x86_64
patterns-sles-sap_server-12-77.8.x86_64
patterns-sles-sap_server-32bit-12-77.8.x86_64
yast2-saptune-1.2-1.5.noarch
sap-locale-32bit-1.0-92.4.x86_64
sapconf-4.1.8-1.18.noarch
sap-locale-1.0-92.4.x86_64
yast2-sap-scp-prodlist-1.0.2-4.2.noarch
# cat /etc/os-release
NAME=”SLES”
VERSION=”12-SP3″
VERSION_ID=”12.3″
PRETTY_NAME=”SUSE Linux Enterprise Server 12 SP3″
ID=”sles”
ANSI_COLOR=”0;32″
CPE_NAME=”cpe:/o:suse:sles_sap:12:sp3″
# uname -a
Linux hana01 4.4.73-7-default #1 SMP Fri Jul 21 13:26:40 UTC 2017 (6beeafd) x86_64 x86_64 x86_64 GNU/Linux

All looks good to me.

I then created an Azure hosted virtual machine using the image “SLES for SAP 12 SP3 (BYOS)”:

 

The Azure VM seems to be missing a lot of the packages that I would expect to be in place:

# which saptune
which: no saptune in (/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/lib/mit/bin)
# rpm -qa | grep sap
patterns-sles-sap_server-12-77.8.x86_64
yast2-sap-scp-prodlist-1.0.2-4.2.noarch
yast2-sap-scp-1.0.3-11.2.noarch
cyrus-sasl-gssapi-2.1.26-7.1.x86_64
sapconf-4.1.10-40.37.1.noarch
# cat /etc/os-release
NAME=”SLES”
VERSION=”12-SP3″
VERSION_ID=”12.3″
PRETTY_NAME=”SUSE Linux Enterprise Server 12 SP3″
ID=”sles”
ANSI_COLOR=”0;32″
CPE_NAME=”cpe:/o:suse:sles_sap:12:sp3″
# uname -a
Linux hana01 4.4.82-6.3-default #1 SMP Mon Aug 14 14:14:02 UTC 2017 (4c72484) x86_64 x86_64 x86_64 GNU/Linux

Notice also that the Kernel release is slightly newer on the Azure image, plus the version of the sapconf package is slightly newer.
The most important point is that the Azure image is missing the saptune package.
This is important as it is a method presented in numerous SAP notes for automatically applying the recommended O/S settings (that’s right, they don’t all get applied out-of-the-box).

Dynamic SSH X-11 Forwarding on RHEL Linux

Sometimes you need to use an X-client in order to perform certain operations with an SAP system.
An example would be using SUM (if the firewall ports are blocked) or SWPM to perform an installation, or if you’re running an AS Java system, the offline editor (configtool).

If you’re already using PuTTY to get into the remote system via SSH, then you are already part way there.

In this post, I will show you how to dynamically adjust the PuTTY and Linux configuration so that you can use MobaXterm (or any other X-client) to connect into the Linux using X11 (X-Windows).
This is not the same as using the standard method of ticking the X-11 forwarding option, because this option must be ticked prior to establishing the connection into the server.

Instead of exiting from your session and re-connection, you can simply use standard port forwarding to enable your X-client connection over SSL.

Step 1 – Connect into Linux using PuTTY.

You should ideally already be connected into the remote system using SSH and PuTTY (that is the aim of this post).
If you’re not already connected, then open your PuTTY session as normal.

Step 2 – Open X-Client on your PC/virtual server.

Open your X-Client on your local PC.  Ideally it will default to display 0.
The “0” simply refers to a subset of a port range.  It’s a virtual display.

In our example, we used the free MobaXterm tool to provide X-client capability.  Mainly because the tool is free and integrated into one nice binary that can be placed on a USB stick for easy access.

In MobaXterm, hover the mouse over the “X” in to top right frame of the window.
It should be green, and if so, it will display the current IP and display port it has been configured with.

However, you can use any other X-Client tool (XManager, Exceed, X-Ming, WRQ  etc).

Step 3 – Adjust Linux SSH configuration.

The Linux SSH configuration may not be setup to enable X11 forwarding.
This is usually performed by the Linux administrator at a global level in the ssh daemon configuration file.
However, it is possible to adjust the config for individual users only.

As your Linux user (in an SAP system this would usually be the <sid>adm user account), enable the X-11 forwarding:

echo “ForwardX11 yes” > $HOME/.ssh/config

The above command simply enables the current user to forward X11 connections over the SSH connection.
It puts this config into a file called “config” within the .ssh directory of the current Linux user’s home directory.

Step 4 – Setup the port forwarding.

Since this tutorial is going to show the setup of X11 dynamically, with an already established SSH connection, from within your already established PuTTY session, select the Windows window menu from the top left of the PuTTY window and select “Change Settings…“:

  

Expand the “Connection -> SSH -> Tunnels” settings area on the left hand tree menu:
(Notice there is no option to enable or disable the usual X-11 forwarding option)

On the right hand side, now add port 6000 as source and loclahost:6000 as destination, then select “Remote” and click “Add“:

Click Apply:

What is the above doing?
It is telling PuTTY to establish a listening port on the Linux server, listening on port 6000 (the port for display “0.0”).

The port is “remote” because it is remote to the PuTTY session (i.e. not on the computer where you are running PuTTY).

We’re telling PuTTY to only consider IPv4 because we don’t use IPv6 and have no interest in having it listen for an IPv6 connection.

The above setup will have now established a port forward over the SSH port.
If you were to run “netstat -an | grep 6000” on the Linux server, you would see one port in status LISTEN.

Now the port forwarding is established, all we need to do is configure our Linux session to make use of the display setting.

Step 5 – Configure DISPLAY

In your Linux session, set the DISPLAY variable appropriately.
In a C-shell you would use:

>  setenv DISPLAY localhost:0.0

In a Bash, Bourne or Korn shell you would use:

$  export DISPLAY=localhost:0.0

From within the Linux session you should now be able to run an X11 application and see the window on your PC/virtual server.
You can use the standard “xeyes” or “xclock” to perform the test, however sometimes on Linux installs these do not get installed.
I’ve found that it’s generally possible to call “firefox” or “totem” to perform an X11 test.

Alternatively, just call your intended SAP X11 application as discussed at the beginning.

That’s it.