This blog contains experience gained over the years of implementing (and de-implementing) large scale IT applications/software.

SUSE Linux 12 – Kernel 4.4.73 – Boot Hang – BTRFS Issue

I had a VMWare guest running SUSE Linux 12 SP3 64bit (kernel 4.4.73).
One day after a power outage, the VM failed to boot.
It would arrive at the SUSE Linux “lizard” splash screen and then just hang.

I noticed prior to this error that the SUSE 12 operating system creates it’s root partition inside a logical volume call “/dev/system/root” and it is then formatted as a BTRFS filesystem.

At this point I decided that I must have a corrupt disk block.
I launched the VM with the CDROM attached and pointing at the SUSE 12 installation ISO file.
While the VM starts you need to press F2 to get to the “BIOS” boot options to enable the CDROM to be bootable before the hard-disks.

Once the installation cdrom was booting, I selected “Recovery” from the SUSE menu.
This drops you into a recovery session with access to the BTRFS filesystem check tools.

Following a fair amount of Google action, I discovered I could run a “check” of the BTRFS file system (much like the old fsck on EXT file systems).

Since I already knew the device name for the root file system, things were pretty easy:

# btrfs check /dev/system/root
Checking filesystem on /dev/system/root

found 5274484736 bytes used err is 0

Looks like the command worked, but it is showing no errors.
So I tried to mount the partition:

# mkdir /old_root
# mount -t btrfs /dev/system/root /old_root

At this point the whole VM hung again!
I had to restart the whole process.
So there was definately an issue with the BTRFS filesystem on the root partition.

Starting the VM again and re-entering the recovery mode of SUSE, I decided to try and mount the partition in recovery mode:

# mkdir /old_root
# mount -t btrfs /dev/system/root /old_root -o ro,recovery

It worked!
No problems.  Weird.
So I unmounted and tried to re-mount in read-write mode again:

# umount /old_root
# mount -t btrfs /dev/system/root /old_root

BAM! The VM hung again.

Starting the VM again and re-entering the recovery mode of SUSE, I decided to just run the btrfs command with the “repair” command (although it says this should be a last resort).

# btrfs check –repair /dev/system/root
enabling repair mode
Checking filesystem on /dev/system/root
UUID: a09b7c3c-9d33-4195-af6e-9519fe550694
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don’t match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 5274484736 bytes used err is 0
total csum bytes: 4909484
total tree bytes: 236126208
total fs tree bytes: 215973888
total extent tree bytes: 13647872
btree space waste bytes: 38681887
file data blocks allocated: 5186543616

Maybe this cache problem that it fixed is the issue.

# mkdir /old_root
# mount -t btrfs /dev/system/root /old_root

Yay!
So, weird problem fixed.
Maybe this is a Kernel level issue and later Kernels have a patch, not sure.  It’s not my primary concern to fix this as I don’t plan on having many power outages, but if it was my production system then I might be more concerned and motivated.

Recovery From: Operation start is not allowed on VM since the VM is generalized – Linux

Scenario: In Azure you had a Linux virtual machine.  In the Azure portal you clicked the “Capture” button in the Portal view of your Linux virtual machine, now you are unable to start the virtual machine as you get the Azure error: “Operation ‘start’ is not allowed on VM ‘abcd’ since the VM is generalized.“.

What this error/information prompt is telling you, is that the “Capture” button actually creates a generic image of your virtual machine, which means it is effectively a template that can be used to create a new VM.
Because the process that is applied to your original VM modifies it in such a way, it is now unable to boot up normally.  The process is called “sysprep”.

Can you recover your original VM?  no.  It’s not possible to recover it properly using the Azure Portal capabilities.  You could do it if you downloaded the disk image, but there’s no need.
Plus, there is no telling what changes have been made to the O/S that might affect your applications that have been installed.

It’s possible for you to create a new VM from your captured image, or even to use your old VM’s O/S disk to create a new VM.
However, both of the above mean you will have a new VM.  Like I said, who knows what changes could have been introduced from the sysprep process.  Maybe it’s better to rebuild…

Because the disk files are still present you can rescue your data and look at the original O/S disk files.
Here’s how I did it.

I’m starting from the point of holding my head in my hands after clicking “Capture“!
The next steps I took were:

– Delete your original VM (just the VM).  The disk files will remain, but at least you can create a new VM of the same name (I liked the original name).

– Create a new Linux VM, same as you did for the one you’ve just lost.
Use the same install image if possible.

– Within the properties of your new VM, go to the “Disks” area.

– Click to add a new data disk.
We will then be able to attach the existing O/S disk to the virtual machine (you will need to find itin the list).
You can add other data disks from the old VM if you need to.

Once your disks are attached to your new Linux VM, you just need to mount them up.
For my specific scenario, I could see that the root partition “/” on my new Linux VM, was of type “ext4” (check the output of ‘df -h’ command).
This means that my old VM’s root partition format would have also been ext4.
Therefore I just needed to find and mount the new disk in the O/S of my new VM.

As root on the new Linux VM find the last disk device added:

# ls -ltr /dev/sd*

The last line is your old VM disk.  Mine was device /dev/sdc and specifically, I needed partition 2 (the whole disk), so I would choose /dev/sdc2.

Mount the disk:

# mkdir /old_vm
# mount -t ext4 /dev/sdc2 /old_vm

I could then access the disk and copy any required files/settings:

# cd /old_vm

Once completed, I unmounted the old O/S disk in the new Linux VM:

# umount /old_vm

Then, back in the Azure Portal in the disks area of the new VM (in Edit mode), I detatched the old disk:

 

Once those disks are not owned by a VM anymore (you can see in the properties for the specific disk), then it’s safe to delete them.

When SLES for SAP is not SLES for SAP

I recently downloaded and installed “SUSE Enterprise Linux for SAP 12 SP3” into a local virtual machine.
It seemed to contain everything that I thought it would contain with regards to included SAP Linux packages.

Noteable were the following in my local VM:

# which saptune
/usr/sbin/saptune
# rpm -qa | grep sap
cyrus-sasl-gssapi-32bit-2.1.26-7.1.x86_64
sap-netscape-link-0.1-1.2.noarch
sap-installation-wizard-3.1.81-3.1.x86_64
yast2-sap-scp-1.0.3-11.2.noarch
saptune-1.1.3-1.1.x86_64
saprouter-systemd-0.2-1.1.noarch
cyrus-sasl-gssapi-2.1.26-7.1.x86_64
patterns-sles-sap_server-12-77.8.x86_64
patterns-sles-sap_server-32bit-12-77.8.x86_64
yast2-saptune-1.2-1.5.noarch
sap-locale-32bit-1.0-92.4.x86_64
sapconf-4.1.8-1.18.noarch
sap-locale-1.0-92.4.x86_64
yast2-sap-scp-prodlist-1.0.2-4.2.noarch
# cat /etc/os-release
NAME=”SLES”
VERSION=”12-SP3″
VERSION_ID=”12.3″
PRETTY_NAME=”SUSE Linux Enterprise Server 12 SP3″
ID=”sles”
ANSI_COLOR=”0;32″
CPE_NAME=”cpe:/o:suse:sles_sap:12:sp3″
# uname -a
Linux hana01 4.4.73-7-default #1 SMP Fri Jul 21 13:26:40 UTC 2017 (6beeafd) x86_64 x86_64 x86_64 GNU/Linux

All looks good to me.

I then created an Azure hosted virtual machine using the image “SLES for SAP 12 SP3 (BYOS)”:

 

The Azure VM seems to be missing a lot of the packages that I would expect to be in place:

# which saptune
which: no saptune in (/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/lib/mit/bin)
# rpm -qa | grep sap
patterns-sles-sap_server-12-77.8.x86_64
yast2-sap-scp-prodlist-1.0.2-4.2.noarch
yast2-sap-scp-1.0.3-11.2.noarch
cyrus-sasl-gssapi-2.1.26-7.1.x86_64
sapconf-4.1.10-40.37.1.noarch
# cat /etc/os-release
NAME=”SLES”
VERSION=”12-SP3″
VERSION_ID=”12.3″
PRETTY_NAME=”SUSE Linux Enterprise Server 12 SP3″
ID=”sles”
ANSI_COLOR=”0;32″
CPE_NAME=”cpe:/o:suse:sles_sap:12:sp3″
# uname -a
Linux hana01 4.4.82-6.3-default #1 SMP Mon Aug 14 14:14:02 UTC 2017 (4c72484) x86_64 x86_64 x86_64 GNU/Linux

Notice also that the Kernel release is slightly newer on the Azure image, plus the version of the sapconf package is slightly newer.
The most important point is that the Azure image is missing the saptune package.
This is important as it is a method presented in numerous SAP notes for automatically applying the recommended O/S settings (that’s right, they don’t all get applied out-of-the-box).

Downloading SAP Download Basket Contents in the Cloud on Linux

Scenario: You’ve made the move to the cloud.  You are about to install some SAP software on a new Linux server hosted in the cloud and you just need to get the installation media uploaded.
Except, you don’t want to have to start X-Windows or setup a specific Windows server just to do this.

You have two options to achieve this, you can either:
Upload the files from your local network using SFTP or SCP (maybe you have ExpressRoute or DirectConnect) and have a fast connection.

or

Why not just download them straight from your SAP download basket into the target server.
What’s the difference between the two?  Well, it depends on whether you’ve already got access to the installation media locally, or if you are intending to first download it locally from SAP, then upload it to your cloud hosted server.
This doesn’t make sense to download then re-upload to the cloud.
Therefore, here’s how you can download straight from your SAP download basket.

What you will need:
– A web browser on your local PC.
– A valid SAP S-user account.
– Disk space on a cloud hosted Linux server (I’m going to show you the Linux way).
– The cloud hosted server will need internet access.

Setup Download Basket:
Before we start downloading, you need to cleanup your download basket.
Go to https://launchpad.support.sap.com/#/downloadbasket
Remove everything already in there that is not the items you want to download to the cloud hosted server.
The reason we must do this is because it’s difficult to know which items relate to which files in the download basket (you’ll see in a moment).
Now you can add the specific items you wish to download, into your download basket.
I’ve got two items:

Export Download Basket:
In the download basket click the “Export Links to Text File” button:

This will generate a text document which you can open on your PC called myDownloadBasketFiles.txt.

Open the text file using notepad:

I have 2 lines for the two items in my download basket.  I don’t know which is which.  I can suppose they are in order, but I don’t really know for sure.
Take the first line and log onto your cloud hosted Linux server as your preferred user.
Ping the softwaredownload.sap.com server using the Linux utility wget to make sure you can see the server.

> wget https://softwaredownloads.sap.com

A successful ping will show a HTTP error 401 and “Authorization failed“:

We have confirmed that our cloud hosted Linux server can see the SAP download server.
Switch to your target download directory (somewhere with disk space):

> cd mydownloadsdirectory

We now call wget again and pass the first item to download:

> wget –http-user=”[your s-user]” –http-password=”[your password]” [the first url] –output-document=1file.SAR

Adjust the command line above to put in your S-user account name and password.
You should also change the last parameter to give your file a name.

We have to guess that it is a SAR file.  Once it’s downloaded you can always re-name and test the extraction using “SAPCAR -tvf thefile.SAR”.

The wget utility will save the file in the current directory.
Repeat the same command line, changing the URL and the output document file, for each of the remaining items in the notepad text file of your download basket.

As you can see, using wget will allow you to script the download process so that you could schedule an overnight download of software.

Dynamic SSH X-11 Forwarding on RHEL Linux

Sometimes you need to use an X-client in order to perform certain operations with an SAP system.
An example would be using SUM (if the firewall ports are blocked) or SWPM to perform an installation, or if you’re running an AS Java system, the offline editor (configtool).

If you’re already using PuTTY to get into the remote system via SSH, then you are already part way there.

In this post, I will show you how to dynamically adjust the PuTTY and Linux configuration so that you can use MobaXterm (or any other X-client) to connect into the Linux using X11 (X-Windows).
This is not the same as using the standard method of ticking the X-11 forwarding option, because this option must be ticked prior to establishing the connection into the server.

Instead of exiting from your session and re-connection, you can simply use standard port forwarding to enable your X-client connection over SSL.

Step 1 – Connect into Linux using PuTTY.

You should ideally already be connected into the remote system using SSH and PuTTY (that is the aim of this post).
If you’re not already connected, then open your PuTTY session as normal.

Step 2 – Open X-Client on your PC/virtual server.

Open your X-Client on your local PC.  Ideally it will default to display 0.
The “0” simply refers to a subset of a port range.  It’s a virtual display.

In our example, we used the free MobaXterm tool to provide X-client capability.  Mainly because the tool is free and integrated into one nice binary that can be placed on a USB stick for easy access.

In MobaXterm, hover the mouse over the “X” in to top right frame of the window.
It should be green, and if so, it will display the current IP and display port it has been configured with.

However, you can use any other X-Client tool (XManager, Exceed, X-Ming, WRQ  etc).

Step 3 – Adjust Linux SSH configuration.

The Linux SSH configuration may not be setup to enable X11 forwarding.
This is usually performed by the Linux administrator at a global level in the ssh daemon configuration file.
However, it is possible to adjust the config for individual users only.

As your Linux user (in an SAP system this would usually be the <sid>adm user account), enable the X-11 forwarding:

echo “ForwardX11 yes” > $HOME/.ssh/config

The above command simply enables the current user to forward X11 connections over the SSH connection.
It puts this config into a file called “config” within the .ssh directory of the current Linux user’s home directory.

Step 4 – Setup the port forwarding.

Since this tutorial is going to show the setup of X11 dynamically, with an already established SSH connection, from within your already established PuTTY session, select the Windows window menu from the top left of the PuTTY window and select “Change Settings…“:

  

Expand the “Connection -> SSH -> Tunnels” settings area on the left hand tree menu:
(Notice there is no option to enable or disable the usual X-11 forwarding option)

On the right hand side, now add port 6000 as source and loclahost:6000 as destination, then select “Remote” and click “Add“:

Click Apply:

What is the above doing?
It is telling PuTTY to establish a listening port on the Linux server, listening on port 6000 (the port for display “0.0”).

The port is “remote” because it is remote to the PuTTY session (i.e. not on the computer where you are running PuTTY).

We’re telling PuTTY to only consider IPv4 because we don’t use IPv6 and have no interest in having it listen for an IPv6 connection.

The above setup will have now established a port forward over the SSH port.
If you were to run “netstat -an | grep 6000” on the Linux server, you would see one port in status LISTEN.

Now the port forwarding is established, all we need to do is configure our Linux session to make use of the display setting.

Step 5 – Configure DISPLAY

In your Linux session, set the DISPLAY variable appropriately.
In a C-shell you would use:

>  setenv DISPLAY localhost:0.0

In a Bash, Bourne or Korn shell you would use:

$  export DISPLAY=localhost:0.0

From within the Linux session you should now be able to run an X11 application and see the window on your PC/virtual server.
You can use the standard “xeyes” or “xclock” to perform the test, however sometimes on Linux installs these do not get installed.
I’ve found that it’s generally possible to call “firefox” or “totem” to perform an X11 test.

Alternatively, just call your intended SAP X11 application as discussed at the beginning.

That’s it.