This blog contains experience gained over the years of implementing (and de-implementing) large scale IT applications/software.

Tag: Database

Writing a Simple Backup Script

Database Backup Script Disks

So you want a backup script to backup your databases to disk.
Sounds like a nice half-a-day scripting job, doesn’t it?
Let’s analyse this requirement a little more and you will start to get the idea that it’s never as simple as it sounds.

Requirements

Digging a little deeper we come up with the following requirements:

Backup Configuration

We need a standard backup location on all database servers.
When we backup a database we need a folder location on the server (unless you’re using a specific device/method like Tivoli, or backint for SAP).
If we are backing up to a local directory (let’s assume this) and you don’t have the same drive/folder structure, then you will need to create a list of locations for each database and have the backup script look at the list, or worst case, hardcode the config into the script.
The easy way to solve this is to move the configuration out of the backup script altogether.
Most databases support the use of a predefined backup configuration.
For example, in ASE we can use “dump configurations”.
In HANA we adjust the ini file of the respective indexserver or nameserver to set the backup path locations.
Having done the above, for each database, we can then “simply” pull out the config from the target database or during the backup command execution, include the name of the config profile to use, which will dictate the target backup location.
We are going to ignore other issues, such as the size of the disk location, type of disk storage tier and other infrastructure related questions.

Access to the Database

We need a way of pulling out the dump/backup path from the database.
As mentioned above, if we move the configuration of the backup location out of the databases, we need to access it from the script.
This is definitely possible, but if it’s stored in the database (let’s assume this), at this point we have no way of logging into the database.
We therefore need a way of logging into the database in a standard way across all databases.
Therefore, we should elect to use a harmonised username for our backups (not a harmonised password!).

Multiple Scripts

We need more than one script.
Pulling out the configuration from the database is never easy due to the differences in the way that the command line interpreters work.
For example, in HANA 1.0, the hdbsql command outputs a slightly different format and provides less capability to customise the output, compared to the HANA 2.0 hdbsql command line tool.
SAP ASE is completely different and again needs a specific script setup.
The same will be for combinations of Linux and Windows systems.  You may need some PowerShell in there!

Modularised Code Packages

We need it to be packaged and easy to read.
Based on the above, we can see that we will need more than one script to cater for different DB vendors and architectures/platforms.
Therefore, one script for backing up HANA databases and one for ASE databases.
What if you need a SQL Server one?  Well again, the DB call is slightly different, so another script is needed.
It’s possible to modularise the scripts in such a way that the DB call itself is a separate script for each DB, leaving your core script the same.
However, we are now into the realms of script readability and simplicity versus complexity and re-use.
But it’s worth considering the options within the limitations of the operating system landscape that you have.

Database Security

We need a secure method of storing the password for the harmonised backup user across all the databases.
Since we will need to log into the database to perform tasks such as getting the backup config settings and actually calling the backup commands, we need to think about how we will be storing the username and password.
In some cases, like HANA, we can just use the secure file system store (hdbuserstore) to add our required username and password.
There are some issues with certain versions of HANA (HANA 1.0 is different to HANA 2.0) in this area.
For ASE we may be able to use the sybxctrl binary to execute our script in an intelligent way, avoiding the need to pass passwords at all.
Recent versions of SAP ASE 16.0 SP03, include a new aseuserstore command, which I will highlight in another post.  It’s very similar to the HANA hdbuserstore, except it allows ASE (for Business Suite) and ASE Enterprise Edition, to use the same method of password-less access.

Scheduling

We need a common backup scheduling capability across the servers.
You can use a central system (e.g. an enterprise scheduler), or something like cron or Windows Task Scheduler, but centrally controlled from a task controller script.
This will rely on either a shared storage ability (e.g. NFS/CIFS) or some method of the scripts talking centrally to a control plane.
Not only does the central location/control provide easy control of the schedule, but you will also find this useful for capturing error situations, where the backup script may have failed and needs to notify the operators.

Logging

We need the logging output of the scripts to be accessible.
When you are backing up over 100 databases, your administrators are not going to want to go onto each individual server to look at logs.
You will need a central location for logging and aggregation of that logging.

O/S Users

We may need common O/S user accounts.
The execution environment of the script needs to include the capability to access the log areas.
The user account used to perform the execution of the script needs to have a common setup across all servers.
If you’re using CIFS or NFS for storing logs, you will have permissions issues if you use different users, unless you configure your NFS/SMB settings appropriately.
In Unix/Linux, it’s easy to create a specific user (could be linked into Windows Active Directory) with the same UID across many servers, or make the user a member of the same group.

Housekeeping

We need housekeeping for our logging capability.
During the execution of your scripts, the logs you generate will need to be kept according to your usual policies.
You may want to see a report of backups for the last month.
You may need to provide audit evidence that a backup has been performed.

Disk Space

We need a defined amount of backup space.
If you are backing up to disk, you may need a way of calculating how much disk space you will need.
In HANA this is fairly easy as it provides views you can query to estimate the backup disk requirements prior to executing a backup.
You will need to call these and check against the target disk location, before your script starts the backup.
How will you account for additional space requirements over time?  Will you just fail or can you provide a warning?
How many backups or backup files will you retain on disk and over how many days?
Will these be removed by your script once they are no longer needed?

Backup Strategies

We should consider different backup strategies.
What type of backup will your script need to handle?
For example, with ASE or SQL Server, you may need to run transaction log backups as well as normal full backups.
Will this be the same script?  Can they run together at the same time?
If you are dumping to disk, performing a full backup once a day, then will you need those transaction log dumps from the previous day?
As well as performing a backup of the databases, your script should also backup the recommended configuration files.
For example on ASE, it is recommended to include the configuration file and also the dumphist file.
On HANA it is recommended to include the ini files, the backup.log and useful to include the trace files.
Will your backup be encrypted and will you need to store the keys somewhere?

Validation

We may need backup validation.
Some databases provide post-backup validation, and some provide inline validation of the blocks.
Do you need to consider to these check on (most are by default turned off)?

Authenticity

We should consider backup file authenticity.
Do you need to know if the backup files have been tampered with?
Or maybe just check that what was sent over the network to the target storage location, is the exact same file that was originally created?
You may need to perform some sort of checksum on the original backup file to help establish and authenticate the backup files.
This process should be the same ideally, for all databases.

Pre-Execution Checks

We should performing checking of the environment.
Before your script starts to run the backup, you may wish to include a common set of pre-checks.
The reason is that common issues can be integrated into the pre-checks over time.

Examples Include:

  • Are you running as the correct O/S user?
  • Do you have execution access to all required sub-scripts/log directories?
  • Is the type of target database that your script supports, installed?
  • Is the target database running?
  • Are any other required processes running (e.g. ASE backupserver)?
  • Is a backup script already executing?
  • Is the version of the database supported by your script?
  • Is the target backup destination available?  (e.g. file/folder location).
  • Is there enough disk space for your backup to complete?
  • Was the last backup a success?  If not, can you remove the previous dump files?

Once you’ve got all the above decided, then it will be a simple task of writing the script.

Forcefully Prevent HANA Tenant DB From Starting

Scenario: Your HANA system is having some serious memory problems.
In the memory you have remaining you can’t get the SYSTEM database to run, so you can’t stop a particular tenant database.

Here’s how you can use the hdbnsutil utility to prevent hdbdaemon from trying to start a tenant database.
As the <sid>adm user, kill off all remaining HANA system processes:

> hdbdaemon -quit

Export the topology:

> hdbnsutil -exportTopology /tmp/topology_export.txt

Edit the file and change the “status” of the tenant database to be “no”:

> vi /tmp/topology_export.txt
…
n3>name
v>HT1
n3>status
v>no
n3>timestamp
…

Save the file and quit vi.
Reload the topology using hdbnsutil:

> hdbnsutil -importTopology

Now restart hdbdaemon:

> hdbdaemon -start

HowTo: Install SAP HANA 2.0 in a VM in less than 30minutes – Part #3

This is the third part of my (quite large) post on how to install an SAP HANA 2.0 database into a SUSE Linux for SAP 12 SP3 virtual machine.

See Part #1 of the post here.
See Part #2 of the post here.

We continue from where we left off in part 2, just after we created a new 50GB disk volume for our new HANA install.
Check the new partition:

# df -h /hana
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/volHANA-lvHANA1   50G  33M   50G   1% /hana

Unmount the CDROM and install VMWare tools (I need it for access to my VMWare shared folder):

# umount /mnt/dvd

Select the option to re-install VMWare tools:


Mount the CD and extract the TAR file:

# mount /dev/sr0 -t iso9660 /mnt/dvd
# cd /tmp
# tar -xvf /mnt/dvd/VMwareTools-10.1.15-6627299.tar.gz
# cd vmware-tools-distrib
# ./vmware-install.pl

Choose YES to ALL prompts (especially to the ones to replace existing files).
Disable some SUSE Linux services that are more than likely not needed (in this specific case) and just consume precious memory:
Disable VMware thin printing:

# chkconfig vmware-tools-thinprint off

Disable Linux printing:

# chkconfig cups off

Disable Linux auditing:

# chkconfig auditd off

Disable Linux eMail SMTP daemon:

# chkconfig postfix off

Disable sound:

# chkconfig alsasound off

Disable NFS ( you might need it…):

# chkconfig nfs off

Disable the Machine Check Events Logging capture:

# chkconfig mcelog off

Double check the IP address of your VM:

# ifconfig | grep inet


Your IP address should be listed (you can see mine is 192.168.174.129).
If you don’t have one, then your VM is not quite setup correctly in the VMWare properties or your networking configuration is not correct, or you don’t have a DHCP server on your local network, or your network security is preventing your VM from registering it’s MAC address.  It’s complex.
Assuming that you have an IP address, check that you can connect to the SSH server in your VM using PUTTY :

Enter the IP address of your VM server:

Log into the server as root:

Now we’ve got access to the VM and disk space to create our HANA database and put the software.
To perform the HANA install, I’ve extracted my HANA patch/install media into a VMWare Shared Folders folder and simply extract the SAR file to my PC using SAPCAR.exe, sharing the directory location through VMware to the guest O/S.
Since I’ve used VMWare shared folders, I need to mount my folder (it’s shared via the VMWare Tools(:

# cd /mnt/hgfs/Downloads      [my VMware share name is “Downloads”]
# cd SAP_HANA_DATABASE      [this is my extracted SAR file]
# ./hdbinst –ignore=check_diskspace,check_min_mem

You will be prompted for certain pieces of information.  Below is what was entered:
Local Host Name: hana01
Installation Path:   /hana/shared
System ID:             H10
Instance Number: 10
Worker Group: default
System Usage: 3 – development
System Administrator Password:  hanahana
System Administrator Home Dir:  /usr/sap/H10/home
System Administrator User ID:  10001
System Administrator Login Shell:  /bin/sh
ID of User Group (sapsys): [I selected any]
Location of Data Volumes:  /hana/shared/H10/global/hdb/data
Location of Log Volumes:   /hana/shared/H10/global/hdb/log
Restrict maximum memory allocation? N
Database SYSTEM user password:   Hanahana1
Restart instance after reboot:  N

Summary before execution:
   Installation Path: /hana/shared
   SAP HANA System ID: H10
   Instance Number: 10
   Database Isolation: low
   System Usage: development
   System Administrator Home Directory: /usr/sap/H10/home
   System Administrator Login Shell: /bin/sh
   System Administrator User ID: 1001
   ID of User Group (sapsys): 79
   Location of Data Volumes: /hana/shared/H10/global/hdb/data
   Location of Log Volumes: /hana/shared/H10/global/hdb/log
   Local Host Name: hana01
   Worker Group: default

Installation will begin:



Installation & instance startup time was around 45 minutes due to the memory swapping.

That’s it for now.
We have a basic SYSTEM database (SYSTEMDB).

Some things to note at this point:
– SYSTEM database data and log files reside in /usr/sap/H10/SYS/global/data and /usr/sap/H10/SYS/global/log directories (linked to /hana/shared/H10/global).
– Initial usage of disk is around 4GB for data and 1 GB for logs.
– Used memory is around 6GB.
– The HANA Cockpit URL would be (if it was installed) https://192.168.80.2:4310/sap/hana/admin/cockpit   or port 8010 for non SSL.
– The above two URLs are served from the xsengine via the webdispatcher.
– You cannot permanently stop the webdispatcher or xsengine (but I can…).
– SAP note 2517761 tells you how to connect via HANA Studio to the system DB.
– You will need to add the h10adm username and password into HANA Studio to allow you to stop/start the system.
– You may need to add the hana01 and it’s FQDN to your PC’s hosts file to be able to successfully stop/start the system from HANA Studio.
******  OPTIONAL ********

We can slightly reduce the memory requirements of the statisticsserver (now embedded into the indexserver process) by following SAP note 2147247 to disable the inifile_checker service in the global.ini:
Switch to h10adm Linux user:

# su – h10adm
> hdbsql -i 10 -u SYSTEM -p Hanahana1 -d SYSTEMDB
hdbsql SYSTEMDB=> ALTER SYSTEM ALTER CONFIGURATION (‘global.ini’, ‘system’) SET (‘inifile_checker’, ‘enable’)=’false’  WITH RECONFIGURE;
hdbsql SYSTEMDB=> quit

******  OPTIONAL ********
We also reduce slightly the system global allocation limit to 12GB, so that we can consequently reduce the VM memory from 24Gb to 18GB:
NOTE: When you do this, you will not be able to run a Tenant Database because the Tenant DB indexserver process will need at least 8GB of memory to start.

> hdbsql -i 10 -u SYSTEM -p Hanahana1 -d SYSTEMDB
hdbsql SYSTEMDB=> ALTER SYSTEM ALTER CONFIGURATION (‘global.ini’, ‘system’) SET (‘memorymanager’, ‘global_allocationlimit’) = ‘12288’ WITH RECONFIGURE;
hdbsql SYSTEMDB=> quit

Restart the HANA system:

> sapcontrol -nr 10 -function Stop

Wait for it to be stopped:

> watch sapcontrol -nr 10 -function GetProcessList

Press CTRL+C once everything is shutdown (apart from the HDB Daemon).
Exit back to root:

> exit

Shutdown the server:

# shutdown -h now

Adjust the VM memory to be 18GB:

Power on the VM:

Log in as h10adm and start the HANA system:

> sapcontrol -nr 10 -function Start

******  OPTIONAL ********
We can create a new tenant database as follows (we would need at least 24GB of memory for SUSE in order to create and run the SYSTEM DB and the Tenant DB):

# su – h10adm
> hdbsql -i 10 -u SYSTEM -p Hanahana1 -d SYSTEMDB
hdbsql SYSTEMDB=> CREATE DATABASE HT1 SYSTEM USER PASSWORD Hanahana1;
hdbsql SYSTEMDB=> quit

If you wish to stop the Tenant database from starting, you can use SQL as per the help.sap.com, or if your SYSTEM DB will not start also, then you can use the temporary method (probably not recommended by SAP) of exporting the topology using hdbnsutil, adjusting the export file to set the Tenant DB “status” to “no” and then re-import the file using hdbnsutil.
Should you need to quickly (and nastily) kill off the SYSTEM DB and Tenant DB processes, you can use the hdbdaemon command: “hdbdaemon -quit”.

HowTo: Install SAP HANA 2.0 in a VM in less than 30minutes – Part #2

This is the second part of a three part post on how to install an SAP HANA 2.0 database into a SUSE Linux for SAP 12 SP3 virtual machine.
See Part #1 here.

During the VM start-up you may be prompted by VMWare to download the VMWare Tools, you should do this (it’s about 1 minute):

The SUSE installation can be started:

Customise the locale settings and accept the terms:

We skipped registration (we don’t need to update SUSE):

Select “SUSE Linux Enterprise Server for SAP Applications” and since we will use SSH, de-select “Enable RDP”:

Click “Network Configuration” in the top right hand corner:

I adjusted my install to use a static IP address, I also setup the hostname and fully qualified domain name at this point (you can change this later using “yast lan” if you want):
IP: 192.168.80.2  (relevant to my VMWare host-only setup)
Subnet: 255.255.255.0
Hostname: hana01.fqdn.corp

On the next page I added the same hostname and FQDN, then set the DNS resolver policy to “Only Manually” which will allow me to not use DNS at all:

We don’t need any addons:

Check the root partition size on /dev/sda1 and click “Edit Proposal Settings”:

We need to adjust the root partition format to be XFS:
NOTE: XFS is the only supported filesystem for the HANA data and log areas, so why not use it for everything.


Set the timezone:

Set the root password:

On the summary screen disable the firewall and ensure that SSH is enabled:

To minimise memory usage, we set the default start-up mode to “Text Mode” (to change click “Default systemd target”):

After all the screen prompts were answered the install time was approx 10 minutes (at least 1 coffee).
NOTE: There were a couple of instances where a package failed to install.  Clicking “Retry” completed the package installation.
We now need to apply the required O/S changes as per SAP note 2205917.  We can use the saptune command to do this:

# saptune solution apply HANA

Enable SAPTUNE to auto-start:

# saptune daemon start

Shutdown the server.

# shutdown –h now

Edit the VM to add a second hard disk for the HANA database:



We assign 50GB in one single file:


Power on the VM.
Log back in as root once it has rebooted.
Check that you can resolve the hostname:

# hostname
hana01
# hostname -f
hana01.fqdn.corp

15 MINUTES HAVE NOW ELAPSED!
Let’s mount the SUSE ISO on the server:

# mkdir /mnt/dvd
# mount /dev/sr0 -t iso9660 /mnt/dvd

Now install the Java runtime:

# cd /mnt/dvd/suse/x86_64
# rpm -i –nodeps java-1_8_0-ibm-*

Check the version is 1.8.0:

# java -version

Now we need to create our HANA database disk partitions.
First check which disk you’re using for the O/S:

# dmsetup deps -o devname


I can see that sda1 (sda) is currently mounted as my primary root and swap disk.
Which means that /dev/sdb will be my new HANA disk:

# ls -l /dev/sd*


WARNING: Adjust the commands below to the finding above, so you use the correct unused disk and don’t overwrite your root disk.
Create the new partition on the disk:

# fdisk /dev/<your disk device e.g. sdb>

Then enter:

n <return>
p <return>
1 <return>
<return>
<return>
t <return>
8e <return>
w <return>

At the end, the fdisk command exits.
Re-run fdisk to check your new partition:

Create the volume group and logical volume:

# pvcreate /dev/sdb1
# vgcreate /dev/volHANA /dev/sdb1
# lvcreate -L 51072M -n lvHANA1 volHANA

Format the new XFS (only one really supported) logical volume:

# mkfs.xfs /dev/volHANA/lvHANA1

Mount the new partition:

# mkdir /hana
# echo “/dev/volHANA/lvHANA1 /hana xfs defaults 0 0”   >> /etc/fstab
# mount -a

That is it for Part #2 of this guide.
Continue on to Part #3 for the completion of our HANA 2.0 install.

HowTo: Install SAP HANA 2.0 in a VM in less than 30minutes – Part #1

For the original post back in 2014 we used SAP HANA 1.0 sps07 and installed into a Virtual Machine running SUSE Linux 11.
Things have moved on since 2014 and we have now seen the arrival of HANA 2.0 with multi-tenant database feature and new HANA Cockpit and SUSE Enterprise Linux 12 with it’s new systemd daemon replacement of the old SYS V init scripts.
I decided it was time to update the post…
Scenario: You want to prototype something for a new HANA 2.0 database.  We can use the power of a virtual machine to get a HANA 2.0 database up and running in less than 30 minutes.
Well, it was supposed to be 30 minutes, and it sure can be 30 minutes, providing you have the right (fast) equipment to hand.
Remember, this is not a “Here’s the standard install process” hand-holding stuff – this is let’s get it installed and use it!
Here’s how…

What you’ll need:
– SAP HANA In Memory DB 2.0 install media from SAP Software Download Centre. 
This can be the Platform Edition (for native HANA systems) or the Enterprise Edition (for S/4HANA or BW/4HANA or any other x/4HANA).
I also cheated a little in my process, since I downloaded the “Installation / Patch” for a HANA database, since this contains the latest entire code line and installer but is much less in size.
In my example I use IMDB_SERVER20_012_3-80002031.SAR which is ~3.5GB in size.

– The SUSE Linux for SAP v12 sp02 or sp03 (recommended) install media (ISO).
This is free to download from https://download.suse.com (although you will need to register an account with SUSE) but you don’t need a license.  This is ~3.6GB in size and you only need the first DVD (DVD1).

– A valid license for the HANA database (platform edition or enterprise edition).

– SAP HANA Studio installed on a PC which can access the virtual HANA server you’re going to create (the Studio install media is contained within the full HANA install media DVD, or you can download it separately from SAP Software Download Centre).
In my example, I’m using IMC_STUDIO2_212_3-80000323.SAR (should be the same revision as the database) which is 734MB in size.
NOTE: The later revisions of HANA come with the HANA Cockpit built-in (web based) so you may not need the HANA Studio, it depends what you want to do with it.  See SAP note 2185556 for more details.

– A host machine to host the virtual machine.  You need at least 20GB of RAM, although if you configure your pagefile (in Windows) on SSD or flash, you could get away with 16GB (I did !!!).
– SAP notes access.  Specifically to read/check SAP notes 1984787, 2205917 & 1944799.
– A downloaded version of SAPCAR.exe on your PC (if, like me, you will be using the VMWare shared folders option to present your downloaded media to the gues O/S).

What we’re going to do:
– We’ll create a basic SUSE Linux for SAP 12 SP3 virtual machine.  You can use any host OS, I’m using Windows 7 64bit and VMWare Workstation Player v14.
– Because most people are using VMs to maximise infrastructure, we’ll go through a couple of steps to really reduce the O/S memory footprint and for efficiency we use SSH and the text mode installer for HANA.  We get this whole thing running in less than 16GB of RAM in the end.
– We’ll install a basic HANA 2.0 database (in multi-tennant mode – this is the future).  Initially we only get the SYSTEM DB, then we create a new tenant DB afterwards.

START THE CLOCK!

Create your basic VM for SUSE Enterprise Linux (I’m using SUSE Linux 12 for SAP SP3).
It will need the following resources:
– More than 16GB of RAM (initially 24GB for installation) on the physical host machine .
– 8GB of disk for the O/S.
– 50GB of disk for the basic HANA DB with nothing in it, plus the installed software.
– 20GB of disk on the physical host  for swapping (if you don’t have 24GB of RAM).
– 2 CPUs if you can spare the cores.
– A hostname and fully qualified domain name.
– Some form of networking (use “Bridged” if you need to access this across the network, I will be using “Host-Only”).
Let’s create the VM and set the CDROM to point to the SUSE Linux 12 SP3 install DVD ISO file:

We choose to do the install later to avoid the VMWare “EasyInstall” feature:



Set the initial hard disk to have 8GB and store it in one big file (it’s up to you really):

Now customise the hardware:
Set the RAM to 24GB or more (you really need 24GB of RAM, but I have only 16GB and will be ready for some serious swapping).  After installation, at a minimum the VM should have 18GB of RAM for day-to-day running:

Give the VM at least 2 cores:

Set the CD/DVD to use the SUS Linux installation ISO you downloaded:

Use bridged networking if you need to access over the network, but only if you have DHCP enabled or you’re a network guru.  I’m using “Host-only”:

I also removed the Sound Card and Printer.
Summary:

Start the VM.

We’re off.
The SUSE install took 12.5 minutes in my testing on a core i5 (unfortunately only 3rd gen 🙁  )
That is it for Part #1 of this guide.
>> Continue on to Part #2 for the completion of our HANA 2.0 install.