This blog contains experience gained over the years of implementing (and de-implementing) large scale IT applications/software.

Category: Shell

Generate HMAC for Azure Storage from KSH

Generating an Azure HMAC Signature for calling Azure Storage Services from KSH

While custom writing an Azure Storage Service blob deletion script, I experienced a problem using the OpenSSL method for generating an HMAC.

For those not familiar with Azure Storage Services (or even signature based authentication) the act of sending a signature as part of an HTTP request serves to prove to the target server that you are in possession of the secret key and that you also would like to perform a specific operation.

The shared key (that you have been given out-of-band) is used to sign the HTTP call. This is so the target server can then perform the same signing operation at its end, and if the signature it obtains matches the one you’ve sent, then it trusts and permits you to perform the specific HTTP operation you’ve requested.

See here for more details: https://docs.microsoft.com/en-us/rest/api/storageservices/authenticate-with-shared-key#Constructing_Element

In my example, the operation is a simple BLOB deletion from an Azure Storage Account, but that is irrelevant to this particular post.
The HMAC generation routine is the same no matter what HTTP operation you wish to perform.

Based on searching in Google, the following OpenSSL method seems popular and able to provide a method of generating an HMAC:

l_input=”your HTTP operation to be signed”
l_key=”your big long Azure storage account key”
l_key_decoded=”$(echo -n “${l_key}”|base64 -d)”
l_hmac=”$(echo -n “${l_input}”|openssl dgst -sha256 -hmac “${l_key_decoded}” -binary | base64)”

The above works, with KSH, most of the time.
There have been one or two occasions when for no apparent reason, an incorrect HMAC is generated.
It’s possible that this stems from the character set interpretation e.g. UTF-8 and/or some strangeness in the way the KSH interpreter works with specific characters. I really wasn’t able to investigate deep enough with the time I had.

Instead of the above, I decided to take a leaf out of the Blobxfer utility team’s book and use a Python based solution instead.
Browsing the Blobxfer source in GitHub, I isolated the specific Python routine that was used to provide the HMAC.
Putting this routine into KSH makes it look like the following:

l_hmac=”$(cat <<EOF | python –
import sys
import hmac
import hashlib
import base64

def _encode_base64(data):
encoded = base64.b64encode(data)
return encoded

def _decode_base64_to_bytes(data):
return base64.b64decode(data)

def _sign_string(key, string_to_sign):
key = _decode_base64_to_bytes(key.encode(‘utf-8’))
string_to_sign = string_to_sign.encode(‘utf-8’)
signed_hmac_sha256 = hmac.HMAC(key, string_to_sign, hashlib.sha256)
digest = signed_hmac_sha256.digest()
encoded_digest = _encode_base64(digest)
return encoded_digest

data = “””${l_input}”””
key = “${l_key}”
print (_sign_string(key,data))
EOF
)”

I’m using a combination of HERE document and KSH in-line sub-shell execution to call python and pass in the stdin containing the python code to be executed.
KSH is responsible for embedding the required variables into the Python code, such as l_input and l_key.

So far, this routine has proved successful 100% of the time.

Downloading SAP Download Basket Contents in the Cloud on Linux

Scenario: You’ve made the move to the cloud.  You are about to install some SAP software on a new Linux server hosted in the cloud and you just need to get the installation media uploaded.
Except, you don’t want to have to start X-Windows or setup a specific Windows server just to do this.

You have two options to achieve this, you can either:
Upload the files from your local network using SFTP or SCP (maybe you have ExpressRoute or DirectConnect) and have a fast connection.

or

Why not just download them straight from your SAP download basket into the target server.
What’s the difference between the two?  Well, it depends on whether you’ve already got access to the installation media locally, or if you are intending to first download it locally from SAP, then upload it to your cloud hosted server.
This doesn’t make sense to download then re-upload to the cloud.
Therefore, here’s how you can download straight from your SAP download basket.

What you will need:
– A web browser on your local PC.
– A valid SAP S-user account.
– Disk space on a cloud hosted Linux server (I’m going to show you the Linux way).
– The cloud hosted server will need internet access.

Setup Download Basket:
Before we start downloading, you need to cleanup your download basket.
Go to https://launchpad.support.sap.com/#/downloadbasket
Remove everything already in there that is not the items you want to download to the cloud hosted server.
The reason we must do this is because it’s difficult to know which items relate to which files in the download basket (you’ll see in a moment).
Now you can add the specific items you wish to download, into your download basket.
I’ve got two items:

Export Download Basket:
In the download basket click the “Export Links to Text File” button:

This will generate a text document which you can open on your PC called myDownloadBasketFiles.txt.

Open the text file using notepad:

I have 2 lines for the two items in my download basket.  I don’t know which is which.  I can suppose they are in order, but I don’t really know for sure.
Take the first line and log onto your cloud hosted Linux server as your preferred user.
Ping the softwaredownload.sap.com server using the Linux utility wget to make sure you can see the server.

> wget https://softwaredownloads.sap.com

A successful ping will show a HTTP error 401 and “Authorization failed“:

We have confirmed that our cloud hosted Linux server can see the SAP download server.
Switch to your target download directory (somewhere with disk space):

> cd mydownloadsdirectory

We now call wget again and pass the first item to download:

> wget –http-user=”[your s-user]” –http-password=”[your password]” [the first url] –output-document=1file.SAR

Adjust the command line above to put in your S-user account name and password.
You should also change the last parameter to give your file a name.

We have to guess that it is a SAR file.  Once it’s downloaded you can always re-name and test the extraction using “SAPCAR -tvf thefile.SAR”.

The wget utility will save the file in the current directory.
Repeat the same command line, changing the URL and the output document file, for each of the remaining items in the notepad text file of your download basket.

As you can see, using wget will allow you to script the download process so that you could schedule an overnight download of software.

HowTo: Script eMail With Attachment Direct via SMTP Server

Have you ever wondered how you can directly send an email with a file attachment, direct from a shell script via an SMTP server?
That is, without needing to rely on any mail or sendmail setup on the UNIX/Linux server.
Well, it’s simple.

Create a script that encodes the file you wish to attach, into BASE64, then type the following to send the file:

telnet somesmtpserver.net 25<<EOF
helo yourserver.net
mail from: noreply@mymail.net
rcpt to: noreply@mymail.net
data
Subject: Test attachment

Mime-Version: 1.0
Content-Type: multipart/mixed; boundary=attachmentbound

--attachmentbound
The email body

--attachmentbound
Content-Type: application/octet-stream; name="attachmentfile"
Content-Disposition: attachment; filename="thefilename.txt"
Content-Transfer-Encoding: base64

VGVzdCBBdHRhY2htZW50     <<-- Insert your BASE64 content here!

.
quit
EOF

You need to have the content already encoded in BASE64.
For this you could use a Perl script using the MIME::Base64 module as shown below:


#!/usr/bin/perl
use MIME::Base64;
#printf ("%s", encode_base64(eval ""$ARGV[0]""));

## Check to see if each and every file exists ##
foreach $file (@ARGV)
{
if (not -e $file)
{
print "File "$file" does not existn";
exit( 1 );
}

# Convert file to BASE64
open( OUTFILE, ">$file.out") or die "Could not open outputfile: $!n";;
open( FILE, $file ) or die "Could not open inputfile: $!n";
{
local($/) = undef;
print OUTFILE encode_base64(<FILE>);
}
close( FILE );
close( OUTFILE );
}


Script Korn Shell to use Bold Output

Here’s how you can use a normal Korn shell script to put out bold text on the screen providing the emulator you are using supports it:

#!/bin/ksh
$ bold=`tput smso`
$ regular=`tput rmso`

$ echo "${bold}Please type in your name: c"
$ echo "${regular}OK"

See the man page for tput for a wild array of other capabilities.
It’s possible you can even use tput to draw in different parts of the screen.
This could be used to create a drop-down box or a menu select box in KSH.
It can be done!  Honest!