This blog contains experience gained over the years of implementing (and de-implementing) large scale IT applications/software.

Tag: http

Capture HTTP POST Using Simple Python Script

In this post I show a simple and quick way to capture a basic HTTP POST using Python to provide a basic HTTP Web Server with cgi capability in just a few lines of code and in most cases, it is executable on almost any Python capable server.

Why I Used This Code

I used this successfully to test an interface which wasn’t particularly clear exactly what data it was going to POST to a target web server.
Usually a developer could use real developer tools to do this analysis, however, the server doing the POST is POSTing to another server, and both servers sit behind firewalls. It was far quicker to create something simple that could be executed direct on the server.

What Does the Code Do?

The code simply creates a HTTP web server on the server on which it is executed.
The web server serves the content that exists in the directory structures from the current working directory and below.
The web server is also able to execute CGI scripts written in Python and stored in the cgi-bin subdirectory.

The Code

#!/usr/bin/python 
# 
import sys, os, cgi, cgitb

# Enable easy error reporting output. 
cgitb.enable()

# Create a custom log output to print to stdout and a log file. 
class CustomLogger(object): 
   def __init__(self): 
      self.terminal = sys.stdout 
      self.log = open("logfile.log", "a")

   def write(self, message): 
      self.terminal.write(message) 
      self.log.write(message)

   def flush(self): 
      pass

# Swap stdout for our custom log class. 
sys.stdout = CustomLogger() 
sys.stderr = sys.stdout

# Call the standard CGI test. 
cgi.test()

### END OF SCRIPT ###

Deploying the Code

To install the code, we need to create a new temporary directory on our host server, I used ssh to do this:

mkdir -p /tmp/dmg/cgi-bin

Put the code into the file called form.py in the cgi-bin directory:

cd /tmp/dmg/cgi-bin

vi form.py
[insert code then press shift-ZZ]

chmod 755 form.py

Still on an ssh session, switch back to the dmg directory and execute the Python CGI handler to listen on port 8080:

cd /tmp/dmg

python -m CGIHTTPServer 8080

Call your HTTP tool to POST to the address:
http://<your-server>:8080/cgi-bin/form.py

If the tool returns output, then you will see the output on your ssh session screen.
The output response from the CGI script is also stored in the /tmp/dmg/logfile.log file.

To quit/end the HTTP web server, simply press CTRL+C multiple times until you are returned to the command prompt.

The output will look like:

Content-type: text/html

Current Working Directory:
/tmp/dmg

Command Line Arguments:
['/tmp/dmg/cgi-bin/form.py', '']

Form Contents:
parameter: <type 'instance'>
MiniFieldStorage('parameter', 'test')

Shell Environment:
...

You will see the POST content in the “Form Contents” section of the output.
The values of fields are pre-fixed with “MiniFieldStorage“.

Also included in the output, is the execution environment which contains the environment variables that contain CGI related variables and their respective values such as HTTP_METHOD.

A Test Form

You can also deploy a simple form in order to test the CGI capability manually from a web browser (although this was not required in my case).
The form is simple HTML that POSTs two text input fields to our form.py CGI script:

<html><body>
<div style="text-align: center;">
<h1>Test Form</h1>
<form action="/cgi-bin/form.py" method="POST">
f1 : <input style="text-align: center;" name="f1" type="text" />
f2 : <input style="text-align: center;" name="f2" type="text" />
<input type="submit" value="Submit" />
</form>
</div>
</body></html>

The form should be saved to a new file called index.html in the /tmp/dmg directory.
You can then manually access the test web server using http://<your-server>:8080 and you will see the form.
Enter two values into the form and click submit, to see the output from your CGI script.

SAP Instance Agent as Mini-Web Server

In this post, I will show you how a little known feature of the SAP Instance Agent (available on every “modern” SAP system) can be used to serve files via HTTP and HTTPS.

You may be thinking, “This is basic stuff, I know this already“, in which case, you may only be interested in the very last paragraph of this post 😉

Why Not Just Use a Real Web Server?

During SAP projects there always comes a point where software needs to be distributed throughout the landscape, where end-users need access to a predefined set of software, or where scripts need to be centralised and downloadable onto multiple target servers.
Essentially, you need a common file distribution point.

Some projects have large budgets and some have small budgets. This post is for those with small budgets. Those projects where using everything twice, if possible, becomes an artform.

Under What Circumstances Would I Possibly Want to Use Such a Method?

Let’s imagine a scenario where you have a set of Korn shell scripts that you would like centralised across the SAP landscape.
There is/was no budget for a common fileshare and there is no budget to have someone setting one up.
Instead you develop an offline deployment approach, where the scripts are pulled down from a central repository on a schedule.
You decide that the central repository needs to be a web server and the scripts will be downloaded by HTTP.

What Is the SAP Instance Agent?

The SAP Instance Agent is the agent that does the work, when you run sapcontrol functions.
It is a small set of executables that come as part of the SAP Kernel and generally you get one Instance Agent installation per SAP instance.

For example, if you have an ASCS instance, there will be an instance agent installed under /usr/sap/<SID>/ASCS<##>/.

You can see the Instance Agent running by querying the list of running processes with ps:

>ps-ef | grep sapstartsrv

as1adm     1969      1  0 06:31 ?        00:00:01 /usr/sap/AS1/ASCS00/exe/sapstartsrv pf=/usr/sap/AS1/SYS/profile/AS1_ASCS00_sapas1ase1 -D -u as1adm

You can see that the binary executable responsible is “sapstartsrv”.

You will also notice that the SAP Host Agent also has a “sapstartsrv”. This is because the Host Agent and the Instance Agent are siblings, sharing a similar code-set, just with different functions.

How Do You Access the SAP Instance Agent?

In the history of the SAP product range, SAP created a Java based GUI tool called SAP MMC. The SAP MMC can be used to administer SAP instances on the local server via the Instance Agent. To be able to start the SAP MMC, it was distributed from the Instance Agent by HTTP. I’m not going to go into the SAP MMC because it will be going away completely eventually.

Generally, whenever you run sapcontrol and call a function, you are accessing the SAP Instance Agent:

sapcontrol -nr 00 -function GetSystemInstanceList

07.05.2020 07:01:32
GetSystemInstanceList
OK
hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
sapas1ase1, 0, 50013, 50014, 1, MESSAGESERVER|ENQUE, GREEN

What Is the Document Root For the Instance Agent?

In Web Server lingo, the “document root” is the highest level directory that the web server can serve files from.

For the Instance Agent, this is /usr/sap/<SID>/<INST>/exe/servicehttp.
Example: /usr/sap/AS1/ASCS00/exe/servicehttp.

If we create a simple text file in the document root, we can access it via HTTP:

echo "Hello Darryl" > /usr/sap/AS1/ASCS00/exe/servicehttp/index.html

We can use wget to access the file like so:

wget -q -O - http://127.0.0.1:50013/index.html

Hello Darryl

Can We Use HTTPS?

You can use HTTPS, the TCP port is the secure port 5##14 (50014 in our example).
Because the certificate used by the SAP Instance Agent is self-signed, you will need to trust the certificate directly.

We can use wget again as follows, but with the secure port and telling wget to not check the SSL certificate:

wget --no-check-certificate -q -O - https://127.0.0.1:50014/index.html

Hello Darryl

Are the Files Persisted Forever?

The files that you create under the document root of the SAP Instance Agent, are not persisted indefinately.
They get removed when the SAP instance is started.
Do not confuse this with when the Instance Agent is started.

An example, the Instance Agent is started up automatically by sapinit on server boot. Only when the SAP instance that is served by the Instance Agent, is started (i.e. sapcontrol -nr 00 -function Start), do the files in the document root location get cleansed, just before the instance is started.

I have previously put together a nice diagram about the interactions of various components during SAP instance startup. See the post here:

How an Azure hosted SAP LaMa Controlled SAP System Starts Up

How Can We Make Our Files Persist?

You can make your files persist, by creating a filesystem soft link from the document root and re-creating this link as part of the SAP instance start process:

mkdir /home/as1adm/docroot
echo "Hello Darryl" > /home/as1adm/docroot/index.html
ln -s /home/as1adm/docroot /usr/sap/AS1/ASCS00/exe/servicehttp/

Then you need to add an entry to the SAP instance profile to re-create the link on startup.
FIrst we need to establish the current number of “Execute” tasks in the profile:

cdpro
grep Execute_ *

AS1_ASCS00_sapas1ase1:Execute_00 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG0)
AS1_ASCS00_sapas1ase1:Execute_01 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG1)
AS1_ASCS00_sapas1ase1:Execute_02 = local rm -f $(_MS)
AS1_ASCS00_sapas1ase1:Execute_03 = local ln -s -f $(DIR_EXECUTABLE)/msg_server$(FT_EXE) $(_MS)
AS1_ASCS00_sapas1ase1:Execute_04 = local rm -f $(_EN)
AS1_ASCS00_sapas1ase1:Execute_05 = local ln -s -f $(DIR_EXECUTABLE)/enserver$(FT_EXE) $(_EN)

We add a new entry as follows:

echo "Execute_05 = local ln -s /home/as1adm/docroot $(DIR_EXECUTABLE)/servicehttp/" >> AS1_ASCS00_sapas1ase1

Upon starting the ASCS instance, the profile is read and the link created.

You would then access the index.html as follows:

wget --no-check-certificate -q -O - https://127.0.0.1:50014/docroot/index.html

Voila!

Even if you don’t use the above method for serving basic file content over HTTP, there is another use if you are running your SAP system in Azure.
We can use the HTTP capability of the SAP Instance Agent as a method to dynamically control which back-end VMs are accessible through an Azure ILB. This is an interesting concept, especially when introduced with a couple of other posts I have written over the past year.
I will elaborate further on this over the next few months.

CORS in a SAP Netweaver Landscape

In this brief article I’m going to try to simplify and articulate what Cross-Origin Resource Sharing (CORS) is, how it works and how in an SAP environment (we use Fiori in our example) we can get around CORS without the complexity of rigidly defining the resource associations in the landscape.

Let’s Look At What CORS Is:

Fundamentally CORS is a protection measure introduced in around 2014 inside Web browsers, to try and prevent in-browser content manipulation issues associated with JavaScipt accessing resources from other websites without the knowledge/consent of the Web browser user.

You may be thinking “Why is this a problem?”, well, it’s complex, but a simple example is that you access content on one Web server, which uses JavaScript to access content on another Web server.  You have no control over where the JavaScript is going and what it is doing.
It doesn’t mean the other Web server in our example, is malicious, it could actually be the intended victim of malicious JavaScript being executed in the context of the source Web server.

What Does “Consent” Mean?

There is no actual consent given by the Web browser user (you). You do not get asked.

It is more of an understanding, built into the Web browser which means the Web browser knows where a piece of JavaScript has been downloaded from (its origin), versus where it is trying to access content from (its target), and causes the Web browser to seek consent from the target Web server before allowing the JavaScript to make its resource request to the target.

A simple analogy:
Your parents are the Web browser.
You (the child) are the untrusted JavaScript downloaded from the source Web server.
You want to go and play at your friend’s house (the target Web server).
Your parents contact your friend’s parents to confirm it’s OK.
Your parents obtain consent for you to go and play and the type of play you will be allowed to perform, before they let you go and play at your friend’s house.

Based on the simple analogy, you can see that the Web browser is not verifying the content on the target, neither is it validating the authenticity of the target (apart from the TLS level verification if using HTTPS).
All the Web browser is doing, is recognising that the origin of the JavaScript is different to its target, and requesting consent from the target, before it lets the JavaScript make it’s resource request.

If the target Web server does not allow the request, then the Web browser will reject the JavaScript request and an error is seen in the Web browser JavaScript debugger/console.

What Does “Accessing” Mean?

When we talk about JavaScript accessing resources on the target Web server, we are saying that it is performing an HTTP call (XML HTTP), usually via the AJAX libraries using one of a range of allowed methods. These methods are the usual HTTP methods such as GET, PUT, POST, HEAD etc.

What is the flow of communication between origin Web server, Web browser and target Web server?

Below I have included a diagram that depicts the flow of communication from a user’s Web browser, between a Fiori Front-End Server (FE1) and a Back-End SAP system (BE2).

In the example, pay attention to the fact that the domain (the DNS domain) of the FE1 and BE2 SAP systems, are different.

So, for example the FE1 server could be fe1.group.corp.net and the BE2 server could be be2.sub.corp.net.

1, The user of the Web browser navigates within Fiori to a tile which will load and execute a JavaScript script from FE1.

2, The JavaScript contains a call to obtain (HTTP PUT) a piece of information into the BE2 system via an XML HTTP Request (XHR) call inside the JavaScript.

3, The user’s Web browser detects the JavaScript’s intention and sends a pre-flight HTTP request to the BE2 system, including the details about the origin of the JavaScript and the HTTP method it would like to perform.

4, The BE2 system responds with an “allow” response (if it wishes to allow the JavaScript’s request).

5, The Web browser permits the JavaScript to make its request and it sends it’s HTTP request to BE2.

What Needs to Be Configured in BE2?

For the above situation to work, the BE2 system needs to be configured to permit the required HTTP methods from JavaScript on the origin FE1.

This means that a light level of trust needs to be added to configuration of BE2. This is documented in SAP notes and help.sap.com for NW 7.40 onwards.

Is There a Simpler Way?

An alternative method to configuring Netweaver itself, is to adjust the ICM on the target (BE2) to rewrite the inbound HTTP request to add a generic “origin” request. This means you can have many domains making the access request, without needing to maintain too much configuration at the cost of security.
I’m thinking more about what needs to be done, not just in production, but it in all DEV, TST and PrePRD systems, plus config re-work after system copies.
Not only this, but it would be difficult for your URL rewrite to be accurate, so it may end up being applied to all URL accesses, no matter where they come from.  This will impact performance of the Web Dispatcher.
You could solve the performance issue by using a different front-end IP address (service name) for your Web Dispatcher, which is used specifically for requests from your origin system (FE1).  Another option could be (if it’s your own code being called in BE2) to apply a URL path designation e.g. “/mystuff/therealstuff”, whereby the ICM on BE2 can match based on “/mystuff” and rewrite the URL to be “/therealstuff”.

How About an Even Simpler Way?

A much better way, which solves the CORS problem altogether and removes the need to place config on individual systems, is to front both the origin and the target behind the same Web Dispatcher.

This way, CORS becomes irrelevant as the domain of the Web Dispatcher is seen by the Web browser, as both the origin and the target.

To enable the above configuration, we need to ensure that we align the Web Dispatcher DNS domain to either the origin or the target.
It has to be aligned to whichever system we use the message server to load balance the HTTP call. This is a SAP requirement.

For the other back-end server (behind the Web Dispatcher), we use the EXTSRV option of the Web Dispatcher to allow it to talk to the BE2 system.
This has the capability of supplying multiple servers for HA and load balancing (round-robin).   It also means the DNS domain of that system can be different to that of the Web Dispatcher’s.

You may also be interested in:

SAP NW 7.31 Java – HTTP Tracing

Scenario:  You would like to trace a HTTP connection to a Netweaver 7.31 Java stack.

Prior to NW 7.31 you would have enabled tracing in either the old Java Administrative Tool, or newer Netweaver Administrator by setting the “enable” flag for tracing parameters of the HTTPProvider dispatcher service (see my previous blog article here).

Since Netweaver 7.31, the Java stack now includes an ICM (Internet Communication Manager) which is the main access point to which HTTP communications are routed to the Java server nodes.
Therefore, tracing in NW 7.31 Java is actually more like the ABAP stack tracing, because you simply increase the trace level in the ICM and then check the dev_icm trace file.

So the next question is, how do you access the ICM to increase the trace level in a Java stack?
Well this can be performed with the help of the SAPMC (only one “M” and not to be confused with SAP MMC for Windows).
The SAPMC was introduced in later NW 7.0 patches and is part of the sapstartsrv service.
The sapstartsrv actually listens on TCP port 5<##>13.  So if you have access through your firewall, navigate to https://<app server>:5<##>13   and the the Java based applet will start up (note that you’ll need Java 1.4 or above on your PC).

Once loaded, you can see the SAP MC and expand the “AS Java” branch to locate the “Process Table” item:

SAP MC management console

From the “Process Table” you should be able to right click the ICM (on the right) and increment the trace level:

SAP MC increase ICM trace level

The trace level is dynamically increased.
To see the trace file, right click again and select “Show Developer Trace“:

SAP MC ICM display trace

You can always see the current trace level by right clicking the ICM and selecting “Parameters…”, or actually right clicking the ICM node further up the tree and selecting “Parameters…”:

SAP MC show ICM parameters

The current trace level is shown as the “rdisp/TRACE” parameter value on the Dispatcher tab:

SAP MC rdisp/TRACE ICM

Again, right clicking the actual ICM node in the tree gives you direct access to the ICM management web page:

MWSnap106 2014-07-23, 09_39_44

From the ICM management page, you can also see the current trace level:

ICM Manager Monitor

Good luck with your tracing.

HowTo: SAP NW731 Java Full HTTP Trace With Headers Content & Timings

Scenario: You have a SAP Java stack which is hosting either some SAP components/modules, or it’s hosting your own custom Java code.
You would like to perform a full HTTP trace so that you can see the HTTP headers and returned content during a HTTP session between your browser and the SAP HTTP Server.  You also wish to see timings for the processing time of the response.
Following SAP note “724719 – How to enable HTTP tracing in the SAP J2EE Engine 6.40/7.0”  connect into the SAP Netweaver Administrator main web page as J2EE_ADMIN:
SAP NWA J2EE_ADMIN
On the left-hand side, select “System Properties”:
SAP NWA System Properties
Select the dispatcher item under “<SID> -> Instance -> Dispatcher”:
SAP NWA System Properties despatcher
On the Services tab, select “http”:
SAP NWA System Properties despatcher http service
In the “Properties” tab at the bottom, select “HttpTrace” and click the “Modify” button:
SAP NWA System Properties dispatcher http service httptrace
image
Now change the trace setting to the following:
SAP NWA System Properties dispatcher http service httptrace enable
The possible values are described in the SAP note as:
enable
         Using this option will cause the whole requests/responses to be written
enableHeaders
        Using this option will cause only the headers of each request/response to be written
enableHex
        Using this option will cause the whole requests/responses to be written in a 16-column hexadecimal format. This option is valid for J2EE Engine SP10 or higher
enableHexHeaders
        Using this option will cause only the headers of each request/response to be written in a 16-column hexadecimal format. This option is valid for J2EE Engine SP10 or higher
The SAP note also recommends to set the property HttpTraceTime, to “true”.
This is described further in SAP note “972540 – Tracing dispatcher and server response times“.
Since you would only get the dispatcher processing time, you should also set the server trace property as described in the above note, to allow you to see the server processing time:

“1) dispatcher + server processing time trace in http access log
– On the dispatcher enable Http Provider#s property #HttpTraceTime#
– On the servers enable Http Provider#s property #LogResponseTime#
Then messages in the httpaccess response file will contain additional fields  d[…] and c[…] where d is dispatcher and server processing time and c is client id.
2) additional traces – if the request is processed for more than XX seconds
– there is a new Http Provider#s property on the server # #TraceResponseTimeAbove#
– if its value is -1 then there will be no additional traces
– if its value is >-1 (the value represents the response time in ms)
    then additional traces will be written in the default trace file only for these requests which response time is >= property#s value
– on the servers put the log severity for HttpProvider service to DEBUG“

Once you have enabled the tracing, no restart of the Java stack is required.  It is activate immediately.
Simply check the trace output file /usr/sap/<SID>/<instance>/j2ee/cluster/dispatcher/log/services/http/req_resp.trc  (for NW731+) using operating system level file viewers (or AL11 in an ABAP stack).
For information, you should be aware of the HTTP 1.1 status codes, so you know what to look for if you are tracing an authorisation error (HTTP authorisation) or some other HTTP related issue  https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html .