This blog contains experience gained over the years of implementing (and de-implementing) large scale IT applications/software.

SAP’s Private Innovations Made Public

In July 2023, SAP announced that future innovations and capabilities will only be available in SAP public cloud and SAP private cloud with a RISE subscription.

The audio version is available to listen to on the SAP Press site, but my favourite go-to for quoted interviews is the diginomica site:

https://diginomica.com/cloud-revenue-growth-misses-sap-q2-future-bright-ai-according-ceo-klein

…It’s also very important to emphasize that SAP’s newest innovations and capabilities will only be delivered in SAP public cloud and SAP private cloud using RISE with SAP as the enabler. This is how we will deliver these innovations with speed, agility, quality and efficiency. Our new innovations will not be available for on-premise or hosted on-premise ERP customers on hyperscalers.

What can we take away from this statement?

Firstly, we should note the absence of the product name “S/4HANA”.  It’s not like SAP to miss the opportunity to include the product name in a discussion.  Not once, but twice the opportunity was not used to insert “S/4HANA” into the conversation.
What the quote is saying is exactly that:  “…SAP’s newest innovations and capabilities will only be delivered in SAP public cloud and SAP private cloud using RISE with SAP…”.

Short version: if you buy RISE, you get the newest innovations and capabilities.  This is not explicitly saying they will be included in the S/4HANA product.

This is because in SAP’s own marketing material, RISE with SAP is a solution that includes:

  • SAP S/4HANA Cloud (one of the two “cloud” editions).
  • Business Process Transformation
  • Business Platform and Analytics
  • SAP Business Network
  • Outcome Driven Services and Tools

There are plenty of places for innovation to happen in that list and it doesn’t not mean S/4HANA specifically.

Also we have to consider this fact; for SAP to branch the S/4HANA code base from S/4HANA Private Cloud Edition and S/4HANA On-Premise Edition, would cause a lot of development and support effort from now until around 2040.

What if an “on-premise” S/4HANA customer, already at a recent S/4HANA version, decided to buy RISE with SAP?  If the code base was massively different it would be a system migration to lift it into a comparable system in RISE.

Instead, by providing these new innovations in some form of BTP hosted service which would only be accessible via RISE with SAP, the S/4HANA code can remain as it is (clean core lovers would like this), albeit with some special user-exits or extensibility points or even an Addon; then the new innovations would be provided by future-proof, containerised BTP services.

This would also allow SAP to leave the option open to eventually offer these innovations, at a much later point in time, to non-RISE customers at a premium, maybe.  Especially if they are truly innovative.  Who would give up that option to get more money by simply un-restricting access for the wider customer base, to what would be at the future time, old innovations.

The second point we note is that these services may not be included in RISE with SAP for free.

It is, after all, a subscription based service.

“…using RISE with SAP as the enabler.

Access is provided/enabled through the RISE subscription, but it sounds to me like this will be another request ticket with some contractual costs or additional consumption credits.

Why Restrict Innovations?

Another line of questioning has to be: why? – Why restrict these new innovations from on-premise customers?

Apart from the obvious suggestion that it simply adds pressure for customers to take a RISE with SAP subscription, there is another idea and it adds to the thinking that the innovation is not being delivered directly in S/4HANA.


These new innovations may not be easily integrated with an on-premise solution.

For “on-premise” we have to bear in mind that it does include both systems hosted physically on-premise (in a customer’s own datacentre) in geographical locations that are far from any SAP Cloud entry/exit points, and also those systems hosted in hyperscalers.  They are one and same version of SAP S/4HANA On-Premise.

SAP BTP hosted services need the SAP Cloud Connector as integration between BTP and an on-premise solution.

The SAP Cloud Connector is a secure one-way TLS tunnel, over which bi-directional application comms can flow.
It is not built for very large datasets and definitely not for precise real-time integration.

For customers hosted under a RISE with SAP subscription, maybe there is some new connectivity solution that can be deployed by SAP that allows a more secure, lower latency connectivity with true bi-directional flow between the SAP system and SAP BTP services.  This is what would be needed to provide innovations that require true real-time AI interaction with large data sets.

Maybe this is the reason why on-premise customers will not get these new innovations outside of RISE with SAP?

SAP S/4HANA != SAP S/4HANA

If you thought S/4HANA Private Cloud Edition  ===  S/4HANA On-Premise Edition  – then you would be wrong!

The S/4HANA Private Cloud Edition and SAP S/4HANA On-Premise Edition are similar but not the same.
The blog post I’ve linked here to goes to lengths to explain that “…the functional differences between SAP S/4HANA Cloud, private edition, and SAP S/4HANA On-Premise are only features that belong to the compatibility scope Matrix“.

If you are migrating to S/4HANA, those differences may not be something to be taken lightly!

Three editions of S/4HANA, three editions with *growing* differences:

  • Private Cloud Edition (for RISE with SAP but a.k.a “Cloud, private edition”).
  • On-Premise Edition (for those that know sense ;-p ).
  • Public Cloud Edition (a.k.a “Cloud, public edition” – but whichever way around you want to use it).

Consider the ramifications from that blog post, if you are debating going for RISE with SAP and S/4HANA Private Cloud Edition.  There may be a lot more to be done during a RISE provided conversion, compared to a slower angle of attack with a non-RISE conversion and S/4HANA On-Premise Edition.

If you are considering which S/4HANA On-Premise Edition you should look towards, then the 2023 version is probably going to be the last possible target you can get to for various reasons explained in my previous post here.

Release 2023 of SAP S/4HANA Last Target

The 2023 release of SAP S/4HANA is due in October 2023.
This release is likely the last release that existing SAP ERP customers can reasonably migrate to.

Here’s why

Any customer migrating to S/4HANA from SAP ERP will need to perform all the analysis and preparation work against this 2023 S/4HANA release (functional capabilities will be known).

The next S/4HANA release will be in late 2025 because after the 2023 release the product moves to a new 2 year release cycle (instead of annually).

This 2025 date is likely to be too late for any reasonably large customer to migrate to, as it would leave only 2 years until 2027 (end of maintenance for SAP ERP) and taking into consideration of the start of end-of-maintenance of Compatibility Scope items (see my other post about those here).

Running an SAP ERP to S/4HANA migration project quicker than 2 years is possible, but this depends heavily on existing environment complexity.

I would bet that for most, 2023 is *the* target S/4HANA release.  The saint release as I’m calling it.

Reference link: https://news.sap.com/2022/09/new-sap-s4hana-release-maintenance-strategy/

SAP S/4HANA Compatibility Pack EOM

I don’t know about you but my LinkedIn feed is becoming uselessly full of “you must move to SAP S/4HANA” posts, with no real business case for doing so apart from the old “you will be left behind…” justification.
It is also becoming full of posts claiming that SAP S/4HANA Private Cloud Edition is the same as SAP S/4HANA On-Premise Edition.

Well, I’m ashamed to say, this is one of those “you will be left behind…” posts, but with an added “…in a boat with a hole and no bucket.”.
As a by-product it will also prove that SAP S/4HANA PCE is most definately not the same as SAP S/4HANA On-Premise Edition.

Some blogs on blogs.sap.com get many, many thousands of views, mainly because they answer the most common questions.
There are some questions however, that few ask and this one has brought me to this excellent blog post linked here: https://blogs.sap.com/2023/05/03/functional-differences-between-sap-s-4hana-cloud-private-edition-and-sap-s-4hana-on-premise.

When moving to SAP S/4HANA, some existing SAP ERP customers were/are allowed to carry over features/developments originally designed for SAP ERP.
These are called “Compatibility Packs” or “Compatibility Scope Items” and are detailed further in links in the blog post that I am linking (I recommend you read the linked blogs from that blog post).

In the blog post I reference it says:

“…functionalities that were originally created for SAP ERP that are currently available in SAP S/4HANA. But SAP has authorized customers to use these functionalities in SAP S/4HANA for a limited time…

Oh really.  Until what time is this “limited time” you may be thinking?

…typically until the end of 2025, with a few exceptions that will be available until 2030.“.

Well, that is great for those customers that have already made the move to S/4HANA.
What about those that are not yet on S/4HANA?   Does that 2025 date extend out to 2027?  No, it does not!

The impact therefore, is if you are late migrating to the S/4HANA party, all the good food is already gone!

Those companies moving to S/4HANA before 2025 will get access to those Compatibility Packs to help smooth their transition from SAP ERP (if they need them).

Those who are moving to S/4HANA between 2025 and 2030, will see some of those Compatibility Packs become unsupported. 

They can be used (SAP will not delete transaction codes etc) but they will be unsupported and unmaintained (think about security, legal, integrity issues and not just bugs).

Finally, regarding SAP S/4HANA On-Premise vs SAP S/4HANA Private Cloud Edition. These are similar but not the same.

The blog post goes to lengths to explain that “…the functional differences between SAP S/4HANA Cloud, private edition, and SAP S/4HANA On-Premise are only features that belong to the compatibility scope Matrix…“.  
Voila!

Three editions of SAP S/4HANA, three products with differences:

  • Private Cloud Edition (for RISE with SAP).
  • On-Premise Edition (for those that know sense ;-p ).
  • Public Cloud Edition.

Cluster Config Issue for SAP ERS Instance

Running SAP Netweaver A(SCS)/ERS in a Pacemaker cluster in Azure, AWS or GCP or potentially even on-premise?

Be aware, there is a configuration issue in the current version of the Microsoft, AWS, GCP and SUSE documentation for the Pacemaker cluster configuration (on SLES with non-native SystemD Startup Framework) for the SAP Enqueue Replication Server (ERS) instance primitive in an ENSA1 (classic Enqueue) architecture.


Having double checked with both the SAP and SUSE documentation (I don’t have access to check RHEL) I believe that the SAP certified SUSE cluster design is correct, but that the instructions to configure it are not inline with the SAP recommendations.

In this post I explain the issue, what the impact is and how to potentially correct it.


NOTE: This is for SLES systems that are not using the new “Native Startup Framework for SystemD” Services for SAP, see here.

Don’t put your SAP system at risk, get a big coffee and let me explain below.

SAP ASCS and High Availability

The Highly Available (HA) cluster configuration for the SAP ABAP Central Services (ASCS) instance is critical to successful operation of the SAP system, with the SAP Enqueue (EN) process being the guardian of the SAP application level logical locks (locks on business objects) and the SAP Enqueue Replication Server (ERS) instance being the guarantor for the EN when a cluster failover occurs.

In a two-node HA SAP ASCS/ERS setup, the EN process is on the first server node and the ERS instance is on the second node.
The EN proccess (within the ASCS instance) is replicating to the ERS instance (or rather, the ERS is pulling from the EN process on a loop).


If the EN process goes down, the surrounding cluster is usually configured to fail over the ASCS instance to the secondary cluster node where the ERS instance is running. The EN process will then take ownership of the replica in-memory lock table:

What is an ideal Architecture design?

In an ideal design, according to the SUSE documentation here.
(again, I’m sure RHEL is similar but if someone can verify that would be great), the ASCS and ERS instances are installed on cluster controlled disk storage on both cluster nodes in the cluster:

We mount the /sapmnt (and potentially /usr/sap/SID/SYS) file system from NFS storage, but these file systems are *not* cluster controlled file systems.
The above design ensures that the ASCS and ERS instances have access to their “local” file system areas before they are started by the cluster. In the event of a failover from node A to node B, the cluster ensures the relevant file system area is present before starting the SAP instance.

We can confirm this by looking at the SAP official filesystem layout design for an HA system here:

What is Microsoft’s Design in Azure?

Let’s look at the cluster design on Microsoft’s documentation here:

It clearly shows that /usr/sap/SID/ASCS and /usr/sap/SID/ERS is being stored on the HA NFS storage.
So this matches with the SAP design.

What is Amazon’s design in AWS?

If we look at the documentation provided by Amazon here:

We can see that they are using EFS (Elastic File Storage) for the ASCS and ERS instance locations, which is mounted using the NFS protocol.
So the design is the same as Microsoft’s and again this matches the SAP design.

What is Google’s design in GCP?

If we look at the documentation provided by Google, the diagram doesn’t show clearly how the filesystems are provided for the ASCS and ERS, so we have to look further into the configuration step here:

The above shows the preparation of the NFS storage.

Later in the process we see in the cluster configuration that the ASCS and ERS file systems are cluster controlled:

and

The above is going to mount /usr/sap/SID/ASCS## or /usr/sap/SID/ERS## and they will be cluster controlled.
Therefore the GCP design is also matching the SAP, Azure and AWS designs.

Where is the Configuration Issue?

So far we have:

  • Understood that /sapmnt is not a cluster controlled file system.
  • established that Azure, AWS, GCP and the SUSE documentation are in alignment regarding the cluster controlled file systems for the ASCS and the ERS.

Now we need to pay closer attention to the cluster configuration of the SAP instances themselves.

The Pacemaker cluster configuration of a SAP instance involves 3 (or more) different cluster resources: Networking, Storage and SAP instance. With the “SAP Instance” resource being the actual running SAP software process(es).

Within the cluster the “SAP Instance” Resource Adapter (RA) is actually called “SAPInstance” and in the cluster configuration is takes a number of parameters specific to the SAP instance that it is controlling.
One of these parameters is called “START_PROFILE” which should point to the SAP instance profile.

The SAP instance profile file is an executable script (on Linux) that contains all the required commands and settings to start (and stop) the SAP instance in the correct way and also contains required configuration for the instance once it has started. It is needed by the SAP Instance Agent (sapstartsrv) and the executable that is the actual SAP instance (ASCS binaries: msg_server and enserver, ERS binary: enrepserver).
Without the profile file, the SAP Instance Agent cannot operate a SAP instance and the process that is the instance itself is unable to read any required parameter settings.

Usually, the SAP Instance Agent knows where to find the instance profile because at server startup, the sapinit script (triggered through either systemd unit-file or the old Sys-V start scripts) will execute the entries in the file /usr/sap/sapservices.
These entries call the SAP Instance Agent for each SAP instance and they pass the location of the start profile.

Here’s a diagram from a prior blog post which shows what I mean:

In our two-node cluster setup example, after a node is started, we will see 2 SAP Instance Agents running, one for ASCS and one for ERS. This happens no matter what the cluster configuration is. The instance agents are always started and they always start with the profile file specific in the /usr/sap/sapservices file.
NOTE: This changes in the latest cluster setup in SLES 15, which is a pure SystemD controlled SAP Instance Agent start process.

The /usr/sap/sapservices file is created at installation time. So it contains the entries that the SAP Software Provisioning Manager has created.
The ASCS instance entry in the sapeservices file, looks like so:

LD_LIBRARY_PATH=/usr/sap/SID/ASCS01/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/SID/ASCS01/exe/sapstartsrv pf=/usr/sap/SID/SYS/profile/SID_ASCS01_myhost -D -u sidadm

But the ERS instance entry looks slightly different:

LD_LIBRARY_PATH=/usr/sap/SID/ERS11/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/SID/ERS11/exe/sapstartsrv pf=/usr/sap/SID/ERS11/profile/SID_ERS11_myhost -D -u sidadm

If we compare the “pf=” (profile) parameter entry between ASCS and ERS after installation, you will notice the specified ERS instance profile location is not using the location accessible under the link /usr/sap/SID/SYS.
Since we have already investigated the file system layout, we know that the “SYS” location only contains links, which point to other locations. In this case, the ASCS is looking for sub-directory “profile”, which is a link to directory /sapmnt/SID/profile.
The ERS on the other hand, is using a local copy of the profile.

This is something that usually would go unnoticed, but the ERS must be using a local copy of the profile for a reason?
Looking at SAP notes, we find SAP note 2954193, which explains that an ERS instance in an ENSA1 architecture should be started using a local instance profile file:

Important part: “this configuration must not be changed“.
Very interesting. It doesn’t go into any further detail, but from what we have understood about the criticality of the SAP ASCS and ERS instances, we have to assume that SAP have tested a specific failure scenario (maybe failure of sapmnt) and deemed it necessary to ensure that the ERS instance profile is always available.
I can’t think of any other reason (maybe you can?).

The next question, how does that ERS profile get created in that local “profile” location? It’s not something the other instances do.
After some testing it would appear that the “.lst” file in the sapmnt location is used by the SAP Instance Agent to determine which files to copy at instance startup:

It is important to notice that the DEFAULT.PFL is also copied by the above process.
Make sure you don’t go removing that “.lst” file from “/sapmnt/SID/profile”, otherwise those local profiles will be outdated!

To summarise in a nice diagram, this setup is BAD:

This is GOOD:

What about sapservices?

When we discussed the start process of the server, we just mentioned that the SAP Instance Agent is always started from the /usr/sap/sapervices file settings. We also noted how in the /usr/sap/sapservices file, the settings for the ERS profile file location are correct.
So why would the cluster affect the profile location of the ERS at all?
It’s a good question, and the answer is not a simple explanation because it requires a specific circumstance to happen in the lifecycle of the cluster.

Here’s the specific circumstance:

  • Cluster starts, the ERS Instance Agent was already running and so it has the correct profile.
  • We can run “ps -ef | grep ERS” and we would see the “er” process has the correct “pf=/path-to-profile” and correctly pointing to the local copy of the profile.
  • If the ERS instance somehow is now terminated (example: “rm /tmp/.sapstream50023”) then the cluster will restart the whole SAP Instance Agent of the ERS (without a cluster failover).
  • At this point, the cluster starts the ERS Instance Agent with the wrong profile location, and the “er” binary now inherits this when it starts. This will be inplace until the next complete shutdown of the ERS Instance Agent.

As you can see, it’s not an easy situation to detect, because from an outside perspective, the ERS died and was successfully restarted.
Except it was restarted with the incorrect profile location.
If a subsequent failure happens to the sapmnt file system, this would render the ERS at risk (we don’t know the exact risk because it is not mentioned in the referenced SAP note that we noted earlier).
What is more, the ERS instance is not monitorable using SAP Solution Manager (out-of-the-box), you would need to create your own monitoring element for it.

Which Documentation has this Issue?

Now we know there is a difference required for the ERS instance profile location, we need to go back and look at how the cluster configurations have been performed, because of “this configuration must not be changed”!

Let’s look first at the SUSE documentation here:

Alright, the above would seem to show that “/sapmnt” is used for the ERS.
That’s not good as it doesn’t comply with the SAP note.

How about the Microsoft documentation for Azure:

No that’s also using /sapmnt for the ERS profile location. That’s not right either.

Looking at the AWS document now:

Still /sapmnt for the ERS.

Finally, let’s look at the GCP document:

This one is a little harder, but essentially, the proposed variable “PATH_TO_PROFILE” looks like it is bound to the same one as the ASCS instance defined just above it, so I’m going to say, it’s going to be “/sapmnt” because when you try and change it on one, it forces the same on the other:

We can say that all documentation available for the main hyperscalers, provides an incorrect configuration of the cluster, which could cause the ERS to operate in a way that is strongly not recommended by SAP.

Should we correct the issue and How can we correct the issue?

I have reported my finding to both Microsoft and SUSE, so I would expect them to validate.
However, in the past when providing such feedback, the relevant SAP note has been updated to exclude or invalidate the information altogether, rather than instigating the effort of fixing or adjusting any incorrect configuration documentation.
That’s just the way it is and it’s not my product, so I have no say in the solution, I can only report on what I know is correct at the time.

If you would like to correct the issue using the information known at this point in time, then the steps to be taken to validate that the ERS is operating and configured in the correct way are provided in a high-level below:

  1. Check the cluster configuration for the ERS instance to ensure it is using the local copy of the instance profile.
  2. Check the current profile location used by the running ERS Instance Agent (on both nodes in the cluster).
  3. Double check someone has not adjusted the /usr/sap/sapservices file incorrectly (on both nodes in the cluster).
  4. Check that the instance profile “.lst” file exists in the /sapmnt/SID/profile directory, so that the ERS Instance Agent can copy the latest versions of the profile and the DEFAULT.PFL to the local directory location when it next starts.
  5. Check for any differences between the current local profile files and the files in the /sapmnt/SID/profile directory and consider executing the “sapcpe” process manually.

Thanks for reading.