This blog contains experience gained over the years of implementing (and de-implementing) large scale IT applications/software.

SAP ASE Error – Process No Longer Found After Startup

This post is about a strange issue I was hitting during the configuration of SAP LaMa 3.0 to start/stop a SAP ABAP 7.52 system (with Kernel 7.53) that was running with a SAP ASE 16.0 database.

During the LaMa start task, the task would fail with an error message: “ASE process no longer found after startup. (fault code: 127)“.

When I logged directly onto the SAP server Linux host, I could see that the database had indeed started up, eventually.
So what was causing the failure?

The Investigation

At first I thought this was related to the Kernel, but having checked the versions of the Kernel components, I found that they were the same as another SAP system that was starting up perfectly fine using the exact same LaMa system.

The next check I did was to turn on tracing on the hostagent itself. This is a simple task of putting the trace value to “3” in the host_profile of the hostagent and restarting it:

service/trace = 3

The trace output is shown in a number of different trace files in the work directory of the hostagent but the trace file we were interested in is called dev_sapdbctrl.

The developer trace file for the sapdbctrl binary executable is important, because the sapdbctrl binary is executed by SAP hostagent (saphostexec) to perform the database start. If you observe the contents of the sapdbctrl trace output, you will see that it loads the Sybase specific shared library which contains the required code to start/stop the ASE database.

The same sapdbctrl also contains the ability to load the required libraries for other database systems.

As a side note, it is still not known to me, how the Sybase shared library comes to exist in the hostagent executable directory. When SAP ASE is patched, this library must also be patched, otherwise how does the hostagent stay in-step with the ASE database that it needs to talk with?

Once tracing was turned on, I shut the SAP ASE instance down again and then used SAP LaMa to initiate the SAP system start once again.
Sure enough, the LaMa start task failed again.

Looking in the trace file dev_sapdbctrl I could see the same error message that I was seeing in SAP LaMa:

Error: Command execution failed. : ASE process no longer found after startup. 
(fault code: 127) Operation ID: 000D3A3862631EEAAEDDA232BE624001
----- Log messages ---- 
Info: saphostcontrol: Executing StartDatabase 
Error: sapdbctrl: ASE process no longer found after startup. 
Error: saphostcontrol: StartDatabase failed (sapdbctrl exit code = 1)

This was great. It confirmed that SAP LaMa was just seeing the symptom of some other issue, since LaMa just calls the hostagent to do the start.

Now I knew the hostagent was seeing the error, I tried using the hostagent directly to perform the start, using the following:

/usr/sap/hostctrl/exe/saphostctrl -debug -function StartDatabase -dbname <SID> -dbtype syb -dbhost <the-ASE-host>

NOTE: The hostagent “-debug” command line option puts out the same information without the need for the hostagent tracing to be turned on in the host_profile.

Once again, the start process failed and the same error message was present in the dev_sapdbctrl trace file.

This was now really strange.
I decided that the next course of action was to start the process of raising the issue with SAP via an incident.
If you suspect that something could take a while to fix, then it’s always best to raise it with SAP early and continue to look at the issue in parallel.

Continuing the Diagnosis

While the SAP incident was in progress, I continued the process of trying to self-diagnose the issue further.
I tried a couple more things such as:

  • Starting and stopping SAP ASE manually using stopdb/startdb commands.
  • Restarting the whole server (this step has a place in every troubleshooting process, eventually).
  • Checking the server patch level.
  • Checking the environment of the Linux user, the shell, the profile files, the O/S limits applied.
  • Checking what happens if McAfee anti-virus was disabled (I’ve seen the ePO blocking processes before).

Eventually exhaustion set in and I decided to give the SAP support processor time to get back to me with some hints.

Some Sleep

I spend a lot of time solving SAP problems. A lot of time.
Something doesn’t work according to the docs, something did work but has stopped working, something has never worked well…
It builds up in your mind and you carry this stuff around in your head.
Subconsciously you think about these problems.

Then, at about 3am when you can’t get back to sleep, you have a revelation.
The hostagent is forking the process to start the database as the syb<sid> Linux user (it uses “su”), from the root user (hostagent runs as the root user).

Linux Domain Users

The revelation I had regarding the forking of the user, was just the trigger I needed to make me consider the way the Linux authentication was setup on this specific server with the problem ASE startup.

I remembered at the beginning of the project that I had hit an issue with the SSSD Linux daemon, which is responsible for interfacing between Linux and Microsoft Active Directory. At that time, the issue was causing the hostagent to hang when operations were executed which required a switch to another Linux user.
This previous issue was actually a hostagent issue that was fixed in a later hostagent patch. During that particular investigation, I requested that the Linux team re-configure the SSSD daemon to be more efficient with its Active Directory traversals, when it was looking to see if the desired user account was local to Linux or if it was a domain account.

With this previous issue in mind, I checked the SSSD configuration on the problem server. This is a simple conf file in /etc/sssd.

The Solution

After all the troubleshooting, the raising of the incident, the sleeping, I had finally got to the solution.

After checking the SSSD daemon configuration file /etc/sssd/sssd.conf, I could clearly see that there was one entry missing compared to the other servers that didn’t experience the SAP ASE start error.

The parameter: “subdomain_enumerate = none” was missing.
Looking at the manual page for SSSD it would seem that without this parameter there is additional overhead during any Active Directory traversal.

I set the parameter accordingly in the /etc/sssd/sssd.conf file and restarted the SSSD daemon using:

service sssd restart

Then I retried the start of the database using the hostagent command shown previously.
It worked!
I then retried with SAP LaMa and that also now started ASE without error messages.

Root Cause

What it seems was happening, was some sort of internal pre-set timeout in the sapdbctrl binary, when hit, the sapdbctrl just abandons and throws the error that I was seeing. This leaves the ASE database to continue and start (the process was initiated), but in the hostagent it looked like it had failed to start.
By adding the “subdomain_enumerate = none” parameter, any “delay”, caused by inappropriate call to Active Directory was massively reduced and subsequent start activities were successful.

SAP PO 7.31 SPS14 JCO NoSuchMethodError

Scenario: We had an outbound interface from SAP PO which was an iDoc being sent to a SAP ABAP stack via HTTP, in the message monitoring log we saw:

javax.ejb.EJBException: nested exception is:
java.lang.RuntimeException:
java.lang.NoSuchMethodError:
com.sap.conn.jco.rt.ClientConnection.execute(Lcom/sap/conn/jco/JCoFunction;Ljava/lang/String;Ljava/lang/String;Lcom/sap/conn/jco/JCoRepository;)V

The message is retried and eventually set to NDLG.

The PO system was a fresh install of PO 7.31, then patched to base SPS13, then we had applied the base 7.31 SPS14 along with the available patches at the time.
During the error, we had the following component levels:

Name    Vendor Location               Version
AJAX-RUNTIME                sap.com               SAP AG 1000.7.31.14.1.20141205212300
BASETABLES       sap.com               SAP AG 1000.7.31.14.0.20141003215100
BI-BASE-B            sap.com               SAP AG 1000.7.31.14.0.20141004141500
BI-BASE-E            sap.com               SAP AG 1000.7.31.14.0.20141004141500
BI-BASE-S            sap.com               SAP AG 1000.7.31.14.0.20141004141500
BI-WDALV           sap.com               SAP AG 1000.7.31.14.0.20141003224600
BI-WDEXT            sap.com               SAP AG 1000.7.31.14.1.20141203045500
BI_UDI  sap.com               SAP AG 1000.7.31.14.0.20141004131400
BPEM-ACC          sap.com               SAP AG 1000.7.31.14.0.20141004155400
BPEM-BASE        sap.com               SAP AG 1000.7.31.14.0.20141004155400
BPEM-BASIS       sap.com               SAP AG 1000.7.31.14.0.20141004155400
BPEM-BUILDT    sap.com               SAP AG 1000.7.31.14.0.20141115023900
BPEM-COLLAB   sap.com               SAP AG 1000.7.31.14.0.20141004155400
BPEM-CONTENT               sap.com               SAP AG 1000.7.31.14.0.20141004155400
BPEM-CORE       sap.com               SAP AG 1000.7.31.14.1.20141203045700
BPEM-CUUI        sap.com               SAP AG 1000.7.31.14.0.20141004155400
BPEM-FACADE  sap.com               SAP AG 1000.7.31.14.1.20141203045700
BPEM-HIM          sap.com               SAP AG 1000.7.31.14.1.20141203045700
BPEM-MM          sap.com               SAP AG 1000.7.31.14.1.20141203012000
BPEM-MON       sap.com               SAP AG 1000.7.31.14.0.20141004155400
BPEM-PP             sap.com               SAP AG 1000.7.31.14.1.20141129055000
BPEM-WDUI      sap.com               SAP AG 1000.7.31.14.1.20141129055000
BRMS-BASE        sap.com               SAP AG 1000.7.31.14.0.20141004130900
BRMS-BUILDT    sap.com               SAP AG 1000.7.31.14.0.20141004130900
BRMS-CORE       sap.com               SAP AG 1000.7.31.14.0.20141004155400
BRMS-FACADE  sap.com               SAP AG 1000.7.31.14.0.20141004155400
BRMS-MON       sap.com               SAP AG 1000.7.31.14.0.20141004155400
BRMS-WDUI      sap.com               SAP AG 1000.7.31.14.0.20141004155400
CAF        sap.com               SAP AG 1000.7.31.14.0.20141004102100
CAF-MF                sap.com               SAP AG 1000.7.31.14.0.20141004102100
CAF-UI  sap.com               SAP AG 1000.7.31.14.0.20141004160600
CFG_ZA                sap.com               SAP AG 1000.7.31.14.0.20141003215100
CFG_ZA_CE        sap.com               SAP AG 1000.7.31.14.0.20141003215100
COLLAB-ADP      sap.com               SAP AG 1000.7.31.14.0.20141004155400
COMP_BUILDT  sap.com               SAP AG 1000.7.31.14.0.20141004104600
CORE-TOOLS      sap.com               SAP AG 1000.7.31.14.1.20141205212100
CU-BASE-JAVA  sap.com               SAP AG 1000.7.31.14.0.20141004130900
CU-BASE-WD     sap.com               SAP AG 1000.7.31.14.0.20141004155400
CU-BASE-WD-EXT            sap.com               SAP AG 1000.7.31.14.0.20141004155400
CU-WD4VC-ADPT            sap.com               SAP AG 1000.7.31.14.0.20141004155400
DATA-MAPPING              sap.com               SAP AG 1000.7.31.14.0.20141004130900
DI_CLIENTS         sap.com               SAP AG 1000.7.31.14.1.20141202210100
ECM-ADMIN      sap.com               SAP AG 1000.7.31.14.0.20141003223800
ECM-APPS          sap.com               SAP AG 1000.7.31.14.0.20141003223800
ECM-CORE          sap.com               SAP AG 1000.7.31.14.0.20141003223800
ECM-JEE-COMP                sap.com               SAP AG 1000.7.31.14.0.20141003223800
ECM-STORE        sap.com               SAP AG 1000.7.31.14.0.20141003223800
ENGFACADE       sap.com               SAP AG 1000.7.31.14.1.20141209182400
ENGINEAPI         sap.com               SAP AG 1000.7.31.14.1.20141130172400
EP-ADMIN          sap.com               SAP AG 1000.7.31.14.1.20141129031400
EP-BASIS              sap.com               SAP AG 1000.7.31.14.2.20141206030600
EP-BASIS-API     sap.com               SAP AG 1000.7.31.14.1.20141130182700
EP-CONNECTIVITY           sap.com               SAP AG 1000.7.31.14.0.20141004132200
EP-MODELING  sap.com               SAP AG 1000.7.31.14.0.20141004132200
EP-RUNTIME      sap.com               SAP AG 1000.7.31.14.0.20141014084300
EP_BUILDT          sap.com               SAP AG 1000.7.31.14.0.20141004104600
ESCONF_BUILDT              sap.com               SAP AG 1000.7.31.14.0.20141004104600
ESI-UI    sap.com               SAP AG 1000.7.31.14.0.20141004155100
ESMP_BUILDT   sap.com               SAP AG 1000.7.31.14.1.20141126014600
ESP_FRAMEWORK          sap.com               SAP AG 1000.7.31.14.0.20141004125900
ESREG-BASIC     sap.com               SAP AG 1000.7.31.14.0.20141004125900
ESREG-SERVICES              sap.com               SAP AG 1000.7.31.14.0.20141004125900
FP-INFRA             sap.com               SAP AG 1000.7.31.14.0.20141003215100
FRAMEWORK    sap.com               SAP AG 1000.7.31.14.0.20141003215100
FRAMEWORK-EXT           sap.com               SAP AG 1000.7.31.14.1.20141206024600
GWJPO sap.com               SAP AG 1000.7.31.14.0.20141004125900
INTG_VIS            sap.com               SAP AG 1000.7.31.14.0.20141004125900
INTG_VIS_DCJ  sap.com               SAP AG 1000.7.31.14.0.20141004125900
J2EE-APPS           sap.com               SAP AG 1000.7.31.14.1.20141125210900
J2EE-FRMW        sap.com               SAP AG 1000.7.31.14.1.20141202210000
JSPM     sap.com               SAP AG 1000.7.31.14.0.20141008191000
KM-KW_JIKS      sap.com               SAP AG 1000.7.31.14.0.20141004131400
LM-CORE             sap.com               SAP AG 1000.7.31.14.2.20141202210000
LM-CTS sap.com               SAP AG 1000.7.31.14.0.20141003224600
LM-CTS-UI          sap.com               SAP AG 1000.7.31.14.0.20141003215100
LM-MODEL-BASE             sap.com               SAP AG 1000.7.31.14.0.20141003224600
LM-MODEL-NW                sap.com               SAP AG 1000.7.31.14.0.20141003224600
LM-SLD sap.com               SAP AG 1000.7.31.14.2.20141202215500
LM-TOOLS           sap.com               SAP AG 1000.7.31.14.0.20141004160100
LMCFG sap.com               SAP AG 1000.7.31.14.1.20141125220400
LMCTC  sap.com               SAP AG 1000.7.31.14.0.20141015180600
LMNWABASICAPPS        sap.com               SAP AG 1000.7.31.14.1.20141128223100
LMNWABASICCOMP      sap.com               SAP AG 1000.7.31.14.1.20141128223100
LMNWABASICMBEAN   sap.com               SAP AG 1000.7.31.14.1.20141209214200
LMNWACDP       sap.com               SAP AG 1000.7.31.14.1.20141128221700
LMNWATOOLS  sap.com               SAP AG 1000.7.31.14.0.20141004160100
LMNWAUIFRMRK            sap.com               SAP AG 1000.7.31.14.1.20141128221700
MESSAGING      sap.com               SAP AG 1000.7.31.14.1.20141206023400
MMR_SERVER   sap.com               SAP AG 1000.7.31.14.0.20141004131400
MOIN_BUILDT  sap.com               SAP AG 1000.7.31.14.0.20141003215000
NWTEC sap.com               SAP AG 1000.7.31.14.0.20141003224600
ODATA-CXF-EXT               sap.com               SAP AG 1000.7.31.14.0.20141003224600
PI-SCP-BUILDT  sap.com               SAP AG 1000.7.31.14.1.20141203013100
PI-SCP-EXT          sap.com               SAP AG 1000.7.31.14.0.20141004125900
PIB2BAS2             sap.com               SAP AG 1000.1.0.4.0.20140929051300
PIB2BOFTP          sap.com               SAP AG 1000.1.0.4.0.20140929051300
PIB2BPGP            sap.com               SAP AG 1000.1.0.4.0.20140929051300
PIB2BSFTP           sap.com               SAP AG 1000.1.0.4.6.20141210073300
PIB2BTOOLKIT   sap.com               SAP AG 1000.1.0.4.1.20141001051200
PIB2BX400           sap.com               SAP AG 1000.1.0.4.0.20140929051300
SAP-XI3RDPARTY             sap.com               SAP AG 1000.7.31.1.0.20110918004000
SAP_BUILDT       sap.com               SAP AG 1000.7.31.14.0.20141003215000
SAP_XIADMIN  sap.com               SAP AG 1000.7.31.14.0.20141004155100
SAP_XIAF            sap.com               SAP AG 1000.7.31.14.2.20141210010200
SAP_XIESR          sap.com               SAP AG 1000.7.31.14.0.20141004125900
SAP_XIGUILIB   sap.com               SAP AG 1000.7.31.14.1.20141201183500
SAP_XITOOL      sap.com               SAP AG 1000.7.31.14.1.20141203013100
SEA-CORE            sap.com               SAP AG 1000.7.31.14.1.20141126010300
SEA-FACADE      sap.com               SAP AG 1000.7.31.14.0.20141004114500
SEA-UI  sap.com               SAP AG 1000.7.31.14.0.20141004144200
SECURITY-EXT    sap.com               SAP AG 1000.7.31.14.0.20141003215100
SERVERCORE      sap.com               SAP AG 1000.7.31.14.1.20141209203300
SERVICE-COMP sap.com               SAP AG 1000.7.31.14.0.20141004130900
SOAMON            sap.com               SAP AG 1000.7.31.14.1.20141126050100
SOAMONBASIC                sap.com               SAP AG 1000.7.31.14.2.20141203045500
SR-UI     sap.com               SAP AG 1000.7.31.14.0.20141004155100
SUPPORTTOOLS               sap.com               SAP AG 1000.7.31.14.0.20141003215100
SWLIFECYCL        sap.com               SAP AG 1000.7.31.14.0.20141004160100
THL-CORE            sap.com               SAP AG 1000.7.31.14.0.20141004155400
TM-WLUI             sap.com               SAP AG 1000.7.31.14.1.20141203045700
UDDI     sap.com               SAP AG 1000.7.31.14.0.20141004125900
UISAPUI5_JAVA               sap.com               SAP AG 1000.7.31.14.1.20141202210100
UKMS_JAVA      sap.com               SAP AG 1000.7.31.14.0.20141004155100
UMEADMIN       sap.com               SAP AG 1000.7.31.14.1.20141202215500
UWLJWF              sap.com               SAP AG 1000.7.31.14.0.20141004141500
VCBASE                sap.com               SAP AG 1000.7.31.14.0.20141004124600
VCCORERT          sap.com               SAP AG 1000.7.31.14.0.20141004124600
VCFRAMEWORK               sap.com               SAP AG 1000.7.31.14.0.20141004124600
VCFREESTYLEKIT               sap.com               SAP AG 1000.7.31.14.0.20141004124600
VCKITBI                sap.com               SAP AG 1000.7.31.14.1.20141203045500
VTP_BUILDT       sap.com               SAP AG 1000.7.31.14.0.20141004104600
WD-ADOBE         sap.com               SAP AG 1000.7.31.14.0.20141003222100
WD-APPS            sap.com               SAP AG 1000.7.31.14.0.20141004101100
WD-FLEX              sap.com               SAP AG 1000.7.31.14.0.20141003222100
WD-RUNTIME   sap.com               SAP AG 1000.7.31.14.2.20141205215300
WD-RUNTIME-EXT          sap.com               SAP AG 1000.7.31.14.0.20141004101100
WDEXTENSIONS               sap.com               SAP AG 1000.7.31.14.1.20141203040100
WSRM  sap.com               SAP AG 1000.7.31.14.0.20141004125900
XI_CNT_SAP_BASIS        sap.com               SAP AG 1000.7.31.14.0.20141004125900

We then downloaded the following patches and applied with SUM:

Name    Version                Current                PatchedLevel
AJAX-RUNTIME                1000.7.31.14.1.20141205212300 1              2
CORE-TOOLS      1000.7.31.14.1.20141205212100 1              2
ENGINEAPI         1000.7.31.14.1.20141130172400 1              2
EP-BASIS              1000.7.31.14.2.20141206030600 2              4
FRAMEWORK    1000.7.31.14.0.20141003215100 0              1
FRAMEWORK-EXT           1000.7.31.14.1.20141206024600 1              2
GWJPO 1000.7.31.14.0.20141004125900 0              1
J2EE-APPS           1000.7.31.14.1.20141125210900 1              4
J2EE-FRMW        1000.7.31.14.1.20141202210000 1              3
LM-CORE             1000.7.31.14.2.20141202210000 2              3
LM-CTS 1000.7.31.14.0.20141003224600 0              1
MESSAGING      1000.7.31.14.3.20150105182200
ODATA-CXF-EXT               1000.7.31.14.0.20141003224600 0              1
SAP_XIAF            1000.7.31.14.4.20150105182200
SAP_XIESR          1000.7.31.14.2.20150107011100
SAP_XIGUILIB   1000.7.31.14.1.20141201183500 1              2
SAP_XITOOL      1000.7.31.14.1.20141203013100 1              2
SERVERCORE      1000.7.31.14.1.20141209203300 1              5
SOAMON            1000.7.31.14.1.20141126050100 1              3
SUPPORTTOOLS               1000.7.31.14.0.20141003215100 0              1
WD-RUNTIME   1000.7.31.14.2.20141205215300 2              4

This fixed the issue.

Starting and Stopping Individual SAP AS Java Server Processes

Should you need to restart either manually or automatically (via script) a specific AS Java server node, it’s possible to use the sapcontrol tool to provide precise control.

The method detailed below is based on the documentation provided here:
Performs a given control function (EnableProcess/StartProcess, DisableProcess/StopProcess,
SoftStopProcess, ActivateProcess, DeactivateProcess, RestartProcess, SoftRestartProcess,
DumpStackTrace, EnableDebugging, DisableDebugging, IncrementTrace, DecrementTrace) on a given AS
Java process. processname must match a process name in the AS Java process list returned by
J2EEGetProcessList. To perform AS Java instance-wide operations (StartInstance, StopInstance,
RestartInstance, BootInstance, RebootInstance), use processname “all”.
Based on the above information, we can query the AS Java processes for a specific AS Java instance, using the following command line as the sidadm user:

sidadm> sapcontrol -host [host] -nr [sys##] -function J2EEGetProcessList
You should substitute the “host” and “sys##” with the host of your application server (virtual host) and the SAP instance number.
The command will return a list of the processes that the instance is running.
For a 2 node AS Java system, you can see the Server0 and Server1 processes in the list.
To restart Server0 on the AS Java instance use the below command line as the sidadm user:

sidadm> sapcontrol -host [host] -nr [sys##] -function J2EEControlProcess “server0” “RestartProcess”
You should substitute the “host” and “sys##” with the host of your application server (virtual host) and the SAP instance number.
Change “server0” for “server1” if you want to restart server1.
Based on the above restart, you can monitor the restart by using the below command line to see when the restart is completed:
sidadm> sapcontrol -host [host] -nr [sys##] -function J2EEGetProcessList | awk -F, ‘{ if ( $2 != “” && FNR > 5 ){ print “Name: “$2″ttState: “$8″ttStarted: “$9 }}’
You should substitute the “host” and “sys##” with the host of your application server (virtual host) and the SAP instance number.