This blog contains experience gained over the years of implementing (and de-implementing) large scale IT applications/software.

HANA Memory Allocation Limit Change With hdbcons for HANA

The SAP HANA command line utility hdbcons can be used to administer the HANA system directly from the Linux operating system command line.

It’s a very powerful utility and I’m sure as time goes by, it will provide invaluable information and functionality for HANA DB administrators to help fine-tune the database.

Log in as the <sid>adm Linux user and run the hdbcons command.
It will automatically connect the indexserver for that instance (if it’s running).
You can then list the available server commands:

hana01:/usr/sap/H10/HDB10> hdbcons
SAP HANA DB Management Client Console (type ‘?’ to get help for client commands)
Try to open connection to server process ‘hdbindexserver’ on system ‘H10′, instance ’10’
SAP HANA DB Management Server Console (type ‘help’ to get help for server commands)
Executable: hdbindexserver (PID: 4092)
[OK]

>
> help

Available commands:
ae_checksize – Check and Recalculate size of columns within the Column Store
authentication – Authentication management.
bye – Exit console client
cd – ContainerDirectory management
checktopic – CheckTopic management
cnd – ContainerNameDirectory management
context – Execution context management (i.e., threads)
converter – Converter management
crash – Crash management
crypto – Cryptography management (SSL/SAML/X509).
deadlockdetector – Deadlock detector.
debug – Debug management
distribute – Handling distributed systems
dvol – DataVolume management
ELF – ELF symbol resolution management
encryption – Persistence encryption management
event – Event management
exit – Exit console client
flightrecorder – Flight Recorder
help – Display help for a command or command list
log – Show information about logger and manipulate logger
mm – Memory management
monitor – Monitor view command
mproxy – Malloc proxy management
mutex – Mutex management
output – Command for managing output from the hdbcons
pageaccess – PageAccess management
page – Page management
pcm – Performance Counter Monitor management
profiler – Profiler
quit – Exit console client
replication – Monitor data and log replication
resman – ResourceManager management
runtimedump – Generate a runtime dump.
savepoint – Savepoint management
snapshot – Snapshot management
statisticsservercontroller – StatisticsServer internals
statreg – Statistics registry command
stat – Statistics management
tablepreload – Manage and monitor table preload
tracetopic – TraceTopic management
trace – Trace management
version – Version management
vf – VirtualFile management

For each of the above commands, additional help can be obtained by re-issuing “help” plus the command name e.g.:

> help mm

Synopsis: mm <subcommand> [options]: Memory management Available subcommands:
   – list|l [-spf] [-l <level>] [<allocator_name>]: List allocators recursively
      -C: work on composite allocators
      -s: Print some allocator statistics (use stat command for full stats)
      -p: Print also peak usage statistics
      -f: Print allocator flags
      -S: Sort output descending by total size
      -l <level>: Print at most <level> levels
   – flag|f [-C] <allocator_name> [-rsdt] <flag_set>: Set allocator flags
      -C: work on composite allocators
      -r: Set flags recursively
      -s: Set given flag(s), default
      -d: Delete given flag(s)
      -t: Toggle given flag(s)
      -a: Also apply changes to associated composite allocators (not allowed in context with ‘-C’)
   – info|i [-f] <address>: Show block information for a given address
      -f: Force block info, even if block not known
   – blocklist|bl [-rtsS] <allocator_name> [-l <count>]: List of allocated blocks in an allocator
      -C: work on composite allocators
      -r: Show blocks in sub-allocators recursively
      -s: Show also allocation stack traces, if known. Cannot be combined with optiont ‘-t’.
      -S: Show only blocks with known allocation stack traces. Cannot be combined with optiont ‘-t’.
      -t: Show top allocation locations sorted by used size (descending).
            Cannot be combined with options ‘-s’ or ‘-S’.
            The default number of printed locations is 10, this can be changed by ‘-l’ option.
      -l <count>: Limit to <count> locations. Valid only if combined with option ‘-t’. Unlimited in cas <count> = 0.
   – maplist|ml: List all mappings used by the allocator
   – mapcheck|mc [-sln]: Check all mappings used by the allocator
      -s: Also show all system mappings
      -l: Also show own mappings as in maplist
      -n: Suppress output of known alloc stack traces for unaccounted memory
   – mapexec|me [-u]: List all known executable module mappings
      -u: Update list of known executable modules
   – reserveallocator|reserveallocators: Print information about OOM Reserve Allocators
   – top [-C] [-sctr] [-l <count>] [-o <fn>] [<allocator_name>]: List top users of memory, <allocator_name> is needed except for option ‘-t’
      -C: work on composite allocators
      -s: Sort call stacks by used size (default)
      -c: Sort call stacks by call count
      -t: Throughput, sort nodes by sum of all allocations
      -r: work recursively also for all suballocators
      -l <count>: Only output top <count> stack traces (0=unlimited)
      -o <fn>: Specify output file name
   – callgraph|cg [-sctrk] [-l <count>] [-o <fn>] [<allocator_name>]: Generate call graph, <allocator_name> is needed except for option ‘-t’
      -C: work on composite allocators
      -s: Sort nodes by used size (default)
      -k: Ouput in kcachegrind format
      -c: Sort nodes by call count
      -t: Throughput, sort nodes by sum of all allocations
      -r: Work recursively also for all suballocators
      -d: Do full demangle (also parameters and template arguments)
      -l <count>: Only output top <count> functions (0=unlimited)
      -o <fn>: Specify output file name (for .dot graph)
   – resetusage|ru [-r] [<allocator_name>]: Reset call stack usage counters, <allocator_name> is needed except for option ‘-r’ or ‘-C’
      -C: work on composite allocators
      -r: Work recursively also for all suballocators
   – limit [-s <size>[K|M|G|T]]: Get current allocation limit in bytes
      -s <size>[K|M|G|T]: Set current allocation limit in bytes, KB, MB or GB
   – globallimit [-s <size>[K|M|G|T]]: get current global (if IPMM is active) allocation limit in bytes
      -s <size>[K|M|G|T]: Set current global (if IPMM is active) allocation limit in bytes, KB, MB or GB
   – garbagecollector|garbagecollection|gc [-f]: Return free segments/blocks to
     operating system
      -f: Also return free fragments in big blocks
   – ipmm: Print Inter Process Memory Management information
      -d: Print detailed information.
   – compactioninfo, ci: Print information about last compaction
   – virtual: Print information about virtual but not resident memory (linux only)
   – requested: Print information about requested allocations (reporting no overhead at all), iterates over all instances of ReportRequestedAllocators
   – blockedmemory [-s <size>[K|M|G|T]]: Get current blocked memory.
      -s <size>[K|M|G|T]: Set current blocked memory in bytes, KB, MB or GB and try to reserve this memory. Common options and arguments:
   – <allocator_name>: Name of the allocator to address
   – <flag_set>: Comma-separated list of following flags: ffence (fence front, writes the pattern 0xaaccaacc in front of the allocated block),
     bfence (fence back, writes the pattern 0xaaccaacc behind the allocated block), astrace (stack trace at allocation),
     dstrace (stack trace at deallocation), areset (overwrite at allocate with pattern 0xd00ffeed),
     dreset (overwrite at deallocate with pattern 0xdeadbeef), all, none, default, !emptyok (allow
     non-empty destruction), preventcheck (prevent changing check flags)
     atrace (trace at allocation), dtrace (trace at deallocation),
     malf (malfunction) or their 2-letter shortcuts [OK]

>

As an example, the current global allocation limit can be displayed:

> mm globallimit
Current global allocation limit=15032385536B.
[OK]

We can adjust the global allocation limit online by issuing the additional “-s” parameter:

> mm globallimit -s 16G
Current global (if IPMM active) allocation limit: 17179869184B
[OK]

>

Now re-check:

mm globallimit
Current global allocation limit=17179869184B.
[OK]

>

What’s the current global allocation limit  in the global.ini you might ask?

hana01:/usr/sap/H10/HDB10> grep ‘^global’  /usr/sap/H10/SYS/global/hdb/custom/config/global.ini

global_allocation_limit = 14336

It hasn’t changed.

So we have confirmed that we can affect the configuration of the HANA system in real-time using hdbcons, but we don’t necessarily preserve the configuration.
You can also check in HANA Studio on the landscape page.
Since each service (process) takes it’s memory allocation percentage from the global allocation, this will automatically change in real-time too.
This means that for analysing “what-if” style scenarios or for operational acceptance testing, you can effectively present a set of configuration values and work through them almost automatically.  Invaluable for those test automation guys in the field.

HowTo: Dynamic SQL Server Memory Change from SAP ST04

Scenario: You have a SQL Server database for your SAP system, and you know that right clicking the database server name in SQL Server Management Studio, selecting “Properties” and then “Memory”, will show you the SQL Server memory settings, but you want to know how you can see/change the same detail in SAP…

SQL Server Management Studio - Dynamic Memory

SQL Server Management Studio - Minimum memory

In SAP, you can use transaction ST04 to see the SQL Server database settings.
The memory details are visible in the “Overview” screen.
You will see the “Current Memory MB” equals the amount of memory allocated to SQL Server, and if the “Min server memory” and “Max server memory” settings have been set equal (recommended by SAP), then the overview screen will show “FIXED” for the “SQL Memory Setting”:

SAP SQL Server memory settings

It is possible to modify this setting directly from ST04.
You will need to expand the “Diagnostics” branch and then double click “SQL Command Editor”:

image

On the right hand side, enter the SQL Server commands to resize the memory (notice that these are slightly different to the SQL statements when using them in SSMS):

SAP ST04 SQL code execute

exec sp_configure 'show advanced options', 1
RECONFIGURE
exec sp_configure 'max server memory', 4096
RECONFIGURE


Click Execute:

image

The output window will be displayed:

SAP ST04 SQL code execute

If you noticed, I didn’t change the “Min” memory setting, only the “Max”.
When I check in the “Performance -> Overview” screen in ST04, I can now see that the “Current Memory MB” setting has not changed, but the “SQL Memory Setting” is now showing “RANGE”:

SAP ST04 SQL Server memory settings

Now if I use the SQL Command Editor to also change the “Min” memory, we will see the ST04 overview screen update:

SAP ST04 SQL execute

exec sp_configure ‘show advanced options’, 1
RECONFIGURE
exec sp_configure ‘max server memory’, 4096
exec sp_configure ‘min server memory’, 4096
RECONFIGURE


image

And the overview screen:

image

Well, we’re at “FIXED” again, but the amount of memory has not changed.
Yet in SSMS, I can see the allocation has changed:

SQL Server Management Studio memory settings

This is a weird, because the Microsoft documentation for SQL Server 2008R2 (my version) says that the setting should take effect straight away.
I guess there’s something within the ST04 screen that doesn’t update.
There is another way…
You can use the “Configuration -> Overview” screen and the “Configuration Options” tab to see both the Min and Max memory settings.
As per out change, these correctly reflect the current memory settings:

SAP ST04 SQL Server memory settings

WARNING: You should be aware that during testing, I was able to set the Min memory value higher than the Max memory value in ST04.
I was then unable to change this through ST04, as the store procedures just produced errors and refused to let me change the values.
In the end I had to change the Max value using SSMS.

Java VM 5.0 Default Heap Size

If you have Java 5.0 (1.5), you may wish to know the default heap size for a JVM.
If you don’t specify -Xmx or -Xms to control the heap size, then the defaults are used.

They are described in detail in the strangely title “Ergonomics in the 5.0 Java Virtual Machine” page here: https://www.oracle.com/technetwork/java/ergo5-140223.html

The page states:
In the J2SE platform version 5.0 a class of machine referred to as a server-class machine has been defined as a machine with
2 or more physical processors
2 or more Gbytes of physical memory

On server-class machines by default the following are selected.
Throughput garbage collector
Heap sizes
initial heap size of 1/64 of physical memory up to 1Gbyte
maximum heap size of ¼ of physical memory up to 1Gbyte
Server runtime compiler

So if you have a “server class” machine, you could expect your Java 5.0 JVM to utilise a maximum of 1GB of heap with no tuning (-Xmx & -Xms).

Although it’s not very clear, it seems that a non-“server class” machine would allocate the same as a Java 1.4.2 virtual machine:

In the J2SE platform version 1.4.2 by default the following selections were made
Serial garbage collector
Heap sizes
initial heap size of 4 Mbyte
maximum heap size of 64 Mbyte
Client runtime compiler

Therefore, a maximum of 64MB of heap would be utilised.

I don’t know if a single dual core CPU is recognised as a “server class” machine, but you can find out what your Java version thinks your machine is by running:

> java -help
Usage: java [-options] class [args...]
(to execute a class)
or java [-options] -jar jarfile [args...]
(to execute a jar file)

where options include:
-d32         use a 32-bit data model if available
-d64         use a 64-bit data model if available
-client       to select the "client" VM
-server      to select the "server" VM
-hotspot    is a synonym for the "client" VM [deprecated]
                 The default VM is server,
                 because you are running on a server-class machine
.

-cp <class search path of directo....

As you can see, the output tells you that it thinks you are running on a “server class” machine.

Either way, it’s probably best to use the “-server” command line option to be sure, plus the “-Xmx” option to restrict memory usage if you don’t need a whole 1GB heap.

SAP Short Dump SYSTEM_NO_TASK_STORAGE

A background job has produced a short dump with SYSTEM_NO_TASK_STORAGE.
The “Heap Memory” section in ST02 showed a maximum use of 1.5GB, but we had allocated nearly 1.9GB according to the “abap/heap_area_nondia” and “abap/heap_area_dia” instance profile parameters.

The ST02 short dump analysis shows the source code line where the problem occurred. The line of code doesn’t look specifically interesting, it is, however, requiring a slight increase in memory allocation.
If the program is running in DIALOG then it will be allocated Extended Memory first, followed by HEAP (local process) memory.
If the program is running in BACKGROUND then it will be allocated HEAP memory first, followed by Extended Memory.

Extended Memory is pre-allocated at system startup according to the EM initial setting in the SAP instance profile.  It is then increased up to the maximum specified in the instance profile per user and per application server.
HEAP memory is only allocated as it is needed within each of the SAP dw.* OS processes, up to the maximum specified (in instance profile) per user or per application server.  It is then released back to the OS when the process finishes processing (SAP restarts the work process) and it has used over a specific amount of memory as set in parameter abap/heaplimit.

Since this was a background job, we can assume that we exhausted HEAP memory and should have automatically switched to Extended Memory.
However, the Kernel section of the short dump showed that we may have experienced an issue obtaining more virtual memory from the OS:

*** Error in libunwind: Out of memory. Try with a higher value >
 > for UNWIND_RESERVE_MEM (current value = 16).
 (0) 0x40000000017ae480 [dw.sapXXX_DVEBMGS01]
 (1) 0x40000000017ae2b0 [dw.sapXXX_DVEBMGS01]
 (2) 0x40000000021de880 [dw.sapXXX_DVEBMGS01]
 (3) 0x40000000021e45d0 [dw.sapXXX_DVEBMGS01]
 (4) 0x400000000115e860 [dw.sapXXX_DVEBMGS01]
 (5) 0x400000000120b3a0 [dw.sapXXX_DVEBMGS01]
 (6) 0x40000000010542f0 [dw.sapXXX_DVEBMGS01]
 (7) 0x4000000001111940 [dw.sapXXX_DVEBMGS01]
 (8) 0x40000000011ae790 [dw.sapXXX_DVEBMGS01]
 (9) 0x40000000012fb090 [dw.sapXXX_DVEBMGS01]
 (10) 0x40000000012fd7d0 [dw.sapXXX_DVEBMGS01]
 (11) 0x400000000188cae0 [dw.sapXXX_DVEBMGS01]
 (12) 0x4000000001896e70 [dw.sapXXX_DVEBMGS01]
 (13) 0x4000000001891670 [dw.sapXXX_DVEBMGS01]
 (14) 0x40000000018949a0 [dw.sapXXX_DVEBMGS01]
 (15) 0x400000000187f1e0 [dw.sapXXX_DVEBMGS01]
 (16) 0x40000000014d1ce0 [dw.sapXXX_DVEBMGS01]
 (17) 0x400000000149e8d0 [dw.sapXXX_DVEBMGS01]
 (18) 0x4000000001496fc0 [dw.sapXXX_DVEBMGS01]
 (19) 0x4000000001364c30 [dw.sapXXX_DVEBMGS01]
 (20) 0x4000000000ed4af0 [dw.sapXXX_DVEBMGS01]
 (21) 0x4000000000ed4a90 [dw.sapXXX_DVEBMGS01]
 (22) 0xc000000000045880 main_opd_entry + 0x50 [/usr/lib/hpux64/dld.so]

Is this case, you could either add more OS memory (essentially you have over allocated memory somewhere), or you could resize the abap/heap_area_nondia parameter to a smaller amount, so that it will switch to EM sooner.