Blog posts

Memory Analyzer is a tool developed in Eclipse that is used to find the cause for the Heap Dump generation in the SAP Java systems. There are several sources from SAP that I will list at the end of the article, but unfortunately this information if a little bit outdated, that is why I would like to clarify here what are the steps that need to be followed in order to install this tool and how can the dump can be analyzed with it.

These are the steps for the installation:

1. The download location is: http://www.eclipse.org/mat/downloads.php
If you have a windows environment you have to download the Windows (x86) kit to your local computer => MemoryAnalyzer-1.7.0.20170613-win32.win32.x86.zip

2. Download the extension pack from the note 1883568 – How to self analyze a Heap Dump using MAT

MemoryAnalyzer-extensions-update-site.zip

3. Unzip MemoryAnalyzer-1.7.0.20170613-win32.win32.x86.zip to your computer and run the MemoryAnalyzer.exe from the mat directory:

4. Install the extension downloaded earlier as follows:

Add the path to extension archive:

And the installation is completed, only a restart of the computer is necessary. That’s it, now the tool is ready to use.

Now, before opening the dump file you have to make sure that you have enough RAM memory available on the machine that you run the heap dump analysis, that is more than the size of the dump. It is possible that for larger dumps, you get this error when opening the file for analysis:

In order to solve this you have to change -Xmx parameter to a higher value from the tool file MemoryAnalyzer.ini located in the mat directory. The value has to be set to at least the size of the dump file:

Now just open the dump file:

The most used option is Leak Suspects Report, this analysis your dump, searches for the memory leak and lists what are the applications/libraries that are causing the problem, split in percentages.

Under each Problem Suspect you should find some keywords, that you can search for on SMP, SCN or if the suspect is a custom code or a third party application you can get in contact with the respective responsible to get to the root cause.
For more information about heap/thread dumps you can find in my post: SAP java heap and thread dumps

Useful related SAP notes and cocumenation:

2063943 – Memory Analyzer Tool stops the heap dump parsing due to insufficient memory

1883568 – How to self analyze a Heap Dump using MAT

https://wiki.scn.sap.com/wiki/pages/viewpage.action?pageId=237307510

 

Definition of thread and heap dump

A thread dump is a snapshot of how the Java threads of the running server processes look at a point in time, whereas a heap dump is a snapshot of the memory of a java process.

Parameters

Automatically, by default, if there is an out of memory situation, the system itself will generate a heap dump. This behavior is assured by the java parameter XX:+HeapDumpOnOutOfMemoryError that you can find in configtool setup:

In case this parameter is deleted, the automatic generation of the dump will not happen. The file will have quite a big size, approximately the size of the java heap size.

Now, there is also a similar parameter -XX:+HeapDumpOnCtrlBreak that it is not set as a default, when this is activated a heap dump will be written together with the thread dump, every time a CTRL_BREAK (win) or SIGQUIT (unix) event is triggered manually or automatically by the system. So if such behavior is not intended, this parameter should be deleted, not to have unnecessary amount of heap dumps generated.

Naming conventions and paths

You can find the dumps in the default location: /usr/sap/SID/instance/j2ee/cluster/serverN. This location is also defined by a parameter in configtool -XX:HeapDumpPath=directory where to save the heap dumps and can be changed, in case for example, you do not have enough space in the default location and you want the dump to be generated somewhere else.
But this parameter path is valid only for dumps that are generated automatically by the parameter -XX:+HeapDumpOnOutOfMemoryError, not also by the parameter HeapDumpOnCtrlBreak.

Heap dumps generated automatically by OutOfmemory errors will have the naming: OOM.hprof and OOM.addons
Heap dumps generated by a user will look like java_pid.hprof and java_pid.addons.
On both situations, these can be found in /usr/sap/SID/instance/j2ee/cluster/serverN.
Standalone thread dumps will have the naming std_serverX.out and can be found in /usr/sap/SID/instance/work/

Generation methods

As of Java 7.3 it is very easy to manually generate a dump, if needed (for example in case of poor performance of the application). Both thread and heap dumps can be generated directly from NWA -> Troubleshooting -> Advanced Troubleshooting -> Thread Dump Analysis or Heap Dump Analysis

Another way to this, but a little bit more complex is from JSMON tool. You can list all the options available like this:

jsmon pf=instance_profile
help => to see all the options available
process sdump idx => to generate a thread dump of a process
or
process hsdump idx => to generate a heap dump of a process

A thread dump can be generated also outside of the system, from Solution Manger. This is how it looks like for Solman 7.1 SP14:

Go to SAP Solution Manager Work Center -> Root Cause Analysis -> System Analysis -> Choose the system for which you want to generated the thread dump -> Expert Links -> Start Embedded -> Thread Dump Analysis -> Trigger Thread Dump for the wanted processes:

Now that we talked about the heap dumps, you can find in my next post how to analyze this memory related dumps and how you can get an idea of what is causing the memory problems: SAP Memory Analyzer – MAT – installation and usage

Useful related SAP documentation:
1004255 – How to create a full HPROF heap dump of J2EE Engine 6.40/7.0
https://wiki.scn.sap.com/wiki/pages/viewpage.action?pageId=237307510
https://wiki.scn.sap.com/wiki/display/ASJAVA/%28JSTTSG%29%28Deploy%29Thread+Dump

In this article I will cover the subject of doing a export/import of Java part of a dual stack system, task that mostly used during a system copy. I would say that is is a topic that is not documented very in detail by SAP, so I hope it will come very handy.
First of all, you have to know that for exporting the Java data, no downtime is required, this is an online task and it is done with the help of SWPM tool.

Preparations

On the target system, make a backup of the below files, as most of them will be overwritten during the import:

1. SAP profiles
2. sidadm environment profiles
3. database profiles $ORACLE_HOME/dbs and if Oracle 12c also /oracle/SID/sapprof
4. listener files from $ORACLE_HOME/network/admin

After all these files are backed up proceed with the following:

1. prepare the location for the source system Java export, no more then 5 Gb should be required
2. Download SWPM tool and extract it
3. Prepare a temporary location for the SWPM logs

export TMP=/usr/sap/SID/temp_sapinst
export TEMP=/usr/sap/SID/temp_sapinst
export TMPDIR=/usr/sap/SID/temp_sapinst
export JAVA_HOME=/usr/sap/SID/SYS/exe/jvm/sunx86_64/sapjvm_4.1.035/sapjvm_4

Export

As root run sapinst on the source system and start the export:

./sapinst SAPINST_USE_HOSTNAME=host_name SAPINST_SLP_MODE=false

Choose the export option:

Give the path to the SAP profiles:

Instances installed will be detected:

Database data is identified:

The database copy was done in this case with database tools, so this option can be selected:

Enter the location where the export will be generated:

If you want to remove the sapinst group that was added to the users:

The summary of the selections made:

Reaching the end of the export:

Import

After the system copy is completed with the help of database tools and also the postprocessing of the ABAP part was done, Java can be imported on the target system.
Prepare the environment parameters and start sapinst with root:

export TMP=/usr/sap/SID/temp_sapinst
export TEMP=/usr/sap/SID/temp_sapinst
export TMPDIR=/usr/sap/SID/temp_sapinst
export JAVA_HOME=/usr/sap/SID/SYS/exe/jvm/sunx86_64/sapjvm_4.1.035/sapjvm_4

Also some archives need to be downloaded from support portal beforehand, kits that will be requested during the import procedure: Kernel, Oracle client, Java component

./sapinst SAPINST_USE_HOSTNAME=host_name SAPINST_SLP_MODE=false

Choose the correct option:

In order to be able to customize some options, select custom:

Download the correct kernel archive for your system and select it here:

Enter the system parameters:

DNS name:

Enter the password for the users that will be created:

Select homogeneous system copy, as the system copy was already done via backup/restore with database tools:

Select database data:

Database version:

If you have Oracle 12c, hopefully installed your software with user oracle, so choose it:

Some more database related parameters:

Enter Oracle passwords:

Listener information:

Enter the path to the Oracle instant client software, that you have to download:

The database is already recovered and running, so choose the appropriate option. OPS$ users can be deleted, as SSFS is implemented:

Key phrase for secure store:

Enter the path to the Java component archive:

You can choose the system numbers for your instances:

Enter message server internal port for SCS:

J2EE_ADMIN password:

SDM password:

No need to install the webdispatcher:

Exit option that does not need to be selected:

DDIC password:

Archives will be unpacked:

If you want to remove the groups sapinst from the users:

Summary of your selections:

Import is now in progress:

Probably that during the import, you will face the following errors that are generated because the database configuration that already exist,s will be overwritten with a default one.
Copy back the old files that you saved over:

Listener files:

Oracle pfile and spfile:

After replacing the files with the old ones, the import should continue with success.