Month: September 2017

Wily Introscope EM performance analysis

manoliv / / Articles

If you have worked with Introscope EM, probably you have also faced some sort of a problem related its performance. For example, I had the situation that the EM would randomly stop itself, apparently for no reason. The version on which I had this error was 9.5.
First, I checked the logs and I could find this error in the em.log file:

java.lang.OutOfMemoryError: GC overhead limit exceeded

The problem was solved by increasing the total java heap memory, as a consequence also the GC duration was increased.

How to change the heap size of the EM? The parameters can be changed in the file Introscope_Enterprise_Manager.lax in the section:

#  LAX.NL.JAVA.OPTION.ADDITIONAL

lax.nl.java.option.additional=-Xms4096m -Xmx4096m -Djava.awt.headless=true -XX:MaxPermSize=512m -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi  -Xss512k -Dsun.ja
va2d.noddraw=true

In my case, I increased it from 2 Gb to 4 Gb and everything got solved out after that.

Unfortunately things do not go that smooth all the time, so what happens when EM does not start with a certain amount of configured memory? For example for version 9.7, I did the exact changes and EM would not start anymore after this, generating the following errors:

./EMCtrl.sh stop: Enterprise Manager stopped
Invalid initial heap size: -Xms4096m
The specified size exceeds the maximum representable size.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Or another error:

./EMCtrl.sh status: Enterprise Manager stopped
Invalid maximum heap size: -Xmx4096m
The specified size exceeds the maximum representable size.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Before making any changes to the java heap memory, I recommend that it is tested how much memory can be allocated to your JVM, especially that each EM runs with different version of JVM.
The command that you can test this is :

/path_of_the_jvm_used_by_EM/java -XmsNNNNm -XmxNNNNm -version

where NNNN is the amount the momory that you want to do the test with.

In case the value is accepted/correct it will return you default java environment info, otherwise it will throw in an error. You can test both initial and maximum heap size or just one of them.
Here are a few examples, so you can make an idea:

/usr/jdk/instances/jdk1.7.0_79/jre/bin/java -Xms3072m  -Xmx3072m -version
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) Server VM (build 24.79-b02, mixed mode)

/usr/jdk/instances/jdk1.7.0_79/jre/bin/java -Xms4096m  -Xmx4096m -version
Invalid initial heap size: -Xms4096m
The specified size exceeds the maximum representable size.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Now that we started to work on configuration files of EM, let’s see what are its most important directories and related files.

Main directories:

EM_HOME directory => by default, the home directory of EM, depending on the version, could be for 8.X /usr/sap/ccms/wilyintroscope and for 9.X and 10.X /usr/sap/ccms/apmintroscope. We can find here the configuration file:

  • Introscope_Enterprise_Manager.lax => the heap memory size can be modified from here, as we saw earlier

logs => contains the logs of EM where you can find quickly information about the agent state:

  • IntroscopeEnterpriseManager.log => you can find here information about the EM and Java VM version, in the format below:
3/10/15 05:55:20.231 PM CET [INFO] [main] [Manager] Introscope Enterprise Manager Release 9.5.6.0 (Build 956044)
3/10/15 05:55:20.232 PM CET [INFO] [main] [Manager] Using Java VM version "SAP Java Server VM 1.6.0_91" from SAP AG
3/10/15 05:55:20.232 PM CET [INFO] [main] [Manager] Using Introscope installation at: /usr/sap/ccms/wilyintroscope/.
3/10/15 05:55:20.232 PM CET [INFO] [main] [Manager] CA Wily Introscope(R) Version 9.5

But also information about any problems with the stop/start is present here.

  • em.log => the most important and readable errors can be found here
  • perflog.txt => more detailed information related to the performance and sizing are permanently written here. After the restart a new file is written and the old information is kept in perflog.txt.previous

bin => directory where you can find the stop/start script

config => various configurations and customizations can be done from here:

  • IntroscopeEnterpriseManager.properties => location of different directories and also many other configurations can be set in the file
  • apm-events-thresholds-config.xml => different thresholds can be set, for example limits the number of agent connections
  • agentclusters.xml => file used for monitoring capabilities

license => license for EM

sap => contains SAP connection related files:

  • SolMan_SID.e2emai.properties => IS Push configuration file used to push Introscope Metrics to Solution Manager Monitoring and Alerting Infrastructure (MAI)
  • TrustedCerts => certificate files used for single sign-on from SolutionManager

 

I started to dig down for any documentation related to EM performance and I was surprised in a good way.
There is a very nice tool from SAP named EMPerflogAnalyzer that will analyze the EM performance based only on the the perflog.txt file.
Here is how the tool looks like:

The checks that are done concern the following topics:

  • Number of Metrics
  • Number of Agents
  • Java Heap Space
  • Harvest Duration
  • Smartstor Duration (collected metrics)

For each point you get a stoplight color depending on whether you have a serious problem (red), just a warning (yellow) or if the configuration is correct (green). Also more details are available for each topic within the tool, guiding you on how to solve the problem.
There is an example on how a result of the tool analysis looked in my case:

The tool can be downloaded from the SAP note 2156320.

You can find other related EM articles on this site at:
http://sapnotebook.com/article-introscope/

More information about the tool and how to use it you can find in the official documentation provided below.

Helpful documentation:

1976769 – Introscope Enterprise Manager rejects new agents after reaching 400 connected agents
1630229 – CA Introscope Enterprise Manager has JAVA heap problems
2156320 – EM perflog analyzer tool – Introscope Enterprise Manager
1791543 – java.lang.OutOfMemoryError: PermGen space

Introscope Troubleshooting: https://wiki.scn.sap.com/wiki/display/TechOps/Introscope+Troubleshooting
Introscope Enterprise Manager.lax parameters: https://docops.ca.com/ca-apm/9-6/en/administrating/properties-files-reference/introscope-enterprise-manager-lax#IntroscopeEnterpriseManager.lax-lax.nl.java.option.additional
EM Perflog Analyzer Tool: https://wiki.scn.sap.com/wiki/display/TechOps/EM+Perflog+Analyzer+Tool
Performance health check: https://support.ca.com/us/knowledge-base-articles.tec604648.html

issues in sapxpg

manoliv / / News

Today SAP announced that there are some problems with the sapxpg binary due to incompatibility in kernel 749.
This affects both SAP_BASIS releases 740 SP >=18 and 750 SP >=8.
SAPXPG is a kernel binary that is used for controlling external programs or the external commands within the same system or in a remote system. The trace for SAPXPG is dev_cp.

There could be two symptoms as per SAP note 2443381:

“1. You use the Unicode sapxpg and the asynchronous execution of external commands no longer works.”

=> in this case for:
– kernel 745, sapxpg_uc has to be patched to PL >=416
– kernel 749, sapxpg has to be patched to PL >=210

“2. You have SAP_BASIS 750 or 740 and SAP Kernel 749, and there are problems with the execution of external commands that have non-ASCII charactes in the parameter string or in their output. These characters are not transferred correctly.”

=> in this case for both 745 and 749 kernels the note 2443381 must be implemented.

But as there are also some other minor problems detected with SAPXPG, it would be best if you update directly to sapxpg_uc PL 510 for kernel 745 and sapxpg PL 310 for kernel 749.

In order to find out the version of sapxpg you can execute:

./sapxpg -VERSION

I recommend that you subscribe to the SAP news page in order to be up to date to the latest kernel related announcements: https://wiki.scn.sap.com/wiki/display/SI/SAP+Kernel%3A+Important+News

More information can be found in notes:

2443381 – Problems during execution of external commands as of SAP_BASIS 740
2479738 – Connection test for sapxpg or sapxpg_uc fails

Introduction to LOBs

manoliv / / Oracle for SAP

What exactly is a LOB? LOB is the acronym for Large OBject. A table can have a column of the type LOB, this column holds unstructured data like documents, images, spreadsheets, xml files … When you create a LOB column for a table, you are actually creating a separate LOB segment. And when creating an LOB segment, also a LOB index is created together with it, that helps to find the data of the LOB, which is stored in different places in a tablespace.

This is how the LOB segments look like:

SYS_LOBXXXXXXXXXXCXXXXX$$, where XXXX is a hexadecimal number

And the index name of the LOB segement:

SYS_ILXXXXXXXXXXXCXXXXX$$, where XXXX is a hexadecimal number.

The hexadecimal numbers for both the LOB segment and the LOB index are the same.

There are several types of LOBs that can be found in the Oracle database:

BLOB: Binary LOBs
CLOB: Character LOBs
NCLOB: National Character LOBs
BFILE: Binary File

But in ABAP dictionary the datatype represented by LOBs are:

RATRING BLOB
STRING CLOB

LOBs can be stored as basicfiles or securefiles. Basicfiles are the old way of storing these segments, the recommended way of storing LOB segments is now securefile. The main reasons for this would be performance, additional features and security. So it is really recommended in case you still have LOB segments stored as basicfiles, to convert them into securefiles. This can be done easily, as a reorganization option via brtools.
It is possible to check the LOB segments that are not stored as secure LOBs via the command:

SELECT DISTINCT TC.TABLE_NAME, 
TE.SECUREFILE
FROM
DBA_TAB_COLUMNS TC, DBA_TABLES T, DBA_LOBS TE
WHERE TC.TABLE_NAME = T.TABLE_NAME and TC.TABLE_NAME = TE.TABLE_NAME and TE.TABLE_NAME = T.TABLE_NAME
and TC.OWNER LIKE 'SAP%' and 
(TC.DATA_TYPE = 'BLOB' OR TC.DATA_TYPE = 'CLOB') and SECUREFILE = 'NO';

These segments can be reorganized as secure LOBs with brtools via this path:

BRTOOLS => Segment management -> Reorganize tables -> choose reorganization option “lob2lob”

In Oracle 11 and 12 there is also a parameter name db_securefile that specifies how should a LOB segment be created by default.
These are the possible values according to Oracle documentation:

” NEVER

Any LOBs that are specified as SecureFiles are created as BasicFiles LOBs. All SecureFiles-specific storage options and features (for example, compress, encrypt, deduplicate) will cause an exception. The BasicFiles LOB defaults will be used for storage options not specified.

PERMITTED

LOBs are allowed to be created as SecureFiles.

PREFERRED

All LOBs are created as SecureFiles unless BASICFILE is explicitly specified in the LOB storage clause or the tablespace is a Manual Segment Space Management tablespace. When PREFERRED is set, cases where BASICFILE would otherwise be inherited from the partition or column level LOB storage are ignored; the LOBs will be created as SecureFiles instead.

ALWAYS

Attempts to create all LOBs as SecureFiles LOBs but creates any LOBs not in an Automatic Segment Space Managed (ASSM) tablespace as BasicFiles LOBs, unless SECUREFILE is explicitly specified. Any BasicFiles LOB storage options that are specified will be ignored and the SecureFiles LOB defaults will be used for all storage options not specified.

IGNORE

The SECUREFILE keyword and all SecureFiles options are ignored “

But SAP mentions in the note 1426979 for Oracle 11g only these options: permitted, never, always and ignore and clearly states that for SAP no other value is allowed, except “permitted”, that is also the default value. This means that a new created LOB, that has no specific options added under Oracle 11g, will be created by default as basicfile and this can be converted to securefile anytime.
But what about Oracle 12c? The problem is that SAP has not updated this information for 12c, but from what I know, is that the new default for 12c is the value PREFFERED which means that LOBs are now created as securefiles directly, if there is nothing else specifically mentioned in the creation clause and it is also the only accepted value by SAP for Oracle 12c.

From my experience I had a lot of situations when LOB segments increased very large and had to be reorganized. I recommend that these are regularly checked and reorganized when feasible. Here are some commands that will for sure help during the process. And remember, do not panic if after the reorganization you will not find that LOB segment in the database, this is recreated during the reorganization with another name.

How to check out the LOBs from a table and the mapped columns:

select SEGMENT_NAME,owner,table_name,column_name from dba_lobs where TABLE_NAME='table_name';

Vice versa you can also check what table belongs to a lob segment:

select owner, table_name, column_name FROM dba_lobs WHERE segment_name = 'lob_segment';

How to check the size of the LOB:

select sum(bytes)/1024/1024 from dba_extents where segment_name='XXXXXXX';

How to get an approximation of how much really does a LOB segment occupy and check if it fragmented:

SELECT 'table_name' TABLE_NAME, 'column_name' COLUMN_NAME, ROUND(SUM(DBMS_LOB.GETLENGTH("column name"))/ 1024 / 1024) "NET (MB)"
FROM "object_owner"."table_name";

Important SAP Notes about this topic:

563359 – Performance optimization for tables with LOB columns
2384534 – LOB conversion and table compression with BRSPACE 7.40 for Oracle 12c
2142497 – What to do for large lobsegment on Oracle database?
500340 – FAQ: LOBS
1426979 – Oracle 11g: SecureFiles – The new way to store LOB data