If you have worked with Introscope EM, probably you have also faced some sort of a problem related its performance. For example, I had the situation that the EM would randomly stop itself, apparently for no reason. The version on which I had this error was 9.5.
First, I checked the logs and I could find this error in the em.log file:
java.lang.OutOfMemoryError: GC overhead limit exceeded
The problem was solved by increasing the total java heap memory, as a consequence also the GC duration was increased.
How to change the heap size of the EM? The parameters can be changed in the file Introscope_Enterprise_Manager.lax in the section:
# LAX.NL.JAVA.OPTION.ADDITIONAL lax.nl.java.option.additional=-Xms4096m -Xmx4096m -Djava.awt.headless=true -XX:MaxPermSize=512m -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -Xss512k -Dsun.ja va2d.noddraw=true
In my case, I increased it from 2 Gb to 4 Gb and everything got solved out after that.
Unfortunately things do not go that smooth all the time, so what happens when EM does not start with a certain amount of configured memory? For example for version 9.7, I did the exact changes and EM would not start anymore after this, generating the following errors:
./EMCtrl.sh stop: Enterprise Manager stopped Invalid initial heap size: -Xms4096m The specified size exceeds the maximum representable size. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.
Or another error:
./EMCtrl.sh status: Enterprise Manager stopped Invalid maximum heap size: -Xmx4096m The specified size exceeds the maximum representable size. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.
Before making any changes to the java heap memory, I recommend that it is tested how much memory can be allocated to your JVM, especially that each EM runs with different version of JVM.
The command that you can test this is :
/path_of_the_jvm_used_by_EM/java -XmsNNNNm -XmxNNNNm -version
where NNNN is the amount the momory that you want to do the test with.
In case the value is accepted/correct it will return you default java environment info, otherwise it will throw in an error. You can test both initial and maximum heap size or just one of them.
Here are a few examples, so you can make an idea:
/usr/jdk/instances/jdk1.7.0_79/jre/bin/java -Xms3072m -Xmx3072m -version java version "1.7.0_79" Java(TM) SE Runtime Environment (build 1.7.0_79-b15) Java HotSpot(TM) Server VM (build 24.79-b02, mixed mode)
/usr/jdk/instances/jdk1.7.0_79/jre/bin/java -Xms4096m -Xmx4096m -version Invalid initial heap size: -Xms4096m The specified size exceeds the maximum representable size. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.
Now that we started to work on configuration files of EM, let’s see what are its most important directories and related files.
Main directories:
EM_HOME directory => by default, the home directory of EM, depending on the version, could be for 8.X /usr/sap/ccms/wilyintroscope and for 9.X and 10.X /usr/sap/ccms/apmintroscope. We can find here the configuration file:
- Introscope_Enterprise_Manager.lax => the heap memory size can be modified from here, as we saw earlier
logs => contains the logs of EM where you can find quickly information about the agent state:
- IntroscopeEnterpriseManager.log => you can find here information about the EM and Java VM version, in the format below:
3/10/15 05:55:20.231 PM CET [INFO] [main] [Manager] Introscope Enterprise Manager Release 9.5.6.0 (Build 956044) 3/10/15 05:55:20.232 PM CET [INFO] [main] [Manager] Using Java VM version "SAP Java Server VM 1.6.0_91" from SAP AG 3/10/15 05:55:20.232 PM CET [INFO] [main] [Manager] Using Introscope installation at: /usr/sap/ccms/wilyintroscope/. 3/10/15 05:55:20.232 PM CET [INFO] [main] [Manager] CA Wily Introscope(R) Version 9.5
But also information about any problems with the stop/start is present here.
- em.log => the most important and readable errors can be found here
- perflog.txt => more detailed information related to the performance and sizing are permanently written here. After the restart a new file is written and the old information is kept in perflog.txt.previous
bin => directory where you can find the stop/start script
config => various configurations and customizations can be done from here:
- IntroscopeEnterpriseManager.properties => location of different directories and also many other configurations can be set in the file
- apm-events-thresholds-config.xml => different thresholds can be set, for example limits the number of agent connections
- agentclusters.xml => file used for monitoring capabilities
license => license for EM
sap => contains SAP connection related files:
- SolMan_SID.e2emai.properties => IS Push configuration file used to push Introscope Metrics to Solution Manager Monitoring and Alerting Infrastructure (MAI)
- TrustedCerts => certificate files used for single sign-on from SolutionManager
I started to dig down for any documentation related to EM performance and I was surprised in a good way.
There is a very nice tool from SAP named EMPerflogAnalyzer that will analyze the EM performance based only on the the perflog.txt file.
Here is how the tool looks like:
The checks that are done concern the following topics:
- Number of Metrics
- Number of Agents
- Java Heap Space
- Harvest Duration
- Smartstor Duration (collected metrics)
For each point you get a stoplight color depending on whether you have a serious problem (red), just a warning (yellow) or if the configuration is correct (green). Also more details are available for each topic within the tool, guiding you on how to solve the problem.
There is an example on how a result of the tool analysis looked in my case:
The tool can be downloaded from the SAP note 2156320.
You can find other related EM articles on this site at:
http://sapnotebook.com/article-introscope/
More information about the tool and how to use it you can find in the official documentation provided below.
Helpful documentation:
1976769 – Introscope Enterprise Manager rejects new agents after reaching 400 connected agents
1630229 – CA Introscope Enterprise Manager has JAVA heap problems
2156320 – EM perflog analyzer tool – Introscope Enterprise Manager
1791543 – java.lang.OutOfMemoryError: PermGen space
Introscope Troubleshooting: https://wiki.scn.sap.com/wiki/display/TechOps/Introscope+Troubleshooting
Introscope Enterprise Manager.lax parameters: https://docops.ca.com/ca-apm/9-6/en/administrating/properties-files-reference/introscope-enterprise-manager-lax#IntroscopeEnterpriseManager.lax-lax.nl.java.option.additional
EM Perflog Analyzer Tool: https://wiki.scn.sap.com/wiki/display/TechOps/EM+Perflog+Analyzer+Tool
Performance health check: https://support.ca.com/us/knowledge-base-articles.tec604648.html