DMIS CONTENT 2011_1_700 is required to upgrade to DMIS CONTENT 2011_1_731 by SAP BASIS 7.31 – SAPK-731BHINSAPBASIS during the queue check

If you are planning to upgrade your Solution Manager to 7.2 release, you have to plan first the update in your Maintenance Planner tool. During this process there is one error that I encountered and that might be popping up for all Solution Mangers that have the add-on TDMS 4.0 installed.
The validation of the software packages in MP will fail with the error:

DMIS CONTENT 2011_1_700 is required to upgrade to DMIS CONTENT 2011_1_731 by SAP BASIS 7.31 – SAPK-731BHINSAPBASIS during the queue check

If you check out for more details provided in the logs you will find the following:


It seems like for SAP TDMS 4.0, the product transition cannot be calculated automatically by MP, so it has to be added as a manual step by you in the MP.
You can find a solution from SAP for this in the note 2367414. But unfortunately, the solution provided by SAP is not 100% clear or complete and an additional step has to be followed.
To be more clear lets review what are the steps that have to be selected in order to get rid of the above error.

First, begin a fresh transaction for the upgrade in MP:

Select Plan a Maintenance:

Select your desired target that is Solution Manager 7.2 and Confirm selection

Then select Install or Maintain an Add-on:

Select SAP Landscape Transformation 2, select SHC for ABA 731 and the desired support package level for this and confirm the selection:

Then what it is very important and it is not mentioned in the note, continue to choose again the Install and Maintain Add-on for selecting also the add-on SAP TDMS 4.0 => TDMS for ABA 7.31 and the desired support package level and confirm this selection as well:

Then continue with the selection as normal, in MP and the validation should go successfully now.
By the way, if you are wandering, the order that you select the two additional add-ons does not matter.
But in case you also have the add-on DMIS EXT 2007_1_700 installed you might get as bonus an additional error for it too:

DMIS EXT 2007_1_700 is required to upgrade to DMIS EXT 2007_1_731 by SAP BASIS 7.31 – SAPK-731BHINSAPBASIS

So in this case, you will also have to select in MP, additionally from what we have talked about at the Install and Maintain an Add-on section also the SAP TDMS EXT BPL 3.0 the TDMS for ABA 7.31 and the wanted SP level. This error is explained more in the note 2417816.

For more details, please consult the offcial SAP documenation:

2417816 – Error: DMIS EXT 2007_1_700 is required to upgrade to DMIS EXT 2007_1_731 by SAP BASIS 7.31 – SAPK-731BHINSAPBASIS during Maintenance Planner Planning activity

2367414 – Error: DMIS 2011_1_700 is required to upgrade to DMIS2011_1_731 by SAP_BASIS 7.31 – SAPK-731BHINSAPBASIS… during Maintenance Planner Planning activity.

https://wiki.scn.sap.com/wiki/display/SL/Maintenance+Planner+-+Plan

Enterprise manager upgrade to 9.7 and ISagent upgrade in Solution Manager 7.1

The EM upgrade is cannot be done by upgrading the agent on top of the old version. The upgrade consists on moving the EM to a backup folder and do a fresh installation of the wanted version and then adjust the configuration to the already existent one.
Here are the steps that should be followed for an upgrade of EM on Solution Manager SPS 12 or higher. In my case my source version was EM 9.1.5 and the target one 9.7 and the environment Solaris Intel.

There are three packages that have to be downloaded:

1. The WILY INTROSCOPE ENTPR MGR 9.7 that can be found in this location:

https://launchpad.support.sap.com/ => Software Downloads => Support packages and patches => By Category => SAP Technology Components => SAP Solution Manager => SAP Solution Manager 7.1 => Entry by component => WILY INTROSCOPE => WILY INTROSCOPE ENTPR MGR 9.7 => Introscope Enterprise Manager 9.7 for SAP SP01

It is very important, that you choose the package for your respective OS type. But in case you do not find your OS from the dropdown list, you have to choose the OS independent version. But there is a catch, the OS independent package does not have a bundled Java VM, like the others do. So in this case you have to, either have the corresponding JAVA version required for your EM already available on the machine, either you download the SAPJVM in order to have the required Java VM while performing the installation.
In such case you can download the SAPJVM from:

https://launchpad.support.sap.com => Software Downloads => Support packages and patches => By Category => SAP NETWEAVER AND COMPLEMENTARY PRODUCTS => SAP NETWEAVER => SAP NETWEAVER 7.4 =>
ENTRY BY COMPONENT => AS JAVA EXTENSIONS => SAPJVM6

2. Wily Introscope enterprise management modules:

https://launchpad.support.sap.com/ => Software Downloads => Support packages and patches => By Category => SAP Technology Components => SAP Solution Manager => SAP Solution Manager 7.1 => Entry by component => WILY INTROSCOPE => WILY INTRO EM MGMT MODULES 9.7 => WILYISMM01P_3-80001009.ZIP (Introscope9.7customizationSP01 forSAP(Management Modules))

3. Third-party external component package files:

http://opensrcd.ca.com/ips/osgi/ => introscope_9.7.1.0/ => download the two files:

osgiPackages.v9.7.1.16.unix.tar
eula.txt

After the files have been uploaded to server, create special directory where to store the files and from where you can start the upgrade/installation from:

mkdir EM_upg_dir
ls
WILYISEM01_0-80001135.zip
WILYISMM01P_3-80001009.zip
eula.txt
osgiPackages.v9.7.1.16.unix.tar

unzip the files:

unzip WILYISEM01_0-80001135.zi
tar -xvf osgiPackages.v9.7.1.16.unix.tar
ls
@LongLink                           ca-eula.txt                                                             osgiPackages.v9.7.1.16.unix.tar     unsigned
WILYISEM01_0-80001135.zip           ca-eula.zh_CN.txt                   installer_console.log               osgiPackagesID.txt                  version.txt
WILYISMM01P_3-80001009.zip          ca-eula.zh_TW.txt                   introscope9.7.1.16otherUnixSAP.jar  server
ca-eula.ja.txt                      eula.txt                            lgpl.txt                            signed
ca-eula.ko.txt                      installer.properties                lib                                 source

Prepare the files for the installation:

1. eula.txt

accept the agreemant by changing: LGPL=accept

2. ca-eula.txt

accept the agreemant by changing: CA-EULA=accept

3. installer.properties => check what the current parameters are for EM and make the changes in the file:

USER_INSTALL_DIR=/usr/sap/ccms/wilyintroscope
externalComponentPackage=osgiPackages.v9.7.1.16.unix.tar
emGuestPassword=
emAdminPassword=
wvPort=8081
wvEmHost=localhost
wvEmPort=6001

If you do not enter anything for the installation directory it will install it by default to the new location /usr/sap/ccms/apmintroscope.

Now that everything is prepared, the actual upgrade can begin:

1. Stop the current EM:

cd /usr/sap/ccms/wilyintroscope/bin
./EMCtrl.sh stop
./EMCtrl.sh stop: Stopping Enterprise Manager...
./EMCtrl.sh stop: Enterprise Manager stopped

2. Move the existing installation to a backup folder

mv /usr/sap/ccms/wilyintroscope /usr/sap/ccms/wilyintroscope_old

3. Install the new version

– the command, if you took the OS dependent package:

cd EM_upg_dir
/java -classpath ./introscope9.7.1.16otherUnixSAP.jar install
Silent installer completed with status of: SUCCESS. For more details, consult install logs at: /usr/sap/ccms/wilyintroscope/install. Please wait for the command prompt to return...

– the command if you have the OS dependent package:

./introscope9.7.1.16linuxAMD64SAP.bin –Djava.awt.headless=true

4. Install the management module

cp WILYISMM01P_3-80001009.zip /usr/sap/ccms/wilyintroscope
cd /usr/sap/ccms/wilyintroscope
unzip WILYISMM01P_3-80001009.zip

5. Where necessary, copy the old files with the configurations over the new EM:

– The Introscope SmartStor that contains the collected metrics. Its location is specified by properties:

introscope.enterprisemanager.smartstor.directory
introscope.enterprisemanager.smartstor.directory.archive

in file /config/IntroscopeEnterpriseManager.properties. If you want to keep the already collected metrics move these folders from the old to the new EM.

# The directory that the EM will use for data storage
introscope.enterprisemanager.smartstor.directory=data
introscope.enterprisemanager.smartstor.directory.archive=data/archive

– The Introscope traces are stored in a folder that is configured with properties:
# The directory that the EM will use for trace storage
introscope.enterprisemanager.transactionevents.storage.dir=traces

If you want to keep the already collected traces move this folders from the old to the new EM.

– Manually transfer any custom management modules (located in /config/modules) from the old to the new location. But do not overwrite newly installed standard SAP modules with older versions.

– Manually transfer any customization made in the old version of EM. These customization can be found in the files:

IntroscopeEnterpriseManager.lax or EMService.conf
/config/IntroscopeEnterpriseManager.properties
/config/em-jetty-config.xml
/config/users.xml
/config/users-sap.xml
/config/domains.xml,
/config/apm-events-thresholds-config.xml
/config/loadbalancing.xml

Do not replace the new configuration files with the old copies since you then may loose any newly introduced properties! Instead, copy and paste any customization from your old files.

– Copy all certificate files used for single sign-on from Solution Manager. You find them in /sap/TrustedCerts.

– For monitoring capabilities of SAP Solution Manager copy file <EM_HOME>/config/agentclusters.xml from the old to the new location.

– Also for monitoring capabilities of SAP Solution Manager copy files /sap/SID.e2emai.properties (see SAP note 1751225).

– If you have a full Introscope license copy this too from folder /license.

– If Emergency Monitoring feature available starting with Introscope version 9.7 is activated copy files /sap/emon/*.

6. EM_HOME/ has to be given write access for SAP group sapsys from default 755 to 775

7. Start EM:

./EMCtrl.sh start
./EMCtrl.sh start: Starting Enterprise Manager...
(Please wait a few minutes for initialization to complete.)
Please check log file for more details.

8. check status

./EMCtrl.sh status
./EMCtrl.sh status: Enterprise Manager (pid XXXX) is running

8. In Solution Manager Basic Configuration Step 3.1, (Configure CA Introscope) Reload the installation. It will detect the new version.

– in case of any problems like Solman is still seeing the EM offline please review the note: 2198743 – Enterprise Manager is offline ” [Manager.UserManagementService] “” Failed to login from host “localhost” “
I had the error and this was due to wrong password of Admin user in Solman. After correcting this in solman_setup, step 3 -> user management the status showed with green status in solman_setup

9. After installing a new EM you have to prepare the EM for pushing monitoring metrics to Solution Manager running activity “Push DPC Configuration to CA Introscope” again. 

ISagent upgrade

After the upgrade of EM, also the ISagent can be upgraded.
You can check the version of ISagent from system information from Java start page:

ISagent_version

You can see that there are two ISagent versions are deployed on Solman, which can be a little bit confusing. But this is not only correct, but also the recommended scenario from SAP.
The IS agent version deployed on the managed system when performing the managed system configuration is dependent of the java version of the managed system and the IS agent version deployed on the Solman.
For Java 1.4 the IS agent 8.X will be deployed. If there is no ISAgent_MIN_J5 deployed on the Solman the IS agent 8.X is also deployed for systems running on Java 1.5 and 1.6. Although this scenario works, it is recommended to use IS agent 9.X for java systems that are running Java 1.5 or higher.
If both IS agent 8.X and IS agent 9.1.X (ISAgent_MIN_J5) are deployed on the Solman as in our case, then the managed systems running on Java 1.5 or higher the IS agent 9.X is deployed. This is the scenario recommended by SAP.
It is possible to have and IS Agent version 8.X deployed in parallel to the Solman IS agent 9.X, but it is NOT possible to have multiple IS agent 9.X versions deployed on the Solman Java stack.
The IS agent version needs to be lower or equal to the EM that it is connected to, that is also why we did first the EM upgrade.

The package that needed for the upgrade can be downloaded from:
https://launchpad.support.sap.com/ => Software Downloads => Support packages and patches => SAP Technology Components => SAP SOlution Manager => SAP Solution Manager 7.1 => Entry by component => Solution Manager Java Stack => WILY INTRO AGT 9.7 MIN J5VIASM => Patch for WILY INTRO AGT 9.7 MIN J5VIASM SP01

First enable the Maintenance Mode in the Agent Administration, deploy IS agent with the help of SUM tool. The upgrade will require a short downtime, as Java stack will be restarted during the process. After the successful deployment, disable the Maintenance Mode in the Agent Administration.

Helpful documents:

EM:

https://wiki.scn.sap.com/wiki/display/TechOps/IntroscopeEMUpgrade

https://websmp210.sap-ag.de/~sapidb/012002523100016951232015E/SAPISDOC97.pdf

Notes:

2198743 – Enterprise Manager is offline ” [Manager.UserManagementService] “” Failed to login from host “localhost” “

ISagent:

https://wiki.scn.sap.com/wiki/pages/viewpage.action?pageId=462049171
https://blogs.sap.com/2014/06/15/byte-code-adapter-introscope-java-agent/

Notes:

2068401 – How to update the ISAgent (Byte Code Adapter) – Solution Manager 7.10

Internal documents related to EM:

http://sapnotebook.com/wily-introscope-em-performance-analysis/
http://sapnotebook.com/article-introscope/

Wily Introscope EM performance analysis

If you have worked with Introscope EM, probably you have also faced some sort of a problem related its performance. For example, I had the situation that the EM would randomly stop itself, apparently for no reason. The version on which I had this error was 9.5.
First, I checked the logs and I could find this error in the em.log file:

java.lang.OutOfMemoryError: GC overhead limit exceeded

The problem was solved by increasing the total java heap memory, as a consequence also the GC duration was increased.

How to change the heap size of the EM? The parameters can be changed in the file Introscope_Enterprise_Manager.lax in the section:

#  LAX.NL.JAVA.OPTION.ADDITIONAL

lax.nl.java.option.additional=-Xms4096m -Xmx4096m -Djava.awt.headless=true -XX:MaxPermSize=512m -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi  -Xss512k -Dsun.ja
va2d.noddraw=true

In my case, I increased it from 2 Gb to 4 Gb and everything got solved out after that.

Unfortunately things do not go that smooth all the time, so what happens when EM does not start with a certain amount of configured memory? For example for version 9.7, I did the exact changes and EM would not start anymore after this, generating the following errors:

./EMCtrl.sh stop: Enterprise Manager stopped
Invalid initial heap size: -Xms4096m
The specified size exceeds the maximum representable size.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Or another error:

./EMCtrl.sh status: Enterprise Manager stopped
Invalid maximum heap size: -Xmx4096m
The specified size exceeds the maximum representable size.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Before making any changes to the java heap memory, I recommend that it is tested how much memory can be allocated to your JVM, especially that each EM runs with different version of JVM.
The command that you can test this is :

/path_of_the_jvm_used_by_EM/java -XmsNNNNm -XmxNNNNm -version

where NNNN is the amount the momory that you want to do the test with.

In case the value is accepted/correct it will return you default java environment info, otherwise it will throw in an error. You can test both initial and maximum heap size or just one of them.
Here are a few examples, so you can make an idea:

/usr/jdk/instances/jdk1.7.0_79/jre/bin/java -Xms3072m  -Xmx3072m -version
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) Server VM (build 24.79-b02, mixed mode)
/usr/jdk/instances/jdk1.7.0_79/jre/bin/java -Xms4096m  -Xmx4096m -version
Invalid initial heap size: -Xms4096m
The specified size exceeds the maximum representable size.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Now that we started to work on configuration files of EM, let’s see what are its most important directories and related files.

Main directories:

EM_HOME directory => by default, the home directory of EM, depending on the version, could be for 8.X /usr/sap/ccms/wilyintroscope and for 9.X and 10.X /usr/sap/ccms/apmintroscope. We can find here the configuration file:

  • Introscope_Enterprise_Manager.lax => the heap memory size can be modified from here, as we saw earlier

logs => contains the logs of EM where you can find quickly information about the agent state:

  • IntroscopeEnterpriseManager.log => you can find here information about the EM and Java VM version, in the format below:
3/10/15 05:55:20.231 PM CET [INFO] [main] [Manager] Introscope Enterprise Manager Release 9.5.6.0 (Build 956044)
3/10/15 05:55:20.232 PM CET [INFO] [main] [Manager] Using Java VM version "SAP Java Server VM 1.6.0_91" from SAP AG
3/10/15 05:55:20.232 PM CET [INFO] [main] [Manager] Using Introscope installation at: /usr/sap/ccms/wilyintroscope/.
3/10/15 05:55:20.232 PM CET [INFO] [main] [Manager] CA Wily Introscope(R) Version 9.5

But also information about any problems with the stop/start is present here.

  • em.log => the most important and readable errors can be found here
  • perflog.txt => more detailed information related to the performance and sizing are permanently written here. After the restart a new file is written and the old information is kept in perflog.txt.previous

bin => directory where you can find the stop/start script

config => various configurations and customizations can be done from here:

  • IntroscopeEnterpriseManager.properties => location of different directories and also many other configurations can be set in the file
  • apm-events-thresholds-config.xml => different thresholds can be set, for example limits the number of agent connections
  • agentclusters.xml => file used for monitoring capabilities

license => license for EM

sap => contains SAP connection related files:

  • SolMan_SID.e2emai.properties => IS Push configuration file used to push Introscope Metrics to Solution Manager Monitoring and Alerting Infrastructure (MAI)
  • TrustedCerts => certificate files used for single sign-on from SolutionManager

 

I started to dig down for any documentation related to EM performance and I was surprised in a good way.
There is a very nice tool from SAP named EMPerflogAnalyzer that will analyze the EM performance based only on the the perflog.txt file.
Here is how the tool looks like:

The checks that are done concern the following topics:

  • Number of Metrics
  • Number of Agents
  • Java Heap Space
  • Harvest Duration
  • Smartstor Duration (collected metrics)

For each point you get a stoplight color depending on whether you have a serious problem (red), just a warning (yellow) or if the configuration is correct (green). Also more details are available for each topic within the tool, guiding you on how to solve the problem.
There is an example on how a result of the tool analysis looked in my case:

The tool can be downloaded from the SAP note 2156320.

You can find other related EM articles on this site at:
http://sapnotebook.com/article-introscope/

More information about the tool and how to use it you can find in the official documentation provided below.

Helpful documentation:

1976769 – Introscope Enterprise Manager rejects new agents after reaching 400 connected agents
1630229 – CA Introscope Enterprise Manager has JAVA heap problems
2156320 – EM perflog analyzer tool – Introscope Enterprise Manager
1791543 – java.lang.OutOfMemoryError: PermGen space

Introscope Troubleshooting: https://wiki.scn.sap.com/wiki/display/TechOps/Introscope+Troubleshooting
Introscope Enterprise Manager.lax parameters: https://docops.ca.com/ca-apm/9-6/en/administrating/properties-files-reference/introscope-enterprise-manager-lax#IntroscopeEnterpriseManager.lax-lax.nl.java.option.additional
EM Perflog Analyzer Tool: https://wiki.scn.sap.com/wiki/display/TechOps/EM+Perflog+Analyzer+Tool
Performance health check: https://support.ca.com/us/knowledge-base-articles.tec604648.html

Introscope Enterprise Manager 10.1 installation

Introscope EM is a third party standalone engine that is installed under Solution Manager and used to monitor the performance of Java applications.
In this article I will present its installation under Solution Manger 7.2 on Linux platform. The version 10.1 is anyway only compatible with Solution Manager 7.2 (>=SP03). But the minimum version that can be installed on Solution Manger 7.2 is 9.7 SP01.

First of all, three packages are needed for the installation:

  1. Introscope Enterprise manager

Location for download: https://support.sap.com/swdc -> Software Downloads -> Support Packages and Patches (2nd tab) -> By Category –> SAP Technology Components -> SAP SOLUTION MANAGER -> SAP SOLUTION MANAGER 7.2 -> WILY INTROSCOPE -> WILY INTROSCOPE ENTPR MGR 10.1 -> (your OS for the enterprise mgr)

2.  Introscope Management Modules to be installed on top of Introscope Enterprise Manager

Location for download: https://support.sap.com/swdc -> Software Downloads -> Support Packages and Patches (2nd tab) -> By Category –> SAP Technology Components -> SAP SOLUTION MANAGER -> SAP SOLUTION MANAGER 7.2 -> -> WILY INTROSCOPE -> WILY INTRO EM MGMT MODULES 10.1 -> OS independent

3. Third-party external component package files based on the release

http://opensrcd.ca.com/ips/osgi/

osgi introscope package
Now that we have the packages, we can create a directory where to place and unzip the Introscope EM. Here we will also put the external component files, as the installer will automatically identify these during the installation.

mkdir wily_ent_mng
unzip WILYISEM00_0-70001413.zip
ls
META-INF                   ca-eula.ja.txt  ca-eula.txt        ca-eula.zh_TW.txt  installer.properties                  osgiPackages.v10.1.0.15.unix.tar  version.txt
WILYISEM00_0-70001413.zip  ca-eula.ko.txt  ca-eula.zh_CN.txt  eula.txt           introscope10.1.0.15linuxAMD64SAP.bin  silent.install.failed.txt

If it does not have already these configurations, modify the installer.properties file as follows and leave the rest of the options as they are:

ca-eulaFile=ca-eula.txt
USER_INSTALL_DIR=/usr/sap/ccms/apmintroscope/
externalComponentPackage=osgiPackages.v10.1.0.15.unix.tar
eulaFile=eula.txt

If you want to use a different password than from the default one for the Introscope users, you can also edit these parameters:

emAdminPassword=
emGuestPassword=

In order to be able to start the installation you must agree with the CA End user license agreement. You can do this by modifying the files this way:

eula.txt => LGPL=accept
ca-eula.txt => CA-EULA=accept

Then start the installation in silent mode with sidadm user (sidadm is the admin user of the Solution Manger instance):

./introscope10.1.0.15linuxAMD64SAP.bin -Djava.awt.headless=true
Preparing to install...
Extracting the JRE from the installer archive...
Unpacking the JRE...
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...

Launching installer...

Silent installer completed with status of: SUCCESS. For more details, consult install logs at: /usr/sap/ccms/apmintroscope/install. Please wait for the command prompt to return...

And the installation is completed. For me the installation did not take more than 5 minutes.

You can check the installation logs under /usr/sap/ccms/apmintroscope/install
In case the installation wasn’t successful you can find more hints in the logfile: /usr/sap/SID/willy_ent_mng/silent.install.failed.txt

The default directory for Introscope EM is /usr/sap/ccms/apmintroscope. Here you can also find the script for start/stop/check state.

After this is installed, install the module. This is done by copying the module zip in the directory of wily: /usr/sap/ccms/apmintroscope and unzip WILYISMM00P_1-70001722.zip here directly.

Now let’s start it:

cd /usr/sap/ccms/apmintroscope
./EMCtrl.sh status
./EMCtrl.sh status: Enterprise Manager stopped
./EMCtrl.sh start
./EMCtrl.sh start: Starting Enterprise Manager...
(Please wait a few minutes for initialization to complete.)
Please check log file for more details.

The link should be reachable: http://hostname:8081/webview

There are two users available for login. If you did not change their credentials before installation the default passwords are:

user Admin, passwd: Admin89
user Guest, passwd: guest12

You can find other related EM articles on this site at:

http://sapnotebook.com/wily-introscope-em-performance-analysis

For the installation, I used the documentation from SAP, where you can find even more information:

Installation guide:

https://service.sap.com/instguides -> SAP Components -> SAP Solution Manger -> Release 7.2 -> Additional Information – Introscope -> Setup Guide Introscope 10.1

Notes:

2285189 – Introscope 10.1 Release Notes for changes and open issues
797147 – Introscope Installation for SAP Customers