Warning

What I'm about to show you is the ugliest hack I think I've ever done.

This is how we get stats from our Websphere 5.x servers into an RRD database. We're not pulling this into Nagios just yet because I'm trying to use perfparse less and looking for an alternate graphing solution.

(FWIW, my only complaint with perfparse is the inability to graph multiple values on a single graph.)

So on to the script.

The first thing that will need to happen is configuration on the WebSphere side. This means installing the PerfServletApp.ear distributed with WebSphere Application Server (which I'll refer to as WAS from here on out).

If you didn't install it when you did the original install, then you'll have to install it manually. Unfortunately (or fortunately if you like the command line like me), you cannot use the GUI to install the perf servlet (or the admin console for that matter). You'll need to use a little Jacl or Jython and wsadmin to do the job. I'm a Jacl kind of guy myself since I picked up tcl many moons ago.

I like to save all the jacl stuff into a script even if I don't think I'll ever use it again. The interactive mode is nice but calling a file is quicker.

Steps to installing the PerfServletApp.ear:

  1. Check and see if you have the perf servlet installed already:

     /opt/WebSphere/AppServer/bin/wsadmin.sh -c '$AdminApp list'
     WASX7209I: Connected to process "server1" on node qawas04 using SOAP
     connector;  The type of process is: UnManagedProcess
     adminconsole
     qa-02.018.04
     qa-02.018.05
    

You can see we have three apps deployed:

  1. Install the PerfServletApp.ear.

    I suggest that this be done using a .jacl file since there are multiple command (okay TWO command) that need to be issued instead of one. The first is a call to install and the second is a call to save the configuration

    Open up vi (natch!) and put these lines in it:

     $AdminApp install PerfServletApp.ear
     $AdminConfig save
    

    Save the file as "perfinstall.jacl"

  2. Install the application into WebSphere.

    Normally you could install the application via the admin interface but in this case you cannot. Using the new Jacl file, we're going to install install the application (making sure the EAR file is in the same directory as the Jacl script:

     /opt/WebSphere/AppServer/bin/wsadmin.sh -conntype NONE -f perfinstall.jacl
    

    The key in this line is to use -conntype NONE. Adminconsole and the perf tool exist outside of your normal WAS applications. The "-conntype NONE" causes wsadmin to use a local connection to the server outside of the normal interfaces.

    Once you run the script, you should get output from the install process that the application was installed successfully:

     WASX7357I: By request, this scripting client is not connected to
     any server process. Certain configuration and application
     operations will be available in local mode.
     ADMA0073W: Custom permissions are found in the
     [(com.ibm.websphere.security.WebSphereRuntimePermission
     AdminPermission),
     (com.ibm.websphere.security.WebSphereRuntimePermission
     accessRuntimeClasses)] policy file. Custom permissions can
     compromise the integrity of Java 2 Security.
     WASX7327I: Contents of was.policy file:
      grant codeBase "file:${application}" {
        permission
        com.ibm.websphere.security.WebSphereRuntimePermission
        "AdminPermission";
          permission
          com.ibm.websphere.security.WebSphereRuntimePermission
          "accessRuntimeClasses";
          };
          grant codeBase "file:${application}" {
            permission java.io.FilePermission
            "${was.install.root}${/}lib${/}ffdc.jar", "read";
            };
            null, false, 0
            ================== qawas04
            ADMA6010I: The tasks are
            [com.ibm.ws.webservices.deploy.WSDeployTask,
            com.ibm.ws.management.application.task.ConfigureTask,
            com.ibm.ws.management.application.task.BackupAppTask]
            ADMA5016I: Installation of perfServletApp started.
            ADMA6017I: Saved document
            /opt/WebSphere/AppServer/wstemp/Script10c22887099/workspace/cells/qawas04/applications/perfServletApp.ear/deployments/perfServletApp/deployment.xml
    
            ....... TRUNCATED TEXT .........
    
            ADMA6016I: Add to workspace
            perfServletApp.war/WEB-INF/web.xml
            ADMA5005I: Application perfServletApp configured in
            WebSphere repository
            ADMA5037I: Starting backup of app at
            /opt/WebSphere/AppServer/wstemp/Script10c22887099/workspace/cells/qawas04/applications/perfServletApp.ear
            ADMA5038I: Completed backup of app at
            /opt/WebSphere/AppServer/wstemp/Script10c22887099/workspace/cells/qawas04/applications/perfServletApp.ear/perfServletApp.ear
            ADMA5001I: Application binaries saved in
            /opt/WebSphere/AppServer/wstemp/Script10c22887099/workspace/cells/qawas04/applications/perfServletApp.ear/perfServletApp.ear
            ADMA6011I: Deleting directory tree /tmp/app_10c228880ce
            ADMA5011I: Cleanup of temp dir for app perfServletApp done.
            ADMA5013I: Application perfServletApp installed
            successfully.
    

    And the application is installed.

4) At this point I would simply bounce WebSphere (you are doing this on a preprod or dev server right?). When WAS is started again, you should be able to access the servlet at:

 http://default_host:default_host_port/wasPerfTool/servlet/perfservlet

This should return an XML document that you can then parse for various stats. If you are having any problems, then it is most likely related to your virtual hosts configuration. Check the Virtual Host mapping for the application and if possible test it with anothe virtual host. In our environment, we use admin_host for the perf servlet as we don't HAVE a default_host. We have custom virtual hosts for our apps.

I'm not going to include the whole XML output here. You can find a sample of all the types of output available on the IBM WebSphere Information Center. I will, however, include the sections that we monitor - JVM Usage and Connection Pool stats:

 Look here - <jvmRuntimeModule>
 Use the sections below
         <totalMemory>
             <PerfLoadInfo currentValue="1074461.0" integral="5.34383300677E11" mean="1074331.6392452905" time="1151632287170" timeSinceCreate="497410.0" uid="pmi1"/>
         </totalMemory>
         <freeMemory>
             <PerfNumericInfo time="1151632287170" uid="pmi2" val="604049.0"/>
         </freeMemory>
         <usedMemory>
             <PerfNumericInfo time="1151632287170" uid="pmi3" val="470411.0"/>
         </usedMemory>
         <upTime>
             <PerfNumericInfo time="1151632287170" uid="pmi4" val="497.0"/>
         </upTime>
 </jvmRuntimeModule>

 Look here - <jdbc_myjdbc>
 Use the sections below
         <poolSize>
             <PerfLoadInfo currentValue="1.0" integral="48751.0" mean="0.9836366571163392" time="1151633157366" timeSinceCreate="49562.0" uid="pmi5" /> 
         </poolSize>
         <freePoolSize>
             <PerfLoadInfo currentValue="1.0" integral="43800.0" mean="0.8837415762075784" time="1151633157366" timeSinceCreate="49562.0" uid="pmi6" /> 
         </freePoolSize>
         <concurrentWaiters>
             <PerfLoadInfo currentValue="0.0" integral="0.0" mean="0.0" time="1151633157366" timeSinceCreate="49565.0" uid="pmi7" /> 
         </concurrentWaiters>
         <faults>
             <PerfNumericInfo time="1151633157366" uid="pmi8" val="0.0" /> 
         </faults>
         <percentUsed>
             <PerfLoadInfo currentValue="0.0" integral="49510.0" mean="0.9989508090876075" time="1151633157366" timeSinceCreate="49562.0" uid="pmi9" /> 
          </percentUsed>
          <percentMaxed>
            <PerfLoadInfo currentValue="0.0" integral="0.0" mean="0.0" time="1151633157366" timeSinceCreate="49562.0" uid="pmi10" /> 
          </percentMaxed>
          <avgUseTime>
             <PerfStatInfo mean="4951.0" num="1" sum_of_squares="2.4512401E7" time="1151633157366" total="4951.0" uid="pmi12" /> 
          </avgUseTime>
          <avgWaitTime>
             <PerfStatInfo mean="0.0" num="0" sum_of_squares="0.0" time="1151633157366" total="0.0" uid="pmi13" /> 
          </avgWaitTime>

Those are the sections we concern ourselves with mostly. We watch memory from within the container and we are concerned about our specific connection pools for our application.

There are shortcuts to specific modules if you are interested:

 http://qawas04:9090/wasPerfTool/servlet/perfservlet?Module=jvmRuntimeModule+connectionPoolModule

Again a listing of all the modules and definitions are on the IBM website.

Now that we have WebSphere giving us some useful information, it's just a matter of a little perl to get the job done. In my case we'll use the following modules:

 LWP::Simple
 XML::Simple
 Data::Dumper
 RRDs

Note

RRDs is a part of the RRD distribution.

I created two rrd files for each stat (jvm and jdbc):

 rrdtool create websphere-jvm.rrd --start 1136809202 --step 300 \
 DS:load:GAUGE:600:0:100 \
 RRA:AVERAGE:0.5:1:105120 \
 RRA:MIN:0.5:1:105120 \
 RRA:MAX:0.5:1:105120 \
 RRA:LAST:0.5:1:105120

 rrdtool create websphere-jdbc.rrd --start 1136809202 --step 300 \
 DS:poolSize:GAUGE:600:0:500 \
 DS:freePoolSize:GAUGE:600:0:500 \
 DS:concurrentWaiters:GAUGE:600:0:500 \
 DS:percentUsed:GAUGE:600:0:100 \
 DS:avgUseTime:GAUGE:600:0:U \
 DS:avgWaitTime:GAUGE:600:0:U \
 RRA:AVERAGE:0.5:1:105120 \
 RRA:MIN:0.5:1:105120 \
 RRA:MAX:0.5:1:105120 \
 RRA:LAST:0.5:1:105120

Note

You may notice that I keep some very detailed stats in my RRDs. While it's true that you CAN have too much information, in this case I like to have 3 years of historical data. This WILL create a much larger RRD file (i.e. 3.3MB for each file) but disk space is cheap.

And here's the perl script that does all the work:

 #!/usr/bin/perl -w
 use LWP::Simple;
 use XML::Simple;
 use Data::Dumper;
 use RRDs;

 my $url =
 "http://qawas04:9090/wasPerfTool/servlet/perfservlet?Module=jvmRuntimeModule+connectionPoolModule";
 my $node = "qawas04";
 my $server = "server1";
 my $jdbc_driver = "DB2_Database_Driver";
 my $connpool = "jdbc_myjdbc";
 my $sysinfo = get $url;
 my $xml = new XML::Simple;

 $data = $xml->XMLin($sysinfo);

 # The below line dumps the xml as perl sees it
 #print Dumper($data);

 my $poolSize = sprintf("%d", $data->{Node}->{$node}->{Server}->{$server}->{connectionPoolModule}->{$jdbc_driver}->{$connpool}->{poolSize}->{PerfLoadInfo}->{currentValue}
 );

 my $freePoolSize = sprintf("%d", $data->{Node}->{$node}->{Server}->{$server}->{connectionPoolModule}->{$jdbc_driver}->{$connpool}->{freePoolSize}->{PerfLoadInfo}->{currentValue}
 );

 my $percentUsed = sprintf ("%d", $data->{Node}->{$node}->{Server}->{$server}->{connectionPoolModule}->{$jdbc_driver}->{$connpool}->{percentUsed}->{PerfLoadInfo}->{currentValue}
 );

 my $concurrentWaiters = sprintf("%d", $data->{Node}->{$node}->{Server}->{$server}->{connectionPoolModule}->{$jdbc_driver}->{$connpool}->{concurrentWaiters}->{PerfLoadInfo}->{currentValue}
 );

 my $avgUseTime = sprintf("%d", $data->{Node}->{$node}->{Server}->{$server}->{connectionPoolModule}->{$jdbc_driver}->{$connpool}->{avgUseTime}->{PerfStatInfo}->{mean}
 );

 my $avgWaitTime = sprintf("%d", $data->{Node}->{$node}->{Server}->{$server}->{connectionPoolModule}->{$jdbc_driver}->{$connpool}->{avgWaitTime}->{PerfStatInfo}->{mean}
 );

 # sample command
 # rrdtool update elmsdb-connections.rrd 1101963372:9
 my $timestamp = time();
 #print "rrdtool update websphere-jvm.rrd $timestamp:$jvmpercent\n";
 RRDs::update ("websphere-jdbc.rrd", "--template", "poolSize:freePoolSize:concurrentWaiters:percentUsed:avgUseTime:avgWaitTime", "N:$poolSize:$freePoolSize:$concurrentWaiters:$percentUsed:$avgUseTime:$avgWaitTime");
 RRDs::update ("websphere-jvm.rrd", "--template", "load", "N:$percentMemUsed");
 print "\n
 $poolSize\n
 $freePoolSize\n
 $concurrentWaiters\n
 $percentUsed\n
 $avgUseTime\n
 $avgWaitTime\n
 \n";

Mind you this is far from a detailed discussion of the delicates of wasPerfTool. I just wanted to show you how to get the data out. In terms of integration with Nagios, you can use check_rrd or your own plugin to get the values out of the RRD file or you can simply get the input directly and send it to Nagios, letting whatever perfdata handler you have do all the work!


Note

Last Modified 07/07/2006 13:12

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 2.5 License.