.. Warning::

    What I'm about to show you is the ugliest hack I think I've ever done.

This is how we get stats from our Websphere 5.x servers into an RRD database.
We're not pulling this into Nagios just yet because I'm trying to use
perfparse less and looking for an alternate graphing solution.

(FWIW, my only complaint with perfparse is the inability to graph multiple
values on a single graph.)

So on to the script.

The first thing that will need to happen is configuration on the WebSphere
side. This means installing the PerfServletApp.ear distributed with WebSphere
Application Server (which I'll refer to as WAS from here on out).

If you didn't install it when you did the original install, then you'll have
to install it manually. Unfortunately (or fortunately if you like the command
line like me), you cannot use the GUI to install the perf servlet (or the
admin console for that matter). You'll need to use a little Jacl or Jython and
wsadmin to do the job. I'm a Jacl kind of guy myself since I picked up tcl
many moons ago.

I like to save all the jacl stuff into a script even if I don't think I'll
ever use it again. The interactive mode is nice but calling a file is quicker.

Steps to installing the PerfServletApp.ear:

    1) Check and see if you have the perf servlet installed already::

        /opt/WebSphere/AppServer/bin/wsadmin.sh -c '$AdminApp list'
        WASX7209I: Connected to process "server1" on node qawas04 using SOAP
        connector;  The type of process is: UnManagedProcess
        adminconsole
        qa-02.018.04
        qa-02.018.05
    
    You can see we have three apps deployed:

        - adminconsole
        - qa-02.018.04
        - qa-02.018.05

    2) Install the PerfServletApp.ear.

        I suggest that this be done using a .jacl file since there are
        multiple command (okay TWO command) that need to be issued instead of
        one. The first is a call to install and the second is a call to save
        the configuration

        Open up vi (natch!) and put these lines in it::

            $AdminApp install PerfServletApp.ear
            $AdminConfig save

        Save the file as "perfinstall.jacl"

    3) Install the application into WebSphere.
        
        Normally you could install the application via the admin interface but
        in this case you cannot. Using the new Jacl file, we're going to
        install install the application (making sure the EAR file is in the
        same directory as the Jacl script::

            /opt/WebSphere/AppServer/bin/wsadmin.sh -conntype NONE -f perfinstall.jacl

        The key in this line is to use -conntype NONE. Adminconsole and the
        perf tool exist outside of your normal WAS applications. The
        "-conntype NONE" causes wsadmin to use a local connection to the
        server outside of the normal interfaces.

        Once you run the script, you should get output from the install
        process that the application was installed successfully::

            WASX7357I: By request, this scripting client is not connected to
            any server process. Certain configuration and application
            operations will be available in local mode.
            ADMA0073W: Custom permissions are found in the
            [(com.ibm.websphere.security.WebSphereRuntimePermission
            AdminPermission),
            (com.ibm.websphere.security.WebSphereRuntimePermission
            accessRuntimeClasses)] policy file. Custom permissions can
            compromise the integrity of Java 2 Security.
            WASX7327I: Contents of was.policy file:
             grant codeBase "file:${application}" {
               permission
               com.ibm.websphere.security.WebSphereRuntimePermission
               "AdminPermission";
                 permission
                 com.ibm.websphere.security.WebSphereRuntimePermission
                 "accessRuntimeClasses";
                 };
                 grant codeBase "file:${application}" {
                   permission java.io.FilePermission
                   "${was.install.root}${/}lib${/}ffdc.jar", "read";
                   };
                   null, false, 0
                   ================== qawas04
                   ADMA6010I: The tasks are
                   [com.ibm.ws.webservices.deploy.WSDeployTask,
                   com.ibm.ws.management.application.task.ConfigureTask,
                   com.ibm.ws.management.application.task.BackupAppTask]
                   ADMA5016I: Installation of perfServletApp started.
                   ADMA6017I: Saved document
                   /opt/WebSphere/AppServer/wstemp/Script10c22887099/workspace/cells/qawas04/applications/perfServletApp.ear/deployments/perfServletApp/deployment.xml

                   ....... TRUNCATED TEXT .........

                   ADMA6016I: Add to workspace
                   perfServletApp.war/WEB-INF/web.xml
                   ADMA5005I: Application perfServletApp configured in
                   WebSphere repository
                   ADMA5037I: Starting backup of app at
                   /opt/WebSphere/AppServer/wstemp/Script10c22887099/workspace/cells/qawas04/applications/perfServletApp.ear
                   ADMA5038I: Completed backup of app at
                   /opt/WebSphere/AppServer/wstemp/Script10c22887099/workspace/cells/qawas04/applications/perfServletApp.ear/perfServletApp.ear
                   ADMA5001I: Application binaries saved in
                   /opt/WebSphere/AppServer/wstemp/Script10c22887099/workspace/cells/qawas04/applications/perfServletApp.ear/perfServletApp.ear
                   ADMA6011I: Deleting directory tree /tmp/app_10c228880ce
                   ADMA5011I: Cleanup of temp dir for app perfServletApp done.
                   ADMA5013I: Application perfServletApp installed
                   successfully.
            
        And the application is installed.

    4) At this point I would simply bounce WebSphere (you are doing this on a
    preprod or dev server right?). When WAS is started again, you should be
    able to access the servlet at::

            http://default_host:default_host_port/wasPerfTool/servlet/perfservlet

    This should return an XML document that you can then parse for various
    stats. If you are having any problems, then it is most likely related
    to your virtual hosts configuration. Check the Virtual Host mapping
    for the application and if possible test it with anothe virtual host.
    In our environment, we use admin_host for the perf servlet as we don't
    HAVE a default_host. We have custom virtual hosts for our apps.

    I'm not going to include the whole XML output here. You can find a
    sample of all the types of output available on the IBM WebSphere
    Information Center. I will, however, include the sections that we
    monitor - JVM Usage and Connection Pool stats::

            

            Look here - <jvmRuntimeModule>
            Use the sections below
                    <totalMemory>
                        <PerfLoadInfo currentValue="1074461.0" integral="5.34383300677E11" mean="1074331.6392452905" time="1151632287170" timeSinceCreate="497410.0" uid="pmi1"/>
                    </totalMemory>
                    <freeMemory>
                        <PerfNumericInfo time="1151632287170" uid="pmi2" val="604049.0"/>
                    </freeMemory>
                    <usedMemory>
                        <PerfNumericInfo time="1151632287170" uid="pmi3" val="470411.0"/>
                    </usedMemory>
                    <upTime>
                        <PerfNumericInfo time="1151632287170" uid="pmi4" val="497.0"/>
                    </upTime>
            </jvmRuntimeModule>

            Look here - <jdbc_myjdbc>
            Use the sections below
                    <poolSize>
                        <PerfLoadInfo currentValue="1.0" integral="48751.0" mean="0.9836366571163392" time="1151633157366" timeSinceCreate="49562.0" uid="pmi5" /> 
                    </poolSize>
                    <freePoolSize>
                        <PerfLoadInfo currentValue="1.0" integral="43800.0" mean="0.8837415762075784" time="1151633157366" timeSinceCreate="49562.0" uid="pmi6" /> 
                    </freePoolSize>
                    <concurrentWaiters>
                        <PerfLoadInfo currentValue="0.0" integral="0.0" mean="0.0" time="1151633157366" timeSinceCreate="49565.0" uid="pmi7" /> 
                    </concurrentWaiters>
                    <faults>
                        <PerfNumericInfo time="1151633157366" uid="pmi8" val="0.0" /> 
                    </faults>
                    <percentUsed>
                        <PerfLoadInfo currentValue="0.0" integral="49510.0" mean="0.9989508090876075" time="1151633157366" timeSinceCreate="49562.0" uid="pmi9" /> 
                     </percentUsed>
                     <percentMaxed>
                       <PerfLoadInfo currentValue="0.0" integral="0.0" mean="0.0" time="1151633157366" timeSinceCreate="49562.0" uid="pmi10" /> 
                     </percentMaxed>
                     <avgUseTime>
                        <PerfStatInfo mean="4951.0" num="1" sum_of_squares="2.4512401E7" time="1151633157366" total="4951.0" uid="pmi12" /> 
                     </avgUseTime>
                     <avgWaitTime>
                        <PerfStatInfo mean="0.0" num="0" sum_of_squares="0.0" time="1151633157366" total="0.0" uid="pmi13" /> 
                     </avgWaitTime>

    Those are the sections we concern ourselves with mostly. We watch
    memory from within the container and we are concerned about our
    specific connection pools for our application.

        There are shortcuts to specific modules if you are interested::

                http://qawas04:9090/wasPerfTool/servlet/perfservlet?Module=jvmRuntimeModule+connectionPoolModule

        Again a listing of all the modules and definitions are on the IBM
        website.

    Now that we have WebSphere giving us some useful information, it's just a
    matter of a little perl to get the job done. In my case we'll use the
    following modules::

        LWP::Simple
        XML::Simple
        Data::Dumper
        RRDs

    .. Note::
    
        RRDs is a part of the RRD distribution.

    I created two rrd files for each stat (jvm and jdbc)::

        rrdtool create websphere-jvm.rrd --start 1136809202 --step 300 \
        DS:load:GAUGE:600:0:100 \
        RRA:AVERAGE:0.5:1:105120 \
        RRA:MIN:0.5:1:105120 \
        RRA:MAX:0.5:1:105120 \
        RRA:LAST:0.5:1:105120

        rrdtool create websphere-jdbc.rrd --start 1136809202 --step 300 \
        DS:poolSize:GAUGE:600:0:500 \
        DS:freePoolSize:GAUGE:600:0:500 \
        DS:concurrentWaiters:GAUGE:600:0:500 \
        DS:percentUsed:GAUGE:600:0:100 \
        DS:avgUseTime:GAUGE:600:0:U \
        DS:avgWaitTime:GAUGE:600:0:U \
        RRA:AVERAGE:0.5:1:105120 \
        RRA:MIN:0.5:1:105120 \
        RRA:MAX:0.5:1:105120 \
        RRA:LAST:0.5:1:105120

    .. Note::

        You may notice that I keep some very detailed stats in my RRDs. While
        it's true that you **CAN** have too much information, in this case I
        like to have 3 years of historical data. This **WILL** create a much
        larger RRD file (i.e. 3.3MB for each file) but disk space is cheap.

    And here's the perl script that does all the work::

        #!/usr/bin/perl -w
        use LWP::Simple;
        use XML::Simple;
        use Data::Dumper;
        use RRDs;

        my $url =
        "http://qawas04:9090/wasPerfTool/servlet/perfservlet?Module=jvmRuntimeModule+connectionPoolModule";
        my $node = "qawas04";
        my $server = "server1";
        my $jdbc_driver = "DB2_Database_Driver";
        my $connpool = "jdbc_myjdbc";
        my $sysinfo = get $url;
        my $xml = new XML::Simple;

        $data = $xml->XMLin($sysinfo);

        # The below line dumps the xml as perl sees it
        #print Dumper($data);

        my $poolSize = sprintf("%d", $data->{Node}->{$node}->{Server}->{$server}->{connectionPoolModule}->{$jdbc_driver}->{$connpool}->{poolSize}->{PerfLoadInfo}->{currentValue}
        );

        my $freePoolSize = sprintf("%d", $data->{Node}->{$node}->{Server}->{$server}->{connectionPoolModule}->{$jdbc_driver}->{$connpool}->{freePoolSize}->{PerfLoadInfo}->{currentValue}
        );

        my $percentUsed = sprintf ("%d", $data->{Node}->{$node}->{Server}->{$server}->{connectionPoolModule}->{$jdbc_driver}->{$connpool}->{percentUsed}->{PerfLoadInfo}->{currentValue}
        );

        my $concurrentWaiters = sprintf("%d", $data->{Node}->{$node}->{Server}->{$server}->{connectionPoolModule}->{$jdbc_driver}->{$connpool}->{concurrentWaiters}->{PerfLoadInfo}->{currentValue}
        );

        my $avgUseTime = sprintf("%d", $data->{Node}->{$node}->{Server}->{$server}->{connectionPoolModule}->{$jdbc_driver}->{$connpool}->{avgUseTime}->{PerfStatInfo}->{mean}
        );

        my $avgWaitTime = sprintf("%d", $data->{Node}->{$node}->{Server}->{$server}->{connectionPoolModule}->{$jdbc_driver}->{$connpool}->{avgWaitTime}->{PerfStatInfo}->{mean}
        );

        # sample command
        # rrdtool update elmsdb-connections.rrd 1101963372:9
        my $timestamp = time();
        #print "rrdtool update websphere-jvm.rrd $timestamp:$jvmpercent\n";
        RRDs::update ("websphere-jdbc.rrd", "--template", "poolSize:freePoolSize:concurrentWaiters:percentUsed:avgUseTime:avgWaitTime", "N:$poolSize:$freePoolSize:$concurrentWaiters:$percentUsed:$avgUseTime:$avgWaitTime");
        RRDs::update ("websphere-jvm.rrd", "--template", "load", "N:$percentMemUsed");
        print "\n
        $poolSize\n
        $freePoolSize\n
        $concurrentWaiters\n
        $percentUsed\n
        $avgUseTime\n
        $avgWaitTime\n
        \n";

    Mind you this is far from a detailed discussion of the delicates of
    wasPerfTool. I just wanted to show you how to get the data out. In terms
    of integration with Nagios, you can use check_rrd or your own plugin to
    get the values out of the RRD file or you can simply get the input
    directly and send it to Nagios, letting whatever perfdata handler you have
    do all the work!

    .. image:: websphere-rrd.png

.. footer:: footer.txt