Jeff Hair (jshair)

Monitoring in JBoss

2009-12-23T17:48:00.000-08:00

While at a client site or within a testing environment, have you ever started to wonder how many users are on the application? How is your application running with regards to memory (heap size)? Are you close to using all of the database connections in the connection pool? For answers to some questions, maybe your application container provides a status (tomcat) or monitoring screen (WebLogic).

To facilitate recording of these statistics when using JBoss, JBoss has included the ability to log/monitor JMX Mbean values. And it's not difficult to install. Once values are being logged, you no longer have to continue refreshing the JMX console to see the values updated.

For installing and monitoring of your web application(s), perform the following steps:

Copy monitor XML files into $JBOSS/server/server_name/deploy
Copy $JBOSS/docs/examples/jmx/logging-monitor/lib/logging-monitor.jar into $JBOSS/server/server_name/lib
Create monitor XML files to monitor JMX MBeans (samples below)

And there you have statistics being saved in a log file...ready for you to parse and create pretty little graphs for the rest of the world to read and understand. For our deployments, we used these logs to track & monitor the following items. As a side affect of monitoring, you also get to see the peak usage times and potentially performance / memory related issues.

DB connections

In use
Available
Max Connections In Use

JVM activity

Heap size
Threads

For additional reading or research on monitoring within JBoss, check out the following links:

DB Connection Monitoring Sample
Here's the XML necessary to monitor a JDBC connection pool (XML comments omitted)

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE server PUBLIC
"-//JBoss//DTD MBean Service 4.0//EN"
"http://www.jboss.org/j2ee/dtd/jboss-service_4_0.dtd">

<server>
<mbean code="org.jboss.services.loggingmonitor.LoggingMonitor"
      name="jboss.monitor:type=LoggingMonitor,name=MY-DSMonitor">

  <attribute name="Filename">${jboss.server.home.dir}/log/my-ds.log</attribute>

  <attribute name="AppendToFile">false</attribute>

  <attribute name="RolloverPeriod">DAY</attribute>

  <attribute name="MonitorPeriod">10000</attribute>

   <attribute name="MonitoredObjects">
      <configuration>
         <monitoredmbean name="jboss.jca:name=MY-DS,service=ManagedConnectionPool"
                         logger="jca.my-ds">
            <attribute>InUseConnectionCount</attribute>
            <attribute>AvailableConnectionCount</attribute>
            <attribute>ConnectionCreatedCount</attribute>
            <attribute>ConnectionDestroyedCount</attribute>
            <attribute>MaxConnectionsInUseCount</attribute>
         </monitoredmbean>
      </configuration>
   </attribute>

  <depends>jboss.jca:name=MY-DS,service=ManagedConnectionPool</depends>
</mbean>

</server>

Utilizing OpenSymphony Caching (OSCache) with iBatis

2009-12-18T06:13:00.000-08:00

Background

My current project is ramping up to be deployed into a centrally hosted data center to be accessed by a large volume of users. In past deployments, we could expect, at most, 200-300 users to be logged into our web application. When translating that number to active users, we can expect somewhere in the range of 25 to 100 active users at any given point in time.

With deployment into a centrally hosted environment, our anticipated user base significantly increases to be approximately 1000 concurrent, active users. With this large number of users, we wanted to investigate caching frequently accessed objects.

The major area in our web application that currently utilizes caching is at the DAO level. Our DAOs are leveraging iBatis and already using iBatis in-memory cache implementation. Moving to a centrally hosted environment, the application will be clustered. Thus, supporting fail-over and high availability. However, using the iBatis cache, we risk the possibility that users see different objects depending upon their assigned clustered instance and when the iBatis cache is refreshed. To synchronize cache flushing across the cache, we decided to investigate incorporating a distributed caching mechanism. With iBatis supporting OSCache, we decided to start there.

Implementation

As mentioned previously, iBatis documentation refers to using OSCache for distributed caching within a clustered environment. Assuming you are using maven to build your project, configuring iBatis to use OSCache is extremely easy and well documented.

Modify the project's pom.xml to include oscache.jar (2.4) as a dependency
Identify and modify the sqlMap / DAO that you wish to use oscache. change cacheModel type to be "OSCACHE"
Optionally include an oscache.properties file on the class path. Within development or continuous integration (CI) environments, this file does not need included as default properties will be applied. Deployment in production or other test areas can include the file.

For configuration options and settings including clustering instructions, refer to OSCache web site. The clustering option sends a flush event that causes all caches within the cluster to be flushed. Currently, OSCache can be configured to send flush events via JMS or JavaGroups (multi-cast). All of the settings are nicely documented.

http://www.opensymphony.com/oscache/wiki/Configuration.html
http://www.opensymphony.com/oscache/wiki/Clustering.html

Retrieving a list of changes for a release in SVN

2009-04-07T07:24:00.000-07:00

Recently for our project, we needed to review the list of SVN commits on a branch. I'm sure there are several ways that possibly even include dates. For our purposes, we wanted to review the entire list and used the following command:

svn log --stop-on-copy –v https://host/svn/project/branches/project-0.8.0902.1

And sha-bam, a nice, lengthy report showing the status of the changes committed to this branch.

Thanks goes to one of our many, in-house, resident SVN experts!

Having to scale a web application?

2008-08-05T05:27:00.000-07:00

In a recent sprint planning meeting, solution owners unveiled how a new customer would be using our web application. Of course, it's a web application and, as such, needs to support many, concurrent users. Up until now, we were looking for our application to support approximately roughly 500 users; probably not concurrent, but potentially.

With learning how the new client will use the application, we may blow those numbers out of the water, both overall users as well as concurrent. So, how do you program and application to support large, very large volumes of users? Will the application scale if we cluster the web application servers? Will performance decrease with more users? Can we just throw a bigger machine with more CPU & memory to handle the load?

I'm a big believer in the fact that scalability needs to be designed from the get-go and then monitored. While some items may not be implemented immediately due to project constraints, scalability needs to be considered from day 1 and the code watched and reviewed to ensure new designs, code, etc will not adversely affect scalability and potentially performance.

Two articles were published in TSS on scaling JEE applications that "hit the nail on the head". If you're faced with having to support volumes of users, these articles are a must read as the writer had (has?) the opportunity to pound on application to test their ability to scale under heavy load AND to analyze why they failed or succeeded.

If you're not having to program for scalability now, the articles are still an excellent read & resource!

Scaling Your Java EE Applications - Part 1
Scaling Your Java EE Applications Part 2

SVN Merge (Trunk to Branch)

2008-07-01T11:17:00.000-07:00

Ever have code changes that need to be pushed into a branch? Or merged back into HEAD or the trunk?

Recently (today), I had the need to merge code changes from HEAD into a newly created branch. Given the fact that my changes spanned a couple of weeks (no lectures, please, as I was on vacation :D), I could not remember all of the lines that were changed in 9 files. I didn't want to blindly copy the files into the branch as I might (shouldn't really) overwrite another developers changes.

After a quick google search and a read of a short blog posting, I had found a quick path forward. For the same reasons that caused the Jake to write a blog, I'm also writing this so that I can easily find it.

I checked in my files into HEAD and got the revision number (7200). Then I changed directories to the directory of the branch and ran the following command:


  svn merge -r 7199:7200 https://phlcvs01/svn/netcds/trunk .

If you want to preview the changes, specify '--dry-run' which causes SVN to list the changes that will occur. Using '-r 7199:7200' causes subversion to only grab the differences between those revisions. Upon executing the command, 'svn stat' shows the modified files that you need to check into the branch.

Simple and easy.

Supporting Multiple DBs using iBatis

2008-03-17T16:53:00.000-07:00

I previously wrote about configuring multiple data sources within JBoss. When writing SQL, you may undoubtedly encounter cases where SQL is not ANSI 92 compliant. In other words, you have DB specific SQL statements. Maybe it's because of performance reasons. Maybe it's because the difference in handling sequential columns. How can you write, configure, & deploy the application without having specific SQL statements being handled in your Java code?

We encountered this issue on our current project and created an elegant, but simple solution.

Using iBatis for our Java to DB mapping framework, all of our SQL statements were contained within XML files. With the potential requirement to support multiple databases (SQL Server & at least Oracle), we wanted a way to reuse the statements that were compliant across both databases.

Step 1:
Within the MappedStatements XML files, we appended the DB name. For example, for Oracle specific SQL, the name would be 'getSomethingData-Oracle'. If the SQL was DB neutral, we omitted the DB designation. How do we modify the app to determine the appropriate SQL for the database at runtime?

Step 2:
Like good little programmers, we created a BaseDao class from which all concrete DAOs extended. Upon initialization of a DAO, our BaseDao class retrieved the connection meta data and determined the connected DB. Now, we know the runtime DB. But how do we choose the appropriate SQL?

Step 3:
We modified BaseDao to extend org.springframework.orm.ibatis.support.SqlMapClientDaoSupport providing our code the ability to retrieve a MappedStatement from the XML files. This class extension gives our code the ability to check for the existence of a MappedStatement - be it DB specific or non-DB specific.

Step 4:
Finally, within the concrete DAO classes, we called 'checkMappedStatement' when attempting to retrieve any MappedStatement. The BaseDao class handles retrieving the appropriate SQL for the runtime DB - be it specific or generic.

The concrete DAOs would retrieve the SQL from iBatis using the following construct. If DB specific SQL existed within the MappingStatement XML files for the runtime DB, that SQL would be returned.


getSqlMapClientTemplate().queryForList(checkMappedStatement("getSomethingData"));

Below is our simple BaseDao class. Quite simple.


public class BaseDaoiBatis extends SqlMapClientDaoSupport {
 private static final Logger log = Logger.getLogger(BaseDaoiBatis.class);
 private String dbProduct = null;

 protected String checkMappedStatement(String id) {
     MappedStatement ms;
     String statementId = id;

     try {
         ms = ((SqlMapClientImpl) getSqlMapClient()).getMappedStatement(id + "-" + getDbProduct());

         // Look for DB specific SQL. If found, return DB specific mapping id
         if (ms != null) {
             statementId = id + "-" + dbProduct;
         }
     }
     catch (SqlMapException sme) {
        // If not found, use default SQL mapping
        log.debug("DB-specific SQL not found, using default SQL mapping");
     }

     return statementId;
 }

  private void initDbProduct() {
      DatabaseMetaData dbMetaData;
      Connection conn = null;

      try {
          conn = getSqlMapClientTemplate().getDataSource().getConnection();
          dbMetaData = conn.getMetaData();

          log.info("Database product name is '" + dbMetaData.getDatabaseProductName() + "'");

          if (dbMetaData.getDatabaseProductName().indexOf("SQL Server") > 0) {
              dbProduct = "MSSQLServer"
          } else if (dbMetaData.getDatabaseProductName().indexOf("Oracle") > 0) {
              dbProduct = "Oracle"
          } else {
              dbProduct = dbMetaData.getDatabaseProductName();
          }

          log.info("Using " + dbProduct + " XML statements...");
      }
      catch (Exception e) {
          log.error("Exception occurred obtaining database information", e);
      }
      finally {
          if (conn != null) {
              try {
                  conn.close();
              } catch (SQLException e) {
                  log.error("SQLException thrown when closing connection");
              }
          }
      }
  }
}

Good use of maven2 profiles

2007-12-12T17:40:00.000-08:00

As my previous blog eluded to, I work on a project that has to support multiple databases. Having continuous integration servers is key to ensuring the quality of your software across multiple databases. We have successfully supported multiple database almost from the inception of the project. Starting with SQL Server (client mandated) and PostgreSql, we're now moving to Oracle.

Our CI environment will run tests against the multiple DBs as necessary. But how can I run integration tests against the different databases, easily, to ensure my code changes work on supported DBs?

Enter, maven profiles.

In my development environment, I have 3 profiles declared just for this project. The first is declared with 'activeByDefault' set to true. All of the generic or common settings for my project are found in this profile. Samples of the settings include my log4j file location, web application home directories (JBoss in our case), and overall maven settings (tests on/off).

The second and third profiles include information specific database information. One profile runs against SQL server and another against Oracle. Both of these profiles are activated by a specified environment variable on the maven command line.

How does this help with testing? If I have a DB instance for both SQL server & Oracle already set to go, I can simply run maven specifying my environment and have integration tests executed against the DB of my choice.

mvn -Denv=ENV1 install

vs

mvn -Denv=ENV2 install

It doesn't get much simpler than that!

JBoss: Configuring Data Sources

2007-12-07T14:58:00.000-08:00

Configuring data sources in JBoss is quite simple but documentation may be a little difficult to find. Hopefully, this will demonstrate HOW simple it really is.

First, let's say that you have a web application (or some other application) that is expecting a data source to be present in JNDI. Maybe you have designed the application that needs to support multiple databases (kudos to you). Or maybe the application will be deployed into multiple environments and you do not want to force clients to configure the data source your way with your connection information. Whatever the reason, the data source configured within the container and handed to your application.

Next comes the real easy steps. To configure the data source in jboss, you need to perform 2 steps.

Drop the JDBC driver jar file into the JBoss classpath. If you're running the 'default' server under JBoss, drop the jar file in $JBOSS_HOME/server/default/lib. JBoss will pick it when the server starts. Here are 2 links to JDBC driver jars:
- PostgreSql
- SQL Server
- Oracle drivers are located under the Oracle installation directory. For 10g, look under C:\oracle\product\10.2.0\db_1\jdbc\lib.

Create the data source XML file in the deploy directory. Once again, if you're running the 'default' server, create the XML file in $JBOSS_HOME/server/default/deploy. Upon startup, JBoss will load this DS file and configure the data source according to the XML file. Sample XML files for the different DBs are located in the $JBOSS_HOME/docs/examples/jca directory.

Sample Oracle DS


<datasources>
  <local-tx-datasource>
    <jndi-name>COP-DS</jndi-name> 
    <connection-url>jdbc:oracle:thin:@localhost:1521:ORCL</connection-url> 
    <driver-class>oracle.jdbc.driver.OracleDriver</driver-class> 
    <user-name>user</user-name> 
    <password>password</password> 
    <valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker</valid-connection-checker-class-name> 
    <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name> 
    <min-pool-size>2</min-pool-size> 
    <max-pool-size>5</max-pool-size> 
    <metadata>
      <type-mapping>Oracle9i</type-mapping> 
    </metadata>
  </local-tx-datasource>
</datasources>

Sample SQL Server DS


<datasources>
  <local-tx-datasource>
    <jndi-name>COP-DS</jndi-name>
    <connection-url>jdbc:jtds:sqlserver://localhost:1433/cop</connection-url>
    <driver-class>net.sourceforge.jtds.jdbc.Driver</driver-class>
    <user-name>affor</user-name>
    <password>password</password>
    <check-valid-connection-sql>select 1</check-valid-connection-sql>
    <min-pool-size>2</min-pool-size>
    <max-pool-size>10</max-pool-size>

    <metadata>
       <type-mapping>MS SQLSERVER2000</type-mapping>
    </metadata>
  </local-tx-datasource>
</datasources>

BEWARE: The data source file must be name with '-ds.xml'. That's a dash and NOT an underscore. That little character cost me a few hours of troubleshooting only to be identified by another person.

For more detailed information or for other DB configurations, visit the JBoss wiki on data sources.

ESB diagramming

2007-09-29T19:05:00.000-07:00

Bear with me on my first blog!

On my current project, I recently found myself having to diagram the architecture of an ESB to demonstrate how applications, our own included, would connect to existing services as well as expose services for others to consume. The diagrams were for both developers and managers to understand how the ESB was going to be the 'glue' that integrates existing systems and provide the ability to make composite applications by integrating services in ways only previously imagined.

Typically, I use Visio to diagram and was going to start using UML but felt that it just was too low level. Class diagrams would be way too low level giving a view into one application. Deployment diagrams are great but I don't feel they are able to depict an SOA/ESB environment very well, from a high level, showing multiple services across disparate, distributed systems.

After googling, I found another architect's SOA blog regarding ESB & SOA Visio Stencils. After trying the EAI stencils, I registered for Sonic's ESB Icon and Diagram library stencils. Having read David Chappell's book Enterprise Service Bus several years ago, the blog & Visio stencils re-opened some old, locked brain cells reminding me of how well the book and diagrams depict an ESB architecture. The icons are a little big but can be reduced in size (or the entire picture reduced). If you work with ESBs, in an SOA, or plan to, this book should definitely be part of your personal library!

When needing to drill down into the details of an implementation on the ESB, other diagrams may be better suited (i.e. UML, EAI, BPMN, etc).