Memory leaks hunting articles

I have been busy lately and once again I will only add interesting links. These are about memory leaks hunting in the JVM. Memory leaks can be hard to spot and these articles were published while I was facing the same problems in my company. So thanks to Java’s community, I have been able to quickly identify the potential origins of the leaks.

Read this very interesting article which also has links to other blogs which deal the issues of using ThreadLocal. The Art of war blog

And this article about Java threads stack which can lead, when not properly sized, to “Out of memory exceptions” even with enough free heap space and lost threads.http://www.javablogs.com/Jump.action?id=215672

And this thread about -Xss JVM option and ulimit on Linux http://forum.java.sun.com/thread.jspa?threadID=261344&tstart=0

Weblogic server 9.0 released

I haven’t seen any special announcement neither on theserverside.com (too late a annoucement has been published) nor on javalobby.org but it seems that Weblogic server 9.0 a.k.a “Diablo” has been released. At least it can be downloaded from dev2dev.bea.com since 07/22

I was waiting for it actively because it runs on Java 5, it’s J2EE 1.4 certified and needed to port some Tomcat 5.5 applications on it.

Enable logging for JDBC's thin driver in Weblogic

Here’s a quick and dirty tip to enable JDBC logging for Oracle thin drivers.Enabling JDBC logging for connections of a pool is straightforward with JDBC connection proxies like p6spy tool. Sadly some applications still don’t use connection pools configured on J2EE application server and connect directly to database without retrieving a connection from a pool and with the JDBC URL hardcoded. For the latters, you can still trace JDBC activity.

To do so:

  • 1) Configure Weblogic server, to use the Oracle’s debug thin driver which is called ojdbc14_g.jar and its located in the $WEBLOGIC_HOME/server/ext/jdbc/oracle/ directory. Just add it in the server’s CLASSPATH the path to this library.

  • 2) Put the following class in the server’s CLASSPATH to control the logging level. The default one (2) is very high it creates a huge amount of log. This class takes as first argument the log levelwhich should go from 1 (lower) to 3 (higher)

import oracle.jdbc.driver.OracleLog;
import weblogic.logging.NonCatalogLogger;
/**
Class that sets the Oracle thin driver log level
*/
public class OracleLoggingSetter {
protected final static NonCatalogLogger logger = new NonCatalogLogger("OracleLoggingSetter");
static final int LOWLOGLEVEL=1;
static final int MEDIUMLOGLEVEL=2;
static final int HIGHLOGLEVEL=3;
public static void main(String[] args) {
int loglevel=0;
try {
loglevel=Integer.parseInt(args[0]);
}
catch(NumberFormatException e) {
logger.error("Wrong oracle log level");
return;
}
switch (loglevel) {
case 1:
logger.info("Setting oracle low log level ");
OracleLog.setLogVolume(LOWLOGLEVEL);
break;
case 2:
logger.info("Setting oracle default log level ");
OracleLog.setLogVolume(MEDIUMLOGLEVEL);
break;
case 3:
logger.info("Setting oracle high log level ");
OracleLog.setLogVolume(HIGHLOGLEVEL);
break;
default:
logger.info("log level is incorrect or unspecified, no action performed ");
break;
}
}
}
  • 3) Configure Weblogic to use this startup class and give the log level as an argument (1 is fineand doesn’t fill too quickly).(On the Startup & shutdown node of the adminisration console )

  • 4) Enable JDBC logging for the Weblogic server in the administration console.(Server node -> Logging tab -> JDBC tab )

  • 5) Restart the server

Now you should see the connection string used, the SQL (prepared)statementsand the Oracle’s session attributes in the JDBC log file.Note that enabling logging has a great impact on performance even when the logging levelis low.

Weblogic impossible to load other JMX classes

I have recently discovered that it not possible to use other JMX classes (JMX 1.1 javax.management.*)than the ones in the weblogic.jar on Weblogic 8.1.
Actually, I was trying to extend CruiseControl web app to offer the ability to start/resume/pauseintegration processes in the cruisecontrol daemon. The CruiseControl daemon has its own mx4j JMX agent with an embedded RMI connector (and an HTML adaptor which I didn’t want to use). My webapp connects to the RMI adapator to contact the MBean server of cruisecontrol.

But I had class cast exceptions when run inside Weblogic on the JMX classes. I tried to use the mx4j JMX classes but actually there is no way to do it. I tried to add them in the Weblogic server’s classpath before the weblogic.jar but Weblogic refuses to boot. So I added them in the WEB-INF/lib directory of my web app and used the special option <prefer-web-inf-classes> in the weblogic.xml file but it didn’t work either. This option is supposed to force the Webapp classloader to first load the classes in the WEB-INF subdirectories lib and classes contrary to the hierarchical classloading mecanism. But fatally, it doesn’t work for javax.management.* classes and it’s not documented. The reason given me by the BEA support is that the javax.management.* classes are already used internally even in the context of the web app classloader for monitoring purpose…

Interesting article about Weblogic Entity EJB clustering

See Dmitri Maximovich's Blog. With very interesting articles about:

  • CMP entity Ejbs clustering behaviour regarding 'Optimistic concurrency' strategy
  • Weblogic server v9 new entity Ejb clustering features

If entity Ejbs 2.1 are a “nightmare” to develop and test compared to transparent persistence frameworks of POJOs like Hibernate, some containers like BEA's one offer nice features to build up a R/W distributed cache (of persistent data).

But it's also true that some distributed cache products can also plug into hibernate… see Hibernate's doc and Tangosol

Proxying Weblogic SNMP Agent

Today many middleware and back end enterprise systems embedtheir own SNMP agent (or at least send SNMP traps). SNMP agentsare polled by SNMP managers or enterprise monitoring systems likeTivoli Netview or HP Openview.

Weblogic offers the ablity to proxy the OS SNMP agent and other SNMP agents butgenerally it’s better to use the OS’s SNMP agent asa proxy for all applications SNMP agents hosted on your machine.(Mostly because if the OS SNMP agent fails it’s likely that the machine isdown and all its applications also ;-) whereas if Weblogic SNMP agent fails the OSmight still be alive).

In order to achieve this goal on Linux/Unix, you can use the net-snmp(a.k.a ucd-snmp) to proxy SNMP requests to Weblogic server’s SNMP agent.And it’s pretty straightforward.

The Weblogic domain is composed here of 2 Weblogic servers one administrationserver adminserver and 1 managed server ejbserver on the hostname machine. The Weblogic SNMP agent runs on port1610 of the administration server and is allowed to received requests and send traps for the weblogicpublic community.

Here’s the net-snmp snmpd.conf I used:

syslocation "SunOS hostname 5.8 Generic_108528-16 sun4u sparc SUNW,Sun-Fire-480R"
syscontact "Root "
rocommunity public
rocommunity weblogicpublic
rouser myuser noauth
proxy -v 1 -c weblogicpublic hostname:1610 .1.3.6.1.4.1.140
proxy -Cn adminserver -v 1 -c weblogicpublic@adminserver hostname:1610 .1.3.6.1.4.1.140
proxy -Cn ejbserver -v 1 -c weblogicpublic@ejbserver hostname:1610 .1.3.6.1.4.1.140

Line 4: we authorize the weblogicpublic community to perform SNMP polling requests.

Line 6: by default SNMP requests are forwarded to Weblogic SNMP agent without context for all requests that concern the Weblogic server MIB (starting with .1.3.6.1.4.1.140 OID)

Line 7: a context is configured to address the adminserver Weblogic server

Line 8: same as above for the ejbserverDon’t forget to add the Weblogic‘s MIB in the net-snmp MIB repository and you’re done.

Weblogic and Active Directory authentication

Ok back to more serious Weblogic posts !

Intranet web applications often require to authenticate users with their Windows domain account.Weblogic can be easily configured to authenticate users through ActiveDirectory Server. Weblogic connects to ADS with LDAP (protocol) andtries to bind to the latter with user’s credentials.

To do so, it’s pretty easy,just create in your security realm an “Active Directory Authenticator”.

You need an account which will have sufficient rights to browse the LDAP’s Directory InformationTree of ADS to retrieve users and groups. Also you need to configure ADS to authorize LDAPrequests: bind,read and lookups at least on the branches where users and groups reside (this seems to be enabled by default once connected as domain user).

When a user will connect to your Web application and be asked for login/password, the following steps will be performed by the authenticator:

  • Retrieve the Distinguished Name of the user by performing a LDAP request to ADS.The search’s filter and base DN can be configured in the users tab of the authenticator in the administration console. Also in that tab you can specify theuser’s attribute name with which the filter to get the user DN will be applied.
  • Then Weblogic binds to ADS with the distinguished name and password given.
  • If previous step is successful, then it tries to find out to which groups the userbelongs. For this, the filter defined in the membership tab (when configuring the authenticator)is used on each static group. Then in the “autentication and authorization” process the other security providers ( authorization and role mapper)are invoked.

As you might have noticed, LDAP requests are numerous and it’s a good idea toenable the cache of LDAP lookups. The cache has a TTL but there’s no option to reset the cache in the console. The AD authenticator MBean doesn’t provide a method to reset the cache programmatically.Here’s a sample configuration extracted from the config.xml file I used for a domain called snakeoil.com.Of course, you might have organised your users and groups differentlyand should adapt your base DN search and filter parameters.

<weblogic.security.providers.authentication.activedirectoryauthenticator
ontrolflag="OPTIONAL" credential="{3DES}8KN5usC0Qp0qSVbgQqtHpg==" displayname="SNAKEOIL_ADS"
groupbasedn="OU=SNAKEOIL_GLOBAL_GROUPS,OU=US,DC=snakeoil,DC=com" host="myhost"
name="Security:Name=myrealmSNAKEOIL_ADS" principal="CN=FirstName
LastName,OU=SNAKEOIL_USERS_INTERNALS,OU=SNAKEOIL_USERS,OU=US,DC=snakeoil,DC=com" realm="Security:Name=myrealm"
userbasedn="OU=SNAKEOIL_USERS,OU=US,DC=snakeoil,DC=com"
userfromnamefilter="(&amp;(sAMAccountName=%u)(objectclass=user))"
usernameattribute="sAMAccountName">
</weblogic.security.providers.authentication.activedirectoryauthenticator>

I didn’t change the membership’s default parameters (they were fine for my environment)that’s why they don’t appear here. The SAMAccountName attribute which contains the Windows login Idis used to find the user DN.

NB1:
By default the JAAS flag of the default authenticator is “required” and you should change it to “Sufficient” or “Optional” otherwise your ADS users will be denied access to your application

NB2:
Their might be a way to transparently authenticate the user once it has an open session on Windows but I haven’t investigated on that (maybe with Kerberos):Update 12/7/2004: I am answering to my ignorance: yes it exists it is NTLM protocol linkBut it seems that there’s no easy way to use it with WLS. It must be better to proxy Weblogic with IIS or use SiteMinder.Update 22/02/2005: Since WLS 8.1 sp4 it’s possible to get automatically authenticated with the Kerberos token obtained once logged in the Windows domain. More explanations on edocs.bea.com

Links:

Presentation slides of Weblogic server 9.0

Nice slides of a presentation of Weblogic server 9.0 made duringJavapolis 2004 are available on-line.

I have discovered some new features like deployment planswhich can be useful when you need to support manyconfiguration environments: test, integration, qualification and production without having to unjar and modify the deployment descriptors.

Some new features are purely technical and useless : like the usage of XMLschema for the config.xml or the usage of log4j as the logging framework.

For developers, the integration of Webservices Metadata looks nice.BTW, this feature is more tied to the usage of J2SDK 5.0 metadata.

Many new features on the JMS stack look promising and make the Weblogic JMS server more attractive as a MOM on its own like Websphere MQ or Sonic MQ.See http://www.javapolis.com/ ( you need to register)

BTW, merry christmas !

JRockit rocks on IA64

I did some benchmark on a web application which runs on Weblogic8.1 sp2. Weblogic was installed on a 4 IA64 CPU Linux server .
The benchmark between the Sun JVM 1.4.1 for IA64 and the BEA JRockit providedwith Weblogic showed that the application ran 10x times fasteron JRockit. The ratio was even higher with more than 50 virtual users.

I couldn't even believe it. I thought I had done mistakes trying to tunethe Sun JVM with the -XX options but no the results were the samewithout the -XX options. I haven't figured outwhy the Sun JVM ran so slowly… But if you need to run Weblogicserver on IA64 give JRockit a try, it's really worth it.

To be honest, the Sun JVM 1.4.1 for Itanium 64seems to be a beta one (java -version gives build 1.4.1-b21) and the JRockit JVM is the only one supported on IA64 by BEA. So the comparision was obviously biased. By the way,i'll do some tests with Weblogic 8.1 sp4 with a JVM 1.4.2