Java Wrapper does whatever it wants

Yesterday I set out to increase the memory settings for a bunch of our ActiveMQ machines. Most of the time these queues work pretty efficiently and their steady-state memory usage is just a few MB, but with some upcoming maintenance we expect these queues to get considerably fuller, so I was growing them preemptively. Anyway, our ActiveMQ installation gets bootstrapped using Java Service Wrapper and the JVM settings for ActiveMQ are managed in some wrapper.conf file. What I wanted to do was pretty straightforward: I increased the VM’s ‘physical’ memory to 16 GB and intended to increase Java’s MaxHeap from its current setting (1536MB aka 1.5GB) to 8192MB (8 GB). In wrapper.conf, I found the “wrapper.java.maxmemory” setting (which was set to 1536), set it to 8192, and restarted ActiveMQ. Here’s where it got weird.

To verify that I had changed the setting correctly, I checked ps -ef and looked at the process args for Java. It showed that Java was started with -Xmx4096m. Huh? I double-checked wrapper.conf and sure enough I had correctly set it to 8192. My first thought was that maybe I wasn’t running a 64-bit VM, but I confirmed that it’s a 64-bit Sun JVM (the identical version we use on other machines). Just to be completely sure that this was the actual JVM being used by ActiveMQ, I connected via JMX and saw that the running JVM was indeed 1.6.0_14 amd64. I restarted ActiveMQ a few more times and each time it came up with a ceiling of 4096MB. Values below 4096 were set properly.

As a workaround, I tried setting the config like so:

wrapper.java.additional.13=-Xmx8192m
wrapper.java.maxmemory=0

When I attempted to restart ActiveMQ, it looked like it worked, but the process wasn’t running. In the wrapper.log file I saw this:

ERROR  | wrapper  | 2013/08/12 14:28:10 | JVM exited while loading the application.
INFO   | jvm 4    | 2013/08/12 14:28:10 | Invalid maximum heap size: -Xmx=8192m
INFO   | jvm 4    | 2013/08/12 14:28:10 | Could not create the Java virtual machine.
STATUS | wrapper  | 2013/08/12 14:28:14 | Launching a JVM...
ERROR  | wrapper  | 2013/08/12 14:28:14 | JVM exited while loading the application.
INFO   | jvm 5    | 2013/08/12 14:28:14 | Invalid maximum heap size: -Xmx=8192m
INFO   | jvm 5    | 2013/08/12 14:28:14 | Could not create the Java virtual machine.
FATAL  | wrapper  | 2013/08/12 14:28:14 | There were 5 failed launches in a row, each lasting less than 300 seconds.  Giving up.
FATAL  | wrapper  | 2013/08/12 14:28:14 |   There may be a configuration problem: please check the logs.
STATUS | wrapper  | 2013/08/12 14:28:15 | <-- Wrapper Stopped

At this point I confirmed that we were using the 64-bit version of wrapper (it ships with both 32 and 64-bit) and later we even renamed the 32-bit version to ensure it wasn’t being used, but nothing seemed to work. What we found that appeared to work was appending the -Xmx to another argument, like so:

wrapper.java.additional.9=-Djava.util.logging.config.file=logging.properties -XX:PermSize=256m -Xms8192m -Xmx8192m
wrapper.java.maxmemory=0

As I said, this appeared to work. The -Xmx8192m shows up in ps, but when connecting through jvisualvm it shows the Max heap size as 3,685,416,960B:
jvv
So, I don’t have any happy ending to this story right now. Maybe I’ll try not using Wrapper at all. But this is pretty weird so I figured I’d waste some internet real estate talking about it.

Advertisements

5 Replies to “Java Wrapper does whatever it wants”

  1. We are seeing the exact same issue. It seems like ActiveMQ does in fact accept the -Xmx memory setting in wrapper.conf so long as you specify a value less than 4096. As soon as you increase the value above 4096, it overrides the specified value and defaults to 4096. I’m not sure whether this is intended behaviour or is some logic that is failing due to a bug caused by 32-bit calculations instead of 64-bit (we too have the correct 64-bit ActiveMQ and Java installations). We are using ActiveMQ 5.5.0 and Java 1.6.0_24-b07 (64-bit).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s