Java Wrapper does whatever it wants

Yesterday I set out to increase the memory settings for a bunch of our ActiveMQ machines. Most of the time these queues work pretty efficiently and their steady-state memory usage is just a few MB, but with some upcoming maintenance we expect these queues to get considerably fuller, so I was growing them preemptively. Anyway, our ActiveMQ installation gets bootstrapped using Java Service Wrapper and the JVM settings for ActiveMQ are managed in some wrapper.conf file. What I wanted to do was pretty straightforward: I increased the VM’s ‘physical’ memory to 16 GB and intended to increase Java’s MaxHeap from its current setting (1536MB aka 1.5GB) to 8192MB (8 GB). In wrapper.conf, I found the “wrapper.java.maxmemory” setting (which was set to 1536), set it to 8192, and restarted ActiveMQ. Here’s where it got weird.

To verify that I had changed the setting correctly, I checked ps -ef and looked at the process args for Java. It showed that Java was started with -Xmx4096m. Huh? I double-checked wrapper.conf and sure enough I had correctly set it to 8192. My first thought was that maybe I wasn’t running a 64-bit VM, but I confirmed that it’s a 64-bit Sun JVM (the identical version we use on other machines). Just to be completely sure that this was the actual JVM being used by ActiveMQ, I connected via JMX and saw that the running JVM was indeed 1.6.0_14 amd64. I restarted ActiveMQ a few more times and each time it came up with a ceiling of 4096MB. Values below 4096 were set properly.

As a workaround, I tried setting the config like so:

wrapper.java.additional.13=-Xmx8192m
wrapper.java.maxmemory=0

When I attempted to restart ActiveMQ, it looked like it worked, but the process wasn’t running. In the wrapper.log file I saw this:

ERROR  | wrapper  | 2013/08/12 14:28:10 | JVM exited while loading the application.
INFO   | jvm 4    | 2013/08/12 14:28:10 | Invalid maximum heap size: -Xmx=8192m
INFO   | jvm 4    | 2013/08/12 14:28:10 | Could not create the Java virtual machine.
STATUS | wrapper  | 2013/08/12 14:28:14 | Launching a JVM...
ERROR  | wrapper  | 2013/08/12 14:28:14 | JVM exited while loading the application.
INFO   | jvm 5    | 2013/08/12 14:28:14 | Invalid maximum heap size: -Xmx=8192m
INFO   | jvm 5    | 2013/08/12 14:28:14 | Could not create the Java virtual machine.
FATAL  | wrapper  | 2013/08/12 14:28:14 | There were 5 failed launches in a row, each lasting less than 300 seconds.  Giving up.
FATAL  | wrapper  | 2013/08/12 14:28:14 |   There may be a configuration problem: please check the logs.
STATUS | wrapper  | 2013/08/12 14:28:15 | <-- Wrapper Stopped

At this point I confirmed that we were using the 64-bit version of wrapper (it ships with both 32 and 64-bit) and later we even renamed the 32-bit version to ensure it wasn’t being used, but nothing seemed to work. What we found that appeared to work was appending the -Xmx to another argument, like so:

wrapper.java.additional.9=-Djava.util.logging.config.file=logging.properties -XX:PermSize=256m -Xms8192m -Xmx8192m
wrapper.java.maxmemory=0

As I said, this appeared to work. The -Xmx8192m shows up in ps, but when connecting through jvisualvm it shows the Max heap size as 3,685,416,960B:

So, I don’t have any happy ending to this story right now. Maybe I’ll try not using Wrapper at all. But this is pretty weird so I figured I’d waste some internet real estate talking about it.