Java Wrapper does whatever it wants

Yesterday I set out to increase the memory settings for a bunch of our ActiveMQ machines. Most of the time these queues work pretty efficiently and their steady-state memory usage is just a few MB, but with some upcoming maintenance we expect these queues to get considerably fuller, so I was growing them preemptively. Anyway, our ActiveMQ installation gets bootstrapped using Java Service Wrapper and the JVM settings for ActiveMQ are managed in some wrapper.conf file. What I wanted to do was pretty straightforward: I increased the VM’s ‘physical’ memory to 16 GB and intended to increase Java’s MaxHeap from its current setting (1536MB aka 1.5GB) to 8192MB (8 GB). In wrapper.conf, I found the “wrapper.java.maxmemory” setting (which was set to 1536), set it to 8192, and restarted ActiveMQ. Here’s where it got weird.

To verify that I had changed the setting correctly, I checked ps -ef and looked at the process args for Java. It showed that Java was started with -Xmx4096m. Huh? I double-checked wrapper.conf and sure enough I had correctly set it to 8192. My first thought was that maybe I wasn’t running a 64-bit VM, but I confirmed that it’s a 64-bit Sun JVM (the identical version we use on other machines). Just to be completely sure that this was the actual JVM being used by ActiveMQ, I connected via JMX and saw that the running JVM was indeed 1.6.0_14 amd64. I restarted ActiveMQ a few more times and each time it came up with a ceiling of 4096MB. Values below 4096 were set properly.

As a workaround, I tried setting the config like so:

wrapper.java.additional.13=-Xmx8192m
wrapper.java.maxmemory=0

When I attempted to restart ActiveMQ, it looked like it worked, but the process wasn’t running. In the wrapper.log file I saw this:

ERROR  | wrapper  | 2013/08/12 14:28:10 | JVM exited while loading the application.
INFO   | jvm 4    | 2013/08/12 14:28:10 | Invalid maximum heap size: -Xmx=8192m
INFO   | jvm 4    | 2013/08/12 14:28:10 | Could not create the Java virtual machine.
STATUS | wrapper  | 2013/08/12 14:28:14 | Launching a JVM...
ERROR  | wrapper  | 2013/08/12 14:28:14 | JVM exited while loading the application.
INFO   | jvm 5    | 2013/08/12 14:28:14 | Invalid maximum heap size: -Xmx=8192m
INFO   | jvm 5    | 2013/08/12 14:28:14 | Could not create the Java virtual machine.
FATAL  | wrapper  | 2013/08/12 14:28:14 | There were 5 failed launches in a row, each lasting less than 300 seconds.  Giving up.
FATAL  | wrapper  | 2013/08/12 14:28:14 |   There may be a configuration problem: please check the logs.
STATUS | wrapper  | 2013/08/12 14:28:15 | <-- Wrapper Stopped

At this point I confirmed that we were using the 64-bit version of wrapper (it ships with both 32 and 64-bit) and later we even renamed the 32-bit version to ensure it wasn’t being used, but nothing seemed to work. What we found that appeared to work was appending the -Xmx to another argument, like so:

wrapper.java.additional.9=-Djava.util.logging.config.file=logging.properties -XX:PermSize=256m -Xms8192m -Xmx8192m
wrapper.java.maxmemory=0

As I said, this appeared to work. The -Xmx8192m shows up in ps, but when connecting through jvisualvm it shows the Max heap size as 3,685,416,960B:
jvv
So, I don’t have any happy ending to this story right now. Maybe I’ll try not using Wrapper at all. But this is pretty weird so I figured I’d waste some internet real estate talking about it.

64-bit Firefox 6 on CentOS 6 x86_64 (or Fedora) with Flash 11

So I’ve gone back to Linux from Mac, due to the SSD issues I had with my Macbook Pro basically making VMware unusable. A Win7 VM would grind the guest and host to a halt on the 7200 RPM SATA OEM drive, and the SSD wouldn’t work, so I put the SSD back in my HP and installed CentOS 6 x86_64. Not really ideal, but at least it works.

CentOS 6 ships with Firefox 3.6.9, which is really old by now. Fine for a server, but I wanted FF6. I grabbed the Firefox 6.0.1 bz2 from VoxCast, unzipped it and copied it to /usr/lib64/firefox-6. Then I ran yum remove firefox to remove 3.6.9 and avoid any issues. I tried the instructions from my older post on this subject but for whatever reason it didn’t work – I’m guessing because I’m not using the distro’s Firefox RPM.

I followed these instructions to get the Flash 11 64-bit plugin installed. It still wasn’t working though.

I ended up creating a symlink:

ln -s /usr/lib64/mozilla/plugins/libflashplayer.so /home/evan/.mozilla/plugins/libflashplayer.so

When I started up Firefox after creating the symlink, Flash worked.

Additionally, to set Firefox 6 as the default browser, run gnome-default-applications-properties, select Custom, and paste /usr/lib64/firefox-6/firefox %s in the Command: field.

Update: I upgraded my home desktop PC from FC12 to FC14 last night and used the above procedure to install Firefox 6.0.2 on it with Flash 11. So, in case anyone was wondering, the above works for Fedora as well.