Reorganizing photos in 1 line with exiftool

A few years ago I wrote a utility in Java to find all JPG files in a directory and move them into a date-based directory structure like /YYYY/MM/DD/ based on the date the photo was taken, extracted from the exif metadata in the file. Well, apparently that was a huge waste of time, as I just discovered that exiftool, an awesome perl utility I’ve used for years to edit/extract the metadata on the command line, can also do this natively. So my entire program can be replaced with this simple command:

$ exiftool -r '-FileName<CreateDate' -d /targetDir/%Y/%Y-%m/%Y-%m-%d/%Y-%m-%d.%%f.%%e /media/EOS_DIGITAL/

This will copy the files directly off the SD card mounted at /media/EOS_DIGITAL/ into the proper structure in /targetDir/.

Graphing SSH dictionary attacks with HighCharts

After my 10-year-old basement Linux server died this week from a power outage, I took the sad step of giving up on it. It’s died before and I’ve patched it back together with a new power supply here or an addon PCI SATA card there, but I finally decided to throw in the towel since I had a newer old computer that had been idle for several years. The one that died was an Athlon K7 750 MHz with 512 MB ram. The new one is an Athlon 2 GHz (3200+) with 1 gig. For my uses, specs don’t really matter that much, but it’s nice to have more power for free.

I put CentOS 6 on it and configured Samba and copied all the data off the old machine and was back up and running within a few hours. Since I forward ports through my FiOS router to this box I did my standard lockdown procedure, including adding myself to the AllowUsers in sshd_config. Afterwards I took a look in /var/log/secure and saw the typical flood of dictionary attacks trying to get in as root or bob or tfeldman or jweisz. I have iptables configured to rate-limit SSH connections to 2 per 5 seconds per IP so the box doesn’t get DoSed out of existence, but some stuff does make it through to sshd.

Looking through /var/log/secure, I got to thinking it would be interesting if there was some way to visualize the attacks in a handy graph. Then I remembered, oh, wait, I can do that.

I wrote a perl script to parse out the attacks from /var/log/secure and insert them into a Postgres DB. This turned out to be pretty easy. Then I thought it would be more interesting to tie the IP of each attack to its originating country. I’ve used MaxMind’s GeoIP DB pretty extensively before, but I was looking something free. That’s when I remembered that MaxMind has a free GeoIP DB: GeoLiteCity. I grabbed it and yum-installed the Perl lib and added the geo data to the attack DB. Rather than worry about normalizing the schema I just shoved the info into the same table. Life is easier this way, and it’s just a for-fun project.

So I got that all working and parsed it against the existing /var/log/secures via

[root@lunix2011 ~]# zcat /var/log/secure-20111117.gz | perl 

I wrote ssh.php to see what’s in the table:

ssh.php list of hacking attempts
ssh.php list of hacking attempts

So now that the data was all in place, time to move on to the graphs, which is what I really wanted to do. Last time I wanted to graph data programmatically I used JPGraph, which does everything in PHP and is super versatile. But I wanted something… cooler. Maybe something interactive. A little Googling turned up Highcharts which is absolutely awesome, and does everything in JavaScript. I basically modified some of their example charts and pumped my data into them and got the charts below.

Pie chart of attacks grouped by country for the past 30 days:

Pie chart by country
Pie chart by country

Bar graph of attacks per day:

Bar graph of daily attacks
Bar graph of daily attacks

So, that’s that. Code is in github if anyone wants to play around with it. I’ve cronned to run every 5 minutes so the data gets updated automatically.

Displaying currently-playing iTunes track in the Mac menu bar

In an attempt to teach myself Objective C, and because I couldn’t find anything that did what I wanted, I wrote a little utility to display the currently-playing iTunes track in the Mac taskbar. Originally I had it display the full track name right in the taskbar but it was too much text for such a small space (especially on a 1440×900 screen), so now you click a little musical note and it shows you the info in a menu.

Here’s a screenshot:

The code is all in github. If you’re looking for a similar utility, and are brave enough to try my first-ever Obj-C app, you can download it here. But the freshest version will probably be in the github project.

As an aside, I was surprised at how easy it was to cobble this together having never written ObjC before. I found some good examples that I mostly ripped off, but it was still remarkably easy to have the app listen to iTunes for track changes, etc.

Benchmarking DNS servers with Java

I’m currently in the process of moving our DNS over to another provider and I was curious as to whether the old or new provider offers faster lookups. dig shows query times, but I didn’t want to just run that over and over. I decided to write something to do this, in Java since I like Java. I found this post, which has the meat of the work done already. I also read some of Sun’s JNDI/DNS lookup info, which was pretty dense. All I want to do is specify the name server’s IP and do the lookup. I don’t even really care about the result, just how long the query takes.

The thing I wrote only looks up A records, but can easily be modified to do CNAMEs or whatever. Here’s how you call it:

$ java -jar DNSTester.jar 25
Resolved to against NS
Performed 25 lookups in 233.29 milliseconds.  Average 9.3316ms per lookup.

$ java -jar DNSTester.jar 25
Resolved to against NS
Performed 25 lookups in 450.034 milliseconds.  Average 18.00136ms per lookup.

Code is in github here. Jar is available here.

MongoDB logrotate script

MongoDB has log rotation functionality built in, but since I run CentOS I like to have everything managed through logrotate. The 10gen RPM I installed doesn’t have a logrotate script, so I wrote one. Create file /etc/logrotate.d/mongod:

/var/log/mongo/mongod.log {
        rotate 30

        /bin/kill -SIGUSR1 `cat /var/lib/mongo/mongod.lock 2> /dev/null` 2> /dev/null || true

logrotate runs at 4 AM daily by default (via the script in /etc/cron.daily/logrotate). The file above rotates the mongod.log file daily, retaining 30 days of files, appending the date to each one after rotation (rather than the “.1” or “.2” suffix) and then gzipping it.

Teaching myself node.js: Part 3

I’ve been playing around with node, JavaScript & MongoDB a bit with this project and one of the things I ran into was MongoDB’s lack of a sequential ID. MongoDB defaults to ObjectIDs for primary keys in collections, and they have good reasons for doing so, but in writing this app, I’d rather have URLs look like “/question/423” than “/question/3001024e521e9c6500000000”.

Fortunately, there’s a relatively convenient workaround for this problem, detailed here. Essentially, create a collection in which each document is a counter. The ID for the document is its “name” and the property “next” contains the next value. This is similar to creating and using a sequence in PostgreSQL, though hopefully in the future they’ll provide an easier way to do this – perhaps a built-in sequence object.

First, in the mongo shell, I created the counter(name) function as specified in the Mongo wiki:

> counter
function counter(name) {
    var ret = db.counters.findAndModify({query:{_id:name}, update:{$inc:{next:1}}, 'new':true, upsert:true});

Then I created the counter document for the “questions” collection (so questions can have numeric IDs):

db.counters.insert({_id:"questions", next: 1});

This is optional – the counter() function will create the document in the counters collection if needed, with an initial value of 1.

That’s it. Each call to counter() will now return an incrementing number:

> db.counters.find();
{ "_id" : "questions", "next" : 3 }
> counter("joe");
> db.counters.find();
{ "_id" : "questions", "next" : 3 }
{ "_id" : "joe", "next" : 1 }
> counter("joe");
> counter("joe");

Cool. So how do we save the counter() function in the DB so we can use it in queries without having to define it every time? Fortunately Mongo makes saving functions on the server easy:

> { _id : "counter" , value : counter });
> db.system.js.find();
{ "_id" : "counter", "value" : function cf__2__f_counter(name) {
    var ret = db.counters.findAndModify({query:{_id:name}, update:{$inc:{next:1}}, 'new':true, upsert:true});
} }

Calling the stored procedure directly in the shell is sort of strange, but it works:

> db.eval("return counter('question')");
> db.eval("return counter('question')");
> bye
[Wed Jun 22 13:14:10 evan@EvanMBP 3 ~]$ ~/Downloads/mongodb-osx-x86_64-1.8.1/bin/mongo questionsMongoDB shell version: 1.8.1
connecting to: questions
> db.eval("return counter('question')");

Unfortunately, I was unable to figure out how to call the stored function from within the mongoskin driver, and ended up writing analogous code, calling findAndModify() within the app code. I finally got it:

                {$inc : {next: 1}},
                function(err, counter) {
                        if (err) { throw new Error(err); }
                        var ins = { date: new Date(),
                                author:, body: req.body.body,
                                tags: tags, tag_count: tagCount,
                                answers: [], votes: 0,
                        db.collection('questions').insert(ins, {});
                        res.end('Added new question: '+req.body.body);

Question detail
Question detail

I tried throwing some code in there to query by either ObjectId() or integer but it didn’t really work (Edit: I tried this again on Linux and it worked fine), so I just deleted all the documents with ObjectIds and everything seems to work fine now. I think the next step is going to have to be some sort of authentication/session stuff, because typing your name in every time kind of sucks (and makes it hard to do things like list questions by user).

Full code for this revision is here.

JavaScript regex for stripping leading & trailing whitespaces

I’m sure there are a bunch of libraries that do this, but sometimes wrestling with regexes is fun.

> var str = "           This string has some mighty fine whitespace.       ";
> str
'           This string has some mighty fine whitespace.       '
> str.replace(/^(\s*)((\S+\s*?)*)(\s*)$/,"$2");
'This string has some mighty fine whitespace.'

Teaching myself node.js: Part 2

I did a little more work today on my little node.js project and added the ability to click on a tag and have the app list all questions with that tag, and a super basic method for adding new questions. Neither of these were really complicated; the most “challenging” part was figuring out how to create links and form inputs with jade. In the end I was surprised at how easy it really was.

For the tag search I just copied the app.get('/questions') route to a new app.get('/tags/:tag') route in app.js and modified the mongo find() query to search for the tag. I created a new Jade template for this, but I suppose the same list.jade template would have worked, since the only thing that was changed was the contents of the questions array.

For adding a new question I created an app.get('/question/new') route that loaded a jade template with the form (addquestion.jade), and which did a POST back to /question and the corresponding'/question'). Right now the form just takes text input fields, but eventually there’ll need to be some notion of users, and other fun functionality. Also after doing the insert into Mongo, it just spits back some plaintext response, but it gets the job done.

I should add that this is my first time writing a webapp using “routes.” It’s been a while since I really did any front-end coding at all and I’m used to the “old school” method of one script per function, where the “add a new question” function would be handled by “addQuestion.php” for instance. Using routes and MVC are part of what I’m trying to learn in doing this.

Commit for this version is here

Teaching myself node.js: Part 1

I’ve been meaning for a while to learn node.js. I read a couple of books and plenty of blog posts but in the end I find, as usual, I can only learn by doing. I have most of the fundamentals to get started, I just need to have a project. The typical tutorials I’ve found focus on creating a blog; I’m thinking of something more along the lines of a Quora question/answer site. I’ll be trying to write this as I go, which will hopefully prod me into actually doing it this time. (I have a tendency to bounce around different tenses and I expect this will be worse than usual – stuff will likely be written in past/present/future at various points).
Continue reading “Teaching myself node.js: Part 1”