Month: September 2013

Posted on

iOS 7 has ruined my life

I was looking forward to iOS 7 for a few reasons. I was glad to be rid of the skeumorphic design elements (stupid shit like the “Notes” app looking like a yellow notepad) but also glad to get the Control Center to turn off Wifi more easily – I only need Wifi to be on when I’m in my house or my office. It’s just a battery drain elsewhere. Ideally iPhone would just do this via geofencing, and the fact that it doesn’t is beyond retarded.

So I upgraded the night it was released. Two days later the novelty had worn off. I tried to roll back to iOS 5 (I never upgraded to 6 due to Maps). Failed with error 3194. I tried DFU mode. Failed with error 3194. I was stuck. They released 7.0.2 this past Friday or Saturday night and I hoped it would solve my problems but it didn’t. Here’s some of what’s happened:

  • Battery life is gone. I lose about 1% per minute at times, even with just 1-2 apps running and the phone in airplane mode. To go from 100% to 75% usually takes about 90 minutes.
  • Phone doesn’t show up on computer at all – not in iTunes, not as a camera, nothing. I’ve reinstalled iTunes several times.
  • It doesn’t check email any more. It tells me no new messages even when I login to Exchange and see new messages. Same for Gmail.
  • Audio is broken. The hardware toggle switch appears to be ignored randomly – it won’t make sound at times, and then other times makes tons of sound. On the train, when I unplugged my headphones while listening to a song, it kept playing the audio out the loudspeaker and the volume buttons were ignored.
  • The phone randomly powers off when it gets under 30% battery life. The behavior is what I used to see when it got around 3%, but now it happens at 30% or so. When I turn it back on, it says there’s ~25% left so I don’t know why it shut down.
  • iCloud backup does not work. It gets to “1% remaining” or something and then says “iCloud backup failed.” The last successful one was from the day I last ran iOS 5.

As of this moment, as I write this, my phone is completely wiped and I’m attempting to restore it from backup. But the phone won’t show up in iTunes for me to restore it. Last time
I went to the Apple Store in Grand Central the “genius” was useless, told me my phone was fine even though it was barely getting 8 hours battery life with no use – his advice was to power the phone off at night. It’s a 4S so not sure what recourse I have, it’s definitely not under warranty anymore. I have work tomorrow and as of right now a useless phone.

I guess my point is, don’t upgrade to iOS 7 if you have an iPhone 4S.

Update, Oct 21, 2013. I took my phone back to the Apple Store and they essentially confirmed that my phone was broken somehow – it wouldn’t even connect to their Macbooks, though it would charge. They offered me a new 4S for $199 (no contract extension) but I didn’t want to do that. Last week I took my phone to Best Buy and got $100 for it towards a new 5S. The 5S is great and IOS7 works fine on it, though it does still have quirks and bugs.

Posted on

Goodbye, pg_dump

I’ve been a Postgres user and administrator for a while. Over the years, my views on backups have evolved.

Originally, like most people, I started out with good old pg_dump. With a reasonably small database (under 50 GB) dumping to a flat text file is a fine option. I’d generally do something like pg_dump -Upostgres dbname | gzip > dbname.sql.gz to compress it on the fly and save space. For years this seemed perfect: dumping the entire database in a single transaction into a single file that can be restored anywhere.

But as my databases started growing larger and larger, the time it took to do a pg_dump grew as well. At a previous job, the database grew to nearly 2TB and the pg_dump took nearly 18 hours. We’d by that point already changed the pg_dump schedule from daily to weekly and then to three times a month and then finally to semi-monthly. Not only was it slow, but since it operated in a single transaction it wreaked havoc with normal database operation for queries that needed locks on tables locked by the dump.

When we moved the database from a physical RAID to a volume on our SAN, that gave us the opportunity to use LUN snapshotting rather than pg_dump (I just remembered I already wrote about that here). This let us move to a monthly pg_dump and more frequent snapshot-level backups that took up very little space. This was ideal on Compellent since the snapshots would auto-expire after however long you specified.

When I started at Yodle we were doing nightly pg_dumps and pretty soon we ran into the same problems I’d seen at Didit with the dump itself interfering with normal DB operation – the dump would start at midnight and run until 7-8 AM when I started, and after a few months it would still be running at noon. We discussed moving to wal archiving and making a basebackup to NFS but that would require a pretty massive amount of space, and as anybody who uses “enterprise storage” knows, that’s not something you want to do. We discussed building a whitebox file server for backups but nobody was really in love with that option – we’re trying to reduce the reliance on physical machines as much as possible. We talked about pushing it all to S3 but that seemed rather difficult.

When I attended NYC PgDay earlier this year, there was lots of discussion about WAL-E. I hadn’t ever head of WAL-E so I looked it up and was impressed. Basically, WAL-E handles archiving of wal to S3, but first compresses and pgp-encrypts it. It also handles pushing the basebackup to S3, also compressed and pgp-encrypted. This was just what we were looking for. We set it up and, amazingly, it worked perfectly. After a few weeks (and confirming we can restore from the wal-e backups) we moved our pg_dump to weekly, on the weekend when it doesn’t interfere with any user processes. We do a wal-e basebackup every 3-4 days or so and retain 3 of them. We retain all the wal so we can restore the DB to any point within the last ~10 days if needed. The best part is it’s faster than pg_dump, and since the basebackup doesn’t operate in a transaction (it’s a filesystem-level backup rather than an application-level backup) it doesn’t mess with user queries. There’s of course elevated IO during this time but our SAN has more than enough bandwidth.

We setup some basic monitoring of S3 (check the age of the most recent WAL and log it in Zabbix) just to ensure the backups are actually happening, and we’re at the point where we’re discussing moving pg_dump to monthly, or simply not doing it at all. Overall, wal-e has been a huge win for us, enabling better, faster backups that don’t interfere with the DB itself, and, while not free, aren’t ridiculously expensive. And since it’s in its own S3 bucket, you can tweak the bucket settings (e.g. enable RRS) to save money, and Amazon tells you exactly how much your backups cost you.