Conversations about life & privacy in the digital age

Ubuntu/Debian APT repository GPG key update

Hello Debian friends!

The GPG key for our Ubuntu/Debian APT repository expires today. We’ve created a new key that you can get here: https://spideroak.com/dist/spideroak-apt-2013.asc. We will have new builds shortly that include the new key.

The new key looks like this:

pub   1024D/08C15DD0 2013-09-20 [expires: 2016-09-19]
      Key fingerprint = FE45 E533 0B11 DCF0 3247  EF49 A6FF 22FF 08C1 5DD0
uid                  SpiderOak Apt Repository 

UPDATE: New .deb packages are available on the Download Page. This will automatically update your apt keys and ensure you continue to get updates.

Advanced users can install the new key with this command:

curl https://spideroak.com/dist/spideroak-apt-2013.asc | sudo apt-key add -

After installing the new key, update your package manager and you’ll be able to upgrade to future versions of SpiderOak without issue.

Securing Your Mail From Site to Site

Many of you know how to secure your email between your mail client and your computer. But if you run your own mail server, did you know you can secure email between servers? Many servers support TLS encryption for outgoing connections, which will protect your mail between your server and the next one. For my favorite mail server, Postfix, add this to your main.cf:

smtp_tls_security_level = may

This will enable “opportunistic” TLS for outbound connections, meaning it will use encryption if the remote server supports it, otherwise it will transmit it unencrypted. If you’re really paranoid and don’t want to talk to servers that don’t support encryption, you can change may to verify or secure to ensure that the remote end uses encryption.

To ensure that your server listens for TLS requests, add this:

smtpd_tls_security_level = may
smtpd_tls_cert_file = ...
smtpd_tls_key_file = ...

Note the small difference between smtp_... and smtpd_. The cert and key parameters configure your SSL certificate. You can also use encrypt here instead of may to force encryption for clients, but this isn’t recommended for a public Internet server.

By default, if Exim is compiled with TLS support, it will attempt TLS for outbound connections. If you want it to accept TLS, though, you’ll have to set:

tls_advertise_hosts = *
tls_certificate = ...
tls_privatekey = ...

It’s important to note that even with these configurations, you can’t guarantee that your mail is completely encrypted in transit, since your mail could be transmitted between several servers. It also doesn’t prevent eavesdropping on the servers themselves. If you want to ensure that only the recipient can read your mail, you should use something like PGP.

I’ll leave other mail servers as an exercise to the reader. Feel free to post further configuration or notes in the comments!

Just Because It’s Encrypted Doesn’t Mean It’s Private

Now that you’ve got a handle on what encryption is and what it can do, it’s important to understand what it can’t do.

Encryption is a tool, and like any tool, it can be used improperly or ineffectively. It may sound a bit strange for us at SpiderOak to disclaim the benefits of encryption, but I hope to show that while encryption is necessary for privacy, it’s not always sufficient.

One prime example of the utility of encryption is HTTPS. By wrapping encryption around regular HTTP, engineers have created a powerful tool for securing content both delivered to you and provided by you. But HTTPS only protects content while it’s in transit. HTTPS will protect your credit card numbers as they travel over the internet to a merchant, but once they arrive on the other end, they’re no longer encrypted and it’s up to the merchant and credit card providers to protect them. Credit card providers and banks have developed PCI DSS regulations to tightly specify the security of credit card processing, but as the frequency of credit card breaches demonstrates, these regulations aren’t sufficient to guarantee privacy.

Another great cryptographic tool is Full Disk Encryption. Whether built-in to your computer hardware or provided by software like TrueCrypt, FDE protects the contents of your hard drive by encrypting every last bit. Anyone who steals your hard drive will find it completely unreadable. But while you’re using the drive, it is readable. While you have your computer on and the drive unlocked, any malicious piece of software running on your computer will find all of your data fully readable. FDE is a valuable tool, but it can only guarantee privacy while the disk isn’t in use.

Privacy is a complex problem that requires attention to many details, one of which is encryption. We’ve tried our best to provide you with the best privacy possible for your important data. If you’re interested in more details about how we protect your privacy, please read our Engineering page. And feel free to ask us about it, we’re always willing to brag!

SysAdmin Day Follow Up

Hello terminal hackers!

We had a really great turnout for the promo last week. Over 600 of you successfully plumbed the depths of SOTSS and claimed 5 extra bonus gigs. Some of you found the bonus gigs program but got the DEVICE BUSY error. That was a bit of rate limiting we put in to prevent abuse. If you never got bonusgigs to work, please mail support@spideroak.com and we’ll hook you up.

For the rest of you who found SOTSS a little daunting, don’t worry. We’re sorry that you felt left out, and we have plenty of prizes left in our promo bag!

Here’s the solution for those of you who who were wondering:

There are two ways you can get started off. The more obvious way is that you can load help and then run help to view the help. From there you can manually load the various programs listed, including the backup program. The less obvious way is to load autoboot.bak, which will load all the items in the help automatically.

Then you can run backup to dump all the programs to the terminal. Inspection of this dump reveals a bonusgigs program. Load this program and run bonusgigs username, where username is your SpiderOak username. That’s it.

Thanks for playing!

Happy System Administrator Appreciation Day!

EDIT: The day is over, and we have shut down the bonus gigs system. Thanks to everyone who participated. We’ll be keeping SOTSS active for a while longer, so you can still tell all your friends about it!

Hello, fellow SysAdmins (and other assorted peons)!

Today is our day! Enjoy it. You earned it.

And to show our excitement and appreciation for all that you do, we’ve created a special kind of bonus for you. It’s a bit of a brain-teaser, and a bit of a retro homage to the bad old days of computing, when a remote terminal was a VT100, memory was measured in kilobytes, and your only hope of understanding a problem was a shelf full of manuals.

Welcome to the SpiderOak Time Stealing System.

Your goal is simple. SOTSS is broken, and you have to get it working. Buried inside the system is a program that will grant you an extra 5GB to your SpiderOak account. Run that program, and the 5GB is yours. That’s it. You’ll need to dig around to figure it out; there’s no manual and the online help is broken. Get creative.

Since this puzzle is for our SysAdmin friends, we’d rather you didn’t post your solutions.

Oh, and have fun. :)

New Expansion Ahoy!

If you’ve been using SpiderOak lately, you might have noticed that it’s been a bit… sluggish. A small portion of our users have been impacted by our recent expansion efforts. We do greatly apologize for the inconvenience. The good news is that we have been working diligently at installing new servers to increase capacity. In a few days, these slowdowns will be a thing of the past!

So please hang in there, we’ll have you right as rain before you know it! :)

The Garfleburgers of Spalagnatz IV

“Do you remember the garfleburgers of Spalagnatz IV? No, I guess you probably don’t. Even nowadays putting a brain in a tin can is a tricky procedure, and it was downright dangerous when you were transcribed. I’m sure a lot of good memories got lost then, but it’s not like we had much of a choice. But don’t worry, I’m here to help recover some of those memories.

“Anyway, the garfleburgers… You begged mom to let you go to Spalagnatz IV for spring break, and she would only let you go if I came along. I wasn’t too happy about having to play chaperone, but I was glad to get out of the house. So that July you and I hopped a spaceliner to Spalagnatz IV where we found the garfleburgers… Well, I should back up a bit.

“Spalagnatz IV was a natural paradise planet in the Hyperion cluster. It orbits a yellow star much like the sun, but it burned a little brighter, so the native plants were blue-green. Humans immediately saw its large tropical zone as an opportunity and turned it into the biggest beach party in the universe.

“After checking in at the hotel, you met this guy, his name was Venutias. Half Beloran, I think. He was also on spring break, and had been to Spalagnatz IV a couple of times before, so he offered to show you around. I, of course, didn’t trust him one bit, so I made a point of it by tagging along.

“We went to the beach which was completely crowded despite there being several thousand miles of it. Their sand is pink… something to do with extra iron in the soil or something. We swam and had a good time. Turns out Venutias was an alright guy. Some stellahead jock tripped over our towel and then started giving you crap for it. Venutias told him in no uncertain terms to get lost, and the jock left without a fight.

“Anyway, we had dinner at this little burger shack he knew about which had the ‘best garfleburgers on the planet.’ A garfle was apparently a native animal. Seemed kind of touristy. Their idea of a hamburger was pretty strange, too. Burger, chili sauce, onion, green pepper, marshmallow creme, and cream cheese in a small pie shell. Frickin’ weird. It was alright, though. You could barely taste the garfle over all the toppings, but I guess that was the point. I looked it up the other day. A garfle is a small scavenger animal, like a squirrel. Apparently they run rampant all over Spalagnatz IV, and selling garfleburgers to tourists is just one of the ways they keep the population under control.

“By the time we left, you definitely had a crush on Venutias. But like most interstellar romances, it was just not meant to happen. We went back to Earth and he went home to Balora, and we never saw him again. I suppose it was for the best. After you died and they translated your mind to digital, there wasn’t much left for you in the physical world. Maybe some day I’ll get to see what it’s like in there.

“Anyway, I have to get back to work. I’ll see you later, sis. I love you.”

More fun with SSL certificate verification failures

Some of you who tried to access spideroak.com a couple hours ago may have noticed a security warning from your browser complaining about an invalid certificate.

Whoops.

No, we didn’t forget to change the storage certificates again. In fact, the new certificate was purchased back in April.

Turns out there was some fun to be had with our new SSL certificate. (SSL is the mechanism that browsers use to encrypt your connection to the server, giving you the nice padlock icon so that you know websites like SpiderOak.com are secure.)

Geotrust changed their certificate roots due to some weaknesses in the old one, which meant that there was not only a new root, but also a new intermediate RapidSSL certificate thrown in for good measure. (The root is the certificate that browsers use to verify that all certificates are genuine. The intermediate certificate establishes a chain of certificates from the root to the certificate used by an individual website.)

This took me a few minutes to figure out, but once I got the extra intermediate certificate thrown in there, the website was happy.

Unfortunately there was another problem: the SpiderOak client didn’t know about the new certificate root. This would have affected anyone who was trying to complete their first signup or create a new device in the SpiderOak client.

The core of the problem is that by default, Python, the language that SpiderOak is mostly written in, does not verify SSL certificates at all, so we were forced to roll our own verification routines. We whipped up our own system that simply packaged the certificates in the client itself, which was better anyway because it didn’t rely on sometimes broken external SSL certificate chains. Today’s problem is the obvious downside. Our developers responded quickly and pushed out new builds with the updated certificate in about an hour.

So if you’ve had problems signing up, we’re sorry. We screwed up. Please download the latest version and try it again. I’ll be over here taking my due flogging.

TL;DR: All your existing backups, syncs, devices, shares, and everything else are fine. The next time you add a new device to your SpiderOak account, you’ll need to download the latest version of SpiderOak.

Update: If you tried to sign up during this time, you should be receiving an email from us shortly, along with an extra gigabyte of free storage to show our appreciation for your patience.

Update 2: It turns out that some older Android phones (older than Android 2.3) don’t include the newer CA roots! (Although, the original iPhone from 2007 does have those roots included via OS updates, and some Android vendors seem to include them also, so it is somewhat unpredictable whether a given phone has them.)

So, we’ve had to add an intermediate certificate to spideroak.com for older Android compatibility. We’ve published the desktop client revision 9830 which also recognizes this additional certificate. Once again, all existing devices, backups, syncs, etc. are fine. You’ll need the newest SpiderOak the next time you add a new device to your account (which is generally the best practice anyway.) -Alan

Wibbly Wobbly Timey Wimey, or How to Gracefully Compensate for an Incorrectly Set Clock

Part of being a good sysadmin is being flexible. Quite often when something breaks you’re solving a new problem, and the developers, users, and management turn to you to find a solution. One day you’ll be sorting out IP misconfiguration, the other you’ll be debugging Python libraries, and another you’ll be working around yet another limitation of your web server. You wind up reading a lot of documentation and learning a little about everything.

We had a particularly puzzling problem on one of our servers the other day. Somehow, its clock had been set eight hours fast1. We were running NTP, but it wasn’t set to jump the clock on start2, so even though NTP was synced to an upstream server, the clock was still a long ways off.

Normally, this is pretty easy to fix — you set the clock back, and you’re done. For us, though, it was a serious problem because uploads are ordered by commit time, which is dependent on the server’s clock. The guy who wrote our nagios time check was summarily flogged, and Alan and I set out to find a solution. We came up with a few ideas:

  1. Wait eight hours for real time to catch up
  2. Rewrite all the logs and database entries to have correct times
  3. Find a way to slow down what our daemons thought was the correct time in order to gradually bring their time in sync with real time

#1 was right out — we weren’t going to stop uploads for that long. #2 was heinously difficult and fraught with peril, so we went with #3. Alan’s quick googling found libfaketime, one of those great utilities that shouldn’t exist, but does. Libfaketime works as a LD_PRELOAD hack to change the way certain time library calls work. It can create a time offset and/or speed up and slow down the passage of time relative to the system clock3.

After two hours pondering and hacking, we put our plan into action. We set our server’s time back to normal and modified our daemons’ time to pretend to be six hours in the future, running at half-speed. Twelve hours later, the times synced up, and we undid the hack. Problem solved, and I’ve added one more unmentionable hack to my toolbox. :)

1: I’m not entirely sure, but I think the reason for this is a recent motherboard swap. We set our system clocks to UTC, and the motherboard probably arrived with its clock set to Pacific time.

2: Debian/Ubuntu admins note: this option is turned off by default in the openntpd package.

3: Interestingly, Erlang provides similar behavior by default. If erl detects a jump in the system clock, it will slowly adjust its concept of current time to match.

FBI Wants Your SpiderOak Data; North Korean Hackers Steeple Fingers in Anticipation

The FBI is again looking to circumvent cryptography to expand their wiretapping capabilities. They want to require that all service providers (like SpiderOak) give them a back door to encrypted communications. To be clear, we have not, nor will we ever, give third parties access to your private data. It so undermines the very core of what SpiderOak believes in, that we would sooner go to jail than comply with such an odious requirement.

Such a provision would put us on a short list. Several countries currently have laws that require decryption keys to be produced on court order, but I could find only one country that requires plaintext access on demand: Iran[1]. Not even China, a country often cited for its severly restricted freedom of speech, has such a requirement.

Aside from the obvious Orwellian issues, there’s a simple technical argument against crypto backdoors: Any cryptographic system that can be broken, even if it’s only by one person, is not secure. It wholly defeats the point of cryptography. Any backdoor made available to the FBI might be found by people with less noble intent, rendering the encryption moot. A lot of our daily life depends on crypto — would you trust your bank knowing that there was a hole in their security just waiting to be found?

And yes, we have a passionate interest in security because it’s our business. If this becomes law, it will terribly pervert or destroy SpiderOak, but ultimately, this is about you. It’s your data we have here, and we want to protect it. Help us help you by raising awareness and contacting lawmakers to make sure this doesn’t get any further.

[1] Crypto Law Survey