Conversations about life & privacy in the digital age

FBI Wants Your Data, part Deux

In a previous post, Chip discussed a recent FBI drive for encryption backdoors. It looks like they’re at it again.

Basically, the WSJ article linked to above details how in the New Information Society information is power, and many countries are either attempting to have their own power over information (such as Saudi Arabia’s wanting a backdoor into RIM’s BlackBerry services). Others are afraid of US government influence and intervention in software, which is most likely the driving force behind Russia’s new push for open-source software. The reason for the worry on the latter point is that recently the FBI directory has been touring Silicon Valley companies, trying to get them to give him back doors into customer data, possibly trying to avoid normal governmental oversight on such things.

This, again, is something we care a great deal about. The whole concept that if you have nothing to hide, you have nothing to fear is a fallacy. Something the Computer Weekly article missed is that the “Nothing-To-Hide, Nothing-To-Fear” (NTH,NTF) concept assumes that a theoretical “nothing to hide” lifestyle coincides with the authority’s concept of “nothing to hide.” Political leanings, sexual orientation, love affairs with those of the “other” group, religious views, reporting abusive members of an otherwise benign government, or even holding governments up to the same NTH,NTF concept can bring severe problems upon an individual.

At SpiderOak, we’ve created a system that makes it impossible to for us to reveal your data to anyone; when you create an account, your client creates private encryption keys that we cannot get. If there was a massive SO data breach, either by accident or interference by a third party, all that The Bad Guys would get would be a mass of encrypted blocks- statistically, little more than random noise. Your data at SpiderOak is safe and is a crypto-nerd’s dream by virtue of being started by a bunch of crypto-nerds. That said there’s always the more likely scenario anyway.

Wibbly Wobbly Timey Wimey, or How to Gracefully Compensate for an Incorrectly Set Clock

Part of being a good sysadmin is being flexible. Quite often when something breaks you’re solving a new problem, and the developers, users, and management turn to you to find a solution. One day you’ll be sorting out IP misconfiguration, the other you’ll be debugging Python libraries, and another you’ll be working around yet another limitation of your web server. You wind up reading a lot of documentation and learning a little about everything.

We had a particularly puzzling problem on one of our servers the other day. Somehow, its clock had been set eight hours fast1. We were running NTP, but it wasn’t set to jump the clock on start2, so even though NTP was synced to an upstream server, the clock was still a long ways off.

Normally, this is pretty easy to fix — you set the clock back, and you’re done. For us, though, it was a serious problem because uploads are ordered by commit time, which is dependent on the server’s clock. The guy who wrote our nagios time check was summarily flogged, and Alan and I set out to find a solution. We came up with a few ideas:

  1. Wait eight hours for real time to catch up
  2. Rewrite all the logs and database entries to have correct times
  3. Find a way to slow down what our daemons thought was the correct time in order to gradually bring their time in sync with real time

#1 was right out — we weren’t going to stop uploads for that long. #2 was heinously difficult and fraught with peril, so we went with #3. Alan’s quick googling found libfaketime, one of those great utilities that shouldn’t exist, but does. Libfaketime works as a LD_PRELOAD hack to change the way certain time library calls work. It can create a time offset and/or speed up and slow down the passage of time relative to the system clock3.

After two hours pondering and hacking, we put our plan into action. We set our server’s time back to normal and modified our daemons’ time to pretend to be six hours in the future, running at half-speed. Twelve hours later, the times synced up, and we undid the hack. Problem solved, and I’ve added one more unmentionable hack to my toolbox. :)

1: I’m not entirely sure, but I think the reason for this is a recent motherboard swap. We set our system clocks to UTC, and the motherboard probably arrived with its clock set to Pacific time.

2: Debian/Ubuntu admins note: this option is turned off by default in the openntpd package.

3: Interestingly, Erlang provides similar behavior by default. If erl detects a jump in the system clock, it will slowly adjust its concept of current time to match.

SpiderOak, Free Games and Poetry to round off the year!

It’s about that time of the year when people – for various reasons – feel the desire to spend their hard earned money on gifts for their loved ones, colleges, spouses and children.

Here at SpiderOak – being the sort of people that like to dive headlong into the spirit of any holiday (from Valentine’s to Groudhog’s day – thought we would take the opportunity to play to your greedy side much like a choir of angels would play Rock Band by offering some of the almighty ‘free loot’ that we all love so much.

Keeping it simple, we have 22 copies of some rather brilliant Indie games that we would like to pass along this holiday season. The games (10 copies of Lead and Gold: Gangs of the Wild West and 12 copies of World of Goo) are both available for installation through the Steam service and compatible with both Mac and PC*. The games are both by brilliant Indie developers and we are sure you won’t mind supporting their efforts (check further down in this post for video trailers of the games).

To score a copy of one of the games mentioned above, all you have to do is: 1) install the SpiderOak client and sign-up for a free 2GB account; and 2) post your SpiderOak username together with a home-made New Years Poem in the comment section.

An impartial jury of SpiderOak employees will then burn the midnight oil in search of 22 worthy winners and email the lucky ones game of their choice on December 31st.

To ensure that our little contest and what will surely be some of the best New Years inspired poetry seen in decades reaches even the farthest corners of the world, we encourage you to share this contest among your friends via Twitter, Facebook, Carrier Pigeon and horseback as much as you desire.

Godspeed and a Happy New Year in advance from the SpiderOak crew!

World of Goo Trailer

Lead and Gold: Gangs of the Wild West Trailer

* Happen to be an all-Linux user or just plain not interested in games? Of course we enjoy your poetry as well! Just go ahead and enter the contest with a little note of this and we will send you a 10GB upgrade for your SpiderOak account instead!

A True Story about iPods, Audio Books, Automatic Weapons, Swat Teams and … Design of Technology

My friend Harry told me a story the other day. It’s an incredible story, but
true. Harry is convinced that his story should be told to others as a
cautionary tale, but he’s too embarrassed to tell it himself. So with the names
and some details changed to protect the embarrassed, here it is.

Harry is a skilled technician who works for a leading company. He travels a
multi-state area installing and fixing specialized hardware for large-scale
production equipment. One day awhile back, Harry was tooling down the freeway
in his company’s large cargo van listening to an audio book. It was an exciting
thriller with an intricate mystery plot. He was just to the part where the
detectives were finally closing in and the perps were executing a desperate
plan to dispose of the evidence and escape.

But in one of those Twilight Zone moments, reality and literature converged!
A patrol car flew by him with lights flashing and sirens shrieking. It got off
at the next exit, dashed over the overpass, and came back down the freeway in
his direction. Then came another – and another – followed by vans and SUVs.
What the…?? All of a sudden, Harry’s car is surrounded by law enforcement
vehicles. Now everybody has their lights and sirens on. “Pull over! Pull over!”
screams the megaphone command. Well, this shouldn’t be dull.

Shouting officers surround his car, rip open his doors, drag him out, take
him to the ground, and handcuff him. Men with automatic weapons are focusing a
lot of attention on him. “You guys must think I’m a real badass or
something…” Harry’s always been rather tactless and combative.

“Where’s the body? Where’s the body?” they keep yelling at him. Other
officers have forced entry into his van and are searching everywhere. “I found
the gurney! Where’s the body?” Harry is nonplussed. Men search him and take his
wallet, his ID, and everything else from his pockets. Somebody calls his
employer. Has the company authorized this man to drive this vehicle? Where is
the vehicle scheduled to be traveling today? On and on and on. He’s just passed
a dam and some rivers. Other officers start heading to those locations.
Everyone is highly suspicious of Harry’s identify. Where’s the real Harry? Did
this imposter dispose of him? Who is this guy really?

Finding no additional physical evidence, the officers start an
interrogation. Harry learns that dispatch received an urgent 911 call from his
phone number. The caller described how they were about to dispose of a body.
Law enforcement had pinged his phone, triangulated his moving location, and
taken him down.

“Wait a minute. What you heard must have come from my audio book! My name
really is Harry, but one of the book characters is named Harry, too.”
Dumbfounded blank stares. They rewind his iPod and, after some rummaging
around, replay that book section. (Harry would later complain that they lost
his place.)

“Learn to use key-lock, asshole!” But Harry’s work-issued phone has a safety
feature. If you hold down the 9 key, the phone will dial 911, even with
key-lock enabled. The officers confirm this for themselves. “Maybe you should
get a new phone.”

Guess all’s well that ends well, but Harry’s story is why I fear Apple may
take over the world. Not because Apple TV is wonderful, but because Apple
understands that my parents shouldn’t have to use three different remotes just
to watch TV and phones shouldn’t make 911 calls when you explicitly tell
them not to. While it’s imperative to ensure that people can easily call 911 in
an emergency, a little more thought could have been put into the solution.
Most of us still have much to learn about the design of everyday

SpiderOak is growing and looking for remote help in engineering and customer service

SpiderOak is growing and we are looking to add new members to our team.

SpiderOak is a distributed, virtual company – we all set our own work
schedule and work from home, coffee shops, or anywhere that can provide a
stable Internet connection (crucial to the job of course). We coordinate via
Wikis, Internet Relay Chat, email, telepathy, and even face-to-face when
possible. We don’t bother with time sheets or other types of wage accounting –
we’re a tight enough group that it would be obvious if someone wasn’t doing her
or his job. Our team members are located across North America and Europe.

We’ve noticed that some of the most accomplished people we know don’t
necessarily have polished or extensive resumes. As such, we don’t care about
formal education, age, gender, geographic location, resume, etc. We like smart
people who love what they do and do it really well. Period.

For support: Everyone new – in any role – begins by doing customer
support for a brief period. This is partly a hazing ritual but, more
importantly, the best way to understand our product is to help users better
understand how they can get the most out of SpiderOak. After all, working with
customers is our most important function as a company. (Even programmers
benefit from one-on-one time with the people who actually use our software.) No
programming languages or computing degrees necessary (though they’re certainly
not discouraged); it’s most important to be friendly, courteous, a good writer,
and moderately tech-savvy.

Our ultimate goal in this department is to develop a solid team where
everyone spends 40% – 50% of their time working with customers and the other
50% – 60% of their time working on creative endeavors that promote SpiderOak.
This can be in the form of marketing, videos, funny stories that can be posted
on our blog, and other such outlets. Creativity is always, always encouraged.

For engineering: We use Python, Django,, WSGI, jQuery,
PostgreSQL, nginx, and varnish, with some occasional heavy-lifting help from C and Erlang. We’re
hoping to meet someone to start helping us with web and API development. Update: Note, we’re also interested in meeting mobile developers, as described previously in our jobs section and on this blog.

If you’re interested in joining SpiderOak for any of these roles,
please send a cover letter to href="">,
replacing DDMMYYYY with the current date.

We look forward to hearing from you!

Update 1 December 2010: We are no longer accepting applications for the engineering or customer support roles. Thanks to everyone who applied, and you should have already received or will shortly receive an email response from us, regardless of the outcome.

Finally here – SpiderOak Android 1.0 Application released!

We at SpiderOak are very proud and excited to finally announce the launch of our SpiderOak Android app; SpiderOak Android 1.0. The launch represents the official release of our second mobile client and gives our customers yet another way of accessing, viewing and sharing their stored data ‘on the go’.

Version 1.0 of SpiderOak Android features functionality allowing the user to:

  1. Seamlessly access all of the data backed up across the SpiderOak Network – viewing office documents, pdf files, pictures, videos, music etc…
  2. Easily access their Share-Rooms as well as the shares of friends, family, colleagues, or clients (depending on access to login credentials)
  3. Email ANY file stored with SpiderOak through the Android email client (via link)

Android users can immediately download and install the SpiderOak 1.0 android app directly via SpiderOak, the Android Marketplace and the Archos Appslib.

As we do with all of our projects, we encourage your ideas and feedback as your feedback drives our development effort and makes SpiderOak better for everyone. Please don’t hesitate to send your comments and suggestions to

For feature hungry users we can also reveal that we already have several new features on our Android development map for release shortly, starting with a one-way file synchronization that will allow the SpiderOak app allowing arbitrary files on your SpiderOak account to be kept up-do-date with your Android device file storage.

Still to come; SpiderOak Blackberry App and a program for the Windows mobile platform, but more info on that as development on those platforms draws to a close.

FBI Wants Your SpiderOak Data; North Korean Hackers Steeple Fingers in Anticipation

The FBI is again looking to circumvent cryptography to expand their wiretapping capabilities. They want to require that all service providers (like SpiderOak) give them a back door to encrypted communications. To be clear, we have not, nor will we ever, give third parties access to your private data. It so undermines the very core of what SpiderOak believes in, that we would sooner go to jail than comply with such an odious requirement.

Such a provision would put us on a short list. Several countries currently have laws that require decryption keys to be produced on court order, but I could find only one country that requires plaintext access on demand: Iran[1]. Not even China, a country often cited for its severly restricted freedom of speech, has such a requirement.

Aside from the obvious Orwellian issues, there’s a simple technical argument against crypto backdoors: Any cryptographic system that can be broken, even if it’s only by one person, is not secure. It wholly defeats the point of cryptography. Any backdoor made available to the FBI might be found by people with less noble intent, rendering the encryption moot. A lot of our daily life depends on crypto — would you trust your bank knowing that there was a hole in their security just waiting to be found?

And yes, we have a passionate interest in security because it’s our business. If this becomes law, it will terribly pervert or destroy SpiderOak, but ultimately, this is about you. It’s your data we have here, and we want to protect it. Help us help you by raising awareness and contacting lawmakers to make sure this doesn’t get any further.

[1] Crypto Law Survey

Looking to hack on Android?

Hello everyone,

It’s been getting kind of hectic around here as we expand out in MobileSpace, and we’re looking for someone who can take lead on Android and/or BlackBerry development at SpiderOak. If you’re capable of programming for Android and/or BlackBerry, and excited about SpiderOak, please send me a cover letter (no more than a page, please) covering:

  • Something about yourself, technical and non-technical
  • Where you think mobile platforms will be going in the next 5-10 years
  • Where you think SpiderOak can fit in with your above Future Vision of Mobile
  • Where you think you can fit in with SpiderOak

If I like your cover letter, I’ll be sending you a sample task to complete as a test of development ability.

SpiderOak is an all-telecommute company, with a team scattered across North America and Europe.

We’ve found great hackers often have an unpolished resume. As such, we don’t care about formal education, age, gender, geographic location, resume, etc. We just want to know you love programming and are a skillful programmer and communicator.

If interested, you can email me at Thanks, and I look forward to hearing from you!

SpiderOak DIY: A space efficient key/value store for arbitrarily large values. Now in beta.

Update: SpiderOak DIY service has been discontinued, and is being replaced by the our new storage service which is a new work based on everything we learned from DIY and our previous internal storage projects. It is also open source, with a fancy new ZeroMQ based architecture. Please visit for more information and to request an invite to use that service. The information below is provided for historical purposes only.

We alpha launched DIY a few months ago to allow SpiderOak customers to directly store data on the SpiderOak storage network via https. It’s similar to Amazon S3, but tuned for large backup and archival class data, and thus much less expensive. It’s also open source, on both the server and client side.

Today DIY is now in beta, and we’ve been using it ourselves to implement new features for some time.

Basically, if you’re already using S3 as a backup storage, switching to DIY will save you a great deal. You could also use the DIY code to run your own space efficient, redundant storage clusters for large data.

One of the things we’re pleased with is how comprehensible the DIY implementation is. It turns out that focusing on space efficiency and high throughput (instead of low latency for each request) allows a number of design simplifications compared to other scalable storage systems.

This is a project you can easily jump in and make progress in quickly. It’s built using zfec for parity striping, Python, gevent, and RabbitMQ, with a framework we created for quickly building small message oriented processes.

Feed back from users and developers is much appreciated.

Mac OS X cats VS. The Linux animals – The Aftermath..


I just want to start off by thanking our fantastic blog audience and customers for all the fun answers, good time banter, support and link-love our recent ‘Who would win in a struggle between all the Mac OS X cats and all the Linux animals?’ post resulted in. You guys sure know how rally around something you find amusing!

Our original idea was just see if we could possibly get 10 people to name a few animals, maybe comment on our little artwork and hand out a few GB of free storage to a few users. Something that turned into over 10.000 hits, 60+ comments and our image and following desktop-wallpaper pack being syndicated to several major blogs and wallpaper websites.

Since Wednesday I have been trying to figure out what to do to reward all you wonderful people who contributed and I think (I hope no one gets upset by this sudden change of rules) since our contest rules were vague to say the least and very few actually got 100% on the naming, order etc. (if we want to be picky) this is what we are going to do.

ANYONE who at the arbitrary time of midnightish between Thursday and Friday September 23rd/24th commented on the original post is welcome to email me at, with the name you used for your comment and where geographically you posted the comment from (we will check this with Geo-ip, cause we are just that awesome!). Then as if by magic, you will receive back a code good for adding an extra 10GB to your free or paid SpiderOak account!

For those interested the correct animals, in order of appearance according to the artist should be as follows: 
[Linux]: (Top) Heron; (Left to Right) Ibex, Gibbon, Gecko/Chameleon, Badger, Koala, Warthog, Lynx, Fawn (Deer), Jackalope, Eft (Newt); (Bottom) Hedgehog, Drake (Duck). 
[Mac]: (Left to Right) Cheetah, Leopard, Snow Leopard, Tiger, Jaguar; (Bottom) Puma, Panther.

Of course we encourage everyone to give our service a shot by downloading the SpiderOak client and signing-up for our free service.

Once again, thank you all for your support and I look forward to ‘seeing’ you all again shortly for something completely different!