Conversations about life & privacy in the digital age

Engineering Matters

If recent events in the cloud storage community have taught us anything, it’s that Engineering Matters. Features essential to you, to your life, cannot be treated as something to worry about later. We’re founded on principles that security and trust do not have to exclude convenience from modern computer use, and we are dismayed by the prevailing thought that it does.

What do we sell? We sell trust. Trust that a cup of coffee won’t wipe out the only copy of your child’s first steps that you captured on your digicam. Trust that you can back up your tax returns without anyone being able to read it. Trust that when it hits the fan, we will have your data safe and secure. That is what we sell, at the core. Trust is an integral, central part of the system. It’s engineered in, every line of Python, every system API call, every system architecture wiki page. Your data cannot be allowed to leave SpiderOak through an act of God or man without your explicit action to make it do so, and we sell you the trust that it won’t.

Placing trust as the central core of the system fundamentally changes how one architects it. The system is first and foremost engineered to be secure and trustworthy. All the fun features in the world will not change the core, fundamental design principles set into the architecture of the system, and given form through solid and tireless engineering. We engineer into the system the expectation that bad things will happen to good disk platters, and design around that. No amount of high-level policy and vague promises of security can replace solid, engineered-in trust. There may be stumbles and falls on the long journey to this combination of trust and ease of use, but that is testament to the fact that we care more about solving the hard problem – trust in your data that transcends hardware and humans – than quick cheap tick-boxes on the feature list.

This is a great new frontier, an amazing and wonderful time in history where mainframe style computing in the data center is meeting the smartphone in your hand somewhere in the middle, and everything Star Trek promised you is starting to come true. We’re moving into an age where instant-access, anywhere access to any information you want is no longer a Hollywood dream, but daily reality. How many of you will be reading this on a phone with smaller size and much higher capability than Kirk’s communicator? Now is when we can stop complaining that things just aren’t possible, and instead move to the wondering when it will be done!

As we move forward, however, how is this always-on, instant-access society impacting you? Do you not expect that the enablers of this magic to take you seriously, with your needs as an individual? Why does adding the phrase “on the internet” suddenly imply that it’s OK to be lax about trust? You have a file store at home: your computer’s hard drive. For those of us renting, we pay someone (the landlord) to house it. How would you react if your landlord or a maintenance man plugged your hard drive into his laptop and downloaded a copy of everything just because some man in a suit asked him to? Why does “file store on the internet” mean anything different? Why should we instantly relax our standards just because it’s online and shiny?

If this sounds like a manifesto, that’s because it is. At the core of this future is the bedrock we lay down today. What you will have tomorrow, the freedoms and limitations of tomorrow, are set in concrete form with the foundations of today. That is the point of this message, that engineering matters! Core design principles will outlive any set of bugs in an implementation, and that is what we do here. Our core is trust, our core is security, our core is safety. The engineering of the system now will have a direct impact for years to come.

In a world of talk, Engineering Matters.

Et Tu, DropBox?

Friends, Netizens, Countrymen, lend me your eyes!
I come to blog on SpiderOak, not to praise it!
The evil that cloud syncs do live after them; the good is oft interred in their bones.

So let it be with SpiderOak.

The noble DropBox hath told you that SpiderOak was untrustworthy,
if it were so, it was a grievous fault. And grievously hath SpiderOak answered it,
Here, under leave of DropBox and the rest; for DropBox is an honourable service.

So are they all, all honourable services!

Come I to scribe in the SpiderOak Git repo, previously it was just my backup service, faithful and just to me. But DropBox says it is untrustworthy, and DropBox is an honourable service.

SpiderOak hath brought home many encrypted blocks to the machine room, whose parity’ed and securely de-dup’d cypther text did the disk platters fill. Did this in SpiderOak seem untrustworthy?

When that the privacy rights are threatened, SpiderOak hath wept.
Yet DropBox says it is untrustworthy, and DropBox is an honourable service.

You all did see on the internets, the FBI thus wants your data.
Which to these demands we refuse; is this untrustworthy?
Yet DropBox says we are untrustworthy, and, sure, DropBox is an honourable service.

I speak not to disprove what DropBox spoke, but here I am to speak what I do know.
You all do love your secure, versioned, and synchronized backups, not without cause; what cause withholds you then, to

O judgement! Thou art fled to brutish beasts, and men have lost their reason. Bear with me;

My data is on the servers at SpiderOak, and there it shall be safely only for me.

FBI Wants Your Data, part Deux

In a previous post, Chip discussed a recent FBI drive for encryption backdoors. It looks like they’re at it again.

Basically, the WSJ article linked to above details how in the New Information Society information is power, and many countries are either attempting to have their own power over information (such as Saudi Arabia’s wanting a backdoor into RIM’s BlackBerry services). Others are afraid of US government influence and intervention in software, which is most likely the driving force behind Russia’s new push for open-source software. The reason for the worry on the latter point is that recently the FBI directory has been touring Silicon Valley companies, trying to get them to give him back doors into customer data, possibly trying to avoid normal governmental oversight on such things.

This, again, is something we care a great deal about. The whole concept that if you have nothing to hide, you have nothing to fear is a fallacy. Something the Computer Weekly article missed is that the “Nothing-To-Hide, Nothing-To-Fear” (NTH,NTF) concept assumes that a theoretical “nothing to hide” lifestyle coincides with the authority’s concept of “nothing to hide.” Political leanings, sexual orientation, love affairs with those of the “other” group, religious views, reporting abusive members of an otherwise benign government, or even holding governments up to the same NTH,NTF concept can bring severe problems upon an individual.

At SpiderOak, we’ve created a system that makes it impossible to for us to reveal your data to anyone; when you create an account, your client creates private encryption keys that we cannot get. If there was a massive SO data breach, either by accident or interference by a third party, all that The Bad Guys would get would be a mass of encrypted blocks- statistically, little more than random noise. Your data at SpiderOak is safe and is a crypto-nerd’s dream by virtue of being started by a bunch of crypto-nerds. That said there’s always the more likely scenario anyway.

Looking to hack on Android?

Hello everyone,

It’s been getting kind of hectic around here as we expand out in MobileSpace, and we’re looking for someone who can take lead on Android and/or BlackBerry development at SpiderOak. If you’re capable of programming for Android and/or BlackBerry, and excited about SpiderOak, please send me a cover letter (no more than a page, please) covering:

  • Something about yourself, technical and non-technical
  • Where you think mobile platforms will be going in the next 5-10 years
  • Where you think SpiderOak can fit in with your above Future Vision of Mobile
  • Where you think you can fit in with SpiderOak

If I like your cover letter, I’ll be sending you a sample task to complete as a test of development ability.

SpiderOak is an all-telecommute company, with a team scattered across North America and Europe.

We’ve found great hackers often have an unpolished resume. As such, we don’t care about formal education, age, gender, geographic location, resume, etc. We just want to know you love programming and are a skillful programmer and communicator.

If interested, you can email me at Thanks, and I look forward to hearing from you!

Two-Way Mobile Clients & You

Hello everyone,

I’ve been taking a bunch of requests for two-way functionality from our mobile clients, and I would like to address that. Let me start by saying that I *fully* understand how much of a killer feature that would be for all of us. I use SpiderOak for my own personal file storage, sync, and share, as well as using it for work (Android beta testers know I use a ShareRoom to distribute the beta builds, for example). I use my SpiderOak iPhone app several times a day. For mobile users, your pain is my pain as well.

That said, traditional two-way interaction with SpiderOak would be nontrivial for a mobile phone. Our zero-knowledge system takes full advantage of the power available in modern computer systems to run the encryption/decryption and file de-dupe locally before uploading to our servers- see our engineering description page for more details on that. I think most of the latest generation of smartphones (1 GHz Android, A4-powered iOS, generally) are finally powerful enough to even think about doing this. That said, there are still a few obstacles to overcome:

  • Most of our application code is written in Python. Either this can be ported to both ObjC and Java, or I can figure out how to tie in some sort of framework to connect it across from Python. Either way, it won’t be pretty or quick.
  • Battery life is generally the biggest concern I have, once the above could be done. SpiderOak would be sucking down the device’s battery, as well as sucking down data usage (important for those of us on metered data plans). Running the CPU as hard as we can certainly do will run down the battery just as hard. CPU and memory usage of our desktop client is one of the most common complaints about our service, and moving that to a phone is only going to exaggerate this.

I have ideas ideas we’re working on to incorporate our very cool DIY storage system into our mobile platforms to offer secured backup to our storage. This would take advantage of built-in cryptography on the phone, and use a shared key between the desktop application and the phone to encrypt data and then upload that to our servers. That can then be tied in desktop clients to offer zero-knowledge backup from mobile widgets. I can’t guarantee this is going to happen overnight, but it’s where I want to take us until the ARM Cortex A15 gives us the power to do this efficiently.

By using our HTTP-based DIY system, we can bypass the most CPU and memory-intensive portions of SpiderOak interactions, and as I don’t anticipate typical mobile load being that much, it shouldn’t overwhelm the desktop client to ask for just a little bit more help.

Android and… THE FUTURE!

In the mobile world here at SpiderOak HQ, there’s two things that have some code being laid down already, and they’re interrelated.

The first is that an Android app is now being actively worked on. Yes, I know we’re late to the party. If it’s “fashionably late”, or “better late than never”, or “about darn time”, I don’t know, but it’s being worked on. When it’s complete, it’s going to be open-source from the get-go so that those with open handsets can play with the app as they please. This is cool, because it’s going to also act as a demo implementation of our Next Big Thing. This app will provide at least the same amount of functionality as the 1.1 version of the iPhone app.

The Next Big Thing

There’s a lot of things you can do with a bunch of cloud storage. They’re all very, very cool. We have very cool web-based APIs, but I know there are difficulties in using our storage for cool uses (like Documents to Go) straightaway because of our ‘Zero-Knowledge’ encryption system.

The solution that I’m working on is a connection library in Objective-C and Java that will use our public APIs to provide high-level operations for working with files in SpiderOak storage, both personal storage and ShareRooms. This library will also be open-sourced as to make it easier to drop into your own projects, or see how we use our own APIs so that you can adapt the use to your own ends.

New iPhone Client Entering Beta

2010-07-12 UPDATE: Beta delayed due to additional work required. See below.

Hello, your friendly local iPhone developer here. I’m happy to say that the first major revision of our client for the iPhone is now about to enter beta testing. There’s a bunch of new features to help make your SpiderOak storage more accessible to you wherever you go.

  • You can now save files from ShareRooms or your storage locally to your phone as Favorites to have them available to you at times when you may not have network connectivity, such as on an airplane.
  • The app will now remember the last few files you visit in your ShareRooms and storage, so commonly-visited files can be easily re-visited.
  • Our graphics are now available for high-res platforms, so those of you with the spiffy iPhone 4 will see spiffy high-res graphics now.
  • The app is now built with LLVM/Clang, which if nothing else, provides much better diagnostic output when something breaks during builds, reducing development time
  • This version represents the first stage of a gently-sweeping refactor that will result in components of the code being made available open-source, as well as a bunch of Shiny New Features being brought forward quicker and with less bugs.

If you’d like to help beta test, please send an email to with your SpiderOak username by 11:59PM, Saturday, July 10th, 2010.

UPDATE: iPad compatibility demands I go and fix some iOS 4 specific parts of the code. The beta is going to be delayed by a few days while I work on it. Thanks for your patience.