Conversations about life & privacy in the digital age

Challenges in compatibility

Recently in the SpiderOak application, we fixed a bug relating to
cross-platform compatibility. As some observers have mentioned, we implement
most of our system in Python.

In the Python world, there is an amazing amount of support libraries and
software available, with differing degrees of maturity. Much software is still
only tested in a limited degree of situations. Even in those whom approach
software testing with a comprehensive mindset may not be able to engineer or
have the resources to test the wide variations in which their software may be
deployed. That’s not necessarily their job though. Most Open Source licenses
come with an explicit disclaimer of warranty, after all.

In our case, we ran into a suprise using Python Crypto Toolkit (PCT). PCT
is a lightweight implementation of common crypto primitives in python,
provided as a mixed python/C library. For public key operations, PCT uses an
internal RSA object behind a generic public-key interface. Problem is, it
actually uses 2 different such objects. If there is a C bignum library
available, it uses that (usually GMP), as it provides math operations on large
numbers that are significantly faster than what standard math implementations
usually provide.

There’s just one little problem…

You think this kind of behavior would be handled at run-time since both
variants of the object handle the same internal data, right? Oops. PCT has 2
different object-types, which one is created depends on whether or not a given
install of PCT has the previously mentioned math module available. However,
when you serialize that object (i.e. save it to disk), that entry is saved,
tagged with the classname of the object. Unless you ensure your serializer
knows this, files saved this way (say, crypto keys), will fail to load again
on a platform installed with a non-fastmath version of this library if the
objects were created by the fastmath variant. To do this in PCT, you have to
patch the module after loading it so fastmath objects don’t get created
and tell the serializer that RSAobj and RSAobj_c (the fastmath one) are
really the same. Thankfully, the PCT developer(s) ensured that these objects
serialize the state the same way. Even when you’re thinking ahead, you can
still get suprised.

Long story short, test and audit EVERYTHING, and have someone else look
too! You WILL be suprised.