Conversations about life & privacy in the digital age

New iPhone Client Entering Beta

2010-07-12 UPDATE: Beta delayed due to additional work required. See below.

Hello, your friendly local iPhone developer here. I’m happy to say that the first major revision of our client for the iPhone is now about to enter beta testing. There’s a bunch of new features to help make your SpiderOak storage more accessible to you wherever you go.

  • You can now save files from ShareRooms or your storage locally to your phone as Favorites to have them available to you at times when you may not have network connectivity, such as on an airplane.
  • The app will now remember the last few files you visit in your ShareRooms and storage, so commonly-visited files can be easily re-visited.
  • Our graphics are now available for high-res platforms, so those of you with the spiffy iPhone 4 will see spiffy high-res graphics now.
  • The app is now built with LLVM/Clang, which if nothing else, provides much better diagnostic output when something breaks during builds, reducing development time
  • This version represents the first stage of a gently-sweeping refactor that will result in components of the code being made available open-source, as well as a bunch of Shiny New Features being brought forward quicker and with less bugs.

If you’d like to help beta test, please send an email to matt@spideroak.com with your SpiderOak username by 11:59PM, Saturday, July 10th, 2010.

UPDATE: iPad compatibility demands I go and fix some iOS 4 specific parts of the code. The beta is going to be delayed by a few days while I work on it. Thanks for your patience.

Thanks,

Matt

An Erlang/OTP SSL proxy and load balancer for Python Twisted’s Perspective Broker

(If the above sounds like gibberish to you, you’re probably not a programmer
and this post won’t be very interesting.)

SpiderOak clients maintain a SSL connection to a Python Twisted Perspective
Broker service to coordinate their actions with the server and with each
other.

To load balance client connections across several Perspective Broker
processes per storage cluster, and route connections from a single public IP to
many storage nodes, we built a proxy server in Erlang. We’ve been running this
in production for several months now.

The design is simple. Erlang/OTP answers the socket, and speaks the
perspective broker protocol just long enough to learn the authentication
credentials the user is attempting to login with. The Erlang server looks up
the user’s assigned storage cluster and node. From there, it simply proxies
the connection (including replaying the authentication sequence) to a Python
Perspective Broker server. After that, it’s a byte-for-byte pass through proxy
server.

The proxy has some added logic to handle connection affinity — multiple
devices for the same SpiderOak user are passed to the same Perspective Broker
process.

This has allowed us to consume fewer public IP addresses (one per proxy
server, instead of one for each storage node) and take advantage of multiple
processors and greater concurrency per storage machine.

Another small benefit is offloading the cost of SSL from the Python
processes. Erlang has it’s own native implementation of SSL (not based on
OpenSSL) which seems to operate with more grace.

This is our first production Erlang/OTP service, and it hasn’t been without
its speed bumps, but these days it’s as stable as any of our other daemons
while handling much greater concurrency and traffic.

Today we’re publishing the code (AGPL3) in case it’s useful to anyone else
(and feedback from the Erlang community is certainly welcome!) It would be
useful to anyone wishing to be able to distribute a Perspective Broker service
across many backend nodes according to user assignment, or perhaps a starting
point for implementing a Perspective Broker server in Erlang. It will likely
require some minor massaging to with your database scheme. Here’s a link to
the tarball: href="https://spideroak.com/dist/spideroak_ssl_proxy.tar.bz2">spideroak_ssl_proxy.tar.bz2

Announcement: We’re now selling storage à la carte via HTTPS

Update: SpiderOak DIY service has been discontinued, and is being replaced by the our new Nimbus.io storage service which is a new work based on everything we learned from DIY and our previous internal storage projects. It is also open source, with a fancy new ZeroMQ based architecture. Please visit nimbus.io for more information and to request an invite to use that service. The information below is provided for historical purposes only.

This is an alpha release for the SpiderOak Do-It-Yourself API for storing and accessing data directly on the SpiderOak storage network. This is similar to Amazon’s S3 and other cloud storage services, but designed specifically for the needs of long term data archival.

We’re happy that this service is open source, top to bottom (including the code we run on the storgae servers.) It’s also offered at the same very affordable prices as regular SpiderOak storage.

During the alpha, this is only available to SpiderOak customers. Every SpiderOak customer can retrieve an API key and get started immediately if they wish. At the beta release (which will be soon) we’ll enable general signup, and we’ll move out of beta shortly after that.

For details on the implementation, architecture, API, the git repositories for server and client code, please visit the DIY API Project Homepage for more
information.

Update 1: Several people have asked why they don’t see a DIY API key option on their billing page. This is because the DIY API is a paid service, so it’s not available with a 2gb free SpiderOak account. Since the storage is so conveniently accessible over HTTPS, we think it likely to be abused if anyone can easily create 2gb free accounts. However, we’ve setup a $1 upgrade you can use to test DIY when you don’t already have a paid account. Just email support and we’ll give you the upgrade code to use.

ShareRoom Embedding

We’ve had a lot of requests for a simple code snippet you can use to add a login form for ShareRooms to your own website. Well, we’ve come up with something for you! Just copy the HTML code below and paste it into your website or blog. It couldn’t be easier! Feel free to customize the code if you’re familiar with HTML, too. The important bits are the form tag, and the share_id and room_key form fields. For future reference, we’ve added a FAQ entry.

Note: If you only want to link to a specific share room, the code below is not necessary. Simply open the share room you want to link to by using the share login form on spideroak.com, and copy the URL shown in the address bar of your web browser. Paste this link into your blog or website just as you would link to any other website.

Feeling disconnected? This is why.

If you’re having connection problems from the SpiderOak client, the solution is to upgrade to version 3.6.9658 or later.

… because 3 years ago when we launched SpiderOak I generated the SpiderOak SSL certificates that the SpiderOak client uses to verify the identify of the storage server. This is to protect against DNS poisoning attacks (i.e. otherwise an attacker that controlled DNS could attempt to convince your SpiderOak client to upload data to a different server.) These are not the same certificates as for the SpiderOak website.

I thought I generated certificates for 10 years, but they were only for the default of 3 years, and thus connections began expiring en masse about an hour ago. Most mistakes you should only make one time, and clearly this would fall under that category.

The verification for the cert is embedded along with the new SpiderOak client. We generated new certs, and fast tracked new builds through testing and release, so please visit the direct download link and all will be well again.

Please accept my deepest apologies; I’ve requested my flogging to be scheduled tomorrow at sunrise.

Upgraded ZIP Functionality for Web Access

We’ve upgraded the zip download in our web shares and web login, and it now creates ZIP64 archives (also sometimes referred to as version 4.5 archives). ZIP64 allows us to create archives with more than 65535 files and archives larger than 4GB.

Unfortunately, not all zip programs support ZIP64. Most notably, Windows XP’s Explorer does not support it. For Windows XP, we recommend using 7-Zip.

ZIP64 has been confirmed to work in OS X 10.6 Finder, Windows Vista/7 Explorer, and Ubuntu 8.10+ Archive Manager (a.k.a. File Roller). It is also supported by Info-Zip 6.0 on the multitude of platforms it supports.

Welcome to the New Blog

If you’re here visiting our blog in person (hello to all our RSS readers out there!), you’ll notice we’ve changed things up. Our blog has a whole new look, and behind the scenes we’ve changed how things work to make blogging easier on us. Our new blog allows us to better communicate with you guys — a problem you’ve made us well aware of.

Let me tell you guys a story about our old blog.

Way back when, we decided we needed a blog, and we created something very quick and simple. Each post was a file in a directory, and the blog software just spewed them out to a single page. This is a pretty easy setup when you’re a Linux hacker and are very used to editing text files. Most of us, however, have no desire to log into our webserver and fiddle with files. Doubly so because our website runs out of a source control repository, meaning you first have to commit the post to the repository, then update the checkout on our server, repeating ad nauseam if you didn’t get things quite right the first time. For our non-technical folks, this involved two people — the guy writing the post, and an admin operating the website. This was a chore and a waste of everyone’s time, and over time we became less and less enthusiastic about writing new blog posts.

Our new blog, by contrast, is database driven, and each of us can go directly to a private webpage, write up a post, and publish it on our own. It’s a lot easier for us to post, which I’m hoping will result in better communication with the people that really matter — you guys.

So watch this space, because you’re going to hear a lot more from us now. :)

User Signups Fixed

If you tried to sign up for a new account, add a device, or reinstall a device in the last day, you probably ran into the dreaded “502 Bad Gateway” error. After a routine upgrade, our new user process stopped working and unfortunately, due to complications with our monitoring systems and most of our customer support being distracted for several hours today, we didn’t find out until today. We’ve fixed the problem, and signups/new devices/reinstalls are working again. We’re very sorry for the downtime. Rest assured that the responsible parties have been flogged.

Thanks again to everyone who wrote into support, and have a nice day. :)

Dramatic Discovery of New Interpretive Compression Algorithm

Here at SpiderOak, we’re always excited about giving back to the community. In the development of SpiderOak, we’ve contributed a number of our internal projects as open source releases. None of those projects had us quite as excited as our latest release, though. We present to you:

Click here for more details on the algorithm.

Download the source code: invertedkernsquish-0.1.0.tar.ksquish

Improve Productivity and Health by Relocating your Chair

A few months ago I started standing during my working day. The center of the
30″ display is at eye level, with the keyboard and trackball slightly above hip
level.

Motivation

I’ve read that sitting throughout the day (with your upper body supported by
leaning against the back of the chair) causes the back and abdominal muscles
which would otherwise be exerted holding your body upright to atrophy. There
seems to be some research to support this. There are even specific types of
chairs designed to enforce self-supporting posture.

I’ve experimented with many hacks to my personal space work arrangement over
the years. Many have been dead ends, but often enough they’ve been useful.

Transition

At first I couldn’t comfortably stand all day. My feet would be sore after
3 to 4 hours, so I would stand in the morning, and transition to sitting
whenever my feet complained. Changing between different pairs of shoes helped,
and being barefoot helped but was cold during the winter. I eventually settled
most often on some good quality slippers that just keep my feet warm with
minimal padding or support. You can find slippers that look almost like
professional footwear.

Positive changes I’ve noticed

  • When I’m in a moment of thought while hacking, I’ve noticed that the
    absence of any required effort to “get up” means that I have a greater tendency
    to step away from the screen while I think. I might pace around or look out
    the window.
  • No lower back pain toward the end of the day.
  • My back is overall stronger (that’s apparent through tracking my regular
    resistance training)
  • I’m warmer (higher in the room where the warmer air is, plus the effort to
    stand does burn more calories and maintains body heat.)
  • Reduced eye strain, likely because of more frequent focusing on distant
    objects.
  • Keyboard and trackball positions are slightly more comfortable with less
    pronation. I still eventually plan switching to a vertical keyboard (typing in
    handshake position instead of palms downward.)
  • Minor but noticeable improvements to digestion and elimination. This may
    sound a bit unusual to discuss, but it’s not surprising. Peristalsis seems to
    suffer from prolonged periods of little body motion.

Negative changes

  • I look like a weirdo with a monitor on a chair on a desk. I’m used to
    standing out, but I’ll get proper display mounts eventually.
  • Less tolerance for long periods of chair sitting. A couple times a week I
    work at a coffee shop. The first couple of hours of this are now actually more
    comfortable, but that ends sooner.
  • People on the internet will laugh at you.