Conversations about life & privacy in the digital age

Upgraded ZIP Functionality for Web Access

We’ve upgraded the zip download in our web shares and web login, and it now creates ZIP64 archives (also sometimes referred to as version 4.5 archives). ZIP64 allows us to create archives with more than 65535 files and archives larger than 4GB.

Unfortunately, not all zip programs support ZIP64. Most notably, Windows XP’s Explorer does not support it. For Windows XP, we recommend using 7-Zip.

ZIP64 has been confirmed to work in OS X 10.6 Finder, Windows Vista/7 Explorer, and Ubuntu 8.10+ Archive Manager (a.k.a. File Roller). It is also supported by Info-Zip 6.0 on the multitude of platforms it supports.

Welcome to the New Blog

If you’re here visiting our blog in person (hello to all our RSS readers out there!), you’ll notice we’ve changed things up. Our blog has a whole new look, and behind the scenes we’ve changed how things work to make blogging easier on us. Our new blog allows us to better communicate with you guys — a problem you’ve made us well aware of.

Let me tell you guys a story about our old blog.

Way back when, we decided we needed a blog, and we created something very quick and simple. Each post was a file in a directory, and the blog software just spewed them out to a single page. This is a pretty easy setup when you’re a Linux hacker and are very used to editing text files. Most of us, however, have no desire to log into our webserver and fiddle with files. Doubly so because our website runs out of a source control repository, meaning you first have to commit the post to the repository, then update the checkout on our server, repeating ad nauseam if you didn’t get things quite right the first time. For our non-technical folks, this involved two people — the guy writing the post, and an admin operating the website. This was a chore and a waste of everyone’s time, and over time we became less and less enthusiastic about writing new blog posts.

Our new blog, by contrast, is database driven, and each of us can go directly to a private webpage, write up a post, and publish it on our own. It’s a lot easier for us to post, which I’m hoping will result in better communication with the people that really matter — you guys.

So watch this space, because you’re going to hear a lot more from us now. :)

User Signups Fixed

If you tried to sign up for a new account, add a device, or reinstall a device in the last day, you probably ran into the dreaded “502 Bad Gateway” error. After a routine upgrade, our new user process stopped working and unfortunately, due to complications with our monitoring systems and most of our customer support being distracted for several hours today, we didn’t find out until today. We’ve fixed the problem, and signups/new devices/reinstalls are working again. We’re very sorry for the downtime. Rest assured that the responsible parties have been flogged.

Thanks again to everyone who wrote into support, and have a nice day. :)

Building a Server

Most companies pay a lot of money to have have third-parties build
and maintain their storage infrastructure, often at an enormous markup
beyond the cost of the hardware. At SpiderOak, they do things a little
bit… differently. They have me.

A big part of my job is following the industry and researching and
testing new ways to build storage servers, and if I do say so myself,
I’m no slouch at building systems. I’ve designed and administered a
small computational cluster and built my fair share of desktops and
servers. So if everything worked perfectly, building SpiderOak machines
would be a doddle.

But they don’t, and it’s not. A few recent examples:

Since our servers are locked away in a data center, we have an array
of remote monitoring and access controls for our machines, core among
which is the BMC, or Baseboard Management Controller. Its job is to
allow all the things you could do with a computer physically, like turn
it off and on again or look at the console display, over a network. With
one of these in your computer, you can install, configure, run, and
destroy your computer from anywhere in the world. It’s a fantastically
convenient piece of kit when it works properly.

The particular BMC we have is made by a company who shall remain
anonymous, but we’ll call them MuperSicro. Now, MuperSicro’s BMC is
designed to share an Ethernet port so that it doesn’t need a dedicated
port. It does still have its own MAC address, though. Or it
should. This particular unit came to me with a MAC address of
00:00:00:00:00:00. Their solution? “Take the MAC address of LAN2 and add
1.” That works, but I would like it if parts came properly configured. I
ordered a BMC, not a Heathkit for one.

More fun comes from our LSI 8888ELP controller. This is a fantastic
SAS RAID controller — internal and external ports, 512MB of cache,
and excellent OS compatibility. The configuration, though, is a bit
daffy. For their BIOS configuration, you have a choice. You can use
WebBIOS, which poorly imitates a webpage, uses a mouse, and it just
about the worst choice of interface for RAID configuration imaginable.
Alternatively, you can use Preboot CLI, which is MegaCLI in standalone
firmware form. The deficiencies of MegaCLI have been href="http://www.kaltenbrunner.cc/blog/index.php?/archives/4-LSIlogic-MegaRAID-SAS-and-the-self-explaining-CLI.html">adequately
discussed by others, and I can say as a man who uses both mplayer
and ffmpeg frequently, it is bar none the most hideous and inconsistent,
poorly-documented piece of crap command-line tool I have ever used.

Without proper documentation, I willy-nilly decided to enable
DirectPdMapping, figuring that it would allow me to get direct access to
the drives. That particular option is, I might add, not documented in
the MegaRaid SAS User’s Guide available on LSI’s website. It said I
needed to reboot, so I did, and I was greeted with this:

Attached Enclosure doesn't support in controller's Direct mapping mode
Please contact your system support.
System has halted due to unsupported configuration.

The controller decides that it can’t use the enclosure in the way I
asked, so it halts the system. It doesn’t offer to turn direct mapping
off (which would be nice) or offer to load the configuration tool (which
would be expected), it just halts. The solution is to open the case and
unplug the SAS cable to the enclosure, which then allows the machine to
boot so that you can change it, then plug it back in and continue on
your way. If this machine was in a rack in the datacenter when this
happened, I’d have to bother the techs to go open it up and fiddle with
it. It allowed me to get into a situation that was unrecoverable without
physical intervention. That is completely unacceptable. Oh, and neither
WebBIOS or Preboot CLI can help you turn that feature off, either. I had
to boot into Linux to switch it back.

Invariably you will find foibles like these when building a new
system, which is why we spend time poking and prodding newly built
systems before putting them in our data centers and entrusting them with
your data.

Notes from the Dungeon

Hi, I’m Chip, SpiderOak’s semi-resident sysadmin. I’m the guy in the picture
below who looks like he’s up to no good. }:-> Mostly, I’m in charge of
keeping the beasts in the server room well-fed and happy, which, like most
admin work, involves generous helpings of my sanity. It’s a job that requires
me to be available around the clock — especially during inopportune times
like weekends and holidays. Nearly every day I’m asked to do something I
haven’t done before. Sometimes it’s fun, but many times it’s not.

But you know what? I love it.

And it’s not just because most days I can wake up at noon and work in the
wee hours of the morning, or that if I need a break I can pop into the other
room and play some Portal (Ah, GLaDOS, an admin after my own heart). I work
with a great bunch of people, and despite the fact that we’re many states and
time zones away from each other, the whole group meshes into a team that I’m
proud to be a part of. (My absence in the group picture below
notwithstanding…)

What exactly goes on behind the scenes at SpiderOak? It’s a really sim…

Gruuuuuuuuuuu…

Whoops, looks like I’ll have to explain later. Duty calls. :)