Conversations about life & privacy in the digital age

LAN Sync Overview

We’re just around the corner from releasing our much-requested LAN (Local Area Network) sync feature, which is now in the final stages of beta testing.

LAN sync enables much more efficient syncing of data between two or more computers. When SpiderOak is processing a sync between devices on the same LAN (i.e. on the same network), the devices can speak to each other locally, passing any needed data blocks directly to each other without the receiving devices having to download them from the SpiderOak servers. The originating device will still upload the changed or added files to SpiderOak to ensure the data is safely backed up and available to other computers in your SpiderOak network.

To illustrate the difference between ‘normal’ sync and LAN sync, please see the diagrams below:

Normal Sync: Device 1 uploads added or changed data to the SpiderOak servers over the Internet. Once the upload is complete and SpiderOak determines that the data is part of a sync, SpiderOak will start downloading the data to Device 2 which also occurs over the Internet. LAN Sync: Device 1 uploads added or changed data to the SpiderOak servers over the Internet (A). Once the upload is complete and SpiderOak determines that the data is part of a sync, SpiderOak again downloads the data to Device 2, only this time the download occurs directly between devices over the LAN (B). This optimization saves you time and bandwidth.

NOTE: The LAN sync feature only works for devices that are on the same LAN. For example, a sync between a home computer and a work computer will still go through the SpiderOak servers, as they do not share the same physical connection.

However, if you carry a laptop between work and home, SpiderOak is smart enough to determine the most efficient sync method, always transferring data blocks over the LAN when possible. At work, your laptop will sync with work devices over the LAN, and home devices over the internet. At home, SpiderOak will use your home LAN to efficiently transfer data from your home devices to your laptop, and data from your work devices will be downloaded over the internet.

In the end, regardless of the sync method SpiderOak uses, you can always feel safe in the knowledge that your important data is stored – securely & privately – in the cloud.

Introducing the SpiderOak Web API

In our efforts to give back to the community, and create an environment
where our customers can use the SpiderOak service how they want to, we have
released a document describing
the SpiderOak Web API.
This is the very same API we use to implement our ShareRoom and My Login
functionality through the SpiderOak.com website.

Additionally, we have packaged up the custom JavaScript we use to implement
the tree navigation structure in our web interface. The widget is a jQuery
plugin that dynamically creates HTML List elements following the structure of
JavaScript objects. It can drill down into the tree structure by retrieving
data using XMLHTTPRequests, enabling great performance for large trees. The
code is available for download on our Code Page.

We’re also offering other more advanced SpiderOak APIs, such as an API for
‘Zero-Knowledge’ remote command and control of all the devices within a SpiderOak
account, or across many accounts.
Email us for details.

We hope these things are useful, and we can’t wait to see how the community
uses the API to use SpiderOak in ways we haven’t even thought of!

IMPORTANT NOTE: Accessing your data remotely through the web is the only
instance when your data does become readable; however, these machines are only
accessible by a select number of SpiderOak employees. For continued
‘Zero-Knowledge’ privacy we only recommend accessing your data through the
SpiderOak client as it downloads the data before decrypting the blocks.

Update 19 May, 2010: The SpiderOak Web API now supports JSONP! Just append a query string with a callback argument to the regular API URLs, et voilĂ !

Arrived: SpiderOak Version 3.6… An Evolution

I must admit that Charles Darwin’s theory on evolution is one of my favorites. (Another happens to be Newton’s law of gravity – the pulling of items to a core – but we can discuss this another time.) What I like about evolution is that it happens all around us and is constant – there is no way to shut it off. I further appreciate the measured and thorough approach – working over long periods of time such that care and thought can prevail.

It is in this evolutionary spirit that I would like to announce the release of our new SpiderOak Version 3.6.9606 – boasting important architectural changes to overall layout and function. To begin, we have re-arranged the tab display such that Status appears first – followed by Backup, View, Sync and Share. The Status tab also now contains an ‘Overview’ section where you can see
high level information on what is occurring within in your SpiderOak Network – what is being backed up, if a sync is active, and so on. Our hope is that this page relays more pertinent information in a more readable format. I would also like to direct attention to the ‘Stats’ section as we will be adding additional elements to this page soon.

You will notice some important changes to the Backup tab. We created a new ‘Basic’ view that will display all of the various categories and their corresponding size in MBs or GBs. For the less technical user it will provide an easier way to navigate the backup process. Of course we did not remove the ‘Advanced’ selection process where you can select individual folders and files
throughout your network (external drives, network volumes, etc…).

And yes – you did read correctly – SpiderOak also now supports the ability to backup individual files. You will notice that both folders and files now have a corresponding checkbox next to them. We have not yet included the ability to share individual files and are curious to know if this is a feature you all might use – please comment below to let us know?

Lastly and on a more technical note, this new version allows for the creation of a user entirely from the command line without the need for GUI interaction – something we have been planning for quite some time and are happy to finally include.

For more information on what else is included in our new SpiderOak Version 3.6.9606, please view our release notes here: https://spideroak.com/release_notes.

Here’s to evolving…

Why and How SpiderOak architecture is different than other online storage services: The surprising consequences on database design from our Zero-Knowledge Approach to privacy.

First, let’s consider the design of products like Mozy or SugarSync. On the server they have a database of all of your files. This database includes foldernames, filenames, last modification times, sizes, etc, all plainly readable. Maybe they encrypt the data (the contents of each file). The backup client on your computer knows which files need to be uploaded by talking to the server and querying for differences between the local file system and the remote database.

In the SpiderOak world, no such central database of your files exists. Rather, you keep your own database [1]. If you have several computers all connected to your SpiderOak account, each of them maintains a local database giving them a full view into your account-wide storage.

This client-side database is updated continuously as uploads from all computers in your account progress. Each upload is a transaction. We stuff changes into a transaction until it reaches 10 meg or 500 files [2]. The contents of the transaction are sequentially numbered data blocks (the data) and entries in sequentially numbered journals (meta data). For each transaction, the server stores everything, and passes the meta-data only along to all the other devices in your account.

In this sense, SpiderOak is really more of a peer-to-peer application than a client-server application. The traffic all goes through central servers but that’s just a conveniently reliable medium for data-passing and storage. The servers can’t read any of it.

There are some clear benefits and challenging drawbacks of peer-to-peer database architecture.

The biggest benefit is obviously the stated goal of preserving full and complete privacy – or ‘Zero-Knowledge’ as we call it.

One drawback is that it’s harder to program. Usually complex systems are built first with central management and then evolve into more peer architecture for scalability reasons. Think of Napster evolving into things like Gnutella and BitTorrent. We enjoy the challenge of creating privacy-preserving software that works just as well as the alternatives. Indeed, this is one of the main
reasons we started SpiderOak. However, it does mean that almost all features require more implementation work than they might if we had chosen unencrypted storage.

Another drawback is that CPU and memory use are sometimes higher. We’re working steadily to minimize this, but ultimately SpiderOak simply has more work to do than products that don’t maintain a ‘Zero-Knowledge’ orientation. SpiderOak is much better in recent versions at minimizing system resources than it has been historically, and – in the next few versions – this will dramatically improve again.

Yet another drawback is that computational work is duplicated on each client. Instead of the server updating a single database once for each change, each of your computers updates its database for all changes. (Obviously this disadvantage doesn’t apply if you only use one computer with SpiderOak.)

That said, a surprising benefit is the implications for total service cost. You may have noticed that SpiderOak offers some of the best pricing per gigabyte for online storage available anywhere. There are other factors contributing to this, but it definitely helps that SpiderOak clients handlemost of the database work. The server’s role is mostly relegated to data storage and retrieval. This lets us focus on building servers with very dense storage without the need for high speed databases and lots of system memory to
run them in. (Although some of those needs reappear for servicing functions like Web-Access and SpiderOak Shares.)

For us, regardless of the advantages and drawbacks of the decisions we made, the choice has always been clear. We set out to build a backup system we ourselves felt comfortable using which is why ‘Zero-Knowledge’ privacy was always the right path for us.

1 If you want you can go examine this database yourself. Hint: Use the libraries from our code page to make more sense of the database structure. The database files implement a complete transactional filesystem inside a database, as well as some relational tables.

2 If you ever wondered why in the upload status, you’ll see several files all uploading at the same time, with the percentage complete changing in unison for all of them, this is why. Each transaction is uploaded as a unit. The server doesn’t know which or how many of your files it contains.

ShareRoom and Web Login Display Issues in Internet Explorer

We are aware of an issue with Internet Explorer 7 and above that causes the ShareRoom and My Login interface to display incorrectly (just the page header is shown, and the folders/files below are blank). We are working on fixing this for all Internet Explorer users, but in the meantime there is a simple workaround.

If you are using Internet Explorer 7 or 8, there is an option you can enable to fix the display, called “Compatibility Mode”. You can enable this by clicking on the “Broken page” icon at the righthand side of the address bar.

Other browsers such as Internet Explorer 6, Safari, Firefox, and Opera are not affected. We apologize for any inconvenience this may cause.

Regarding Recent Connection Issues (Updated Oct 13)

Final Update: To begin – I need to thank you all again for your patience during the last two weeks. We have worked tirelessly to bring all of your accounts active as quickly and as safely as possible.

At this time, all affected user accounts should be live and the upload process should be fully engaged. As mentioned previously, all of the data you have backed up will be present in the View tab although certain files may still not be available for download as we work to backfill the remaining data.

I would also like to apologize to the users who contacted us and did not get a timely response. Unfortunately, due to unforeseen circumstances with one of our employees, we had to adjust and we do appreciate greatly your understanding. Customer service has always been a top priority for us and we will do whatever we can to live up to the highest standards moving forward.

Lastly – for those interested – we will be posting a full disclosure of what occurred and the process taken for reactivation of accounts.

If you are still experiencing any connectivity related problems, please do email us at support@spideroak.com and we will address your issue immediately.


To the users affected by this recent connection issue: we apologize greatly for any inconvenience this has caused and thank you for your continued patience and understanding during this process.

By way of explanation, one of SpiderOak’s storage clusters (which contains your data) had an unexpected fault during a routine maintenance procedure earlier this week. We have corrected the fault and are working to restore normal operations.

Because this fault happened in a very unusual way and operating in an abundance of caution, we have completed a copy of all the data from this cluster to a new cluster. We believe it is most prudent for us to complete a full file system integrity check and then a full cluster storage audit before resuming operations. For very large file systems like the ones we use, these operations can take days to complete from start to finish.

It is important to note that this situation does not pose a risk of data loss. We have other measures of redundancy, including additional backups (of our backups). To reiterate, a very small portion of SpiderOak customers are affected and – quite unfortunately – if you have experienced connection issues over the last few days then you are among them.

Obviously we will stay in contact with you throughout this process and let you know when things are back online. In the meantime, if you have urgent backup requirements that need to proceed, we would like to offer you a second SpiderOak account with equal space at no charge and which can be brought online immediately. Please do let us know and we will start working with you to make this happen.

Thank you again for your patience and understanding during this outage; we are working day and night to bring the data back online in as safe and protected a way as possible. Please don’t hesitate to send further questions or thoughts at support@spideroak.com.

Our best wishes and deepest apologizes for the inconvenience,

The SpiderOak Team

Update: Again – to the users affected by this outage – your patience and understanding is greatly appreciated. We do apologize for not providing this update sooner but we have been working around the clock to bring the data back online as fast and as safe as possible. You will notice that your clients are now able to connect to the SpiderOak backend (a green light will appear in the Control Center). Unfortunately they are still unable to upload data but have been successfully migrated to a new cluster – the first and necessary step.

Over the next several hours and days, we will be slowly bringing accounts back online and to a fully operational status. The process of restoring from archival backups is slow and its speed not always predictable but we hope by the end of the weekend to have many of our users affected up and running.

In the meantime – if you do have additional questions – please don’t hesitate to contact us at support@spideroak.com and we will respond as soon as possible.

Update 2: We wanted to provide another update to inform you all of our progress. At this stage we have now started bringing users back online and to a fully functional status. This process will continue over the next several days until all of our users affected by this issue have been fully restored to operation.

If your account is still not connecting properly or you have more specific questions, please do get in touch with us at
support@spideroak.com and we will work to resolve any remaining or outstanding issues.

Thank you again for your continued patience and understanding throughout this process. We apologize greatly that it has taken longer than anticipated but the safety of restoring your data properly was and is always the priority.

Update 3: I am writing to provide another update on the recent connectivity problems we experienced.

To begin – I would like to thank you all for your patience during this outage. We have been and will continue to work diligently until the entire situation is resolved.

As it relates to the current status of bringing accounts online, we are continuing to make progress everyday but – unfortunately – the process has taken much longer than we had anticipated. To speed things up, we have devised a new method of activating user accounts that is faster. This will allow users to begin uploading their data to SpiderOak – ensuring changed and/or new data is being securely backed up. And whereas users will be able to see all of the folders and files already stored in their SpiderOak Network on the View tab, the contents will not be available for download at the moment as we will backfill the data in the days to follow.

Our sincere hope is to have everyone’s account online in the next couple of days. If you are still experiencing problems with your
account, please do send us an email to support@spideroak.com and we will be sure to provide you with additional information as to when your account is live. And just as a reminder, we have not lost or misplaced any of your data – it is all accounted for and will be available for you to access.

On a related note, I would also like to apologize for the delayed responses to many of your customer service inquires. Sadly, one of our team members suffered a terrible health problem and we are now getting caught up on your requests. Thank you in advance for your understanding.

Schedule Maintenance/Upgrade – Friday 2 pm – 6 pm CST

I wanted to send out a quick follow-up notification to our users – alerting
you all that we will be performing a schedule maintenance/upgrade for 4 hours
on Friday the 6th from 2 pm to 6 pm CST. It is important to note that this
outage will not effect all of our users; however, those that are effected will
lose connection to the SpiderOak servers. This downtime will not have any
consequence on the security or integrity of your data; rather, the upload
process will stop and then resume as soon as the connection is
re-established.

Thank you in advance for your patience and understanding and we do
apologize for any inconvenience this may cause. If you find the timing to be a
problem, please don’t hesitate to contact us immediately at
support@spideroak.com.

Very best wishes,
The SpiderOak Team

SpiderOak v2.3 beta: New Command Line Interface

Recognizing that the command line interface lacked some capabilities
compared to the GUI, we’ve given it a full makeover as the central improvement
in 2.3.

We’ve also added a folder changelog feature that lets you see exactly how a
folder changed over time, and a collection of other things.

Full release notes here.

Unless something unexpected arises, this should be the last SpiderOak
release before we merge in the Sync code. So — for those who have inquired –
look for it right around the corner.

Beta download links for all platforms are below. The usual cautions about
running beta software apply, of course. If you decide to download a beta
version, pleease send us a note at beta at spideroak.com so we can update you
on future beta releases.

Here’s the new “--help” text for using the command line. This works
the same across Windows, Mac, and Linux.


Alan@Alan ~ $ /Applications/SpiderOak.app/Contents/MacOS/SpiderOak --help
Usage: SpiderOak basic command line usage:

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -v, --verbose         be verbose

  Operational Modes and Commands:
    --backup=TARGET     ad hoc operation: backup whatever exists at TARGET in
                        the filesystem and exit (ignores existing backup
                        selection.)
    --restore=item      Restore a folder, file, or version.
                        Run "--restore help" for more info
    --headless          run in headless mode (without the graphical interface)
    --batchmode         like headless, but will exit when all available work
                        is done
    --scan-only         scan the filesystem for changes and report a summary
    --scan-and-build-only
                        scan the filesystem, and build all possible file
                        system changes as shelved upload transactions, and
                        exit without uploading them

  Information Commands:
    --userinfo, --user-info
                        Show user and device info
    --space             Show space usage information by category and by device
    --tree              Show the hierarchy of stored backup folders
    --tree-changelog    Show a log of how the hierarchy of stored backup
                        folders has changed over time
    --journal-changelog=folder_or_journal
                        Show the changelog of a given folder
    --shelved-x, --print-shelved-x
                        Show information about each shelved upload transaction

  Backup Selection Manipulation Commands:
    --selection, --print-selection
                        Show a list of selected and excluded backup items
    --reset-selection   Reset selection (but preserve excluded files)
    --exclude-file=EXCLUDE_FILE
                        Exclude the given file from the selection
    --exclude-dir=EXCLUDE_DIR
                        Exclude the given directory from the selection
    --include-dir=INCLUDE_DIR
                        Include the given directory in the selection
    --force             Do in/exclusion even if the path doesn't exist

  Maintenance Commands:
    --vacuum            Vacuum SpiderOak's local database (rebuilds indexs and
                        reclaims space)
    --rebuild-reference-database
                        rebuild the SpiderOak reference database (can take
                        awhile)

  Dangerous/Support Commands:
    Caution: Do not use these commands unless advised by SpiderOak
    support.  They can damage your installation if used improperly.

    --empty-garbage-bin
                        purge all deleted items on the current device
    --destroy-shelved-x
                        destroy each shelved upload transaction (not intended
                        for general use -- this will damage your break your
                        account if not used correctly)
    --apply-subscription-xact
                        apply all transactions previously received from remote
                        devices -- (not intended for general use -- this
                        normally happens automatically)

General Beta Download (autodetect OS and architecture, except Ubuntu Gutsy):
https://spideroak.com/directdownload?beta=yes

Platform Specific Downloads

Mac OS X 10.4 and 10.5 (Universal Binary for Intel and PowerPC):
https://spideroak.com/directdownload?beta=yes&platform=mac

Windows 2000, Server 2003, XP, Vista:
https://spideroak.com/directdownload?beta=yes&platform=win

Linux Ubuntu “Hardy” and “Intrepid” 32 bit:
https://spideroak.com/directdownload?beta=yes&platform=ubuntu32

Linux Ubuntu “Hardy” and “Intrepid” 64 bit:
https://spideroak.com/directdownload?beta=yes&platform=ubuntu64

Linux Ubuntu “Gutsy” 32 bit:
https://spideroak.com/directdownload?beta=yes&platform=ubuntu_old32

Linux Ubuntu “Gutsy” 64 bit:
https://spideroak.com/directdownload?beta=yes&platform=ubuntu_old64

Slackware >= 12.1 (preliminary support):
https://spideroak.com/static/main/spideroak-6218.tgz