Conversations about life & privacy in the digital age

Download The Newest Version of SpiderOak

Today we are proud to release our newest version – SpiderOak 5.0.

DOWNLOAD 5.0 – HERE

The most significant addition is SpiderOak Hive – a simple drag and drop folder that allows you to sync files across all your devices. For more information on Hive, feel free to visit https://spideroak.com/hive/.

 

You can read about several other new features, enhancements and bug fixes at SpiderOak Launches 5.0 Beta.

Please reach out with any thoughts or questions. We look forward to continuing development and making SpiderOak the best it can be.

Mind in The Cloud

The individual mind is immanent but not only in the body. It is immanent also in pathways and messages outside the body; and there is a larger Mind of which the individual mind is only a subsystem. Gregory Bateson

I believe that our consciousness has evolved within historical time. And that this evolution is chiefly based on external storage. This evolution is disruptive. There is a long period of transition.

We are entering an era of unprecedented access to external storage. I think that we are at the beginnings of a new mode of consciousness.

While our user’s information is inaccessible, SpiderOak spends a lot of time and resources making company information available.

Bruce Sterling (that smirking twit) imagines tracking objects: Spime. But we have the resources to track ideas. Who would have imagined Amazon Reviews becoming a literary form? A modern book is more than a physical artifact. It becomes enfolded in a cloud of comment and interpretation.

The book may surface on the net long before it is published, and may continue to interact with readers. The modern author maintains a conversation with his readers.

For a person of degraded sensibilities, as I am, YouTube becomes a primary repository of music. Incredible new forms are evolving:

Features 101: 5 Characteristics That Make SpiderOak Different

Welcome to SpiderOak University. If you’re a student, new user, or a lover of continuous learning, this month we’re talking to you.

We’ll be posting a couple video shorts each week where SpiderOak CEO Ethan Oberman uses a whiteboard to explain some of our basic product functionalities. School yourself and keep an eye out for our next POP QUIZ tomorrow so you can receive extra GBs.


Apologies for the brief blackout in this video

Whether you need to access a document you have stored on a remote server, synchronize data between a Mac, Windows or Linux device, share important business documents with your clients, or just rest easy knowing all of your data is safely, securely, and automatically backed up – SpiderOak is there for you. Ethan Oberman shows off 5 distinct characteristics.

Do you have a .edu email address? Don’t forget – you can enjoy 50% off your private backup/sync/share account:

Sign up today.

Explaining The Cloud to my Grandparents

Granny and Papa

Pleasure to meet you! I’m new to the SpiderOak team. And I’m new to the cloud technologies space – I come from four years of work with an international nonprofit, Water.org.

I’m completely fascinated. Intrigued. Excited, even. I find myself not only spending time each day learning more about the world of backup, sync, share, and access (mobile), and all things related, but trying to adequately explain to someone else in my life what it all is. It’s good practice.

I recently visited my beloved, hospitable and humorous grandparents – Granny and Papa – in Memphis, Tennessee. As I told them about SpiderOak, they asked the question I have come to expect on a regular basis: “What is the cloud?”

A few weeks ago, SpiderOak’s Jovan Washington wrote a blog post called “Living the CloudLife: Cloud Computing 101,” in which he rightfully called cloud computing a critical trend, and asked “How would you explain the cloud to your mother?”

I took on that challenge. But let me give you a little background: Papa gets on AOL every morning to check his email, the news, his stocks, and forwards the latest funny email, such as “Wal-Martians”. He also keeps tabs on some of the family via Facebook ( i.e. “lurking”). I helped my Granny get on Pinterest (although she loved it, I don’t think she’s active), and she has an e-reader. As far as grandparents go, I think they are doing pretty good with progressive technology.

So I told Granny and Papa:

“The past few years, I haven’t had my own personal laptop, just my work computer. And I obviously had to turn that back in when I left. Since I’d had it for years, it had all my personal music, photos, and documents on it too, besides work stuff.

So, I opened a SpiderOak account, and had it backup, or save, everything off my computer. Then, I completely erased everything on my computer, and turned it back into work, empty. Now, whenever I buy a new computer, I can login my SpiderOak account, and grab anything I want that I had saved off of my old computer. I can just access it, or save it to my new computer. But it’s all there – on the cloud. And no one can get to it but me. And if my computer burns in a fire, everything will still be there for me in the future.”

Even within these past few weeks, I’ve learned to tell most people – “Actually, you know what the cloud is, you just don’t know you do – all of our photos on Facebook, our email in gmail, anything in Google Docs, or if you have photos on Flickr – that is cloud storage, or cloud-based sharing.”

What do you think? How did I do? What did I miss? How do you explain the cloud to someone who doesn’t know?

I’m excited and honored to be a part of the SpiderOak team, getting to know you – the loyal SpiderOak user, and the ever-growing space. In fact, you probably recently heard that Google announced its contribution: Google Drive.

If you missed it, last week, our CEO Ethan Oberman was interviewed on InvestorPlace about the Facebook IPO. I also enjoyed the 6 Myths About Cloud Backup You Probably Thought Were True (as well as the Zero-Knowledge shout out that linked to our mention).

Cheers! Thank you for the warm welcome, and see you here again very soon…
Erin Swanson

P.S. Stay tuned for a SpiderOak announcement this week, particularly of interest to universities.

What Cloud Computing Can Mean for Small Businesses

Cloud storage will be a necessity within five to ten years for businesses and for individuals so we have invited a professional to talk a little about the future of cloud computing. Biz-it Pro has distinguished himself as an online backup reviewer for many years. He has graciously agreed to contribute to the SpiderOak blog and talk about what cloud computing can mean for small businesses.

Cloud computing is changing the way businesses utilize IT infrastructure. Enterprise organizations are deploying on a massive scale to integrate entire business processes to help them scale their business. However, even smaller businesses can benefit from what the cloud has to offer. Take a look at some of the ways a small business can streamline their operations with help from the cloud by SpiderOak.

How Cloud Computing Works

Instead of developing an in-house infrastructure, complete with expensive servers, physical copies of software and lots of fully equipped computers, cloud computing puts all the processing and hosting responsibility on a professional third-party company. Employees access their programs, email and data via thin clients, browser-like tools that remotely connect them to everything they need to do their jobs. There are numerous benefits to this configuration.

Flexibility and Mobility

Since cloud computing services store all the documents, programs and related data in a online, cloud computing system, your employees will have more flexibility and mobility than ever before. They’ll never be tied to a specific computer to work on a presentation or use a special program. This is excellent for traveling salesmen or executives that are often on the go.

Since programs are not stored locally, bug fixes and updates can be done seamlessly from the server side without causing any trouble for the user. This creates a stable, flexible platform that is also accessible via smartphones and laptops. This gives your users the ultimate level of flexibility.

Backup and Redundancy

Cloud systems also make it easy to backup existing data and provide disaster recovery . Easy-to-use systems like SpiderOak allow users to create and sync their local documents with a cloud version, which they can later access from any device. These systems even save revisions of documents so users can go back if they make a mistake and retrieve a previous version. Customer reviews of SpiderOak point out that this cloud data specialist manages to make this technically difficult process very easy for businesses of all sizes.


Many cloud providers have comprehensive small business data backups that regularly copy and protect your data. You’ll also benefit from on-site security, professional-grade cooling systems and a team of professionals who keep the hardware running smoothly.

Collaboration In The Cloud

Cloud systems also make it easy to collaborate on documents, presentations and projects. Since data is stored in the cloud, users can “check out” a file to work on it and then add it back to the collection for co-workers to view. Systems like email and chat are integrated into word processing and productivity tools for a very intuitive and flexible collaboration process.

Get Started Today

You might be surprised at just how easy it is to start using cloud computing tools provided by SpiderOak . Many small businesses discover they can switch to the cloud and even stay on their existing equipment! Instead of paying for an expensive server and technicians to maintain them, a small monthly fee can cover all the needs of your organization. Making changes and expanding is also intrinsically easy thanks to the cloud’s dynamic design. Find out today exactly what the cloud can do for your organization!

Living the CloudLife: Cloud Computing 101

Ever wonder just how far into the clouds you and your data are traveling? You are likely using multiple cloud-based services (including SpiderOak, we hope). Google offers software as a free online service to billions of users across the globe. The Internet giant hosts a set of online productivity tools and applications that live in the cloud such as email (Gmail), word processing (Google Docs), calendars (Google Calendar), photo sharing (Picassa), and website creation tools. And whereas you may not think of those services in ‘cloud’ terms, that is exactly what they are. So when someone asks you to define cloud computing, it may seem difficult at first to explain. Try asking that question to 5 of your friends and you’ll probably get five different answers.

There are many ideas of the ‘cloud’. Often wrapped in marketing lingo, definitions fly all over the net. And while they don’t clearly define cloud computing and what makes it different, they sure make it sound good. In essence, cloud computing means having every piece of data available via the Internet anytime you need it. Wikipedia defines it as “the delivery of computing as a service rather than a product, whereby shared resources, software, and information are provided to computers and other devices as a utility (like the electricity grid) over a network (typically the Internet).”

Do you (or can you) remember when there wasn’t a cloud? Before the cloud emerged, there was software-as-a-service computing. Instead of data pipes and routers and servers, the cloud brings full fledged services. The underlying hardware and software of networking remains of course but there are now higher level service capabilities available to build applications.

Every day new start-ups present options for cloud computing. If we follow these trends, the computer merely becomes a gateway into the cloud – removing the need for onboard storage and freeing consumers to leap from one device to another depending on their requirements.

The days of downloading and installing memory-hogging applications on their device will be gone and replaced a powerful Internet browser capable of accessing and computing. However, we are not quite there yet as there are many drawbacks to this system still. The first is accessibility as we still don’t live in a fully connected world where the Internet is accessible everywhere. Further, there are critical security and privacy concerns and who ultimately has access to what data (an issue we worked hard to solve at SpiderOak with our ‘zero-knowledge’ privacy).

The idea of handing over critical data to a 3rd party company still worries many people – and with good reason. Ultimately, users adopt a hybrid approach where some data lives in the cloud and runs off of cloud applications while other more valuable data stays firmly on their machine.

As you are aware, we at SpiderOak are very passionate about privacy and security. In fact, the founders – Ethan and Alan – created SpiderOak to dis-spell the myth that just because data is online doesn’t mean that it cannot be completely private. We believe, as hope more and more companies will agree, it is in the users best interest to employ the most advanced techniques around protecting user data. This will only grow in importance as the cloud becomes more ubiquitous.

Another somewhat lesser known complaint about the ‘cloud’ relates to outages. In the ‘cloud’ world, we have come to expect that data should always be available. And when it is not we become very very impatient and frustrated. For example, imagine not being able to access email or an important business document. All companies suffer from outages as nothing can be perfect all the time (including us humans). Amazon, generally considered the gold standard, had devastating downtime last year that had millions of users suffering. Though no company can promise an outage-free service, there are methods in place to quickly restore service and make sure data is not lost.

Despite some of the drawbacks and a bit of hype, cloud computing remains a critical trend. Many of you have voiced your opinion on Facebook and Twitter on cloud computing. We would love for you to also sound off in the comments. How would you explain cloud computing to your mother? What do you think of cloud computing? What are some of your predictions for the next 5 to 10 years?

I’d love to look back at this blog post in 5 to 10 years to compare your thoughts!

SpiderOak’s new Amazon S3 alternative is half the cost and open source

As 37signals famously described, in the software business we almost always create valuable byproducts. To build a privacy-respecting backup and sync service that was affordable, we also had to build a world class long term archival storage system.

We had to do it. Most companies in the online backup space (including BackBlaze, Carbonite, Mozy, and SpiderOak to name a few) have made substantial investments in creating an internal system to cost effectively store data at massive scale. Those who haven’t such as Dropbox and JungleDisk are not price competitive per GB and put their efforts into competing on other factors.

Long term archival data is different than everyday data. It’s created in bulk, generally ignored for weeks or months with only small additions and accesses, and restored in bulk (and then often in a hurried panic!)

This access pattern means that a storage system for backup data ought to be designed differently than a storage system for general data. Designed for this purpose, reliable long term archival storage can be delivered at dramatically lower prices.

Unfortunately, the storage hardware industry does not offer great off-the-shelf solutions for reliable long term archival data storage. For example, if you consider NAS, SAN and RAID offerings across the spectrum of storage vendors, they are not appropriate for one or both of these reasons:

  1. Unreliable: They do not protect against whole machine failure. If you have enough data on enough RAID volumes, over time you will lose a few of them. RAID failures happen every day.
  2. Expensive: Pricy hardware and high power consumption. This is because you are paying for low-latency performance that does not matter in the archival data world.

Of course #1 is solvable by making #2 worse. This is the approach of existing general purpose redundant distributed storage systems. All offer excellent reliability and performance but require overpaying for hardware. Examples include GlusterFS, Linux DRBD, MogileFS, and more recently Riak+Luwak. All of these systems replicate data to multiple whole machines making the combined cluster tolerant of machine failure at the cost of 3x or 4x overhead. Nimbus.IO takes a different approach using parity striping instead of replication, for only 1.25x overhead.

Customers purchasing long term storage don’t typically notice or care about the difference between a transfer starting in 0.006 seconds or 0.6 seconds. That’s two orders of magnitude of latency. Customers care greatly about throughput (megabyte per second of transfer speed) but latency (how long until the first byte begins moving) is not relevant the way it is if you’re serving images on a website.

Meanwhile the added cost to support those two orders of magnitude of latency performance is huge. It impacts all three of the major cost components – bandwidth, hardware, and power consumption.

A service designed specifically for bulk, long-term, high-throughput storage is easily less than half the cost to provide.

Since launching SpiderOak in 2007, we’ve rewritten the storage backend software four times and gone through five different major hardware revisions for the nodes in our storage clusters. Nimbus.IO is a new software architecture leveraging everything we’ve learned so far.

The Nimbus.IO online service is noteworthy in that the backend hardware and software is also open source, making it possible for people to either purchase storage from Nimbus.IO similar to S3, or run storage clusters locally on site.

If you are currently using or planning to adopt cloud storage, we hope you will give Nimbus.IO some consideration. Chances are we can eliminate 2/3 of your monthly bill.

SpiderOak DIY: A space efficient key/value store for arbitrarily large values. Now in beta.

Update: SpiderOak DIY service has been discontinued, and is being replaced by the our new Nimbus.io storage service which is a new work based on everything we learned from DIY and our previous internal storage projects. It is also open source, with a fancy new ZeroMQ based architecture. Please visit nimbus.io for more information and to request an invite to use that service. The information below is provided for historical purposes only.

We alpha launched DIY a few months ago to allow SpiderOak customers to directly store data on the SpiderOak storage network via https. It’s similar to Amazon S3, but tuned for large backup and archival class data, and thus much less expensive. It’s also open source, on both the server and client side.

Today DIY is now in beta, and we’ve been using it ourselves to implement new features for some time.

Basically, if you’re already using S3 as a backup storage, switching to DIY will save you a great deal. You could also use the DIY code to run your own space efficient, redundant storage clusters for large data.

One of the things we’re pleased with is how comprehensible the DIY implementation is. It turns out that focusing on space efficiency and high throughput (instead of low latency for each request) allows a number of design simplifications compared to other scalable storage systems.

This is a project you can easily jump in and make progress in quickly. It’s built using zfec for parity striping, Python, gevent, and RabbitMQ, with a framework we created for quickly building small message oriented processes.

Feed back from users and developers is much appreciated.

Announcement: We’re now selling storage à la carte via HTTPS

Update: SpiderOak DIY service has been discontinued, and is being replaced by the our new Nimbus.io storage service which is a new work based on everything we learned from DIY and our previous internal storage projects. It is also open source, with a fancy new ZeroMQ based architecture. Please visit nimbus.io for more information and to request an invite to use that service. The information below is provided for historical purposes only.

This is an alpha release for the SpiderOak Do-It-Yourself API for storing and accessing data directly on the SpiderOak storage network. This is similar to Amazon’s S3 and other cloud storage services, but designed specifically for the needs of long term data archival.

We’re happy that this service is open source, top to bottom (including the code we run on the storgae servers.) It’s also offered at the same very affordable prices as regular SpiderOak storage.

During the alpha, this is only available to SpiderOak customers. Every SpiderOak customer can retrieve an API key and get started immediately if they wish. At the beta release (which will be soon) we’ll enable general signup, and we’ll move out of beta shortly after that.

For details on the implementation, architecture, API, the git repositories for server and client code, please visit the DIY API Project Homepage for more
information.

Update 1: Several people have asked why they don’t see a DIY API key option on their billing page. This is because the DIY API is a paid service, so it’s not available with a 2gb free SpiderOak account. Since the storage is so conveniently accessible over HTTPS, we think it likely to be abused if anyone can easily create 2gb free accounts. However, we’ve setup a $1 upgrade you can use to test DIY when you don’t already have a paid account. Just email support and we’ll give you the upgrade code to use.

SpiderOak v2.3 beta: New Command Line Interface

Recognizing that the command line interface lacked some capabilities
compared to the GUI, we’ve given it a full makeover as the central improvement
in 2.3.

We’ve also added a folder changelog feature that lets you see exactly how a
folder changed over time, and a collection of other things.

Full release notes here.

Unless something unexpected arises, this should be the last SpiderOak
release before we merge in the Sync code. So — for those who have inquired –
look for it right around the corner.

Beta download links for all platforms are below. The usual cautions about
running beta software apply, of course. If you decide to download a beta
version, pleease send us a note at beta at spideroak.com so we can update you
on future beta releases.

Here’s the new “--help” text for using the command line. This works
the same across Windows, Mac, and Linux.


Alan@Alan ~ $ /Applications/SpiderOak.app/Contents/MacOS/SpiderOak --help
Usage: SpiderOak basic command line usage:

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -v, --verbose         be verbose

  Operational Modes and Commands:
    --backup=TARGET     ad hoc operation: backup whatever exists at TARGET in
                        the filesystem and exit (ignores existing backup
                        selection.)
    --restore=item      Restore a folder, file, or version.
                        Run "--restore help" for more info
    --headless          run in headless mode (without the graphical interface)
    --batchmode         like headless, but will exit when all available work
                        is done
    --scan-only         scan the filesystem for changes and report a summary
    --scan-and-build-only
                        scan the filesystem, and build all possible file
                        system changes as shelved upload transactions, and
                        exit without uploading them

  Information Commands:
    --userinfo, --user-info
                        Show user and device info
    --space             Show space usage information by category and by device
    --tree              Show the hierarchy of stored backup folders
    --tree-changelog    Show a log of how the hierarchy of stored backup
                        folders has changed over time
    --journal-changelog=folder_or_journal
                        Show the changelog of a given folder
    --shelved-x, --print-shelved-x
                        Show information about each shelved upload transaction

  Backup Selection Manipulation Commands:
    --selection, --print-selection
                        Show a list of selected and excluded backup items
    --reset-selection   Reset selection (but preserve excluded files)
    --exclude-file=EXCLUDE_FILE
                        Exclude the given file from the selection
    --exclude-dir=EXCLUDE_DIR
                        Exclude the given directory from the selection
    --include-dir=INCLUDE_DIR
                        Include the given directory in the selection
    --force             Do in/exclusion even if the path doesn't exist

  Maintenance Commands:
    --vacuum            Vacuum SpiderOak's local database (rebuilds indexs and
                        reclaims space)
    --rebuild-reference-database
                        rebuild the SpiderOak reference database (can take
                        awhile)

  Dangerous/Support Commands:
    Caution: Do not use these commands unless advised by SpiderOak
    support.  They can damage your installation if used improperly.

    --empty-garbage-bin
                        purge all deleted items on the current device
    --destroy-shelved-x
                        destroy each shelved upload transaction (not intended
                        for general use -- this will damage your break your
                        account if not used correctly)
    --apply-subscription-xact
                        apply all transactions previously received from remote
                        devices -- (not intended for general use -- this
                        normally happens automatically)

General Beta Download (autodetect OS and architecture, except Ubuntu Gutsy):
https://spideroak.com/directdownload?beta=yes

Platform Specific Downloads

Mac OS X 10.4 and 10.5 (Universal Binary for Intel and PowerPC):
https://spideroak.com/directdownload?beta=yes&platform=mac

Windows 2000, Server 2003, XP, Vista:
https://spideroak.com/directdownload?beta=yes&platform=win

Linux Ubuntu “Hardy” and “Intrepid” 32 bit:
https://spideroak.com/directdownload?beta=yes&platform=ubuntu32

Linux Ubuntu “Hardy” and “Intrepid” 64 bit:
https://spideroak.com/directdownload?beta=yes&platform=ubuntu64

Linux Ubuntu “Gutsy” 32 bit:
https://spideroak.com/directdownload?beta=yes&platform=ubuntu_old32

Linux Ubuntu “Gutsy” 64 bit:
https://spideroak.com/directdownload?beta=yes&platform=ubuntu_old64

Slackware >= 12.1 (preliminary support):
https://spideroak.com/static/main/spideroak-6218.tgz