On Windows, Quicken files should back up without problems. As with other files with proprietary file formats, we recommend that you exit the application before backing up the data. This prevents saving a copy of the data in an inconsistent state, which would show up as a corrupted database on restore. If Quicken has an "Export" or "Save as..." menu option, using these to save the data in a separate file before backing up would be the best option to use.
On MacOS, Quicken files are saved in package files. SpiderOak handles these files how the underlying operating system presents them, which is a directory. When restoring these file types, you should restore the entire directory (as presented by the SpiderOak client), not individual files. The Sync function should not be used for these file types.
It is recommended that you do not use Sync with Quicken files. Like other applications, the data is updated frequently, and SpiderOak may try to update it before the data file is in a consistent state. With the proper precautions, it may be possible to sync with a manual or carefully coordinated schedule. Syncing will not be possible between MacOS and Windows.
More details on the Mac-to-Windows issue are available from Intuit's website at Converting Quicken for Mac files to Quicken for Windows.