Syncthing: decentralized file sync software

When it comes to file synchronization, there are several choices based on free software:

I have tried both and I preferred the latter for personal use. Nextcloud is obviously a great way of sharing files with your friends. It enables you to collaborate with them and work on the same project at the same time. However, setting up a service on a central server, which is often physically out of your hand, is a little bit overkill for personal use. As long as you synchronize your files among your own devices you would not need the server, which can be a single point of failure (data can be lost if it fails).

On the other hand, #Syncthing does not require a central server at all. By default it transmits data only among your devices. For example, you can set up Syncthing on your desktop, laptop, and smartphone. If they are connected with LAN data can be synchronized locally, and it is not sent outside of it. You can decide by yourself where to store your data, and there is not a single point of failure.

Syncthing is available on not only Windows, macOS, Android, and iOS but also GNU/Linux distributions and FreeBSD. This level of availability cannot be expected from proprietary software, of course.

Encrypt private data

If you synchronize private data such as recovery passphrase of your online accounts I highly recommend you to encrypt them beforehand to make sure that nobody but you can read the data.

From the perspective of versatility you may use GnuPG for data #encryption. As long as the private key is kept safely, practically there is almost zero chance for other people to read data, even if you lost your device and someone got an access to the encrypted files.

After you create the key pair you can upload the public key to a distribution service such as https://keys.openpgp.org/. Once you have the public key, you can encrypt the file without the private key. It will not be required until you decrypt the file to read it.

In order to keep the private key secure, you can store it on Yubikey, following the official guide available here. Because the private key is stored on the hardware key, it cannot be hacked online.

Create a backup on Storj DCS with rclone

Syncthing provides you a function to store file history, but data loss can happen for whatever reason. You may set it up in a wrong way or there may be a bug on Syncthing. If your devices were broken from a natural disaster like flood, your data would be lost forever, so you should always prepare backup in case.

Following the golden rule of data backup, it should be outside of your Syncthing ecosystem. This means that it should be available online, meaning it comes to full circle: storing data online, outside of the local area network.

The backup itself should also be encrypted and decentralized. For if the backup data was available to someone, what is the point of setting up decentralized file synchronization among your devices in the first place?

For backup solution I would pick up Storj DCS this case as well. #Storj DCS, renamed from #Tardigrade recently, provides an encrypted and decentralized cloud storage. I have covered the service on this site multiple times such as here.

To synchronize local data with cloud storage services, rclone is one of the popular choices. After installing, you can configure it for Storj DCS.

For daily use you can run a .sh script like this:

#!/bin/sh
rclone sync -i --progress /home/local/directory/ remote:bucket/path/to/dir/ --exclude=".stversions/**"

Running that script synchronizes the source to the destination, changing the destination only, deleting any excess files. Folders inside .stversions store versioning files, which Syncthing on each computer saves for itself, so they should only stay local and be excluded from synchronization between Storj DCS.

Since the command can cause data loss, test with --dry-run flag at first to see exactly what would be copied and deleted.

Warning: you are not supposed to upload a huge number of files as the fee is calculated not only how large files are, but also how much files are uploaded! You should read ToS carefully before using the service.

Restore backup

If data loss or conflicts happen locally, you may recover the latest state from backup with this command:

#!/bin/sh
rclone sync -i --progress remote:bucket/path/to/dir/ /home/local/directory/ --exclude=".stversions/**"

It downloads data from Storj DCS, changing the local directories only, deleting any excess files. Test with --dry-run in this case as well.

Copyright © 2021 Suguru Hirahara. This work is available under GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation. See https://blog.progressiv.dev/yq31akw3jj for copying conditions.