au QuaStation - Part 8 - Setting up Resilio Sync
Parts of this guide are borrowed from: Resilio Sync's install page.
echo "deb http://linux-packages.resilio.com/resilio-sync/deb resilio-sync non-free" | sudo tee /etc/apt/sources.list.d/resilio-sync.list
curl -L https://linux-packages.resilio.com/resilio-sync/key.asc | sudo apt-key add
sudo apt-get update sudo apt-get install resilio-sync
nano /etc/resilio-sync/config.json
nano /etc/resilio-sync/user_config.json
If the IP address shows 127.0.0.1, change it to 0.0.0.0. (This is a security feature to only allow access to the interface from the local host, but since this machine has no screen we will not be accessing it locally). I would also turn of the "Force HTTPS" option for testing, though you can turn it back on later.
Next, set the service to auto-start.
sudo systemctl enable resilio-sync
Finally, start it up
/etc/init.d/resilio-sync start
If you left the HTTPS option on, then you will have to ignore the security warning since it is using a self signed certificate from Resilio, Inc.
On first access, it will ask you to set up the web admin account. This is the account you will use to log into the Resilio Sync Web interface on this machine, and has nothing to do with the unix accounts in Linux.
Next, you will have to create an ID for this machine. This ID will only live for a few minutes in my case, so I don't care what I will call it. If you are not going to use Sync Pro, then you may want to carefully consider the name.
After you enter the information and click the "Get started" button, it will ask you for a username and password, which is the account you just created in the previous step.
After that you will be greeted with a mostly blank screen, since you haven't created (or linked to) any shares yet.
Now you want to go to a machine that already has Sync running on it so you can get an ID. This can be of a shared folder (in case you are not running Sync Pro), or your actual person ID if you are running pro. I'm going to call this machine the "donor" machine.
Click on the gear icon in the upper-righthand corner, and then click on "Identity" on the left-hand side. You will see the "Link Device" option.
A key for linking your other device will be displayed with the idea that you may key it manually. As fun as that sounds, we will skip the typing practice and click the "Copy Key" button.
Depending on your donor system, a copy key dialog will appear, or the key will be directly copied to the clipboard. If a separate dialog appears, then click the Copy button or hit Ctrl-C.
Now go to Resilio Sync on the recipient system (the Qua Station you are setting up), and again click on the gear icon to enter the settings menu. This time select "License" on the left-hand side.
Select the "Manual Connection" button on the right-hand side of the screen, and a new dialog for entering your connection key will appear.
Paste the key your copied into this field and click "Next".
Another dialog will appear while the system links the devices.
There are three modes you can select from, so let's discuss about that a little bit here:
1. Selective Sync - This mode will sync only the file names, and when you click on one in the Desktop UI, that will will be immediately synched and opened.
If you copy a new file into a folder with selective sync enabled, it will be copied to peers.
Likewise, if you modify a synced file, the changed will be replicated to all peers.
If you delete a file, it will simply be replaced by a placeholder file. (i.e. there is no way to delete the file from all peers from a machine with selective sync installed without using the shell extensions).
All of this means that this is a great mode for clients where you don't want to the full contents of an entire folder, but you want a way to sync single files as you need them, and the easy ability to delete them when you are done without affecting the other peers. This also makes it a lousy choice for a server.
2. Connected - This option causes the server to sync full copies of all of the files in the shared folders under your ID.
If you have a server with a huge hard drive, this might be the option for you. If you use this option with Sync Pro, then as soon as you set it up, it will immediately start downloading all of the files from all of your shared folders.
All files will be synched all of the time, so there is no manual download feature when using this mode. (There are no placeholder files in this mode).
Just as with selective sync mode, modifying a file will cause the modifications to ripple to other peers.
Unlike selective sync mode, deleting a file will cause the deletion to occur on other peers as well.
When this option is set as default, sync will start syncing any new folders you make on any other client. This means you can add a share to your server just by adding it to any client. This can be both good and bad, depending on your setup and intentions.
This would be a good option for a server that should sync all of your files continuously, however if your server is to small to hold all of the files you have now and might have in the future, then this is a bad option to set initially.
3. Disconnected - If a folder is disconnected, then it is not synched in any way, so it would seem like this option is useless, but setting this as the default option means that the server won't go crazy trying to sync everything automatically - and you can instead select the shares you want to sync later.
If you have < 1TB of data, then select "Connected", otherwise select "Disconnected".
Click the folder icon to return to the list of shared folders.
Assuming you had selected "Disconnected" above, you will have to manually set the folders you want to sync. Use the chain link icon on the right side to do this:
For high value data - I would recommend keeping at least 3 copies of anything really important. That way, if one server dies, there are still two copies of it available. (One copy should theoretically be enough, but it has been noted that when trying to pull a lot of data off of a drive that is already near the end of its life, that alone may kill it, so it's better to have two remaining copied).
Lower importance data can be classified as "bulk" (i.e. things that are large and not super important, but you may not be able to re-download), and "disposable" (i.e. things you can always re-download, DVDs you can always re-rip, etc).
I keep two copies for bulk data and 1 copy for disposable data.
In this way, you can have your family photos, etc. mirrored to 3 or more separate disks, and this keeps them secure against hardware failure as well as speeding up data transfer. You can have two copies of things like your skydiving footage which will still keep them fairly safe, and one copy of things like Linux ISOs, etc. that you could re-download if desired.
For really large folders, I separate them into separate shares of ~ 100GB. This is just to allow me to flexibly assign them to different machines. f.e. if I had a folder that was 750GB and only 300GB free on each machine, I wouldn't be able to sync the folder at all in it's fill size form. By splitting it into 7 or 8 chunks, I can sync those chunks to different servers and use the space more efficiently.
Cautions and Notes:
Some people would point out that running an online system like this is essentially the same as RAID, and should not be considered a backup. I don't fully agree, but the reasons given are somewhat valid. If something goes wrong with the sync software, the operating system, or even the hardware, then it is possible that corrupted files could be synched everywhere. In reality, Google does something like this for their backend storage for Gmail and other services and it has never caused a large scale problem. The files are normally modified on client machines which use SSD, and only synched to/from the servers. I use BTRFS with check-summing so I can see if there is any corruption on any of the nodes, and I could re-sync the file if necessary. Resilio sync itself keeps backup copies of recently changed files under 1GB, so I am not using the snapshotting capabilities of BTRFS to protect against idiocy. BTRFS also allows for redundant data copies, but since each machine only has one disk and I am anyway keeping copied of the same data on multiple machines, this seems redundant.
Realistically, the biggest danger is something like a lightning storm or house fire wiping out all of the machines. To account for this eventuality I have one off-site machine which has only the most important folders synched. This backup could be another Resilio Sync Node, or a more traditional backup destination which is copied from one of the nodes that has the most important files synched to it.
For those looking for an open source solution, SyncThing is probably your best bet. It is not as full featured as Resilio sync, but would be something to consider if you were running a large scale commercial system since Resilio Sync is only licenses for personal use. (For personal use, however, Selective Synch is a killer feature which SyncThing doesn't have!)
Continue to Quastation Part 9 - Economics
Comments
Post a Comment