Sync Server Update

 Originally when I set up my sync solution, I had Two Epson Endeavor servers, a mac mini, and I added a Synology NAS.  These were all "Server" devices, in that they were expected to always be on, and to sync of all or most of the files.  

The machines I had were:

1. Endeavor - 320GB Atom Epson PC

2. Pegatron - 1TB Atom based Epson PC

3. Synology DiskStation NAS

4. Mac Mini 2012 i7 with 2TB (hosted at MacMini Vault)

5. Macbook Pro 15 inch with most of the files synched.

6. Other miscellaneous laptops with only the selective sync enabled, so that they only have the files they need.

As time went by, some of this got degraded as described below:

a. I no longer needed the Mac Mini for the other things I was using it for, and it was overkill (and expensive) to use just for Sync.  This was decommissioned maybe 2 years ago - but that was okay because I had everything synched to the NAS

b. I upgraded the OS on my NAS from DSM 6 to DSM 7 - along with that, unfortunately ResilioSync stopped working.  I managed to reinstall it using Docker, but the original metadata was done, and the Docker container can't read the underlying disk by default.  

c. I sold the Macbook pro 2015.  This was okay because everything was synched up somewhere, right?

Okay, so I knew everything should be on Pegatron - but Pegatron wasn't responding - neither was Endeavor - what's going on?  And how long had they not been responding for?!

Unfortunately, these were questions that I had no idea about the answer to.  These are headless servers and I no longer had a monitor laying around that could use a D-Sub VGA connector.  

I scanned for them using NMAP and no luck.  The power light came on, and the HDD light flashed for a while for Endeavor, which seemed to indicate that it was booting - but I had no way to know any more.  (Pegatron was set up to boot from a USB drive, which flashed only briefly at startup).  

I remembered that I had purchased a cheap LCD monitor and left it at the office, so I went to check that, and sure enough, it had a D-Sub connector.  

First: Endeavor:

Connecting the monitor and powering up Endeavor allowed me to see that everything worked fine, and the boot process has no issues until it came to the network - where the machine hung for around 5 minutes, trying to find an IP address.  I tried different cables and different router ports, but no luck.  

Once it did boot up, I could operate it locally, but not bring up the network interface.  Out of curiosity, I found a USB TypeA to Ethernet adaptor intended for the original MacBook Air, and plugged that in.  Linux recognized it without issue.  I plugged the network cable into that, and dhclient was able to get an Ip address straight away.  It would seem that the network port on Endeavor had been fried.  

Still, using the USB network adaptor, it should work, right?  It took a while, but eventually ResilioSync did find the other peers and start syncing.

Enter Pegatron:

What about Pegatron?  Interestingly enough, this was a different issue.  Firstly, the BIOD and Grub screens would show, following by "Uncompression Error" - Does that mean decompression?

Searching online showed that this usually indicates a RAM fault, so I let it be for a few days.  Pegatron was set to boot from a USB disk for ease of upgrading the HDD.  The idea was that I could easily swap out the HDD and the system would still boot from the same USB drive - so no reinstall necessary.  

I decided to try to boot my PC from the USB drive to see if it was the drive that was the issue, or indeed the memory as various forum posts has suggested.  Unfortunately, I stuck the drive into my PC was I was getting ready to reboot it, and hitting "enter" to all of the "are you sure you want to quit?" type prompts.  One of them that popped up was something like "do you want to format this disk?", and I ended up formatting the main partition of the USB drive as a windows drive - obliterating the Ubuntu root partition - that was incredibly stupid of me - but it also is a demo of one of the features I hate most about windows - focus stealing.  Windows computers are useless while they are still starting up, for example, because (for example): If you log in, and immediately open notepad and start typing, some other stupid app like the mouse driver splash screen, etc., will pop up and steal the focus away from notepad.  You will have to switch back to it to continue.  Since most Windows computers start all sorts of junk at startup it is pretty much unavoidable that you simply can't really use the computer for several minutes.  Even after startup, a popup can happen at any time from any app - and if you are in the middle of typing, you will probably make some selection and close it with unintended consequences.  

Also, by now, Windows should not say "Windows can not read this disk...", but instead say "This is a Linux formatted disk, you need to type FORMATME to format it." or something similar.   

Alas, though, Pegatron was already not working before I nuked the drive, and I still wanted to figure out why.  That didn't take long - the boot drive and GRUB had survived the format, and the same uncompression error still appeared.  I mounted the boot drive from Pegatron on Endeaver and checked out the contents.  fsck indicated no issues - but it was 100% full, and the latest kernel files were truncated.  This is something that the installer should really check for!  Manually selecting an old (intact) kernel version would boot the kernel and ramdisk fine, which would then try to mount the root file system and dump me into a busybox shell when that didn't work.  At least now I know what happened in the first place.  

It's worth noting that busybox on Pegatron also didn't manage to get an IP for the ethernet.  I don't know whether this indicates that Pegatron's ethernet is also fried, or is just a difference in functionality.  

The Path Forward

The idea of these servers was to be as low cost as possible, and since I spend less than $100 each, I would say they achieved that goal.  What's more, they ran for several years without issue.  Resilio Sync still seems to be the best available option, but the Ubuntu LTS 16 is a bit out of date now.  

I installed the latest version of Ubuntu server LTS  onto the USB drive and get Pegatron up and running again, since the HDD is be fine and still have all of my data - it's just not being synced anymore.  The sync metadata was stored on the USB drive, so Pegatron will have to be synched from scratch (more about that below).  As for how to do that, the normal way would be to set up another USB disk with an ISO install image, boot from that, and then install onto the boot disk.  An easier way is to simply download an ISO and mount it as a CD-ROM in VMware to install to the bootable USB drive.  The USB drive can then be transferred to the physical machine.  That is what I did, and once I replaced the USB drive back into Pegatron it booted up normally - but the networking did not come up, so whatever event fried the ethernet port on Endeaver did the same for Pegatron.  That is interesting, since nothing else connected to the network was damaged.  

Once I had Pegatron booted from USB, I edited fstab to set the hard drive to mount as /home as it had been before, and used chown to reset the file owners to the appropriate user IDs.  I downloaded the ResilioSync .deb, and installed that.  Next I edited the ResilioSync configuration to allow remote logins, and logged in from my laptop.  I set up the admin user name and password, and linked this copy of ResilioSync to my ID, with a default of "Disconnected".  That is important, since otherwise, ResilioSync would create all new folders suffixed with (1), and start syncing everything as if it was a new client - and the existing files would be ignored.  I added the relevant folders one by one, selecting the previous folder name when adding them.  ResilioSync will hash all of the files and compare them with the other peers, and then perform updates on files where the hashes differ, using the file modification date to choose the "winner".  Because of this, it is important to make sure the time and time zone of the server are set correctly, though ResilioSync will refuse to Sync if they are very far off anyway.  I also checked the file dates on the hard drive before I started syncing anyway, just to be safe.

Interestingly, I got sync errors for some files, with a message indicating that the files could not be downloaded because a peer had not been online for too long.  Those files all already existed on Pegatron anyway, so I chose the ignore option.  

In the meantime, I had set up a new Mac Mini server a few months ago just to ensure redundancy, and copied over all of the files from the main area of the NAS.  Once the copy was done, I added those folders to ResilioSync, so that they would Sync back to the ResilioSync Docker container on the NAS.  That is not the most efficient way, as there are ways to give Docker containers access to the underlying file system, but it seemed the most sure-fire.

Thus, once I got Pegatron up and running, it was able to copy most all of the files from the new Mac Mini and the NAS.  (Some of the files also of course also synched from my laptops).  

That means that at present, I have 3 copies of most of the files (Except for Bike dash cam videos,  of which I have several terabytes and I am not concerned with backing up).  The most important files have 4 copies too, since Endeavor has synched.  

Since only the changes were transferred, and not the videos, there was only a total of 1.8GB actually re-synched.  

Legacy?

Endeaver and Pegatron are slow Atom machines with only 1GB of RAM, but they were plenty fast enough for what I use them for.  Whenever I clicked on a file I wanted to open on one of my laptops, all three of them would start seeding, and the file would transfer very quickly, maxing out the network connection.  In addition, any changes to any files would also be immediately synched to those machines as well - this means that not only do I have a backup, but I can always delete any files I want from my laptops to save space.  If I accidentally broke a file and the broken version synched, I could always go into the snapshots on my NAS and bring out an old version.  (Technically this is also possible on Pegatron and Endeaver, since they also use BTRFS with snapshots once per hour).  

Still, there are various limitations with the old Epson hardware:

1. The USB is only USB2, which makes any external disks slow.  Hence I only use USB for the boot disk for now.  

2. They are both using physical hard drives, which can outperform SSDs in terms of total write cycles, but they tend to die after a certain age just from spinning all the time, even if they aren't written too much.  In fact, it's interesting that what died wasn't the HDDs, but the network ports..  

3. The 1 GB of memory means that the disk cache was probably not very large, and the fact that Resilio Sync scans the file system periodically to check for any changes that it wasn't notified of by the kernel means that the disks tend to spin up often.  

4. Physical HDDs also use more power and generate more heat than SSDs in general, especially in the type of access patterns used by Sync.  (Relatively constant low level reading, less writing).  

5. The networking was never that fast anyway, though having multiple machines made downloading files quite fast anyway.  

Due to the above, I am also going to set up.a new server to replace Endeaver, since the drive is too small, and the network over USB is a bit slow.    

I still want a low end budget option, so I have chosen a Celeron based Intel NUC with 8GB of RAM and a 1GB SSD.  This should be noticeably snappier, and have HDMI video as an added bonus.  I chose the NUC because it is basically a reference commodity platform, well documented, well supported by Linux, and it should be easy to get replacement parts, etc. for.  I was somewhat tempted to go with an ARM based system, but there could be unforeseen compatibility issues with the ResilioSync binaries, etc.  I also considered using the AsRock BeeBox, but they seem to be less available these days.  

Once the new machine is set up and running, I will decommission the Mac Mini server and shop Pegatron to a friend's house for an off-site backup.  

One Note: I did take this opportunity to evaluate alternatives to ResilioSync, especially open source ones.  My criteria were as follows:

1. Should be distributed (i.e. easily allow multiple servers which automatically sync the same data).

2. It should only transfer differences when files are updated.

3. It should have a selective sync feature, so you can have a synched folder which only downloads files when you try to open them.  

4. It should run (decently) on OS X, Linux, and preferably Windows as well.  Android support would be nice.  

5. It should be self hosted and have no monthly service fee.

My Synology NAS has a sync software (Synology Drive), which has most of these features, and supports multiple operating systems -  but it only works with the NAS as the server, and doesn't have the same kind of multi-server functionality.

There are open source alternatives like SyncThing, but they don't typically have the Selective Sync functionality.  

AeroFS is a close contender, but that was bought out by Redbooth a while ago and is no longer offered as a stand-alone free solution (and it also required a separate VM for the license server is I remember correctly).

SeaFile and some others are again centralized, which means one server to go down.  

So at this point, ResilioSync is really the only thing that fits my criteria.  I was also interested in iFolder, and was waiting for that to mature all the way back in 2002 - but it seems to not be well supported anymore, so I did not investigate it further.  

I do have to say that it is a bit disappointing that ResilioSync is not better supported than it is.  It is such a great product, but it seems to be languishing.  It still works fine on the main platforms, but updated are few and far between, and the lack of support for the lates Synology OS is a bit disappointing.