au Qua Station - Setting up snapshots for BTRFS to prevent data loss

 I installed most of the machines with BTRFS as the filesystem for the /home partition, mainly for the copy-on-write functionality.  If power is lost, this is much less likely to cause data loss, and can be checked much more quickly and reliably.

One thing I neglected to do was add the last "2" at the end of the /etc/fstab file.  This was set up as 0 on the distribution I used for some reason, and I just left it that way for a long time.  I think BTRFS always does a recovery on mounting anyway, but it doesn't hurt to add it to the fstab file anyway.

I have mentioned elsewhere about adjusting the advanced parameters of Resilio-Sync to prevent data loss by extending the archive retention period and increasing the size limit of files to be archived, but bear in mind that those settings may not cover all cases - including bugs in Resilio Sync itself.

What can cover those types of issues is the BTRFS snapshot feature.

This is a filesystem feature that allows you to take a snapshot of the filesystem at a certain point of time, and access and/or restore it later on if needed.  The main issue is that the filesystem feature existing does not make it immediately available to use in a practical and friendly to use form.

I searched online to find front-ends, and then checked to see which of those might be available by default in the versions of Ubuntu I am using.  I came across "Snapper", and installed that on the machines where everything was already going well.

For the machines that were still using ext4, I used the btrfs-convert command to convert them to btrfs format first.  Note that this requires un-mounting the drive, so it is lucky that I had the foresight to make /home a separate partition on all of the machines when I installed them.  Even so, I had to log in as root and then stop all the services that were using the /home mount.  Also note that this process can take over an hour for a 1TB drive on these machines.  (I suppose it depends more on the drive than the processor).  Once the conversion was complete, I updated /etc/fstab and mounted using mount /home to test that everything would work before continuing.  

To install snapper, I used the following command:

apt -y install snapper

Once this was done, I did the following:

snapper create-config /

snapper -c home create-config /home

in cases there the root partition was formatted as ext4, I did the following instead

snapper create-config /home

This means the config is slightly different on the different machines, but this could not be helped at the moment.  (There is no way to convert a mounted ext4 root to btrfs, and I can't unmount it and still get into the machine).  For some of the machines, the root filesystem is on a USB flash drive SD Card, so I could theoretically take them out and perform the conversion on a different machine.  The problem is that even if I could convert it to btrfs, the filesystem type is contained in the boot parameter on the eMMC of the machine, which can only safely be reprogrammed by taking the machine apart to access the serial console.  The OS of the machine is much less important than the data, so that is not something I was prepared to do at the moment.

At some point I will standardize the machines so they all boot exactly the same way.  

Once snapper is installed and the configuration is set up, it can be texted with snapper list, which should show a single entry.

Next, snapper create will create a new snapshot.

I created a file in /home/console in between the two commands to make sure there would be a difference between the snapshots, but it turns out that wasn't necessary, as /var/log has plenty of changes.  

I used snapper status 0..1 to list out the changes, which worked as I had hoped.  

The snapper utility comes with a cron job set up to automatically once per hour, day, week, etc., and it automatically prunes old snapshots.  You can manually delete snapshots if you are running out of space, or change the settings to make sure this happens as you desire automatically.  

Linux is a fantastic operating system, but disaster is just one command away, so I highly recommend you install snapper or another snap-shotting tool if you can.