Friday 24 Nov 2023

Monitor Directories for Missing or Damaged Files

Image copyright

I haven't written on my journal for the entire month. This is with good reason. I have been learning Golang (Go) -- a C-like programming language. The best way to learn a new language is to solve a problem. I couldn't think of anything new, so a rewrite of my python script - ArchiveGuardian - was in order with a few differences.

Introducing DEFIANT-FG

DirEctory FIle hAsh moNiTor (DEFIANT) File Guard, or DEFIANT-FG, is a Linux command-line utility to monitor a directory. You point to a directory, run the program then FG will producing a hash database that can be queried again at a later date to check for corrupted files or report missing files.

This program keeps your drives honest - no matter what the drive reports if the hash doesn't match the file has been changed. There is a potential problem of bit rot whereby one bit, e.g. a 1 or 0, may get changed which, depending where the bit it, may destroy a photograph.

DEFIANT-FG is open source and I welcome contribution. Visit my GitHub page for the source and binaries.

How I use?

I have pictures on my backup NAS running ZFS. I know that the data is good - as ZFS would report problems during access or scrubbing. So this is my source of truth. I run FG on this directory to get a database. Then, with the database, I put this on my working drive (attached to laptop) and on backup drives (including cloud). Now I can run FG on my laptop and know my attached drive is returning intact files. I think this is a great method to monitor for problems. Using FG has given me the confidence to maintain backups on ext4 hard drives and in the cloud. Although FG cannot scan the cloud, I can test restore and make sure the files have downloaded correctly.

If you want to build from source (this is easy) to run on your own hardware, follow this build guide.

Roadmap

• I would like to build a graphical interface
• Faster SQL writes (many thousands of files can take a few hours to write)
• Run on Windows (SQL plugin does not work on Windows)
• More control with additional flags, such as skip certain tasks
• Automatic database backup