Saturday 02 Sep 2023

Checking File Integrity

Image copyright

I wanted the ability to check file integrity within a directory structure; this would be useful should I ever need to restore from the cloud or test that an external hard drive is telling the truth (help detect bitrot). As I have thousands of pictures doing this check one-by-one would be a pain.

MD5Deep is an excellent tool I can use to check every file within a directory structure. As my picture directory does not change, it was a great starting place to test this script. The application will check each time, generate a checksum, and store within a defined text file. You can choose different hashing methods. In my case I used SHA256.

Let's get started.

Install the hashing tools:

sudo apt install md5deep

Once installed, let's get hashing. This may take some time if you are hashing a large directory.

sha256deep -r /foo/bar > hash.sha256

The -r switch is recursive so will follow directories. You can store the hash file whenever you want, such as ~/Documents/hashes/pictures.sha256

Great, you now have a list of file hashes to play with. Back these up and you're good.

Once you are ready to check the directory against your hash file, use this command:

sha256deep -X hash.sha256 -r /foo/bar

If you want to watch progress, add -e to the end. However, because I was hashing pictures and short videos, the progress hashed so quickly this was not helpful, unless hashing large files.

Now that I have a list of hashes, I can be more confident my working hard drive is not failing and corrupting files. I can run this check monthly like my own "scrub" tool. Should I need to restore files I will know whether they are damaged.