Unlocking the Secrets of the Veeam Hardened Repository Part 2
In my previous blog, I discussed the Veeam Hardened Repository, why we need it, its uses, and its limitations. Today, I want to discuss this subject in more detail while providing links to build instructions with commentary. There are numerous ways to build a Veeam Hardened Repository, and one must choose carefully which method to use, as any errors or miscalculations will be very difficult to solve further down the road.
It is also important to remember that building a Veeam Hardened Repository is not necessarily that difficult a task, especially if you follow the instructions provided by Veeam in their user guide or guides offered by various bloggers and community experts. The difficulties and serious responsibilities come later, on day 2. Problems can arise if you do not constantly monitor and patch your repository to avoid security vulnerabilities and other issues. I will discuss those problems and possible solutions in the third part of my blog series.
Planning
It is important to note that the Veeam Hardened Repository should be a physical server. While creating a virtual VHR is okay for testing, it defeats the purpose of hardening if used in production because the VM is vulnerable to the virtualization layer. Anyone with administrative privileges can easily delete the virtual machine or modify/encrypt the data, thus eliminating any OS hardening that was present.
It is of the utmost importance that the physical server is in a secure and monitored environment. If a bad actor somehow gets physical access to your server, they can destroy it or even boot from a flash drive and logically destroy the disks and information contained in them.
Hardware
The next question that often arises is, “What type of server should be purchased, and what will be the physical design?” This question depends on many factors, from who your preferred vendor is to what server technology you are most familiar with. The answer is to go with what you know and trust. Remember to plan to use RAID (either HW or SW). For example, a simple solution would be to leverage DAS and Raid 6 or 60.
Sizing
When it comes to sizing, the most important consideration is immutability. Unlike other repositories, the Veeam Hardened repository’s immutable storage sizing can be tricky. There is minimal room for errors since your backup data is unchangeable, and you can’t delete it. Of course, on a Veeam Hardened repository, nothing is stopping you from logging in as root and applying imperative commands to remove the immutability (or even format the drive, for that matter), but this goes against the whole principle of having a secure hardened repository in the first place.
Building
Many instruction sets are now available to help you build a Veeam Hardened Repository. Faced with many choices, one must decide which method to use based on a few critical considerations. First and foremost, you must leverage an operating system you are familiar with. This is not an area for experimentation or trial-and-error learning. Veeam recommends leveraging Ubuntu 20.04, and they provide a step-by-step instruction set in their user guide.
On a side note, they also mention leveraging the repository as a VMware proxy in Network mode. Still, I would stick to limiting the role to that solely of a Veeam repository. It is important to remember that any additional roles will increase the attack surface of the VHR. The concept of server hardening is based on minimalism. The less there is to attack, the better.
There are many other guides and blogs on setting up a VHR (also known as a Linux Hardened Repository), and I recommend looking through them as well. Some may be older, and Veeam is constantly evolving, so I would nevertheless use the official Veeam documentation as the preliminary point of reference.
Veeam Vanguard Didier Van Hoye has done a deep dive setup blog.
He also has a more recent YouTube video available.
If you intend to leverage Red Hat Linux as your Server OS, Marco Escobar created a blog on that subject a few years ago.
Another excellent source of up-to-date information is the Veeam R&D forums. Make sure to sign up and follow the threads. Veeam Project Managers closely monitor these forums and can also be the source of information concerning new features.
The Veeam Hardened Repository ISO
Veeam has also tried simplifying the VHR build process by providing its own Hardened repository ISO. The ISO has all the desired hardening settings baked into it by default. The idea is to help users avoid any errors during setup, which could unintentionally expose the OS and repository to attack.
Hannes Kasparick, Senior Analyst in Product Management at Veeam, has provided excellent step-by-step setup instructions using the download link here.
Rick Vanover, Senior Director of Product Strategy for Veeam Software, has a post dedicated to the VHR and promises some critical updates at VeeamON 2024.
He also has a markdown file on GitHub with essential details on the VHR ISO.
Timothy Dewin, Veeam Solutions Architect, created a script that could be modified or used as a baseline. Again, it is essential to remember to check the dates of all the instructions and scripts you find on the internet concerning the VHR and check to make sure they are being kept up to date.
Veeam’s instructions are for Ubuntu 20.04, but if you prefer to be more bleeding-edge, Eric Henry of Epic IT has a blog with instructions for creating a VHR with Ubuntu 22.04 on HPE hardware.
Locking down
A hardened Veeam repository is hardened because it has been locked down. So, what does lock down a Linux server entail?
- MFA and strong passwords
- Disable SSH
(if you need to periodically enable SSH for updating software or firmware, then leverage SSH Keys instead of passwords, disabling the latter) - Update the System regularly and apply emergency security patches ad hoc
- Install the bare minimum software packages necessary for a repository to function.
- Disable the Root account.
- Turn on the firewall and allow only the ports needed for Veeam.
- Leverage AppArmor or SELinux
Make sure to follow the DISA-STIG, which is available for Ubuntu 20.04. Veeam has also listed what sections of this apply to the Veeam Hardened Repository and how to apply them:
Linux, but which Linux?
There are many different flavors of Linux, or if stated more correctly, distributions of Linux. This creates a challenge for Vendors and users alike. The distribution of Linux that a company uses or has expertise in might differ from the version contained in Veeam’s official documentation. All these factors must be weighed up when building your hardened repository.
Veeam’s documentation leverages Ubuntu, but from a technical point of view, you can use Red Hat, Oracle, or any other distribution that supports the XFS file system.
Conclusion
The Veeam Hardened Repository is an excellent solution for organizations with the appropriate quantity of qualified staff to maintain it. There are numerous ways to build the VHR; in each case, organizations should leverage what they are most comfortable with economically and resource-wise. Setting up the VHR is only the first step and perhaps the easiest. What comes next is critical when protecting your properly backed up data. Even a well-built VHR will become vulnerable if not adequately monitored and updated. In the next part of this blog series, I will review these Day 2 tasks and stress the importance of each.