Skip Navigation

Upgrade vs Reinstall

I'm a generalist SysAdmin. I use Linux when necessary or convenient. I find that when I need to upgrade a specific solution it's often easier to just spin up an entirely new instance and start from scratch. Is this normal or am I doing it wrong? For instance, this morning I'm looking at a Linux VM whose only task is to run Acme.sh to update an SSL cert. I'm currently upgrading the release. When this is done I'll need to upgrade acme.sh. I expect some kind of failure that will require several hours to troubleshoot, at which point I'll give up and start from scratch. I'm wondering if this is my ignorance of Linux or common practice?

13 comments
  • There's nothing ignorant IMO about avoiding headaches, and keeping environments new-car-smell fresh.

  • I think yes. In general if you have good setup instructions (preferably automated) then it will be easier to start from scratch. This is because when starting from scratch you need to worry about the new setup. But when upgrading you need to worry about the new setup as well as any cruft that has been carried over from the previous setup. Basically starting clean has some advantages.

    However it is important to make sure that you can go back to the old working state if required. Either via backups or leaving the old machine around working until the new one has been proven to be operational.

    I also really like NixOS for this reason. It means that you can upgrade your system with very little cruft carrying over. Basically it behaves like a clean install every update. But it is easier to roll back if you need to.

  • It depends on what you need to upgrade/do. I usually upgrade stuff, but at the same time I also have templates in case I quickly need to spin up something new.

    If that's the case, I seed the new instance whatever conf files are needed and I am up and running quickly. Consider that in my work environment we rarely use containers (more of a philosophy at this point than a real reason, since we also have a relatively big K8s cluster for big data).

    Linux sysadmin here, for the past 25 years.

  • I'm a sysadmin as well and I consider spinning up a new instance and rebuilding a system from scratch to be an essential part of the backup and recovery process.

    Upgrades are fine, but they can sometimes be risky and over a long enough period of time your system is likely accumulating many changes that are not documented and it can be difficult to know exactly which settings or customizations are important to running your applications. VM snapshots are great but they aren't always portable and they don't solve the problem of accumulating undocumented changes over time.

    Instead if you can reinstall an OS, copy data, apply a config and get things working again then you know exactly what configuration is necessary and when something breaks you can more easily get back to a healthy state.

    Generally these days I use a preseed file for my Linux installs to partition disks, install essential packages, add users and set ssh keys. Then I use Ansible playbooks to deploy a config and install/start applications. If I ever break something that takes longer than 20 minutes to fix I can just reinstall the whole OS and be back up and running, no problem.

  • It depends on the type of machine you're talking about. Pet machines, bare metal or VMs, such as workstations, desktops, laptops are generally upgraded because it takes a while to re-setup everything. Cattle machines such as servers are generally recreated. With that said, creation of such machines typically involves some sort of automation that does the work for you. Setup scripts are the very basic, however configuration as code systems such as Ansible, SaltStack are much preferable. So if I had a VM that runs acme.sh, I'd write an Ansible runbook that creates it from a vanilla OS installation. I stop here for my own infrastructure. When we do this in cloud environments where we need to spin up more than one such VM and quickly, we'd have the OS install and Ansible run in a Jenkins job which builds a VM image that's pushed to the cloud. Then we spin up ready acme.sh VMs from that image which takes seconds.

13 comments