How to Rollback a Failed Proxmox VE Upgrade

Understanding Rollback Limitations

Rolling back a Proxmox VE upgrade is not officially supported by Proxmox and should be treated as an emergency measure. Unlike upgrading, which follows a well-tested path, downgrading involves reversing hundreds of package changes and can introduce subtle inconsistencies. The best rollback strategy is one you prepared before the upgrade: snapshots, backups, and tested recovery procedures. If you did not prepare these in advance, your options are more limited but not hopeless.

Option 1: Restore from a ZFS or LVM Snapshot

If you created a root filesystem snapshot before the upgrade, this is by far the cleanest and fastest rollback method. It restores the entire system to its exact pre-upgrade state.

# For ZFS root - check available snapshots
zfs list -t snapshot | grep rpool

# Roll back to the pre-upgrade snapshot
# WARNING: This destroys all changes made after the snapshot
zfs rollback rpool/ROOT/pve-1@pre-pve8-upgrade

# For LVM root - check available snapshots
lvs | grep snap

# Merge LVM snapshot back (requires reboot)
lvconvert --merge /dev/pve/root-snap

# Reboot to complete the rollback
reboot

For ZFS, if the snapshot has child snapshots (created automatically during the upgrade), you need the -r flag to recursively destroy them:

# Force recursive rollback
zfs rollback -r rpool/ROOT/pve-1@pre-pve8-upgrade

After restoring the snapshot, verify the system is running the pre-upgrade versions and that all services start correctly.

Option 2: Boot the Previous Kernel

If the system fails to boot with the new kernel but the old kernel packages are still installed, you can select the previous kernel at the boot menu. This does not undo the userspace upgrade but can get the system bootable.

For GRUB-based systems, hold Shift during boot (BIOS) or press Escape (EFI) to access the GRUB menu. Select "Advanced options for Proxmox VE" and choose the older kernel version.

For systemd-boot (common with ZFS root), press Space or a key during early boot to access the boot menu and select the older kernel entry.

# Once booted on the old kernel, verify
uname -r

# Make the old kernel the default boot option for GRUB
# Edit /etc/default/grub and set:
#   GRUB_DEFAULT="Advanced options for Proxmox VE>Proxmox VE GNU/Linux, with Linux 6.5.x-x-pve"
update-grub

# For systemd-boot, set the default entry
proxmox-boot-tool kernel pin 6.5.13-5-pve
proxmox-boot-tool refresh

Option 3: APT Version Pinning to Downgrade Packages

If the system boots but specific packages are causing problems, you can attempt to downgrade individual packages using APT. This only works if the older package versions are still available in a repository.

# Re-add the old version repositories temporarily
# Example: add back Bullseye repos for a PVE 7 rollback
echo "deb http://deb.debian.org/debian bullseye main contrib" > /etc/apt/sources.list.d/rollback.list

# Update package index
apt update

# Install a specific older version of a package
apt install pve-manager=7.4-17 proxmox-ve=7.4-2

# Pin packages to prevent re-upgrade
cat > /etc/apt/preferences.d/rollback-pin << 'EOF'
Package: pve-manager
Pin: version 7.4*
Pin-Priority: 1001
EOF

# After rollback is complete, remove the old repo
rm /etc/apt/sources.list.d/rollback.list
apt update

Be aware that selectively downgrading packages can create dependency conflicts. APT will warn you about broken dependencies. If the dependency chain is too complex, this approach may not be viable.

Option 4: Manual dpkg Rollback

In extreme cases, you can use dpkg directly to force-install older package versions from cached .deb files. APT caches downloaded packages in /var/cache/apt/archives/, and if the cache was not cleared, older versions may still be available.

# Check the APT cache for older package versions
ls /var/cache/apt/archives/pve-manager*
ls /var/cache/apt/archives/proxmox-ve*

# Force install an older version
dpkg --force-downgrade -i /var/cache/apt/archives/pve-manager_7.4-17_amd64.deb

# Fix any broken dependencies afterward
apt --fix-broken install

This method is fragile and should be used only when other options have been exhausted. Each forced downgrade can break dependencies that cascade into further issues.

Option 5: Emergency Rescue Mode

If the system does not boot at all, use the Proxmox VE installer ISO to access a rescue environment:

# Boot from Proxmox VE installer ISO
# Select "Rescue Boot" or open a shell from the installer

# For ZFS root systems:
zpool import -f -R /mnt rpool
zfs mount rpool/ROOT/pve-1

# For ext4/LVM root systems:
mount /dev/pve/root /mnt
mount /dev/sda2 /mnt/boot  # adjust device as needed

# Bind mount required filesystems
mount --rbind /dev /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys /mnt/sys

# Chroot into the installed system
chroot /mnt /bin/bash

# From here, you can:
# - Roll back a ZFS snapshot
# - Install/remove kernel packages
# - Fix GRUB configuration
# - Rebuild initramfs
# - Edit network configuration

# When done:
exit
umount -R /mnt
reboot

When Rollback Is Not Possible

In some situations, a rollback is not feasible:

• ZFS pool was upgraded with zpool upgrade - feature flags cannot be reverted, and older ZFS versions cannot import the pool
• Ceph was upgraded and new on-disk formats were applied - older Ceph versions cannot read the updated OSD data
• No snapshots or backups exist - there is nothing to roll back to
• Configuration database was migrated - the /etc/pve cluster filesystem format may have changed in ways that older versions cannot read

In these cases, the only path forward is to fix the upgrade rather than roll it back. Identify the specific failing component, consult the Proxmox forums and bug tracker, and address each issue individually. Often what appears to be a catastrophic failed upgrade is actually one or two specific problems that can be resolved.

# Gather diagnostic information for troubleshooting
pveversion -v
uname -r
systemctl --failed
journalctl -p err -b --no-pager > /root/upgrade-errors.log
dmesg | grep -i error > /root/dmesg-errors.log

The most effective safeguard against needing a rollback is thorough preparation. Always snapshot before upgrading, always run the upgrade checker, and always have remote console access ready. Tools like ProxmoxR can help you verify your environment is healthy before attempting upgrades, reducing the likelihood of needing emergency rollback procedures in the first place.