Recovering a PBS Datastore After a Failed Disk Expand
After expanding a virtual disk attached to a Proxmox Backup Server, the datastore wouldn’t mount on reboot. The partition table looked fine, the disk was visible, but the filesystem was gone as far as the OS was concerned. This is what recovery looked like.
This covers diagnosing a missing filesystem after a disk expand, recovering a corrupted ext4 superblock, and getting the datastore back online without losing any backup data.
The Goal
- Identify why the datastore won’t mount after a disk expand
- Recover the ext4 filesystem without data loss
- Re-mount and make it persistent across reboots
- Understand what went wrong and how to avoid it next time
Part 1 - Diagnose the Mount Failure
The first sign of trouble is a failed mount with a generic error:
1
2
mount /dev/sdb1 /mnt/datastore/Backups
# mount: /mnt/datastore/Backups: wrong fs type, bad option, bad superblock on /dev/sdb1
Start by checking what the kernel and filesystem tools can actually see:
1
2
3
blkid /dev/sdb1
lsblk -f /dev/sdb
dmesg | tail -30
If blkid returns PTTYPE="PMBR" instead of a filesystem type, the partition is being misread. If lsblk -f shows no FSTYPE for your partition, the filesystem signature is missing or unreadable.
Check the datastore config to confirm the expected path:
1
cat /etc/proxmox-backup/datastore.cfg
Part 2 - Inspect the Partition Table
Confirm the GPT partition table is intact and the partition covers the right range:
1
gdisk -l /dev/sdb
If the output shows the partition spanning the full disk with code 8300 (Linux filesystem), the partition itself is fine. The problem is the filesystem inside it.
Also check wipefs to see what signatures are present:
1
2
wipefs /dev/sdb
wipefs /dev/sdb1
A PMBR entry on /dev/sdb1 at offset 0x1fe is a red flag. This is a leftover artifact from the disk expand that can sit right over the area where the filesystem superblock flags need to be written, which blocks repair tools from completing.
Part 3 - Confirm the Superblock is Intact
Before doing anything destructive, check whether the primary ext4 superblock is still readable:
1
dd if=/dev/sdb1 bs=1 skip=1024 count=256 2>/dev/null | hexdump -C
Look for 53 ef in the output around offset 0x37. That’s the ext4 magic number. If it’s there, the superblock survived and recovery is straightforward. You should also see your mount path embedded in the hex output as plain text, something like /mnt/datastore/Backups, which confirms this is the right partition.
Also check what backup superblock locations are available (this is a dry run, nothing is written):
1
2
mke2fs -n /dev/sdb1
# Proceed anyway? y
This prints a list of backup superblock locations that e2fsck can use if the primary is damaged.
Part 4 - Clear the PMBR Artifact
If wipefs /dev/sdb1 showed a PMBR entry at 0x1fe, remove it before attempting a filesystem repair. This signature causes e2fsck to fail with “unable to set superblock flags” even when the superblock itself is readable.
1
wipefs --force -o 0x1fe /dev/sdb1
The --force flag is required because wipefs normally refuses to touch signatures on non-whole-disk devices. This only removes the two bytes at that offset and leaves everything else untouched.
Part 5 - Repair the Filesystem
Run e2fsck using a backup superblock. The first backup at block 32768 is the standard starting point:
1
e2fsck -b 32768 -f -y /dev/sdb1
On a large partition (7-8 TiB) this will take a while. Pass 1 alone can run for an hour or more as it checks every inode. Let it finish.
You will likely see output like this as it works through the repair:
1
2
3
4
5
6
7
8
9
10
11
12
13
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free inodes count wrong for group #30080 (8192, counted=8075).
Fix? yes
...
Free inodes count wrong (326634545, counted=325990453).
Fix? yes
Padding at end of inode bitmap is not set. Fix? yes
/dev/sdb1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdb1: 1689547/327680000 files (1.5% non-contiguous), 1250423031/1310719739 blocks
Inode count mismatches and bitmap padding errors are expected after an unclean shutdown during a disk expand. These are metadata inconsistencies, not data loss. Once e2fsck completes without the “Filesystem still has errors” warning, the filesystem is clean.
Part 6 - Mount and Make Persistent
Mount the recovered filesystem:
1
mount /dev/sdb1 /mnt/datastore/Backups
Confirm the data is there:
1
2
ls /mnt/datastore/Backups
df -h /mnt/datastore/Backups
Get the UUID for a stable fstab entry:
1
blkid /dev/sdb1
Add it to /etc/fstab:
1
echo "UUID=your-uuid-here /mnt/datastore/Backups ext4 defaults 0 2" >> /etc/fstab
Test the fstab entry before rebooting:
1
mount -a
Then restart the PBS service so it picks up the datastore:
1
systemctl restart proxmox-backup
Part 7 - Expand the Filesystem
The partition was already resized during the original disk expand, but the filesystem inside it never grew to match. Do that now:
1
2
3
4
umount /mnt/datastore/Backups
resize2fs /dev/sdb1
mount /mnt/datastore/Backups
df -h /mnt/datastore/Backups
resize2fs with no size argument automatically fills the partition. The datastore should now report the full expanded capacity in the PBS UI under Datastore, then Summary.