One Reminder, One Long Day: Forum Upgrade, File Cleanup, and a Network Card That Vanished

One Reminder, One Long Day: Forum Upgrade, File Cleanup, and a Network Card That Vanished

It started with a simple reminder: the forum software needed updating. It turned into a full day of server housekeeping. Here's what I ran into and the commands I used. Anything in ALL CAPS is a placeholder, so swap in your own values. Back up before you change anything.

1. The upgrade that always failed

The one-click Invision upgrade never worked for me, so I'd been uploading updates by hand. The cause was file ownership. Years of unpacking files as the administrator left hundreds of them owned by the wrong user, and the site runs as a normal account that couldn't overwrite them. Check for it:

ls -ld /PATH/TO/FORUM

find /PATH/TO/FORUM ! -user YOUR_SITE_USER | wc -l

If that count isn't 0, fix the ownership and look for world-writable (777) folders:

sudo chown -R YOUR_SITE_USER:YOUR_SITE_USER /PATH/TO/FORUM

find /PATH/TO/FORUM -type d -perm -0002 | head

find /PATH/TO/FORUM -type d -perm -0002 -exec chmod 755 {} \;

After that, the one-click upgrade ran clean. Test a photo upload afterward.

Lesson: when an auto-updater fails, check file ownership before blaming the firewall.

2. Cleaning personal files off the server

Photos, music, and documents had piled up where they didn't belong. Before deleting anything I made sure it was all safely on the desktop. First, check that no sync program (like Syncthing) is set up on the server, because deleting synced files can delete them everywhere:

sudo find /home -maxdepth 5 -name config.xml -path '*syncthing*'

sudo grep -hE '<folder |<device id' /PATH/TO/config.xml | cut -c1-200

Then run a dry run (-n changes nothing) to see what the desktop is missing, and copy only those files:

rsync -rn --ignore-existing --exclude='.stfolder*' --out-format='%n' YOUR_USER@SERVER_IP:/SERVER/PHOTOS/ /DESKTOP/PHOTOS/ > /tmp/missing.txt

wc -l /tmp/missing.txt

rsync -av --ignore-existing --files-from=/tmp/missing.txt YOUR_USER@SERVER_IP:/SERVER/PHOTOS/ /DESKTOP/PHOTOS/

For music and documents, I compared filenames anywhere on the desktop:

export LC_ALL=C

ssh YOUR_USER@SERVER_IP 'cd /SERVER/HOME && find music documents -type f -printf "%f\n"' | sort -u > /tmp/srv-files.txt

find ~ -type f -printf '%f\n' | sort -u > /tmp/desk-files.txt

comm -23 /tmp/srv-files.txt /tmp/desk-files.txt

Finally, don't delete. Move the old copies to a hold folder until everyone has looked things over:

mkdir -p ~/_hold-to-delete-YYYY-MM-DD

mv ~/photos ~/music ~/documents ~/_hold-to-delete-YYYY-MM-DD/

Lesson: verify, copy, hold, then delete.

3. The network card that fell off the bus

A wired network card died and the link showed NO-CARRIER. First, find the card, its driver, and what the log says:

lspci -k | grep -A3 -i ethernet

ip link

sudo dmesg | grep -i YOUR_DRIVER | tail -20

My log said "PCIe link lost," which means reloading the driver won't help. Removing the device from the PCI bus and rescanning brought it back without a reboot (use the address from lspci, like 06:00.0):

echo 1 | sudo tee /sys/bus/pci/devices/0000:BUS:SLOT.F/remove

sleep 2

echo 1 | sudo tee /sys/bus/pci/rescan

ip link

Confirm it has an address, and that Wake-on-LAN is still on if you rely on it:

ip -4 addr show YOUR_INTERFACE

sudo ethtool YOUR_INTERFACE | grep -iE 'wake|link detected'

To make a repeat less likely, I turned off PCIe power-saving (takes effect at the next reboot):

sudo cp /etc/default/grub /etc/default/grub.bak

sudo sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT="\(.*\)"/GRUB_CMDLINE_LINUX_DEFAULT="\1 pcie_aspm=off"/' /etc/default/grub

sudo update-grub

Lesson: read the kernel log first. It tells you which kind of problem you have.

4. When the spam filter eats your login code

I'd tightened the mail filtering to deal with spam, and it then blocked a verification email I needed. I looked in the mail log to see who the real sender was, then whitelisted that domain, but only for mail that passes authentication:

grep -hiE 'SENDER_NAME' /var/log/maillog | tail -15

sudo cp /etc/mail/spamassassin/local.cf /etc/mail/spamassassin/local.cf.bak

echo 'whitelist_auth *@SENDER-DOMAIN.com' | sudo tee -a /etc/mail/spamassassin/local.cf

spamassassin --lint && sudo systemctl restart spamassassin

File locations vary by mail setup, so adjust the paths for yours.

Lesson: tighten the filter, then whitelist the few services you really use.

Questions or similar war stories? Drop them below.