Linux

Debian sid a few years ago: Uninstalled Python2, system became unusable and couldn't neither reinstall from APT neither recompile it

When Ubuntu 16.04 had just been released, I tried upgrading my 14.04, the whole system broke and I had to install another os (Manjaro won).

That day I learned Ubuntu too can be a bit stupid.

It was my first time using a Linux GUI. I was comfortable with CLI, but it was my first time having it installed on a laptop instead of just sshing into a server somewhere.

So naturally, instead of learning how the GUI worked, I tried changing it to be exactly like Windows. I was doing things like making it so I could double click shell scripts and other code files and they would run instead of opening them up in an editor. I think you see where this is going, but I sure as hell didn't.

Well, one of my coworkers comes over and asks me to run this code on this device we were developing. We were still in the very early stages of development, we didn't even have git set up, so he brought the code over on a USB stick. I pop it into my laptop. I went to check it once by opening it in an editor by double clicking on it... Only it ran the code that was written for our device on my laptop instead of opening in an editor.

To this day, I have no idea what it did to fuck my laptop so bad. I spent maybe an hour trying to figure out what was wrong, but I was so inexperienced with Linux, that I decided to just reinstall the OS. I had only installed it the day before anyway, so I wasn't losing much.

One day on my main Arch installation I created a container inside a directory, and "booted" into it by using systemd-nspawn. When I was done with it I decided to do a rm -rf / inside the container just to be funny. Then I noticed that my DE on the host froze and I couldn't do anything. Then I realized that systemd-nspawn mounts some important host's directories on the container, and I deleted those when I did the rm -rf /. I didn't lose anything, but it was scary.

So I am sort of an embedded developer, and I like to mess around with weird configurations. So the craziest experiment I did was trying to reflash a rasberry pi from a system running in the pi's RAM. It honestly might have worked, but during the prep work I forgot to resize the filesystem before mucking with the paritions and had to reflash the normal way before I could try again. Ended up just turning it into a pihole instead, but I still learned a lot about pivot_root

One time I rebooted. The system never recovered.

I've literally done the rm -rf / thing. I thought I was in a different subdirectory, but I was in / and did rm -rf .

When it didn't return after half a second, I looked at the command again and hit CTRL+C about 20 times in the span of 3 seconds.

I had to rebuild the install, but luckily didn't lose anything in /home.

Ubuntu GUI/apt fail

Back when I used ubuntu, Unity was stuck with old gnome packages. This meant that the version gnome-terminal packaged with ubuntu (up to at least 18.04) didn't have text reflow on window size changes.

You could add the upstream sources, upgrade the specific text reflow package only, and then disable the sources.

I forgot to disable the sources, or typed dist-upgrade (this happened multiple times...). Broke the whole desktop/lightdm setup with half upgraded packages, and half removed packages (for preparation to install new versions). Way easier to reinstall the os than to disentangle. Unity was a mess then anyway.

Moral: Actually read the package change summaries when doing updates/removes/installs, and [ y/N ] means actually check what the fuck you think you're agreeing to.

BtrFS snapshots for idiots

I've also run automated snapshots on my btrfs partition, then run out of space doing multi-hop system upgrade on fedora (dnf has a plugin that creates a snapshot every time it kicks in.

You can imagine there were many changes happenning per snapshot, and I effectively could have rolled back 4 major fedora versions... Til I ran out of space.

I couldn't get a replacement drive in time, and I had an hour to rebuild my laptop before needing to be on a customer site, so sadly I couldn't preserve my drive for later investigation. My best guess is the high-water-mark was configured incorrectly, and somehow it was able to 'write' data past the extents of the filesystem.

Rollback did work for my home partition, but I had to mount it from another OS to get it to work - so no data loss!

By that time I'd already reinstalled the os to the root partition/subvolume however, so I couldn't determine the exact cause of failure :(

Moral: Snapshots are not backups, and 'working' is not 'tested'

sudo usermod -a cdrom

Forgot the -G and wasn't sudo anymore...

I did recover eventually, but it was not nice.

ran chown -R www-data: ./ from /var instead of /var/www.

Built a new desktop, backed up everything on my old laptop, next step was to format an Arch installer USB. Instead of formatting the USB, I formatted my laptop's /boot partition. No big loss since I had the backup and was done with that old toaster, but oops.

I was new to Linux, I made the not so calculated decision to use manjaro as my daily, deleted xorg to in an attempt to reinstall xorg to then hopefully fix the stuttering. Everything went wrong, no display obviously, /boot/ files where corrupt. I now use arch and am wiser

I ran firejail config or something, which replaces a lot of home directory app files. Not sure if binaries or desktop entries.

But things broke, randomly, screenshots not working, not even inside firefox etc. I reinstalled the system and imported the home folder... and it was there again!

I set up a progressive backup of my home folder... to my home folder. By the time I got home that day it was impossible to log in because there was no room to create a login record. Had to fix that by deleting the backup file using a live CD.

I once did an apt-get upgrade in the middle of when debian testing was recompiling all packages and moving to a new gcc version. I get it, using testing invites stuff like this. But come on, there should at least be a way to warn people beforehand.

@fl42v I have thousands from my early days, but my only recent-ish one was pretty funny.

On an Arch install that hadn't been updated for a while, in a rush, had an app that needed OpenSSL 3. Instead of updating the whole system, I just updated the openssl package.

*Everything* broke immediately. Turns out a lot of stuff depends on openssl. Who knew?

To fix, booted to the arch installer, chrooted into my env, and reverted to the previous version of the package — then updated properly.

I was trying to extract some files from a a Linux image of one of those ARM boards. It was packed into the cpio format, and I had never used the format before. Of course I was trying to extract to a root owned directory and I sudo'ed it. I effed up the command and overwrote all my system directories (/bin, /usr, /lib, etc...). Thankfully I had backed up my system recently and was able to get it working again.

It was only in a container on a Chromebook, but I'll share it anyway. One time, I had installed Android Studio but found it mildly annoying that I got a line when using apt about Android Studio and some error on a certain line of this one file. I believe the file was something related to dpkg, and after changing some things within the file, I seemed to have broken apt. Luckily, I had a backup, but it was a few days old, so I had to reinstall some apps.

Once I succumbed to a proprietary software's allure, post-usage, I felt like a digital pariah! To rid myself of the taint, I wiped my system clean – reinstall time!

I had issues with a new version of glibc that prevented me from working on music in Ardour on Manjaro. I then proceeded to force-downgrade glibc (in the hopes of letting me get back to work) and that broke sudo and some other things, which I found out after rebooting. That was an interesting learning experience. Now I snapshot before I do stupid stuff. :]

sudo rm -r /run/timeshift

Accidentally executed a JPEG (on an NTFS partition) and the shell started going crazy. reboot was not successful =[

I cant remember anymore... Let me explain ... My first computer was with at-the-time-very-new windows xp, using primary for games, after some time it got bloated with stuff so i had to reinstall again and again over time. Then i discovered redhat,centos and debian... I started heavily distro hopping. My passion for software grew to the point that I was installing new software on daily basis, just to explore new things. But nothing seemed stable enough, ubuntu, fedora, sabayon, gentoo, arch... And their derivatives all broke under my fingers to the point that i had to do more fixing than discovering new software, I took it as a challenge and continue. At around the time of university I discovered NixOS, as with any new technology I went head on with it. It took a lot of trial and error since at the time there were no documentation for any of it. I spent months reading the code, but I never gave up, since what I have found was a gem. I found the OS that is resistant to my curiosity, I just cant seem to be able to break it. Now I use NixOS everywhere that I can, even on my work computer. I do not need to reinstall after initial installation. Well... only when hardware fails...

The first time I wanted to try Linux I did by installing elementary OS in dual boot mode (with windows) and everything went well, I played with it a bit and then I returned to Windows..

So, few days after that I realize that I have a lot of space in the Linux partition and I didn't have plans to use it anymore so I go to drive's & partition's manager on windows to delete my elementary OS partition..

Oh Lord when I restarted my PC, grub was showing nonsenses and I couldn't boot on windows again, I was in panic, I spent the rest of the day trying to fix grub to boot windows. At the end of the day I did it and save all my files and I uninstall grub properly, but what a day 😂

I used to work at this place that had a gigantic QNX install. I don't know if QNX that we used back then had any relation to q&x now They certainly don't look very close.

It was in the '90s and they had it set up so that particular nodes handled particular jobs. One node to handle boot images and serve as a net boot provider, one node handled all of the arcnet to ethernet communication, one node handled all the serial to mainframe, a number of the nodes were main worker nodes that collected data and operated machinery and diverters. All of these primary systems were on upper-end 386s or 486s ,they all had local hard disks.

The last class of node they called slave nodes. They were mainly designed for user data ingest, data scanning stations, touch screen terminals, simple things that weren't very high priority.

These nodes could have hard discs in them, and if they did, they would attempt to boot from them saving the net boot server a few cycles.

If for some reason they were unable to boot from their local hard drive, They would netboot format their local hard drive and rewrite their local file system.

If they were on able to rewrite their local file system they could still operate perfectly fine purely off the net boot. The Achilles heel of the system was that you had no idea that they had net booted unless you looked into the log files. If you boot it off your local hard drive of course your root file system would be on your local hard drive. If you had net booted, and it could not rebuild your local file system, your local root file / was actually the literal partition on the boot server. Because of the design of the network boot, nothing looked like it was remotely mounted.

SOP for problems on one of the slave nodes was to wipe the hard disk and reboot, in the process it would format the hard drive and either fix itself or show up as unreliable and you could then replace the disc or just leave the disc out of it. Of course If the local disk had failed and the box had already rebooted off netboot without a technician standing there to witness it, rm -Rf would wipe out the master boot node.

I wasn't the one that wiped it, but I fully understand why the guy did.

Turns out we were on a really old version of QNX, we were kind of a remote warehouse mostly automated. They just shut us down for about a week. Flew a team out. Rebuilt the system from newer software, and setup backups.

I'm not sure how funny this will be, but here's how I broke my system twice in a single case. Step by step:

1. Migrated from Manjaro KDE to EndeavourOS KDE. Kept the previous home directory.
2. After a few updates, there was a problem with Plasma. Applications were not starting from the panels or the .desktop files (they worked from the terminal. The terminal emulator was in startup and worked that way)
3. After a few google searches, found out that downgrading glibc would do something, so downgraded... Worked for a while
4. While using pacman -Syu, I always checked for warnings (foolishly thinking that the downgraded and ignored glibc would cause a pacman warning if it broke dependencies) and there were none. So, the updated OS stopped working due to unmatched glibc. BREAK 1
5. To fix it, I opened one of my multiple boots (another EndeavourOS) and made a script using pacman -Ql and cp to copy new glibc related files into the broken system (because I was too lazy to learn how to do it the correct way with pacman and chroot didn't work because glibc is needed by bash).
6. Turned out the script I made was wrong and I hadn't checked the intermediate output from pacman -Ql, which was telling cp to copy the whole /etc /usr and other directories. (just if I hadn't given the -r to cp) BREAK 2

In the end, I just made a new installation, this time with a new home and hand-picked whatever settings I wanted from the previous home, Viva la multi-HDD

A regular update I guess...