This is an automated archive made by the Lemmit Bot.
The original was posted on /r/maliciouscompliance by /u/backgroundnerd on 2024-06-21 23:46:12+00:00.
This starts as malicious compliance and ends with Petty Revenge. I posted in both. I hope cross-posting is not an issue here!
It’s a fairly long story but I think I can keep to the essentials. A place I worked at in 2004 had three DBA teams (Data Base Admin). The team I had been on since I joined was tight and ON-call rotation was pretty easy. If anyone got paged for a database that you were the primary on, then you had to buy the team drinks. You rarely got paged. :)
So End of an era. Do you remember “Al Bundy” doing commercials to save big money on long-distance phone calls? That was us! Cell phones and free long-distance put us under.
During the death throws they had massive layoffs and combined 30 DBAs on three teams into one team of 7. I made the cut. HOWEVER, they refused to offline any systems so the workload of 30 remained! My old boss did not make the cut and my new boss was a toxic butt-kisser. I don’t even care anymore-- his name was Steve. Worst boss ever! He would defend ANYONE but someone on his team. I believe he legit thought throwing his people under every bus made him a “team player” Other departments quickly figured out, “Blame them and he will blame his people” so they did!
My first on-call rotation on this new combined team was utter hell! The pager went off every 10 to 15 minutes. Messages like:
Job 104 completed step 5.
Job 110 starting step 12
Job 110 completed step 12.
In case you did not know-
- THESE ARE NOT THINGS YOU PAGE OUT FOR!!
- I have no idea what any of those jobs are, where they are kept or what do about them if they should fail. No idea!
On and on all night long - You can’t ignore them as you might miss a REAL message about an actual database problem! So you get zero sleep and with the diminished staff they expect you there the next day.
When I started ranting the next day the folks left from that particular team said it’s always been that way. That was nuts and we are short staffed so I set out to fix the problem. I looked at every message that got sent to the DBA pager queue and added the prefixes “INFO and WARN” to every message (kept it short for the pager) Then I changed the pager daemon to route anything starting with INFO to e-mail and ONLY route “WARN” to pagers.
Worked like a champ! We went from 50 pages a night to maybe one per night and often none.
One day a team meeting was called and Boss Steve was upset that a print job had failed and our on-call person (me) did not respond.
“Why would I respond to a print job failure? We (database admins) have nothing to with that.”
“When you get a page you respond. Period”
“Why would I get a page that a print job failed?”
“Because we always have, didn’t you get the page?”
“Ohhhh, that, no I fixed that awhile back. DBA’s only get paged for DBA stuff now.”
“WHAT? You need to put that back the way it was!”
Ok, I am cutting a bunch out here for brevity – I said no! The people who were actually responsible should get paged not us-- we ended up in a shouting match! He was yelling he would take me to HR and I told him that worked both ways as he was flat-out abusing us not letting us sleep AND expecting us in at 8:00 AM sharp.
That slowed him down a little and then he calmly said, “Our department must know everything that happens overnight so we can answer for it in the morning, make sure we get the pages!
Oh the evil light bulb went off. I smiled and said, “Yes, sir!” Ding! Ding! Ding! “Our department” eh? “We” eh? Okie Dokey then!
I reworked the pager daemon so that all pages went to HIS pager but left the code that only routed WARN to the on-call pager in place. Now WE and OUR DEPARTMENT know everything that happens overnight! Yes, I am so very proud of that one! I slept soundly that night!
The next morning I kept waiting for him to come over, and either tear into me again OR admit it was a horrible night, that he did not grasp how bad it was but now let’s get a better fix in place. I was disappointed-- He said NOTHING to me at all. Did he get all those pages? I double-checked that everything was routed to his pager and it was. He just would not admit he was wrong! I was impressed! I thought he was tough (but stupid) getting all those pages every night and never saying a word!
END OF MALICIOUS COMPLIANCE
Start petty revenge. :)
A few weeks later I was doing an upgrade outage overnight. We were getting to a critical point and needed a decision to roll back or continue and exceed the outage window. I paged him several times. No answer! We all sleep through pages from time to time so I rolled back and e-mailed all concerned. No big. But he NEVER got back to me. That was odd. The next Monday I mentioned it to the team and several folks said yeah, he does not answer pages after hours anymore!
AAAAAA-HA! He was not tough! He just started turning his pager off before bed! Still too chicken to admit he was wrong.
A few weeks later I was on call again and late Sunday night I got paged to come in immediately! To keep it brief the system that went down was a federally mandated system with severe financial penalties for downtime beyond a limit. We were reaching that limit! It was a huge amount of money! All hands on deck! Even the CEO and CIO were there. Developers, sys admin, network, security and DBA with their bosses (with one exception) were all on laptops in the conference room. This was BIG! Thank god it was not a DB problem – It turned out to be network. The CIO said he appreciated us and wanted all there JUST in case. No problem… sometimes that is the life we choose! :)
At about 5 am Monday after a long night, the CIO says to me, “Hey, I have paged your boss like a dozen times -- he is not answering”
OH GLORIOUS DAY! “With a shrug, I very casually say, “Oh yeah, yeah, Steve **never** answers pages at night.”
He said nothing out loud but I saw his face cloud up.
About 7:30 issues are resolved and we are watching transactions flow across to the feds. I am watching the DB to make sure the huge backlog is not gumming it up but I am just watching as all is well.
At about 7:45 Ole Steve comes strutting in the door, fresh-faced with a full night's sleep! As he walks past the full conference room he does a double take and stops. When he sees all the heavy hitters (and us plebes) in the room, his face falls, he knew right then he screwed up!
The CIO Says, loud and mean, "Where have you been?!" Then the CIO and CEO take him out of the room. I have no idea what was said but I bet it was not pretty! lol
The CIO and CEO come back in about 10 minutes. They heartily thanked us all for the dedication then sent us home and said don’t come back until tomorrow! Good leaders!
I came in the next day and no Steve. I never saw him again!
I sure am glad I kept that short! :)