The hardest part of running the Columbia Interlocking Services company was figuring out how to get paid when we were acting as sub-contractors for one of our own competitors. We had tried everything from cash under the table to IOU's but after a few broken promises that led to bad feelings we ended up using our friend Daryl the Dispatcher to keep track of hours and jobs.
That's how we got the call from Daryl saying there was an easy chance to make up for some of the hours we owed to Northwest . It wasn't always us saving them and we recently needed their help on a big job not to mention always borrowing their van for the big stuff like that Cisco 9300 switch that wouldn't fit either one of our cars.
Northwest was the original contractor who then called in the guys from Solution Integrators who together spent the first 4 days gathering data and by the end of the week they were calling us in desperation, specifically Charlie. Luckily we were already done working for the day and promised to come over after lunch. Charlie wanted to go back to the All You can eat buffet to fill his plate with crawdads but we had been there twice in a row already and I insisted on the Burnt Bun burger joint where you can build your own creation, my favorite is guacamole bacon burger with no cheese.
We arrive at the customer site, a service company called Timberlodge, along with a rush of workers returning from lunch as we were escorted through the security system to a conference room where all us consultants were gathered to finally get to the bottom of the issue. I was hoping to break the ice with their main guy who had a big tag on his shirt that said he is the Computer Manager whose name is MAURICE so I gave him my best hand shake along with a "Glad to meet you 'Moreece'," but right away he says "That's how they say it in France but where I comes from it's 'Morris". I made sure to correct myself out loud and made a silent promise to call everyone Buddy.
In the meeting room are all the documents collected showing the logs and reboot times. There are 150 users on these 20 servers providing remote services for every department across the entire campus. The reboots were random and hit all the servers at various different times with only one recurring pattern every day between 2:10 and 2:20pm. The logs were not much help but so far all of the Analysts were blaming the servers but not Charlie. Right away he was more interested in the users.
There were two groups preparing for the coming reboot at 2:15pm. Half the team headed for the server room to see if they can notice some coincidence with the AC or power while the rest will stay in the Crisis Room watching the server consoles to see if the verbose logging they setup might isolate the issue.
Those were good ideas but not for Charlie, he went straight out to the user cube farm and started talking with the area Leaders. He found that half the users come back from lunch at 1Pm and the others at 2pm. Following a hunch he went to the printing station located at the center of the area where the 2pm lunch crowd would have just returned. I asked him what he was up to and he said, "If the answer was in the server room they would have found it by now."
Then at the appointed time Charlie raised his finger in triumph pointing at the sound of a distant user shouting out in frustration; "Crap, every time I print, I loose my server session." Following the sound we found the user coming towards us and the printers. We introduced ourselves to Sharon who had long suffered this problem and even filed a ticket once but had since learned to live with it.
Sharon always knew it was weird that she had to start a new server session so often but since the print jobs did go through successfully no one ever made the connection between printing and server crashing until Charlie heard her print. It turns out she has an old Unix workstation that uses a malformed print command that caused a panic in the process on the other end.
We never did fix the issue, we just got her to print from another machine. The guys from Northwest were so pleased they offered to take us out for beers so we had to go and listen how they had done all the real work. I kept trying to get Charlie the credit he deserved but every time he told the story the most important part was "How that girl cussed!"
No comments:
Post a Comment