We were standing around the warming fire during our 7:30Am Safety Meeting at Milepost 568 between Black Butte and Edgewood when we got a call to stop tamping ties in the Montague yard and get on over to the branch line to help the nearby section gang clear a slow order. We hated tamping by hand and were glad to go but we got there only to find out the slow order was a curve full of low joints that had to be tamped by hand.
Mike Anzo was the Section Foreman on the gang in Mt. Shasta and I had the one in Weed so it was natural for us to end up competing to be the best new Foreman in our District. We worked together on urgent issues and this slow-order limiting all trains to 5 Mph had to be removed today or else. We had to wait for a train to go by and even at that slow speed you could see the boxcars rock and roll as they hit the low joints.
We were both young whippersnappers in our early 20's from two different schools, his Dad was a cat-skinner and they lived in company housing since the day he was born where I was a new comer from the system steel gang. I was schooled by the book and he did things off the cuff in the old school way. I was the expert when it came to installing new rails or ties but when it came to maintaining the existing track structure I still needed some schooling and Mike was pleased to show off.
The first point of discussion was replacing the old cinder ballast with new heavy grade gravel. I assumed since it was cinder to begin with we had to keep that standard but Mike had his Dad haul in several truck loads of gravel that would fix this curve for good. This made the guys even madder because we had been tamping red cinder rock which is light as sugar compared to the heavy gage gravel which was so much harder to tamp we would have to use tamping bars instead of shovels and that meant a whole new set of sore muscles for that night.
Then we argued about the elevation of the hi-side rail. The official ballast chart said 1 inch maximum but the book of rules said a curve that sharp needed 2 inches. I was all for leaving things the way they were until we got the official order but Mike was more than willing to change things on his own authority. Then next point of contention had to do with the rail joints. The standard was to tighten the bolts first and then do the tamping but Mike insisted on raising first and then tightening. I didn't even argue that one since it was not mentioned in the book of rules and it seemed to make sense.
We broke for lunch and both gangs were still chowing down when the 30 minute lunch ended and I was calling for everyone to get back to work but Mike told me to take it easy even though it was clear we were no where near half way done. I already knew Mike made lots of money working overtime and assumed this would be another case but then his Dad showed up unexpectedly again this time with a full sized tamping machine that would make it easy for us to finish early without breaking our backs.
The next item that came up did get us to arguing about how high we should hump the joints. The old guys had shown me how to go to level with the track jack and then a notch or two at most above level but good old Mike liked to really hump the joints higher than I ever would. He figured the bump after should be the same size as the dip before since the ballast needed to get packed down by the train. We settled on "sky high" as the upper limit.
We watched another train rocking and rolling over the same track after our work and there was no difference from this morning maybe even worse, but sure enough a week later it was smoother than ever. I started out all mad and insulted for being shown up so many times in one day but I couldn't bring myself to keep it up since I was having so much fun learning new tricks. I had to admit Mike really did know what he was doing.
Thursday, July 7, 2011
Pieces of Fight
No two fights are ever exactly alike but they all have certain things in common. These then are the various parts and pieces necessary to carry on a decent fight. Whether it be the Worlds Nations at war or a married couple at the dinner table every fight will always contain some of these required components.
The sequence is not absolute but generally follows the usual pattern. Some contestants will always jump straight to their favorite device and stick with it to the bitter end while others think the secret is covering as many of the steps as possible. Some fights go straight to the end and some get stuck in the middle in a repeating circle. Once the gloves come off all bets are equally off along with the rules.
The understanding...
They say you only hurt the one you love and so you can only fight with someone with whom you have an understanding of some sort from which there can be an event that will lead to the fight.
The misunderstanding...
The fight begins with one party acting as the victim of some wrong to which they must respond. Even if both parties already know in advance what the other will do it still feels good to act all insulted and surprised.
The mistake…
In every argument there is always something said that is soon regretted but instead of stopping to apologize we invariably ratchet up the heat with another shot across the bow. The mistake and the misunderstanding are often the same thing.
Defending the mistake…
So we defend the mistake no matter how ridiculous and that leads us down a pitiful path on the way to the next steps on the list. Normally we would not defend such a silly mistake but that would be a normal conversation. During a fight we are required to defend any mistake, that's what makes it a fight.
Shifting the blame…
It starts with denial and then gets worse through the series of sub-distractions that make up the blame shifting category including the top two; sarcasm and insult. Admit it, you know we all do it. Telling an even bigger lie is an excellent way to change the subject.
Loud Voice…
Each person has their quiet mean voice followed by their Loud Voice. Usually that's enough because no one wants to resort to the dreaded Louder Voice which leaves no other choice but the Loudest Voice and after that there are no more weapon left in the arsenal and that means:
The Blowout…
Sooner or later one or both parties give up and move on. There needs to be a dramatic end or else it feels like there never really was a fight, so this is one of the required steps. Once the dust settles from the blowout it’s time for:
Licking wounds…
Which tastes bad so we rush to:
Reconciliation…
If there is no makeup then there will be no more fights. Most fighters would rather keep their option for another engagement down the road.
Unfortunately, having this parts list doesn't make you any better at fighting but at least it gives you a program guide so you can tell where you are if you get lost during your next fight.
The sequence is not absolute but generally follows the usual pattern. Some contestants will always jump straight to their favorite device and stick with it to the bitter end while others think the secret is covering as many of the steps as possible. Some fights go straight to the end and some get stuck in the middle in a repeating circle. Once the gloves come off all bets are equally off along with the rules.
The understanding...
They say you only hurt the one you love and so you can only fight with someone with whom you have an understanding of some sort from which there can be an event that will lead to the fight.
The misunderstanding...
The fight begins with one party acting as the victim of some wrong to which they must respond. Even if both parties already know in advance what the other will do it still feels good to act all insulted and surprised.
The mistake…
In every argument there is always something said that is soon regretted but instead of stopping to apologize we invariably ratchet up the heat with another shot across the bow. The mistake and the misunderstanding are often the same thing.
Defending the mistake…
So we defend the mistake no matter how ridiculous and that leads us down a pitiful path on the way to the next steps on the list. Normally we would not defend such a silly mistake but that would be a normal conversation. During a fight we are required to defend any mistake, that's what makes it a fight.
Shifting the blame…
It starts with denial and then gets worse through the series of sub-distractions that make up the blame shifting category including the top two; sarcasm and insult. Admit it, you know we all do it. Telling an even bigger lie is an excellent way to change the subject.
Loud Voice…
Each person has their quiet mean voice followed by their Loud Voice. Usually that's enough because no one wants to resort to the dreaded Louder Voice which leaves no other choice but the Loudest Voice and after that there are no more weapon left in the arsenal and that means:
The Blowout…
Sooner or later one or both parties give up and move on. There needs to be a dramatic end or else it feels like there never really was a fight, so this is one of the required steps. Once the dust settles from the blowout it’s time for:
Licking wounds…
Which tastes bad so we rush to:
Reconciliation…
If there is no makeup then there will be no more fights. Most fighters would rather keep their option for another engagement down the road.
Unfortunately, having this parts list doesn't make you any better at fighting but at least it gives you a program guide so you can tell where you are if you get lost during your next fight.
Railroad Days: Snow Removal - Day 1
Snow Removal was a big deal for the gang assigned to Fields Station between Oakridge and Crescent Lake in Central Oregon. Fields is at the end of a steep 7 mile long dirt road that can only be called a thrill ride. The Company provided free housing to the workers willing to put up with the commute. There had already been a few heavy storms in October and November 1988 but the snow had all melted away before long. The storm that started in early December just kept going and soon the snow and ice were piling up.
Elevation makes all the difference when it comes to snow. Oakridge at 1500 feet hardly ever gets more than a few inches at a time and it always soon melts. Fields is at 2500 feet where the snow starts sooner and sticks around longer than the lower elevations. The summit at Odell Lake is just under 5000 feet and the snow gets deep and stays deep.
For the last few weeks we had been cleaning snow by hand with our shovels and brooms keeping the switches clear beyond the reach of the gas powered rail heaters. There are big propane tanks that feed the flame torch heaters attached to the rails for about 75 feet on either side of the swi just enough to keep the switch open at both ends of the various sidings along the way. Our job was to clear a space for the train to dump any snow they might be plowing before filling the switch.
We started our day as usual cleaning each switch as we worked our way up the mountain following trains from switch to switch as we had been doing for weeks. We were making good time since the wind was blowing and the snow was light and dry so it could never get high enough to cover the top of the rails before it blew away. Then around 10am the wind stopped and the snow was coming straight down in a thick blanket.
By lunch we only had two of the 6 sets of switches done compared to the last few days when we had been eating lunch at the summit. It was obvious today would be overtime if we didn't get help from above, meaning the gang from Crescent Lake would have to come down help. We were encouraged for a minute when we heard over the radio that they were on the way but then we got word that the snow was piling up down the hill where it usually rained so now we would have to go down to help after we finish on top and it would still be overtime.
Then the heavy snow started and before we knew what to do our motor car was stuck. We could not go up or down the tracks and we were forced to hole up in the warming shack at the west end of the siding at Cruzette waiting to be rescued. We had radio contact with the dispatcher and he said there would be one last train coming down the hill and then no more after that, all trains are stopped until further notice. Our best chance was following that last train down the hill but we were on the siding and had to dig our way out by hand and then it was too late to catch up with the train. The snow was coming too fast and falling back on us in the cuts and soon we were stuck again.
We made a brief effort to shovel our way out but it was obvious that we were worse off than ever. Now we were stuck between sidings and would present a hazard if another train did come. We discussed taking the motor car off the rails but that would be a last resort. So instead we made our way to the nearest tunnel and built a warming fire near the entrance where we had stashed diesel fuel and wood for such situations. It was already 3pm and the sun goes down early that time of year and we only had a couple of small flashlights which are not much help during a blizzard at night. Everyone took stock of the leftovers in their lunch box as we prepared for the worst.
Then we heard the good news, the work train was on the way with the Flanger. The Flanger is a single 40 foot custom car with a snow plow on the front that just fit over the rails with one Train Engine to push it along up to 40 miles an hours. Now we could finish the run from Oakridge to the Summit and back in less than four hours. Once they cleared a path to our stranded motor car we were able to take it back down the hill to the motor car shed at the east end of Frazier. It was a warm and comfortable relief to finally climb aboard the Flanger where there were supplies to go along with the stove.
There were only two positions for the Flanger blade, up and down. There was one big red button to toggle the position and the operator had to know just when to go up in order to miss the switches, road crossings, rail greasers and other locations that were marked with red reflector posts positioned to stick out of the deepest snow. It was not uncommon for the flanger to derail but it was easy to put back due to specially equipped outriggers that kept it from going to far off the rails.
We made it up to the summit and back to Fields by 9pm and they were already calling for the Flanger to make another run back on top. The Superintendent even got involved and told us to take our 8 hours rest since the weather report was calling for even heavier snow tomorrow. The Flanger and crew went all the way to Oakridge to spend the night in the Motor Lodge while the rest of us returned to our Company housing in Fields.
We were dog tired but It was a fitful night of sleep knowing what was coming the next day. And that will be the next chapter in the series of stories about Railroad Days : Snow Removal Day 2.
Elevation makes all the difference when it comes to snow. Oakridge at 1500 feet hardly ever gets more than a few inches at a time and it always soon melts. Fields is at 2500 feet where the snow starts sooner and sticks around longer than the lower elevations. The summit at Odell Lake is just under 5000 feet and the snow gets deep and stays deep.
For the last few weeks we had been cleaning snow by hand with our shovels and brooms keeping the switches clear beyond the reach of the gas powered rail heaters. There are big propane tanks that feed the flame torch heaters attached to the rails for about 75 feet on either side of the swi just enough to keep the switch open at both ends of the various sidings along the way. Our job was to clear a space for the train to dump any snow they might be plowing before filling the switch.
We started our day as usual cleaning each switch as we worked our way up the mountain following trains from switch to switch as we had been doing for weeks. We were making good time since the wind was blowing and the snow was light and dry so it could never get high enough to cover the top of the rails before it blew away. Then around 10am the wind stopped and the snow was coming straight down in a thick blanket.
By lunch we only had two of the 6 sets of switches done compared to the last few days when we had been eating lunch at the summit. It was obvious today would be overtime if we didn't get help from above, meaning the gang from Crescent Lake would have to come down help. We were encouraged for a minute when we heard over the radio that they were on the way but then we got word that the snow was piling up down the hill where it usually rained so now we would have to go down to help after we finish on top and it would still be overtime.
Then the heavy snow started and before we knew what to do our motor car was stuck. We could not go up or down the tracks and we were forced to hole up in the warming shack at the west end of the siding at Cruzette waiting to be rescued. We had radio contact with the dispatcher and he said there would be one last train coming down the hill and then no more after that, all trains are stopped until further notice. Our best chance was following that last train down the hill but we were on the siding and had to dig our way out by hand and then it was too late to catch up with the train. The snow was coming too fast and falling back on us in the cuts and soon we were stuck again.
We made a brief effort to shovel our way out but it was obvious that we were worse off than ever. Now we were stuck between sidings and would present a hazard if another train did come. We discussed taking the motor car off the rails but that would be a last resort. So instead we made our way to the nearest tunnel and built a warming fire near the entrance where we had stashed diesel fuel and wood for such situations. It was already 3pm and the sun goes down early that time of year and we only had a couple of small flashlights which are not much help during a blizzard at night. Everyone took stock of the leftovers in their lunch box as we prepared for the worst.
Then we heard the good news, the work train was on the way with the Flanger. The Flanger is a single 40 foot custom car with a snow plow on the front that just fit over the rails with one Train Engine to push it along up to 40 miles an hours. Now we could finish the run from Oakridge to the Summit and back in less than four hours. Once they cleared a path to our stranded motor car we were able to take it back down the hill to the motor car shed at the east end of Frazier. It was a warm and comfortable relief to finally climb aboard the Flanger where there were supplies to go along with the stove.
There were only two positions for the Flanger blade, up and down. There was one big red button to toggle the position and the operator had to know just when to go up in order to miss the switches, road crossings, rail greasers and other locations that were marked with red reflector posts positioned to stick out of the deepest snow. It was not uncommon for the flanger to derail but it was easy to put back due to specially equipped outriggers that kept it from going to far off the rails.
We made it up to the summit and back to Fields by 9pm and they were already calling for the Flanger to make another run back on top. The Superintendent even got involved and told us to take our 8 hours rest since the weather report was calling for even heavier snow tomorrow. The Flanger and crew went all the way to Oakridge to spend the night in the Motor Lodge while the rest of us returned to our Company housing in Fields.
We were dog tired but It was a fitful night of sleep knowing what was coming the next day. And that will be the next chapter in the series of stories about Railroad Days : Snow Removal Day 2.
Sunday, June 12, 2011
Curious Case #3: The Password Problem
With all the free work we had been doing to pay back our competitors it was time to actually make some money and I made sure to find work at my job for Charlie to come in at least once a week but in between I had asked Daryl the dispatcher to keep any eye out for any guaranteed money opportunities and that was how we ended up at the home of the CEO of a local Sports Shoe magnate who kept having problems logging into certain systems.
It took two days leading up to the visit working out the details with his Executive Assistant a fascinating lady named Sparkle Silverfox. I couldn't help asking how her parents came up with that name and she laughed explaining how they had named her Elizabeth Hamilton but she had changed it on her own back in the '70s. I should have known she was the old hippie and not her parents but she didn't look that old. We could tell she was the perfect fit for the equally eccentric CEO.
On the appointed Tuesday we were told the meeting would begin between at 1 PM but we already knew enough to expect a delay and spent the time waiting at a nearby Sandwich shop which was the only thing close to his secluded mansion hidden way up in the hills . While we waited I did battle with my favorite foot long meatball, while Charlie had to agonize over the entire menu before finally compromising with himself by ordering 6" each of Turkey and Tuna.
When we finally stood next to the CEO and watched him login for the first time we were surprised at how long it took to login. Charlie jumped right in saying "Whoa, what're you doing writing a book? No one should have a password that long, it's just too much typing. " The Chief Executive was indignant at first explaining how he had important data to protect and needed the strongest possible security.
I was starting to get uncomfortable but Charlie found a way of turning the tension into a learnable moment for all of us by explaining with complete sincerity that when it comes to passwords the strength is not in the length, but in the special characters and avoiding dictionary words. No one ever thought to question the big boss on how long his password should be but Charlie gave him the training he needed. We went on in detail how Passwords are used in different "namespaces" for example; LDAP for Windows, NIS for unix and sometimes a home grown web page will use a local list of people and passwords . That was the problem in this case when the program could not process that long password.
The discussion included a viewing of his own Company intranet web page for proper password management where it says the password must be exactly 8 characters. The Chief Exec was determined to change that policy so we knew we had to convince him once and for all that it wouldn’t matter anyway. We had him telnet to a terminal and forced him to type only the first 8 characters of his super long password and sure enough it let him login thereby proving that all those extra password characters were a waste of time.
The solution was simple, change the password one more time following the rules and things worked as expected and we went on our way. You can imagine my angry response later on when Daryl said the CEO refused to pay the bill since we didn't actually fix anything. Charlie didn't seem to mind and acted like it was no big deal while I took it as a personal insult until I found out later Charlie had a date with Sparkle Silverfox and that’s why he felt properly compensated.
Monday, June 6, 2011
The Dog Ate My Resume
We had a Geometry teacher in High School who was well liked for telling great stories and giving easy grades but if a student asked a hard question his famous response was "I missed that day at school." After a while we got to thinking he must have been very sickly or delinquent for all the days he missed but we never thought to question his teaching skills, he just didn't know the answer. His main job was Basketball Coach but he had to teach other classes to make his pay and Geometry was on his schedule even if it wasn't his strong suit. At least he was honest.
Back in the first grade I did get sick and miss a day of school and when I got back the next day the teacher took time out to help me catch up with the previous day's work which consisted of one word: pie. Seriously that was a day’s work for a first grader in my day. That was the last time they ever helped me catch up on stuff that was missed. You didn't have to miss a whole day to miss the lesson, it could be anything from daydreaming to dyslexia, the results are the same.
Then later on in life I caught myself using that same lame excuse and I realized that using the ‘missed day at school’ routine was more trouble than good, so then I had to take time out of my adult life to unlearn that teacher's bad habit. That was when I first started looking for other bad things I might have accidentally learned along the way but how do you know it's wrong before it has been revealed? Everything still looks OK so there is no easy way to know which is the wrong idea.
I may not be sure how much bad stuff I did learn but once I got to looking I found all kinds of stuff that I didn't learn. Simple things that would seem obvious to anyone like, How to think. I never had that class, no how to guide, no FAQ , nothing. There should be memory tricks to make things easier to remember, and exercises to remember better. How to smell, how to listen, how to look and see, these were all taken for granted but I can tell I am still not using those skills to the full potential. The brain is such a powerful tool but there is no user manual. It feels like I’m not getting all the features and benefits.
Another variation on the "missed that day at school" is this similar excuse " They didn't teach that at my school". This puts the blame on the teacher, the principal, the school board or even the student body, anybody but the student themselves. Using these excuses is like carrying on the old tradition of the dog ate my homework, then he ate my resume and now the dog ate my career.
Back in the first grade I did get sick and miss a day of school and when I got back the next day the teacher took time out to help me catch up with the previous day's work which consisted of one word: pie. Seriously that was a day’s work for a first grader in my day. That was the last time they ever helped me catch up on stuff that was missed. You didn't have to miss a whole day to miss the lesson, it could be anything from daydreaming to dyslexia, the results are the same.
Then later on in life I caught myself using that same lame excuse and I realized that using the ‘missed day at school’ routine was more trouble than good, so then I had to take time out of my adult life to unlearn that teacher's bad habit. That was when I first started looking for other bad things I might have accidentally learned along the way but how do you know it's wrong before it has been revealed? Everything still looks OK so there is no easy way to know which is the wrong idea.
I may not be sure how much bad stuff I did learn but once I got to looking I found all kinds of stuff that I didn't learn. Simple things that would seem obvious to anyone like, How to think. I never had that class, no how to guide, no FAQ , nothing. There should be memory tricks to make things easier to remember, and exercises to remember better. How to smell, how to listen, how to look and see, these were all taken for granted but I can tell I am still not using those skills to the full potential. The brain is such a powerful tool but there is no user manual. It feels like I’m not getting all the features and benefits.
Another variation on the "missed that day at school" is this similar excuse " They didn't teach that at my school". This puts the blame on the teacher, the principal, the school board or even the student body, anybody but the student themselves. Using these excuses is like carrying on the old tradition of the dog ate my homework, then he ate my resume and now the dog ate my career.
Friday, June 3, 2011
Curious Case #2: The Printing Problem
The hardest part of running the Columbia Interlocking Services company was figuring out how to get paid when we were acting as sub-contractors for one of our own competitors. We had tried everything from cash under the table to IOU's but after a few broken promises that led to bad feelings we ended up using our friend Daryl the Dispatcher to keep track of hours and jobs.
That's how we got the call from Daryl saying there was an easy chance to make up for some of the hours we owed to Northwest . It wasn't always us saving them and we recently needed their help on a big job not to mention always borrowing their van for the big stuff like that Cisco 9300 switch that wouldn't fit either one of our cars.
Northwest was the original contractor who then called in the guys from Solution Integrators who together spent the first 4 days gathering data and by the end of the week they were calling us in desperation, specifically Charlie. Luckily we were already done working for the day and promised to come over after lunch. Charlie wanted to go back to the All You can eat buffet to fill his plate with crawdads but we had been there twice in a row already and I insisted on the Burnt Bun burger joint where you can build your own creation, my favorite is guacamole bacon burger with no cheese.
We arrive at the customer site, a service company called Timberlodge, along with a rush of workers returning from lunch as we were escorted through the security system to a conference room where all us consultants were gathered to finally get to the bottom of the issue. I was hoping to break the ice with their main guy who had a big tag on his shirt that said he is the Computer Manager whose name is MAURICE so I gave him my best hand shake along with a "Glad to meet you 'Moreece'," but right away he says "That's how they say it in France but where I comes from it's 'Morris". I made sure to correct myself out loud and made a silent promise to call everyone Buddy.
In the meeting room are all the documents collected showing the logs and reboot times. There are 150 users on these 20 servers providing remote services for every department across the entire campus. The reboots were random and hit all the servers at various different times with only one recurring pattern every day between 2:10 and 2:20pm. The logs were not much help but so far all of the Analysts were blaming the servers but not Charlie. Right away he was more interested in the users.
There were two groups preparing for the coming reboot at 2:15pm. Half the team headed for the server room to see if they can notice some coincidence with the AC or power while the rest will stay in the Crisis Room watching the server consoles to see if the verbose logging they setup might isolate the issue.
Those were good ideas but not for Charlie, he went straight out to the user cube farm and started talking with the area Leaders. He found that half the users come back from lunch at 1Pm and the others at 2pm. Following a hunch he went to the printing station located at the center of the area where the 2pm lunch crowd would have just returned. I asked him what he was up to and he said, "If the answer was in the server room they would have found it by now."
Then at the appointed time Charlie raised his finger in triumph pointing at the sound of a distant user shouting out in frustration; "Crap, every time I print, I loose my server session." Following the sound we found the user coming towards us and the printers. We introduced ourselves to Sharon who had long suffered this problem and even filed a ticket once but had since learned to live with it.
Sharon always knew it was weird that she had to start a new server session so often but since the print jobs did go through successfully no one ever made the connection between printing and server crashing until Charlie heard her print. It turns out she has an old Unix workstation that uses a malformed print command that caused a panic in the process on the other end.
We never did fix the issue, we just got her to print from another machine. The guys from Northwest were so pleased they offered to take us out for beers so we had to go and listen how they had done all the real work. I kept trying to get Charlie the credit he deserved but every time he told the story the most important part was "How that girl cussed!"
That's how we got the call from Daryl saying there was an easy chance to make up for some of the hours we owed to Northwest . It wasn't always us saving them and we recently needed their help on a big job not to mention always borrowing their van for the big stuff like that Cisco 9300 switch that wouldn't fit either one of our cars.
Northwest was the original contractor who then called in the guys from Solution Integrators who together spent the first 4 days gathering data and by the end of the week they were calling us in desperation, specifically Charlie. Luckily we were already done working for the day and promised to come over after lunch. Charlie wanted to go back to the All You can eat buffet to fill his plate with crawdads but we had been there twice in a row already and I insisted on the Burnt Bun burger joint where you can build your own creation, my favorite is guacamole bacon burger with no cheese.
We arrive at the customer site, a service company called Timberlodge, along with a rush of workers returning from lunch as we were escorted through the security system to a conference room where all us consultants were gathered to finally get to the bottom of the issue. I was hoping to break the ice with their main guy who had a big tag on his shirt that said he is the Computer Manager whose name is MAURICE so I gave him my best hand shake along with a "Glad to meet you 'Moreece'," but right away he says "That's how they say it in France but where I comes from it's 'Morris". I made sure to correct myself out loud and made a silent promise to call everyone Buddy.
In the meeting room are all the documents collected showing the logs and reboot times. There are 150 users on these 20 servers providing remote services for every department across the entire campus. The reboots were random and hit all the servers at various different times with only one recurring pattern every day between 2:10 and 2:20pm. The logs were not much help but so far all of the Analysts were blaming the servers but not Charlie. Right away he was more interested in the users.
There were two groups preparing for the coming reboot at 2:15pm. Half the team headed for the server room to see if they can notice some coincidence with the AC or power while the rest will stay in the Crisis Room watching the server consoles to see if the verbose logging they setup might isolate the issue.
Those were good ideas but not for Charlie, he went straight out to the user cube farm and started talking with the area Leaders. He found that half the users come back from lunch at 1Pm and the others at 2pm. Following a hunch he went to the printing station located at the center of the area where the 2pm lunch crowd would have just returned. I asked him what he was up to and he said, "If the answer was in the server room they would have found it by now."
Then at the appointed time Charlie raised his finger in triumph pointing at the sound of a distant user shouting out in frustration; "Crap, every time I print, I loose my server session." Following the sound we found the user coming towards us and the printers. We introduced ourselves to Sharon who had long suffered this problem and even filed a ticket once but had since learned to live with it.
Sharon always knew it was weird that she had to start a new server session so often but since the print jobs did go through successfully no one ever made the connection between printing and server crashing until Charlie heard her print. It turns out she has an old Unix workstation that uses a malformed print command that caused a panic in the process on the other end.
We never did fix the issue, we just got her to print from another machine. The guys from Northwest were so pleased they offered to take us out for beers so we had to go and listen how they had done all the real work. I kept trying to get Charlie the credit he deserved but every time he told the story the most important part was "How that girl cussed!"
Friday, May 20, 2011
Curious Case #1: The Power Problem
The first real job that came along for Columbia Interlocking Services was at a small software company called Forest Park that had several racks of servers and one rack kept having servers burn out the motherboard. The service provider would come and replace the motherboard and then later it would happen again, sometimes on another server. It was just they kind of mystery we loved.
Charlie was coming onsite for our regular Tuesday morning consulting anyway so it was convenient to take the afternoon off to handle this new challenge. Usually we go straight to lunch and then go the customer after but this was a cold call on a hot issue and we decided we better go right away.
The traffic was light but it still took 30 minutes to complete the 4 mile trip door to door and that was enough time for us to argue over the music, best route, window position and what was for lunch. Charlie said he thinks my car has a spark plug that is misfiring but it all seems good to me.
We get to the place and find that two of the Nasties from Northwest have already been here and replaced all the power components in the server room: power strip, cables, and even had an electrician onsite to change the circuit breaker just to eliminate suspects. Then last night another server gets smoked and now this morning their CEO is ready to fire somebody.
We met their local IT leader named Harish which I made the mistake of pronouncing with two syllables when he rudely interrupts to correct me by blurting out; "Harsh!". At first I thought he was describing his personality but then I realized that was how he said his name. In spite of my recent training in international cultural sensitivity I agreed to use his pronunciation. We have a policy not to argue with clients.
First thing Charlie does is stand in back of the problem rack and check each cable starting at the bottom making sure each was plugged in all the way even though he knew they had just replaced them all. About half way up he stops and says "Here it is" in a matter of fact voice. He is holding up a network cable to everyone’s amazement when we were so sure the problem was power related, no one had looked at the network.
It turns out there had once been remote network device in this rack that required power over the Ethernet cable. This unusual setup originally required a red cable with a yellow label showing it was not a standard link. Somewhere over the years the cable was removed and eventually someone plugged a regular network cable into the powered injected jack. Amazingly the server would work for a while before failing so no one thought to blame the network cable until Charlie said he could feel the power in the cable.
Harish didn't believe it until we proved it with a power meter. It was only 5 Volts of Direct Current but enough to finally kill the machine. In hindsight it is easy to blame the service provider for replacing the motherboard without providing the root cause but these motherboards have the network adapter built-in which makes it more vulnerable and harder to debug compared to servers with a separate network adapter card.
Later on we were at the Flower Blossom restaurant filling our plates from the All You Can Eat buffet while pretending we knew how to use chopsticks. I had a better grip than Charlie but we both ended up with forks to finish our plates. That's when Charlie told me the rest of the story about how he fell off the power line pole after accidentally connecting himself to the wrong two wires. It smoked both his gloves then burned all the hair off the top of his head and it never did grow back. But it did leave him with the uncanny ability to feel electricity.
Charlie was coming onsite for our regular Tuesday morning consulting anyway so it was convenient to take the afternoon off to handle this new challenge. Usually we go straight to lunch and then go the customer after but this was a cold call on a hot issue and we decided we better go right away.
The traffic was light but it still took 30 minutes to complete the 4 mile trip door to door and that was enough time for us to argue over the music, best route, window position and what was for lunch. Charlie said he thinks my car has a spark plug that is misfiring but it all seems good to me.
We get to the place and find that two of the Nasties from Northwest have already been here and replaced all the power components in the server room: power strip, cables, and even had an electrician onsite to change the circuit breaker just to eliminate suspects. Then last night another server gets smoked and now this morning their CEO is ready to fire somebody.
We met their local IT leader named Harish which I made the mistake of pronouncing with two syllables when he rudely interrupts to correct me by blurting out; "Harsh!". At first I thought he was describing his personality but then I realized that was how he said his name. In spite of my recent training in international cultural sensitivity I agreed to use his pronunciation. We have a policy not to argue with clients.
First thing Charlie does is stand in back of the problem rack and check each cable starting at the bottom making sure each was plugged in all the way even though he knew they had just replaced them all. About half way up he stops and says "Here it is" in a matter of fact voice. He is holding up a network cable to everyone’s amazement when we were so sure the problem was power related, no one had looked at the network.
It turns out there had once been remote network device in this rack that required power over the Ethernet cable. This unusual setup originally required a red cable with a yellow label showing it was not a standard link. Somewhere over the years the cable was removed and eventually someone plugged a regular network cable into the powered injected jack. Amazingly the server would work for a while before failing so no one thought to blame the network cable until Charlie said he could feel the power in the cable.
Harish didn't believe it until we proved it with a power meter. It was only 5 Volts of Direct Current but enough to finally kill the machine. In hindsight it is easy to blame the service provider for replacing the motherboard without providing the root cause but these motherboards have the network adapter built-in which makes it more vulnerable and harder to debug compared to servers with a separate network adapter card.
Later on we were at the Flower Blossom restaurant filling our plates from the All You Can Eat buffet while pretending we knew how to use chopsticks. I had a better grip than Charlie but we both ended up with forks to finish our plates. That's when Charlie told me the rest of the story about how he fell off the power line pole after accidentally connecting himself to the wrong two wires. It smoked both his gloves then burned all the hair off the top of his head and it never did grow back. But it did leave him with the uncanny ability to feel electricity.
Subscribe to:
Posts (Atom)