Curious Case #4: The Personnel Problem

We got a call from Daryl the Dispatcher with a special request for our services at a company called Toll Free Support who made their mark by offering toll free tech support to the callers while charging the backend company either by the minute or by the call.

Last year we helped setup their two big accounts, Intel and IBM, so we already knew the people. The Intel contact was nice lady named Li Xu and the IBM contact was Radjendiramane Venkatachalapathy. We called him Radj and we called her...Li. Right off the bat Charlie took a liking to Li but seemed to resent Radj for making us type so many characters in an email address.

Intel was one of the companies that paid by the call and IBM paid by the minute. There were two databases, one for the total phone calls taken every day and another for the written reports entered by the workers. Intel didn't care about the written reports, they only looked at the phone call totals. IBM on the other hand only wanted to see nice long written reports justifying the minutes and never asked to see the phone call numbers.

The two databases came up with the same numbers every day but it was clear there was something wrong due to the large number of complaints from dropped calls and people who swore they had called before but there was no record of them in either database. It may seem primitive now but they had no reason to compare the two databases. It would be hard to do anyway because the two didn't share the same data types and it would have taken lots of work to make the simple comparison.

We spent our first day walking the main floor where all of the tiny cubes were lined up in a big "L" shape with a row of offices all along one wall where the senior Level 3 experts were still just as crowded with three people in a small office, but it was still an office. Charlie soon lost interest on the main floor and from the second day on concentrated on the offices where he easily disarmed each group as he asked how they liked the products they were supporting while offering advice on how to handle tough situations.

He also kept talking Li into taking us to lunch onsite at the Intel campus a few blocks away, which first happened when we went for training at the beginning of the contract. I didn't think the Intel cafeteria was that good but Charlie couldn't get enough.

We finally got back from lunch on the fourth day when Charlie announced he had solved the mystery. Once again he walked straight up to the problem knowing the majority of the users on the open call floor had already been eliminated it had to be the people in the offices. All he had to do was get them talking and sure enough they were so proud of their achievements they couldn't help but brag even if it meant exposing their crime.

It all started quite a while back with only two workers. Dennis was on the Intel team and several times a day he would take the time to enter a written report, but the rest of the time with the easy calls he just took the next call and kept going with no written reports. Meanwhile, several times a day Christine would get flustered and double-click her phone which would dump one caller and skip to the next and then later she would take the time to enter two reports anyway. The dropped call didn't last the required two seconds and would not register in the days totals which is why they were cancelling each other out. He had calls with no reports and she had reports with no calls.

They were both successful in their own group, Dennis taking big numbers of calls albeit without written reports and Christine was famous for such great written reports though there were no calls to go with them. Because of their success they were each given an office with two assistants who in turn each soon learned the same bad habits and before long there were six experts adding to the overall problem.

In truth all of the guilty users figured the Company knew what was going on all along. And even though the Company now knew the truth beyond a doubt the only steps they took was to have Christine's team to stop taking calls and start writing reports for Dennis and his crew. We gave a recommendation to make the two databases compatible and start including all calls in the phone report, even those lasting less than two seconds.

Afterward Charlie told me he knew the solution on the first day but it took four trips to the Intel cafeteria to finally get his fill.

Bigger Than An Elephant

Back in the late '90's I worked at a small hi-tech call center called 800 Support. They had recently changed their name from 900 Support because 900 numbers were synonymous with phone sex at that time. Changing the name didn't change the culture. We had a small budget for our Company picnic and had to settle for a Clown as our entertainment. There was lots of talk in the social committee meetings about how we could easily have a better picnic next year. The big Blow-Up slide was the clear favorite in the unofficial voting.

Then we arrived at Cook Park to find the place is full because Intel is also having their Company Picnic at the same time. We were limited to a small corner of the park with our Clown while Intel got the rest of the park for their Petting Zoo, Carnival equipment and best of all an Elephant for the kids to ride. Talk about picnic envy, we were wanting to crash that party for sure.

We kept wondering what they would do to top that picnic next year. What could they possibly get that is bigger than an Elephant? A Whale might be bigger but the kids can't ride. It's like the escalating Birthday Party for kids that go from costume themes to the famous Clown again and it can only end with a Pony to ride for entertainment. But what do you get the year after the Pony? A herd of Ponies? Clowns on Ponies? Ponies on Clowns?

The next year we couldn't even afford the cost of a full picnic so we had to settle for a Bowling trip and all the people complained that last year was better. That was when the Social Committee turned into the anti-Social Committee and we ended up with no picnic the year after that.

There was a time when we were jealous of Intel and their Elephant but then it turned into proof of their excess. We finally realized you can't live up to last year so don't even try. Now days most people are happy to go Bowling but I am still dreaming of the day we get a Dinosaur for our next Company Picnic. Nothing could be bigger than a Dino.

Curious Case #3: The Password Problem

With all the free work we had been doing to pay back our competitors it was time to actually make some money and I made sure to find work at my job for Charlie to come in at least once a week but in between I had asked Daryl the dispatcher to keep any eye out for any guaranteed money opportunities and that was how we ended up at the home of the CEO of a local Sports Shoe magnate who kept having problems logging into certain systems.

It took two days leading up to the visit working out the details with his Executive Assistant a fascinating lady named Sparkle Silverfox. I couldn't help asking how her parents came up with that name and she laughed explaining how they had named her Elizabeth Hamilton but she had changed it on her own back in the '70s. I should have known she was the old hippie and not her parents but she didn't look that old. We could tell she was the perfect fit for the equally eccentric CEO.

On the appointed Tuesday we were told the meeting would begin between at 1 PM but we already knew enough to expect a delay and spent the time waiting at a nearby Sandwich shop which was the only thing close to his secluded mansion hidden way up in the hills . While we waited I did battle with my favorite foot long meatball, while Charlie had to agonize over the entire menu before finally compromising with himself by ordering 6" each of Turkey and Tuna.

When we finally stood next to the CEO and watched him login for the first time we were surprised at how long it took to login. Charlie jumped right in saying "Whoa, what're you doing writing a book? No one should have a password that long, it's just too much typing. " The Chief Executive was indignant at first explaining how he had important data to protect and needed the strongest possible security.

I was starting to get uncomfortable but Charlie found a way of turning the tension into a learnable moment for all of us by explaining with complete sincerity that when it comes to passwords the strength is not in the length, but in the special characters and avoiding dictionary words. No one ever thought to question the big boss on how long his password should be but Charlie gave him the training he needed. We went on in detail how Passwords are used in different "namespaces" for example; LDAP for Windows, NIS for unix and sometimes a home grown web page will use a local list of people and passwords . That was the problem in this case when the program could not process that long password.

The discussion included a viewing of his own Company intranet web page for proper password management where it says the password must be exactly 8 characters. The Chief Exec was determined to change that policy so we knew we had to convince him once and for all that it wouldn’t matter anyway. We had him telnet to a terminal and forced him to type only the first 8 characters of his super long password and sure enough it let him login thereby proving that all those extra password characters were a waste of time.

The solution was simple, change the password one more time following the rules and things worked as expected and we went on our way. You can imagine my angry response later on when Daryl said the CEO refused to pay the bill since we didn't actually fix anything. Charlie didn't seem to mind and acted like it was no big deal while I took it as a personal insult until I found out later Charlie had a date with Sparkle Silverfox and that’s why he felt properly compensated.

Curious Case #2: The Printing Problem

The hardest part of running the Columbia Interlocking Services company was figuring out how to get paid when we were acting as sub-contractors for one of our own competitors. We had tried everything from cash under the table to IOU's but after a few broken promises that led to bad feelings we ended up using our friend Daryl the Dispatcher to keep track of hours and jobs.

That's how we got the call from Daryl saying there was an easy chance to make up for some of the hours we owed to Northwest . It wasn't always us saving them and we recently needed their help on a big job not to mention always borrowing their van for the big stuff like that Cisco 9300 switch that wouldn't fit either one of our cars.

Northwest was the original contractor who then called in the guys from Solution Integrators who together spent the first 4 days gathering data and by the end of the week they were calling us in desperation, specifically Charlie. Luckily we were already done working for the day and promised to come over after lunch. Charlie wanted to go back to the All You can eat buffet to fill his plate with crawdads but we had been there twice in a row already and I insisted on the Burnt Bun burger joint where you can build your own creation, my favorite is guacamole bacon burger with no cheese.

We arrive at the customer site, a service company called Timberlodge, along with a rush of workers returning from lunch as we were escorted through the security system to a conference room where all us consultants were gathered to finally get to the bottom of the issue. I was hoping to break the ice with their main guy who had a big tag on his shirt that said he is the Computer Manager whose name is MAURICE so I gave him my best hand shake along with a "Glad to meet you 'Moreece'," but right away he says "That's how they say it in France but where I comes from it's 'Morris". I made sure to correct myself out loud and made a silent promise to call everyone Buddy.

In the meeting room are all the documents collected showing the logs and reboot times. There are 150 users on these 20 servers providing remote services for every department across the entire campus. The reboots were random and hit all the servers at various different times with only one recurring pattern every day between 2:10 and 2:20pm. The logs were not much help but so far all of the Analysts were blaming the servers but not Charlie. Right away he was more interested in the users.

There were two groups preparing for the coming reboot at 2:15pm. Half the team headed for the server room to see if they can notice some coincidence with the AC or power while the rest will stay in the Crisis Room watching the server consoles to see if the verbose logging they setup might isolate the issue.

Those were good ideas but not for Charlie, he went straight out to the user cube farm and started talking with the area Leaders. He found that half the users come back from lunch at 1Pm and the others at 2pm. Following a hunch he went to the printing station located at the center of the area where the 2pm lunch crowd would have just returned. I asked him what he was up to and he said, "If the answer was in the server room they would have found it by now."

Then at the appointed time Charlie raised his finger in triumph pointing at the sound of a distant user shouting out in frustration; "Crap, every time I print, I loose my server session." Following the sound we found the user coming towards us and the printers. We introduced ourselves to Sharon who had long suffered this problem and even filed a ticket once but had since learned to live with it.

Sharon always knew it was weird that she had to start a new server session so often but since the print jobs did go through successfully no one ever made the connection between printing and server crashing until Charlie heard her print. It turns out she has an old Unix workstation that uses a malformed print command that caused a panic in the process on the other end.

We never did fix the issue, we just got her to print from another machine. The guys from Northwest were so pleased they offered to take us out for beers so we had to go and listen how they had done all the real work. I kept trying to get Charlie the credit he deserved but every time he told the story the most important part was "How that girl cussed!"

Curious Case #1: The Power Problem

The first real job that came along for Columbia Interlocking Services was at a small software company called Forest Park that had several racks of servers and one rack kept having servers burn out the motherboard. The service provider would come and replace the motherboard and then later it would happen again, sometimes on another server. It was just they kind of mystery we loved.

Charlie was coming onsite for our regular Tuesday morning consulting anyway so it was convenient to take the afternoon off to handle this new challenge. Usually we go straight to lunch and then go the customer after but this was a cold call on a hot issue and we decided we better go right away.

The traffic was light but it still took 30 minutes to complete the 4 mile trip door to door and that was enough time for us to argue over the music, best route, window position and what was for lunch. Charlie said he thinks my car has a spark plug that is misfiring but it all seems good to me.

We get to the place and find that two of the Nasties from Northwest have already been here and replaced all the power components in the server room: power strip, cables, and even had an electrician onsite to change the circuit breaker just to eliminate suspects. Then last night another server gets smoked and now this morning their CEO is ready to fire somebody.

We met their local IT leader named Harish which I made the mistake of pronouncing with two syllables when he rudely interrupts to correct me by blurting out; "Harsh!". At first I thought he was describing his personality but then I realized that was how he said his name. In spite of my recent training in international cultural sensitivity I agreed to use his pronunciation. We have a policy not to argue with clients.

First thing Charlie does is stand in back of the problem rack and check each cable starting at the bottom making sure each was plugged in all the way even though he knew they had just replaced them all. About half way up he stops and says "Here it is" in a matter of fact voice. He is holding up a network cable to everyone’s amazement when we were so sure the problem was power related, no one had looked at the network.

It turns out there had once been remote network device in this rack that required power over the Ethernet cable. This unusual setup originally required a red cable with a yellow label showing it was not a standard link. Somewhere over the years the cable was removed and eventually someone plugged a regular network cable into the powered injected jack. Amazingly the server would work for a while before failing so no one thought to blame the network cable until Charlie said he could feel the power in the cable.

Harish didn't believe it until we proved it with a power meter. It was only 5 Volts of Direct Current but enough to finally kill the machine. In hindsight it is easy to blame the service provider for replacing the motherboard without providing the root cause but these motherboards have the network adapter built-in which makes it more vulnerable and harder to debug compared to servers with a separate network adapter card.

Later on we were at the Flower Blossom restaurant filling our plates from the All You Can Eat buffet while pretending we knew how to use chopsticks. I had a better grip than Charlie but we both ended up with forks to finish our plates. That's when Charlie told me the rest of the story about how he fell off the power line pole after accidentally connecting himself to the wrong two wires. It smoked both his gloves then burned all the hair off the top of his head and it never did grow back. But it did leave him with the uncanny ability to feel electricity.

Curious Case # 0 : The Prolog Problem

I met Charlie Pressley at my first hi-tech job after finishing Trade School in '91. He was that old guy from the Phone Department and I was that new guy in the newly formed Network group of the also new Computer department. The rules required technicians work in teams of two so we ended up working together since he was in a department by himself and tradition held that the newest guy in our group had to work with Charlie.

Back in the old days Charlie was the Department lead with people working under him and a big budget to spend money managing five hundred phones and faxes but now all the money was going to the new Network project. Suddenly phones had lost all their all their budget and importance. Charlie ended up working with me as his reluctant helper on loan from another group.

The other teams called us Katz and Pressley but managed to make it sound disrespectful and they made sure we got all the worst jobs. We were working at a mid-size hardware company that was moving from a product line of dialup modems to network adapter cards. Charlie and I spent most of our time pulling cables and punching down connectors in a constant flurry of office moves and cube relocations as the company grew. Occasionally we got called on real issues for the computer group but it was never enough for me.

At first Charlie and I didn't have a lot of respect for each other. I thought he was a crazy old man and he thought I was a young punk. He had these super thick glasses with lots of hair coming out all around his head except in the expected places. He also had a deep voice with a slow delivery that was a little too deliberate leaving an overall bad impression. He didn't think much of me either as I liked punk music and was more interested in setting up Computer Bulletin Boards than learning about phones.

Then a funny thing happened. The more we worked together the more we gradually gained mutual respect which in turn led to our recurring success. As much as I liked taking the credit it became clear that Charlie was the root to the solution for most problems even though he could barely read and write. He was lucky more than anything. He was always stumbling into the stinkiest mess and coming out smelling like a rose.

Then we started getting more and more real jobs assigned to us and to everyone's surprise we became one of the best teams in the department . As we developed a reputation for getting things done we started getting the hardest tickets and that's when I realized just how much help Charlie could be in spite of his limitations. He always made friends with anyone we met along the way and he always told funny stories to keep things going.

There was a recurring debate regarding the correct pronunciation of his name. Charlie insisted it was just like Elvis and sounded like Presley while I kept telling him that the double "s" meant it had to rhyme with "dress" as in Pressley. He ought to know how to say his own name and it just shows how arrogant I was back then and just how little I respected him at first.

Then we started going to lunch every so often and he opened up about his personal life and his various interests such as ham radio, a collection of adding machines and even raising chickens. He brought examples of each the next day in the form of the FAA operators manual, a classic hand powered adder and one dozen fresh brown eggs.

Our lunch evolved from every so often into almost every day as we had more calls to the branch offices. Between rides in the van and meals on the road I started to learn more about how Charlie came up through the ranks over the years. One story told how he got his limp from falling off a telephone pole when he was just starting out as a young lineman and after that he ended up working in the phone switch room.

A few years later we were the top team for troubleshooting and firefighting and finally had some respect in our department. Then the world changed and they laid Charlie off. The old modems weren't selling anymore and the new network cards weren't selling yet so there was a big layoff to save money. All the various computer departments were consolidated into a single IT department and all the phone work was to be outsourced.

Charlie didn't seem to mind and was starting to enjoy his early retirement but I soon found I couldn't get anything done without his help. The first time we got in a crunch at work the boss agreed to pay Charlie one time just to help out. Then it happened again and before long we ended up calling him back to help us on a regular part time basis.

The HR folks suggested it would be easier if Charlie had a business and he asked for my help and that's how we ended up starting a small company together so we could get contract work hours with the Corporate Headquarters. Charlie and I went into business and that led to us getting work on the side with other high tech companies in our area. It was always my intention to go into semi-retirement by contracting so this gave me a chance to practice without having to actually quit yet.

We called our little company Columbia Interlocking Services. Our two main competitors were Northwest Systems Technologies and Solution Integrators Network. We called them the Nasties and the Sinners and they called us the Sissies. We often ended up working on the same jobs after the other guy failed to put out the fire or the customer wanted a second opinion.

We got the same jobs because we all got them from the same dispatcher who had once been classmates with myself and the guys running the Nasties and the Sinners. There was mutual respect among everyone when the only goal was profit and the only enemy was stupid computers.

With our new company all setup and customers waiting we were in the perfect situation to find the best and most interesting of all the challenging jobs in our area. That's how we ended up with the first of our series of unusual incidents starting with Curious Case #1: The Power Problem. That is next week's Chapter.

Can't Talk Now

It's not just a joke about people who can't rub their belly and pat their head at the same time, it's true that the average person has a one track mind when it comes to talking. For some reason, talking is a whole brain process. We can multi-task at most things but It is well known that driving a car while talking on the phone is almost as debilitating as being drunk. Talking takes the whole brain because we are so busy listening for the next word while planning our next response there is no bandwidth left for any other task.

It must go back to our ancient ancestors and we are locked into full attention when we talk. But now in the modern age we can't afford to give all our attention to one thing so unfortunately, talking is the odd-man out. Sorry buddy, you gotta go.

At work I can carry on five simultaneous chat conversations with no stress and trouble with accents from different cultures and there is a written record. But I can still only do one phone call at a time. That's why meetings are so unproductive, only one person can talk at a time. If everyone stayed at their desk and chatted they would get more work done.

Having a conversation is the most intimate way to communicate and it is very good for conveying gestures and other personal feel good type interactions but it is not so good for business. Those gestures and feel-good tricks are only confusing and clutter up the workspace when clear communication is needed.

We are less likely to blurt out in writing something we might regret the way we might do in person. Like when the Scarecrow from the Wizard of Oz said "Some people without brains do an awful lot of talking." Conversely, some people with nothing to say have done an awful lot of thinking.

That is why we are evolving into people who might prefer to write than fight. Meanwhile I will call back once I write down what I want to say, but I can't talk now, I have to think instead.

Hi-Tech Series : Five Nines is for Wimps

Five Nines refers to 99.999% uptime for computers. Uptime is an Industry Standard in the Hi-Tech world that measures the time that a computer system runs between unscheduled down time. Scheduled down times are called maintenance windows while any unscheduled down time is considered an Outage. Two Nines allows for 3 days downtime a year, while Three Nines equals 8 hours, Four Nines is 50 minutes and Five Nines means only 5 minutes downtime per year. Anything more than that is a joke because you can't really measure 5 seconds per year which would eqal Six Nines or 99.9999%.

Most companies are happy to have 99% uptime but some have spent large amounts of time and money in the quest to get to 99.9% and so on in search the holy grail of uptime the infamous five nines: 99.999%. Everyone agrees it would be pointless to go any further. But not my new Boss, oh no, his first remark in the first meeting was: "Five Nines is for Wimps, we are going for 100%." He might as well have said 111% since it is well known you can't get there without spending more than we can afford.

Every new Boss has to make noise like a genius and they always start out with an "All Hands" meeting so they can lay out their agenda. The last new boss was going to make his mark by getting everyone a new laptop. He had the right idea but the wrong brand, we had to switch back to Lenovo after HP couldn't support our needs. HP is better for servers but Lenovo is best for laptops. The boss before that made an equally epic failure when he insisted on upgrading the company Windows version. Once again the right idea but the wrong version; too bad he chose Vista. That was his first and last mistake.

Then there was the Boss who demanded there shall be “No simple outages”. It was true we had suffered a series of avoidable breakdowns but we all knew he should be careful what he wished for and sure enough we stopped all the simple breakdowns and started on the big complicated outages. There was no going back after that and we soon had another new Boss.

This process has been going on all the way back to the Y2k panic at the turn of the Millennium. As soon as things settled down there was a move to switch to linux which we did do in spite of the complaints and protests from the existing unix departments. That new Boss was promoted to Vice President and everyone agreed he was brilliant. Later on that same Boss decided we should get rid of Windows which we did not do in spite of years spent wasting time and money on the effort. He is no longer considered brilliant or Boss.

I'm looking forward to the arrival of the next new Boss because I know it will continue a long standing tradition of starting out with some hair brain scheme that is bound to exasperate and entertain us at the same time.

Dog Food Shopping

If you Google on “dogfooding” you can find out about one of the concepts in Hi-Tech Industry called the Dog Food effect. The idea is that you have to use your own products so you can tell your customers that you know what they are going through because you eat your own dog food, so to speak. The idea is you should be familiar with your own product, something like an updated version of“Practice What You Preach”.

I first heard of this concept when Microsoft was found to be running linux servers and they had to promise to stop the same way Intel can’t run any computers with AMD chips and the Pepsi delivery man can’t be caught drinking Coke. It makes sense up to a point and then it starts to hurt more than it’s worth and people end up cheating to sneak their i-Pad in the door. It’s good to shop local when you can but it’s still OK to go Mall once in a while.

Another related idea is to go “Shopping” for your own product to experience what your users have to go through when they shop for your stuff. When I was taking support calls the shopping amounted to our bosses calling in to us as if they were customers to see how we handled their complaints.

I remember one of these incidents when I was taking support calls for Intel modems where the customer gave the standard complaint about the modems being hard to use and I gave the standard response about having to make system configuration changes and then the guy got mad. I kept my cool but he was angry and demanded a picture of my boss. It seemed like an unusual request but I said I would make the report and then he asked for a picture of the boss with his dog and that was the moment when I should have recognized the joke but they could have kept going asking for pictures of the cat and the pet bird and the even the dentist and I would have kept going along with the joke all day long.

The same concept works with your competitors in the business world where you want to go buy their product and then call their tech support. You should know the look and feel of the other products in your space so you can honestly point out the differences. Similarly you should use equipment from the company you are trying to sell to, we found out the hard way when HP told us we would get no more business until our Sales department switched from IBM to HP laptops. They didn’t like us going to their building with the competitors laptops.

The whole concept of having to eat your own dog food reminds me of the free trade concept of “Buy American” where it sounds alot easier than it is to actually do in reality. My company tries to buy products with our chips but it is impossible to operate with only our own stuff. Besides, I tried eating the dog food when I was a kid and while it wasn’t that bad, I still wouldn’t recommend it as the only diet option.

Systems Analyst Series – Chapter 4: How to be a Systems Analyst

Here are the tricks of the trade for the Systems Analyst.

Be The Machine

Seriously, get in the server room and touch those servers. Listen to them for whining fans or clicking drives. Smell for the lingering odor of the telltale burnt component. See that red light on Rack #5? That could mean something… Take the temperature in the server room, check the humidity and overall weather conditions to prevent runaway heat or loss of AC.

Monitor and Alert

There are a number of free monitoring and alert tools that will send an email, text message, phone call or turn a web page red to let you know there is a problem with any piece of equipment. In the old days we would get a page when a server was down but now we use smart phones instead. My favorite alerting software is called bigbrother which morphed into hobbit which had to change its name to xymon:http://sourceforge.net/projects/xymon/ At least this one is free and just as good as the expensive solutions.

Be Flexible

Sometimes you have to be physically flexible while working in tight spots behind the racks in the server room or sorting out cables under the user’s desk but the most common flexibility is interacting with other users. You have to open your calendar to accommodate any time, open you heart to engage with any user and open your mind to facilitate any scenario. Like the chameleon who matches the background the Analyst must fit in with all groups from the temps, to the execs or anyone in between. It helps to know a little about every group.

Be There

My cube is on the corner of First and Main and people always say I should move to a corner cube but my location says I am open for business and stop in any time. I could demand my own time and space but it seems like a cliché to be the grouchy old computer guy and this is not the season for being hard to work with, especially for us baby boomers.

Mess around with Stuff

It might seem counterintuitive but you need to break things on purpose so you get exposed to the error messages and solutions. If you are afraid of breaking things then you are a bad analyst. Anyone can break things and not feel bad about it; they are called regular users. The only thing that separates an Analyst from a regular user is the Analyst feels bad about it and wants to fix it as fast as possible. The regular user rarely realizes just how bad it's broken to begin with.

Be Picky

It's only natural for an Analyst to be a picky person. They can't help looking for faults and vulnerabilities. In everything. But don't get the idea that they are omnipotent or all observing because they are selective about what they scrutinize. What do you expect from a picky bastard?

Ignore Things

The other primary instinct of the average Analyst is knowing when to ignore things. You can't look at everything so it's really a matter of looking past most things to find the real rub. No sense chasing after wild geese when the cooked goose will come home to roost soon enough. So the next time you see a Systems Analyst looking all spaced out it might be their best work of the day.

Be Real

You might think an analyst would be clever and sharp but if they had those qualities they would get a better job. Instead you have those people who couldn't quite concentrate hard enough to be a real engineer. Like a Minor League player who never makes it to the Big League, they love to play, but not the pay, which is never enough.

The Server Is Down Again

When we first rolled out the Citrix Servers we had a few issues and there were a number of support calls that led to the creation of a generic category called Citrix Server Down. We then spent time and money to stabilize the farm and now the servers are very reliable with no outages for months and months, but it's too late, every problem that comes along gets thrown in the Citrix Server Down bucket.

Every morning the report comes in from our Bangalore office that the Citrix Server is Down Again. The twist is, their morning is our night so for me the calls come in between 8pm and 11pm with that same old complaint. These users are in various ODC's (Offshore Design Centers) which are basically branch offices off our branch office in India. These users have laptops and linux workstations that are maintained by a third party.

Here are the trouble report solutions for a typical series of tickets.

Day 1: Users can't even get to Google, obviously a network issue . Turns out all the local desktops were patched which broke their network software. The same thing happens on the Second Tuesday of each month with the latest Microsoft patch release.

Day 2: Some of the users cannot connect some of the time. I finally found a friendly user who admitted that it only happens when they use Firefox instead of Internet Explorer. It is true that Firefox is faster in most cases but since it would not connect it wasn't really faster.

Day 3: Half the users can't connect half the time because they are trying to use “the other network” that they claim is faster. Ping and traceroute show they are not taking the right route. Turns out they are trying to use the unsecure network which is faster but also a security violation.

Day 4: No issues...Holiday in India.

Day 5: Many users cannot connect and all users have slow performance. As soon as I logon to one of their machines to investigate there is a big honking error message about a virus outbreak. I end up helping the Local Admin build a cleanup USB stick to pass around to all users.

Day 6: All users on one server are running slow. The CPU is maxed out at 100% due to a wreckless user running file sharing software. Kill the process but not the user.

Day7: One server locked up with all users frozen. There is a runaway process sending the memory past the limit. Their backup software is running at the wrong time due the time being off. Corrected the time and killed the backup.

In every case it was not the server at fault but I still had to fix it to prove it was their fault and not mine. After too many reports they formed an Action Team to meet every day with special reports to all the big shots in spite of my protests. That finally led to a meeting where they got the ODC Local Admin on the line and he sheepishly admitted that putting calls in the server down queue was the only way to get anything fixed. I will yell sooner next time…

Hi-Tech Series: Chapter 8: The Business of Lifecycles

There are several stages to the business lifecycle. If you search for Business Lifecycle there will be sites with 7 stages, 5 stages, 4 stages but I am here to tell you there are only three. The typical lifecycle goes from Change to Containment followed by Consolidation. Any given company may get stuck at one stage and never move on while others go through the whole cycle every year. Every company has to go through the cycle sooner or later, like taxes. Technically there are 5 stages if you include birth and death but only the other three keep recurring in a never-ending loop.

Change is the first stage of the business lifecycle. If there is no change, there is no lifecycle. Change is always the result of some disruption that caused the change. We like to use the catch phrase Change Management but it is a contradiction in terms when management is all about stability and predictability while change is not. There was a time we tried to manage the change but now we accept that Change is in charge and we never will be able to manage it.

Here are some of the telltale signs that your company is suffering Change.

• Lower Standards: It’s much easier to change if you don’t have to follow the rules.
• Increased Budget: Change always costs lots of money.
• Increased Headcount: Someone has to take the blame. There is always plenty of blame to go around during change, and who better than new hires.
• Buying other Companies: Sooner or later there has to be a good one.
• Bad News: Scandals are inevitable.
• Good News: Here comes the microscope.

After a period of Change companies must evolve into the next stage; Containment. It should be obvious that this includes layoffs, reduced budgets and all the other money saving efforts of cost containment but it goes even further where they try to contain the workers and products from leaving. When things get bad companies will do anything to contain the disruption that caused the change. Contain the bad news… contain the competition… buyout the competition.

Another branch of the Containment bureaucracy is Reorganization aka; re-org. It can go by other names; retool, retrofit, right-size and smart-staff are some of the insulting euphemisms in current use but in the end it amounts to musical chairs. It can be as simple as a name change or a complete shake down. My computer services department has been moved from the original place in the Engineering department, to the HR group, then Facilities, then Financials and finally to the current location under the Operations department.

Finally comes Consolidation. If you didn’t die then you must be stronger. Things either get back to normal or you get bought out by some other company that is in another stage of the life cycle. Consolidation brings a period of stability that can lead to productivity and worst of all profitability. That’s just the kind of thing that can start the ball rolling again. If you are lucky enough to survive all the way through the lifecycle to the Consolidation stage then you know it’s only a matter of time until the Change will start again.

The System’s Analyst Chapter Two: The Curse of the Systems Analyst

Here it is, the Bottom Line on top: Never take a job with “anal” in the title. If your job has an anal in it anywhere chances are you will end up acting or feeling like one.

When you finally get good at being an analyst you can’t help but go too far and end up over-analyzing everything. You know you have gone too far when you start questioning basic things like what day it is or what time it is, even though you just looked at your calendar or watch. Then in an effort to compensate you head toward the other extreme where you skim over everything and ignore all the details in a recurring vicious cycle. Your relatives, friends and co-workers will be sure to let you know when you are analyzing too much or too little by starting many of their sentences with “No one cares…” or “Wait a minute…”.

The analyst is cursed because everyone assumes they know it all. Engineers get to specialize in just one thing but analyst are supposed to cover everything. If we knew it all we would have a better job than this! People also assume you will do anything and expect you to cover everything from the Enterprise Infrastructure down to the paper towels in the restroom. They will also accuse you of reading all their data or listening to their calls even though you can’t keep up with your own email and phone calls. Again, if we were that sophisticated we would find better stuff to snoop.

All your co-workers and customers will assume you are available anytime, especially vacations and holidays. They are allowed to be freaked out during the disaster but you have to stay cool. Worst of all, everyone assumes you want to help them with their friends, relatives and acquaintances for any and all issues and projects. In the same conversation where they were complaining about being overworked they will freely assign you non-company related tasks and never think twice.

The analyst usually works. You have to be self-motivated and not rely on others for help while at the same time you must also work well with others as part of the job. This creates a bi-polar response where the analyst can’t get along with the other team members when they are working on a team of one. Sometimes a Systems Analyst has been around so long they need to separate them from the herd so they reassign them as Senior Systems Analyst. It’s not a promotion, just a distinction so the other regular analysts will know to leave him alone.

If I had it to do all over again, I would get me an honest job without any anal in it.

Server Room Series – Chapter 5: Quarterly Outages

I had a friend who was a Millwright and every summer they had a two-week “Shutdown” at the mill so they could rebuild all the machinery that had to run 24\7 the rest of the year. My friend was mad that he had to work double-time for two weeks while everyone else had the time off, so he eventually switched from millwright to truck driver even though the pay was less and the job was not as interesting. No one wants to do the dirty work.

In the hi-tech world the shutdowns are more frequent and our company has settled on once every three months around the second week of February, May, August and November. We have two basic types; minor and major Quarterly Outages. A minor outage has to have basic services up so people can keep working on projects while the major outages require every device be turned off for a complete fresh start.

The timing of the shutdown is critical so the last one down has to be the first one up or every other machine will sit there on boot up waiting for the other guy to start the handshake. The network has to be first, the DNS name resolvers are second, and then come all the secondary service providers so that none are left hanging. Each type of server has further dependencies to get the desired clean start and it is not uncommon to have to go back and start over.

There is that one brief moment in the middle of the major Outages when everything is turned off and the server room is almost silent for once, except for the background hum of the AC and UPS units. The place is almost dark with all the racks silent and missing their blinking lights it is as spooky and feels as bad as having the power go out completely. It is almost a peaceful moment except for the anticipation of something going wrong on reboot and we are always anxious to get it over.

Even though we have test machines that we practice on before each outage and we try to anticipate every contingency there is still always some fallout after the Outage is over. Every outage is good for a few panic attacks and there is almost always that one server or service that does not come back up and some patch has to be removed or the whole machine rolled back to a previous version. Finally we declare the Outage is over and take a comp day off to recover from the all 24 hour marathon but right after that we immediately start planning for the next Quarterly Outage.

Governance Risk and Compliance

Governance, Risk and Compliance or GRC is another business process that does not quite match how we live our day to day lives. In the real world we live by laws (Governance) and if we get caught breaking laws we get punished (Risk), and then we promise to be good again (Compliance). In the Business world they get off easier. First, they get to write their own rules and then they hire their own Auditors who are always nice when it comes to the punishment.

In our industry the Governance is called ISO9000 and you can’t do business without it so everyone gets Audited, no exceptions. It all starts with a written document of what the Company promises to do and then another document to prove they did what they said in the first document. If they don’t match the Auditor declares a violation and you have to change the documents.

Here is a real world example of the changes in my Backup document.

2000 Version 1: Tapes will be shipped off site every Tuesday.

2002 Version 2: Tapes will be shipped offsite once a week.

2005 Version 3: Tapes will be shipped offsite once a month.

2008 Version 4: Tapes will be shipped offsite as needed.

2010 Version 5: Tapes have been replaced by hard drives

Each of these revisions was the result of a failed Audit where I had missed a Tuesday or we decided to decrease our frequency to save costs. I wish we had made it generic from the beginning but we had to learn the hard way.

In the Business world the Governance is self determined, they get to write their own rules, in the real world the Governance has more bite and we don’t get to write our own laws. In the business world the Risk is all monetary but in real life the Risk is all too real. In the business world the Compliance is voluntary but in real life Compliance is enforced with a heavy hand.

When it comes to enforcement in the Business world, it seems they get away with more money for less punishment compared to the real world where criminals serve more time for stealing less money.

The Last Typewriter Repairman

While attending Trade School in the early ‘90’s I ended up taking an outside job for the one semester when there was no Work-study job for me and I accepted a part time position at a business called Cascade Computers. While they did so some work on computers, the majority of the workers were involved with typewriters. Repairing, maintenance and cleaning typewriters was a big contract that once was a major money maker involving several businesses but they all ended up closing down one by one until only Cascade Computers was left servicing the Typewriter Industry.

At first I was embarrassed to work on the old fashion machines and was longing to work with computers. But once I finally settled down long enough to watch the expert guy put the pulley spring back on the rollers using two long skinny crochet hooks I could tell there was real skill involved. He was able to dig into the guts of that contraption and fix things again with the incredible dexterity. Most computers are quite simple to service but if you have to take apart an electric typewriter there will be lots of parts and pieces.

These were top of the line IBM Selectric Machines that were not like your regular Typewriter. These had a Golf Ball with letters instead of keys and no reaching up to push the carriage back for each line, instead the golf ball moves back and forth. These devices were state of the art in their day but when I got there they were already well on the way to extinction. The thing I remember most about them was the humming noise they made even while unattended that made them sound more like a refrigerator than a computer. And that was nothing compared to the noise they made when the worker started typing.

The job of cleaning these dodo-birds consisted of several steps requiring the use of toxic Chemicals to treat the different kinds of rubber on the rollers and spacers. This only added to the overall discomfort of having to disturb workers in their office to clean the thing and then their eyes would get bigger and bigger when the smell of those chemicals hit their noses. Usually I went by myself to the various places on the list of units to be cleaned. For the big Jobs we would fire up the old van out back and get all the workers together for an assembly line approach.

The funniest part was trying to find the typewriter at most places and then when we finally did find it, it didn’t need cleaning because it hadn’t been used. There were some places that really did need cleaning and then we earned our money but most of the time we felt like the Maytag repairman.

I love to kid around by saying I was also the Last Railroad Gandy Dancer. Sure there are people who still do that same job today but they never heard the songs or felt the thrill of working on the traveling System gangs way back in the last Century . I even go so far as to tell the joke that I will someday be the last computer Systems Analyst but when it comes to the Last Typewriter job, I swear it’s true. Seriously, when was the last time you saw a working Typewriter?

Stupid Work Tricks

This was going to be the Top 10 Ways to Win at Work but I couldn’t make it all the way there to 10 and nobody wants a Top 7 so I changed the name instead. Everyone knows how to use the standard excuses at work; “It’s not my fault”, “It’s not my job”, “I tried my best” but these are nothing more than the adult version of “The dog ate my homework.” The modern age requires more sophisticated explanations. Herewith are my favorite selections that are equally suited for any work environment.

Every Minute Spent

One of the favorite work manipulations is to tell people that you don’t have time for them because, “Every minute spent doing (Thing A) is a minute I won’t be able to spend on (Thing B).” Where Thing A is what they are asking for and Thing B is something even more important. There is always something more important, it could be from another department the user knows nothing about or anything the Boss wants on a tight deadline. The “Every Minute Spent” machination is not perfect and can even backfire in certain cases, be sure not to use it on anyone who wears a stopwatch around their neck.

It’s Not Supposed To Do That

I can’t take credit for this one but it works so well I added it to my play list and let it work its way to the top of the charts. I first heard this from the electrician who promised everything would be fine when we put the UPS into bypass mode and then: BANG~! the whole building goes dark and the batteries are not in the loop so all the servers crashed. Anytime something goes wrong you can bet “it’s not supposed to do that”.

Sunspots, Gamma Ray Bursts and Atomic Instabilities

This is the perfect ploy for any problem reported by any user. If they want to discuss this scientific subject your real job is done and if they don’t you still win. It is well established that certain hi-tech equipment is susceptible to damage resulting from sunspots, though it is in practice quite rare and unique but as long as it is ever true it can always be a suspect. Gamma Ray Bursts are less well known and harder to prove while Atomic Instabilities are pure fiction made up on the spot. You always have to be quick on your feet.

Virtual Cloud Security Infrastructure

Blame everything and anything on the virtual cloud security infrastructure. Everybody has one but no one can reveal it so it is therefore the perfect catchall for mysterious events. If you are ever arrested you must remember to ask for a lawyer and if you are ever in an uncomfortable computer situation, you must invoke the Security card. This trumps everything and after that even a lawyer won’t help.

ROI, VPN, ABC, ETC…

Any back-to-back-back three-letter-acronym will immediately hypnotize any human who is still listening. In fact no one ever listens past the second one because they feel the hypnosis coming. It’s still fun to play the game.

Those IT Guys Did Something

Yes, even the people working in the same department are reduced to blaming each other in order to maintain efficiency when all the user really wants is someone to blame. They have been trained to expect nothing more.

At Least No One was Injured

Back on the Railroad injuries were so common that we just happy no one got killed and even that was not always true, but in the hi-tech world all the injuries are of the personality and hardly ever the person.

15 Centimeters of Fame

As an Aspiring Artist and Wannabe Writer I am always looking for places to practice or display my work. One such place is the little whiteboard in the break room ( it is gray in color actually) where we fill in the status report for the ongoing coffee project. This miniature billboard is split in two with decaf on the left and regular on the right and the whole point being to put the day and time of the last brew so we can know how old it is. Wouldn’t want to throw out anything fresh or worse yet accidentally drink last night’s leftovers.

Because we never bother to change the time, it is perpetually set at 8am so the only info that changes is the day and this is in the abbreviated form of Mon, Tue, Wed etc… So there it is, my three letter marquee where people can read my latest writing or view my latest art, depending on my mood when I post. I go from calligraphy to swirly font and even tried a foreign language, Spanish, but stick with the basics for the most part.

M – T – W – Th – F …feeling lazy

t T * ( = …feeling cryptic

It is not a big deal, in fact everything about it is small. The area on the board is small because it is surrounded by further instructions that have faded in time but still restrict the actual size of the area in which to work. The potential audience is very small because there are only 70 people in the office and they don’t all go to the kitchen and of those only a few will take the time to read the coffee news. But in spite of those limitations I still manage to make the most of that recurring art show and find myself spending a few moments now and then imagining future works.

So it must be a big deal only for me. But we take what we can get in life eh? . If I squeeze the letters together I can get a fourth letter in there like these:

Moon – Tooz – Wenz – Thuz - Friz

Mony – Tuey – Wedy - Thry - Friy

I was going to condense the letters even further so it can go up to five characters per day but then I realized that would be more writing than I am willing to commit to.

Tales From The Server Room

There are strange things done amid the grinding hum of the machines in the server room.

There are many a fright on a sleepless night in that land of doom and gloom.

The unlabeled cables, those disarming alarms, red lights are blinking and no time for thinking,

Oh the troubles you’ll shoot amid the endless reboots, in the shadow of the server room.

In order to properly appreciate the complexities of the activities that take place in the server room we must first set the stage so people who have never been there can get the right feel for the atmosphere in such a place.

The server room is noisy. The noise is bad enough to consider using ear protection but that would imply a commitment to staying for a long time and it's always better to get in and out as fast as possible. There are big headphone style noise-suppressors that feel heavy and clumsy and make it even harder for a big guy to slither around behind the racks. There are little foam-rubber ear-pugs that are irritating and unsanitary. In all cases the little relief from too much noise does not justify the loss of actual hearing for the sounds you do want to hear; the cell phone (notwithstanding vibrate mode), the rare visitor (usually site manager) trying to get your attention, or any of the many equipment alarms that might go off.

The server room is cold. I have a sweater on the back of the tall tan rolling swivel chair but I don't always put in on as the first order of business for the same reason as the noise suppression; I don't want to be in there that long. On a hot summer day it is nice to go there just for a blast of cold air. I tend to sweat like a dog with just the slightest amount of effort so I end up in there to cool off all year long.

The server room is scary. That's where all the bad things happen. All the scariest things are in there. There have been hours with the sweater and earplugs left in even when I leave the server room because I know I am going right back in there very soon. There are dangerous electrical circuit panels and distribution cabinets. Two HVAC units the size of $29 U-haul truck and one transformer that raises your hair in the air in a 20 foot radius. The HVAC units take turns vibrating the floor like a constant 2.0- earthquake.

Going into the server room is like going under water. The temptation to hold my breath is soon replaced by an urge to get out again. Sometimes it feels like a full blown panic attack and it is small wonder that some people can't handle it. My best defence is to use my old railroad tactics to tolerate this fear. What is the worse that can happen? What will we do then? Just having a plan is a big part of the relief and just knowing we've survived similar things helps a lot too but the single best tension reducer is using the word "we" instead of "I" even though you are still alone in the server room.

Some people like to call their server room the Data Center, but it’s still just a server room. You can call it a closet or a gymnasium but if there are servers in there it can only be a server room.

Pickers and Peckers

I was amazed the first time I saw a famous writer on TV typing with just the two forefingers…tipping and tapping away like a real beginner. I could not believe they could finish their book let alone be productive with such primitive typing skills. Then I saw a well known Journalist on TV showing the same technique, then I saw it again and again and it finally dawned on me that this is the secret to their success in literary endeavor.

If it was just Andy Rooney and Studds Terkel then it might be on account of eccentricity but this phenomenon includes all the great masters, Pearl S. Buck, James A. Michener and even the venerable Walter Cronkite, why I bet Shakespeare would’ve done it if he could’ve got his hands on a typewriter. At first I thought maybe it is just a few Genius that benefit from this oddity but then I started seeing it in the hi-tech world where I know that person did get a good education and keyboard skills had to be included.

I remember learning to type the proper way back in the 10^th grade and looking down on anyone who couldn’t keep up with 35 words per minute, or even find the home row. Now that very same snooty skill set seems more of a hindrance than help. No wonder I can’t get ahead; I never learned to pick and peck.

Could it be these people do know how to type with all fingers but are deliberately going with just two on purpose? Do they somehow benefit from the loss of speed that better matches the speed of their thoughts? Maybe those two fingers are directly connected to a special part of the brain that connects straight to the memory banks while the other fingers have to stand in line to send or receive data. Ten finger typing is great for secretaries and stenographers but great writers can’t be creative 10 ways at time. They need to channel down to the core of one thought rather than trying manage 10 trains of thought down 10 different tracks to the roundhouse.

In the computer world data can be sent using Serial data ( one bit at a time) or parallel ( 4, 8, 16… bits at a time) which in theory means more throughput but in the end the data still has to go back to the one bit at a time mode for processing so in some scenarios it is faster to stick with serial. And because we type so much slower than we think maybe it does make sense that we should slow down to increase quality.

So I tried to pick and peck but it wouldn’t work. I have already been programmed the wrong way and it is too late to break the mold and get anything useful in the end. So I will have to stick it out with my old fashioned methodologies, and suffer the possibility that I am too good a typist to be any good at writing.