The New Job
What have I got myself into? This network is a mess! Let me back up …I started my new job two weeks ago now. Monday morning I arrived at 7:10AM. When asked what time work started my boss had said “people start coming in between 7:00AM and 8:00AM”. But I could see that she would rather folks come in earlier. That was fine with me as I am trying to avoid my local inner-city rush hour. I ended up arriving a little too early, about 7:10AM. Only one other guy was there, and I could see that he was wondering why the hell I was there so early. Everyone else shows up at 7:55AM. No matter, I was able to spend some time cleaning junk off my new desk.
When my boss showed up, she grabbed me and said “follow me!” Then, out the door we went and up a hallway to a strangely situated room near the elevators. She cracked open the door, and ushered me into this roomy “junk closet”. “Grab one of those”, she said. I picked up a Dell box, and carried it back to my desk. I had a new PC. Woo-hoo! Next we headed for the elevators, ducked inside, and I watched as she cracked open a special panel and requested the “7th floor”. This floor is reserved for hospital staff only. Once there, the doors opened to what seemed to be a general storage area. It was pretty loud, as you had to duck and jump over duct work and generator-like devices. Amongst the rubble was old hospital beds, food preparation equipment, ancient PC’s, and … a rack of 19 inch flat panel displays. I grabbed one, and back we went to the elevators. While waiting for it to come back, I heard a “lullaby jingle” which rang loud and clear from all directions. I turned to look, but all I could see was a radiation cage for the decaying of radiology equipment. “Did that jingle come from within there?” “No, that’s the intercom … a baby was just born”.
Once back to my area the boss says “work on setting up your computer, I will be back later”. So I unboxed everything and starting going through the motions. I must say, I was right at home. When I logged in with my new account, I all ready had a couple of mapped drives – one of which was full of various software packages. I got to work installing Office, antivirus stuff, and some utilities that would probably come in helpful like Ethereal.
About the time I was wrapping up and checking my mail for the first time – the boss returned.
She turned to one of the veterans of the department and said “remember that project I had you working on … well it needs done now … so have him work on that and give him the details”. Then she was gone. What was this project? Track down every port on every patch panel in the hospital – and record what port on what switch that connects to. Basically, I was asked to build a compiled list of locations of wall jacks to patch panels. Then, patch panels to switching equipment. This might take … a while.
An hour into the project I could see that the wiring was a Goddamn mess. The good news is that I could complain about it all I wanted, and tell everyone how awful it was – because nobody that works there now had anything to do with it. The old regime from years past is completely gone (disposed of) and after only one day I could see why.
Three days into the project I felt I was making some headway, thanks to the help of an intern that they had assigned me. The intern had no knowledge of networking but had been working for the hospital off and on for years. The advantage was that he knew where everything was located now (and for that matter were things USED to be in years past).
Now here are some tips for anyone who might run network cabling.
1) Label the wall jacks. Even if you don’t have a fancy sticker labeler – you can write onto the wall jack with a permanent marker. But hey – write the correct thing on the wall jack while you are at it.
2) When you put in a patch panel, they are labeled one to twenty four (or more). When you put in a second panel, it too will be marked one through twenty four. Don’t be a fucking retard and leave it that way. You can’t have two jacks that are both port “18”. Duh.
3) When you run out of ports on the patch panel, and you are running some more cable – don’t reach around the back and yank out stuff you aren’t using at the time. Spend the extra fifty bucks and get another panel.
4) Don’t put three piece oak furniture in front of network jacks. If you do, you should put something on the wall or on the furniture which tells the network administrator what’s behind the obstruction.
Most of these seem like common sense. Perhaps I could sum all these up with one master rule – “Make sure your network administrator has some idea of what the fuck they are doing”. That being said – the boss approached me on my third day to see how things were going, and then asked “how long until this is done”.
I was honest. Maybe too honest. I went on to tell her that this network (at least the first floor) was in the worst condition that I have ever seen in my 12 years of IT. “If I were to do this right, we are looking at several more weeks of running around this place”. Her eyes widened and I could see that she was about to be … not happy. “Not weeks! Days!”, she said, “so let’s try this again … how many days will this take”. I wasn’t sure how to retort. But she is a serious woman, who doesn’t kid much, and wants it all straight. “I should have a somewhat useful list and diagram in the next two days … but it could be months before this all gets straightened out”. She understood. She said “I understand why you say months … we don’t have months … you get it done in two days!”.
Two days later I had a pretty good list of where various ports went to. But there were other problems. Some wall jacks were labeled incorrectly, and we didn’t have time to test each port. Running around with a toner (fox and hound Fluke device) was taking far too long (even with two people working for nearly 10 hours a day). Even now, two weeks later, I am upset that it’s such a mess.
It was made pretty clear to me that this was my network. My baby. And my baby is quite ill. I hope that in the years that follow I can turn it all around. But certain things have to stop. Like the build maintenance people running our network drops for us.
My second week of work, I spent the first two days in orientation. You learn all sorts of interesting stuff there. HIPAA rules and regulations, how to deal with spilled body fluids, how to help people in need, emergency codes, infection prevention … the list goes on. For two solid days we were packed into a hot little room where speaker after speaker came in to entertain us with odd movies, slides, and other such presentations. It was all good information, and it was all necessary. But still, I was quite ready to “get back to work”.
The rest of the week was broken up into a few tasks. They would like for me to take over some of the printer/copier/scanner support for Ricoh devices (the hospital has dozens of them everywhere). I went out and fixed a few printer jams to help the “PC technician”. I even helped out on a strange wireless problem. Oh, here is an interesting story.
The emergency room had called a couple times one morning. They have two carts with laptops in them that they roll around to perform patient registrations with. Of these carts had gone belly up on them and was rolled back to us for repair. I had inspected it, ran several tests and determined that “it’s fucked”. Simply put, they damned thing was overheating and shutting itself off. It was years out of warranty and would need flat out replaced.
In the mean time, their one other cart was experiencing issues. As they rolled it around the department it would sometimes loose it’s address and begin dropping packets. I had a look, and confirmed their claims. There were nine access points in this area, most of which were about eight feet apart. It was complete overkill and I had to wonder if there were enough channels to keep all those signals separated. I also noticed that while there were nine access points, there were only eight cables going into a switch. Like everything else in the hospital, nothing was labeled so it took me an hour or so of climbing up and down a ladder and running back and forth to track down and label all of the connections. Sure enough, one of the access points was getting power from a POE injected patch panel … but was not sending or receiving data through the network. Basically, every time a cart was lucky enough to connect to this access point, traffic would begin dropping.
I also noticed that two other access points were alive and well – but would also drop packets when connected. Was it that the switch was managed into a VLAN of some sort which was configured poorly? They only started having this problem a few days before when this new switch was put into production. Since the consultant that was hired into the project is the only one that would know, I asked him about it. I also asked if I could gain access to this switch for the purpose of troubleshooting. He declined to give me that information or to assist in my problem.
The following morning I went in with a mission. I removed all of the access points from the Cisco switch and plugged them into an old Netgear 24 port switch from a “junk pile”. Sure enough, everything worked fine. I wandered around with the cart a few times letting it connect to various access points, and renew its address each time. All the while, I ran continuous ping back to the default gateway. I dropped a single packet throughout the entire test. It was clear to me that I had made my point.
I headed back to my department, and I grabbed the veteran. “I think I have narrowed down the problem to that new switch”. I explained what I had done. “I guess you had better tell Rob then”. Rob is the “consultant” for a Cisco equipment upgrade. I went into the little office which was set up for Rob and relayed what I had done, and what I had found out. “Are they plugged into the Cisco now?”. “No”, I said, “but I can put them back in for you. I just wanted to show you that it works when they are not in the Cisco device”.
With that I headed back to the emergency department, and started plugging the access
points back into the Cisco. When I did, none of the link lights came back. That was odd. Then the door to this room cracks open. It was my pal from our department. “I thought you were kidding about that. You really pulled them out of the switch?”. Here is where I’m really confused. I thought he was kidding. Yes, I un-plugged the access points from the Cisco switch, and put them into an old Netgear. Yes, I told the Cisco consultant what I had done. Yes, I fail to see a problem with doing so.
“When you left”, he adds, “Rob asked me if you had access to these switches. He thinks someone is going around screwing with them, and now he thinks we’re doing it”.
I genuinely felt bad. The consultant can kiss my ass. I think he is a huge waste of money. If I have to unplug a device and plug it back in, I should not have to go ask permission or wonder if it’s going to break something. But I did feel bad for my department buddy. I had just made life a little harder on this guy, and he is all ready pretty well stressed out.
Having plugged everything back in, I headed back with my tail between my legs to talk to “captain dickhead” the consultant. I apologized (against my better judgement). “Sorry about that”, I said, “I guess I didn’t see a problem with moving devices from one switch to another to troubleshoot that issue – and I wanted to ensure it was a problem at the switch before I bothered you about it”. He continued to scorn me, “well that’s the problem … you can’t just go unplugging things”. I failed to see his point. “Well the users are going to be unplugging things all the time. We can’t stop that. My question is, why are the ports all dead?”. He connected to the switch and had a look. “All of those ports are in ‘error/disable’ mode”. “ah, well can’t we just re-enable them then?”. “I don’t know”, he says.
What don’t you know about this? Aren’t you the fucking “expert”? Well I’m not Cisco certified, but I would be willing to bet that the ports just need re-enabled. Oh, by the way – can you fix it so that they won’t do that again for when the power goes out – or the POE patch panel finally bites the dust? Thanks!
He re-enabled the ports. It was fine. And after some more mucking about in the settings, things were back to normal. Thank you Mr. Cisco genius. I could see that I had made an enemy for life with this guy, but seriously, who gives a shit?
One thing about that whole mess troubled me though. How much of a network administrator can I be if someone else manages all my equipment? Maybe I am overstepping my bounds here, but I was hired because I had routing and switching experience. What’s going to happen when this guy is “done”? Will he just leave, and we will have to figure out how to make all of this stuff work? Is he going to provide support after the project is done? It seems that there is so much work to do, and so little time to do it, that most of the staff would rather not be involved. But I don’t come from that kind of environment and I am going to have a hard time adopting that mentality. I guess I just need to let that one go.
In the mean time, this is the “nicest” place I have ever worked. Aside from the vendors and consultants (the outsiders) the hospital is like one big family. Everyone there is friendly, outgoing, and busts their asses to get the job done. I like being part of that. I like greeting people in the hall. I like watching how all of these departments, offices, people, equipment, etc. all work together rather flawlessly. It’s like one big ant farm, and I have a magic key which allows me to explore the whole damn thing 24 hours a day (with the exception of surgical areas of
course!).
Oh, I have blogged far too much this time. More exciting news later!
Posts
