[ Previous entry: "One Person at a Time" ] [Main Index] [ Next entry: "Odd and Ends" ]

10/15/2003 Entry: "Ghost in the Machine" No, not the album by The Police, but something that has been bugging me at work ever since I started this QA job started in 1999: someone, or something, is screwing with the machines we work with. I started with International, where we places machines in remote sites around the world. Some places were never any trouble. Some had rampant security problems. In Australia, for instance, some of our testing machines were turned into mirrors of someone's mp3 collections, and often people would install all kind of other software on them. The head of the Australia tech team just blew it off. In Germany, one site had problems were people would use the machine to browse the web. We'd log in through a desktop remote viewing program and watch someone browsing porn on the web. We'd even log it (this was a machine that tested web page speed). "Nein," said the person in charge of the tech center. "We have no records of anyone coming or going out of that locked room." We later found out "records" meant a clipboard with a pen on a nail next to the door, and "locked" meant "in a locked building, behind a door with a lock," that is, the room the machine was in was wide open. "People who enter that room must sign in and out, and no one did, so no one must have gone in there." Uh huh. The activity stopped when we threatened to put in a web cam. When I was folded into the domestic (US) side of the group, I thought having everything local meant no more of these problems. Wrong. At first, it was just a matter of petty theft. Small things vanishing, like power cords, label makers, RAM, LAN cables, and so on. Part of this was because we were in a lab, we'll call Lab1. Lab1 had restricted access, but by "restricted" it meant "not many employees and all the guards and cleaning staff." While our badge readers are supposed to track who comes in and out, there were "ghostings" where people could just follow behind the person who badged in (there was no guard or even a camera). When our group started locking things in a cabinet, the theft problem stopped. Then weird stuff started to happen to our systems. How weird? Weird enough to be crafty, and for a while, untraceable. It was done by someone who really was subtle. Like they'd change a crucial configuration file, or unplug random machines (either from the network or the power cord itself). At first, we thought it was just by accident, but then we started to notice that some issues were very specific, like someone KNEW which file to change, and what bit to change from a 1 to a 0, for instance. Many of it was by software we wrote, and only we knew how it worked. This never was released on a grand scale, but obviously by someone testing our security in some way. It worked, because we started locking down stuff with passwords, screen savers, and permissions. Most of these shenanigans dropped sharply in number, and were restricted to hardware only (usually unplugging every other machine in one rack). But then software "issues" began to creep back up. A lot of it is random. Like someone will change a shortcut in the startup menu that will point to another program, or no program at all. Or they will delete a file, change its CR/LF, or other things that are hard to trace down until you've exhausted all the logical options. This isn't done daily, but usually in bursts of every 2-3 months. Most of it is not really very damaging, but just annoying. The whole thing smells of someone deliberately being prankish because they are bored, or rebellious. Then we moved to Lab2. Lab2 was bigger, had better security, and much more restricted access. It even has a camera. The pranks stopped when we moved in there, and didn't come back up again until a few months ago, when again, it started back up. It coincided with the time another group we used to share Lab1 with started also using Lab2. I have a pretty good idea who's responsible, but this clown knows that he can do anything with a certain login we use for remote scripts. Yes, I have proof. No, no one cares. But he's not responsible for ALL of it. So who is? I mean, once you filter out user error (on my part - about 90%), something you didn't expect (like hardware issues - 9%), you have 1% unaccounted for. Someone respells "mount 10.0.10.3" to "muont 10.0.10.3" in a script, puts leading zeros in an IP address, adds one space to the end of a line to change a variable, and you know someone did it because the "date modified" has changed, but the trace login and IP address are masked, and the only way that would happen is if someone was doing it intentionally. Could be him. Could be others. I have no idea. So every few months, something goes down because someone saved a text file as an MS Word .doc format, deleted a log file, or unplugged some LAN cables, inserted paper into the port, and plugged it back in. And I have to put up with it, because the prankster never actually does anything horribly damaging, just stays under the radar as "anonymously annoying." Then I have to ask why. I can guess it's a boredom issue, or "revenge against the ruling class," or maybe "because I like to mess things up for the sake of messing them up." It's not a personal attack, because it's done to a lot of people in our group, other groups, and beyond. Still, it's unprofessional. Maybe I'll never know the real reason. Maybe it's a life lesson I have to learn from. It's still annoying.