and Words
Something′s at the edge of your mind
You don’t know what it is
Something you were hoping to find
But you′re not sure what it is
Then you hear the music
And it all comes crystal clear
The music does the talking
Says the things you want to hear
I’m young, I’m wild and I′m free
Got the magic power of the music in me
I′m young, I’m wild and I′m free
Got the magic power of the music in me
Lyric excerpts from Musixmatch.
On a typical morning in 2000, I wandered into work. I was a software developer on Microsoft Windows. Traditionally, Microsoft prided itself on very lenient flex time, allowing employees to come in whenever they liked and get their work done. The particular hours didn’t matter. Unfortunately, some coordination with other teams meant that this was suspended for now. I arrived around 10 am and immediately looked at the most urgent issues, the stress failures.
Most programmers had several computers. They instructed everyone to install a recent build of Windows and run a set of stress tests as we left for the day. All these computers had a remote debugger attached to investigate any problems. These computers, generally used for testing, were to be returned to normal use by 1 pm. Arriving at 10 am gave me a three-hour window to investigate any issues we found overnight. The command used to connect to that computer was named remote, so we referred to each such investigation request as a ‘remote’.
On my busiest day, I got 30 such remotes. If we do the quick math, and with my arriving at 10 am, that means I had about six minutes to investigate each one. When asked to describe the difference between programming as a developer or tester on the Windows team, my most common response is this. Coding as a developer is like programming with a gun to your head.
A remote is not a conventional debugging session
Well, it is and it’s not. It’s not what most modern programmers will recognize as a debugger. You don’t see your source code neatly arranged on a window. You don’t see your local variables neatly enumerated, with their values displayed as you hover your mouse over them. We got effectively a terminal to an interactive session with a remote instance of ntsd.
Each session dropped us into assembly code, just a symbolic representation of raw machine code. We could see the actual zeroes and ones (or their hexadecimal equivalents) in memory as the processor interprets them. We could display the contents of the registers and memory. Each developer, armed with a set of memorized commands, deciphers the state of the computer. The remote terminal allowed you to execute a command, and it responded with a stream of text. That’s all you got.
Each command retrieved diagnostic information like the contents of the stack, disassembled a portion of memory (displayed it as assembly code), or displayed the raw contents of memory (or as a particular type).
However, this was par for the course. All Windows developers who investigated these computer failures needed to learn to conduct their investigations, and it was often trial by fire. It typically starts small with one such remote debugging session, and you cut your teeth by fumbling around with each command. If you’re lucky, you’ll have a mentor who shepherds you through this process. If you’re curious and crafty, you’ll connect to someone else’s debug session and observe what they did to diagnose the issue. If you’re neither lucky nor crafty, you’ll spend a sizeable amount of time with the help file listing all the debugger commands.
Finding yourself in the middle of nowhere
In the vast majority of these instances, the debugger knows precisely where you are and what caused you to land here. It does so by analyzing symbol files against the running binaries and overlaying them. These often surface when executing instructions like “go to the local grocery store and buy a Coca-Cola,” which seem trivial. What happens if you go to Whole Foods, which doesn’t carry it? What happens if they’re simply out of stock? Or if the store is closed? In these cases, file the bug and move on with life. You’ll fix it later.
Occasionally, the debugger has little clue about where you are. It cannot find the symbols, or they fail to match. The debugger does the equivalent of pushing you out of a plane with a parachute (and a dead phone) as it tells you, “You’re in Montana; good luck.” Most of Windows (and many other components) are built into what we call Dynamic Link Libraries or DLLs. They can tell you which particular DLL the problem lies in, but not which line of code, nor what it was trying to do. One such DLL can span thousands of lines of code. It’s the proverbial needle in a haystack.
In such instances, many developers throw their hands up in the air and give up. Extracting useful information from such a debug session is difficult and typically not worth the trouble. If this happened once, it’ll likely happen again; it’s simply a question of time. To resolve such a failure in this way is reasonable.
How to navigate that forest
That said, there were instances when that computer in suspended animation held the secret to a very elusive bug. Your choices are to let it go, knowing that it may hold the key to a particular problem, or to jump out of the figurative plane somewhere into Montana. Having done this often enough (remember as many as 30 in one day), I meticulously developed skills to navigate these sessions, even when we couldn’t resolve symbols:
- The vast majority of DLLs name their exports. The oversimplification is that we may not be able to listen to conversations, but we can track outgoing calls. If we determine that a figurative call goes to your spouse, your doctor, or your financial advisor, it helps us narrow down the scenario.
- Examine the parameters on the stack. Even if you don’t know where this code is calling into, there’s a convention in which it passes information from one call to the next. No, you may not immediately know which internal function it’s calling, but you can safely determine that it takes four parameters instead of three. This will help you narrow it down.
- Look at the contents of memory. I’m not suggesting scanning through a hexdump of all the memory. Parameters passed through the stack are often pointers, which are, more often than not, a data structure. Looking at that memory may help you determine where you are and what you were doing. For instance, to this day, I can recognize the contents of a .wav file if I see it in memory.
There are, of course, many other tools and techniques you can use. These are the ones that come to mind from decades ago.
My Magic Power
I’ve listened to ‘Magic Power’ from a playlist many times. This song adorned my teens and into my adulthood. I’ve even seen Triumph in concert; they gave a phenomenal show. Though I’ve sung the words to this song many times along to the music, it took years before that phrase even registered: “Got the magic power.” Only in that instant did I ponder, “Do I have a magic power?” That idea flies in the face of humility.
Debugging obscure scenarios like this is likely the closest I’ll have to a magic power. I solved puzzles that others had abandoned as impossible. Occasionally, others asked what I did to solve the mystery, and I’d happily demonstrate. At times, they’d respond with, “I would’ve never thought to check that or do that.” On rare occasions, someone asked me to help them debug their bug.
Good debugging (and programming) is the lovechild of several traits. First, experience will inform the possibilities. What set of commands do you have? How are parameters stored in the stack? Second, intuition will yield discoveries. Occasionally, objects will seem oddly out of place, much like noticing the odd flowerpot on the porch where you’d find the key to the house. Third, perseverance will fuel your quest for an answer. In my case, it bordered on pigheadedness.
I won’t tell you that I was the best at this; some others were far better than I. However, I worked the numbers in my head. Operating systems programmers are rare; perhaps a few hundred or thousand people do this work. Only a small subset of them were able and willing to ‘find that needle in a haystack’. Still, I was among a very exclusive set of people.
Moreover, the tools have improved, and few people need to do this sort of debugging anymore. I look back fondly upon those days. Those moments on that terminal, where characters scrolling on the screen yielded the answer to a mystery that only I could decipher.