How To: Debug the WRK on Mac OS X Using VMware Fusion

By Alexander Schmidt

Debugging the WRK on Mac OS requires two virtual machines (VM): the WRK VM, which will be used to debug the kernel, and the debugger VM, which hosts the kernel debugger. Both VMs will use their serial port to communicate with each other. In a previous post, we have shown for VirtualBox how to connect both serial ports over a named pipe. Now, we are going to describe how to configure VMware Fusion 3.0 virtual machines to accomplish the same goal.

Before we begin, here is a sketch of the architecture of our WRK debug scenario:

WRK debug architecture on Mac OS.

WRK debug architecture on Mac OS.

In this tutorial, we will not describe how to configure WinDbg or the WRK to communicate via the COM1 port since this is described elsewhere. We only concentrate on the configuration of the named pipe to connect both VMs. You will notice that the following screenshots were taken on a German system installation. However, the menu locations should be the same in your language and we will also describe what is shown in the image.

If you want to configure the serial port in VMware Fusion 3.0, you will only get this dialog:

Serial Port Configuration in VMware Fusion.

Serial Port Configuration in VMware Fusion.

Which means that the standard configuration for the serial port is used to connect your virtual machine with printers available on Mac OS. The only other option in the GUI you have is to connect the serial port with a file, which is not appropriate for our purpose.

The solution is to manually configure the VM configuration file. Configuration files have the suffix .vmx attached. If you are using a version of VMware Fusion prior to version 3, you will find the configuration file right next to the virtual hard disk file. On Mac OS, virtual machines typically reside in ~/Library/Application Support/VMware Fusion/Virtual Machines/, but you may be using a different location for your VMs. If you are using VMware Fusion 3, VMs and their associated files are organized in packages with the suffix .vmwarevm. To locate the VM configuration file, simply right click the .vmwarevm package in Finder and select Show Packet Contents, as shown in the following figure for the Boot Camp VM.

Use the context menu to see packet contents.

Use the context menu to see packet contents.

After locating the .vmx files for the WRK VM and the debugger VM, open each file with a text editor and locate the lines containing serial0. If your VM is configured to connect with printers, you will find the following two lines in your configuration file:

serial0.present = "TRUE"
serial0.fileType = "thinprint"

Please note that the contents may differ or that the serial connection may not be present at all.

Since we are using a pipe, we need to define a server, which creates the pipe, and a client, which connects to the pipe. My proposal here is to use the WRK VM as a client and the debugger VM as the server. For example, on my Mac, I am using my Boot Camp partition as debugger VM. To configure the server end of the named pipe, replace all the lines starting with serial0 in your debugger VM configuration file with the following:

serial0.present = "TRUE"
serial0.fileType = "pipe"
serial0.fileName = "/private/tmp/com1"
serial0.tryNoRxLoss = "FALSE"
serial0.pipe.endPoint = "server"

This tells VMware that the serial port (serial0) is available (present) and represented by a pipe. The named pipe must be located on the local file system in a directory, for which VMware Fusion has write permissions. My suggestion is to use /tmp/com1 or /private/tmp/com1, since /tmp is only a link. The endpoint property tells VMware that this virtual machine is supposed to create the named pipe.

To configure the client end of the named pipe, replace all the lines starting with serial0 in your WRK VM configuration file with the following:

serial0.present = "TRUE"
serial0.fileType = "pipe"
serial0.fileName = "/private/tmp/com1"
serial0.tryNoRxLoss = "FALSE"
serial0.pipe.endPoint = "client"

Again, we are using the named pipe /private/tmp/com1 but this time as a client, which means that VMware expects the named pipe to already being created. Apparently, this specification has implications on the boot order of your virtual machines: since the debugger VM is the server, it must be started prior to the WRK VM.

Now safe the configuration files and close the editor. If you check the virtual machine settings in VMware Fusion for either VM, you will now find the following options dialog for the serial port (which reads: Use user defined, unsupported settings):

Serial port configuration after manually setting the configuration file.

Serial port configuration after manually setting the configuration file.

If you boot the debugger VM first and then the WRK VM in /debug mode with /debugport=com1, you should be able to connect the kernel debugger with the WRK.

DISCLAIMER

Please note that this description is provided “as is” with absolutely no warranties. Since we are using undocumented properties, the configuration described herein may not be working in future releases of VMware Fusion. If that should be the case or you found out another way of how to connect two VMs with a named pipe in a documented fashion, please feel free to post your experiences here.

Comments

2 Responses to “How To: Debug the WRK on Mac OS X Using VMware Fusion”

  1. Bilal Khan on January 30th, 2011 04:00

    I want to thank you for your wonderful post. Even though I am still struggling to get it to work, the fact that someone out there has really succeeded in getting this to work gives me the courage to keep going. Thank you for being so generous and writing it up. It’s like a present from overseas in the new year!

    I do not want to trouble you, but even following your instructions precisely, I do not get the WinDbg to connect to the WRK which is set to run in /debug mode over /com1.

    Two things appear suspicious in my setup and I was wondering if you could confirm what they look like on yours:
    1) The debugger VM (in my case running Windows 7) shows 2 COM ports under device manager even though only 1 serial port is listed under VMware settings and the vmx file specifies only serial0…

    2) The WRK VM (in my case Windows 2003 SP1) shows only port under device manager but it is COM2 port (no COM1), and only 1 serial port under VMware settings); the vmx file specifies only serial0… Note that the kernel debugport was set to COM1.

    Now, if I change the WRK kernel debugport to COM2, then the WRK VM (upon reboot) shows only the COM1 port under device manager but no is COM2 port, and only 1 serial port under VMware settings); the vmx file specifies only serial0. It is as though the port is deliberately trying to run away :(

    Regardless, I am not able to get WinDbg to connect remotely to the WRK… the “Debugee not connected”.

    Any clues would be greatly appreciated.

    Thanks,
    Bilal

  2. Bilal Khan on January 30th, 2011 08:07

    This might be useful to others:

    http://msdn.microsoft.com/en-us/library/ff556253(v=vs.85).aspx

    “When you enable kernel debugging on a serial port, Windows removes the specified port from the system device list. As a result, on computers with an ACPI BIOS, the port does not appear in any device lists, such as the one in Device Manager. On computers that do no have an ACPI BIOS, the port appears with an error message, such as “Not enough resources to use this port.” These messages indicate that the port is under the control of the host debugging computer; they do not indicate a malfunction.”