The DevKit Loves libxml2

After an 18 month slumber, libxml2 snapped awake, downed a few Irish coffee's, and recently pushed out a 2.8.0 release.

I just read your mind.

"I wish Jon would write a short post on how to build libxml2 from source on Windows using the DevKit."

Bored stiff with your real work I see. Remember, work? The thing you do to fund your debaucheries and sundry bad habits?

But who am I to stand in the way of a bit of slacky hacking. Thankfully, there's just three trivial steps thanks to the work of the libxml2 contributors:

  1. Build or download a DevKit.
  2. Download the libxml source via FTP.
  3. Build and test.

Get an Edgy DevKit and libxml2-2.8.0

For the fiendishly edgy, clone the RubyInstaller repo and build a mingw-w64, gcc 4.7.1 DevKit with rake devkit sfx=1 dkver=mingw64-32-4.7.1. Run rake devkit:ls if you want a list of available DevKit's to build.

For the mildly edgy, download a mingw, gcc 4.6.2 DevKit from TheCodeShop downloads page.

Next, download the libxml2 2.8.0 source tarball and extract it to a location that contains no spaces in the pathname. C:\temp\libxml2-2.8.0 looks just fine.

Build and Test

Although 2.8.0 added lzma compression support, I've gone the way of the Luddite and slavishly stuck with just zlib support.

If you don't already have the zlib dev headers/libraries, you can always build libxml2 without zlib support by passing the ./configure script --without-zlib instead of --with-zlib=c:/path/to/zlib/devstuff. Or if you really want zlib support, scan this RubyInstaller ML post and see what interesting URLs you discover.

I'll swing back and update the post once I've toyed with the new lzma capability.

c:\temp\libxml2-2.8.0>\DevKit-4.7.1\devkitvars.bat
Adding the DevKit to PATH...

# formatted for post...use a single command line when building
C:\temp\libxml2-2.8.0>sh -c "./configure --prefix=c:/devlibs/libxml2-2.8.0 \
                            --with-zlib=c:/devlibs/zlib-1.2.7 \
                            --without-iconv --without-docbook"

checking build system type... i686-pc-mingw32
checking host system type... i686-pc-mingw32
...
checking for ld used by gcc... c:/devkit-4.7.1/mingw/i686-w64-mingw32/bin/ld.exe
...
Checking zlib
checking zlib.h usability... yes
checking zlib.h presence... yes
checking for zlib.h... yes
checking for gzread in -lz... yes
...
checking lzma.h presence... no
...
checking whether to enable IPv6... no
...
Found Python version 2.7
could not find python2.7/Python.h or /include/Python.h
./configure: line 14262: python2.7-config: command not found
Checking configuration requirements
Enabling multithreaded support
Disabling Docbook support
Disabling ICONV support
Disabling ICU support
Enabled Schematron support
Enabled Schemas/Relax-NG support
...
Done configuring

C:\temp\libxml2-2.8.0>make
...
make[1]: Leaving directory `/c/temp/libxml2-2.8.0'

To test the libxml2 API using the generated testapi.exe helper, first, add the following file to the C:\temp\libxml2-2.8.0 source directory.

<!-- file: gatolog.xml -->

<catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">

  <system systemId="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"
    uri="dtd/xhtml1/xhtml1-strict.dtd"/>
 
  <system systemId="http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"
    uri="dtd/xhtml1/xhtml1-transitional.dtd"/>
 
  <system systemId="http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"
    uri="dtd/xhtml11/xhtml11-flat.dtd"/>

</catalog>

Finally, grab a cup of Rishi Masala Chai tea, set an env var, and fire off the API tests.

C:\temp\libxml2-2.8.0>set XML_CATALOG_FILES=C:/temp/libxml2-2.8.0/gatolog.xml

C:\temp\libxml2-2.8.0>testapi.exe
Testing HTMLparser : 32 of 38 functions ...
Testing HTMLtree : 18 of 18 functions ...
Testing SAX2 : 38 of 38 functions ...
...
Testing nanoftp : 14 of 22 functions ...
Testing nanohttp : 13 of 17 functions ...
Testing parser : 61 of 70 functions ...
Testing parserInternals : 33 of 90 functions ...
...
Testing relaxng : 14 of 24 functions ...
...
Testing xmlIO : 39 of 48 functions ...
...
Testing xmlreader : 76 of 86 functions ...
...
Testing xmlschemas : 15 of 25 functions ...
Testing xmlschemastypes : 26 of 34 functions ...
...
Testing xmlwriter : 51 of 79 functions ...
Testing xpath : 30 of 38 functions ...
Testing xpathInternals : 106 of 117 functions ...
Testing xpointer : 17 of 21 functions ...
Total: 1161 functions, 291375 tests, 0 errors

 
Profiling MinGW Apps - Part 1

If you've tried to profile Windows apps built with a MinGW toolkit you've likely run into a couple of problems. Either your profiler doesn't understand MinGW's debug symbols, or the profiler is so intrusive that you can't build the application you're trying to profile.

A number of well-known and useful profilers such as Windows Performance Tools only understand binaries with debug information in PDB files. Your poor MinGW application built with -g, -ggdb, or -gstabs sadly can't be profiled because the profiler simply doesn't understand the integrated debugging information. Usually this means that assembly and/or raw memory addresses appear in the output rather than source code and functions names. Not the easiest way to determine what's happening in your code.

Other profilers such as gprof can be so finicky that it's next to impossible to find the right combination of compile options required for your application to build. Most frustrating, if you do finally discover the incantation needed to build, you're often confronted with runtime failures such as The procedure entry point mcount could not be located... It's maddening if it turns out that gprof just doesn't play nicely with your app.

So I was pleased to discover that the freely avalailable AQtime Standard profiling toolkit from SmartBear Software natively understands .NET, Java, PDB, and MinGW's metadata and debug information. Check out the AQtime screencasts for some great getting started information.

Let's take AQtime Standard for a quick spin to figure out why the following C code performs so poorly. Let's also pretend you've reviewed the source and still can't figure out why it takes so long for the MessageBox to appear.

/* file: slowhello.c
 *
 * build with:
 *     gcc -Wall -O2 -g2 -gstabs+ -o slowhello.exe slowhello.c -luser32
 *
 */
#include <sys/stat.h>
#include <windows.h>

#define STAT_COUNT 50000

void rb_mongo_stat(const char *filename)
{
    int i, rc;
    struct _stat buf;

    for (i = 0; i < STAT_COUNT; i++)
    {
        rc = _stat(filename, &buf);
    }
}

void hello(void)
{
    rb_mongo_stat("slowhello.exe");

    MessageBox(NULL, "Hi Speedy G!", "Pokey", MB_SETFOREGROUND);
}

int main(int argc, char **argv)
{
    hello();

    return 0;
}

Profiling with AQtime follows the pattern typical with most other profilers:

  1. Compile your application with metadata/debug information
  2. Create a "project" in the AQtime tool with profiling setup and configuration information
  3. Run the application to be profiled
  4. Analyze the results

I'll assume you've successfully installed AQtime. If you don't already have a MinGW toolchain installed, swing over to the RubyInstaller project and download the DevKit for a quick and painless install. After following the installation instructions, don't forget to add the DevKit to your PATH environment variable by running somthing similar to:

c:\cdev>\Devkit\devkitvars.bat
Adding the DevKit to PATH...

c:\cdev>

Compiling for Profiling

The only trick here is to ensure you compile with the -gstabs+ build option. For example, compile the example code like:

c:\cdev>gcc -Wall -O2 -g2 -gstabs+ -o slowhello.exe slowhello.c -luser32

c:\cdev>

Creating and Profiling an AQtime Project

The SmartBear's AQtime documentation has more detailed info on how to create a project, but a quick way to create a project is to use the File -> New Project From Module... dialog and select the slowhello.exe app you just built. As this test app is simple and doesn't depend on other custom DLLs you simply need to add the slowhello.exe module to the project.

If your app was more complex and depended upon other custom DLLs you would need to add those DLLs to the project. Although slowhello.exe depends upon msvcrt.dll, kernel32.dll, and user32.dll you do not need (or want) to add these Windows system DLLs to your project.

For a project this simple, no additional configuration is required. Simply press the F5 key, click on the green run icon, or select the Run -> Run menu item to start profiling the slowhello.exe app. After accepting the defaults for a couple of AQtime dialogs, slowhello.exe's dialog box will appear. Click OK and AQtime will complete it's profiling and show it's results in both a Summary tab view and a Report tab view.

With my AQtime configuration, the results from the Report tab and other views looks like the following.

AQtime slowhello.exe profiling results

While it's almost impossible to read the results, the highlighted line near the middle of the graphic shows that the rb_mongo_stat routine took up the most time with 2.27 seconds followed by the hello routine taking 0.45 seconds. Selecting each routine name changes the information displayed in the source code view on the right and the different views (Disassembler, Parent/Child Details, Call Graph, and Call Tree) at the bottom.

AQtime quickly showed you that the problem child causing MessageBox to be delayed is the rb_mongo_stat function. Time to refactor that implementation to stop unnecessarily stat-ing. A shockingly elusive result I know.

Conclusion

Although this has been a whirlwind summary of how to build and profile a very simple C program with SmartBear's AQtime Standard product, I hope you see how straight forward it can be to begin profiling your MinGW built Windows applications.

Future Post Spoiler

Now that you're a profiling expert with AQtime Standard, it's time to pick up the pace and dig into something a bit more challenging.

# file: override/aqtime_build.rb
# custom build config overrides
if ENV['AQTIME'] then
  puts '[INFO] Overriding to enable AQtime profiled Ruby 1.9.x...'

  RubyInstaller::Ruby19.dependencies = [ :ffi, :yaml, :zlib ]

  RubyInstaller::Ruby19.configure_options << "optflags='-O2'"
  RubyInstaller::Ruby19.configure_options << "debugflags='-g2 -gstabs+'"
end
Categories
Lua (1)
Python (2)
Go (1)
Ruby (7)