The custom stylesheet is based on the standard DocBook XSL XHTML stylesheet. However, instead of inserting an HTML header, it inserts PHP calls to include the WordPress headers and footers. The only tricky part of this is to tell WordPress what the title of the resulting page should be. I hacked this in by creating a class with the bare minimum fields required by wp_title() (which is used by WordPress to print the title of the document). The resulting template looks like this:

<?php
  require('../wordpress/wp-blog-header.php');
  class MyPost { var $post_title = "My Title"; }
  $wp_query->is_home = false;
  $wp_query->is_single = true;
  $wp_query->queried_object = new MyPost();
  get_header();
?>
<div id="content" class="narrowcolumn" role="main">
  ...
</div>
<?php
  get_sidebar();
  get_footer();
?>

The DocBook kit is available from my Git repository (or on GitHub, or as a ZIP file). More information about the kit can be found here.