<p>What are the relative merits of the xdoc and apt formats for writing websites of Maven projects? Is one more expressive than the other? Has one got better tool support than the other?</p>

The XDOC format is definitely a richer mechanism for creating documents and is required if you want to produce documents with TOC/TOF, headers, footers or footnotes (and other document attributes), since the APT format doesn't support these.
That being said, I tend to use the APT format for almost all internal documents as I enjoy writing in the APT format.  When compared to writing XDOC (with all its XML loveliness), APT is a breeze.  By the same token, when I'm writing a plugin that generates content, I tend to use the XDOC format, since it's pretty easy to write software that creates the required XML.