<p>Has anyone here ever used C++'s "placement new"? If so, what for? It looks to me like it would only be useful on memory-mapped hardware.</p>

It is useful if you are building a kernel - where do you place the kernel code you read from disk or the pagetable? You need to know where to jump to.
Or in other, very rare circumstances such as when you have loads of allocated room and want to place a few structures behind each other. They can be packed this way without the need for the offsetof() operator. There are other tricks for that too, though.
I also believe some STL implementations make use of placement new, like std::vector. They allocate room for 2^n elements that way and don't need to always realloc.