<p>Can standard pointers in .Net do this? Or does one need to resort to P/invoke?</p>

<p>Note that I'm not talking about object references; I'm talking about actual C# pointers in unsafe code.</p>

C#, as a managed and protected run time engine, does not allow low level hardware access and the memory locations associated with actual hardware are not available.
You'll need to use a port driver or write your own in C++ or C with the proper Windows API to access the memory mapped I/O regions of interest.  This will run in a lower ring than the C# programs are capable of.
This is why you don't see drivers written in C#, although I understand many are writing access routines with C++, but the main driver logic in C#.  It's tricky, though, as crashes and restarting can become tricky, not to mention synchronization and timing issues (which are somewhat more concrete in C++ at a lower ring, even though windows is far from a real-time system).
-Adam