<p>What files can we modify so that our solution is still supported by Microsoft?</p>

<p>Is it allowed to customize error pages? 
Can we modify the web.config files to use custom HTTPHandlers?</p>

You can certainly edit the web.config file for your sites.  The one thing that you should be aware of, however, is that when you start editing files manually on the file system, you will have to remember to manually make those changes across all servers in the farm (assuming a farm exists).  In addition to this, when you edit files in the 12 hive, it's important to understand that you will be making a change to all SharePoint sites hosted on the server(s) for which the files were edited.
Personally, if I were going to create a custom error page, I would simply add a <customErrors> section to my web.config.  I avoid
editing
any
existing
files in the 12 hive, but I have
added
files (though it's rare).