<p>I'm looking for a way to limit the maximum number of running processes in Windows Server 2003.</p>

<p>Is there a registry key somewhere that controls it?
If so, which one is it?</p>

if you are talking about processes as in the items listed in task manager, then there is no way to do it natively, and you can do it with a program,
but there should be no real valid reason to do so.
If you are talking about making your application only ever launch one EXE no matter how many times it is called, then you are looking for singleton-ing. Example for .NET at:
http://www.thescarms.com/dotnet/SingleInstance.aspx
--EDIT For another language, google for "singleton" and your language.
If you are asking about something else, please elaborate.