<p>We are currently thinking of building a cache-system to hold data pulled out of an SQL database and make it available to a couple of other applications (website, webservice, etc). We imagine the cache to be running as a windows service and basically consist of a smart dictionary which holds the cache entries. My question is, is there a limit to the working set of the application (it will be running under windows server 2003)? Or is the amount of physical memory the limit?</p>

As with any other Windows program, you're limited by address space. That is: on 32-bit, you can have 2GB of address space. On x64, you can have 8TB.
If you don't have 8TB of physical memory, it will start to page.