<p>When creating a new ASP.NET project in Visual Studio should I chose create: website or project? </p>

<p>I understand that web application project was the way to do it back in the day with VS 2003 but is it still applicable today? What are some of the caveats using one over the other?</p>

There's a
pretty good comparison chart on MSDN
.
Website projects are simple, in that all files added to the project folders are automatically compiled and included, which was supposedly added to make it more palatable to classic ASP and PHP developers. Once benefit is that it includes build providers, which allow for certain actions to be associated with a filetype - that's how the first release of SubSonic would rebuild the data access layer when you added a .abp file to the site.
Web Application Projects are a lot more flexible, though. For instance, all class libraries in a Website Project need to be in the App_Code folder, which is frustrating in a complex application. There are a lot of scenarios which just don't work for a Website Project.
You can convert from one to another, although if you're unsure I'd recommend just starting with a Web Application.