<p>I have an existing standalone application which is going to be extended by a 3rd-party, using a network protocol. The capabilities are already implemented, all I need is to expose them to the outside.</p>

<p>Assuming the transport protocol is already chosen (UDP), are there any resources that will help me to design my application protocol?</p>

<p>There seems to be a lot of information about software design, but not on protocol design.
I've already looked at <a href="http://www.faqs.org/docs/artu/ch05s03.html" rel="noreferrer">Application Protocol Design</a>.</p>

First off, UDP is primarily a one-way broadcast transport method. Also, it is
potentially lossy
, so you need to be able handle missing packets and out-of-order packets. If you need any level of reliability from UDP, or require two-way connections, you will end up needing just about everything from TCP, so you might as well go with that to start with and let the network stack take care of it.
Next up, if your data is potentially larger than a single IP packet then you will need some way of identifying the start and end of each packet, and a means of handling illegal or corrupt packets. I would recommend some kind of header with packet length, some kind of footer, and maybe a checksum.
Then you need some way of encoding the messages and responses. There are many RPC protocols around. You could look at SOAP, or design a custom XML-based protocol, or a binary one.