<p>Is there ever a good reason to <em>not</em> declare a virtual destructor for a class?  When should you specifically avoid writing one?</p>

There is no need to use a virtual destructor when any of the below is true:
No intention to derive classes from it
No instantiation on the heap
No intention to store with access via a pointer to a superclass
No specific reason to avoid it unless you are really so pressed for memory.