<p>For an ecommerce website how do you measure if a change to your site actually improved usability?  What kind of measurements should you gather and how would you set up a framework for making this testing part of development?</p>

Multivariate testing
and reporting is a great way to actually measure these kind of things.
It allows you to test what combination of page elements has the greatest conversion rate, providing continual improvement on your site design and usability.
Google Web Optimiser
has support for this.