<p>We are currently testing a Java Swing application for it's performance. I wonder if there is a good tool to automate this?</p>

A few years ago I used
JMeter
for such a task. I generally enjoyed using it, though I never did much research on what else is available and I don't know if it's still actively developed.