<p>Can we somehow extend the RuleSetDialog class and host in our windows application?</p>

You can not extend the RuleSetDialog class (since it is sealed), but you can host it in your app:
RuleSetDialog Class
Hosting the workflow rules designer
Rules Engine via Windows Workflow Foundation
Filtering Intellisense lists in the WF RuleSetDialog
Windows Workflow Foundation Rules for Generic Types
Tweaking RuleSetDialog to Resize
PowerShell Rules Engine: How-To Technical Details