<p>I am using the WebBrowser control to add a WebInterface to C# app. My desire is to verify that only such app is able to connect to our Web server using SSL client certificates.</p>

<p>My idea was to embed the client certificate in the app and just use when connecting via my app. Anybody have a sugestion on how to do this? Or the only way to make it work is to load the key in the X509Store.</p>

<p>If I put it in X509Store, will it make my key available for general Internet Explorer Usage?</p>

Are you sure this is what you want to do? If you embed the private key in your application (as your approach entails), an attacker can extract it and use it to authenticate their rogue software.
A server cannot authenticate client software. It can only test whether a client possesses some secret. When you embed a private key in your client and distribute it, it will not be a secret anymore.
I'd recommend authenticating
users
of your software, rather than the software itself. You need to let users generate their own secret, whether it is a password or a private key, and give them an incentive to protect it.