<p>If you call javascript window.open and pass a url to a .xls file it open on some machines in the browser window.  How can you force it into Excel?</p>

Only the users machine can "force" it into Excel.  That said, 99% of the time if you send the correct mime-type and a user has Excel, then it will open in Excel assuming they approve.
And only the server can send the correct mime-type.  The document type you pass to a JavaScript window.open call will have no effect on this.  In fact, calling window.open will at best just open a superfluous window.
It's best to just link to the document with
```
<a href="foo.xls">
```
.  And provided your server is sending a mime-type of
```
application/x-excel
```
or
```
application/x-msexcel
```
this will almost always nudge the browser into opening a new window with the Excel document.