<p>Does anyone know tools like <a href="http://www.nektra.com/products/spystudio/" rel="noreferrer">SpyStudio</a> to hook processes?
The features I would like to have are:</p>

<ul>
<li>log function calls on other processes</li>
<li>GUI</li>
<li>scriptable</li>
<li>opensource</li>
</ul>

<p>I already looked at some tools like:</p>

<ul>
<li><a href="http://www.nektra.com/products/spystudio/" rel="noreferrer">SpyStudio</a></li>
<li><a href="http://research.microsoft.com/sn/detours/" rel="noreferrer">Detours</a></li>
<li><a href="http://www.dyninst.org/" rel="noreferrer">DynInst</a></li>
</ul>

<p>but none really fully satisfies me.</p>

It likely uses the debugging API and places debug hooks at key points in the application to to trap events. You may even be able to do what you want using
ollydbg
.
EDIT: I just looked at the site and saw the demonstration videos and I would say that they almost certainly do this through the debugging API. They likely place breakpoints in the target at the desired function entry points. When a trap event fires, it looks like they use an RPC call to notify the user of their API and allow them to look at what happened and potentially modify some things. Sending back what to do next also in an RPC call. A nice design, but just a glorified debugger.