<p>I have a neural network written in Erlang, and I just bought a GeForce GTX 260 card with a 240 core GPU on it. Is it trivial to use <a href="http://en.wikipedia.org/wiki/CUDA" rel="noreferrer">CUDA</a> as glue to run this on the graphics card?</p>

No, using CUDA is not a trivial matter.
The CUDA programming model basically uses C (with some additions) but in order to get the most of the GPGPU's capabilities you would have to ensure that your algorithms follow the CUDA guidelines. (see
NVidia CUDA Programming Guide
)
For example in order to get the best memory performance (somewhere around 70Gbps) you need to access memory in streaming mode with coalescing, also branches are very costly on the GPUs so you should avoid conditionals as much as possible. Check out the guide and samples provided with the SDK, they'll provide an excellent starting point