<p>MTOM is the W3C Message Transmission Optimization Mechanism, a method of efficiently sending binary data to and from web services.</p>

<p>How does it work in general?</p>

If you put
Wireshark
(or enabled
System.Net Logging
) on the non-MTOM enabled service, you should see the SOAP requests with the binary data encoded as BASE64. Sending it as BASE64 increases the size of the binary data but (I assume) makes it more interoperable.
With MTOM, the SOAP messages are sent as MIME messages with the BASE64 encoding being replaced with a placeholder. The binary data is then placed between delimiters (which happens for each piece of binary data), and then placed at the end of the SOAP request. The binary data is then sent unencoded. IIRC, MTOM also determines whether sending it as a MIME message will increase the size of the SOAP call and if doesn't provide a saving, it will send it as a normal SOAP message.
This
provides an example of what the message sent over the wire looks like.