<p>I like how SQLite library can be included into an application and the application has a full-fledged database to use.</p>

<p>Similarly, is there an open source versioning library that I can include into my application so that I can save versions of files as well as do diffs and merges?</p>

In my opinion
Firebird
is one of the best choices for embedded DB scenarios.
Also
Microsoft SQL Server Compact
(closed source, but free) might be suitable, however it less capable than Firebird.
EDIT:
I misread you question. If you don't need RDBMS, you can try to embed
SVN
to your application.