Add anti-quote-memorization checks
#23
by
jbakerx - opened
To reduce risk of reproducing long verbatim passages:
run n-gram overlap checks vs your training corpus
filter or down-rank generations with high overlap
add a decoding constraint (simple post-filter) in demos
We will consider this enhancement for inclusion in version 2.0.0.