What are your favorite text chunkers/splitters? 
text
What are your favorite text chunkers/splitters? 
new_line
Mine are: 
text
Mine are: 
new_line
- 
text
- 
https://github.com/benbrandt/text-splitter
https://github.com/benbrandt/text-splitter
link
 (Rust/Python, battle-tested, Wasm version coming soon) 
text
 (Rust/Python, battle-tested, Wasm version coming soon) 
new_line
- 
text
- 
https://github.com/umarbutler/semchunk
https://github.com/umarbutler/semchunk
link
 (Python, really performant but some issues with huge docs)
text
 (Python, really performant but some issues with huge docs)
new_line
new_line
I tried the huge Jina AI regex, but it failed for my (admittedly messy) documents, e.g. from EUR-LEX. Their free segmenter API is really cool but unfortunately times out on my huge docs (~100 pages): 
text
I tried the huge Jina AI regex, but it failed for my (admittedly messy) documents, e.g. from EUR-LEX. Their free segmenter API is really cool but unfortunately times out on my huge docs (~100 pages): 
https://jina.ai/segmenter/
https://jina.ai/segmenter/
link
new_line
new_line
Also, I tried to write a Vanilla JS chunker with a simple, adjustable hierarchical logic (inspired from the above). I think it does a decent job for the few lines of code: 
text
Also, I tried to write a Vanilla JS chunker with a simple, adjustable hierarchical logic (inspired from the above). I think it does a decent job for the few lines of code: 
https://do-me.github.io/js-text-chunker/
https://do-me.github.io/js-text-chunker/
link
new_line
new_line
Happy to hear your thoughts! 
text
Happy to hear your thoughts! 
What are your favorite text chunkers/splitters? 
Mine are: 
- https://github.com/benbrandt/text-splitter (Rust/Python, battle-tested, Wasm version coming soon) 
- https://github.com/umarbutler/semchunk (Python, really performant but some issues with huge docs)

I tried the huge Jina AI regex, but it failed for my (admittedly messy) documents, e.g. from EUR-LEX. Their free segmenter API is really cool but unfortunately times out on my huge docs (~100 pages): https://jina.ai/segmenter/

Also, I tried to write a Vanilla JS chunker with a simple, adjustable hierarchical logic (inspired from the above). I think it does a decent job for the few lines of code: https://do-me.github.io/js-text-chunker/

Happy to hear your thoughts! 
👀
cschroeder John6666
❤️
Felladrin
625026749d39e8be3166132f
/avatars/f32291df2054c1bb4a01889d1b41c0d5.svg
Christopher Schröder
cschroeder
user