Support for Text Chunking with Overlap in TokenTextSplitter

[alfredogangemi]

Expected Behavior

The TokenTextSplitter class should support splitting text into chunks with an optional overlap between them. This would enable better context preservation when processing long texts that are chunked for downstream tasks such as embeddings or text analysis.

TokenTextSplitter splitter = TokenTextSplitter.builder().withChunkSize(800).withOverlapSize(50).build();

Current Behavior

Currently, the TokenTextSplitter class splits text into chunks without any overlap. While this works for cases where chunks are fully independent, it may result in loss of context between adjacent chunks, particularly in tasks like embeddings where continuity can improve results.

Context

Why it's needed
Chunking with overlap is critical for use cases where preserving context across chunks is necessary. For example:

• Embedding generation, where continuity between chunks improves semantic representations.
• Language models that benefit from adjacent context during token predictions.

Alternatives considered
The functionality can be implemented by extending TokenTextSplitter locally. However, a native implementation would ensure consistency and reusability across projects.

[xiunchen]

This led me trun to langchain4j

[jrgleason]

This would be nice I would hate to put all this time into doing things with Spring AI only to have to figure out how to port everything over to LangChain for java or equivalent

[jingbio]

I have custom implemented TextSplitter

[checkHup]

I am looking forward to implementing this feature.

[pabloxtiyo]

This is definitely a great feature to support. Same as all the people on this issue i may need to move to langchain.