[ML] Add recursive chunking strategy with Markdown splitters

[dan-rubinstein]

TODO: Update this once there's been some time to research this.

The goal is to be able to chunk markdown documents and the best way seems to be through recursive chunking (splitting on a list of characters until chunks reach a desired size). We might as well create a generic recursive chunker that we can extend to use markdown chunking. We can look at LangChain's recursive splitter https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/. They also offer some interesting extra features with markdown chunking that include metadata in the chunked results. We might be able to utilize this metadata in the future so let's look into how we can expand this in the future.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[dan-rubinstein]

Questions about recursive chunking:

1. What's the desired behavior if a chunk of a desired size can not be produced?

Questions for markdown chunking recursively:

1. How should we differentiate between markdown/default/custom splitters?
   1. Separate chunker/chunking strategy (may be tedious to test)? Have separator_type and separators?
   2. Have separators and separator_set?
   3. Will there be custom logic for any of the separator sets (ex. does markdown do anything on top of recursive chunking?
2. How do we handle non-fixed length separators (ex. ===== or -----)? - Maybe separators should just be treated as regex instead of regular strings?1. Do we want to retain things like header information separately? Do we have a use for this?
3. Should we wait on markdown or recursive chunking until after we've collected more information about the use case?

[dan-rubinstein]

Plan:

• Created a hierarchical chunker that maximally fills each chunk based on a hierarchy of splitters.
  • For default cases: "\n\n", "\n", "." (maybe use a keyword for sentence to use sentence chunker instead of just the character)
  • What do we do if we hit the end of the separators list? Let's fall back on sentence which will fall back on word.
  • Support regex instead of just keywords.
• For markdown:
  • Discuss with the team how to differentiate between markdown/default/custom splitters.