The calculation method of token count needs to be abstracted.

[xbotter]

The calculation method of the number of tokens in LLM is always related to LLM, as well as the maximum number of tokens.
However, in SearchClient, it is specified as the applicable token calculation method for OpenAI.

kernel-memory/dotnet/CoreLib/Search/SearchClient.cs

Lines 183 to 186 in a26407b

	var tokensAvailable = 8000
	- GPT3Tokenizer.Encode(this._answerPrompt).Count
	- GPT3Tokenizer.Encode(question).Count
	- AnswerTokens;

My idea is to extract int MaxTokens {get;} and int TokenCount(string text) together into ITextGeneration, so that providers of LLM can implement this part on their own.

[dluc]

I agree about the abstraction, I'm still hoping to delegate that work to SK. The alternative would be to package each model with its own tokenizer, that's a lot of work. My current thinking is that the tokenizer doesn't need to be precise, as long as it affects only chunking and chunking doesn't need to be exact. There will be a need for precise tokenization when working on costs, but I think for now we could wait. Thoughts?
I would be careful touching the existing interfaces, it might be easier thinking about the tokenizer as yet another dependency that can be injected.

By the way, do you have a scenario that requires exact token count?

[xbotter]

Based on the implementation of SearchClient, I understand that the calculation of tokens is to use as many facts as possible. There is no problem with this. I am currently adapting KernelMemory in LlamaSharp. Since the context of Llama Model is usually small(4096), the default MaxTokens of 8000 can easily cause unexpected issues with the model. Therefore, I hope to be able to configure this parameter myself.
It would be better to abstract token count separately. Currently, one common issue encountered is during the chunking stage of embedding, especially for different language embedding models where there can be significant differences in token calculation. Additionally, some smaller embedding models have a smaller upper limit for tokens, so they are particularly sensitive to token calculation, such as bge-large where the maximum context is limited to 512.

[dlucr]

Thinking about token count, I think every model should have a dedicated tokenizer. For instance, using a GPT tokenizer for Mistral doesn't seem logical, nor does employing an LLAMA2 tokenizer for GPT-4 make sense in most cases.

In other words, perhaps it makes sense to extend the ITextGenerator and IEmbeddingGenerator interfaces to have a Tokenize method, so that when providing one or the other, the consumer object has a consistent way of measuring tokens.

Also, there are at least two different tokenization scenarios:

1. when calculating embeddings, each model accepts a max limit of tokens input. Some models throw an exception when giving too many tokens (e.g. OpenAI Ada2) and other models just ignore the extra tokens, losing semantic.
2. when calculating answers with TextGeneration, the amount of grounding information that can fit depends on the model used. This number is different from the number used by (1) and tokens are also calculated differently when mixing models from different providers, e.g. OpenAI embeddings with LLama text generation.

Also, (1) affects chunking: chunks should not be bigger than the max amount of tokens accepted by the embedding generator, to avoid losing semantics (and runtime exceptions). Currently, the default chunking/partitioning settings are fine tuned for OpenAI Ada2, and the application has no way to validate the values (e.g. _maxTokensPerParagraph=1000) for the chosen models.

[dluc]

@xbotter PR #187 should help, hope you have a chance to take a look - thanks!

[dlucr]

code moved to #189

[dluc]

#189 merged, tokenization is now abstracted and configurable.