Text Generation Pipeline doesn't take Truncation = True

[dunalduck0]

🚀 Feature request

Motivation

This request is similar to #9432 but for text generation pipeline. Truncation is not accepted by text generation pipeline. GPT-J would crash if the input prompt exceeds the limit of 1024 tokens.

[LysandreJik]

Could you provide a reproducible code example as well as your software versions as asked by the templates? Thank you!

cc @Narsil

[dunalduck0]

Environment info

• transformers version: 4.11.0.dev0
• Platform: Linux-5.4.0-84-generic-x86_64-with-glibc2.17
• Python version: 3.8.10
• PyTorch version (GPU?): 1.9.0+cu111 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

This code crashed because the input is too long.

from transformers import pipeline
generator = pipeline('text-generation', model='EleutherAI/gpt-j-6B', device=0)
string = generator('a '*6000, do_sample=True, max_new_tokens=10, temperature=0.9, top_k=10, top_p=0.92, num_return_sequences=1)
print(string)

Token indices sequence length is longer than the specified maximum sequence length for this model (6001 > 1024). Running this sequence through the model will result in indexing errors
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
Traceback (most recent call last):
  File "test-deepspeed.py", line 5, in <module>
    string = generator('a '*6000, do_sample=True, max_new_tokens=10, temperature=0.9, top_k=10, top_p=0.92, num_return_sequences=1)
  File "/home/meiyang/src/transformers/src/transformers/pipelines/text_generation.py", line 150, in __call__
    return super().__call__(text_inputs, **kwargs)
  File "/home/meiyang/src/transformers/src/transformers/pipelines/base.py", line 915, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/home/meiyang/src/transformers/src/transformers/pipelines/base.py", line 922, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/home/meiyang/src/transformers/src/transformers/pipelines/base.py", line 871, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/meiyang/src/transformers/src/transformers/pipelines/text_generation.py", line 166, in _forward
    generated_sequence = self.model.generate(input_ids=input_ids, **generate_kwargs)  # BS x SL
  File "/home/meiyang/miniconda3/envs/gptj_server/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/home/meiyang/src/transformers/src/transformers/generation_utils.py", line 1016, in generate
    return self.sample(
  File "/home/meiyang/src/transformers/src/transformers/generation_utils.py", line 1529, in sample
    outputs = self(
  File "/home/meiyang/miniconda3/envs/gptj_server/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/meiyang/src/transformers/src/transformers/models/gptj/modeling_gptj.py", line 774, in forward
    transformer_outputs = self.transformer(
  File "/home/meiyang/miniconda3/envs/gptj_server/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/meiyang/src/transformers/src/transformers/models/gptj/modeling_gptj.py", line 630, in forward
    outputs = block(
  File "/home/meiyang/miniconda3/envs/gptj_server/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/meiyang/src/transformers/src/transformers/models/gptj/modeling_gptj.py", line 275, in forward
    attn_outputs = self.attn(
  File "/home/meiyang/miniconda3/envs/gptj_server/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/meiyang/src/transformers/src/transformers/models/gptj/modeling_gptj.py", line 224, in forward
    attn_output, attn_weights = self._attn(query, key, value, attention_mask, head_mask)
  File "/home/meiyang/src/transformers/src/transformers/models/gptj/modeling_gptj.py", line 146, in _attn
    attn_weights = torch.where(causal_mask, attn_weights, self.masked_bias.to(attn_weights.dtype))
RuntimeError: The size of tensor a (2048) must match the size of tensor b (6001) at non-singleton dimension 3

I wanted to pass "truncate=True" to tokenizer within the pipeline. But adding it to both pipeline() and generator() didn't work, and resulted in the same crash.

from transformers import pipeline
generator = pipeline('text-generation', model='EleutherAI/gpt-j-6B', device=0, truncate=True)
string = generator('a '*6000, do_sample=True, max_new_tokens=10, temperature=0.9, top_k=10, top_p=0.92, num_return_sequences=1, truncate=True)
print(string)

My current work around is to add "truncate=True" directly to preprocess() in the file src/transformers/src/transformers/pipelines/text_generation.py

    def preprocess(self, prompt_text, prefix=""):
        inputs = self.tokenizer(
            prefix + prompt_text, padding=False, add_special_tokens=False, return_tensors=self.framework, truncation=True
        )

        inputs["prompt_text"] = prompt_text
        return inputs

The request is to expose the parameter "truncate" to either pipeline() or generator().

[Narsil]

Hi @dunalduck0 .

tl;dr: it might be more complex than you imagine to produce what you actually want.

Truncation will truncate the right of your prompt (abc -> ab), which is not desirable in your case I think.
You probably want to truncate left (discarding earlier part of the prompt). Otherwise, you' re going to generate from the start to a random part of the prompt, which is unlikely to be the desired thing. you most likely want to drop the first part of the prompt that exceeds the capacity, and generate from there.

Having max_length kind of prompts is actually not trivial and already lead to some internal discussions (without "clean" solution right now). The generation methods use past_key_values to accelerate generation by reusing past attentions. That works very well when < model_max_length, but it actually is counter productive when reaching model_max_length because the position_embeddings (when they exist which is in most cases) will actually be wrong, since you're shifting all of them by 1 when generating subsequent characters.

• We could start dropping past_key_values at that point, but that' s increased complexity on the code (which is already complex with sampling, beam_search and grouped beam_search etc..). That complexity would likely need to hit ALL branches. It's also slightly incorrect, since we're truncating more than you intend. The performance hit is pretty significant too.
• We could instead cut the input to model_max_length - generate_length so that the generated outputs have enough room to grow so that the position_ids cached can be kept in cache. But if the generate_length is too big, then the prompt will be smaller or non existant, which is also undesired (depends on the numbers and application).
• We can continue using the cache, keep the performance, at the cost of degraded outputs when the generated length exceeds model_max_length (wether it's in the prompt or in the generated output).

Overall, the current consensus internally (afaik, feel free to correct me), is that generation that far exceeds the model capacity is non trivial, and cannot be handled automatically as it' s using models outside their intended boundaries. This means all solutions have to compromise somewhere, performance/complexity/correctness, and no solution is actually 100% correct (since you have to drop part of the prompt anyway).
Because of that, we feel that this library shouldn't make a choice on the user's behalf.

All that being said, making it easier to choose between those 3 solutions for text-generation is something we could look into (first solution being the hardest).

Am I correct in my assumptions in what you want to achieve ? Are my propositions to solve the problem making sense to you ? Which solution would be relevant in your case ?

[dunalduck0]

Thank you @Narsil for the detailed answer. You are right that abc --> ab was not a good solution in general. In my own application, it doesn't matter because all truncation is equally bad and I only need to ensure no crash that interrupts the generation pipeline.

If I read your reply correctly, I think the best trade-off is to preprocess my input before calling generator():

1. Find out the lengths of all input by calling tokenizer() myself.
2. Do something about inputs that len(input) + max_new_tokens > 1024 (GPT-J max_model_length seems to be 1024). The choices include truncating, dropping the input completely, and etc..
3. Call generator() with only inputs that satisfy len(input) + max_new_tokens <= 1024

Does it sound right?

[Narsil]

Yes that's the solution 2 I proposed.

We can definitely include that in the pipeline in some form too to make it easier, I first of all wanted to make sure I understood your problem correctly and that the proposed solutions made sense.

[Narsil]

#14118

[dunalduck0]

Thank you Narsil for confirming. As for my problem, my prompt are programming language functions. Some functions are simply too long because they contain literal string definition. Since it's not natural language, it's equally bad to truncate beginning (the function signature) or end (the immediate code before generation). Truncating the middle might work but I've not tried.

I think the ideal solution is NOT truncating at all, but split a long prompt into K chunks. Do mini fine-tuning by feeding the first K-1 chunks and then generate by feeding the last chunk. Of coz, I'm asking too much :).

[Narsil]

I don't think it fits the scope of the pipelines then.

It seems too much knowledge about the underlying model is required to do things correctly it seems.
Also with regard to truncating left, if you trained your model in that way, it should work decently too actually.

Crashing on prompts too long is also not necessarily that bad (if someone feeds you the entire linux kernel it's going to be hard t generate something with everything in context I guess.)

[jbmaxwell]

I think the ideal solution is NOT truncating at all, but split a long prompt into K chunks. Do mini fine-tuning by feeding the first K-1 chunks and then generate by feeding the last chunk. Of coz, I'm asking too much :).

I'm facing a similar problem. Could you explain your "mini fine-tuning" idea a little more? Is the idea to just run inference on the K-1 chunks, to update attentions, then just use your Kth chunk as the generation prompt? This is similar to something I was thinking, but I wasn't sure if it made sense... will something of the K-1 chunks actually be retained by the model (if there's no recursion)?