Update Model Card for Encoder Decoder Model #39272
Open
+96
−78
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stevhliu
reviewed
Jul 9, 2025
| @@ -14,8 +14,6 @@ rendered properly in your Markdown viewer. | |||
| --> | |||
|
|
|||
| # Encoder Decoder Models | |||
|
|
|||
| <div class="flex flex-wrap space-x-1"> | |||
There was a problem hiding this comment.
Wrap the badges with the below to align them to the right
<div style="float: right;">
...
</div>| The effectiveness of initializing sequence-to-sequence models with pretrained checkpoints for sequence generation tasks | ||
| was shown in [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://huggingface.co/papers/1907.12461) by | ||
| Sascha Rothe, Shashi Narayan, Aliaksei Severyn. | ||
| [`EncoderDecoderModel`](https://huggingface.co/papers/1706.03762) has two main parts: |
There was a problem hiding this comment.
Suggested change
| [`EncoderDecoderModel`](https://huggingface.co/papers/1706.03762) has two main parts: | |
| [`EncoderDecoderModel`](https://huggingface.co/papers/1706.03762) initializes a sequence-to-sequence model with any pretrained autoencoder and pretrained autoregressive model. It is effective for sequence generation tasks as demonstrated in [Text Summarization with Pretrained Encoders](https://huggingface.co/papers/1908.08345) which uses [`BertModel`] as the encoder and decoder. |
Comment on lines
+29
to
+30
| Encoder: This part reads the input text and converts it into a set of numerical features capturing the meaning and context | ||
| of the input. |
There was a problem hiding this comment.
Suggested change
| Encoder: This part reads the input text and converts it into a set of numerical features capturing the meaning and context | |
| of the input. |
|
|
||
| An application of this architecture could be to leverage two pretrained [`BertModel`] as the encoder | ||
| and decoder for a summarization model as was shown in: [Text Summarization with Pretrained Encoders](https://huggingface.co/papers/1908.08345) by Yang Liu and Mirella Lapata. | ||
| Decoder: This part takes the numerical features from the encoder and generates the output text step by step. It uses the information from the encoder to produce meaningful and relevant output, such as a translation, a summary, or an answer. |
There was a problem hiding this comment.
Suggested change
| Decoder: This part takes the numerical features from the encoder and generates the output text step by step. It uses the information from the encoder to produce meaningful and relevant output, such as a translation, a summary, or an answer. |
Comment on lines
+34
to
+35
| We can be use this model class to initialize a sequence-to-sequence model with any pretrained autoencoding model as the | ||
| encoder and any pretrained autoregressive model as the decoder. |
There was a problem hiding this comment.
Suggested change
| We can be use this model class to initialize a sequence-to-sequence model with any pretrained autoencoding model as the | |
| encoder and any pretrained autoregressive model as the decoder. |
Comment on lines
+132
to
+133
| - [`EncoderDecoderModel`] can be randomly initialized from an encoder and a decoder config. In the following example, we show | ||
| how to do this using the default [`BertModel`] configuration for the encoder and the default [`BertForCausalLM`] configuration for the decoder. |
There was a problem hiding this comment.
Suggested change
| - [`EncoderDecoderModel`] can be randomly initialized from an encoder and a decoder config. In the following example, we show | |
| how to do this using the default [`BertModel`] configuration for the encoder and the default [`BertForCausalLM`] configuration for the decoder. | |
| - [`EncoderDecoderModel`] can be randomly initialized from an encoder and a decoder config as shown below. |
| ``` | ||
|
|
||
| ## Training | ||
| ## Notes |
There was a problem hiding this comment.
Add a note here about initializing from pretrained encoder/decoder
| >>> config = EncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder) | ||
| >>> model = EncoderDecoderModel(config=config) | ||
| ```python | ||
| import torch |
There was a problem hiding this comment.
Would be better to adapt the Pipeline, AutoModel and transformers-cli examples for summarization since thats what the checkpoint was fine-tuned for
|
|
||
| - The Encoder Decoder Model can also be used for translation of different languages. The example below demonstrates a |
There was a problem hiding this comment.
I think Helsinki-NLP/opus-mt-en-de is already an encoder-decoder model versus combining a separate encoder and decoder model together. You'll either need to find an existing finetuned EncoderDecoderModel checkpoint for translation or initialize a pretrained encoder/decoder
Comment on lines
+31
to
+32
| > This model was contributed by [Molbap](https://huggingface.co/Molbap), with tremendous help from | ||
| > [Anton Vlasjuk](https://github.com/vasqu). |
There was a problem hiding this comment.
Suggested change
| > This model was contributed by [Molbap](https://huggingface.co/Molbap), with tremendous help from | |
| > [Anton Vlasjuk](https://github.com/vasqu). | |
| > This model was contributed by [Molbap](https://huggingface.co/Molbap) and [Anton Vlasjuk](https://huggingface.co/AntonV). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
As described in the issue, this PR updates the model card for the encoder decoder model with an additional translation example. I have also re-added the contributor names for the Mamba and Mamba-2 models which were previously removed by me. Please let me know if any modifications are required and I will make the necessary changes.
Fixes #8944
Refs #36979
Before submitting
Who can review?
@stevhliu