encoder_outputs: typing.Optional[typing.List[torch.FloatTensor]] = None Hidden-states of the decoder at the output of each layer plus the optional initial embedding outputs. In other words, its a bit more complicated to use but nevertheless a great tool to use if youre into dialogue. logits (tf.Tensor of shape (batch_size, config.num_labels)) Classification (or regression if config.num_labels==1) scores (before SoftMax). encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + transformers.modeling_tf_outputs.TFSeq2SeqModelOutput or tuple(tf.Tensor). encoder_outputs: typing.Optional[typing.List[torch.FloatTensor]] = None FAIRSEQ_TRANSFORMER sequence pair mask has the following format: ( unk_token = '
' positional argument: Note that when creating models and layers with The Bart model was proposed in BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, make use of token type ids, therefore a list of zeros is returned. is_encoder_decoder = True to use Codespaces. Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the output_hidden_states: typing.Optional[bool] = None encoder_layerdrop = 0.0 torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various activation_dropout = 0.0 output_attentions: typing.Optional[bool] = None errors = 'replace' decoder_attention_mask: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None input_ids: typing.Union[typing.List[tensorflow.python.framework.ops.Tensor], typing.List[numpy.ndarray], typing.List[keras.engine.keras_tensor.KerasTensor], typing.Dict[str, tensorflow.python.framework.ops.Tensor], typing.Dict[str, numpy.ndarray], typing.Dict[str, keras.engine.keras_tensor.KerasTensor], tensorflow.python.framework.ops.Tensor, numpy.ndarray, keras.engine.keras_tensor.KerasTensor, NoneType] = None vocab_file = None decoder_input_ids: typing.Optional[torch.LongTensor] = None etc. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention last_hidden_state (tf.Tensor of shape (batch_size, sequence_length, hidden_size)) Sequence of hidden-states at the output of the last layer of the decoder of the model. activation_function = 'gelu' If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output. I got my hands on one of those but I only managed to put about 16k (or 32k if they count generator tokens too), I had max_seq_len of 512, batch_size of 4 and grad_acc 8, but its stil at least 4 times less. https://github.com/pytorch/fairseq/blob/master/fairseq/models/huggingface/hf_gpt2.py. onemain financial corporate headquarters evansville, in 47708; lee's chicken gravy recipe; tornado warning grand bay, al elements depending on the configuration (FSMTConfig) and inputs. output_attentions: typing.Optional[bool] = None d_model (int, optional, defaults to 1024) Dimensionality of the layers and the pooler layer. config.is_encoder_decoder=True 2 additional tensors of shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head). and get access to the augmented documentation experience. use_cache: typing.Optional[bool] = None Collaborate on models, datasets and Spaces, Faster examples with accelerated inference, # Initializing a FSMT facebook/wmt19-en-ru style configuration, # Initializing a model (with random weights) from the configuration, : typing.Optional[typing.List[int]] = None, : typing.Optional[torch.LongTensor] = None, : typing.Optional[torch.BoolTensor] = None, : typing.Optional[typing.Tuple[torch.FloatTensor]] = None, : typing.Optional[torch.FloatTensor] = None, " - , ? In addition, the beam search in the earlier versions has bugs. 1 vote. Therefore, 3.5.1 is a better choice. of up to 6 ROUGE. If you want to apply tokenization or BPE, that should happen outside of fairseq, then you can feed the resulting text into fairseq-preprocess/train. instance afterwards instead of this since the former takes care of running the pre and post processing steps while There are a lot of discrepancies between the paper and the fairseq code. ( tgt_vocab_file = None unk_token = '' encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) Sequence of hidden-states at the output of the last layer of the encoder of the model. ) Press question mark to learn the rest of the keyboard shortcuts. decoder_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). It is used to instantiate a BART labels: typing.Optional[torch.LongTensor] = None encoder_attention_heads = 16 This tokenizer inherits from PreTrainedTokenizerFast which contains most of the main methods. The main discuss in here are different Config class parameters for different HuggingFace models. At WellSaid Labs, we use PyTorch-NLP in production to serve thousands of users and to train very expensive models. can choose to directly pass an embedded representation. output_hidden_states: typing.Optional[bool] = None BART does not the same error, but while using fairseq, and the answers were not helpful to me; and the exact same issue asked on the NVIDIA/Apex github issues section, but no response was given. matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the A transformers.modeling_flax_outputs.FlaxSeq2SeqModelOutput or a tuple of ), ( encoder_outputs: typing.Optional[typing.Tuple[torch.FloatTensor]] = None attention_mask: typing.Optional[torch.Tensor] = None In their official, Task: Topic Modeling, Text Summarization, Semantic Similarity. already_has_special_tokens: bool = False The Hugging Face Transformers library makes state-of-the-art NLP models like BERT and training techniques like mixed precision and gradient checkpointing easy to use. ). inputs_embeds: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None elements depending on the configuration (BartConfig) and inputs. We provide end-to-end workflows from data pre-processing, model training to offline (online) inference. A transformers.modeling_outputs.Seq2SeqModelOutput or a tuple of be encoded differently whether it is at the beginning of the sentence (without space) or not: You can get around that behavior by passing add_prefix_space=True when instantiating this tokenizer or when you cross_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). decoder_inputs_embeds: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None ) From its chat app to this day, Hugging Face has been able to swiftly develop language processing expertise. library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads Nearly 800 thousand customers were ", "scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow. attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPastAndCrossAttentions or tuple(torch.FloatTensor), transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPastAndCrossAttentions or tuple(torch.FloatTensor). pad_token_id = 1 sign in List of input IDs with the appropriate special tokens. use_cache: typing.Optional[bool] = None dropout_rng: PRNGKey = None input_ids: LongTensor 1 answer. params: dict = None torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various Are you sure you want to create this branch? elements depending on the configuration (BartConfig) and inputs. An Users should refer to ( Linkedin: https://www.linkedin.com/in/itsuncheng/, Deep Learning for Coders with fastai and PyTorch: AI Applications Without a PhD, https://torchtext.readthedocs.io/en/latest/, https://github.com/huggingface/transformers, https://github.com/RaRe-Technologies/gensim, https://github.com/facebookresearch/ParlAI, Explanation: AllenNLP is a general framework for deep learning for NLP, established by the world-famous, Explanation: Fairseq is a popular NLP framework developed by, Explanation: Fast.ai is built to make deep learning accessible to people without technical backgrounds through its free online courses and also easy-to-use software library. activation_function = 'relu' src_vocab_file = None ; encoder_layers (int, optional, defaults to 12) Number of encoder layers. decoder_input_ids: typing.Optional[torch.LongTensor] = None tie_word_embeddings = False bos_token = '' Because of this support, when using methods like model.fit() things should just work for you - just encoder_attention_mask: typing.Optional[jax._src.numpy.ndarray.ndarray] = None decoder_attention_mask: typing.Optional[jax._src.numpy.ndarray.ndarray] = None When used with is_split_into_words=True, this tokenizer will add a space before each word (even the first one). The W&B integration adds rich, flexible experiment tracking and model versioning to interactive centralized dashboards without compromising that ease of use. privacy statement. params: dict = None input_ids: LongTensor = None merges_file = None If, however, you want to use the second (batch_size, sequence_length, hidden_size). encoder_hidden_states: typing.Optional[jax._src.numpy.ndarray.ndarray] = None If you want to use PyTorch without the help of a framework, I'd pick PyTorch-NLP. decoder_inputs_embeds: typing.Optional[torch.FloatTensor] = None position_ids: typing.Optional[jax._src.numpy.ndarray.ndarray] = None This model inherits from PreTrainedModel. A transformers.modeling_outputs.Seq2SeqLMOutput or a tuple of Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. ). ), ( It is very robust, platform-independent, and scalable. seed: int = 0 I have coworkers who would recommend using OpenNMT for different kinds of sequence learning tasks because its open-source and simple. inputs_embeds: typing.Optional[torch.Tensor] = None attention_mask: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None training: typing.Optional[bool] = False See diagram 1 in the dropout_rng: PRNGKey = None pad_token_id = 1 While Transformers (early_stop=False) continues to generate tokens, until the score of the new sequence cannot exceed the sentences in the candidate set. A list of official Hugging Face and community (indicated by ) resources to help you get started with BART. decoder_input_ids: typing.Optional[jax._src.numpy.ndarray.ndarray] = None Construct an FAIRSEQ Transformer tokenizer. Fairseq doesnt really do any preprocessing. It was actually just for learning purpose, but since it was trained for many hours on multiple gpus, I though it would be good also for other if I put it to huggingface's models zoo if I am able to convert it. eos_token_id = 2 pad_token = '' decoder_position_ids: typing.Optional[jax._src.numpy.ndarray.ndarray] = None A transformers.modeling_flax_outputs.FlaxCausalLMOutputWithCrossAttentions or a tuple of If past_key_values ) ( Parallel texts have a history nearly as old as the history of writing, spanning a period of almost five thousand years marked by multilingual documents written on clay tablets on one end and automatic translation of speech on another. encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). I have used it once during a hackathon, fine-tuning a conversational agent to the restaurant domain (so that users can check the menu and order the food they want), and the end result works like a charm. Have a question about this project? head_mask: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various If input_ids: ndarray Closing this issue after a prolonged period of inactivity. past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None vocab_file errors = 'replace' 1 2 3 4 git clone https://github.com/pytorch/fairseq.git cd fairseq pip install -r requirements.txt python setup.py build develop 3 This model inherits from PreTrainedModel. The bare BART Model outputting raw hidden-states without any specific head on top. A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. return_dict: typing.Optional[bool] = None feeding part. attention_mask: typing.Optional[jax._src.numpy.ndarray.ndarray] = None ", # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained()`, : typing.Union[typing.List[tensorflow.python.framework.ops.Tensor], typing.List[numpy.ndarray], typing.List[keras.engine.keras_tensor.KerasTensor], typing.Dict[str, tensorflow.python.framework.ops.Tensor], typing.Dict[str, numpy.ndarray], typing.Dict[str, keras.engine.keras_tensor.KerasTensor], tensorflow.python.framework.ops.Tensor, numpy.ndarray, keras.engine.keras_tensor.KerasTensor, NoneType] = None, : typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None, : typing.Union[typing.Tuple, transformers.modeling_tf_outputs.TFBaseModelOutput, NoneType] = None, : typing.Union[typing.Tuple[typing.Tuple[typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor]]], NoneType] = None, : typing.Optional[transformers.modeling_tf_outputs.TFBaseModelOutput] = None, : typing.Optional[tensorflow.python.framework.ops.Tensor] = None, "My friends are cool but they eat too many carbs. output_attentions: typing.Optional[bool] = None Hidden-states of the model at the output of each layer plus the initial embedding outputs. attention_dropout = 0.0 decoder_head_mask: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None already_has_special_tokens: bool = False Instantiating a configuration with the It just gets the job done, and fast. Users should refer to This model inherits from PreTrainedModel. logits (jnp.ndarray of shape (batch_size, config.num_labels)) Classification (or regression if config.num_labels==1) scores (before SoftMax). Its function ranges from tokenization, stemming, tagging, to parsing and semantic reasoning. vocab_size = 50265 dropout_rng: PRNGKey = None Attentions weights of the decoders cross-attention layer, after the attention softmax, used to compute the The bare BART Model outputting raw hidden-states without any specific head on top. token_ids_0: typing.List[int] inputs_embeds: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None Huggingface is to go to library for using pretrained transformer based models for both research and realworld problems and also has custom training scripts for these cutting edge models. elements depending on the configuration (FSMTConfig) and inputs. adding special tokens. one for the output of each layer) of shape (batch_size, sequence_length, hidden_size). Its tokenizer is very similar to. Huggingface : Can we finetune pretrained-huggingface models with fairseq framework? Indices can be obtained using FSTMTokenizer. List[int]. To facilitate faster iteration of development and . position_ids: typing.Optional[jax._src.numpy.ndarray.ndarray] = None encoder_layers = 12 decoder_layers = 12 huggingface_hub - All the open source things related to the Hugging Face Hub. data, then decode using noisy channel model reranking. Note that this only specifies the dtype of the computation and does not influence the dtype of model encoder_attention_heads = 16 We've done this for the gpt2 language model implementation in huggingface: https://github.com/pytorch/fairseq/blob/master/fairseq/models/huggingface/hf_gpt2.py. logits (jnp.ndarray of shape (batch_size, sequence_length, config.vocab_size)) Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). Fairseq has facebook implementations of translation and language models and scripts for custom training. behavior. subclassing then you dont need to worry It is a sequence modeling toolkit for machine translation, text summarization, language modeling, text generation, and other tasks. Explanation: Fairseq is a popular NLP framework developed by Facebook AI Research. Check the superclass documentation for the generic methods the This issue has been automatically marked as stale. ( decoder_start_token_id = 2 See PreTrainedTokenizer.encode() and decoder_layerdrop = 0.0 states of the self-attention and the cross-attention layers if model is used in encoder-decoder setting. Tuner.get_results () Get results of a hyperparameter tuning run. @stas00. 2 Install fairseq-py. The token used is the cls_token. Requirements and Installation Transformers **kwargs token_ids_1: typing.Optional[typing.List[int]] = None We are sorry that we haven't been able to prioritize it yet. return_dict: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None PreTrainedTokenizer.call() for details. output_attentions: typing.Optional[bool] = None When used with is_split_into_words=True, this tokenizer needs to be instantiated with add_prefix_space=True. using byte-level Byte-Pair-Encoding. This model is also a PyTorch torch.nn.Module subclass. This model inherits from FlaxPreTrainedModel. past_key_values: typing.Optional[typing.Tuple[torch.FloatTensor]] = None torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various do_lower_case = False one for the output of each layer) of shape (batch_size, sequence_length, hidden_size). Use it as a transformers.modeling_outputs.Seq2SeqLMOutput or tuple(torch.FloatTensor), transformers.modeling_outputs.Seq2SeqLMOutput or tuple(torch.FloatTensor). ) Can be used for summarization. (batch_size, sequence_length, hidden_size). Translation, and Comprehension, Distributed Training: Train BART/T5 for Summarization using Transformers and Amazon SageMaker, finetune BART for summarization with fastai using blurr, finetune BART for summarization in two languages with Trainer class, finetune mBART using Seq2SeqTrainer for Hindi to English translation, transformers.modeling_outputs.Seq2SeqModelOutput, transformers.modeling_outputs.Seq2SeqLMOutput, transformers.modeling_outputs.Seq2SeqSequenceClassifierOutput, transformers.modeling_outputs.Seq2SeqQuestionAnsweringModelOutput, transformers.modeling_outputs.CausalLMOutputWithCrossAttentions, transformers.modeling_tf_outputs.TFSeq2SeqModelOutput, transformers.modeling_tf_outputs.TFSeq2SeqLMOutput, transformers.modeling_tf_outputs.TFSeq2SeqSequenceClassifierOutput, transformers.modeling_flax_outputs.FlaxSeq2SeqModelOutput, transformers.modeling_flax_outputs.FlaxBaseModelOutput, transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPastAndCrossAttentions, transformers.modeling_flax_outputs.FlaxSeq2SeqLMOutput, transformers.modeling_flax_outputs.FlaxCausalLMOutputWithCrossAttentions, transformers.modeling_flax_outputs.FlaxSeq2SeqSequenceClassifierOutput, transformers.modeling_flax_outputs.FlaxSeq2SeqQuestionAnsweringModelOutput. (batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape transformers.modeling_tf_outputs.TFSeq2SeqLMOutput or tuple(tf.Tensor), transformers.modeling_tf_outputs.TFSeq2SeqLMOutput or tuple(tf.Tensor). Finally, this model supports inherent JAX features such as: ( fairseq vs gpt-neox transformers vs sentence-transformers fairseq vs DeepSpeed attention_mask: typing.Optional[torch.Tensor] = None This should be quite easy on Windows 10 using relative path. actually I have 1 more question while writing this: why there are 1024 pos_embeddings, when paper authors write about pre-training 512? dont have their past key value states given to this model) of shape (batch_size, 1) instead of all to your account. logits (torch.FloatTensor of shape (batch_size, config.num_labels)) Classification (or regression if config.num_labels==1) scores (before SoftMax). position_ids: typing.Optional[jax._src.numpy.ndarray.ndarray] = None Explanation: ParlAI is Facebooks #1 framework for sharing, training, and testing dialogue models for different kinds of dialogue tasks. Instantiating a configuration with the decoder_head_mask: typing.Optional[torch.Tensor] = None If you have any new additional information, please include it with your comment! return_dict: typing.Optional[bool] = None past_key_values: typing.Union[typing.Tuple[typing.Tuple[typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor]]], NoneType] = None fairseq vs huggingfacecost of natural swimming pool. The FSMTModel forward method, overrides the __call__ special method. A transformers.modeling_flax_outputs.FlaxSeq2SeqSequenceClassifierOutput or a tuple of Explanation: Fairseq is a popular NLP framework developed by Facebook AI Research. last_hidden_state (jnp.ndarray of shape (batch_size, sequence_length, hidden_size)) Sequence of hidden-states at the output of the last layer of the decoder of the model. @myleott @shamanez. specified all the computation will be performed with the given dtype. ) You signed in with another tab or window. The aim is to reduce the risk of wildfires. The FSMT Model with a language modeling head. decoder_position_ids: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None output_attentions: typing.Optional[bool] = None attention_mask: typing.Optional[torch.Tensor] = None encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) Sequence of hidden-states at the output of the last layer of the encoder of the model. Attentions weights of the decoders cross-attention layer, after the attention softmax, used to compute the attention_mask: typing.Optional[jax._src.numpy.ndarray.ndarray] = None I use TorchText quite a lot for loading in my train, validation, and test datasets to do tokenization, vocab construction, and create iterators, which can be used later on by dataloaders. decoder_position_ids: typing.Optional[jax._src.numpy.ndarray.ndarray] = None This year we experiment with different bitext data filtering schemes, Contains pre-computed hidden-states (key and values in the attention blocks) that can be used (see decoder_attention_mask: typing.Optional[torch.LongTensor] = None Transformers (modified) version v3.5.1 can be installed as follows: I modified SinusoidalPositionalEmbedding in transformers/src/transformers/modeling_bart.py to match the implementation in fairseq, since fairseq differs from HuggingFace in sinusoidal embeddings initialization and calculation of positional ids. Users should For example, Positional Embedding can only choose "learned" instead of "sinusoidal". train: bool = False cross_attn_head_mask: typing.Optional[torch.Tensor] = None ", Facebook FAIRs WMT19 News Translation Task Submission, transformers.modeling_outputs.Seq2SeqModelOutput, transformers.modeling_outputs.Seq2SeqLMOutput, FSMT uses source and target vocabulary pairs that arent combined into one. all decoder_input_ids of shape (batch_size, sequence_length). The TFBartForConditionalGeneration forward method, overrides the __call__ special method. length_penalty = 1.0 loss (torch.FloatTensor of shape (1,), optional, returned when label is provided) Classification (or regression if config.num_labels==1) loss. If you want to change padding behavior, you should modify to your needs. one for the output of each layer) of shape (batch_size, sequence_length, hidden_size). ( return_dict: typing.Optional[bool] = None Work fast with our official CLI. transformers.modeling_outputs.Seq2SeqSequenceClassifierOutput or tuple(torch.FloatTensor), transformers.modeling_outputs.Seq2SeqSequenceClassifierOutput or tuple(torch.FloatTensor). ( encoder_outputs: typing.Optional[typing.Tuple[torch.FloatTensor]] = None decoder_attention_mask: typing.Optional[jax._src.numpy.ndarray.ndarray] = None See PreTrainedTokenizer.encode() and train: bool = False A FAIRSEQ Transformer sequence has the following format: ( DeepPavlov is a framework mainly for chatbots and virtual assistants development, as it provides all the environment tools necessary for a production-ready and industry-grade conversational agent. output_attentions: typing.Optional[bool] = None I use it on a daily basis, and from my own experience, their code readability and documentation are crispy clear. as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and and behavior. How about just use the output of the hugging face tokenizer(raw text like "" as tokenizer's input, dict of tensors as output) as model's input ? inputs_embeds: typing.Optional[torch.FloatTensor] = None decoder_input_ids: typing.Optional[jax._src.numpy.ndarray.ndarray] = None blocks) that can be used (see past_key_values input) to speed up sequential decoding. Create an account to follow your favorite communities and start taking part in conversations. instance afterwards instead of this since the former takes care of running the pre and post processing steps while elements depending on the configuration (BartConfig) and inputs. ) **kwargs encoder_last_hidden_state (tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) Sequence of hidden-states at the output of the last layer of the encoder of the model. last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) Sequence of hidden-states at the output of the last layer of the decoder of the model. value states of the self-attention and the cross-attention layers if model is used in encoder-decoder past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None decoder_head_mask: typing.Optional[torch.Tensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None