Adaptor

Base Classes

BaseAdaptor

class ofasys.adaptor.base.BaseAdaptor(embed_tokens: Embedding, dictionary: Dictionary, is_src: bool, general_adaptor, cfg: BaseAdaptorConfig)

IO Adaptors convert modality data between its tensor form that is represented by computer program and embedding sequence that is readable by the universal computation model, e.g. OFA. In order to keep the Input Adaptors and the Output Adaptors are used in pairs, we use two methods in the same class to represent a pair of IO adaptors (forward for Input Adaptor and forward_output for Output Adaptor).

Parameters

• embed_tokens (Embedding) – global embedding matrix.

• dictionary (Dictionary) – global vocab.

• is_src (bool) – where is the adaptor used for .

• general_adaptor (GeneralAdaptor) – instance of GeneralAdaptor.

• cfg (BaseAdaptorConfig) – adaptor config.

expand_rel_pos_bias(values: Tensor, batch_size: int)

Expand and permute attention bias.

Parameters

• values (Tensor) – origin self attention bias of shape (seq_length, seq_length, num_attention_heads).

• batch_size (Int) – batch size of input data.

Returns

expanded attention bias of shape (batch_size, num_attention_heads, seq_length, seq_length)

Return type

Tensor

abstract forward(inputs: Union[Slot, List[Slot]], **kwargs) → AdaptorOutput

The Adaptor work as the InputAdaptor, takes corresponding data in tensor format as input, and then output sequences in the same format -ref AdaptorOutput.

Parameters

inputs (Slot) – preprocessed input data.

Returns

adaptor_output: adaptor output for the input slot.

Return type

AdaptorOutput

forward_hook_fn(inputs, output: AdaptorOutput)

This hook will be called every time after forward() has computed an output. Position embedding, type_embedding, layernorm_embedding and layernorm_position will be added to adaptor output. If output does not contain self_attn_bias, this hook will generate self_attn_bias list by calling get_rel_pos_bias() and expand them according to batch size.

Parameters

• inputs – model input.

• output – AdaptorOutput computed by the adaptor.

Returns

modified adaptor_output

Return type

AdaptorOutput

forward_output(x: Tensor, extra: Dict[str, Any], slot: Slot, **kwargs)

The Adaptor work as the OutputAdaptor, takes hidden states from model as input, and then output the modality data in their own form, e.g. probs on vocabulary.

Parameters

• x (Tensor) – hidden states from model in the shape of (batch_size, seq_length, embed_dim)

• extra (Dict[str, Any]) – extra model output information.

• slot (Slot) – input preprocessed data.

Returns

• x (Tensor): modality data in Tensor form.

• extra (Dict[str, Any]): model output with any modality-specific information.

Return type

tuple

abstract get_rel_pos_bias(batch_size, seq_length, idx, **kwargs)

Get relative position bias of self attention.

Parameters

• batch_size – batch size of input data.

• seq_length – sequence length of input data.

• idx – layer index.

Returns

attention bias.

Return type

Tensor

BaseAdaptorConfig

class ofasys.adaptor.base.BaseAdaptorConfig(_name: Union[str, NoneType] = None, is_active: bool = False, layernorm_embedding: bool = True, layernorm_position: bool = True, add_type_embedding: bool = True, entangle_position_embedding: bool = False, no_scale_embedding: bool = True, scale_embedding_gradient: float = 1.0, dropout: float = None, embed_dim: int = None, num_attention_heads: int = None, encoder_layers: int = None, decoder_layers: int = None, max_position: int = None, use_self_attn_bias: bool = None, share_attn_bias: bool = None)

add_type_embedding: bool = True

decoder_layers: int = None

dropout: float = None

embed_dim: int = None

encoder_layers: int = None

entangle_position_embedding: bool = False

is_active: bool = False

layernorm_embedding: bool = True

layernorm_position: bool = True

max_position: int = None

no_scale_embedding: bool = True

num_attention_heads: int = None

parse_from_model_cfg(model_cfg)

scale_embedding_gradient: float = 1.0

share_attn_bias: bool = None

use_self_attn_bias: bool = None

AdaptorOutput

class ofasys.adaptor.base.AdaptorOutput(embed: FloatTensor, masks: BoolTensor, pos_embed: FloatTensor, self_attn_bias: List[FloatTensor], modal_mask: Optional[IntTensor] = None)

Parameters

• embed (torch.FloatTensor) – the processed embedding for OFA of shape (batch_size, seq_length, hidden_size)

• masks (torch.BoolTensor) – the positions of padding elements of shape (batch, src_len)

• pos_embed (torch.FloatTensor) – the position embeddings of shape (batch_size, seq_length, hidden_size)

• self_attn_bias (List[torch.FloatTensor], optional) – attention bias in self attention of shape (batch_size, num_attention_heads, seq_length, seq_length)

OFAGeneralAdaptor

class ofasys.adaptor.general.OFAGeneralAdaptor(cfg, dictionary, is_src)

General adaptor will dispatch slot to its adaptor (or default adaptor for its Modality). General will init each adaptor (if is_activate). Like ** BaseAdaptor , GeneralAdaptor can work for both IO Adaptors (**forward for Input Adaptor and forward_output for Output Adaptor).

Parameters

• cfg – model config.

• dictionary (Dictionary) – global vocab.

• is_src (bool) – where is the adaptor used for .

build_embedding(cfg, dictionary)

Parameters

• cfg – model config.

• dictionary (Dictionary) – global vocab.

Returns

global embedding matrix.

Return type

Embedding

concat(modality_outputs: List[AdaptorOutput]) → AdaptorOutput

Concatenate all adaptor outputs into a large AdaptorOutput in order.

Parameters

modality_outputs (List[AdaptorOutput]) – AdaptorOutput from different slots.

Returns

concatenated AdaptorOuptut, which will be fed into the computation model.

Return type

AdaptorOutput

forward(slots: List[Slot], **kwargs)

When work as GeneranlInputAdaptor, GeneralAdaptor will dispatch each slot to its adaptor, then gather all AdaptorOutputs and concatenate them to one AdaptorOutput by using self.concat(). return a tuple instead of AdaptorOutput as checkpoint_activations need iterable object.

Parameters

slots – preprocessed input slots.

Returns

concatenated embedding.

Return type

tuple

forward_output(x: Tensor, extra: Dict[str, Any], slots: List[Slot], **kwargs)

When work as GeneralOutputAdaptor, GeneralAdaptor will dispatch hidden states from model to the target Output Adaptor ( by calling method forward_output()).

Note

Only one Output Adaptor is supported now, which means we only allow one Slot in the target sequence of the Instruction.

Parameters

• x (Tensor) – hidden states from model in the shape of (batch_size, seq_length, embed_dim)

• extra (Dict[str, Any]) – extra model output information.

• slots (List[Slot]) – input preprocessed data.

Returns

• x (Tensor): modality data in Tensor form.

• extra (Dict[str, Any]): model output with any modality-specific information.

Return type

tuple

get_adaptor(slot: Slot) → BaseAdaptor

Get Adaptor for the given Slot. If the Slot is not assigned with a adaptor name in the Instruction, we will use the default Adaptor for its modality.

Parameters

slot (Slot) – preprocessed input data.

Returns

Adaptor for Slot.

Return type

BaseAdaptor

upgrade_state_dict_named(state_dict, name)

Upgrade a (possibly old) state dict for new versions of ofa.

Image

ImageResnetAdaptor

class ofasys.adaptor.image_resnet.ImageResnetAdaptor(embed_tokens: Embedding, dictionary: Dictionary, is_src: bool, general_adaptor, cfg: ImageResnetAdaptorConfig)

Bases: BaseAdaptor

IO Adaptors convert modality data between its tensor form that is represented by computer program and embedding sequence that is readable by the universal computation model, e.g. OFA. In order to keep the Input Adaptors and the Output Adaptors are used in pairs, we use two methods in the same class to represent a pair of IO adaptors (forward for Input Adaptor and forward_output for Output Adaptor).

Parameters

• embed_tokens (Embedding) – global embedding matrix.

• dictionary (Dictionary) – global vocab.

• is_src (bool) – where is the adaptor used for .

• general_adaptor (GeneralAdaptor) – instance of GeneralAdaptor.

• cfg (BaseAdaptorConfig) – adaptor config.

forward(slot: Slot, **kwargs) → AdaptorOutput

Parameters

slot (Slot) –

• ModalityType: IMAGE

• value: stacked image tensors of size (batch_size, num_channels, height, width)

Returns

• embed (Tensor): the processed embedding for OFA of shape (src_len, batch, embed_dim)

• padding_masks (ByteTensor): the positions of padding elements of shape (batch, src_len)

• pos_embedding (Tensor): the position embeddings of shape (batch, src_len, embed_dim)

• self_attn_bias (List[Tensor]): attention bias in self attention of shape (batch, num_attention_heads, src_len, src_len).

Return type

AdaptorOutput

get_patch_images_info(patch_images)

Parameters

patch_images (Tensor) – stacked image tensors of size (batch_size, num_channels, height, width)

Returns

• image_embed (Tensor): the processed embedding for OFA of shape (src_len, batch, embed_dim)

• image_num_patches (int): the number of image patches.

• image_padding_masks (ByteTensor): the positions of padding elements of shape (batch, src_len)

• image_position_ids (Tensor): the position ids of shape (batch, src_len)

• image_pos_embed (Tensor): the position embeddings of shape (batch, src_len, embed_dim)

Return type

Tuple

get_rel_pos_bias(batch_size, seq_length, idx, **kwargs)

Get relative position bias of self attention.

Parameters

• batch_size – batch size of input data.

• seq_length – sequence length of input data.

• idx – layer index.

Returns

attention bias.

Return type

Tensor

train(mode=True)

Sets the module in training mode.

This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

Parameters

mode (bool) – whether to set training mode (True) or evaluation mode (False). Default: True.

Returns

self

Return type

Module

training: bool

ImageResnetAdaptorConfig

class ofasys.adaptor.image_resnet.ImageResnetAdaptorConfig(_name: Union[str, NoneType] = None, is_active: bool = False, layernorm_embedding: bool = True, layernorm_position: bool = True, add_type_embedding: bool = True, entangle_position_embedding: bool = False, no_scale_embedding: bool = True, scale_embedding_gradient: float = 1.0, dropout: float = None, embed_dim: int = None, num_attention_heads: int = None, encoder_layers: int = None, decoder_layers: int = None, max_position: int = None, use_self_attn_bias: bool = None, share_attn_bias: bool = None, resnet_type: ofasys.configure.constants.Choices = 'resnet152', resnet_drop_path_rate: float = 0.0, sync_bn: bool = False, freeze_resnet: bool = False, image_bucket_size: int = 42, pretrained_ckpt_path: str = '')

Bases: BaseAdaptorConfig

freeze_resnet: bool = False

image_bucket_size: int = 42

pretrained_ckpt_path: str = ''

resnet_drop_path_rate: float = 0.0

resnet_type: Choices = 'resnet152'

sync_bn: bool = False

ImageVqganAdaptor

class ofasys.adaptor.image_vqgan.ImageVqganAdaptor(embed_tokens: Embedding, dictionary: Dictionary, is_src: bool, general_adaptor, cfg: ImageVqganAdaptorConfig)

Bases: BaseAdaptor

IO Adaptors convert modality data between its tensor form that is represented by computer program and embedding sequence that is readable by the universal computation model, e.g. OFA. In order to keep the Input Adaptors and the Output Adaptors are used in pairs, we use two methods in the same class to represent a pair of IO adaptors (forward for Input Adaptor and forward_output for Output Adaptor).

Parameters

• embed_tokens (Embedding) – global embedding matrix.

• dictionary (Dictionary) – global vocab.

• is_src (bool) – where is the adaptor used for .

• general_adaptor (GeneralAdaptor) – instance of GeneralAdaptor.

• cfg (BaseAdaptorConfig) – adaptor config.

forward(slot: Slot, **kwargs) → AdaptorOutput

Parameters

slot (Slot) – ModalityType.IMAGE

Returns

• embed (Tensor): the processed embedding for OFA of shape (src_len, batch, embed_dim)

• padding_masks (ByteTensor): the positions of padding elements of shape (batch, src_len)

• pos_embedding (Tensor): the position embeddings of shape (batch, src_len, embed_dim)

Return type

AdaptorOutput

forward_output(x: Tensor, extra: Dict[str, Any], slot: Slot, **kwargs)

Parameters

• x (Tensor) – hidden states from model in the shape of (batch_size, seq_length, embed_dim)

• extra (Dict[str, Any]) – extra model output information.

• slot (Slot) – input preprocessed data.

Returns

• x (Tensor): Tensor of shape (batch_size, seq_length, vocab_size).

• extra (Dict[str, Any]): model output with any modality-specific information.

Return type

tuple

get_rel_pos_bias(batch_size, seq_length, idx, **kwargs)

Get relative position bias of self attention.

Parameters

• batch_size – batch size of input data.

• seq_length – sequence length of input data.

• idx – layer index.

Returns

attention bias.

Return type

Tensor

training: bool

update_sample(sample: Dict)

preprocess sample on gpu.

Parameters

sample (Dict) – preprocessed data named dict

Returns

sample: add vqgan encoded images to slot.value

Return type

Dict

upgrade_state_dict_named(state_dict, name)

Upgrade a (possibly old) state dict for new versions of ofa.

ImageVqganAdaptorConfig

class ofasys.adaptor.image_vqgan.ImageVqganAdaptorConfig(_name: Union[str, NoneType] = None, is_active: bool = False, layernorm_embedding: bool = True, layernorm_position: bool = True, add_type_embedding: bool = True, entangle_position_embedding: bool = False, no_scale_embedding: bool = True, scale_embedding_gradient: float = 1.0, dropout: float = None, embed_dim: int = None, num_attention_heads: int = None, encoder_layers: int = None, decoder_layers: int = None, max_position: int = None, use_self_attn_bias: bool = None, share_attn_bias: bool = None, code_image_size: int = 256, code_bucket_size: int = 42, vqgan_factor: int = 8, vqgan_model_path: str = 'oss://ofasys/tasks/image_gen/vqgan/last.ckpt', vqgan_config_path: str = 'oss://ofasys/tasks/image_gen/vqgan/model.yaml', use_encode: bool = True, code_entry_prefix: str = 'code')

Bases: BaseAdaptorConfig

code_bucket_size: int = 42

code_entry_prefix: str = 'code'

code_image_size: int = 256

use_encode: bool = True

vqgan_config_path: str = 'oss://ofasys/tasks/image_gen/vqgan/model.yaml'

vqgan_factor: int = 8

vqgan_model_path: str = 'oss://ofasys/tasks/image_gen/vqgan/last.ckpt'

Text

TextAdaptor

class ofasys.adaptor.text.TextAdaptor(embed_tokens: Embedding, dictionary: Dictionary, is_src: bool, general_adaptor, cfg: TextAdaptorConfig)

Bases: BaseAdaptor

IO Adaptors convert modality data between its tensor form that is represented by computer program and embedding sequence that is readable by the universal computation model, e.g. OFA. In order to keep the Input Adaptors and the Output Adaptors are used in pairs, we use two methods in the same class to represent a pair of IO adaptors (forward for Input Adaptor and forward_output for Output Adaptor).

Parameters

• embed_tokens (Embedding) – global embedding matrix.

• dictionary (Dictionary) – global vocab.

• is_src (bool) – where is the adaptor used for .

• general_adaptor (GeneralAdaptor) – instance of GeneralAdaptor.

• cfg (BaseAdaptorConfig) – adaptor config.

build_output_projection(dictionary)

forward(slot: Slot, **kwargs) → AdaptorOutput

Parameters

slot (Slot) – ModalityType.Text

Returns

• embed (Tensor): the processed embedding for OFA of shape (src_len, batch, embed_dim)

• padding_masks (ByteTensor): the positions of padding elements of shape (batch, src_len)

• pos_embedding (Tensor): the position embeddings of shape (batch, src_len, embed_dim)

Return type

AdaptorOutput

forward_output(x: Tensor, extra: Dict[str, Any], slot: Slot, **kwargs)

Parameters

• x (Tensor) – hidden states from model in the shape of (batch_size, seq_length, embed_dim)

• extra (Dict[str, Any]) – extra model output information.

• slot (Slot) – input preprocessed data.

Returns

• x (Tensor): Tensor of shape (batch_size, seq_length, vocab_size).

• extra (Dict[str, Any]): model output with any modality-specific information.

Return type

tuple

get_rel_pos_bias(batch_size, seq_length, idx, **kwargs)

Get relative position bias of self attention.

Parameters

• batch_size – batch size of input data.

• seq_length – sequence length of input data.

• idx – layer index.

Returns

attention bias.

Return type

Tensor

training: bool

TextAdaptorConfig

class ofasys.adaptor.text.TextAdaptorConfig(_name: Union[str, NoneType] = None, is_active: bool = False, layernorm_embedding: bool = True, layernorm_position: bool = True, add_type_embedding: bool = True, entangle_position_embedding: bool = False, no_scale_embedding: bool = True, scale_embedding_gradient: float = 1.0, dropout: float = None, embed_dim: int = None, num_attention_heads: int = None, encoder_layers: int = None, decoder_layers: int = None, max_position: int = None, use_self_attn_bias: bool = None, share_attn_bias: bool = None, token_bucket_size: int = 256, share_input_output_embed: bool = True, output_embed_dim: Union[int, NoneType] = 512, output_dim: Union[int, NoneType] = None, output_bias: bool = False)

Bases: BaseAdaptorConfig

output_bias: bool = False

output_dim: Optional[int] = None

output_embed_dim: Optional[int] = 512

share_input_output_embed: bool = True

token_bucket_size: int = 256

Audio

AudioFbankAdaptor

class ofasys.adaptor.audio.AudioFbankAdaptor(embed_tokens: Embedding, dictionary: Dictionary, is_src: bool, general_adaptor, cfg: AudioFbankAdaptorConfig)

Bases: BaseAdaptor

IO Adaptors convert modality data between its tensor form that is represented by computer program and embedding sequence that is readable by the universal computation model, e.g. OFA. In order to keep the Input Adaptors and the Output Adaptors are used in pairs, we use two methods in the same class to represent a pair of IO adaptors (forward for Input Adaptor and forward_output for Output Adaptor).

Parameters

• embed_tokens (Embedding) – global embedding matrix.

• dictionary (Dictionary) – global vocab.

• is_src (bool) – where is the adaptor used for .

• general_adaptor (GeneralAdaptor) – instance of GeneralAdaptor.

• cfg (BaseAdaptorConfig) – adaptor config.

apply_mask(x, padding_mask, mask_indices=None, mask_channel_indices=None, mask_prob=None)

build_encoder_layer(cfg, drop_path_rate=0.0)

forward(slot: Slot, **kwargs) → AdaptorOutput

Parameters

slot (Slot) – ModalityType.AUDIO

Returns

• embed (Tensor): the processed embedding for OFA of shape (src_len, batch, embed_dim)

• padding_masks (ByteTensor): the positions of padding elements of shape (batch, src_len)

• pos_embedding (Tensor): the position embeddings of shape (batch, src_len, embed_dim)

• self_attn_bias (List[Tensor]): attention bias in self attention of shape (batch, num_attention_heads, src_len, src_len).

Return type

AdaptorOutput

forward_hook_fn(inputs, output: AdaptorOutput)

This hook will be called every time after forward() has computed an output. Position embedding, type_embedding, layernorm_embedding and layernorm_position will be added to adaptor output. If output does not contain self_attn_bias, this hook will generate self_attn_bias list by calling get_rel_pos_bias() and expand them according to batch size.

Parameters

• inputs – model input.

• output – AdaptorOutput computed by the adaptor.

Returns

modified adaptor_output

Return type

AdaptorOutput

forward_output(x: Tensor, extra: Dict[str, Any], slot: Slot, **kwargs)

The Adaptor work as the OutputAdaptor, takes hidden states from model as input, and then output the modality data in their own form, e.g. probs on vocabulary.

Parameters

• x (Tensor) – hidden states from model in the shape of (batch_size, seq_length, embed_dim)

• extra (Dict[str, Any]) – extra model output information.

• slot (Slot) – input preprocessed data.

Returns

• x (Tensor): modality data in Tensor form.

• extra (Dict[str, Any]): model output with any modality-specific information.

Return type

tuple

get_mask_indices(B, T, C, mask_prob=None)

get_rel_pos_bias(batch_size, seq_length, idx, **kwargs)

Get relative position bias of self attention.

Parameters

• batch_size – batch size of input data.

• seq_length – sequence length of input data.

• idx – layer index.

Returns

attention bias.

Return type

Tensor

training: bool

AudioFbankAdaptorConfig

class ofasys.adaptor.audio.AudioFbankAdaptorConfig(_name: Union[str, NoneType] = None, is_active: bool = False, layernorm_embedding: bool = True, layernorm_position: bool = True, add_type_embedding: bool = True, entangle_position_embedding: bool = False, no_scale_embedding: bool = True, scale_embedding_gradient: float = 1.0, dropout: float = None, embed_dim: int = None, num_attention_heads: int = None, encoder_layers: int = None, decoder_layers: int = None, max_position: int = None, use_self_attn_bias: bool = None, share_attn_bias: bool = None, output_frame_dim: int = 80, n_frames_per_step: int = 1, is_transformer_layers: bool = False, encoder_config: ofasys.module.transformer_config.EncDecBaseConfig = EncDecBaseConfig(_name=None, embed_path=None, embed_dim=512, ffn_embed_dim=2048, layers=6, attention_heads=8, normalize_before=False, learned_pos=False, layerdrop=0, layers_to_keep=None), encode_drop_path_rate: float = 0.0, checkpoint_activations: bool = False, min_params_to_wrap: int = 100000000, attn_scale_factor: float = 2, scale_attn: bool = True, scale_fc: bool = True, scale_heads: bool = True, scale_resids: bool = False, use_fused: bool = True, prenet_layers: int = 2, prenet_dim: int = 256, prenet_dropout: float = 0.5, postnet_conv_dim: int = 512, postnet_conv_kernel_size: int = 5, postnet_layers: int = 5, postnet_dropout: float = 0.5, use_mask: bool = False, mask_length: int = 10, mask_prob: float = 0.65, mask_selection: ofasys.configure.constants.Choices = 'static', mask_other: float = 0, no_mask_overlap: bool = False, mask_min_space: int = 1, mask_channel_length: int = 10, mask_channel_prob: float = 0.0, mask_channel_selection: ofasys.configure.constants.Choices = 'static', mask_channel_other: float = 0, no_mask_channel_overlap: bool = False, mask_channel_min_space: int = 1, mask_channel_before: bool = False, require_same_masks: bool = True, mask_dropout: float = 0.0)

Bases: BaseAdaptorConfig

attn_scale_factor: float = 2

checkpoint_activations: bool = False

encode_drop_path_rate: float = 0.0

encoder_config: EncDecBaseConfig = EncDecBaseConfig(_name=None, embed_path=None, embed_dim=512, ffn_embed_dim=2048, layers=6, attention_heads=8, normalize_before=False, learned_pos=False, layerdrop=0, layers_to_keep=None)

is_transformer_layers: bool = False

mask_channel_before: bool = False

mask_channel_length: int = 10

mask_channel_min_space: int = 1

mask_channel_other: float = 0

mask_channel_prob: float = 0.0

mask_channel_selection: Choices = 'static'

mask_dropout: float = 0.0

mask_length: int = 10

mask_min_space: int = 1

mask_other: float = 0

mask_prob: float = 0.65

mask_selection: Choices = 'static'

min_params_to_wrap: int = 100000000

n_frames_per_step: int = 1

no_mask_channel_overlap: bool = False

no_mask_overlap: bool = False

output_frame_dim: int = 80

postnet_conv_dim: int = 512

postnet_conv_kernel_size: int = 5

postnet_dropout: float = 0.5

postnet_layers: int = 5

prenet_dim: int = 256

prenet_dropout: float = 0.5

prenet_layers: int = 2

require_same_masks: bool = True

scale_attn: bool = True

scale_fc: bool = True

scale_heads: bool = True

scale_resids: bool = False

use_fused: bool = True

use_mask: bool = False

AudioTargetFbankAdaptor

class ofasys.adaptor.audio.AudioTargetFbankAdaptor(embed_tokens: Embedding, dictionary: Dictionary, is_src: bool, general_adaptor, cfg: AudioTargetFbankAdaptorConfig)

Bases: BaseAdaptor

IO Adaptors convert modality data between its tensor form that is represented by computer program and embedding sequence that is readable by the universal computation model, e.g. OFA. In order to keep the Input Adaptors and the Output Adaptors are used in pairs, we use two methods in the same class to represent a pair of IO adaptors (forward for Input Adaptor and forward_output for Output Adaptor).

Parameters

• embed_tokens (Embedding) – global embedding matrix.

• dictionary (Dictionary) – global vocab.

• is_src (bool) – where is the adaptor used for .

• general_adaptor (GeneralAdaptor) – instance of GeneralAdaptor.

• cfg (BaseAdaptorConfig) – adaptor config.

forward(slot: Slot, **kwargs) → AdaptorOutput

Parameters

slot (Slot) – ModalityType.AUDIO

Returns

• embed (Tensor): the processed embedding for OFA of shape (src_len, batch, embed_dim)

• padding_masks (ByteTensor): the positions of padding elements of shape (batch, src_len)

• pos_embedding (Tensor): the position embeddings of shape (batch, src_len, embed_dim)

• self_attn_bias (List[Tensor]): attention bias in self attention of shape (batch, num_attention_heads, src_len, src_len).

Return type

AdaptorOutput

forward_output(x: Tensor, extra: Dict[str, Any], slot: Slot, **kwargs)

The Adaptor work as the OutputAdaptor, takes hidden states from model as input, and then output the modality data in their own form, e.g. probs on vocabulary.

Parameters

• x (Tensor) – hidden states from model in the shape of (batch_size, seq_length, embed_dim)

• extra (Dict[str, Any]) – extra model output information.

• slot (Slot) – input preprocessed data.

Returns

• x (Tensor): modality data in Tensor form.

• extra (Dict[str, Any]): model output with any modality-specific information.

Return type

tuple

get_rel_pos_bias(batch_size, seq_length, idx, **kwargs)

Get relative position bias of self attention.

Parameters

• batch_size – batch size of input data.

• seq_length – sequence length of input data.

• idx – layer index.

Returns

attention bias.

Return type

Tensor

training: bool

AudioFbankAdaptorConfig

class ofasys.adaptor.audio.AudioTargetFbankAdaptorConfig(_name: Union[str, NoneType] = None, is_active: bool = False, layernorm_embedding: bool = True, layernorm_position: bool = True, add_type_embedding: bool = True, entangle_position_embedding: bool = False, no_scale_embedding: bool = True, scale_embedding_gradient: float = 1.0, dropout: float = None, embed_dim: int = None, num_attention_heads: int = None, encoder_layers: int = None, decoder_layers: int = None, max_position: int = None, use_self_attn_bias: bool = None, share_attn_bias: bool = None, output_frame_dim: int = 80, n_frames_per_step: int = 1, conv_kernel_size: int = 5, prenet_layers: int = 2, prenet_dim: int = 256, prenet_dropout: float = 0.5, postnet_conv_dim: int = 512, postnet_conv_kernel_size: int = 5, postnet_layers: int = 5, postnet_dropout: float = 0.5)

Bases: BaseAdaptorConfig

conv_kernel_size: int = 5

n_frames_per_step: int = 1

output_frame_dim: int = 80

postnet_conv_dim: int = 512

postnet_conv_kernel_size: int = 5

postnet_dropout: float = 0.5

postnet_layers: int = 5

prenet_dim: int = 256

prenet_dropout: float = 0.5

prenet_layers: int = 2

Motion

Motion6dAdaptor

class ofasys.adaptor.motion_6d.Motion6dAdaptor(embed_tokens: Embedding, dictionary: Dictionary, is_src: bool, general_adaptor, cfg: Motion6dAdaptorConfig)

Bases: TextAdaptor

IO Adaptors convert modality data between its tensor form that is represented by computer program and embedding sequence that is readable by the universal computation model, e.g. OFA. In order to keep the Input Adaptors and the Output Adaptors are used in pairs, we use two methods in the same class to represent a pair of IO adaptors (forward for Input Adaptor and forward_output for Output Adaptor).

Parameters

• embed_tokens (Embedding) – global embedding matrix.

• dictionary (Dictionary) – global vocab.

• is_src (bool) – where is the adaptor used for .

• general_adaptor (GeneralAdaptor) – instance of GeneralAdaptor.

• cfg (BaseAdaptorConfig) – adaptor config.

forward(slot: Slot, **kwargs) → AdaptorOutput

Parameters

slot (Slot) – ModalityType.Motion

Returns

• embed (Tensor): the processed embedding for OFA of shape (src_len, batch, embed_dim)

• padding_masks (ByteTensor): the positions of padding elements of shape (batch, src_len)

• pos_embedding (Tensor): the position embeddings of shape (batch, src_len, embed_dim)

Return type

AdaptorOutput

forward_output(x: Tensor, extra: Dict[str, Any], slot: Slot, **kwargs)

Parameters

• x (Tensor) – hidden states from model in the shape of (batch_size, seq_length, embed_dim)

• extra (Dict[str, Any]) – extra model output information.

• slot (Slot) – input preprocessed data.

Returns

• x (Tensor): Tensor of shape (batch_size, seq_len, data_dim).

• extra (Dict[str, Any]): model output with any modality-specific information.

Return type

tuple

output_dim: int

output_embed_bias: bool

training: bool

Motion6dAdaptorConfig

class ofasys.adaptor.motion_6d.Motion6dAdaptorConfig(_name: Union[str, NoneType] = None, is_active: bool = False, layernorm_embedding: bool = True, layernorm_position: bool = True, add_type_embedding: bool = True, entangle_position_embedding: bool = False, no_scale_embedding: bool = True, scale_embedding_gradient: float = 1.0, dropout: float = None, embed_dim: int = None, num_attention_heads: int = None, encoder_layers: int = None, decoder_layers: int = None, max_position: int = None, use_self_attn_bias: bool = None, share_attn_bias: bool = None, token_bucket_size: int = 256, share_input_output_embed: bool = True, output_embed_dim: Union[int, NoneType] = 512, output_dim: Union[int, NoneType] = None, output_bias: bool = False, max_data_dim: int = 512, max_noise_levels: int = 1024)

Bases: TextAdaptorConfig

max_data_dim: int = 512

max_noise_levels: int = 1024

Video

VideoImageSequenceAdaptor

class ofasys.adaptor.video_image_sequence.VideoImageSequenceAdaptor(embed_tokens: Embedding, dictionary: Dictionary, is_src: bool, general_adaptor, cfg: VideoImageSequenceAdaptorConfig)

Bases: BaseAdaptor

IO Adaptors convert modality data between its tensor form that is represented by computer program and embedding sequence that is readable by the universal computation model, e.g. OFA. In order to keep the Input Adaptors and the Output Adaptors are used in pairs, we use two methods in the same class to represent a pair of IO adaptors (forward for Input Adaptor and forward_output for Output Adaptor).

Parameters

• embed_tokens (Embedding) – global embedding matrix.

• dictionary (Dictionary) – global vocab.

• is_src (bool) – where is the adaptor used for .

• general_adaptor (GeneralAdaptor) – instance of GeneralAdaptor.

• cfg (BaseAdaptorConfig) – adaptor config.

forward(slot: Slot, **kwargs) → AdaptorOutput

Parameters

slot (Slot) – ModalityType.VIDEO

Returns

• embed (Tensor): the processed embedding for OFA of shape (src_len, batch, embed_dim)

• padding_masks (ByteTensor): the positions of padding elements of shape (batch, src_len)

• pos_embedding (Tensor): the position embeddings of shape (batch, src_len, embed_dim)

• self_attn_bias (List[Tensor]): attention bias in self attention of shape (batch, num_attention_heads, src_len, src_len).

Return type

AdaptorOutput

get_clip_videos_info(clip_videos: Tensor)

get_image_resnet_adaptor() → ImageResnetAdaptor

get_rel_pos_bias(batch_size, seq_length, idx, **kwargs)

Get relative position bias of self attention.

Parameters

• batch_size – batch size of input data.

• seq_length – sequence length of input data.

• idx – layer index.

Returns

attention bias.

Return type

Tensor

train(mode=True)

Sets the module in training mode.

This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

Parameters

mode (bool) – whether to set training mode (True) or evaluation mode (False). Default: True.

Returns

self

Return type

Module

training: bool

upgrade_state_dict_named(state_dict, name)

VideoImageSequenceAdaptorConfig

class ofasys.adaptor.video_image_sequence.VideoImageSequenceAdaptorConfig(_name: Union[str, NoneType] = None, is_active: bool = False, layernorm_embedding: bool = True, layernorm_position: bool = True, add_type_embedding: bool = True, entangle_position_embedding: bool = False, no_scale_embedding: bool = True, scale_embedding_gradient: float = 1.0, dropout: float = None, embed_dim: int = None, num_attention_heads: int = None, encoder_layers: int = None, decoder_layers: int = None, max_position: int = None, use_self_attn_bias: bool = None, share_attn_bias: bool = None, token_bucket_size: int = 256)

Bases: BaseAdaptorConfig

token_bucket_size: int = 256