Generator

BaseGenerator

class ofasys.generator.base.Generator

Base Class for Generator. As OFASys follows the encoder-decoder architecture, the output is mainly produced by the decoder. However, as the decoder accepts both E-slots and D-slots of various slot types, different generators are provided in OFASys to complement the differences in generation paradigm.

forward(model, sample, **kwargs)

Calls the generate() function to generate result by default.

Parameters

• model (the model object.) –

• sample (preprocessed batch input data.) –

abstract generate(model, sample, **kwargs)

Generate function. Should be overridden by all subclasses.

Parameters

• model (the model object.) –

• sample (preprocessed batch input data.) –

GeneratorOutput

class ofasys.generator.base.GeneratorOutput

Base Class for output of Generator. Users can directly get the output of the supported modality through calling the modality name. This class will also provide interfaces to dump output data as a file (e.g. png, wav, gif). Original output in tensor format and extra information are also provided.

SequenceGenerator

class ofasys.generator.sequence_generator.SequenceGenerator(tgt_dict, beam_size: int = 1, return_n_best: int = -1, max_len_a: int = 0, max_len_b: int = 200, max_len: int = 256, min_len: int = 1, normalize_scores: bool = True, len_penalty: float = 1.0, unk_penalty: float = 0.0, temperature: float = 1.0, match_source_len: bool = False, no_repeat_ngram_size: int = 0, search_strategy: Optional[Search] = None, lm_model=None, lm_weight: float = 1.0, constraint_trie: Optional[Trie] = None, constraint_range: Optional[str] = None, **unused_kwargs)

Bases: Generator

A autoregressive generator for discrete token sequences .

Modified from fairseq.

Parameters

• tgt_dict (Dictionary) – target dictionary

• beam_size (int, optional) – beam width (default: 1)

• return_n_best (int, optional) – -1, which indicates beam_size)

• max_len_a/b (int, optional) – generate sequences of maximum length ax + b, where x is the source length

• max_len (int, optional) – the maximum length of the generated output (not including end-of-sentence)

• min_len (int, optional) – the minimum length of the generated output (not including end-of-sentence)

• normalize_scores (bool, optional) – normalize scores by the length of the output (default: True)

• len_penalty (float, optional) – length penalty, where <1.0 favors shorter, >1.0 favors longer sentences (default: 1.0)

• unk_penalty (float, optional) – unknown word penalty, where <0 produces more unks, >0 produces fewer (default: 0.0)

• temperature (float, optional) – temperature, where values >1.0 produce more uniform samples and values <1.0 produce sharper samples (default: 1.0)

• match_source_len (bool, optional) – outputs should match the source length (default: False)

finalize_hypos(step: int, bbsz_idx, eos_scores, tokens, scores, finalized: List[Union[GeneratorOutput, List[GeneratorOutput]]], finished: List[bool], beam_size: int, attn: Optional[Tensor], src_lengths, max_len: int)

Finalize hypothesis, store finalized information in finalized, and change finished accordingly. A sentence is finalized when {beam_size} finished items have been collected for it.

Returns number of sentences (not beam items) being finalized. These will be removed from the batch and not processed further.

generate(model, sample, **kwargs)

Generate function.

Parameters

• models (ofasys.model.OFAModel) – OFAModel

• sample (dict) – batch

is_finished(step: int, unfin_idx: int, max_len: int, finalized_sent_len: int, beam_size: int)

Check whether decoding for a sentence is finished, which occurs when the list of finalized sentences has reached the beam size, or when we reach the maximum length.

replicate_first_beam(tensor, mask, beam_size: int)

training: bool

SequenceGeneratorOutput

class ofasys.generator.sequence_generator.SequenceGeneratorOutput(tokens: LongTensor, score: FloatTensor, attention: FloatTensor, positional_scores: FloatTensor, text: Optional[str] = None, image: Optional[Image] = None, box: Optional[Tensor] = None)

Bases: GeneratorOutput

Output of SequenceGenerator. Output with origin data format (e.g. string, png image file) of different modalities are available. Original output in tensor format and extra information are also provided.

attention: FloatTensor

box: Optional[Tensor] = None

image: Optional[Image] = None

positional_scores: FloatTensor

save_box(image_name: str)

save_image(image_name: str)

Save the output image to a file. :param image_name: :type image_name: image save path

score: FloatTensor

text: Optional[str] = None

tokens: LongTensor

SpeechGenerator

class ofasys.generator.speech_generator.SpeechGenerator(src_dict, stats_npz_path: Optional[str] = None)

Bases: Generator

Base Generator class for Audio modality.

gcmvn_denormalize(x)

training: bool

SpeechGeneratorOutput

class ofasys.generator.speech_generator.SpeechGeneratorOutput(feature: Union[FloatTensor, ndarray], eos_prob: FloatTensor, attn: FloatTensor, alignment: Tensor, text: Optional[str] = None, waveform: Optional[Union[FloatTensor, ndarray]] = None, targ_feature: Optional[Union[FloatTensor, ndarray]] = None, targ_waveform: Optional[Union[FloatTensor, ndarray]] = None)

Bases: GeneratorOutput

Output of SpeechGeneratorOutput. Output with origin data format (e.g. string, audio wav) of different modalities are available. Original output in tensor format and extra information are also provided.

alignment: Tensor

attn: FloatTensor

eos_prob: FloatTensor

feature: Union[FloatTensor, ndarray]

save_audio(audio_name: str, sample_rate: int = 22050, target: bool = False)

save_fbank(fbank_name: str, target: bool = False)

targ_feature: Optional[Union[FloatTensor, ndarray]] = None

targ_waveform: Optional[Union[FloatTensor, ndarray]] = None

text: Optional[str] = None

waveform: Optional[Union[FloatTensor, ndarray]] = None

AutoRegressiveSpeechGenerator

class ofasys.generator.speech_generator.AutoRegressiveSpeechGenerator(src_dict, stats_npz_path: Optional[str] = None, max_iter: int = 6000, eos_prob_threshold: float = 0.5, **unused_kwargs)

Bases: SpeechGenerator

A autoregressive generator for contiguous audio feature sequences . Modified from fairseq.

Parameters

• src_dict – source dictionary.

• stats_npz_path – gcmvn_stats path.

• max_iter – max iteration steps.

• eos_prob_threshold – threshold for generating end of sequence.

generate(model, sample, **kwargs)

Generate function.

training: bool

DiffusionGenerator

class ofasys.generator.diffusion_generator.DiffusionGenerator(general_preprocess, diffuser_args, **kwargs)

Bases: Generator

Diffusion generator.

Parameters

• general_preprocess – object of general preprocessor.

• diffuser_args – arguments passed to the __init__ of a Diffusion implementation such as GaussianDiffusion

generate(model, sample, **kwargs)

Generate function. Should be overridden by all subclasses.

training: bool

MotionOutput

class ofasys.generator.diffusion_generator.MotionOutput(feature: Union[FloatTensor, ndarray], target_feature: Optional[Union[FloatTensor, ndarray]] = None, prompt: Optional[str] = None, bvh_header: Optional[BvhHeader] = None, bvh_motion: Optional[ndarray] = None)

Bases: GeneratorOutput

Output of DiffusionGenerator. Output with origin data format (e.g. bvh, gif) are available. Original output in tensor format and extra information are also provided.

bvh_header: Optional[BvhHeader] = None

bvh_motion: Optional[ndarray] = None

feature: Union[FloatTensor, ndarray]

prompt: Optional[str] = None

save_as_bvh(bvh_name: str)

save output as a bvh file.

Parameters

bvh_name – save file path.

save_as_gif(gif_name: str)

save output as a gif file.

Parameters

gif_name – save file path.

save_features(feature_name: str)

save output feature as a npz file.

Parameters

feature_name – save file path.

target_feature: Optional[Union[FloatTensor, ndarray]] = None