Preprocessor

Base Classes

BasePreprocess

class ofasys.preprocessor.default.base.BasePreprocess(global_dict: Dictionary, cfg: PreprocessConfig)

The preprocessor converts the raw modal data of a single sample into the batch input accepted by the neural network. The preprocessor runs inside the dataloader and can be parallelized on multiple processes through setting num_workers.

Each mode has its own preprocessors. Preprocessing generally consists of four sequential phases, namely instruction_map, map, group_map, and collate.

Parameters

• global_dict (Dictionary) – global vocab

• cfg (PreprocessConfig) – preprocess config

abstract collate(slots: List[Slot]) → CollateOutput

The collate phase of the preprocessor takes a batch of Slot as input, collate their values and outputs a CollateOutput. This phase defines how to collate a batch for a single modal.

dummy_slot(slot)

Set dummy value for slot, which is used for inference.

group_key(slot: Slot) → ModalityType

group_key returns a key for reducing continuous modal.

abstract group_map(slots: List[Slot]) → List[Slot]

The group_map phase of the preprocessor takes a list of Slot as input, processes their values and outputs a list of Slot. This phase defines how to reduce continuous modes in a sample.

instruction_map(ist_data: Instruction) → Instruction

The instruction_map phase of the preprocessor takes the whole Instruction as input and outputs a preprocessed one. This function is mainly used to cooperatively process multiple types of modal inputs.

abstract map(slot: Slot) → Slot

The map phase of the preprocessor takes a Slot as input and outputs a preprocessed Slot. This phase defines how to preprocess the raw data of a single modal.

Parameters

inputs (Slot) – raw input data.

Returns

preprocessed data of a single modal

Return type

output (Slot)

postprocess(outputs, **sample)

BasePreprocessConfig

class ofasys.preprocessor.default.base.PreprocessConfig(_name: Union[str, NoneType] = None, is_active: bool = False, pad_to_multiple: int = 1)

is_active: bool = False

pad_to_multiple: int = 1

SafeBasePreprocess

class ofasys.preprocessor.default.base.SafeBasePreprocess(global_dict, cfg: PreprocessConfig, modality_type: ModalityType, sanity_check: bool = True)

collate(slots: List[Slot]) → CollateOutput

The collate phase of the preprocessor takes a batch of Slot as input, collate their values and outputs a CollateOutput. This phase defines how to collate a batch for a single modal.

group_map(slots: List[Slot]) → List[Slot]

The group_map phase of the preprocessor takes a list of Slot as input, processes their values and outputs a list of Slot. This phase defines how to reduce continuous modes in a sample.

map(slot: Slot) → Slot

The map phase of the preprocessor takes a Slot as input and outputs a preprocessed Slot. This phase defines how to preprocess the raw data of a single modal.

Parameters

inputs (Slot) – raw input data.

Returns

preprocessed data of a single modal

Return type

output (Slot)

GeneralPreprocess

class ofasys.preprocessor.general.GeneralPreprocess(cfg: PreprocessConfig, global_dict: Dictionary)

property bos

property bpe

collate(samples: List[Instruction]) → Dict

property eos

get_name2pre(cfg)

get_preprocess(slot: Slot) → BasePreprocess

property pad

postprocess(outputs, **sample)

prepare_for_generation(closed_set, **kwargs)

Image

DefaultImagePreprocess

class ofasys.preprocessor.default.image.DefaultImagePreprocess(global_dict, cfg: ImagePreprocessConfig)

Bases: SafeBasePreprocess

collate(slots: List[Slot]) → CollateOutput

The collate phase of the preprocessor takes a batch of Slot as input, collate their values and outputs a CollateOutput. This phase defines how to collate a batch for a single modal.

map(slot: Slot) → Slot

The map phase of the preprocessor takes a Slot as input and outputs a preprocessed Slot. This phase defines how to preprocess the raw data of a single modal.

Parameters

inputs (Slot) – raw input data.

Returns

preprocessed data of a single modal

Return type

output (Slot)

DefaultImagePreprocessConfig

class ofasys.preprocessor.default.image.ImagePreprocessConfig(_name: Union[str, NoneType] = None, is_active: bool = False, pad_to_multiple: int = 1, patch_image_size: int = 480, imagenet_default_mean_and_std: bool = False, interpolation: str = 'bicubic')

Bases: PreprocessConfig

imagenet_default_mean_and_std: bool = False

interpolation: str = 'bicubic'

patch_image_size: int = 480

ImagenetImagePreprocess

class ofasys.preprocessor.default.image.ImagenetImagePreprocess(global_dict, cfg: ImagePreprocessConfig)

Bases: DefaultImagePreprocess

VQGANCodePreprocess

class ofasys.preprocessor.default.image_code.VQGANCodePreprocess(global_dict: Dictionary, cfg: VQGANCodePreprocessConfig)

Bases: SafeBasePreprocess

collate(slots: List[Slot]) → CollateOutput

The collate phase of the preprocessor takes a batch of Slot as input, collate their values and outputs a CollateOutput. This phase defines how to collate a batch for a single modal.

decode(tokens: LongTensor, **kwargs)

dummy_slot(slot)

Set dummy value for slot, which is used for inference.

map(slot: Slot) → Slot

Inputs:

code: (str or List or Tensor) could be:

A string separated by single-whitespaces like 6674 4336 4532 5334… ; Tokens of a numpy or torch Tensor after user-defined preprocess

Returns

1-d int64 torch.Tensor

Return type

Torch.LongTensor

preprocess_image(image, **kwargs)

rerank_with_clip(images, text)

split_str(tokens_str)

VQGANCodePreprocessConfig

class ofasys.preprocessor.default.image_code.VQGANCodePreprocessConfig(_name: Union[str, NoneType] = None, is_active: bool = False, pad_to_multiple: int = 1, code_image_size: int = 256, vqgan_factor: int = 8, code_dict_size: int = 8192, code_entry_prefix: str = 'code', use_encode: bool = True, clip_model: str = 'oss://ofasys/tasks/image_gen/clip/ViT-B-16.pt')

Bases: PreprocessConfig

clip_model: str = 'oss://ofasys/tasks/image_gen/clip/ViT-B-16.pt'

code_dict_size: int = 8192

code_entry_prefix: str = 'code'

code_image_size: int = 256

use_encode: bool = True

vqgan_factor: int = 8

Text

DefaultTextPreprocess

class ofasys.preprocessor.default.text.DefaultTextPreprocess(global_dict: Dictionary, cfg: TextPreprocessConfig, sanity_check=False)

Bases: SafeBasePreprocess

add_prefix(s, prefix)

build_ans2label()

build_bpe(cfg)

build_constraint_trie()

collate(slots: List[Slot]) → CollateOutput

Inputs:

samples: List of Tensors after preprocess

Returns

src_tokens (Tensor): batched tokens with shape [batch, seq_len]

Return type

dict

decode(tokens, escape_unk=False)

dummy_slot(slot)

Set dummy value for slot, which is used for inference.

encode(text)

group_map(slots: List[Slot]) → List[Slot]

The group_map phase of the preprocessor takes a list of Slot as input, processes their values and outputs a list of Slot. This phase defines how to reduce continuous modes in a sample.

map(slot: Slot) → Slot

Inputs:

text: (str or Tensor) could be:

A raw text string Tokens of a numpy or torch Tensor after user-defined preprocess

Returns

1-d int64 torch.Tensor

Return type

Torch.LongTensor

postprocess(outputs, **sample)

prepare_for_generation(closed_set, **kwargs)

remove_prefix(s, prefix)

DefaultTextPreprocessConfig

class ofasys.preprocessor.default.text.TextPreprocessConfig(_name: Union[str, NoneType] = None, is_active: bool = False, pad_to_multiple: int = 1, ans2label: Union[str, NoneType] = None, bpe: ofasys.configure.constants.Choices = 'gpt2', mask_span_distribution: Union[str, NoneType] = 'span-poisson', poisson_lambda: float = 3.0, random_ratio: float = 0.0, replace_length: int = -1, max_src_length: int = 1024, max_tgt_length: int = 1024)

Bases: PreprocessConfig

ans2label: Optional[str] = None

bpe: Choices = 'gpt2'

mask_span_distribution: Optional[str] = 'span-poisson'

max_src_length: int = 1024

max_tgt_length: int = 1024

poisson_lambda: float = 3.0

random_ratio: float = 0.0

replace_length: int = -1

TextForPhonePreprocess

class ofasys.preprocessor.default.text.TextForPhonePreprocess(global_dict: Dictionary, cfg: TextForPhonePreprocessConfig)

Bases: DefaultPhonePreprocess, DefaultTextPreprocess

build_bpe(cfg)

collate(slots: List[Slot]) → CollateOutput

Inputs:

samples: List of Tensors after preprocess

Returns

src_tokens (Tensor): batched tokens with shape [batch, seq_len]

Return type

dict

decode(tokens, escape_unk=False)

dummy_slot(slot)

Set dummy value for slot, which is used for inference.

encode(text)

map(slot: Slot) → Slot

Inputs:

text: (str or Tensor) could be:

A raw text string Tokens of a numpy or torch Tensor after user-defined preprocess

Returns

1-d int64 torch.Tensor

Return type

Torch.LongTensor

TextForPhonePreprocessConfig

class ofasys.preprocessor.default.text.TextForPhonePreprocessConfig(_name: Union[str, NoneType] = None, is_active: bool = False, pad_to_multiple: int = 1, phone_dict_file: str = 'oss://ofasys/tasks/tts/vocab.txt', use_t2p: bool = True, lang: str = 'en', ans2label: Union[str, NoneType] = None, bpe: ofasys.configure.constants.Choices = 'gpt2', mask_span_distribution: Union[str, NoneType] = 'span-poisson', poisson_lambda: float = 3.0, random_ratio: float = 0.0, replace_length: int = -1, max_src_length: int = 1024, max_tgt_length: int = 1024, dict_bpe: Union[str, NoneType] = None)

Bases: TextPreprocessConfig, PhonePreprocessConfig

bpe: Choices = 'gpt2'

dict_bpe: Optional[str] = None

lang: str = 'en'

use_t2p: bool = True

BOX

DefaultBoxPreprocess

class ofasys.preprocessor.default.box.DefaultBoxPreprocess(global_dict: Dictionary, cfg: BoxPreprocessConfig)

Bases: DefaultTextPreprocess

decode(tokens, w_resize_ratio, h_resize_ratio)

encode(region_coord)

group_key(slot: Slot)

group_key returns a key for reducing continuous modal.

instruction_map(ist_data: Instruction) → Instruction

The instruction_map phase of the preprocessor takes the whole Instruction as input and outputs a preprocessed one. This function is mainly used to cooperatively process multiple types of modal inputs.

map(slot: Slot) → Slot

Inputs:

text: (str or Tensor) could be:

A raw text string Tokens of a numpy or torch Tensor after user-defined preprocess

Returns

1-d int64 torch.Tensor

Return type

Torch.LongTensor

postprocess(outputs, **sample)

DefaultBoxPreprocessConfig

class ofasys.preprocessor.default.box.BoxPreprocessConfig(_name: Union[str, NoneType] = None, is_active: bool = False, pad_to_multiple: int = 1, box_dict_size: int = 1000, max_image_size: int = 512, patch_image_size: int = 512, imagenet_default_mean_and_std: bool = False)

Bases: PreprocessConfig

box_dict_size: int = 1000

imagenet_default_mean_and_std: bool = False

max_image_size: int = 512

patch_image_size: int = 512

Phone

DefaultPhonePreprocess

class ofasys.preprocessor.default.phone.DefaultPhonePreprocess(global_dict: Dictionary, cfg: PhonePreprocessConfig)

Bases: SafeBasePreprocess

add_dict_phone_tokens()

collate(slots: List[Slot]) → CollateOutput

Inputs:

samples: List of Tensors after preprocess

Returns

src_tokens (Tensor): batched tokens with shape [batch, seq_len]

Return type

dict

decode(tokens, escape_unk=False)

dummy_slot(slot)

Set dummy value for slot, which is used for inference.

encode(phone_item)

group_key(slot: Slot)

group_key returns a key for reducing continuous modal.

map(slot: Slot) → Slot

The map phase of the preprocessor takes a Slot as input and outputs a preprocessed Slot. This phase defines how to preprocess the raw data of a single modal.

Parameters

inputs (Slot) – raw input data.

Returns

preprocessed data of a single modal

Return type

output (Slot)

postprocess(outputs, **sample)

DefaultPhonePreprocessConfig

class ofasys.preprocessor.default.phone.PhonePreprocessConfig(_name: Union[str, NoneType] = None, is_active: bool = False, pad_to_multiple: int = 1, phone_dict_file: str = 'oss://ofasys/tasks/tts/vocab.txt', use_t2p: bool = False, lang: str = 'zh')

Bases: PreprocessConfig

lang: str = 'zh'

phone_dict_file: str = 'oss://ofasys/tasks/tts/vocab.txt'

use_t2p: bool = False

Audio

DefaultAudioPreprocess

class ofasys.preprocessor.default.audio.DefaultAudioPreprocess(global_dict, cfg: AudioPreprocessConfig)

Bases: SafeBasePreprocess

collate(slots: List[Slot]) → CollateOutput

Inputs:

samples: List of Tensors after preprocess

Returns

src_tokens (Tensor): batched tokens with shape [batch, seq_len]

Return type

dict

decode(feature: Tensor)

Convert frequency domain features to time domain features, i.e., convert fbank features to waveform. This function aims to single input.

dummy_slot(slot)

Set dummy value for slot, which is used for inference.

map(slot: Slot) → Slot

The map phase of the preprocessor takes a Slot as input and outputs a preprocessed Slot. This phase defines how to preprocess the raw data of a single modal.

Parameters

inputs (Slot) – raw input data.

Returns

preprocessed data of a single modal

Return type

output (Slot)

maybe_normalize_waveform(wav)

pack_frames(feature: Tensor, n_frames_per_step=None)

postprocess(outputs, **sample)

prepare_fbank(fbank, split='eval', n_frames_per_step=None)

property vocoder

DefaultAudioPreprocessConfig

class ofasys.preprocessor.default.audio.AudioPreprocessConfig(_name: Union[str, NoneType] = None, is_active: bool = False, pad_to_multiple: int = 1, original_sample_rate: Union[int, NoneType] = None, target_sample_rate: int = 16000, max_seconds: int = 120, input_type: str = 'wave', output_type: str = 'fbank', vocoder: str = 'hifigan', spec_bwd_max_iter: int = 8, speed_augmentation: str = '[1.0]', config_yaml: Union[str, NoneType] = 'oss://ofasys/tasks/asr/config.yaml', output_frame_dim: int = 80, n_frames_per_step: int = 1, normalize: bool = True, random_crop: Union[bool, NoneType] = True, pad_audio: Union[bool, NoneType] = True, normalize_volume: Union[bool, NoneType] = False, win_length: Union[int, NoneType] = 1024, hop_length: Union[int, NoneType] = 256, n_fft: Union[int, NoneType] = 1024, f_min: Union[int, NoneType] = 0, f_max: Union[int, NoneType] = 8000)

Bases: PreprocessConfig

config_yaml: Optional[str] = 'oss://ofasys/tasks/asr/config.yaml'

f_max: Optional[int] = 8000

f_min: Optional[int] = 0

hop_length: Optional[int] = 256

input_type: str = 'wave'

max_seconds: int = 120

n_fft: Optional[int] = 1024

n_frames_per_step: int = 1

normalize: bool = True

normalize_volume: Optional[bool] = False

original_sample_rate: Optional[int] = None

output_frame_dim: int = 80

output_type: str = 'fbank'

pad_audio: Optional[bool] = True

random_crop: Optional[bool] = True

spec_bwd_max_iter: int = 8

speed_augmentation: str = '[1.0]'

target_sample_rate: int = 16000

vocoder: str = 'hifigan'

win_length: Optional[int] = 1024

Motion

Motion6dPreprocess

class ofasys.preprocessor.default.motion_6d.Motion6dPreprocess(global_dict: Dictionary, cfg: Motion6dPreprocessConfig)

Bases: SafeBasePreprocess

batch_decode(slot: Slot, outputs)

build_clamp_fn(slot)

collate(slots: List[Slot]) → CollateOutput

The collate phase of the preprocessor takes a batch of Slot as input, collate their values and outputs a CollateOutput. This phase defines how to collate a batch for a single modal.

custom_reg_loss(slot: Slot, prediction, target, sample_weights)

get_data_dim() → int

map(slot: Slot) → Slot

The map phase of the preprocessor takes a Slot as input and outputs a preprocessed Slot. This phase defines how to preprocess the raw data of a single modal.

Parameters

inputs (Slot) – raw input data.

Returns

preprocessed data of a single modal

Return type

output (Slot)

postprocess(outputs, **sample)

Motion6dPreprocessConfig

class ofasys.preprocessor.default.motion_6d.Motion6dPreprocessConfig(_name: Union[str, NoneType] = None, is_active: bool = False, pad_to_multiple: int = 1, bvh_header: str = 'oss://ofasys/data/human_motion/smplh_bvh_header.bvh', inbetween_args: str = '')

Bases: PreprocessConfig

bvh_header: str = 'oss://ofasys/data/human_motion/smplh_bvh_header.bvh'

inbetween_args: str = ''

Video

DefaultVideoPreprocess

class ofasys.preprocessor.default.video.DefaultVideoPreprocess(global_dict, cfg: VideoPreprocessConfig)

Bases: SafeBasePreprocess

collate(slots: List[Slot]) → CollateOutput

The collate phase of the preprocessor takes a batch of Slot as input, collate their values and outputs a CollateOutput. This phase defines how to collate a batch for a single modal.

map(slot: Slot) → Slot

The map phase of the preprocessor takes a Slot as input and outputs a preprocessed Slot. This phase defines how to preprocess the raw data of a single modal.

Parameters

inputs (Slot) – raw input data.

Returns

preprocessed data of a single modal

Return type

output (Slot)

DefaultVideoPreprocessConfig

class ofasys.preprocessor.default.video.VideoPreprocessConfig(_name: Union[str, NoneType] = None, is_active: bool = False, pad_to_multiple: int = 1, decoding_backend: str = 'pyav', patch_image_size: int = 256, imagenet_default_mean_and_std: bool = True, interpolation: str = 'bicubic', train_jitter_scales_min: int = 256, train_jitter_scales_max: int = 320, train_crop_size: int = 256, test_crop_size: int = 256, train_jitter_scales_relative_min: float = 0.08, train_jitter_scales_relative_max: float = 1.0, train_jitter_aspect_relative_min: float = 0.75, train_jitter_aspect_relative_max: float = 1.3333, num_frames: int = 16, sampling_rate: float = -1, target_fps: int = 30, train_jitter_fps: float = 0.0, train_crop_num_spatial: int = 1, train_aug_num_sample: int = 1, train_auto_augment_type: str = '', train_random_erase_prob: float = 0.25, train_random_erase_mode: str = 'pixel', train_random_erase_count: int = 1)

Bases: PreprocessConfig

decoding_backend: str = 'pyav'

imagenet_default_mean_and_std: bool = True

interpolation: str = 'bicubic'

num_frames: int = 16

patch_image_size: int = 256

sampling_rate: float = -1

target_fps: int = 30

test_crop_size: int = 256

train_aug_num_sample: int = 1

train_auto_augment_type: str = ''

train_crop_num_spatial: int = 1

train_crop_size: int = 256

train_jitter_aspect_relative_max: float = 1.3333

train_jitter_aspect_relative_min: float = 0.75

train_jitter_fps: float = 0.0

train_jitter_scales_max: int = 320

train_jitter_scales_min: int = 256

train_jitter_scales_relative_max: float = 1.0

train_jitter_scales_relative_min: float = 0.08

train_random_erase_count: int = 1

train_random_erase_mode: str = 'pixel'

train_random_erase_prob: float = 0.25