Instruction

Instruction

class ofasys.preprocessor.instruction.Instruction(template: str, split: str = 'train', decoder_plain_with_loss: bool = False)

The instruction’s template should format as “… [MODE] … -> … [MODE] …”, where MODE should be one of ModalityType and “…” could contains more [MODE].

For example, the instruction of image caption could be:

• Illustration 1. Image Captioning:

  [IMAGE:img] what does the image describe? -> [TEXT:cap]
  

• Illustration 2. MNLI Task in Glue Benchmark:

  can text1 [TEXT:sent1] imply text2 [TEXT:sent2]? -> [TEXT:label,closed_set]
  
  # Or we can use the prompt tuning which prepends some text prompts to decoder.
  can text1 [TEXT:sent1] imply text2 [TEXT:sent2]? ->  can text1 [TEXT:sent1,no_loss] imply text2 [TEXT:sent2,no_loss]? [TEXT:label,closed_set]
  

• Illustration 3. Object Detection Task with variable-length output

  [IMAGE:img] detect the objects in the image. -> [[BOUNDING_BOX] [TEXT]]*
  

• Illustration 4. Interleaved Image Text context with variable-length pairs:

  -> ([IMAGE] [TEXT])*
  

Parameters

• template – instruction template string.

• split – data split: train, valid, or test.

• decoder_plain_with_loss – whether compute loss (for decoder)

format(*args, **kwargs)

Fill template with input data. The formatted instruction can be used for model inference.

Usage:

>>> model = OFASys.from_pretrain('OFASys.ckpt')
>>> sample = Instruction(
...     "[IMAGE] what does the region describe in the image? region: [BOUNDING_BOX] -> [TEXT]"
... ).format(
...     image_data, box_data
... )
>>> text = model.inference(sample)

get_slot_names() → List[str]

Slot

class ofasys.preprocessor.instruction.Slot(modality: ModalityType, is_src: bool, value: Optional[Any], global_position: Optional[int] = None, column_name: Optional[str] = None, attributes: Optional[List[str]] = None, preprocess: Optional[str] = None, is_plaintext: bool = False, split: str = 'train', decoder_plain_with_loss: bool = False)

Slot is the core concept of the multi-modal abstraction in OFASys. Each slot contains only one modality data that spans consecutive positions. A Slot is described by modality type, reference Name as well as several arguments for training or inference, marked as attr. Given different positions appeared in the instruction, we denote the slot appears in the encoder and decoder sentence by E-slot and D-slot, respectively.

attr2kwargs()

attributes: Optional[List[str]] = None

column_name: Optional[str] = None

decoder_plain_with_loss: bool = False

get_attr(attr_key: str, class_factory: Optional[type] = None) → Optional[Any]

static get_target_slot_from_sample(sample: Dict)

static get_target_slot_from_slots(slots: List)

global_position: Optional[int] = None

has_attr(attr_key: str) → bool

is_plaintext: bool = False

is_src: bool

modality: ModalityType

pin_memory()

preprocess: Optional[str] = None

split: str = 'train'

value: Optional[Any]