OFASys#

What is OFASys?#

OFASys is a multi-modal multi-task learning system designed to make multi-modal tasks declarative, modular, and task-scalable. With OFASys, it is easy to:

Rapidly introduce new multi-modal tasks/datasets by defining a declarative one-line instruction.

Develop new or reuse existing modality-specific components.

Jointly train multiple multi-modal tasks together without manual processing of multi-modal data collating.

This system aims to allow users to rapidly deploy the model across customized datasets/tasks/modalities, and provide engineers and researchers with a solution for training one single model/checkpoint to process multiple (multi-modal) tasks at the near-SOTA level simultaneously.

What does OFASys have?#

For now, OFASys supports seven modalities. Including: seven modalities: TEXT, IMAGE, VIDEO, AUDIO, MOTION, BOX, STRUCT

OFASys supports more than 20 classes of multi-modal tasks, including:

Modality	Task	Dataset	Metrics	OFA+ (Specialist)	OFA+ (Generalist)	OFA+ (Generalist MoE)
Text	NLU	GLUE	Avg Score ↑	83.1*	-	-
	Multiple Tasks	Natural instruction v2	ROUGE-L ↑	30.49	26.97	27.74
	Text Summarization	Gigaword	ROUGE-L ↑	34.24	34.68	33.95
	Text Infilling^p	Pile/ Wikicorpus/ Bookcorpus	-	-	-	-
Image	Image Classification	ILSVRC	top1 acc ↑	83.31	72.56	78.95
	Visual Entailment	SnliVE	Acc ↑	88.88	85.84	86.18
	Image Captioning	MsCoco	Cider ↑	134.8	122.6	125.2
	Visual Question Answering	VQA-v2	VQA score ↑	78.72	68.86	72.27
	Text-to-Image Generation	COCO	clip_ti ↑	0.317	0.289	0.294
	Image Infilling ^p	-	-	-	-	-
BOX	Visual Grounding	Refcoco	Acc @ 0.5 ↑	88.12	80.08	83.06
	Grounded Caption ^p	-	-	-	-	-
	Object Detection ^p	-	-	-	-	-
Video	Video Classification	kinetics400	Acc ↑	74.30	64.58	69.47
	Video Captioning	MSR-VTT	Cider ↑	70.80	59.10	63.00
	Video Question Answering	MSR-VTT QA	VQA score ↑	42.10	41.73	40.00
Audio	Automatic Speech Recognition	LibriSpeech	WER ↓	7.5	8.5	8.1
Audio	Text to Speech	-	mcd loss ↓	1.187	1.443	1.429
Structural Language	Text-to-SQL Generation	Spider	Exact Match ↑	45.70	39.20	40.50
	Table-to-Text Generation	Dart	BLEU ↑	51.24	50.86	50.88
	Table Question Answering	Fetaqa	BLEU ↑	31.56*	-	-
	Sudoku	-	Solved Acc ↑	99.8*	-	-
Motion	Text-to-Motion Generation	AMASS/KIT /AIST++	-	-	-	-

Tasks with ^p are used in pretraining only.

Scores with * are finetuned with a large model.

Here, GLUE Benchmark contains seven tasks including COLA, MNLI, MRPC, QNLI, QQP, RTE and SST2.

You can finetune the above individual tasks to achieve some sota results reported in the OFA paper by following Installation and Usage in 15 minutes, or you are free to arbitrarily combine these tasks for larger-scale joint pre-training. Besides, you can also add new tasks or even new modalities by extending the base classes provided by OFASys.

Contents#

The documentation is organized into five sections:

GET STARTED provides a quick tour of the library and installation instructions to get up and running.
HOW-TO GUIDES show you how to achieve a specific goal, like how to add a new task or how to write a custom module.
CONCEPTUAL GUIDES offer more discussion and explanation of the underlying concepts and the design philosophy of OFASys.
Task Gallery lists all supported tasks.
API describes all classes and functions.