Video-Related Tasks#

Video Classification#

Task Introduction#

The video classification task is a fundamental task in the field of video understanding where the model needs to predict the label for a given video clip. We evaluate our model on the Kinetics-400 dataset, which contains ~300k video clips from 400 classes. We report the accuracy on the val split of the Kinetics-400 dataset.

Default Template#

[VIDEO:video] what is in the video? -> [TEXT:label_name,closed_set]

Usage#

>>> instruction = '[VIDEO:video] what is in the video? -> [TEXT:label_name,closed_set]'
>>> data = {'video': 'oss://ofasys/datasets/kinetics_data/k400.256p/test/-7aeB7vFtB4_000037_000047.mp4'}
>>> output = model.inference(instruction, data=data)
>>> print(output.text)

CASE#

input:

output:

"playing the piano"

Video Captioning#

Task Introduction#

The video captioning task requires the model to generate a textual description for a given video clip. We evaluate the proposed method on MSR-VTT caption dataset, which contains 10K video clips 200K descriptions of the videos. Following, We report CIDEr scores on the val split of the MSR-VTT dataset.

Default Template#

[VIDEO:video] what does the video describe? -> [TEXT:cap]

Usage#

>>> instruction = '[VIDEO:video] what does the video describe? -> [TEXT:cap]'
>>> data = {'video': 'oss://ofasys/datasets/msrvtt_data/videos/video7030.mp4'}
>>> output = model.inference(instruction, data=data)
>>> print(output.text)

CASE#

input:

output:

"a group of people are dancing"

Video Question Answering#

Task Introduction#

The video captioning task requires the model to generate a answer for a given video clip and a question related to that video clip. We evaluate the proposed method on MSR-VTT QA dataset, which contains question-answer pairs extracted from the original MSR-VTT dataset. We report the accuracy on the val split of MSR-VTT QA dataset.

Default Template#

[VIDEO:video] [TEXT:question] -> [TEXT:answer,is_label]

Usage#

>>> instruction = '[VIDEO:video] [TEXT:question] -> [TEXT:answer,is_label]'
>>> data = {'video': 'oss://ofasys/datasets/msrvtt_data/videos/video9585.mp4', 'question': 'what is a person decorating?'}
>>> output = model.inference(instruction, data=data)
>>> print(output.text)

CASE#

input:

'what is a person decorating?'

output:

"a person is decorating a cake"