Utils

Dictionary

class ofasys.preprocessor.dictionary.Dictionary(*, bos='<s>', pad='<pad>', eos='</s>', unk='<unk>', extra_special_symbols=None)

A mapping from symbols to consecutive integers

add_from_file(f, prefix=None, check=True)

Loads a pre-existing dictionary from a text file and adds its symbols to this instance.

add_symbol(word, n=1, overwrite=False, check=True)

Adds a word to the dictionary

bos()

Helper to get index of beginning-of-sentence symbol

eos()

Helper to get index of end-of-sentence symbol

finalize(threshold=-1, nwords=-1, padding_factor=8)

Sort symbols by frequency in descending order, ignoring special ones.

Parameters

• count (- threshold defines the minimum word) –

• dictionary (- nwords defines the total number of words in the final) –

• symbols (including special) –

• 8 (- padding_factor can be used to pad the dictionary size to be a multiple of) –

• hardware (which is important on some) –

index(sym)

Returns the index of the specified symbol

classmethod load(f)

Loads the dictionary from a text file with the format:

` <symbol0> <count0> <symbol1> <count1> ... `

pad()

Helper to get index of pad symbol

pad_to_multiple_(padding_factor)

Pad Dictionary size to be a multiple of padding_factor.

save(f)

Stores dictionary into a text file

string(tensor, bpe_symbol=None, escape_unk=False, extra_symbols_to_ignore=None, unk_string=None, include_eos=False, separator=' ')

Helper for converting a tensor of token indices to a string.

Can optionally remove BPE symbols or escape <unk> words.

unk()

Helper to get index of unk symbol

unk_string(escape=False)

Return unknown string, optionally escaped as: <<unk>>

update(new_dict)

Updates counts from new dictionary.