Module piston_meta::tokenizer [] [src]

Tracking tokenizer state.

Structs

TokenizerState

Stores the number of tokens received.

Functions

read_data

Reads meta data.