Model Database's logo

Tokenizers documentation

Visualizer

Model Database's logo
Join the Model Database community

and get access to the augmented documentation experience

to get started

Visualizer

Python
Rust
Node

Annotation

class tokenizers.tools.Annotation

< >

( start: int end: int label: str )

EncodingVisualizer

class tokenizers.tools.EncodingVisualizer

< >

( tokenizer: Tokenizer default_to_notebook: bool = True annotation_converter: typing.Union[typing.Callable[[typing.Any], tokenizers.tools.visualizer.Annotation], NoneType] = None )

Parameters

  • tokenizer (Tokenizer) — A tokenizer instance
  • default_to_notebook (bool) — Whether to render html output in a notebook by default
  • annotation_converter (Callable, optional) — An optional (lambda) function that takes an annotation in any format and returns an Annotation object

Build an EncodingVisualizer

__call__

< >

( text: str annotations: typing.List[tokenizers.tools.visualizer.Annotation] = [] default_to_notebook: typing.Optional[bool] = None )

Parameters

  • text (str) — The text to tokenize
  • annotations (List[Annotation], optional) — An optional list of annotations of the text. The can either be an annotation class or anything else if you instantiated the visualizer with a converter function
  • default_to_notebook (bool, optional, defaults to False) — If True, will render the html in a notebook. Otherwise returns an html string.

Build a visualization of the given text

←Decoders