papyri.tokens

Tokenisation helpers for papyri gen.

Two unrelated tokenisers live here, both used to turn user-authored Python snippets into structured form:

• Jedi-based identifier inference for example blocks (parse_script). Each identifier in the snippet is enriched with its fully-qualified name when Jedi can infer one.

• Pygments token-class extraction for syntax-highlighting hints (get_classes / _add_classes).

A small on-disk cache under ~/.cache/papyri/jedi/ keyed by content hash + day stops Jedi inference from re-running on every gen build.