Section 124 · Chapter 16, Introspection: White-Box Testing Networks
Input Token Tables
Before builders inspect attention or activations, they need to see exactly what the model received as tokens.
tokenizationattentionactivationinput token tables
What to do
- Define runnable checks that exercise tokenization, attention, and activation.
- Set acceptable outcomes and blocker failures for tokenization, attention, and activation before running the evaluation.
- Run representative cases for tokenization, attention, and activation and preserve the failures that would change the decision.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for tokenization, attention, activation, input token tables needed to reproduce work on Input Token Tables.
- Report results for tokenization, attention, activation, input token tables by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Token tables should include tokenizer version, position, token id, byte span, original text span, and any preprocessing. This matters for multilingual text, code, identifiers, numbers, names, and pasted documents.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Input Token Tables." Testing AI Knowledge Edition, section 124.
https://jarbon.ai/testing-ai/knowledge/ch124-input-token-tables.html