Figure 5
How a number becomes one token
A language model answers with tokens, not numbers. So we turn a screen position into a choice between 256 tokens, and read the model's whole distribution over them in one pass.
1
Figure 5
How a number becomes one token
A language model answers with tokens, not numbers. So we turn a screen position into a choice between 256 tokens, and read the model's whole distribution over them in one pass.