Self-play vs. text pretraining:
what kind of in-context learning
does each produce?

Same architecture. Three prompt formats.

The learner is a byte-level Llama-style transformer with a 4,096-byte context. Each task gives worked examples, then a query. Evaluation uses greedy decoding with no fine-tuning.

Printable-text and word answers require every byte to be correct. The raw reverse task uses per-byte accuracy, following the original paper. Suite scores average fixed task cells at m = 32–128; the word headline excludes word-order reversal. The explorer still includes the zero-scoring tasks.

cat=t
mouth=b
rabbit=

Read the uncertainty with the result.

Self-play has four released seeds. Bands and whiskers are 95% Student’s t intervals over seeds, clipped to [0, 1] for display. Text has two seeds at 1M, 3M and 6M; one at the other sizes. Text runs appear individually, without a seed interval. Exact, unclipped bounds are in the tables.

Warm starts use one seed and begin at self-play round 8191 (51.54B prior learner tokens). Additional text tokens exclude that cost. The universal prior uses different program counts per round and has no released 24M checkpoint.

These are small-model observations across a narrow task set. The size trends are not established scaling laws, and the token-matched comparison does not establish a general advantage per unit of total compute.