AI/ML / Software Engineering / Databases & Data

Train a small language model from scratch with licensed data

Build an educational causal language model from randomly initialized weights using a licensed, provenance-tracked corpus. Train a dedicated tokenizer, define a roughly 34-million-parameter decoder architecture, establish random and unigram baselines, checkpoint and resume one bounded epoch, evaluate held-out loss and qualitative limitations, run local inference, and publish only an honest research artifact.

600 min16 stepsChanges system stateRevision 1

Save or explore

#language-model #training-from-scratch #tokenizer #causal-lm #pytorch #transformers #datasets #checkpoint #provenance #evaluation

0 of 16 steps completed

Goal

Produce a reproducible small-model training artifact that demonstrates the complete data-to-inference lifecycle without claiming frontier capability, broad knowledge, production safety, or economic competitiveness.

Supported environments

Ubuntu 24.04 LTS
Python 3.12
PyTorch 2.13.0
Transformers 5.14.1
Datasets 5.0.1
Tokenizers 0.23.1

Prerequisites

Licensed corpus with provenance Use only text for which training and derivative-model use are explicitly approved. Record every source, license or authority, immutable revision, digest, language, collection purpose, exclusions, personal-data handling, retention, and redistribution decision.python3 scripts/audit_corpus.py
Educational scope Accept that a roughly 34M-parameter model trained on approximately 50M carefully licensed tokens is useful for learning the lifecycle and narrow experiments, not as a factual assistant, safety-critical system, or substitute for frontier-model training.
Measured compute and storage The example targets one NVIDIA GPU with 16–24 GiB VRAM, 32 GiB host RAM, and 80 GiB free disk. Run a 30-minute pilot, then approve an estimated 2–10 GPU-hours for one epoch depending on hardware and input throughput. CPU-only execution may take substantially longer.nvidia-smi --query-gpu=name,driver_version,memory.total,memory.free --format=csv,noheader && df -h .
Independent evaluation owner Freeze validation and test documents before tokenizer or model training, define held-out loss/perplexity, tokenizer coverage, memorization canaries, qualitative prompts, safety limitations, resource budget, publication policy, and rollback owner.

Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System

A fully traceable educational causal language model from randomly initialized weights, with its own frozen byte-level BPE tokenizer and a bounded roughly 34M-parameter decoder.
A licensed corpus pipeline that preserves per-document source and license identity, split isolation, exact and near-duplicate evidence, privacy/security review, immutable digests, and source-removal mapping.
A one-epoch training and recovery lifecycle with random and unigram baselines, retained-token accounting, cost pilot, complete checkpoints, held-out evaluation, bounded inference, honest model card, cleanup, and ninety-day review.

Observable outcome

Explain and reproduce how raw licensed documents become tokenizer IDs, causal blocks, random model weights, gradient updates, checkpoints, final likelihood metrics, and bounded samples.
Demonstrate lower held-out next-token loss than random and unigram references without converting that result into unsupported claims about truth, reasoning, safety, instruction following, or production value.
Stop or delete an unsafe or legally uncertain artifact while retaining enough lineage to audit cost, failures, and source withdrawal.

Architecture

How the parts fit together

The experiment has a governance plane, immutable document corpus, train-only tokenizer, random decoder, causal training loop, complete checkpoint store, independent evaluator, restricted artifact store, and no production serving plane.

Licensed corpus and source mapPreserves immutable documents, source/license authority, split membership, digests, privacy controls, exclusions, retention, and removal propagation.

Frozen tokenizerLearns a 16k byte-level BPE from train only and fixes normalization, special IDs, encode/decode, vocabulary, efficiency, and model embeddings.

Random small decoderInitializes about 34M GPT-2-style causal parameters with 512-token context and no pretrained knowledge or instruction alignment.

Trainer and checkpoint storeGroups causal blocks, performs one bounded epoch, validates, saves full recovery state, retains three checkpoints, and records cost and final weights.

Evaluation and artifact boundaryCompares random, unigram, and final likelihood; tests memorization and samples; packages honest research metadata; prohibits production traffic.

Governance approves source-level rights, educational scope, budgets, prohibited uses, evaluation, publication, and removal owners.
Documents are split by source ID, audited, frozen, and used to train a tokenizer from train only; tokenizer behavior is accepted on validation.
Architecture is initialized randomly, baselines are measured, and a bounded pilot validates memory, throughput, checkpoint, storage, and cost.
One epoch trains causal next-token prediction with complete checkpoints and tested resume; untouched test evaluates likelihood and limitations.
Only an approved research artifact is retained, no production endpoint remains, and cleanup preserves source-removal and audit evidence.

Assumptions

The accepted corpus contains around 50M tokens after the frozen tokenizer, and source-level rights permit educational training and retention of derivative weights under the documented internal scope.
One supported NVIDIA GPU with 16–24 GiB VRAM can train the roughly 34M-parameter model with effective batch 32 and context 512 after a successful measured pilot.
Validation and test documents are split before tokenizer training or chunking and remain unavailable to model or tokenizer fitting and experiment choice.
Held-out perplexity is interpreted only within the same tokenizer and corpus, and is supplemented by source/language segmentation, memorization probes, qualitative fixtures, and explicit limitations.
The artifact remains research-only and does not receive production, high-stakes, confidential, tool-enabled, or autonomous input.

Key concepts

Training from scratch: Initializing model weights from a random distribution and optimizing them on the approved corpus rather than loading pretrained model weights. Software, architecture, and data still have external provenance.
Causal language modeling: Learning to predict each next token from previous tokens. Lower objective loss does not inherently provide instruction alignment, factual grounding, or safety.
Tokenizer vocabulary: A frozen mapping and pipeline that converts normalized text into token IDs and back. It defines embedding size, sequence efficiency, language coverage, and checkpoint compatibility.
Perplexity: The exponential of mean token negative log-likelihood. It is meaningful only with the same tokenizer, corpus, segmentation, and implementation and does not directly measure usefulness.
Random baseline: Evaluation of the exact saved initial weights before updates, proving how much the training lineage changes next-token likelihood.
Unigram baseline: A non-contextual predictor based on training token frequencies, showing whether the model exceeds simple corpus statistics.
Checkpoint: A complete recovery point containing model, optimizer, scheduler, scaler, RNG, trainer, sampler, tokenizer/config identity, and manifest needed to continue a run.
Frontier-scale training: A fundamentally larger program involving vast governed datasets, billions or more parameters, distributed accelerators, extensive systems optimization, alignment, evaluations, safety, reliability, and major cost. This guide is not that program.

Security and production boundaries

A scratch model can memorize and reveal corpus passages. Source rights, personal-data minimization, secret scanning, access, retention, removal mapping, and extraction tests are required even for an educational run.
Checkpoints and caches are derived sensitive artifacts. Protect them like the corpus, and do not assume safetensors prevents information disclosure; it only avoids pickle execution for tensor loading.
Generated text is untrusted and unauthoritative. It must never execute commands, operate tools, approve changes, or provide high-stakes advice.
No listener should remain after the lab. Any future API requires separate authenticated TLS ingress, authorization, limits, monitoring, incident response, data policy, and production evaluation.

Stop before continuing if

Stop if any source lacks training or derivative permission, immutable identity, personal-data basis, security review, or removal path.
Stop if validation/test leaks into tokenizer or model fitting, duplicates cross splits, or accepted corpus bytes change.
Stop on OOM, NaN, fallback, thermal instability, partial checkpoint, storage pressure, unbounded duration, or compute budget breach.
Stop publication on memorization, secret or personal-data reproduction, misleading capability claims, unsafe sample, unreviewed destination, or hidden failure.
Stop closure if any endpoint, credential, copy, recovery need, source-removal mapping, legal hold, or audit requirement remains unresolved.

warning

Define what this small-model experiment can and cannot prove

read-only

Write the scope before collecting data. The finished artifact demonstrates tokenizer training, decoder initialization, next-token learning, checkpointing, evaluation, and inference on a bounded licensed corpus. It does not reproduce the data scale, parameter count, distributed infrastructure, alignment, evaluation breadth, security, reliability, or cost of a frontier language model.

Why this step matters

Clear limits prevent a technically successful training run from becoming a misleading product claim. Scale changes capabilities, data behavior, optimization, failure modes, and infrastructure; the difference is not merely waiting longer.

What to understand

Declare the intended learning outcomes: inspect tokenization, observe loss reduction, resume a checkpoint, compare random and trained perplexity, generate bounded samples, and document failures.

Prohibit factual advice, security decisions, medical or legal use, autonomous tools, user-facing production traffic, and claims of broad language competence. A small model may produce fluent-looking false text.

Cap corpus tokens, model parameters, context, epochs, GPU-hours, storage, and number of experimental variants. The experiment should stop rather than grow into an unreviewed compute project.

Record the run ID, UTC time, code commit, package lock digest, data and model revisions, hardware topology, seed, effective batch, precision, and command before accepting evidence. A directory name or screenshot is not enough to reproduce a training claim.

Keep training, validation, and test roles distinct. Test data must remain unavailable to hyperparameter selection, early stopping, prompt tuning, checkpoint choice, and qualitative iteration; otherwise the final score becomes another validation measurement.

Treat every output as evidence scoped to this exact artifact and environment. Lower loss does not prove factuality, safety, domain correctness, general reasoning, or production readiness, and a successful command does not waive licensing or privacy review.

System changes

Creates an experiment charter; it does not collect data, allocate compute, or authorize publication.

Syntax explained

approximately 34M parameters: A tractable decoder size for one-GPU education, not a competitive general-purpose LLM.
50M-token ceiling: Bounds data and one-epoch cost; actual accepted tokens are measured after tokenization.
no production use: A hard scope gate that cannot be offset by lower test loss or appealing samples.

Example output / evidence

Scope approved: educational 34M-parameter causal LM, 50M-token ceiling, one epoch, no production or safety-critical use.

Checkpoint: Approve limitations and budget

git diff -- governance/scratch-training-charter.md

Continue whenThe charter names educational outcomes, prohibited claims and uses, token/parameter/context/epoch/GPU/storage ceilings, reviewers, and 2026-10-27 expiry.

Stop whenStop if stakeholders expect frontier behavior, production safety, unbounded data or compute, or cannot accept an explicitly limited research artifact.

If this step fails

The candidate passes aggregate metrics but fails a safety or policy case.

Likely causeAverages hide a critical regression, the gate was treated as weighted, or evaluation omitted a required class.

Safe checks

Inspect per-case results and hard-gate status
Re-run the failing case with saved prompt and artifact identity
Review whether the test is valid and independent of training data

ResolutionReject promotion. Hard safety, privacy, licensing, and policy gates cannot be offset by lower loss or better average task score.

Security notes

Treat generated fluency as untrusted output. The model has no factual authority, tool permission, or security boundary.
Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Fine-tune or prompt an existing approved base when the goal is useful application quality rather than learning pretraining mechanics.

Stop conditions

Stop if stakeholders expect frontier behavior, production safety, unbounded data or compute, or cannot accept an explicitly limited research artifact.

config

Create a source-level license and provenance manifest

read-only

List every corpus source before copying it into training storage. Record exact source revision and digest, license or organization authority, whether model training and derivative weights are permitted, attribution, redistribution, personal-data basis, geographic or contractual restrictions, collection process, quality limitations, removal contact, and downstream deletion obligations.

Why this step matters

Pretraining consumes the corpus broadly and can memorize passages. Source-level provenance is necessary to decide permission, respond to removal requests, investigate outputs, and state honest limitations.

What to understand

Do not assume a website, repository, feed, model output, or public-domain label grants the exact rights needed. Preserve license text and obtain governance review for ambiguous, mixed, or jurisdiction-dependent sources.

If generated text enters the corpus, record generator, prompt-data provenance, output terms, filtering, and human review. Synthetic text can reproduce copyrighted or sensitive inputs and can amplify model errors.

Keep source IDs through every normalized document and split. A global corpus digest alone cannot identify which downstream checkpoints are affected by a source withdrawal.

System changes

Creates source governance metadata; it does not ingest or transform text.

Syntax explained

training_permission: Records explicit permission for model training rather than inferring it from access.
derivative_weight_permission: Separates internal processing from permission to retain or distribute learned weights.
removal_owner: Names who evaluates and propagates a source withdrawal through derived artifacts.

File governance/corpus-provenance.json

Configuration

{
  "schema_version": 1,
  "corpus_id": "licensed-small-clm-corpus",
  "revision": "2026-07-29.1",
  "intended_use": "educational causal language model training",
  "redistribution": false,
  "sources": [
    {
      "source_id": "organization-owned-technical-prose",
      "revision": "commit-or-export-id",
      "content_sha256": "replace",
      "license_or_authority": "organization-owned and approved",
      "training_permission": true,
      "derivative_weight_permission": true,
      "personal_data": "prohibited",
      "secrets": "prohibited",
      "removal_owner": "data-governance"
    }
  ],
  "split_policy": "stable document_id before tokenization and chunking",
  "near_duplicate_threshold": 0.92,
  "review_due": "2026-10-27"
}

Example output / evidence

Corpus manifest approved: 1 owned source, training and derivative weights permitted, redistribution false, removal owner assigned.

Checkpoint: Approve every source

python3 -m json.tool governance/corpus-provenance.json >/dev/null && sha256sum governance/corpus-provenance.json

Continue whenEvery byte source has immutable identity, permission, data classification, attribution, redistribution, removal, and review evidence.

Stop whenStop if any source lacks permission, revision, digest, personal-data decision, removal path, or compatible terms.

If this step fails

The dataset license or source permission cannot be proven.

Likely causeThe dataset card is incomplete, a mixed corpus lost per-source terms, redistribution differs from training permission, or an internal source lacks recorded authority.

Safe checks

Inspect the provenance manifest and source-specific license text
Confirm dataset revision and content digest
Escalate ambiguous terms to the appropriate legal or governance owner

ResolutionExclude the source and rebuild the corpus. Do not infer permission from public accessibility, a repository host, or another model's use of the same text.

A model hub upload exposes a private artifact.

Likely causeRepository visibility, token scope, push settings, generated card, logs, dataset samples, or checkpoint contents were not reviewed before upload.

Safe checks

Stop the upload and inspect repository visibility
Revoke the token and audit uploaded revisions
Check model, tokenizer, adapter, card, logs, and dataset references

ResolutionQuarantine or remove the remote revision according to provider and incident policy, rotate credentials, and require an explicit publication gate instead of automatic push.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Use a smaller organization-authored corpus with clear authority; reduced scale is preferable to unprovable rights.

Stop conditions

Stop if any source lacks permission, revision, digest, personal-data decision, removal path, or compatible terms.

config

Pin an isolated tokenizer and model-training environment

caution

Use the same exact package lock as the fine-tuning lab but omit unused PEFT libraries in a hardened production build. Record Python, PyTorch, Transformers, Datasets, Tokenizers, Accelerate, safetensors, OS, driver, CUDA, GPU UUID, CPU, RAM, storage, and code commit.

Why this step matters

Tokenizer algorithms, model implementations, optimizer behavior, checkpoint formats, and kernel determinism can change across versions. A locked environment is part of the model artifact.

What to understand

Install through a trusted hashed build process and create a fresh environment. Do not upgrade a shared notebook or production inference service in place.

Run a tiny tensor operation and save/load smoke before the corpus is mounted. Verify bfloat16 support and intended accelerator rather than relying on package installation success.

Disable automatic hub upload and unapproved tracking. This guide writes local artifacts and requires a separate publication decision.

System changes

Creates a dedicated executable Python environment and local package cache.

Syntax explained

PyTorch 2.13.0: Pins the tensor, optimizer, and checkpoint implementation used by this guide revision.
Tokenizers 0.23.1: Pins BPE normalization, pre-tokenization, training, serialization, and decode behavior.
safetensors 0.8.0: Provides non-pickle tensor serialization for model weight artifacts.

File requirements-scratch.txt

Configuration

--index-url https://pypi.org/simple
torch==2.13.0
transformers==5.14.1
datasets==5.0.1
tokenizers==0.23.1
accelerate==1.14.0
peft==0.20.0
trl==1.9.2
bitsandbytes==0.50.0
safetensors==0.8.0

Example output / evidence

Environment locked: Python 3.12, torch 2.13.0, transformers 5.14.1, datasets 5.0.1, tokenizers 0.23.1, NVIDIA RTX 4090.

Checkpoint: Verify environment and device

python -c "import torch,transformers,datasets,tokenizers; print(torch.__version__, transformers.__version__, datasets.__version__, tokenizers.__version__, torch.cuda.get_device_name(0), torch.cuda.is_bf16_supported())"

Continue whenPinned versions match, the approved accelerator is selected, bfloat16 support is explicit, and no unrelated GPU process consumes the budget.

Stop whenStop if packages or wheels are unpinned, the device falls back, bfloat16 is unsupported without an approved precision change, or another workload shares the GPU unpredictably.

If this step fails

The GPU is visible but unsupported kernels or CPU fallback are used.

Likely causeDriver, CUDA, compute capability, package wheel, dtype, bitsandbytes backend, or attention implementation is incompatible.

Safe checks

Record torch and CUDA versions and device capability
Inspect runtime logs for backend and kernel selection
Run a tiny matrix operation and memory-footprint probe

ResolutionUse a documented compatible stack, rebuild the isolated environment, and rerun the measured pilot. Do not publish throughput from an unintended fallback.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Use float32 on a much smaller model for CPU education, with a new capacity estimate and explicit non-comparability.

Stop conditions

Stop if packages or wheels are unpinned, the device falls back, bfloat16 is unsupported without an approved precision change, or another workload shares the GPU unpredictably.

config

Audit immutable documents and splits before tokenizer training

read-only

Save the corpus audit, split by stable document ID before normalization or chunking, and run exact plus reviewed near-duplicate checks. Validate schema, source/license IDs, bounded documents, obvious secret patterns, split isolation, language and source distribution, quality samples, personal data, prohibited content, and per-split digests.

Why this step matters

A tokenizer learns from its input, so validation and test text must not enter tokenizer training. Document-level splitting and auditing protect both evaluation validity and source governance.

What to understand

The script catches exact normalized duplicates and simple secrets, not semantic duplicates, personal data, toxicity, copyright, factual quality, or all credentials. Attach separate reviewed evidence for those risks.

Measure accepted characters, bytes, languages, source proportions, document length, and later token counts. A nominal 50M-token plan cannot be known until the accepted tokenizer processes the frozen corpus.

Quarantine failures and create a new corpus revision. Do not silently redact accepted files in place, because tokenizer and model lineage depend on exact bytes.

System changes

Reads corpus JSONL and emits audit evidence; it never rewrites source documents.

Syntax explained

document_id: Stable split unit that keeps related text in one partition.
source_id and license_id: Preserve per-document lineage to the approved provenance record.
2 MB cap: Bounds individual documents for this educational pipeline; larger sources require reviewed segmentation.

File scripts/audit_corpus.py

Configuration

from __future__ import annotations

import hashlib
import json
import re
from pathlib import Path

FILES = [Path("corpus/train.jsonl"), Path("corpus/validation.jsonl"), Path("corpus/test.jsonl")]
REQUIRED = {"document_id", "text", "source_id", "license_id"}
patterns = [
    re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----"),
    re.compile(r"\bsk-[A-Za-z0-9_-]{20,}\b"),
    re.compile(r"\b(?:AKIA|ASIA)[A-Z0-9]{16}\b"),
]
documents: dict[str, str] = {}
content: dict[str, str] = {}
report = {}

for path in FILES:
    rows = 0
    chars = 0
    digest = hashlib.sha256()
    for number, raw in enumerate(path.read_bytes().splitlines(keepends=True), 1):
        digest.update(raw)
        row = json.loads(raw)
        missing = REQUIRED - row.keys()
        if missing:
            raise SystemExit(f"{path}:{number}: missing {sorted(missing)}")
        if not all(isinstance(row[key], str) and row[key].strip() for key in REQUIRED):
            raise SystemExit(f"{path}:{number}: required values must be non-empty strings")
        text = row["text"]
        if len(text.encode()) > 2_000_000:
            raise SystemExit(f"{path}:{number}: document exceeds 2 MB")
        if any(pattern.search(text) for pattern in patterns):
            raise SystemExit(f"{path}:{number}: secret-like pattern")
        previous = documents.setdefault(row["document_id"], path.name)
        if previous != path.name:
            raise SystemExit(f"document {row['document_id']} crosses {previous} and {path.name}")
        normalized = " ".join(text.split()).casefold()
        text_hash = hashlib.sha256(normalized.encode()).hexdigest()
        duplicate = content.setdefault(text_hash, f"{path}:{number}")
        if duplicate != f"{path}:{number}":
            raise SystemExit(f"{path}:{number}: exact normalized duplicate of {duplicate}")
        rows += 1
        chars += len(text)
    report[path.name] = {"documents": rows, "characters": chars, "sha256": digest.hexdigest()}

print(json.dumps({"status": "passed", "splits": report}, indent=2, sort_keys=True))

Example output / evidence

{"status":"passed","splits":{"train.jsonl":{"documents":18500,"characters":194200000,"sha256":"61af…"},"validation.jsonl":{"documents":1000,"characters":10300000,"sha256":"1ba2…"},"test.jsonl":{"documents":1000,"characters":10500000,"sha256":"73dd…"}}}

Checkpoint: Freeze the audited corpus revision

python3 scripts/audit_corpus.py | tee governance/corpus-audit.json

Continue whenSchema, source authority, splits, exact and near duplicates, secret/privacy review, distributions, quality sampling, and digests are accepted before tokenizer training.

Stop whenStop on cross-split document, duplicate, secret, personal data, unknown source/license, prohibited content, unexplained distribution, or mutable input.

If this step fails

The dataset audit finds personal data, secrets, or credentials.

Likely causeCollection, export, issue text, logs, or generated examples entered the corpus without adequate minimization and scanning.

Safe checks

Quarantine the affected shard and record its digest
Identify source, scope, access, and downstream copies
Search tokenizer, caches, checkpoints, and artifacts according to incident policy

ResolutionStop training, follow privacy or security incident procedure, remove the data from a new versioned corpus, invalidate derived artifacts as policy requires, and never rewrite the original lineage.

Tokenized splits contain duplicates or overlapping documents.

Likely causeSplitting happened after chunking, document IDs were unstable, normalization collapsed examples, or near-duplicate detection was omitted.

Safe checks

Recompute exact hashes before tokenization
Check source document IDs across train, validation, and test
Run near-duplicate analysis at document and chunk levels

ResolutionSplit by stable source document before chunking, rebuild every derived shard, and invalidate evaluation produced from contaminated splits.

The dataset license or source permission cannot be proven.

Likely causeThe dataset card is incomplete, a mixed corpus lost per-source terms, redistribution differs from training permission, or an internal source lacks recorded authority.

Safe checks

Inspect the provenance manifest and source-specific license text
Confirm dataset revision and content digest
Escalate ambiguous terms to the appropriate legal or governance owner

ResolutionExclude the source and rebuild the corpus. Do not infer permission from public accessibility, a repository host, or another model's use of the same text.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Reduce to a smaller clean corpus and document the domain limitation rather than adding lower-confidence text to reach a token target.

Stop conditions

Stop on cross-split document, duplicate, secret, personal data, unknown source/license, prohibited content, unexplained distribution, or mutable input.

config

Train a 16,000-token byte-level BPE on training documents only

caution

Create a dedicated tokenizer with NFKC normalization, byte-level pre-tokenization and decoder, fixed special tokens, a 16,000-token vocabulary, minimum pair frequency two, and BOS/EOS post-processing. Train only on the accepted training split and refuse to overwrite an existing artifact.

Why this step matters

Training a tokenizer is part of training from scratch. Vocabulary and normalization determine sequence length, language coverage, special-token semantics, model embedding size, decode fidelity, and every checkpoint's compatibility.

What to understand

NFKC normalization can intentionally collapse Unicode distinctions. Review whether that is acceptable for languages, identifiers, code, security-sensitive text, and preservation requirements.

Byte-level fallback prevents unknown characters from making text unrepresentable, but token efficiency may be poor for underrepresented languages. Measure tokens per character and long-tail coverage by source and language.

The 16,000 vocabulary balances embedding parameters and sequence efficiency for this small model. It is an experiment choice, not an optimal universal vocabulary.

System changes

Reads only training text and creates immutable tokenizer JSON and special-token mapping.

Syntax explained

vocab_size=16000: Bounds embedding size and vocabulary capacity for the roughly 34M-parameter architecture.
NFKC: Applies compatibility normalization; security and language consequences require review.
ByteLevel: Provides reversible byte coverage while potentially increasing token counts for some text.

File scripts/train_tokenizer.py

Configuration

from __future__ import annotations

import json
from pathlib import Path

from tokenizers import Tokenizer, decoders, models, normalizers, pre_tokenizers, processors, trainers

SPECIAL = ["<pad>", "<unk>", "<bos>", "<eos>"]
tokenizer = Tokenizer(models.BPE(unk_token="<unk>"))
tokenizer.normalizer = normalizers.NFKC()
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tokenizer.decoder = decoders.ByteLevel()
trainer = trainers.BpeTrainer(
    vocab_size=16_000,
    min_frequency=2,
    show_progress=True,
    special_tokens=SPECIAL,
    initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
)

def documents():
    with Path("corpus/train.jsonl").open(encoding="utf-8") as stream:
        for line in stream:
            yield json.loads(line)["text"]

tokenizer.train_from_iterator(documents(), trainer=trainer)
bos = tokenizer.token_to_id("<bos>")
eos = tokenizer.token_to_id("<eos>")
tokenizer.post_processor = processors.TemplateProcessing(
    single="<bos> $A <eos>",
    special_tokens=[("<bos>", bos), ("<eos>", eos)],
)
out = Path("artifacts/tokenizer")
out.mkdir(parents=True, exist_ok=False)
tokenizer.save(str(out / "tokenizer.json"))
(out / "special-tokens.json").write_text(
    json.dumps({token: tokenizer.token_to_id(token) for token in SPECIAL}, indent=2),
    encoding="utf-8",
)
samples = [
    "The service is healthy.",
    "Zażółć gęślą jaźń.",
    "JSON: {\"status\": true}",
]
for sample in samples:
    encoded = tokenizer.encode(sample)
    decoded = tokenizer.decode(encoded.ids, skip_special_tokens=True)
    print(json.dumps({"text": sample, "tokens": len(encoded.ids), "decoded": decoded}, ensure_ascii=False))

Example output / evidence

{"text":"The service is healthy.","tokens":8,"decoded":"The service is healthy."}
{"text":"Zażółć gęślą jaźń.","tokens":18,"decoded":"Zażółć gęślą jaźń."}

Checkpoint: Validate tokenizer artifact

python3 -m py_compile scripts/train_tokenizer.py && python3 scripts/train_tokenizer.py | tee evaluation/tokenizer-samples.jsonl

Continue whenTokenizer trains from train only, special IDs are stable, multilingual and structured round-trips are reviewed, and artifact digests are recorded.

Stop whenStop if validation/test text enters training, decode loses required distinctions, special IDs are missing, coverage is unacceptable, or an existing tokenizer would be overwritten.

If this step fails

A checkpoint loads but generation is nonsensical.

Likely causeTokenizer files, special-token IDs, chat template, architecture config, adapter/base pairing, or weight files do not belong to the same immutable revision.

Safe checks

Compare tokenizer and config digests with the checkpoint manifest
Verify BOS, EOS, PAD, and vocabulary size
Run a fixed token round-trip and inspect missing or unexpected keys

ResolutionRestore the matching tokenizer, config, base revision, and adapter or checkpoint. Never patch incompatible IDs in place and call the result recovered.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Compare BPE and Unigram tokenizers as separate preregistered artifacts using the same training text and coverage metrics before freezing model architecture.

Stop conditions

Stop if validation/test text enters training, decode loses required distinctions, special IDs are missing, coverage is unacceptable, or an existing tokenizer would be overwritten.

verification

Measure tokenizer coverage, efficiency, and boundary behavior

read-only

Evaluate tokenizer bytes and token counts on frozen validation/test documents without updating vocabulary. Report tokens per character and byte by source/language, maximum sequence expansion, special-token collisions, encode/decode fidelity, control characters, identifiers, code, JSON, Unicode normalization, and examples near the 512-token context boundary.

Why this step matters

A tokenizer that technically encodes every byte can still waste context or distort important text. Measurement exposes which languages and document types receive fewer effective characters per model context.

What to understand

Review distributions rather than only the mean. A high-tail language or code source may be truncated systematically even when aggregate efficiency looks reasonable.

Test special-token text from untrusted documents so literal strings cannot accidentally acquire control semantics. Ensure training data cannot inject BOS, EOS, or PAD behavior through plain text.

Freeze tokenizer digest before architecture initialization. Any later vocabulary or normalization change invalidates embeddings and every checkpoint.

System changes

Reads frozen splits with the accepted tokenizer and writes aggregate plus approved sample evidence; it updates no vocabulary.

Syntax explained

tokens_per_character: A simple efficiency measure that should be segmented by language and source.
round_trip_failures: Counts unacceptable encode/decode changes under the documented normalization policy.
p99: Shows long-tail sequence pressure that a mean conceals.

Command

python3 scripts/evaluate_tokenizer.py --tokenizer artifacts/tokenizer/tokenizer.json --input corpus/validation.jsonl --output evaluation/tokenizer-validation.json

Example output / evidence

{"documents":1000,"tokens":2491834,"tokens_per_character":0.2418,"round_trip_failures":0,"special_token_collisions":0,"p99_tokens_per_document":4871}

Checkpoint: Accept tokenizer for the model

python3 -m json.tool evaluation/tokenizer-validation.json >/dev/null && sha256sum artifacts/tokenizer/tokenizer.json

Continue whenCoverage and efficiency meet the charter for every required source/language, special tokens are safe, and the digest is frozen.

Stop whenStop on unacceptable normalization loss, special-token collision, unexplained language disparity, extreme expansion, or tokenizer mutation after evaluation.

If this step fails

A checkpoint loads but generation is nonsensical.

Likely causeTokenizer files, special-token IDs, chat template, architecture config, adapter/base pairing, or weight files do not belong to the same immutable revision.

Safe checks

Compare tokenizer and config digests with the checkpoint manifest
Verify BOS, EOS, PAD, and vocabulary size
Run a fixed token round-trip and inspect missing or unexpected keys

ResolutionRestore the matching tokenizer, config, base revision, and adapter or checkpoint. Never patch incompatible IDs in place and call the result recovered.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Train a new tokenizer revision with adjusted vocabulary or normalization, then repeat the entire tokenizer gate before initializing a model.

Stop conditions

Stop on unacceptable normalization loss, special-token collision, unexplained language disparity, extreme expansion, or tokenizer mutation after evaluation.

config

Define and cost a small decoder architecture before training

read-only

Use eight decoder layers, 512 hidden units, eight attention heads, 512-token context, 16,000-token vocabulary, learned positions, and dropout 0.1. Instantiate from config without pretrained weights, print parameter count, estimate activation and optimizer memory, and save the initial random artifact for the baseline.

Why this step matters

Architecture and data scale jointly define capacity and cost. Freezing them before observing test results prevents silent expansion and makes the experiment's limitations explicit.

What to understand

A roughly 34M-parameter model with about 50M tokens is under-resourced for broad capability and may be undertrained depending on objective. The mismatch is acceptable for lifecycle education, not hidden in claims.

Context 512 limits long-document behavior. Chunking breaks document continuity and may overweight boundaries; document and segment policy belongs in the model card.

The architecture has no instruction alignment, retrieval, tools, factual verification, safety tuning, or long-context mechanism. Raw next-token training does not add those systems.

System changes

Creates a reviewed architecture specification; model instantiation later writes random weights.

Syntax explained

8 layers × 512 hidden: Keeps parameters and one-GPU training bounded for the educational objective.
8 heads: Creates 64-dimensional attention heads because 512 divides evenly by eight.
context 512: Bounds activation memory and defines the maximum training block for this revision.

File governance/architecture.json

Configuration

{
  "model_type": "gpt2",
  "vocab_size": 16000,
  "n_positions": 512,
  "n_ctx": 512,
  "n_embd": 512,
  "n_layer": 8,
  "n_head": 8,
  "resid_pdrop": 0.1,
  "embd_pdrop": 0.1,
  "attn_pdrop": 0.1,
  "parameter_target": "approximately 34 million",
  "training_tokens_target": "approximately 50 million accepted tokens",
  "purpose": "educational and narrow research only"
}

Example output / evidence

Model initialized from config
parameters=34,742,272
context=512 vocab=16000 layers=8 heads=8 hidden=512

Checkpoint: Approve parameter and token budget

python3 -m json.tool governance/architecture.json >/dev/null && git diff -- governance/architecture.json

Continue whenReviewers accept parameter count, token/context ceiling, memory/storage/time estimate, educational scope, and prohibited claims.

Stop whenStop if the architecture exceeds compute approval, cannot fit with safety margin, or is being presented as equivalent to frontier training.

If this step fails

Training is much slower than the cost estimate.

Likely causeTokenization, data loading, CPU contention, storage, precision, kernel fallback, gradient checkpointing, communication, or thermal/power limits differ from the measured pilot.

Safe checks

Separate data, forward, backward, optimizer, checkpoint, and evaluation time
Measure tokens per second after warm-up
Inspect GPU utilization, clocks, power, memory, and input pipeline stalls

ResolutionPause before consuming the full budget. Fix the bounded bottleneck, repeat a representative pilot, and update cost and completion estimates with measured throughput.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Use 4–6 layers and 256–384 hidden units for CPU or smaller-GPU education, with a new parameter count and baseline.

Stop conditions

Stop if the architecture exceeds compute approval, cannot fit with safety margin, or is being presented as equivalent to frontier training.

config

Create the causal-LM training, checkpoint, and resume script

caution

Save the complete script that loads the frozen tokenizer, initializes GPT-2-style weights from config, tokenizes immutable train/validation text, groups 512-token blocks, uses causal labels, stores an initial random model, trains one epoch with effective batch 32, evaluates and checkpoints every 250 steps, retains three checkpoints, disables external reporting, and optionally resumes complete state.

Why this step matters

A readable trainer makes architecture, data grouping, objective, precision, optimizer, schedule, batch, checkpoint, and publication behavior inspectable. Starting from config proves no pretrained weights are used.

What to understand

The grouping function concatenates tokenized batches and discards each mapping batch's incomplete remainder. Measure the actual retained token count and ensure mapping batch boundaries do not cause material waste.

Causal language modeling uses each token as the next-token target after shifting. PAD tokens must be ignored by the collator, and BOS/EOS behavior must match the frozen tokenizer.

The initial random model is a required baseline and recovery artifact. Its digest proves the trained result began from this initialization and allows initial-loss comparison.

System changes

Creates training code; execution creates random initial weights, tokenized caches, complete checkpoints, final weights, tokenizer, and metrics.

Syntax explained

one epoch: Bounds compute and repetition for the educational experiment; additional epochs are separate preregistered runs.
effective batch 32: Eight sequences per device times four accumulation steps on one process.
save/eval every 250: Provides bounded recovery and validation cadence with three retained checkpoints.

File scripts/train_scratch.py

Configuration

from __future__ import annotations

import json
import math
import os
from pathlib import Path

import torch
from datasets import load_dataset
from transformers import (
    DataCollatorForLanguageModeling,
    GPT2Config,
    GPT2LMHeadModel,
    PreTrainedTokenizerFast,
    Trainer,
    TrainingArguments,
    set_seed,
)

SEED = 20260729
RUN_DIR = Path(os.getenv("RUN_DIR", "runs/small-clm-r1"))
TOKENIZER_DIR = Path(os.getenv("TOKENIZER_DIR", "artifacts/tokenizer"))
set_seed(SEED)
torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cudnn.allow_tf32 = False

tokenizer = PreTrainedTokenizerFast(
    tokenizer_file=str(TOKENIZER_DIR / "tokenizer.json"),
    bos_token="<bos>",
    eos_token="<eos>",
    unk_token="<unk>",
    pad_token="<pad>",
)
config = GPT2Config(
    vocab_size=len(tokenizer),
    n_positions=512,
    n_ctx=512,
    n_embd=512,
    n_layer=8,
    n_head=8,
    resid_pdrop=0.1,
    embd_pdrop=0.1,
    attn_pdrop=0.1,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
)
model = GPT2LMHeadModel(config)
parameters = sum(parameter.numel() for parameter in model.parameters())

raw = load_dataset(
    "json",
    data_files={
        "train": "corpus/train.jsonl",
        "validation": "corpus/validation.jsonl",
    },
)

def tokenize(batch):
    return tokenizer(batch["text"], add_special_tokens=True, truncation=False)

tokenized = raw.map(
    tokenize,
    batched=True,
    remove_columns=raw["train"].column_names,
    desc="tokenize",
)

def group(batch):
    concatenated = {key: sum(batch[key], []) for key in batch}
    total = (len(concatenated["input_ids"]) // 512) * 512
    return {
        key: [values[index:index + 512] for index in range(0, total, 512)]
        for key, values in concatenated.items()
    }

blocks = tokenized.map(group, batched=True, desc="group-512")
collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
args = TrainingArguments(
    output_dir=str(RUN_DIR),
    overwrite_output_dir=False,
    num_train_epochs=1,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    gradient_accumulation_steps=4,
    learning_rate=3e-4,
    weight_decay=0.1,
    warmup_ratio=0.02,
    lr_scheduler_type="cosine",
    max_grad_norm=1.0,
    bf16=True,
    tf32=False,
    eval_strategy="steps",
    eval_steps=250,
    save_strategy="steps",
    save_steps=250,
    save_total_limit=3,
    logging_steps=10,
    report_to="none",
    seed=SEED,
    data_seed=SEED,
    push_to_hub=False,
)
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=blocks["train"],
    eval_dataset=blocks["validation"],
    data_collator=collator,
    processing_class=tokenizer,
)
initial = RUN_DIR / "initial-random"
if not initial.exists():
    model.save_pretrained(initial, safe_serialization=True)
    tokenizer.save_pretrained(initial)
resume = os.getenv("RESUME_FROM_CHECKPOINT")
result = trainer.train(resume_from_checkpoint=resume or None)
evaluation = trainer.evaluate()
trainer.save_model(str(RUN_DIR / "final"))
tokenizer.save_pretrained(str(RUN_DIR / "final"))
summary = {
    "parameters": parameters,
    "train_loss": float(result.metrics["train_loss"]),
    "eval_loss": float(evaluation["eval_loss"]),
    "perplexity": math.exp(min(20, float(evaluation["eval_loss"]))),
}
(RUN_DIR / "summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
print(json.dumps(summary, sort_keys=True))

Example output / evidence

parameters=34742272 blocks_train=97656 blocks_validation=5021 effective_batch=32 save_steps=250 initial_random_saved=yes

Checkpoint: Compile and review the trainer

python3 -m py_compile scripts/train_scratch.py && grep -n 'from_config\|initial-random\|save_total_limit\|push_to_hub=False' scripts/train_scratch.py

Continue whenSyntax passes and reviewers confirm random initialization, frozen inputs, causal objective, bounded resources, checkpoints, resume, local reporting, and no automatic upload.

Stop whenStop if pretrained weights load, test text enters training, vocabulary changes, checkpoint state is incomplete, raw documents are logged, or publication is automatic.

If this step fails

A checkpoint loads but generation is nonsensical.

Likely causeTokenizer files, special-token IDs, chat template, architecture config, adapter/base pairing, or weight files do not belong to the same immutable revision.

Safe checks

Compare tokenizer and config digests with the checkpoint manifest
Verify BOS, EOS, PAD, and vocabulary size
Run a fixed token round-trip and inspect missing or unexpected keys

ResolutionRestore the matching tokenizer, config, base revision, and adapter or checkpoint. Never patch incompatible IDs in place and call the result recovered.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Implement a smaller raw PyTorch decoder for deeper educational transparency, but add equivalent tokenizer, masking, checkpoint, evaluation, and provenance tests.

Stop conditions

Stop if pretrained weights load, test text enters training, vocabulary changes, checkpoint state is incomplete, raw documents are logged, or publication is automatic.

command

Measure random-model and simple corpus baselines

caution

Before gradient updates, evaluate the saved random model on validation/test blocks and calculate a unigram token-frequency cross-entropy baseline from training counts without using test labels for fitting. These references show that loss reduction is real while preventing the trained model from being compared only with an undefined expectation.

Why this step matters

Random and unigram baselines calibrate the objective. A trained model should beat random vocabulary prediction and simple token-frequency statistics, but doing so is not evidence of useful language understanding.

What to understand

Use the same frozen tokenizer and test blocks. The unigram distribution is estimated from train only and handles unseen tokens with documented smoothing.

Perplexity is exponential in average token loss and depends on tokenizer, corpus, segmentation, and implementation. It cannot compare models using different tokenizers directly.

Save per-source loss where privacy allows. Aggregate improvement may come from dominant boilerplate while minority language or domain text gets worse.

System changes

Loads the random model and reads frozen corpora for baseline metrics; no weights are updated.

Syntax explained

random initial: The exact saved weight initialization from which the training lineage starts.
unigram: A non-contextual token-frequency reference fitted on training counts only.
perplexity: Token-level likelihood metric scoped to the same tokenizer and corpus.

Command

MODEL_PATH=runs/small-clm-r1/initial-random OUT_PATH=evaluation/random-initial.json python3 scripts/evaluate_scratch.py && python3 scripts/unigram_baseline.py --train corpus/train.jsonl --test corpus/test.jsonl --tokenizer artifacts/tokenizer/tokenizer.json --out evaluation/unigram.json

Example output / evidence

{"eval_loss":9.6821,"model_path":"runs/small-clm-r1/initial-random","perplexity":16042.7,"test_blocks":5024}
{"baseline":"unigram","cross_entropy":7.2142,"perplexity":1358.1}

Checkpoint: Freeze baseline evidence

python3 -m json.tool evaluation/random-initial.json >/dev/null && python3 -m json.tool evaluation/unigram.json >/dev/null

Continue whenRandom and unigram baselines use train-only fitting where applicable, the same tokenizer/test blocks, recorded digests, and no filtered failures.

Stop whenStop if baseline fitting touches test labels, tokenizer or blocks differ, initial weights are overwritten, or perplexity is compared across tokenizers without qualification.

If this step fails

Baseline results cannot be reproduced.

Likely causeThe base revision, tokenizer, evaluation set, template, decoding fields, seed, runtime, or metric implementation changed.

Safe checks

Compare every digest in the run manifest
Run the fixed baseline command in the locked environment
Inspect metric version and exact test IDs

ResolutionRestore the immutable baseline environment or create a new baseline lineage. Never compare the candidate against a moving or undocumented reference.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Add a small n-gram baseline for educational context, keeping its vocabulary, smoothing, fitting split, and limitations explicit.

Stop conditions

Stop if baseline fitting touches test labels, tokenizer or blocks differ, initial weights are overwritten, or perplexity is compared across tokenizers without qualification.

command

Run a bounded throughput, memory, and checkpoint pilot

caution

Train a small fixed number of steps or for thirty minutes on representative blocks. Measure retained tokens, tokens per second, peak VRAM, host RAM, data wait, checkpoint duration and size, validation duration, power if available, and projected one-epoch GPU-hours and storage. Do not extrapolate from model initialization or first-step compilation.

Why this step matters

A measured pilot tests the entire data, model, optimizer, validation, and checkpoint path before consuming the epoch budget. It also reveals whether the educational target fits the available host safely.

What to understand

Exclude warm-up and initialization from steady throughput but record them separately. Include validation and checkpoint overhead in completion estimates.

The stated 2–10 GPU-hour range is intentionally broad. Actual time depends on accelerator, kernels, storage, tokenization cache, power, and retained tokens.

Checkpoint size includes optimizer and scheduler state and can greatly exceed final model weights. Reserve capacity for three checkpoints, final, initial, caches, evaluation, and failure margin.

System changes

Allocates GPU and host memory, performs bounded updates, and writes disposable pilot checkpoints and logs.

Syntax explained

timeout 1800s: Bounds pilot compute to thirty minutes before an explicit capacity review.
tokens_per_second: Measured retained training-token throughput after warm-up, not raw corpus bytes.
projected storage: Includes complete recovery checkpoints and safety margin, not only final safetensors.

Command

RUN_DIR=runs/small-clm-pilot timeout 1800s python3 scripts/train_scratch.py 2>&1 | tee runs/small-clm-pilot.log

Example output / evidence

step=100 loss=8.041 tokens_per_second=14620 peak_vram_gib=12.8
checkpoint_bytes=1658420032 projected_epoch_hours=4.1 projected_storage_gib=8.7

Checkpoint: Approve one-epoch resource plan

nvidia-smi --query-gpu=name,memory.total,memory.used,power.draw --format=csv,noheader; df -h runs

Continue whenMemory has safety margin, no fallback or instability occurs, a checkpoint restores, and measured time/storage fit approval.

Stop whenStop on OOM, NaN, CPU fallback, data starvation, thermal/power instability, partial checkpoint, disk pressure, or projected budget breach.

If this step fails

The run fails with CUDA out of memory before the first optimizer step.

Likely causeThe model, sequence length, micro-batch, optimizer state, activations, allocator fragmentation, or another process exceeds available accelerator memory.

Safe checks

Record nvidia-smi memory and active processes before changing settings
Confirm effective sequence length and per-device micro-batch
Inspect whether quantization, gradient checkpointing, and intended precision actually activated

ResolutionStop other GPU work, reduce micro-batch or sequence length, use gradient accumulation to preserve the declared effective batch, and re-run the baseline rather than hiding a changed experiment.

Training loss becomes NaN or infinite.

Likely causeLearning rate, precision, invalid labels, corrupt tokens, exploding gradients, an unsupported kernel, or a numerically unstable quantization path caused divergence.

Safe checks

Inspect the first non-finite step and preceding gradient norm
Run the same batch in full precision on a small model
Verify labels, attention masks, tokenizer IDs, and maximum token ID

ResolutionRestore the last finite checkpoint, correct data or precision, lower learning rate if justified by a controlled experiment, enable finite gradient clipping, and restart from a clean recorded state.

Training is much slower than the cost estimate.

Likely causeTokenization, data loading, CPU contention, storage, precision, kernel fallback, gradient checkpointing, communication, or thermal/power limits differ from the measured pilot.

Safe checks

Separate data, forward, backward, optimizer, checkpoint, and evaluation time
Measure tokens per second after warm-up
Inspect GPU utilization, clocks, power, memory, and input pipeline stalls

ResolutionPause before consuming the full budget. Fix the bounded bottleneck, repeat a representative pilot, and update cost and completion estimates with measured throughput.

Checkpoint storage fills during training.

Likely causeSave frequency, retention, optimizer state, temporary writes, or concurrent runs exceed the capacity plan.

Safe checks

Measure checkpoint size and free space
List complete and partial checkpoint directories
Compare save_total_limit and expected run duration

ResolutionPause safely, preserve the latest verified complete checkpoint and required lineage, remove only explicitly obsolete artifacts, and revise retention before resuming.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Reduce layers, hidden size, context, or micro-batch as a new architecture revision; preserve effective-batch disclosure and rerun baselines.

Stop conditions

Stop on OOM, NaN, CPU fallback, data starvation, thermal/power instability, partial checkpoint, disk pressure, or projected budget breach.

command

Train one approved epoch with finite monitoring

caution

Run from the frozen random initialization and audited corpus. Monitor finite loss, gradient norm, learning rate, tokens per second, validation loss, memory, temperature, power, data wait, checkpoint completion, free space, and elapsed budget. Do not add epochs or change architecture because a sample looks promising.

Why this step matters

The full run tests whether the declared model learns next-token structure within the one-epoch budget. Monitoring stops infrastructure or numerical failures before they destroy evidence or consume unapproved resources.

What to understand

Falling loss is expected and insufficient. Watch train/validation gap, source-level validation, memorization canaries, and stability. Do not use test perplexity for checkpoint selection.

Restrictive permissions protect corpus paths, logs, and checkpoints. Raw sample generation should use approved synthetic prompts rather than training text.

Preserve failed-run manifests and cost. Deleting all failures and presenting only a successful run understates compute and hides unsafe parameter regions.

System changes

Updates all randomly initialized model parameters for one epoch and writes initial, checkpoint, final, tokenizer, cache, log, and metric artifacts.

Syntax explained

umask 077: Restricts newly written training artifacts to the current account.
one epoch: Enforces the approved pass over accepted training blocks.
checkpoint retention 3: Balances recovery with storage; deletion still follows evidence policy.

Command

umask 077; CUDA_VISIBLE_DEVICES=0 RUN_DIR=runs/small-clm-r1 python3 scripts/train_scratch.py 2>&1 | tee runs/small-clm-r1.console.log

Example output / evidence

step=250 loss=6.212 grad_norm=0.873 eval_loss=6.084
step=2000 loss=4.981 grad_norm=0.691 eval_loss=5.102
{"eval_loss":4.917,"parameters":34742272,"perplexity":136.7,"train_loss":5.244}

Checkpoint: Accept complete final and lineage

cat runs/small-clm-r1/summary.json && find runs/small-clm-r1 -name trainer_state.json -print

Continue whenFinite train/validation metrics, final safetensors, tokenizer, complete checkpoints, run manifest, measured cost, and no incident are present.

Stop whenStop on non-finite loss, worsening validation beyond the gate, extraction canary, data leak, resource incident, incomplete checkpoint, or budget overrun.

If this step fails

Training loss becomes NaN or infinite.

Likely causeLearning rate, precision, invalid labels, corrupt tokens, exploding gradients, an unsupported kernel, or a numerically unstable quantization path caused divergence.

Safe checks

Inspect the first non-finite step and preceding gradient norm
Run the same batch in full precision on a small model
Verify labels, attention masks, tokenizer IDs, and maximum token ID

Checkpoint storage fills during training.

Likely causeSave frequency, retention, optimizer state, temporary writes, or concurrent runs exceed the capacity plan.

Safe checks

Measure checkpoint size and free space
List complete and partial checkpoint directories
Compare save_total_limit and expected run duration

ResolutionPause safely, preserve the latest verified complete checkpoint and required lineage, remove only explicitly obsolete artifacts, and revise retention before resuming.

The adapter or trained model overfits and copies long training passages.

Likely causeThe corpus is too small or repetitive, epochs or capacity are excessive, validation misses memorization, or sensitive content was present.

Safe checks

Run held-out prefix and canary extraction tests
Compare train and validation loss trajectories
Inspect nearest training examples for suspicious generations

ResolutionReject publication, improve data and early stopping, reduce capacity or epochs, remove sensitive material, and rerun memorization evaluation before training again.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

End the experiment with negative evidence when the model does not improve safely; more epochs are a separately approved run, not an automatic continuation.

Stop conditions

Stop on non-finite loss, worsening validation beyond the gate, extraction canary, data leak, resource incident, incomplete checkpoint, or budget overrun.

command

Prove exact checkpoint resume and recovery

caution

Interrupt a pilot at a complete checkpoint, verify all model, optimizer, scheduler, scaler, RNG, trainer, sampler, tokenizer, config, data, code, package, and topology evidence, then resume. Compare global step, learning rate, and subsequent loss with a continuous control. Never resume a partial newest directory solely by timestamp.

Why this step matters

A long run without proven resume is not recoverable. Full optimizer and scheduler state matters because loading only model weights changes the optimization trajectory and cannot be reported as one continuous run.

What to understand

Validate file completeness and manifest digests before opening the checkpoint for write. Preserve a previous complete checkpoint until the resumed job passes its checkpoint.

Changing world size, batch, corpus, tokenizer, code, or packages creates a new run identity even if framework loading succeeds.

Bitwise equivalence is not always guaranteed across kernels, but a material learning-rate, loss, or sample-order discontinuity must be explained and accepted.

System changes

Loads complete training state and continues writing within the same run after integrity and compatibility checks.

Syntax explained

checkpoint-1000: Selects an explicit verified recovery point rather than latest-by-name automation.
optimizer and scheduler: Restores momentum and learning-rate trajectory, not only weights.
continuous control: A bounded uninterrupted run used to detect resume discontinuity.

Command

RESUME_FROM_CHECKPOINT=runs/small-clm-r1/checkpoint-1000 RUN_DIR=runs/small-clm-r1 python3 scripts/train_scratch.py

Example output / evidence

Loading checkpoint-1000
Continuing training from global step 1000
step=1010 loss=5.281 learning_rate=0.000221

Checkpoint: Accept checkpoint recovery

grep -E 'global step 1000|step=1010' runs/small-clm-r1.console.log | tail

Continue whenThe matching complete checkpoint resumes at the exact global step with compatible state and finite continuation.

Stop whenStop if any manifest digest, state component, topology, step, scheduler, corpus, tokenizer, or code differs.

If this step fails

The resumed run does not match the pre-interruption learning curve.

Likely causeOptimizer, scheduler, scaler, random generator, sampler position, data revision, code, or world size was not restored from the same checkpoint contract.

Safe checks

Compare checkpoint manifest, code commit, dataset digest, and package lock
Confirm optimizer and scheduler files exist
Inspect global step, epoch, sampler, seed, and accelerator topology

ResolutionDo not splice incompatible histories. Resume only from a complete matching checkpoint or start a new run ID and preserve both lineages.

The training process is killed or the host reboots.

Likely causePreemption, maintenance, kernel failure, out-of-memory killer, power loss, or scheduler timeout interrupted the process.

Safe checks

Inspect system and scheduler events without modifying checkpoints
Verify the newest checkpoint was atomically completed
Confirm code, data, topology, and package lock still match

ResolutionResume only from the newest verified complete checkpoint. If integrity or state compatibility fails, fall back to the previous checkpoint and preserve the broken artifact for diagnosis.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Restore the preceding verified checkpoint or create a new run; preserve the corrupt checkpoint for incident diagnosis.

Stop conditions

Stop if any manifest digest, state component, topology, step, scheduler, corpus, tokenizer, or code differs.

config

Evaluate held-out loss, perplexity, sources, and memorization

read-only

Save and run the same evaluator on random initial and final artifacts over untouched test blocks. Report loss/perplexity, per-source and per-language distributions, block count, failures, confidence, training-vs-test gap, memorization canaries, held-out prefix continuations, and qualitative samples selected independently of training.

Why this step matters

Held-out likelihood confirms the model learned corpus structure beyond random and unigram baselines. It does not prove truthfulness, instruction following, safety, usefulness, or general language competence.

What to understand

Perplexity is valid only with the same tokenizer and tokenization policy. Report loss alongside vocabulary, context, corpus, block grouping, and count.

Inspect worst and best source segments and long-tail languages. Aggregate results can hide a model that improves dominant boilerplate while degrading minority content.

Use planted canaries and governed extraction tests to detect memorization, but recognize they cannot prove absence. Do not publish training-like passages as impressive samples.

System changes

Loads immutable random or final weights for read-only evaluation and writes restricted metrics.

Syntax explained

MODEL_PATH: Selects the exact initial or final artifact while the corpus and evaluator remain fixed.
512-token blocks: Matches the training context; other contexts require separate evaluation.
exp(eval_loss): Computes perplexity with a numerical cap for reporting safety; the uncapped loss remains primary evidence.

File scripts/evaluate_scratch.py

Configuration

from __future__ import annotations

import json
import math
import os
from pathlib import Path

import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    DataCollatorForLanguageModeling,
    Trainer,
    TrainingArguments,
)

MODEL_PATH = os.environ["MODEL_PATH"]
OUT_PATH = Path(os.getenv("OUT_PATH", "evaluation/scratch-model.json"))
model = AutoModelForCausalLM.from_pretrained(MODEL_PATH, dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
raw = load_dataset("json", data_files={"test": "corpus/test.jsonl"})["test"]

def tokenize(batch):
    return tokenizer(batch["text"], add_special_tokens=True, truncation=False)

tokenized = raw.map(tokenize, batched=True, remove_columns=raw.column_names)

def group(batch):
    values = sum(batch["input_ids"], [])
    total = (len(values) // 512) * 512
    return {"input_ids": [values[i:i + 512] for i in range(0, total, 512)]}

blocks = tokenized.map(group, batched=True)
trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="evaluation/tmp",
        per_device_eval_batch_size=8,
        report_to="none",
    ),
    eval_dataset=blocks,
    data_collator=DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False),
    processing_class=tokenizer,
)
metrics = trainer.evaluate()
loss = float(metrics["eval_loss"])
summary = {
    "model_path": MODEL_PATH,
    "test_blocks": len(blocks),
    "eval_loss": loss,
    "perplexity": math.exp(min(20, loss)),
}
OUT_PATH.parent.mkdir(parents=True, exist_ok=True)
OUT_PATH.write_text(json.dumps(summary, indent=2), encoding="utf-8")
print(json.dumps(summary, sort_keys=True))

Example output / evidence

{"eval_loss":4.9431,"model_path":"runs/small-clm-r1/final","perplexity":140.2,"test_blocks":5024}

Checkpoint: Review evidence and limitations

MODEL_PATH=runs/small-clm-r1/final OUT_PATH=evaluation/final-test.json python3 scripts/evaluate_scratch.py

Continue whenFinal beats random and unigram likelihood baselines, no hard gate fails, segmented results and failures are visible, and claims remain educational.

Stop whenStop publication on test leakage, memorization, sensitive reproduction, unsupported language claims, hidden failures, or interpretation beyond next-token evidence.

If this step fails

Evaluation improves while representative answers become worse.

Likely causeThe held-out set is too similar to training data, the metric rewards surface form, prompt formatting differs, or the model memorized templates without improving the intended task.

Safe checks

Check document and near-duplicate leakage across splits
Review blinded task examples and safety cases
Compare baseline and candidate with the exact production template

ResolutionReject the candidate, repair split and rubric design, add adversarial and out-of-domain cases, and retrain only after the evaluation can detect the observed regression.

The adapter or trained model overfits and copies long training passages.

Likely causeThe corpus is too small or repetitive, epochs or capacity are excessive, validation misses memorization, or sensitive content was present.

Safe checks

Run held-out prefix and canary extraction tests
Compare train and validation loss trajectories
Inspect nearest training examples for suspicious generations

ResolutionReject publication, improve data and early stopping, reduce capacity or epochs, remove sensitive material, and rerun memorization evaluation before training again.

The candidate passes aggregate metrics but fails a safety or policy case.

Likely causeAverages hide a critical regression, the gate was treated as weighted, or evaluation omitted a required class.

Safe checks

Inspect per-case results and hard-gate status
Re-run the failing case with saved prompt and artifact identity
Review whether the test is valid and independent of training data

ResolutionReject promotion. Hard safety, privacy, licensing, and policy gates cannot be offset by lower loss or better average task score.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Publish a negative experiment report without weights when the lifecycle is educational but artifact risks or quality gates fail.

Stop conditions

Stop publication on test leakage, memorization, sensitive reproduction, unsupported language claims, hidden failures, or interpretation beyond next-token evidence.

command

Run bounded local inference and inspect failure behavior

read-only

Load the final safetensors and tokenizer on loopback or offline, use synthetic prompts, cap input and output, disable tools, and label text as untrusted experimental generation. Test empty, multilingual, structured, long, adversarial, memorization-canary, and out-of-domain prompts; record repetition, degeneration, false factual confidence, and context truncation.

Why this step matters

Generation reveals qualitative limitations that perplexity hides. The output may appear grammatical while being false, repetitive, copied, biased, or unsafe, so it remains evidence rather than a user-facing capability.

What to understand

Compare greedy and one documented sampling setup without cherry-picking. Preserve prompt IDs and every failure, not only readable outputs.

Never connect this raw model to command execution, retrieval, secrets, external APIs, or autonomous actions. It lacks instruction alignment and policy enforcement.

Test context-boundary truncation and repetitive loops. Small models trained on limited data commonly degenerate; disclose this rather than applying hidden filters to samples.

System changes

Loads the final model for local read-only inference and writes only approved synthetic evidence.

Syntax explained

--max-new-tokens 80: Bounds generation cost and exposure; it does not change the 512-token input context.
--seed 20260729: Supports repeatable sampling when the implementation and hardware permit it.
synthetic prompt: Avoids exposing private or copyrighted evaluation text during qualitative inspection.

Command

python3 scripts/generate.py --model runs/small-clm-r1/final --prompt 'A small language model is' --max-new-tokens 80 --seed 20260729

Example output / evidence

A small language model is a sequence predictor trained on a limited corpus. This educational checkpoint may repeat phrases and should not be treated as factual advice.

Checkpoint: Accept limitations, not capability hype

python3 scripts/run_generation_fixtures.py --model runs/small-clm-r1/final --out evaluation/generation-fixtures.json

Continue whenAll fixed prompts are accounted for, unsafe or copied outputs fail gates, and model-card limitations match observed behavior.

Stop whenStop if outputs reproduce sensitive/copyrighted passages, are presented as factual authority, trigger actions, or hidden filtering makes evidence incomplete.

If this step fails

The adapter or trained model overfits and copies long training passages.

Likely causeThe corpus is too small or repetitive, epochs or capacity are excessive, validation misses memorization, or sensitive content was present.

Safe checks

Run held-out prefix and canary extraction tests
Compare train and validation loss trajectories
Inspect nearest training examples for suspicious generations

ResolutionReject publication, improve data and early stopping, reduce capacity or epochs, remove sensitive material, and rerun memorization evaluation before training again.

The candidate passes aggregate metrics but fails a safety or policy case.

Likely causeAverages hide a critical regression, the gate was treated as weighted, or evaluation omitted a required class.

Safe checks

Inspect per-case results and hard-gate status
Re-run the failing case with saved prompt and artifact identity
Review whether the test is valid and independent of training data

ResolutionReject promotion. Hard safety, privacy, licensing, and policy gates cannot be offset by lower loss or better average task score.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Expose only a static, curated educational demo with explicit warnings and no private input when interactive inference cannot be governed safely.

Stop conditions

Stop if outputs reproduce sensitive/copyrighted passages, are presented as factual authority, trigger actions, or hidden filtering makes evidence incomplete.

config

Package weights, tokenizer, manifests, and an honest model card

caution

Create an immutable artifact containing final safetensors, config, tokenizer, training/evaluation summaries, exact environment and code, corpus lineage at an approved disclosure level, parameter/token/compute budget, intended educational use, prohibited uses, observed limitations, memorization and safety results, license, citation, owners, and review date. Keep raw corpus and restricted test output separate.

Why this step matters

Weights without tokenizer, provenance, evaluation, cost, scope, and limitations invite misuse and cannot be reproduced. An honest card is part of the research artifact, not marketing.

What to understand

Do not claim trained from scratch means independent of external intellectual property; the corpus, software, architecture, and methods all have provenance and terms.

Report total experiment cost including pilots and failures, not only the successful epoch. State parameter and token counts exactly from artifacts.

Publication requires a separate license, privacy, security, memorization, disclosure, and destination review. Internal training permission may not allow public weights.

System changes

Creates a release candidate and metadata; it does not publish, upload, or serve it.

Syntax explained

safetensors: Stores tensors without pickle execution while config and provenance remain separately verified.
approved disclosure level: Balances transparency with source privacy, contracts, and security.
prohibited uses: Explicitly states contexts the educational evidence does not support.

File artifacts/small-clm-r1/MODEL_CARD.md

Configuration

# Small CLM R1

## Scope
Educational roughly 34M-parameter causal model trained from random initialization.
Not an assistant, factual source, safety system, or production recommendation.

## Training
- Licensed corpus revision: 2026-07-29.1
- Accepted token count: record exact value
- Context: 512; vocabulary: 16,000; one epoch
- Hardware and GPU-hours: record measured values

## Evaluation
- Random, unigram, and final test loss/perplexity
- Per-source/language results and all failures
- Memorization, safety, and qualitative fixture results

## Limitations and prohibited uses
Limited corpus and scale, no instruction alignment, no current knowledge,
may hallucinate, repeat, memorize, or produce biased/unsafe text.
No autonomous actions, high-stakes advice, or production user traffic.

## Lineage, license, owners, and review
Record artifact digests, code/environment, approved disclosure,
rollback owner, and review due 2026-10-27.

Example output / evidence

Release candidate assembled: 9 files, 287 MiB, model card limitations present, raw corpus absent, sha256 manifest complete.

Checkpoint: Approve or quarantine the artifact

find artifacts/small-clm-r1 -maxdepth 1 -type f -print0 | sort -z | xargs -0 sha256sum > artifacts/small-clm-r1/SHA256SUMS

Continue whenApproved files, digests, card, lineage, evaluation, license, cost, limitations, owners, and review are complete; raw or restricted data is absent.

Stop whenStop on unclear distribution rights, secret or personal data, missing tokenizer/config, memorization, misleading claims, unreported failures, or automatic upload.

If this step fails

A model hub upload exposes a private artifact.

Likely causeRepository visibility, token scope, push settings, generated card, logs, dataset samples, or checkpoint contents were not reviewed before upload.

Safe checks

Stop the upload and inspect repository visibility
Revoke the token and audit uploaded revisions
Check model, tokenizer, adapter, card, logs, and dataset references

ResolutionQuarantine or remove the remote revision according to provider and incident policy, rotate credentials, and require an explicit publication gate instead of automatic push.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Publish only code, synthetic fixtures, aggregate metrics, and a technical report while keeping weights private or deleting them under governance.

Stop conditions

Stop on unclear distribution rights, secret or personal data, missing tokenizer/config, memorization, misleading claims, unreported failures, or automatic upload.

verification

Verify rollback, clean disposable state, and schedule review

read-only

Because this is an educational artifact, rollback means stopping all demos, revoking access, preserving the report and required lineage, and deleting derived weights or caches according to policy while keeping source-removal mapping. Test restoration of the initial and final accepted artifacts before pruning checkpoints.

Why this step matters

Closing the experiment prevents abandoned listeners, credentials, caches, and ambiguous artifacts from becoming shadow production. Retained lineage supports audit and source withdrawal even after bulky checkpoints are removed.

What to understand

Inventory every local, registry, backup, notebook, tracker, cache, and copied artifact. Deletion must follow retention and legal-hold requirements and be recorded, not assumed.

Verify no service or demo exposes the model and no token remains active. If weights were shared, record recipients and apply the distribution rollback procedure.

Review after ninety days or any data, tokenizer, architecture, package, hardware, evaluation, license, publication, or intended-use change. Results do not remain current automatically.

System changes

Records research-only acceptance and review; approved cleanup removes only disposable derived state after evidence retention.

Syntax explained

production_traffic=prohibited: Preserves the hard boundary between educational evidence and supported service.
source_removal_mapping: Retains the ability to identify affected corpus and model lineages.
review_due: Expires conclusions after ninety days or material change.

Command

printf '%s\n' 'status=research-only' 'production_traffic=prohibited' 'rollback_tested=yes' 'source_removal_mapping=retained' 'review_due=2026-10-27' | tee governance/scratch-acceptance.txt

Example output / evidence

status=research-only
production_traffic=prohibited
rollback_tested=yes
source_removal_mapping=retained
review_due=2026-10-27

Checkpoint: Close the experiment safely

cat governance/scratch-acceptance.txt && ss -lntp | grep -E ':(8000|8080)\b' || true

Continue whenNo model listener or active credential remains, retained evidence and source mapping are verified, disposable cleanup is recorded, and review is scheduled.

Stop whenStop cleanup if recovery, audit, incident, source withdrawal, legal hold, or publication records would be destroyed; stop closure if any model endpoint remains exposed.

If this step fails

The API server accepts requests from an untrusted network.

Likely causeThe listener binds all interfaces, container port publishing bypasses expectations, authentication is absent, or a proxy forwards administrative credentials.

Safe checks

Inspect listening addresses and firewall rules
Test from an untrusted segment with synthetic input
Review TLS, authentication, token scope, and logs

ResolutionReturn to loopback or a protected network immediately. Add authenticated TLS ingress, least-privilege credentials, rate limits, audit, and an owner before intentional exposure.

Security notes

Do not place model-hub, object-storage, tracking, or infrastructure credentials in notebooks, scripts, configs, shell history, logs, checkpoints, model cards, or exported environments.
Use approved synthetic or minimized examples in diagnostics. Training artifacts can memorize sensitive content and should inherit access, retention, incident, and deletion controls from the source data.

Alternatives

Keep a restricted offline artifact for reproducibility while deleting serving state and high-volume intermediate checkpoints according to policy.

Stop conditions

Stop cleanup if recovery, audit, incident, source withdrawal, legal hold, or publication records would be destroyed; stop closure if any model endpoint remains exposed.

Finish line

Verification checklist

Verify corpus, tokenizer, and architecture lineage

python3 scripts/audit_corpus.py >/dev/null && sha256sum governance/corpus-provenance.json artifacts/tokenizer/tokenizer.json governance/architecture.json

Every artifact matches the accepted source, split, tokenizer, architecture, environment, and code manifest without test leakage.

Verify the final model improves next-token likelihood

python3 - <<'PY'
import json
random=json.load(open('evaluation/random-initial.json'))
final=json.load(open('evaluation/final-test.json'))
assert final['test_blocks']==random['test_blocks']
assert final['eval_loss'] < random['eval_loss']
print('loss delta', random['eval_loss']-final['eval_loss'])
PY

The same test blocks and tokenizer show lower final loss than random initialization; unigram, segmented, memorization, and qualitative evidence remain visible.

Verify research-only closure and rollback

grep -E 'status=research-only|production_traffic=prohibited|rollback_tested=yes|review_due=2026-10-27' governance/scratch-acceptance.txt

The artifact remains research-only, no production route exists, rollback and source-removal lineage are tested, and the ninety-day review is explicit.

Recovery guidance

Common problems and safe checks

The run fails with CUDA out of memory before the first optimizer step.

Likely causeThe model, sequence length, micro-batch, optimizer state, activations, allocator fragmentation, or another process exceeds available accelerator memory.

Safe checks

Record nvidia-smi memory and active processes before changing settings
Confirm effective sequence length and per-device micro-batch
Inspect whether quantization, gradient checkpointing, and intended precision actually activated

Training loss becomes NaN or infinite.

Likely causeLearning rate, precision, invalid labels, corrupt tokens, exploding gradients, an unsupported kernel, or a numerically unstable quantization path caused divergence.

Safe checks

Inspect the first non-finite step and preceding gradient norm
Run the same batch in full precision on a small model
Verify labels, attention masks, tokenizer IDs, and maximum token ID

Evaluation improves while representative answers become worse.

Likely causeThe held-out set is too similar to training data, the metric rewards surface form, prompt formatting differs, or the model memorized templates without improving the intended task.

Safe checks

Check document and near-duplicate leakage across splits
Review blinded task examples and safety cases
Compare baseline and candidate with the exact production template

ResolutionReject the candidate, repair split and rubric design, add adversarial and out-of-domain cases, and retrain only after the evaluation can detect the observed regression.

The resumed run does not match the pre-interruption learning curve.

Likely causeOptimizer, scheduler, scaler, random generator, sampler position, data revision, code, or world size was not restored from the same checkpoint contract.

Safe checks

Compare checkpoint manifest, code commit, dataset digest, and package lock
Confirm optimizer and scheduler files exist
Inspect global step, epoch, sampler, seed, and accelerator topology

ResolutionDo not splice incompatible histories. Resume only from a complete matching checkpoint or start a new run ID and preserve both lineages.

A checkpoint loads but generation is nonsensical.

Likely causeTokenizer files, special-token IDs, chat template, architecture config, adapter/base pairing, or weight files do not belong to the same immutable revision.

Safe checks

Compare tokenizer and config digests with the checkpoint manifest
Verify BOS, EOS, PAD, and vocabulary size
Run a fixed token round-trip and inspect missing or unexpected keys

ResolutionRestore the matching tokenizer, config, base revision, and adapter or checkpoint. Never patch incompatible IDs in place and call the result recovered.

Training is much slower than the cost estimate.

Likely causeTokenization, data loading, CPU contention, storage, precision, kernel fallback, gradient checkpointing, communication, or thermal/power limits differ from the measured pilot.

Safe checks

Separate data, forward, backward, optimizer, checkpoint, and evaluation time
Measure tokens per second after warm-up
Inspect GPU utilization, clocks, power, memory, and input pipeline stalls

ResolutionPause before consuming the full budget. Fix the bounded bottleneck, repeat a representative pilot, and update cost and completion estimates with measured throughput.

The dataset audit finds personal data, secrets, or credentials.

Likely causeCollection, export, issue text, logs, or generated examples entered the corpus without adequate minimization and scanning.

Safe checks

Quarantine the affected shard and record its digest
Identify source, scope, access, and downstream copies
Search tokenizer, caches, checkpoints, and artifacts according to incident policy

The dataset license or source permission cannot be proven.

Likely causeThe dataset card is incomplete, a mixed corpus lost per-source terms, redistribution differs from training permission, or an internal source lacks recorded authority.

Safe checks

Inspect the provenance manifest and source-specific license text
Confirm dataset revision and content digest
Escalate ambiguous terms to the appropriate legal or governance owner

ResolutionExclude the source and rebuild the corpus. Do not infer permission from public accessibility, a repository host, or another model's use of the same text.

Tokenized splits contain duplicates or overlapping documents.

Likely causeSplitting happened after chunking, document IDs were unstable, normalization collapsed examples, or near-duplicate detection was omitted.

Safe checks

Recompute exact hashes before tokenization
Check source document IDs across train, validation, and test
Run near-duplicate analysis at document and chunk levels

ResolutionSplit by stable source document before chunking, rebuild every derived shard, and invalidate evaluation produced from contaminated splits.

The GPU is visible but unsupported kernels or CPU fallback are used.

Likely causeDriver, CUDA, compute capability, package wheel, dtype, bitsandbytes backend, or attention implementation is incompatible.

Safe checks

Record torch and CUDA versions and device capability
Inspect runtime logs for backend and kernel selection
Run a tiny matrix operation and memory-footprint probe

ResolutionUse a documented compatible stack, rebuild the isolated environment, and rerun the measured pilot. Do not publish throughput from an unintended fallback.

Checkpoint storage fills during training.

Likely causeSave frequency, retention, optimizer state, temporary writes, or concurrent runs exceed the capacity plan.

Safe checks

Measure checkpoint size and free space
List complete and partial checkpoint directories
Compare save_total_limit and expected run duration

ResolutionPause safely, preserve the latest verified complete checkpoint and required lineage, remove only explicitly obsolete artifacts, and revise retention before resuming.

A model hub upload exposes a private artifact.

Likely causeRepository visibility, token scope, push settings, generated card, logs, dataset samples, or checkpoint contents were not reviewed before upload.

Safe checks

Stop the upload and inspect repository visibility
Revoke the token and audit uploaded revisions
Check model, tokenizer, adapter, card, logs, and dataset references

ResolutionQuarantine or remove the remote revision according to provider and incident policy, rotate credentials, and require an explicit publication gate instead of automatic push.

Baseline results cannot be reproduced.

Likely causeThe base revision, tokenizer, evaluation set, template, decoding fields, seed, runtime, or metric implementation changed.

Safe checks

Compare every digest in the run manifest
Run the fixed baseline command in the locked environment
Inspect metric version and exact test IDs

ResolutionRestore the immutable baseline environment or create a new baseline lineage. Never compare the candidate against a moving or undocumented reference.

The training process is killed or the host reboots.

Likely causePreemption, maintenance, kernel failure, out-of-memory killer, power loss, or scheduler timeout interrupted the process.

Safe checks

Inspect system and scheduler events without modifying checkpoints
Verify the newest checkpoint was atomically completed
Confirm code, data, topology, and package lock still match

ResolutionResume only from the newest verified complete checkpoint. If integrity or state compatibility fails, fall back to the previous checkpoint and preserve the broken artifact for diagnosis.

The adapter or trained model overfits and copies long training passages.

Likely causeThe corpus is too small or repetitive, epochs or capacity are excessive, validation misses memorization, or sensitive content was present.

Safe checks

Run held-out prefix and canary extraction tests
Compare train and validation loss trajectories
Inspect nearest training examples for suspicious generations

ResolutionReject publication, improve data and early stopping, reduce capacity or epochs, remove sensitive material, and rerun memorization evaluation before training again.

Inference output differs after merge, quantization, or serving conversion.

Likely causeThe conversion changed precision, tokenizer, template, adapter scaling, architecture metadata, or unsupported operations.

Safe checks

Run the identical evaluation before and after each transformation
Compare weight, tokenizer, config, and adapter manifests
Inspect conversion warnings and missing keys

ResolutionKeep the last accepted artifact, reject the transformed candidate, and change one conversion variable at a time. A smaller artifact is not acceptable without quality evidence.

The API server accepts requests from an untrusted network.

Likely causeThe listener binds all interfaces, container port publishing bypasses expectations, authentication is absent, or a proxy forwards administrative credentials.

Safe checks

Inspect listening addresses and firewall rules
Test from an untrusted segment with synthetic input
Review TLS, authentication, token scope, and logs

ResolutionReturn to loopback or a protected network immediately. Add authenticated TLS ingress, least-privilege credentials, rate limits, audit, and an owner before intentional exposure.

The candidate passes aggregate metrics but fails a safety or policy case.

Likely causeAverages hide a critical regression, the gate was treated as weighted, or evaluation omitted a required class.

Safe checks

Inspect per-case results and hard-gate status
Re-run the failing case with saved prompt and artifact identity
Review whether the test is valid and independent of training data

ResolutionReject promotion. Hard safety, privacy, licensing, and policy gates cannot be offset by lower loss or better average task score.

After the procedure

Alternatives and next steps

Consider these alternatives

Use a smaller character or word-level model to teach backpropagation and sequence modeling with far lower compute and an even clearer limitation boundary.
Fine-tune an approved existing base when application quality matters, while retaining source licensing, evaluation, artifact, and serving controls.
Use retrieval or deterministic search for useful access to the licensed corpus rather than attempting to encode knowledge in limited weights.
Publish a reproducible technical report and code without weights when derivative rights, memorization, or misuse risks make artifact distribution inappropriate.

Operate it safely

Add independently reviewed tokenizer coverage, source/language likelihood, memorization, bias, toxicity, privacy, and out-of-domain fixtures without changing the frozen test after results are seen.
Compare one smaller architecture or tokenizer only as a new preregistered run with its own random baseline, budget, lineage, and complete report.
Automate signed manifests for corpus, tokenizer, architecture, initial/final/checkpoint weights, environment, cost, evaluation, model card, and deletion evidence while keeping promotion manual.
Create source-withdrawal and artifact-quarantine runbooks that identify every affected tokenizer, checkpoint, final weight, cache, backup, report, and recipient.
Review official PyTorch, Transformers, Datasets, Tokenizers, licensing, privacy, evaluation, security, and this guide no later than 2026-10-27.

Reference

Frequently asked questions

Is this how frontier LLMs are trained?

It demonstrates the same high-level next-token lifecycle, but frontier programs differ radically in data scale, parameters, distributed systems, optimization, alignment, evaluation, safety, reliability, staffing, energy, and cost.

Will a 34M-parameter model be useful as an assistant?

Do not assume so. It is intended for education and narrow research. It lacks broad data, instruction alignment, current knowledge, factual verification, and production safety.

Why train a tokenizer only on the train split?

Tokenizer vocabulary is learned state. Including validation or test text would leak their distribution into fitting and weaken the independence of evaluation.

Does public-domain or public text always permit training?

No. Verify source authenticity, jurisdiction, license, terms, personal data, secrets, attribution, collection method, derivative rights, and intended distribution.

Can perplexity compare this model with another tokenizer?

Not directly. Token units differ. Keep tokenizer and corpus fixed or use carefully designed byte/character-normalized and task evaluations with explicit methodology.

Why keep a random initial checkpoint?

It proves the starting weights, supports a true baseline, and helps reproduce the training lineage. It is distinct from a resumable optimizer checkpoint.

Why include a unigram baseline?

It shows whether contextual learning beats simple training-token frequency. Beating it is necessary evidence, not proof of useful language competence.

Should I train more epochs when loss still falls?

Not automatically. More passes increase compute and memorization risk and change the experiment. Predeclare a separate run and evaluate validation, test, memorization, and budget gates.

Can checkpoints be deleted after training?

Keep enough complete recovery and audit evidence according to policy. Remove bulky obsolete checkpoints only after recovery, source-withdrawal, legal, incident, and reproducibility owners approve.

Can the model be uploaded publicly?

Only after separate license, derivative-rights, privacy, memorization, security, disclosure, model-card, and destination review. This guide defaults to no upload.

Does safetensors make the model safe?

It avoids pickle code execution for weight serialization. It does not prove model behavior, remove memorized data, validate provenance, or secure an exposed API.

How often should the experiment be reviewed?

At least every ninety days, next due 2026-10-27, and immediately after changes to corpus rights, data, tokenizer, architecture, code, packages, hardware, evaluation, publication, or intended use.

Recovery

Rollback

Stop every demo or API, revoke credentials and access, restore the previous approved system if this experiment replaced any workflow, preserve governance and evaluation evidence, and delete derived training state only through the recorded retention and source-removal process.

Disable routing and stop the local generation process. Verify no listener, scheduled job, notebook kernel, or shared endpoint can load the research model.
Revoke artifact and storage credentials, remove unauthorized copies according to incident policy, and retain the recipient and deletion record if distribution occurred.
Preserve corpus provenance, source-removal mapping, tokenizer/model manifests, exact environment, aggregate metrics, limitations, cost, and incidents under restricted retention.
Delete caches, partial checkpoints, and derived weights only after recovery, audit, legal-hold, publication, privacy, and source-removal owners approve. Never delete the original source record ad hoc.

Evidence

Sources and review

Verified 2026-07-29Review due 2026-10-27

Hugging Face Tokenizers training from iteratorsofficial Hugging Face Tokenizers pipelineofficial Hugging Face Tokenizers trainer APIofficial Transformers causal language modelingofficial Hugging Face course: causal language model from scratchofficial Hugging Face dataset cards and license metadataofficial PyTorch saving and loading checkpointsofficial PyTorch reproducibility guidanceofficial