Command line¶
Every command and option, from tinydistill <command> --help.
| Command | What it does |
|---|---|
label |
score every label with the teacher |
train |
distil the student |
export |
export the student to ONNX |
eval |
quality and speed table |
predict |
classify texts with an exported model |
serve |
serve an exported model over HTTP (JSON) |
plot |
charts of a run as PNG files: training, speed and accuracy, per label, confusions, cascade |
web |
a static web page that runs the model in the browser |
synth |
have the teacher write unlabelled example texts |
init |
write a starter task file from a dataset's labels |
push |
upload an exported model to the Hugging Face Hub, with a model card |
report |
rebuild RUN/results.md from RUN/results.json |
tune |
choose teacher, prompt and few-shot on a dev split |
refresh |
retrain RUN with the texts its cascade sent to the teacher (a new run directory) |
suggest |
which training texts a person should label next |
analyse |
where the teacher and student go wrong (RUN/analysis.html) |
status |
how far a run is, and whether it is still running |
compare |
one markdown table comparing several runs (e.g. student sizes) |
env |
versions and hardware, for bug reports |
run |
label, train, export and eval |
label¶
Score every label with the teacher.
usage: tinydistill label [-h] --data DATA [--test TEST] --out OUT [--teacher TEACHER]
[--load-4bit] [--cache-dir CACHE_DIR] [--vllm-url VLLM_URL]
[--api-url API_URL] [--api-key-env VAR] [--shortlist K]
[--shortlist-model SHORTLIST_MODEL] [--shortlist-keep IDS] [--few-shot K]
[--teacher-tokens TEACHER_TOKENS] [--limit LIMIT]
[--test-limit TEST_LIMIT] [--device DEVICE] [--threads THREADS]
[--seed SEED]
task
positional arguments:
task task YAML file
options:
-h, --help show this help message and exit
--data DATA unlabelled texts: .csv, .jsonl or hf:<dataset>[:<config>]:<split>
--test TEST optional gold-labelled test set, same formats
--out OUT run directory
--teacher TEACHER Hugging Face causal LM; several, comma-separated, are ensembled
--load-4bit load the teacher in 4 bit (bitsandbytes, CUDA)
--cache-dir CACHE_DIR
where teacher scores are cached (default: the run directory); runs that
share it label each text once
--vllm-url VLLM_URL score with this vLLM server (OpenAI API base, e.g. http://host:8000/v1),
not a local model
--api-url API_URL score with a hosted OpenAI-compatible chat API (base URL, e.g.
https://api.openai.com/v1; --teacher names the model there): the labels
are shown as options A, B, C, ... and scored by the answer letter's log-
probability; single-label tasks with at most 20 labels
--api-key-env VAR environment variable holding the API key for --api-url (default
OPENAI_API_KEY; unset: no key is sent, as for a local server)
--shortlist K score only the K labels most similar to each text (for tasks with very
many labels)
--shortlist-model SHORTLIST_MODEL
sentence-embedding model for --shortlist (default: MiniLM)
--shortlist-keep IDS label ids the teacher always scores besides the shortlist, comma-
separated: catch-all labels such as other or out of scope, whose names
resemble no particular text
--few-shot K show K gold training examples per class in the teacher's prompt (needs
{examples} in it)
--teacher-tokens TEACHER_TOKENS
tokens per teacher forward pass; short texts with few labels are packed
together
--limit LIMIT use only the first N training texts
--test-limit TEST_LIMIT
use only the first N test texts
--device DEVICE auto, cpu or cuda (teacher labelling and training)
--threads THREADS CPU threads for torch and onnxruntime
--seed SEED seed for example selection and training
train¶
Distil the student.
usage: tinydistill train [-h] [--student STUDENT] [--epochs EPOCHS] [--val-share S]
[--patience PATIENCE] [--batch-size BATCH_SIZE] [--lr LR]
[--max-length MAX_LENGTH] [--gold-file GOLD_FILE] [--gold-per-class K]
[--ce-weight CE_WEIGHT] [--gold-only] [--seeds SEEDS] [--upper-bound]
[--device DEVICE] [--threads THREADS] [--seed SEED]
run
positional arguments:
run run directory
options:
-h, --help show this help message and exit
--student STUDENT Hugging Face encoder, or `static` / `static:<encoder>` for a static-
embedding student (no attention: much smaller and faster; try --epochs 20
--lr 1e-3)
--epochs EPOCHS passes over the training texts (default 10)
--val-share S hold out this share of the training texts and keep the epoch with the
lowest loss on them, stopping early after --patience epochs without
improvement (e.g. 0.05; default: off, keep the last)
--patience PATIENCE epochs without improvement before stopping
--batch-size BATCH_SIZE
--lr LR
--max-length MAX_LENGTH
--gold-file GOLD_FILE
gold labels for some training texts (same text and label fields as the
task), e.g. the texts `tinydistill suggest` picked, labelled by a person;
used like --gold-per-class
--gold-per-class K also use the gold labels of K training examples per class (cross-entropy
term)
--ce-weight CE_WEIGHT
weight of that cross-entropy term
--gold-only also train the student on the K gold labels per class only, without the
teacher (a baseline)
--seeds SEEDS training seeds per student (mean and std in eval)
--upper-bound also train the student on all gold training labels, without the teacher
--device DEVICE auto, cpu or cuda (teacher labelling and training)
--threads THREADS CPU threads for torch and onnxruntime
--seed SEED seed for example selection and training
export¶
Export the student to onnx.
usage: tinydistill export [-h] [--int8] [--device DEVICE] [--threads THREADS] [--seed SEED] run
positional arguments:
run run directory
options:
-h, --help show this help message and exit
--int8 also write an int8 dynamically quantised model
--device DEVICE auto, cpu or cuda (teacher labelling and training)
--threads THREADS CPU threads for torch and onnxruntime
--seed SEED seed for example selection and training
eval¶
Quality and speed table.
usage: tinydistill eval [-h] [--teacher-devices TEACHER_DEVICES] [--skip-teacher-bench]
[--reps REPS] [--teacher-reps TEACHER_REPS] [--device DEVICE]
[--threads THREADS] [--seed SEED]
run
positional arguments:
run run directory
options:
-h, --help show this help message and exit
--teacher-devices TEACHER_DEVICES
comma-separated devices to time the teacher on (default: the labelling
device)
--skip-teacher-bench report teacher quality only
--reps REPS timed runs per student benchmark at batch 1
--teacher-reps TEACHER_REPS
timed runs of the teacher (batch 1)
--device DEVICE auto, cpu or cuda (teacher labelling and training)
--threads THREADS CPU threads for torch and onnxruntime
--seed SEED seed for example selection and training
predict¶
Classify texts with an exported model.
usage: tinydistill predict [-h] [--device DEVICE] [--threads THREADS] [--seed SEED] [--data DATA]
[--text-field TEXT_FIELD] [--fp32] [--cascade RUN] [--log LOG]
[--threshold THRESHOLD]
model [texts ...]
positional arguments:
model directory written by `export` (run/onnx), or hf:user/name on the Hub
texts
options:
-h, --help show this help message and exit
--device DEVICE auto, cpu or cuda (teacher labelling and training)
--threads THREADS CPU threads for torch and onnxruntime
--seed SEED seed for example selection and training
--data DATA file with texts (.csv, .jsonl or hf:...)
--text-field TEXT_FIELD
--fp32 use the fp32 model even if an int8 one exists
--cascade RUN send texts the student is unsure about to RUN's teacher
--log LOG with --cascade: append the texts the teacher answered (for `refresh`)
--threshold THRESHOLD
with --cascade: the student's top probability below which the teacher
answers
serve¶
Serve an exported model over http (json).
usage: tinydistill serve [-h] [--device DEVICE] [--threads THREADS] [--seed SEED] [--host HOST]
[--port PORT] [--cascade RUN] [--threshold THRESHOLD] [--log LOG]
model
positional arguments:
model directory written by `export` (run/onnx), or hf:user/name on the Hub
options:
-h, --help show this help message and exit
--device DEVICE auto, cpu or cuda (teacher labelling and training)
--threads THREADS CPU threads for torch and onnxruntime
--seed SEED seed for example selection and training
--host HOST interface to listen on (default: this machine only)
--port PORT
--cascade RUN send texts the student is unsure about to RUN's teacher
--threshold THRESHOLD
with --cascade: see `predict --help`
--log LOG with --cascade: append the texts the teacher answered (for `refresh`)
plot¶
Charts of a run as png files: training, speed and accuracy, per label, confusions, cascade.
usage: tinydistill plot [-h] [--out OUT] run
positional arguments:
run run directory
options:
-h, --help show this help message and exit
--out OUT folder for the PNG files (default: RUN/plots)
web¶
A static web page that runs the model in the browser.
usage: tinydistill web [-h] [--out OUT] [--title TITLE] [--note NOTE] model
positional arguments:
model directory written by `export` (run/onnx), or hf:user/name on the Hub
options:
-h, --help show this help message and exit
--out OUT folder to write the page and model into
--title TITLE page title (default: from the task name)
--note NOTE an extra footer line, e.g. the dataset's licence and source
synth¶
Have the teacher write unlabelled example texts.
usage: tinydistill synth [-h] [--device DEVICE] [--threads THREADS] [--seed SEED]
--per-label PER_LABEL --out OUT [--teacher TEACHER]
[--temperature TEMPERATURE] [--load-4bit]
task
positional arguments:
task task YAML file
options:
-h, --help show this help message and exit
--device DEVICE auto, cpu or cuda (teacher labelling and training)
--threads THREADS CPU threads for torch and onnxruntime
--seed SEED seed for example selection and training
--per-label PER_LABEL
texts to write per label
--out OUT JSONL file to write (usable as --data)
--teacher TEACHER the model that writes (the first if several)
--temperature TEMPERATURE
--load-4bit load the model in 4 bit (bitsandbytes, CUDA)
init¶
Write a starter task file from a dataset's labels.
usage: tinydistill init [-h] [--out OUT] [--text-field TEXT_FIELD] [--label-field LABEL_FIELD]
[--force]
data
positional arguments:
data .csv, .jsonl or hf:<dataset>[:<config>]:<split> with gold labels
options:
-h, --help show this help message and exit
--out OUT
--text-field TEXT_FIELD
--label-field LABEL_FIELD
--force overwrite an existing file
push¶
Upload an exported model to the hugging face hub, with a model card.
usage: tinydistill push [-h] --repo REPO [--private] [--license LICENSE] model
positional arguments:
model directory written by `export` (run/onnx)
options:
-h, --help show this help message and exit
--repo REPO Hub repository, e.g. user/my-classifier
--private
--license LICENSE licence identifier for the model card (check student, teacher and data)
report¶
Rebuild run/results.md from run/results.json.
usage: tinydistill report [-h] run
positional arguments:
run run directory
options:
-h, --help show this help message and exit
tune¶
Choose teacher, prompt and few-shot on a dev split.
usage: tinydistill tune [-h] [--device DEVICE] [--threads THREADS] [--seed SEED] --data DATA
[--teacher TEACHER] [--few-shot FEW_SHOT] [--prompt PROMPT]
[--dev-size DEV_SIZE] [--limit LIMIT] [--load-4bit] [--out OUT]
task
positional arguments:
task task YAML file
options:
-h, --help show this help message and exit
--device DEVICE auto, cpu or cuda (teacher labelling and training)
--threads THREADS CPU threads for torch and onnxruntime
--seed SEED seed for example selection and training
--data DATA training data with gold labels for some texts
--teacher TEACHER one or more teachers, comma-separated
--few-shot FEW_SHOT comma-separated values of K to try (0 = zero-shot)
--prompt PROMPT another task file to try (same labels)
--dev-size DEV_SIZE gold-labelled training texts to score
--limit LIMIT use only the first N training texts
--load-4bit load teachers in 4 bit (bitsandbytes, CUDA)
--out OUT writes OUT.md and OUT.json
refresh¶
Retrain run with the texts its cascade sent to the teacher (a new run directory).
usage: tinydistill refresh [-h] [--student STUDENT] [--epochs EPOCHS] [--val-share S]
[--patience PATIENCE] [--batch-size BATCH_SIZE] [--lr LR]
[--max-length MAX_LENGTH] [--gold-file GOLD_FILE] [--gold-per-class K]
[--ce-weight CE_WEIGHT] [--gold-only] [--seeds SEEDS] [--upper-bound]
[--int8] [--teacher-devices TEACHER_DEVICES] [--skip-teacher-bench]
[--reps REPS] [--teacher-reps TEACHER_REPS] [--device DEVICE]
[--threads THREADS] [--seed SEED] --log LOG --out OUT
run
positional arguments:
run run directory
options:
-h, --help show this help message and exit
--student STUDENT Hugging Face encoder, or `static` / `static:<encoder>` for a static-
embedding student (no attention: much smaller and faster; try --epochs 20
--lr 1e-3)
--epochs EPOCHS passes over the training texts (default 10)
--val-share S hold out this share of the training texts and keep the epoch with the
lowest loss on them, stopping early after --patience epochs without
improvement (e.g. 0.05; default: off, keep the last)
--patience PATIENCE epochs without improvement before stopping
--batch-size BATCH_SIZE
--lr LR
--max-length MAX_LENGTH
--gold-file GOLD_FILE
gold labels for some training texts (same text and label fields as the
task), e.g. the texts `tinydistill suggest` picked, labelled by a person;
used like --gold-per-class
--gold-per-class K also use the gold labels of K training examples per class (cross-entropy
term)
--ce-weight CE_WEIGHT
weight of that cross-entropy term
--gold-only also train the student on the K gold labels per class only, without the
teacher (a baseline)
--seeds SEEDS training seeds per student (mean and std in eval)
--upper-bound also train the student on all gold training labels, without the teacher
--int8 also write an int8 dynamically quantised model
--teacher-devices TEACHER_DEVICES
comma-separated devices to time the teacher on (default: the labelling
device)
--skip-teacher-bench report teacher quality only
--reps REPS timed runs per student benchmark at batch 1
--teacher-reps TEACHER_REPS
timed runs of the teacher (batch 1)
--device DEVICE auto, cpu or cuda (teacher labelling and training)
--threads THREADS CPU threads for torch and onnxruntime
--seed SEED seed for example selection and training
--log LOG the log written by predict/serve --cascade RUN --log FILE
--out OUT new run directory
suggest¶
Which training texts a person should label next.
usage: tinydistill suggest [-h] [--n N] [--out OUT] run
positional arguments:
run run directory
options:
-h, --help show this help message and exit
--n N how many texts
--out OUT JSONL to write, with an empty label to fill in
analyse¶
Where the teacher and student go wrong (run/analysis.html).
usage: tinydistill analyse [-h] [--top TOP] run
positional arguments:
run run directory
options:
-h, --help show this help message and exit
--top TOP how many confusions and disagreements to list
status¶
How far a run is, and whether it is still running.
usage: tinydistill status [-h] run
positional arguments:
run run directory
options:
-h, --help show this help message and exit
compare¶
One markdown table comparing several runs (e.g. student sizes).
usage: tinydistill compare [-h] runs [runs ...]
positional arguments:
runs run directories or results.json files
options:
-h, --help show this help message and exit
env¶
Versions and hardware, for bug reports.
usage: tinydistill env [-h]
options:
-h, --help show this help message and exit
run¶
Label, train, export and eval.
usage: tinydistill run [-h] --data DATA [--test TEST] --out OUT [--teacher TEACHER] [--load-4bit]
[--cache-dir CACHE_DIR] [--vllm-url VLLM_URL] [--api-url API_URL]
[--api-key-env VAR] [--shortlist K] [--shortlist-model SHORTLIST_MODEL]
[--shortlist-keep IDS] [--few-shot K] [--teacher-tokens TEACHER_TOKENS]
[--limit LIMIT] [--test-limit TEST_LIMIT] [--student STUDENT]
[--epochs EPOCHS] [--val-share S] [--patience PATIENCE]
[--batch-size BATCH_SIZE] [--lr LR] [--max-length MAX_LENGTH]
[--gold-file GOLD_FILE] [--gold-per-class K] [--ce-weight CE_WEIGHT]
[--gold-only] [--seeds SEEDS] [--upper-bound] [--int8]
[--teacher-devices TEACHER_DEVICES] [--skip-teacher-bench] [--reps REPS]
[--teacher-reps TEACHER_REPS] [--device DEVICE] [--threads THREADS]
[--seed SEED]
task
positional arguments:
task task YAML file
options:
-h, --help show this help message and exit
--data DATA unlabelled texts: .csv, .jsonl or hf:<dataset>[:<config>]:<split>
--test TEST optional gold-labelled test set, same formats
--out OUT run directory
--teacher TEACHER Hugging Face causal LM; several, comma-separated, are ensembled
--load-4bit load the teacher in 4 bit (bitsandbytes, CUDA)
--cache-dir CACHE_DIR
where teacher scores are cached (default: the run directory); runs that
share it label each text once
--vllm-url VLLM_URL score with this vLLM server (OpenAI API base, e.g. http://host:8000/v1),
not a local model
--api-url API_URL score with a hosted OpenAI-compatible chat API (base URL, e.g.
https://api.openai.com/v1; --teacher names the model there): the labels
are shown as options A, B, C, ... and scored by the answer letter's log-
probability; single-label tasks with at most 20 labels
--api-key-env VAR environment variable holding the API key for --api-url (default
OPENAI_API_KEY; unset: no key is sent, as for a local server)
--shortlist K score only the K labels most similar to each text (for tasks with very
many labels)
--shortlist-model SHORTLIST_MODEL
sentence-embedding model for --shortlist (default: MiniLM)
--shortlist-keep IDS label ids the teacher always scores besides the shortlist, comma-
separated: catch-all labels such as other or out of scope, whose names
resemble no particular text
--few-shot K show K gold training examples per class in the teacher's prompt (needs
{examples} in it)
--teacher-tokens TEACHER_TOKENS
tokens per teacher forward pass; short texts with few labels are packed
together
--limit LIMIT use only the first N training texts
--test-limit TEST_LIMIT
use only the first N test texts
--student STUDENT Hugging Face encoder, or `static` / `static:<encoder>` for a static-
embedding student (no attention: much smaller and faster; try --epochs 20
--lr 1e-3)
--epochs EPOCHS passes over the training texts (default 10)
--val-share S hold out this share of the training texts and keep the epoch with the
lowest loss on them, stopping early after --patience epochs without
improvement (e.g. 0.05; default: off, keep the last)
--patience PATIENCE epochs without improvement before stopping
--batch-size BATCH_SIZE
--lr LR
--max-length MAX_LENGTH
--gold-file GOLD_FILE
gold labels for some training texts (same text and label fields as the
task), e.g. the texts `tinydistill suggest` picked, labelled by a person;
used like --gold-per-class
--gold-per-class K also use the gold labels of K training examples per class (cross-entropy
term)
--ce-weight CE_WEIGHT
weight of that cross-entropy term
--gold-only also train the student on the K gold labels per class only, without the
teacher (a baseline)
--seeds SEEDS training seeds per student (mean and std in eval)
--upper-bound also train the student on all gold training labels, without the teacher
--int8 also write an int8 dynamically quantised model
--teacher-devices TEACHER_DEVICES
comma-separated devices to time the teacher on (default: the labelling
device)
--skip-teacher-bench report teacher quality only
--reps REPS timed runs per student benchmark at batch 1
--teacher-reps TEACHER_REPS
timed runs of the teacher (batch 1)
--device DEVICE auto, cpu or cuda (teacher labelling and training)
--threads THREADS CPU threads for torch and onnxruntime
--seed SEED seed for example selection and training