# Training

A training job fine-tunes a base model on your dataset and produces a new model you can deploy or continue training from.

## Start a job

```python
job = model.train(
    dataset,
    hyperparameters={
        "METHOD": "lora",
        "NUM_EPOCHS": 3,
        "LEARNING_RATE": 2e-5,
        "LORA_R": 16,
    },
)
job.monitor()
```

Fleet auto-selects the most cost-effective GPU that fits your model. To target a specific tier, pass `hardware`:

```python
job = model.train(dataset, hardware="fleet:pro", hyperparameters={...})
```

The `dataset` argument accepts a `Dataset` object, a dataset ID string, or a path to a local `.jsonl` file — Fleet uploads it automatically if needed. All hyperparameters are optional; defaults are listed below.

## Training methods

| Method | When to use |
|--------|-------------|
| `full` | Maximum fidelity. Trains all weights. Requires more GPU memory. |
| `lora` | Fast, memory-efficient adapters. Best default choice. |
| `qlora` | 4-bit quantised LoRA. Largest models on smallest GPUs. |
| `dpo` | Direct preference optimisation. Aligning a model to preferred responses. |

Set the method with `"METHOD": "lora"` in hyperparameters. Defaults to `full`.

DPO requires a dataset with `prompt`, `chosen`, and `rejected` fields per row:

```json
{"prompt": "What is the capital of France?", "chosen": "Paris.", "rejected": "London."}
```

## Hyperparameters

| Key | Default | Description |
|-----|---------|-------------|
| `METHOD` | `full` | Training method: `full`, `lora`, `qlora`, `dpo` |
| `NUM_EPOCHS` | `3` | Number of training epochs |
| `LEARNING_RATE` | `2e-5` | Optimizer learning rate |
| `BATCH_SIZE` | `4` | Per-device training batch size |
| `GRAD_ACCUM` | `1` | Gradient accumulation steps |
| `WARMUP_RATIO` | `0.0` | Fraction of steps used for LR warmup |
| `LR_SCHEDULER` | `linear` | Schedule: `linear`, `cosine`, `cosine_with_restarts`, `constant` |
| `OPTIMIZER` | `adamw` | Optimizer: `adamw`, `adafactor`, `adamw8bit` |
| `WEIGHT_DECAY` | `0.0` | L2 regularisation weight decay |
| `MAX_STEPS` | none | Override epoch count with a fixed step limit |
| `SAVE_STEPS` | `500` | Save a checkpoint every N steps |
| `PRECISION` | `auto` | Mixed precision: `auto`, `fp16`, `bf16`, `fp32` |
| `LORA_R` | `8` | LoRA rank (LoRA/QLoRA only) |
| `LORA_ALPHA` | `16` | LoRA alpha scaling (LoRA/QLoRA only) |
| `LORA_DROPOUT` | `0.05` | LoRA dropout (LoRA/QLoRA only) |
| `LORA_TARGET_MODULES` | auto | Comma-separated target modules, e.g. `q_proj,v_proj` |
| `LORA_BIAS` | `none` | Bias training: `none`, `all`, `lora_only` |
| `BNB_QUANT_TYPE` | `nf4` | Quantisation type: `nf4`, `fp4` (QLoRA only) |
| `USE_UNSLOTH` | `0` | Set to `1` to enable Unsloth acceleration where supported |
| `DPO_BETA` | `0.1` | KL penalty coefficient (DPO only) |
| `DPO_LOSS` | `sigmoid` | Loss type: `sigmoid`, `hinge`, `ipo` (DPO only) |

## Hardware tiers

| Tier | GPU memory | Best for |
|------|-----------|----------|
| `fleet:micro` | 6–12 GB | Small models, quick tests |
| `fleet:economy` | 16–20 GB | GPT-2, small Llama variants |
| `fleet:standard` | 24–32 GB | Most fine-tunes (default) |
| `fleet:pro` | 40–48 GB | Large models, longer context |
| `fleet:ultra` | 80 GB+ | Frontier models |

Omit `hardware` entirely and Fleet picks the most cost-effective GPU that fits your model automatically. Specifying a tier constrains selection to that tier — useful if you need a specific memory size or want predictable placement.

Training is billed by the hour at the actual GPU rate. Pricing varies within a tier depending on which GPU is assigned. Check live rates:

```python
client.hardware()
```

## Monitoring progress

`job.monitor()` blocks until the job finishes and streams a live progress bar with loss and learning rate:

```python
job.monitor()
```

To also print every log line as it arrives:

```python
job.monitor(verbose=True)
```

To fetch logs manually:

```python
for line in job.logs():
    print(line)
```

## Training metrics

Fleet records loss and learning rate during training. Metrics are posted every 10 steps (configurable via the `METRICS_STEPS` hyperparameter) and stored server-side so you can query them at any time, even after the job finishes.

```python
data = job.metrics()
print(data["latest"])
# {'step': 100, 'loss': 2.34, 'lr': 1.38e-05, 'val_loss': None}

for point in data["metrics"]:
    print(point["step"], point["loss"])
```

The `metrics` array is the full time-series ordered by step; feed it directly into a chart. `val_loss` is populated when validation runs (same step number as the corresponding training point).

Control how often metrics are posted:

```python
job = model.train(dataset, hyperparameters={"METRICS_STEPS": 5})
```

From the CLI:

```bash
fleet jobs metrics <job_id>
```

## Recovering a job

If your script exits while a job is running, reconnect by job ID:

```python
job = client.jobs.get("job_abc123")
job.monitor()  # reattaches and continues watching
```

## Retrieving the output model

Once complete, register the fine-tuned model and get a `Model` object back:

```python
fine_tuned = job.model()
print(fine_tuned.id)
```

## Checkpoints

If `SAVE_STEPS` is set, Fleet saves intermediate checkpoints during training. List them:

```bash
fleet checkpoints <job_id>
```

Each checkpoint is a model you can retrieve, deploy, or continue training from.

Fleet enforces a rolling checkpoint limit to control storage costs. The default is 1. When a new checkpoint is saved, the oldest beyond the limit is deleted automatically from both storage and your model list. Adjust the limit:

```bash
fleet settings set max_checkpoints 5
```

Or via the SDK:

```python
client.update_settings(max_checkpoints=5)
```

By default, all checkpoints are deleted automatically when a job completes successfully. The final output model is the canonical artifact. To retain checkpoints after completion:

```bash
fleet settings set prune_checkpoints_on_completion false
```

```python
client.update_settings(prune_checkpoints_on_completion=False)
```

## Evaluating a model

Run a forward-pass evaluation on a held-out dataset to measure how well your model performs after training. No weights are updated — Fleet loads the model, runs every row through a forward pass, and returns loss and perplexity.

```python
job = model.evaluate("./held-out.jsonl")
job.monitor()

results = job.eval_results()
print(f"Loss:        {results['loss']:.4f}")
print(f"Perplexity:  {results['perplexity']:.4f}")
print(f"Samples:     {results['num_samples']}")
```

The dataset must be a `.jsonl` file in the same format as your training data. Use a held-out split that was not included in training for a meaningful result.

Evaluation jobs use the same GPU selection and billing as training — you are charged only for the time the forward pass takes, typically 10–20% of the equivalent training run.

## Cancelling a job

```python
job.cancel()
```

```bash
fleet jobs cancel <job_id>
```

Cancellation terminates the GPU immediately. You are charged only for time used up to that point.