Fine-Tune LFM3, a Large Language Model, into a Decision Model with Unsloth: Lead Qualification with Claude Code over MCP

This page trains the second model of the tutorial, a reply router. It reads a prospect’s reply to a first sales email, picks one of 11 intents, and says whether the reply calls for a next step.

On 124 unseen test replies, it reaches a macro-F1 of 0.975, against 0.705 for a TF-IDF classifier. The page also shows the step that made the difference: the first version trained for only 20 optimizer steps and scored 0.778 (its metrics).

The notebook is 02-reply-routing.ipynb, and the models are the reply router and its merged copy.

Step 12: Run the reply notebook

The environment is the one of page 1: the same machine, the same requirements-experiment.txt and the same pinned base model. The reply notebook loads its own copy of the base model and uses no object from page 1. Run it from the repository root:

jupyter nbconvert --to notebook --execute --inplace --ExecutePreprocessor.timeout=-1 notebooks/02-reply-routing.ipynb

The whole notebook ran in 1,962 s, the four candidate trainings and the Hub upload included. Set HF_USERNAME to your own account in the first cell, or set PUBLISH = False.

Run options

import os
import time
from pathlib import Path

if Path.cwd().name == "notebooks":
    os.chdir(Path.cwd().parent)

HF_USERNAME = "raoofaltaher"
PUBLISH = True
LOAD_IN_4BIT = True
TIMINGS = {}

print("Working directory:", Path.cwd())

Check the GPU

import platform
import torch

print("Python:", platform.python_version())
print("PyTorch:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())

if torch.cuda.is_available():
    print("GPU:", torch.cuda.get_device_name(0))
    print("CUDA runtime:", torch.version.cuda)
else:
    raise RuntimeError("A supported GPU runtime is required for this training run.")

Imports and the base-model pin

import json
from collections import Counter

import numpy as np
import pandas as pd
from datasets import Dataset
from sklearn.model_selection import StratifiedGroupKFold
from unsloth import DecisionTrainer, FastDecisionModel, is_bfloat16_supported
from transformers import TrainingArguments

MODEL_ID = "LiquidAI/LFM3-1.7B"
MODEL_REVISION = "ac2586a34f1a1c03329dbdcdefabd8b6478ddaa7"
print("Model:", MODEL_ID, "pinned revision:", MODEL_REVISION)


def to_decision_dataset(frame, builder):
    rows = [
        builder(row)
        for row in frame.to_dict(orient="records")
    ]
    return Dataset.from_list(rows)

Step 13: The synthetic reply dataset

No public dataset holds B2B sales replies with intent labels, and support tickets or sentiment data are not sales replies. So the repository ships its own synthetic dataset: 617 English replies to one fictional first-touch email, 51 to 61 per intent, every row marked as synthetic. It is a teaching dataset, not a production one.

Intent	Meaning
interested	Expresses interest without asking for a meeting
meeting_requested	Asks to schedule or discuss a meeting
more_information	Asks for details, pricing or documentation
referral	Points to another relevant person
not_now	Asks to revisit the topic later
objection	Raises a concern or a reason not to proceed
wrong_person	Says they are not responsible for the topic
out_of_office	States an absence or a return date
not_interested	Declines further engagement
opt_out	Asks to stop marketing or to unsubscribe
ambiguous	Does not support a reliable classification

The dataset was built in four steps, all recorded in data/replies:

1. Seeds. Eleven seed replies, one per intent, each in its own scenario group. A scenario group holds paraphrases of one underlying situation.
2. Generation. Claude Opus 5.5 generated 624 more replies from the generation brief, one intent per batch.
3. Blind relabel. A separate agent on Claude Sonnet labelled every candidate from its text and the intent definitions alone, under shuffled ids. It agreed with the intended label on 97.2% of the rows.
4. Automated checks. tools/check_replies.py dropped the 18 rows where the two labels disagreed, checked the schema, the taxonomy, duplicates and the group counts, and wrote the kept rows with their SHA-256 and a report.

Seventeen of the dropped rows were declines with a reason, such as “We’ve already chosen another provider”, which the relabel called objections. No person reviewed the rows. The notebook refuses a file whose SHA-256 differs from the published one:

Load and validate the reply data

REPLY_INTENTS = [
    "interested",
    "meeting_requested",
    "more_information",
    "referral",
    "not_now",
    "objection",
    "wrong_person",
    "out_of_office",
    "not_interested",
    "opt_out",
    "ambiguous",
]

reply_df = pd.read_csv("data/replies/reply_examples.csv")

required_reply_columns = {
    "reply_id",
    "scenario_group",
    "outreach_context",
    "reply_text",
    "reply_intent",
    "is_synthetic",
}

missing_reply_columns = required_reply_columns - set(reply_df.columns)
if missing_reply_columns:
    raise ValueError(f"Missing reply columns: {sorted(missing_reply_columns)}")

reply_df["is_synthetic"] = (
    reply_df["is_synthetic"]
    .astype(str)
    .str.strip()
    .str.lower()
    .eq("true")
)

reply_df["reply_intent"] = (
    reply_df["reply_intent"]
    .astype(str)
    .str.strip()
)

if not reply_df["is_synthetic"].all():
    raise ValueError("This experiment accepts only explicitly synthetic reply rows.")

if reply_df["reply_id"].duplicated().any():
    raise ValueError("reply_id values must be unique.")

normalized_reply_text = (
    reply_df["reply_text"]
    .astype(str)
    .str.strip()
    .str.casefold()
)

if normalized_reply_text.duplicated().any():
    raise ValueError(
        "Exact duplicate reply text exists. Deduplicate or review it before splitting."
    )

required_text_columns = [
    "reply_id",
    "scenario_group",
    "outreach_context",
    "reply_text",
]

if reply_df[required_text_columns].isna().any().any():
    raise ValueError("Reply IDs, groups, context and text cannot be null.")

if (
    reply_df[required_text_columns]
    .astype(str)
    .apply(lambda column: column.str.strip().eq("").any())
    .any()
):
    raise ValueError("Reply IDs, groups, context and text cannot be blank.")

if not set(reply_df["reply_intent"]).issubset(set(REPLY_INTENTS)):
    raise ValueError("The dataset contains an unknown reply_intent.")

if reply_df.groupby("scenario_group")["reply_intent"].nunique().gt(1).any():
    raise ValueError("A scenario_group cannot contain multiple reply_intent labels.")

if len(reply_df) < 500:
    raise ValueError("The reply dataset must hold at least 500 rows.")

class_counts = reply_df["reply_intent"].value_counts()
print(class_counts)

missing_classes = set(REPLY_INTENTS) - set(class_counts.index)
if missing_classes:
    raise ValueError(f"Missing reply classes: {sorted(missing_classes)}")

if class_counts.min() < 20:
    raise ValueError("Add more examples to the under-represented classes.")

group_counts = reply_df.groupby("reply_intent")["scenario_group"].nunique()
if group_counts.min() < 10:
    raise ValueError(
        "Each reply intent needs at least ten independent scenario groups "
        "for the five-fold grouped split."
    )

from hashlib import sha256
from pathlib import Path

reply_dataset_hash = sha256(
    Path("data/replies/reply_examples.csv").read_bytes()
).hexdigest()
published_hash = Path("data/replies/reply_examples.sha256").read_text(encoding="utf-8").split()[0]

if reply_dataset_hash != published_hash:
    raise ValueError("reply_examples.csv does not match its published SHA-256.")

print("Reply dataset SHA-256 verified:", reply_dataset_hash)

reply_intent
interested           61
referral             61
objection            61
wrong_person         61
ambiguous            60
not_interested       56
more_information     53
meeting_requested    51
not_now              51
out_of_office        51
opt_out              51
Name: count, dtype: int64
Reply dataset SHA-256 verified: c57a26d99d05b29b3ce63c5e627d2a79dd03ed78d3bd35ce1afb1f07fca72439

Step 14: Write the questions and records

Each record holds a state with the outreach context and the reply, two questions and the gold answers. The intent is a choice question; the follow-up is a noul question whose label comes from a written rule: every intent except not_interested and opt_out calls for a next step. “Follow-up recommended” never means “send another email now”.

REPLY_QUESTIONS = {
    "reply_intent": {
        "type": "choice",
        "instructions": (
            "Classify the recipient's reply using the defined intent taxonomy. "
            "Do not infer interest that the recipient did not express."
        ),
        "criteria": {
            "interested": "The recipient expresses interest without requesting a meeting.",
            "meeting_requested": "The recipient asks to schedule or discuss a meeting.",
            "more_information": "The recipient requests details, pricing or documentation.",
            "referral": "The recipient points to another relevant person.",
            "not_now": "The recipient asks to revisit the topic later.",
            "objection": "The recipient raises a concern or reason not to proceed.",
            "wrong_person": "The recipient says they are not responsible for this topic.",
            "out_of_office": "The recipient indicates an absence or return date.",
            "not_interested": "The recipient declines further engagement.",
            "opt_out": "The recipient asks to stop marketing or unsubscribe.",
            "ambiguous": "The reply does not support a reliable intent classification.",
        },
    },
    "follow_up_recommended": {
        "type": "noul",
        "instructions": (
            "Does this reply call for an appropriate next step? "
            "A next step may be a handoff, requested information, a pause "
            "or human review. It does not mean an automatic marketing email."
        ),
    },
}

NO_FOLLOW_UP_INTENTS = {
    "not_interested",
    "opt_out",
}

def make_reply_record(row):
    intent = row["reply_intent"]

    state = {
        "task": "B2B sales reply assessment",
        "outreach_context": row["outreach_context"],
        "recipient_reply": row["reply_text"],
    }

    return {
        "state": json.dumps(state, sort_keys=True, ensure_ascii=False),
        "questions": REPLY_QUESTIONS,
        "gold": {
            "reply_intent": intent,
            "follow_up_recommended": intent not in NO_FOLLOW_UP_INTENTS,
        },
    }

Step 15: Split by scenario group

Paraphrases of one situation must not land on both sides of an evaluation boundary. The notebook splits by scenario_group with StratifiedGroupKFold, then splits whole groups out of the training fold for calibration.

Make the group-disjoint folds

reply_splitter = StratifiedGroupKFold(
    n_splits=5,
    shuffle=True,
    random_state=3407,
)

reply_folds = list(
    reply_splitter.split(
        reply_df,
        y=reply_df["reply_intent"],
        groups=reply_df["scenario_group"],
    )
)

reply_test_indices = reply_folds[0][1]
reply_validation_indices = reply_folds[1][1]
reply_train_indices = np.concatenate(
    [reply_folds[2][1], reply_folds[3][1], reply_folds[4][1]]
)

reply_train_df = reply_df.iloc[reply_train_indices].reset_index(drop=True)
reply_validation_df = reply_df.iloc[reply_validation_indices].reset_index(drop=True)
reply_test_df = reply_df.iloc[reply_test_indices].reset_index(drop=True)

def scenario_groups(frame):
    return set(frame["scenario_group"])

assert scenario_groups(reply_train_df).isdisjoint(scenario_groups(reply_validation_df))
assert scenario_groups(reply_train_df).isdisjoint(scenario_groups(reply_test_df))
assert scenario_groups(reply_validation_df).isdisjoint(scenario_groups(reply_test_df))

for name, frame in [
    ("train", reply_train_df),
    ("validation", reply_validation_df),
    ("test", reply_test_df),
]:
    missing = set(REPLY_INTENTS) - set(frame["reply_intent"].unique())
    if missing:
        raise ValueError(
            f"{name} split is missing reply classes: {sorted(missing)}. "
            "Add independent scenario groups or revise the split."
        )

    print(name, len(frame))
    print(frame["reply_intent"].value_counts())

train 369
[... 13 lines cut ...]
validation 124
[... 13 lines cut ...]
test 124
[... 13 lines cut ...]

Split a group-disjoint calibration holdout

reply_calibration_splitter = StratifiedGroupKFold(
    n_splits=5,
    shuffle=True,
    random_state=3408,
)

reply_fit_indices, reply_calibration_indices = next(
    reply_calibration_splitter.split(
        reply_train_df,
        y=reply_train_df["reply_intent"],
        groups=reply_train_df["scenario_group"],
    )
)

reply_fit_df = reply_train_df.iloc[reply_fit_indices].reset_index(drop=True)
reply_calibration_df = reply_train_df.iloc[reply_calibration_indices].reset_index(drop=True)

assert scenario_groups(reply_fit_df).isdisjoint(
    scenario_groups(reply_calibration_df)
)

# The approximate overall proportions are 48% fit, 12% calibration,
# 20% validation and 20% test; grouped row counts will vary.

for name, frame in [
    ("optimizer training", reply_fit_df),
    ("calibration", reply_calibration_df),
]:
    missing = set(REPLY_INTENTS) - set(frame["reply_intent"].unique())
    if missing:
        raise ValueError(
            f"{name} split is missing reply classes: {sorted(missing)}. "
            "Add independent scenario groups or revise the split."
        )

    print(name, len(frame))

optimizer training 296
calibration 73

The result is 296 optimizer-training, 73 calibration, 124 validation and 124 test replies, with 10 to 13 test replies per intent.

Step 16: The baseline and the base model

The comparison is a TF-IDF vectorizer with a logistic regression (TfidfVectorizer). It sees only the reply text and trains on the same 296 rows.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, f1_score
from sklearn.pipeline import Pipeline

reply_text_baseline = Pipeline(
    steps=[
        (
            "tfidf",
            TfidfVectorizer(
                ngram_range=(1, 2),
                min_df=2,
                max_features=50000,
                sublinear_tf=True,
            ),
        ),
        (
            "classifier",
            LogisticRegression(
                max_iter=2000,
                class_weight="balanced",
            ),
        ),
    ]
)

new_reply_model loads the pinned base model in 4 bits and attaches LoRA adapters with lora_alpha equal to the rank (recommended settings). The rank is a parameter, because Step 17 compares two ranks. build_dataset then converts each partition into Unsloth’s items and reports skipped or truncated decisions:

Load the base model and build the items

def new_reply_model(lora_rank=16):
    decision_model, decision_tokenizer = FastDecisionModel.from_pretrained(
        model_name=MODEL_ID,
        revision=MODEL_REVISION,
        trust_remote_code=True,
        max_seq_length=2048,
        load_in_4bit=LOAD_IN_4BIT,
    )

    decision_model = FastDecisionModel.get_peft_model(
        decision_model,
        r=lora_rank,
        lora_alpha=lora_rank,
        lora_dropout=0.0,
        use_gradient_checkpointing="unsloth",
        random_state=3407,
    )
    return decision_model, decision_tokenizer


reply_model, reply_tokenizer = new_reply_model()

reply_train_raw = to_decision_dataset(reply_fit_df, make_reply_record)
reply_calibration_raw = to_decision_dataset(reply_calibration_df, make_reply_record)
reply_validation_raw = to_decision_dataset(reply_validation_df, make_reply_record)
reply_test_raw = to_decision_dataset(reply_test_df, make_reply_record)

reply_train_items_all, reply_train_report = FastDecisionModel.build_dataset(
    reply_train_raw,
    reply_tokenizer,
    reply_model,
)

reply_calibration_items, reply_calibration_report = FastDecisionModel.build_dataset(
    reply_calibration_raw,
    reply_tokenizer,
    reply_model,
)

reply_validation_items, reply_validation_report = FastDecisionModel.build_dataset(
    reply_validation_raw,
    reply_tokenizer,
    reply_model,
)

reply_test_items, reply_test_report = FastDecisionModel.build_dataset(
    reply_test_raw,
    reply_tokenizer,
    reply_model,
)

for name, report in [
    ("train", reply_train_report),
    ("calibration", reply_calibration_report),
    ("validation", reply_validation_report),
    ("test", reply_test_report),
]:
    print(name, report)
    if report["skipped"]:
        raise ValueError(
            f"{name} reply dataset contains skipped decisions: {report['reason']}"
        )

[... 14 lines cut ...]
train {'total': 592, 'skipped': 0, 'reason': None, 'truncated': 0}
calibration {'total': 146, 'skipped': 0, 'reason': None, 'truncated': 0}
validation {'total': 248, 'skipped': 0, 'reason': None, 'truncated': 0}
test {'total': 248, 'skipped': 0, 'reason': None, 'truncated': 0}

Fit the baseline and score the untrained head

reply_train_items = reply_train_items_all

print("Reply optimizer-training rows:", len(reply_train_items))
print("Reply calibration rows:", len(reply_calibration_items))

reply_calibration_counts = Counter(
    question_type
    for item in reply_calibration_items
    for question_type in item["qtypes"]
)

if (
    len(reply_calibration_counts) != len(REPLY_QUESTIONS)
    or min(reply_calibration_counts.values()) < 10
):
    raise ValueError(
        "Provide at least ten calibration decisions for both reply question types. "
        f"Observed counts: {dict(reply_calibration_counts)}"
    )

# Fit the classical baseline on exactly the same scenario groups as Unsloth.
reply_text_baseline.fit(
    reply_fit_df["reply_text"],
    reply_fit_df["reply_intent"],
)

reply_validation_text_predictions = reply_text_baseline.predict(
    reply_validation_df["reply_text"]
)

print(
    classification_report(
        reply_validation_df["reply_intent"],
        reply_validation_text_predictions,
        labels=REPLY_INTENTS,
        zero_division=0,
    )
)

print(
    "TF-IDF validation macro-F1:",
    f1_score(
        reply_validation_df["reply_intent"],
        reply_validation_text_predictions,
        labels=REPLY_INTENTS,
        average="macro",
        zero_division=0,
    ),
)

reply_baseline = FastDecisionModel.evaluate(
    reply_model,
    reply_tokenizer,
    reply_validation_items,
)

print("Unfine-tuned reply baseline:", reply_baseline)

[... 20 lines cut ...]
TF-IDF validation macro-F1: 0.5652430331292474
Unfine-tuned reply baseline: {'accuracy': 0.13306451612903225, 'ece': 0.16774805512038932, 'loss': 1.5470689388532792, 'record_accuracy': 0.0}

The untrained decision head gets no reply right on both questions. The baseline reaches a validation macro-F1 of 0.565.

Step 17: Choose the training setup on validation data

This is the step that made the model. The first version of this notebook copied the lead model’s settings: two epochs with a batch of 8 and 4 accumulation steps. On 296 replies, that is 20 optimizer updates, and the model reached a validation macro-F1 of only 0.68. Unsloth’s guide says that a small dataset needs more epochs (recommended settings), and its own examples run 60 to 240 steps.

The fix is a small, pre-registered grid: 3 or 6 epochs, LoRA rank 16 or 64, with a batch of 8 and no accumulation, so that each epoch gives 37 updates. Each candidate trains from a fresh copy of the base model, calibrates on the holdout and scores validation macro-F1. The winner is chosen on validation only; the test set is scored once, in Step 19. predict_replies is the helper that asks the model both questions for every row:

The prediction helper

from sklearn.metrics import confusion_matrix, f1_score

def predict_replies(frame, decision_model, decision_tokenizer):
    predictions = []

    for row in frame.to_dict(orient="records"):
        example = make_reply_record(row)

        result = FastDecisionModel.predict(
            decision_model,
            decision_tokenizer,
            example["state"],
            REPLY_QUESTIONS,
        )

        predictions.append(
            {
                "reply_intent": result["reply_intent"]["answer"],
                "reply_intent_confidence": result["reply_intent"]["confidence"],
                "follow_up_recommended": result["follow_up_recommended"]["answer"],
            }
        )

    return pd.DataFrame(predictions)

REPLY_CANDIDATES = [
    {"epochs": 3, "lora_rank": 16},
    {"epochs": 6, "lora_rank": 16},
    {"epochs": 3, "lora_rank": 64},
    {"epochs": 6, "lora_rank": 64},
]

def train_reply_model(num_epochs, lora_rank):
    decision_model, decision_tokenizer = new_reply_model(lora_rank)

    trainer = DecisionTrainer(
        model=decision_model,
        processing_class=decision_tokenizer,
        train_dataset=reply_train_items,
        eval_dataset=reply_validation_items,
        head_learning_rate=1e-4,
        args=TrainingArguments(
            per_device_train_batch_size=8,
            gradient_accumulation_steps=1,
            warmup_steps=10,
            num_train_epochs=num_epochs,
            learning_rate=2e-4,
            lr_scheduler_type="cosine",
            weight_decay=0.01,
            bf16=is_bfloat16_supported(),
            fp16=not is_bfloat16_supported(),
            eval_strategy="epoch",
            logging_steps=5,
            save_strategy="no",
            output_dir=f"outputs/reply-decision-{num_epochs}-epochs-r{lora_rank}",
            report_to="none",
            seed=3407,
        ),
    )

    started = time.perf_counter()
    trainer.train()
    training_seconds = round(time.perf_counter() - started, 1)

    calibration = FastDecisionModel.calibrate(
        decision_model,
        decision_tokenizer,
        reply_calibration_items,
    )

    FastDecisionModel.for_inference(decision_model)
    validation_predictions = predict_replies(reply_validation_df, decision_model, decision_tokenizer)

    return decision_model, decision_tokenizer, {
        "epochs": num_epochs,
        "lora_rank": lora_rank,
        "optimizer_steps": trainer.state.global_step,
        "training_seconds": training_seconds,
        "validation_macro_f1": float(
            f1_score(
                reply_validation_df["reply_intent"],
                validation_predictions["reply_intent"],
                labels=REPLY_INTENTS,
                average="macro",
                zero_division=0,
            )
        ),
        "calibration": calibration,
        "log_history": trainer.state.log_history,
    }

import gc

del reply_model, reply_tokenizer
gc.collect()
torch.cuda.empty_cache()

reply_selection = []
reply_best = None

for settings in REPLY_CANDIDATES:
    candidate = train_reply_model(settings["epochs"], settings["lora_rank"])
    reply_selection.append(candidate[2])
    print(
        f"{settings['epochs']} epochs, LoRA rank {settings['lora_rank']}: "
        f"{candidate[2]['optimizer_steps']} optimizer steps, "
        f"{candidate[2]['training_seconds']} s, "
        f"validation macro-F1 {candidate[2]['validation_macro_f1']:.4f}"
    )
    if reply_best is None or candidate[2]["validation_macro_f1"] > reply_best[2]["validation_macro_f1"]:
        reply_best = candidate
    del candidate
    gc.collect()
    torch.cuda.empty_cache()

reply_model, reply_tokenizer, reply_chosen = reply_best
del reply_best

reply_calibration_metrics = reply_chosen["calibration"]
TIMINGS["reply_training"] = reply_chosen["training_seconds"]
TIMINGS["reply_epoch_selection"] = round(sum(result["training_seconds"] for result in reply_selection), 1)
peak_vram_gb = round(torch.cuda.max_memory_allocated() / 1024**3, 2)

print("Chosen on validation macro-F1:", reply_chosen["epochs"], "epochs, LoRA rank", reply_chosen["lora_rank"])
print("Calibration of the chosen model:", reply_calibration_metrics)
print("Peak VRAM GB:", peak_vram_gb)

[... 15 lines cut ...]
3 epochs, LoRA rank 16: 111 optimizer steps, 153.8 s, validation macro-F1 0.8182
[... 13 lines cut ...]
6 epochs, LoRA rank 16: 222 optimizer steps, 201.7 s, validation macro-F1 0.9306
[... 13 lines cut ...]
3 epochs, LoRA rank 64: 111 optimizer steps, 148.5 s, validation macro-F1 0.8777
[... 13 lines cut ...]
6 epochs, LoRA rank 64: 222 optimizer steps, 207.0 s, validation macro-F1 0.9453
Chosen on validation macro-F1: 6 epochs, LoRA rank 64
Calibration of the chosen model: {'accuracy': 0.9657534246575342, 'ece': 0.02417528955903771, 'loss': 0.07532362670451807, 'record_accuracy': 0.9452054794520548, 'fitted_types': [0, 2]}
Peak VRAM GB: 4.6

Six epochs beat three at both ranks, and rank 64 beats rank 16 at both epoch counts, so 6 epochs at rank 64 wins with 0.945. The loss curves alone do not pick the winner: the three-epoch runs also end near zero, yet they score lower on validation (0.818 and 0.878). More passes over the data helped the held-out score, not the training fit. The six-epoch runs fit their training replies completely, and the validation score, not the training loss, says whether that is overfitting. Here it is not.

Step 18: Calibrate and evaluate on validation

Each candidate was calibrated on the group-disjoint holdout inside train_reply_model. Unsloth’s evaluator gives the decision metrics over both questions:

reply_validation_metrics = FastDecisionModel.evaluate(
    reply_model,
    reply_tokenizer,
    reply_validation_items,
)

print(reply_validation_metrics)

{'accuracy': 0.967741935483871, 'ece': 0.019779227553836796, 'loss': 0.10463338459266519, 'record_accuracy': 0.9354838709677419}

Per intent, with the follow-up question and the confusion matrix:

_predict_start = time.perf_counter()

Per-intent validation metrics

FastDecisionModel.for_inference(reply_model)

reply_validation_predictions = predict_replies(
    reply_validation_df,
    reply_model,
    reply_tokenizer,
)

print(
    classification_report(
        reply_validation_df["reply_intent"],
        reply_validation_predictions["reply_intent"],
        labels=REPLY_INTENTS,
        zero_division=0,
    )
)

validation_follow_up_labels = reply_validation_df["reply_intent"].map(
    lambda intent: intent not in NO_FOLLOW_UP_INTENTS
)

print(
    "Follow-up decision metrics:",
    classification_report(
        validation_follow_up_labels,
        reply_validation_predictions["follow_up_recommended"],
        labels=[False, True],
        zero_division=0,
    ),
)

print(
    "Macro-F1:",
    f1_score(
        reply_validation_df["reply_intent"],
        reply_validation_predictions["reply_intent"],
        labels=REPLY_INTENTS,
        average="macro",
        zero_division=0,
    ),
)

print(
    confusion_matrix(
        reply_validation_df["reply_intent"],
        reply_validation_predictions["reply_intent"],
        labels=REPLY_INTENTS,
    )
)

                   precision    recall  f1-score   support

       interested       0.91      0.83      0.87        12
meeting_requested       1.00      1.00      1.00        10
 more_information       1.00      0.90      0.95        10
         referral       0.92      1.00      0.96        12
          not_now       0.91      0.91      0.91        11
        objection       1.00      1.00      1.00        13
     wrong_person       1.00      0.83      0.91        12
    out_of_office       0.92      1.00      0.96        11
   not_interested       1.00      1.00      1.00        11
          opt_out       1.00      1.00      1.00        10
        ambiguous       0.79      0.92      0.85        12

         accuracy                           0.94       124
        macro avg       0.95      0.94      0.95       124
     weighted avg       0.95      0.94      0.94       124
[... 10 lines cut ...]
Macro-F1: 0.9452537310827306
[... 11 lines cut ...]

mean_predict_ms = round((time.perf_counter() - _predict_start) * 1000 / len(reply_validation_df), 1)
print("Mean milliseconds per reply prediction:", mean_predict_ms)

Mean milliseconds per reply prediction: 75.6

Macro-F1 gives each intent the same weight, which matters here because the intents that lead to different actions, such as not_now, not_interested and opt_out, are the ones a model mixes up most easily.

Step 19: Score the test set once

Evaluate the chosen model on the test replies

reply_test_metrics = FastDecisionModel.evaluate(
    reply_model,
    reply_tokenizer,
    reply_test_items,
)

FastDecisionModel.for_inference(reply_model)

reply_test_predictions = predict_replies(
    reply_test_df,
    reply_model,
    reply_tokenizer,
)

print("Unsloth reply test metrics:", reply_test_metrics)

print(
    classification_report(
        reply_test_df["reply_intent"],
        reply_test_predictions["reply_intent"],
        labels=REPLY_INTENTS,
        zero_division=0,
    )
)

reply_test_text_predictions = reply_text_baseline.predict(
    reply_test_df["reply_text"]
)

print(
    "TF-IDF test macro-F1:",
    f1_score(
        reply_test_df["reply_intent"],
        reply_test_text_predictions,
        labels=REPLY_INTENTS,
        average="macro",
        zero_division=0,
    ),
)

print(
    classification_report(
        reply_test_df["reply_intent"],
        reply_test_text_predictions,
        labels=REPLY_INTENTS,
        zero_division=0,
    )
)

test_follow_up_labels = reply_test_df["reply_intent"].map(
    lambda intent: intent not in NO_FOLLOW_UP_INTENTS
)

print(
    "Follow-up test metrics:",
    classification_report(
        test_follow_up_labels,
        reply_test_predictions["follow_up_recommended"],
        labels=[False, True],
        zero_division=0,
    ),
)

print(
    "Test macro-F1:",
    f1_score(
        reply_test_df["reply_intent"],
        reply_test_predictions["reply_intent"],
        labels=REPLY_INTENTS,
        average="macro",
        zero_division=0,
    ),
)

Unsloth reply test metrics: {'accuracy': 0.9758064516129032, 'ece': 0.01909714937210081, 'loss': 0.09047679677076949, 'record_accuracy': 0.9596774193548387}
                   precision    recall  f1-score   support

       interested       1.00      1.00      1.00        13
meeting_requested       1.00      1.00      1.00        11
 more_information       1.00      1.00      1.00        10
         referral       1.00      1.00      1.00        13
          not_now       0.90      0.90      0.90        10
        objection       1.00      1.00      1.00        12
     wrong_person       1.00      1.00      1.00        12
    out_of_office       1.00      1.00      1.00        10
   not_interested       0.85      1.00      0.92        11
          opt_out       1.00      0.90      0.95        10
        ambiguous       1.00      0.92      0.96        12

         accuracy                           0.98       124
        macro avg       0.98      0.97      0.97       124
     weighted avg       0.98      0.98      0.98       124
[... 1 line cut ...]
TF-IDF test macro-F1: 0.7048601336254758
[... 18 lines cut ...]
Follow-up test metrics:               precision    recall  f1-score   support

       False       0.88      1.00      0.93        21
        True       1.00      0.97      0.99       103

    accuracy                           0.98       124
   macro avg       0.94      0.99      0.96       124
weighted avg       0.98      0.98      0.98       124
[... 1 line cut ...]
Test macro-F1: 0.9745960751681576

Three test replies are wrong. Two of them went to not_interested: one not_now reply and one opt_out reply. The opt-out error is the one that matters, because an opt-out must suppress marketing. So the routing in Step 20 suppresses every opt_out answer, and the gate of page 3 (Step 24) also looks for opt-out phrases in the text itself, whatever the model says.

The confidence of the chosen intent is high for every test reply and matches the accuracy, so a confidence threshold would change little here. The plot cells:

Plot style

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
from IPython.display import Image, display

SURFACE = "#fcfcfb"
INK = "#0b0b0b"
INK_SECONDARY = "#52514e"
MUTED = "#898781"
GRID = "#e1e0d9"
SERIES = ["#2a78d6", "#eb6834", "#1baf7a", "#eda100"]
FIGURES = Path("outputs/figures")
FIGURES.mkdir(parents=True, exist_ok=True)

plt.rcParams.update(
    {
        "figure.facecolor": SURFACE,
        "axes.facecolor": SURFACE,
        "axes.edgecolor": GRID,
        "axes.labelcolor": INK_SECONDARY,
        "axes.titlecolor": INK,
        "axes.titlesize": 12,
        "axes.titleweight": "semibold",
        "axes.titlelocation": "left",
        "axes.grid": True,
        "grid.color": GRID,
        "grid.linewidth": 0.6,
        "axes.spines.top": False,
        "axes.spines.right": False,
        "xtick.color": MUTED,
        "ytick.color": MUTED,
        "legend.frameon": False,
        "legend.labelcolor": INK_SECONDARY,
        "lines.linewidth": 2,
        "font.size": 10,
    }
)


def save_figure(figure, name):
    path = FIGURES / f"{name}.png"
    figure.savefig(path, dpi=150, bbox_inches="tight", facecolor=SURFACE)
    plt.close(figure)
    display(Image(filename=str(path)))
    print("Saved", path)

from sklearn.calibration import calibration_curve

Draw the four figures and save the test predictions

reply_test_df[["reply_id", "reply_intent"]].reset_index(drop=True).assign(
    predicted_intent=reply_test_predictions["reply_intent"],
    confidence=reply_test_predictions["reply_intent_confidence"],
    tfidf_intent=reply_test_text_predictions,
).to_csv("outputs/reply-test-predictions.csv", index=False)

tfidf_validation_macro_f1 = f1_score(
    reply_validation_df["reply_intent"],
    reply_validation_text_predictions,
    labels=REPLY_INTENTS,
    average="macro",
    zero_division=0,
)
figure, axis = plt.subplots(figsize=(8, 4.5))
positions = range(len(reply_selection))
values = [result["validation_macro_f1"] for result in reply_selection]
bars = axis.bar(positions, values, width=0.5, color=SERIES[0])
axis.bar_label(bars, labels=[f"{value:.3f}" for value in values], padding=4, color=INK)
axis.axhline(tfidf_validation_macro_f1, color=SERIES[1], linewidth=2, linestyle="--", label=f"TF-IDF baseline {tfidf_validation_macro_f1:.3f}")
axis.set_xticks(
    list(positions),
    [f"{result['epochs']} epochs, rank {result['lora_rank']}\n{result['optimizer_steps']} steps" for result in reply_selection],
)
axis.set_ylim(0, 1)
axis.set_title(
    f"Reply model: validation macro-F1 by setting (chosen: {reply_chosen['epochs']} epochs, rank {reply_chosen['lora_rank']})"
)
axis.set_ylabel("Validation macro-F1")
axis.legend(loc="lower right")
axis.grid(axis="x", visible=False)
save_figure(figure, "reply-epoch-choice")

figure, axis = plt.subplots(figsize=(8, 4.5))
for result, color in zip(reply_selection, SERIES):
    points = [(entry["step"], entry["loss"]) for entry in result["log_history"] if "loss" in entry]
    label = f"{result['epochs']} epochs, rank {result['lora_rank']}"
    axis.plot(*zip(*points), color=color, label=label)
    axis.annotate(label, points[-1], xytext=(6, 0), textcoords="offset points", va="center", color=INK_SECONDARY)
axis.set_title("Reply model: training loss for each candidate setting")
axis.set_xlabel("Optimizer step")
axis.set_ylabel("Training loss")
axis.legend(loc="upper right")
save_figure(figure, "reply-training-loss")

test_matrix = confusion_matrix(reply_test_df["reply_intent"], reply_test_predictions["reply_intent"], labels=REPLY_INTENTS)
row_share = test_matrix / test_matrix.sum(axis=1, keepdims=True).clip(min=1)
figure, axis = plt.subplots(figsize=(8, 7))
axis.imshow(row_share, cmap=matplotlib.colors.LinearSegmentedColormap.from_list("blue", [SURFACE, "#cde2fb", "#6da7ec", "#2a78d6", "#104281"]), vmin=0, vmax=1)
for true_index in range(len(REPLY_INTENTS)):
    for predicted_index in range(len(REPLY_INTENTS)):
        count = test_matrix[true_index, predicted_index]
        if count:
            axis.text(predicted_index, true_index, count, ha="center", va="center", fontsize=9, color="#ffffff" if row_share[true_index, predicted_index] > 0.5 else INK)
axis.set_xticks(range(len(REPLY_INTENTS)), REPLY_INTENTS, rotation=45, ha="right")
axis.set_yticks(range(len(REPLY_INTENTS)), REPLY_INTENTS)
axis.set_xlabel("Predicted intent")
axis.set_ylabel("True intent")
axis.grid(False)
axis.set_title("Reply model: test confusion matrix (counts; shade = share of the true intent)")
save_figure(figure, "reply-confusion-matrix")

figure, axis = plt.subplots(figsize=(8, 4.5))
correct = (reply_test_predictions["reply_intent"] == reply_test_df["reply_intent"].reset_index(drop=True)).astype(int)
observed, predicted = calibration_curve(correct, reply_test_predictions["reply_intent_confidence"], n_bins=5, strategy="quantile")
axis.plot([0, 1], [0, 1], color=MUTED, linewidth=1, linestyle="--", label="Perfect calibration")
axis.plot(predicted, observed, color=SERIES[0], marker="o", markersize=8, label="Fine-tuned LFM3")
axis.set_title("Reply model: confidence of the chosen intent against accuracy, test replies")
axis.set_xlabel("Mean confidence (5 equal-count bins)")
axis.set_ylabel("Share answered correctly")
axis.legend(loc="upper left")
save_figure(figure, "reply-calibration")

Reply model results

The notebook writes every number to outputs/reply-metrics.json, including the validation score of every candidate.

Measure	Split	Fine-tuned LFM3-1.7B	TF-IDF + logistic regression
Intent macro-F1	Validation	0.945	0.565
Intent macro-F1	Test	0.975	0.705
Intent accuracy	Validation	0.94	0.57
Intent accuracy	Test	0.98	0.71
Follow-up F1, “no next step”	Test	0.93	not applicable
ECE, both questions	Test	0.019	not applicable

• Per intent on test. Seven intents score an F1 of 1.0. The other four are not_now (0.90), not_interested (0.92), opt_out (0.95) and ambiguous (0.96).
• Spread between runs. The same notebook ran twice with seed 3407. The first run stopped in a cleanup cell after it scored the test set, and its chosen model scored 0.924 on test; the recorded second run scored 0.975. GPU kernels are not fully deterministic, and 124 test replies make one reply worth about 0.01 of macro-F1. Report a range, not a point, when you adapt this.
• Against the first version. The 20-step model of the first version scored 0.778 on the same test replies (its metrics), and its follow-up head answered yes for every reply. More optimizer updates fixed both.
• Cost. The chosen setting trained in 207 s; the grid took 711 s. Peak GPU memory was 4.6 GB. A prediction took 75.6 ms per reply on average.

Step 20: Map replies to actions

The model classifies; the application routes. route_reply maps the intent to a controlled action, and three rules come before the model: a suppressed contact, an explicit opt-out event or an opt_out answer always gives suppress_marketing; an unknown suppression status goes to verification; a malformed answer goes to human review. When the intent calls for a next step but the model’s follow-up answer is no, the reply goes to human review. Every branch returns send as False.

Predicted intent	Action
interested, meeting_requested	Hand off to sales
more_information	Prepare the requested information for review
referral	Review the referral
not_now	Schedule later, after a person confirms the date
out_of_office	Pause until a person confirms the return date
objection, wrong_person, ambiguous	Human review
not_interested	Stop the sequence
opt_out	Suppress marketing

The routing policy, route_reply

REPLY_ACTIONS = {
    "interested": "handoff_to_sales",
    "meeting_requested": "handoff_to_sales",
    "more_information": "prepare_requested_information",
    "referral": "review_referral",
    "not_now": "schedule_later_after_date_confirmation",
    "objection": "human_review",
    "wrong_person": "human_review",
    "out_of_office": "pause_until_return_date_confirmation",
    "not_interested": "stop_sequence",
    "opt_out": "suppress_marketing",
    "ambiguous": "human_review",
}

def route_reply(
    predicted_intent,
    model_follow_up_recommended,
    contact_is_suppressed,
    explicit_opt_out_event,
):
    if isinstance(contact_is_suppressed, np.bool_):
        contact_is_suppressed = bool(contact_is_suppressed)
    if isinstance(explicit_opt_out_event, np.bool_):
        explicit_opt_out_event = bool(explicit_opt_out_event)

    if (
        contact_is_suppressed is True
        or explicit_opt_out_event is True
        or predicted_intent == "opt_out"
    ):
        return {
            "action": "suppress_marketing",
            "model_follow_up_recommended": (
                model_follow_up_recommended
                if isinstance(model_follow_up_recommended, (bool, np.bool_))
                else None
            ),
            "policy_follow_up_recommended": False,
            "send": False,
        }

    if (
        contact_is_suppressed is not False
        or explicit_opt_out_event is not False
    ):
        return {
            "action": "verify_suppression_and_opt_out_status",
            "model_follow_up_recommended": None,
            "policy_follow_up_recommended": False,
            "send": False,
        }

    if not isinstance(model_follow_up_recommended, (bool, np.bool_)):
        return {
            "action": "human_review",
            "reason": "invalid_follow_up_model_output",
            "model_follow_up_recommended": None,
            "policy_follow_up_recommended": False,
            "send": False,
        }

    model_follow_up_recommended = bool(model_follow_up_recommended)

    if predicted_intent not in REPLY_ACTIONS:
        return {
            "action": "human_review",
            "reason": "unknown_reply_intent",
            "model_follow_up_recommended": model_follow_up_recommended,
            "policy_follow_up_recommended": False,
            "send": False,
        }

    policy_follow_up_recommended = (
        predicted_intent not in NO_FOLLOW_UP_INTENTS
    )

    if not policy_follow_up_recommended:
        return {
            "action": REPLY_ACTIONS[predicted_intent],
            "model_follow_up_recommended": model_follow_up_recommended,
            "policy_follow_up_recommended": False,
            "send": False,
        }

    if not model_follow_up_recommended:
        return {
            "action": "human_review",
            "reason": "intent_and_follow_up_decisions_disagree",
            "model_follow_up_recommended": False,
            "policy_follow_up_recommended": True,
            "send": False,
        }

    return {
        "action": REPLY_ACTIONS[predicted_intent],
        "model_follow_up_recommended": True,
        "policy_follow_up_recommended": True,
        "send": False,
    }

The classifier extracts no dates, names or requested items, so not_now and out_of_office wait for a person before anything is scheduled. Route the first validation reply:

sample_reply = reply_validation_predictions.iloc[0]

print(
    route_reply(
        predicted_intent=sample_reply["reply_intent"],
        model_follow_up_recommended=sample_reply["follow_up_recommended"],
        contact_is_suppressed=False,
        explicit_opt_out_event=False,
    )
)

{'action': 'schedule_later_after_date_confirmation', 'model_follow_up_recommended': True, 'policy_follow_up_recommended': True, 'send': False}

Regression checks for the policy gates

import numpy as np

assert route_reply(
    predicted_intent="interested",
    model_follow_up_recommended=True,
    contact_is_suppressed=False,
    explicit_opt_out_event=True,
)["action"] == "suppress_marketing"

assert route_reply(
    predicted_intent="interested",
    model_follow_up_recommended=True,
    contact_is_suppressed=None,
    explicit_opt_out_event=False,
)["action"] == "verify_suppression_and_opt_out_status"

assert route_reply(
    predicted_intent="interested",
    model_follow_up_recommended=False,
    contact_is_suppressed=False,
    explicit_opt_out_event=False,
)["action"] == "human_review"

assert route_reply(
    predicted_intent="interested",
    model_follow_up_recommended="False",
    contact_is_suppressed=False,
    explicit_opt_out_event=False,
)["reason"] == "invalid_follow_up_model_output"

assert route_reply(
    predicted_intent="opt_out",
    model_follow_up_recommended="invalid",
    contact_is_suppressed=False,
    explicit_opt_out_event=False,
)["action"] == "suppress_marketing"

assert route_reply(
    predicted_intent="interested",
    model_follow_up_recommended="invalid",
    contact_is_suppressed=np.bool_(True),
    explicit_opt_out_event=np.bool_(False),
)["action"] == "suppress_marketing"

integration/policy.py holds the same route_reply for page 3. There, the gate also looks for opt-out phrases in the reply text, and tests/test_gate.py checks that a mixed reply with an opt-out phrase, such as “Sounds useful, but please take me off your list”, is suppressed whatever the model answers.

Step 21: Save, publish and reload

save_pretrained saves the adapter and the decision head; save_pretrained_merged exports a merged 16-bit model (Unsloth: calibrate and save).

reply_model.save_pretrained("outputs/reply-decision-adapter")
reply_model.save_pretrained_merged(
    "outputs/reply-decision-merged",
    reply_tokenizer,
)

[... 6 lines cut ...]
/home/dev/work/lead-decision-models/unsloth_env/lib/python3.13/site-packages/unsloth_zoo/saving_utils.py:3849: UserWarning: Unsloth: `LiquidAI/LFM3-1.7B` was loaded without a recorded commit, so its repo code was not copied into the export. Copy the repo's *.py files at the revision you loaded into `/home/dev/work/lead-decision-models/outputs/reply-decision-merged/.unsloth-save-6turp0u0` before loading it with trust_remote_code=True.
  warnings.warn(
[... 9 lines cut ...]
Unsloth: Merge process complete. Saved to `/home/dev/work/lead-decision-models/outputs/reply-decision-merged/.unsloth-save-6turp0u0`

As on page 1, Unsloth warns that it did not copy LFM3’s Python files into the export, and the merged folder still holds them in the recorded run. Check your own export before you load it with trust_remote_code=True.

REPLY_REPO = f"{HF_USERNAME}/lead-decision-models-reply-router"

if PUBLISH:
    reply_model.push_to_hub(REPLY_REPO, reply_tokenizer)
    reply_model.push_to_hub_merged(f"{REPLY_REPO}-merged", reply_tokenizer, save_method="merged_16bit")
    print("Published", REPLY_REPO, "and", f"{REPLY_REPO}-merged")

[... 1 line cut ...]
Unsloth: Saved the decision model to https://huggingface.co/raoofaltaher/lead-decision-models-reply-router
[... 13 lines cut ...]
Unsloth: Saved the decision model to https://huggingface.co/raoofaltaher/lead-decision-models-reply-router-merged
Published raoofaltaher/lead-decision-models-reply-router and raoofaltaher/lead-decision-models-reply-router-merged

Both Hub models fall under the LFM Open License v1.0. They are teaching models trained on synthetic replies. Release the training model, reload the merged export and compare one prediction:

import gc

del reply_model, reply_tokenizer
gc.collect()
torch.cuda.empty_cache()

reloaded_reply_model, reloaded_reply_tokenizer = FastDecisionModel.from_pretrained(
    model_name="outputs/reply-decision-merged",
    trust_remote_code=True,
    max_seq_length=2048,
    load_in_4bit=LOAD_IN_4BIT,
)

FastDecisionModel.for_inference(reloaded_reply_model)

reply_example = make_reply_record(reply_test_df.iloc[0])
reply_reload_prediction = FastDecisionModel.predict(
    reloaded_reply_model,
    reloaded_reply_tokenizer,
    reply_example["state"],
    REPLY_QUESTIONS,
)

print(reply_reload_prediction)

reply_before_export = reply_test_predictions.iloc[0]
print(
    "Reply class agreement after export:",
    reply_reload_prediction["reply_intent"]["answer"]
    == reply_before_export["reply_intent"],
    reply_reload_prediction["follow_up_recommended"]["answer"]
    == bool(reply_before_export["follow_up_recommended"]),
)

[... 11 lines cut ...]
{'reply_intent': {'type': 'choice', 'choice': 'interested', 'confidence': 0.9996, 'probabilities': {'interested': 0.9996, 'meeting_requested': 0.0, 'more_information': 0.0, 'referral': 0.0, 'not_now': 0.0, 'objection': 0.0, 'wrong_person': 0.0, 'out_of_office': 0.0, 'not_interested': 0.0, 'opt_out': 0.0, 'ambiguous': 0.0002}, 'answer': 'interested'}, 'follow_up_recommended': {'type': 'noul', 'noul': 0.9997, 'answer': True, 'probabilities': {'true': 0.9997, 'false': 0.0003}}}
Reply class agreement after export: True True

The last cell writes outputs/reply-metrics.json and outputs/experiment-manifest.json, which records the base-model revision, the data checksums, the splits, the seeds, the chosen training settings and the package versions of both notebooks:

Write the metrics and the experiment manifest

import hashlib
import importlib.metadata
import platform

def installed_version(distribution):
    try:
        return importlib.metadata.version(distribution)
    except importlib.metadata.PackageNotFoundError:
        return None

reply_metrics = {
    "model_id": MODEL_ID,
    "model_revision": MODEL_REVISION,
    "load_in_4bit": LOAD_IN_4BIT,
    "splits": {
        "optimizer_training": len(reply_fit_df),
        "calibration": len(reply_calibration_df),
        "validation": len(reply_validation_df),
        "test": len(reply_test_df),
    },
    "baseline_untrained": reply_baseline,
    "tfidf": {
        "validation_macro_f1": float(f1_score(reply_validation_df["reply_intent"], reply_validation_text_predictions, labels=REPLY_INTENTS, average="macro", zero_division=0)),
        "test_macro_f1": float(f1_score(reply_test_df["reply_intent"], reply_test_text_predictions, labels=REPLY_INTENTS, average="macro", zero_division=0)),
    },
    "epoch_selection": [{key: value for key, value in result.items() if key not in ("calibration", "log_history")} for result in reply_selection],
    "chosen_epochs": reply_chosen["epochs"],
    "calibration": reply_calibration_metrics,
    "validation": {**reply_validation_metrics, "macro_f1": float(f1_score(reply_validation_df["reply_intent"], reply_validation_predictions["reply_intent"], labels=REPLY_INTENTS, average="macro", zero_division=0))},
    "test": {**reply_test_metrics, "macro_f1": float(f1_score(reply_test_df["reply_intent"], reply_test_predictions["reply_intent"], labels=REPLY_INTENTS, average="macro", zero_division=0))},
    "test_report": classification_report(reply_test_df["reply_intent"], reply_test_predictions["reply_intent"], labels=REPLY_INTENTS, output_dict=True, zero_division=0),
    "follow_up_report": classification_report(test_follow_up_labels, reply_test_predictions["follow_up_recommended"], labels=[False, True], output_dict=True, zero_division=0),
    "confusion_matrix": {"labels": REPLY_INTENTS, "matrix": confusion_matrix(reply_test_df["reply_intent"], reply_test_predictions["reply_intent"], labels=REPLY_INTENTS).tolist()},
    "timings_seconds": TIMINGS,
    "peak_vram_gb": peak_vram_gb,
    "mean_predict_ms": mean_predict_ms,
}
Path("outputs/reply-metrics.json").write_text(json.dumps(reply_metrics, indent=2, default=float), encoding="utf-8")

lead_metrics = json.loads(Path("outputs/lead-metrics.json").read_text(encoding="utf-8"))

experiment_manifest = {
    "experiment": "unsloth-b2b-lead-and-reply-decision-models",
    "dataset": {
        "leadforge_commit": lead_metrics["dataset"]["leadforge_commit"],
        "leadforge_release": "v0.1.0-alpha",
        "leadforge_tier": "intermediate",
        "leadforge_task": "converted_within_90_days",
        "lead_target_counts": lead_metrics["dataset"]["target_counts"],
        "reply_dataset": "synthetic-llm-generated-automatically-checked",
        "reply_dataset_sha256": reply_dataset_hash,
        "reply_intent_counts": {key: int(value) for key, value in reply_df["reply_intent"].value_counts().items()},
    },
    "base_model": {"id": MODEL_ID, "revision": MODEL_REVISION, "load_in_4bit": LOAD_IN_4BIT},
    "splits": {"lead": lead_metrics["splits"], "reply": reply_metrics["splits"]},
    "seed": 3407,
    "calibration_split_seed": 3408,
    "training": {
        "learning_rate": 2e-4,
        "head_learning_rate": 1e-4,
        "max_seq_length": 2048,
        "lead": {"epochs": 2, "per_device_train_batch_size": 8, "gradient_accumulation_steps": 4, "lora_rank": 16, "lora_alpha": 16},
        "reply": {"epochs": reply_chosen["epochs"], "per_device_train_batch_size": 8, "gradient_accumulation_steps": 1, "lora_rank": reply_chosen["lora_rank"], "lora_alpha": reply_chosen["lora_rank"]},
    },
    "timings_seconds": {**lead_metrics["timings_seconds"], **TIMINGS},
    "peak_vram_gb": {"lead": lead_metrics["peak_vram_gb"], "reply": peak_vram_gb},
    "runtime": {
        "python": platform.python_version(),
        "pytorch": torch.__version__,
        "cuda_runtime": torch.version.cuda,
        "gpu": torch.cuda.get_device_name(0),
        "packages": {
            name: installed_version(name)
            for name in ["unsloth", "unsloth_zoo", "transformers", "bitsandbytes", "datasets", "pandas", "scikit-learn", "huggingface-hub"]
        },
    },
    "policy_versions": {
        "icp_policy": "fictional-procurement-icp-v1",
        "reply_taxonomy": "b2b-reply-intents-v1",
        "outreach_policy": "human-approval-and-external-suppression-v1",
    },
}

Path("outputs/experiment-manifest.json").write_text(json.dumps(experiment_manifest, indent=2, default=float), encoding="utf-8")
print("Wrote outputs/reply-metrics.json and outputs/experiment-manifest.json")