Skip to content
Raoof Altaher

Fine-Tune LFM3, a Large Language Model, into a Decision Model with Unsloth: Lead Qualification with Claude Code over MCP

Page 1 of 3: Lead qualification

  • unsloth
  • fine-tuning
  • lfm3
  • decision-models
  • claude-code
  • mcp
  • case-study
  • practical-ai

This tutorial turns LiquidAI/LFM3-1.7B, a small large language model, into two decision models with Unsloth’s FastDecisionModel. Then it hands both models to a Claude Code agent over MCP.

A dark title card with an orange edge that reads Fine-Tune LFM3, a Large Language Model, into a Decision Model with Unsloth: Lead Qualification with Claude Code over MCP, by Raoof Altaher

The first model qualifies B2B leads: does the account match the Ideal Customer Profile (ICP), and did the lead convert within 90 days? The second model reads a prospect’s reply and picks one of 11 intents. Application code turns every answer into a recommendation that a person approves. Nothing in the setup can send a message.

The tutorial has three pages:

  1. Lead qualification (this page): the setup, the data, the training and the evaluation of the lead model.
  2. Reply routing: the reply model, and how its training setup was chosen on validation data.
  3. Claude Code over MCP: a small MCP server for both models, the Claude Code connection and a recorded case study with real companies.

Everything you need is public:

Every Python block is a cell of an executed notebook, and every text block under it is that cell’s output from the recorded run. Collapsed blocks hold the helper cells; open them when you reproduce a step. [... n lines cut ...] marks a shortened output.

Results at a glance

Section titled “Results at a glance”
Result Fine-tuned LFM3-1.7B Baseline
Reply intent, macro-F1 on 124 test replies 0.975 0.705 (TF-IDF + logistic regression)
Lead ICP fit, accuracy on 1,002 test leads 1.000 not applicable
Lead 90-day conversion, ROC-AUC 0.869 0.876 (logistic regression)
Lead 90-day conversion, Brier score (lower is better) 0.1091 0.1105

On text, the fine-tuned model wins clearly. On tabular CRM features, a logistic regression still ranks conversions slightly better, and four training seeds confirm it.

The ICP score is perfect because the ICP label is a rule over three fields that are never empty in this data (What the ICP accuracy of 1.0 means).

In the Claude Code case study, the agent researched ten real companies. The gate refused eight for missing evidence, the lead model answered the other two correctly, and the reply model gave each of the eight case-study replies the intent its text states.

What a decision model is

Section titled “What a decision model is”

A decision model does not write text that your code must parse. Unsloth formats a state and typed questions, and a decision head scores the allowed options directly. The guide defines three question types (Unsloth dataset guide):

Type Purpose Gold label
choice Select one named option The option key
noul Answer yes or no Python True or False
score Select an ordered level A zero-based integer

This tutorial uses choice for ICP fit and noul for the conversion outcome. The answers feed four separate decisions, and only one of them is the model’s:

Decision Who decides Evidence
Account ICP fit The model, trained on a written policy Industry, region, employee band
90-day conversion The model, as an experiment only LeadForge’s simulated outcome label
Contact persona fit Application code Buyer role in the CRM
Permission to contact Application code and a person Suppression, consent and legal checks

A model probability is never permission to contact anyone.

Replicate the setup

Section titled “Replicate the setup”

What you need

Section titled “What you need”
Item Recorded run Minimum to reproduce
GPU NVIDIA GeForce RTX 5090 Laptop GPU, 24 GB, driver 617.42 An NVIDIA GPU with CUDA support; this run peaked at 2.73 GB, but only this GPU was tested
System Windows 11 with Ubuntu 24.04.4 LTS on WSL2 Linux, or WSL2 on Windows
Python 3.13.18 3.13
PyTorch 2.14.1+cu132 (CUDA runtime 13.2) From requirements-experiment.txt
Unsloth and Unsloth Zoo 2026.10.11 Pinned
Transformers 5.22.1 Pinned; see below
Base model LiquidAI/LFM3-1.7B at revision ac2586a34f1a1c03329dbdcdefabd8b6478ddaa7, 4-bit QLoRA Pinned
Disk 6.4 GB for the environment, 8.1 GB for the outputs of both notebooks About 25 GB free, with the model cache (estimate)
Accounts Hugging Face None to train; a Hugging Face write token to publish
Time 1,917 s for the lead notebook and 1,962 s for the reply notebook, uploads included Depends on the GPU

On Windows, install WSL2 with wsl --install (Microsoft: Install WSL). Install the NVIDIA driver on Windows only. NVIDIA’s guide says not to install a Linux display driver inside WSL (CUDA on WSL User Guide). The tutorial uses Unsloth Core, the Python package, not Unsloth Studio.

Install the recorded environment

Section titled “Install the recorded environment”

Install uv, clone the repository and install the exact package set of the recorded run:

curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/raoofaltaher/lead-decision-models
cd lead-decision-models
uv venv unsloth_env --python 3.13
source unsloth_env/bin/activate
uv pip sync --torch-backend=auto requirements-experiment.txt
python tools/check_unsloth_pin.py --commit 53eeebdc9be3c3e1a27f891b668a38df0e1abb88

--torch-backend=auto picks the CUDA 13.2 build of PyTorch when the driver supports it (uv: Using uv with PyTorch). To install the same files on any driver, use the two-index form:

uv pip sync --index-url https://pypi.org/simple --extra-index-url https://download.pytorch.org/whl/cu132 --index-strategy unsafe-best-match requirements-experiment.txt

tools/check_unsloth_pin.py compares the installed decision.py and _decision_common.py with Unsloth’s commit 53eeebdc9be3c3e1a27f891b668a38df0e1abb88 by SHA-256. It prints MATCH for both files. Every link to Unsloth’s code on this page points at that commit.

uv pip check then reports four incompatibilities, and all four are deliberate. unsloth and unsloth-zoo declare transformers<=5.17.0 and datasets<5.0.0. The environment installs Transformers 5.22.1 because LFM3’s code needs it, and datasets 5.1.0 with fsspec 2026.7.0 because they close two security advisories.

Why Transformers 5.22.1

Section titled “Why Transformers 5.22.1”

tools/probe_lfm3.py loads LFM3-1.7B through FastDecisionModel at the pinned revision and trains five steps, in 4-bit and in 16-bit. Every Transformers release that Unsloth 2026.10.11 declares fails with LFM3’s own code:

Transformers Declared by Unsloth Probe result
5.12.1 Yes modeling_lfm3.py does not import: cannot import name 'use_kernel_func_from_hub_with_fallback'
5.16.1 Yes The forward pass fails with KeyError: 'linear_attention'
5.17.0 Yes, the highest The configuration does not load: AutoConfig error: 'conv'
5.22.1 No Loads and trains in both modes; the 4-bit loss falls from 0.974 to 0.886 in five steps

If you change either version, run the probe again before you train. The repository keeps each result in outputs/lfm3-probe-*.json.

Log in and run the notebook

Section titled “Log in and run the notebook”

To publish the trained model, log in once with a Hugging Face token that has write access (User access tokens, hf CLI). Skip this step if you set PUBLISH = False.

hf auth login

Run the notebook in Jupyter, or without a browser:

jupyter nbconvert --to notebook --execute --inplace --ExecutePreprocessor.timeout=-1 notebooks/01-lead-qualification.ipynb

The first cell sets the run options. Set HF_USERNAME to your own account, or set PUBLISH = False. TIMINGS collects the wall-clock times that the results report.

import os
import time
from pathlib import Path


if Path.cwd().name == "notebooks":
    os.chdir(Path.cwd().parent)


HF_USERNAME = "raoofaltaher"
PUBLISH = True
LOAD_IN_4BIT = True
TIMINGS = {}


print("Working directory:", Path.cwd())
Working directory: /home/dev/work/lead-decision-models
import platform
import torch


print("Python:", platform.python_version())
print("PyTorch:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())


if torch.cuda.is_available():
    print("GPU:", torch.cuda.get_device_name(0))
    print("CUDA runtime:", torch.version.cuda)
else:
    raise RuntimeError("A supported GPU runtime is required for this training run.")
Python: 3.13.18
PyTorch: 2.14.1+cu132
CUDA available: True
GPU: NVIDIA GeForce RTX 5090 Laptop GPU
CUDA runtime: 13.2

Step 1: Load the LeadForge data

Section titled “Step 1: Load the LeadForge data”

The experiment uses the intermediate tier of LeadForge, a public, MIT-licensed synthetic CRM dataset for a fictional B2B procurement SaaS vendor. It holds 5,000 leads with account, contact, engagement and sales features. It is not real customer data. The release is alpha, so the notebook pins commit 39ebe8c9e6cf25fe33a003662e18557b4de55142 (release v0.1.0-alpha) and checks the SHA-256 of every file before it loads it.

Download and verify the three task files
import hashlib
import shutil
from pathlib import Path
from urllib.request import urlopen


from datasets import load_dataset


DATASET_COMMIT = "39ebe8c9e6cf25fe33a003662e18557b4de55142"
TASK_BASE = (
    "https://raw.githubusercontent.com/leadforge-dev/leadforge-datasets/"
    f"{DATASET_COMMIT}/releases/v0.1.0-alpha/intermediate/"
    "tasks/converted_within_90_days"
)


# SHA-256 values from the pinned intermediate manifest.
TASK_FILES = {
    "train": ("train.parquet", "ba7dd8f6ff24add8d6d4897d9870f46e818bb0ca3432a59c263c2d4e16f79184"),
    "validation": ("valid.parquet", "61029542d352f493b56d1ad85cf52a4d4970c651ed87cf5bdf05978d222e3592"),
    "test": ("test.parquet", "d428c07decc2b1fdf8b5f56a1a63c65799897f1e22b61afd9f5d517f74593f09"),
}


DATA_DIR = Path("data/leadforge-intermediate")
DATA_DIR.mkdir(parents=True, exist_ok=True)


def sha256_file(path):
    digest = hashlib.sha256()
    with path.open("rb") as file:
        for chunk in iter(lambda: file.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()


local_files = {}


for split_name, (filename, expected_sha256) in TASK_FILES.items():
    destination = DATA_DIR / filename
    url = f"{TASK_BASE}/{filename}"


    with urlopen(url, timeout=60) as response, destination.open("wb") as output:
        shutil.copyfileobj(response, output)


    actual_sha256 = sha256_file(destination)
    if actual_sha256 != expected_sha256:
        raise ValueError(
            f"{split_name} checksum mismatch: expected {expected_sha256}, "
            f"received {actual_sha256}"
        )


    local_files[split_name] = str(destination)
    print(split_name, "SHA-256 verified:", actual_sha256)


lead_splits = load_dataset(
    "parquet",
    data_files=local_files,
)


for split_name, split in lead_splits.items():
    print(split_name, len(split), split.column_names)
train SHA-256 verified: ba7dd8f6ff24add8d6d4897d9870f46e818bb0ca3432a59c263c2d4e16f79184
validation SHA-256 verified: 61029542d352f493b56d1ad85cf52a4d4970c651ed87cf5bdf05978d222e3592
test SHA-256 verified: d428c07decc2b1fdf8b5f56a1a63c65799897f1e22b61afd9f5d517f74593f09
[... 6 lines cut ...]
Inspect the target
import pandas as pd


LABEL_COLUMN = "converted_within_90_days"


frames = []
for split_name, split in lead_splits.items():
    frame = split.to_pandas()
    frame["release_split"] = split_name
    frames.append(frame)


leads = pd.concat(frames, ignore_index=True)


print("Rows:", len(leads))
print("Unique leads:", leads["lead_id"].nunique())
print("Target distribution:")
print(leads[LABEL_COLUMN].value_counts(dropna=False))
print("Target rate:")
print(leads[LABEL_COLUMN].mean())


assert len(leads) == 5000
assert leads["lead_id"].is_unique
assert leads[LABEL_COLUMN].notna().all()
Rows: 5000
Unique leads: 5000
Target distribution:
converted_within_90_days
False    3951
True     1049
Name: count, dtype: int64
Target rate:
0.2098

The target converted_within_90_days is LeadForge’s label: a simulated conversion within 90 days of lead creation. The features cover only the first 30 days (dataset card). So a positive label can describe a lead that already converted by day 30. The public tables/leads.parquet holds each conversion timestamp, and the next cell counts those leads:

Count conversions on or before day 30
LEADS_TABLE_SHA256 = "7a4bf1b7afeb54e0b8d74b7f89b309ceee83455f734ca67f59e94e32d430fa95"
LEADS_TABLE_URL = (
    "https://raw.githubusercontent.com/leadforge-dev/leadforge-datasets/"
    f"{DATASET_COMMIT}/releases/v0.1.0-alpha/intermediate/tables/leads.parquet"
)


leads_table_path = DATA_DIR / "leads.parquet"


with urlopen(LEADS_TABLE_URL, timeout=60) as response, leads_table_path.open("wb") as output:
    shutil.copyfileobj(response, output)


if sha256_file(leads_table_path) != LEADS_TABLE_SHA256:
    raise ValueError("leads.parquet checksum mismatch")


conversion_times = pd.read_parquet(
    leads_table_path,
    columns=["lead_id", "conversion_timestamp"],
)


timed_leads = leads.merge(
    conversion_times,
    on="lead_id",
    how="left",
    validate="one_to_one",
)


assert timed_leads["conversion_timestamp"].notna().eq(
    timed_leads[LABEL_COLUMN].astype(bool)
).all()


created_at = pd.to_datetime(timed_leads["lead_created_at"])
converted_at = pd.to_datetime(timed_leads["conversion_timestamp"])
days_to_conversion = (converted_at - created_at).dt.days


converted_by_day30 = days_to_conversion.le(30)


clean_leads = timed_leads.loc[~converted_by_day30].copy()
clean_leads["converted_after_day30"] = days_to_conversion.loc[~converted_by_day30].lt(90)


print("Published positive labels:", int(timed_leads[LABEL_COLUMN].sum()))
print("Converted on or before day 30:", int(converted_by_day30.sum()))
print("Leads in the clean risk set:", len(clean_leads))
print("Clean positives:", int(clean_leads["converted_after_day30"].sum()))
Published positive labels: 1049
Converted on or before day 30: 22
Leads in the clean risk set: 4978
Clean positives: 1027

Only 22 of the 1,049 positives (2.1%) converted by day 30, but the label is still a retrospective outcome, not a clean forecast. This tutorial keeps the published label, so that the experiment follows the released task, and never uses the conversion score to rank live prospects. The task manifest words the window differently from the generator code; the tutorial follows the generator and the dataset card.

Exclude leakage and identifiers

Section titled “Exclude leakage and identifiers”

The model state uses a whitelist of features. Identifiers, the target, the split name and total_touches_all stay out. total_touches_all counts touches across the whole 90-day horizon, so it is a documented leakage trap (LeadForge README). lead_source and first_touch_channel stay out too, because the experiment does not recommend a channel.

The feature whitelist
MODEL_FEATURES = [
    "industry",
    "region",
    "employee_band",
    "estimated_revenue_band",
    "process_maturity_band",
    "role_function",
    "seniority",
    "buyer_role",
    "touch_count",
    "inbound_touch_count",
    "outbound_touch_count",
    "session_count",
    "pricing_page_views",
    "demo_page_views",
    "total_session_duration_seconds",
    "touches_week_1",
    "touches_last_7_days",
    "days_since_first_touch",
    "activity_count",
    "days_since_last_touch",
    "opportunity_created",
    "has_open_opportunity",
    "opportunity_estimated_acv",
    "expected_acv",
]


required_columns = {
    *MODEL_FEATURES,
    "account_id",
    "contact_id",
    "lead_id",
    LABEL_COLUMN,
}


missing_columns = required_columns - set(leads.columns)
if missing_columns:
    raise ValueError(f"Dataset schema changed. Missing columns: {sorted(missing_columns)}")


assert "total_touches_all" in leads.columns
assert "total_touches_all" not in MODEL_FEATURES
assert LABEL_COLUMN not in MODEL_FEATURES
assert "account_id" not in MODEL_FEATURES
assert "contact_id" not in MODEL_FEATURES
assert "lead_id" not in MODEL_FEATURES

Missing values stay as JSON null, because a missing value is not evidence of a negative outcome. In this release, only numeric engagement and opportunity fields have gaps:

print(leads[MODEL_FEATURES].isna().sum().loc[lambda counts: counts > 0])
touch_count                        400
inbound_touch_count                416
outbound_touch_count               403
session_count                      373
pricing_page_views                 398
demo_page_views                    384
total_session_duration_seconds     400
touches_week_1                     411
touches_last_7_days                406
days_since_first_touch             465
activity_count                     389
days_since_last_touch              469
opportunity_estimated_acv         3049
expected_acv                       391
dtype: int64

Step 2: Split by account

Section titled “Step 2: Split by account”

Several leads can belong to one account. A row-level split puts related leads on both sides of the evaluation boundary, and the audit below shows that LeadForge’s supplied split does exactly that. The notebook therefore pools the three supplied partitions and makes new folds with StratifiedGroupKFold, with account_id as the group. It asserts that no account and no contact crosses a boundary. From scikit-learn 1.8, the class also keeps stratification when shuffle=True (release notes).

Audit the supplied split and make account-disjoint folds
from itertools import combinations
from sklearn.model_selection import StratifiedGroupKFold
import numpy as np


release_frames = {
    name: split.to_pandas()
    for name, split in lead_splits.items()
}


for left, right in combinations(release_frames, 2):
    left_accounts = set(release_frames[left]["account_id"].dropna())
    right_accounts = set(release_frames[right]["account_id"].dropna())
    left_contacts = set(release_frames[left]["contact_id"].dropna())
    right_contacts = set(release_frames[right]["contact_id"].dropna())


    print(
        left,
        right,
        "shared accounts:",
        len(left_accounts & right_accounts),
        "shared contacts:",
        len(left_contacts & right_contacts),
    )


if leads["account_id"].isna().any():
    raise ValueError("Account IDs are required for the grouped evaluation split.")


splitter = StratifiedGroupKFold(
    n_splits=5,
    shuffle=True,
    random_state=3407,
)


folds = list(
    splitter.split(
        leads,
        y=leads[LABEL_COLUMN].astype(int),
        groups=leads["account_id"],
    )
)


test_indices = folds[0][1]
validation_indices = folds[1][1]
train_indices = np.concatenate([folds[2][1], folds[3][1], folds[4][1]])


train_df = leads.iloc[train_indices].reset_index(drop=True)
validation_df = leads.iloc[validation_indices].reset_index(drop=True)
test_df = leads.iloc[test_indices].reset_index(drop=True)


def group_ids(frame, column):
    return set(frame[column].dropna())


for column in ["account_id", "contact_id"]:
    assert group_ids(train_df, column).isdisjoint(group_ids(validation_df, column))
    assert group_ids(train_df, column).isdisjoint(group_ids(test_df, column))
    assert group_ids(validation_df, column).isdisjoint(group_ids(test_df, column))


for name, frame in [
    ("train", train_df),
    ("validation", validation_df),
    ("test", test_df),
]:
    print(
        name,
        "rows:", len(frame),
        "accounts:", frame["account_id"].nunique(),
        "conversion rate:", frame[LABEL_COLUMN].mean(),
    )
train validation shared accounts: 487 shared contacts: 372
train test shared accounts: 518 shared contacts: 403
validation test shared accounts: 243 shared contacts: 110
train rows: 2998 accounts: 767 conversion rate: 0.209472981987992
validation rows: 1000 accounts: 262 conversion rate: 0.21
test rows: 1002 accounts: 251 conversion rate: 0.21057884231536927

Unsloth’s FastDecisionModel.split_holdout makes a seeded calibration holdout, but it does not know about accounts. So the notebook splits whole accounts out of the training fold for calibration:

Split an account-disjoint calibration holdout
calibration_splitter = StratifiedGroupKFold(
    n_splits=5,
    shuffle=True,
    random_state=3408,
)


fit_indices, calibration_indices = next(
    calibration_splitter.split(
        train_df,
        y=train_df[LABEL_COLUMN].astype(int),
        groups=train_df["account_id"],
    )
)


lead_fit_df = train_df.iloc[fit_indices].reset_index(drop=True)
lead_calibration_df = train_df.iloc[calibration_indices].reset_index(drop=True)


assert group_ids(lead_fit_df, "account_id").isdisjoint(
    group_ids(lead_calibration_df, "account_id")
)
assert group_ids(lead_fit_df, "contact_id").isdisjoint(
    group_ids(lead_calibration_df, "contact_id")
)


for name, frame in [
    ("optimizer training", lead_fit_df),
    ("calibration", lead_calibration_df),
]:
    if frame[LABEL_COLUMN].nunique() != 2:
        raise ValueError(
            f"{name} split must contain both conversion classes. "
            "Collect more independent accounts or revise the grouped split."
        )


    print(name, "leads:", len(frame))
    print(name, "accounts:", frame["account_id"].nunique())
    print(name, "conversion rate:", frame[LABEL_COLUMN].mean())
optimizer training leads: 2400
optimizer training accounts: 614
optimizer training conversion rate: 0.20958333333333334
calibration leads: 598
calibration accounts: 153
calibration conversion rate: 0.20903010033444816

The result is 2,400 optimizer-training, 598 calibration, 1,000 validation and 1,002 test leads, about 48/12/20/20. The test fold stays untouched until the operating policy is fixed.

Step 3: Write the ICP as code

Section titled “Step 3: Write the ICP as code”

The ICP label comes from a written policy, not from LeadForge. The example policy is narrower than LeadForge’s market, so both classes occur:

  • Industry: manufacturing, logistics or professional services.
  • Region: US or UK.
  • Employee band: 200–499, 500–999 or 1,000–1,999.

A missing field returns insufficient_evidence. The 2000+ band is not_fit, because it does not prove that the company has 2,000 employees or fewer.

The ICP policy and its label counts
TARGET_INDUSTRIES = {
    "manufacturing",
    "logistics",
    "professional_services",
}


TARGET_REGIONS = {"US", "UK"}


TARGET_EMPLOYEE_BANDS = {
    "200-499",
    "500-999",
    "1000-1999",
}


def icp_label(row):
    required = ["industry", "region", "employee_band"]


    allowed_values = {
        "industry": TARGET_INDUSTRIES,
        "region": TARGET_REGIONS,
        "employee_band": TARGET_EMPLOYEE_BANDS,
    }


    known_mismatch = any(
        not pd.isna(row.get(column))
        and row[column] not in allowed
        for column, allowed in allowed_values.items()
    )


    if known_mismatch:
        return "not_fit"


    if any(pd.isna(row.get(column)) for column in required):
        return "insufficient_evidence"


    return "fit"


ICP_CLASSES = ["fit", "not_fit", "insufficient_evidence"]
TRAINABLE_ICP_CLASSES = ["fit", "not_fit"]


for name, frame in [
    ("optimizer training", lead_fit_df),
    ("calibration", lead_calibration_df),
    ("validation", validation_df),
    ("test", test_df),
]:
    labels = frame.apply(icp_label, axis=1)
    missing = set(TRAINABLE_ICP_CLASSES) - set(labels.unique())
    if missing:
        raise ValueError(
            f"{name} split is missing ICP classes: {sorted(missing)}. "
            "Review the policy and grouped split sizes before training."
        )


    print(name, labels.value_counts().reindex(ICP_CLASSES, fill_value=0).to_dict())
optimizer training {'fit': 1610, 'not_fit': 790, 'insufficient_evidence': 0}
calibration {'fit': 383, 'not_fit': 215, 'insufficient_evidence': 0}
validation {'fit': 683, 'not_fit': 317, 'insufficient_evidence': 0}
test {'fit': 672, 'not_fit': 330, 'insufficient_evidence': 0}

insufficient_evidence never occurs, because LeadForge never leaves one of the three fields empty. The model cannot learn a class that it never sees, so the application checks for missing evidence in code before the model runs.

Step 4: Turn rows into decision records

Section titled “Step 4: Turn rows into decision records”

Unsloth expects each record to hold a state, the questions and the gold answers (dataset guide). These are the two questions:

LEAD_QUESTIONS = {
    "icp_fit": {
        "type": "choice",
        "instructions": (
            "Apply the stated fictional ICP using only industry, region and employee_band. "
            "Ignore contact role, seniority, lead source, engagement and opportunity fields "
            "when deciding account fit. Return not_fit if any known required value is "
            "outside the target. Otherwise use insufficient_evidence when a required "
            "value is missing. Do not infer facts that are not present."
        ),
        "criteria": {
            "fit": "Target industry, US or UK region, and an employee band from 200 to 1999.",
            "not_fit": "At least one known firmographic value falls outside the target ICP.",
            "insufficient_evidence": "At least one required field is missing, and no known value rules out ICP fit.",
        },
    },
    "conversion_outcome_90d": {
        "type": "noul",
        "instructions": (
            "Classify whether the simulator records a closed_won event within 90 days "
            "of lead creation. Features cover only the first 30 days, so the label "
            "may include outcomes that occurred before the feature cutoff. "
            "Do not interpret this as a day-30-to-day-90 forecast."
        ),
    },
}

to_decision_dataset serializes the whitelisted features as JSON and adds the gold labels. Print one record and check that the state holds only approved features:

import json
from datasets import Dataset


def to_python_value(value):
    if pd.isna(value):
        return None
    if hasattr(value, "item"):
        return value.item()
    return value


def make_lead_record(row):
    feature_state = {
        name: to_python_value(row[name])
        for name in MODEL_FEATURES
    }


    state = {
        "task": "B2B lead qualification",
        "snapshot": "Features available through day 30 after lead creation.",
        "features": feature_state,
    }


    return {
        "state": json.dumps(state, sort_keys=True, ensure_ascii=False),
        "questions": LEAD_QUESTIONS,
        "gold": {
            "icp_fit": icp_label(row),
            "conversion_outcome_90d": bool(row[LABEL_COLUMN]),
        },
    }


def to_decision_dataset(frame, builder):
    rows = [
        builder(row)
        for row in frame.to_dict(orient="records")
    ]
    return Dataset.from_list(rows)


lead_train_raw = to_decision_dataset(lead_fit_df, make_lead_record)
lead_calibration_raw = to_decision_dataset(lead_calibration_df, make_lead_record)
lead_validation_raw = to_decision_dataset(validation_df, make_lead_record)
lead_test_raw = to_decision_dataset(test_df, make_lead_record)


print(lead_train_raw[0])
{'state': '{"features": {"activity_count": null, "buyer_role": "technical_evaluator", "days_since_first_touch": 30.333835742092244, "days_since_last_touch": 11.003359485085898, "demo_page_views": 0.0, "employee_band": "500-999", "estimated_revenue_band": "$10M-$50M", "expected_acv": 58295.50663158452, "has_open_opportunity": false, "inbound_touch_count": 5.0, "industry": "logistics", "opportunity_created": false, "opportunity_estimated_acv": null, "outbound_touch_count": 0.0, "pricing_page_views": 0.0, "process_maturity_band": "high", "region": "UK", "role_function": "it_director", "seniority": "c_suite", "session_count": 2.0, "total_session_duration_seconds": 632.0, "touch_count": 5.0, "touches_last_7_days": null, "touches_week_1": 1.0}, "snapshot": "Features available through day 30 after lead creation.", "task": "B2B lead qualification"}', 'questions': {'icp_fit': {'type': 'choice', 'instructions': 'Apply the stated fictional ICP using only industry, region and employee_band. Ignore contact role, seniority, lead source, engagement and opportunity fields when deciding account fit. Return not_fit if any known required value is outside the target. Otherwise use insufficient_evidence when a required value is missing. Do not infer facts that are not present.', 'criteria': {'fit': 'Target industry, US or UK region, and an employee band from 200 to 1999.', 'not_fit': 'At least one known firmographic value falls outside the target ICP.', 'insufficient_evidence': 'At least one required field is missing, and no known value rules out ICP fit.'}}, 'conversion_outcome_90d': {'type': 'noul', 'instructions': 'Classify whether the simulator records a closed_won event within 90 days of lead creation. Features cover only the first 30 days, so the label may include outcomes that occurred before the feature cutoff. Do not interpret this as a day-30-to-day-90 forecast.'}}, 'gold': {'icp_fit': 'fit', 'conversion_outcome_90d': False}}
Validate every record before tokenization
for name, dataset in [
    ("train", lead_train_raw),
    ("calibration", lead_calibration_raw),
    ("validation", lead_validation_raw),
    ("test", lead_test_raw),
]:
    for row in dataset:
        assert set(row["questions"]) == set(row["gold"])
        assert row["gold"]["icp_fit"] in {
            "fit",
            "not_fit",
            "insufficient_evidence",
        }
        assert isinstance(row["gold"]["conversion_outcome_90d"], bool)


    print(name, len(dataset))
train 2400
calibration 598
validation 1000
test 1002

Step 5: Load LFM3 and attach LoRA

Section titled “Step 5: Load LFM3 and attach LoRA”

The notebook loads LiquidAI/LFM3-1.7B in 4 bits with a 2,048-token limit. FastDecisionModel attaches its Clef decision head to a plain language model by default (Unsloth: train with code). LFM3 ships its own modeling code, so every load passes trust_remote_code=True, and the revision pin fixes which code runs. Read the Python files of the model repository at that revision before you run them.

from unsloth import (
    FastDecisionModel,
    DecisionTrainer,
    is_bfloat16_supported,
)
from transformers import TrainingArguments


MODEL_ID = "LiquidAI/LFM3-1.7B"
MODEL_REVISION = "ac2586a34f1a1c03329dbdcdefabd8b6478ddaa7"


print("Model:", MODEL_ID)
print("Pinned revision:", MODEL_REVISION)


model, tokenizer = FastDecisionModel.from_pretrained(
    model_name=MODEL_ID,
    revision=MODEL_REVISION,
    trust_remote_code=True,
    max_seq_length=2048,
    load_in_4bit=LOAD_IN_4BIT,
)
[... 6 lines cut ...]
Model: LiquidAI/LFM3-1.7B
Pinned revision: ac2586a34f1a1c03329dbdcdefabd8b6478ddaa7
Unsloth: WARNING `trust_remote_code` is True.
Are you certain you want to do remote code execution?
==((====))==  Unsloth 2026.10.11: Fast Lfm3 patching. Transformers: 5.22.1.
   \\   /|    NVIDIA GeForce RTX 5090 Laptop GPU. Num GPUs = 1. Max memory: 23.863 GB. Platform: Linux.
O^O/ \_/ \    Torch: 2.14.1+cu132. CUDA: 12.0. CUDA Toolkit: 13.2. Triton: 3.8.0
\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.35. FA2 = False]
 "-____-"     Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
[... 1 line cut ...]
Unsloth: Repaired remote modeling code so it trains: Lfm3ForCausalLM.forward.
Unsloth: compiling each transformer block as one region (Lfm3DecoderLayer).
[... 3 lines cut ...]

The banner confirms Unsloth 2026.10.11 with Transformers 5.22.1. Attach LoRA adapters with the guide’s settings: rank 16, alpha 16, no dropout (recommended settings, PEFT LoRA reference).

model = FastDecisionModel.get_peft_model(
    model,
    r=16,
    lora_alpha=16,
    lora_dropout=0.0,
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

Step 6: Build Unsloth’s training items

Section titled “Step 6: Build Unsloth’s training items”

FastDecisionModel.build_dataset converts each partition into the items that DecisionTrainer uses. Read its report: any skipped or truncated decision needs a fix before training.

lead_train_items_all, lead_train_report = FastDecisionModel.build_dataset(
    lead_train_raw,
    tokenizer,
    model,
)


lead_calibration_items, lead_calibration_report = FastDecisionModel.build_dataset(
    lead_calibration_raw,
    tokenizer,
    model,
)


lead_validation_items, lead_validation_report = FastDecisionModel.build_dataset(
    lead_validation_raw,
    tokenizer,
    model,
)


lead_test_items, lead_test_report = FastDecisionModel.build_dataset(
    lead_test_raw,
    tokenizer,
    model,
)


for name, report in [
    ("train", lead_train_report),
    ("calibration", lead_calibration_report),
    ("validation", lead_validation_report),
    ("test", lead_test_report),
]:
    print(name, report)
    if report["skipped"]:
        raise ValueError(
            f"{name} dataset contains skipped decisions: {report['reason']}"
        )
train {'total': 4800, 'skipped': 0, 'reason': None, 'truncated': 0}
calibration {'total': 1196, 'skipped': 0, 'reason': None, 'truncated': 0}
validation {'total': 2000, 'skipped': 0, 'reason': None, 'truncated': 0}
test {'total': 2004, 'skipped': 0, 'reason': None, 'truncated': 0}

Each lead holds two questions, so 2,400 leads give 4,800 decisions, and none is skipped or truncated. Calibration needs at least ten decisions of each question type, so the notebook checks the type codes of the calibration items:

Check the calibration items
lead_train_items = lead_train_items_all


print("Training rows:", len(lead_train_items))
print("Calibration rows:", len(lead_calibration_items))
print("Validation rows:", len(lead_validation_items))
print("Test rows:", len(lead_test_items))


from collections import Counter


lead_calibration_counts = Counter(
    question_type
    for item in lead_calibration_items
    for question_type in item["qtypes"]
)


if (
    len(lead_calibration_counts) != len(LEAD_QUESTIONS)
    or min(lead_calibration_counts.values()) < 10
):
    raise ValueError(
        "Provide at least ten calibration decisions for both lead question types. "
        f"Observed counts: {dict(lead_calibration_counts)}"
    )
Training rows: 2400
Calibration rows: 598
Validation rows: 1000
Test rows: 1002

Step 7: Measure two baselines

Section titled “Step 7: Measure two baselines”

A fine-tuned model needs something to beat. The first baseline is the untrained decision head:

lead_baseline = FastDecisionModel.evaluate(
    model,
    tokenizer,
    lead_validation_items,
)


print("Untrained decision-head validation metrics:", lead_baseline)
Untrained decision-head validation metrics: {'accuracy': 0.395, 'ece': 0.31961952917277814, 'loss': 0.9056575514376164, 'record_accuracy': 0.0}

It answers both questions correctly for no lead. The second baseline is a logistic regression on the same features and the same optimizer-training rows. On tabular CRM data, it is the honest comparison.

Fit the logistic-regression baseline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.metrics import average_precision_score, brier_score_loss, roc_auc_score


CATEGORICAL_FEATURES = [
    "industry",
    "region",
    "employee_band",
    "estimated_revenue_band",
    "process_maturity_band",
    "role_function",
    "seniority",
    "buyer_role",
]


NUMERIC_FEATURES = [
    name
    for name in MODEL_FEATURES
    if name not in CATEGORICAL_FEATURES
]


numeric_pipeline = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="median")),
        ("scaler", StandardScaler()),
    ]
)


categorical_pipeline = Pipeline(
    steps=[
        (
            "imputer",
            SimpleImputer(
                strategy="constant",
                fill_value="__missing__",
            ),
        ),
        (
            "onehot",
            OneHotEncoder(
                handle_unknown="ignore",
                min_frequency=10,
            ),
        ),
    ]
)


preprocessor = ColumnTransformer(
    transformers=[
        ("numeric", numeric_pipeline, NUMERIC_FEATURES),
        ("categorical", categorical_pipeline, CATEGORICAL_FEATURES),
    ]
)


logistic_baseline = Pipeline(
    steps=[
        ("preprocessor", preprocessor),
        ("classifier", LogisticRegression(max_iter=2000)),
    ]
)


logistic_baseline.fit(
    lead_fit_df[MODEL_FEATURES],
    lead_fit_df[LABEL_COLUMN].astype(int),
)


positive_class_index = list(
    logistic_baseline.named_steps["classifier"].classes_
).index(1)


logistic_validation_probabilities = logistic_baseline.predict_proba(
    validation_df[MODEL_FEATURES]
)[:, positive_class_index]


validation_y = validation_df[LABEL_COLUMN].astype(int)


print(
    "Logistic baseline validation 90-day outcome ROC-AUC:",
    roc_auc_score(validation_y, logistic_validation_probabilities),
)
print(
    "Logistic baseline validation 90-day outcome average precision:",
    average_precision_score(validation_y, logistic_validation_probabilities),
)
print(
    "Logistic baseline validation Brier score:",
    brier_score_loss(validation_y, logistic_validation_probabilities),
)
Logistic baseline validation 90-day outcome ROC-AUC: 0.8627305605786617
Logistic baseline validation 90-day outcome average precision: 0.5303466286953371
Logistic baseline validation Brier score: 0.11834991428419404

LeadForge publishes its own logistic baseline, ROC-AUC 0.880 on its supplied test split (baseline results). That figure uses the row-level split and includes the leakage field, so it is a reference, not a target for this account-disjoint split.

Step 8: Fine-tune

Section titled “Step 8: Fine-tune”

The settings follow Unsloth’s guide: learning rate 2e-4 for the adapters, 1e-4 for the decision head, two epochs. A per-device batch of 8 with 4 accumulation steps gives 32 leads, or 64 decisions, per optimizer update (recommended settings). eval_strategy="epoch" records the validation loss after each epoch.

trainer = DecisionTrainer(
    model=model,
    processing_class=tokenizer,
    train_dataset=lead_train_items,
    eval_dataset=lead_validation_items,
    head_learning_rate=1e-4,
    args=TrainingArguments(
        per_device_train_batch_size=8,
        gradient_accumulation_steps=4,
        warmup_steps=10,
        num_train_epochs=2,
        learning_rate=2e-4,
        lr_scheduler_type="cosine",
        weight_decay=0.01,
        bf16=is_bfloat16_supported(),
        fp16=not is_bfloat16_supported(),
        eval_strategy="epoch",
        logging_steps=10,
        output_dir="outputs/lead-decision",
        report_to="none",
        seed=3407,
    ),
)
_train_start = time.perf_counter()
trainer_stats = trainer.train()
print(trainer_stats.metrics)
Unsloth: Will smartly offload gradients to save VRAM!
Unsloth: Double buffering enabled (parallel H2D + compute) for backward pass.
[... 2 lines cut ...]
{'train_runtime': 967.5274, 'train_samples_per_second': 4.961, 'train_steps_per_second': 0.155, 'total_flos': 0.0, 'train_loss': 0.23644883632659913, 'epoch': 2.0}
TIMINGS["lead_training"] = round(time.perf_counter() - _train_start, 1)
peak_vram_gb = round(torch.cuda.max_memory_allocated() / 1024**3, 2)
print("Training seconds:", TIMINGS["lead_training"], "| peak VRAM GB:", peak_vram_gb)
Training seconds: 1005.0 | peak VRAM GB: 2.73
Lead model training loss over 150 optimizer steps: it falls from 0.73 at step 10 to 0.15 at step 60, then stays between 0.14 and 0.21; the validation loss is 0.20 after epoch 1 and about 0.18 after epoch 2

The training loss falls fast and flattens after about 60 steps. The validation loss still falls between the two epochs, so two epochs show no sign of overfitting. If memory runs out, lower per_device_train_batch_size and raise gradient_accumulation_steps.

Step 9: Calibrate, then evaluate

Section titled “Step 9: Calibrate, then evaluate”

Calibration fits temperatures on the account-disjoint holdout, after training and before any test evaluation (Unsloth: calibrate and save). It changes how confident the probabilities are; it does not change which answer is right.

lead_calibration_metrics = FastDecisionModel.calibrate(
    model,
    tokenizer,
    lead_calibration_items,
)


print(lead_calibration_metrics)
{'accuracy': 0.8954849498327759, 'ece': 0.005321504728849912, 'loss': 0.1711510666748503, 'record_accuracy': 0.7909698996655519, 'fitted_types': [0, 2]}

fitted_types lists two type codes, one for each question. Evaluate on validation first:

lead_validation_metrics = FastDecisionModel.evaluate(
    model,
    tokenizer,
    lead_validation_items,
)


print("Fine-tuned validation metrics:", lead_validation_metrics)
Fine-tuned validation metrics: {'accuracy': 0.8985, 'ece': 0.016541349023580572, 'loss': 0.1805455202129442, 'record_accuracy': 0.797}

The evaluator reports overall decision metrics. A helper calls FastDecisionModel.predict for each lead and computes metrics per question; a timer measures the mean prediction time.

_predict_start = time.perf_counter()
Per-question validation metrics
from sklearn.metrics import (
    average_precision_score,
    brier_score_loss,
    classification_report,
    roc_auc_score,
)


def option_probability(probabilities, option):
    for key, value in probabilities.items():
        if str(key).lower() == option.lower():
            return float(value)
    raise ValueError(f"No {option!r} probability in {probabilities}")


def true_probability(probabilities):
    return option_probability(probabilities, "true")


def predict_leads(frame, decision_model, decision_tokenizer):
    predictions = []


    for row in frame.to_dict(orient="records"):
        example = make_lead_record(row)


        result = FastDecisionModel.predict(
            decision_model,
            decision_tokenizer,
            example["state"],
            LEAD_QUESTIONS,
        )


        predictions.append(
            {
                "icp_fit": result["icp_fit"]["answer"],
                "icp_fit_probability": option_probability(
                    result["icp_fit"]["probabilities"],
                    "fit",
                ),
                "conversion_outcome_prediction": result["conversion_outcome_90d"]["answer"],
                "outcome_probability": true_probability(
                    result["conversion_outcome_90d"]["probabilities"]
                ),
            }
        )


    return pd.DataFrame(predictions)


def precision_at_k(y_true, scores, k=100):
    k = min(k, len(y_true))
    selected = np.argsort(scores)[-k:]
    return float(np.asarray(y_true)[selected].mean())


FastDecisionModel.for_inference(model)


lead_validation_predictions = predict_leads(
    validation_df,
    model,
    tokenizer,
)


validation_icp_labels = validation_df.apply(icp_label, axis=1)


print(
    classification_report(
        validation_icp_labels,
        lead_validation_predictions["icp_fit"],
        labels=["fit", "not_fit", "insufficient_evidence"],
        zero_division=0,
    )
)


validation_y = validation_df[LABEL_COLUMN].astype(int)
validation_scores = lead_validation_predictions["outcome_probability"]


print("90-day outcome ROC-AUC:", roc_auc_score(validation_y, validation_scores))
print("90-day outcome average precision:", average_precision_score(validation_y, validation_scores))
print("90-day outcome Brier score:", brier_score_loss(validation_y, validation_scores))
print("90-day outcome Precision@100:", precision_at_k(validation_y, validation_scores, 100))
                       precision    recall  f1-score   support


                  fit       1.00      1.00      1.00       683
              not_fit       1.00      1.00      1.00       317
insufficient_evidence       0.00      0.00      0.00         0


             accuracy                           1.00      1000
            macro avg       0.67      0.67      0.67      1000
         weighted avg       1.00      1.00      1.00      1000


90-day outcome ROC-AUC: 0.8607142857142857
90-day outcome average precision: 0.5331324006774082
90-day outcome Brier score: 0.12077270216
90-day outcome Precision@100: 0.55
mean_predict_ms = round((time.perf_counter() - _predict_start) * 1000 / len(validation_df), 1)
print("Mean milliseconds per lead prediction:", mean_predict_ms)
Mean milliseconds per lead prediction: 92.1

ROC-AUC measures ranking across all thresholds. Average precision suits a rare positive class. Brier score measures probability error, and precision at 100 asks how many of the top 100 leads converted (scikit-learn: model evaluation). Fix the policy on validation, then score the test fold once:

Score the test fold once
lead_test_metrics = FastDecisionModel.evaluate(
    model,
    tokenizer,
    lead_test_items,
)


FastDecisionModel.for_inference(model)


lead_test_predictions = predict_leads(
    test_df,
    model,
    tokenizer,
)


test_icp_labels = test_df.apply(icp_label, axis=1)


print(
    classification_report(
        test_icp_labels,
        lead_test_predictions["icp_fit"],
        labels=["fit", "not_fit", "insufficient_evidence"],
        zero_division=0,
    )
)


test_y = test_df[LABEL_COLUMN].astype(int)
test_scores = lead_test_predictions["outcome_probability"]


print("Unsloth test metrics:", lead_test_metrics)
print("90-day outcome ROC-AUC:", roc_auc_score(test_y, test_scores))
print("90-day outcome average precision:", average_precision_score(test_y, test_scores))
print("90-day outcome Brier score:", brier_score_loss(test_y, test_scores))
print("90-day outcome Precision@100:", precision_at_k(test_y, test_scores, 100))


logistic_test_probabilities = logistic_baseline.predict_proba(
    test_df[MODEL_FEATURES]
)[:, positive_class_index]


print(
    "Logistic baseline test 90-day outcome ROC-AUC:",
    roc_auc_score(test_y, logistic_test_probabilities),
)
print(
    "Logistic baseline test 90-day outcome average precision:",
    average_precision_score(test_y, logistic_test_probabilities),
)
print(
    "Logistic baseline test Brier score:",
    brier_score_loss(test_y, logistic_test_probabilities),
)
print(
    "Logistic baseline test 90-day outcome Precision@100:",
    precision_at_k(test_y, logistic_test_probabilities, 100),
)
                       precision    recall  f1-score   support


                  fit       1.00      1.00      1.00       672
              not_fit       1.00      1.00      1.00       330
insufficient_evidence       0.00      0.00      0.00         0


             accuracy                           1.00      1002
            macro avg       0.67      0.67      0.67      1002
         weighted avg       1.00      1.00      1.00      1002


Unsloth test metrics: {'accuracy': 0.8932135728542914, 'ece': 0.0010656011616160988, 'loss': 0.15622826673260332, 'record_accuracy': 0.7864271457085829}
90-day outcome ROC-AUC: 0.8691889203779486
90-day outcome average precision: 0.5166622361399306
90-day outcome Brier score: 0.10905676169660679
90-day outcome Precision@100: 0.51
Logistic baseline test 90-day outcome ROC-AUC: 0.8757526917154481
Logistic baseline test 90-day outcome average precision: 0.5495348201164751
Logistic baseline test Brier score: 0.11052304954493032
Logistic baseline test 90-day outcome Precision@100: 0.54
ROC and precision-recall curves on 1,002 test leads: the fine-tuned LFM3 model reaches ROC-AUC 0.869 and average precision 0.517, the logistic regression 0.876 and 0.550

The two ROC curves almost overlap. The logistic regression keeps a small lead at the top of the ranking, which is where average precision and precision at 100 look.

Reliability diagram for the 90-day conversion probability on test leads, in ten equal-count bins: the fine-tuned model's points stay near the diagonal but stop at a predicted probability of about 0.5, while the logistic regression reaches 0.65

The reliability diagram shows why the Brier score is good: the model sorts leads into two groups. Most leads get a probability near zero, and few of them convert. The rest get a probability near 0.5, and about half of them convert. It also shows a limit: the highest test probability is 0.504, so the model almost never answers yes.

Plot style
import matplotlib


matplotlib.use("Agg")
import matplotlib.pyplot as plt
from IPython.display import Image, display


SURFACE = "#fcfcfb"
INK = "#0b0b0b"
INK_SECONDARY = "#52514e"
MUTED = "#898781"
GRID = "#e1e0d9"
SERIES = ["#2a78d6", "#eb6834", "#1baf7a", "#eda100"]
FIGURES = Path("outputs/figures")
FIGURES.mkdir(parents=True, exist_ok=True)


plt.rcParams.update(
    {
        "figure.facecolor": SURFACE,
        "axes.facecolor": SURFACE,
        "axes.edgecolor": GRID,
        "axes.labelcolor": INK_SECONDARY,
        "axes.titlecolor": INK,
        "axes.titlesize": 12,
        "axes.titleweight": "semibold",
        "axes.titlelocation": "left",
        "axes.grid": True,
        "grid.color": GRID,
        "grid.linewidth": 0.6,
        "axes.spines.top": False,
        "axes.spines.right": False,
        "xtick.color": MUTED,
        "ytick.color": MUTED,
        "legend.frameon": False,
        "legend.labelcolor": INK_SECONDARY,
        "lines.linewidth": 2,
        "font.size": 10,
    }
)




def save_figure(figure, name):
    path = FIGURES / f"{name}.png"
    figure.savefig(path, dpi=150, bbox_inches="tight", facecolor=SURFACE)
    plt.close(figure)
    display(Image(filename=str(path)))
    print("Saved", path)
Draw the three figures
from sklearn.calibration import calibration_curve
from sklearn.metrics import precision_recall_curve, roc_curve


pd.DataFrame(
    {
        "converted_within_90_days": test_y.to_numpy(),
        "model_probability": test_scores.to_numpy(),
        "logistic_probability": logistic_test_probabilities,
        "icp_label": test_icp_labels.to_numpy(),
        "icp_predicted": lead_test_predictions["icp_fit"].to_numpy(),
    }
).to_csv("outputs/lead-test-predictions.csv", index=False)


train_points = [(entry["step"], entry["loss"]) for entry in trainer.state.log_history if "loss" in entry]
eval_points = [(entry["step"], entry["eval_loss"]) for entry in trainer.state.log_history if "eval_loss" in entry]
figure, axis = plt.subplots(figsize=(8, 4.5))
axis.plot(*zip(*train_points), color=SERIES[0], label="Training loss")
axis.plot(*zip(*eval_points), color=SERIES[1], marker="o", markersize=8, label="Validation loss (end of each epoch)")
axis.set_title("Lead model: loss during fine-tuning")
axis.set_xlabel("Optimizer step")
axis.set_ylabel("Decision loss")
axis.legend(loc="upper right")
save_figure(figure, "lead-training-loss")


figure, (roc_axis, pr_axis) = plt.subplots(1, 2, figsize=(10, 4.5))
for scores, label, color in [
    (test_scores, f"Fine-tuned LFM3 (AUC {roc_auc_score(test_y, test_scores):.3f})", SERIES[0]),
    (logistic_test_probabilities, f"Logistic regression (AUC {roc_auc_score(test_y, logistic_test_probabilities):.3f})", SERIES[1]),
]:
    false_positive_rate, true_positive_rate, _ = roc_curve(test_y, scores)
    roc_axis.plot(false_positive_rate, true_positive_rate, color=color, label=label)
roc_axis.plot([0, 1], [0, 1], color=MUTED, linewidth=1, linestyle="--", label="Random ranking")
roc_axis.set_title("ROC curve, 1,002 test leads")
roc_axis.set_xlabel("False positive rate")
roc_axis.set_ylabel("True positive rate")
roc_axis.legend(loc="lower right")
for scores, label, color in [
    (test_scores, f"Fine-tuned LFM3 (AP {average_precision_score(test_y, test_scores):.3f})", SERIES[0]),
    (logistic_test_probabilities, f"Logistic regression (AP {average_precision_score(test_y, logistic_test_probabilities):.3f})", SERIES[1]),
]:
    precision, recall, _ = precision_recall_curve(test_y, scores)
    pr_axis.plot(recall, precision, color=color, label=label)
pr_axis.axhline(test_y.mean(), color=MUTED, linewidth=1, linestyle="--", label=f"Base rate {test_y.mean():.2f}")
pr_axis.set_title("Precision-recall curve")
pr_axis.set_xlabel("Recall")
pr_axis.set_ylabel("Precision")
pr_axis.legend(loc="upper right")
figure.tight_layout()
save_figure(figure, "lead-roc-pr")


figure, axis = plt.subplots(figsize=(8, 4.5))
axis.plot([0, 1], [0, 1], color=MUTED, linewidth=1, linestyle="--", label="Perfect calibration")
for scores, label, color in [
    (test_scores, f"Fine-tuned LFM3 (Brier {brier_score_loss(test_y, test_scores):.4f})", SERIES[0]),
    (logistic_test_probabilities, f"Logistic regression (Brier {brier_score_loss(test_y, logistic_test_probabilities):.4f})", SERIES[1]),
]:
    observed, predicted = calibration_curve(test_y, scores, n_bins=10, strategy="quantile")
    axis.plot(predicted, observed, color=color, marker="o", markersize=8, label=label)
axis.set_title("Calibration of the 90-day conversion probability, test leads")
axis.set_xlabel("Mean predicted probability (10 equal-count bins)")
axis.set_ylabel("Observed conversion rate")
axis.legend(loc="upper left")
save_figure(figure, "lead-calibration")

Lead model results

Section titled “Lead model results”

The notebook writes every number to outputs/lead-metrics.json. No account appears in two partitions.

90-day conversion Split Fine-tuned LFM3-1.7B Logistic regression
ROC-AUC Validation 0.8607 0.8627
ROC-AUC Test 0.8692 0.8758
Average precision Validation 0.5331 0.5303
Average precision Test 0.5167 0.5495
Brier score (lower is better) Validation 0.1208 0.1183
Brier score (lower is better) Test 0.1091 0.1105
Precision at 100 Validation 0.55 0.61
Precision at 100 Test 0.51 0.54
  • Both questions. On test, the decision accuracy over both questions is 0.893, and the ECE is 0.001. The record accuracy, the share of leads with both answers right, is 0.786.
  • The conversion yes/no answer. Every ICP answer is right, so the record accuracy is the accuracy of the conversion answer. On test, 0.786 is slightly below the 0.789 of a constant “no”; on validation, 0.797 is slightly above 0.790. For this question, use the probability, not the yes/no answer.
  • Cost. Training took 1,005 s by the notebook’s timer, with a peak of 2.73 GB of GPU memory. A prediction took 92.1 ms per lead on average. These figures come from one GPU and one seed.

Four training seeds

Section titled “Four training seeds”

One run says little about spread. tools/seed_study.py trained the lead model three more times with training seeds 1, 2 and 3 on the same splits. It stops if the logistic regression differs between runs, which would mean a different split. The summary is in outputs/seed-study.json.

Test metric, four runs Mean Range Seed 3407 (published) Logistic regression
90-day ROC-AUC 0.871 0.869 to 0.874 0.869 0.876
90-day average precision 0.531 0.517 to 0.548 0.517 0.550
90-day Brier score (lower is better) 0.1092 0.1087 to 0.1099 0.1091 0.1105
90-day precision at 100 0.53 0.51 to 0.55 0.51 0.54
ECE, both questions 0.005 0.001 to 0.012 0.001 not applicable

By ROC-AUC and average precision, every run ranks conversions slightly below the logistic regression, and every run has a slightly lower Brier score. Precision at 100 moves around the baseline’s 0.54. Four runs show the spread of one setup; they are not a confidence interval.

What the ICP accuracy of 1.0 means

Section titled “What the ICP accuracy of 1.0 means”

The ICP label is a deterministic function of three fields that are never empty here, so the model learns a lookup rule on complete records. Real records are often incomplete. In the first recorded session of page 3, the model answered fit with a probability of about 1.0 for real companies whose records lacked the employee band or the region. Since then, build_lead_request in integration/gate.py refuses a record with a missing field or an unknown value before any model call. Handle missing evidence in code, not in the model.

What the run shows

Section titled “What the run shows”

On tabular features, a fine-tuned 1.7B decision model does not beat a logistic regression at ranking conversions. It comes close (ROC-AUC 0.869 against 0.876), with a slightly lower Brier score (0.1091 against 0.1105) and two typed answers from one call. If you only need a conversion score from a CRM table, start with the logistic regression. A decision model earns its place when the state is text, when one call must answer several typed questions, or when an agent needs one interface for many decisions. Page 2 shows the text case, where the fine-tuned model clearly wins.

Step 10: Turn predictions into actions

Section titled “Step 10: Turn predictions into actions”

The model predicts; the application decides. lead_action gives an ICP-fit account high priority, sends an account with missing evidence to research and keeps a non-fit account out of the queue. A separate persona rule checks the contact’s buyer role, and an external eligibility status (approved, ineligible or unknown) blocks or escalates. Every branch returns send as False: the function recommends an action, and a person approves every outreach.

TARGET_BUYER_ROLES = {
    "economic_buyer",
    "champion",
    "technical_evaluator",
}


def evaluate_contact_persona_fit(buyer_role):
    if pd.isna(buyer_role):
        return None


    return str(buyer_role).strip().lower() in TARGET_BUYER_ROLES
The outreach policy, lead_action
def lead_action(
    prediction,
    suppressed,
    contact_persona_fit,
    contact_eligibility,
):
    if isinstance(suppressed, np.bool_):
        suppressed = bool(suppressed)
    if isinstance(contact_persona_fit, np.bool_):
        contact_persona_fit = bool(contact_persona_fit)


    if suppressed is True:
        return {
            "priority": "blocked",
            "action": "suppress",
            "send": False,
        }


    if suppressed is not False:
        return {
            "priority": "review",
            "action": "verify_suppression_status",
            "send": False,
        }


    fit = prediction.get("icp_fit")


    if fit not in {"fit", "not_fit", "insufficient_evidence"}:
        return {
            "priority": "review",
            "action": "human_icp_review",
            "send": False,
        }


    if fit == "insufficient_evidence":
        return {
            "priority": "review",
            "action": "research_missing_account_evidence",
            "send": False,
        }


    if fit == "not_fit":
        return {
            "priority": "low",
            "action": "do_not_recommend_outreach",
            "send": False,
        }


    if contact_persona_fit is False:
        return {
            "priority": "high",
            "icp_fit_probability": prediction.get("icp_fit_probability"),
            "action": "find_relevant_contact",
            "send": False,
        }


    if contact_persona_fit is not True:
        return {
            "priority": "high",
            "icp_fit_probability": prediction.get("icp_fit_probability"),
            "action": "human_persona_review",
            "send": False,
        }


    if contact_eligibility == "ineligible":
        return {
            "priority": "blocked",
            "action": "do_not_contact",
            "send": False,
        }


    if contact_eligibility != "approved":
        return {
            "priority": "high",
            "icp_fit_probability": prediction.get("icp_fit_probability"),
            "action": "human_eligibility_review",
            "send": False,
        }


    return {
        "priority": "high",
        "icp_fit_probability": prediction.get("icp_fit_probability"),
        "action": "recommend_outreach_for_approval",
        "send": False,
    }

Apply the policy to the first validation lead, which is not an ICP fit:

sample_lead = lead_validation_predictions.iloc[0].to_dict()


print(
    lead_action(
        sample_lead,
        suppressed=False,
        contact_persona_fit=evaluate_contact_persona_fit("champion"),
        contact_eligibility="unknown",
    )
)
{'priority': 'low', 'action': 'do_not_recommend_outreach', 'send': False}

The notebook also asserts the fail-closed branches: a suppressed contact, an unknown suppression status, an ineligible contact, an unexpected ICP label, and a suppression flag that arrives as a NumPy boolean.

Regression checks for the policy gates
fit_prediction = {"icp_fit": "fit"}


assert lead_action(
    fit_prediction,
    suppressed=True,
    contact_persona_fit=True,
    contact_eligibility="approved",
)["action"] == "suppress"


assert lead_action(
    fit_prediction,
    suppressed=None,
    contact_persona_fit=True,
    contact_eligibility="approved",
)["action"] == "verify_suppression_status"


assert lead_action(
    fit_prediction,
    suppressed=False,
    contact_persona_fit=True,
    contact_eligibility="ineligible",
)["action"] == "do_not_contact"


assert lead_action(
    {"icp_fit": "unexpected_label"},
    suppressed=False,
    contact_persona_fit=True,
    contact_eligibility="approved",
)["action"] == "human_icp_review"


assert lead_action(
    {"icp_fit": "fit"},
    suppressed=np.bool_(True),
    contact_persona_fit=True,
    contact_eligibility="approved",
)["action"] == "suppress"

integration/policy.py holds the same lead_action for page 3, and tests/test_gate.py pins the gates around it. Commercial email law still applies to B2B outreach (FTC CAN-SPAM guide, ICO B2B marketing).

Step 11: Save, publish and reload

Section titled “Step 11: Save, publish and reload”

save_pretrained saves the adapter and the decision head. save_pretrained_merged exports a merged 16-bit model (Unsloth: calibrate and save).

model.save_pretrained("outputs/lead-decision-adapter")
model.save_pretrained_merged(
    "outputs/lead-decision-merged",
    tokenizer,
)
[... 6 lines cut ...]
/home/dev/work/lead-decision-models/unsloth_env/lib/python3.13/site-packages/unsloth_zoo/saving_utils.py:3849: UserWarning: Unsloth: `LiquidAI/LFM3-1.7B` was loaded without a recorded commit, so its repo code was not copied into the export. Copy the repo's *.py files at the revision you loaded into `/home/dev/work/lead-decision-models/outputs/lead-decision-merged/.unsloth-save-ieqciakh` before loading it with trust_remote_code=True.
  warnings.warn(
Found HuggingFace hub cache directory: /home/dev/.cache/huggingface/hub
Checking cache directory for required files...
[... 1 line cut ...]
Successfully copied all 1 files from cache to `/home/dev/work/lead-decision-models/outputs/lead-decision-merged/.unsloth-save-ieqciakh`
Checking cache directory for required files...
Cache check failed: tokenizer.model not found in local cache.
Not all required files found in cache. Will proceed with downloading.
[... 2 lines cut ...]
Unsloth: Merge process complete. Saved to `/home/dev/work/lead-decision-models/outputs/lead-decision-merged/.unsloth-save-ieqciakh`

Unsloth warns that it did not copy LFM3’s Python files into the export. In the recorded run, the merged folder still holds configuration_lfm3.py, modeling_lfm3.py and lfm3_conv.py, and the reload below works. Check your own export for these files before you load it with trust_remote_code=True.

When PUBLISH is True, the next cell uploads the adapter and the merged model:

LEAD_REPO = f"{HF_USERNAME}/lead-decision-models-lead-qualifier"


if PUBLISH:
    model.push_to_hub(LEAD_REPO, tokenizer)
    model.push_to_hub_merged(f"{LEAD_REPO}-merged", tokenizer, save_method="merged_16bit")
    print("Published", LEAD_REPO, "and", f"{LEAD_REPO}-merged")
[... 1 line cut ...]
Unsloth: Saved the decision model to https://huggingface.co/raoofaltaher/lead-decision-models-lead-qualifier
[... 13 lines cut ...]
Unsloth: Saved the decision model to https://huggingface.co/raoofaltaher/lead-decision-models-lead-qualifier-merged
Published raoofaltaher/lead-decision-models-lead-qualifier and raoofaltaher/lead-decision-models-lead-qualifier-merged

Both Hub models derive from LiquidAI/LFM3-1.7B and fall under the LFM Open License v1.0, which licenses commercial use only below a revenue threshold. They are teaching models trained on synthetic data. outputs/hub-check.json records that each Hub copy holds the files of the run, reloads and predicts the same classes.

Release the training model, then reload the merged checkpoint and compare one prediction with the prediction before export:

import gc


del trainer, model, tokenizer, trainer_stats
gc.collect()
torch.cuda.empty_cache()
reloaded_lead_model, reloaded_lead_tokenizer = FastDecisionModel.from_pretrained(
    model_name="outputs/lead-decision-merged",
    trust_remote_code=True,
    max_seq_length=2048,
    load_in_4bit=LOAD_IN_4BIT,
)


FastDecisionModel.for_inference(reloaded_lead_model)


lead_example = make_lead_record(test_df.iloc[0])
lead_reload_prediction = FastDecisionModel.predict(
    reloaded_lead_model,
    reloaded_lead_tokenizer,
    lead_example["state"],
    LEAD_QUESTIONS,
)


print(lead_reload_prediction)


lead_before_export = lead_test_predictions.iloc[0]
print(
    "Lead class agreement after export:",
    lead_reload_prediction["icp_fit"]["answer"] == lead_before_export["icp_fit"],
    lead_reload_prediction["conversion_outcome_90d"]["answer"]
    == bool(lead_before_export["conversion_outcome_prediction"]),
)
Unsloth: WARNING `trust_remote_code` is True.
Are you certain you want to do remote code execution?
==((====))==  Unsloth 2026.10.11: Fast Lfm3 patching. Transformers: 5.22.1.
   \\   /|    NVIDIA GeForce RTX 5090 Laptop GPU. Num GPUs = 1. Max memory: 23.863 GB. Platform: Linux.
O^O/ \_/ \    Torch: 2.14.1+cu132. CUDA: 12.0. CUDA Toolkit: 13.2. Triton: 3.8.0
\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.35. FA2 = False]
 "-____-"     Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
[... 1 line cut ...]
Unsloth: Repaired remote modeling code so it trains: Lfm3ForCausalLM.forward.
Unsloth: compiling each transformer block as one region (Lfm3DecoderLayer).
{'icp_fit': {'type': 'choice', 'choice': 'fit', 'confidence': 1.0, 'probabilities': {'fit': 1.0, 'not_fit': 0.0, 'insufficient_evidence': 0.0}, 'answer': 'fit'}, 'conversion_outcome_90d': {'type': 'noul', 'noul': 0.0047, 'answer': False, 'probabilities': {'true': 0.0047, 'false': 0.9953}}}
Lead class agreement after export: True True

Both classes match. A merged 16-bit checkpoint reloaded in 4 bits can move a probability slightly; investigate only a change of class or route.

del reloaded_lead_model, reloaded_lead_tokenizer
gc.collect()
torch.cuda.empty_cache()

The last cell writes the metrics file that the tables above quote:

Write outputs/lead-metrics.json
import json


def outcome_metrics(y_true, scores):
    return {
        "roc_auc": float(roc_auc_score(y_true, scores)),
        "average_precision": float(average_precision_score(y_true, scores)),
        "brier": float(brier_score_loss(y_true, scores)),
        "precision_at_100": float(precision_at_k(y_true, scores, 100)),
    }


lead_metrics = {
    "model_id": MODEL_ID,
    "model_revision": MODEL_REVISION,
    "load_in_4bit": LOAD_IN_4BIT,
    "dataset": {
        "leadforge_commit": DATASET_COMMIT,
        "target_counts": {str(key): int(value) for key, value in leads[LABEL_COLUMN].value_counts().items()},
    },
    "splits": {
        "optimizer_training": len(lead_fit_df),
        "calibration": len(lead_calibration_df),
        "validation": len(validation_df),
        "test": len(test_df),
    },
    "baseline_untrained": lead_baseline,
    "logistic": {
        "validation": outcome_metrics(validation_y, logistic_validation_probabilities),
        "test": outcome_metrics(test_y, logistic_test_probabilities),
    },
    "calibration": lead_calibration_metrics,
    "validation": {**lead_validation_metrics, **outcome_metrics(validation_y, validation_scores)},
    "test": {**lead_test_metrics, **outcome_metrics(test_y, test_scores)},
    "icp_report": {
        "validation": classification_report(validation_icp_labels, lead_validation_predictions["icp_fit"], output_dict=True, zero_division=0),
        "test": classification_report(test_icp_labels, lead_test_predictions["icp_fit"], output_dict=True, zero_division=0),
    },
    "timings_seconds": TIMINGS,
    "peak_vram_gb": peak_vram_gb,
    "mean_predict_ms": mean_predict_ms,
}


Path("outputs").mkdir(exist_ok=True)
Path("outputs/lead-metrics.json").write_text(json.dumps(lead_metrics, indent=2, default=float), encoding="utf-8")
print(json.dumps({key: lead_metrics[key] for key in ["splits", "logistic", "test", "peak_vram_gb", "mean_predict_ms", "timings_seconds"]}, indent=2, default=float))

Troubleshooting

Section titled “Troubleshooting”

The GPU is not visible

Section titled “The GPU is not visible”

Sync the environment again, or use the two-index form if --torch-backend=auto does not pick the CUDA 13.2 build. On WSL2, check nvidia-smi inside Ubuntu; the driver comes from Windows.

LFM3 does not load or train

Section titled “LFM3 does not load or train”

use_kernel_func_from_hub_with_fallback, KeyError: 'linear_attention' or AutoConfig error: 'conv' means a Transformers release that the probe recorded as failing. Sync requirements-experiment.txt again, and do not let a resolver move Transformers back into Unsloth’s declared range.

Unsloth skips decisions

Section titled “Unsloth skips decisions”

Read the build_dataset report. Check that every question has a matching gold key and that every label matches its question type (dataset builder).

Calibration looks good, but live results fall

Section titled “Calibration looks good, but live results fall”

Calibration describes only the data that fitted it. Check for changes in industry, region, company size, data collection and label definition. Recalibrate on recent data and keep a later untouched test period.

The model recommends outreach to a suppressed contact

Section titled “The model recommends outreach to a suppressed contact”

The policy is incomplete. Suppression must run in code, before any action. Fix the gate and add a regression test; more training data does not fix it.