---
title: "Fine-Tune LFM3, a Large Language Model, into a Decision Model with Unsloth: Lead Qualification with Claude Code over MCP"
url: "https://raoofaltaher.com/tutorials/fine-tune-lfm3-decision-model-with-unsloth/claude-code-mcp"
description: "Serve two fine-tuned Unsloth decision models to Claude Code through a small MCP server, research real prospects with Firecrawl, and gate every action for human approval."
---

# Fine-Tune LFM3, a Large Language Model, into a Decision Model with Unsloth: Lead Qualification with Claude Code over MCP

Page 3 of 3: Claude Code over MCP

-   unsloth
-   fine-tuning
-   lfm3
-   decision-models
-   claude-code
-   mcp
-   case-study
-   practical-ai

This page puts the two models to work. A Claude Code agent researches real companies with Firecrawl, asks the fine-tuned lead and reply models each bounded question through a small MCP server, and hands every answer to a deterministic gate. The gate writes a recommended action to a queue that a person approves. No tool in the setup can send a message, and every result carries `"send": false`. The page teaches three things: how to write an MCP server for a local model, how to connect it to Claude Code, and how to record an agent session so that others can check it.

[Page 1](/tutorials/fine-tune-lfm3-decision-model-with-unsloth) and [page 2](/tutorials/fine-tune-lfm3-decision-model-with-unsloth/reply-routing) trained the two models. Every code block on this page is a file of the companion repository [lead-decision-models](https://github.com/raoofaltaher/lead-decision-models) or an excerpt of one. The integration code is in [`integration/`](https://github.com/raoofaltaher/lead-decision-models/tree/main/integration), and the transcript is [`outputs/case-study-session.md`](https://github.com/raoofaltaher/lead-decision-models/blob/main/outputs/case-study-session.md).

## The case study

[Section titled “The case study”](#the-case-study)

Tillbrook Procurement is an imaginary company that sells procurement automation software. Its Ideal Customer Profile (ICP) is the example ICP of page 1: manufacturing, logistics or professional services, in the US or the UK, with 200 to 1,999 employees. Claude Code acts as Tillbrook’s research agent:

flowchart LR
    A\[Claude Code agent\] -->|search, scrape| F\[Firecrawl MCP\]
    A -->|lead-request, reply-request| G\[gate.py\]
    G -->|exact decide request| A
    A -->|decide| L\[lead-decisions MCP server\]
    A -->|decide| R\[reply-decisions MCP server\]
    L -->|answer| A
    R -->|answer| A
    A -->|lead, reply| G
    G -->|action, send: false| Q\[approval-queue.jsonl\]
    Q --> H\[A person approves\]

The agent decides no action itself. The models answer bounded questions, the gate turns the answers into actions with fixed rules, and a person reviews the queue. The lead model learned from synthetic LeadForge records, so its answers about real companies demonstrate the integration; they are not validated predictions, and this page labels each of them.

## Case study results

[Section titled “Case study results”](#case-study-results)

The recorded session ran 128 turns in 391 s and made 127 tool calls: 7 Firecrawl searches, 20 scrapes, 2 lead `decide` calls, 8 reply `decide` calls, 45 shell commands, 32 file writes, 12 reads and 1 tool search. Claude Code’s permission list refused 16 of those calls, the gate refused 8 records with exit code 2, and 1 scrape failed on a certificate error.

| Result | Count |
| --- | --- |
| Companies researched | 10 |
| Records with all three ICP fields on the company’s own pages | 2 |
| Records the gate refused before any model call | 8 |
| Lead model answers that matched the ICP rule | 2 of 2 (one fit, one not_fit) |
| Reply model answers that match the intent the reply text states | 8 of 8 |
| Queue entries, all with send: false | 10 |
| Warm decision latency | 55 ms median (reply); 52 ms and 666 ms (lead, two calls) |

> Not validated: trained on synthetic LeadForge data.

The gate did most of the work on the lead side: eight companies state no exact headcount on their pages, and the prompt forbids turning “more than 300” into a band. That is the point of the design. A model answers only when the evidence is complete, and a person reads the evidence before anything is sent.

## What you need for Claude Code

[Section titled “What you need for Claude Code”](#what-you-need-for-claude-code)

| Item | Recorded run |
| --- | --- |
| Machine and environment | The machine and unsloth_env of page 1, with both notebooks run, so that outputs/lead-decision-merged/ and outputs/reply-decision-merged/ exist |
| Models | The merged exports of both notebooks, or the same files downloaded from the Hub: lead-qualifier-merged and reply-router-merged |
| MCP Python SDK | mcp 2.3.0, in requirements-experiment.txt |
| Claude Code | 2.1.317, native install, with claude-opus-5-5 |
| Firecrawl | The hosted MCP server with OAuth sign-in; a Firecrawl account, no key in the repository |
| GPU memory | Both servers loaded at once fit in 24 GB; each holds one 4-bit model |

To skip the training runs, download the merged models of the recorded run into the folders that `.mcp.json` names ([hf CLI](https://huggingface.co/docs/huggingface_hub/guides/cli)). Each is about 3.3 GB:

```
hf download raoofaltaher/lead-decision-models-lead-qualifier-merged --local-dir outputs/lead-decision-mergedhf download raoofaltaher/lead-decision-models-reply-router-merged --local-dir outputs/reply-decision-merged
```

Install Claude Code with the native installer and log in by running `claude` once ([Claude Code setup](https://code.claude.com/docs/en/setup)). Claude Code needs a Pro, Max, Team, Enterprise or Console account.

```
curl -fsSL https://claude.ai/install.sh | bashclaude --version
```

Firecrawl’s hosted MCP server has two URLs: `https://mcp.firecrawl.dev/v2/mcp` with an API key, and `https://mcp.firecrawl.dev/v2/mcp-oauth`, which opens a browser sign-in and needs no key ([Firecrawl MCP server](https://docs.firecrawl.dev/mcp-server)). The repository uses the OAuth one. Every search query and every scraped URL goes to Firecrawl under your account, so read its terms before you send it anything sensitive.

## Why an own server and not Unsloth Studio

[Section titled “Why an own server and not Unsloth Studio”](#why-an-own-server-and-not-unsloth-studio)

Unsloth Studio serves decision models through a Decision API and an MCP endpoint, so it is the obvious choice. Studio 2026.10.11 cannot load LFM3: its runtime loads a model without `trust_remote_code`, and its environment pins Transformers 5.5.0, while LFM3’s code needs 5.22.1. The fix, [unslothai/unsloth#16096](https://github.com/unslothai/unsloth/pull/16096), is merged and waits for a release ([#15588](https://github.com/unslothai/unsloth/issues/15588)). Until then, the repository serves the models with its own server, which keeps Studio’s `decide` contract: the same tool, request shape, limits and error messages. When Studio can load LFM3, the agent side of this page does not change.

## Step 22: Write the MCP server

[Section titled “Step 22: Write the MCP server”](#step-22-write-the-mcp-server)

[`integration/decision_server.py`](https://github.com/raoofaltaher/lead-decision-models/blob/main/integration/decision_server.py) serves one merged model folder as an MCP `decide` tool over stdio, with the [MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk):

```
import argparseimport contextlibimport jsonimport sysimport threadingfrom collections.abc import Callablefrom pathlib import Pathfrom typing import Any, Union
from mcp.server import MCPServerfrom mcp.server.mcpserver.exceptions import ToolErrorfrom pydantic import BaseModel, ConfigDict
QUESTION_TYPES = ("noul", "choice", "score")MAX_QUESTIONS = 64MAX_CHOICES = 255MAX_SCORE_LEVELS = 10MAX_STATE_CHARS = 200_000MAX_QUESTION_CHARS = 20_000
DECIDE_DESCRIPTION = (    "Ask the fine-tuned decision model typed questions about a state (text or JSON). "    'questions maps a name you pick to {"type", "instructions", "criteria"}. '    '"noul" is yes/no and answers a probability; "choice" needs criteria mapping each option name '    'to a description; "score" needs criteria listing 1 to 10 levels, lowest first.')
JSONContent = Union[str, dict[str, Any], list[Any]]Decider = Callable[[JSONContent, dict[str, dict[str, Any]]], dict[str, Any]]

class Question(BaseModel):    model_config = ConfigDict(extra="forbid")
    type: str    instructions: JSONContent | None = None    criteria: JSONContent | None = None

class DecisionRequestError(ValueError):    pass

def validate_request(state: JSONContent, questions: dict[str, dict[str, Any]]) -> None:    if not questions:        raise DecisionRequestError("At least one question is required")    state_chars = len(state) if isinstance(state, str) else len(json.dumps(state, ensure_ascii=False))    if state_chars > MAX_STATE_CHARS:        raise DecisionRequestError(f"State is longer than {MAX_STATE_CHARS} characters")    if len(questions) > MAX_QUESTIONS:        raise DecisionRequestError(f"At most {MAX_QUESTIONS} questions per request")    for name, question in questions.items():        _validate_question(name, question)

def _validate_question(name: str, question: dict[str, Any]) -> None:    if len(name) + len(json.dumps(question, ensure_ascii=False)) > MAX_QUESTION_CHARS:        raise DecisionRequestError(f"Question is longer than {MAX_QUESTION_CHARS} characters")    kind = question.get("type")    if kind not in QUESTION_TYPES:        raise DecisionRequestError(f'Question "{name}" has unknown type "{kind}"')    criteria = question.get("criteria")    if kind == "noul":        if criteria is not None and (not isinstance(criteria, dict) or set(criteria) - {"true", "false"}):            raise DecisionRequestError(f'Noul "{name}" criteria may only have "true" and "false"')    elif kind == "choice":        if not isinstance(criteria, dict) or not criteria:            raise DecisionRequestError(f'Choice "{name}" needs criteria naming its options')        if len(criteria) > MAX_CHOICES:            raise DecisionRequestError(f'Choice "{name}" has more than {MAX_CHOICES} options')    elif not isinstance(criteria, list) or not 1 <= len(criteria) <= MAX_SCORE_LEVELS:        raise DecisionRequestError(f'Score "{name}" needs 1 to {MAX_SCORE_LEVELS} criteria levels')    if criteria is not None:        entries = criteria.items() if isinstance(criteria, dict) else enumerate(criteria)        for key, value in entries:            if value is not None and not isinstance(value, (str, dict, list)):                raise DecisionRequestError(                    f'Question "{name}" criteria[{key!r}] must be text, an object, an array or null'                )

def load_merged_model(path: str) -> Decider:    from unsloth import FastDecisionModel
    model, tokenizer = FastDecisionModel.from_pretrained(        path,        trust_remote_code=True,        max_seq_length=2048,        load_in_4bit=True,    )    FastDecisionModel.for_inference(model)
    def decide(state: JSONContent, questions: dict[str, dict[str, Any]]) -> dict[str, Any]:        return FastDecisionModel.predict(model, tokenizer, state, questions)
    return decide

class DecisionService:    def __init__(self, model_path: str, loader: Callable[[str], Decider]):        self.model_path = model_path        self._loader = loader        self._decider: Decider | None = None        self._lock = threading.Lock()
    def decide(self, state: JSONContent, questions: dict[str, dict[str, Any]]) -> dict[str, Any]:        validate_request(state, questions)        with self._lock, contextlib.redirect_stdout(sys.stderr):            if self._decider is None:                self._decider = self._loader(self.model_path)            return {"answers": self._decider(state, questions)}

def build_server(service: DecisionService) -> MCPServer:    server = MCPServer("Unsloth decision model")
    @server.tool(description=DECIDE_DESCRIPTION)    def decide(state: JSONContent, questions: dict[str, Question]) -> dict[str, Any]:        try:            return service.decide(                state, {name: question.model_dump(exclude_none=True) for name, question in questions.items()}            )        except ValueError as error:            raise ToolError(str(error)) from None
    return server

def main(argv: list[str] | None = None, loader: Callable[[str], Decider] = load_merged_model) -> None:    parser = argparse.ArgumentParser(description="Serve one merged decision model as an MCP decide tool over stdio.")    parser.add_argument("--model", required=True, help="Local folder of a merged decision model")    args = parser.parse_args(argv)    if not (Path(args.model) / "config.json").is_file():        parser.error(f"{args.model} has no config.json; pass a local merged model folder")    build_server(DecisionService(args.model, loader)).run("stdio")

if __name__ == "__main__":    main()
```

The parts that matter when you write your own:

-   **One tool, typed.** `MCPServer` turns the `decide` function into a tool, and its type hints become the JSON schema that Claude Code shows the model. `Question` is a pydantic model with `extra="forbid"`, so a misspelled field is refused before anything runs.
-   **Validate before you load.** `validate_request` applies Studio’s limits: at least one question, a state of at most 200,000 characters, at most 64 questions, the types `noul`, `choice` and `score`, at most 255 options, 1 to 10 score levels. An invalid request returns a `ToolError` with Studio’s message and never loads the model.
-   **Load lazily, once, under a lock.** The first `decide` call runs `FastDecisionModel.from_pretrained` in 4 bits, the same call that page 2 uses to reload the merged export. One lock covers the load and every prediction, so two first calls load the model once.
-   **Keep standard output clean.** MCP over stdio sends JSON-RPC on standard output, and Unsloth prints banners while it loads. The server redirects standard output to standard error during the load and every prediction. [`tests/test_decision_server.py`](https://github.com/raoofaltaher/lead-decision-models/blob/main/tests/test_decision_server.py) starts the server as a client and checks that every line on standard output is JSON-RPC.
-   **Serve a local folder only.** `main` stops when `--model` holds no `config.json`. The LFM3 code that runs is the Python files in the merged folder; read them first.

The setup is local, but not offline: Unsloth contacts the Hugging Face Hub while it loads, and the serving log records those requests.

## Step 23: Connect Claude Code

[Section titled “Step 23: Connect Claude Code”](#step-23-connect-claude-code)

Claude Code reads project MCP servers from `.mcp.json` in the repository root ([Claude Code: MCP](https://code.claude.com/docs/en/mcp)):

```
{  "mcpServers": {    "lead-decisions": {      "type": "stdio",      "command": "unsloth_env/bin/python",      "args": ["integration/decision_server.py", "--model", "outputs/lead-decision-merged"]    },    "reply-decisions": {      "type": "stdio",      "command": "unsloth_env/bin/python",      "args": ["integration/decision_server.py", "--model", "outputs/reply-decision-merged"]    },    "firecrawl": {      "type": "http",      "url": "https://mcp.firecrawl.dev/v2/mcp-oauth"    }  }}
```

| Server | Transport | What it runs |
| --- | --- | --- |
| lead-decisions | stdio | decision_server.py on outputs/lead-decision-merged |
| reply-decisions | stdio | decision_server.py on outputs/reply-decision-merged |
| firecrawl | HTTP with OAuth | Firecrawl’s hosted MCP server |

The same entries can be written from the command line. `--scope project` writes to `.mcp.json`, and `--` separates Claude’s options from the server command:

```
claude mcp add --scope project lead-decisions -- unsloth_env/bin/python integration/decision_server.py --model outputs/lead-decision-mergedclaude mcp add --scope project reply-decisions -- unsloth_env/bin/python integration/decision_server.py --model outputs/reply-decision-mergedclaude mcp add --scope project --transport http firecrawl https://mcp.firecrawl.dev/v2/mcp-oauth
```

Then, in the repository folder:

1.  Start `claude`. Claude Code asks you to approve the three project servers, because `.mcp.json` comes from the repository.
2.  Run `/mcp` and sign in to Firecrawl in the browser.
3.  Run `claude mcp list` and check that all three servers show as connected.

The decision servers start `unsloth_env/bin/python` with paths relative to the repository root, so run `claude` from that root. No server listens on a network port.

## Step 24: The gate

[Section titled “Step 24: The gate”](#step-24-the-gate)

[`integration/gate.py`](https://github.com/raoofaltaher/lead-decision-models/blob/main/integration/gate.py) is the only path from a model answer to an action. It has four commands:

| Command | What it does |
| --- | --- |
| lead-request --features f.json | Validates a lead record against the categories the model was trained on and prints the exact decide request. Exit code 2 means a field is missing or a value is unknown: research the lead instead of calling the model |
| reply-request --reply-text "..." | Prints the exact decide request for a reply |
| lead --domain d --answers a.json ... | Applies the outreach policy of page 1 and appends one queue entry per domain |
| reply --reply-text "..." --answers a.json ... | Applies the routing policy of page 2; an opt-out phrase in the text always suppresses marketing |

The values that `lead-request` accepts are the values the lead model saw in training, and the same block holds the opt-out phrases:

```
TRAINED_VALUES = {    "industry": {"manufacturing", "logistics", "professional_services", "healthcare_non_clinical"},    "region": {"US", "UK"},    "employee_band": {"200-499", "500-999", "1000-1999", "2000+"},}OPT_OUT_PHRASES = (    "unsubscribe",    "remove me",    "take me off",    "stop emailing",    "opt out",    "opt me out",    "do not contact",    "don't contact",)
```

`build_lead_request` collects every problem before it refuses a record:

```
def build_lead_request(features: dict) -> dict:    unknown_fields = sorted(set(features) - set(MODEL_FEATURES))    problems = [f"unknown fields {unknown_fields}"] if unknown_fields else []    for field, allowed in TRAINED_VALUES.items():        value = features.get(field)        if value is None:            problems.append(f"missing ICP evidence: {field}")        elif value not in allowed:            problems.append(f"{field}={value!r} is outside the training categories {sorted(allowed)}")    if problems:        raise ValueError("; ".join(problems))    return {"state": make_lead_state(features), "questions": LEAD_QUESTIONS}
```

A malformed model answer counts as no answer, so the policy fails closed to a review action. A domain is queued once. Every result carries `"send": false`, and [`tests/test_gate.py`](https://github.com/raoofaltaher/lead-decision-models/blob/main/tests/test_gate.py) pins these rules.

The case study researches companies, not people, so the prompt passes `--suppressed false --persona-fit unknown --eligibility unknown` for every lead. The best action a lead can then get is `human_persona_review`: a person must still find a contact and check that contacting them is allowed. In a deployment, the suppression flag comes from your suppression records, never from a prompt.

## Step 25: The prompt and the recording

[Section titled “Step 25: The prompt and the recording”](#step-25-the-prompt-and-the-recording)

[`integration/case_study_prompt.md`](https://github.com/raoofaltaher/lead-decision-models/blob/main/integration/case_study_prompt.md) gives the agent the ICP, the allowed values, the gate commands and the order of the steps. It never asks the agent for an outreach decision.

The prompt of the recorded session

```
You are the research agent of Tillbrook Procurement, an imaginary company that sells procurement automation software. You find and research prospects, and you hand every bounded decision to two fine-tuned decision models. You never decide an outreach action yourself, and you never send any message to anyone.
Tillbrook's Ideal Customer Profile: companies in manufacturing, logistics or professional services, based in the US or the UK, with 200 to 1,999 employees.
You have these tools:- `firecrawl` (MCP): web search and page scraping.- `lead-decisions` (MCP, tool `decide`): the fine-tuned lead qualification model.- `reply-decisions` (MCP, tool `decide`): the fine-tuned reply routing model.- Bash, to run `python integration/gate.py`, the deterministic policy gate. Use the active virtual environment.
Part A, prospects. Work through these steps for each company:
1. With Firecrawl search, find 8 to 12 real companies that may match the profile. Spread the searches across the three industries and both regions.2. Scrape up to three pages per company, the About page first, to learn what it does, where it is based and how many people it employs.3. Write `outputs/case-study/<domain>.features.json` with only the fields `industry`, `region` and `employee_band`. Use only these values, and only when the pages support them: industry `manufacturing`, `logistics`, `professional_services` or `healthcare_non_clinical`; region `US` or `UK`; employee_band `200-499`, `500-999`, `1000-1999` or `2000+`. A headcount stated only as a lower bound supports a band only when the bound guarantees it: "over 2,000" supports `2000+`, but "over 500" or "800-plus" supports no band. A figure for a parent group is not the company's own headcount. When the pages do not support a value, leave that field out. Do not guess. The gate refuses a record that lacks any of the three fields and sends that company to research.4. Write `outputs/case-study/<domain>.evidence.json` with each value you set, the sentence that supports it, and the page URL.5. Run `python integration/gate.py lead-request --features outputs/case-study/<domain>.features.json`. If it exits with code 2, record the error and do not call the model for that company.6. Call `lead-decisions` `decide` with exactly the `state` and `questions` that command printed. Save the full answer as `outputs/case-study/<domain>.answers.json`.7. Run `python integration/gate.py lead --domain <domain> --answers outputs/case-study/<domain>.answers.json --suppressed false --persona-fit unknown --eligibility unknown --queue outputs/approval-queue.jsonl` and keep its printed action.
Part B, replies. For each item in `data/case-study/replies.json`:
1. Run `python integration/gate.py reply-request --reply-text "<reply_text>"`.2. Call `reply-decisions` `decide` with exactly the printed `state` and `questions`. Save the answer as `outputs/case-study/<reply_id>.answers.json`.3. Run `python integration/gate.py reply --reply-text "<reply_text>" --answers outputs/case-study/<reply_id>.answers.json --suppressed <contact_is_suppressed> --opt-out-event <explicit_opt_out_event> --queue outputs/approval-queue.jsonl` and keep its printed action. Write the two JSON booleans in lower case, `true` or `false`.
When you finish, print one table for the prospects (domain, ICP answer and its probability, gate action) and one for the replies (reply_id, intent answer and its probability, gate action). End with one sentence stating that no message was sent.
```

[`integration/record_case_study.sh`](https://github.com/raoofaltaher/lead-decision-models/blob/main/integration/record_case_study.sh) runs the prompt headless with `claude -p` and renders the result:

```
#!/usr/bin/env bashset -uo pipefailexport PATH="${HOME}/.local/bin:${PATH}"command -v claude > /dev/null || { echo "claude is not on PATH"; exit 127; }cd "$(dirname "$0")/.."source unsloth_env/bin/activatemkdir -p outputs/case-study outputs/logsrm -f outputs/approval-queue.jsonlstart=$(date +%s)claude -p "$(cat integration/case_study_prompt.md)" \  --mcp-config .mcp.json \  --strict-mcp-config \  --setting-sources project \  --allowedTools \    "mcp__firecrawl__firecrawl_search" \    "mcp__firecrawl__firecrawl_scrape" \    "mcp__lead-decisions__decide" \    "mcp__reply-decisions__decide" \    "Bash(python integration/gate.py *)" \    "Edit(outputs/case-study/**)" \    "Read(data/case-study/**)" \    "Read(outputs/case-study/**)" \  --output-format stream-json \  --verbose \  > outputs/logs/case-study-session.raw.jsonl 2> outputs/logs/case-study.errstatus=$?end=$(date +%s)echo "claude exit=${status} seconds=$((end - start))"python tools/render_session.py \  --raw outputs/logs/case-study-session.raw.jsonl \  --prompt integration/case_study_prompt.md \  --queue outputs/approval-queue.jsonl \  --out-jsonl outputs/case-study-session.jsonl \  --out-md outputs/case-study-session.md
```

-   `--mcp-config .mcp.json --strict-mcp-config` loads the three servers and no other.
-   `--setting-sources project` loads the project settings only.
-   `--allowedTools` allows exactly the Firecrawl search and scrape tools, the two `decide` tools, the gate, file writes under `outputs/case-study/` and reads under `data/case-study/` and `outputs/case-study/`. Every other call needs a permission that nobody can grant in a headless run, so Claude Code refuses it, and the transcript shows the refusal. Read-only shell commands such as `ls` and `cat` are the exception: Claude Code runs them without a rule.
-   `--output-format stream-json --verbose` prints every event. [`tools/render_session.py`](https://github.com/raoofaltaher/lead-decision-models/blob/main/tools/render_session.py) keeps only known-safe fields, cuts every Firecrawl result to 600 characters, and adds “Not validated: trained on synthetic LeadForge data.” after every lead answer.

## What the first recording taught

[Section titled “What the first recording taught”](#what-the-first-recording-taught)

The first recording, kept in [`outputs/case-study-first-run/`](https://github.com/raoofaltaher/lead-decision-models/tree/main/outputs/case-study-first-run), used an earlier gate that let a record with missing fields reach the model. The lead model answered `fit` with a probability between 0.9998 and 1.0 for all nine companies, although seven records had no employee band and two of those had no region:

| Domain | Fields set | ICP answer |
| --- | --- | --- |
| evapco.com | manufacturing, US | fit (0.9998) |
| xtrac.com | manufacturing | fit (0.9999) |
| denholmgoodlogistics.com | logistics, UK | fit (1.0) |
| schneiderdowns.com | professional_services | fit (1.0) |
| ksmcpa.com | professional_services, US, 500-999 | fit (1.0) |

> Not validated: trained on synthetic LeadForge data.

The question says that a missing field means `insufficient_evidence`, but the model gave that answer a probability of 0 every time. LeadForge never leaves a field empty, so the model never saw that answer in training ([page 1](/tutorials/fine-tune-lfm3-decision-model-with-unsloth#what-the-icp-accuracy-of-10-means)). The agent also turned lower bounds into bands: “800-plus CPAs” became `500-999`. Two fixes followed. The gate now refuses an incomplete record with exit code 2 and the next action `research_missing_account_evidence`, and the prompt accepts a lower bound only when it guarantees a band (“over 2,000” supports `2000+`; “over 500” supports nothing). The lesson is general: a model cannot give an answer that its data never showed, so handle missing evidence in code.

## The recorded session

[Section titled “The recorded session”](#the-recorded-session)

[`outputs/case-study-session.md`](https://github.com/raoofaltaher/lead-decision-models/blob/main/outputs/case-study-session.md) is the recording. Claude Code 2.1.317 ran it with `claude-opus-5-5` in permission mode `default`, with the three servers connected.

### The approval queue

[Section titled “The approval queue”](#the-approval-queue)

| # | Kind | Subject | Model answer | Action | Send |
| --- | --- | --- | --- | --- | --- |
| 1 | lead | verstlogistics.com | not_fit (1.00) | do_not_recommend_outreach | false |
| 2 | lead | chemringcm.com | fit (1.00) | human_persona_review | false |
| 3 | reply | case-01: “Thanks for the note about Tillbrook. Could we set up 30 minutes next Thursday to see how the approval workflow works?” | meeting_requested (1.00) | handoff_to_sales | false |
| 4 | reply | case-02: “What would this cost for roughly 300 requesters across two plants?” | more_information (1.00) | prepare_requested_information | false |
| 5 | reply | case-03: “We’re mid-way through an ERP upgrade. Please come back to me in early March.” | not_now (1.00) | schedule_later_after_date_confirmation | false |
| 6 | reply | case-04: “I’m out of the office until 24 November with limited access to email.” | out_of_office (1.00) | pause_until_return_date_confirmation | false |
| 7 | reply | case-05: “I look after facilities, not purchasing systems, so I’m the wrong person for this.” | wrong_person (1.00) | human_review | false |
| 8 | reply | case-06: “Thanks, but we’re not going to take this forward.” | not_interested (1.00) | stop_sequence | false |
| 9 | reply | case-07: “Please unsubscribe me from these emails.” | opt_out (1.00) | suppress_marketing | false |
| 10 | reply | case-08: “Sounds genuinely useful for our buyers, but please take me off your mailing list.” | opt_out (1.00) | suppress_marketing | false |

> Not validated: trained on synthetic LeadForge data.

### Prospects

[Section titled “Prospects”](#prospects)

The agent researched ten companies, up to three pages each. Only two had pages that supported all three ICP fields:

| Domain | Fields the pages supported | Gate result | Reason from the evidence file |
| --- | --- | --- | --- |
| chemringcm.com | manufacturing, UK, 200-499 | Model called: fit (1.00), human_persona_review, priority high | The unit’s page states “we employ 400 people”; the agent noted that this is the unit’s headcount, not the group’s |
| verstlogistics.com | logistics, US, 2000+ | Model called: not_fit (1.00), do_not_recommend_outreach | The About page states “2,500 Total Employees”, above the ICP range |
| evapco.com | manufacturing, US | Refused: no employee band | No headcount on the About page; the careers page returned 404 |
| ki.com | manufacturing, US | Refused: no employee band | No headcount on the About or careers page |
| wagnerlogistics.com | logistics, US | Refused: no employee band | The only figure on the page belongs to an executive’s previous employer |
| tenens.com | logistics, UK | Refused: no employee band | No headcount on its pages; “800+ colleagues” in a social post is a lower bound |
| rettew.com | professional_services, US | Refused: no employee band | “More than 300 professionals” is a lower bound that supports no band |
| hcrlaw.com | professional_services, UK | Refused: no employee band | “More than 1,000 people” does not guarantee 1000-1999 |
| watermanaspen.co.uk | professional_services, UK | Refused: no employee band | “More than 850 people” is a lower bound |
| stanleyconsultants.com | professional_services | Refused: no region, no employee band | No page states where the firm is based today; “1,000+ Employees” is a lower bound |

> Not validated: trained on synthetic LeadForge data.

Both model answers match the ICP rule: 400 employees in a UK manufacturer is inside the band, and 2,500 is outside it. The gate checks that a value is present and allowed, not that it is true, so the person who reviews the queue still reads the evidence files, which hold the sentence and the URL behind every value. The agent saved the eight refusals in `outputs/case-study/lead-gate-errors.json`.

### Replies

[Section titled “Replies”](#replies)

The eight replies are fixed inputs in `data/case-study/replies.json`, with `contact_is_suppressed` and `explicit_opt_out_event` set to false, so only the text and the model’s answers decide the action. The reply model, retrained in [page 2](/tutorials/fine-tune-lfm3-decision-model-with-unsloth/reply-routing), gave each reply the intent its text states, each with a probability above 0.999, and its follow-up answer was no for the three replies that end the conversation. case-08 is the mixed reply that praises the product and asks to come off the list: the model answered `opt_out` at 0.9996, and the gate would suppress it anyway, because “take me off” is one of its opt-out phrases.

### From the transcript

[Section titled “From the transcript”](#from-the-transcript)

Transcript step 26 scrapes the page of Chemring Countermeasures. The published transcript keeps the first 600 characters of each Firecrawl result:

````
### 26. `mcp__firecrawl__firecrawl_scrape`
```json{  "url": "https://www.chemring.co.uk/about-us/our-business/chemring-countermeasures-uk",  "formats": [    "markdown"  ],  "onlyMainContent": true,  "objective": "Learn what the company does, where it is based and how many people it employs, for ICP qualification.",  "conversationId": "34af5291-4fca-4e07-b682-8ed1cb89a7ee"}```
[... 28 lines cut ...]Result of step 26:
```text{"markdown":"This site uses cookies. Cookies that are strictly necessary for the website to function are always on and cannot be switched off. To accept functional, performance and targeting cookies, click “Accept All”. Alternatively, you can reject these cookies by clicking the “Necessary Cookies only” button. View our [cookies page](https://www.chemring.com/site-services/cookie-statement \"Opens in a new window\").\n\nAccept all cookiesDecline cookiesChange cookie preferences\n\nClose\n\n- [About us](https://www.chemring.com/about-us)\n- [What we do](https://www.chemring.com/what-we-do)\n- [[21,051 characters cut]```
````

Transcript step 77 runs the gate on RETTEW, whose record has no employee band. The gate refuses it with exit code 2, so the model is never called:

````
### 77. `Bash`
```json{  "command": "python integration/gate.py lead-request --features outputs/case-study/rettew.com.features.json",  "description": "Gate lead-request for rettew.com"}```[... 8 lines cut ...]Step 77 refused or failed:
```textExit code 2{"error": "missing ICP evidence: employee_band", "next_action": "research_missing_account_evidence", "send": false}```
````

Transcript step 82 sends the lead model exactly the request that the gate printed for Chemring. Three of the 24 features have values; the other 21 are null, and the answer comes back with the not-validated label:

````
### 82. `mcp__lead-decisions__decide`
```json{  "state": {    "features": {      "activity_count": null,      "buyer_role": null,      "days_since_first_touch": null,      "days_since_last_touch": null,      "demo_page_views": null,      "employee_band": "200-499",[... 48 lines cut ...]Result of step 82:
```text{"answers":{"icp_fit":{"type":"choice","choice":"fit","confidence":1,"probabilities":{"fit":1,"not_fit":0,"insufficient_evidence":0},"answer":"fit"},"conversion_outcome_90d":{"type":"noul","noul":0.0149,"answer":false,"probabilities":{"true":0.0149,"false":0.9851}}}}```
> Not validated: trained on synthetic LeadForge data.
````

The permission list refused the agent’s shortcuts, and the transcript shows every refusal. Examples: a Python script in a shell here-document at transcript step 37, a `for` loop over the feature files at step 60, and `; echo "exit=$?"` after a gate command at steps 61 to 70. After the refusals, the agent wrote each file with a separate write call and ran each gate command on its own.

## Latency

[Section titled “Latency”](#latency)

[`tools/serving_check.py`](https://github.com/raoofaltaher/lead-decision-models/blob/main/tools/serving_check.py) starts each server over MCP stdio, as Claude Code does, sends the first request once to load the model, then sends every request of the session and compares each answer with the saved one. The recorded check, [`outputs/serving-check.json`](https://github.com/raoofaltaher/lead-decision-models/blob/main/outputs/serving-check.json):

```
{  "lead": {    "calls": 3,    "first_call_s": 22.17,    "warm_median_ms": 358.8,    "warm_max_ms": 665.9,    "matches_session": {      "chemringcm.com": true,      "verstlogistics.com": true    }  },  "reply": {    "calls": 9,    "first_call_s": 20.1,    "warm_median_ms": 55.4,    "warm_max_ms": 416.5,    "matches_session": {      "case-01": true,      "case-02": true,      "case-03": true,      "case-04": true,      "case-05": true,      "case-06": true,      "case-07": true,      "case-08": true    }  }}
```

-   **Reply server.** The first call took 20.1 s, the model load included. Over eight warm calls, the median was 55.4 ms and the maximum 416.5 ms.
-   **Lead server.** The first call took 22.2 s. The two warm calls took 52 ms and 666 ms; with only two calls, the tool’s median of 358.8 ms is their mean and says little.
-   **Answers.** Every answer matched the session: both lead answers and all eight reply answers.

The check ran on an NVIDIA GeForce RTX 5090 Laptop GPU, with both servers started one after the other.

The transcript does not time the calls: the timestamps of the stream can lag a call by up to about a second, too coarse for a decision that takes tens of milliseconds.

## Security, privacy and compliance

[Section titled “Security, privacy and compliance”](#security-privacy-and-compliance)

-   **No send capability.** No allowed tool can send a message, every result carries `"send": false`, and a person reviews the queue.
-   **A fixed permission list.** The recording allows only the tools it names and refused the agent’s own scripts and shell loops. Read-only shell commands still run, so keep secrets out of the folder where the agent runs.
-   **Remote code at a pinned revision.** LFM3 runs its own Python code through `trust_remote_code=True`, loaded only from a local merged folder that holds the files of revision `ac2586a34f1a1c03329dbdcdefabd8b6478ddaa7`.
-   **Untrusted page text.** A scraped page can carry text written to steer an agent. Page text can change the agent’s notes, but not the allowed tools or the rules of the gate.
-   **Real company names, labelled.** The case study stores only company facts with the sentence and URL behind each, and labels every lead prediction as not validated.
-   **No network listener.** The decision servers talk to Claude Code over stdio only. Never expose an unauthenticated decision API to the internet.

Keep suppression records in a deterministic system outside the model, keep the source URL of every extracted claim, log predictions, decisions and approvals, and review the law for the sender, the recipient and the channel. The FTC’s [CAN-SPAM guide](https://www.ftc.gov/business-guidance/resources/can-spam-act-compliance-guide-business) covers B2B email and requires prompt opt-out handling; the ICO’s [B2B marketing guidance](https://ico.org.uk/for-organisations/direct-marketing-and-privacy-and-electronic-communications/business-to-business-marketing/) covers the UK. This page is not legal advice.

## Limitations

[Section titled “Limitations”](#limitations)

-   **Synthetic training data.** The lead model learned from LeadForge’s synthetic CRM records, and the reply model from 617 synthetic replies that one Claude model wrote and a second labelled blind. No person reviewed the replies, and neither model is validated on real companies or real inboxes.
-   **Small test sets.** The reply test set holds 124 replies. Two runs of the same notebook scored 0.924 and 0.975, so read a range, not a point.
-   **Tabular data favours simple models.** On the lead features, a logistic regression ranks conversions slightly better than the fine-tuned model.
-   **Missing evidence.** The lead model never saw a record with an empty ICP field, so the gate, not the model, handles missing evidence.
-   **One recorded session.** Another run researches other prospects. The gate checks that each field is present and allowed, not that the evidence is true.
-   **One machine.** Every time and latency figure comes from one RTX 5090 Laptop GPU.
-   **No outreach text.** The tutorial classifies replies and routes actions. It does not write or optimize outreach messages, on purpose.

## Work with me

[Section titled “Work with me”](#work-with-me)

Every company has its own ICP, its own CRM history and its own inbox. The method of this tutorial transfers to all three.

Write the policy as code. Label your own data. Fine-tune a small decision model on one GPU, a laptop GPU in this tutorial, and compare it with a simple baseline. Then put every action behind a gate and a person.

The models run on your own hardware, so lead data can stay inside your infrastructure. Claude Code, or any MCP client, reaches them through one small server, and every session leaves a transcript that a reviewer can read.

Do you want a decision model trained on your own ICP, CRM history or reply taxonomy, self-hosted and connected to your agents? Do you want me to check first whether a fine-tuned model or a simpler classifier fits your data better? [Contact me](/#contact).

## References

[Section titled “References”](#references)

**Unsloth, the base model and the models of this tutorial**

-   [Unsloth: Train Your Own Decision Model](https://unsloth.ai/docs/basics/train-your-own-decision-model-with-unsloth)
-   [Unsloth decision-model implementation at the pinned commit](https://github.com/unslothai/unsloth/blob/53eeebdc9be3c3e1a27f891b668a38df0e1abb88/unsloth/models/decision.py)
-   [Unsloth Studio](https://unsloth.ai/docs/basics/unsloth-studio) and the LFM3 fix [unslothai/unsloth#16096](https://github.com/unslothai/unsloth/pull/16096)
-   [LFM3-1.7B on Hugging Face](https://huggingface.co/LiquidAI/LFM3-1.7B) and its [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM3-1.7B/blob/ac2586a34f1a1c03329dbdcdefabd8b6478ddaa7/LICENSE)
-   [lead-decision-models on GitHub](https://github.com/raoofaltaher/lead-decision-models)
-   [Lead qualifier](https://huggingface.co/raoofaltaher/lead-decision-models-lead-qualifier) and its [merged copy](https://huggingface.co/raoofaltaher/lead-decision-models-lead-qualifier-merged)
-   [Reply router](https://huggingface.co/raoofaltaher/lead-decision-models-reply-router) and its [merged copy](https://huggingface.co/raoofaltaher/lead-decision-models-reply-router-merged)

**Claude Code, MCP and Firecrawl**

-   [Claude Code: setup](https://code.claude.com/docs/en/setup)
-   [Claude Code: MCP](https://code.claude.com/docs/en/mcp)
-   [MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk)
-   [Model Context Protocol specification](https://modelcontextprotocol.io/specification/latest)
-   [Firecrawl MCP server](https://docs.firecrawl.dev/mcp-server)

**Data**

-   [LeadForge dataset repository at the pinned commit](https://github.com/leadforge-dev/leadforge-datasets/tree/39ebe8c9e6cf25fe33a003662e18557b4de55142)
-   [LeadForge intermediate dataset card](https://github.com/leadforge-dev/leadforge-datasets/blob/39ebe8c9e6cf25fe33a003662e18557b4de55142/releases/v0.1.0-alpha/intermediate/dataset_card.md)
-   [LeadForge baseline results](https://github.com/leadforge-dev/leadforge-datasets/blob/39ebe8c9e6cf25fe33a003662e18557b4de55142/releases/v0.1.0-alpha/BASELINES.md)
-   [LeadForge simulation engine](https://github.com/leadforge-dev/leadforge/blob/8dc21c12aa5a407823eda93747abb6cff4ac3d2a/leadforge/simulation/engine.py)
-   [The reply dataset card](https://github.com/raoofaltaher/lead-decision-models/blob/main/data/replies/DATASET_CARD.md)

**Setup, training and evaluation**

-   [uv: Using uv with PyTorch](https://docs.astral.sh/uv/guides/integration/pytorch/)
-   [Microsoft: Install WSL](https://learn.microsoft.com/windows/wsl/install)
-   [NVIDIA: CUDA on WSL](https://docs.nvidia.com/cuda/wsl-user-guide/index.html)
-   [Hugging Face: user access tokens](https://huggingface.co/docs/hub/security-tokens) and the [hf CLI](https://huggingface.co/docs/huggingface_hub/guides/cli)
-   [Hugging Face PEFT: LoRA](https://huggingface.co/docs/peft/package_reference/lora)
-   [scikit-learn: StratifiedGroupKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html)
-   [scikit-learn: model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html)
-   [scikit-learn: probability calibration](https://scikit-learn.org/stable/modules/calibration.html)
-   [scikit-learn: common pitfalls](https://scikit-learn.org/stable/common_pitfalls.html)

**Compliance and research**

-   [FTC: CAN-SPAM Act compliance guide](https://www.ftc.gov/business-guidance/resources/can-spam-act-compliance-guide-business)
-   [ICO: business-to-business marketing](https://ico.org.uk/for-organisations/direct-marketing-and-privacy-and-electronic-communications/business-to-business-marketing/)
-   [The relevance of lead prioritization: a B2B lead scoring model based on machine learning](https://doi.org/10.3389/frai.2025.1554325)

This tutorial is an implementation guide and a controlled experiment on synthetic data. It is not a production sales system, a legal opinion or evidence of real-world results.