Fine-Tune LFM3, a Large Language Model, into a Decision Model with Unsloth: Lead Qualification with Claude Code over MCP

Page 3 of 3: Claude Code over MCP

This page puts the two models to work. A Claude Code agent researches real companies with Firecrawl, asks the fine-tuned lead and reply models each bounded question through a small MCP server, and hands every answer to a deterministic gate. The gate writes a recommended action to a queue that a person approves. No tool in the setup can send a message, and every result carries "send": false. The page teaches three things: how to write an MCP server for a local model, how to connect it to Claude Code, and how to record an agent session so that others can check it.

Page 1 and page 2 trained the two models. Every code block on this page is a file of the companion repository lead-decision-models or an excerpt of one. The integration code is in integration/, and the transcript is outputs/case-study-session.md.

The case study

Tillbrook Procurement is an imaginary company that sells procurement automation software. Its Ideal Customer Profile (ICP) is the example ICP of page 1: manufacturing, logistics or professional services, in the US or the UK, with 200 to 1,999 employees. Claude Code acts as Tillbrook’s research agent:

flowchart LR
    A[Claude Code agent] -->|search, scrape| F[Firecrawl MCP]
    A -->|lead-request, reply-request| G[gate.py]
    G -->|exact decide request| A
    A -->|decide| L[lead-decisions MCP server]
    A -->|decide| R[reply-decisions MCP server]
    L -->|answer| A
    R -->|answer| A
    A -->|lead, reply| G
    G -->|action, send: false| Q[approval-queue.jsonl]
    Q --> H[A person approves]

The agent decides no action itself. The models answer bounded questions, the gate turns the answers into actions with fixed rules, and a person reviews the queue. The lead model learned from synthetic LeadForge records, so its answers about real companies demonstrate the integration; they are not validated predictions, and this page labels each of them.

Case study results

The recorded session ran 128 turns in 391 s and made 127 tool calls: 7 Firecrawl searches, 20 scrapes, 2 lead decide calls, 8 reply decide calls, 45 shell commands, 32 file writes, 12 reads and 1 tool search. Claude Code’s permission list refused 16 of those calls, the gate refused 8 records with exit code 2, and 1 scrape failed on a certificate error.

Result	Count
Companies researched	10
Records with all three ICP fields on the company’s own pages	2
Records the gate refused before any model call	8
Lead model answers that matched the ICP rule	2 of 2 (one `fit`, one `not_fit`)
Reply model answers that match the intent the reply text states	8 of 8
Queue entries, all with `send: false`	10
Warm decision latency	55 ms median (reply); 52 ms and 666 ms (lead, two calls)

Not validated: trained on synthetic LeadForge data.

The gate did most of the work on the lead side: eight companies state no exact headcount on their pages, and the prompt forbids turning “more than 300” into a band. That is the point of the design. A model answers only when the evidence is complete, and a person reads the evidence before anything is sent.

What you need for Claude Code

Item	Recorded run
Machine and environment	The machine and `unsloth_env` of page 1, with both notebooks run, so that `outputs/lead-decision-merged/` and `outputs/reply-decision-merged/` exist
Models	The merged exports of both notebooks, or the same files downloaded from the Hub: lead-qualifier-merged and reply-router-merged
MCP Python SDK	`mcp` 2.3.0, in `requirements-experiment.txt`
Claude Code	2.1.317, native install, with `claude-opus-5-5`
Firecrawl	The hosted MCP server with OAuth sign-in; a Firecrawl account, no key in the repository
GPU memory	Both servers loaded at once fit in 24 GB; each holds one 4-bit model

To skip the training runs, download the merged models of the recorded run into the folders that .mcp.json names (hf CLI). Each is about 3.3 GB:

hf download raoofaltaher/lead-decision-models-lead-qualifier-merged --local-dir outputs/lead-decision-merged
hf download raoofaltaher/lead-decision-models-reply-router-merged --local-dir outputs/reply-decision-merged

Install Claude Code with the native installer and log in by running claude once (Claude Code setup). Claude Code needs a Pro, Max, Team, Enterprise or Console account.

curl -fsSL https://claude.ai/install.sh | bash
claude --version

Firecrawl’s hosted MCP server has two URLs: https://mcp.firecrawl.dev/v2/mcp with an API key, and https://mcp.firecrawl.dev/v2/mcp-oauth, which opens a browser sign-in and needs no key (Firecrawl MCP server). The repository uses the OAuth one. Every search query and every scraped URL goes to Firecrawl under your account, so read its terms before you send it anything sensitive.

Why an own server and not Unsloth Studio

Unsloth Studio serves decision models through a Decision API and an MCP endpoint, so it is the obvious choice. Studio 2026.10.11 cannot load LFM3: its runtime loads a model without trust_remote_code, and its environment pins Transformers 5.5.0, while LFM3’s code needs 5.22.1. The fix, unslothai/unsloth#16096, is merged and waits for a release (#15588). Until then, the repository serves the models with its own server, which keeps Studio’s decide contract: the same tool, request shape, limits and error messages. When Studio can load LFM3, the agent side of this page does not change.

Step 22: Write the MCP server

integration/decision_server.py serves one merged model folder as an MCP decide tool over stdio, with the MCP Python SDK:

import argparse
import contextlib
import json
import sys
import threading
from collections.abc import Callable
from pathlib import Path
from typing import Any, Union

from mcp.server import MCPServer
from mcp.server.mcpserver.exceptions import ToolError
from pydantic import BaseModel, ConfigDict

QUESTION_TYPES = ("noul", "choice", "score")
MAX_QUESTIONS = 64
MAX_CHOICES = 255
MAX_SCORE_LEVELS = 10
MAX_STATE_CHARS = 200_000
MAX_QUESTION_CHARS = 20_000

DECIDE_DESCRIPTION = (
    "Ask the fine-tuned decision model typed questions about a state (text or JSON). "
    'questions maps a name you pick to {"type", "instructions", "criteria"}. '
    '"noul" is yes/no and answers a probability; "choice" needs criteria mapping each option name '
    'to a description; "score" needs criteria listing 1 to 10 levels, lowest first.'
)

JSONContent = Union[str, dict[str, Any], list[Any]]
Decider = Callable[[JSONContent, dict[str, dict[str, Any]]], dict[str, Any]]


class Question(BaseModel):
    model_config = ConfigDict(extra="forbid")

    type: str
    instructions: JSONContent | None = None
    criteria: JSONContent | None = None


class DecisionRequestError(ValueError):
    pass


def validate_request(state: JSONContent, questions: dict[str, dict[str, Any]]) -> None:
    if not questions:
        raise DecisionRequestError("At least one question is required")
    state_chars = len(state) if isinstance(state, str) else len(json.dumps(state, ensure_ascii=False))
    if state_chars > MAX_STATE_CHARS:
        raise DecisionRequestError(f"State is longer than {MAX_STATE_CHARS} characters")
    if len(questions) > MAX_QUESTIONS:
        raise DecisionRequestError(f"At most {MAX_QUESTIONS} questions per request")
    for name, question in questions.items():
        _validate_question(name, question)


def _validate_question(name: str, question: dict[str, Any]) -> None:
    if len(name) + len(json.dumps(question, ensure_ascii=False)) > MAX_QUESTION_CHARS:
        raise DecisionRequestError(f"Question is longer than {MAX_QUESTION_CHARS} characters")
    kind = question.get("type")
    if kind not in QUESTION_TYPES:
        raise DecisionRequestError(f'Question "{name}" has unknown type "{kind}"')
    criteria = question.get("criteria")
    if kind == "noul":
        if criteria is not None and (not isinstance(criteria, dict) or set(criteria) - {"true", "false"}):
            raise DecisionRequestError(f'Noul "{name}" criteria may only have "true" and "false"')
    elif kind == "choice":
        if not isinstance(criteria, dict) or not criteria:
            raise DecisionRequestError(f'Choice "{name}" needs criteria naming its options')
        if len(criteria) > MAX_CHOICES:
            raise DecisionRequestError(f'Choice "{name}" has more than {MAX_CHOICES} options')
    elif not isinstance(criteria, list) or not 1 <= len(criteria) <= MAX_SCORE_LEVELS:
        raise DecisionRequestError(f'Score "{name}" needs 1 to {MAX_SCORE_LEVELS} criteria levels')
    if criteria is not None:
        entries = criteria.items() if isinstance(criteria, dict) else enumerate(criteria)
        for key, value in entries:
            if value is not None and not isinstance(value, (str, dict, list)):
                raise DecisionRequestError(
                    f'Question "{name}" criteria[{key!r}] must be text, an object, an array or null'
                )


def load_merged_model(path: str) -> Decider:
    from unsloth import FastDecisionModel

    model, tokenizer = FastDecisionModel.from_pretrained(
        path,
        trust_remote_code=True,
        max_seq_length=2048,
        load_in_4bit=True,
    )
    FastDecisionModel.for_inference(model)

    def decide(state: JSONContent, questions: dict[str, dict[str, Any]]) -> dict[str, Any]:
        return FastDecisionModel.predict(model, tokenizer, state, questions)

    return decide


class DecisionService:
    def __init__(self, model_path: str, loader: Callable[[str], Decider]):
        self.model_path = model_path
        self._loader = loader
        self._decider: Decider | None = None
        self._lock = threading.Lock()

    def decide(self, state: JSONContent, questions: dict[str, dict[str, Any]]) -> dict[str, Any]:
        validate_request(state, questions)
        with self._lock, contextlib.redirect_stdout(sys.stderr):
            if self._decider is None:
                self._decider = self._loader(self.model_path)
            return {"answers": self._decider(state, questions)}


def build_server(service: DecisionService) -> MCPServer:
    server = MCPServer("Unsloth decision model")

    @server.tool(description=DECIDE_DESCRIPTION)
    def decide(state: JSONContent, questions: dict[str, Question]) -> dict[str, Any]:
        try:
            return service.decide(
                state, {name: question.model_dump(exclude_none=True) for name, question in questions.items()}
            )
        except ValueError as error:
            raise ToolError(str(error)) from None

    return server


def main(argv: list[str] | None = None, loader: Callable[[str], Decider] = load_merged_model) -> None:
    parser = argparse.ArgumentParser(description="Serve one merged decision model as an MCP decide tool over stdio.")
    parser.add_argument("--model", required=True, help="Local folder of a merged decision model")
    args = parser.parse_args(argv)
    if not (Path(args.model) / "config.json").is_file():
        parser.error(f"{args.model} has no config.json; pass a local merged model folder")
    build_server(DecisionService(args.model, loader)).run("stdio")


if __name__ == "__main__":
    main()

The parts that matter when you write your own:

One tool, typed. MCPServer turns the decide function into a tool, and its type hints become the JSON schema that Claude Code shows the model. Question is a pydantic model with extra="forbid", so a misspelled field is refused before anything runs.
Validate before you load. validate_request applies Studio’s limits: at least one question, a state of at most 200,000 characters, at most 64 questions, the types noul, choice and score, at most 255 options, 1 to 10 score levels. An invalid request returns a ToolError with Studio’s message and never loads the model.
Load lazily, once, under a lock. The first decide call runs FastDecisionModel.from_pretrained in 4 bits, the same call that page 2 uses to reload the merged export. One lock covers the load and every prediction, so two first calls load the model once.
Keep standard output clean. MCP over stdio sends JSON-RPC on standard output, and Unsloth prints banners while it loads. The server redirects standard output to standard error during the load and every prediction. tests/test_decision_server.py starts the server as a client and checks that every line on standard output is JSON-RPC.
Serve a local folder only. main stops when --model holds no config.json. The LFM3 code that runs is the Python files in the merged folder; read them first.

The setup is local, but not offline: Unsloth contacts the Hugging Face Hub while it loads, and the serving log records those requests.

Step 23: Connect Claude Code

Claude Code reads project MCP servers from .mcp.json in the repository root (Claude Code: MCP):

{
  "mcpServers": {
    "lead-decisions": {
      "type": "stdio",
      "command": "unsloth_env/bin/python",
      "args": ["integration/decision_server.py", "--model", "outputs/lead-decision-merged"]
    },
    "reply-decisions": {
      "type": "stdio",
      "command": "unsloth_env/bin/python",
      "args": ["integration/decision_server.py", "--model", "outputs/reply-decision-merged"]
    },
    "firecrawl": {
      "type": "http",
      "url": "https://mcp.firecrawl.dev/v2/mcp-oauth"
    }
  }
}

Server	Transport	What it runs
`lead-decisions`	stdio	`decision_server.py` on `outputs/lead-decision-merged`
`reply-decisions`	stdio	`decision_server.py` on `outputs/reply-decision-merged`
`firecrawl`	HTTP with OAuth	Firecrawl’s hosted MCP server

The same entries can be written from the command line. --scope project writes to .mcp.json, and -- separates Claude’s options from the server command:

claude mcp add --scope project lead-decisions -- unsloth_env/bin/python integration/decision_server.py --model outputs/lead-decision-merged
claude mcp add --scope project reply-decisions -- unsloth_env/bin/python integration/decision_server.py --model outputs/reply-decision-merged
claude mcp add --scope project --transport http firecrawl https://mcp.firecrawl.dev/v2/mcp-oauth

Then, in the repository folder:

Start claude. Claude Code asks you to approve the three project servers, because .mcp.json comes from the repository.
Run /mcp and sign in to Firecrawl in the browser.
Run claude mcp list and check that all three servers show as connected.

The decision servers start unsloth_env/bin/python with paths relative to the repository root, so run claude from that root. No server listens on a network port.

Step 24: The gate

integration/gate.py is the only path from a model answer to an action. It has four commands:

Command	What it does
`lead-request --features f.json`	Validates a lead record against the categories the model was trained on and prints the exact `decide` request. Exit code 2 means a field is missing or a value is unknown: research the lead instead of calling the model
`reply-request --reply-text "..."`	Prints the exact `decide` request for a reply
`lead --domain d --answers a.json ...`	Applies the outreach policy of page 1 and appends one queue entry per domain
`reply --reply-text "..." --answers a.json ...`	Applies the routing policy of page 2; an opt-out phrase in the text always suppresses marketing

The values that lead-request accepts are the values the lead model saw in training, and the same block holds the opt-out phrases:

TRAINED_VALUES = {
    "industry": {"manufacturing", "logistics", "professional_services", "healthcare_non_clinical"},
    "region": {"US", "UK"},
    "employee_band": {"200-499", "500-999", "1000-1999", "2000+"},
}
OPT_OUT_PHRASES = (
    "unsubscribe",
    "remove me",
    "take me off",
    "stop emailing",
    "opt out",
    "opt me out",
    "do not contact",
    "don't contact",
)

build_lead_request collects every problem before it refuses a record:

def build_lead_request(features: dict) -> dict:
    unknown_fields = sorted(set(features) - set(MODEL_FEATURES))
    problems = [f"unknown fields {unknown_fields}"] if unknown_fields else []
    for field, allowed in TRAINED_VALUES.items():
        value = features.get(field)
        if value is None:
            problems.append(f"missing ICP evidence: {field}")
        elif value not in allowed:
            problems.append(f"{field}={value!r} is outside the training categories {sorted(allowed)}")
    if problems:
        raise ValueError("; ".join(problems))
    return {"state": make_lead_state(features), "questions": LEAD_QUESTIONS}

A malformed model answer counts as no answer, so the policy fails closed to a review action. A domain is queued once. Every result carries "send": false, and tests/test_gate.py pins these rules.

The case study researches companies, not people, so the prompt passes --suppressed false --persona-fit unknown --eligibility unknown for every lead. The best action a lead can then get is human_persona_review: a person must still find a contact and check that contacting them is allowed. In a deployment, the suppression flag comes from your suppression records, never from a prompt.

Step 25: The prompt and the recording

integration/case_study_prompt.md gives the agent the ICP, the allowed values, the gate commands and the order of the steps. It never asks the agent for an outreach decision.

The prompt of the recorded session

You are the research agent of Tillbrook Procurement, an imaginary company that sells procurement automation software. You find and research prospects, and you hand every bounded decision to two fine-tuned decision models. You never decide an outreach action yourself, and you never send any message to anyone.

Tillbrook's Ideal Customer Profile: companies in manufacturing, logistics or professional services, based in the US or the UK, with 200 to 1,999 employees.

You have these tools:
- `firecrawl` (MCP): web search and page scraping.
- `lead-decisions` (MCP, tool `decide`): the fine-tuned lead qualification model.
- `reply-decisions` (MCP, tool `decide`): the fine-tuned reply routing model.
- Bash, to run `python integration/gate.py`, the deterministic policy gate. Use the active virtual environment.

Part A, prospects. Work through these steps for each company:

1. With Firecrawl search, find 8 to 12 real companies that may match the profile. Spread the searches across the three industries and both regions.
2. Scrape up to three pages per company, the About page first, to learn what it does, where it is based and how many people it employs.
3. Write `outputs/case-study/<domain>.features.json` with only the fields `industry`, `region` and `employee_band`. Use only these values, and only when the pages support them: industry `manufacturing`, `logistics`, `professional_services` or `healthcare_non_clinical`; region `US` or `UK`; employee_band `200-499`, `500-999`, `1000-1999` or `2000+`. A headcount stated only as a lower bound supports a band only when the bound guarantees it: "over 2,000" supports `2000+`, but "over 500" or "800-plus" supports no band. A figure for a parent group is not the company's own headcount. When the pages do not support a value, leave that field out. Do not guess. The gate refuses a record that lacks any of the three fields and sends that company to research.
4. Write `outputs/case-study/<domain>.evidence.json` with each value you set, the sentence that supports it, and the page URL.
5. Run `python integration/gate.py lead-request --features outputs/case-study/<domain>.features.json`. If it exits with code 2, record the error and do not call the model for that company.
6. Call `lead-decisions` `decide` with exactly the `state` and `questions` that command printed. Save the full answer as `outputs/case-study/<domain>.answers.json`.
7. Run `python integration/gate.py lead --domain <domain> --answers outputs/case-study/<domain>.answers.json --suppressed false --persona-fit unknown --eligibility unknown --queue outputs/approval-queue.jsonl` and keep its printed action.

Part B, replies. For each item in `data/case-study/replies.json`:

1. Run `python integration/gate.py reply-request --reply-text "<reply_text>"`.
2. Call `reply-decisions` `decide` with exactly the printed `state` and `questions`. Save the answer as `outputs/case-study/<reply_id>.answers.json`.
3. Run `python integration/gate.py reply --reply-text "<reply_text>" --answers outputs/case-study/<reply_id>.answers.json --suppressed <contact_is_suppressed> --opt-out-event <explicit_opt_out_event> --queue outputs/approval-queue.jsonl` and keep its printed action. Write the two JSON booleans in lower case, `true` or `false`.

When you finish, print one table for the prospects (domain, ICP answer and its probability, gate action) and one for the replies (reply_id, intent answer and its probability, gate action). End with one sentence stating that no message was sent.

integration/record_case_study.sh runs the prompt headless with claude -p and renders the result:

#!/usr/bin/env bash
set -uo pipefail
export PATH="${HOME}/.local/bin:${PATH}"
command -v claude > /dev/null || { echo "claude is not on PATH"; exit 127; }
cd "$(dirname "$0")/.."
source unsloth_env/bin/activate
mkdir -p outputs/case-study outputs/logs
rm -f outputs/approval-queue.jsonl
start=$(date +%s)
claude -p "$(cat integration/case_study_prompt.md)" \
  --mcp-config .mcp.json \
  --strict-mcp-config \
  --setting-sources project \
  --allowedTools \
    "mcp__firecrawl__firecrawl_search" \
    "mcp__firecrawl__firecrawl_scrape" \
    "mcp__lead-decisions__decide" \
    "mcp__reply-decisions__decide" \
    "Bash(python integration/gate.py *)" \
    "Edit(outputs/case-study/**)" \
    "Read(data/case-study/**)" \
    "Read(outputs/case-study/**)" \
  --output-format stream-json \
  --verbose \
  > outputs/logs/case-study-session.raw.jsonl 2> outputs/logs/case-study.err
status=$?
end=$(date +%s)
echo "claude exit=${status} seconds=$((end - start))"
python tools/render_session.py \
  --raw outputs/logs/case-study-session.raw.jsonl \
  --prompt integration/case_study_prompt.md \
  --queue outputs/approval-queue.jsonl \
  --out-jsonl outputs/case-study-session.jsonl \
  --out-md outputs/case-study-session.md

--mcp-config .mcp.json --strict-mcp-config loads the three servers and no other.
--setting-sources project loads the project settings only.
--allowedTools allows exactly the Firecrawl search and scrape tools, the two decide tools, the gate, file writes under outputs/case-study/ and reads under data/case-study/ and outputs/case-study/. Every other call needs a permission that nobody can grant in a headless run, so Claude Code refuses it, and the transcript shows the refusal. Read-only shell commands such as ls and cat are the exception: Claude Code runs them without a rule.
--output-format stream-json --verbose prints every event. tools/render_session.py keeps only known-safe fields, cuts every Firecrawl result to 600 characters, and adds “Not validated: trained on synthetic LeadForge data.” after every lead answer.

What the first recording taught

The first recording, kept in outputs/case-study-first-run/, used an earlier gate that let a record with missing fields reach the model. The lead model answered fit with a probability between 0.9998 and 1.0 for all nine companies, although seven records had no employee band and two of those had no region:

Domain	Fields set	ICP answer
evapco.com	manufacturing, US	fit (0.9998)
xtrac.com	manufacturing	fit (0.9999)
denholmgoodlogistics.com	logistics, UK	fit (1.0)
schneiderdowns.com	professional_services	fit (1.0)
ksmcpa.com	professional_services, US, 500-999	fit (1.0)

Not validated: trained on synthetic LeadForge data.

The question says that a missing field means insufficient_evidence, but the model gave that answer a probability of 0 every time. LeadForge never leaves a field empty, so the model never saw that answer in training (page 1). The agent also turned lower bounds into bands: “800-plus CPAs” became 500-999. Two fixes followed. The gate now refuses an incomplete record with exit code 2 and the next action research_missing_account_evidence, and the prompt accepts a lower bound only when it guarantees a band (“over 2,000” supports 2000+; “over 500” supports nothing). The lesson is general: a model cannot give an answer that its data never showed, so handle missing evidence in code.

The recorded session

outputs/case-study-session.md is the recording. Claude Code 2.1.317 ran it with claude-opus-5-5 in permission mode default, with the three servers connected.

The approval queue

#	Kind	Subject	Model answer	Action	Send
1	lead	verstlogistics.com	not_fit (1.00)	do_not_recommend_outreach	false
2	lead	chemringcm.com	fit (1.00)	human_persona_review	false
3	reply	case-01: “Thanks for the note about Tillbrook. Could we set up 30 minutes next Thursday to see how the approval workflow works?”	meeting_requested (1.00)	handoff_to_sales	false
4	reply	case-02: “What would this cost for roughly 300 requesters across two plants?”	more_information (1.00)	prepare_requested_information	false
5	reply	case-03: “We’re mid-way through an ERP upgrade. Please come back to me in early March.”	not_now (1.00)	schedule_later_after_date_confirmation	false
6	reply	case-04: “I’m out of the office until 24 November with limited access to email.”	out_of_office (1.00)	pause_until_return_date_confirmation	false
7	reply	case-05: “I look after facilities, not purchasing systems, so I’m the wrong person for this.”	wrong_person (1.00)	human_review	false
8	reply	case-06: “Thanks, but we’re not going to take this forward.”	not_interested (1.00)	stop_sequence	false
9	reply	case-07: “Please unsubscribe me from these emails.”	opt_out (1.00)	suppress_marketing	false
10	reply	case-08: “Sounds genuinely useful for our buyers, but please take me off your mailing list.”	opt_out (1.00)	suppress_marketing	false

Not validated: trained on synthetic LeadForge data.

Prospects

The agent researched ten companies, up to three pages each. Only two had pages that supported all three ICP fields:

Domain	Fields the pages supported	Gate result	Reason from the evidence file
chemringcm.com	manufacturing, UK, 200-499	Model called: fit (1.00), human_persona_review, priority high	The unit’s page states “we employ 400 people”; the agent noted that this is the unit’s headcount, not the group’s
verstlogistics.com	logistics, US, 2000+	Model called: not_fit (1.00), do_not_recommend_outreach	The About page states “2,500 Total Employees”, above the ICP range
evapco.com	manufacturing, US	Refused: no employee band	No headcount on the About page; the careers page returned 404
ki.com	manufacturing, US	Refused: no employee band	No headcount on the About or careers page
wagnerlogistics.com	logistics, US	Refused: no employee band	The only figure on the page belongs to an executive’s previous employer
tenens.com	logistics, UK	Refused: no employee band	No headcount on its pages; “800+ colleagues” in a social post is a lower bound
rettew.com	professional_services, US	Refused: no employee band	“More than 300 professionals” is a lower bound that supports no band
hcrlaw.com	professional_services, UK	Refused: no employee band	“More than 1,000 people” does not guarantee 1000-1999
watermanaspen.co.uk	professional_services, UK	Refused: no employee band	“More than 850 people” is a lower bound
stanleyconsultants.com	professional_services	Refused: no region, no employee band	No page states where the firm is based today; “1,000+ Employees” is a lower bound

Not validated: trained on synthetic LeadForge data.

Both model answers match the ICP rule: 400 employees in a UK manufacturer is inside the band, and 2,500 is outside it. The gate checks that a value is present and allowed, not that it is true, so the person who reviews the queue still reads the evidence files, which hold the sentence and the URL behind every value. The agent saved the eight refusals in outputs/case-study/lead-gate-errors.json.

Replies

The eight replies are fixed inputs in data/case-study/replies.json, with contact_is_suppressed and explicit_opt_out_event set to false, so only the text and the model’s answers decide the action. The reply model, retrained in page 2, gave each reply the intent its text states, each with a probability above 0.999, and its follow-up answer was no for the three replies that end the conversation. case-08 is the mixed reply that praises the product and asks to come off the list: the model answered opt_out at 0.9996, and the gate would suppress it anyway, because “take me off” is one of its opt-out phrases.

From the transcript

Transcript step 26 scrapes the page of Chemring Countermeasures. The published transcript keeps the first 600 characters of each Firecrawl result:

### 26. `mcp__firecrawl__firecrawl_scrape`

```json
{
  "url": "https://www.chemring.co.uk/about-us/our-business/chemring-countermeasures-uk",
  "formats": [
    "markdown"
  ],
  "onlyMainContent": true,
  "objective": "Learn what the company does, where it is based and how many people it employs, for ICP qualification.",
  "conversationId": "34af5291-4fca-4e07-b682-8ed1cb89a7ee"
}
```

[... 28 lines cut ...]
Result of step 26:

```text
{"markdown":"This site uses cookies. Cookies that are strictly necessary for the website to function are always on and cannot be switched off. To accept functional, performance and targeting cookies, click “Accept All”. Alternatively, you can reject these cookies by clicking the “Necessary Cookies only” button. View our [cookies page](https://www.chemring.com/site-services/cookie-statement \"Opens in a new window\").\n\nAccept all cookiesDecline cookiesChange cookie preferences\n\nClose\n\n- [About us](https://www.chemring.com/about-us)\n- [What we do](https://www.chemring.com/what-we-do)\n- [
[21,051 characters cut]
```

Transcript step 77 runs the gate on RETTEW, whose record has no employee band. The gate refuses it with exit code 2, so the model is never called:

### 77. `Bash`

```json
{
  "command": "python integration/gate.py lead-request --features outputs/case-study/rettew.com.features.json",
  "description": "Gate lead-request for rettew.com"
}
```
[... 8 lines cut ...]
Step 77 refused or failed:

```text
Exit code 2
{"error": "missing ICP evidence: employee_band", "next_action": "research_missing_account_evidence", "send": false}
```

Transcript step 82 sends the lead model exactly the request that the gate printed for Chemring. Three of the 24 features have values; the other 21 are null, and the answer comes back with the not-validated label:

### 82. `mcp__lead-decisions__decide`

```json
{
  "state": {
    "features": {
      "activity_count": null,
      "buyer_role": null,
      "days_since_first_touch": null,
      "days_since_last_touch": null,
      "demo_page_views": null,
      "employee_band": "200-499",
[... 48 lines cut ...]
Result of step 82:

```text
{"answers":{"icp_fit":{"type":"choice","choice":"fit","confidence":1,"probabilities":{"fit":1,"not_fit":0,"insufficient_evidence":0},"answer":"fit"},"conversion_outcome_90d":{"type":"noul","noul":0.0149,"answer":false,"probabilities":{"true":0.0149,"false":0.9851}}}}
```

> Not validated: trained on synthetic LeadForge data.

The permission list refused the agent’s shortcuts, and the transcript shows every refusal. Examples: a Python script in a shell here-document at transcript step 37, a for loop over the feature files at step 60, and ; echo "exit=$?" after a gate command at steps 61 to 70. After the refusals, the agent wrote each file with a separate write call and ran each gate command on its own.

Latency

tools/serving_check.py starts each server over MCP stdio, as Claude Code does, sends the first request once to load the model, then sends every request of the session and compares each answer with the saved one. The recorded check, outputs/serving-check.json:

{
  "lead": {
    "calls": 3,
    "first_call_s": 22.17,
    "warm_median_ms": 358.8,
    "warm_max_ms": 665.9,
    "matches_session": {
      "chemringcm.com": true,
      "verstlogistics.com": true
    }
  },
  "reply": {
    "calls": 9,
    "first_call_s": 20.1,
    "warm_median_ms": 55.4,
    "warm_max_ms": 416.5,
    "matches_session": {
      "case-01": true,
      "case-02": true,
      "case-03": true,
      "case-04": true,
      "case-05": true,
      "case-06": true,
      "case-07": true,
      "case-08": true
    }
  }
}

Reply server. The first call took 20.1 s, the model load included. Over eight warm calls, the median was 55.4 ms and the maximum 416.5 ms.
Lead server. The first call took 22.2 s. The two warm calls took 52 ms and 666 ms; with only two calls, the tool’s median of 358.8 ms is their mean and says little.
Answers. Every answer matched the session: both lead answers and all eight reply answers.

The check ran on an NVIDIA GeForce RTX 5090 Laptop GPU, with both servers started one after the other.

The transcript does not time the calls: the timestamps of the stream can lag a call by up to about a second, too coarse for a decision that takes tens of milliseconds.

Security, privacy and compliance

No send capability. No allowed tool can send a message, every result carries "send": false, and a person reviews the queue.
A fixed permission list. The recording allows only the tools it names and refused the agent’s own scripts and shell loops. Read-only shell commands still run, so keep secrets out of the folder where the agent runs.
Remote code at a pinned revision. LFM3 runs its own Python code through trust_remote_code=True, loaded only from a local merged folder that holds the files of revision ac2586a34f1a1c03329dbdcdefabd8b6478ddaa7.
Untrusted page text. A scraped page can carry text written to steer an agent. Page text can change the agent’s notes, but not the allowed tools or the rules of the gate.
Real company names, labelled. The case study stores only company facts with the sentence and URL behind each, and labels every lead prediction as not validated.
No network listener. The decision servers talk to Claude Code over stdio only. Never expose an unauthenticated decision API to the internet.

Keep suppression records in a deterministic system outside the model, keep the source URL of every extracted claim, log predictions, decisions and approvals, and review the law for the sender, the recipient and the channel. The FTC’s CAN-SPAM guide covers B2B email and requires prompt opt-out handling; the ICO’s B2B marketing guidance covers the UK. This page is not legal advice.

Limitations

Synthetic training data. The lead model learned from LeadForge’s synthetic CRM records, and the reply model from 617 synthetic replies that one Claude model wrote and a second labelled blind. No person reviewed the replies, and neither model is validated on real companies or real inboxes.
Small test sets. The reply test set holds 124 replies. Two runs of the same notebook scored 0.924 and 0.975, so read a range, not a point.
Tabular data favours simple models. On the lead features, a logistic regression ranks conversions slightly better than the fine-tuned model.
Missing evidence. The lead model never saw a record with an empty ICP field, so the gate, not the model, handles missing evidence.
One recorded session. Another run researches other prospects. The gate checks that each field is present and allowed, not that the evidence is true.
One machine. Every time and latency figure comes from one RTX 5090 Laptop GPU.
No outreach text. The tutorial classifies replies and routes actions. It does not write or optimize outreach messages, on purpose.

Work with me

Every company has its own ICP, its own CRM history and its own inbox. The method of this tutorial transfers to all three.

Write the policy as code. Label your own data. Fine-tune a small decision model on one GPU, a laptop GPU in this tutorial, and compare it with a simple baseline. Then put every action behind a gate and a person.

The models run on your own hardware, so lead data can stay inside your infrastructure. Claude Code, or any MCP client, reaches them through one small server, and every session leaves a transcript that a reviewer can read.

Do you want a decision model trained on your own ICP, CRM history or reply taxonomy, self-hosted and connected to your agents? Do you want me to check first whether a fine-tuned model or a simpler classifier fits your data better? Contact me.

References

Unsloth, the base model and the models of this tutorial

Unsloth: Train Your Own Decision Model
Unsloth decision-model implementation at the pinned commit
Unsloth Studio and the LFM3 fix unslothai/unsloth#16096
LFM3-1.7B on Hugging Face and its LFM Open License v1.0
lead-decision-models on GitHub
Lead qualifier and its merged copy
Reply router and its merged copy

Claude Code, MCP and Firecrawl

Data

Setup, training and evaluation

Compliance and research

This tutorial is an implementation guide and a controlled experiment on synthetic data. It is not a production sales system, a legal opinion or evidence of real-world results.