The reference implementation¶

examples/docplan is a small, complete application built on artifactr: a person and an agent write a document together, then plan the work it describes. It uses only the library's public API, a test enforces that, and it is held to the same quality gates as the library (ADR-0024). Read it when you want to see every piece of this documentation working together.

What it contains¶

Run it¶

docplan is a member of the repository's uv workspace, so it runs against the library in the same checkout. From the repository root, with an Anthropic API key:

make install                    # or: uv sync --all-packages
export ANTHROPIC_API_KEY=...
uv run docplan-serve            # in one terminal
uv run docplan --user alice     # in another

To use another provider, set DOCPLAN_MODEL to any pydantic-ai model name and that provider's key. The server listens on 127.0.0.1:8000; set DOCPLAN_HOST and DOCPLAN_PORT to change it.

By default the server keeps its workspaces in memory, and they are gone when it stops. Set DOCPLAN_DATABASE_URL to keep them in SQLite or PostgreSQL with SQL storage; the server migrates the database to artifactr's schema when it starts:

DOCPLAN_DATABASE_URL=sqlite+aiosqlite:///docplan.db uv run docplan-serve
DOCPLAN_DATABASE_URL=postgresql+asyncpg://user:password@localhost/docplan uv run docplan-serve

Then ask for a document and a plan. The agent edits the document directly; its changes to the plan arrive as proposals you accept with /accept or reject with /reject. Edit anything yourself and the agent is told what you changed on its next turn. When it asks a question, your next message answers it. The docplan README lists every client command and flag, and shows a sample session.

The same server speaks the other surfaces too: REST under /v1 (the demo trusts an x-user header, as in curl -H 'x-user: alice' localhost:8000/v1/workspaces/main/artifacts), and MCP at http://127.0.0.1:8000/mcp/ for external agents.

Evaluate it¶

docplan's agent is told to keep its edits small. People say when it did not with /edits ok|big, an edit_size on the turn, and whether a thread did what they asked with /done yes|no 1-5, a task_completion. evals.py closes the loop over that feedback with the [evals] extra (Evaluation):

1. dataset turns people's edit_size feedback into an evalr dataset with a LogFeedbackSource, and turn_edits builds each example's input from the turn as it ended: the request, and the docs before and after.
2. edit_size_judge is a function evaluator of the same input, and calibrate has evalr's BestOf choose its threshold by agreement with people. With DOCPLAN_JUDGE_MODEL set and the dspy extra installed, the judge is a DSPy judge instead.
3. With DOCPLAN_EVAL_SAMPLE_RATE set, the server's Runner judges that share of turns with an OnlineEvaluator, within DOCPLAN_EVAL_BUDGET a day, and records each verdict as feedback from the judge.
4. replay replays the dataset's turns against an agent with replay_task, and the same judge judges them.
5. measures computes task completion, drop-off (from thread_sessions) and the rewrite rate (from artifact_histories).

docplan-eval runs the steps against the server's database:

export DOCPLAN_DATABASE_URL=sqlite+aiosqlite:///docplan.db
DOCPLAN_EVAL_SAMPLE_RATE=1 uv run docplan-serve   # judge every turn; chat, then /edits and /done
uv run docplan-eval judge                         # the dataset, and the judge that agrees best
uv run docplan-eval replay                        # the turns replayed against the agent
uv run docplan-eval measures                      # task completion, drop-off, rewrite rate

The docplan README describes each variable and option.

Test it¶

The tests run the whole stack with a scripted FunctionModel, so they need no API key:

uv run pytest examples/docplan/tests

Testing your application explains the pattern.

How it maps to the guides¶