Six real problems.What we built. What changed.

Stack

Terms, translated

Deterministic cohort matching

Fixed rules check whether the patients in a study look like this patient. The same input always gives the same result.

ERS evidence score

Each study gets a score for its design, the strength of its results, how recent it is and how credible the source is. Anyone can check why one study ranks above another.

Hybrid search: dense and sparse vectors

Two searches run side by side. One looks for meaning, the other for exact words. A study is found even when it uses other terms for the same thing.

Embeddings, 1,536 dimensions

Each text becomes a list of 1,536 numbers. Texts with similar meaning get similar numbers, so a computer can measure how close they are.

Reciprocal Rank Fusion (RRF)

Merges the two result lists by position, not by raw score, so neither search wins by accident.

Guideline mapping

Each study is labelled as in line with, in conflict with or not covered by the ESMO guidelines, the European standard for cancer care.

How it is tested

• Baseline: an oncologist's manual research and plain keyword search. The system has to beat it, not just sound fluent.
• Measures: search correctness, citation and answer faithfulness (the answer says only what the source says), time to evidence, agreement between oncologists and error analysis.
• Test packs: 10 to 15 patient cases with starter PubMed and ESMO sets, small enough to check by hand and break on purpose. A confusion matrix tracks false negatives (relevant studies the system missed).
• Frozen benchmark, 15 patient cases, current build: recall 40% (share of expected studies found; the rest are false negatives), first label correct in 8 of 15 cases (53%), exact guideline topic in 4 of 15 (27%). Low, and shown on the project page as low.
• Hybrid search on the same pack: 487 extra studies found, and 214 more evidence items aligned after the rules checked them (704 instead of 490). The rules engine still sets every final label.

Ruled out

Free-form LLM answers as the authority. They change from run to run and are hard to audit. The LLM layer stays optional: grounded summaries and import help, never the clinical source of truth.

What broke

A strict rules engine can still fail on meaning. To a doctor, egfr-tki, tki and targeted can mean the same treatment. To the code they were three different tags, until the team built a shared vocabulary across patient cases, PubMed and ESMO.

Build order

The rules engine and its test harness came first. Then small test packs, data cleanup, the shared vocabulary, the web app and the live rules engine. Hybrid search came seventh. The optional AI summary came last.

What it records

Each answer keeps its source document, the evidence fragment, the data version, the search logic, the model and prompt settings, and the review trail.

Sources

• Project page with the full stack

Stack

Terms, translated

State machine

A contract can only move along allowed steps. It cannot jump from a draft straight to paid.

Append-only event log

Every change is added as a new line and never overwritten. The full history of each contract can be read back.

A spreadsheet as the database

In this version a Google Sheet holds the data. Only one layer of the code may read or write it, and IDs come from the server, never from row numbers.

Webhooks

The task board and the signature service tell the app when something changes. Nobody has to check by hand.

Outbox and retry jobs

If an outside service is down, the request waits in a queue and goes out again later. It does not get lost.

Magic link

The freelancer opens a personal link. No account, no password.

AES-GCM encrypted drafts

A half-filled form is saved encrypted, so a freelancer can finish later and the data is not stored in plain text.

How it is tested

• Baseline: the outsourced process, about 30% on top of each contract's value, with little visibility.
• Success means the whole process runs in-house, with visible status, fewer handoffs, no loss of legal control and a lower cost than the baseline. Result: the outside firm is fully replaced.
• Built in seven two-week sprints from February to May 2026, then hardened in real use. More than 900 automated tests.

Ruled out

LLM contract generation. Approved legal templates are the only source of contract text, and only a small set of fields can change.

What broke

A spreadsheet has request limits. Under load the app hit Google's quota, so it now retries with back-off (each retry waits a little longer) and defers work instead of failing.

What it records

Each contract keeps its status, every event, its documents, the signature trail, the handoffs and the decisions. ID numbers, bank details and document contents never go into the event log.

Stack

Terms, translated

Textual elicitation

AI generates a text response, similar to an interview answer, instead of selecting a number directly.

Semantic Similarity Rating (SSR)

Each answer is compared with reference sentences. For a purchase question, these range from "I would definitely not buy this" to "I would definitely buy this". How close it is to each one becomes a score distribution.Technically: free-text answer → embedding → cosine similarity to Likert anchor statements → probability over the scale → segment distribution

Reference anchors

The model sentences that stand for each point of the scale. Different anchors give different scores, so we test several sets.

SSR temperature

How sharply an answer is pinned to one score. It has to be tuned for each type of question. One value does not fit all.

Pre-registered controls

Before a run we add a concept that should clearly win and one that should clearly lose. If the method cannot separate them by a set margin, the run does not count.

Frozen runs

Prompts, models, settings, anchors and inputs are saved with each result, so any run can be repeated and challenged.

How it is tested

• Client use: over 100 concepts screened, 98% lower cost per run than a panel, agreement with real panel results above 88%.
• The method was first tested on public survey data with real answers (Mendeley, CC BY 4.0).
• New-product survey, 6 questions, 438 real respondents: KS similarity 0.887 (how much the AI and human answer patterns overlap, 1 is identical), mean absolute error 0.036 per answer option, and the average score 0.39 points off on a 7-point scale.
• A harder survey on green purchases, 253 respondents: KS similarity 0.73, then 0.78 after a prompt revision. Useful for direction, not a replacement for a panel.
• Model choice: with the same embeddings, Qwen3 30B came out ahead of DeepSeek on all three measures, and ahead of Gemini Flash Lite on score error. In the embedding comparison, bge-m3 scored best, ahead of qwen3-embedding-8b and text-embedding-3-small.

Ruled out

Simple LLM opinions, classic synthetic surveys and fully simulated agent respondents. Text answers scored with SSR kept the reasoning and still gave numbers that can be compared.

What broke

A model comparison failed on plumbing, not quality. One provider kept hitting rate limits (it refuses requests when too many arrive) and returned broken JSON. Since then every comparison reports reliability next to quality. And when a run missed our pre-set control margin by a few hundredths, we marked it failed, not nearly passed.

What it records

Each result keeps the prompt version, the model and provider, the run settings, audience segments, SSR anchors, raw answers, final scores, cost and time.

Sources

• The SSR method paper, PyMC Labs (arXiv 2510.08338)
• SSR reference implementation
• Benchmark data: New Product Purchase Intentions (Mendeley)

Stack

Terms, translated

Baseline

The simplest forecast anyone could make: the average for that weekday, plus the events we already know about. A model has to beat it to earn its place.

Backtest on a closed month

We hide a past month, forecast it and compare the forecast with what really happened. No live risk.

Forecast error

How far the forecast was from the real number, on average.

Seasonal drift

A forecast slowly gets worse as seasons and habits change. We watch for it.

How it is tested

• Baseline today: manager experience, calendars, reservations, manual reports and simple rules by weekday, season and weather.
• Measures: forecast error, understaffed and overstaffed shifts, time to plan the schedule, lost or served demand, and feedback from managers and reception.

Ruled out

Going straight to a live model. The simple baseline may be good enough, or the backtest may show that the data is not ready yet. Both are useful answers.

What broke

Nothing yet. This part fills in after the first backtest.

What it records

Each forecast will show the history period it used, the data sources, the weather and event inputs, the model version and every correction a manager made.

Stack

Terms, translated

Read-only adapter

The board reads the booking system's public availability data and never writes to it. If that data changes shape, the board says so instead of guessing.

Freshness states

Every column shows when it was last read and whether the data is fresh, stale or failed, so a coordinator knows when to double-check.

Parity check

Compares the board with the booking system's own staff panel. Over two days, times and spaces matched exactly for every fixed-slot activity.

Realtime sync

When one person picks a slot, everyone else's board updates at once, without a refresh.

Internal reservation

A hold inside the board only. It stops two colleagues planning the same slot, but it never blocks or books anything in the real system.

Isolated columns

All activities load at the same time. If one fails, only its column shows the error. The rest of the board keeps working.

How it is tested

• Baseline: 15 to 20 minutes to plan one event in the booking system, as reported by the events team.
• Target for the availability check: about 1 minute.
• September: 353 reservations on the board, about 30 hours of planning at a cautious 5 minutes each, against 88 to 117 hours for the same checks in the booking system. Estimates, not measurements.
• The events team reviewed the board in a working session and accepted it without material changes.
• A three-week evidence stage ended in a continue, change or stop decision.

Ruled out

Writing bookings into the booking system, rebuilding its capacity logic and sending messages to customers automatically. The booking system stays the source of truth, and people talk to customers.

What broke

Not a failure yet, but the main risk. With no API, the board reads the public availability data that the booking website uses, and that data can change shape without notice. So a failed, stale or changed read is a normal state on the board, shown to the team, never hidden.

What it records

Every availability read is stored with its time and result, so the team can see how stable the data is and what changed between reads. Every reservation and change is linked to the person who made it.

Stack

Terms, translated

Vision-language model (VLM)

An AI model that looks at a photo and answers questions about it. Here: how wide and how high is the opposite wall?

Scale from known objects

Doors, switches and furniture have typical sizes. The model uses them, plus the usual 72° field of view of a phone camera, to turn pixels into centimetres.

Segmentation (Mask2Former)

Labels every pixel of the photo as wall, window, floor or ceiling, so the wizard knows where curtains can hang.

Wall × box fractions

The model measures the whole wall. The area the customer marks is a share of it, so its size in centimetres follows from the wall's.

Percentile trimming (p10/p90)

Ignores the most extreme 10% at each edge of the detected wall, so a stray shadow does not stretch the measurement.

Artist texture pipeline

Fabric textures made in Blender. The renderer picks the fold type (wave, flex, double flex) from the pleat and the material, so the preview looks like the sewn curtain.

How it is tested

• Measurement tests: within 3 to 10% of the real size from the photo alone, against 20 to 40% with the A4-sheet method.
• A benchmark script runs photos against measured ground truth and logs the time each model takes.
• Model choice: Qwen VL gave the best accuracy for its cost. GPT-4o-mini was too imprecise for centimetres.
• No before-number for the business: the old way of ordering was not online, so there is nothing like-for-like to compare.

Ruled out

The A4 sheet as a ruler in every photo. It was one more task for the customer, and it failed on glass, mirrors and steep angles. Scale from known objects replaced it.

What broke

iPhones save photos as HEIC, which the pipeline could not read. The wizard now converts them in the browser, with a server fallback. And when the GPU service was down, large phone photos made the fallback model slow, so uploads are now resized before the call.

What it records

Each measurement keeps the model's raw answer, the prompt, the time it took, the warnings and the cache key, so the accuracy can be tuned.

Sources

• zaslony.com