Research · 26 September 2026

Asking instead of guessing

People rarely write down the rules they work by. A support lead knows which tickets are urgent; a sales assistant knows when a discount applies. We asked how far an ordinary 7B open model gets when it has to learn such a rule from twelve examples, and what closes the gap. Three changes took it from under half right to 87–95% on requests it had never seen.

Three rules nobody wrote down

We chose three skills where the answer follows a rule the examples only hint at. Ticket codes combine an area with a severity: a login problem that loses data is AUTH-S1, a misaligned search button is SRCH-S3. Order prices apply a 10% discount from ten items and a further 5% for members. Shipping costs come from a weight band multiplied by a regional factor. The test requests use wording, products and places that never appear in the examples.

Results

Share of test requests answered exactly right. Each column adds to the one before it.

What made the difference

• One question, asked when stuck. When the model keeps getting a skill wrong, it asks the person how they decide. A single sentence in reply moved ticket codes from 25% to 77%. Given the rule alone, prices got worse: the model applied every clause of the rule to every order.
• Checking each condition against the request. Before using the rule, the model states for each condition whether this request meets it, quoting the request's own words. Things everyone knows about a named place or item count as stated; anything else the request does not say is not met. This stopped discounts on two lamps and member prices for customers who never mentioned a card.
• A calculator and a vote. The model writes its arithmetic as a sum, which is worked out exactly and written in the format of the examples. Where two readings of a request disagree, two more are taken and the majority wins. Slips rarely repeat.

What did not work

A larger 14B model did not help shipping. Having the model write a Python program for the rule failed on new places, because its programs memorised the example prices instead of the bands. Rules that merely listed the examples' outputs looked accurate on the examples and failed on everything else; we now detect and reject them. Loosening the condition check so that anything "implied" counts raised shipping but cost ticket codes twenty points.

Limitations

The person's reply to a question comes from a written rulebook, so the answers are cleaner than a real colleague's. The skills and test requests are generated, and twenty requests per skill leave room for noise of several points. We are moving to larger and more varied test sets.

All research

Research · 28 September 2026

Eight tasks, nothing forgotten

A model that keeps learning after it is deployed has one job above all: learn the new thing without losing the old. TRACE is the standard public test of this for language models. It teaches a model eight very different tasks in a row, then tests it on all of them. Most methods lose ground on earlier tasks as later ones arrive. Fintan, on an ordinary 7B open model, scored 67.3 overall and forgot nothing.

The test

The eight tasks share almost nothing: the stance of Chinese social media posts, the tone of central bank statements, summarising city council meetings, completing Python code, science questions, two sets of maths word problems, and simplifying German news. Each is learned once, in order, and the model never sees it again before the final test. Two numbers matter. The overall score is the average across all eight tasks at the end. Forgetting measures how much each task dropped between the day it was learned and the end; zero means nothing was lost.

Results

Scored with TRACE's own evaluation: accuracy for the classification, science and maths tasks, ROUGE-L for meetings, edit similarity for code, SARI for German simplification.

Against published results

The strongest recent published method on the same model, trained on ten times more data per task, reports 65.45 overall with some forgetting (−2.29). In the original TRACE paper, fine-tuning a 7B chat model on each task in turn scores 48.7 and loses 8.3 points on earlier tasks; replaying old data scores 55.5. Those used a different 7B model, so they show the scale of the problem rather than a like-for-like comparison.

Why nothing is lost

Each task is learned into its own part of a knowledge layer inside the model, and the model's original weights are never changed. Learning a new task cannot overwrite an old one. For each request, Fintan decides which learned task it belongs to; it chose correctly on 99–100% of test questions without being told.

Limitations

This is a preliminary run: 500 training examples per task and 100 test questions per task, where the full benchmark uses 5,000 and larger test sets. At this size a single task can move by several points between runs. The full-size run, with same-model baselines trained under identical conditions, comes next, and we will publish its predictions and scoring code with it.

All research

Research · 27 September 2026

Learning into its own weights

A notebook lets a model do a new skill on day one. The goal is for the model to keep the skill in its own weights, so it does the work without the notebook, faster and more cheaply. We measured what decides whether that happens. Four things matter: the size of the model, learning the working and not only the answer, practice that covers every case, and a learning rate matched to the kind of skill.

Size matters most

Same practice material for every model, one night of learning.

A small model can copy formats but struggles to carry the reasoning behind a price or a shift length. In practice the learning model is the one an organisation already runs, which is usually 7B or larger, so we now use the same model for both roles: it reads the notebook by day and learns it into the model at night.

Learning the working, not only the answer

Given two hundred real past records per skill and nothing else, the model learned formats well (routing 95%, order records 100%, emails 90%) but not calculations (prices 10%, shipping 15%, timesheets 20%). The final number alone does not show how it was reached. When the model instead learns the notebook's step-by-step working, checked against the real answer, prices rose to 55% from records alone, and to 95% when the person's rule was also available.

Practice has to cover every case

The inbox-sorting skill stopped at 65% for every model size. Its twelve examples contained no email from the manager and one from finance, so practice written from those examples never contained them either. A model cannot learn a category it has never practised. Writing practice for every category the person uses raised the skill from 62% to 78% in one run.

A learning rate for each kind of skill

With a gentle rate, ticket codes reached 95% and inbox labels 82%, but prices fell to 10%. With a stronger rate, prices held at 75% and ticket codes fell to 65%. Short category answers need a light touch; long pieces of working need a firmer one. The system now chooses the rate from how a skill is learned.

Where it stands

On these six hard skills the model's own weights reach 55–66%, while the notebook reaches about 76%. On simpler skills such as order records and date formats, the weights already match the notebook. Closing the remaining gap on reasoning-heavy work is the centre of our research.

All research