AI & agents

Why the parse errors stopped and the mistakes did not

The mistake

You asked a model to classify support tickets and to answer in JSON. Most of the replies were valid JSON. Then one of them opened with “Sure, here is the JSON:”, and json_decode returned null. So you moved to a typed answer, where the model picks one of four labels and can write nothing else.

Your parse errors went to zero that afternoon. You read that as the problem being solved.

The machine

Simulator · typed answers route

reading them as Parse the prose3 of 12 handled

1Double charged on my card this monthbilling
2The CSV export comes out emptybug
3Could you add dark mode?feature
4I cannot log in since the password resetwaiting
5I was charged twice, please refund onewaiting
6Is your company hiring backend engineers?waiting
7Partnership enquiry from a reseller in Spainwaiting
8Login fails after I changed my emailwaiting
9Invoice 4471 does not match my planwaiting
10Two factor codes never arrivewaiting
11Change my billing address to Milanwaiting
12Webhook retries fire twicewaiting

0failed, and you saw it

0wrong, and nothing said so

3right

the reply you had to parse

{"intent": "feature"}

Both routes run on the tested reducer, over twelve replies written by hand. The prose route runs a real JSON.parse, so a parse failure here is a parse failure. No model was called.

Reading them as parse the prose, 3 of 12 handled. 0 failures you can see, 0 wrong answers nothing flagged, 3 right.

Drive it

Twelve replies have come back from the model. Three are already handled, and each of those parsed cleanly.

Press “Handle next output” until all twelve are done. Six of them produce no answer at all. Four replies are not valid JSON, and two carry a label that is not one of your four.
Now press the button again. The button has changed. It offers the same twelve replies as typed answers, so watch the two counters below the list while you press it.
Switch the route back to “Parse the prose” from the picker at the top. Read row 6 and row 7 in each route. Those two tickets are about recruiting and about a reseller partnership. No label on your list fits either one.

The mechanism

A route to a label gives you one of two things. Either it answers, or it fails and tells you what went wrong.

The prose route fails in two ways. The reply is not valid JSON, so the parse throws. Or the reply parses, and the value is not a label you recognise. Your own check is what rejects the second one. Both land in your error tracker, where you can see them.

A typed answer closes that off. You send the model a list of options. The answer that comes back can only hold one of them. There is no text to parse, so a malformed reply is impossible. The six failures in the simulator become zero, and that part is real.

Now look at what those six failures were. Four of them, rows 4, 5, 10 and 12, come back as correct answers once the route is typed. Three of those were malformed JSON: a preamble, a trailing comma, a reply that stopped halfway. The fourth answered “billing_issue”, which is not on your list. It is near enough to “billing” that the constrained answer landed there. That is a genuine improvement, and this page is not arguing with it.

Two of the six come back wrong. They are rows 6 and 7, and what they have in common is not the shape of the reply. Row 6 parsed, and its label was rejected. Row 7 never parsed at all. What those two share is the ticket. One asks whether you are hiring, and the other is a reseller asking about a partnership. No label on your list fits either one. The model said so in the only way it could. Row 6 answered “recruiting”. The reply that failed to parse on row 7 has “sales” sitting inside it. Those two failures were the only signal you had that your four labels do not cover everything that arrives.

The typed route cannot send that signal. It cannot return “recruiting”, because “recruiting” is not one of the values the answer can hold. So it returns “account” for the recruiting enquiry, and “feature” for the partnership. Both answers are well formed. Both are in your taxonomy. Both are wrong, and nothing in the response says so.

That is the move this page is about. The errors did not go away. Four were fixed. Two were converted from a failure you could see into an answer you cannot check.

The guarantee is worth reading closely. OpenAI’s structured outputs guide opens by saying you no longer need to worry about the model “hallucinating an invalid enum value”. That is true, and it is also the whole of the promise. A valid enum value and a correct one are two different properties. Only the first one is guaranteed.

The distribution says more than the label does. Look at row 1 and row 9 on the typed route. Row 1 puts 91% on one label. Row 9 puts 40% on two labels, then resolves the tie to one of them. Both come back as a single string, and reading that string cannot tell the two apart. The answer object carries the whole distribution. Almost no code reads it.

So the number to watch is not the count of parse errors. It is the count of answers you never verified. That count went up while every indicator you had went green.

In your code

A Choice question in laravel/ai takes the option names and a short description of each. The answer comes back as one of those names.

use Laravel\Ai\Classification;
use Laravel\Ai\Classification\Choice;

$answer = Classification::of($ticket->body)
    ->question('intent', new Choice('What does this ticket need?', [
        'billing' => 'Charges, invoices and refunds',
        'bug' => 'Something is broken',
        'feature' => 'A request for something new',
        'account' => 'Sign in, profile and access',
        'other' => 'None of these four',
    ]))
    ->classify()
    ->answer('intent');

$answer->choice;            // one of the five names, always
$answer->probabilities;     // ['billing' => 0.41, 'bug' => 0.39, ...]
$answer->confidence;        // null when the provider cannot measure it

if ($answer->choice === 'other') {
    $ticket->escalate();
}

The other option is the cheapest fix on this page. Without it, the model cannot say that all of your labels miss. It picks the closest one instead. With it, what you used to learn from a rejected label comes back as an answer you can route.

probabilities maps every option name to a float. probabilityOf() reads one of them. confidence is nullable, and the library’s own docblock says why: a provider that cannot measure the distribution’s certainty returns null rather than a number. Code that reads choice and nothing else throws both of those away.

Choice refuses to build with fewer than two options, and throws an InvalidArgumentException. Classification::fake() and assertClassified() cover all of this with no network call. What a fake cannot tell you is whether your five options describe the tickets you actually receive.

The fine print

The twelve replies are written by hand. Nothing here was measured, and none of it is a claim about how a real model behaves. They exist so that “the parse failed” and “the answer was wrong” each have an exact meaning the test suite can assert. The prose route runs a real JSON.parse over the reply, so a parse failure on this page is a parse failure and not a flag we set.

The two routes are held to one authored rule. Where the prose parses to a label on the list, the distribution puts that same label on top. So the type never improves an answer that already worked, and every difference between the two columns comes from a reply that produced nothing. A real model is not that tidy.

How a constrained answer is produced is left out. The provider restricts what the model can emit, and those mechanics sit outside your application.

Where to set a threshold is left out on purpose. That is the confidence threshold page, and the two questions are separate. This page is about the shape of the answer. That one is about what the number beside it is worth.

Also left out: more than one label per ticket, nested schemas, and a label set that changes while the system is running. Each of those makes the picture harder without changing the finding.

answer() returns Answer, which is an abstract base class. The choice, probabilities and confidence properties are declared on ChoiceAnswer. That is the subclass a Choice question produces. The snippet runs, and PHPStan or Psalm will still flag the property access on the base type. Narrow the type with an instanceof check first.

The code is read against laravel/ai v1.2.0. The classification API first shipped in v1.0.0 on 23/09/2026, so a v0.x install does not have it.