Postmortem: the eval that took the chat down

On 12 October 2026, the chat on this site stopped answering questions about me from 17:45 onward. None of the Worker's metrics showed an error. The cause was my own eval, which used up the free Workers AI quota that the chat's search also depends on.

The charts with the day's data are on the incident dashboard. Times are in Brasília time (UTC-3).

Summary

• Impact: questions about me, the site or the chat came back with no text, with any model in the picker. General questions still worked.
• Cause: the buscar_site tool embeds the question with Workers AI. With the daily quota used up, that call failed with error 4006 and the whole tool failed.
• Why nobody saw it: /chat returns 200 before the search runs, and the tool failure wrote nothing to the log.
• Fix: #287, merged at 18:24. Without an embedding, the search falls back to the BM25 ranking and logs the fallback.

Timeline

• 11:45 and 12:45. Two runs of the chat eval with Llama 3.3 70B on Workers AI. Each run sends the same questions to every model in the picker.
• 17:30 to 17:40. Third run. Usage for the day goes past 10,000 neurons, the free daily quota.
• 17:45. First calls rejected with 4006, on both the embedding and the model. From here on the chat's search fails.
• 18:24. #287 is merged.
• 21:00. The quota resets (midnight UTC).

The day ended with 11,877 neurons used and 182 calls rejected by Workers AI.

What the metrics showed

In the Worker's metrics the whole day looks clean: 0 errors in 632 requests, all with status success. That is because /chat opens the stream and returns 200 right away. The search runs later, inside the stream, and its error becomes an event in the chat protocol. As far as the runtime knows, the request succeeded.

The only signal in the Worker was latency. The median dropped from 2.9 s to 0.5 s when the errors started. Responses got faster because the model never got search results and had nothing to write. In a chat, a sudden latency drop with no code change deserves as much attention as a spike.

Getting to the cause

With the data that existed, this is the path:

1. Reproduce the symptom. A question about me comes back empty. A simple math question works. So the problem is in the search, not in the model or the whole Worker.
2. Read the stream, not the status. The response stream has a tool error event with the Workers AI message: 4006: you have used up your daily free allocation of 10,000 neurons.
3. Confirm it in Workers AI Analytics. Calls with errorCode 4006 start at 17:45, on both the embedding and the model.
4. Find what used the quota. Adding up the day's neurons by model, almost all of it is Llama 3.3 70B, in three blocks that match the eval runs. The search's embeddings used very few neurons.

Step 2 is the slow one when there are no logs. With no line in the Worker saying the tool had failed, the only way to see the error was to open the stream of a response.

Root cause

Three things together:

• The eval and the production chat share the same account and free Workers AI quota.
• The search depended on the question's embedding and had no alternative when it failed. The BM25 ranking was already in memory, unused.
• The tool failure left no line in the log.

The fix and the reproduction

#287 makes retrieval continue with BM25 alone when the embedding fails, and writes embed_error_bm25_fallback to the log. To check it, I ran the Worker bundle from before and after the fix with the embedding returning the quota error. That is 8 questions from the chat eval that depend on the search, scored by the same checks as the benchmark:

Before the fix, every question fails and none leaves a log line. With the fix, 8 of 8 pass and every fallback shows up in the log. In the normal run, one question failed because of model variance, unrelated to the incident. The reproduction also shows the latency drop: a 1.6 s median without the search, against 5.4 s with it.

Still open

The fix handles the symptom. These are still open:

• Separate the eval from production. Run the eval under another limit or account, so it cannot use up the quota the chat depends on.
• A usage alert. Warn when the day's neurons pass 80% of the quota. By 13:00, usage was already past 9,500.
• Persisted logs. Workers Logs are not enabled on this Worker. On the day, there were only aggregate metrics.
• Count empty answers. Track answers with no text as a metric. That count would have shown the incident as it happened.
• The Workers AI model in the picker is still unavailable when the quota runs out. Only the paid plan fixes that.

See the data

• Incident dashboard
• The fix (#287)
• Reproduction script

Back to home