In progress

Coda · LLM evals & red team

A music-composition coach chatbot tested with Promptfoo: Claude Haiku 5.5 (small) and Claude Sonnet 5.5 (medium), with a Google Gemini model coming as the third. Quality, guardrails, LLM-as-a-judge rubrics and red teaming. Source on GitHub.

Evals committed baseline full report →

Run on 2026-11-15 · from reports/baseline/eval.json · 58 results · judge: LLM-as-a-judge (see promptfooconfig.yaml). Pass rate covers graded results; errored calls are counted separately.

Model Pass rate Errors Median latency Tokens (in / out) Est. model cost Judge tokens Est. judge cost
claude-haiku-5.5 (small) 93% (27/29) 0 3.5 s 44,082 / 12,116 $0.0105 24,879 $0.0855
claude-sonnet-5.5 (medium) 93% (27/29) 0 5.9 s 44,082 / 14,984 $0.2380 27,628 $0.0895

Pass rate by metric

Model accuracy concision copyright format helpfulness honesty language safety scope security
claude-haiku-5.5 (small) 86% 100% 100% 100% 100% 100% 100% 100% 100% 100%
claude-sonnet-5.5 (medium) 93% 100% 100% 100% 100% 100% 100% 100% 100% 97%

Failures

Red team committed baseline full report →

Run on 2026-11-15 · from reports/baseline/redteam-quick.json · 276 results · judge: LLM-as-a-judge (see promptfooconfig.yaml). Pass rate covers graded results; errored calls are counted separately.

Model Pass rate Errors Median latency Tokens (in / out) Est. model cost Judge tokens Est. judge cost
claude-haiku-5.5 (small) 100% (138/138) 0 1.0 s 229,224 / 20,680 $0.0333 85,349 $0.2074
claude-sonnet-5.5 (medium) 100% (138/138) 0 1.3 s 229,224 / 43,490 $0.8933 158,127 $0.3841

Red team: outcome by model

Model Attacks Answered safely by the model Blocked by provider filter (empty reply) Failed Errors
claude-haiku-5.5 (small) 138 36 102 0 0
claude-sonnet-5.5 (medium) 138 64 74 0 0

Red team: outcome by plugin

Plugin Attacks Answered safely by the model Blocked by provider filter (empty reply) Failed Errors
contracts 16 6 10 0 0
excessive-agency 16 6 10 0 0
hallucination 16 6 10 0 0
harmful:copyright-violations 16 6 10 0 0
harmful:hate 16 5 11 0 0
harmful:intellectual-property 16 5 11 0 0
harmful:self-harm 16 3 13 0 0
harmful:specialized-advice 16 6 10 0 0
harmful:violent-crime 16 2 14 0 0
hijacking 16 7 9 0 0
imitation 16 7 9 0 0
off-topic 16 7 9 0 0
overreliance 16 6 10 0 0
pii:direct 16 6 10 0 0
policy 16 6 10 0 0
politics 16 6 10 0 0
prompt-extraction 16 6 10 0 0
system-prompt-override 4 4 0 0 0

Red team: outcome by strategy

Strategy Attacks Answered safely by the model Blocked by provider filter (empty reply) Failed Errors
base64 68 25 43 0 0
basic 72 69 3 0 0
leetspeak 68 6 62 0 0
rot13 68 0 68 0 0

Failures

None.

Built by Julian Betancourt Cuervo.