How to compare two AI models on the same coding task

Run one task, with a clear finish line, on two models from the same starting commit, then compare three things: did the result pass, how long did it take, and what did it cost. Benchmarks tell you how models do on someone else's code; this tells you how they do on yours.

The key is keeping everything except the model the same: the code, the prompt, the tools and the check. The steps below set that up with git worktrees, so both runs can even happen side by side.

Steps

1. Pick a real, representative task. Choose something you actually need done and can check, such as a bug with a failing test or a small feature. A task with a test command is ideal, because "it passes" is not a matter of opinion.

1. Write the task once and save it. Use the same text for both models, word for word.

Goal: <outcome>. Where: <folders>. Limits: <rules>.
Done when: <exact command> passes.

See how to write a task for a coding agent for a full template.

1. Freeze the starting point. Commit or stash your work and note the commit.

git status
git rev-parse --short HEAD

1. Create one worktree per model, from that same commit.

git worktree add ../myapp-model-a -b compare/model-a
git worktree add ../myapp-model-b -b compare/model-b

Install dependencies in each one. The commands are the same on Mac, Linux and Windows.

1. Run the task with model A in its folder, and model B in the other. Use the same tool and the same settings for both. Note the start and end time, or time the check:

time npm test

On Windows PowerShell, use Measure-Command { npm test }.

1. Check each result yourself. In each worktree, run the done command and read the diff:

git diff --stat
git diff

1. Score them side by side. A simple table is enough:

Read cost from your provider's usage page, or from your tool's usage report.

1. Repeat on two or three tasks before you decide. Models vary from run to run. One task is a hint, not a verdict. Hard bugs, simple edits and long refactors can each favour a different model.

1. Clean up.

git worktree remove ../myapp-model-a
git worktree remove ../myapp-model-b

What usually matters

• Correctness first. A slow, correct answer beats a fast one you have to redo.
• The size of the diff. Smaller, focused changes are easier to review.
• Cost per finished task, not cost per token. A model that is dearer per token but needs fewer attempts can cost less overall.

Doing it with Mel

Mel works with Claude, GPT, Gemini, DeepSeek and Qwen, with 10 providers in the app, so you can compare models from the same window.

Pin a model per conversation. Click the model chip in the agent input, or type /model. Models are grouped by provider, and hovering one shows its Model Specs card with intelligence, speed and cost bars and its context window. Select one to pin it for that conversation.

Run them side by side. Track your repo from the + menu ("Track this repo"), then use + → Worktree to create a checkout on a new branch for each model. Open a conversation in each, pin a different model in each, and paste the same task. You can also split one tab with ctrl+D and run a conversation per pane.

Read the usage report. Every agent response ends with a line you can expand into a usage summary: the model used, tokens in and out, tool calls, time to first token and total time. That gives you most of the scoring table above.

Or let Auto choose. Leave the chip on Auto, and Mel picks a model for each turn, with a "why this model" line in the usage summary. You can bias Auto toward Cost-efficient, Balanced or Genius. Comparing Auto against a pinned model on the same task is a useful test too.

Then open the code review panel (ctrl+shift+=, or cmd+shift+= on macOS) in each worktree to read the two diffs.

FAQ

How do I compare AI models for coding fairly? Same task text, same starting commit, same tool and the same check. Change only the model, and repeat on a few tasks.

Should I trust public benchmarks? They are a useful first filter. Your own code, tests and conventions are what decide which model works best for you.

How do I measure cost? Use your provider's usage page, or your tool's usage report. Compare cost per finished task, not per token.

Why did the same model give different results twice? Model outputs vary between runs. That is why one comparison is not enough; try two or three tasks.

Can I use different models for different tasks? Yes. Many people pin a strong model for hard bugs and use a faster one for simple edits. Mel's Auto mode picks a model per task for you.

• How to write a task for a coding agent
• How to use your own API key with an AI coding tool
• How to run several AI agents at once
• What is a multi-model coding agent?