Picking an image model family without trusting a leaderboard

Updated 2026-10-02

Image model rankings go stale in weeks, and a leaderboard average says little about your use case: product shots, poster text, character consistency and photo edits stress different things. This page gives a decision framework based on what you can verify, then a method for settling the question with your own prompts through one endpoint.

The families in the catalogue

The catalogue spans several model creators. The model families most people shortlist are:

• GPT Image (OpenAI): gpt-image-1, gpt-image-1.5, gpt-image-2 and related variants.
• FLUX (Black Forest Labs): FLUX.2 variants such as dev, pro, max and flex, Kontext editing models, and flux-3-image.
• Seedream (ByteDance): Seedream 4.0, 4.5 and 5.0 variants, including Flash, Lite and Pro.
• Nano Banana (Google): Nano Banana 2 and Pro, along with Gemini image models that work through the same endpoint.
• Qwen-Image (Alibaba), Ideogram, Grok Imagine, Recraft and others.

Browse the models index to see exactly what is live; ids change as models are added.

Decide by constraint first, quality second

Most selection problems are settled by hard constraints before any visual judgement:

After constraints, the remaining differences are largely qualitative: aesthetic defaults, how literally the model follows long prompts, how well it keeps a subject consistent across edits. I will not rank these here, because any ranking not backed by your prompts is a guess.

A tiering pattern that holds up

1. Draft tier: a fast, low-cost model for exploration, thumbnails and internal review.
2. Final tier: a premium model for assets that ship.
3. Edit tier: an edit-capable model for revisions of approved outputs.

The same endpoint serves all three, so the tiers are a config map from role to model id, not three integrations.

Per image, before VideoRouter's 2% platform fee. For tiered models each row compares the resolution tier with the widest host-to-host gap. Built 2026-10-02 from the live catalog.

What to write down about each model

As you test, keep a short record per candidate: the exact id, whether it accepts references and how many, whether it honours size controls, what the cost basis is, and any prompt style it prefers. That table becomes your routing config and saves the next engineer repeating the experiment. Note failures as well as successes: a model that returns a refusal or an error on a class of prompts you need is ruled out as quickly as one with poor output.

How to A/B models on your own prompts

Because every model is called the same way, a comparison harness is a loop over model ids. Pull the exact ids from each model's page.

import requests, base64, pathlib

BASE = "https://videorouter.sh/api/v1"
H = {"Authorization": "Bearer llmr_sk_live_..."}
MODELS = ["gpt-image-2", "flux-3-image", "qwen-image-3.0"]   # use ids from the model pages
PROMPTS = [l.strip() for l in open("prompts.txt") if l.strip()]

out = pathlib.Path("ab"); out.mkdir(exist_ok=True)
rows = []
for p_i, prompt in enumerate(PROMPTS):
    for model in MODELS:
        r = requests.post(BASE + "/images", headers=H, timeout=120,
                          json={"model": model, "prompt": prompt}).json()
        if "error" in r:
            rows.append((p_i, model, None, r["error"]["message"])); continue
        d = r["data"][0]
        img = base64.b64decode(d["b64_json"]) if d.get("b64_json") else requests.get(d["url"]).content
        f = out / f"{p_i}_{model.replace('/', '_')}.png"
        f.write_bytes(img)
        rows.append((p_i, model, r["usage"]["cost"], str(f)))

Then review the outputs blind. Shuffle the file names or put them in a spreadsheet without the model column, score each against a short rubric (prompt adherence, text legibility, subject fidelity, artifacts), and join the scores back to the model and usage.cost. The number to compare is cost per accepted image: price divided by the fraction of outputs you would actually ship. A cheaper model with a lower acceptance rate is often not cheaper.

Practical tips for a fair test

• Use 20 to 50 real prompts from your product, including the hard ones.
• Generate more than one sample per prompt for the models you shortlist; single samples overstate differences.
• Keep size and aspect ratio equal across models where they are honoured, and note models that ignore them.
• Test the editing path separately with your own images; see text-to-image vs image-to-image.
• Re-run the test when a new model appears. It costs one list entry.

Hosts also matter: the same model can be served by more than one host at different prices, so check the price spread before pinning one. Cost levers are covered in image generation cost control. Create a key and run the harness above; the quickstart has the base request.

Frequently asked questions

Keep reading

• Choosing an Image Generation API — Price, Quality and Control
• Text-to-Image vs Image-to-Image API: Modes, Fields, Pitfalls
• Image Generation API Cost Control: Billing, Drafts, Caching
• GPT Image API Parameters: Sizes, Quality, Edits and Output