02 / AI SYSTEMS · MULTI-OBJECTIVE SEARCH

Foundry.

Compare AI routing options against a quality target, latency limit, and budget.

81 strategies computedHYPOTHETICAL EVALUATIONS

Within all constraints Tradeoff outside constraints Your cost / quality limitsEach point is a complete routing policy

Feasible strategies15Of 81 combinations

Best feasible pass rate92.4%Weighted by workload mix

Best policy cost$2.90Per 1,000 modeled tasks

Expected mean latency2393 msWeighted mean, not tail latency

These profiles are hypothetical, with invented example measurements. Replace the table with your evaluations to make a real routing decision.

Inspect a strategy

Issue triage Local small

Code changes Deep reasoning

Design review Deep reasoning

Private context Local small

This policy meets your current limits. Nondominated means no other eligible policy improves one of the three objectives without worsening another.

The evidence behind the points

No named model gets an invented benchmark here. The starter table uses fictional profiles. Quality is an observed task pass rate when you supply your own rows.

Task / profile	Pass	Mean	$/1k	n
triage / local	91%	180 ms	0	100
triage / balanced	95%	480 ms	1.2	100
triage / deep	97%	2100 ms	5.8	100
code / local	65%	420 ms	0	100
code / balanced	87%	1100 ms	1.2	100
code / deep	96%	4200 ms	5.8	100
review / local	55%	380 ms	0	100
review / balanced	80%	1400 ms	1.2	100
review / deep	94%	5200 ms	5.8	100
secrets / local	84%	210 ms	0	100
secrets / balanced	92%	600 ms	1.2	100
secrets / deep	96%	2400 ms	5.8	100

Use your evaluation results

Paste the same schema with a row for every task/profile pair. Set source to user-supplied. Costs, latency and pass rates should come from comparable held-out evaluations.

{
  "name": "Illustrative engineering workload",
  "source": "hypothetical",
  "models": [
    {
      "id": "local",
      "name": "Local small",
      "local": true
    },
    {
      "id": "balanced",
      "name": "Balanced cloud",
      "local": false
    },
    {
      "id": "deep",
      "name": "Deep reasoning",
      "local": false
    }
  ],
  "tasks": [
    {
      "id": "triage",
      "name": "Issue triage",
      "share": 40,
      "private": false
    },
    {
      "id": "code",
      "name": "Code changes",
      "share": 30,
      "private": false
    },
    {
      "id": "review",
      "name": "Design review",
      "share": 20,
      "private": false
    },
    {
      "id": "secrets",
      "name": "Private context",
      "share": 10,
      "private": true
    }
  ],
  "rows": [
    {
      "task": "triage",
      "model": "local",
      "passRate": 0.91,
      "latencyMs": 180,
      "costPer1k": 0,
      "samples": 100
    },
    {
      "task": "triage",
      "model": "balanced",
      "passRate": 0.95,
      "latencyMs": 480,
      "costPer1k": 1.2,
      "samples": 100
    },
    {
      "task": "triage",
      "model": "deep",
      "passRate": 0.97,
      "latencyMs": 2100,
      "costPer1k": 5.8,
      "samples": 100
    },
    {
      "task": "code",
      "model": "local",
      "passRate": 0.65,
      "latencyMs": 420,
      "costPer1k": 0,
      "samples": 100
    },
    {
      "task": "code",
      "model": "balanced",
      "passRate": 0.87,
      "latencyMs": 1100,
      "costPer1k": 1.2,
      "samples": 100
    },
    {
      "task": "code",
      "model": "deep",
      "passRate": 0.96,
      "latencyMs": 4200,
      "costPer1k": 5.8,
      "samples": 100
    },
    {
      "task": "review",
      "model": "local",
      "passRate": 0.55,
      "latencyMs": 380,
      "costPer1k": 0,
      "samples": 100
    },
    {
      "task": "review",
      "model": "balanced",
      "passRate": 0.8,
      "latencyMs": 1400,
      "costPer1k": 1.2,
      "samples": 100
    },
    {
      "task": "review",
      "model": "deep",
      "passRate": 0.94,
      "latencyMs": 5200,
      "costPer1k": 5.8,
      "samples": 100
    },
    {
      "task": "secrets",
      "model": "local",
      "passRate": 0.84,
      "latencyMs": 210,
      "costPer1k": 0,
      "samples": 100
    },
    {
      "task": "secrets",
      "model": "balanced",
      "passRate": 0.92,
      "latencyMs": 600,
      "costPer1k": 1.2,
      "samples": 100
    },
    {
      "task": "secrets",
      "model": "deep",
      "passRate": 0.96,
      "latencyMs": 2400,
      "costPer1k": 5.8,
      "samples": 100
    }
  ]
}

Smarter allocation, with visible assumptions

The search enumerates all category-to-profile assignments, then compares cost, expected pass rate, and mean latency. It does not train a router, call paid models, predict tail latency, or establish statistical confidence. A zero API price for a local profile excludes hardware and electricity. Routing research such as RouteLLM motivates the tradeoff; the results shown here come only from this table. Re-evaluate on fresh tasks before using a policy in production.

The person behind the project

A note from Luis.

My coding-agent evaluation work makes me interested in what an AI system can demonstrate, including where it fails. This notebook exposes the quality, cost, and latency assumptions. Its starting model profiles are hypothetical, not a benchmark I have measured.

Who it helps

Engineers comparing the tradeoffs of AI routing policies.

Try this

Change the quality target or budget and inspect why a routing option becomes infeasible.

Inspect a reproducible system handoff