Optimization
What a warm start actually guarantees, and why it broke my benchmark
A note on the difference between an engineering guarantee and an empirical result, using a mistake I shipped.
The setup
I built a field-service dispatch optimizer: a backlog of jobs, a crew of technicians with different skills, service-level deadlines, travel times, and an overtime budget. The question is which technician goes to which job, in what order. It is a vehicle-routing problem with time windows sitting on top of an assignment problem, and I modeled it in OR-Tools CP-SAT.
To show the model was worth anything, I needed something to compare it to, so I implemented a manual dispatch baseline: priority-first assignment, roughly what a dispatcher does with a whiteboard. Then I did something that felt obviously correct at the time. I fed the baseline solution into the solver as a starting point, using CP-SAT's hint mechanism, so the solver begins from a known-feasible plan rather than from nothing.
This is a warm start, and for a production tool it is the right call. It has a property that matters enormously to an operator: the optimizer can never hand back a plan worse than what the dispatcher already had. The baseline is a feasible incumbent. The solver will only replace it with something that scores better on the objective. Worst case, you get your own plan back.
The problem
I then reported the comparison as a result. On the canonical scenario, the optimized plan produced 18 fewer service-level breaches and removed 10.5 overtime hours against the baseline. I put that number on my homepage.
It took someone asking one question to show me what was wrong with it: what would it have looked like if the optimizer had lost?
It could not. That was the whole point of the warm start. I had constructed a comparison whose direction was fixed before the solver ran. The optimizer beating the baseline was not evidence about the optimizer; it was a restatement of how I had wired the solver. The only empirical content in "18 fewer breaches" is the magnitude, and the magnitude is a property of the scenario I generated, not of the method.
This is the difference between a guarantee and a measurement. A guarantee tells you what cannot happen. A measurement tells you what did happen when it could have gone either way. I had published a guarantee wearing a measurement's clothes.
The second problem, which is worse
Once I started looking at the baseline as an experimental control rather than as scaffolding, it got harder to defend. On the canonical scenario my priority-first baseline breached 32 of the 77 jobs it scheduled. Measured properly across 200 randomized scenarios, it misses a median of 45.6 percent of the deadlines it takes on.
No dispatcher runs at 45 percent. An operation like that does not keep its customers long enough to buy optimization software. I had not built a baseline; I had built a strawman, and then beaten it by a margin that says more about the strawman than about CP-SAT.
A fair baseline for this problem is a greedy nearest-qualified assignment with a 2-opt local-search pass over each technician's route, plus a cheapest-insertion pass to backfill the time the reordering frees. It is what a competent engineer writes in an afternoon without a solver, and it is a genuinely hard thing to beat. Mine came to about 300 lines once feasibility, travel re-simulation, and the backfill were handled properly, which is itself worth noting: "a simple baseline" is rarely as simple as it sounds, and that is part of why people skip building one.
What the honest version looks like
Keep the warm start. It is correct for the product, and removing it to make the benchmark cleaner would be optimizing the demo at the expense of the tool. Instead, separate the two claims that were tangled together:
The engineering claim: the optimizer is warm-started from the incumbent, so it is monotone by construction and an operator can adopt it without downside risk. State this as a design property, which is what it is, and stop presenting it as an outcome.
The empirical claim: run a few hundred randomized scenarios, varying crew size, skill mix, emergency rate, and travel scale. Report the service-level delta as a distribution with a median and a 5th-to-95th percentile band, against the 2-opt greedy baseline rather than the strawman.
So I ran it: 200 randomized scenarios, four arms, at the product's own eight-second budget. The greedy-plus-2-opt dispatcher takes a median of 12 breaches off the naive baseline, 5th to 95th percentile 3 to 22, better on 199 of 200 scenarios. That arm is fully deterministic and reproduces bit-identically on a second machine.
Then the part I did not expect. Against that stronger baseline, CP-SAT at eight seconds is not comfortably ahead. It is five breaches behind. I nearly published "constraint programming loses to a greedy heuristic here," which would have been a great headline and wrong. A follow-up sweep at 2, 8, 30, and 60 seconds moved the median difference from +9 and +5 breaches worse to 3 and 5 better. The model was fine. The shipped time budget was below the point where it pays for itself, and the strawman baseline had hidden that, because eight seconds is plenty when your opponent misses 45 percent of its deadlines.
One more thing fell out of it, and it is the reason I am not quoting per-scenario win rates. The solver runs eight search workers against a wall-clock limit, so random_seed=42 does not make it deterministic. Re-running the same scenarios flips the verdict on about a fifth of them. Its numbers are meaningful as a distribution and unreliable one at a time. I would rather say that than publish a win-loss tally that will not reproduce for anyone who checks.
That version is weaker as a headline and much stronger as evidence. Every figure in it is something a reader can argue with. "18 fewer breaches" was not, because there was nothing there to argue with.
The general version
The trap is not specific to CP-SAT or to dispatch. Any time you build both the baseline and the challenger, you control the outcome, and the temptation is to build a baseline that makes your work look good. It rarely feels like cheating in the moment. It feels like writing a simple reference implementation.
The test I use now, before reporting any comparison: describe, concretely, the world in which my method loses. If I cannot describe it, I have not run an experiment. If I can describe it but never checked, I have not run it yet.
The optimizer is still good work. The model handles asymmetric travel without modification because CP-SAT routes over directional arcs, the core is testable without a database or a web framework, and it reports its optimality gap so an operator can see how far the incumbent sits from the proven bound. None of that needed a rigged comparison propping it up.
The project
The dispatch optimizer is live, and the case study now carries this correction in its own limitations section rather than in a footnote.