The most expensive bug in a weather backtest
Weather markets are unusually pleasant to backtest. Settlement is objective, the history is public, and the data is free. That pleasantness is exactly the problem. Everything lines up so neatly that it is easy to build a test which quietly hands your past self a forecast that did not exist yet. We walked into three separate versions of this over two days. Each one looked like a discovery. Each one was a bug. Here is what they were and what they cost.
Trap one: the forecast you could not have had
Weather APIs will happily serve you historical forecasts, and the field names are seductive. Ask for the value predicted one day ahead and you get a clean column of numbers indexed by the day being forecast. Join it to market prices, sweep entry times, and you get a curve.
Our curve had a beautiful peak. Entering at a particular hour returned twenty three percent with a z-score of four, well clear of the surrounding hours. It looked like a real structural feature.
It was not, and the reason is mundane. The historical forecast column is anchored to the moment being forecast, not to the moment of publication. A value labelled as a one-day-ahead prediction for Tuesday noon was issued at Monday noon. If your simulated trade happens Monday morning, you have used a forecast that will not exist for several hours.
The fix is to model publication explicitly. Weather centres run on a schedule: a model initialises at a fixed hour, then takes roughly six hours of compute before anyone can read it. Once we required that the run used at any simulated entry had actually been published by then, the twenty three percent peak fell to under eight percent and the z-score dropped from four to under one. The entire feature was an artifact of reading tomorrow's newspaper.
Trap two: the calendar that is not your calendar
These markets settle on the local calendar day at the settlement station. Your forecast data arrives in UTC. Those are not the same window, and the gap is a full day at the extremes.
We took daily maxima by slicing forecast data on UTC dates. For a European city the error is invisible, because the afternoon peak sits inside the UTC day either way. For a city twelve hours offset it is catastrophic: the local day begins in the previous UTC day and the window you sliced covers a different afternoon entirely.
Measured against station observations, the disagreement between a UTC-day maximum and a local-day maximum ranged from half a percent of days in one city to nearly a third of days in another. The worst offenders were exactly the ones furthest from Greenwich.
What makes this trap nasty is its direction. It degrades your forecast rather than inflating it, so the strategy still appears to work and you never go looking. We only found it by auditing the pipeline after the fact. Refetching everything on local-day boundaries changed about three percent of all values, concentrated almost entirely in the two most offset cities.
Trap three: choosing the winners after seeing them
The third trap is not a data bug, it is a discipline failure, and it is the one that survives peer review most often.
We ran a strategy across a list of cities. Some cities performed well and some badly. Dropping the bad ones lifted the aggregate return from about nineteen percent to twenty nine percent, with a much better z-score. Every instinct says to drop them.
You cannot, because you chose them using the same data you are now reporting. With per-city samples in the teens and twenties, several cities will look terrible by chance alone, and removing them is a guaranteed way to manufacture a result.
The only defensible exclusions are the ones you can date to before the test. Two of our cities had been flagged as negative across several earlier analyses, on different data cuts and different time axes. Dropping those is a prior decision being applied, not a selection being made. Dropping the others was not, so we did not, and we reported both figures side by side so the difference is visible.
The cost that is not lookahead at all
One more thing that is not technically lookahead but behaves like it. Price history gives you a series of traded prices. Backtests read one and assume you transacted there. You cannot. You transact at the offer, and the offer sits above the last trade by whatever the spread happens to be.
We measured live spreads on the leading bucket over a full market lifecycle. In the hours right after a market opens, the median spread across cities was in the double digits of cents, with individual cities showing thirty cents or more and only a handful of shares resting on either side. Within a few hours the same markets tightened to one or two cents.
The variation between cities at the same instant was larger than the variation over time. At one moment a European capital sat at four cents while another European city on the same ladder sat at thirty five. Any backtest that applies one uniform cost assumption across cities and hours is wrong in both directions, and it is most wrong exactly where it looks most profitable, because thin early markets show the largest apparent edges.
How to tell if your curve is real
Three checks caught all of ours.
Smooth before you believe. A single hour that beats its neighbours by five points is almost always noise. We were fooled twice by isolated peaks that vanished under a three-point average. The neighbours of a real feature are also elevated, because the underlying mechanism does not switch on for exactly sixty minutes.
Name the mechanism, then check its clock. If the peak is real there is a reason for it, and reasons have timestamps. Ours turned out to be the model publication cycle, which is a fixed offset from the market close. Once you can state the mechanism you can verify whether your simulated trade respected it.
Ask what your past self could actually see. For every input, write down the moment it became available and confirm your entry happens after it. This is tedious and it is the only reliable defence.
Where it ended up
After all three fixes the strategy was still positive, at roughly a third of the return the first version advertised, on a sample small enough that the honest confidence interval is wide. That is a much less exciting sentence than the one we could have written on day one, and it is the only one worth acting on.
The general lesson is that a backtest is not a measurement, it is a simulation of a decision. Every number in it needs a defensible answer to one question: could the person making that decision have known this? Weather markets make that question easy to forget, because the data is so clean and so free.
Frequently asked
Is a z-score of four not conclusive? Only conditional on the test being correct. A high z-score on a biased test measures the bias precisely. Ours dropped below one once the bias was removed.
How much history do you need? More than is available, usually. Single-run forecast archives are retained for a limited window, in our case a little under four months, which caps the sample regardless of how far the market history goes back.
Does forward testing solve this? It removes lookahead by construction, which is why we run it. It replaces the problem with a slower one, since accumulating a few hundred honest trades takes months.
Where to watch this live
Forward testing is the only clean answer, which means watching books in real time rather than reconstructing them. SmartX carries the Polymarket temperature ladders with live depth, which is the part price history cannot give you.