Do model decisions survive trading costs?

What happened

In the saved replay, none of the six models beat the capped SPY control (+$3.25). Every model ended below its $100 start. Phi-4 Mini (−$11.86) and Gemini (−$14.54) lost less than random (−$17.86), but Gemini failed on 152 of 191 sessions (79.6%). Its result includes those missed decisions.

What the saved figures show

Costs can turn gains into losses. Mistral recorded −$56.62 net P/L and $67.53 in costs. Adding those costs back gives +$10.91. Random similarly moves to +$13.51 before costs. This is an accounting breakdown of the saved trades; a fresh simulation without costs could produce different trades and position sizes.

Final balances hide large drawdowns. Phi-4 Mini finished down 11.86%, while its daily account value fell 49.40% from an earlier peak. The chart records closed trades, so a flat line can still conceal changes in open positions.

How to read these results

These are outcomes from one shared market period. They show how the saved decisions performed under the recorded execution rules. They do not establish a repeatable advantage or predict future returns.

Gemini’s errors matter. Its archived cache contains 152 quota-error sessions as well as valid decisions. Existing positions can remain open during a failed session; an error does not put the whole account into cash. Less opportunity to trade can reduce losses and costs; its ranking should be read alongside the failed-session count. A failed response is different from an intentional decision to hold cash.

The inputs were not identical. Local models received up to five rows per market group; Gemini’s older cache used eight. The local-model context files report full coverage of their configured views, but the groups shown to each model were not identical. A comparison with common inputs and consistent error handling is part of the next-run plan.

Inactivity has different causes. Gemma recorded 68 sessions with no orders and no error. Qwen still emitted orders on all 191 sessions; its flat tail followed heavy account losses, rather than a decision to stop trading. The $5 minimum position and 34% cap can leave a depleted account unable to open new positions.

How the test worked

The shared period is 2016-03-31 through 2016-12-29: 191 trading sessions and 1,102 closed trades. Each method starts with $100 and uses saved decisions with cached, split-adjusted US equity prices from the consolidated SIP feed. The date range is the intersection of all ten saved runs, selected for coverage rather than return.

The models: Qwen 2.5 7B, Llama 3.1 8B, Gemma 2 9B, Mistral 7B, Phi-4 Mini, and Gemini 2.5 Flash.

The controls: random selection, the original momentum strategy, mean reversion, and buy & hold SPY. Together they compare model decisions with randomized, rule-based, and passive approaches.

What the models saw and returned

The first stage generates a closing decision for each trading session and saves it as JSON, after a 60-session indicator warmup. The second stage replays the saved decisions without calling a model. Each model’s decision is independent of the replayed account: it does not receive current holdings, previous fills, or prior errors, and it has no callable quantitative toolkit.

The model sees samples drawn from the full-market universe, including names that later delisted. Nine groups organize the snapshot: movers up, movers down, most active, gappers, liquid core, oversold, overbought, near the 20-day high, and high volatility. Each group contains up to five rows for local models or eight in Gemini’s older cache; names can appear in more than one group.

Each row includes the nominal price, gap, daily change, 5- and 20-day returns, volatility, average dollar volume, relative volume, RSI, and distance from the 20-day high. A small SPY index summary supplies market context. These are precomputed indicators, not tools the model can invoke.

The requested JSON contains a short thesis and up to three long-only orders, with an equity weight and optional stop, target, and holding horizon. Structured output constrains the response format. Unknown tickers are dropped, and orders for names already held are skipped. Models do not issue explicit exits in these caches; the engine closes positions at stops, targets, holding limits, or the window’s end.

An empty orders list means no requested trades. Quota failures, timeouts, and missing order fields are logged separately as errors. Neither supplies new entry orders, but existing holdings still follow the engine’s exit rules.

How the controls chose trades

All controls use the same replay settings, but their selection rules, requested weights, and exits differ. Here, ADV means 20-session average dollar volume, ATR measures recent price range, and R is the distance from entry to the stop.

• Random: up to two names per session from stocks priced at least $1 with ADV of at least $1 million. Sampling weights are the square root of ADV. Each request uses a 30% weight, a stop at 2×ATR, a target at 2R, and a five-session horizon.
• Legacy momentum: up to one name with a 4–12% daily change, relative volume of at least 1.5, ADV of at least $500,000, and price of at least $1. Requests use a 34% weight, a 1.5×ATR stop, a 3R target, and a one-session horizon. The change screen uses the full day’s move, rather than the overnight gap alone.
• Mean reversion: up to two names with 14-day RSI of 25 or less, ADV of at least $5 million, and price of at least $2. Requests use a 34% weight, a 2.5×ATR stop, a 1.5R target, and a five-session horizon.
• Buy & hold SPY: requests an index position with no stop or target. This shared-period replay applies its 34% position cap and holds until the end.

Execution settings and measurements

The recorded replay uses fractional shares, a 34% position cap, a 100% gross-exposure cap, at most five positions, no shorting, a $5 minimum position, and immediate settlement. The liquidity cap is 2% of that day’s dollar volume. The final session takes no new entries, and remaining positions close at the end of the window.

Market fills pay a half-spread with a one-cent tick floor, illiquidity and volatility terms, a 5% cap, and square-root price impact. Resting limits pay no spread; stops pay double the half-spread. Sales also incur modeled SEC and FINRA fees. If a bar touches both stop and target, the stop takes precedence; a gap through the stop fills at the open. Limit entries require the day’s low to reach the limit, although all entries in these shared-period caches are market orders.

A 20% daily-loss setting exists in the policy, but the replay resets it before the next session’s entries. It did not enforce a brake on this run.

Entry orders come from the previous session’s closing decisions. The timing correction planned for the next run is described below.

The position cap applies to SPY too. Its return includes idle cash and excludes dividends; it is not a fully invested SPY total-return benchmark.

Net realized P/L sums closed-trade gains and losses after modeled costs. Final balance is the account value after all positions close. Profitable trades have positive net P/L. Maximum drawdown is the largest percentage fall from an earlier peak in daily account value, including open positions.

What I’d change next time

These changes are planned for a future run. The results above remain the recorded outcomes of the completed test.

1. Correct the replay timing. Size morning purchases using information available at the open. In a regression example, changing only a later closing price changed an earlier purchase from 3.40 to 4.56 shares. Check the order of exits, cash availability, and entries too. Fix the engine, add tests that prevent future information from changing earlier trades, and rerun the comparison. Returns and rankings may change.
2. Use a common market view. Log each model’s complete input, version, and context coverage. Keep the market snapshot and decision format consistent, and document model-specific limits.
3. Separate failures from no-trade decisions. Set consistent timeouts and retries, and report failed sessions beside returns.
4. Keep execution rules consistent. Freeze timing, sizing, liquidity limits, and cost assumptions. Include both the capped SPY control and a separately labeled fully invested benchmark.
5. Repeat the comparison. Choose development and test periods in advance, cover different market conditions, and repeat the random baseline with different seeds. Account for overlapping periods when estimating uncertainty.
6. Test costs, exposure, and risk. Compare predeclared cost levels and report trading volume, exposure, daily account value, and drawdown. Set the success criterion before scoring: repeatable gains over the controls after costs, with acceptable drawdown.

Historical background

An audit of the original intraday bot covered July 2023–June 2025 and 1,520 trades. It reported −$835.86 net P/L, +$10.62 before costs, and profit factors of 0.771 net and 1.003 gross. Reported slippage was $846.48. Performance before costs was near break-even in that sample, which motivated testing explicit controls in the Arena.

The July 25, 2026 control run below used an earlier engine. Later reviews found extra trades on each window’s final day, delisted positions closed at their entry prices, and split-adjusted prices used in screening and cost calculations. These are the original reported figures, retained as the history of the experiment.

This separate run uses overlapping 180-day windows. Its median balances should not be combined with the single shared-period comparison above.