Diplomacy, ‘fun’ for both Man and Machine
Some of you may have heard of “Diplomacy: the game that ruins friendships”, the deterministic 7 player board game from the 1950s. Some of you may even still have friends that play it with you. Given the wonderful reputation that it holds, I wondered: how would today’s LLM models play?
Would they be honest with each other? Maintain alliances? Knowingly lie about their intentions? Backstab a long-term ally, like a true Diplomacy player?
Or would they be too agreeable, and trust everything their compatriots said? Would they listen to cries for mercy, and leave opposing players alive even when they could be removed?
I decided to find out, and built a tournament.
My initial attempt isn’t published, and ended up testing long-context behavior and compaction more than anything else. Instructions were provided at the very beginning, and all press, orders, and resolutions were appended in one conversation from 1901 until 1920. This led to both low performance and high costs, with Luna losing to a gunboat bot in one game.
The second design
Instead of a single context for the entire game, now each movement phase was a context - three turns of press, one of movement, one of resolutions (retreats), one of builds/disbands (in winter) and a final call to ask the model to write a memory for the next phase. This worked fine in initial testing, and I moved forward with it.
I started with 35 LLMs and two deterministic no-press gunboat players.
I then had to abandon some LLMs due to persistent order failures, but added in some higher-reasoning models to compensate.
So with this in mind and some initial smoke tests complete, I started games and left things overnight.
In the morning, I discovered that Opus and Fable 5 cost dramatically more than the next most expensive model (Sonnet 5) with my harness (through OpenRouter), and I needed to rebalance things. (Sorry everyone, only two games with Fable and three with Opus because five games across those two models ended up being about 1/4th the cost of the whole tournament.) So I added some less expensive models on max reasoning effort instead - every other model was run with low effort but an unlimited token budget.
The Opus and Fable games are still included, but their results are less reliable as they did not play nearly as many opposing models as other entrants.
Results and Analysis
The big models won, as expected. Increased reasoning helped DeepSeek V4 Pro, Terra, and Luna, but hurt GLM 5.2. All but the smallest models managed to beat the Gunboats.
And yes, the models were willing to lie, to betray, and to stab.
Were they good at it? Absolutely not. But it was interesting.
Some models were dramatically more aggressive than others. Given the opportunity, Opus 5 attempted to walk into an adjacent enemy-controlled unoccupied center 13 times in 60 opportunities. (the two deterministic models took the chance in 28 of 73 opportunities (normal) and 85 of 223 (aggressive)) In contrast, Sonnet 5 did this 8 times in 230. It did manage to actually enter the center 7 of those 8 times, but surely this isn’t an optimal level of aggression.
Even the most competent models were quite likely to sit in place and do nothing, unfortunately. Often a model would enter holds - not even supports - for all but one unit.
The full results, along with model inputs and outputs, can be found on diplobench.com, along with any future runs.
Power strengths
Models did dramatically better playing as England, with Germany faring terribly. Adjusted scores take into account what model was playing, not merely the raw averages. The sole win by Turkey was by Fable in one of its two games.
| Rank | Power | Adj SC/G | Raw SC/G | Solos | Survive |
|---|---|---|---|---|---|
| 1 | England | 7.07 | 6.85 | 3 | 90% |
| 2 | France | 5.47 | 5.49 | 2 | 77% |
| 3 | Russia | 4.89 | 4.74 | 1 | 74% |
| 4 | Austria | 4.50 | 4.62 | 0 | 82% |
| 5 | Italy | 4.47 | 4.49 | 0 | 72% |
| 6 | Turkey | 3.88 | 4.10 | 1 | 64% |
| 7 | Germany | 3.72 | 3.72 | 0 | 54% |
X-high reasoning variants
Enabling reasoning for OpenAI models helped much more than the tested Chinese ones, with GLM-5.2 performing worse with higher reasoning.
| Base | X-high | Bayesian score Δ | SC/G Δ | Cost/G multiple |
|---|---|---|---|---|
| deepseek-v4-pro | deepseek-v4-pro-xhigh | -0.2% | +1.21 | 2.72× |
| glm-5.2 | glm-5.2-xhigh | -15.4% | -2.80 | 1.60× |
| gpt-5.6-luna | gpt-5.6-luna-xhigh | +30.1% | +4.43 | 5.65× |
| gpt-5.6-terra | gpt-5.6-terra-xhigh | +15.1% | +3.12 | 2.66× |
Cost–performance Pareto frontier
No listed paid model is both cheaper per game and has a higher conservative Bayesian pairwise-score estimate than another listed model. Both movement-only expert system Gunboat models beat granite-4.1-8b, though.
| Model | Conservative score | Central score | Reported cost/G | $/active year |
|---|---|---|---|---|
| granite-4.1-8b | 4.1% | 16.4% | $0.04 | $0.0046 |
| ling-3.0-flash | 9.6% | 29.9% | $0.06 | $0.0044 |
| gpt-5.6-luna | 17.5% | 44.3% | $0.09 | $0.0055 |
| deepseek-v4-flash-0731 | 19.7% | 47.2% | $0.28 | $0.02 |
| gpt-5.6-luna-xhigh | 41.9% | 74.4% | $0.49 | $0.03 |
| gpt-5.6-terra-xhigh | 57.4% | 87.0% | $3.01 | $0.15 |
| gpt-5.6-sol | 59.6% | 86.8% | $6.33 | $0.35 |
What’s Next
The obvious followup would be a single game with the top 7 players, and of course to play again with more models as time goes on. Or to actually finish out a full set of Fable games - aborting for cost reasons is pretty painful. Another would be to run a more comprehensive reasoning effort test - choose a few models and compare no reasoning through extra high, and see how model performance changes. Finally, a variance test would be interesting - how different are the results when the same matchup is run multiple times? Will I? No idea.
If you have comments or suggestions, please reach out to diplobench @ laverya.com.
All posts