choices13k Has No Learning Curve

The dataset ships a column called Block that runs from 1 to 5. It looks like an experience trajectory. It isn't one — and the difference matters if you're modelling learning, adaptation, or anything that accumulates over trials.

Pulakesh Upadhyaya · 11 September 2026 · ~6 min

choices13k is the largest public dataset of human decisions under risk — 13,006 gamble pairs, roughly 16 participants each, collected by Peterson, Bourgin, Agrawal, Reichman and Griffiths for their 2021 Science paper. It has become a default benchmark for models of risky choice, and it is a genuinely excellent resource.

It also contains a trap I walked straight into, and I suspect I'm not the only one. This is a short note on what the Block column is, what it is not, and how to check the difference in about ten lines of code.

Among feedback problems, Block is assigned uniformly at random from {2, 3, 4, 5}. It is a label, not a position in a learning sequence. Participants did not accumulate experience across it, and the data confirm they didn't.

1.Why the column looks like a trajectory

The trap is inherited, not invented. choices13k follows the presentation format of the 2015 and 2018 Choice Prediction Competitions, and in those designs the block parameter meant exactly what you'd expect: participants played five successive blocks of five trials with feedback, and the block index recorded how much experience they had accumulated. Block 5 really was later than block 2.

choices13k reduced that to two blocks — one without feedback, one with. But the column name and its 1-to-5 range survived the reduction. So you get a variable called Block, spanning the same values it spans in the parent literature, that no longer carries the same meaning.

To the dataset's credit this is documented. The README states plainly that for problems with no feedback Block is always 1, and otherwise it “was sampled uniformly at random from {2, 3, 4, 5}.” The information is right there. It's just easy to read past when the column has a familiar name and your analysis wants a time axis.

2.What the data say

Random assignment has one happy consequence: it makes the check clean. If Block carried experiential content, later blocks should show better choices. Because assignment is random, any difference would be causal. So just look.

The natural outcome measure is accuracy: the rate at which participants pick the gamble with the higher expected value.

3.How flat is flat?

“No effect” is only a claim if you say how large an effect you could have detected. The linear trend in accuracy is +0.00019 per block with a 95% interval of [−0.0027, +0.0031]. Taking the top of that interval and running it across the full span from block 2 to block 5 gives a ceiling of about 0.009.

Set that against the effect that is in the data. Comparing the same problem with and without feedback — 1,562 problems appear in both conditions — gives a gain of +0.029. So the largest block drift the data can hide is under a third of the description-to-experience effect measured on the very same problems.

Choice extremity tells the same story: the trend in |bRate − 0.5| is −0.00098, p = 0.338. A one-way ANOVA across the four blocks on accuracy gives F = 0.43, p = 0.730. There is nothing there.

One honest caveat. Raw bRate — the unsigned rate of choosing gamble B — does show a weak upward trend across blocks: +0.0038 per block, p = 0.035. It is one marginal result among three tests, it does not survive as accuracy or extremity, and gamble B is an arbitrary label rather than a normatively better option, so the direction carries no interpretation. I mention it because I'd want to know about it, not because I think it's real.

4.What to do instead

None of this makes choices13k less useful. It makes one specific design unavailable, and points at the one that works.

1. Don't treat Block as time, trial count, or accumulated experience. Don't fit learning curves over it, and don't use it as an ordinal predictor in a model of adaptation.
2. Do use the feedback contrast instead. 1,562 problems appear in both the no-feedback and feedback conditions, which gives a clean within-problem description-to-experience comparison with every problem characteristic held fixed.
3. Do pool blocks 2–5 when you want the feedback condition. Since assignment is random and the outcome is flat, pooling costs you nothing and quadruples your cell sizes.
4. Go elsewhere for genuine trial-level dynamics. CPC18 publishes raw data with real within-subject block sequences. choices13k aggregates over roughly 16 participants × 5 trials, so even subject-level heterogeneity in learning rate is averaged away before you see it.

5.Check it yourself

The whole thing is one groupby. If you're using choices13k for anything that accumulates over trials, this is worth thirty seconds.

import pandas as pd, scipy.stats as st

d  = pd.read_csv("c13k_selections.csv")
fb = d[d.Feedback]

# bRate is the rate of choosing gamble B; flip it where A is the higher-EV option
# (evA / evB computed from c13k_problems.json)
print(fb.groupby("Block").pmax.agg(["mean", "sem", "size"]))

r = st.linregress(fb.Block, fb.pmax)
print(r.slope, r.slope - 1.96*r.stderr, r.slope + 1.96*r.stderr, r.pvalue)
# +0.00019   -0.00270   +0.00309   0.895