Calibrated Confidence

Author and veteran trader Brent Donnelly’s recent post, #37, in his highly educational, free Substack series 50 Trades In 50 Weeks dissects common traits of successful traders:

Week 37: Common Traits of Top Traders (10 min read)

It’s a terrific read for what it validates, but also for what is surprising (“You can succeed in trading with any level of risk appetite.”) It’s also a great read to see how he constructs a survey to challenge his beliefs.

I want to zoom in on one of the key 5 key traits he identifies: “calibrated confidence”.

When we hosted the StockSlam sessions in October we gave a couple of homeworks before the in-person events. One of the homeworks involved this module:

You will be asked to make 90% confidence intervals on some facts.

90% confidence interval can be understood with an example:

I have not been outside for a few hours, and without looking up the answer, “I’m 90% confident that the temperature outside my window is between 45°F and 60°F, or (45,60).”  If I wanted to be more (like 99%) confident, I would widen the interval, and conversely a tighter interval would coincide with less confidence.

Example questions included:

• Without looking anything up, what is your 90% confidence interval on the number of bones in the human body?
• What is your 90% confidence interval on the length (# of letters) of the longest word in English?

Note what you are doing. You are making markets where you think fair value is 90% to be inside your bid/ask spread.

This is highly relevant to trading and handicapping.

If you get 10 out 10 markets “right” then you are more conservative than we asked you to be. In other words, you were too wide. In markets, this means you will never trade. You are bidding $50 for a stock trading $60. You will get no market share.

If you get say 5 out of 10 markets right, then you were too confident. In markets, everyone will trade with you and you will be sadder for it.

Trading is only partially about knowing “fair value”. It’s in your meta-knowledge about fair value that the magic happens. You are always dealing in uncertainty…your feel for the degree of uncertainty is how you recognize opportunity or defend yourself. This is not just market-maker talk. It’s the essence of what Buffet understood when he said “margin of safety”.

Google “Paul Slovic’s horse bettor study” and you will find several posts such as:

How Can Confidence Kill Investment Returns? (4 min read)

They all talk about a 1973 study where experienced horse handicappers are given a few pieces of data of their own choosing. Armed with their preferred data, they are able to make good bets, but critically, they are well-calibrated about their accuracy. Their confidence and accuracy were in agreement.

However, as the bettors are given increasing amounts of data their accuracy falls, but their confidence shoots up. No bueno.

Presumably, they were less experienced in weighing the additional data which turns out to be noise to their handicapping process, but the presence of more info gave them the illusion that they would be better. A disastrous combo.

Improving Our Handicapping Skills

It sounds discouraging to realize that decision-making is not just an exercise in accurate prediction but also judging how wide our error bars should be. But there’s plenty of good news. This is not a skill anyone is born with. It is learned.

[This is why prop trading firms who recruit elite students from top schools screen for teachability which I suspect is randomly allocated across skill. There are elite students or athletes who can remain humble learners despite the advantages of their talents and there are insufferably overconfident talents who have yet to get hit by the Mack truck of peak competition. This latter group is dangerous to themselves and if given enough rope, to their backers too.]

3 suggestions to get better:

1. Self-diagnose. You can take a fun calibration test to establish some context:
   http://confidence.success-equation.com/
   
   This thread includes my results as well as many others including several side threads of interpretation and discussion.
2. Read Superforecasting: The Art and Science of Prediction
   
   This book is a great primer on what it takes to predict but most importantly frames prediction as a skill we can improve at. I took extensive notes which weave my thoughts into some of the key points.
3. Bet on stuff
   
   How long will it take to finish your school or work assignment?
   
   What percent of random sample of people from your contact list knows what date the winter solstice falls on?
   
   (My wife and I bet on what facts are “common knowledge” when one of us gets defensive to the other’s attack “how did you not know that?” We’ll poll some people with a text message and bet on the consensus. These “studies” are highly correlated with one of us having “one drink too many”.)
   
   Calibration is a nice benefit of a betting culture (1 min read).