Holden at GiveWell has posted a very interesting analysis of Why We Can’t Take Expected Value Estimates Literally. I've always been suspicious of the idea that we should treat rough subjective estimates of risk (e.g., an "X% probability" that [insert scary futuristic technology here] will destroy the world) equivalently to robustly established probabilities (e.g., an X% chance that a large asteroid will hit the earth within a century). Holden's analysis backs up this intuition, by appealing to the idea that we need to adjust "explicit expected value" calculations by the variance in our "estimate error".
The upshot: robustly established estimates count for nearly their full weight, whereas highly uncertain estimates should barely move us away from our priors. To illustrate: "It seems fairly clear that a restaurant with 200 Yelp reviews, averaging 4.75 stars, ought to outrank a restaurant with 3 Yelp reviews, averaging 5 stars." Why? Because a mere three reviews is not robust enough evidence to shift us far from our prior expectation (i.e. that the restaurant is just average).
Anyway, this strikes me as a very important (and intuitive) result, which my rough summary here doesn't really do justice to. So, go read the whole thing!
Thursday, August 18, 2011
1 comment:
Visitors: check my comments policy first.
Non-Blogger users: If the comment form isn't working for you, email me your comment and I can post it on your behalf. (If your comment is too long, first try breaking it into two parts.)
Note: only a member of this blog may post a comment.
Subscribe to:
Post Comments (Atom)
Posts
Using imprecise probability to model rational credence should help with this. In the "rough estimate" type cases we should have very spread out credences. In the case of more reasoned or evidentially established cases credences may be more precise. See James Joyce's recent paper "A Defense of Imprecise Probability in Inference and Decision Making" for an explication and defense. I have some subtle differences with Joyce, but my understanding and writing is still in progress.
ReplyDeleteI would not put it in terms of how moved we should be from our (presumably precise) prior, but rather in terms of how confident we should be that given odds are advantageous based on objective updating of imprecise priors. I have been thinking in terms of the choice of variance in the Imprecise Dirichlet Model. Here is a preliminary sketch of an application of that model to the pragmatic encroachment question in epistemology:
http://uiowa.academia.edu/JeremyShipley/Papers/328946/Varying_Variance_Invariantly_essay_on_pragmatic_encroachment_and_thick_credence_slides_