On a related note, I am always wary of the term "anecdotal evidence". This phrase is usually used, in my experience, as an excuse to dismiss certain information as irrelevant, usually by someone who would not like the conclusions supported by that information. However, regardless of any dictionary definitions, the "anecdotal" label tends to be used to describe two very different things, depending on context: either a "sample of one" ("Well, in my case, this happened...") or information from an "indirect source" ("I once heard of a guy who...").
The second class tend to be unsubstantiated, and of course we must be wary of giving any weight to unsubstantiated hypotheticals. However, it is all too easy to generalise and write off any "anecdotal evidence" in this way. I find the argument that a sample of one is not completely representative and therefore irrelevant to be unsound. By that logic, since any sample that is not the entire population is unrepresentative to some extent, any incomplete sample is irrelevant. The difference is quantitative, not qualitative. Similarly, only in the world of scientific conceits do we believe that perfect studies with ideal control groups exist.
Newspapers often quote survey results with sample sizes of just over 1,000 members of the public. This is not a coincidence; it gives us a certain percentage level of statistical reliability, and by convention we accept that level. However, that level has no special mathematical properties. It is only a convention, and all such statistical tests ultimately come down to getting one answer or the other based on an arbitrary number. Consultants are, of course, very good at picking that number to get answers their clients will like!
Thus while we may reasonably choose to give more weight to formally-structured, well-controlled studies with higher levels of statistical reliability, we should not entirely discount verifiable information just because it comes from smaller samples, particularly if that's all we have available. Even a single verifiable counter-example, anecdotal as it may be, is enough to support scribb1e's third point: if you thought something was impossible, but it definitely happened at least once, then it can be at most "highly unlikely", not "impossible". (Obviously you still need enough control information that it's reasonable to consider the counter-example verified.)
Always remember, statistics is little more than the study of quantifying faith.
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
timeasmymeasure
no subject
Date: 2006-09-10 02:51 pm (UTC)The second class tend to be unsubstantiated, and of course we must be wary of giving any weight to unsubstantiated hypotheticals. However, it is all too easy to generalise and write off any "anecdotal evidence" in this way. I find the argument that a sample of one is not completely representative and therefore irrelevant to be unsound. By that logic, since any sample that is not the entire population is unrepresentative to some extent, any incomplete sample is irrelevant. The difference is quantitative, not qualitative. Similarly, only in the world of scientific conceits do we believe that perfect studies with ideal control groups exist.
Newspapers often quote survey results with sample sizes of just over 1,000 members of the public. This is not a coincidence; it gives us a certain percentage level of statistical reliability, and by convention we accept that level. However, that level has no special mathematical properties. It is only a convention, and all such statistical tests ultimately come down to getting one answer or the other based on an arbitrary number. Consultants are, of course, very good at picking that number to get answers their clients will like!
Thus while we may reasonably choose to give more weight to formally-structured, well-controlled studies with higher levels of statistical reliability, we should not entirely discount verifiable information just because it comes from smaller samples, particularly if that's all we have available. Even a single verifiable counter-example, anecdotal as it may be, is enough to support
Always remember, statistics is little more than the study of quantifying faith.