This maybe makes a good case for the rule of thumb “report odds ratios, not posterior priors”. If the statistician simply reports the formula , then people who know the context of the experiment can decide how important a very skinny peak slightly to the right of matters.

]]>Suppose I have a prior that puts nonzero probability on p=1/2. I’m worried you can construct a stopping rule such that, as p approaches 1/2 without becoming equal to it, the probability that I conclude that p=1/2 approaches 1.

Or, worse, maybe there’s a neighborhood around p=1/2 in which the stopping rule guarantees that I’ll conclude p=1/2.

In fact, the more I think about it, the more strongly I’m driven to conjecture that this is the case. So I don’t really believe that putting nonzero prior probability on p=1/2 protects Bayesian analysis from stopping rules.

]]>I am always comfortable using a Bayesian analysis with a prior that matches my prior belief. In practice this is hard and I use frequentist analyses a lot because I don’t feel like dealing with prior specification. But what I’m mainly concerned with is whether Bayesianism is correct.

All this example shows me is that there are certain experimental designs where you have to include prior information that you can normally get away with leaving out (specifically, the possibility of the parameter being exactly .5). Nobody says that systems with significant air resistance disprove Newtonian physics, just because it’s common practice to leave out air resistance.

]]>By the way, it occurs to me that the drug example may not be the best story to consider wrapping around this problem, because it really shouldn’t matter if a drug is equal to placebo or better. As an alternate story, suppose the experimenter is Chien-Shiung Wu trying to demonstrate parity violation in weak interactions. Now any nonzero effect, in either direction, is Nobel prize material; we really do care about whether the value is exactly or not. Are you comfortable analyzing that data by this method ?

]]>Last sentence of the intro should match the comment

Below the disjunction sign and summation sign in section one, I wrote “i=ceil(N/2+sqrt(N))” where I meant “N=1″. Long story.

I should clarify. First of all, I am _not_ suggesting you update on the drug company’s protocol. I’m assuming that they don’t have any more information about the drug than we do. If I thought they did know something more about the drug, then I _would_ update on their protocol, as you said, but that’s kind of leaving behind the thought experiment.

What I’m saying is, if our prior distribution is identical to our prior degrees of belief, we end up in one of two situations:

1. We don’t think the drug company can make us believe theta > 1/2

2. Our prior dist for theta is strongly peaked near 1/2

The significance of this is that, in neither case do we think of the thought experiment in the OP as a reason not to do a Bayesian analysis.

The math backing this up was too much to put in a comment, so here it is as a pdf:

https://drive.google.com/file/d/0B6198rcSwqpyb0FseGlkZUJWcVE/view?usp=sharing

]]>