21 September 2010
An interesting 1992 paper by Bayarri and DeGroot entitled “Difficulties and Ambiguities in the Definition of a Likelihood Function” (gated version) grapples with the problem of defining the likelihood when auxiliary variables are at hand. Here is the abstract:
The likelihood function plays a very important role in the development of both the theory and practice of statistics. It is somewhat surprising to realize that no general rigorous definition of a likelihood function seem to ever have been given. Through a series of examples it is argued that no such definition is possible, illustrating the difficulties and ambiguities encountered specially in situations involving “random variables” and “parameters” which are not of primary interest. The fundamental role of such auxiliary quantities (unfairly called “nuisance”) is highlighted and a very simple function is argued to convey all the information provided by the observations.
The example that resonates with me in on pages 4-6, where they describe the ambiguity of using defining the likelihood function when there is an observation y which is a measurement of x subject to (classical) error. There are several different ways of writing a likelihood in that case, depending on how you handle the latent, unobserved data x. One can condition on it, marginalize across it, or include it in the joint distribution of the data. Each of these can lead to a different MLE.
Their point is that situations like this involve subjective choices (though, all modeling requires subjective choice) and the hermetic seal between the “model” and the “prior” is less airtight than we think.
14 September 2010
The Misconception: You take randomness into account when determining cause and effect.
The Truth: You tend to ignore random chance when the results seem meaningful or when you want a random event to have a meaningful cause.