Thanks for your analysis, Anthony. Admittedly, I did not look beyond the Guardian article, but then, I generally don't have time to fact-check everything I link. You seem to suggest that's irresponsible, but if I expected to be held to that level of rigor, I'd at least hope to get paid for my blogging. I don't view blogging as a way to trade authoritative facts so much as a way to have conversations about information as it emerges. You've made an invaluable contribution to the conversation about this Lancet study, and for that we should all be greatful. Thank you.

You seem to suggest that's irresponsible, but if I expected to be held to that level of rigor, I'd at least hope to get paid for my blogging.

I don't think it's irresponsible, in the sense of being negligent, Ambimb, but I do think it goes to credibility. I'd much rather read someone who's a partisan democrat if I know that when presented with a fact that seems just too good (a high casualty report coming out just days before an election?), he'll take fifteen minutes to check it before posting. There's a lot of things I read and like, but don't link to because I can't verify it: I either lack time or experience.

What I hope is that my Democratic readers, when looking at this partisan Republican, will at least have confidence that I'm doing more than just feeding them garbage. That way if I do say something about, say, Guantanamo, they don't come back and say, "Oh, yeah, but last week you told us about those 100,000 deaths that weren't, too."

Having read the thing (and sent you a longer email) had the Guardian prefaced their comments with 'The most accurate study to date shows that about...' they'd have been fine.

I should point out that 100 000 remains *the most likely figure* with a 66% chance that the results are between 48 thousand and 148 thousand.

Martin:

That's a very low, flat bell curve you've got there, supported by data inconsistent with other studies.

(Remember, if the pre-war death rate is understated, then this estimate collapses as well, and the infant mortality rate understates previous assumptions by a factor of four. Indeed, didn't you used to show me Guardian articles on the bus in the morning with such figures in, back when the debate was whether sanctions were a great evil and the war was still far off?)

But thanks for going on-record in support. When this document is adequately peer-reviewed, it'll be interesting to watch what happens.

Tony,

Thanks for defending me over at The Yin Blog, though I think Prof. Heller had plenty of room to criticize my initial comment for its lack of content. I didn't want to get into an argument with him or point out the argument from authority because that's just not my thing. It's just so frustrating to see obviously smart people (law profs) post this kind of stuff with no critical analysis. I expect more from law professors, though maybe I shouldn't. It goes to weight, not to admissibility.

The criticism of the Lancet study has been of awful quality and has been taken as gospel by pro-war bloggers. I had to create a new subject category for all my posts correcting this nonsense. Here.

Tim,

Afraid I simply don't buy it. You seem to have refuted some carelessly-written comments at TCS, true, but it doesn't change the salient facts: first, that the range of error in the study is enormous; and second, that the use of a cluster study in these conditions is pretty ridiculous, because over assymetrical data the results will be incorrect in ways not shown in the confidence index.

I'd accept a statement along the lines of, "Somewhere between 8,000 and 196,000 'extra deaths' have occurred in Iraq." But 100,000 as a standalone number, as reported in the press, is fairly silly. And while you've poked wholes in an argument or two on the margins, you've still not butressed the argument actually in the Lancet.

Your reply is unfortunately typical of critics of the study. You don't know much about statistics and you don't like the result so you will accept any silly argument about why the statstics are wrong, even though you don't understand them. Clustered sampling is a perfectly standard way of doing these sorts of studies and we know how to estimate sampling error for these sorts of studies.

The confidence interval is wide but it indicates increased deaths. Nor is every number in the confidence interval equally likely--it is much more likely to be near 100,000.

And if you are looking for a Dan Rather analogue in this story, I give Michael Fumento, who continues to insist that the 100,000 number includes Falluja.

Tim:

You are, I suppose, a statistician? I'm sorry, but I really don't feel like looking through your blog for a biography. Someone who just shows up and--typical of the supporters of the study--call those critiquing ignorant without putting forward an argument isn't likely to gain my effort.

Your counterargument--if you take out the insults--is basically a tissue of nothing:

• Yes, the range covers a number greater than zero. (Actually an argument you link to at Crooked Timber.) But no one thought it would be negative, or at least that would have astounded me. There was, after all, an invasion.
  
• Cluster sampling is "perfectly standard"? Please, cite a source or two. I'm sure it's done quite often when it's the best way of gathering data, but in this case the "best" simply isn't very good. Hence the wide range of error.
  
• "[I]t is much more likely to be near 100,000": Yes, yes, if you were to plot the probabilities it should look like a bell curve centered on 98,000. The point is that it's a rather shallow bell curve: while a number around 100,000 is more probable than anything else, it's not much more probable.

But anyway, please explain to me some things, since you're welcome to post here if you want to make a convincing argument:

A) How do you know that the casualties were civilians? The report does not ever tie the headline number of 100,000 together with the term "civilian" and even disclaims that it can prove so on page seven.

B) Do you think that the numbers would look different if you tried to estimate the death toll regionally or by governate? The study's authors unfortunately didn't give us the data to evaluate that.

C) You insist that the study was done by random sampling. Nonetheless, Page 2 of the report suggests that the sampling was not random, but modified based upon the ability of interviewers to reach certain areas. You don't think this might add a structural error to the data that would not show up in the CI?

D) Finally, the report does not attempt to take into account adjustments for displacement and immigration. This would seem an odd assumption to make in any kind of invasion scenario, where you would expect at least a percentage of the population to move from more dangerous population centers to less. Do you think that would show up in the CI, or that it's a variable not reflected in the model?

E) The report suggests 25,000 civilians have died in air strikes, excluding the one outlier cluster. Does that sound about right to you?

Hi,

Just a couple of things - I'm pretty sure Tim has never said 100000 civilian deaths just that the paper says 98000 excess deaths.
On your point C) I thought that the pairing of governates with similar prewar characteristics, randomly selecting one and then randoming selecting areas within the selected governate was farily logical and unlikely to impart bias. The clusters were assigned without regard to the perceived threats but the pairing of the governates was carried out to reduce the amount of travel and therefore reduce some of the risks faced by the researchers. The authors also state that this was likely to reduce the accuracy of the final numbers (ie increase the CI).

Cluster sampling is a standard method - try typing 'cluster sampling' into PubMed. Cluster sampling is a standard method taught in universities and discussed in statistical textbooks.

And I think with E) the 'does that sound right to you?' line is a way of dismissing the report because it doesn't agree with what you presume to be the truth.

I actually think that this paper reasonable but it should be kept in mind that this is a single study and more studies need to be carried out before the 'true' numbers are arrived at.

Simon:

With regards to (C), I think you overlook aspects that are likely to impart bias. In countries at war, there is often significant population mobility, especially among those with the resources or contacts to move "out of harm's way," as it were. If one has reason to believe that families able to move towards the borders would be safer (because, for instance, they moved beyond dangerous zones before they became so), that would skew your distribution in a manner that may very well not be reflected in the CI.

As for cluster-sampling, yes, it's a common statistical methodology. Like all such standard methodologies, it works well in some instances and less well in others. No one is saying that it should never be used, merely that its use in a population in which you'd expect a great amount of asymmetry in the data is inappropriate.

Look at it this way: the data suggests that 25,000 people have died from air strikes. But in order to get that data, it suggests that an equal number of individuals died from air strikes all over the country. (The final tally is derived from subjecting the gross increase in mortality across the entire population.)

Now, given that the air strikes of the U.S. and its allies have actually been localized in combat areas, doesn't that suggest that averaging such things over the entire country might be overstating the case?

And yes, the 25,000 number for casualties due to air strikes does seem fairly high to me, especially given how it's been derived.

1. It is not true that no-one thought that it was negative. In fact, it the standard warblogger response if anyone brought up civilian deaths was to claim that on net, lives had been saved. For instance, Glenn Reynolds: WHY ISN'T THE IRAQ BODY COUNT METER RUNNING IN REVERSE?

2. You can find references to the use of cluster sampling in the paper. Note that the previous surveys of mortality in Iraq that you seem to think refute this study also used cluster sampling.

3. A 67% CI goes from 50,000 to 150,000. This is the range that most likely contains the true number.

Anthony,

The authors did make an attempt at working out the 'moving out of harms way' factor by recording the number of vacant households in the clusters they sampled. You would expect that most of the clusters would represent the safety of most of the country, some would be significantly more dangerous and some much more safe. To use a known dangerous place as an example - Falluja - 44% of the homes were vacant. Now if all the clusters were as about as dangerous as Falluja then the other clusters would have a similar vacancy rate. The average was less than 8%. Also, unlike Falluja, none of the houses were abondoned after all or most of the residents had died.