Social Science Statistics Blog

« November 21, 2009 | Main | November 28, 2009 »

26 November 2009

(not) growing up to be a lawyer

I went to law school before I ended up as a graduate student, so I read with some interest a recent essay by Vanderbilt Law Professor Herwig Schlunk entitled "Mamas, don't let your babies grow up to be...lawyers" (an online version is at the Wall Street Journal's Law Blog).

Maybe the title gives it all away, but the gist is that a legal education might not always pay off. While I wholeheartedly agree with this, I'm less enthusiastic about the author's methodology. The author essentially constructs three hypothetical law students: "Also Ran," a legal slacker who attends a lower-ranked law school; "Solid Performer," a middling kind of person who attends a middling kind of law school; and "Hot Prospect," a high-flying and well-placed law student. The essay then more or less "follows" them through their legal "careers" to see if their discounted expected gains in salary match what they "paid" in terms of opportunity costs, tuition, and interest on their student loans. (I know, I'm using a lot of air quotes here.) Unsurprisingly, a legal education isn't a very good investment for any of the individuals.

What's interesting about the paper is that it's essentially an exercise in counterfactuals -- what Also Ran would have earned after going to law school, what Hot Prospect would have made had she not gone, etc., etc. To that extent, it's very fun think about. But, on the flip side, that's kind of what it is -- a thought experiment. Maybe an interesting extension would be a do an empirical causal analysis -- maybe matching pre-law undergraduates along a slew of covariates and then seeing how the "treatment" of law school affects or does not affect their earnings. I'd certainly find that a lot more persuasive (although I imagine that the kind of data that you'd need to pull this off would be well-nigh impossible to collect).

The article has gotten quite a bit of attention from legal types, including from the WSJ's Law Blog and from the NYTimes Economix blog. Thoughts?

Posted by Maya Sen at 10:58 AM

breast cancer, rare diseases, and bayes rule, revised

Happy Thanksgiving!

Last Thursday, I posted about the recent government recommendations regarding breast cancer screening in women ages 40-49. At least one of you wrote me to say that one of my calculations might have been slightly off (they were), and so I did some more investigation on this issue, as well as on new recommendations on cervical pap smears. (Sorry --it took
me a few days to get around to all of this!)

To back up a second, here's what the controversial new recommendations (made by the US Preventative Services Task Force) say:

• The USPSTF recommends against routine screening mammography in women aged 40 to 49 years. The decision to start regular, biennial screening mammography before the age of 50 years should be an individual one and take patient context into account, including the patient's values regarding specific benefits and harms.
• The USPSTF recommends biennial screening mammography for women aged 50 to 74 years.
• The USPSTF concludes that the current evidence is insufficient to assess the additional benefits and harms of screening mammography in women 75 years or older.
• The USPSTF recommends against teaching breast self-examination (BSE).

As I noted in my original post, these recommendations really upend existing wisdom. For years, women have been told that routine self-examinations -- followed by regular mammograms past age 40 -- are the best way to counter breast cancer. In fact, women as young as 18 are regularly given directions on how to do self-exams, and it's not unusual for women in their 30s to undergo mammograms.

So all of this got me thinking that this could maybe be a straightforward application of the "rare diseases" example of Bayes' Rule (which many people see in their first probability course). I did some (more) digging around in one of the government reports, and here's
how the probabilities break down:

• Experts estimate that 12.3% of women will develop the more invasive form of breast cancer in their lifetimes (p. 3).
• Second, the probability of a woman developing breast cancer ages 40-49 is 1 in 69, but increases as she gets older. In women ages 50-59 it's 1 in 38 (p. 3).
• Third, breast cancer is also more difficult to detect accurately in younger women -- likely because their breast tissue is more dense and fibrous. Experts estimate that there is a false-positive rate of 97.8 per 1,000 screened for women in their 40s, but a 86.6 per 1,000 rate for women in their 50s (Table 7).
• Fourth, there is a false-negative rate of 1.0 per 1,000 screened for women their 40s, and a 1.1 per 1,000 rate for women in their 50s. (This last part is what I goofed up in my last post; thanks to those who caught it! It's also in Table 7 of the report, although, as we see later, it doesn't change the probabilities by that much.)

Now bear with me while I go through the mechanics of Bayes' Rule. For women in their 40s, here are the pertinent probabilities:

P(cancer) = 1/69
P(no cancer) = 1-1/69 = 68/69
P(positive|no cancer) = 97.8/1000
P(negative|no cancer) = 1 - 97.8/1000 = 902.2/1000
P(negative|cancer) =1/1000
P(positive|cancer) = 1 - 1/1000 = 999/1000

And for women in their 50s:

P(cancer) = 1/38
P(no cancer) = 1-1/38 = 37/38
P(positive|no cancer) = 86.6/1000
P(negative|no cancer) = 1 - 86.6/1000 = 913.4/1000
P(negative|cancer) =1.1/1000
P(positive|cancer) = 1 - 1.1/1000 = 998.9/1000

The probability we are interested is the probability of cancer given that a woman has tested positive, P(cancer|positive). Using Bayes' Rule:

P(cancer|positive) = P(positive|cancer)*P(cancer)/P(positive)
P(cancer|positive) = P(positive|cancer)*P(cancer)/P(positive|no
cancer)P(no cancer)+P(positive|cancer)P(cancer)

We now have all of the moving parts. Let's first look at a woman in her 40s:

P(cancer|positive) = (999/1000*1/69)/(97.8/1000*68/69+999/1000*1/69)
= 0.1305985

and for a woman in her 50s:

P(cancer|positive) = (998.9/1000*1/38)/(86.6/1000*37/38+998.9/1000*1/38)
0.2376579

All this very simple analysis suggests is that mammograms do appear to be a less reliable test for younger women. Whether these recommendations make sense is another matter. Insurance companies might use these recommendations as an excuse to deny coverage for women with higher than average risk. In addition, as some of you noted, a 13% risk is nothing to sneeze at, and it's much higher than the 1/69 rate for women in their 40s (though comparable to a woman's lifetime 12% risk). Lastly, I also refer folks to Andrew Thomas's post , where he discusses the metrics used by the task force and notes that the confidence interval for women in their 40s lies completely within the confidence interval for women in their 50s.

I also did some very brief investigation regarding the new cervical cancer guidelines. For those of you unfamiliar with this story, the American College of Obstetricians and Gynecologists recently issued recommendations that women up to the age of 21 no longer receive pap test and that older women receive paps less often -- also advice contrary to what women have been told for decades.

It was much harder to pinpoint the false positive and false negative rates involved with pap tests (a lot of medical jargon, different levels of detection, and human and non-human error made things confusing). I did manage to find this article in the NEJM. The researchers there looked at women ages 30 to 69 (a different subgroup, unfortunately, from the under 21 group), but they do report that the sensitivity of Pap testing was 55.4% and the specificity was 96.8%. This corresponds to a false negative rate somewhere around 44.6% and a false positive rate somewhere around 3.2%. (Other references I've seen elsewhere hint that the false negative rate could be anywhere from 15 to 40%, depending on the quality of the lab and the cells collected.)

The other thing to note is that cervical cancer is very rare in young women and, unlike forms of cancer, it grows relatively slowly. According to the New York Times, 1-2 cases occur per 1,000,000 girls ages 15 to 19. This, combined with the high false negative rates, resulted in the ACOG recommendations.

My sense is that the ACOG recommendations are on more solid footing, but if people have comments, I'd be keen to hear them.

Posted by Maya Sen at 10:03 AM