18 April 2008
In last week's debate in Philadelphia,
Last week's debate provides a small but interesting corpus to analyze the candidates' favorite linguistic formulations. Overall,
So all in all, the candidates spoke about the same number of words. But which words? We can test that using a basic corpus comparison method. In all, there were 1,971 unique words. For each of these, we test the hypothesis that the candidates spoke the word with equal probability, using a simple chi-squared test. Next we sort all words by their p-values so that the most differentially expressed words percolate to the top. Here are the top 20 words by p-value, along with their frequencies from Obama and Clinton.
Sometimes control words (I, it, etc.) are excluded from analysis, but here I thought it would be fun to leave them in so we could see each candidate's preferred constructions. Besides the points listed above, here are a few interesting notes:
- Clinton used the word "I" 205 times to Obama's 150
- Obama loves to start sentences with "That's:" "That's why I'm...", "That's what we're," etc.
- Obama loves the word "decade" -- evidently he used the phrase "decades after decades" several times
Of course, unigrams -- single words -- can only tell you so much. If we do the same analysis using bigrams, a few more bits of information drip out:
So Clinton always punctuates her thoughts with "you know," while Obama attributes his goals to the "American people."
It will be interesting when McCain gets into the mix with one of these two. I think it would be fun to construct a language model -- a model for the probability that each candidate spoke a certain sentence. Given the differences, I bet that given a sentence, it could easily figure out whether Obama, Clinton or McCain said it!