{"id":1244,"date":"2013-07-08T12:04:50","date_gmt":"2013-07-08T12:04:50","guid":{"rendered":"http:\/\/garysieling.com\/blog\/?p=1244"},"modified":"2013-07-08T12:04:50","modified_gmt":"2013-07-08T12:04:50","slug":"nlp-analysis-using-modal-verbs","status":"publish","type":"post","link":"https:\/\/www.garysieling.com\/blog\/nlp-analysis-using-modal-verbs\/","title":{"rendered":"NLP Analysis in Python using Modal Verbs"},"content":{"rendered":"

Modal verbs are auxiliary verbs which indicate semantic information about an action, i.e. likelihood (will, should) , permission (could, may), obligation (shall\/must). One interesting concept to explore is whether the presence of these verbs varies over different types of text, and whether that means anything.<\/p>\n

“Natural Language Processing with Python<\/a>” (read my review<\/a>) has an example of how to start this process, comparing verb frequencies across various genres of text using the Brown corpus, a well-known collection of texts assembled in the 60’s for language research.<\/p>\n

I extended the example to include an additional corpus of court cases, and extra helper verbs- This includes the contents of ~15,000 legal documents.<\/p>\n

We first define a function to retrieve genres of literature, and a second to retrieve words from the genre. For the legal documents, I am reading from an index I previously built of n-grams<\/a> (i.e. word\/phrase counts).<\/p>\n

\nimport nltk\nimport os\n\ndef get_genres():\nyield 'legal'\nfor genre in brown.categories():\nyield genre\n\nmodals = ['can', 'could', 'may', 'might', 'must', 'will', 'would', 'should']\n\ndef get_words(genre):\n  if (genre == 'legal'):\n    grams = open('1gram', 'rU')\n    for line in grams:\n      vals = line.split(' ')\n      word = vals[0]\n      count = int(vals[1])\n      if (word in modals):\n        for index in range(0, count):\n          yield word\n        else:\n          yield word\n    grams.close()\n  else:\n    for word in brown.words(categories=genre):\n      yield word\n<\/pre>\n
The Natural Language Toolkit provides a class for tracking frequencies of “experiment” results – here we track the use of different verb tenses.<\/p>\n
\ncfd = nltk.ConditionalFreqDist(\n  (genre, word)\n  for genre in get_genres()\n  for word in get_words(genre)\n)\n\ngenres = [g for g in get_genres()]\ncfd.tabulate(conditions=genres, samples=modals)\n\ncfd.tabulate(conditions=genres, samples=modals)\n<\/pre>\n
The tabulate method is provided by NTLK, and makes a nicely formatted chart (in a  command line it makes everything line up neatly)<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\ncan<\/td>\ncould<\/td>\nmay<\/td>\nmight<\/td>\nmust<\/td>\nwill<\/td>\nwould<\/td>\nshould<\/td>\n<\/th>\n
legal<\/td>\n13059<\/td>\n7849<\/td>\n26968<\/td>\n1762<\/td>\n15974<\/td>\n20757<\/td>\n19931<\/td>\n13916<\/td>\n<\/tr>\n
adventure<\/td>\n46<\/td>\n151<\/td>\n5<\/td>\n58<\/td>\n27<\/td>\n50<\/td>\n191<\/td>\n15<\/td>\n<\/tr>\n
belles_lettres<\/td>\n246<\/td>\n213<\/td>\n207<\/td>\n113<\/td>\n170<\/td>\n236<\/td>\n392<\/td>\n102<\/td>\n<\/tr>\n
editorial<\/td>\n121<\/td>\n56<\/td>\n74<\/td>\n39<\/td>\n53<\/td>\n233<\/td>\n180<\/td>\n88<\/td>\n<\/tr>\n
fiction<\/td>\n37<\/td>\n166<\/td>\n8<\/td>\n44<\/td>\n55<\/td>\n52<\/td>\n287<\/td>\n35<\/td>\n<\/tr>\n
government<\/td>\n117<\/td>\n38<\/td>\n153<\/td>\n13<\/td>\n102<\/td>\n244<\/td>\n120<\/td>\n112<\/td>\n<\/tr>\n
hobbies<\/td>\n268<\/td>\n58<\/td>\n131<\/td>\n22<\/td>\n83<\/td>\n264<\/td>\n78<\/td>\n73<\/td>\n<\/tr>\n
humor<\/td>\n16<\/td>\n30<\/td>\n8<\/td>\n8<\/td>\n9<\/td>\n13<\/td>\n56<\/td>\n7<\/td>\n<\/tr>\n
learned<\/td>\n365<\/td>\n159<\/td>\n324<\/td>\n128<\/td>\n202<\/td>\n340<\/td>\n319<\/td>\n171<\/td>\n<\/tr>\n
lore<\/td>\n170<\/td>\n141<\/td>\n165<\/td>\n49<\/td>\n96<\/td>\n175<\/td>\n186<\/td>\n76<\/td>\n<\/tr>\n
mystery<\/td>\n42<\/td>\n141<\/td>\n13<\/td>\n57<\/td>\n30<\/td>\n20<\/td>\n186<\/td>\n29<\/td>\n<\/tr>\n
news<\/td>\n93<\/td>\n86<\/td>\n66<\/td>\n38<\/td>\n50<\/td>\n389<\/td>\n244<\/td>\n59<\/td>\n<\/tr>\n
religion<\/td>\n82<\/td>\n59<\/td>\n78<\/td>\n12<\/td>\n54<\/td>\n71<\/td>\n68<\/td>\n45<\/td>\n<\/tr>\n
reviews<\/td>\n45<\/td>\n40<\/td>\n45<\/td>\n26<\/td>\n19<\/td>\n58<\/td>\n47<\/td>\n18<\/td>\n<\/tr>\n
romance<\/td>\n74<\/td>\n193<\/td>\n11<\/td>\n51<\/td>\n45<\/td>\n43<\/td>\n244<\/td>\n32<\/td>\n<\/tr>\n
science_fiction<\/td>\n16<\/td>\n49<\/td>\n4<\/td>\n12<\/td>\n8<\/td>\n16<\/td>\n79<\/td>\n3<\/td>\n<\/tr>\n<\/table>\n
Looking at these numbers, it is clear that we need to add a concept of normalization. My added corpus has a lot more tokens than the Brown corpus, which makes it hard to compare across. <\/p>\n
The frequency distribution class exists to count things, and I didn’t see a good way to normalize the rows. I re-wrote the tabulate function to do this – it simply finds the max for each row, divides by that, and multiplies by 100.<\/p>\n
\ndef tabulate(cfd, conditions, samples):\n  max_len = max(len(w) for w in conditions)\n  sys.stdout.write(\" \" * (max_len + 1))\n  for c in samples:\n    sys.stdout.write(\"%-s\\t\" % c)\n    sys.stdout.write(\"\\n\")\n  for c in conditions:\n    sys.stdout.write(\" \" * (max_len - len(c)))\n    sys.stdout.write(\"%-s\" % c)\n    sys.stdout.write(\" \")\n    dist = cfd[c]\n    norm = sum([dist[w] for w in modals])\n    for s in samples:\n      value = 100 * dist[s] \/ norm\n      sys.stdout.write(\"%-d\\t\" % value)\n      sys.stdout.write(\"\\n\")\n\ntabulate(cfd, genres, modals)\n<\/pre>\n
This makes it easier to scan up and down the chart-<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\ncan<\/td>\ncould<\/td>\nmay<\/td>\nmight<\/td>\nmust<\/td>\nwill<\/td>\nwould<\/td>\nshould<\/td>\n<\/th>\n
legal<\/td>\n10<\/td>\n6<\/td>\n22<\/td>\n1<\/td>\n13<\/td>\n17<\/td>\n16<\/td>\n11<\/td>\n<\/tr>\n
adventure<\/td>\n8<\/td>\n27<\/td>\n0<\/td>\n10<\/td>\n4<\/td>\n9<\/td>\n35<\/td>\n2<\/td>\n<\/tr>\n
belles_lettres<\/td>\n14<\/td>\n12<\/td>\n12<\/td>\n6<\/td>\n10<\/td>\n14<\/td>\n23<\/td>\n6<\/td>\n<\/tr>\n
editorial<\/td>\n14<\/td>\n6<\/td>\n8<\/td>\n4<\/td>\n6<\/td>\n27<\/td>\n21<\/td>\n10<\/td>\n<\/tr>\n
fiction<\/td>\n5<\/td>\n24<\/td>\n1<\/td>\n6<\/td>\n8<\/td>\n7<\/td>\n41<\/td>\n5<\/td>\n<\/tr>\n
government<\/td>\n13<\/td>\n4<\/td>\n17<\/td>\n1<\/td>\n11<\/td>\n27<\/td>\n13<\/td>\n12<\/td>\n<\/tr>\n
hobbies<\/td>\n27<\/td>\n5<\/td>\n13<\/td>\n2<\/td>\n8<\/td>\n27<\/td>\n7<\/td>\n7<\/td>\n<\/tr>\n
humor<\/td>\n10<\/td>\n20<\/td>\n5<\/td>\n5<\/td>\n6<\/td>\n8<\/td>\n38<\/td>\n4<\/td>\n<\/tr>\n
learned<\/td>\n18<\/td>\n7<\/td>\n16<\/td>\n6<\/td>\n10<\/td>\n16<\/td>\n15<\/td>\n8<\/td>\n<\/tr>\n
lore<\/td>\n16<\/td>\n13<\/td>\n15<\/td>\n4<\/td>\n9<\/td>\n16<\/td>\n17<\/td>\n7<\/td>\n<\/tr>\n
mystery<\/td>\n8<\/td>\n27<\/td>\n2<\/td>\n11<\/td>\n5<\/td>\n3<\/td>\n35<\/td>\n5<\/td>\n<\/tr>\n
news<\/td>\n9<\/td>\n8<\/td>\n6<\/td>\n3<\/td>\n4<\/td>\n37<\/td>\n23<\/td>\n5<\/td>\n<\/tr>\n
religion<\/td>\n17<\/td>\n12<\/td>\n16<\/td>\n2<\/td>\n11<\/td>\n15<\/td>\n14<\/td>\n9<\/td>\n<\/tr>\n
reviews<\/td>\n15<\/td>\n13<\/td>\n15<\/td>\n8<\/td>\n6<\/td>\n19<\/td>\n15<\/td>\n6<\/td>\n<\/tr>\n
romance<\/td>\n10<\/td>\n27<\/td>\n1<\/td>\n7<\/td>\n6<\/td>\n6<\/td>\n35<\/td>\n4<\/td>\n<\/tr>\n
science_fiction<\/td>\n8<\/td>\n26<\/td>\n2<\/td>\n6<\/td>\n4<\/td>\n8<\/td>\n42<\/td>\n1<\/td>\n<\/tr>\n
<\/td>\n<\/tr>\n<\/table>\n
One thing this makes clear is most genres have numerous references to ‘would’ and few have ‘should’.<\/p>\n
It might be nice to see these on a scale of 1-10 – seeing the columns of numbers communicates something in the lengths.<\/p>\n
\ndef tabulate(cfd, conditions, samples):\n  max_len = max(len(w) for w in conditions)\n  sys.stdout.write(\" \" * (max_len + 1))\n  for c in samples:\n    sys.stdout.write(\"%-s\\t\" % c)\n    sys.stdout.write(\"\\n\")\n    for c in conditions:\n      sys.stdout.write(\" \" * (max_len - len(c)))\n      sys.stdout.write(\"%-s\" % c)\n      sys.stdout.write(\" \")\n      dist = cfd[c]\n      norm = sum([dist[w] for w in modals])\n  for s in samples:\n    value = 10 * float(dist[s]) \/ norm\n    sys.stdout.write(\"%.1f\\t\" % value)\n    sys.stdout.write(\"\\n\")\n\ntabulate(cfd, genres, modals)\n<\/pre>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\ncan<\/td>\ncould<\/td>\nmay<\/td>\nmight<\/td>\nmust<\/td>\nwill<\/td>\nwould<\/td>\nshould<\/td>\n<\/th>\n
legal<\/td>\n1.1<\/td>\n0.7<\/td>\n2.2<\/td>\n0.1<\/td>\n1.3<\/td>\n1.7<\/td>\n1.7<\/td>\n1.2<\/td>\n<\/tr>\n
adventure<\/td>\n0.8<\/td>\n2.8<\/td>\n0.1<\/td>\n1.1<\/td>\n0.5<\/td>\n0.9<\/td>\n3.5<\/td>\n0.3<\/td>\n<\/tr>\n
belles_lettres<\/td>\n1.5<\/td>\n1.3<\/td>\n1.2<\/td>\n0.7<\/td>\n1.0<\/td>\n1.4<\/td>\n2.3<\/td>\n0.6<\/td>\n<\/tr>\n
editorial<\/td>\n1.4<\/td>\n0.7<\/td>\n0.9<\/td>\n0.5<\/td>\n0.6<\/td>\n2.8<\/td>\n2.1<\/td>\n1.0<\/td>\n<\/tr>\n
fiction<\/td>\n0.5<\/td>\n2.4<\/td>\n0.1<\/td>\n0.6<\/td>\n0.8<\/td>\n0.8<\/td>\n4.2<\/td>\n0.5<\/td>\n<\/tr>\n
government<\/td>\n1.3<\/td>\n0.4<\/td>\n1.7<\/td>\n0.1<\/td>\n1.1<\/td>\n2.7<\/td>\n1.3<\/td>\n1.2<\/td>\n<\/tr>\n
hobbies<\/td>\n2.7<\/td>\n0.6<\/td>\n1.3<\/td>\n0.2<\/td>\n0.8<\/td>\n2.7<\/td>\n0.8<\/td>\n0.7<\/td>\n<\/tr>\n
humor<\/td>\n1.1<\/td>\n2.0<\/td>\n0.5<\/td>\n0.5<\/td>\n0.6<\/td>\n0.9<\/td>\n3.8<\/td>\n0.5<\/td>\n<\/tr>\n
learned<\/td>\n1.8<\/td>\n0.8<\/td>\n1.6<\/td>\n0.6<\/td>\n1.0<\/td>\n1.7<\/td>\n1.6<\/td>\n0.9<\/td>\n<\/tr>\n
lore<\/td>\n1.6<\/td>\n1.3<\/td>\n1.6<\/td>\n0.5<\/td>\n0.9<\/td>\n1.7<\/td>\n1.8<\/td>\n0.7<\/td>\n<\/tr>\n
mystery<\/td>\n0.8<\/td>\n2.7<\/td>\n0.3<\/td>\n1.1<\/td>\n0.6<\/td>\n0.4<\/td>\n3.6<\/td>\n0.6<\/td>\n<\/tr>\n
news<\/td>\n0.9<\/td>\n0.8<\/td>\n0.6<\/td>\n0.4<\/td>\n0.5<\/td>\n3.8<\/td>\n2.4<\/td>\n0.6<\/td>\n<\/tr>\n
religion<\/td>\n1.7<\/td>\n1.3<\/td>\n1.7<\/td>\n0.3<\/td>\n1.2<\/td>\n1.5<\/td>\n1.4<\/td>\n1.0<\/td>\n<\/tr>\n
reviews<\/td>\n1.5<\/td>\n1.3<\/td>\n1.5<\/td>\n0.9<\/td>\n0.6<\/td>\n1.9<\/td>\n1.6<\/td>\n0.6<\/td>\n<\/tr>\n
romance<\/td>\n1.1<\/td>\n2.8<\/td>\n0.2<\/td>\n0.7<\/td>\n0.6<\/td>\n0.6<\/td>\n3.5<\/td>\n0.5<\/td>\n<\/tr>\n
science_fiction<\/td>\n0.9<\/td>\n2.6<\/td>\n0.2<\/td>\n0.6<\/td>\n0.4<\/td>\n0.9<\/td>\n4.2<\/td>\n0.2<\/td>\n<\/tr>\n
<\/td>\n<\/tr>\n<\/table>\n
It would be nice to see how similar these genres are – we can compute that by imagining the counts of modals as describing vectors. The angle between vectors approximates “similarity”. The nice thing about this measure is that it removes other words (words which may only exist in one text – some of this will be due to how well the data is cleaned, which does not reflect on the genre of literature).<\/p>\n
\nimport math\n\ndef distance(cfd, conditions, samples, base):\n  base_cond = cfd[base]\n  base_vector = [base_cond[w] for w in samples]\n  base_length = math.sqrt(sum(a * a for a in base_vector))\n  for c in conditions:\n    cond = cfd[c]\n    cond_vector = [cond[w] for w in samples]\n    dotp = sum(a * b for (a,b) in zip(base_vector, cond_vector))\n    cond_length = math.sqrt(sum(a * a for a in cond_vector))\n    angle = math.acos(dotp \/ (cond_length * base_length))\n    percent = (math.pi \/ 2 - angle) \/ (math.pi \/ 2) * 100\n    print \"%-s similarity to %-s: %-.1f\" % (c, base, percent)\n<\/pre>\n
The result are interesting – the genres showing closes to legal in this case are  government and religion. <\/p>\n
As an interesting side-note, belles_lettres means “fine writing”, i.e. poems, drama, fiction. <\/p>\n
\nlegal similarity to legal: 100.0\nadventure similarity to legal: 41.6\nbelles_lettres similarity to legal: 72.4\neditorial similarity to legal: 68.8\nfiction similarity to legal: 42.9\ngovernment similarity to legal: 80.6\nhobbies similarity to legal: 63.5\nhumor similarity to legal: 50.1\nlearned similarity to legal: 80.6\nlore similarity to legal: 78.6\nmystery similarity to legal: 41.3\nnews similarity to legal: 58.1\nreligion similarity to legal: 81.2\nreviews similarity to legal: 73.5\nromance similarity to legal: 42.9\nscience_fiction similarity to legal: 41.8\n<\/pre>\n
Some genres appear similar to legal documents – it is possible, however, that some verbs are not independent. For instance, you might see “may” and “might” with equal similarity. One way to test this might be to flip what we track for distance (make a vector for each modal, rather than genre)<\/p>\n
The following code tracks the distance between each modal and the mean, using the different genres as dimensions. Since each of them contributes to the mean somewhat, there is guaranteed to be some similarity, but note that some are closer than others. Note also that these have to be normalized, like the last example, or the answer will be defined by the ‘legal’ genre. <\/p>\n
\ndef distance(cfd, conditions, samples):\n  base_vector = [0.0 for w in conditions]\n  norm = {}\n  for c_i in range(0, len(conditions)):\n    cond_name = conditions[c_i]\n    cond = cfd[cond_name]\n    norm[cond_name] = float(sum(cond[s] for s in samples))\n    for s in samples:\n      base_vector[c_i] = base_vector[c_i] + float(cond[s]) \/ norm[cond_name]\n      base_length = math.sqrt(sum(a * a for a in base_vector))\n  for s in samples: # compute each vector - which, might, etc\n    sample_vector = []\n    for c in conditions: # find condition for each vector\n      sample_vector.append(cfd[c][s] \/ norm[c])\n      dotp = sum(a * b for (a,b) in zip(base_vector, sample_vector))\n      sample_length = math.sqrt(sum(a * a for a in sample_vector))\n      angle = math.acos(dotp \/ (sample_length * base_length))\n      percent = (math.pi \/ 2 - angle) \/ (math.pi \/ 2) * 100\n      print \"%-s similarity to mean: %-.1f\" % (s, percent)\n\ndistance(cfd, genres, modals)\n<\/pre>\n
What I’d infer from this is that the least helpful verb for distinguishing genres is “must,” and the most helpful is “may.”<\/p>\n
\ncan similarity to mean: 76.0\ncould similarity to mean: 67.6\nmay similarity to mean: 61.5\nmight similarity to mean: 70.0\nmust similarity to mean: 79.7\nwill similarity to mean: 67.7\nwould similarity to mean: 73.6\nshould similarity to mean: 74.2\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
Modal verbs are auxiliary verbs which indicate semantic information about an action, i.e. likelihood (will, should) , permission (could, may), obligation (shall\/must). One interesting concept to explore is whether the presence of these verbs varies over different types of text, and whether that means anything. “Natural Language Processing with Python” (read my review) has an … <\/p>\n
Continue reading “NLP Analysis in Python using Modal Verbs”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[5,6],"tags":[368,385,386,447,567],"aioseo_notices":[],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/1244"}],"collection":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/comments?post=1244"}],"version-history":[{"count":0,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/1244\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/media?parent=1244"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/categories?post=1244"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/tags?post=1244"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}