The U.S. Congress recently released a series of XML documents containing U.S. Laws. The structure of these documents allow us to find which sections of the law are most commonly cited. Examining which citations occur most frequently allows us to see what Congress has spent the most time thinking about.<\/p>\n
Citations occur for many reasons: a justification for addition or omission in subsequent laws, clarifications, or amendments, or repeals. As we might expect, the most commonly cited sections involve the IRS (Income Taxes, specifically), Social Security, and Military Procurement.<\/p>\n
To arrive at this result, we must first see how U.S. Code is laid out. The laws are divided into a hierarchy of units, which allows anything from an entire title to individual sentences to cited. These sections have an ID and an identifier – “identifier” is used an an citation reference within the XML documents, and has a different form from the citations used by the legal community, comes in a form like “25 USC Chapter 21 \u00a7\u20091901”.<\/p>\n
The XML hierarchy defines seventeen different levels which can be cited: ‘title’, ‘subtitle’, ‘chapter’, ‘subchapter’, ‘part’, ‘subpart’, ‘division’, ‘subdivision’, ‘article’, ‘subarticle’, ‘section’, ‘subsection’, ‘paragraph’, ‘subparagraph’, ‘clause’, ‘subclause’, and ‘item’.<\/p>\n
We can use a simple XPath expression to retrieve one of these, like section:<\/p>\n
\u00a7?104.<\/num>\n Federal Highway Administration<\/heading>\n \n (a)<\/num>\n The Federal Highway Administration is an administration\nin the Department of Transportation.<\/content>\n<\/subsection><\/pre>\n A portion of the human readable citation is contained in “num”. In order to retrieve a citation that a lawyer would recognize, we need to look at “num” for the parent element as well.<\/p>\n
from elementtree import ElementTree as ET\nimport os\n\ndir = \"G:\\\\us_code\\\\xml_uscAll@113-21\"\n\ndef getParent(parent_map, elt, idx):\n try:\n parent = elt\n for i in range(idx):\n parent = parent_map.get(parent)\n \n return \\\n parent.findall('{http:\/\/xml.house.gov\/schemas\/uslm\/1.0}num')[0].text + \n ' ' + \n parent.findall('{http:\/\/xml.house.gov\/schemas\/uslm\/1.0}heading')[0].text\n except:\n return \"--No Heading--\"\n<\/pre>\nOnce we find the parent, we need to traverse all the way up the tree:<\/p>\n
def getTree(parent_map, t):\n tree = []\n parent = \"\"\n idx = 0\n while (parent != \"--No Heading--\"):\n parent = getParent(parent_map, t, idx)\n tree.append(parent)\n idx += 1\n return tree\n\nusc26.xml: Title 26\u2014 Subtitle A\u2014 CHAPTER 1\u2014 \n<\/pre>\nThis forms the basis for a function which builds a citation index – a list of every XML node that can be used in a citation, along with it’s human-readable citation and name. This takes some time, so if you reproduce this effort, you may want to save the results to a file.<\/p>\n
dir = \"G:\\\\us_code\\\\xml_uscAll@113-21\"\nurls = {}\n\ndef findElements(xpath, urls):\n for root, dirs, files in os.walk(dir):\n for f in files:\n if f.endswith('.xml'):\n tree = ET.parse(dir + \"\\\\\" + f)\n parent_map = dict((c, p) for p in tree.getiterator() for c in p)\n sections = tree.findall(xpath)\n for t in sections: \n urls[t.attrib.get('identifier')] = \\\n (t.attrib.get('id'), \n getTree(parent_map, t),\n f)\n\nrefs = {}\nrefTypes = ['title', 'subtitle', 'chapter', \\\n 'subchapter', 'part', 'subpart', 'division', \\\n 'subdivision', 'article', 'subarticle', 'section', \\\n 'subsection', 'paragraph', 'subparagraph', 'clause', \\\n 'subclause', 'item']\n\nfor ref in refTypes:\n findElements('.\/\/{http:\/\/xml.house.gov\/schemas\/uslm\/1.0}' + ref, refs)\n\nrefs.items()[20]\n('\/us\/usc\/t2\/s2102\/b',\n ('id8a923648-f59b-11e2-8dfe-b6d89e949a2c',\n ['(b) Issuance and publication of regulations',\n u'\\xa7\\u202f2102. Duties of Commission',\n u'Part B\\u2014 Senate Commission on Art',\n u'SUBCHAPTER V\\u2014 HISTORICAL PRESERVATION AND FINE ARTS',\n u'CHAPTER 30\\u2014 OPERATION AND MAINTENANCE OF CAPITOL COMPLEX',\n u'Title 2\\u2014 THE CONGRESS',\n '--No Heading--'],\n 'usc02.xml'))\n<\/pre>\nNow that we know how to look up a citation we need to find the actual citations. Like HTML, the U.S. code documents use the “a href=” tag to reference a node, as well as “ref href=”. The same XPath technique used above allows us to find refs:<\/p>\n
hrefs = {}\ntitles = {}\nrefpath = '.\/\/{http:\/\/xml.house.gov\/schemas\/uslm\/1.0}ref'\nfor root, dirs, files in os.walk(dir):\n for f in files:\n if f.endswith('.xml'):\n tree = ET.parse(dir + \"\\\\\" + f)\n root = tree.getroot()\n h = {t.attrib.get('href'): f + ' ' + t.text \\\n for t in tree.findall(refpath)}\n hrefs = dict(hrefs.items() + h.items())\n\n\nhrefs.items()[0]\nOut[55]: \n('\/us\/pl\/109\/280\/s601\/a\/3',\n u'usc29.xml Pub. L. 109\\u2013280, title VI, \\xa7\\u202f601(a)(3)')\n<\/pre>\nWe have everything we need to find which sections are commonly cited, we just need to combine them. Most of the complexity here is dealing with missing entries (e.g. due to the fact that a citation can point anywhere in the hierarchy).<\/p>\n
from collections import Counter\n\ndef countCitations(urls, hrefs):\n titles = Counter()\n subtitles = Counter()\n chapters = Counter()\n not_found = []\n for key in hrefs.keys():\n found = urls.get(key)\n \n title = \"None\"\n subtitle = \"None\"\n chapter = \"None\"\n file = \"None\" \n\n if (found != None):\n (id, history, file) = found\n if len(history) >= 2:\n title = history[-2]\n if len(history) >= 3:\n subtitle = history[-3]\n if len(history) >= 4:\n chapter = history[-4] \n else:\n not_found.append(key)\n \n titles[file + \": \" + title] += 1 \n subtitles[file + \": \" + title + \" - \" + subtitle] += 1 \n chapters[file + \": \" + title + \" - \" + subtitle + \" - \" + chapter] += 1 \n return (titles, subtitles, chapters, not_found)\n\n(t, s, c, none) = countCitations(refs, hrefs)\n<\/pre>\nThis returns results that are rolled up to titles, subtitles, and chapters. In particular note how as we drill down, the results provide clarity as to what was most important in the priort section. Within “The Public Health and Welfare,” we see that Social Security is important, and within “Armed Forces,” we see that “General Military Law – Personnel” is important.<\/p>\n
None: None: 359662\nusc42.xml: Title 42\u2014 THE PUBLIC HEALTH AND WELFARE: 6679\nusc10.xml: Title 10\u2014 ARMED FORCES: 2078\nusc16.xml: Title 16\u2014 CONSERVATION: 2068\nusc42.xml: None: 1965\nusc15.xml: Title 15\u2014 COMMERCE AND TRADE: 1796\nusc07.xml: Title 7\u2014 AGRICULTURE: 1689\nusc22.xml: Title 22\u2014 FOREIGN RELATIONS AND INTERCOURSE: 1684\nusc20.xml: Title 20\u2014 EDUCATION: 1660\nusc26.xml: Title 26\u2014 INTERNAL REVENUE CODE: 1610\n<\/pre>\nNone: None - None: 359662\nusc42.xml: None - None: 1965\nusc42.xml: Title 42\u2014 THE PUBLIC HEALTH AND WELFARE - CHAPTER 7\u2014 SOCIAL SECURITY: 1573\nusc10.xml: Title 10\u2014 ARMED FORCES - Subtitle A\u2014 General Military Law: 1490\nusc42.xml: Title 42\u2014 THE PUBLIC HEALTH AND WELFARE - CHAPTER 6A\u2014 PUBLIC HEALTH SERVICE: 1220\nusc26.xml: Title 26\u2014 INTERNAL REVENUE CODE - Subtitle A\u2014 Income Taxes: 841\nusc05.xml: None - None: 736\nusc10.xml: None - None: 639\nusc16.xml: Title 16\u2014 CONSERVATION - CHAPTER 1\u2014 NATIONAL PARKS, MILITARY PARKS, MONUMENTS, AND SEASHORES: 616\nusc20.xml: Title 20\u2014 EDUCATION - CHAPTER 70\u2014 STRENGTHENING AND IMPROVEMENT OF ELEMENTARY AND SECONDARY SCHOOLS: 531\n<\/pre>\nNone: None - None - None: 359662\nusc42.xml: None - None - None: 1965\nusc26.xml: Title 26\u2014 INTERNAL REVENUE CODE - Subtitle A\u2014 Income Taxes - CHAPTER 1\u2014 NORMAL TAXES AND SURTAXES: 817\nusc10.xml: Title 10\u2014 ARMED FORCES - Subtitle A\u2014 General Military Law - PART II\u2014 PERSONNEL: 738\nusc05.xml: None - None - None: 736\nusc42.xml: Title 42\u2014 THE PUBLIC HEALTH AND WELFARE - CHAPTER 7\u2014 SOCIAL SECURITY - SUBCHAPTER XVIII\u2014 HEALTH INSURANCE FOR AGED AND DISABLED: 663\nusc10.xml: None - None - None: 639\nusc38.xml: None - None - None: 497\nusc10.xml: Title 10\u2014 ARMED FORCES - Subtitle A\u2014 General Military Law - PART IV\u2014 SERVICE, SUPPLY, AND PROCUREMENT: 496\nusc15.xml: None - None - None: 428\n<\/pre>\nFuture work in this area will involve cleaning up the results to remove some of the “None” entries, building a visualization of the results, and training a tagger to recognize the human-readable versions of citation in court documents. In the long run, I hope these developments help make legal information more accessible to everyone, rather than being locked up in expensive databases.<\/p>\n","protected":false},"excerpt":{"rendered":"
The U.S. Congress recently released a series of XML documents containing U.S. Laws. The structure of these documents allow us to find which sections of the law are most commonly cited. Examining which citations occur most frequently allows us to see what Congress has spent the most time thinking about. Citations occur for many reasons: … <\/p>\n