Building an Directory Structure Index in Python

I’m working through examples in “Natural Language Processing with Python” (read my review) and found that the corpus I have to work with is large enough to require special performance tuning exercises.

If you have a large enough directory structure, it becomes difficult to walk with os.walk – for instance any failure in longer scripts require starting from scratch. This is a common issue in larger systems – typically they manage file listings through a relational database, and directory storage is obfuscated in some way.

In this environment it takes at an hour for Windows to count the files, and python seems to take longer.

It’s worth generating a list of the files in advance – this lists which PDFs and HTML documents exist, and for which a text extract has been generated (see a Node.JS approach here). This supports a few uses, including generating missing text renditions.

import os
import re
import datetime
pdf_idx = open('pdfs.idx', 'w')
rendition_idx = open('txts.idx', 'w')
html_idx = open('htmls.idx', 'w')
xml_idx = open('xmls.idx', 'w')
for root, dirs, files in os.walk('.'):
  for f in files:
    if f.endswith(".pdf"):
      rend = root + os.sep + f + ".textrendition.txt"
          with open(rend):
              rendition_idx.write(rend + "\n")
      except IOError:
      pdf_idx.write(root + os.sep + f + "\n")
    if f.endswith(".html") or f.endswith('.htm'):
      html_idx.write(root + os.sep + f + "\n")
    if f.endswith(".xml"):
      xml_idx.write(root + os.sep + f + "\n")      

What this allows is quite useful – you can read the file quickly, and select a random subset for training and test data for NLP algorithms. This could also be done by storing all the names in a database, but this is probably the simplest and fastest for my current needs.

rendition_idx = open('txts1.idx', 'r')
files = [f[:-1] for f in rendition_idx]
>>> files[:4]
import random
contents = [open(f[2:]).read() for f in files[:100]]

Tags: , ,


#1 Eli on 07.11.13 at 2:04 pm

Instead of:

root + os.sep + f + “\n”

Use os.path.join

Also, use context managers for opening files instead of closing manually.

#2 bitcycle on 07.12.13 at 9:38 pm

Title should be “Building a directory structure index in Python”.

Leave a Comment