Wikipedia has extensive text on articles it discusses – in some cases so much that a lot of language processing APIs won’t accept it. Alchemy API (now seemingly marketed as “IBM Watson”) has an endpoint to parse text from a website, but it only accepts 600KB pages (50K of output text). Consequently, it quickly becomes easier to just get the text yourself.<\/p>\n
To do this, I recommend Apache Tika, which seems to include one of the better \/ best libraries for extracting text, and has every imaginable interface – Java, command line, REST, and a GUI(!). <\/p>\n
You only need a Java jar for this-<\/p>\n
\ncurl http:\/\/apache.spinellicreations.com\/tika\/tika-app-1.13.jar > tika-app.jar\n<\/pre>\nTika has a complex set of options for detecting content types1<\/a><\/sup>, but it seems to respond to file extensions, and when I was testing this I found that it was more reliable when I specified these:<\/p>\n\ncurl https:\/\/en.wikipedia.org\/wiki\/Barack_Obama > Barack_Obama.html\n<\/pre>\nInvoking Tika is simple:<\/p>\n
\njava -jar tika-app.jar -t data\/$1.html > out\/$1.txt\n<\/pre>\nProblem is, Wikipedia has a ton of extra content wrapping the text. You could handle this in a few ways – pre-process the file to select out what you want, customize Tika to have it parse out the bits you want (probably a good option if you want to get just captions or headings), or hack at the output.<\/p>\n
For my case I chose the last option. The following script will remove the table of contents, most captions, and the bogus header \/ footer information that shows up at the end of the file. Tune to your liking (I removed the references as well).<\/p>\n
\nimport fileinput\nimport re\n\nstart = re.compile(\"Jump to:.*navigation,.*search\")\n \nend = re.compile(\"^Notes and references$\")\n\nstarted = False\nended = False\nblank = False\n\nignore = re.compile(\n \"^(Main article: .*|\" +\n \"Main articles: .*|\" + \n \"See also: .*|\" + \n \"\\s*[0-9]+.[0-9]+ .*\" + \n \"|\\s*[0-9]+.[0-9]+.[0-9]+ .*)\\s*$\")\n\nfootnote = re.compile(\"\\[[0-9]+\\]\")\n\nfor line in fileinput.input():\n\n if (re.match(end, line.strip())):\n ended = True\n\n if (started and not ended):\n if (not blank or line.strip() != \"\"):\n if (not re.match(ignore, line)):\n if (\".\" in line or len(line) > 150 or len(line.strip()) == 0):\n print(re.sub(footnote, \"\", line.strip()))\n\n if (line.strip() == \"\"):\n blank = True\n else:\n blank = False\n\n if (not started):\n if (re.search(start, line) != None):\n started = True\n\n pass\n<\/pre>\n- https:\/\/tika.apache.org\/1.1\/detection.html#Content_Detection [↩<\/a>]<\/span><\/li><\/ol>","protected":false},"excerpt":{"rendered":"
How to extract text content from Wikipedia articles programmatically<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"footnotes":""},"categories":[12],"tags":[300,447],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t