This query implies that we should exclude or de-rank documents with phrases like \u201cwriting css\u201d or \u201cwriting php\u201d, preferring results with “poetry”, “fiction”, or \u201ccopyediting.\u201d <\/p>\n
This scenario is a ticket I received from a user of www.findlectures.com<\/a>. People want ways to research the latest developments in their field, without search results being cluttered with talks that mention their interests in passing.<\/p>\n For a more complex example, lets say we were searching recipes:<\/p>\n “Vegan cooking” would be an appropriate result for this – as would be recipes for people with food allergies.<\/p>\n The negative search for dairy implies a hierarchy of items we wish to remove: milk, cheese and any specific types, components, or brands of these (e.g. Parmesan, whey, Kraft, respectively). It’s unlikely that the searcher has thought this far into what they want, but implicitly expect it.<\/p>\n For a final example:<\/p>\n This search implies a geography and time range. This article about the founding of Arden, Delaware<\/a>, would be an appropriate match because it is nearby.<\/p>\n The notion of a concept hierarchy brings the WordNet1<\/a><\/sup> database to mind, although it is not sufficiently detailed for our purposes. Let’s suppose you wanted to search for talks, excluding politics and religion. If you start at “religion” and work down the tree, you about 200 results:<\/p>\n This list is pretty good, but far too short – Wikipedia’s list of Christian sects includes nearly 200 entries just under “types of Baptist”2<\/a><\/sup>. <\/p>\n The tree of concepts for politics has nearly nothing. Of the results it returns, none of these are what people mean by “no politics”. A political talk is determined by time – whatever is in the news now, where I’m at, is political today, and history tomorrow.<\/p>\n Machine learning techniques are a compelling alternative to using a database maintained by a team, because you can rely on a computer to find patterns, and update your model as new text becomes available. Word embeddings3<\/a><\/sup> are a compelling tool, Word2vec can discover implicit relationships, such as gender or country capitals. Varations of Word2vec represent word meanings from how they are used in context, using mathematical vectors.<\/p>\n Much of the practical research I’ve found on Word2Vec and search uses it to generate synonyms. There are a few papers suggesting that Word2Vec can discover other types of relationship4<\/a><\/sup> (e.g. more general\/specific terms). In a Word2vec model trained on English the “nearest” term to a noun is often the plural of the term, since those two terms are often used together. I expect this would be different in a language with cases (German, Greek, Latin), as related verbs and adjectives in change their endings to match.<\/p>\n I’m using concept search to generate personalized email\u00a0digests<\/a> resembling the excellent Cooper Press newsletters (Javascript Weekly<\/a>, Postgres Weekly<\/a>, etc). Each email has unique articles from Reddit and conference talks from FindLectures.com<\/a>, chosen with machine learning.<\/p>\n Concept search shines for users who enter multiple search terms. For “python” and “machine learning”, we really want to see pieces about scikit-learn, Tensorflow, and Keras. If we enter “java” and “machine learning”, we instead expect to see work by people using Stanford NLP or Deeplearning4j.<\/p>\n Traditional full text search tools (Lucene) query for the presence or absence of terms, weighted by how often they occur. A variation of this is the simple and fast Rocchio Algorithm5<\/a><\/sup>. The Rocchio Algorithm essentially does the following:<\/p>\n This improves the quality of results, and it’s very fast. There is an excellent talk on this6<\/a><\/sup> by Simon Hughes \/ Dice.com, who has a Solr plugin that implements the algorithm7<\/a><\/sup>.<\/p>\n Here are the results when we query for articles submitted to Reddit on Python + Machine Learning:<\/p>\n The Rocchio algorithm does a good job here, but I suspect that Word2Vec can do better because it maintains a concept of how similar two terms are.<\/p>\n Understanding how Word2Vec defines similarity is foundational to the work we want to do with concepts: The distance between two term vectors is the cosine of the angle between them. This produces a score from 0 to 1. Like full text scores, higher is better, and it’s not mathematically valid to add the scores.<\/p>\n The dataset for FindLectures.com<\/a> includes machine transcriptions for most talks. When I ran Word2vec on this dataset it identified the nearest word to “code” as “coat” – a logical mistake for a machine. This suggests that Word2vec trained on articles could be used to improve machine transcription by incorporating the probability of a term in real usage. This paper Let’s consider a simple change to the Rocchio algorithm: use synonyms suggested by Word2vec, but incorporate the distance from the source query to the synonym as a weighting factor. <\/p>\n This code uses the angle between terms as a boost. The following text shows the resulting Solr query looks like, The caret(^) is a boost. These boosts are multiplied by the weights Solr maintains internally (BM25)9<\/a><\/sup>. This accounts for how often a term occurs – really common synonyms will have essentially no effect on the output.<\/p>\n Here are the results for Python + Machine Learning:<\/p>\n Here are the results for just “writing”:<\/p>\n These results look really good – we’ve removed all the “code” oriented results. <\/p>\n In my first implementation I used the cosine similarity as the boost, rather than the angle it represented. This is almost as good, but it returned an article titled: “Re-writing, Re-reading, Re-thinking \u2013 Web Design in Words”. Clearly this has the correct terms, but is not actually about the topic we’re looking for.<\/p>\n To improve these results further, I’d like to measure whether the document is “about” the query terms. An easy way to do this is to average all the terms in the query and document, and compute the cosine similarity Graphically, we’re comparing the “average” for the document to the “average” for the query:<\/p>\n This is not fast – computing all of these term vectors and averaging them took 5 minutes 45 seconds on my machine with 16 parallel threads. This could be significantly improved by pre-computing the averages for each document and running on a GPU.<\/p>\n Here are the new results for “writing” (the previous result, re-sorted by “aboutness”):<\/p>\n Again, these look good.<\/p>\n Some People who set up email alerts enter all of their interests (Art, Hiking), and some enter terms that modify each other (Python, Programming). We need to identify whether each term is related so that we can choose between “AND” and “OR” in the queries we generate.<\/p>\n A simple approach to this is to compute the distance between each query term and segment them into clusters. <\/p>\n Here are the distances for our example:<\/p>\n While we’re now getting good results we often see articles on the same subject. For an email list dedicated to learning, it’s no use giving people the same article written multiple times.<\/p>\n In one email a user who chose “writing” as a topic got these two talks back:<\/p>\n Even worse, an alert for Python returned these results, which are all re-written versions of the same article:<\/p>\n Improving the diversity of search results is a fascinating problem. If this was an e-commerce site, producing varied results for a broad search like “shoes” gives the users hints about what a site has to offer, as well as prompts to refine the search. <\/p>\n Solr can do k-means clustering of documents around phrases it discovers11<\/a><\/sup> using Carrot212<\/a><\/sup> – we could improve diversity by choosing a talk from each cluster.<\/p>\n The clusters I get for “transcript:python” are as follows:<\/p>\n Each of these clusters has a list of talks within it. Here is what we get if we pick a talk per cluster:<\/p>\n This approach is really fast, but difficult to combine with the prior techniques (pick N results, reshuffle).<\/p>\n A Word2vec alternative could do this: Pick the top result, then find least related result. Average these two, then find the next most unrelated talk.<\/p>\n We’re doing the following comparison iteratively: Again this was a bit slow – 1 min, 30 seconds @ 16 parallel threads.<\/p>\n To prove this works I changed the query to “python and pandas” to make it harder – that guarantees the original three articles show up.<\/p>\n In general, each technique builds upon the last by obtaining the top results and re-shuffling. Adding more computing resources improves more relevance, but it takes some time to build and retrain Word2vec. In the next iteration of this, I’m intending to explore generating sequences of talks that build on each other, as well as dealing with geography (“history of philadelphia”).<\/p>\n Exploring the issues around concept search<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[10],"tags":[121,185,348,385,517,599],"aioseo_notices":[],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/5515"}],"collection":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/comments?post=5515"}],"version-history":[{"count":0,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/5515\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/media?parent=5515"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/categories?post=5515"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/tags?post=5515"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Vegetarian Food NOT Dairy \n<\/pre>\n
\nPhiladelphia, History\n<\/pre>\n
Wordnet<\/h2>\n
'religion', 'faith', 'organized religion', 'taoism', 'shinto', 'sect', 'religious sect',\n'religious order', 'zurvanism', 'waldenses', 'vaudois', 'vaishnavism', 'vaisnavism', 'sunni',\n'sunni islam', 'sisterhood', 'shuha shinto', 'shua', 'shivaism', 'sivaism', 'shiah', 'shia',\n'shiah islam', 'shaktism', 'saktism', 'shakers', \n'united society of believers in christ\\'s second appearing', 'religious society of friends', \n'society of friends', 'quakers', 'order', 'monastic order', 'society of jesus', 'jesuit order', \n'franciscan order', 'dominican order', 'carthusian order', 'carmelite order', \n'order of our lady of mount carmel', 'benedictine order', 'order of saint benedict', \n'augustinian order', 'austin friars', 'augustinian hermits', 'augustinian canons', \n'kokka shinto', 'kokka', 'karaites', 'jainism', 'high church', 'high anglican church', \n'haredi', 'hare krishna', 'international society for krishna consciousness', 'iskcon', \n'brethren', 'amish sect', 'albigenses', 'cathars', 'cathari', 'abecedarian', 'scientology', \n'church of scientology', 'khalsa', 'judaism', 'hebraism', 'jewish religion', 'reform judaism', \n'orthodox judaism', 'jewish orthodoxy', 'hasidim', 'hassidim', 'hasidism', 'chasidim', \n'chassidim', 'conservative judaism', 'hinduism', 'hindooism', 'brahmanism', 'brahminism', \n'established church', 'cult', 'cargo cult', 'wicca', 'voodoo', 'rastafarian', 'rastafari', \n'rastas', 'obeah', 'obi', 'macumba', 'church', 'christian church', 'unification church', \n'protestant church', 'protestant', 'pentecostal religion', 'nestorian church', 'coptic church', \n'catholic church', ...142 more]\n<\/pre>\n
\n'political sympathies', 'political science', 'government', 'realpolitik', 'practical politics', \n'geopolitics', 'geostrategy', 'political relation'\n<\/pre>\n
Word2Vec<\/h2>\n
![\"image_4\"](\"http:\/\/www.garysieling.com\/blog\/wp-content\/uploads\/2017\/09\/image_4-300x140.png\")
<\/p>\nUse Cases<\/h2>\n
![\"email2\"](\"http:\/\/172.104.26.128\/wp-content\/uploads\/2017\/10\/email2-1.png\")
<\/p>\n![\"email1\"](\"http:\/\/172.104.26.128\/wp-content\/uploads\/2017\/10\/email1-1.png\")
<\/p>\nRocchio Algorithm<\/h2>\n
\n
Using the scikit-learn machine learning library in Ruby using PyCall\nPythons Positive Press Pumps Pandas\nImage Recognition with Python, Clarifai and Twilio\nDeepSchool.io Open Source Deep Learning course\nKeep it simple! How to understand Gradient Descent algorithm\nML-From-Scratch: Library of bare bones Python implementations of Machine Learning models and algorithms\nEpoch vs Batch Size vs Iterations: Machine Learning\n<\/pre>\n
![\"cosine\"](\"http:\/\/www.garysieling.com\/blog\/wp-content\/uploads\/2017\/10\/cosine-300x187.png\")
<\/p>\nDataset<\/h2>\n
\n8<\/a><\/sup> shows a promising example, using French language newspaper text to improve transcription of broadcasts.<\/p>\nSynonyms<\/h2>\n
\nList(\"python\", \"machine\", \"learning\").map(\n (queryTerm) =>\n \"(\" +\n model.wordsNearest(\n List(queryTerm), \/\/ positive terms\n List(), \/\/ negative terms\n 25\n ).map(\n (nearWord) =>\n \"transcript:\" + term2 +\n \"^\" + \n (1 + \n (Math.PI - \n Math.acos(\n model.similarity(nearWord, term2))))\n ).mkString(\" OR \") \n + \")\"\n).mkString(\" AND \")\n<\/pre>\n
\ntitle_s:python^10 OR title_s:\"machine learning\"^10 \u2026\n(title_s: software^1.21 OR title_s:database^1.20 OR title_s:format^1.18 \ntitle_s:applications^1.14 OR title_s:browser^1.14 OR title_s:setup^1.13 \ntitle_s:bootstrap^1.13 OR title_s:in-class^1.13 OR title_s:campesina^1.12 OR \ntitle_s:excel^1.12 OR title_s:hardware^1.11 OR title_s:programming^1.11 OR\ntitle_s:api^1.11 OR title_s:prototype^1.11 OR title_s:middleware^1.11 OR \ntitle_s:openstreetmap^1.10 OR title_s:product^1.10 OR title_s:app^1.09 OR \ntitle_s:hbp^1.09 OR title_s:programmers^1.09 OR title_s:application^1.09 OR \ntitle_s:databases^1.09 OR title_s:idiomatic^1.09 OR title_s:spreadsheet^1.09 \nOR title_s:java^1.09 \u2026\nAND (\u2026)\n<\/pre>\n
\nPython for Data Analysis\nThe \/r\/playrust Classifier: Real World Rust Data Science\nAndreas Mueller - Commodity Machine Learning\nJose Quesada - A full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons\nYOW! 2016 Mark Hibberd & Ben Lever - Lab to Factory: Robust Machine Learning Systems\nHow To Get Started With Machine Learning? | Two Minute Papers\nA Gentle Introduction To Machine Learning\nBurc Arpat - Why Python is Awesome When Working With Data at any Scale\nLeverage R and Spark in Azure HDInsight for scalable machine learning\nMachine Learning with Scala on Spark by Jose Quesada\n<\/pre>\n
\nIs Nonfiction Literature?\n\"Oh, you liar, you storyteller\": On Fibbing, Fact and Fabulation \nThe Value of the Essay in the 21st Century\nMaking the Case for American Fiction: Connecting the Dots\nSiri Hustvedt in Conversation with Paul Auster\nIssues Related to the Teaching of Creative Writing\nAspen New York Book Series: The Art of the Memoir\nH.G. Adler - A Survivor's Dual Reverie\nSixth Annual Leon Levy Biography Lecture: David Levering Lewis\nContemporary Writing from Korea\n<\/pre>\n
Aboutness<\/h2>\n
\n10<\/a><\/sup>. Per the linked paper from Microsoft Research, this is a good technique if it is used to re-shuffle top results (i.e., you would not want to replace full text search with this).<\/p>\n\nval queryMean = model.getWordVectorsMean(List(\u201cwriting\u201d))\nval mean = model.getWordVectorsMean(NLP.getWords(document._1))\nval distance = Transforms.cosineSim(vec._2, queryMean)\n<\/pre>\n
![\"\"](\"http:\/\/www.garysieling.com\/blog\/wp-content\/uploads\/2017\/10\/3-2-300x177.png\")
<\/p>\n\nIssues Related to the Teaching of Creative Writing: 0.43\nAutobiography: 0.41\nContemporary Indian Writers: The Search for Creativity: 0.41\nMarjorie Welish: Lecture: 0.40\nHistory and Literature: The State of Play: A Roundtable Discussion: 0.40\nCritical Reading of Great Writers: Albert Camus: 0.40\nDaniel Schwarz: In Defense of Reading: 0.39\nThe Journey To The West by Professor Anthony C. Yu: 0.39\nBlogs, Twitter, the Kindle: The Future of Reading: 0.39\n<\/pre>\n
Overlapping Search Terms<\/h2>\n
\nterms.map(\n (term1) =>\n terms.map(\n (term2) => (term1, term2)\n )\n).flatten.filter(\n (tuple) => tuple._1 < tuple._2\n).map(\n (tuple) => \n (tuple._1, tuple._2, w2v.model.get.similarity(tuple._1, tuple._2))\n)\n<\/pre>\n
\ndistance(programming, python): 0.61\ndistance(art, hiking): 0.1\n<\/pre>\n
Topic Diversity<\/h2>\n
\nA Conversation with David Gerrold, Writer of Star Trek: The Trouble with Tribbles (58 minutes)\u00a0\nStar Trek: Science Fiction to Science Fact - STEM in 30 (28 minutes)\u00a0\n<\/pre>\n
\nPythons Positive Press Pumps\u00a0Pandas\u00a0\u00a0\nWhy is Python Growing So Quickly? \nPython explosion blamed on\u00a0pandas\u00a0\n<\/pre>\n
\nDeal with Unicode\nFalse Really Are Equal to True\nInteractive Debugger\nOutputs a Path\nPiece of Fortran\nPython 2.7 Point 10\nEmbedding Situation\nAndroid App\nAwesome Capability\nBinary Multiply\nDepended on Pandas have Pandas\n<\/pre>\n
\nGoogle I\/O 2011: Python@Google\nPorting Django apps to Python 3\nPython, C, C++, and Fortran Relationship Status: It\u2019s Not That Complicated\nSupercharging C++ Code with Embedded Python\nRules for Radicals: Changing the Culture of Python at Facebook\nPython for Ruby Programmers\nSaturday Morning Keynote (Brett Cannon)\nAll Things Open 2013 | Jessica McKellar | Python Foundation\n<\/pre>\n
\n![\"\"](\"http:\/\/www.garysieling.com\/blog\/wp-content\/uploads\/2017\/10\/3-3-300x156.png\")
<\/p>\n\nPython explosion blamed on pandas: 1.0\nConsidering Python's Target Audience: 0.97\nAnimated routes with QGIS and Python: 0.97\nI can't get some SQL to commit reading data from a database: 0.97\nUsing Python to build an AI Twitter bot people trust: 0.96\nGetting a Job as a Self-Taught Python Developer: 0.96\nDownload and Process DEMs in Python: 0.96\nHow to mine newsfeed data and extract interactive insights in Python: 0.94\nDifferential Equation Solver In MATLAB, R, Julia, Python, C, Mathematica, \nMaple, and Fortran: 0.86\nMy personal data science toolbox written in Python: 0.75\n<\/pre>\n
Conclusion<\/h2>\n