March 7th, 2014 — Code Examples, Data Science, Proof of Concepts
I’ve written previously about the mechanics of building decision trees: extract data from some system, build a model on it, then save the model in a file for later:
Once you get that far, you’ll likely find that you want to try different models or change the parameters on the ones you’re using.
Ideally, you want to write code like this, which runs several different experiments in sequence, saving the results to a file:
args = {'criterion': 'entropy', 'max_depth': 5, 'min_samples_split': 10}
createModels(treeParams=args, folds = 3)
args = {'criterion': 'entropy', 'max_depth': 10, 'min_samples_split': 10}
createModels(treeParams=args, folds = 3)
args = {'criterion': 'entropy', 'max_depth': 15, 'min_samples_split': 10}
createModels(treeParams=args, folds = 3) |
Once you start running experiments, you’ll get loads of files, so it’s important to name them based on what you’re running. Having saved off the results, you’ll be able to save models in source control and keep a permanent record of what experiments you ran.
Like unit tests, you can write out the results of accuracy tests as well, which lets you see how well you’re doing.
To make the filing system clear, the key is to take a python dictionary and turn it into a file name. With that in hand, we convert the model to JSON, and write it and the test results to a file.
def save(trees, features, args):
print "Saving..."
idx = 0
params = ""
for k in sorted(args.keys()):
params = params + k + "-" + str(args[k]) + " "
for t, report in trees:
f = open('D:\\projects\\tree\\tree ' + params + \
" fold " + str(idx) + ".json", 'w')
f.write(report)
f.write(treeToJson(t, features))
f.close()
idx = idx + 1 |
When we define the createModels function, it forces us to define and extract what this model-building process requires – for instance, what database tables to pull data from, a function to extract the value we want to predict, and how many folds we want to run.
The database operation in particular can be pretty heavy (pulling hundreds of thousands of records into memory), so memoization is key here.
def createModels(treeParams = None, folds = None, _cache = {}):
print "Creating models..."
def f(a):
if '>' in a:
return 1
else:
return 0
if len(_cache) == 0:
_cache['data'], _cache['outputs'] = \
extract('income_trn', f, 'category')
_cache['intData'], _cache['features'] = \
transform(_cache['data'], _cache['outputs'])
trees = runTests(_cache['intData'], _cache['outputs'], \
treeParams, folds)
save(trees, _cache['features'], treeParams) |
In any project where you reshape real data, you typically spend an inordinate amount of time on data transformations, and much less on fun steps like model training or performance tuning. This I’ve found true whether you do machine learning, data warehousing, or data migration projects.
Fortunately most commercial products and libraries that work in those areas provide utilities for common operations, and this is an area where scikit-learn really shines.
For instance, when we run tests we might train the model on 75% of the data, then validate it on 25% of the data. There are different strategies like this, but in this case I’d like to do that four times (each time removing a different 25% of the data). If I had to write the code to do that, it would be a pain – one more source of defects.
Scikit-learn, provide around a half dozen different implementations for splitting data, so if you run into a problem, chances are all you need to do is change which class you instantiate in the cross_validation package:
def runTests(intData, classify, treeParams, folds):
print "Testing data..."
rs = cross_validation.ShuffleSplit(len(intData), n_iter=folds,
test_size=.25, random_state=0)
foldNumber = 1
trees = []
for train_index, test_index in rs:
print "Fold %s..." % (foldNumber)
train_data = [intData[idx] for idx in train_index]
test_data = [intData[idx] for idx in test_index]
train_v = [classify[idx] for idx in train_index]
test_v = [classify[idx] for idx in test_index]
clf = tree.DecisionTreeClassifier(**treeParams)
clf = clf.fit(train_data, train_v)
predictions = [clf.predict(r)[0] for r in test_data]
report = classification_report(test_v, predictions)
trees.append( (clf, report) )
foldNumber = foldNumber + 1
return trees |
Similarly, one of the painful parts of the tree implementation is figuring out how to use data that has both text and numeric attributes. This is a specific case of the general problem of transforming data, and scikit-learn has dozens of classes to help in this area.
def transform(measurements, classify):
print "Transforming data..."
vec = DictVectorizer()
intData = vec.fit_transform(measurements).toarray()
featureNames = vec.get_feature_names()
return intData, featureNames |
As a digression, there is an interesting technical problem that lies hidden under the surface of the API. Trees specifically require that you make all your rows of data arrays of numbers – while this is very frustrating when you come in with strings, this is by design and not by accident and the code below using DictVectorizer will transform the data as you need.
When I first began researching python for data analysis, I was doing a broad search for tools, products, and libraries that might replace parts of the SQL + Relational database ecosystem. This includes tools that fall under the NoSQL umbrella, like Cassandra, Hadoop, and Solr, but also some ETL and data wrangling tools, like R, SAS, Informatica, and python.
If someone specifically wanted to avoid SQL as a language, for instance, there are quite a few options: swap in a different database underneath, use an ORM library to not have to see SQL, or use some sort of in-memory structure.
The python ORM (sqlalchemy) is quite nice for the purposes used in this article, as we can trivially read an entire table into RAM, converting each row into a dictionary:
def extract(tableName, resultXForm, resultColumn):
print "Loading from database..."
from sqlalchemy import create_engine, MetaData, Table
from sqlalchemy.sql import select
engine = create_engine(
"postgresql+pg8000://postgres:postgres@localhost/pacer",
isolation_level="READ UNCOMMITTED"
)
conn = engine.connect()
meta = MetaData()
table = Table(tableName, meta, autoload=True, autoload_with=engine)
def toDict(row, columns):
return {columns[i]: x for i, x in enumerate(row) \
if columns[i] != resultColumn}
s = select([table])
result = conn.execute(s)
columns = [c.name for c in table.columns]
resultIdx = columns.index(resultColumn)
data = [[c for c in row] for row in result]
split = [toDict(row, columns) for row in data]
classes = [resultXForm(row[resultIdx]) for row in data]
return (split, classes) |
There’s lots of code out there that looks like this, and it’s kind of kludgy, because eventually you’ll have so much data you wish you could switch algorithms as it grows, while serializing infrequently used structures to disk…which is exactly what relational databases give you.
One of the key value points of the scientific python stack is NumPy, which bridges a part of that gap (if you read “NumPy Internals“, it sounds an awful lot like something out of a database textbook).
This internal storage, then, is the reason for the above mentioned issue with Trees: internally, they use a massive array of floats, using NumPy underneath, so that it can be really fast on large datasets.
And with that, with a little bit of effore, we have a script that is almost exactly two hundred lines of code, allowing us to harness hundreds of thousands of engineering and grad student hours to find patterns out in the world.
March 4th, 2014 — Code Examples
If you do the following:
You’ll sometimes get this error:
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 8: ordinal not in range(128)
This indicates that you have non-ascii characters in the data, and should use a wider type. You can fix it by doing the following (alternately, you can do something like base64 encoding, although I’m not sure why you’d want to do that):
[unicode(x) for x in data] |
March 4th, 2014 — Code Examples
If you do this:
from sklearn import preprocessing
lb = preprocessing.LabelBinarizer()
lb.fit(["a", 2]) |
You will get the following error:
ValueError: Mix of label input types (string and number)
When you mix numbers and strings, it’s unclear whether you are mixing different types of classes, or if you’re mixing continuous and non-continuous data. If the latter- you don’t want the LabelBinarizer to run on the continuous data, and you should remove it, then re-add to the data later. If the former, you can convert the integers to strings.
March 3rd, 2014 — Uncategorized
I’m running some tests on sklearn decision trees, and the lessons learned so far may be interesting.
I’ve put my measurement code at the end – I’m tracking % correct, number of tests that are positive, negative, and false positives and negatives.
- When running predictions, if you have a defect where you include the ‘answer’ column in the test columns, the above code gives you a division by zero, which is a good check.
- For my data, when I run with criterion=’entropy’ I get 2% increase in accuracy, but other people I talked to on twitter have had the opposite
- criterion=’entropy’ is noticeably slower than the default (‘gini’)
- The default decision tree settings create trees that are very deep (~20k nodes for ~100k data points)
- For my use case, I found that limiting the depth of trees and forcing each node to have a large number of samples (50-500) made much simpler trees with only a small decrease in accuracy.
- In forcing nodes to have more samples, the accuracy decreased ~0-5%, roughly along the range of how many samples were included at each node (50-500)
- I found that I needed to remove a lot of my database columns to get a meaningful result. For instance originally I had ID columns, which lets sklearn pick up data created in a certain time window (since the IDs are sequential) but I don’t think this is useful for what I want to do.
- You have to turn class based attribute values into integers (it appears to be using a numpy float class internally for performance reasons)
- SKLearn appears to only use range based rules. Combine this with the above and you get a lot of rules like “status > 1.5″
- The tree could conceivably generate equality conditions within the structure, although it’d be hard to tell (e.g. “status > 1.5″, “status < 2.5" would be equivalent to "status = 2" if status is an integer)
- I’m more interesting in discovering useful rules than in future predictions; it helps a lot to generate JSON
- Within the JSON, the “entropy” and “impurity” field shows you how clean the rule is (0 = good). The “value” field shows how many items fit the rule (small numbers are probably not useful, at least for me)
testsRun = 0
testsPassed = 0
testsFalseNegative = 0
testsFalsePositive = 0
testsPositive = 0
testsNegative = 0
for t in test:
prediction = clf.predict(t)[0]
if prediction == 0:
testsNegative = testsNegative + 1
else:
testsPositive = testsPositive + 1
if prediction == test_v[testsRun]:
testsPassed = testsPassed + 1
else:
if prediction == 0:
testsFalseNegative = testsFalseNegative + 1
else:
testsFalsePositive = testsFalsePositive + 1
testsRun = testsRun + 1
print "Percent Pass: {0}".format(100 * testsPassed / testsRun)
print "Percent Positive: {0}".format(100 * testsPositive / testsRun)
print "Percent Negative: {0}".format(100 * testsNegative / testsRun)
print "Percent False positive: {0}".format(100 * testsFalseNegative / (testsFalsePositive + testsFalseNegative))
print "Percent False negative: {0}".format(100 * testsFalsePositive / (testsFalsePositive + testsFalseNegative)) |
March 3rd, 2014 — Code Examples, Data Mining, Data Science
SKLearn has a function to convert decision trees to “graphviz” (for rendering) but I find JSON more helpful, as you can read it more easily, as well as use it in web apps. The function below will give you JSON.
The reason this is necessary (vs the JSON.dumps) library is that the Decision Tree interfaces don’t support the interfaces the JSON library needs to run. Additionally, even if it did, the JSON library in python dies on very small floating point numbers, which is why it’s not used at all in my version.
def treeToJson(decision_tree, feature_names=None):
from warnings import warn
js = ""
def node_to_str(tree, node_id, criterion):
if not isinstance(criterion, sklearn.tree.tree.six.string_types):
criterion = "impurity"
value = tree.value[node_id]
if tree.n_outputs == 1:
value = value[0, :]
jsonValue = ', '.join([str(x) for x in value])
if tree.children_left[node_id] == sklearn.tree._tree.TREE_LEAF:
return '"id": "%s", "criterion": "%s", "impurity": "%s", "samples": "%s", "value": [%s]' \
% (node_id,
criterion,
tree.impurity[node_id],
tree.n_node_samples[node_id],
jsonValue)
else:
if feature_names is not None:
feature = feature_names[tree.feature[node_id]]
else:
feature = tree.feature[node_id]
if "=" in feature:
ruleType = "="
ruleValue = "false"
else:
ruleType = "<="
ruleValue = "%.4f" % tree.threshold[node_id]
return '"id": "%s", "rule": "%s %s %s", "%s": "%s", "samples": "%s"' \
% (node_id,
feature,
ruleType,
ruleValue,
criterion,
tree.impurity[node_id],
tree.n_node_samples[node_id])
def recurse(tree, node_id, criterion, parent=None, depth=0):
tabs = " " * depth
js = ""
left_child = tree.children_left[node_id]
right_child = tree.children_right[node_id]
js = js + "\n" + \
tabs + "{\n" + \
tabs + " " + node_to_str(tree, node_id, criterion)
if left_child != sklearn.tree._tree.TREE_LEAF:
js = js + ",\n" + \
tabs + ' "left": ' + \
recurse(tree, \
left_child, \
criterion=criterion, \
parent=node_id, \
depth=depth + 1) + ",\n" + \
tabs + ' "right": ' + \
recurse(tree, \
right_child, \
criterion=criterion, \
parent=node_id,
depth=depth + 1)
js = js + tabs + "\n" + \
tabs + "}"
return js
if isinstance(decision_tree, sklearn.tree.tree.Tree):
js = js + recurse(decision_tree, 0, criterion="impurity")
else:
js = js + recurse(decision_tree.tree_, 0, criterion=decision_tree.criterion)
return js |
March 3rd, 2014 — Code Examples
SQL Alchemy makes it easy to get types out of the database:
from sqlalchemy import *
engine = create_engine(
"postgresql+pg8000://postgres:postgres@localhost/postgres",
isolation_level="READ UNCOMMITTED"
)
c = engine.connect()
meta = MetaData()
t = Table('table', meta, autoload=True, autoload_with=engine, schema='test')
columns = [col for col in t.columns] |
And then from there, you can filter the column list down to things you want.
import sqlalchemy.sql.sqltypes
def useColumn(c):
if (type(c.type) is TIMESTAMP):
return False
if (type(c.type) is VARCHAR):
if (c.type.length == 24):
return False
if (type(c.type) is DATE):
return False
if (c.name.startswith("internal_")):
return False
if (c.name == "dont_use_me"):
return False
return True |
For certain columns, SQL Alchemy will give you an object that is a database specific type (e.g. there is a Postgres namespace), but you can find the exact class name:
columns[0].type.__class__
...sqlalchemy.sql.sqltypes.INTEGER |
March 2nd, 2014 — Data Mining, Data Science
The scikit-learn documentation has an argument to control how the decision tree algorithm splits nodes:
criterion : string, optional (default=”gini”)
The function to measure the quality of a split.
Supported criteria are “gini” for the Gini impurity
and “entropy” for the information gain.
It seems like something that could be important since this determines the formula used to partition your dataset at each point in the dataset.
Unfortunately the documentation tells you nothing about what you should use, other than trying each to see what happens, so here’s what I found (spoiler: it doesn’t appear to matter):
- Gini is intended for continuous attributes, and Entropy for attributes that occur in classes (e.g. colors
- “Gini” will tend to find the largest class, and “entropy” tends to find groups of classes that make up ~50% of the data((http://paginas.fe.up.pt/~ec/files_1011/week%2008%20-%20Decision%20Trees.pdf))
- “Gini” to minimize misclassification
- “Entropy” for exploratory analysis
- Some studies show this doesn’t matter – these differ less than 2% of the time
- Entropy may be a little slower to compute
February 27th, 2014 — Code Examples
Fiddler (HTTP proxy for Windows) is a valuable tool for finding out what is happening in an external environment (e.g. when a client uses IE).
It’s sometimes helpful to pull it’s data into Excel – this allows you to highlight dramatic deviations.
The scripting API allows you to do this – drop the following code into the Fiddler script file-
public static ToolsAction("Copy Request Timings to Excel")
function DoHighlightSlowRequests() {
var oSessions = FiddlerApplication.UI.GetAllSessions();
var s: String = "";
s = s +
"Index\t" +
"URL\t" +
"Is HTTPS\t" +
"Response Code\t" +
"Request Bytes\t" +
"Response Bytes\t" +
"ClientConnected\t" +
"ClientBeginRequest\t" +
"ClientDoneRequest\t" +
"ServerConnected\t" +
"FiddlerBeginRequest\t" +
"ServerGotRequest\t" +
"ServerBeginResponse\t" +
"ServerDoneResponse\t" +
"ClientBeginResponse\t" +
"ClientDoneResponse\t" +
"DNSTime\t" +
"GatewayDeterminationTime\t" +
"TCPConnectTime\t" +
"HTTPSHandshakeTime\t" +
"Request Transmission Time\t" +
"Server Time Spent\t" +
"Response Transmission Time\t" +
"Transmission time (down + up)\t" +
"Total Round Trip Time" +
"\r\n";
for (var x:int = 0; x < oSessions.Length; x++){
var session = oSessions[x]
var timer = session.Timers
// use tabs intead of CSV because
// Excel and .NET have incompatible
// expectations for unicode format
var t = oSessions[x].Timers
var transmissionTime =
new TimeSpan(t.ServerGotRequest.Ticks -
t.FiddlerBeginRequest.Ticks).Milliseconds
var serverTimeSpent =
new TimeSpan(t.ServerDoneResponse.Ticks -
t.ServerGotRequest.Ticks).Milliseconds
var responseTransmissionTime =
new TimeSpan(t.ServerDoneResponse.Ticks -
t.ServerBeginResponse.Ticks).Milliseconds
var totalTransferTime =
transmissionTime + responseTransmissionTime
var roundTripTime =
new TimeSpan(t.ClientDoneResponse.Ticks -
t.ClientBeginRequest.Ticks).Milliseconds
var transmissionTimeStr =
transmissionTime < 0 ? "" : transmissionTime + ""
var serverTimeSpentStr =
serverTimeSpent < 0 ? "" : serverTimeSpent + ""
var responseTransmissionTimeStr =
responseTransmissionTime < 0 ? "" : responseTransmissionTime + ""
var totalTransferTimeStr =
totalTransferTime < 0 ? "" : totalTransferTime + ""
var roundTripTimeStr =
roundTripTime < 0 ? "" : roundTripTime + ""
s = s +
x + "\t" +
oSessions[x].url + "\t" +
oSessions[x].isHTTPS + "\t" +
oSessions[x].responseCode + "\t" +
oSessions[x].requestBodyBytes.Length + "\t" +
oSessions[x].responseBodyBytes.Length + "\t" +
oSessions[x].Timers.ClientConnected.Ticks + "\t" +
oSessions[x].Timers.ClientBeginRequest.Ticks + "\t" +
oSessions[x].Timers.ClientDoneRequest.Ticks + "\t" +
oSessions[x].Timers.ServerConnected.Ticks + "\t" +
oSessions[x].Timers.FiddlerBeginRequest.Ticks + "\t" +
oSessions[x].Timers.ServerGotRequest.Ticks + "\t" +
oSessions[x].Timers.ServerBeginResponse.Ticks + "\t" +
oSessions[x].Timers.ServerDoneResponse.Ticks + "\t" +
oSessions[x].Timers.ClientBeginResponse.Ticks + "\t" +
oSessions[x].Timers.ClientDoneResponse.Ticks + "\t" +
oSessions[x].Timers.DNSTime + "\t" +
oSessions[x].Timers.GatewayDeterminationTime + "\t" +
oSessions[x].Timers.TCPConnectTime + "\t" +
oSessions[x].Timers.HTTPSHandshakeTime + "\t" +
transmissionTimeStr + "\t" +
serverTimeSpentStr + "\t" +
responseTransmissionTimeStr + "\t" +
totalTransferTimeStr + "\t" +
roundTripTimeStr +
"\r\n";
}
System.Windows.Forms.Clipboard.SetText(
s.ToString(),
TextDataFormat.Text);
} |
February 26th, 2014 — Code Examples
Scikit-learn provides routines to export decision trees to a format called Graphviz, although typically this is used to provide an image of a chart.
For some applications this is valuable, but if the product of machine learning is a the ability to generate models (rather than predictions), it would be preferable to provide interactive models.
Automatically construction decision trees, for instance, might allow you to discover patterns in underlying data, e.g. determining that many failures are caused by a particular device or vendor. In this scenario, being able to predict failure is relatively useless, since the goal is to take corrective action.
There are many awesome interactive tree examples with D3.js and the example that follows will show how to link these two products together.
Scikit-learn provides a function called export_graphviz, which I pulled changed to export JSON (the library would probably benefit from adding API calls that let you iterate over their trees, so this is not needed)
The whole function is a bit long, so feel free to skip to the next section.
def viz(decision_tree, feature_names=None):
from warnings import warn
js = ""
def node_to_str(tree, node_id, criterion):
if not isinstance(criterion, sklearn.tree.tree.six.string_types):
criterion = "impurity"
value = tree.value[node_id]
if tree.n_outputs == 1:
value = value[0, :]
if tree.children_left[node_id] == sklearn.tree._tree.TREE_LEAF:
return '{"id": "%s", "criterion": "%s", "impurity": "%s", "samples": "%s", "value": "%s"}' \
% (node_id,
criterion,
tree.impurity[node_id],
tree.n_node_samples[node_id],
value)
else:
if feature_names is not None:
feature = feature_names[tree.feature[node_id]]
else:
feature = tree.feature[node_id]
return '"id": "%s", "rule": "%s <= %.4f", "%s": "%s", "samples": "%s"' \
% (node_id,
feature,
tree.threshold[node_id],
criterion,
tree.impurity[node_id],
tree.n_node_samples[node_id])
def recurse(tree, node_id, criterion, parent=None, depth=0):
tabs = " " * depth
js = ""
left_child = tree.children_left[node_id]
right_child = tree.children_right[node_id]
js = js + "\n" + \
tabs + "{\n" + \
tabs + " " + node_to_str(tree, node_id, criterion)
if left_child != sklearn.tree._tree.TREE_LEAF and depth < 6:
js = js + ",\n" + \
tabs + ' "left": ' + \
recurse(tree, \
left_child, \
criterion=criterion, \
parent=node_id, \
depth=depth + 1) + ",\n" + \
tabs + ' "right": ' + \
recurse(tree, \
right_child, \
criterion=criterion, \
parent=node_id,
depth=depth + 1)
js = js + tabs + "\n" + \
tabs + "}"
return js
if isinstance(decision_tree, sklearn.tree.tree.Tree):
js = js + recurse(decision_tree, 0, criterion="impurity")
else:
js = js + recurse(decision_tree.tree_, 0, criterion=decision_tree.criterion)
return js
cols = dict()
for i, c in enumerate(income_trn.columns):
cols[i] = c.name
print viz(clf, feature_names=cols) |
I’m extending a previous example I wrote, which builds a decision tree for income data from Postgres data. The ‘rules’ here are what determines whether the left or right path down a tree applies to a data point. The internal structure of the tree numbers these as column 1, column 2, etc, but the function above lets you provide a mapping, so these can be turned back into column headers.
x = {
id: '0', rule: 'marital_status <= 12.0000', 'gini': '0.360846845083', 'samples': '16281',
left:
{
id: '1', rule: 'capital_gain <= 86.5000', 'gini': '0.0851021980364', 'samples': '5434',
left:
{
id: '2', rule: 'capital_loss <= 2372.0000', 'gini': '0.0634649568944', 'samples': '5212',
left:
{
id: '3', rule: 'education <= 46.0000', 'gini': '0.0584505343', 'samples': '5177'
},
right:
{
id: '570', rule: 'capital_loss <= 2558.5000', 'gini': '0.489795918367', 'samples': '35'
}
},
right:
{
id: '577', rule: 'education_num <= 47.0000', 'gini': '0.43507020534', 'samples': '222',
left:
{
id: '578', rule: 'capital_gain <= 221.5000', 'gini': '0.272324674466', 'samples': '123'
},
right:
{
id: '607', rule: 'capital_gain <= 817.5000', 'gini': '0.499540863177', 'samples': '99'
}
}
},
right:
{
id: '632', rule: 'marital_status <= 55.0000', 'gini': '0.44372508662', 'samples': '10847',
left:
{
id: '633', rule: 'education_num <= 47.0000', 'gini': '0.493880410242', 'samples': '7403',
left:
{
id: '634', rule: 'capital_gain <= 22.0000', 'gini': '0.454579586462', 'samples': '4363'
},
right:
{
id: '2885', rule: 'education_num <= 113.5000', 'gini': '0.486689750693', 'samples': '3040'
}
},
right:
{
id: '4292', rule: 'capital_gain <= 86.5000', 'gini': '0.164770726851', 'samples': '3444',
left:
{
id: '4293', rule: 'education_num <= 47.0000', 'gini': '0.126711048456', 'samples': '3207'
},
right:
{
id: '4902', rule: 'education <= 46.0000', 'gini': '0.478627000659', 'samples': '237'
}
}
}
} |
To make this render with D3, I picked one example structure, which looks like this:
The nice thing about this visualization is that while it shows leaf nodes, it groups them into the tree hierarchy and lets you selectively drill down by clicking to zoom.
The JSON structure for every example isn’t necessarily guaranteed to be the same, so I’ve written a function to restructure the above tree into exactly what this needs (this is easier than rewriting the above python code for every test case, or fixing the D3 examples)
function toJson(x)
{
var result = {};
result.name = x.rule;
if ( (!!x.left && !x.left.value) ||
(!!x.right && !x.right.value) )
result.children = [];
else
result.size = parseInt(x.samples);
var index = 0;
if (!!x.left && !x.left.value)
result.children[index++] = toJson(x.left);
if (!!x.right && !x.right.value)
result.children[index++] = toJson(x.right);
return result;
} |
Then, the only change you need to make to the D3 example is to add this function and add a call:
node = root = toJson(data); |
Now, this isn’t the prettiest and is only one view of the tree (leaves), but it wires up enough parts to get you set up to find the right visualization for what you’re doing.
February 21st, 2014 — Code Examples
This example uses a twenty year old data set that you can use to predict someone’s income from demographic data.
The purpose of this example is to show how to go from data in a relational database to a predictive model, and note what problems you may encounter.
One of the nice things about this data is that the authors of the paper provide the accuracy of various algorithms, which we can use to smoke test our results:
Algorithm Error
-- ---------------- -----
1 C4.5 15.54
2 C4.5-auto 14.46
3 C4.5 rules 14.94
4 Voted ID3 (0.6) 15.64
5 Voted ID3 (0.8) 16.47
6 T2 16.84
7 1R 19.54
8 NBTree 14.10
9 CN2 16.00
10 HOODG 14.82
11 FSS Naive Bayes 14.05
12 IDTM (Decision table) 14.46
13 Naive-Bayes 16.12
14 Nearest-neighbor (1) 21.42
15 Nearest-neighbor (3) 20.35
16 OC1 15.04
To load this dataset into Postgres you’ll need to remove the blank lines at the end of the author’s files, and the “1×0 Cross Validator” line.
The following will load their data – If this seems contrived, I’m doing this to set up test cases for a real Postgres database.
Note that Postgres can read UNC paths, which shouldn’t be surprising, but considering Visual Studio still can’t, it’s always nice when it works.
DROP TABLE income_trn;
CREATE TABLE income_trn
(age INTEGER,
workclass text,
fnlwgt INTEGER,
education text,
education_num INTEGER,
marital_status text,
occupation text,
relationship text,
race text,
sex text,
capital_gain INTEGER,
capital_loss INTEGER,
hours_per_week INTEGER,
native_country text,
category text);
COPY income_trn
FROM '\\\\nas\\Files\\Data\\income\\adult.data' DELIMITER ',' CSV;
DROP TABLE income_test;
CREATE TABLE income_test
(age INTEGER,
workclass text,
fnlwgt INTEGER,
education text,
education_num INTEGER,
marital_status text,
occupation text,
relationship text,
race text,
sex text,
capital_gain INTEGER,
capital_loss INTEGER,
hours_per_week INTEGER,
native_country text,
category text);
COPY income_test
FROM '\\\\nas\\Files\\Data\\income\\adult.test' DELIMITER ',' CSV; |
You can load this data in python easily with sqlalchemy. However, the Postgres driver (“pg8000″) seems to be flaky, as it will throw these errors at random: (Edit 2/26/14 – one of the pg8000 contributors fixed the issue after I posted this – Thanks!)
ProgrammingError: (ProgrammingError)
('ERROR', '34000',
'portal "pg8000_portal_12" does not exist')
None None
These can be caused by using an old version of Postgres (I’m on 9.3), although the error is inconsistent – it appears to be an issue where the code reads from a closed cursor.
from sqlalchemy import *
engine = create_engine(
"postgresql+pg8000://postgres:postgres@localhost/pacer",
isolation_level="READ UNCOMMITTED"
)
c = engine.connect()
meta = MetaData()
income_trn = Table('income_trn', meta, autoload=True, autoload_with=engine)
income_test = Table('income_test', meta, autoload=True, autoload_with=engine) |
For a large data set it is likely valuable to stream the results of queries into the model, but this dataset is small, so I haven’t attempted that. If your data is all in one table, you’ll need to find a way to split it in half randomly that also performs well enough.
from sqlalchemy.sql import select
def get_data(table):
s = select([table])
result = c.execute(s)
return [row for row in result]
test_data = get_data(income_trn)
trn_data = get_data(income_test) |
When this data comes in, it has text labels for some columns, and integers for others (e.g. age vs occupation). This is oddly difficult for the python machine learning libraries (or at least, the decision trees). This is a little concerning considering that continuous data is very different than data that is a set of values.
The library expects that everything is a list of values, but they have to be numbers, but we can build a global dictionary to map back and forth:
maxVal = 0
vals = dict()
rev_vals = dict()
def f(x):
global maxVal
global vals
if (not x in vals):
maxVal = maxVal + 1
vals[x] = maxVal
rev_vals[maxVal] = x
return vals[x] |
Then, we have to split the attributes into the output and the attributes used to predict the output:
def get_selectors(data):
return [ [f(x) for x in t[0:-1]] for t in data]
def get_predictors(data):
return [0 if "<" in t[14] else 1 for t in data]
trn = get_selectors(trn_data)
trn_v = get_predictors(trn_data) |
The most compelling thing about this whole set-up to me is how trivially easy it is to build a model:
from sklearn import tree
clf = tree.DecisionTreeRegressor()
clf = clf.fit(trn, trn_v) |
I ended up writing my own test method, because the confusion matrix calculator didn’t like data in classes.
test = get_selectors(test_data)
test_v = get_predictors(test_data)
testsRun = 0
testsPassed = 0
for t in test:
if clf.predict(t) == test_v[testsRun]:
testsPassed = testsPassed + 1
testsRun = testsRun + 1
100 * testsPassed / testsRun
DecisionTreeClassifier: 78%
DecisionTreeRegressor: 79% |
If you look at the the scikit-learn documentation, all the examples have awesome charts. One of the things I discovered from this is that the Decision Trees can be quite long – they may have thousands of rules included, which doesn’t lend itself to charting, except in the simplest cases.
