Decision Trees: “Gini” vs. “Entropy” criteria

The scikit-learn documentation1 has an argument to control how the decision tree algorithm splits nodes:

criterion : string, optional (default=”gini”)
The function to measure the quality of a split. 
Supported criteria are “gini” for the Gini impurity 
and “entropy” for the information gain.

It seems like something that could be important since this determines the formula used to partition your dataset at each point in the dataset.

Unfortunately the documentation tells you nothing about what you should use, other than trying each to see what happens, so here’s what I found (spoiler: it doesn’t appear to matter):

  • Gini is intended for continuous attributes, and Entropy for attributes that occur in classes (e.g. colors2
  • “Gini” will tend to find the largest class, and “entropy” tends to find groups of classes that make up ~50% of the data((http://paginas.fe.up.pt/~ec/files_1011/week%2008%20-%20Decision%20Trees.pdf))
  • “Gini” to minimize misclassification3
  • “Entropy” for exploratory analysis3
  • Some studies show this doesn’t matter – these differ less than 2% of the time4
  • Entropy may be a little slower to compute5
  1. http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html []
  2. http://paginas.fe.up.pt/~ec/files_1011/week%2008%20-%20Decision%20Trees.pdf []
  3. http://www.quora.com/Machine-Learning/Are-gini-index-entropy-or-classification-error-measures-causing-any-difference-on-Decision-Tree-classification [] []
  4. https://rapid-i.com/rapidforum/index.php?topic=3060.0 []
  5. http://stats.stackexchange.com/questions/19639/which-is-a-better-cost-function-for-a-random-forest-tree-gini-index-or-entropy []

5 Replies to “Decision Trees: “Gini” vs. “Entropy” criteria”

  1. If you find the best split using the entropy criterion. Does the “feature importances” attribute of a tree return the information gain measures or does it return the gini importance?

  2. whats meaning of area under curve of Gini and Entropy ??
    area and other attributes of their curves???
    answer please

Leave a Reply

Your email address will not be published. Required fields are marked *