balkian/sitc
1
0
Fork 0
mirror of https://github.com/gsi-upm/sitc synced 2024-11-05 07:31:41 +00:00
Code Issues Releases Wiki Activity
47fe85d527
sitc/ml1/2_5_2_Decision_Tree_Model.ipynb

600 lines
146 KiB
Plaintext
Raw Normal View History

Add ml1
2016-03-15 12:55:14 +00:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](files/images/EscUpmPolit_p.gif \"UPM\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Course Notes for Learning Intelligent Systems"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
added update -all
2019-02-28 11:26:33 +00:00
"Department of Telematic Engineering Systems, Universidad Politécnica de Madrid, © Carlos A. Iglesias"
Add ml1
2016-03-15 12:55:14 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## [Introduction to Machine Learning](2_0_0_Intro_ML.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Table of Contents\n",
"* [Decision Tree Learning](#Decision-Tree-Learning)\n",
"* [Load data and preprocessing](#Load-data-and-preprocessing)\n",
"* [Train classifier](#Train-classifier)\n",
"* [Evaluating the algorithm](#Evaluating-the-algorithm)\n",
"\t* [Precision, recall and f-score](#Precision,-recall-and-f-score)\n",
"\t* [Confusion matrix](#Confusion-matrix)\n",
"\t* [K-Fold cross validation](#K-Fold-cross-validation)\n",
"* [References](#References)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Decision Tree Learning"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
Fix visualization section
2017-12-11 17:12:06 +00:00
"The goal of this notebook is to learn how to create a classification object using a [decision tree learning algorithm](https://en.wikipedia.org/wiki/Decision_tree_learning). \n",
Add ml1
2016-03-15 12:55:14 +00:00
"\n",
"There are a number of well known machine learning algorithms for decision tree learning, such as ID3, C4.5, C5.0 and CART. The scikit-learn uses an optimised version of the [CART (Classification and Regression Trees) algorithm](https://en.wikipedia.org/wiki/Predictive_analytics#Classification_and_regression_trees).\n",
"\n",
added update -all
2019-02-28 11:26:33 +00:00
"This notebook will follow the same steps that the previous notebook for learning using the [kNN Model](2_5_1_kNN_Model.ipynb), and details some peculiarities of the decision tree algorithms.\n",
"\n",
"You need to install pydotplus: `conda install pydotplus` for the visualization."
Add ml1
2016-03-15 12:55:14 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data and preprocessing\n",
"\n",
"Here we repeat the same operations for loading data and preprocessing than in the previous notebooks."
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 1,
"metadata": {},
added update -all
2019-02-28 11:26:33 +00:00
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/cif/anaconda3/lib/python3.5/site-packages/sklearn/utils/fixes.py:313: FutureWarning: numpy not_equal will not check object identity in the future. The comparison did not return the same result as suggested by the identity (`is`)) and will change.\n",
" _nan_object_mask = _nan_object_array != _nan_object_array\n"
]
}
],
Add ml1
2016-03-15 12:55:14 +00:00
"source": [
"# library for displaying plots\n",
"import matplotlib.pyplot as plt\n",
"# display plots in the notebook \n",
"%matplotlib inline\n",
"\n",
"## First, we repeat the load and preprocessing steps\n",
"\n",
"# Load data\n",
"from sklearn import datasets\n",
"iris = datasets.load_iris()\n",
"\n",
"# Training and test spliting\n",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"from sklearn.model_selection import train_test_split\n",
Add ml1
2016-03-15 12:55:14 +00:00
"\n",
"x_iris, y_iris = iris.data, iris.target\n",
"# Test set will be the 25% taken randomly\n",
"x_train, x_test, y_train, y_test = train_test_split(x_iris, y_iris, test_size=0.25, random_state=33)\n",
"\n",
"# Preprocess: normalize\n",
"from sklearn import preprocessing\n",
"scaler = preprocessing.StandardScaler().fit(x_train)\n",
"x_train = scaler.transform(x_train)\n",
"x_test = scaler.transform(x_test)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Train classifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The usual steps for creating a classifier are:\n",
"1. Create classifier object\n",
"2. Call *fit* to train the classifier\n",
"3. Call *predict* to obtain predictions\n",
"\n",
"*DecisionTreeClassifier* is capable of both binary (where the labels are [-1, 1]) classification and multiclass (where the labels are [0, ..., K-1]) classification."
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 2,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"data": {
"text/plain": [
"DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=3,\n",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
" max_features=None, max_leaf_nodes=None,\n",
added update -all
2019-02-28 11:26:33 +00:00
" min_impurity_decrease=0.0, min_impurity_split=None,\n",
" min_samples_leaf=1, min_samples_split=2,\n",
" min_weight_fraction_leaf=0.0, presort=False, random_state=1,\n",
" splitter='best')"
Add ml1
2016-03-15 12:55:14 +00:00
]
},
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 2,
Add ml1
2016-03-15 12:55:14 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.tree import DecisionTreeClassifier\n",
"import numpy as np\n",
"\n",
"from sklearn import tree\n",
"\n",
"max_depth=3\n",
"random_state=1\n",
"\n",
"# Create decision tree model\n",
"model = tree.DecisionTreeClassifier(max_depth=max_depth, random_state=random_state)\n",
"\n",
"# Train the model using the training sets\n",
"model.fit(x_train, y_train) "
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 3,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Prediction [1 0 1 1 1 0 0 1 0 2 0 0 1 2 0 1 2 2 1 1 0 0 2 0 0 2 1 1 2 2 2 2 0 0 1 1 0\n",
" 1 2 1 2 0 2 0 1 0 2 1 0 2 2 0 0 2 0 0 0 2 2 0 1 0 1 0 1 1 1 1 1 0 1 0 1 2\n",
" 0 0 0 0 2 2 0 1 1 2 1 0 0 2 1 1 0 1 1 0 2 1 2 1 2 0 1 0 0 0 2 1 2 1 2 1 2\n",
" 0]\n",
"Expected [1 0 1 1 1 0 0 1 0 2 0 0 1 2 0 1 2 2 1 1 0 0 2 0 0 2 1 1 2 2 2 2 0 0 1 1 0\n",
" 1 2 1 2 0 2 0 1 0 2 1 0 2 2 0 0 2 0 0 0 2 2 0 1 0 1 0 1 1 1 1 1 0 1 0 1 2\n",
" 0 0 0 0 2 2 0 1 1 2 1 0 0 1 1 1 0 1 1 0 2 2 2 1 2 0 1 0 0 0 2 1 2 1 2 1 2\n",
" 0]\n"
]
}
],
"source": [
"print(\"Prediction \", model.predict(x_train))\n",
"print(\"Expected \", y_train)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively, the probability of each class can be predicted, which is the fraction of training samples of the same class in a leaf:"
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 4,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Predicted probabilities [[ 0. 0.97368421 0.02631579]\n",
" [ 1. 0. 0. ]\n",
" [ 0. 0.97368421 0.02631579]\n",
" [ 0. 0.97368421 0.02631579]\n",
" [ 0. 0.97368421 0.02631579]\n",
" [ 1. 0. 0. ]\n",
" [ 1. 0. 0. ]\n",
" [ 0. 0.97368421 0.02631579]\n",
" [ 1. 0. 0. ]\n",
" [ 0. 0. 1. ]]\n"
]
}
],
"source": [
"# Print the \n",
"print(\"Predicted probabilities\", model.predict_proba(x_train[:10]))"
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 5,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy in training 0.982142857143\n"
]
}
],
"source": [
"# Evaluate Accuracy in training\n",
"\n",
"from sklearn import metrics\n",
"y_train_pred = model.predict(x_train)\n",
"print(\"Accuracy in training\", metrics.accuracy_score(y_train, y_train_pred))"
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 6,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy in testing 0.921052631579\n"
]
}
],
"source": [
"# Now we evaluate error in testing\n",
"y_test_pred = model.predict(x_test)\n",
"print(\"Accuracy in testing \", metrics.accuracy_score(y_test, y_test_pred))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we are going to visualize the DecisionTree classification. It will plot the decision boundaries for each class.\n",
"\n",
"The current version of pydot does not work well in Python 3.\n",
"For obtaining an image, you need to install `pip install pydotplus` and then `conda install graphviz`.\n",
"\n",
"You can skip this example. Since it can require installing additional packages, we include here the result.\n",
"![Decision Tree](files/images/cart.png)"
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 7,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
Fixed age charts
2016-03-29 10:48:56 +00:00
{
added update -all
2019-02-28 11:26:33 +00:00
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAukAAAJBCAIAAAB06bzfAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzddzxV7x8A8M+9pOw9m1Yoo2kXUlZDqUhpL0lSUdmbaG/t+ja0h2h8GyqjUCqlrFA0NBAyrjt+f5zv737v17gkuro+79f5w3me5zznc67k43mecw6JwWAAQgghhNAfgszpABBCCCGEfgLmLgghhBD6k2DughBCCKE/CS+nA0AcwGAwCgsLCwsLy8vLccETah8ymSwmJqaoqKioqEgikTgdDkKoG8HcpRuh0Wjx8fExMTE3b94sLy/ndDiIS4iLi1taWs6cOdPGxoaHh4fT4SCEuB8J/+zuJmJjY9esWfPmzRtTk9ETxo830NdTUVaWkBAnk3HeELUHnU4vKyvPf/Pm4aPUuPj4e/cfKCsrb968edKkSZwODSHE5TB34X75+fnLly+/deuWo4N9YICfirIypyNCXCj/zZvAoJCYM2fHjRu3e/duFRUVTkeEEOJa+Dc3l7tz546uru7n0tL7d2+f+OsoJi6ok6goK5/46+j9u7c/l5bq6ureuXOH0xEhhLgW5i7c7MCBA9bW1lYW41IS7xkbGXI6HMT9jI0MUxLvWVmMs7a2PnDgAKfDQQhxJ1yry7ViYmKWLl3q5+sd4OuDt4Gg36ZXr14n/jqqOlB16dKlQkJCjo6OnI4IIcRtcL0Ld3r8+LGJicmypYs3Rm7gdCyom/JYu25P9P67d+8aGBhwOhaEEFfBOSMu9O3btwkTJpibmUVGhHMwjEFaQ8h8/J3aA5mPf5DWkA45dUNDw0h9w/hr138uRLZeZ2f3V1L99q2sA/v8g0RtiBhnbj5lypRv375xOhaEEFfB3IUL+fv7k8mk48cOd/j9z3bT7FesXNWxfXaIXw8sMDikZ8+e422sOyokANBQVzcfY+bssrwdxzY0NLS9MZ1OD4uIHK6rLyopYzTa9NCRo208cILtlJaSvNUeno3yQjIff0tbsz2QyeTjxw6TySR/f/+2XwtCCLUK17twm6ysrP379x/cHy0iItLhnV+Ovaquptbh3bbkcWpKG+c0fzGwgsLCqE1bzp2JaXcPLfFYs0pTZ9jtO3fHmo9p4yGlpZ/3HzwUvX//+7eFbTzEwXHWhUuXzUxNXF2cr9+4uXjpsqKiopCgQPZH7d4bfe36jWar3hQUHP3ruJysHGvhvDmzm7a8cOmyjLR0S6cQEREJDw1ZtMR5yZIlOjo6rV8JQgi1Aa534Ta2trafPn58mHS/M9bnkvn41dXUXr141pbGg7SGZOfk0Cm1HR5G03gaBfazp3Z2cb1w6dKHd0U9evTo8CBH6hsKCwvfvXWz1ZaPn2Ts2r3n9NlzQkKC8+bM3hQV2Zb+0x8/0TM0njxp4oVzZ0gkUm1treEo09y8vKL8XGlpqZaOep2dPULPsK6ujsFgsH5QkRs3P3nyJP76jdra2la/1+cuXJzpNCfx3l19Pd2W2jAYDANjEzl5+StXrrTlchBCqFU4Z8RVSkpK4uPj16x2Z5O4qA/WJvPx19bWurq5K6mq91NUmTNvwZcvX5kNqFRqWESknqGxsLiUitogHz//qqoqACCmBrJzcphzBHQ6/ehfx41NzOT69BMSk1QfrL3e24dozIbZWAtxaTkajUbsbt+5i8zHP37SZGYDx1mzeXsJfv36rdGylX0HDo4yHSMuLTdIa4izi+v379+J8qaBESorK+ctWNRfSVVKTsHBcdanT6XNxlNdXX3s+IlJEyawJi4NDQ2BwSHDdfVFJKRH6hsGBocQkzhESD9+/Fi8dFl/JdV+iiourm5UKjXl4SNzCytJWXmFfgOWOLtUV1czu5pqZ3fv/oOsV69a+kCoVOrZ8xdGmY7RNTB6nZ0dvWdXceGbNiYuALB7z14AcF/pRnzT+fn5nZcurq2tZTNzRKFQnObMMzYyVG3yBLlHj1Irvn83Mmx9dW1p6WcX1xW+3l5sEhcAIJFIq1etjI+PLykpacvlIIRQqzB34SpXrlwRFBS0nTiBTRsiabC1m/amoMBplqOystKJUzG6BkZEzkGj0cZaWvsFBJLJZI/V7sOGDomI3GhuYVVXV3frxjUA6NunD/EFALiv9liwaMmr16+tLS3d3VyFhYWiNm1ZuHgp+yCtLC2/f/+e8fSfP+iTkpIBICk5hZnN3HvwYOSI4VJSkqxHzZm3YNnyFcXFJXNnO423sbqbcE/faDRR1TQwgrmFVc+ePVevWqmspHTuwsUly1yajefWnbv19fWGLL+taTSa2ViL4NBwWVnZtR5rBqqqhoZvsLAezxykHD9psrCw0FrPNWJiYtH7D5iNtZhgO1lPd2RQgL+oiOjBw0cCgoKZvRka6ANA/LXmZ2fCN0QpqqjNX7h4oKpq2sPk1JSkeXNm8/P/xELjnNxcXl5e1mzDZPQoAMjLy2vpEL+AwKK3bw8f3N90RdSlC2f/vh7/9/X4Vs/r7OKqIK/g47Wu1ZaTJ00UEBCIjY1ttSVCCLUJA3GRqVOn2k6cSKfUstkG9O8PAK4uy4hdWn3NgnlzASA0OIhOqd2zawcA2FhbNdRWEw22bdkEAJER4cTMgrqaGrMrSUkJADh96gSxS6mpkpKSFBAQIHaJBShNA3j2JA0AIsJCiV1ZWRktTU0ASE1JolNqiUmKAD9f1h6uXb0CAEOH6JR9/kQcVVn2RU93JDOeRoERB3qsdid2q8q/CggIMANrtC1bugQAnj5OZZYQH4Kb63JmSYCfDwDcvXWT6Hnn9q1E+cvnGcTPUXzsZdarGz5sKPPYqvKvADDWfEyzZwcAPj6+vbt3Umqq2H/jWtp6KyjIyEizlnz5+B4ALMaNbbb9nb9vkMnkmJPH2XyPmn6kjbYb8VcB4HpcbBuDtJ04cerUqZz++UAIcQkcd+EqmZmZQ4a0siKSGN7w8/EmdkkkUlCgPwBcjYsDgJjTZwDA19uL+UJgV5dlffv0udLcH8352a/Kv3yaZjeF2K2srKJQGmpqatgHoK2l1VtBIeHePQDIzcsrLf3s7bWORCLdf5AIAAn37gOAjbUV6yHnzl8AgLCQYDExUaJESEgoKKCVu1cWLVxAfCEoKNind++WAit6+xYAWNelEh+Cj9d6ZsmypUt3bt8qI/PPolRHBwfiCw11dQCQkpK0trIkSjQHDwaAHz/+PZegoKCIiEhhYVGzZ0+6nzDFdtKKlasGKA8MCYtoaWKLjS9fvwoLCbOWiIqKAEBp6eemjcvLK+bOX+joYO8wfdrPnoiJRqN5rvMaN9bc0mJcGw8ZMkTnxYsX7T4jQgixwtyFq3z48KFv3z7s29BoNDk5WdZVnL0VFKSlpQoKCwEgOycHAHh5eXNyc4ktLz9fUXEAUd6IqKhoZVXVyZjTHmvXmY216N1fsbKysi1xWllaJCWnUCiUxKRkXl7eCTbW2lpaDxITAeD+/QfS0lLDhw1lbf86OxsAdEeOZC0cPmwY+7MoDhjA/JrN7eIfP34EAAkJcWZJTm6urKwM60ckIyO9fJnzIA0NYpfZmFhiIiX5b8tmTyQpIfHh48dmz25ooB9z8nhhfs78eXN27dnTX1nVac68h49SW4q26Z3JkpIS1T+qWdtUVlYBgLi4WNPDXVxXkEikndu3tdR/W5w6febFy5e+3l5tP6RPn94fW/gEEELoZ2HuwlVqamoEBQXZt6HRaU0LyWQyhdIAAFQqFQB0DYw0NHWY24PEpOrqH02Pir92XVNnmIurW2np50ULF2RlPlUbOLAtcVpaWtTU1KSlP05KSh4+bKigoKCZ6ejEpGQajXbvwQMrC4tGGQAfH1+zMbM/SxtvGhIQEACA+vp6ZgmFQuEh87Tl2Daqr68XEGC3hKW3gkJIUOC7gvz90Xuyc3KMRpuO0Gt+tezrl8+ZG1GiIC9fVlZOp9OZbb5++0r02ejYq/HxZ86dX7fWo/RzKZGYEledk5ub2/LimKZ279mrNnDgKGOjth8iJCTEun4ZIYR+BeYuXIXBYLR6azSNRist/cx6Y9HHj59KSz+rDVQFgIGqqgBQ/uV
"text/plain": [
"<IPython.core.display.Image object>"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
Add ml1
2016-03-15 12:55:14 +00:00
}
],
"source": [
"from IPython.display import Image \n",
"from sklearn.externals.six import StringIO\n",
"import pydotplus as pydot\n",
"\n",
"dot_data = StringIO() \n",
"tree.export_graphviz(model, out_file=dot_data, \n",
" feature_names=iris.feature_names, \n",
" class_names=iris.target_names, \n",
" filled=True, rounded=True, \n",
" special_characters=True) \n",
"\n",
"\n",
"graph = pydot.graph_from_dot_data(dot_data.getvalue()) \n",
"graph.write_png('iris-tree.png')\n",
"Image(graph.create_png()) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we show a graph of the decision tree boundaries. For each pair of iris features, the decision tree learns decision boundaries made of combinations of simple thresholding rules inferred from the training samples.\n",
"\n",
"We are going to import a function defined in the file [util_ds.py](files/util_ds.py) using the *magic command* **%run**."
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 8,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
added update -all
2019-02-28 11:26:33 +00:00
"ename": "AttributeError",
"evalue": "module 'matplotlib.colors' has no attribute 'to_rgba'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-8-7a926dac062b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;31m# display plots in the notebook\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_line_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'matplotlib'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'inline'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mplot_tree_iris\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m~/GoogleDrive/cursos/sitc/sitc/ml1/util_ds.py\u001b[0m in \u001b[0;36mplot_tree_iris\u001b[0;34m()\u001b[0m\n\u001b[1;32m 115\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msuptitle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Decision surface of a decision tree using paired features\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 116\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlegend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 117\u001b[0;31m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m~/anaconda3/lib/python3.5/site-packages/matplotlib/pyplot.py\u001b[0m in \u001b[0;36mshow\u001b[0;34m(*args, **kw)\u001b[0m\n\u001b[1;32m 242\u001b[0m \"\"\"\n\u001b[1;32m 243\u001b[0m \u001b[0;32mglobal\u001b[0m \u001b[0m_show\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 244\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_show\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 245\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 246\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/anaconda3/lib/python3.5/site-packages/ipykernel/pylab/backend_inline.py\u001b[0m in \u001b[0;36mshow\u001b[0;34m(close, block)\u001b[0m\n\u001b[1;32m 37\u001b[0m display(\n\u001b[1;32m 38\u001b[0m \u001b[0mfigure_manager\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcanvas\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 39\u001b[0;31m \u001b[0mmetadata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0m_fetch_figure_metadata\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfigure_manager\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcanvas\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 40\u001b[0m )\n\u001b[1;32m 41\u001b[0m \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/anaconda3/lib/python3.5/site-packages/ipykernel/pylab/backend_inline.py\u001b[0m in \u001b[0;36m_fetch_figure_metadata\u001b[0;34m(fig)\u001b[0m\n\u001b[1;32m 172\u001b[0m \u001b[0;34m\"\"\"Get some metadata to help with displaying a figure.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 173\u001b[0m \u001b[0;31m# determine if a background is needed for legibility\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 174\u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0m_is_transparent\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_facecolor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 175\u001b[0m \u001b[0;31m# the background is transparent\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 176\u001b[0m ticksLight = _is_light([label.get_color()\n",
"\u001b[0;32m~/anaconda3/lib/python3.5/site-packages/ipykernel/pylab/backend_inline.py\u001b[0m in \u001b[0;36m_is_transparent\u001b[0;34m(color)\u001b[0m\n\u001b[1;32m 193\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_is_transparent\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcolor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 194\u001b[0m \u001b[0;34m\"\"\"Determine transparency from alpha.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 195\u001b[0;31m \u001b[0mrgba\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcolors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_rgba\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcolor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 196\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mrgba\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;36m.5\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mAttributeError\u001b[0m: module 'matplotlib.colors' has no attribute 'to_rgba'"
]
Add ml1
2016-03-15 12:55:14 +00:00
}
],
"source": [
"%run util_ds\n",
"\n",
"# display plots in the notebook \n",
"%matplotlib inline\n",
"plot_tree_iris()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we are going to export the pseudocode of the the learnt decision tree."
]
},
{
"cell_type": "code",
added update -all
2019-02-28 11:26:33 +00:00
"execution_count": null,
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"metadata": {},
added update -all
2019-02-28 11:26:33 +00:00
"outputs": [],
Add ml1
2016-03-15 12:55:14 +00:00
"source": [
"%run util_ds\n",
"get_code(model, iris.feature_names, iris.target_names)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also obtain the feature importance of the fitted model as follows."
]
},
{
"cell_type": "code",
added update -all
2019-02-28 11:26:33 +00:00
"execution_count": null,
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"metadata": {},
added update -all
2019-02-28 11:26:33 +00:00
"outputs": [],
Add ml1
2016-03-15 12:55:14 +00:00
"source": [
"print(iris.feature_names)\n",
"print(model.feature_importances_)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see that the most important feature for this classifier is `petal width`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Evaluating the algorithm"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Precision, recall and f-score"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For evaluating classification algorithms, we usually calculate three metrics: precision, recall and F1-score\n",
"\n",
"* **Precision**: This computes the proportion of instances predicted as positives that were correctly evaluated (it measures how right our classifier is when it says that an instance is positive).\n",
"* **Recall**: This counts the proportion of positive instances that were correctly evaluated (measuring how right our classifier is when faced with a positive instance).\n",
"* **F1-score**: This is the harmonic mean of precision and recall, and tries to combine both in a single number."
]
},
{
"cell_type": "code",
added update -all
2019-02-28 11:26:33 +00:00
"execution_count": null,
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"metadata": {},
added update -all
2019-02-28 11:26:33 +00:00
"outputs": [],
Add ml1
2016-03-15 12:55:14 +00:00
"source": [
"print(metrics.classification_report(y_test, y_test_pred,target_names=iris.target_names))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Confusion matrix"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Another useful metric is the confusion matrix"
]
},
{
"cell_type": "code",
added update -all
2019-02-28 11:26:33 +00:00
"execution_count": null,
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"metadata": {},
added update -all
2019-02-28 11:26:33 +00:00
"outputs": [],
Add ml1
2016-03-15 12:55:14 +00:00
"source": [
"print(metrics.confusion_matrix(y_test, y_test_pred))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see we classify well all the 'setosa' and 'versicolor' samples. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### K-Fold cross validation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In order to avoid bias in the training and testing dataset partition, it is recommended to use **k-fold validation**.\n",
"\n",
"Sklearn comes with other strategies for [cross validation](http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation), such as stratified K-fold, label k-fold, Leave-One-Out, Leave-P-Out, Leave-One-Label-Out, Leave-P-Label-Out or Shuffle & Split."
]
},
{
"cell_type": "code",
added update -all
2019-02-28 11:26:33 +00:00
"execution_count": null,
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"metadata": {},
added update -all
2019-02-28 11:26:33 +00:00
"outputs": [],
Add ml1
2016-03-15 12:55:14 +00:00
"source": [
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"from sklearn.model_selection import cross_val_score, KFold\n",
Add ml1
2016-03-15 12:55:14 +00:00
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import StandardScaler\n",
"\n",
"# create a composite estimator made by a pipeline of preprocessing and the KNN model\n",
"model = Pipeline([\n",
" ('scaler', StandardScaler()),\n",
" ('DecisionTree', DecisionTreeClassifier())\n",
"])\n",
"\n",
"# create a k-fold cross validation iterator of k=10 folds\n",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"cv = KFold(10, shuffle=True, random_state=33)\n",
Add ml1
2016-03-15 12:55:14 +00:00
"\n",
"# by default the score used is the one returned by score method of the estimator (accuracy)\n",
"scores = cross_val_score(model, x_iris, y_iris, cv=cv)\n",
"print(scores)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"We get an array of k scores. We can calculate the mean and the standard error to obtain a final figure"
]
},
{
"cell_type": "code",
added update -all
2019-02-28 11:26:33 +00:00
"execution_count": null,
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"metadata": {},
added update -all
2019-02-28 11:26:33 +00:00
"outputs": [],
Add ml1
2016-03-15 12:55:14 +00:00
"source": [
"from scipy.stats import sem\n",
"def mean_score(scores):\n",
" return (\"Mean score: {0:.3f} (+/- {1:.3f})\").format(np.mean(scores), sem(scores))\n",
"print(mean_score(scores))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So, we get an average accuracy of 0.947."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## References"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* [Plot the decision surface of a decision tree on the iris dataset](http://scikit-learn.org/stable/auto_examples/tree/plot_iris.html)\n",
"* [Learning scikit-learn: Machine Learning in Python](http://proquest.safaribooksonline.com/book/programming/python/9781783281930/1dot-machine-learning-a-gentle-introduction/ch01s02_html), Raúl Garreta; Guillermo Moncecchi, Packt Publishing, 2013.\n",
"* [Python Machine Learning](http://proquest.safaribooksonline.com/book/programming/python/9781783555130), Sebastian Raschka, Packt Publishing, 2015.\n",
"* [Parameter estimation using grid search with cross-validation](http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_digits.html)\n",
"* [Decision trees in python with scikit-learn and pandas](http://chrisstrelioff.ws/sandbox/2015/06/08/decision_trees_in_python_with_scikit_learn_and_pandas.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Licence\n",
"The notebook is freely licensed under under the [Creative Commons Attribution Share-Alike license](https://creativecommons.org/licenses/by/2.0/). \n",
"\n",
added update -all
2019-02-28 11:26:33 +00:00
"© Carlos A. Iglesias, Universidad Politécnica de Madrid."
Add ml1
2016-03-15 12:55:14 +00:00
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
added update -all
2019-02-28 11:26:33 +00:00
"version": "3.5.6"
},
"latex_envs": {
"LaTeX_envs_menu_present": true,
"autocomplete": true,
"bibliofile": "biblio.bib",
"cite_by": "apalike",
"current_citInitial": 1,
"eqLabelWithNumbers": true,
"eqNumInitial": 1,
"hotkeys": {
"equation": "Ctrl-E",
"itemize": "Ctrl-I"
},
"labels_anchors": false,
"latex_user_defs": false,
"report_style_numbering": false,
"user_envs_cfg": false
Add ml1
2016-03-15 12:55:14 +00:00
}
},
"nbformat": 4,
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"nbformat_minor": 1
Add ml1
2016-03-15 12:55:14 +00:00
}
Reference in New Issue Copy Permalink