balkian/sitc
1
0
Fork 0
mirror of https://github.com/gsi-upm/sitc synced 2024-11-05 07:31:41 +00:00
Code Issues Releases Wiki Activity
13b3b2606a
sitc/ml1/2_5_2_Decision_Tree_Model.ipynb

655 lines
58 KiB
Plaintext
Raw Normal View History

Add ml1
2016-03-15 12:55:14 +00:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](files/images/EscUpmPolit_p.gif \"UPM\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Course Notes for Learning Intelligent Systems"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Department of Telematic Engineering Systems, Universidad Politécnica de Madrid, © 2016 Carlos A. Iglesias"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## [Introduction to Machine Learning](2_0_0_Intro_ML.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Table of Contents\n",
"* [Decision Tree Learning](#Decision-Tree-Learning)\n",
"* [Load data and preprocessing](#Load-data-and-preprocessing)\n",
"* [Train classifier](#Train-classifier)\n",
"* [Evaluating the algorithm](#Evaluating-the-algorithm)\n",
"\t* [Precision, recall and f-score](#Precision,-recall-and-f-score)\n",
"\t* [Confusion matrix](#Confusion-matrix)\n",
"\t* [K-Fold cross validation](#K-Fold-cross-validation)\n",
"* [References](#References)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Decision Tree Learning"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
Fix visualization section
2017-12-11 17:12:06 +00:00
"The goal of this notebook is to learn how to create a classification object using a [decision tree learning algorithm](https://en.wikipedia.org/wiki/Decision_tree_learning). \n",
Add ml1
2016-03-15 12:55:14 +00:00
"\n",
"There are a number of well known machine learning algorithms for decision tree learning, such as ID3, C4.5, C5.0 and CART. The scikit-learn uses an optimised version of the [CART (Classification and Regression Trees) algorithm](https://en.wikipedia.org/wiki/Predictive_analytics#Classification_and_regression_trees).\n",
"\n",
Typo pecualarities fixed
2017-03-06 13:42:05 +00:00
"This notebook will follow the same steps that the previous notebook for learning using the [kNN Model](2_5_1_kNN_Model.ipynb), and details some peculiarities of the decision tree algorithms."
Add ml1
2016-03-15 12:55:14 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data and preprocessing\n",
"\n",
"Here we repeat the same operations for loading data and preprocessing than in the previous notebooks."
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 1,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [],
"source": [
"# library for displaying plots\n",
"import matplotlib.pyplot as plt\n",
"# display plots in the notebook \n",
"%matplotlib inline\n",
"\n",
"## First, we repeat the load and preprocessing steps\n",
"\n",
"# Load data\n",
"from sklearn import datasets\n",
"iris = datasets.load_iris()\n",
"\n",
"# Training and test spliting\n",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"from sklearn.model_selection import train_test_split\n",
Add ml1
2016-03-15 12:55:14 +00:00
"\n",
"x_iris, y_iris = iris.data, iris.target\n",
"# Test set will be the 25% taken randomly\n",
"x_train, x_test, y_train, y_test = train_test_split(x_iris, y_iris, test_size=0.25, random_state=33)\n",
"\n",
"# Preprocess: normalize\n",
"from sklearn import preprocessing\n",
"scaler = preprocessing.StandardScaler().fit(x_train)\n",
"x_train = scaler.transform(x_train)\n",
"x_test = scaler.transform(x_test)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Train classifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The usual steps for creating a classifier are:\n",
"1. Create classifier object\n",
"2. Call *fit* to train the classifier\n",
"3. Call *predict* to obtain predictions\n",
"\n",
"*DecisionTreeClassifier* is capable of both binary (where the labels are [-1, 1]) classification and multiclass (where the labels are [0, ..., K-1]) classification."
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 2,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"data": {
"text/plain": [
"DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=3,\n",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
" max_features=None, max_leaf_nodes=None,\n",
" min_impurity_split=1e-07, min_samples_leaf=1,\n",
Add ml1
2016-03-15 12:55:14 +00:00
" min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
" presort=False, random_state=1, splitter='best')"
]
},
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 2,
Add ml1
2016-03-15 12:55:14 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.tree import DecisionTreeClassifier\n",
"import numpy as np\n",
"\n",
"from sklearn import tree\n",
"\n",
"max_depth=3\n",
"random_state=1\n",
"\n",
"# Create decision tree model\n",
"model = tree.DecisionTreeClassifier(max_depth=max_depth, random_state=random_state)\n",
"\n",
"# Train the model using the training sets\n",
"model.fit(x_train, y_train) "
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 3,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Prediction [1 0 1 1 1 0 0 1 0 2 0 0 1 2 0 1 2 2 1 1 0 0 2 0 0 2 1 1 2 2 2 2 0 0 1 1 0\n",
" 1 2 1 2 0 2 0 1 0 2 1 0 2 2 0 0 2 0 0 0 2 2 0 1 0 1 0 1 1 1 1 1 0 1 0 1 2\n",
" 0 0 0 0 2 2 0 1 1 2 1 0 0 2 1 1 0 1 1 0 2 1 2 1 2 0 1 0 0 0 2 1 2 1 2 1 2\n",
" 0]\n",
"Expected [1 0 1 1 1 0 0 1 0 2 0 0 1 2 0 1 2 2 1 1 0 0 2 0 0 2 1 1 2 2 2 2 0 0 1 1 0\n",
" 1 2 1 2 0 2 0 1 0 2 1 0 2 2 0 0 2 0 0 0 2 2 0 1 0 1 0 1 1 1 1 1 0 1 0 1 2\n",
" 0 0 0 0 2 2 0 1 1 2 1 0 0 1 1 1 0 1 1 0 2 2 2 1 2 0 1 0 0 0 2 1 2 1 2 1 2\n",
" 0]\n"
]
}
],
"source": [
"print(\"Prediction \", model.predict(x_train))\n",
"print(\"Expected \", y_train)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively, the probability of each class can be predicted, which is the fraction of training samples of the same class in a leaf:"
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 4,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Predicted probabilities [[ 0. 0.97368421 0.02631579]\n",
" [ 1. 0. 0. ]\n",
" [ 0. 0.97368421 0.02631579]\n",
" [ 0. 0.97368421 0.02631579]\n",
" [ 0. 0.97368421 0.02631579]\n",
" [ 1. 0. 0. ]\n",
" [ 1. 0. 0. ]\n",
" [ 0. 0.97368421 0.02631579]\n",
" [ 1. 0. 0. ]\n",
" [ 0. 0. 1. ]]\n"
]
}
],
"source": [
"# Print the \n",
"print(\"Predicted probabilities\", model.predict_proba(x_train[:10]))"
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 5,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy in training 0.982142857143\n"
]
}
],
"source": [
"# Evaluate Accuracy in training\n",
"\n",
"from sklearn import metrics\n",
"y_train_pred = model.predict(x_train)\n",
"print(\"Accuracy in training\", metrics.accuracy_score(y_train, y_train_pred))"
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 6,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy in testing 0.921052631579\n"
]
}
],
"source": [
"# Now we evaluate error in testing\n",
"y_test_pred = model.predict(x_test)\n",
"print(\"Accuracy in testing \", metrics.accuracy_score(y_test, y_test_pred))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we are going to visualize the DecisionTree classification. It will plot the decision boundaries for each class.\n",
"\n",
"The current version of pydot does not work well in Python 3.\n",
"For obtaining an image, you need to install `pip install pydotplus` and then `conda install graphviz`.\n",
"\n",
"You can skip this example. Since it can require installing additional packages, we include here the result.\n",
"![Decision Tree](files/images/cart.png)"
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 7,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
Fixed age charts
2016-03-29 10:48:56 +00:00
{
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"ename": "ModuleNotFoundError",
"evalue": "No module named 'pydotplus'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-7-1bf5ec7fb043>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mIPython\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisplay\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mImage\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexternals\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msix\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mStringIO\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mpydotplus\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mpydot\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mdot_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mStringIO\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'pydotplus'"
Fixed age charts
2016-03-29 10:48:56 +00:00
]
Add ml1
2016-03-15 12:55:14 +00:00
}
],
"source": [
"from IPython.display import Image \n",
"from sklearn.externals.six import StringIO\n",
"import pydotplus as pydot\n",
"\n",
"dot_data = StringIO() \n",
"tree.export_graphviz(model, out_file=dot_data, \n",
" feature_names=iris.feature_names, \n",
" class_names=iris.target_names, \n",
" filled=True, rounded=True, \n",
" special_characters=True) \n",
"\n",
"\n",
"graph = pydot.graph_from_dot_data(dot_data.getvalue()) \n",
"graph.write_png('iris-tree.png')\n",
"Image(graph.create_png()) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we show a graph of the decision tree boundaries. For each pair of iris features, the decision tree learns decision boundaries made of combinations of simple thresholding rules inferred from the training samples.\n",
"\n",
"We are going to import a function defined in the file [util_ds.py](files/util_ds.py) using the *magic command* **%run**."
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 8,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"data": {
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEjCAYAAAAypHaFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXmcXEW1+L+nu2fpWZNJJpNMIAkJayIIiAEjmwluLCL+FMXlqTzE7T0E5YkgT3FFXFBRUBF874mIAi4oxiVEEcIWdhASIAlJSDKTTLaetWemu8/vj3t7ppd7u29P7zP1/XzmM933Vtc991bdOlWnTp0SVcVgMBgMBl+5BTAYDAZDZWAUgsFgMBgAoxAMBoPBYGMUgsFgMBgAoxAMBoPBYGMUgsFgMBiAKaQQROTPIvJBD+n6RWRhKWSaKCJymIg8KSJ9InJRueUBEJFzROQV+/kdU+C8PyQia/LMY54tmz9LuveJyN/yuVa1IyLPicip5ZYjERE5SUReKGB+p4rItgznXy8iL9l15u2Fum6lI5W0DkFENgMdQASIAs8DPwduVNVYGUWrKETkZqBXVS8ptyxxRGQj8GlVvasIeX8IuEBVTyx03qXCbmB/oaoHlFsWQ/byEJHVwB9U9fsFuNZmrPp7T755FZtKHCGcparNwHzgG8BlwM3lFakyEJGA/XE+8Fw5ZXGgEmWqKhLK15ADRXpuFVOfS1ovVLVi/oDNwGkpx5YCMeBV9vc64NvAVmAn8GMgmJD+bOApoBfYCLzFPn4vlpYGOBj4JxACdgO/Tvi9Agfbn1uxRig9wBbgSsBnn/sQsMaWZR/wMvDWDPd2GbAd6ANeAFbYx/8X+GpCulOBbSnP5DLgGWAY+DvW6CkM9AOHAmcAT9r3/ApwVcq1TwQeBPbb5z/k5Vmm5OGz738LsMt+Lq12Hv32cxsANrr8/vv2tXuBx4GTMjyrGcAf7LRrga8AaxLOHw6sAvbaz/LchHNB4Du2nCG7jILAAlvGQEL5bbLL42XgfYnlmpDfMuBRO69HgWUJ5+61ZXvAzudvwEyH+2kEhrDqcb/91wlcBdwJ/MK+1wvs5/w5rLq7B7gdaEvI64SEsnwaODXDcxyry6l1DZgJ3G3nsxe4n/G6vRn7PbRlvN0u7z6sRvK4hDyPxap7fcAdwK9JqM8p8nzIflY/sJ/neuz3wD7/YWCdndcm4KM5vBcB+5n+But9fRm4KKVe/C/Wu/o88F+J+aXIudEuqyG7rOqw6vrNQBfWe/xVwG+nX4T1Xu7Bak9uBabZ525Jyeuzqffi8sw91wug3k67xy7PR4GOCbXBhWrMC/GHg0Kwj28FPm5//h5WY9EGNAN/BK62zy21K9ob7Qc4Fzg84eWNK4TbgM/baeqBE51eIqyX4C77OguAF4F/T6jco8BHAD/wcWAHthkuRf7DsBrDTvv7AmBR6kuaoeI/BRyI3Vgn3kvCb4607+corMb97fa5eVgv2HlADVZje3S2Z+lwD+cDG4CFQBPwW+AWt8bH4ffvt68dAD4DdAP1Lml/ZVf4RuBVWC/gGvtco/0sP2zndSzWS7jEPn+9/Xzm2uWyDOuFXmDLGLDz6AUOs38zJ+H3H0q4VhtWA/IB+3fn2d9nJJTDRiylHLS/f8PlnpLKNeHFHwXebpddELgYeBg4wJb7J8Btdvq5WC/96Xb6N9rf212umUkhXI3VAaix/05i3IS8meTGKWxf02//7mH7XC2W4v2Uncc7gBEyK4QIcImd/t1Y72u8YTsDq3EV4BRgEDjWy3thP4/HgS/Yci3EUipvttN/A0vptdm/+VdqeWRqi4Df22XRCMzC6qh81D53sF0WdUA7cB/wvQx5OdWF1GeeS734KNa722CX0WuAlgm1wfk24oX8S31wCccfxmrABasXuijh3OuAl+3PPwG+65L3vYwrhJ8DNwIHuL1E9oMdBhYnnPsocG9C5d6QcK7B/u1shzwPxupVnwbUpJz7X7IrhPPd7sXlXr8Xfw7A5cDvHNJkfJYO6VcDn0j4fphdaeM97owKwSG/fcCrHY777XwPTzj2dcYb6XcD96f85ifAF+2XZ8gl3wUkK4T9wP8jZUREskL4ALA25fxDjI+w7gWuTDj3CeAvLvebVK72sauA+1KOrSO51zwn/pyxesS3pKT/K/BBl2tmUghfxurspJUZ6Y3TPQnnFgND9ueTsZS1JJxfQ2aFsCMl/VrgAy7pfw98yst7ARwPbE35/eXA/9ifN2FbC+zvF6aWR4Zn0IHVFiRaIs4D/uHy27cDTzrllaEupD7zXOrF+VijxqO8vn9uf5U4h+DEXKxhbTtWw/u4iOwXkf3AX+zjYGn+jR7y+yxWg7jW9qg43yHNTMZ7QHG22LLE6Y5/UNVB+2NTakaqugFLw18F7BKRX4lIpwc547yS6aSIHC8i/xCRHhEJAR+z5Qf3Z5LtWabSSfqzCGC9LFkRkc+IyDoRCdnXak2QMVWuAMn3nHjd+cDxcZntvN4HzLbzqydLHVDVASzF8jGgS0T+JCKHOyRNvee4LI51AKtHm1b+WUgt2/nA7xLubR2WibDDPveulHs/EatxyJVvYY34/iYim0TkcxnSpt5jvW3X7gS2q91CudxPKqnpt9j5ICJvFZGHRWSvfW+n41xHnK41H+hMeTZXMF4/O3GvU9mYjzWi6UrI+ydYIwVEZJb9Tm8XkV4s800mub2QS724Batj8CsR2SEi3xSRmolctOIVgoi8FusFXINlGhjCGt5Ps/9aVTX+Er6CNeTMiKp2q+pHVLUTq9d/g4gcnJJsN5YGnp9wbB5WjyhnVPWXannJzMfquV1jnxrAapjjzHb6eZbsf4ll+jlQVVuxTAFin3N7JtmeZSo7SH8WESzzVEZE5CSs3u25wHRVnYZlKhCH5D12vgemXCvOK8A/E2SepqpNqvpx+57CLvebhKr+VVXfiNWYrgd+6pAs9Z7jskykDriVYerxV7DmohLvr15Vt9vnbkk516iq33DJexCXuqWqfar6GVVdCJwFfFpEVuR4T13AXBFJLMcD3RLbpKafB+wQkTos+/+3sezf04CVONeRsdtI+PwK1ug28dk0q+rpCbK61alsvII1QpiZkHeLqi6xz19ty3KUqrZgmUcT5U4t46R33naFTu2Iea4Xqjqqql9S1cVYJtIzgX/L4f7GqFiFICItInImlj35F6r6rFqupz8Fvisice08V0TebP/sZuDDIrJCRHz2ubSen4i8S0Ti7mb7sB5+NDGNqkax7NhfE5FmEZkPfBpL++d6L4eJyHK70oexGuL49Z4CTheRNhGZjTWSyJVmYK+qhkVkKfDehHO3AqeJyLkiEhCRGSJytIdnmcptwCUicpCINGGZcX6tqhGP8kWwGvuAiHwBaHFKaD/33wJXiUiDiCwGPpiQ5G7gUBH5gIjU2H+vFZEj7Hv6GXCtiHSKiF9EXmc/9zFEpENE3iYijVgvej8p5W+z0r7We+1n924sk8ndHu45lZ3ADBFpzZLux1h1br4ta7uInG2f+wVwloi82b63erH86d1cWZ8C3munfQuWXR473zNF5GC7ce7Fun+nZ5CJh+zf/If9fM7GmsfLxCzgIrvc3gUcgfWca7Fs4z1ARETeCrwpB1nWAr0icpmIBO17fpXdoQTrXb5cRKbbz+s/vWasql1YDgPfsdsln4gsEpH482zGqkP7RWQu1oR1Ijux5jTivIg1yjrD7slfad97JlzrhYi8QUSOtBVLL1ZHNteyBCpTIfxRRPqwNOLngWuxJhDjXIY11H3YHp7dg2XPRlXX2mm/i9UD/SfpPTyA1wKPiEg/Vs/6U6r6skO6/8TS5puwRii/xGpwcqUOa1JrN9bwexbWcBas4d7TWDbEv2F5aeTKJ4Av28/tC1iVHwBV3Yo19P4MltntKeDV9mnXZ+nAz2xZ78Py4Ajj/aX6K/BnrBdhi/3bTKaF/8AyvXRj2b3/J+F++rAaivdg9eC7sUZb8RfqUuBZLE+Lvfa51Hruw3oeO+w0p2A9wyRUdQ9Wb+szWJO3nwXOVNXdXm46Ja/1WEp1kz3sdzMZfh+rTv7NLs+HsezjqOorWF50V2A1nK9gNT5u7/GnsHr/cbPa7xPOHYJV3v1YDfsNqnp
Add ml1
2016-03-15 12:55:14 +00:00
"text/plain": [
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"<matplotlib.figure.Figure at 0x7f6e096d9eb8>"
Add ml1
2016-03-15 12:55:14 +00:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%run util_ds\n",
"\n",
"# display plots in the notebook \n",
"%matplotlib inline\n",
"plot_tree_iris()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we are going to export the pseudocode of the the learnt decision tree."
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 9,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"if ( petal width (cm) <= -0.415243178606 ) {\n",
" return setosa ( 42 examples )\n",
"}\n",
"else {\n",
" if ( petal width (cm) <= 0.861689627171 ) {\n",
" if ( petal length (cm) <= 0.818572163582 ) {\n",
" return versicolor ( 37 examples )\n",
" return virginica ( 1 examples )\n",
" }\n",
" else {\n",
" return versicolor ( 1 examples )\n",
" return virginica ( 2 examples )\n",
" }\n",
" }\n",
" else {\n",
" if ( petal length (cm) <= 0.707377433777 ) {\n",
" return versicolor ( 1 examples )\n",
" }\n",
" else {\n",
" return virginica ( 28 examples )\n",
" }\n",
" }\n",
"}\n"
]
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
},
{
"data": {
"text/plain": [
"<matplotlib.figure.Figure at 0x7f6e0709def0>"
]
},
"metadata": {},
"output_type": "display_data"
Add ml1
2016-03-15 12:55:14 +00:00
}
],
"source": [
"%run util_ds\n",
"get_code(model, iris.feature_names, iris.target_names)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also obtain the feature importance of the fitted model as follows."
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 10,
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']\n",
"[ 0. 0. 0.05947455 0.94052545]\n"
]
}
],
"source": [
"print(iris.feature_names)\n",
"print(model.feature_importances_)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see that the most important feature for this classifier is `petal width`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Evaluating the algorithm"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Precision, recall and f-score"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For evaluating classification algorithms, we usually calculate three metrics: precision, recall and F1-score\n",
"\n",
"* **Precision**: This computes the proportion of instances predicted as positives that were correctly evaluated (it measures how right our classifier is when it says that an instance is positive).\n",
"* **Recall**: This counts the proportion of positive instances that were correctly evaluated (measuring how right our classifier is when faced with a positive instance).\n",
"* **F1-score**: This is the harmonic mean of precision and recall, and tries to combine both in a single number."
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 11,
"metadata": {},
Review J
2016-03-28 10:26:20 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" precision recall f1-score support\n",
"\n",
" setosa 1.00 1.00 1.00 8\n",
" versicolor 0.79 1.00 0.88 11\n",
" virginica 1.00 0.84 0.91 19\n",
"\n",
"avg / total 0.94 0.92 0.92 38\n",
"\n"
]
}
],
Add ml1
2016-03-15 12:55:14 +00:00
"source": [
"print(metrics.classification_report(y_test, y_test_pred,target_names=iris.target_names))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Confusion matrix"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Another useful metric is the confusion matrix"
]
},
{
"cell_type": "code",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"execution_count": 12,
"metadata": {},
Review J
2016-03-28 10:26:20 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[ 8 0 0]\n",
" [ 0 11 0]\n",
" [ 0 3 16]]\n"
]
}
],
Add ml1
2016-03-15 12:55:14 +00:00
"source": [
"print(metrics.confusion_matrix(y_test, y_test_pred))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see we classify well all the 'setosa' and 'versicolor' samples. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### K-Fold cross validation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In order to avoid bias in the training and testing dataset partition, it is recommended to use **k-fold validation**.\n",
"\n",
"Sklearn comes with other strategies for [cross validation](http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation), such as stratified K-fold, label k-fold, Leave-One-Out, Leave-P-Out, Leave-One-Label-Out, Leave-P-Label-Out or Shuffle & Split."
]
},
{
"cell_type": "code",
Fixed age charts
2016-03-29 10:48:56 +00:00
"execution_count": 14,
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ 1. 0.8 1. 0.93333333 0.93333333 1. 1.\n",
" 1. 0.86666667 0.93333333]\n"
]
}
],
"source": [
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"from sklearn.model_selection import cross_val_score, KFold\n",
Add ml1
2016-03-15 12:55:14 +00:00
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import StandardScaler\n",
"\n",
"# create a composite estimator made by a pipeline of preprocessing and the KNN model\n",
"model = Pipeline([\n",
" ('scaler', StandardScaler()),\n",
" ('DecisionTree', DecisionTreeClassifier())\n",
"])\n",
"\n",
"# create a k-fold cross validation iterator of k=10 folds\n",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"cv = KFold(10, shuffle=True, random_state=33)\n",
Add ml1
2016-03-15 12:55:14 +00:00
"\n",
"# by default the score used is the one returned by score method of the estimator (accuracy)\n",
"scores = cross_val_score(model, x_iris, y_iris, cv=cv)\n",
"print(scores)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"We get an array of k scores. We can calculate the mean and the standard error to obtain a final figure"
]
},
{
"cell_type": "code",
Fixed age charts
2016-03-29 10:48:56 +00:00
"execution_count": 15,
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"metadata": {},
Add ml1
2016-03-15 12:55:14 +00:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Mean score: 0.947 (+/- 0.022)\n"
]
}
],
"source": [
"from scipy.stats import sem\n",
"def mean_score(scores):\n",
" return (\"Mean score: {0:.3f} (+/- {1:.3f})\").format(np.mean(scores), sem(scores))\n",
"print(mean_score(scores))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So, we get an average accuracy of 0.947."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## References"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* [Plot the decision surface of a decision tree on the iris dataset](http://scikit-learn.org/stable/auto_examples/tree/plot_iris.html)\n",
"* [Learning scikit-learn: Machine Learning in Python](http://proquest.safaribooksonline.com/book/programming/python/9781783281930/1dot-machine-learning-a-gentle-introduction/ch01s02_html), Raúl Garreta; Guillermo Moncecchi, Packt Publishing, 2013.\n",
"* [Python Machine Learning](http://proquest.safaribooksonline.com/book/programming/python/9781783555130), Sebastian Raschka, Packt Publishing, 2015.\n",
"* [Parameter estimation using grid search with cross-validation](http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_digits.html)\n",
"* [Decision trees in python with scikit-learn and pandas](http://chrisstrelioff.ws/sandbox/2015/06/08/decision_trees_in_python_with_scikit_learn_and_pandas.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Licence\n",
"The notebook is freely licensed under under the [Creative Commons Attribution Share-Alike license](https://creativecommons.org/licenses/by/2.0/). \n",
"\n",
"© 2016 Carlos A. Iglesias, Universidad Politécnica de Madrid."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"version": "3.6.3"
Add ml1
2016-03-15 12:55:14 +00:00
}
},
"nbformat": 4,
adapted some calls to new scikit version
2018-02-23 14:48:59 +00:00
"nbformat_minor": 1
Add ml1
2016-03-15 12:55:14 +00:00
}
Reference in New Issue Copy Permalink