{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](files/images/EscUpmPolit_p.gif \"UPM\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Course Notes for Learning Intelligent Systems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Department of Telematic Engineering Systems, Universidad Politécnica de Madrid, © 2016 Carlos A. Iglesias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Introduction to Machine Learning](2_0_0_Intro_ML.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "\n",
    "* [Model Tuning](#Model-Tuning)\n",
    "* [Load data and preprocessing](#Load-data-and-preprocessing)\n",
    "* [Train classifier](#Train-classifier)\n",
    "* [More about Pipelines](#More-about-Pipelines)\n",
    "* [Tuning the algorithm](#Tuning-the-algorithm)\n",
    "\t* [Grid Search for Parameter optimization](#Grid-Search-for-Parameter-optimization)\n",
    "* [Evaluating the algorithm](#Evaluating-the-algorithm)\n",
    "\t* [K-Fold validation](#K-Fold-validation)\n",
    "* [References](#References)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous [notebook](2_5_2_Decision_Tree_Model.ipynb), we got an accuracy of 9.47. Could we get a better accuracy if we tune the parameters of the estimator?\n",
    "\n",
    "The goal of this notebook is to learn how to tune an algorithm by opimizing its parameters using grid search."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data and preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# library for displaying plots\n",
    "import matplotlib.pyplot as plt\n",
    "# display plots in the notebook \n",
    "%matplotlib inline\n",
    "\n",
    "## First, we repeat the load and preprocessing steps\n",
    "\n",
    "# Load data\n",
    "from sklearn import datasets\n",
    "iris = datasets.load_iris()\n",
    "\n",
    "# Training and test spliting\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "x_iris, y_iris = iris.data, iris.target\n",
    "# Test set will be the 25% taken randomly\n",
    "x_train, x_test, y_train, y_test = train_test_split(x_iris, y_iris, test_size=0.25, random_state=33)\n",
    "\n",
    "# Preprocess: normalize\n",
    "from sklearn import preprocessing\n",
    "scaler = preprocessing.StandardScaler().fit(x_train)\n",
    "x_train = scaler.transform(x_train)\n",
    "x_test = scaler.transform(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Train classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As previously, we train the model and evaluate the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean score: 0.940 (+/- 0.021)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score, KFold\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "import numpy as np\n",
    "\n",
    "# create a composite estimator made by a pipeline of preprocessing and the KNN model\n",
    "model = Pipeline([\n",
    "        ('scaler', StandardScaler()),\n",
    "        ('ds', DecisionTreeClassifier())\n",
    "])\n",
    "\n",
    "# Fit the model\n",
    "model.fit(x_train, y_train) \n",
    "\n",
    "# create a k-fold cross validation iterator of k=10 folds\n",
    "cv = KFold(10, shuffle=True, random_state=33)\n",
    "\n",
    "# by default the score used is the one returned by score method of the estimator (accuracy)\n",
    "scores = cross_val_score(model, x_iris, y_iris, cv=cv)\n",
    "\n",
    "from scipy.stats import sem\n",
    "def mean_score(scores):\n",
    "    return (\"Mean score: {0:.3f} (+/- {1:.3f})\").format(np.mean(scores), sem(scores))\n",
    "print(mean_score(scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We obtain an accuracy of 0.947."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More about Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we use a Pipeline, every chained estimator is stored in the dictionary *named_steps* and as a list in *steps*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ds': DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "             max_features=None, max_leaf_nodes=None,\n",
       "             min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "             min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "             presort=False, random_state=None, splitter='best'),\n",
       " 'scaler': StandardScaler(copy=True, with_mean=True, with_std=True)}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.named_steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)),\n",
       " ('ds',\n",
       "  DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "              max_features=None, max_leaf_nodes=None,\n",
       "              min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "              min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "              presort=False, random_state=None, splitter='best'))]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get the list of parameters of the model. As you will observe, the parameters of the estimators in the pipeline can be accessed using the &lt;estimator&gt;__&lt;parameter&gt; syntax. We will use this for tuning the parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['steps', 'scaler', 'ds', 'scaler__copy', 'scaler__with_mean', 'scaler__with_std', 'ds__class_weight', 'ds__criterion', 'ds__max_depth', 'ds__max_features', 'ds__max_leaf_nodes', 'ds__min_impurity_split', 'ds__min_samples_leaf', 'ds__min_samples_split', 'ds__min_weight_fraction_leaf', 'ds__presort', 'ds__random_state', 'ds__splitter'])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_params().keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what happens if we change a parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)), ('ds', DecisionTreeClassifier(class_weight='balanced', criterion='gini',\n",
       "            max_depth=None, max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            presort=False, random_state=None, splitter='best'))])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.set_params(ds__class_weight='balanced')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another alternative is to create the pipeline with the values we want to set, but it can be useful to access the estimators of the Pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)), ('ds', DecisionTreeClassifier(class_weight='balanced', criterion='gini',\n",
       "            max_depth=None, max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            presort=False, random_state=None, splitter='best'))])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = Pipeline([\n",
    "        ('scaler', StandardScaler()),\n",
    "        ('ds', DecisionTreeClassifier(class_weight='balanced'))\n",
    "])\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same approach can be used for accessing attributes such as *feature_importances_* we saw in the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.01834862  0.01910853  0.05728223  0.90526062]\n"
     ]
    }
   ],
   "source": [
    "# Fit the model\n",
    "model.fit(x_train, y_train) \n",
    "# Using named_steps\n",
    "my_decision_tree = model.named_steps['ds']\n",
    "print(my_decision_tree.feature_importances_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.01834862  0.01910853  0.05728223  0.90526062]\n"
     ]
    }
   ],
   "source": [
    "#Using steps, we take the last step (-1) or the second step (1)\n",
    "#name, my_desision_tree = model.steps[1]\n",
    "name, my_desision_tree = model.steps[-1]\n",
    "print(my_decision_tree.feature_importances_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning the algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the most important feature for this classifier is `petal width`.\n",
    "\n",
    "Look at the [API](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) of *scikit-learn* to understand better the algorithm, as well as which parameters can be tuned. As you see, we can change several ones, such as *criterion*, *splitter*, *max_features*, *max_depth*, *min_samples_split*, *class_weight*, etc.\n",
    "\n",
    "We can get the full list parameters of an estimator with the method *get_params()*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ds': DecisionTreeClassifier(class_weight='balanced', criterion='gini',\n",
       "             max_depth=None, max_features=None, max_leaf_nodes=None,\n",
       "             min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "             min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "             presort=False, random_state=None, splitter='best'),\n",
       " 'ds__class_weight': 'balanced',\n",
       " 'ds__criterion': 'gini',\n",
       " 'ds__max_depth': None,\n",
       " 'ds__max_features': None,\n",
       " 'ds__max_leaf_nodes': None,\n",
       " 'ds__min_impurity_split': 1e-07,\n",
       " 'ds__min_samples_leaf': 1,\n",
       " 'ds__min_samples_split': 2,\n",
       " 'ds__min_weight_fraction_leaf': 0.0,\n",
       " 'ds__presort': False,\n",
       " 'ds__random_state': None,\n",
       " 'ds__splitter': 'best',\n",
       " 'scaler': StandardScaler(copy=True, with_mean=True, with_std=True),\n",
       " 'scaler__copy': True,\n",
       " 'scaler__with_mean': True,\n",
       " 'scaler__with_std': True,\n",
       " 'steps': [('scaler',\n",
       "   StandardScaler(copy=True, with_mean=True, with_std=True)),\n",
       "  ('ds', DecisionTreeClassifier(class_weight='balanced', criterion='gini',\n",
       "               max_depth=None, max_features=None, max_leaf_nodes=None,\n",
       "               min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "               min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "               presort=False, random_state=None, splitter='best'))]}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can try different values for these parameters and observe the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grid Search for Parameter optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Changing manually the parameters to find their optimal values is not practical. Instead, we can consider to find the optimal value of the parameters as an *optimization problem*. \n",
    "\n",
    "The sklearn comes with several optimization techniques for this purpose, such as  **grid search** and  **randomized search**. In this notebook we are going to introduce the former one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sklearn provides an object that, given data, computes the score during the fit of an estimator on a parameter grid and chooses the parameters to maximize the cross-validation score. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best score:  0.946428571429\n",
      "Best params:  {'max_depth': 3}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "import numpy as np\n",
    "\n",
    "param_grid = {'max_depth': np.arange(3, 10)} \n",
    "\n",
    "gs = GridSearchCV(DecisionTreeClassifier(), param_grid)\n",
    "\n",
    "gs.fit(x_train, y_train)\n",
    "\n",
    "# summarize the results of the grid search\n",
    "print(\"Best score: \", gs.best_score_)\n",
    "print(\"Best params: \", gs.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now we are going to show the results of grid search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.946 (+/-0.075) for {'max_depth': 3}\n",
      "0.929 (+/-0.024) for {'max_depth': 4}\n",
      "0.946 (+/-0.075) for {'max_depth': 5}\n",
      "0.929 (+/-0.024) for {'max_depth': 6}\n",
      "0.946 (+/-0.075) for {'max_depth': 7}\n",
      "0.946 (+/-0.075) for {'max_depth': 8}\n",
      "0.929 (+/-0.024) for {'max_depth': 9}\n"
     ]
    }
   ],
   "source": [
    "# We print the score for each value of max_depth\n",
    "for i, max_depth in enumerate(gs.cv_results_['params']):\n",
    "    print(\"%0.3f (+/-%0.03f) for %r\" % (gs.cv_results_['mean_test_score'][i],\n",
    "                                        gs.cv_results_['std_test_score'][i] * 2,\n",
    "                                        max_depth))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now evaluate the KFold with this optimized parameter as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean score: 0.953 (+/- 0.020)\n"
     ]
    }
   ],
   "source": [
    "# create a composite estimator made by a pipeline of preprocessing and the KNN model\n",
    "model = Pipeline([\n",
    "        ('scaler', StandardScaler()),\n",
    "        ('ds', DecisionTreeClassifier(max_depth=3))\n",
    "])\n",
    "\n",
    "# Fit the model\n",
    "model.fit(x_train, y_train) \n",
    "\n",
    "# create a k-fold cross validation iterator of k=10 folds\n",
    "cv = KFold(10, shuffle=True, random_state=33)\n",
    "\n",
    "# by default the score used is the one returned by score method of the estimator (accuracy)\n",
    "scores = cross_val_score(model, x_iris, y_iris, cv=cv)\n",
    "def mean_score(scores):\n",
    "    return (\"Mean score: {0:.3f} (+/- {1:.3f})\").format(np.mean(scores), sem(scores))\n",
    "print(mean_score(scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have got an *improvement* from 0.947 to 0.953 with k-fold.\n",
    "\n",
    "We are now to try to fit the best combination of the parameters of the algorithm. It can take some time to compute it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Tuning hyper-parameters for precision\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "/opt/conda/lib/python3.6/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "/opt/conda/lib/python3.6/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters set found on development set:\n",
      "\n",
      "{'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.964 (+/-0.092) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.943 (+/-0.084) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.123) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.936 (+/-0.122) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.953 (+/-0.126) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.973 (+/-0.068) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.968 (+/-0.132) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.946 (+/-0.123) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.943 (+/-0.081) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.123) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.919 (+/-0.251) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.975 (+/-0.079) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.953 (+/-0.126) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.123) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.951 (+/-0.118) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.953 (+/-0.126) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.943 (+/-0.113) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.953 (+/-0.126) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.948 (+/-0.108) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.953 (+/-0.126) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.961 (+/-0.081) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.953 (+/-0.126) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.928 (+/-0.165) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.953 (+/-0.126) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.949 (+/-0.118) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.953 (+/-0.134) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.946 (+/-0.123) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.123) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.942 (+/-0.067) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.980 (+/-0.062) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.945 (+/-0.141) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.949 (+/-0.095) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.953 (+/-0.126) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.946 (+/-0.123) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.953 (+/-0.126) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.961 (+/-0.114) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.972 (+/-0.069) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.953 (+/-0.126) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.950 (+/-0.118) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.123) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.946 (+/-0.125) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.938 (+/-0.142) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.956 (+/-0.121) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.968 (+/-0.082) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.956 (+/-0.097) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.906 (+/-0.296) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.932 (+/-0.110) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.955 (+/-0.121) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.921 (+/-0.132) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.942 (+/-0.132) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.948 (+/-0.108) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.945 (+/-0.123) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.897 (+/-0.187) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.944 (+/-0.148) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.948 (+/-0.107) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.950 (+/-0.118) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.961 (+/-0.081) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.939 (+/-0.117) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.949 (+/-0.090) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.972 (+/-0.068) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.950 (+/-0.118) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.906 (+/-0.162) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.947 (+/-0.146) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.950 (+/-0.118) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.946 (+/-0.123) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.953 (+/-0.134) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.975 (+/-0.079) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.932 (+/-0.136) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.940 (+/-0.146) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.953 (+/-0.082) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.979 (+/-0.064) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.952 (+/-0.108) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.968 (+/-0.082) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.919 (+/-0.106) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.941 (+/-0.129) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.943 (+/-0.113) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.956 (+/-0.094) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.954 (+/-0.154) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.949 (+/-0.158) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.943 (+/-0.113) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.893 (+/-0.163) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.916 (+/-0.186) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.943 (+/-0.113) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.961 (+/-0.081) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.947 (+/-0.108) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.912 (+/-0.120) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.960 (+/-0.082) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.943 (+/-0.113) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.962 (+/-0.113) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.966 (+/-0.070) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.962 (+/-0.113) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.943 (+/-0.113) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.949 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.954 (+/-0.112) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.943 (+/-0.113) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.955 (+/-0.097) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.974 (+/-0.081) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.947 (+/-0.175) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.950 (+/-0.117) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.943 (+/-0.113) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.935 (+/-0.075) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.954 (+/-0.129) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.940 (+/-0.142) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.934 (+/-0.155) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.927 (+/-0.112) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.943 (+/-0.113) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.934 (+/-0.184) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.932 (+/-0.136) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.968 (+/-0.082) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.903 (+/-0.240) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.939 (+/-0.179) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.975 (+/-0.079) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.923 (+/-0.094) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.967 (+/-0.083) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.944 (+/-0.115) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.938 (+/-0.177) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.943 (+/-0.113) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.964 (+/-0.092) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.950 (+/-0.117) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.895 (+/-0.229) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.944 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.930 (+/-0.199) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.953 (+/-0.126) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.949 (+/-0.116) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.943 (+/-0.113) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.922 (+/-0.177) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.959 (+/-0.067) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.933 (+/-0.136) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.933 (+/-0.125) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.943 (+/-0.113) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.939 (+/-0.117) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.946 (+/-0.123) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.918 (+/-0.155) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.945 (+/-0.123) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.931 (+/-0.153) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.944 (+/-0.113) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.957 (+/-0.120) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.972 (+/-0.069) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.968 (+/-0.082) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.950 (+/-0.118) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.955 (+/-0.111) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      1.00      1.00         8\n",
      "          1       0.92      1.00      0.96        11\n",
      "          2       1.00      0.95      0.97        19\n",
      "\n",
      "avg / total       0.98      0.97      0.97        38\n",
      "\n",
      "\n",
      "# Tuning hyper-parameters for recall\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_search.py:667: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters set found on development set:\n",
      "\n",
      "{'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.893 (+/-0.215) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.955 (+/-0.092) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.929 (+/-0.155) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.938 (+/-0.138) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.929 (+/-0.155) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.920 (+/-0.241) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.946 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.866 (+/-0.268) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.884 (+/-0.218) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.911 (+/-0.179) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.973 (+/-0.081) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.929 (+/-0.155) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.893 (+/-0.177) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.938 (+/-0.137) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.920 (+/-0.162) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.920 (+/-0.187) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.929 (+/-0.104) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.911 (+/-0.191) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.938 (+/-0.141) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.946 (+/-0.114) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.929 (+/-0.155) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.911 (+/-0.137) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.902 (+/-0.148) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.929 (+/-0.158) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.946 (+/-0.113) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.938 (+/-0.137) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.938 (+/-0.137) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.920 (+/-0.147) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.893 (+/-0.255) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.938 (+/-0.117) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.159) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.938 (+/-0.141) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.955 (+/-0.115) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.893 (+/-0.139) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.920 (+/-0.168) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.137) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.911 (+/-0.179) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.137) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.946 (+/-0.146) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.946 (+/-0.121) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.911 (+/-0.179) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.955 (+/-0.115) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.938 (+/-0.141) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.955 (+/-0.121) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.920 (+/-0.119) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.929 (+/-0.109) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.938 (+/-0.137) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.938 (+/-0.183) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.946 (+/-0.120) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.929 (+/-0.168) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.938 (+/-0.183) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.955 (+/-0.147) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.955 (+/-0.121) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.929 (+/-0.158) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.920 (+/-0.168) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.866 (+/-0.202) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.938 (+/-0.137) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.929 (+/-0.155) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.938 (+/-0.141) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.920 (+/-0.154) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.920 (+/-0.151) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.964 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.955 (+/-0.115) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.920 (+/-0.140) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.946 (+/-0.120) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.929 (+/-0.131) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.920 (+/-0.181) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.920 (+/-0.204) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.929 (+/-0.154) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.938 (+/-0.137) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.955 (+/-0.146) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.955 (+/-0.121) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.929 (+/-0.136) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.964 (+/-0.121) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.946 (+/-0.086) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.929 (+/-0.175) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.946 (+/-0.114) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.938 (+/-0.173) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.938 (+/-0.110) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.929 (+/-0.175) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.920 (+/-0.168) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.929 (+/-0.131) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.964 (+/-0.119) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.938 (+/-0.110) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.938 (+/-0.110) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.946 (+/-0.140) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.938 (+/-0.159) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.946 (+/-0.120) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.920 (+/-0.147) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.920 (+/-0.127) for {'class_weight': None, 'criterion': 'gini', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.955 (+/-0.115) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.893 (+/-0.213) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.875 (+/-0.216) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.929 (+/-0.196) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.911 (+/-0.173) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 3, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.938 (+/-0.163) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.938 (+/-0.115) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.946 (+/-0.140) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 4, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.920 (+/-0.187) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.920 (+/-0.187) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.929 (+/-0.131) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.929 (+/-0.155) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 5, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.946 (+/-0.140) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.920 (+/-0.127) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.938 (+/-0.159) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.946 (+/-0.120) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 6, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.938 (+/-0.137) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.902 (+/-0.179) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.929 (+/-0.175) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.902 (+/-0.148) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 7, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.929 (+/-0.132) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.955 (+/-0.146) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.955 (+/-0.169) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.964 (+/-0.121) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.929 (+/-0.136) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 8, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'best'}\n",
      "0.920 (+/-0.147) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': None, 'splitter': 'random'}\n",
      "0.946 (+/-0.140) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'best'}\n",
      "0.938 (+/-0.137) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 5, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'best'}\n",
      "0.929 (+/-0.168) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 10, 'splitter': 'random'}\n",
      "0.938 (+/-0.138) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'best'}\n",
      "0.946 (+/-0.120) for {'class_weight': None, 'criterion': 'entropy', 'max_depth': 9, 'max_leaf_nodes': 20, 'splitter': 'random'}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      1.00      1.00         8\n",
      "          1       1.00      0.64      0.78        11\n",
      "          2       0.83      1.00      0.90        19\n",
      "\n",
      "avg / total       0.91      0.89      0.89        38\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_search.py:667: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "# Set the parameters by cross-validation\n",
    "\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "# set of parameters to test\n",
    "tuned_parameters = [{'max_depth': np.arange(3, 10),\n",
    "#                     'max_weights': [1, 10, 100, 1000]},\n",
    "                     'criterion': ['gini', 'entropy'], \n",
    "                     'splitter': ['best', 'random'],\n",
    "                    # 'min_samples_leaf': [2, 5, 10],\n",
    "                     'class_weight':['balanced', None],\n",
    "                     'max_leaf_nodes': [None, 5, 10, 20]\n",
    "                    }]\n",
    "\n",
    "scores = ['precision', 'recall']\n",
    "\n",
    "for score in scores:\n",
    "    print(\"# Tuning hyper-parameters for %s\" % score)\n",
    "    print()\n",
    "\n",
    "    # cv = the fold of the cross-validation cv, defaulted to 5\n",
    "    gs = GridSearchCV(DecisionTreeClassifier(), tuned_parameters, cv=10, scoring='%s_weighted' % score)\n",
    "    gs.fit(x_train, y_train)\n",
    "\n",
    "    print(\"Best parameters set found on development set:\")\n",
    "    print()\n",
    "    print(gs.best_params_)\n",
    "    print()\n",
    "    print(\"Grid scores on development set:\")\n",
    "    print()\n",
    "    for params, mean_score, scores in gs.grid_scores_:\n",
    "        print(\"%0.3f (+/-%0.03f) for %r\" % (mean_score, scores.std() * 2, params))\n",
    "    print()\n",
    "\n",
    "    print(\"Detailed classification report:\")\n",
    "    print()\n",
    "    print(\"The model is trained on the full development set.\")\n",
    "    print(\"The scores are computed on the full evaluation set.\")\n",
    "    print()\n",
    "    y_true, y_pred = y_test, gs.predict(x_test)\n",
    "    print(classification_report(y_true, y_pred))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Let's evaluate the resulting tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean score: 0.907 (+/- 0.015)\n"
     ]
    }
   ],
   "source": [
    "# create a composite estimator made by a pipeline of preprocessing and the KNN model\n",
    "model = Pipeline([\n",
    "        ('scaler', StandardScaler()),\n",
    "        ('ds', DecisionTreeClassifier(max_leaf_nodes=20, criterion='gini', \n",
    "                                      splitter='random', class_weight='balanced', max_depth=3))\n",
    "])\n",
    "\n",
    "# Fit the model\n",
    "model.fit(x_train, y_train) \n",
    "\n",
    "# create a k-fold cross validation iterator of k=10 folds\n",
    "cv = KFold(10, shuffle=True, random_state=33)\n",
    "\n",
    "# by default the score used is the one returned by score method of the estimator (accuracy)\n",
    "scores = cross_val_score(model, x_iris, y_iris, cv=cv)\n",
    "def mean_score(scores):\n",
    "    return (\"Mean score: {0:.3f} (+/- {1:.3f})\").format(np.mean(scores), sem(scores))\n",
    "print(mean_score(scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we get an average accuracy of 0.96!! Better than 0.947 (without tuning) and 0.953 (tuning only *max_depth*)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [Plot the decision surface of a decision tree on the iris dataset](http://scikit-learn.org/stable/auto_examples/tree/plot_iris.html)\n",
    "* [Learning scikit-learn: Machine Learning in Python](http://proquest.safaribooksonline.com/book/programming/python/9781783281930/1dot-machine-learning-a-gentle-introduction/ch01s02_html), Raúl Garreta; Guillermo Moncecchi, Packt Publishing, 2013.\n",
    "* [Python Machine Learning](http://proquest.safaribooksonline.com/book/programming/python/9781783555130), Sebastian Raschka, Packt Publishing, 2015.\n",
    "* [Parameter estimation using grid search with cross-validation](http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_digits.html)\n",
    "* [Decision trees in python with scikit-learn and pandas](http://chrisstrelioff.ws/sandbox/2015/06/08/decision_trees_in_python_with_scikit_learn_and_pandas.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Licence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The notebook is freely licensed under under the [Creative Commons Attribution Share-Alike license](https://creativecommons.org/licenses/by/2.0/).  \n",
    "\n",
    "© 2016 Carlos A. Iglesias, Universidad Politécnica de Madrid."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
