balkian/sitc
1
0
Fork 0
mirror of https://github.com/gsi-upm/sitc synced 2024-11-05 07:31:41 +00:00
Code Issues Releases Wiki Activity
adaf4b1223
sitc/ml1/2_6_Model_Tuning.ipynb

1291 lines
87 KiB
Plaintext
Raw Normal View History

Add ml1
2016-03-15 12:55:14 +00:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](files/images/EscUpmPolit_p.gif \"UPM\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Course Notes for Learning Intelligent Systems"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Department of Telematic Engineering Systems, Universidad Politécnica de Madrid, © 2016 Carlos A. Iglesias"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## [Introduction to Machine Learning](2_0_0_Intro_ML.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Table of Contents\n",
"\n",
"* [Model Tuning](#Model-Tuning)\n",
"* [Load data and preprocessing](#Load-data-and-preprocessing)\n",
"* [Train classifier](#Train-classifier)\n",
"* [More about Pipelines](#More-about-Pipelines)\n",
"* [Tuning the algorithm](#Tuning-the-algorithm)\n",
"\t* [Grid Search for Parameter optimization](#Grid-Search-for-Parameter-optimization)\n",
"* [Evaluating the algorithm](#Evaluating-the-algorithm)\n",
"\t* [K-Fold validation](#K-Fold-validation)\n",
"* [References](#References)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Model Tuning"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the previous [notebook](2_5_2_Decision_Tree_Model.ipynb), we got an accuracy of 9.47. Could we get a better accuracy if we tune the parameters of the estimator?\n",
"\n",
"The goal of this notebook is to learn how to learn how to tune an algorithm by opimizing their parameters using grid search."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data and preprocessing"
]
},
{
"cell_type": "code",
"execution_count": 111,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# library for displaying plots\n",
"import matplotlib.pyplot as plt\n",
"# display plots in the notebook \n",
"%matplotlib inline\n",
"\n",
"## First, we repeat the load and preprocessing steps\n",
"\n",
"# Load data\n",
"from sklearn import datasets\n",
"iris = datasets.load_iris()\n",
"\n",
"# Training and test spliting\n",
"from sklearn.cross_validation import train_test_split\n",
"\n",
"x_iris, y_iris = iris.data, iris.target\n",
"# Test set will be the 25% taken randomly\n",
"x_train, x_test, y_train, y_test = train_test_split(x_iris, y_iris, test_size=0.25, random_state=33)\n",
"\n",
"# Preprocess: normalize\n",
"from sklearn import preprocessing\n",
"scaler = preprocessing.StandardScaler().fit(x_train)\n",
"x_train = scaler.transform(x_train)\n",
"x_test = scaler.transform(x_test)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Train classifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As previously, we train the model and evaluate the result."
]
},
{
"cell_type": "code",
"execution_count": 128,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([ 1. , 0.8 , 1. , 0.93333333, 0.93333333,\n",
" 0.93333333, 1. , 1. , 0.86666667, 0.93333333])"
]
},
"execution_count": 128,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.cross_validation import cross_val_score, KFold\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import StandardScaler\n",
"\n",
"# create a composite estimator made by a pipeline of preprocessing and the KNN model\n",
"model = Pipeline([\n",
" ('scaler', StandardScaler()),\n",
" ('ds', DecisionTreeClassifier())\n",
"])\n",
"\n",
"# Fit the model\n",
"model.fit(x_train, y_train) \n",
"\n",
"# create a k-fold cross validation iterator of k=10 folds\n",
"cv = KFold(x_iris.shape[0], 10, shuffle=True, random_state=33)\n",
"\n",
"# by default the score used is the one returned by score method of the estimator (accuracy)\n",
"scores = cross_val_score(model, x_iris, y_iris, cv=cv)\n",
"\n",
"from scipy.stats import sem\n",
"def mean_score(scores):\n",
" return (\"Mean score: {0:.3f} (+/- {1:.3f})\").format(np.mean(scores), sem(scores))\n",
"print(mean_score(scores))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We obtain an accuracy of 0.947."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## More about Pipelines"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When we use a Pipeline, every chained estimator is stored in the dictionary *named_steps* and as a list in *steps*."
]
},
{
"cell_type": "code",
"execution_count": 115,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"{'ds': DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
" max_features=None, max_leaf_nodes=None, min_samples_leaf=1,\n",
" min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
" presort=False, random_state=None, splitter='best'),\n",
" 'scaler': StandardScaler(copy=True, with_mean=True, with_std=True)}"
]
},
"execution_count": 115,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.named_steps"
]
},
{
"cell_type": "code",
"execution_count": 116,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)),\n",
" ('ds',\n",
" DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
" max_features=None, max_leaf_nodes=None, min_samples_leaf=1,\n",
" min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
" presort=False, random_state=None, splitter='best'))]"
]
},
"execution_count": 116,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.steps"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can get the list of parameters of the model. As you will observe, the parameters of the estimators in the pipeline can be accessed using the <estimator>__<parameter> syntax. We will use this for tuning the parameters."
]
},
{
"cell_type": "code",
"execution_count": 117,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"dict_keys(['ds__min_samples_split', 'ds__max_features', 'ds__max_leaf_nodes', 'scaler__copy', 'scaler__with_mean', 'ds__class_weight', 'ds__splitter', 'ds__min_weight_fraction_leaf', 'ds', 'ds__max_depth', 'ds__min_samples_leaf', 'scaler', 'scaler__with_std', 'ds__random_state', 'ds__presort', 'ds__criterion', 'steps'])"
]
},
"execution_count": 117,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.get_params().keys()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we are going to change a parameter"
]
},
{
"cell_type": "code",
"execution_count": 118,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Pipeline(steps=[('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)), ('ds', DecisionTreeClassifier(class_weight='balanced', criterion='gini',\n",
" max_depth=None, max_features=None, max_leaf_nodes=None,\n",
" min_samples_leaf=1, min_samples_split=2,\n",
" min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
" splitter='best'))])"
]
},
"execution_count": 118,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.set_params(ds__class_weight='balanced')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Another alternative is to create the pipeline with the values we want to set, but it can be useful to access the estimators of the Pipeline."
]
},
{
"cell_type": "code",
"execution_count": 119,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Pipeline(steps=[('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)), ('ds', DecisionTreeClassifier(class_weight='balanced', criterion='gini',\n",
" max_depth=None, max_features=None, max_leaf_nodes=None,\n",
" min_samples_leaf=1, min_samples_split=2,\n",
" min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
" splitter='best'))])"
]
},
"execution_count": 119,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = Pipeline([\n",
" ('scaler', StandardScaler()),\n",
" ('ds', DecisionTreeClassifier(class_weight='balanced'))\n",
"])\n",
"model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The same approach can be used for accessing attributes such as *feature_importances_* we saw in the previous notebook."
]
},
{
"cell_type": "code",
"execution_count": 121,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ 0.01428571 0.03745715 0.55176597 0.39649117]\n"
]
}
],
"source": [
"# Fit the model\n",
"model.fit(x_train, y_train) \n",
"# Using named_steps\n",
"my_decision_tree = model.named_steps['ds']\n",
"print(my_decision_tree.feature_importances_)"
]
},
{
"cell_type": "code",
"execution_count": 122,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ 0.01428571 0.03745715 0.55176597 0.39649117]\n"
]
}
],
"source": [
"#Using steps, we take the last step (-1) or the second step (1)\n",
"#name, my_desision_tree = model.steps[1]\n",
"name, my_desision_tree = model.steps[-1]\n",
"print(my_decision_tree.feature_importances_)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Tuning the algorithm"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see that the most important feature for this classifier is `petal width`.\n",
"\n",
"Look at the [API](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) of *scikit-learn* to understand better the algorithm, as well as which parameters can be tuned. As you see, we can change several ones, such as *criterion*, *splitter*, *max_features*, *max_depth*, *min_samples_split*, *class_weight*, etc.\n",
"\n",
"We can get the full list parameters of an estimator with the method *get_params()*. "
]
},
{
"cell_type": "code",
"execution_count": 123,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"{'ds': DecisionTreeClassifier(class_weight='balanced', criterion='gini',\n",
" max_depth=None, max_features=None, max_leaf_nodes=None,\n",
" min_samples_leaf=1, min_samples_split=2,\n",
" min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
" splitter='best'),\n",
" 'ds__class_weight': 'balanced',\n",
" 'ds__criterion': 'gini',\n",
" 'ds__max_depth': None,\n",
" 'ds__max_features': None,\n",
" 'ds__max_leaf_nodes': None,\n",
" 'ds__min_samples_leaf': 1,\n",
" 'ds__min_samples_split': 2,\n",
" 'ds__min_weight_fraction_leaf': 0.0,\n",
" 'ds__presort': False,\n",
" 'ds__random_state': None,\n",
" 'ds__splitter': 'best',\n",
" 'scaler': StandardScaler(copy=True, with_mean=True, with_std=True),\n",
" 'scaler__copy': True,\n",
" 'scaler__with_mean': True,\n",
" 'scaler__with_std': True,\n",
" 'steps': [('scaler',\n",
" StandardScaler(copy=True, with_mean=True, with_std=True)),\n",
" ('ds', DecisionTreeClassifier(class_weight='balanced', criterion='gini',\n",
" max_depth=None, max_features=None, max_leaf_nodes=None,\n",
" min_samples_leaf=1, min_samples_split=2,\n",
" min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
" splitter='best'))]}"
]
},
"execution_count": 123,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.get_params()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can try different values for these parameters and observe the results."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Grid Search for Parameter optimization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Changing manually the parameters to find their optimal values is not practical. Instead, we can consider to find the optimal value of the parameters as an *optimization problem*. \n",
"\n",
"The sklearn comes with several optimization techniques for this purpose, such as [**grid search**](https://en.wikipedia.org/wiki/Hyperparameter_optimization#Grid_search) and **randomized search**. In this notebook we are going to introduce the former one."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The sklearn provides an object that, given data, computes the score during the fit of an estimator on a parameter grid and chooses the parameters to maximize the cross-validation score. "
]
},
{
"cell_type": "code",
"execution_count": 124,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best score: 0.946428571429\n",
"Best params: {'max_depth': 3}\n"
]
}
],
"source": [
"from sklearn.grid_search import GridSearchCV\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"import numpy as np\n",
"\n",
"param_grid = {'max_depth': np.arange(3, 10)} \n",
"\n",
"gs = GridSearchCV(DecisionTreeClassifier(), param_grid)\n",
"\n",
"gs.fit(x_train, y_train)\n",
"\n",
"# summarize the results of the grid search\n",
"print(\"Best score: \", gs.best_score_)\n",
"print(\"Best params: \", gs.best_params_)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"Now we are going to show the results of grid search"
]
},
{
"cell_type": "code",
"execution_count": 133,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.946 (+/-0.075) for {'max_depth': 3}\n",
"0.929 (+/-0.025) for {'max_depth': 4}\n",
"0.929 (+/-0.025) for {'max_depth': 5}\n",
"0.938 (+/-0.050) for {'max_depth': 6}\n",
"0.929 (+/-0.025) for {'max_depth': 7}\n",
"0.946 (+/-0.075) for {'max_depth': 8}\n",
"0.946 (+/-0.075) for {'max_depth': 9}\n"
]
}
],
"source": [
"# We print the score for each value of max_depth\n",
"for params, mean_score, scores in gs.grid_scores_:\n",
" print(\"%0.3f (+/-%0.03f) for %r\" % (mean_score, scores.std() * 2, params))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now evaluate the KFold with this optimized parameter as follows."
]
},
{
"cell_type": "code",
"execution_count": 132,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Mean score: 0.953 (+/- 0.020)\n"
]
}
],
"source": [
"# create a composite estimator made by a pipeline of preprocessing and the KNN model\n",
"model = Pipeline([\n",
" ('scaler', StandardScaler()),\n",
" ('ds', DecisionTreeClassifier(max_depth=3))\n",
"])\n",
"\n",
"# Fit the model\n",
"model.fit(x_train, y_train) \n",
"\n",
"# create a k-fold cross validation iterator of k=10 folds\n",
"cv = KFold(x_iris.shape[0], 10, shuffle=True, random_state=33)\n",
"\n",
"# by default the score used is the one returned by score method of the estimator (accuracy)\n",
"scores = cross_val_score(model, x_iris, y_iris, cv=cv)\n",
"def mean_score(scores):\n",
" return (\"Mean score: {0:.3f} (+/- {1:.3f})\").format(np.mean(scores), sem(scores))\n",
"print(mean_score(scores))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have got an *improvement* from 0.947 to 0.953 with k-fold.\n",
"\n",
"We are now to try to fit the best combination of the parameters of the algorithm. It can take some time to compute it."
]
},
{
"cell_type": "code",
"execution_count": 134,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# Tuning hyper-parameters for precision\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/cif/anaconda3/lib/python3.5/site-packages/sklearn/metrics/classification.py:1074: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
" 'precision', 'predicted', average, warn_for)\n",
"/home/cif/anaconda3/lib/python3.5/site-packages/sklearn/metrics/classification.py:1074: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
" 'precision', 'predicted', average, warn_for)\n",
"/home/cif/anaconda3/lib/python3.5/site-packages/sklearn/metrics/classification.py:1074: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
" 'precision', 'predicted', average, warn_for)\n",
"/home/cif/anaconda3/lib/python3.5/site-packages/sklearn/metrics/classification.py:1074: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
" 'precision', 'predicted', average, warn_for)\n",
"/home/cif/anaconda3/lib/python3.5/site-packages/sklearn/metrics/classification.py:1074: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
" 'precision', 'predicted', average, warn_for)\n",
"/home/cif/anaconda3/lib/python3.5/site-packages/sklearn/metrics/classification.py:1074: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
" 'precision', 'predicted', average, warn_for)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best parameters set found on development set:\n",
"\n",
"{'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"\n",
"Grid scores on development set:\n",
"\n",
"0.964 (+/-0.092) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.955 (+/-0.078) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.946 (+/-0.123) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.945 (+/-0.126) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.962 (+/-0.080) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.976 (+/-0.078) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.974 (+/-0.080) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.953 (+/-0.126) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.957 (+/-0.090) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.968 (+/-0.082) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.972 (+/-0.068) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.945 (+/-0.162) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.953 (+/-0.126) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.964 (+/-0.092) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.953 (+/-0.126) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.934 (+/-0.123) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.940 (+/-0.146) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.936 (+/-0.165) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.946 (+/-0.123) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.952 (+/-0.105) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.953 (+/-0.126) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.956 (+/-0.142) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.947 (+/-0.109) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.946 (+/-0.123) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.941 (+/-0.167) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.953 (+/-0.126) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.949 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.941 (+/-0.135) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.953 (+/-0.126) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.956 (+/-0.098) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.972 (+/-0.068) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.947 (+/-0.073) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.953 (+/-0.126) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.940 (+/-0.131) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.953 (+/-0.126) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.924 (+/-0.148) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.953 (+/-0.126) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.968 (+/-0.082) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.878 (+/-0.325) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.959 (+/-0.068) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.950 (+/-0.087) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.962 (+/-0.113) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.966 (+/-0.070) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.968 (+/-0.082) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.953 (+/-0.134) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.957 (+/-0.102) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.946 (+/-0.130) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.935 (+/-0.102) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.964 (+/-0.092) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.959 (+/-0.066) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.907 (+/-0.168) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.910 (+/-0.090) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.933 (+/-0.136) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.967 (+/-0.087) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.912 (+/-0.160) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.913 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.944 (+/-0.138) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.944 (+/-0.146) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.944 (+/-0.138) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.936 (+/-0.121) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.975 (+/-0.078) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.946 (+/-0.123) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.942 (+/-0.168) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.968 (+/-0.082) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.950 (+/-0.086) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.949 (+/-0.116) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.972 (+/-0.069) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.938 (+/-0.089) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.962 (+/-0.084) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.924 (+/-0.151) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.960 (+/-0.082) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.959 (+/-0.110) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.970 (+/-0.099) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.939 (+/-0.118) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.925 (+/-0.128) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.973 (+/-0.068) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.941 (+/-0.155) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.945 (+/-0.177) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.922 (+/-0.157) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.904 (+/-0.091) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.915 (+/-0.150) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.949 (+/-0.118) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.943 (+/-0.081) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.962 (+/-0.115) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.946 (+/-0.146) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.913 (+/-0.163) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.953 (+/-0.080) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.915 (+/-0.153) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.936 (+/-0.170) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.934 (+/-0.160) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.968 (+/-0.082) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.879 (+/-0.258) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.902 (+/-0.257) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.946 (+/-0.130) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.960 (+/-0.083) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.943 (+/-0.138) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.950 (+/-0.089) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.936 (+/-0.133) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.945 (+/-0.123) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.953 (+/-0.134) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.900 (+/-0.248) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.949 (+/-0.091) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.972 (+/-0.068) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.909 (+/-0.296) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.952 (+/-0.105) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.930 (+/-0.133) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.970 (+/-0.099) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.965 (+/-0.115) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.926 (+/-0.138) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.950 (+/-0.087) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.956 (+/-0.094) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.934 (+/-0.141) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.943 (+/-0.113) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.959 (+/-0.069) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.957 (+/-0.120) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.961 (+/-0.081) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.965 (+/-0.070) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.950 (+/-0.118) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.938 (+/-0.160) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"\n",
"Detailed classification report:\n",
"\n",
"The model is trained on the full development set.\n",
"The scores are computed on the full evaluation set.\n",
"\n",
" precision recall f1-score support\n",
"\n",
" 0 1.00 1.00 1.00 8\n",
" 1 0.85 1.00 0.92 11\n",
" 2 1.00 0.89 0.94 19\n",
"\n",
"avg / total 0.96 0.95 0.95 38\n",
"\n",
"\n",
"# Tuning hyper-parameters for recall\n",
"\n",
"Best parameters set found on development set:\n",
"\n",
"{'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"\n",
"Grid scores on development set:\n",
"\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.920 (+/-0.243) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.902 (+/-0.164) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.955 (+/-0.146) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.955 (+/-0.117) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.938 (+/-0.137) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.929 (+/-0.155) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.911 (+/-0.146) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.973 (+/-0.108) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.955 (+/-0.121) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.964 (+/-0.119) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.929 (+/-0.155) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.911 (+/-0.182) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.938 (+/-0.137) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.920 (+/-0.151) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.929 (+/-0.155) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.938 (+/-0.152) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.946 (+/-0.114) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.955 (+/-0.115) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.938 (+/-0.141) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.929 (+/-0.155) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.902 (+/-0.163) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.911 (+/-0.141) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.920 (+/-0.162) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.920 (+/-0.148) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.929 (+/-0.155) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.911 (+/-0.121) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.946 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.955 (+/-0.121) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.938 (+/-0.141) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.938 (+/-0.142) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.929 (+/-0.155) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.902 (+/-0.169) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.955 (+/-0.115) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.857 (+/-0.147) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.902 (+/-0.169) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.938 (+/-0.137) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.938 (+/-0.112) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.938 (+/-0.137) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.964 (+/-0.113) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 3}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.911 (+/-0.137) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.955 (+/-0.115) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.893 (+/-0.149) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 4}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.902 (+/-0.124) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.920 (+/-0.190) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.946 (+/-0.120) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.929 (+/-0.175) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 5}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.929 (+/-0.155) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.920 (+/-0.198) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.955 (+/-0.115) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.920 (+/-0.147) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 6}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.964 (+/-0.121) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.929 (+/-0.136) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.964 (+/-0.089) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.920 (+/-0.207) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 7}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.938 (+/-0.141) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.920 (+/-0.147) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.911 (+/-0.179) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.938 (+/-0.137) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 8}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.929 (+/-0.110) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.902 (+/-0.156) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.911 (+/-0.160) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': 'balanced', 'max_depth': 9}\n",
"0.955 (+/-0.115) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.920 (+/-0.167) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.938 (+/-0.142) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.955 (+/-0.122) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.955 (+/-0.141) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.946 (+/-0.114) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.946 (+/-0.119) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.929 (+/-0.168) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.955 (+/-0.121) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.884 (+/-0.232) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.920 (+/-0.168) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.911 (+/-0.159) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.964 (+/-0.119) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.929 (+/-0.212) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.920 (+/-0.187) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.929 (+/-0.131) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.938 (+/-0.109) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.946 (+/-0.120) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.929 (+/-0.155) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.938 (+/-0.164) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.920 (+/-0.171) for {'max_leaf_nodes': None, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.893 (+/-0.227) for {'max_leaf_nodes': 5, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.902 (+/-0.169) for {'max_leaf_nodes': 10, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.902 (+/-0.148) for {'max_leaf_nodes': 20, 'criterion': 'gini', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.955 (+/-0.115) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.875 (+/-0.228) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.920 (+/-0.147) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.938 (+/-0.137) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 3}\n",
"0.929 (+/-0.116) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 3}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.911 (+/-0.160) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.866 (+/-0.234) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.911 (+/-0.183) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 4}\n",
"0.938 (+/-0.109) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 4}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.946 (+/-0.088) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.929 (+/-0.174) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.938 (+/-0.152) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 5}\n",
"0.938 (+/-0.116) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 5}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.946 (+/-0.120) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.938 (+/-0.164) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 6}\n",
"0.929 (+/-0.187) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 6}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.920 (+/-0.147) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.920 (+/-0.153) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.911 (+/-0.137) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 7}\n",
"0.929 (+/-0.158) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 7}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.920 (+/-0.168) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.920 (+/-0.162) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 8}\n",
"0.938 (+/-0.159) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 8}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.929 (+/-0.158) for {'max_leaf_nodes': None, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.893 (+/-0.194) for {'max_leaf_nodes': 5, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.938 (+/-0.138) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.946 (+/-0.140) for {'max_leaf_nodes': 10, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"0.929 (+/-0.132) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'best', 'class_weight': None, 'max_depth': 9}\n",
"0.929 (+/-0.155) for {'max_leaf_nodes': 20, 'criterion': 'entropy', 'splitter': 'random', 'class_weight': None, 'max_depth': 9}\n",
"\n",
"Detailed classification report:\n",
"\n",
"The model is trained on the full development set.\n",
"The scores are computed on the full evaluation set.\n",
"\n",
" precision recall f1-score support\n",
"\n",
" 0 1.00 1.00 1.00 8\n",
" 1 0.91 0.91 0.91 11\n",
" 2 0.95 0.95 0.95 19\n",
"\n",
"avg / total 0.95 0.95 0.95 38\n",
"\n",
"\n"
]
}
],
"source": [
"# Set the parameters by cross-validation\n",
"\n",
"from sklearn.metrics import classification_report\n",
"\n",
"# set of parameters to test\n",
"tuned_parameters = [{'max_depth': np.arange(3, 10),\n",
"# 'max_weights': [1, 10, 100, 1000]},\n",
" 'criterion': ['gini', 'entropy'], \n",
" 'splitter': ['best', 'random'],\n",
" # 'min_samples_leaf': [2, 5, 10],\n",
" 'class_weight':['balanced', None],\n",
" 'max_leaf_nodes': [None, 5, 10, 20]\n",
" }]\n",
"\n",
"scores = ['precision', 'recall']\n",
"\n",
"for score in scores:\n",
" print(\"# Tuning hyper-parameters for %s\" % score)\n",
" print()\n",
"\n",
" # cv = the fold of the cross-validation cv, defaulted to 5\n",
" gs = GridSearchCV(DecisionTreeClassifier(), tuned_parameters, cv=10, scoring='%s_weighted' % score)\n",
" gs.fit(x_train, y_train)\n",
"\n",
" print(\"Best parameters set found on development set:\")\n",
" print()\n",
" print(gs.best_params_)\n",
" print()\n",
" print(\"Grid scores on development set:\")\n",
" print()\n",
" for params, mean_score, scores in gs.grid_scores_:\n",
" print(\"%0.3f (+/-%0.03f) for %r\" % (mean_score, scores.std() * 2, params))\n",
" print()\n",
"\n",
" print(\"Detailed classification report:\")\n",
" print()\n",
" print(\"The model is trained on the full development set.\")\n",
" print(\"The scores are computed on the full evaluation set.\")\n",
" print()\n",
" y_true, y_pred = y_test, gs.predict(x_test)\n",
" print(classification_report(y_true, y_pred))\n",
" print()"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"New we evaluate the resulting tuning."
]
},
{
"cell_type": "code",
"execution_count": 135,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Mean score: 0.960 (+/- 0.018)\n"
]
}
],
"source": [
"# create a composite estimator made by a pipeline of preprocessing and the KNN model\n",
"model = Pipeline([\n",
" ('scaler', StandardScaler()),\n",
" ('ds', DecisionTreeClassifier(max_leaf_nodes=20, criterion='gini', \n",
" splitter='random', class_weight='balanced', max_depth=3))\n",
"])\n",
"\n",
"# Fit the model\n",
"model.fit(x_train, y_train) \n",
"\n",
"# create a k-fold cross validation iterator of k=10 folds\n",
"cv = KFold(x_iris.shape[0], 10, shuffle=True, random_state=33)\n",
"\n",
"# by default the score used is the one returned by score method of the estimator (accuracy)\n",
"scores = cross_val_score(model, x_iris, y_iris, cv=cv)\n",
"def mean_score(scores):\n",
" return (\"Mean score: {0:.3f} (+/- {1:.3f})\").format(np.mean(scores), sem(scores))\n",
"print(mean_score(scores))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So, we get an average accuracy of 0.96!! Better than 0.947 (without tuning) and 0.953 (tuning only *max_depth*)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## References"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* [Plot the decision surface of a decision tree on the iris dataset](http://scikit-learn.org/stable/auto_examples/tree/plot_iris.html)\n",
"* [Learning scikit-learn: Machine Learning in Python](http://proquest.safaribooksonline.com/book/programming/python/9781783281930/1dot-machine-learning-a-gentle-introduction/ch01s02_html), Raúl Garreta; Guillermo Moncecchi, Packt Publishing, 2013.\n",
"* [Python Machine Learning](http://proquest.safaribooksonline.com/book/programming/python/9781783555130), Sebastian Raschka, Packt Publishing, 2015.\n",
"* [Parameter estimation using grid search with cross-validation](http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_digits.html)\n",
"* [Decision trees in python with scikit-learn and pandas](http://chrisstrelioff.ws/sandbox/2015/06/08/decision_trees_in_python_with_scikit_learn_and_pandas.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Licence"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The notebook is freely licensed under under the [Creative Commons Attribution Share-Alike license](https://creativecommons.org/licenses/by/2.0/). \n",
"\n",
"© 2016 Carlos A. Iglesias, Universidad Politécnica de Madrid."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.1"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Reference in New Issue Copy Permalink