Notebook · Trees & Random Forests

Decision Trees¶

Decision trees are a type of supervised learning algorithm used for both classification and regression tasks. Their aim is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features.

Characteristics of Decision Trees:¶

• Structure: A decision tree consists of nodes, where each node represents a feature (or attribute), each link (or branch) represents a decision/rule, and each leaf represents an outcome.
• Ease of Interpretation: They are easy to understand and interpret, as they can be visually represented.
• Versatility: Can handle both numerical and categorical data.
• Non-parametric: They do not assume any distribution of the data, making them suitable for analysis where the data does not meet certain assumptions.

Uses of Decision Trees:¶

• Classification and Regression: Used for solving both classification and regression problems.
• Exploratory Data Analysis: Help in identifying significant patterns and relevant variables.
• Predictive Modeling: Useful in predictive modeling, especially when model interpretation is important.

Differences from Other Algorithms:¶

• Simplicity: Unlike more complex models like neural networks, decision trees are simpler and easier to interpret.
• Non-linearity: They can capture non-linear relationships between features and the target variable, unlike linear models like linear regression.
• Data Sensitivity: More sensitive to variations in the data and can easily overfit, unlike more robust models like random forests.

Example: Decision Tree Classification in Python¶

In this example, we will use a dataset from scikit-learn, convert it to a DataFrame, describe it, and then apply a decision tree model for classification. After obtaining the results, we will analyze the precision, recall, accuracy, and ROC curve.

In [ ]:

import pandas as pd
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import classification_report, roc_curve, auc
import matplotlib.pyplot as plt

# Load and prepare the dataset
iris = load_iris()
iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)
iris_df['target'] = iris.target

# Display basic information about the dataset
iris_df.info()
iris_df.describe()

# Split the dataset into training and test sets
X_train, X_test, y_train, y_test = train_test_split(iris_df[iris.feature_names], iris_df['target'], test_size=0.3, random_state=42)

# Initialize and train the decision tree classifier
clf = DecisionTreeClassifier(random_state=42)
clf.fit(X_train, y_train)

# Make predictions and evaluate the model
y_pred = clf.predict(X_test)
report = classification_report(y_test, y_pred)
print('Classification Report:\n', report)

# Calculate ROC curve and AUC
fpr, tpr, thresholds = roc_curve(y_test, clf.predict_proba(X_test)[:,1], pos_label=1)
roc_auc = auc(fpr, tpr)

# Plotting the ROC curve
plt.figure()
plt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic')
plt.legend(loc='lower right')
plt.show()

import pandas as pd from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import classification_report, roc_curve, auc import matplotlib.pyplot as plt # Load and prepare the dataset iris = load_iris() iris_df = pd.DataFrame(iris.data, columns=iris.feature_names) iris_df['target'] = iris.target # Display basic information about the dataset iris_df.info() iris_df.describe() # Split the dataset into training and test sets X_train, X_test, y_train, y_test = train_test_split(iris_df[iris.feature_names], iris_df['target'], test_size=0.3, random_state=42) # Initialize and train the decision tree classifier clf = DecisionTreeClassifier(random_state=42) clf.fit(X_train, y_train) # Make predictions and evaluate the model y_pred = clf.predict(X_test) report = classification_report(y_test, y_pred) print('Classification Report:\n', report) # Calculate ROC curve and AUC fpr, tpr, thresholds = roc_curve(y_test, clf.predict_proba(X_test)[:,1], pos_label=1) roc_auc = auc(fpr, tpr) # Plotting the ROC curve plt.figure() plt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % roc_auc) plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--') plt.xlim([0.0, 1.0]) plt.ylim([0.0, 1.05]) plt.xlabel('False Positive Rate') plt.ylabel('True Positive Rate') plt.title('Receiver Operating Characteristic') plt.legend(loc='lower right') plt.show()

Example: Decision Tree Classification with a Different Dataset¶

In this example, we will use another dataset from scikit-learn, convert it to a DataFrame, describe it, and then apply a decision tree model for classification. We will follow the same steps as the previous example, including analyzing precision, recall, accuracy, and the ROC curve.

In [ ]:

import pandas as pd
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import classification_report, roc_curve, auc
import matplotlib.pyplot as plt

# Load and prepare the dataset
wine = load_wine()
wine_df = pd.DataFrame(wine.data, columns=wine.feature_names)
wine_df['target'] = wine.target

# Display basic information about the dataset
wine_df.info()
wine_df.describe()

# Split the dataset into training and test sets
X_train, X_test, y_train, y_test = train_test_split(wine_df[wine.feature_names], wine_df['target'], test_size=0.3, random_state=42)

# Initialize and train the decision tree classifier
clf = DecisionTreeClassifier(random_state=42)
clf.fit(X_train, y_train)

# Make predictions and evaluate the model
y_pred = clf.predict(X_test)
report = classification_report(y_test, y_pred)
print('Classification Report:\n', report)

# Calculate ROC curve and AUC for multi-class
fpr = dict()
tpr = dict()
roc_auc = dict()
for i in range(wine.target_names.size):
    fpr[i], tpr[i], _ = roc_curve(y_test, clf.predict_proba(X_test)[:, i], pos_label=i)
    roc_auc[i] = auc(fpr[i], tpr[i])

# Plotting the ROC curve for each class
plt.figure()
for i in range(wine.target_names.size):
    plt.plot(fpr[i], tpr[i], lw=2, label='ROC curve of class %d (area = %0.2f)' % (i, roc_auc[i]))
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic for each class')
plt.legend(loc='lower right')
plt.show()

import pandas as pd from sklearn.datasets import load_wine from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import classification_report, roc_curve, auc import matplotlib.pyplot as plt # Load and prepare the dataset wine = load_wine() wine_df = pd.DataFrame(wine.data, columns=wine.feature_names) wine_df['target'] = wine.target # Display basic information about the dataset wine_df.info() wine_df.describe() # Split the dataset into training and test sets X_train, X_test, y_train, y_test = train_test_split(wine_df[wine.feature_names], wine_df['target'], test_size=0.3, random_state=42) # Initialize and train the decision tree classifier clf = DecisionTreeClassifier(random_state=42) clf.fit(X_train, y_train) # Make predictions and evaluate the model y_pred = clf.predict(X_test) report = classification_report(y_test, y_pred) print('Classification Report:\n', report) # Calculate ROC curve and AUC for multi-class fpr = dict() tpr = dict() roc_auc = dict() for i in range(wine.target_names.size): fpr[i], tpr[i], _ = roc_curve(y_test, clf.predict_proba(X_test)[:, i], pos_label=i) roc_auc[i] = auc(fpr[i], tpr[i]) # Plotting the ROC curve for each class plt.figure() for i in range(wine.target_names.size): plt.plot(fpr[i], tpr[i], lw=2, label='ROC curve of class %d (area = %0.2f)' % (i, roc_auc[i])) plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--') plt.xlim([0.0, 1.0]) plt.ylim([0.0, 1.05]) plt.xlabel('False Positive Rate') plt.ylabel('True Positive Rate') plt.title('Receiver Operating Characteristic for each class') plt.legend(loc='lower right') plt.show()

Random Forests¶

Random Forests are an ensemble learning method, primarily used for classification and regression. They operate by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees.

Characteristics of Random Forests:¶

• Ensemble Method: Random Forests combine multiple decision trees to improve the predictive performance and control over-fitting.
• Robustness: They are less prone to overfitting than individual decision trees.
• Handling of Unbalanced Data: Effective in handling unbalanced datasets by balancing error in the minority class.
• Feature Importance: They provide insights into the importance of each feature in making predictions.

Differences from Decision Trees:¶

• Complexity: Random Forests are generally more complex than single decision trees.
• Performance: They often provide better accuracy due to the averaging of multiple trees.
• Interpretability: While individual trees are easy to interpret, the ensemble nature of Random Forests makes them more complex to interpret.

Metrics Used in Random Forests:¶

• Accuracy: Measures the proportion of correct predictions.
• Precision and Recall: Useful for evaluating class imbalance.
• F1 Score: Harmonic mean of precision and recall.
• AUC-ROC Curve: Measures the performance across all possible classification thresholds.

Random Forest Example with Iris Dataset¶

Following the same steps as in the decision tree examples, we will now apply a Random Forest model to the Iris dataset. We will load the dataset, convert it to a DataFrame, describe it, apply the Random Forest model for classification, and analyze the precision, recall, accuracy, and ROC curve.

In [ ]:

from sklearn.ensemble import RandomForestClassifier

# Initialize and train the Random Forest classifier
rf_clf = RandomForestClassifier(random_state=42)
rf_clf.fit(X_train, y_train)

# Make predictions and evaluate the model
y_pred_rf = rf_clf.predict(X_test)
rf_report = classification_report(y_test, y_pred_rf)
print('Random Forest Classification Report:\n', rf_report)

# Calculate ROC curve and AUC
rf_fpr, rf_tpr, rf_thresholds = roc_curve(y_test, rf_clf.predict_proba(X_test)[:,1], pos_label=1)
rf_roc_auc = auc(rf_fpr, rf_tpr)

# Plotting the ROC curve
plt.figure()
plt.plot(rf_fpr, rf_tpr, color='darkorange', lw=2, label='Random Forest ROC curve (area = %0.2f)' % rf_roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Random Forest Receiver Operating Characteristic')
plt.legend(loc='lower right')
plt.show()

from sklearn.ensemble import RandomForestClassifier # Initialize and train the Random Forest classifier rf_clf = RandomForestClassifier(random_state=42) rf_clf.fit(X_train, y_train) # Make predictions and evaluate the model y_pred_rf = rf_clf.predict(X_test) rf_report = classification_report(y_test, y_pred_rf) print('Random Forest Classification Report:\n', rf_report) # Calculate ROC curve and AUC rf_fpr, rf_tpr, rf_thresholds = roc_curve(y_test, rf_clf.predict_proba(X_test)[:,1], pos_label=1) rf_roc_auc = auc(rf_fpr, rf_tpr) # Plotting the ROC curve plt.figure() plt.plot(rf_fpr, rf_tpr, color='darkorange', lw=2, label='Random Forest ROC curve (area = %0.2f)' % rf_roc_auc) plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--') plt.xlim([0.0, 1.0]) plt.ylim([0.0, 1.05]) plt.xlabel('False Positive Rate') plt.ylabel('True Positive Rate') plt.title('Random Forest Receiver Operating Characteristic') plt.legend(loc='lower right') plt.show()

Random Forest Example with Wine Dataset¶

Now, we will apply the Random Forest model to the Wine dataset, following the same procedure as before. We will load the dataset, convert it to a DataFrame, describe it, apply the Random Forest model for classification, and analyze the precision, recall, accuracy, and ROC curve for each class.

In [ ]:

# Initialize and train the Random Forest classifier for the Wine dataset
rf_clf_wine = RandomForestClassifier(random_state=42)
rf_clf_wine.fit(X_train_wine, y_train_wine)

# Make predictions and evaluate the model
y_pred_rf_wine = rf_clf_wine.predict(X_test_wine)
rf_report_wine = classification_report(y_test_wine, y_pred_rf_wine)
print('Random Forest Classification Report for Wine Dataset:\n', rf_report_wine)

# Calculate ROC curve and AUC for multi-class
rf_fpr_wine = dict()
rf_tpr_wine = dict()
rf_roc_auc_wine = dict()
for i in range(wine.target_names.size):
    rf_fpr_wine[i], rf_tpr_wine[i], _ = roc_curve(y_test_wine, rf_clf_wine.predict_proba(X_test_wine)[:, i], pos_label=i)
    rf_roc_auc_wine[i] = auc(rf_fpr_wine[i], rf_tpr_wine[i])

# Plotting the ROC curve for each class
plt.figure()
for i in range(wine.target_names.size):
    plt.plot(rf_fpr_wine[i], rf_tpr_wine[i], lw=2, label='Random Forest ROC curve of class %d (area = %0.2f)' % (i, rf_roc_auc_wine[i]))
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Random Forest Receiver Operating Characteristic for Wine Dataset')
plt.legend(loc='lower right')
plt.show()

# Initialize and train the Random Forest classifier for the Wine dataset rf_clf_wine = RandomForestClassifier(random_state=42) rf_clf_wine.fit(X_train_wine, y_train_wine) # Make predictions and evaluate the model y_pred_rf_wine = rf_clf_wine.predict(X_test_wine) rf_report_wine = classification_report(y_test_wine, y_pred_rf_wine) print('Random Forest Classification Report for Wine Dataset:\n', rf_report_wine) # Calculate ROC curve and AUC for multi-class rf_fpr_wine = dict() rf_tpr_wine = dict() rf_roc_auc_wine = dict() for i in range(wine.target_names.size): rf_fpr_wine[i], rf_tpr_wine[i], _ = roc_curve(y_test_wine, rf_clf_wine.predict_proba(X_test_wine)[:, i], pos_label=i) rf_roc_auc_wine[i] = auc(rf_fpr_wine[i], rf_tpr_wine[i]) # Plotting the ROC curve for each class plt.figure() for i in range(wine.target_names.size): plt.plot(rf_fpr_wine[i], rf_tpr_wine[i], lw=2, label='Random Forest ROC curve of class %d (area = %0.2f)' % (i, rf_roc_auc_wine[i])) plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--') plt.xlim([0.0, 1.0]) plt.ylim([0.0, 1.05]) plt.xlabel('False Positive Rate') plt.ylabel('True Positive Rate') plt.title('Random Forest Receiver Operating Characteristic for Wine Dataset') plt.legend(loc='lower right') plt.show()