11) Recalling a Gradient Boosting Machine.

This Blog entry is from the Probability and Trees section in Learn R.

Recalling the GBM is quite initiative and obeys the standardised predict signature.  To recall the GBM:

GBMPredictions <- predict(GBM,CreditRisk,type = "response")
1.png

Run the line of script to console:

2.png

A distinct peculiarity, given that the CreditRisk data frame has a dependent variable which is a factor, is that the binary classification has been modelled between 1 and 2, being the levels of the factor with 1 being Bad, and Good being two:

3.png

 It follows that predictions that are closer to 2, than 1 would be considered to be Good, whereas vice versa, 1.  To appraise the model performance, a confusion matrix should be created.  Create a vector using the ifelse() function to classify between Good and Bad:

CreditRiskGBMClassifications <- ifelse(GBMPredictions >= 1.5,"Good","Bad")
4.png

Run the line of script to console:

5.png

Create a confusion matrix between the actual value and the value predicted by the GBM:

CrossTable(CreditRisk$Dependent, CreditRiskGBMClassifications)
6.png

Run the line of script to console:

7.png

It can be seen in this example that the GBM has mustered a strong performance.  Of 220 accounts that were bad, it can be seen that the GBM classified 182 of them correctly, which gives an overall accuracy rating of 82%. This is a more realistic figure when compared to C5 boosting, as over-fitting will have been contended with.

10) Creating a Gradient Boosting Machine.

This Blog entry is from the Probability and Trees section in Learn R.

A relatively underutilised classification tool, which is built upon the concept of boosted decision trees, is the Gradient Boosting Machine, or GBM.  The GBM is a fairly black box implementation of the methods covered thus far, in this section.  The concept of Boosting refers to taking under-performing classifications and singling them out for boosting, or rather creating a dedicated model targeting the weaker performing data.  The GBM is part of the GBM package, as such install that package:

1.png

Click Install to download and install the package:

2.png

Load the library:

library(GBM)
3.png

Run the line of script to console:

4.png

The warning messages can be ignored as we can be reasonably assured of backward compatibility between the package build and this version of R.

Creating a GBM is similar to the familiar interfaces of regression, except for having a few parameters relating to the taming of the GBM:

gbm = gbm(Dependent ~., CreditRisk,
          n.trees=1000,
          shrinkage=0.01,
          distribution="gaussian",
          interaction.depth=7,
          bag.fraction=0.9,
          cv.fold=10,
          n.minobsinnode = 50
)

Run the block of script to console:

5.png

Run the line of script to console, it may take some time:

6.png

To review the performance statistics of the GBM, simply recall the model:

gbm
7.png

Run the line of script to console:

8.png

The most salient information from this summary is that 1000 iterations were performed, with the cross validation diverging at tree 542.  A visual inspection of the cross validation can be presented by:

gbm.perf(gbm)
9.png

Run the line of script to console:

10.png

It can be seen that the line was drawn at the point divergence started:

11.png

As decision trees can become a little unwieldy, it might be prudent to inspect the relative importance of each of the independent variables with a view to pruning and rerunning the GBM training.  To understand the importance of each Independent Variable, wrap the summary function around the GBM:

summary(GBM)
12.png

Run the line of script to console:

13.png

The most useful and important variable is written out first, with the less important being written out last.  This is also displayed in a bar chart giving the overall usefulness of the independent variables at a glance:

14.png

9) Boosting and Recalling in C5.

This Blog entry is from the Probability and Trees section in Learn R.

Boosting is a mechanism inside the C5 package that will create many different models, then give opportunity for each model to vote a classification, with the most widely suggested classification being the prevailing classification.  The majority classification voted for wins. It could be argued that this is a form of Abstraction.

Simply add the argument 10 to indicate that there should be ten trials to vote:

C50Tree <- C5.0(CreditRisk[-1],CreditRisk$Dependent,trials = 10) 
1.png

Run the line of script to console:

2.png

The summary function will produce a report:

summary(C50Tree)
3.png

In this instance, however, upon scrolling up, it can be seen that several different models \ trials have been created:

4.png

In the above example the decision tree for the 9th trial has been evidenced.  Prediction takes place in exactly the same manner, using the predict() function,  except for it will run several models and established a voted majority classification.  This is boosting:

CreditRiskPrediction <- predict(C50Tree,CreditRisk)
5.png

Run the line of script to console:

6.png

In the above example the decision tree for the 9th trial has been evidenced.  Prediction takes place in exactly the same manner, using the predict() function,  except for it will run several models and established a voted majority classification.  This is boosting:

CreditRiskPrediction <- predict(C50Tree,CreditRisk)
7.png

Run the line of script to console:

8.png

A confusion matrix can be created to compare this object with that created in procedure 100:

CrossTable(CreditRisk$Dependent, CreditRiskPrediction)
9.png

Run the line of script to console:

10.png

In this example, it can be observed that there were 281 accounts where predicted to be bad, taking the CreditRiskPrediction column-wise, it can be observed there was a 1 account classification as bad in error.  Out of 281 classifications as bad, it can be said that the error rate is just 0.3%.  Referring to the original model as created, it can be seen that an 11% increase in performance has been achieved from boosting.

There is such a thing as a model being too good, which would indicate that the model is perhaps over-fit. Over-fitting is dealt with in more detail while exploring Gradient Boosting Machines and Neural Networks, however, at this stage it is sufficient to explain that one should never test a model on the same data used to train.

8) Expressing Business Rules from C5.

This Blog entry is from the Probability and Trees section in Learn R.

In traversing the C5 decision tree it is almost certain that when coming to deploy the model, beyond using the predict() function,  that it will be expressed or programmed as logical statements,  for example:

If Status_Of_Existing_Checking_Account < 200 EUR

AND Credit_History in ("All_Paid","No_Credit_Open_Or_All_Paid")

AND Housing = "Owner"

AND Purpose = "New Car"

AND Duration_In_Month < 22 THEN "Good"

1.png

To display the model as rules rather than a tree, it is necessary to rebuild the model specifying rules argument to be true:

C50Tree <- C5.0(CreditRisk[-1],CreditRisk$Dependent,rules=TRUE)
2.png

Thereafter, the summary() function can be used to output a series of rules created in the rebuild as opposed to a decision tree:

summary(C50Tree)
3.png

Run the line of script to console:

4.png

Scrolling up in the console, it can be observed, towards the top, that in place of a decision tree a series of rules has been created:

5.png

These rules can be deployed with very small modification far more intuitively in a variety of languages, not least SQL.

6) Creating a Confusion Matrix for a C5 Decision Tree.

This Blog entry is from the Probability and Trees section in Learn R.

Beyond the summary statistic created, the confusion matrix is the most convenient means to appraise the utility of a classification model. The confusion matrix for the C5 decision tree model will be created using the CorssTable function of the gmodels() package:

library("gmodels")
CrossTable(CreditRisk$Dependent, CreditRiskPrediction)
1.png

Run the line of script to console:

2.png

The overall utility of the C5 decision tree model can be inferred in the same manner as procedure 100.

The confusion matrix classified 206 records as being bad correctly, taking CreditRiskPrediction column wise, it can be seen that 28 records were classified as Bad yet they were in fact Good.  It can be said that there is an 11.9% error rate on records classified as bad by the model.  Taking note of this metric, in procedure 112 boosting will be attempted which should bring about improvement of this model.