Loan data analysis using Azure ML, part 3

This blog post focuses on model evaluation.  Accuracy and AUC (Area Under Curve) are two metrics we focus on since they are good for binary classifications.

Accuracy deals with ones and zeros, meaning you either got the class label right or you didn’t based on a certain threshold cur off.  The most natural threshold is 0.5 which means <0.5 is one of the output values and >0.5 is the other.  However many classifiers quantify uncertainty of the output by providing a probability value.  ROC tries all of the cutpoint and plots the sensitivity and specificity.  An ROC curve is a very common way to visualize the performance of a binary classifier such as logistic regression, which is what we used in my experiment.  In an ROC curve the true positive rate (Sensitivity) is plotted in function of the false positive rate (100-Specificity) for different cut-off points of a parameter.  AzureML Studio provides an easy way to construct this curve.  Below’s the curve from my experiment.   You can move the threshold slide bar and see the accuracy change.  The AUC remains constant.



The area under a ROC curve quantifies the overall ability of the test to discriminate between those individuals with the disease and those without the disease.  A truly useless test (one no better at identifying true positives than flipping a coin) has an area of 0.5.  A perfect test (one that has zero false positives and zero false negatives) has an area of 1.00.  An AUC value of 0.943 as in this case is considered very good.