Multinomial AUC (Area Under the ROC Curve)¶
This model metric is used to evaluate how well a multinomial classification model is able to distinguish between true positives and false positives across all domains. The metric is composed of these outputs:
where (c) is the number of classes and (text{AUC}(j, rest_j)) is the AUC with class (j) as the positive class and rest classes (rest_j) as the negative class. The result AUC is normalized by number of classes.
where (c) is the number of classes, (text{AUC}(j, rest_j)) is the AUC with class (j) as the positive class and rest classes (rest_j) as the negative class and (p(j)) is the prevalence of class (j) (number of positives of class (j)). The result AUC is normalized by sum of all weights.
where (c) is the number of classes and (text{AUC}(j, k)) is the AUC with class (j) as the positive class and class (k) as the negative class. The result AUC is normalized by number of all class combinations.
where (c) is the number of classes, (text{AUC}(j, k)) is the AUC with class (j) as the positive class and class (k) as the negative class and (p(j cup k)) is prevalence of class (j) and class (k) (sum of positives of both classes). The result AUC is normalized by sum of all weights.
Result Multinomial AUC table could look for three classes like this:
Note Macro and weighted average values could be the same if the classes are same distributed.
type
first_class_domain
second_class_domain
auc
1 vs Rest
1
None
0.996891
2 vs Rest
2
None
0.996844
3 vs Rest
3
None
0.987593
Macro OVR
None
None
0.993776
Weighted OVR
None
None
0.993776
1 vs 2
1
2
0.969807
1 vs 3
1
3
1.000000
2 vs 3
2
3
0.995536
Macro OVO
None
None
0.988447
Weighted OVO
None
None
0.988447
Default value of AUC
Multinomial AUC metric can be used for early stopping and during grid search as binomial AUC. In case of Multinomial AUC only one value need to be specified. The AUC calculation is disabled (set to NONE) by default. However this option can be changed using auc_type model parameter to any other average type of AUC and AUCPR - MACRO_OVR, WEIGHTED_OVR, MACRO_OVO, WEIGHTED_OVO.
Example
Notes - Calculation of this metric can be very expensive on time and memory when the domain is big. So it is disabled by default. - To enable it setup system property sys.ai.h2o.auc.maxClasses to a number.
