Performance and Prediction¶

Multinomial AUC (Area Under the ROC Curve)¶

This model metric is used to evaluate how well a multinomial classification model is able to distinguish between true positives and false positives across all domains. The metric is composed of these outputs:

where (c) is the number of classes and (text{AUC}(j, rest_j)) is the AUC with class (j) as the positive class and rest classes (rest_j) as the negative class. The result AUC is normalized by number of classes.

where (c) is the number of classes, (text{AUC}(j, rest_j)) is the AUC with class (j) as the positive class and rest classes (rest_j) as the negative class and (p(j)) is the prevalence of class (j) (number of positives of class (j)). The result AUC is normalized by sum of all weights.

where (c) is the number of classes and (text{AUC}(j, k)) is the AUC with class (j) as the positive class and class (k) as the negative class. The result AUC is normalized by number of all class combinations.

where (c) is the number of classes, (text{AUC}(j, k)) is the AUC with class (j) as the positive class and class (k) as the negative class and (p(j cup k)) is prevalence of class (j) and class (k) (sum of positives of both classes). The result AUC is normalized by sum of all weights.

Result Multinomial AUC table could look for three classes like this:

Note Macro and weighted average values could be the same if the classes are same distributed.

type

first_class_domain

second_class_domain

auc

1 vs Rest

1

None

0.996891

2 vs Rest

2

None

0.996844

3 vs Rest

3

None

0.987593

Macro OVR

None

None

0.993776

Weighted OVR

None

None

0.993776

1 vs 2

1

2

0.969807

1 vs 3

1

3

1.000000

2 vs 3

2

3

0.995536

Macro OVO

None

None

0.988447

Weighted OVO

None

None

0.988447

Default value of AUC

Multinomial AUC metric can be used for early stopping and during grid search as binomial AUC. In case of Multinomial AUC only one value need to be specified. The AUC calculation is disabled (set to NONE) by default. However this option can be changed using auc_type model parameter to any other average type of AUC and AUCPR - MACRO_OVR, WEIGHTED_OVR, MACRO_OVO, WEIGHTED_OVO.

Example

Notes - Calculation of this metric can be very expensive on time and memory when the domain is big. So it is disabled by default. - To enable it setup system property sys.ai.h2o.auc.maxClasses to a number.