comp_accu_freq
computes a list of current accuracy metrics
from the 4 essential frequencies (hi
,
mi
, fa
, cr
)
that constitute the current confusion matrix and
are contained in freq
.
comp_accu_freq(hi = freq$hi, mi = freq$mi, fa = freq$fa, cr = freq$cr, w = 0.5)
The number of hits hi
(or true positives).
The number of misses mi
(or false negatives).
The number of false alarms fa
(or false positives).
The number of correct rejections cr
(or true negatives).
The weighting parameter w
(from 0 to 1)
for computing weighted accuracy wacc
.
Default: w = .50
(i.e., yielding balanced accuracy bacc
).
A list accu
containing current accuracy metrics.
Currently computed accuracy metrics include:
acc
: Overall accuracy as the proportion (or probability)
of correctly classifying cases or of dec_cor
cases:
acc = dec_cor/N = (hi + cr)/(hi + mi + fa + cr)
Values range from 0 (no correct prediction) to 1 (perfect prediction).
wacc
: Weighted accuracy, as a weighted average of the
sensitivity sens
(aka. hit rate HR
, TPR
,
power
or recall
)
and the the specificity spec
(aka. TNR
)
in which sens
is multiplied by a weighting parameter w
(ranging from 0 to 1) and spec
is multiplied by
w
's complement (1 - w)
:
wacc = (w * sens) + ((1 - w) * spec)
If w = .50
, wacc
becomes balanced accuracy bacc
.
mcc
: The Matthews correlation coefficient (with values ranging from -1 to +1):
mcc = ((hi * cr) - (fa * mi)) / sqrt((hi + fa) * (hi + mi) * (cr + fa) * (cr + mi))
A value of mcc = 0
implies random performance; mcc = 1
implies perfect performance.
See Wikipedia: Matthews correlation coefficient for additional information.
f1s
: The harmonic mean of the positive predictive value PPV
(aka. precision
)
and the sensitivity sens
(aka. hit rate HR
,
TPR
, power
or recall
):
f1s = 2 * (PPV * sens) / (PPV + sens)
See Wikipedia: F1 score for additional information.
Notes:
Accuracy metrics describe the correspondence of decisions (or predictions) to actual conditions (or truth).
There are several possible interpretations of accuracy:
Computing exact accuracy values based on probabilities (by comp_accu_prob
) may differ from
accuracy values computed from (possibly rounded) frequencies (by comp_accu_freq
).
When frequencies are rounded to integers (see the default of round = TRUE
in comp_freq
and comp_freq_prob
) the accuracy metrics computed by
comp_accu_freq
correspond to these rounded values.
Use comp_accu_prob
to obtain exact accuracy metrics from probabilities.
Consult Wikipedia: Confusion matrix for additional information.
accu
for all accuracy metrics;
comp_accu_prob
computes exact accuracy metrics from probabilities;
num
for basic numeric parameters;
freq
for current frequency information;
txt
for current text settings;
pal
for current color settings;
popu
for a table of the current population.
Other metrics:
accu
,
acc
,
comp_accu_prob()
,
comp_acc()
,
comp_err()
,
err
Other functions computing probabilities:
comp_FDR()
,
comp_FOR()
,
comp_NPV()
,
comp_PPV()
,
comp_accu_prob()
,
comp_acc()
,
comp_comp_pair()
,
comp_complement()
,
comp_complete_prob_set()
,
comp_err()
,
comp_fart()
,
comp_mirt()
,
comp_ppod()
,
comp_prob_freq()
,
comp_prob()
,
comp_sens()
,
comp_spec()
comp_accu_freq() # => accuracy metrics for freq of current scenario
#> $acc
#> [1] 0.774
#>
#> $w
#> [1] 0.5
#>
#> $wacc
#> [1] 0.7986667
#>
#> $mcc
#> [1] 0.5279731
#>
#> $f1s
#> [1] 0.6523077
#>
comp_accu_freq(hi = 1, mi = 2, fa = 3, cr = 4) # medium accuracy, but cr > hi
#> $acc
#> [1] 0.5
#>
#> $w
#> [1] 0.5
#>
#> $wacc
#> [1] 0.452381
#>
#> $mcc
#> [1] -0.08908708
#>
#> $f1s
#> [1] 0.2857143
#>
# Extreme cases:
comp_accu_freq(hi = 1, mi = 1, fa = 1, cr = 1) # random performance
#> $acc
#> [1] 0.5
#>
#> $w
#> [1] 0.5
#>
#> $wacc
#> [1] 0.5
#>
#> $mcc
#> [1] 0
#>
#> $f1s
#> [1] 0.5
#>
comp_accu_freq(hi = 0, mi = 0, fa = 1, cr = 1) # random performance: wacc and f1s are NaN
#> Warning: accu$mcc: A denominator of 0 was corrected to 1, resulting in mcc = 0.
#> $acc
#> [1] 0.5
#>
#> $w
#> [1] 0.5
#>
#> $wacc
#> [1] NaN
#>
#> $mcc
#> [1] 0
#>
#> $f1s
#> [1] NaN
#>
comp_accu_freq(hi = 1, mi = 0, fa = 0, cr = 1) # perfect accuracy/optimal performance
#> $acc
#> [1] 1
#>
#> $w
#> [1] 0.5
#>
#> $wacc
#> [1] 1
#>
#> $mcc
#> [1] 1
#>
#> $f1s
#> [1] 1
#>
comp_accu_freq(hi = 0, mi = 1, fa = 1, cr = 0) # zero accuracy/worst performance, but see f1s
#> $acc
#> [1] 0
#>
#> $w
#> [1] 0.5
#>
#> $wacc
#> [1] 0
#>
#> $mcc
#> [1] -1
#>
#> $f1s
#> [1] NaN
#>
comp_accu_freq(hi = 1, mi = 0, fa = 0, cr = 0) # perfect accuracy, but see wacc and mcc
#> Warning: accu$mcc: A denominator of 0 was corrected to 1, resulting in mcc = 0.
#> $acc
#> [1] 1
#>
#> $w
#> [1] 0.5
#>
#> $wacc
#> [1] NaN
#>
#> $mcc
#> [1] 0
#>
#> $f1s
#> [1] 1
#>
# Effects of w:
comp_accu_freq(hi = 3, mi = 2, fa = 1, cr = 4, w = 1/2) # equal weights to sens and spec
#> $acc
#> [1] 0.7
#>
#> $w
#> [1] 0.5
#>
#> $wacc
#> [1] 0.7
#>
#> $mcc
#> [1] 0.4082483
#>
#> $f1s
#> [1] 0.6666667
#>
comp_accu_freq(hi = 3, mi = 2, fa = 1, cr = 4, w = 2/3) # more weight to sens
#> $acc
#> [1] 0.7
#>
#> $w
#> [1] 0.6666667
#>
#> $wacc
#> [1] 0.6666667
#>
#> $mcc
#> [1] 0.4082483
#>
#> $f1s
#> [1] 0.6666667
#>
comp_accu_freq(hi = 3, mi = 2, fa = 1, cr = 4, w = 1/3) # more weight to spec
#> $acc
#> [1] 0.7
#>
#> $w
#> [1] 0.3333333
#>
#> $wacc
#> [1] 0.7333333
#>
#> $mcc
#> [1] 0.4082483
#>
#> $f1s
#> [1] 0.6666667
#>
## Contrasting comp_accu_freq and comp_accu_prob:
# (a) comp_accu_freq (based on rounded frequencies):
freq1 <- comp_freq(N = 10, prev = 1/3, sens = 2/3, spec = 3/4) # => hi = 2, mi = 1, fa = 2, cr = 5
accu1 <- comp_accu_freq(freq1$hi, freq1$mi, freq1$fa, freq1$cr) # => accu1 (based on rounded freq).
# accu1
#
# (b) comp_accu_prob (based on probabilities):
accu2 <- comp_accu_prob(prev = 1/3, sens = 2/3, spec = 3/4) # => exact accu (based on prob).
# accu2
all.equal(accu1, accu2) # => 4 differences!
#> [1] "Component “acc”: Mean relative difference: 0.03174603"
#> [2] "Component “wacc”: Mean relative difference: 0.02586207"
#> [3] "Component “mcc”: Mean relative difference: 0.1306675"
#> [4] "Component “f1s”: Mean relative difference: 0.07692308"
#
# (c) comp_accu_freq (exact values, i.e., without rounding):
freq3 <- comp_freq(N = 10, prev = 1/3, sens = 2/3, spec = 3/4, round = FALSE)
accu3 <- comp_accu_freq(freq3$hi, freq3$mi, freq3$fa, freq3$cr) # => accu3 (based on EXACT freq).
# accu3
all.equal(accu2, accu3) # => TRUE (qed).
#> [1] TRUE