Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #421 +/- ##
==========================================
- Coverage 92.88% 92.81% -0.08%
==========================================
Files 31 31
Lines 3289 3074 -215
==========================================
- Hits 3055 2853 -202
+ Misses 234 221 -13 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
This is how benchmark results would change (along with a 95% confidence interval in relative change) if f08b1cb is merged into master:
Further explanation regarding interpretation and methodology can be found in the documentation. |
jgabry
left a comment
There was a problem hiding this comment.
One small suggestion, otherwise looks good to me
| return(khat_r_i) | ||
| } | ||
| # scale so that the largest |h * r| is 1, which avoids underflow for small |h| | ||
| log_hr <- log(abs(x_i)) + log_ratios_i |
There was a problem hiding this comment.
| log_hr <- log(abs(x_i)) + log_ratios_i | |
| log_hr <- log(abs(x_i)) + log_ratios_i | |
| if (max(log_hr) == -Inf) { | |
| return(khat_r_i) | |
| } |
This avoids potentially passing NaN to posterior::pareto_khat(). Right now it works ok to pass NaN but it's undocumented behavior in posterior so I think better to avoid it in case something changes in the future.
There was a problem hiding this comment.
I added the check in f08b1cb, but as if (!is.finite(max(log_hr))) instead of == -Inf. E_loo() accepts log_ratios with NA if the psis() object is valid, and then max(log_hr) == -Inf is NA.
What E_loo() gives for the k (posterior 1.8.0):
| input | before | == -Inf |
!is.finite() |
|---|---|---|---|
| regular | max(k of h * r, psis() k) |
same | same |
h * r is 0 for all draws |
psis() k, warning |
psis() k |
psis() k |
some log ratios are -Inf |
max(k of h * r, psis() k) |
same | same |
one log ratio is NA or NaN |
psis() k, warning |
error | psis() k |
one log ratio is +Inf |
psis() k, warning |
psis() k, warning |
psis() k |
where
- warning:
pareto_khat()getsNaNorNAand warns "Input contains infinite or NA values ...". - error:
E_loo()stops with "missing value where TRUE/FALSE needed".
Summary
E_loo()gives anNAPareto k wherepsis()givesInffor the same log ratios. This occurs when many of the largest ratiosexp(log_ratios - max(log_ratios))underflow to 0, so that k cannot be estimated from the tail.psis()sets k toInfin this case.E_loo()estimates k from the tail of the same ratios a second time withposterior::pareto_khat(), which givesNAwith posterior 1.8.0.Example
Changes
With this PR,
E_loo()takes the k of the ratios frompsis_object, wherepsis()stored it, and estimates k only for the tails ofh * r. The k of the ratios then follows the rules ofpsis().psis()givesInfwhen it cannot estimate k from the tail. This occurs when the tail has fewer than 5 draws, when all tail values are equal, or when a quarter or more of the tail values are tied at the low end of the tail (as with the zeros from underflow). The PR also removes the code for posterior versions that returned a list (loo requires posterior >= 1.7.0) and the code for twoNAvalues.Other changes of the returned k:
log_ratiosofpsis_objectand a finitepsis()k, the k changes only by rounding (at most 3.6e-14 forexample_loglik_matrix()and for one example with a heavy tail).log_ratios, the k of the ratios is now thepsis()k of the raw ratios. Before, it was the k of the smoothed weights. For the 32 columns ofexample_loglik_matrix()andtype = "quantile", the change of the k is between -0.043 and 0.068. The k ofh * rstill uses the smoothed weights in this case.E_loo()now givesInf, aspsis()does. Before, it gave the k ofhfor a continuousx, andNAfor a binaryx.E_loo()now givesInf, aspsis()does. Before, it gaveNA(tested with S = 20).loo_approximate_posterior(), the k of the ratios is now the k of the corrected ratios that the weights use. Before, it was the k of thelog_ratiosargument.loo_moment_match()replaces the weights and the k in a saved psis object for the observations that it improves.E_loo()withxfrom the original draws is not valid for these observations, before and after this PR. With the originallog_ratios,E_loo()gave the high k of the original ratios for them. It now gives the moment-matched k.The k of
h * rThis PR changes also the scaling of
h * rbefore callingposterior::pareto_khat.E_loo()now scalesh * ron the log scale so that the largest|h * r|is 1. The k ofh * rthen does not change whenxis multiplied by a constant. On master,E_loo()scales only the ratios so that the largest ratio is 1. For1e-307 * xand the 32 columns ofexample_loglik_matrix(),E_loo()on master returns k =Inffor each column, becauseh * rhas subnormal values.This PR does not change the rule for the k that
E_loo()returns.E_loo()estimates k for the left tail and for the right tail ofh * rwithposterior::pareto_khat(). For a tail, k cannot be estimated when all tail values are equal, or when a quarter or more of them are tied.E_loo()returns the largest of the k of the ratios and the k values of the two tails ofh * r, and leaves out a tail for which k cannot be estimated. If k cannot be estimated for either tail ofh * r,E_loo()returns the k of the ratios.The table shows examples with S = 1000 and a tail of 95 draws (posterior 1.8.0). In rows 4 to 6, only 150, 100 and 30 ratios are nonzero; the other ratios underflow to 0. Only row 6 changes with this PR.
psis())h * r, left tailh * r, right tailE_loo()k, masterE_loo()k, this PRxx, 62% zerosxis 0 for 95% of the drawsIn the
NAcells of the table, a quarter or more of the tail values are exact zeros. In rows 2 and 3, exact zeros ofxcause the ties, andE_loo()uses the other tail (row 2) or the k of the ratios (row 3). In rows 4 and 5, the underflow of the ratios causes the ties.h * rthen has fewer nonzero values on each side of 0 than the tail has draws. The k of the ratios is still finite in these rows. With fewer zeros in the tail, k can be estimated and is high (row 4, left tail). TheE_loo()k is thus not monotone in the number of nonzero ratios. This PR does not change the rule for when k can be estimated.The documentation of
pareto_know says where the k of the ratios comes from. The documentation oflog_ratiosnow says that they are the log ratios thatpsis()used to computepsis_object.Tests
New tests in
tests/testthat/test_E_loo.Rcover:x,type = "quantile", nolog_ratios);h * r;psis()k for quantiles and a binaryx;xscaled by 1e-307 (added to the existing test of the scale ofx).On master, the new tests give 16 failures. The full test suite passes (1365 expectations) with R 4.6.2 and posterior 1.8.0.
This PR was assisted by Claude.