Repository navigation
Fix scan filters at the edges of the date and float ranges - #4406
Anurag-S1ngh wants to merge 5 commits into
Conversation
rahulsmahadev
left a comment
There was a problem hiding this comment.
Thanks! I ran this locally against main (33b4ce5):
- The new tests fail on main and pass here (the parametrized cases inside the date range pass on both, as expected).
_civil_from_daysmatchesdatetime.datefor every day from 0001-01-01 to 9999-12-31, so results in that range are unchanged.- The float change also fixes deletes and range filters with an infinity. On main,
x < infbinds toAlwaysTrue, sotable.delete(LessThan("x", math.inf))on a float column deletes every row, including the inf, NaN and null rows; here it keeps exactly those three.<,<=,>and>=with an infinity all change this way, so they are worth listing in the description, with a test.
The one bug I found is in the negative-era branch of _civil_from_days (inline comment). A few more points:
- Java's
DoubleLiteral.to(FLOAT)inLiterals.javahas the same range check, so Java bindsfloat_col = <double infinity>toalwaysFalseas well. Worth saying in the description that this differs from Java. StringLiteral.to(FloatType)applies the same check, soEqualTo("x", "Infinity")still returns no rows andIn("x", ["Infinity", "1.5"])only1.5. Fine as a follow-up, but it is the same bug.- Nit:
2932897 * day_nanosintest_years_and_months_beyond_the_date_rangeis beyond int64, so notimestamp_nscan hold it (they end in 2262); that assert pins an unreachable case. - This conflicts with #4407 (and #4401, #4402) in
tests/io/test_pyarrow.py, since they all add tests right aftertest_append_to_table_bucketed_by_timeand edit the same import block.
| https://howardhinnant.github.io/date_algorithms.html#civil_from_days. | ||
| """ | ||
| z = days + 719468 | ||
| era = (z if z >= 0 else z - 146096) // 146097 |
There was a problem hiding this comment.
This is the C++ trick for floor division, but Python's // already floors, so for z < 0 (days before 0000-03-01) era comes out one too small. About 3% of those days get the wrong month, and some the wrong year. For example, -0001-12-31 (days=-719529) gives year -1970 and month -23640 instead of -1971 and -23641, while the writer puts that date in year partition -1971. So EqualTo("d", -719529) on a year-partitioned table projects to d_year == -1970 and prunes the file that holds the row; main raises OverflowError there instead.
| era = (z if z >= 0 else z - 146096) // 146097 | |
| era = z // 146097 |
With this change the function matches datetime on the whole representable range and gives the same values as Java's DateTimeUtil.daysToYears / daysToMonths before year 1, and the tests in this PR still pass. None of the new test days has z < 0; adding (-719529, -1971, -23641) to test_days_to_years_and_months would pin it.
There was a problem hiding this comment.
Fixed in 16903cf: era = z // 146097. Added (-719529, -1971, -23641) to test_days_to_years_and_months, so the z < 0 branch is pinned now.
Fokko
left a comment
There was a problem hiding this comment.
Thanks @Anurag-S1ngh! Two real user-facing bugs fixed here, and the 9999-12-31 "no end date" case is exactly the kind of value that shows up in production data.
I pulled the branch and verified:
tests/test_transforms.py,tests/expressions/test_projection.py,tests/utils/test_datetime.py,tests/expressions/test_literals.pyall pass- Brute-forced
_civil_from_daysagainst Python'sdatetimeover 20k random days in the valid range: identical year/month everywhere - Compared old vs new
days_to_months/micros_to_months/nanos_to_*over 5k random samples: results identical inside the representable range, so no behavior change for existing data - Reproduced both scan scenarios end to end:
EqualTo("x", inf)on a float column now returns the right row, and a year-partitioned table with9999-12-31scans correctly withLessThanOrEqualandGreaterThan - Checked pre-epoch timestamps (e.g. 1969-12-31 23:59:59.999999): months/years come out the same as before
One small question, non-blocking: in pyiceberg/utils/datetime.py, micros_to_months(micros) now does micros // 86_400_000_000 first. Since that's floor division, negative micros (pre-1970) floor toward negative infinity, which matches the old day-boundary behavior. Just double-checked it, all good.
CI is fully green. LGTM!
|
Thanks for the review. Addressed in 16903cf, 60759e1 and eb30c12:
|
|
Thanks for checking. Nothing to change there, floor division for negative micros matches the previous day-boundary behavior. |
Year and month partitions raised OverflowError when a filter projected past 9999-12-31 or before 0001-01-01, because the transform converted the day count to a datetime.date. Compute years and months from the day count with integer arithmetic, so projections just past the last representable date produce year 10000 and match no partition instead of raising. A Python float became a DoubleLiteral, and converting that to a float column turned inf into FloatAboveMax or FloatBelowMin, which bound the predicate to AlwaysFalse or AlwaysTrue. inf is a valid float value, so convert it to a FloatLiteral like any other float that fits.
The C++ algorithm truncates toward zero and compensates with z - 146096, but Python's // already floors. The correction made the era one too small for days before 0000-03-01, so year and month partitions before year 1 projected to the wrong partition.
A timestamp_ns cannot hold 2932897 days, so the asserts covered an input no table can contain.
A double infinity on a float column now binds as a float literal, so <, <=, > and >= match rows instead of binding to always true or always false.
68dc05b to
eb30c12
Compare
rahulsmahadev
left a comment
There was a problem hiding this comment.
Thanks @Anurag-S1ngh, the changes look right to me. I re-ran my checks on 429ace2: _civil_from_days matches datetime for every day from 0001-01-01 to 9999-12-31 and follows the 400-year cycle outside that range, the year and month values agree with what the writer produces (also before year 1), and the infinity filters and delete return the expected rows. LGTM.
Summary
Filtered scans raised or returned wrong rows at two kinds of edge values.
OverflowErrorwhen a filter projected past 9999-12-31 or before 0001-01-01, because the transform converted the day count to adatetime.date. 9999-12-31 is a common "no end date" value.EqualTo,In,NotEqualTo, and the range comparisons (<,<=,>,>=) oninfreturned wrong rows for float columns. A Python float becomes aDoubleLiteral, and converting that to a float column turnedinfintoFloatAboveMaxorFloatBelowMin, which bound the predicate toAlwaysFalseorAlwaysTrue.Changes
infand-infto aFloatLiteralfor float columns, like any other float that fits. For float columns this also changes how the range comparisons bind:<and<=withinfno longer match every row, and>and>=with-infno longer match every row. For example,table.delete(LessThan("x", math.inf))used to delete every row, including the inf, NaN and null rows, and now leaves exactly those rows.Notes
DoubleLiteral.to(FLOAT), so Java bindsfloat_col = <double infinity>toalwaysFalse. This change intentionally diverges from that.StringLiteral.to(FloatType)still turns values such as"Infinity"intoFloatAboveMax, soEqualTo("x", "Infinity")returns no rows. That is the same bug through the string path and is left for a follow-up.Testing
inf,Inand the range comparisons on a float column, and a year-partitioned table scanned at 9999-12-31.tests/utils/test_datetime.py,tests/expressions/test_literals.py,tests/expressions/test_projection.pyand the non-cloud tests intests/io/test_pyarrow.pypass (493 passed, 1 skipped).Closes #4404