Repository navigation
Support downcasting time64[ns] to time64[us] on write (#1169) - #4471
Open
entertainingrover01 wants to merge 1 commit into
Open
entertainingrover01 wants to merge 1 commit into
entertainingrover01 wants to merge 1 commit into
Conversation
Iceberg time is microsecond precision, so time64[ns] columns previously failed schema conversion with 'Unsupported type: time64[ns]'. Mirror the existing timestamp[ns] downcast behavior: with the 'downcast-ns-timestamp-to-us-on-write' property set, time64[ns] converts to TimeType (with a warning) and time64[ns] arrays are cast to time64[us] on write; without it, conversion raises a TypeError pointing at the property. time64[us] behavior is unchanged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1169
Rationale for this change
PyArrow
pa.time64("ns")was unsupported on write and raisedUnsupported type: time64[ns], even though Iceberg'stimetype is microsecond precision by spec. Users had to manually downcast atime64[ns]column before writing, while the analogoustimestamp[ns]case has been handled by an opt-in downcast since #848.This change mirrors that existing
ns -> ustimestamp behavior fortime, gated on the samedowncast-ns-timestamp-to-us-on-writeconfiguration property:pyiceberg/io/pyarrow.py,_ConvertToIceberg.primitive: atime64[ns]PyArrow type now maps to IcebergTimeType()(with a warning) whendowncast-ns-timestamp-to-us-on-writeis set, and otherwise raises aTypeErrorpointing the user at that property.time64[us]keeps working unchanged.pyiceberg/io/pyarrow.py,ArrowProjectionVisitor._cast_if_needed: a newTimeTypebranch casts atime64[ns]array totime64[us](safe=False) on write when the flag is set, so the data, not just the schema mapping, is actually downcast.test_pyarrow_time64_ns_to_icebergand addedtest_pyarrow_time64_ns_to_iceberg_downcast,test__to_requested_schema_time_ns_downcast, andtest__to_requested_schema_time_ns_without_downcast_raises_exception.Are these changes tested?
Yes —
pytest tests/io/test_pyarrow_visitor.py tests/io/test_pyarrow.pypasses (296 passed, 3 skipped), including the 3 new tests and the updated one.ruff checkandruff format --checkpass on all changed files.