Skip to content

Support downcasting time64[ns] to time64[us] on write (#1169) - #4471

Open
entertainingrover01 wants to merge 1 commit into
apache:mainfrom
entertainingrover01:fix/time64-ns-downcast-1169
Open

entertainingrover01 wants to merge 1 commit into
apache:mainfrom
entertainingrover01:fix/time64-ns-downcast-1169

Conversation

@entertainingrover01

Copy link
Copy Markdown

Closes #1169

Rationale for this change

PyArrow pa.time64("ns") was unsupported on write and raised Unsupported type: time64[ns], even though Iceberg's time type is microsecond precision by spec. Users had to manually downcast a time64[ns] column before writing, while the analogous timestamp[ns] case has been handled by an opt-in downcast since #848.

This change mirrors that existing ns -> us timestamp behavior for time, gated on the same downcast-ns-timestamp-to-us-on-write configuration property:

  • pyiceberg/io/pyarrow.py, _ConvertToIceberg.primitive: a time64[ns] PyArrow type now maps to Iceberg TimeType() (with a warning) when downcast-ns-timestamp-to-us-on-write is set, and otherwise raises a TypeError pointing the user at that property. time64[us] keeps working unchanged.
  • pyiceberg/io/pyarrow.py, ArrowProjectionVisitor._cast_if_needed: a new TimeType branch casts a time64[ns] array to time64[us] (safe=False) on write when the flag is set, so the data, not just the schema mapping, is actually downcast.
  • Tests: updated test_pyarrow_time64_ns_to_iceberg and added test_pyarrow_time64_ns_to_iceberg_downcast, test__to_requested_schema_time_ns_downcast, and test__to_requested_schema_time_ns_without_downcast_raises_exception.

Are these changes tested?

Yes — pytest tests/io/test_pyarrow_visitor.py tests/io/test_pyarrow.py passes (296 passed, 3 skipped), including the 3 new tests and the updated one. ruff check and ruff format --check pass on all changed files.

Iceberg time is microsecond precision, so time64[ns] columns previously
failed schema conversion with 'Unsupported type: time64[ns]'. Mirror the
existing timestamp[ns] downcast behavior: with the
'downcast-ns-timestamp-to-us-on-write' property set, time64[ns] converts
to TimeType (with a warning) and time64[ns] arrays are cast to
time64[us] on write; without it, conversion raises a TypeError pointing
at the property. time64[us] behavior is unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature request] Support Time64Type[ns]

2 participants