Skip to content

Relabel UTC aliases when projecting timestamptz_ns columns - #4468

Open
g0itb4 wants to merge 1 commit into
apache:mainfrom
g0itb4:fix-timestamptz-ns-utc-alias
Open

g0itb4 wants to merge 1 commit into
apache:mainfrom
g0itb4:fix-timestamptz-ns-utc-alias

Conversation

@g0itb4

@g0itb4 g0itb4 commented Nov 22, 2026

Copy link
Copy Markdown

Closes #4467

Rationale for this change

_cast_if_needed relabels a UTC alias (+00:00, Etc/UTC, Z) to UTC for timestamptz
columns but not for timestamptz_ns. A table whose files come from writers that label UTC
differently, or where some files predate the column, then fails in pa.concat_tables. This adds
the same handling for TimestamptzNanoType, for nanosecond values only: no unit conversion, so a
value is never widened or truncated.

Are these changes tested?

Yes. test__to_requested_schema_timestamptz_ns_utc_alias_concatenates projects a file without the
column and a file with a +00:00-labelled timestamp[ns] column, checks the label is UTC, and
concatenates them; nanosecond digits are kept. It fails on main and passes with the change.

Are there any user-facing changes?

Scans of such tables succeed instead of raising ArrowTypeError. No API change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scan fails when a timestamptz_ns column's files label UTC as "+00:00" and other files lack the column

1 participant