Skip to content

Leave intermediate snapshots out of the snapshot log - #4470

Open
ShivamSharma43 wants to merge 1 commit into
apache:mainfrom
ShivamSharma43:intermediate-snapshots-in-log
Open

ShivamSharma43 wants to merge 1 commit into
apache:mainfrom
ShivamSharma43:intermediate-snapshots-in-log

Conversation

@ShivamSharma43

Copy link
Copy Markdown
Contributor

Rationale for this change

A commit that sets main more than once adds every one of those snapshots to the snapshot log. Table.overwrite does this when it commits a delete and an append together, so time travel can resolve to the delete snapshot, which was never the committed state of the table:

table.append(pa.table({"x": [1]}))
table.overwrite(pa.table({"x": [2]}))

_, delete, _ = table.snapshots()
snapshot = table.snapshot_as_of_timestamp(delete.timestamp_ms)
table.scan(snapshot_id=snapshot.snapshot_id).to_arrow()  # no rows, expected x = 1

This happens in catalogs where PyIceberg applies the updates itself. Java's TableMetadata.Builder suppresses these entries (intermediateSnapshotIdSet and updateSnapshotLog). This change does the same: snapshots that are added and set as main, but are not current once all updates are applied, are removed from the log. They stay in snapshots.

Are these changes tested?

Yes, with a unit test on update_table_metadata and a test that overwrites a table in the memory and SQL catalogs. Both fail without the fix.

Are there any user-facing changes?

Yes. Table.history() and inspect.history() have one entry for a commit that replaces its own snapshots, and snapshot_as_of_timestamp no longer returns the replaced ones. Tables that already have such entries in their log are not rewritten.

A commit that sets main more than once, such as an overwrite, added every
snapshot to the snapshot log, so time travel could resolve to a snapshot
that was never the committed state of the table. Keep only the last one,
like Java does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 22, 2026 17:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants