Repository navigation
Fix duplicate file checks after commit retry - #4446
Conversation
Revalidate add_files duplicate paths against the refreshed branch snapshot before rebuilding a retried snapshot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fokko
left a comment
There was a problem hiding this comment.
Great catch @GayathriSrividya 🙌 And thanks for working on this. I left one suggestion. LMKWYT
| producer._validate_retry() | ||
| producer._validate_concurrency() |
There was a problem hiding this comment.
These two methods look similar, could we merge the _validate_retry() into _validate_concurrency()
There was a problem hiding this comment.
Thanks for the suggestion @Fokko , I merged the retry validation into _validate_concurrency() while preserving fast append’s behavior. The commit-retry tests pass locally.
There was a problem hiding this comment.
Appreciate the quick follow-up 🙌 Let me check!
| break | ||
| self._rebuild_snapshot_updates() | ||
| except (CommitFailedException, ValidationException): | ||
| except (CommitFailedException, ValidationException, ValueError): |
There was a problem hiding this comment.
One more comment! I think this is a risky change. The ValueError can be general catch-all. Let's say that the commit succeeded, but there is an issue with the response. In this situation all the files will be cleaned up. Can we avoid this change? For example, when it find a duplicate file, that we raise a ValidationException instead?
There was a problem hiding this comment.
Thanks for catching this, @Fokko . You’re right that catching ValueError broadly could trigger cleanup when the commit outcome isn’t known to be safe. I’ve updated retry-time duplicate detection to raise ValidationException and removed ValueError from the cleanup handler, while preserving ValueError for the initial duplicate check. Sorry for overlooking that cleanup-safety implication in my earlier change. The focused commit-retry tests pass (174 passed).
Raise ValidationException when a duplicate is discovered during a commit retry so cleanup only runs for validation failures known to be safe. Keep ValueError for the initial duplicate check.
Fokko
left a comment
There was a problem hiding this comment.
Thanks @GayathriSrividya for your patience, this looks great! Thanks for catching this edge-case!
Closes #4445
Rationale for this change
add_files(check_duplicate_files=True)checked duplicate paths only before the first commit attempt. During a conflict retry, the snapshot producer rebuilt the append using refreshed metadata without repeating the duplicate-path validation, allowing concurrent writers to reference the same file twice.This change registers retry-time validation on snapshot producers and reruns it after refreshing metadata. Duplicate paths now raise
ValueErrorbefore the retried snapshot is rebuilt.Are these changes tested?
ruff check pyiceberg/table/__init__.py pyiceberg/table/update/snapshot.py tests/table/test_commit_retry.py— passedruff format --check pyiceberg/table/__init__.py pyiceberg/table/update/snapshot.py tests/table/test_commit_retry.py— passedpython -m compileall -q pyiceberg/table/__init__.py pyiceberg/table/update/snapshot.py tests/table/test_commit_retry.py— passedgit diff --check— passedAre there any user-facing changes?
Yes. Concurrent
add_filescalls can no longer add the same file twice after a commit retry. Duplicate files are rejected with the existing duplicate-fileValueError.