Skip to content

Bazel: Retry transient Git LFS batch failures - #23190

Open
josefs wants to merge 1 commit into
mainfrom
josefs/lfs-transport-hardening
Open

josefs wants to merge 1 commit into
mainfrom
josefs/lfs-transport-hardening

Conversation

@josefs

@josefs josefs commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This is an attempt at deflaking CI. With some copilot investigation, it's clear that LFS connection failure is a common recurring problem. This PR adds retries in an attempt at hardening the code. Specifically it retries timeouts, connection errors (including RemoteDisconnected), and HTTP 408/429/500/502/503/504.

Evidence

Pre-change CI baseline

Seven complete UTC days: November 13–19, 2026 (2026-11-13T00:00:00Z inclusive to 2026-11-20T00:00:00Z exclusive).

Automated transient-failure reports recorded 11 distinct workflow runs failing at the LFS batch/probe step. Each was confirmed against its original first-attempt job log.

LFS batch/probe failure Workflow runs
HTTP 429 rate limiting 8
RemoteDisconnected 2
Request timeout 1
Total 11

Counting rule: select reports by their creation timestamp in the UTC window, then count each workflow run once. Exclude failures downloading the actual LFS archives, failures in other services, and duplicate downstream errors. These are counts of recorded failures, not an exhaustive CI census or a failure rate. Ten of the eleven were reported on November 17.

For the after comparison, repeat the same collection over seven complete UTC days once CI is using the landed helper. Baseline collected November 20, 2026; per-run evidence retained separately.

Regression tests

  • Before: Run the new tests against the helper from origin/main: Ran 8 tests, FAILED (failures=4, errors=12) including parameterized cases. Disconnects/timeouts escape, rate limits are not retried, and malformed-response fallback sends the wrong payload.
    After: The same suite passes all 8 tests, including Retry-After seconds/dates, malformed and expired headers, the 30s/31s boundary, exhaustion, permanent errors, authenticated payload reuse, and safe diagnostics.
  • python3 -m unittest discover -s misc/bazel/internal -p 'test_git_lfs_probe.py' -v — passes.
  • bazel test //misc/bazel/internal:test_git_lfs_probe --jobs=2 --loading_phase_threads=2 --test_output=errors — 1/1 test target passes.
  • git diff --check — clean.

Network, Git subprocesses, and time are mocked. No live LFS service test or full CodeQL build was run.

Merge Danger

Door: two-way

Blast Radius: Builds

Only the LFS batch-request helper changes. Transient failures can incur bounded extra attempts and waits before endpoint fallback; cached downloads are unaffected.

@josefs
josefs requested a review from a team as a code owner November 20, 2026 16:45
Copilot AI adaptive review requested due to automatic review settings November 20, 2026 16:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A truncated response can raise an uncaught IncompleteRead, aborting the probe instead of retrying or trying the next endpoint; retry handling and a regression test are needed.

1 open issue
What changed in this PR

This PR makes Bazel’s Git LFS batch-request helper more resilient to transient failures while preserving authentication and the existing download/cache path.

Changes:

  • Add bounded retries with exponential backoff and Retry-After handling.
  • Preserve the batch payload across endpoint fallback and avoid sensitive exception details in diagnostics.
  • Add mocked regression tests and a Bazel test target.
File Description
misc/​bazel/​internal/​test_git_lfs_probe.py Tests retries, delays, fallback, authentication reuse, and safe diagnostics.
misc/​bazel/​internal/​git_lfs_probe.py Adds retry handling and preserves request data during fallback.
misc/​bazel/​internal/​BUILD.bazel Registers the Python regression suite as a Bazel test.

Review effort: Balanced (adaptive)

Comment thread misc/bazel/internal/git_lfs_probe.py Resolved
Bound retries for transient batch-request errors, respect Retry-After, and preserve authenticated endpoint fallback without logging sensitive response details.

Handle truncated response bodies as well as connection errors: json.load can raise IncompleteRead, which is not a ConnectionError. Cover recovery and exhausted retries using in-memory HTTP responses alongside the other deterministic regressions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@josefs
josefs force-pushed the josefs/lfs-transport-hardening branch from 247b106 to ea6608d Compare November 20, 2026 17:03
@josefs
josefs requested a review from kaspersv November 20, 2026 17:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants