Skip to content

Fail with exit 3 when a scan comparison returns no artifacts - #461

Draft
David Larsen (dc-larsen) wants to merge 6 commits into
mainfrom
fix/null-diff-artifacts-exit-3
Draft

David Larsen (dc-larsen) wants to merge 6 commits into
mainfrom
fix/null-diff-artifacts-exit-3

Conversation

@dc-larsen

@dc-larsen David Larsen (dc-larsen) commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

If the diff-scan poll returns a finished comparison with artifacts: null and incomplete unset, 2.13.0 logs

Diff scan comparison failed with Exception(Error fetching diff scan ...: unexpected response: ...), falling back to the streaming scan comparison
Diff comparison mode: requested=diff-scan effective=streaming reason=Exception

and finishes on the streaming comparison. The streaming response has no incomplete flag, so the run reports on a comparison that nothing has checked for completeness. With incomplete: true the same response correctly exits 3 (#453). #453's description reads as if both cases fail ("including when its artifacts are null or absent"). That holds only when incomplete is also set. Found while checking 2.13.0 for a customer who previously ran their own null-artifacts check in CI.

Root Cause

Core.get_diff_scan_artifacts raises IncompleteComparisonError for incomplete: true but a plain Exception for a missing artifacts object. Core.get_added_and_removed_packages re-raises only IncompleteComparisonError. Every other exception is treated as "the diff-scans flow is unavailable" and falls back to fullscans.stream_diff.

Is the fallback deliberate for this shape?

The fallback (added in #284) is deliberate for failures of the diff-scans flow: a token without the diff-scans:* scopes (403), deployments without the endpoint, the 30-minute polling backstop, and a response truncated on a huge dependency tree (response.json() fails). All of those still fall back. This PR doesn't change them.

A finished (non-202) response that returns the diff_scan object without an artifacts object is not one of those cases:

  • The API's finished response always carries an artifacts object with an incomplete boolean. omit_unchanged sets artifacts.unchanged to null, not artifacts, and the CLI already handles that (artifacts_dict.get(key) or []; covered by a new test here).
  • Large diffs don't come back as null artifacts. They fail as a truncated body, which still falls back. A response with no diff_scan object at all (204, unfollowed 3xx) is a flow failure too, and it still falls back.
  • The streaming endpoint's response has no incomplete field, so there is no completeness check to add on that path. Falling back turns a comparison the API couldn't deliver into one nobody checked.

So the choice is to fail with the infrastructure code, not to keep the fallback with a check that can't exist.

Fix

  • A finished poll response that has a diff_scan object whose artifacts is missing, null or not an object raises IncompleteComparisonError with its own message ("The scan comparison returned no results (diff scan ...): the API returned the diff scan without its artifacts object ..."). It reuses the existing class so it skips the fallback and maps through cli() like incomplete: true: exit 3, the --exit-code-on-api-error value, or 0 under --disable-blocking. That matches the README's "incomplete scan comparison" row, so the exit table is unchanged.
  • A response with no diff_scan object at all keeps 2.13.0's streaming fallback. The SDK's diffscans.get returns {} for a status it doesn't raise on (not 200/202, below 400: in practice a 204 or an unfollowed 3xx). That says nothing about the comparison, so it's treated like the other flow failures (403, 404, timeout, truncated body) instead of exiting 3 with a message claiming the API finished. The fallback warning now reads "response has no diff_scan object". A top-level incomplete: true still exits 3, as before.
  • Comments, the get_diff_scan_artifacts docstring, the IncompleteComparisonError docstring and docs/ci-cd.md updated.

Testing

  • tests/core/test_exit_codes.py: four response shapes with a diff_scan object (artifacts: null with incomplete: false, artifacts: null with incomplete unset, artifacts absent, artifacts: []) raise IncompleteComparisonError and never call stream_diff; the exit mapping is pinned for default / --exit-code-on-api-error 42 / --disable-blocking / both; unchanged: null under omit_unchanged still returns a result; incomplete: true still raises. New: {} and {"diff_scan": null} raise a plain exception and reach stream_diff, and a real HTTP 204 run through the SDK's own DiffScans.get() falls back to streaming. The 8 shape tests fail on main; the 204 test passes on main and failed on this PR's previous head.
  • Full suite with API key variables unset: uv run pytest: 1017 passed (1001 on main). ruff check socketsecurity tests, uv lock --locked --offline, git diff --check, and uv build + twine check pass.
  • End to end against a test org: a real PR-branch comparison (--scm api --integration api --ignore-commit-files --strict-blocking) on two projects: npm (lodash 4.17.11 + handlebars 4.0.11 baseline, the branch adds vm2 3.9.11) and PyPI (pyyaml 5.3 + django 2.2, the branch adds requests 2.19.1). The only change is the final diff-scan poll response, patched in-process (or, for 204, a real 204 response fed through the SDK's get()). Before = 2.13.0 from PyPI, after = this branch run twice; both after runs were identical on both projects:
Final poll response Flags 2.13.0 This branch
unpatched default / 42 / --disable-blocking diff-scan, exit 1 / 1 / 0 same
artifacts: null, incomplete unset default / 42 / --disable-blocking streaming, exit 1 / 1 / 0 exit 3 / 42 / 0, "returned no results"
artifacts absent; artifacts: [] default streaming, exit 1 exit 3
artifacts: null, incomplete: true default / 42 / --disable-blocking exit 3 / 42 / 0 same
SDK returns {}; {"diff_scan": null} default streaming, exit 1 same, "response has no diff_scan object"
real HTTP 204 via the SDK default / 42 / --disable-blocking streaming, exit 1 / 1 / 0 same
truncated JSON body (200); create_from_ids 403 default streaming, exit 1 same

Other open 2.13.x fixes

2.13.1 and 2.13.2 were released from main on 2026-11-20, so this branch merges main at v2.13.2 and bumps to 2.13.3 in pyproject.toml, socketsecurity/__init__.py and uv.lock, with its CHANGELOG entries under a new ## 2.13.3 heading. #459, #460, #461, #462, #463, #464 and #465 all make the same bump, so the version files merge cleanly between them, and once one merges the rest ride main's unreleased 2.13.3 (Version Check accepts a version equal to main's while it is above the latest PyPI release). Merging a second one conflicts only in CHANGELOG.md: keep one ## 2.13.3 heading and combine the bullets under its subsections (Upgrade notes, Changed, Fixed, Documentation, in that order). There are no code conflicts between them. If main releases 2.13.3 before this merges, bump to the next patch again.

Public Changelog

A scan comparison the API returns without its package lists now fails the run with the infrastructure-error code (3, or the --exit-code-on-api-error value) instead of falling back to a comparison that can't report whether it is complete.

When the diff-scan poll returned a finished comparison whose artifacts
were null and whose incomplete flag was unset, get_diff_scan_artifacts
raised a plain Exception. get_added_and_removed_packages treated that
like any other failure of the diff-scans flow and fell back to the
streaming comparison, whose response has no incomplete flag. The run
then reported on a comparison nobody had checked for completeness.

A finished comparison without an artifacts object now raises
IncompleteComparisonError, which skips the fallback and maps to the
infrastructure exit code (3, --exit-code-on-api-error, or 0 under
--disable-blocking). Failures of the flow itself (missing token scopes,
transient errors, the polling timeout, a truncated response) still fall
back to streaming as before.
@lelia lelia changed the title Fail with exit 3 when a scan comparison returns no artifacts Fail with exit 3 when a scan comparison returns no artifacts Nov 20, 2026
@lelia lelia self-assigned this Nov 20, 2026
The SDK's diffscans.get returns {} for statuses it doesn't raise on (204, an
unfollowed 3xx). That response says nothing about the comparison, so it falls
back to streaming as it did in 2.13.0 instead of exiting 3 with a message
claiming the API finished the comparison. Only a returned diff_scan whose
artifacts are null, missing or not an object raises the incomplete-comparison
error, and its message now says that.
2.13.2 was released from main, so this change needs the next patch version.

This branch was successfully deployed

1 active deployment
socket-firewall — d183dd18 Deployed Nov 20, 2026 by dc-larsen via python-sfw-smoke-enterprise #664
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants