Skip to content

fix: isolate tests from the user's Helm config, harden JAR download - #464

Merged
manusa merged 3 commits into
mainfrom
fix/test-isolation
Nov 22, 2026
Merged

manusa merged 3 commits into
mainfrom
fix/test-isolation

Conversation

@manusa

@manusa manusa commented Nov 22, 2026 •

Copy link
Copy Markdown
Owner

Test hygiene follow-ups to #463 (context: #409). The Java and Go tests wrote registry logins, repository indexes and lock files into the developer's own Helm config, cache and temp directory. The test OCI servers were never stopped. The remote native JAR download could hang forever, or cache a truncated JAR.

Changes

  • Java test isolation (helm-java/pom.xml)
    • Surefire points HELM_CACHE_HOME and HELM_DATA_HOME at helm-java/target/helm-home/ (the helm.test.home property). It points HELM_CONFIG_HOME and DOCKER_CONFIG at helm-java/target/test-classes/helm-home/.
    • The registry configs there come from the test resources (src/test/resources/helm-home/), so Maven and IDEs both copy them. They hold a placeholder entry. Without one, Helm's registry client (oras-go DetectDefaultNativeStore) stores logins in the OS credential store: Keychain, Credential Manager, pass or Secret Service.
    • excludedEnvironmentVariables drops every HELM_* variable Helm v3.22.1 reads that would redirect its files or change its behaviour: those in pkg/cli/environment.go, plus HELM_DRIVER (read by native/internal/helm/helm.go) and HELM_DRIVER_SQL_CONNECTION_STRING (pkg/action/action.go). That list is the same as in native/internal/test/env.go.
    • HelmPackageTest.withoutChart packages an empty temp directory instead of the working directory. The working directory now holds the cached repository indexes, which are over Helm's 5 MiB file limit.
    • AGENTS.md documents all of this, including the trade-off: Testcontainers' KinD and Ryuk image pulls are anonymous.
  • Go test isolation (native/internal/test)
    • Both TestMains now really run their cleanup. Before, os.Exit skipped the deferred cleanup and left a helm-test-* directory behind on every run.
    • The new test.RepositoryConfig(t, content) puts repositories.yaml in t.TempDir(). With os.CreateTemp("", "repositories.yaml"), RepoAdd left a shared $TMPDIR/repositories.lock behind.
    • The registry configs are seeded like on the Java side, and a failed seed write now fails the tests.
    • The excluded HELM_* list matches the Java one.
  • Test OCI servers are stopped: RepoServerStop / RepoServerStopAll call distribution's Registry.Shutdown with a bounded context. The stray os.RemoveAll(server.id), which ran on a relative path in the working directory, is gone. HelmRegistryTest and HelmShowTest stop their servers in @AfterEach, as HelmPushTest does.
  • RemoteJarLoader
    • Connect and read timeouts (10 s / 30 s).
    • A downloaded JAR is checked before it is cached. A connection dropped mid-body reads as a normal end of stream, so a truncated JAR used to be cached as if complete. The checks:
      • its size against the Content-Length, a cheap first check;
      • then it must open as a ZIP. The central directory is at the end of the file, so this also catches truncated chunked or close-delimited responses.
    • It downloads to a unique .part file and moves it into place atomically, so no JVM sees or deletes another one's partial file.
    • On Windows, if a concurrent JVM already holds a complete JAR of the same size open, that one is used.
    • A failed download is no longer swallowed. When no native library can be found, the IllegalStateException from NativeLibrary.getInstance() keeps its message and now carries the download or metadata failure as its cause, for example a timeout, an HTTP 404 or a truncated download. This addresses the review comment about swallowed download failures.
  • Test fixtures
    • The Go repo-server tests call RepoServerStart directly, with no goroutines, busy-polling or fixed sleeps, and use an HTTP client with a timeout.
    • The OCI server stop is now covered.
    • HelmTestServerTest sets connection timeouts.

Release notes

RemoteJarLoader (the fallback that downloads the platform JAR when none is on the classpath):

  • Downloads now time out: 10 s to connect, 30 s without data.
  • A truncated download is rejected instead of cached: one shorter than its Content-Length, or one that doesn't open as a valid ZIP (its end record and central directory; entry data isn't checked).
  • With com.marcnuri.jkube-helm.forceUpdate, the previously cached JAR is kept when the new download fails. Before, it was deleted first.
  • When no native library is found and the download failed, the No NativeLibrary implementation found error now includes the underlying download failure as its cause.

Testing

CI on 7970ee6 (this PR's first commit) passed on Linux, macOS and Windows, including the envtest suite, HelmKubernetesTest (KinD, with the isolated DOCKER_CONFIG) and NativeLibraryTest. The follow-up commits change RemoteJarLoader and the seed location, and only CI exercises the real Maven Central download and the macOS/Windows paths, so CI runs again on them. On c4c06dc, macOS failed only NativeLibraryTest.getSnapshotInstanceFromRemoteJar, with the generic message. The snapshot JAR it fetched is complete (its Content-Length matches, and unzip -t passes), so it was most likely a network failure. The swallowed exception hid the reason, which is why the error now carries it.

Locally, on Java 8 (linux/amd64), in a sandbox where ~/.config/helm and ~/.cache/helm are read-only, so passing tests can't have written there:

  • lib/api: 51/51. RemoteJarLoaderTest runs against a local HttpServer, with no mocks. It covers the release and snapshot (maven-metadata.xml) lookups, and that a missing JAR, missing metadata or invalid metadata surfaces as an exception naming the URL.
  • NativeLibraryTest: the offline cases pass, and with the remote JAR disabled the error has no cause, as before. The remote cases can't reach Maven Central from the sandbox, and their failure now shows the cause (UnknownHostException: repo1.maven.org) instead of only the generic message.
  • helm-java: Install, Template, Search, Repo, Registry, Push, Show, Package, Dependency, TestServer and SignalHandler all pass, 91/91, after a clean, with only the chart hosts reachable.
    • HelmSearchTest ran before HelmPackageTest.
    • The repository indexes land in target/helm-home, not in target/test-classes.
  • Go: both packages pass, except the envtest tests, which need ~/.local/share/kubebuilder-envtest. No repositories* file or helm-test-* directory is left in $TMPDIR.
  • Earlier on this branch: with HELM_REGISTRY_CONFIG, HELM_REPOSITORY_CACHE and HELM_REPOSITORY_CONFIG set to bogus paths, the tests passed and used the isolated homes.

Controls and mutations, each of which fails without the fix:

  • The old withoutChart fails once a large index is cached.
  • Without the exclusions, the bogus HELM_REPOSITORY_CACHE breaks HelmInstallTest.
  • Without the seed resources, after a clean, a fake docker-credential-secretservice on PATH receives every login and credsStore gets written. With them it is never called, after a clean or on a re-run.
  • The old Go tests leave repositories.lock and a helm-test-* directory behind.
  • Removing Shutdown fails both Go stop tests.
  • RemoteJarLoaderTest catches each of these:
    • dropping the ZIP check: a truncated chunked download is cached;
    • restoring the old delete-first with forceUpdate;
    • dropping the length check;
    • a 0 timeout constant;
    • ignoring the read timeout;
    • cleaning up the wrong file;
    • moving to the wrong target.

The Windows move fallback can't be triggered on Linux without root, so it is covered by review only.

manusa and others added 2 commits November 22, 2026 04:51
The Java and Go tests wrote registry logins, repository indexes and lock
files into the developer's own Helm config, cache and temp directory,
and the remote native JAR download could hang forever or cache a
truncated JAR.
- Java tests: Surefire points the Helm and Docker homes at
  target/helm-home, drops the HELM_* overrides, and seeds the registry
  configs so logins never reach an OS credential store
- Go tests: TestMain cleanup actually runs, repositories.yaml files live
  in t.TempDir() (test.RepositoryConfig), registry configs are seeded
- RepoServerStop/StopAll now shut the test OCI servers down
- RemoteJarLoader: connect/read timeouts, rejects downloads shorter
  than their Content-Length, downloads to a .part file and moves it
  into place
- Go repo-server fixtures without goroutines or sleeps, bounded HTTP
  clients in the fixture tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marc Nuri <marc@marcnuri.com>
Follow-ups from the review of the test isolation change:
- RemoteJarLoader opens the downloaded .part file as a ZIP before
  moving it into place, so a truncated chunked or close-delimited
  response (no Content-Length) is no longer cached as complete
- RemoteJarLoaderTest covers that, the cache-hit path, and keeping
  the cached JAR with forceUpdate when the download fails
- the excluded HELM_* variables (Surefire and the Go TestMain) are
  every one Helm reads that redirects files or changes behavior, and
  the two lists match
- the registry config seeds are test resources, so IDE runs on a
  clean target/ get them too (no extra Maven execution)
- SetupIsolatedEnv fails when it can't write the seeds

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marc Nuri <marc@marcnuri.com>

@manusa manusa left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of #464

Verdict: approve. It's posted as a comment only because this is my own PR. Two review rounds found no Critical or Important issues. Round 1's minors 1 to 5 and the description update are addressed in c4c06dc. The rest are follow-ups, listed below.

CI on 7970ee6

All three jobs passed:

  • Linux (Java 8): the Go native and internal/helm suites, envtest included. Java ran 166 tests with 0 failures, including:
    • NativeLibraryTest 4/4, with real downloads from Maven Central;
    • HelmKubernetesTest 55/55 on KinD with the isolated DOCKER_CONFIG.
  • Windows: 112 tests, 0 failures. NativeLibraryTest, plus the Registry, Push and Show tests with the seeded configs.
  • macOS: green.

CI runs again on c4c06dc, which changes RemoteJarLoader and the seed location.

Isolation

  • Helm paths: every Helm path the tests use resolves through helmpath/cli.New() in Helm v3.22.1. registry.NewClient defaults to helmpath.ConfigPath("registry/config.json"), so excluding HELM_REGISTRY_CONFIG is purely defensive.
  • Excluded variables: the 19 excluded HELM_* variables are complete and match between Surefire and env.go:
    • every variable in pkg/cli/environment.go;
    • HELM_DRIVER;
    • HELM_DRIVER_SQL_CONNECTION_STRING.
  • Credential-store seed: the placeholder auths entry is what makes oras-go v2.6.2's IsAuthConfigured skip native credential-store detection. To check it, a fake docker-credential-secretservice was put on PATH, after a clean:
    • with the seeds, it was never called;
    • without them, it was called on every login, and credsStore was written.
  • Seed location: the seeds now come from test resources, so IDE runs get them too. The large repository indexes stay in target/helm-home, outside test-classes.
  • OCI servers: Registry.Shutdown has no race with a server that hasn't started serving yet. Removing it fails both Go stop tests.
  • Go TestMain: the exit and cleanup ordering is correct. No os.CreateTemp("", "repositories.yaml") is left.

RemoteJarLoader

  • Timeouts, the Content-Length check, and the atomic .part + move behave correctly with redirects, chunked or unknown lengths, file: URLs, the cache-hit path, forceUpdate, and 32 concurrent downloads to the same target. It uses Java 8 APIs only, and the cached JAR's permissions are unchanged.

  • ZIP validation before the move, tested on Java 8 and 21:

    • every truncation length of a 6-entry JAR fails;
    • about 8,900 truncation points of the real 29.8 MB linux-amd64 JAR fail, including every byte of the last 8 KiB;
    • a zip64 JAR with 70,000 entries still opens.

    The release note was narrowed to "truncated or not a valid ZIP": entry data and CRCs aren't checked.

  • New tests cover the cache hit, forceUpdate keeping the old JAR when a download fails, and a truncated chunked response. Mutations of the ZIP check, the length check, delete-first, and the timeouts are each caught.

  • Windows move fallback: covered by review only.

Follow-ups (not blocking)

  • Security, not introduced here: the remote JAR is cached at a predictable path in a shared java.io.tmpdir and loaded on a cache hit without verification. A separate PR will add .sha1 verification and a per-user cache.
  • Error reporting: download failures surface to users only as "No NativeLibrary implementation found". The cause should be kept or logged.
  • Orphaned .part files: they're left behind when a JVM is killed mid-download.
  • Eclipse/m2e: it may not apply Surefire's environment variables, so Eclipse runs would use the real Helm config, as before this PR.

RemoteJarLoader swallowed every download and metadata failure, so a
timeout, an HTTP 404 or a truncated download all surfaced only as "No
NativeLibrary implementation found, please add one of the supported
dependencies to your project", with no hint of the cause.
- remoteJar() now throws the IOException (XML errors in the snapshot
  metadata are wrapped with its URL); it still returns null when the
  remote JAR is disabled or the platform is unknown
- NativeLibrary.getInstance() keeps its message and attaches that
  failure as the cause
- RemoteJarLoaderTest covers the release and snapshot lookups and the
  failures through a package-private remoteJar(...) seam

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marc Nuri <marc@marcnuri.com>
@manusa
manusa merged commit f5cbd6f into main Nov 22, 2026
3 checks passed
@manusa
manusa deleted the fix/test-isolation branch November 22, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant