Repository navigation
fix(deploy): Run the service on a dedicated-core VM - #74
Conversation
The e2-small VM sustains only 0.5 vCPU. On 2026-11-21, when about 35 getsentry/junior eval runs used Roach at the same time, request p50 went from about 70ms to about 500ms, and integration evals timed out.
|
What's this change from a cost profile |
|
The VM cost goes up about 4x, from about $12 to about $49 a month. These are on-demand list prices for us-east1 at 730 hours a month:
So the change adds about $37 a month. E2 types get no sustained-use discount. A 1-year commitment would bring There's a cheaper option, Prices are from https://gcloud-compute.com/e2-standard-2.html and https://gcloud-compute.com/e2-small.html, which mirror the GCP price list. |
Change the default
machine_typefrome2-smalltoe2-standard-2.e2-smallis a shared-core type. It sustains 0.5 vCPU and can go above that only while it has burst credit. The proxy uses CPU for each request (TLS, JSON key, redaction), so its throughput stops at the shared-core limit.On 2026-11-21, from about 22:45 UTC, about 35 getsentry/junior eval runs used Roach in each 15-minute window. Before that, it was 5 to 15. The
sentry/roachspans show the service, not the upstreams:roach.request, replayedroach.request, passthroughroach.store.readThe total request rate stayed at about 14k per 15 minutes while latency went up about 10 times: the service was saturated. Replays (GCS) and passthrough calls (api.vercel.com) slowed at the same time.
roach.keyis synchronous CPU work with no I/O, and its p95 went from 17ms to 200ms. That points to CPU starvation on the VM, not to GCS or an upstream. The getsentry/junior integration evals went from about 4 minutes of test time per shard to about 26 minutes and hit the 30-minute job limit. Their Roach hit rate did not change (532 of 535 model requests replayed).e2-standard-2has 2 dedicated vCPUs, so the single Node process gets a full core and GC gets the other one. It uses the same boot disk type. One Node process is still limited to one core for JavaScript. If a load spike saturates it again, the next step is to run more than one process.To deploy, run
terraform apply -var-file=roach.tfvarsindeploy/gcp/. The Deploy workflow does not apply Terraform.allow_stopping_for_updateis set, so the apply stops the VM for about a minute to resize it, and eval runs in progress at that time fail.via @dcramer.
--
View Session [Sentry]