)]}'
{
  "commit": "f17fbaa6c0544eb1c7b2069d6a8ef39480a9effa",
  "tree": "34634b630fda16437edd62574339c73f48cd93be",
  "parents": [
    "595d9f3b0657705528e06fa8c64776aaabce4afe"
  ],
  "author": {
    "name": "Liang-Chi Hsieh",
    "email": "viirya@gmail.com",
    "time": "Fri Sep 04 00:33:40 2026 -0700"
  },
  "committer": {
    "name": "Liang-Chi Hsieh",
    "email": "liangchi.hsieh@databricks.com",
    "time": "Fri Sep 04 00:33:40 2026 -0700"
  },
  "message": "[SPARK-59224] Add an end-to-end workflow running the gateway on kind with a real PySpark client\n\n### What changes were proposed in this pull request?\n\nThe `deploy/examples/e2e-smoke` walkthrough exercises the whole deployment path —\ngateway image, Helm chart, Kubernetes Endpoints discovery, session affinity, audit\nand metrics — but only by hand. Nothing in CI covered it, so a break in the chart,\nthe Dockerfile or the K8s pool would only surface when someone next ran the\nwalkthrough manually.\n\nThis adds an `E2E` workflow that automates it:\n\n1. build the gateway image\n2. create a kind cluster, load the image into it\n3. deploy two `apache/spark:4.0.0` Spark Connect servers\n4. install the gateway with the Helm chart + the example values\n5. assert the gateway discovered **both** backends (K8s Endpoints watch)\n6. run the repo\u0027s own `test/integration/client_smoke.py` through a port-forward\n7. assert `scg_backend_pool_size`, the `ExecutePlan` counters, and that the audit\n   log recorded `ExecutePlan`\n\n`kind` and `helm` are installed with plain `curl` at pinned versions rather than\nthird-party actions — both to stay clear of the ASF GitHub Actions allowlist and to\nkeep the versions explicit. The only action used is `actions/checkout`. On failure\nthe job dumps pod state and gateway/Spark logs; the kind cluster is always deleted.\n\n**Cost on a hosted runner — measured, not estimated:** **6m22s**, broken down as\n`docker build` 208s (55%), PySpark install + client 57s, `kind create cluster` 41s,\nSpark backends ready (~700 MiB image pull) 43s, Helm install + gateway ready 11s,\ntool install and assertions ~14s. It runs in parallel with the existing `build`\njob (2m19s), so it adds roughly four minutes to a PR\u0027s wall clock in exchange for\ncovering the whole deployment path. Most of that is the release build, so there is\nheadroom later if it needs trimming (caching the Rust build layers).\n\nThe PySpark client is pinned to `4.0.0` to match the `apache/spark:4.0.0` backend:\npyspark 4.2\u0027s `createDataFrame` reads the SQL config\n`spark.sql.session.localRelationSizeLimit`, which Spark 4.0 does not have, and the\nserver answers `SQL_CONF_NOT_FOUND`.\n\n### Why are the changes needed?\n\nThis is the only coverage of the real deployment path: the Helm chart, the\nKubernetes Endpoints-watch pool, and session affinity as observed by an actual\nSpark client. The in-process integration tests cover the proxy logic, but nothing\nelse verifies that the chart deploys a working gateway.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nRan the entire walkthrough locally before writing the workflow. The PySpark client\nreturned correct results including a TempView query — the meaningful check that\nsession affinity held, since a TempView lives in one driver\u0027s memory and a\nmisrouted follow-up RPC would fail it. `scg_backend_pool_size 2`, the RPC counters\nand the audit records were all as expected. This PR\u0027s own run exercises the\nworkflow on a hosted runner.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nYes, co-authored with Claude Code.\n\nCloses #20 from viirya/ci-04-e2e-nightly.\n\nAuthored-by: Liang-Chi Hsieh \u003cviirya@gmail.com\u003e\nSigned-off-by: Liang-Chi Hsieh \u003cliangchi.hsieh@databricks.com\u003e\n",
  "tree_diff": [
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "ca974bbe887a487f0647a97f5840fad3c030e5e5",
      "new_mode": 33188,
      "new_path": ".github/workflows/e2e.yml"
    }
  ]
}