tree a52cd2f4626e1ee0247a0d0226718288d813b88c
parent 34ee0d8414b2f919a5f40e4e4b1ab4cfd033b696
author Niranjan Jayakar <nija@databricks.com> 1715243011 +0900
committer Hyukjin Kwon <gurwls223@apache.org> 1715243011 +0900

[SPARK-47986][CONNECT][PYTHON] Unable to create a new session when the default session is closed by the server

### What changes were proposed in this pull request?

This is a follow-up to a previous improvement - 7d04d0f0.

In some cases, particularly when running older versions of the Spark
cluster (3.5), the error actually manifests as a mismatch in the
observed server-side session id between calls.

With this fix, we also capture this case and ensure that this case is
also handled.

Further, we improve the implementation of `getActiveSession()`
and introduce a similar `getDefaultSession()` that accounts for
stopped sessions.
This ensures that all places where default or active session is used,
stopped sessions are considered neither default nor active.

### Why are the changes needed?

Explained above.

### Does this PR introduce _any_ user-facing change?

Previously, when client encounters a session mismatch, a user
cannot create a new session. With this change, a user can
call `getOrCreate()` on the SparkSession builder and create a
new session.

### How was this patch tested?

Attached unit tests.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #46435 from nija-at/session-expires-part2.

Lead-authored-by: Niranjan Jayakar <nija@databricks.com>
Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>