[SPARK-49090][CORE] Support `JWSFilter`
### What changes were proposed in this pull request?
This PR aims to support `JWSFilter` which is a servlet filter that requires `JWS`, a cryptographically signed JSON Web Token, in the header via `spark.ui.filters` configuration.
- spark.ui.filters=org.apache.spark.ui.JWSFilter
- spark.org.apache.spark.ui.JWSFilter.param.key=YOUR-BASE64URL-ENCODED-KEY
To simply put, `JWSFilter` will check the following for all requests.
- The HTTP request should have `Authorization: Bearer <jws>` header.
- `<jws>` is a string with three fields, `<header>.<payload>.<signature>`.
- `<header>` is supposed to be a base64url-encoded string of `{"alg":"HS256","typ":"JWT"}`.
- `<payload>` is a base64url-encoded string of fully-user-defined content.
- `<signature>` is a signature based on `<header>.<payload>` and a user-provided key parameter.
For example, the value of `<header>` will be `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9` always and the value of `payload` can be `e30` if the payload is empty, `{}`. The `<signature>` part is changed by the shared value of `spark.org.apache.spark.ui.JWSFilter.param.key` between the server and client.
```
jshell> java.util.Base64.getUrlEncoder().encodeToString("{\"alg\":\"HS256\",\"typ\":\"JWT\"}".getBytes())
$2 ==> "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9"
jshell> java.util.Base64.getUrlEncoder().encodeToString("{}".getBytes())
$3 ==> "e30="
```
### Why are the changes needed?
To provide a little better security on WebUI consistently including Spark Standalone Clusters.
For example,
**SETTING**
```
$ jshell
| Welcome to JShell -- Version 17.0.12
| For an introduction type: /help intro
jshell> java.util.Base64.getUrlEncoder().encodeToString("Visit https://spark.apache.org to download Apache Spark.".getBytes())
$1 ==> "VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4="
```
```
$ cat conf/spark-defaults.conf
spark.ui.filters org.apache.spark.ui.JWSFilter
spark.org.apache.spark.ui.JWSFilter.param.key VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4=
```
**SPARK-SHELL**
```
$ build/sbt package
$ cp jjwt-impl-0.12.6.jar assembly/target/scala-2.13/jars
$ cp jjwt-jackson-0.12.6.jar assembly/target/scala-2.13/jars
$ bin/spark-shell
```
Without JWS (ErrorCode: 403 Forbidden)
```
$ curl -v http://localhost:4040/
* Host localhost:4040 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:4040...
* connect to ::1 port 4040 from ::1 port 61313 failed: Connection refused
* Trying 127.0.0.1:4040...
* Connected to localhost (127.0.0.1) port 4040
> GET / HTTP/1.1
> Host: localhost:4040
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 403 Forbidden
< Date: Fri, 02 Aug 2024 01:27:23 GMT
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=iso-8859-1
< Content-Length: 472
<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 403 Authorization header is missing.</title>
</head>
<body><h2>HTTP ERROR 403 Authorization header is missing.</h2>
<table>
<tr><th>URI:</th><td>/</td></tr>
<tr><th>STATUS:</th><td>403</td></tr>
<tr><th>MESSAGE:</th><td>Authorization header is missing.</td></tr>
<tr><th>SERVLET:</th><td>org.apache.spark.ui.JettyUtils$$anon$2-3b39bee2</td></tr>
</table>
</body>
</html>
* Connection #0 to host localhost left intact
```
With JWS,
```
$ curl -v -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e30.4EKWlOkobpaAPR0J4BE0cPQ-ZD1tRQKLZp1vtE7upPw" http://localhost:4040/
* Host localhost:4040 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:4040...
* connect to ::1 port 4040 from ::1 port 61311 failed: Connection refused
* Trying 127.0.0.1:4040...
* Connected to localhost (127.0.0.1) port 4040
> GET / HTTP/1.1
> Host: localhost:4040
> User-Agent: curl/8.7.1
> Accept: */*
> Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e30.4EKWlOkobpaAPR0J4BE0cPQ-ZD1tRQKLZp1vtE7upPw
>
* Request completely sent off
< HTTP/1.1 302 Found
< Date: Fri, 02 Aug 2024 01:27:01 GMT
< Cache-Control: no-cache, no-store, must-revalidate
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Location: http://localhost:4040/jobs/
< Content-Length: 0
<
* Connection #0 to host localhost left intact
```
**SPARK MASTER**
Without JWS (ErrorCode: 403 Forbidden)
```
$ curl -v http://localhost:8080/json/
* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:8080...
* connect to ::1 port 8080 from ::1 port 61331 failed: Connection refused
* Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080
> GET /json/ HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 403 Forbidden
< Date: Fri, 02 Aug 2024 01:34:03 GMT
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=iso-8859-1
< Content-Length: 477
<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 403 Authorization header is missing.</title>
</head>
<body><h2>HTTP ERROR 403 Authorization header is missing.</h2>
<table>
<tr><th>URI:</th><td>/json/</td></tr>
<tr><th>STATUS:</th><td>403</td></tr>
<tr><th>MESSAGE:</th><td>Authorization header is missing.</td></tr>
<tr><th>SERVLET:</th><td>org.apache.spark.ui.JettyUtils$$anon$1-6c52101f</td></tr>
</table>
</body>
</html>
* Connection #0 to host localhost left intact
```
With JWS
```
$ curl -v -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e30.4EKWlOkobpaAPR0J4BE0cPQ-ZD1tRQKLZp1vtE7upPw" http://localhost:8080/json/
* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:8080...
* connect to ::1 port 8080 from ::1 port 61329 failed: Connection refused
* Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080
> GET /json/ HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.7.1
> Accept: */*
> Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e30.4EKWlOkobpaAPR0J4BE0cPQ-ZD1tRQKLZp1vtE7upPw
>
* Request completely sent off
< HTTP/1.1 200 OK
< Date: Fri, 02 Aug 2024 01:33:10 GMT
< Cache-Control: no-cache, no-store, must-revalidate
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Content-Type: text/json;charset=utf-8
< Vary: Accept-Encoding
< Content-Length: 320
<
{
"url" : "spark://M3-Max.local:7077",
"workers" : [ ],
"aliveworkers" : 0,
"cores" : 0,
"coresused" : 0,
"memory" : 0,
"memoryused" : 0,
"resources" : [ ],
"resourcesused" : [ ],
"activeapps" : [ ],
"completedapps" : [ ],
"activedrivers" : [ ],
"completeddrivers" : [ ],
"status" : "ALIVE"
* Connection #0 to host localhost left intact
}%
```
### Does this PR introduce _any_ user-facing change?
No, this is a new filter.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47575 from dongjoon-hyun/SPARK-49090.
Lead-authored-by: Dongjoon Hyun <dhyun@apple.com>
Co-authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
diff --git a/LICENSE-binary b/LICENSE-binary
index b697179..6a8dc8c 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -267,6 +267,7 @@
io.fabric8:kubernetes-model-storageclass
io.fabric8:zjsonpatch
io.github.java-diff-utils:java-diff-utils
+io.jsonwebtoken:jjwt-api
io.netty:netty-all
io.netty:netty-buffer
io.netty:netty-codec
diff --git a/core/pom.xml b/core/pom.xml
index adb1b30..ff7fa04 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -118,6 +118,23 @@
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</dependency>
+ <dependency>
+ <groupId>io.jsonwebtoken</groupId>
+ <artifactId>jjwt-api</artifactId>
+ <version>0.12.6</version>
+ </dependency>
+ <dependency>
+ <groupId>io.jsonwebtoken</groupId>
+ <artifactId>jjwt-impl</artifactId>
+ <version>0.12.6</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>io.jsonwebtoken</groupId>
+ <artifactId>jjwt-jackson</artifactId>
+ <version>0.12.6</version>
+ <scope>test</scope>
+ </dependency>
<!-- Jetty dependencies promoted to compile here so they are shaded
and inlined into spark-core jar -->
diff --git a/core/src/main/scala/org/apache/spark/ui/JWSFilter.scala b/core/src/main/scala/org/apache/spark/ui/JWSFilter.scala
new file mode 100644
index 0000000..e942bce
--- /dev/null
+++ b/core/src/main/scala/org/apache/spark/ui/JWSFilter.scala
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import javax.crypto.SecretKey
+
+import io.jsonwebtoken.{JwtException, Jwts}
+import io.jsonwebtoken.io.Decoders
+import io.jsonwebtoken.security.Keys
+import jakarta.servlet.{Filter, FilterChain, FilterConfig, ServletRequest, ServletResponse}
+import jakarta.servlet.http.{HttpServletRequest, HttpServletResponse}
+
+/**
+ * A servlet filter that requires JWS, a cryptographically signed JSON Web Token, in the header.
+ *
+ * Like the other UI filters, the following configurations are required to use this filter.
+ * {{{
+ * - spark.ui.filters=org.apache.spark.ui.JWSFilter
+ * - spark.org.apache.spark.ui.JWSFilter.param.key=BASE64URL-ENCODED-YOUR-PROVIDED-KEY
+ * }}}
+ * The HTTP request should have {@code Authorization: Bearer <jws>} header.
+ * {{{
+ * - <jws> is a string with three fields, '<header>.<payload>.<signature>'.
+ * - <header> is supposed to be a base64url-encoded string of '{"alg":"HS256","typ":"JWT"}'.
+ * - <payload> is a base64url-encoded string of fully-user-defined content.
+ * - <signature> is a signature based on '<header>.<payload>' and a user-provided key parameter.
+ * }}}
+ */
+private class JWSFilter extends Filter {
+ private val AUTHORIZATION = "Authorization"
+
+ private var key: SecretKey = null
+
+ /**
+ * Load and validate the configurtions:
+ * - IllegalArgumentException will happen if the user didn't provide this argument
+ * - WeakKeyException will happen if the user-provided value is insufficient
+ */
+ override def init(config: FilterConfig): Unit = {
+ key = Keys.hmacShaKeyFor(Decoders.BASE64URL.decode(config.getInitParameter("key")));
+ }
+
+ override def doFilter(req: ServletRequest, res: ServletResponse, chain: FilterChain): Unit = {
+ val hreq = req.asInstanceOf[HttpServletRequest]
+ val hres = res.asInstanceOf[HttpServletResponse]
+ hres.setHeader("Cache-Control", "no-cache, no-store, must-revalidate")
+
+ try {
+ val header = hreq.getHeader(AUTHORIZATION)
+ header match {
+ case null =>
+ hres.sendError(HttpServletResponse.SC_FORBIDDEN, s"${AUTHORIZATION} header is missing.")
+ case s"Bearer $token" =>
+ val claims = Jwts.parser().verifyWith(key).build().parseSignedClaims(token)
+ chain.doFilter(req, res)
+ case _ =>
+ hres.sendError(HttpServletResponse.SC_FORBIDDEN, s"Malformed ${AUTHORIZATION} header.")
+ }
+ } catch {
+ case e: JwtException =>
+ // We intentionally don't expose the detail of JwtException here
+ hres.sendError(HttpServletResponse.SC_FORBIDDEN, "JWT Validate Fail")
+ }
+ }
+}
diff --git a/core/src/test/scala/org/apache/spark/ui/JWSFilterSuite.scala b/core/src/test/scala/org/apache/spark/ui/JWSFilterSuite.scala
new file mode 100644
index 0000000..a094a5e
--- /dev/null
+++ b/core/src/test/scala/org/apache/spark/ui/JWSFilterSuite.scala
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import java.util.{Base64, HashMap => JHashMap}
+
+import scala.jdk.CollectionConverters._
+
+import jakarta.servlet.{FilterChain, FilterConfig, ServletContext}
+import jakarta.servlet.http.{HttpServletRequest, HttpServletResponse}
+import org.mockito.ArgumentMatchers.{any, eq => meq}
+import org.mockito.Mockito.{mock, times, verify, when}
+
+import org.apache.spark._
+
+class JWSFilterSuite extends SparkFunSuite {
+ // {"alg":"HS256","typ":"JWT"} => eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9, {} => e30
+ private val TOKEN =
+ "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e30.4EKWlOkobpaAPR0J4BE0cPQ-ZD1tRQKLZp1vtE7upPw"
+
+ private val TEST_KEY = Base64.getUrlEncoder.encodeToString(
+ "Visit https://spark.apache.org to download Apache Spark.".getBytes())
+
+ test("Should fail when a parameter is missing") {
+ val filter = new JWSFilter()
+ val params = new JHashMap[String, String]
+ val m = intercept[IllegalArgumentException] {
+ filter.init(new DummyFilterConfig(params))
+ }.getMessage()
+ assert(m.contains("Decode argument cannot be null"))
+ }
+
+ test("Succeed to initialize") {
+ val filter = new JWSFilter()
+ val params = new JHashMap[String, String]
+ params.put("key", TEST_KEY)
+ filter.init(new DummyFilterConfig(params))
+ }
+
+ test("Should response with SC_FORBIDDEN when it cannot verify JWS") {
+ val req = mockRequest()
+ val res = mock(classOf[HttpServletResponse])
+ val chain = mock(classOf[FilterChain])
+
+ val filter = new JWSFilter()
+ val params = new JHashMap[String, String]
+ params.put("key", TEST_KEY)
+ val conf = new DummyFilterConfig(params)
+ filter.init(conf)
+
+ // 'Authorization' header is missing
+ filter.doFilter(req, res, chain)
+ verify(res).sendError(meq(HttpServletResponse.SC_FORBIDDEN),
+ meq("Authorization header is missing."))
+ verify(chain, times(0)).doFilter(any(), any())
+
+ // The value of Authorization field is not 'Bearer <token>' style.
+ when(req.getHeader("Authorization")).thenReturn("Invalid")
+ filter.doFilter(req, res, chain)
+ verify(res).sendError(meq(HttpServletResponse.SC_FORBIDDEN),
+ meq("Malformed Authorization header."))
+ verify(chain, times(0)).doFilter(any(), any())
+ }
+
+ test("Should succeed on valid JWS") {
+ val req = mockRequest()
+ val res = mock(classOf[HttpServletResponse])
+ val chain = mock(classOf[FilterChain])
+
+ val filter = new JWSFilter()
+ val params = new JHashMap[String, String]
+ params.put("key", TEST_KEY)
+ val conf = new DummyFilterConfig(params)
+ filter.init(conf)
+
+ when(req.getHeader("Authorization")).thenReturn(s"Bearer $TOKEN")
+ filter.doFilter(req, res, chain)
+ verify(chain, times(1)).doFilter(any(), any())
+ }
+
+ private def mockRequest(params: Map[String, Array[String]] = Map()): HttpServletRequest = {
+ val req = mock(classOf[HttpServletRequest])
+ when(req.getParameterMap()).thenReturn(params.asJava)
+ req
+ }
+
+ class DummyFilterConfig (val map: java.util.Map[String, String]) extends FilterConfig {
+ override def getFilterName: String = "dummy"
+
+ override def getInitParameter(arg0: String): String = map.get(arg0)
+
+ override def getInitParameterNames: java.util.Enumeration[String] =
+ java.util.Collections.enumeration(map.keySet)
+
+ override def getServletContext: ServletContext = null
+ }
+}
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 53e2086..492733b 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -138,6 +138,7 @@
jettison/1.5.4//jettison-1.5.4.jar
jetty-util-ajax/11.0.21//jetty-util-ajax-11.0.21.jar
jetty-util/11.0.21//jetty-util-11.0.21.jar
+jjwt-api/0.12.6//jjwt-api-0.12.6.jar
jline/2.14.6//jline-2.14.6.jar
jline/3.25.1//jline-3.25.1.jar
jna/5.14.0//jna-5.14.0.jar
