)]}' { "log": [ { "commit": "228b889750caf12529ea90fab1ca98f075fcc8d3", "tree": "6f0af2de365540f89aa30628315d7724bd2496b8", "parents": [ "1aa25ed384064da37942c78a421e7dec189bd6f4" ], "author": { "name": "nileshrathi345", "email": "113672051+nileshrathi345@users.noreply.github.com", "time": "Fri Jul 10 00:51:12 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 09 15:21:12 2026 -0400" }, "message": "[LIVY-1059][FOLLOWUP] Fix missing validation condition and add unit test\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up fix for [LIVY-1059](https://github.com/apache/livy/pull/529).\n\nWhen a client sends `\"queue\": \"\"` in the session create JSON, Livy was treating it as a valid queue value instead of falling back to the server default `livy.spark.yarn.queue`.\n\nThis change treats an empty queue string the same as a missing queue, so the default YARN queue from Livy config is used when the client passes \"\".\n\nUpdated in both batch and interactive session creation:\n`request.queue.filterNot(_.isEmpty).orElse(livyConf.sparkYarnQueue())`\n\n## How was this patch tested?\n\n- Added unit test in `BatchSessionSpec` for `queue \u003d Some(\"\")` and verified the default queue is passed to spark-submit\n- Added unit test in `InteractiveSessionSpec` for `queue \u003d Some(\"\")` and verified the session uses the default queue from `LivyConf`\n- Existing tests for `queue \u003d None` and user-provided queue values still pass" }, { "commit": "1aa25ed384064da37942c78a421e7dec189bd6f4", "tree": "cd2c3506c6bba05aec44bd9c986c0364cbddf79b", "parents": [ "e5accb9830627959b829c3025d3dc907d3e70e1b" ], "author": { "name": "nileshrathi345", "email": "113672051+nileshrathi345@users.noreply.github.com", "time": "Tue Jul 07 03:30:21 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 06 18:00:21 2026 -0400" }, "message": "[LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue\n\n## What changes were proposed in this pull request?\n\nThis PR introduces support for configuring a default YARN queue at the Livy server level via a new configuration property: `livy.spark.yarn.queue`.\n\nPreviously, if a client did not explicitly specify a YARN queue parameter (`queue`) in their session creation payload, the session would fall back to the global Hadoop cluster\u0027s default YARN queue. This change allows cluster administrators to isolate Livy-generated workloads into a specific default queue without requiring end-users to pass it manually in every API request.\n\nWhen a client provides `queue` in the session creation request, that value takes priority over the server default.\n\n## How was this patch tested?\n\n- Added unit tests in `BatchSessionSpec` that verify `--queue` is passed to spark-submit when using the LivyConf default or a user-provided queue\n- Added unit tests in `InteractiveSessionSpec` that verify the resolved queue is stored on the session\n- Manual API validation against a YARN cluster" }, { "commit": "e5accb9830627959b829c3025d3dc907d3e70e1b", "tree": "aff7f5634af59344f53149025a19a411b6bd2290", "parents": [ "320df286d117184f489d1e728a48b9d31eced486" ], "author": { "name": "Gabor Roczei", "email": "roczei@cloudera.com", "time": "Wed Jun 17 18:19:12 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jun 17 12:19:12 2026 -0400" }, "message": "[LIVY-1040] Migrate logging framework from Reload4j to Log4j2\n\n## What changes were proposed in this pull request?\n\nReplaced the reload4j dependency with the Log4j2 library across all modules.\n\nCo-authored by Peter Toth and Gyorgy Gal:\n\n- Added log4j-api, log4j-core, log4j-slf4j-impl and log4j-1.2-api (backward-compat bridge) to dependencies\n- Removed ch.qos.reload4j:reload4j from all module poms\n- Migrated all log4j.properties test config files to log4j2.properties using the new Log4j2 property-file syntax (rootLogger, appender.* keys)\n\n## How was this patch tested?\n\nIt has been manually tested with the existing unit tests.\n\n## Was this patch authored or co-authored using generative AI tooling?\n\nNo\n\nCo-authored-by: Peter Toth \u003c7253827+peter-toth@users.noreply.github.com\u003e\nCo-authored-by: Gyorgy Gal \u003c27883675+gyogal@users.noreply.github.com\u003e" }, { "commit": "320df286d117184f489d1e728a48b9d31eced486", "tree": "e3ef07306d9261ee840e37fc03cfd661448856e0", "parents": [ "ce48d4f3d13c45f0eef4ee09a71530167896b6d0" ], "author": { "name": "Gabor Roczei", "email": "roczei@cloudera.com", "time": "Wed Jun 17 17:41:13 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jun 17 11:41:13 2026 -0400" }, "message": "[LIVY-1049] Remove the \u0027-incubating\u0027 designator\n\n## What changes were proposed in this pull request?\n\nAs Apache Livy has graduated to a Top-Level Project (TLP), this commit removes the \u0027-incubating\u0027 suffix from all files, build configurations and documentation.\n\nSee announcement: https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-new-top-level-project-4\n\n## How was this patch tested?\n\nManually verified\n\n## Was this patch authored or co-authored using generative AI tooling?\n\nNo" }, { "commit": "ce48d4f3d13c45f0eef4ee09a71530167896b6d0", "tree": "373ce4e5e9fc58866ae412591f35fbd1998061a3", "parents": [ "4457bdc15e2403ab2f6ab429b9af509c9a48d0ef" ], "author": { "name": "Gabor Roczei", "email": "roczei@cloudera.com", "time": "Wed Jun 10 21:03:02 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jun 11 00:33:02 2026 +0530" }, "message": "[LIVY-1042] Add SSL/TLS support for ZooKeeper connection\n\n* [LIVY-1042] Add SSL/TLS support for ZooKeeper connection\n\nIntroduced an optional TLS encryption for the Curator based ZooKeeper\nclient used during session recovery.\n\nChanges by Asif Khatri:\n\n- Added five LivyConf entries (LIVY_ZK_CLIENT_SECURE, LIVY_ZK_CLIENT_SOCKET,\n LIVY_ZK_KEYSTORE_PASS, LIVY_ZK_TRUSTSTORE_FILE, LIVY_ZK_TRUSTSTORE_PASS)\n and document them in livy.conf.template.\n- Added ZooKeeperManager.createZKClientConfig that assembles a ZKClientConfig\n from keystore/truststore settings; applied to the CuratorFramework builder\n only when livy.server.zk.client.secure\u003dtrue.\n\nTest improvements:\n\n- Extracted mockCurator() helper in ZooKeeperStateStoreSpec to eliminate\n repeated CuratorFramework mock setup boilerplate.\n- Added an \"SSL config\" describe block with withSslMock\n helper (SslTestFixture) and tests covering SSL property mapping and\n both the secure and non-secure construction paths.\n\nWas this patch authored or co-authored using generative AI tooling?\n\nYes, this was co-authored using Cursor to help generate the new test cases.\n\nGenerated-by: Cursor 3.5.17\n\nCo-authored-by: Asif Khatri \u003c123077165+askhatri@users.noreply.github.com\u003e\n\n* Implemented Arnav\u0027s suggestion\n\nWhen livy.server.zk.client.secure\u003dtrue, the following are required: livy.keystore,\nlivy.server.zk.ssl.keyStore.password livy.server.zk.ssl.truststore.location, livy.server.zk.ssl.truststore.password.\n\n---------\n\nCo-authored-by: Asif Khatri \u003c123077165+askhatri@users.noreply.github.com\u003e" }, { "commit": "4457bdc15e2403ab2f6ab429b9af509c9a48d0ef", "tree": "d34ef8253d8a5a6bd1502e5ff811521082fb8a26", "parents": [ "d0e846e7b8625506e0aab72349ef1b6f5998122c" ], "author": { "name": "Arnav Balyan", "email": "60175178+ArnavBalyan@users.noreply.github.com", "time": "Tue Jun 09 21:53:59 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jun 09 12:23:59 2026 -0400" }, "message": "[LIVY-1056] Update Livy readme to reflect post graduation status\n\n## What changes were proposed in this pull request?\n\n - Livy has graduated, the repo has now been formally renamed\n - Updating the readme to point to updated links.\n\n## How was this patch tested?\n\n - CI" }, { "commit": "d0e846e7b8625506e0aab72349ef1b6f5998122c", "tree": "8f7f92ef07bc72963c6a85a809b48c916804e96b", "parents": [ "45e96b669ed98d2930664b57af164cc2dda8990b" ], "author": { "name": "Gabor Roczei", "email": "roczei@cloudera.com", "time": "Mon Jun 08 10:07:53 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 08 13:37:53 2026 +0530" }, "message": "[LIVY-1050] Remove the DISCLAIMER file from the Livy repo\n\nRemove the DISCLAIMER file as Apache Livy has graduated to a Top-Level Project (TLP).\nhttps://news.apache.org/foundation/entry/the-apache-software-foundation-announces-new-top-level-project-4" }, { "commit": "45e96b669ed98d2930664b57af164cc2dda8990b", "tree": "0a1192791e36c1f412866edf7d1ae5892ef2643d", "parents": [ "fc4202ea31722979566f7f730f141420828e4d6b" ], "author": { "name": "Gabor Roczei", "email": "roczei@cloudera.com", "time": "Fri May 29 16:11:02 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 29 16:11:02 2026 +0200" }, "message": "[LIVY-1047] Update PR template to align with ASF Generative Tooling Guidance (#524)\n\n## What changes were proposed in this pull request?\nThis adds a section to the PR template asking \"Was this patch authored or co-authored using generative AI tooling?\" to align with the ASF Generative Tooling Guidance: https://www.apache.org/legal/generative-tooling.html\n\nThis was created based on the Spark template with minor adjustments to ensure the formatting matches the previous two sections: https://github.com/apache/spark/blob/master/.github/PULL_REQUEST_TEMPLATE\n\n## How was this patch tested?\nI tried it out in my own git repository." }, { "commit": "fc4202ea31722979566f7f730f141420828e4d6b", "tree": "3b8eb171815b74d258a461a51f457fccd509d8ff", "parents": [ "717bf7478233fd0f92f55bfe18d6816723697e78" ], "author": { "name": "Arnav Balyan", "email": "60175178+ArnavBalyan@users.noreply.github.com", "time": "Tue May 19 20:00:13 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue May 19 10:30:13 2026 -0400" }, "message": "[LIVY-1046] Avoid decoding user info twice in HTTP client\n\n## What changes were proposed in this pull request?\n\n - The HTTP client decodes the user info in url twice. \n - Java `URI` returns percent decoded output which is passed to .decode() again. Any password sent with special characters is wrongly decoded.\n - Remove the second decode call and ensure `:` inside the password is not treated as a delimiter\n - Closes LIVY-1046\n\n## How was this patch tested?\n\n - UT" }, { "commit": "717bf7478233fd0f92f55bfe18d6816723697e78", "tree": "8741d3b2498b118c3981633577d5a75caa8225cb", "parents": [ "70dbf978cee856f086d28f4f777244ec8ea0c593" ], "author": { "name": "Arnav Balyan", "email": "60175178+ArnavBalyan@users.noreply.github.com", "time": "Mon Apr 27 19:24:32 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Apr 27 09:54:32 2026 -0400" }, "message": "[LIVY-1043] Track ZK connection transitions in Livy\n\n## What changes were proposed in this pull request?\n\n - Currently ZKManager does not observe connection transitions. When underlying connection changes, Livy is not aware of it, only ZK side logs show up.\n - Additionally register to ConnectionStateListener, which adds/listens to Livy owned logs for the connection changes.\n - Same events can be used for recovery in the future.\n\n## How was this patch tested?\n\n - Unit tests" }, { "commit": "70dbf978cee856f086d28f4f777244ec8ea0c593", "tree": "c21f903804dcf33f801603c6be6e14d0551cd4ae", "parents": [ "fd21e1dbd03636ec152780bb7caf1029eb2d6df4" ], "author": { "name": "Arnav Balyan", "email": "60175178+ArnavBalyan@users.noreply.github.com", "time": "Thu Apr 09 08:23:24 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Apr 09 11:23:24 2026 -0400" }, "message": "[LIVY-1039] Redact sensitive credentials/tokens from log output\n\n## What changes were proposed in this pull request?\n\n - Logs in several places are logging sensitive data.\n - Ensure log line does not display this in plain text, credentials should not be logged \n\n## How was this patch tested?\n\n - Existing UTs" }, { "commit": "fd21e1dbd03636ec152780bb7caf1029eb2d6df4", "tree": "ee413ce4fec0c7e9d3ac475ea5dcd582e51c8890", "parents": [ "0a32275c3717d7486f54c97ef6ecfb3be6b16c66" ], "author": { "name": "Arnav Balyan", "email": "60175178+ArnavBalyan@users.noreply.github.com", "time": "Thu Mar 26 22:34:35 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Mar 26 13:04:35 2026 -0400" }, "message": "[LIVY-1038] Provide configs for ZK SASL authentication\n\n## What changes were proposed in this pull request?\n\n - Allow configuring configs for ZK SASL authentication\n - Users can enable auth, and pass in custom login context\n - Closes #487 \n \n## How was this patch tested?\n\n - UT, setup a KDC, ZK and tested Livy auth to zk" }, { "commit": "0a32275c3717d7486f54c97ef6ecfb3be6b16c66", "tree": "4fa980a247454bec2e4ba54f11050389e1725337", "parents": [ "5da3da4e9704c7f2a9dbdfb6e50da4f13a99118e" ], "author": { "name": "Arnav Balyan", "email": "60175178+ArnavBalyan@users.noreply.github.com", "time": "Mon Mar 02 22:21:31 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Mar 02 11:51:31 2026 -0500" }, "message": "[LIVY-789] Ensure Spark data files are available in interactive session\n\n## What changes were proposed in this pull request?\n\n - Ensures the CWD for interactive sessions is set to Spark root directory for data files to be available. For interactive sessions this bug causes --files argument to not work since the shell does not have access to the staging directory where files are made available. \n - Deprecated unused Spark 1.x code.\n - Adds UTs to ensure files are available when set for interactive sessions.\n\n## How was this patch tested?\n\n - UT" }, { "commit": "5da3da4e9704c7f2a9dbdfb6e50da4f13a99118e", "tree": "5a8eff0da415f3ef06fb171d2961a0693d209220", "parents": [ "b284a868ed8c81a8af73801f605f8a8b6fbf4f49" ], "author": { "name": "György Gál", "email": "27883675+gyogal@users.noreply.github.com", "time": "Thu Feb 26 11:28:34 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Feb 26 11:28:34 2026 -0500" }, "message": "[LIVY-1037] Fix typo in add-exports causing warning on launch\n\n** What changes were proposed in this pull request?\n\nIn the \u0027livy-server\u0027 startup script there is a typo in the \u0027add-exports\u0027\nlist causing a warning to be logged on startup (\u0027ALL_UNNAMED\u0027 instead of\n\u0027ALL-UNNAMED\u0027).\n\n** How was this patch tested?\n\nTested by running `livy-server` locally." }, { "commit": "b284a868ed8c81a8af73801f605f8a8b6fbf4f49", "tree": "567ecfa81ee3c36358208cd275330281be799e2d", "parents": [ "b141e0aa8b9ff54d360254ab7ed4f8f3ef684207" ], "author": { "name": "Takanobu Asanuma", "email": "tasanuma@apache.org", "time": "Wed Feb 25 04:12:20 2026 +0900" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Feb 24 14:12:20 2026 -0500" }, "message": "[LIVY-1016] Improve exception handling for Spark YARN Job Startup Failures\n\n## What changes were proposed in this pull request?\n\nWhen jobs fail at startup, such as those specifying an undefined queue, are executed multiple times, the Livy Thrift Server gets stuck.\n\n**Cause of the Problem** \n\n- When a problematic query is submitted, the `getAppIdFromTag` method throws an `IllegalStateException(\"spark-submit start failed\")`.\n - https://github.com/apache/incubator-livy/blob/d6822aaa04df87fb17e7e061d39cbc05cedae5c3/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala#L196\n- Subsequently, a `yarn application kill` is executed in the `kill()` method, but the above `IllegalStateException` is not caught and slips through, resulting in `yarnAppMonitorThread.interrupt()` not being executed.\n - https://github.com/apache/incubator-livy/blob/d6822aaa04df87fb17e7e061d39cbc05cedae5c3/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala#L153-L160\n- If the `yarnAppMonitorThread` is not stopped, it seems that the Livy Thrift Server eventually hangs.\n\n**Proposed Fix:** Change the exception capture target to `NonFatal` to catch all exceptions.\n\n## How was this patch tested?\n\n- unit test (I added a new unit test in SparkYarnAppSpec.)\n" }, { "commit": "b141e0aa8b9ff54d360254ab7ed4f8f3ef684207", "tree": "621959b0b28e09548a2e3bec421ec7f1eb33ca07", "parents": [ "01523b89073523611000c6ac577d5d9f12e9084e" ], "author": { "name": "György Gál", "email": "27883675+gyogal@users.noreply.github.com", "time": "Sat Feb 21 18:59:06 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Feb 21 18:59:06 2026 -0500" }, "message": "[LIVY-1036] Run both unit tests and integration tests on new PRs\n\n## What changes were proposed in this pull request?\n\nCurrently only integration tests are run on new PRs. Unit tests could also be\nlaunched to ensure code quality.\n\n## How was this patch tested?\n\n- Tested on Github PR." }, { "commit": "01523b89073523611000c6ac577d5d9f12e9084e", "tree": "11514489a27f15d7b8c2330e5c7d447c6981b83d", "parents": [ "147fc5e920ab8367c6b0a1bb62343f1e55ba25f1" ], "author": { "name": "György Gál", "email": "27883675+gyogal@users.noreply.github.com", "time": "Sat Feb 21 18:54:56 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Feb 21 18:54:56 2026 -0500" }, "message": "[LIVY-1031][DOC] README build instructions are out-of-date (#510)\n\n## What changes were proposed in this pull request?\n\n * The project\u0027s `README.md` states that `spark3` and `spark-2.12` are\n default profiles, however this is not the case.\n * The profile `thriftserver` is added to `README.md`.\n\n## How was this patch tested?\n\nTested with unit test and integration test workflows." }, { "commit": "147fc5e920ab8367c6b0a1bb62343f1e55ba25f1", "tree": "e68d7a63a5b6a40ad7cf58bc4bf1a18f092ada67", "parents": [ "d432b4c7a990a7899491108fc911139ce61278c3" ], "author": { "name": "Gyorgy Gal", "email": "27883675+gyogal@users.noreply.github.com", "time": "Fri Jan 02 15:27:39 2026 -0500" }, "committer": { "name": "György Gál", "email": "27883675+gyogal@users.noreply.github.com", "time": "Sat Feb 21 18:52:59 2026 -0500" }, "message": "Update Copyrights in NOTICE and footer for 2026\n" }, { "commit": "d432b4c7a990a7899491108fc911139ce61278c3", "tree": "831b0f046aac627b04d11979e11b496ea5a42923", "parents": [ "e79ebe379af3cc2f1d4dce52f0b79b37c9e7939e" ], "author": { "name": "Gyorgy Gal", "email": "27883675+gyogal@users.noreply.github.com", "time": "Mon Dec 01 15:00:25 2025 -0500" }, "committer": { "name": "György Gál", "email": "27883675+gyogal@users.noreply.github.com", "time": "Sat Feb 21 18:46:46 2026 -0500" }, "message": "Update Copyrights in NOTICE and footer\n" }, { "commit": "e79ebe379af3cc2f1d4dce52f0b79b37c9e7939e", "tree": "3ea7ae01f4d23c76a6c0f26baf1baa83fedbac8e", "parents": [ "27c8603ca9252e5359902deffba867cb750083fa" ], "author": { "name": "Gyorgy Gal", "email": "27883675+gyogal@users.noreply.github.com", "time": "Mon Dec 01 14:59:49 2025 -0500" }, "committer": { "name": "György Gál", "email": "27883675+gyogal@users.noreply.github.com", "time": "Sat Feb 21 18:32:26 2026 -0500" }, "message": "Update directory configuration\n" }, { "commit": "27c8603ca9252e5359902deffba867cb750083fa", "tree": "1cc49beb330174930c25704bd74e30f33d6fecb9", "parents": [ "db99f620ca6d5adef59040e83971df191b9054e4" ], "author": { "name": "Gabor Roczei", "email": "roczei@gmail.com", "time": "Thu Nov 13 15:41:08 2025 -0500" }, "committer": { "name": "György Gál", "email": "27883675+gyogal@users.noreply.github.com", "time": "Sat Feb 21 09:53:32 2026 -0500" }, "message": "Update pre-defined run configuration\n" }, { "commit": "db99f620ca6d5adef59040e83971df191b9054e4", "tree": "bdbca36480fce2c127d0c7404179d70b84874ef2", "parents": [ "c97fd58eeaa8b52a26836afd9916e4bb6dd58640" ], "author": { "name": "Arnav Balyan", "email": "60175178+ArnavBalyan@users.noreply.github.com", "time": "Fri Feb 20 08:56:12 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Feb 19 22:26:12 2026 -0500" }, "message": "update (#511)\n\n" }, { "commit": "c97fd58eeaa8b52a26836afd9916e4bb6dd58640", "tree": "6a02c218923b807b67d89ee21b9ce6eec9e111e6", "parents": [ "674ac00ab28c4a259cb5e9b3ccf9afe706b9aad1" ], "author": { "name": "Edward Capriolo", "email": "edlinuxguru@gmail.com", "time": "Wed Jan 21 14:31:00 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jan 21 14:31:00 2026 -0500" }, "message": "LIVY-1032 include owasp plugin (#507) (edwardcapriolo via lmccay)\n\n" }, { "commit": "674ac00ab28c4a259cb5e9b3ccf9afe706b9aad1", "tree": "b8752d44efecb991dd164ae0f30e8a7a05ba12cd", "parents": [ "284467b98b8f086cb3cf9a6813873141bced5dc2" ], "author": { "name": "Edward Capriolo", "email": "edlinuxguru@gmail.com", "time": "Wed Jan 21 14:29:41 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jan 21 14:29:41 2026 -0500" }, "message": "LIVY-1030 cleanup of context and clients (#506) (edwardcapriolo via lmccay)\n\n" }, { "commit": "284467b98b8f086cb3cf9a6813873141bced5dc2", "tree": "27ce57ed08f262da838750945363263d5e617667", "parents": [ "22923a1f1ba8908c1bd16154b46bf4177172bab9" ], "author": { "name": "Edward Capriolo", "email": "edlinuxguru@gmail.com", "time": "Fri Jan 16 10:24:14 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jan 16 10:24:14 2026 -0500" }, "message": "LIVY-1029 modernize versions (#505)\n\n" }, { "commit": "22923a1f1ba8908c1bd16154b46bf4177172bab9", "tree": "4f2447fc932c4df64db5d163e7a7557868180df8", "parents": [ "26ccdebff41af59822ac0039a41c00c3ee530dda" ], "author": { "name": "Damon P. Cortesi", "email": "d.lifehacker@gmail.com", "time": "Fri Dec 12 08:48:57 2025 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Dec 12 11:48:57 2025 -0500" }, "message": "[LIVY-981] Exclude test and provided scopes from THIRD-PARTY license generation\n\n## What changes were proposed in this pull request?\n\n- Added `test,provided` to `excludedScopes` property for license maven plugin.\n- https://issues.apache.org/jira/browse/LIVY-981\n\n## How was this patch tested?\n\n- Generated licenses manually, identified test-scope license removed from generated file" }, { "commit": "26ccdebff41af59822ac0039a41c00c3ee530dda", "tree": "838c4203283a1177e4832fb18bef269587f5c87d", "parents": [ "4c2549ac92159f8e7ab886c9a836632ec5633281" ], "author": { "name": "György Gál", "email": "27883675+gyogal@users.noreply.github.com", "time": "Sat Dec 06 22:53:01 2025 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Dec 06 22:53:01 2025 -0500" }, "message": "[LIVY-1026] Fix Content-Types for nosniff header\n\n## What changes were proposed in this pull request?\n\nIn LIVY-785 the `X-Content-Type-Options: nosniff` HTTP header was added,\nhowever this breaks transferring some files where the Content-Type is\nnot correctly specified.\n\n## How was this patch tested?\n\nTested by running the server locally." }, { "commit": "4c2549ac92159f8e7ab886c9a836632ec5633281", "tree": "eed043453bfd52fa8da437461258e463aa24715c", "parents": [ "5b4e3480c51298d02242eb427717d97d91eae011" ], "author": { "name": "Arnav Balyan", "email": "60175178+ArnavBalyan@users.noreply.github.com", "time": "Fri Dec 05 01:53:23 2025 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Dec 04 15:23:23 2025 -0500" }, "message": "[LIVY-1025][TEST] Clean up Python2InterpreterSpec post Python deprecation\n\n## What changes were proposed in this pull request?\n\n * Cleans up Python2InterpreterSpec since python 2 is deprecated.\n * Also removed the Python2SessionSpec counterpart\n\n## How was this patch tested?\n\n * CI" }, { "commit": "5b4e3480c51298d02242eb427717d97d91eae011", "tree": "49416626a889cfb34dd7a5616ae704c239847e59", "parents": [ "49a36164784f00a652df3d86f4eaf9652fb50f13" ], "author": { "name": "Arnav Balyan", "email": "60175178+ArnavBalyan@users.noreply.github.com", "time": "Thu Dec 04 20:34:38 2025 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Dec 04 10:04:38 2025 -0500" }, "message": "[LIVY-1024] Upgrade Livy to Python3\n\n## What changes were proposed in this pull request?\n\n * Python2 was deprecated a while back, most systems/environments don\u0027t support it anymore\n * Bump Livy to Python3, remove Python2 support since Spark 2 has also now been deprecated\n * Closes LIVY-1024\n\n## How was this patch tested?\n\n * CI / Locally tested the python3 files manually" }, { "commit": "49a36164784f00a652df3d86f4eaf9652fb50f13", "tree": "db5d839d6c76d7d4e47e45038702dff633089cd9", "parents": [ "0efa2d7e508c6d955cebe48a5159ffc3202e57a1" ], "author": { "name": "Edward Capriolo", "email": "edlinuxguru@gmail.com", "time": "Wed Dec 03 15:10:16 2025 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Dec 03 15:10:16 2025 -0500" }, "message": "[LIVY-1023] Missing double quote causes server not to start\n\n## What changes were proposed in this pull request?\n\nbin/livy-server doesnt work at all.\n\n## How was this patch tested?\n\n```\nedward@fedora:~/incubator-livy$ bin/livy-server \nWARNING: Unknown module: ALL_UNNAMED specified to --add-exports\nlog4j:WARN No appenders could be found for logger (org.apache.livy.server.AccessManager).\nlog4j:WARN Please initialize the log4j system properly.\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\nException in thread \"main\" java.lang.IllegalArgumentException: Livy requires the SPARK_HOME environment variable\n\tat org.apache.livy.utils.LivySparkUtils$$anonfun$1.apply(LivySparkUtils.scala:55)\n\tat org.apache.livy.utils.LivySparkUtils$$anonfun$1.apply(LivySparkUtils.scala:55)\n\tat scala.Option.getOrElse(Option.scala:121)\n\tat org.apache.livy.utils.LivySparkUtils$.testSparkHome(LivySparkUtils.scala:54)\n\tat org.apache.livy.server.LivyServer.start(LivyServer.scala:79)\n\tat org.apache.livy.server.LivyServer$.main(LivyServer.scala:439)\n\tat org.apache.livy.server.LivyServer.main(LivyServer.scala)\nedward@fedora:~/incubator-livy$ git status\n```" }, { "commit": "0efa2d7e508c6d955cebe48a5159ffc3202e57a1", "tree": "9a6ba6720c6c3df35489401c360b0399d31c181b", "parents": [ "acb11e11ebaf36c5792e0b51d90421964c57c88d" ], "author": { "name": "Arnav Balyan", "email": "60175178+ArnavBalyan@users.noreply.github.com", "time": "Tue Dec 02 01:27:42 2025 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Dec 01 14:57:42 2025 -0500" }, "message": "[LIVY-1022][TEST] Update NM Disk utilization percentage for UT\n\n## What changes were proposed in this pull request?\n\n * Currently Livy UT for `JobApiIT.scala` fails due to disk being 92% which causes the node to be lost.\n * Updated this to 100% for the UT to ensure the tests pass even if the disk utilization is over 90%\n * Closes LIVY-1022\n\n## How was this patch tested?\n\n * UT fails on master without this change\n * This PR should allow UT to go through" }, { "commit": "acb11e11ebaf36c5792e0b51d90421964c57c88d", "tree": "2a6daa199175e47a0224be51221b1d0b97430e96", "parents": [ "c7d1c26435b72dc64620a632853837d2fa57d991" ], "author": { "name": "Arnav Balyan", "email": "60175178+ArnavBalyan@users.noreply.github.com", "time": "Fri Nov 14 07:11:09 2025 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Nov 13 20:41:09 2025 -0500" }, "message": "[LIVY-1019] Deprecate Spark 2 support\n\n## What changes were proposed in this pull request?\n\n * Spark 2.x is at end of life and no longer supported, newer Spark versions (Spark 4) are now available.\n * Deprecate Spark 2.x\n * Also offboard Scala 2.11 support\n\n## How was this patch tested?\n\n * CI and local builds are successful" }, { "commit": "c7d1c26435b72dc64620a632853837d2fa57d991", "tree": "87c6969c74ce04872f4f0a0e8e1179547ff9b24e", "parents": [ "47db26909ca5e681a8c53653683dd33c5240d079" ], "author": { "name": "György Gál", "email": "27883675+gyogal@users.noreply.github.com", "time": "Sun Nov 09 18:13:25 2025 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Nov 09 18:13:25 2025 -0500" }, "message": "Update main to 0.10.0-incubating-SNAPSHOT\n\n" }, { "commit": "47db26909ca5e681a8c53653683dd33c5240d079", "tree": "0db270f71e8269c5a89320078e44a9ea9780f3bd", "parents": [ "1fb8f67e68912f271a0b5f0756c5302646bcdc01" ], "author": { "name": "Damon P. Cortesi", "email": "d.lifehacker@gmail.com", "time": "Mon Nov 03 10:36:45 2025 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Nov 03 13:36:45 2025 -0500" }, "message": "[LIVY-985] Remove unused Cat X licenses\n\n## What changes were proposed in this pull request?\n\nExclude unnecessary Cat X licenses where possible.\n\n## How was this patch tested?\n\nBuilding project during release process." }, { "commit": "1fb8f67e68912f271a0b5f0756c5302646bcdc01", "tree": "c0a681b5d560a62ea661a3ed257c68b0295a1559", "parents": [ "f3e23ed0b34da9d44ec06c724d5d03c7cf15aebe" ], "author": { "name": "Damon P. Cortesi", "email": "d.lifehacker@gmail.com", "time": "Mon Oct 27 10:23:53 2025 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 27 13:23:53 2025 -0400" }, "message": "[LIVY-980] Update release script with modern commands\n\n## What changes were proposed in this pull request?\n\nDuring the previous release, several changes had to be made to the release script to get it to work. This PR includes those changes.\n\n## How was this patch tested?\n\nManually during the release process." }, { "commit": "f3e23ed0b34da9d44ec06c724d5d03c7cf15aebe", "tree": "d318ea5eea3a660fbef0344a766e3c8ab99be92c", "parents": [ "370a71561dcf60a860d3b97ef20810b95bf90980" ], "author": { "name": "György Gál", "email": "27883675+gyogal@users.noreply.github.com", "time": "Wed Oct 22 10:29:20 2025 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 22 10:29:20 2025 -0400" }, "message": "[LIVY-1017] Support Java 17, set up JDK 17 based tests\n\n## What changes were proposed in this pull request?\n\nEnable Livy to run with Java 17, set up unit test and integration test runs with JDK 17.\n\n## How was this patch tested?\n\nUnit and integration tests." }, { "commit": "370a71561dcf60a860d3b97ef20810b95bf90980", "tree": "3d0608caa65621afa8f5b2edde09ba9aebb4db4d", "parents": [ "710c0784dae73aa7c5a6ce0bac264bcfdefd7681" ], "author": { "name": "Arnav Balyan", "email": "60175178+ArnavBalyan@users.noreply.github.com", "time": "Tue Oct 07 20:00:17 2025 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 07 10:30:17 2025 -0400" }, "message": "[LIVY-1018] Extract field comments from Spark schema metadata and set in Datatypeutils\n\n## What changes were proposed in this pull request?\n\n * During schema fetch, ensure we can extract and persist the comments.\n * Column comments extracted from Spark metadata for complexToDataType and schemaFromSparkJson\n * Closes Extract comment metadata for complexToDataType and schemaFromSparkJson #483\n\n## How was this patch tested?\n\nCI" }, { "commit": "710c0784dae73aa7c5a6ce0bac264bcfdefd7681", "tree": "5544eceb0409d2dc0dcc8c3a2dcf5ea46bf79380", "parents": [ "c28f2fce58e1fc2e69b1c7470bd9c02d8cb196a6" ], "author": { "name": "György Gál", "email": "27883675+gyogal@users.noreply.github.com", "time": "Tue Aug 12 19:38:36 2025 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 12 19:38:36 2025 +0200" }, "message": "[LIVY-1010] Add support for Spark 3.5.6\n\n## What changes were proposed in this pull request?\n\nKeep Spark 3 support up to date with the latest Spark 3 release, upgrade dependencies.\n\n## How was this patch tested?\n\nUnit and integration tests." }, { "commit": "c28f2fce58e1fc2e69b1c7470bd9c02d8cb196a6", "tree": "9e490f59afc17d0d02ade0151b3da00354c47a2d", "parents": [ "adb10848e259fbd9687198b137a8e141abcdbc73" ], "author": { "name": "nileshrathi345", "email": "113672051+nileshrathi345@users.noreply.github.com", "time": "Mon Aug 11 17:52:41 2025 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 11 14:22:41 2025 +0200" }, "message": "[LIVY-1009] Add support for global TTL for sessions\n\n## What changes were proposed in this pull request?\n\nAdded two new Livy configuration parameters as below:\n\n```\nEnabled to check whether TTL Livy sessions should be stopped.\nlivy.server.session.ttl-check \u003d false\n\nTime in milliseconds on how long Livy will wait before TTL is an inactive session.\nNote that the inactive session could be busy running jobs.\nlivy.server.session.ttl \u003d 8h\n```\n\n## How was this patch tested?\n\nRead LIVY-1009 for details." }, { "commit": "adb10848e259fbd9687198b137a8e141abcdbc73", "tree": "5428d562d0b8cb850b70c1dffa411a69fe374339", "parents": [ "d6822aaa04df87fb17e7e061d39cbc05cedae5c3" ], "author": { "name": "Yi Wei", "email": "yiwei.in.cyber@gmail.com", "time": "Sat Jun 14 02:11:04 2025 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jun 13 20:11:04 2025 +0200" }, "message": "[MINOR][DOC] Fix the missing fields in the Batch model\n\n" }, { "commit": "d6822aaa04df87fb17e7e061d39cbc05cedae5c3", "tree": "21fbbdaec5293ce380d7b85706185ec89fb7768c", "parents": [ "df691033b11440837c2a06fe624510bdbf815fd5" ], "author": { "name": "idzikovsky", "email": "23353239+idzikovsky@users.noreply.github.com", "time": "Tue May 20 13:26:32 2025 +0300" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue May 20 12:26:32 2025 +0200" }, "message": "[MINOR][DOC] Fix cursor parameter documentation for completion endpoint\n\n" }, { "commit": "df691033b11440837c2a06fe624510bdbf815fd5", "tree": "98cb133f6994815b4e643f7508d02e3a20a4cbda", "parents": [ "f580e71b5805c0b2ed2b93005fce9bc6396354e7" ], "author": { "name": "Surendra Singh Lilhore", "email": "45159337+surendralilhore@users.noreply.github.com", "time": "Wed Apr 23 19:15:58 2025 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Apr 23 15:45:58 2025 +0200" }, "message": "[LIVY-1012] Use SslContextFactory.Server() instead of SslContextFactory\n\n## What changes were proposed in this pull request?\n\nUse SslContextFactory.Server() instead of SslContextFactory() when constructing the factory.\n\n## How was this patch tested?\n\nWith Jetty 9.4.50, we should call SslContextFactory.Server(), instead of SslContextFactory(), to create SslContextFactory. Otherwise we get the following error when using a KeyStore with multiple certificates in it.\n\n```\nException in thread \"main\" java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.\nSslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)\nat org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1289)\n```" }, { "commit": "f580e71b5805c0b2ed2b93005fce9bc6396354e7", "tree": "5b35bda2cc78d2f0fa329c9c7b58080107ad12e3", "parents": [ "826ec291fa83c26fd715b5f043629af2a48014cb" ], "author": { "name": "nileshrathi345", "email": "113672051+nileshrathi345@users.noreply.github.com", "time": "Mon Apr 14 19:25:39 2025 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Apr 14 15:55:39 2025 +0200" }, "message": "[LIVY-785] Enabled adding security related HTTP headers\n\n## What changes were proposed in this pull request?\n\nThis change introduces a new configuration option `livy.server.security-headers.enabled`.\nWhen this property is set to true, the following security headers are added to HTTP\nresponses by default:\n\n * X-XSS-Protection\n * X-Frame-Options\n * X-Content-Type-Options\n * Strict-Transport-Security\n * Content-Security-Policy\n\nAlso, adds content type information to all responses as required when using content type option nosniff\n\n## How was this patch tested?\n\nTested manually" }, { "commit": "826ec291fa83c26fd715b5f043629af2a48014cb", "tree": "1afbbbf7a3ead7e149091de56401cbe13d73b584", "parents": [ "1f6bd7ab55db3e5b08ff133d554b32db83a23890" ], "author": { "name": "György Gál", "email": "27883675+gyogal@users.noreply.github.com", "time": "Wed Apr 02 13:23:42 2025 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Apr 02 13:23:42 2025 +0200" }, "message": "[LIVY-1011] Upgrade CI image, fix Spark 3 tests\n\n## What changes were proposed in this pull request?\n\nThe CI environment is broken due to library version conflicts between\nthe image and the checkout action.\n\nThe livy-ci Docker image needs to be upgraded to a more recent version\nof Ubuntu, ensuring that the unit tests and integration tests all pass.\n\nDue to a bug in Livy\u0027s GitHub workflows, tests had only been executed\nwith Spark 2. Now that this is fixed, we also need to make updates to\nsome of the Python-based tests to work with Python 3 because Spark 3.2\ndoes not support Python 2.\n\nTwo R-based integration tests are ignored in the Spark 2 integration\ntest runs because SparkR 2 does not work with R 4, and the last version\nof Ubuntu with R 3.6 is Ubuntu 20. If we want to re-enable these tests,\nwe may need to build R 3.6 for Ubuntu 24.\n\n## How was this patch tested?\n\nCI and unit test runs in a private fork of the repo." }, { "commit": "1f6bd7ab55db3e5b08ff133d554b32db83a23890", "tree": "e430bcc25002260f94664c9c87cbcd1c87e8c6a4", "parents": [ "6097af1cdd536ebbe1d7eacb1513a440a5fa2784" ], "author": { "name": "Moritz Meister", "email": "8422705+moritzmeister@users.noreply.github.com", "time": "Tue Nov 12 14:33:33 2024 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Nov 12 14:33:33 2024 +0100" }, "message": "[LIVY-863] Missing JVM class imports for Spark3\n\n## What changes were proposed in this pull request?\r\n\r\nDescription of the problem: https://issues.apache.org/jira/browse/LIVY-863\r\n\r\nThe proposed fix consists of adding the missing imports that upstream Spark has when it\u0027s initiating the Java Gateway.\r\nSee the imports of Spark here: https://github.com/apache/spark/blob/87bf6b0ea4ca0618c8604895d05037edce8b7cb0/python/pyspark/java_gateway.py#L153\r\n\r\nAs far as I am aware, the java_import() does not fail or fails silently if the imported class does not exist.\r\nBut I might need to add some code to account for different Spark versions, looking for some guidance on this.\r\n\r\n## How was this patch tested?\r\n\r\nTested with both valid and \"bogus\" imports by reviewers." }, { "commit": "6097af1cdd536ebbe1d7eacb1513a440a5fa2784", "tree": "4e501b70a0217871dc591c45e2ccbc851838982f", "parents": [ "6dcb29422f5a43e396a6a2006819284e491de169" ], "author": { "name": "wangdengshan", "email": "wangdengshan01@163.com", "time": "Fri Sep 13 16:10:30 2024 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 13 10:10:30 2024 +0200" }, "message": "[LIVY-1003][RSC] Interactive session - Setting large value of rsc.server.connect.timeout blocks other tasks\n\n## What changes were proposed in this pull request?\r\n\r\nThe main adjustment here is the thread pool used when creating and closing sessions asynchronously. Scala\u0027s default thread pool size is limited, which will cause the waiting thread to be blocked.\r\n\r\nhttps://issues.apache.org/jira/browse/LIVY-1003\r\n\r\n## How was this patch tested?\r\n\r\nHow to reproduce:\r\n1. Set `livy.rsc.server.connect.timeout` to something high like 24h.\r\n2. Create enough interactive livy sessions in YARN so that they are queued in ACCEPTED state. The number of sessions that are stuck in ACCEPTED state should be equal to global execution context [thread pool size](https://docs.scala-lang.org/overviews/core/futures.html#the-global-execution-context) (Runtime.availableProcessors)\r\n3. Try to delete a session using DELETE /sessions/{sessionId}\r\nand it should not be hang until one of the sessions is no longer stuck in ACCEPTED state.\r\n" }, { "commit": "6dcb29422f5a43e396a6a2006819284e491de169", "tree": "695bbae4ef42409745138a0e1e9199885a04e3c9", "parents": [ "b089dd68277f5b97bd939787f05a953694302fba" ], "author": { "name": "Asif Khatri", "email": "123077165+askhatri@users.noreply.github.com", "time": "Mon Aug 12 16:56:29 2024 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 12 13:26:29 2024 +0200" }, "message": "[LIVY-1007] Livy should not spawn one thread per job to track the job on Kubernetes\n\n## What changes were proposed in this pull request?\r\n\r\nInstead of spawning a monitor thread for every session, create a centralised thread to monitor all Kubernetes apps.\r\n\r\nJIRA link: https://issues.apache.org/jira/browse/LIVY-1007\r\n\r\n## How was this patch tested?\r\n\r\n**Unit Tests:** The patch has been verified through comprehensive unit tests.\r\n**Manual Testing:** Conducted manual testing using Kubernetes on Docker Desktop.\r\n\r\nEnvironment: Helm charts. For detailed instructions on testing using Helm charts, please refer to the documentation available at [livycluster](https://github.com/askhatri/livycluster).\r\n\r\nCo-authored-by: Asif Khatri \u003casif.khatri@cloudera.com\u003e" }, { "commit": "b089dd68277f5b97bd939787f05a953694302fba", "tree": "6b60e5e070b85075676a600e7c4db0c8276b32e0", "parents": [ "03ceb4ae1f764b7299998ffafffda0a18d8a1602" ], "author": { "name": "Asif Khatri", "email": "123077165+askhatri@users.noreply.github.com", "time": "Wed Jul 10 14:35:17 2024 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 10 11:05:17 2024 +0200" }, "message": "[LIVY-702] Submit Spark apps to Kubernetes (#451)\n\nThis pull request (PR) is the foundational PR for adding Kubernetes support in Apache Livy, originally found here (#249). This update includes a newer version of the Kubernetes client and adds code to display the Spark UI.\r\n\r\n## Summary of the Proposed Changes\r\n\r\nThis PR introduces a method to submit Spark applications to a Kubernetes cluster. The key points covered include:\r\n\r\n * Submitting batch sessions\r\n * Submitting interactive sessions\r\n * Monitoring sessions, collecting logs, and gathering diagnostic information\r\n * Restoring session monitoring after restarts\r\n * Garbage collection (GC) of created Kubernetes resources\r\n\r\nJIRA link: https://issues.apache.org/jira/browse/LIVY-702\r\n\r\n## How was this patch tested?\r\n\r\n * Unit Tests: The patch has been verified through comprehensive unit tests.\r\n * Manual Testing: Conducted manual testing using Kubernetes on Docker Desktop.\r\n * Environment: Helm charts.\r\n\r\nFor detailed instructions on testing using Helm charts, please refer to the documentation available at https://github.com/askhatri/livycluster\r\n\r\nCo-authored-by: Asif Khatri \u003casif.khatri@cloudera.com\u003e\r\nCo-authored-by: Alex Sasnouskikh \u003cjahstreetlove@gmail.com\u003e" }, { "commit": "03ceb4ae1f764b7299998ffafffda0a18d8a1602", "tree": "0664807170a704bd6c200462eee23745005aa1d9", "parents": [ "21ca618e13137ff60553e91284e7b6f26b571819" ], "author": { "name": "Javi Roman", "email": "jroman.espinar@gmail.com", "time": "Wed Jun 26 10:21:03 2024 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jun 26 10:21:03 2024 +0200" }, "message": "[LIVY-997][DOC] Add file .sdkmanrc to .gitignore (#444)\n\n## What changes were proposed in this pull request?\r\n\r\nAdd new ignore pattern to .gitignore file for .sdkmanrc file." }, { "commit": "21ca618e13137ff60553e91284e7b6f26b571819", "tree": "de6e927752761b3123c9ee4d7cc1517603bf067b", "parents": [ "78ddd595953711aeed2a40b76408b58faba971f1" ], "author": { "name": "RajshekharMuchandi", "email": "rsmuchandi@gmail.com", "time": "Tue Jun 25 19:54:52 2024 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jun 25 16:24:52 2024 +0200" }, "message": "[LIVY-991][SERVER] Facing issues with the Livy UI Driver link (#437)\n\nAdded conditional check on finished state to set driverlogUrl to null\r\n\r\nCo-authored-by: Rajshekhar Muchandi \u003crajshekhar.muchandi@cloudera.com\u003e" }, { "commit": "78ddd595953711aeed2a40b76408b58faba971f1", "tree": "15058b089e7dd1f3135fcfa654e4227bfbc8314e", "parents": [ "728fcf0ae9b94b7690c8a3f1eaa4d05507bfbea6" ], "author": { "name": "RajshekharMuchandi", "email": "rsmuchandi@gmail.com", "time": "Tue Jun 25 19:41:02 2024 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jun 25 16:11:02 2024 +0200" }, "message": "[LIVY-992][SERVER] In livy logs, user is not able to differentiate between interactive session and batch session (#435)\n\nChanged log info to provide session type details\r\n\r\nCo-authored-by: Rajshekhar Muchandi \u003crajshekhar.muchandi@cloudera.com\u003e" }, { "commit": "728fcf0ae9b94b7690c8a3f1eaa4d05507bfbea6", "tree": "d95b8fc88f7967479343a0d786ba3c923ca81ea8", "parents": [ "f615f272e9130d02170024832ea308516b907195" ], "author": { "name": "Asif Khatri", "email": "123077165+askhatri@users.noreply.github.com", "time": "Mon Jun 10 13:46:06 2024 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 10 10:16:06 2024 +0200" }, "message": "[LIVY-998][THRIFT] Support connecting to existing sessions using session name via Thrift Server\n\nCo-authored-by: Asif Khatri \u003casif.khatri@cloudera.com\u003e" }, { "commit": "f615f272e9130d02170024832ea308516b907195", "tree": "807b5bb3a2469e2c4f4a63d034fdcc6d9dd75700", "parents": [ "96b1eb617b18302ae842d0ebed298e5ec61429f5" ], "author": { "name": "RajshekharMuchandi", "email": "rsmuchandi@gmail.com", "time": "Tue Mar 12 20:57:17 2024 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Mar 12 16:27:17 2024 +0100" }, "message": "[LIVY-977][SERVER][CONF] Livy can not be started if HDFS is still in safe mode (#440)\n\n## What changes were proposed in this pull request?\r\n\r\nHDFS safe mode is checked when livy session is created. If safe mode is ON, then IllegalStateException is thrown after max retry attempts (configurable) with safe mode interval (configurable) checks are done. If safe mode is OFF, then livy will be able to create session.\r\n\r\nhttps://issues.apache.org/jira/browse/LIVY-977\r\n\r\n## How was this patch tested?\r\n\r\nAdded unit test cases to validate code changes. Also, done manual testing in CDP cluster by creating livy sessions with HDFS safe mode check ON/OFF." }, { "commit": "96b1eb617b18302ae842d0ebed298e5ec61429f5", "tree": "e7ba69990db8eac6b57dc1e00e67a176652c1dcc", "parents": [ "86fc823893ee96d4effaa4f2b8ef6603cea9d77a" ], "author": { "name": "jianzhenwu", "email": "117174379+jianzhenwu@users.noreply.github.com", "time": "Fri Jan 26 07:56:55 2024 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jan 25 18:56:55 2024 -0500" }, "message": "[LIVY-995][REPL] JsonParseException is thrown when closing Livy session when using python profile (#439)\n\n* [LIVY-995][REPL] JsonParseException is thrown when closing Livy session when using python profile" }, { "commit": "86fc823893ee96d4effaa4f2b8ef6603cea9d77a", "tree": "e74aac11924532634df6736b03fd7da70515fb28", "parents": [ "024efa549bb490b6682ac6bd7fa5bf8b556b0d0c" ], "author": { "name": "Asif Khatri", "email": "123077165+askhatri@users.noreply.github.com", "time": "Mon Oct 30 19:28:53 2023 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 30 14:58:53 2023 +0100" }, "message": "[LIVY-989] Livy core support for interactive session idleTimeout (#426)\n\n## What changes were proposed in this pull request?\r\n\r\nCurrently, a Livy interactive session has a field called ttl, which kills the session if it has been idle for a given amount of time. However, here is the expected behavior:\r\n\r\n * ttl: kills the session if it has been active for a certain duration, irrespective of idleness.\r\n * idleTimeout: kills the session if it has been idle for the given duration. (The current TTL behaves in this manner.)\r\n\r\nJIRA: https://issues.apache.org/jira/browse/LIVY-989\r\n\r\n## How was this patch tested?\r\n\r\nTested manually by creating interactive session with idle time and TTL." }, { "commit": "024efa549bb490b6682ac6bd7fa5bf8b556b0d0c", "tree": "e9b2e5234c29944261d97878f129a0f6762f47dc", "parents": [ "8b2e29fe9fd42c20395c63e1571f2492f005162b" ], "author": { "name": "mvanderlee", "email": "918128+mvanderlee@users.noreply.github.com", "time": "Tue Oct 24 22:41:22 2023 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 24 13:41:22 2023 -0700" }, "message": "Fix testing with Python3 (#424)\n\nFix print statement in tests to be Python 2 and Python 3 compatible. #408" }, { "commit": "8b2e29fe9fd42c20395c63e1571f2492f005162b", "tree": "ae26dd0faf0a540509e6070c61d276fdd886e1fa", "parents": [ "bf30e3bd3105f72a6a7fd2a276a16dabe8bdf104" ], "author": { "name": "Marco Gaido", "email": "mgaido@fbk.eu", "time": "Thu Sep 07 15:56:45 2023 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 07 21:56:45 2023 +0800" }, "message": "[DEV] Support python3 for merge_livy_pr script (#418)\n\n" }, { "commit": "bf30e3bd3105f72a6a7fd2a276a16dabe8bdf104", "tree": "78ea49696aa3f34d47b3ddf6094ce17ebc7d825f", "parents": [ "5dccc479c6087112f048a7e5cff0723855ef14e9" ], "author": { "name": "jianzhen.wu", "email": "jianzhen.wu@shopee.com", "time": "Tue Sep 05 11:57:36 2023 +0200" }, "committer": { "name": "Marco Gaido", "email": "mgaido@fbk.eu", "time": "Tue Sep 05 11:57:36 2023 +0200" }, "message": "[LIVY-987] NPE when waiting for thrift session to start timeout.\n\n## What changes were proposed in this pull request?\nFix NPE when waiting for thrift session to start timeout.\nhttps://issues.apache.org/jira/browse/LIVY-987\n\n## How was this patch tested?\nmanual tests by beeline and set timeout to 10s.\n\n0: jdbc:hive2://username:passwordthrift-server\u003e select 123;\n\nRSC client is executing SQL query: select 123, statementId \u003d 681bc017-8f37-4665-a575-da355db77254, session \u003d SessionHandle [c17f1729-6ee1-4260-b82b-aebec3b08e14]\n\nLivy session has not yet started. Please wait for it to be ready...\n\nError: java.util.concurrent.TimeoutException: Futures timed out after [10000 milliseconds] (state\u003d,code\u003d0)\n\n0: jdbc:hive2://username:passwordthrift-server\u003e Closing: 0: jdbc:hive2://username:passwordthrift-server\n\nPlease review https://livy.incubator.apache.org/community/ before opening a pull request.\n\nAuthor: jianzhen.wu \u003cjianzhen.wu@shopee.com\u003e\nAuthor: jianzhenwu \u003c117174379+jianzhenwu@users.noreply.github.com\u003e\n\nCloses #416 from jianzhenwu/LIVY-987.\n" }, { "commit": "5dccc479c6087112f048a7e5cff0723855ef14e9", "tree": "52addb86bb080ee43d0b17c3ac1c712ae028a7cb", "parents": [ "4cee47df13348b6a3daedc5215d0cec0f98c83ee" ], "author": { "name": "Andras Beni", "email": "anna.andras.beni@gmail.com", "time": "Fri Aug 18 13:57:43 2023 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 18 13:57:43 2023 +0200" }, "message": "[LIVY-793] Make keystore type configurable (#395)\n\n## What changes were proposed in this pull request?\r\n\r\nThis change introduces configuration parameter livy.keystore.type.\r\nThe default value is JKS which is equivalent to current functionality.\r\n\r\n## How was this patch tested?\r\n\r\nThis change was tested by running existing tests and manually verifying functionality using non-JKS keystore." }, { "commit": "4cee47df13348b6a3daedc5215d0cec0f98c83ee", "tree": "42e50aa6c6279511dd584bd30925ed3f0cb27529", "parents": [ "074d6a284281f5d08d4572f97216716696b1ca9b" ], "author": { "name": "Asif Khatri", "email": "123077165+askhatri@users.noreply.github.com", "time": "Fri Aug 18 17:23:06 2023 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 18 13:53:06 2023 +0200" }, "message": "[LIVY-986][SERVER] Adding null pointer check for SessionInfo (#412)\n\nWe need to add a null check for SessionInfo in InteractiveSessionServlet.scala to avoid an exception that might occur during a Livy upgrade." }, { "commit": "074d6a284281f5d08d4572f97216716696b1ca9b", "tree": "8b46a58c41c43d40f394b31e871baf1083cb343a", "parents": [ "b175ba1bd1210e2d1db61bdf19a3dbfc1374dd8c" ], "author": { "name": "lmccay", "email": "lmccay@apache.org", "time": "Sun Aug 13 15:56:15 2023 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Aug 13 15:56:15 2023 -0400" }, "message": "Livy 984 (#410)\n\nLIVY-984 - Update Copyrights in NOTICE and footer" }, { "commit": "b175ba1bd1210e2d1db61bdf19a3dbfc1374dd8c", "tree": "ab6084c026286f16af7b1beaf2bba94629174f22", "parents": [ "713cb3e7938863969d5de26b0d5a20d8990ff584" ], "author": { "name": "Asif Khatri", "email": "123077165+askhatri@users.noreply.github.com", "time": "Sat Jul 22 09:10:56 2023 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jul 22 11:40:56 2023 +0800" }, "message": "LIVY-978: Update the application tag generation logic in Livy (#409)\n\nCo-authored-by: Asif Khatri \u003casif.khatri@cloudera.com\u003e" }, { "commit": "713cb3e7938863969d5de26b0d5a20d8990ff584", "tree": "36c97dbb74ad72c4838cb4cb7cef3b4621a592eb", "parents": [ "314f2def423954a77eb75f72e9bf7f4e51a6f7f4" ], "author": { "name": "Damon P. Cortesi", "email": "d.lifehacker@gmail.com", "time": "Wed Jul 19 23:07:25 2023 +0000" }, "committer": { "name": "Damon P. Cortesi", "email": "d.lifehacker@gmail.com", "time": "Wed Jul 19 23:07:25 2023 +0000" }, "message": "Update main to 0.9.0-incubating-SNAPSHOT\n" }, { "commit": "314f2def423954a77eb75f72e9bf7f4e51a6f7f4", "tree": "91925a3b085ec5336b9bc111038791930fb568ff", "parents": [ "c3dd6457c45662de13be72919532ed0ef1b57ab3" ], "author": { "name": "Sumit Kumar", "email": "ksumit@users.noreply.github.com", "time": "Fri Jun 16 17:03:57 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jun 16 17:03:57 2023 -0700" }, "message": "[LIVY-969] Create docker based integration environment for local debugging (#407)\n\n* [LIVY-969] Create docker based integration environment for local\r\ndebugging\r\n* Add missing newlines, move customization steps towards the end in Dockerfile so we can reuse previous layers and avoid recreating it for every image build\r\n* Revert changes in module list, still update the scala-maven-plugin version to avoid issues compiling against 2.12\r\n* Upgrade spark3 version to spark 3.2.3\r\n* Fix Dockerfile path in ci image build pipeline, \r\n* Update instructions to use livy-ci container image name\r\n\r\n---------\r\n\r\nCo-authored-by: Sumit Kumar \u003cksumit@apache.org\u003e" }, { "commit": "c3dd6457c45662de13be72919532ed0ef1b57ab3", "tree": "bdb40e7a75ae767a5a6e65d2870632665690023b", "parents": [ "cb484ea66d80381ba28d6b3bdc84d1828084c84d" ], "author": { "name": "Sumit Kumar", "email": "ksumit@users.noreply.github.com", "time": "Tue May 16 11:09:49 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue May 16 11:09:49 2023 -0700" }, "message": " [LIVY-593] Add flags to support hadoop, spark and scala version specific builds (#405)\n\nEnable flags to build livy against specific hadoop, spark and scala versions. Currently supports:\r\n1. scala 2.11 and scala 2.12 with Spark 2 and Hadoop 2\r\n2. scala 2.12 with Spark 3 and hadoop 2 (Spark 3 doesn\u0027t come with scala 2.11 builds)\r\n\r\nCo-authored-by: Sumit Kumar \u003cksumit@apache.org\u003e" }, { "commit": "cb484ea66d80381ba28d6b3bdc84d1828084c84d", "tree": "b6a6d6c2a2d8a9edf4cda162cc3aa016308587c3", "parents": [ "7a6b194583946d618b827293368544c1f2abb578" ], "author": { "name": "Asif Khatri", "email": "123077165+askhatri@users.noreply.github.com", "time": "Tue May 09 13:11:28 2023 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue May 09 09:41:28 2023 +0200" }, "message": "[LIVY-975][SERVER] Fix compatibility issue related to a change on the Spark side (#403)\n\n## What changes were proposed in this pull request?\r\n\r\nRecent change in spark required us to fix compatibility issue related to a change on the Spark side.\r\n\r\n## How was this patch tested?\r\n\r\nVerified manually by creating interactive / batch sessions via REST API call in a local Yarn cluster. Also, we have updated the unit tests." }, { "commit": "7a6b194583946d618b827293368544c1f2abb578", "tree": "1ce617a128d235fd756abd6b6864a23430c5a732", "parents": [ "12505dd88fac4c2b9da92a020205ffb72524a9c8" ], "author": { "name": "Asif Khatri", "email": "123077165+askhatri@users.noreply.github.com", "time": "Tue May 09 12:42:59 2023 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue May 09 09:12:59 2023 +0200" }, "message": "LIVY-974: Remove verbose output on Livy UI error pages (#404)\n\n## What changes were proposed in this pull request?\r\n\r\nOn error, the Livy UI shows verbose output on error pages including the Jetty version number. This could be considered as a security vulnerability. We can make it configurable and avoid sending server version details. The Jetty version is there in every response header as well.\r\n\r\n## How was this patch tested?\r\n\r\nVerified manually by calling get sessions via REST API call in a local Yarn cluster." }, { "commit": "12505dd88fac4c2b9da92a020205ffb72524a9c8", "tree": "26670d5d38ea8b18d7f7f9be7b78b0b9004012e9", "parents": [ "80c6303b5bc7ac5d07281b0366dc2c4a2d7f082a" ], "author": { "name": "Damon P. Cortesi", "email": "d.lifehacker@gmail.com", "time": "Wed Apr 26 10:14:29 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Apr 26 10:14:29 2023 -0700" }, "message": "[LIVY-878] Replace log4j with reload4j (#392)\n\n* Replace log4j with reload4j along with relevant exclusions in dependencies\r\n* Bump surefire to fix build\r\n* Swap slf4j-api with slf4j-reload4j" }, { "commit": "80c6303b5bc7ac5d07281b0366dc2c4a2d7f082a", "tree": "b3ea69b3029aab16b23f17240b5da7912c739d32", "parents": [ "830a27e16efa9d1e6265988d7d71dc1de6ed2b39" ], "author": { "name": "Asif Khatri", "email": "123077165+askhatri@users.noreply.github.com", "time": "Tue Apr 18 17:45:55 2023 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Apr 18 14:15:55 2023 +0200" }, "message": "[LIVY-967][SERVER] Return session information with livy sessions APIs (#383)\n\n## What changes were proposed in this pull request?\r\n\r\nCurrently livy GET /Sessions doesn\u0027t return fields like driver-executor memory, spark configuration, etc. Ideally a session response should return all the values set in session request api call(POST /sessions).\r\n\r\nJIRA: https://issues.apache.org/jira/browse/LIVY-967\r\n\r\n## How was this patch tested?\r\n\r\nVerified manually by creating interactive session via REST API call in a local Yarn cluster. Also, we have updated the unit tests." }, { "commit": "830a27e16efa9d1e6265988d7d71dc1de6ed2b39", "tree": "753435d856963fa4eb53acc498d56d606a2d61bb", "parents": [ "e4c2676c161edf585bf9bb543a9277aaf92283a5" ], "author": { "name": "RajshekharMuchandi", "email": "rsmuchandi@gmail.com", "time": "Wed Apr 12 13:26:39 2023 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Apr 12 09:56:39 2023 +0200" }, "message": "[LIVY-973][RSC] Always close RSCClient\u0027s EventLoopGroup (#398)\n\n## What changes were proposed in this pull request?\r\n\r\nAdded try-catch block to jobs\u0027 setFailure() and log entries if there is any exception. eventLoopGroup.shutdownGracefully(); statement will be executed after exception is caught." }, { "commit": "e4c2676c161edf585bf9bb543a9277aaf92283a5", "tree": "896153335779d78c31a9f0b629b84ff61d1911c7", "parents": [ "64521ce806c8263cd243e221982b026edd9cdf28" ], "author": { "name": "Asif Khatri", "email": "123077165+askhatri@users.noreply.github.com", "time": "Tue Apr 11 17:44:33 2023 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Apr 11 14:14:33 2023 +0200" }, "message": "[LIVY-970][SERVER] Ordering and pagination support in GET /statement (#389)\n\n## What changes were proposed in this pull request?\r\n\r\nGET /sessions/id/statements returns a list of statements run in a session. However the ordering of the statements are older ones first. Livy could expose a parameter (something like orderBy\u003dlatest) that will order the list of statements as latest first.\r\nJIRA: https://issues.apache.org/jira/browse/LIVY-970\r\n\r\n## How was this patch tested?\r\n\r\nVerified manually by creating interactive session and statements via REST API call in a local Yarn cluster." }, { "commit": "64521ce806c8263cd243e221982b026edd9cdf28", "tree": "b6e26190fff120e8d82ac2e6290d09a56162198d", "parents": [ "da246ff30d11705d8f9772950c34c4ae1c82cf09" ], "author": { "name": "RajshekharMuchandi", "email": "rsmuchandi@gmail.com", "time": "Tue Apr 11 17:42:40 2023 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Apr 11 14:12:40 2023 +0200" }, "message": "[LIVY-763][SERVER] Fix File Descriptor leak from DirectoryStream (#397)\n\n## What changes were proposed in this pull request?\r\n\r\nTook separate handle for directory stream. Closed directory stream handle in finally block. Code changes done in InteractiveSession.scala.\r\n\r\nJIRA: https://issues.apache.org/jira/browse/LIVY-763\r\n\r\n## How was this patch tested?\r\n\r\nRan lsof | grep python command to see the Directory FDs for {SPARK_HOME}/python/lib. Tested by creating interactive session for pyspark and verified FDs are not created." }, { "commit": "da246ff30d11705d8f9772950c34c4ae1c82cf09", "tree": "f422132f11bfb92d19c4b6cd99347a237f44db56", "parents": [ "45e07fec68f2b9ad1dc7ebce8db08ad8a778dddc" ], "author": { "name": "Damon P. Cortesi", "email": "d.lifehacker@gmail.com", "time": "Sat Apr 08 11:42:44 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Apr 08 20:42:44 2023 +0200" }, "message": "[LIVY-972] Add GitHub Action for CI image (#393)\n\n* Add GitHub Action for CI image\r\n\r\n* Update push action\r\n\r\n* Add sample unit test\r\n\r\n* Add matrix based on spark version\r\n\r\n* Update spark version to be strings for proper comparison\r\n\r\n* Fix syntax\r\n\r\n* Add maven cache and integration tests\r\n\r\n* Just use actions/cache\r\n\r\n* Be more explicty with pom locations because container build is running as root. https://github.com/actions/runner/issues/449\r\n\r\n* Add MAVEN_OPTS to avoid timeouts[1] and ignore livy in m2 cache\r\n[1] https://github.com/apache/iotdb/issues/3121\r\n\r\n* Update cache paths\r\n\r\n* Add codecov" }, { "commit": "45e07fec68f2b9ad1dc7ebce8db08ad8a778dddc", "tree": "21c43ab918458cdc2349ab5c06478654a0391e02", "parents": [ "8d07eee831221093f55203190d38d793234c5db7" ], "author": { "name": "Asif Khatri", "email": "123077165+askhatri@users.noreply.github.com", "time": "Fri Feb 10 21:42:31 2023 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Feb 10 11:12:31 2023 -0500" }, "message": "[LIVY-968][SERVER] Provide ttl field for a livy session (#384)\n\n* [LIVY-968][SERVER] Provide ttl field for a livy session via askhatri" }, { "commit": "8d07eee831221093f55203190d38d793234c5db7", "tree": "b88daa4b97d444089e3c993f48bf46d4c477fc28", "parents": [ "cee2c79db8607f87b1a92f8fde6c1aaa84c3a60c" ], "author": { "name": "Praneet Sharma", "email": "praneet10061@iiitd.ac.in", "time": "Mon Dec 26 21:33:54 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Dec 27 00:33:54 2022 -0500" }, "message": "[LIVY-888] AddJar or AddFile call on a duplicate file should not result in failure for Livy session (#341)\n\n* [LIVY-888] AddJar or AddFile call on a duplicate file should not result in failure for Livy session" }, { "commit": "cee2c79db8607f87b1a92f8fde6c1aaa84c3a60c", "tree": "b181de756a060b18a07c88a2ce3c69224460e72d", "parents": [ "9855a9e990f8513d17a445e5b76cc0cb87c3dc06" ], "author": { "name": "Praneet Sharma", "email": "praneet10061@iiitd.ac.in", "time": "Mon Dec 26 12:38:12 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Dec 26 15:38:12 2022 -0500" }, "message": "LIVY-889 initial commit (#344)\n\n* addJar, addFile called during a running livy session will take care of adding this jar to spark driver classpath and appropriate interpreter." }, { "commit": "9855a9e990f8513d17a445e5b76cc0cb87c3dc06", "tree": "5e1cc8b50ffe038acd9771307da8bdd8550dfae1", "parents": [ "25e9b8e8fe427c286894309a553c2f417c16384d" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Sat Dec 17 00:18:11 2022 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Dec 17 00:18:11 2022 -0500" }, "message": "Bump junit from 4.11 to 4.13.1 (#375)\n\nBumps [junit](https://github.com/junit-team/junit4) from 4.11 to 4.13.1." }, { "commit": "25e9b8e8fe427c286894309a553c2f417c16384d", "tree": "e36c06b24504ef09d1d3605e278c33a8edb90930", "parents": [ "04917edc809434b7810ac37effceaf3e5e218f56" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Fri Dec 16 19:24:19 2022 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Dec 16 19:24:19 2022 -0500" }, "message": "Bump httpclient from 4.5.3 to 4.5.13 (#374)\n\nBumps httpclient from 4.5.3 to 4.5.13." }, { "commit": "04917edc809434b7810ac37effceaf3e5e218f56", "tree": "416232db951247c651f080d6a87d0330e54b1e9c", "parents": [ "7e87c6cfa078af9721a9dcc221a4cbf8c8a4efc5" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Fri Dec 16 18:13:28 2022 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Dec 16 18:13:28 2022 -0500" }, "message": "Bump commons-io from 2.5 to 2.7 in /integration-test (#376)\n\nBumps commons-io from 2.5 to 2.7." }, { "commit": "7e87c6cfa078af9721a9dcc221a4cbf8c8a4efc5", "tree": "014800c538bfdc551bcc1a8b3dc010019deab242", "parents": [ "ff1f65bbf1dc9ddeb208b086c307642d3530e7a7" ], "author": { "name": "lmccay", "email": "lmccay@apache.org", "time": "Fri Dec 16 10:19:34 2022 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Dec 16 10:19:34 2022 -0500" }, "message": "Livy 904 - Address Jetty Dependency Upgrades for 0.8.0 (#372)\n\n* LIVY-904 - Address Jetty Dependency Upgrades for 0.8.0\r\n* LIVY-904 - Bump apacheds dependencies to align with jetty upgrade" }, { "commit": "ff1f65bbf1dc9ddeb208b086c307642d3530e7a7", "tree": "8b2c0949ecc052139bfef55ea22fb4c08738805f", "parents": [ "41304e8ef1d82127740820b4998fd54f8137ee60" ], "author": { "name": "lmccay", "email": "lmccay@apache.org", "time": "Wed Dec 14 20:27:28 2022 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Dec 14 20:27:28 2022 -0500" }, "message": "LIVY-903 - Address Netty Upgrades for 0.8.0 (#371)\n\n* LIVY-904 - Address Jetty Dependency Upgrades for 0.8.0" }, { "commit": "41304e8ef1d82127740820b4998fd54f8137ee60", "tree": "6313c31a974c566584a3e689dcced1bc3a7393bb", "parents": [ "e5599757798c3ef59076ad4b2d56e722f39488ed" ], "author": { "name": "Damon P. Cortesi", "email": "d.lifehacker@gmail.com", "time": "Sun Dec 11 01:12:53 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Dec 11 17:12:53 2022 +0800" }, "message": "Bump jackson to 2.12 - closes #364 (#369)\n\n" }, { "commit": "e5599757798c3ef59076ad4b2d56e722f39488ed", "tree": "ebc4df2fc450733499f4a4d003a1258407861857", "parents": [ "58e3ebbd1d828be9b054191093715b0b49b2963c" ], "author": { "name": "Damon P. Cortesi", "email": "d.lifehacker@gmail.com", "time": "Sat Dec 10 10:26:44 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Dec 10 10:26:44 2022 -0800" }, "message": "Add a Dockerfile for the build (#367)\n\n* Add a Dockerfile for the build\r\n* Add UTF-8 locale so python2 unicode test passes and add volume mapping for .m2 directory\r\n* Squish python3 deps per comments and reformat" }, { "commit": "58e3ebbd1d828be9b054191093715b0b49b2963c", "tree": "c8c52403e49134f38ce0b91e152c15b410524e60", "parents": [ "50faa9f68a6f1c79f605a4cb70e452f9c7f21528" ], "author": { "name": "Damon P. Cortesi", "email": "d.lifehacker@gmail.com", "time": "Fri Dec 09 11:27:24 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Dec 09 11:27:24 2022 -0800" }, "message": "Bump version dependencies for docs (#368)\n\n- Closes #361\r\n- Closes #357 (not the full version bump)\r\n- Closes #351" }, { "commit": "50faa9f68a6f1c79f605a4cb70e452f9c7f21528", "tree": "0cf6aa6da53aa89563d0461266d2afd1a0299f59", "parents": [ "b81656823e019c64bf98ab3e61670e9b48fc3797" ], "author": { "name": "leesf", "email": "490081539@qq.com", "time": "Wed Nov 16 11:05:58 2022 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 16 11:05:58 2022 +0800" }, "message": "[MINOR] Add ASF YAML (#365)\n\n" }, { "commit": "b81656823e019c64bf98ab3e61670e9b48fc3797", "tree": "cd00281af61d7efefe68cd226abc22b4c61e4d50", "parents": [ "2b25c71008e98dc645f6117d379e1ca790688730", "1a57012cb586621fe7c4cc3d3afe25c92ea4a5cc" ], "author": { "name": "Jeffrey(Xilang) Yan", "email": "7855100+yantzu@users.noreply.github.com", "time": "Wed Nov 09 18:02:29 2022 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 09 18:02:29 2022 +0800" }, "message": "Merge pull request #313 from Minutis/master\n\n[LIVY-783] Use shaded kryo library" }, { "commit": "2b25c71008e98dc645f6117d379e1ca790688730", "tree": "f851fef1f8ca1a3b2b3789b91c8b9c37511d4832", "parents": [ "8c6f60b74633f25c211ca248ba290de65aee89f9" ], "author": { "name": "Gabriel Magno", "email": "gabrielmagno1@gmail.com", "time": "Tue Nov 08 22:03:34 2022 -0300" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 09 09:03:34 2022 +0800" }, "message": "[LIVY-795] Add support for PySpark with Python \u003e\u003d 3.8 (#314)\n\n" }, { "commit": "8c6f60b74633f25c211ca248ba290de65aee89f9", "tree": "75adb83a1aebfb0bed7372ca477e339caa8a267d", "parents": [ "1954868ccb185cbe9c87ec885edc208658a55869" ], "author": { "name": "Jeff Xu", "email": "49769179+jeff-xu-z@users.noreply.github.com", "time": "Tue Nov 08 01:39:15 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Nov 08 17:39:15 2022 +0800" }, "message": "[LIVY-896] Livy not capture spark-submit error exit if timing is right (#358)\n\n" }, { "commit": "1954868ccb185cbe9c87ec885edc208658a55869", "tree": "51cd931b5c1a52055f6099a51736fd8c153933ba", "parents": [ "4d8a912699683b973eee76d4e91447d769a0cb0d" ], "author": { "name": "leesf", "email": "490081539@qq.com", "time": "Mon Nov 07 10:36:05 2022 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Nov 07 10:36:05 2022 +0800" }, "message": "[MINOR] fix travis (#362)\n\n" }, { "commit": "1a57012cb586621fe7c4cc3d3afe25c92ea4a5cc", "tree": "adc8669e4e1fe99b3bba0205a0ab0bff9ca95bb0", "parents": [ "4d8a912699683b973eee76d4e91447d769a0cb0d" ], "author": { "name": "Minutis", "email": "minutis@exacaster.com", "time": "Mon Dec 28 23:13:40 2020 +0200" }, "committer": { "name": "Minutis", "email": "minutis@exacaster.com", "time": "Mon Dec 28 23:13:40 2020 +0200" }, "message": "Fixes [LIVY-783] https://issues.apache.org/jira/browse/LIVY-783\n" }, { "commit": "4d8a912699683b973eee76d4e91447d769a0cb0d", "tree": "8c1d941aa75e5232b8b703a579d0d872a266757f", "parents": [ "97cf2f75929ef6c152afc468adbead269bd0758f" ], "author": { "name": "Marco Gaido", "email": "mgaido@apache.org", "time": "Fri Aug 14 17:25:54 2020 -0700" }, "committer": { "name": "Alex Bozarth", "email": "ajbozart@us.ibm.com", "time": "Fri Aug 14 17:25:54 2020 -0700" }, "message": "Add html escape to session name\n\n## What changes were proposed in this pull request?\n\nThe PR adds HTML escaping to session names.\n\n## How was this patch tested?\n\nManual test.\n\nAuthor: Marco Gaido \u003cmgaido@apache.org\u003e\n\nCloses #302 from mgaido91/escape_html.\n" }, { "commit": "97cf2f75929ef6c152afc468adbead269bd0758f", "tree": "dcfc8b20d3a423cbc8aa3209b7ebc4dc8c363cea", "parents": [ "3b9bbefbaf8ddb3116952caf8c15b97d676386a7" ], "author": { "name": "jerryshao", "email": "jerryshao@tencent.com", "time": "Thu Jul 02 15:44:12 2020 +0800" }, "committer": { "name": "jerryshao", "email": "jerryshao@tencent.com", "time": "Thu Jul 02 15:44:12 2020 +0800" }, "message": "[LIVY-756] Add Spark 3.0 and Scala 2.12 support\n\n## What changes were proposed in this pull request?\n\nThis PR is based tprelle \u0027s PR #289 , and address all the left issues in that PR:\n\n1. multi-scala version support in one build (Scala 2.11 and 2.12 support).\n2. make SparkR work.\n\nAlso reverts most of the unnecessary changes. Besides this PR remove the build below 2.4 (2.2, 2.3), since Spark 2.2 and 2.3 only ships with Scala 2.11, hard to maintain multiple version. But user could still use 2.2 and 2.3 without changes.\n\nAll credits to tprelle.\n\n## How was this patch tested?\n\nRun UT and IT with Spark 2.4.5 and 3.0.0 locally.\n\nAuthor: jerryshao \u003cjerryshao@tencent.com\u003e\n\nCloses #300 from jerryshao/LIVY-756.\n" }, { "commit": "3b9bbefbaf8ddb3116952caf8c15b97d676386a7", "tree": "9331a17d22a594c59b9d037feb84c04151ef4536", "parents": [ "ee7fdfc45d90c0478dcd446bc8a19a217eebe04d" ], "author": { "name": "Wing Yew Poon", "email": "wypoon@cloudera.com", "time": "Wed May 20 19:37:20 2020 +0200" }, "committer": { "name": "Marco Gaido", "email": "mgaido@apache.org", "time": "Wed May 20 19:37:20 2020 +0200" }, "message": "[LIVY-754][THRIFT] Encode precision and scale for decimal type.\n\n## What changes were proposed in this pull request?\n\nWhen a `org.apache.livy.thriftserver.session.DataType.DECIMAL` is converted to a `org.apache.hive.service.rpc.thrift.TTypeDesc` for sending a Thrift response to a client request for result set metadata, the `TTypeDesc` contains a `TPrimitiveTypeEntry(TTypeId.DECIMAL_TYPE)` without `TTypeQualifiers` (which are needed to capture the precision and scale).\nWith this change, we include the qualifiers in the `TPrimitiveTypeEntry`. We use both the name and the `DataType` of a field type to construct the `TTypeDesc`. We are able to do this without changing the existing internal representation for data types because we can obtain the precision and scale from the name of the decimal type.\n\n## How was this patch tested?\n\nUse beeline to connect to the Thrift server. Do a select from a table with a column of decimal type.\nAlso extended an existing integration test.\n\nAuthor: Wing Yew Poon \u003cwypoon@cloudera.com\u003e\n\nCloses #288 from wypoon/LIVY-754.\n" }, { "commit": "ee7fdfc45d90c0478dcd446bc8a19a217eebe04d", "tree": "8d2556b488a25f48df79af7ba03ad1da18f321da", "parents": [ "e39d8fee43adbddf88acb2e04b470aa14b713785" ], "author": { "name": "Shingo Furuyama", "email": "sfuruyam@yahoo-corp.jp", "time": "Thu Mar 26 14:59:21 2020 +0800" }, "committer": { "name": "jerryshao", "email": "jerryshao@tencent.com", "time": "Thu Mar 26 14:59:21 2020 +0800" }, "message": "[MINOR] Modify the description of POST /sessions/{sessionId}/completion\n\n## What changes were proposed in this pull request?\n\nJust modified a description of POST /sessions/{sessionId}/completion in the api-doc.\n\n## How was this patch tested?\n\nSince the change is quite small, I didn\u0027t test the patch. If I have an instruction, I will follow it.\n\nAuthor: Shingo Furuyama \u003csfuruyam@yahoo-corp.jp\u003e\n\nCloses #285 from marblejenka/mod-doc-completion.\n" }, { "commit": "e39d8fee43adbddf88acb2e04b470aa14b713785", "tree": "a8236813b315d42d26dd2f5dc7a8d6b817a3d0ad", "parents": [ "d07d103f22941525d3cfa2f07f647e310ffb34a1" ], "author": { "name": "Shingo Furuyama", "email": "sfuruyam@sfuruyam2.ssk.ynwm.yahoo.co.jp", "time": "Thu Mar 26 14:07:42 2020 +0800" }, "committer": { "name": "jerryshao", "email": "jerryshao@tencent.com", "time": "Thu Mar 26 14:07:42 2020 +0800" }, "message": "[LIVY-751] Livy server should allow to customize LIVY_CLASSPATH\n\n## What changes were proposed in this pull request?\n\nThe purpose and background is https://issues.apache.org/jira/browse/LIVY-751\n\n## How was this patch tested?\n\nI tested the following two manually.\n\n1. To confirm there is no degradation, I run 0.7.0-incubating livy server with sources in this PR. I also run an example jobs, and it completed without error.\n2. To confirm our workaround works, I build 0.7.0-incubating branch with specifying `-Dhadoop.scope\u003dprovided` and sources with this PR. After that, I added `export LIVY_CLASSPATH\u003d\"$LIVY_HOME/jars/*:$(hadoop classpath)\"` in conf/livy-env.sh and boot livy server. I also run an example jobs, and it completed without error.\n\nAuthor: Shingo Furuyama \u003csfuruyam@sfuruyam2.ssk.ynwm.yahoo.co.jp\u003e\nAuthor: Shingo Furuyama \u003csfuruyam@yahoo-corp.jp\u003e\n\nCloses #282 from marblejenka/livy-classpath.\n" }, { "commit": "d07d103f22941525d3cfa2f07f647e310ffb34a1", "tree": "fab1cb2e62a87667272373e9d6775fd2db05417b", "parents": [ "3f9a1a5222a3d0acd828e6d08997f01c1abfa42e" ], "author": { "name": "Shingo Furuyama", "email": "sfuruyam@yahoo-corp.jp", "time": "Thu Mar 26 13:55:51 2020 +0800" }, "committer": { "name": "jerryshao", "email": "jerryshao@tencent.com", "time": "Thu Mar 26 13:55:51 2020 +0800" }, "message": "[MINOR] Add description of POST /batches\n\n## What changes were proposed in this pull request?\n\nJust added a description of POST /batches in the api-doc.\n\n## How was this patch tested?\n\nSince the change is quite small, I didn\u0027t test the patch. If I have an instruction, I will follow it.\n\nAuthor: Shingo Furuyama \u003csfuruyam@yahoo-corp.jp\u003e\n\nCloses #283 from marblejenka/add-description.\n" }, { "commit": "3f9a1a5222a3d0acd828e6d08997f01c1abfa42e", "tree": "b2dbfcb79433d70c0e27c65fff34d35cfe7b0326", "parents": [ "06a8d4f67c910d0cd8327b1547134d641d793acb" ], "author": { "name": "Wing Yew Poon", "email": "wypoon@cloudera.com", "time": "Mon Mar 16 17:46:29 2020 +0100" }, "committer": { "name": "Marco Gaido", "email": "mgaido@apache.org", "time": "Mon Mar 16 17:46:29 2020 +0100" }, "message": "[LIVY-752][THRIFT] Fix implementation of limits on connections.\n\n## What changes were proposed in this pull request?\n\n`LivyThriftSessionManager` keeps a `ConcurrentHashMap[String, AtomicLong]` named `connectionsCount` to track the number of connections per user, etc. The `incrementConnectionsCount` and `decrementConnectionsCount` methods in `LivyThriftSessionManager` check that `connectionsCount` does not contain a key (instead of contains the key) before getting the value and incrementing or decrementing the count (leading to a `NullPointerException`). Even accounting for the incorrect condition, they do not use the `ConcurrentHashMap` correctly. There is a race -- a thread can get a count, find that it\u0027s within a limit, create a new session and then increment the count, while in the meantime, another thread could have incremented the count and so the limit is now actually exceeded.\n\nWe increment all relevant counts optimistically before creating a new session, check if any limits are violated, and if so, decrement all incremented counts.\n\n## How was this patch tested?\n\nTested by deploying the change on a cluster and setting livy.server.thrift.limit.connections.per.user. Verified that the number of connections reaches but does not exceed the limit.\nAlso, added basic unit tests that connection limits are enforced.\n\nAuthor: Wing Yew Poon \u003cwypoon@cloudera.com\u003e\n\nCloses #284 from wypoon/LIVY-752.\n" }, { "commit": "06a8d4f67c910d0cd8327b1547134d641d793acb", "tree": "5e08511d240dc6b2a0c3f2b6b77c08ef2d55233d", "parents": [ "3a2685643670c1cd1530feb7a54516b01784e781" ], "author": { "name": "Roger Liu", "email": "roliu@microsoft.com", "time": "Mon Mar 02 10:03:39 2020 +0800" }, "committer": { "name": "jerryshao", "email": "jerryshao@tencent.com", "time": "Mon Mar 02 10:03:39 2020 +0800" }, "message": "[LIVY-748] Add support for running Livy Integration tests against secure external clusters\n\n## What changes were proposed in this pull request?\n\nhttps://issues.apache.org/jira/projects/LIVY/issues/LIVY-748?filter\u003dreportedbyme\n\nThis PR configures the integration tests to work on Livy Servers instances that require basic or kerberos authentication. To do this we had to replace the async httpclient with apaches normal httpclient, as there were issues found getting kerberos authentication to work with the async client.\n\nThis PR also contains a minor fix to the InteractiveIT test suite. One of the tests would occasionally fail as a statement submitted to the interactive session would not be finished processing when the result was checked. The correctness check was also changed to correctly check for code completition\n\n## How was this patch tested?\nThis patch was tested on local deployments of livy server, both on a instance that required kerberos authentication and on one that required basic authentication.\n\nAuthor: Roger Liu \u003croliu@microsoft.com\u003e\nAuthor: roliu \u003croliu@microsoft.com\u003e\n\nCloses #278 from RogPodge/LivyKerberosIT.\n" }, { "commit": "3a2685643670c1cd1530feb7a54516b01784e781", "tree": "18840b31b5405c2a9b22b1f4ae96ed631b2b673f", "parents": [ "f4ab5ef5d389d5743410f5839ffc79aea8943c9c" ], "author": { "name": "Wing Yew Poon", "email": "wypoon@cloudera.com", "time": "Sat Feb 15 22:42:47 2020 +0100" }, "committer": { "name": "Marco Gaido", "email": "mgaido@apache.org", "time": "Sat Feb 15 22:42:47 2020 +0100" }, "message": "[LIVY-745] Ensure that a single RSCClientFactory gets loaded.\n\n## What changes were proposed in this pull request?\n\nIn LivyClientBuilder, a ServiceLoader is used to load configured LivyClientFactory providers (HttpClientFactory and RSCClientFactory). Correct behavior of RSCClientFactory implicitly depends on there being a single instance of it in the Livy server.\nInstead of instantiating a new ServiceLoader every time the build method is called on a LivyClientBuilder, keep a single ServiceLoader in a static field. ServiceLoader instances are not safe for use by multiple concurrent threads, so have the ServiceLoader load the LivyClientFactory implementations into a static List. Then LivyClientBuilder#build will use the single instance of either HttpClientFactory or RSCClientFactory in this List to create a LivyClient.\n\n## How was this patch tested?\n\nTested by having 20 clients connect to the Livy Thrift server simultaneously. Before the change, the behavior described in https://issues.apache.org/jira/browse/LIVY-745 was encountered. With the change, the problem is not encountered.\nAlso added a unit test that fails before this change to LivyClientBuilder, and passes with the change.\n\nAuthor: Wing Yew Poon \u003cwypoon@cloudera.com\u003e\n\nCloses #275 from wypoon/LIVY-745.\n" }, { "commit": "f4ab5ef5d389d5743410f5839ffc79aea8943c9c", "tree": "5eb505d3125ad37f466b16fdf782ac0426b45294", "parents": [ "66b5833e413bc10e39e3b92b585f496444c147d4" ], "author": { "name": "Wing Yew Poon", "email": "wypoon@cloudera.com", "time": "Tue Feb 04 16:01:52 2020 +0800" }, "committer": { "name": "jerryshao", "email": "jerryshao@tencent.com", "time": "Tue Feb 04 16:01:52 2020 +0800" }, "message": "[MINOR] Fix CI breakage in python-api unit tests.\n\n## What changes were proposed in this pull request?\n\nFreeze python mock library at 3.0.5 to avoid pulling in 4.0.0b1.\n\n## How was this patch tested?\n\nExisting unit tests.\n\nAuthor: Wing Yew Poon \u003cwypoon@cloudera.com\u003e\n\nCloses #279 from wypoon/CI_fix.\n" } ], "next": "66b5833e413bc10e39e3b92b585f496444c147d4" }