)]}' { "log": [ { "commit": "82c53aaad53c69f99ef5130658d9f93093a1e477", "tree": "03e2edd996562c3279c7d2ecbd47ca3755163cb9", "parents": [ "23f52e98d6426a5d0e11bc4e8c8de588b60f0406" ], "author": { "name": "Jon Bringhurst", "email": "jon@bringhurst.org", "time": "Fri May 02 15:22:30 2025 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 02 15:22:30 2025 -0700" }, "message": "SAMZA-2804: Concurrency issues identified in run-class.sh on samza-yarn (#1716)\n\n* Add annotations for each line identified as having a potential issue.\n\n* Resolve multiple concurrency issues\n\n## Race condition in pathing jar manifest creation\n\nA race condition exists when setting up the classpath during container launch.\n\nDuring container launch using samza-yarn, run-class.sh creates a pathing jar file (which holds the classpath for the container launch). However, during the creation of this pathing jar, temporary files, as well as the pathing jar itself is not placed in a location unique to the container. This results in multiple containers writing to the same pathing jar location and temporary file location, which results in a race condition.\n\nThis race condition may show up in several ways, such as when Yarn removes jars from a finished container (other containers will point to a classpath which no longer exists) or when multiple run-class.sh scripts attempt to write the manifest.txt or pathing jar at the same time.\n\nNote that host affinity being enabled will make this problem worse. The pathing.jar is written to the usercache, so when the container which created the pathing.jar is finished and removed, any new container which launches on that host will point to jar files which do not exist anymore. When host affinity is enabled, it will not move to a new host and just keep failing.\n\n## Container logging directory fallback is not unique for each container\n\nThe fallback log directory is the same among all containers running on the same host. It should be unique per-container.\n\n## Container tmp dir is not unique per-container\n\nThe JAVA_TMP_DIR directory is the same for all containers. We should make sure that it\u0027s safe to use the same directory for all containers.\n\n* Simplify comments and print manifest file locations" }, { "commit": "23f52e98d6426a5d0e11bc4e8c8de588b60f0406", "tree": "8f1b218901753abe80c60da33240c42755a63fe0", "parents": [ "28149ead1884345d9a60d8358b295ef09813098f" ], "author": { "name": "Niall Pemberton", "email": "niall.pemberton@gmail.com", "time": "Tue Mar 04 23:18:14 2025 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Mar 04 15:18:14 2025 -0800" }, "message": "SAMZA-2803 Remove Google Analytics from the Samza Website (#1715)\n\nCo-authored-by: Niall Pemberton \u003cniallp@apache.org\u003e" }, { "commit": "28149ead1884345d9a60d8358b295ef09813098f", "tree": "1d9f03d6c11845e1de6a4614375477e6c72ddac2", "parents": [ "ef8bc7629e8b72e1855adbae9e1f7fbba2c12bb2" ], "author": { "name": "Shekhar Sharma", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Mon Feb 24 18:28:06 2025 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Feb 24 18:28:06 2025 -0800" }, "message": "[Docs] Add Shekhar Sharma to committers list and GPG keys" }, { "commit": "ef8bc7629e8b72e1855adbae9e1f7fbba2c12bb2", "tree": "ef813c430210d3bd3c6a54a8007152e9084cde34", "parents": [ "63c86b5e661b0a0a9a0b33a0851379b8786ee36e" ], "author": { "name": "ajo thomas", "email": "ajo.thomas24@gmail.com", "time": "Wed Dec 18 16:34:43 2024 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Dec 18 16:34:43 2024 -0800" }, "message": "Clear local logged stores if input checkpoints are empty (#1713)\n\n" }, { "commit": "63c86b5e661b0a0a9a0b33a0851379b8786ee36e", "tree": "c5095a3d91a60d35c4d059dba27f1813f14a160a", "parents": [ "0cf9f9a8d44544170919464357310bc2d41a2e11" ], "author": { "name": "Haolan Ye", "email": "yehaolan@gmail.com", "time": "Mon Nov 25 10:56:07 2024 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Nov 25 10:56:07 2024 -0800" }, "message": "Add new configuration allowing to keep processing when there are fatal exceptions or timeout (#1708)\n\n" }, { "commit": "0cf9f9a8d44544170919464357310bc2d41a2e11", "tree": "11b7440bcd7c2b06fa53f3b67a67fb72e0706f85", "parents": [ "d6ffeebbe4775c0b88c47c0a09fecd3e98505bd8" ], "author": { "name": "ajo thomas", "email": "ajo.thomas24@gmail.com", "time": "Tue Oct 15 14:39:43 2024 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 15 14:39:43 2024 -0700" }, "message": "Fix get future exception message (#1712)\n\n" }, { "commit": "d6ffeebbe4775c0b88c47c0a09fecd3e98505bd8", "tree": "d44a731b253ff900ea4be51d02875128e44f278c", "parents": [ "56c62677c8e2c4852afd91d14762105fb5d9421a" ], "author": { "name": "ajo thomas", "email": "ajo.thomas24@gmail.com", "time": "Wed Oct 09 15:46:09 2024 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 09 15:46:09 2024 -0700" }, "message": "Log bob deserialization exception for snapshot blobs (#1711)\n\n" }, { "commit": "56c62677c8e2c4852afd91d14762105fb5d9421a", "tree": "26b7de4f0d2baa6142b75f32c8de755b7fb796a0", "parents": [ "a52ef9db29bf9c7d6ff2d149a4a46e7d703e5533" ], "author": { "name": "ajo thomas", "email": "ajo.thomas24@gmail.com", "time": "Tue Oct 01 12:33:18 2024 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 01 12:33:18 2024 -0700" }, "message": "Update versions and use io.github.rburgst instead of com.burgstaller for okhttp-digest (#1709)\n\n" }, { "commit": "a52ef9db29bf9c7d6ff2d149a4a46e7d703e5533", "tree": "a846b0797b7b3ef7c31448d5bdb550c4e56ada0e", "parents": [ "2a94cadae5e1884c5d65526b9f2251dcb23bda90" ], "author": { "name": "Haolan Ye", "email": "yehaolan@gmail.com", "time": "Mon Aug 26 11:41:16 2024 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 26 11:41:16 2024 -0700" }, "message": "Add Default Configuration for RocksDB MaxManifestFileSize (#1706)\n\n* add default configuration for rocksdb max manifest file size\r\n\r\n* update doc, clarify the configuration\r\n\r\n* create constant for default store name\r\n\r\n* address minor comments\r\n\r\n* table format\r\n\r\n* one-line" }, { "commit": "2a94cadae5e1884c5d65526b9f2251dcb23bda90", "tree": "5c03e6f815ba3f26155455a2b40381c41fd1c356", "parents": [ "3cb57c22784448b991d1ef4e3a2928f1901b0f3a" ], "author": { "name": "Ayush Khandelwal", "email": "ayushkhandelwal93@gmail.com", "time": "Fri Aug 23 03:40:04 2024 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 22 15:10:04 2024 -0700" }, "message": "New interface \u0026 factory for AzureHTTPClient providers (#1700)\n\n* New interface for AzureHTTPClient providers\r\n\r\n* Added license headers\r\n\r\n* Updated method spec definitions\r\n\r\n* Updated spec headers doc\r\n\r\n* Updated config property in readme file\r\n\r\n* Modified doc of method\r\n\r\n* Fixed checkstyle failures" }, { "commit": "3cb57c22784448b991d1ef4e3a2928f1901b0f3a", "tree": "247b37f4e8a9d37aa28462e17b014a779843c569", "parents": [ "50280c60e3835c922e139c2853a227d1cce1af8f" ], "author": { "name": "Xinyu Liu", "email": "xinyuliu.us@gmail.com", "time": "Wed Aug 21 14:28:05 2024 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 21 14:28:05 2024 -0700" }, "message": " SAMZA-2801: fix the case when waterrmark is not aggregated when quorum is not met (#1707)\n\n" }, { "commit": "50280c60e3835c922e139c2853a227d1cce1af8f", "tree": "da3e36da019cf1ea54a08dd8f372ca4cabe0eb59", "parents": [ "5b4b1b70b824c1d9ab97428b1bf0c50c1cf7818b" ], "author": { "name": "Xinyu Liu", "email": "xinyuliu.us@gmail.com", "time": "Fri Aug 16 09:48:07 2024 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 16 09:48:07 2024 -0700" }, "message": "SAMZA-2801: Support excluding tasks from watermark computation when exceeding idle time (#1705)\n\n" }, { "commit": "5b4b1b70b824c1d9ab97428b1bf0c50c1cf7818b", "tree": "8f9a20944bf54e5d51635ff68bac5bb2c8d4fd48", "parents": [ "17e552cd59b04d899f72203f11cda284ca1fa138" ], "author": { "name": "Michael Barskii", "email": "mikebarskiy@gmail.com", "time": "Thu Aug 08 11:05:33 2024 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 08 11:05:33 2024 -0700" }, "message": "Improve Samza AM retry count logging (#1701)\n\n* LISAMZA-43659 Improve Samza AM retry count logging\r\n\r\n* Output Current Timestamp at run-class.sh script (#1702)\r\n\r\n* print current timestamp\r\n\r\n* Fix typo\r\n\r\n* fix build issue about grolifant okhttp\r\n\r\n---------\r\n\r\nCo-authored-by: Haolan Ye \u003chye@linkedin.com\u003e\r\n\r\n* Revert \"Output Current Timestamp at run-class.sh script (#1702)\"\r\n\r\nThis reverts commit 1e84ac05eb70d7b1880bb2a62a03e5e1a4b3ef16.\r\n\r\n---------\r\n\r\nCo-authored-by: Michael Barskii \u003cmbarskii@linkedin.com\u003e\r\nCo-authored-by: Haolan Ye \u003cyehaolan@gmail.com\u003e\r\nCo-authored-by: Haolan Ye \u003chye@linkedin.com\u003e" }, { "commit": "17e552cd59b04d899f72203f11cda284ca1fa138", "tree": "efd994bed41d733f977c84bf27a993cbace62c85", "parents": [ "f9c3241b87ce5d7a568368d3ffec5dea174f7692" ], "author": { "name": "Haolan Ye", "email": "yehaolan@gmail.com", "time": "Thu Aug 08 08:10:40 2024 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 08 08:10:40 2024 -0700" }, "message": "BUG FIX: set target file size multiplier (#1704)\n\n" }, { "commit": "f9c3241b87ce5d7a568368d3ffec5dea174f7692", "tree": "a5cb47ce6a501225798e3143b5e8848c4b6411f2", "parents": [ "2eb556a5bcfb4aff83f3ba00fc221108d6cba0b2" ], "author": { "name": "Haolan Ye", "email": "yehaolan@gmail.com", "time": "Fri Aug 02 16:43:11 2024 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 02 16:43:11 2024 -0700" }, "message": "Output Current Timestamp at run-class.sh script (#1702)\n\n* print current timestamp\r\n\r\n* Fix typo\r\n\r\n* fix build issue about grolifant okhttp\r\n\r\n---------\r\n\r\nCo-authored-by: Haolan Ye \u003chye@linkedin.com\u003e" }, { "commit": "2eb556a5bcfb4aff83f3ba00fc221108d6cba0b2", "tree": "a3a7325d91a3e8dd8400d960cee2548bdd06d521", "parents": [ "fb9f5cdcb48ab4ab8c3e309395440bc31152cb0b" ], "author": { "name": "Shekhar Sharma", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Fri Apr 05 10:38:42 2024 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Apr 05 10:38:42 2024 -0700" }, "message": "Create store directory paths in CSM constructor for disk space monitor (#1697)\n\n* Create store directory paths in CSM constructor to be able to monitor the disk usage of the store directories\r\n\r\n* Fix stylecheck issues\r\n\r\n* Refactor - init all store paths together and do not mutate the storeDirPaths. Added test\r\n\r\n* Remove ununsed method\r\n\r\n* Remove ununsed method\r\n\r\n* Stylecheck, Remove ununsed import" }, { "commit": "fb9f5cdcb48ab4ab8c3e309395440bc31152cb0b", "tree": "5fe01d79a6170e1270037b9fd7c71ae7feebf304", "parents": [ "93b982840a6beba8ba8a48c5c7b4645385349b07" ], "author": { "name": "Bharath Kumarasubramanian", "email": "bharathkk@apache.org", "time": "Mon Jan 22 13:36:41 2024 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jan 22 13:36:41 2024 -0800" }, "message": "SAMZA-2799: Remove worker.opts handling in shell command builder (#1696)\n\n" }, { "commit": "93b982840a6beba8ba8a48c5c7b4645385349b07", "tree": "69d3ea3bd9aa43a08e784cc3d129fc8dff8a1d94", "parents": [ "082899d85c9031c2a75eae9dd01675a78d944634" ], "author": { "name": "Daniel Chen", "email": "xrchen@uwaterloo.ca", "time": "Fri Jan 19 18:46:09 2024 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jan 19 18:46:09 2024 -0800" }, "message": "SAMZA-2784: Remove excessive commit logs (#1695)\n\n" }, { "commit": "082899d85c9031c2a75eae9dd01675a78d944634", "tree": "e5d97f154a326d3f20502d594c05680a69644bcd", "parents": [ "8ed3572bee37c04481be7c831fa455a8305f3fe8" ], "author": { "name": "Shekhar Sharma", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Fri Dec 08 16:40:47 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Dec 08 16:40:47 2023 -0800" }, "message": "Add MAX_BACKGROUND_JOBS config for RocksDB (#1694)\n\n" }, { "commit": "8ed3572bee37c04481be7c831fa455a8305f3fe8", "tree": "94583aa96a33e1fa086a64b61bdb5ee001bbf0ec", "parents": [ "e1816f3e7f09c27b3642a3e62a356198feb020f7" ], "author": { "name": "Bharath Kumarasubramanian", "email": "bharathkk@apache.org", "time": "Wed Nov 22 13:49:33 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 22 13:49:33 2023 -0800" }, "message": "SAMZA-2798: Populate worker.opts in environment variable only if available (#1693)\n\nDescription\r\nPopulate worker.opts in the environment variable only if available in the configs.\r\n\r\nChanges\r\nCheck if worker.opts is present and then add it to environment variable\r\n\r\nTests\r\nUpdated unit tests\r\n" }, { "commit": "e1816f3e7f09c27b3642a3e62a356198feb020f7", "tree": "3c3c63369ff405ac2b25e2cd0ad61802c669bb3c", "parents": [ "66495b677a728ff75a8674b217672cd51aece640" ], "author": { "name": "ajo thomas", "email": "ajo.thomas24@gmail.com", "time": "Tue Nov 21 17:21:29 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Nov 21 17:21:29 2023 -0800" }, "message": "SAMZA-2797: Call flush during stop from CoordinatorStreamWriter (#1692)\n\n" }, { "commit": "66495b677a728ff75a8674b217672cd51aece640", "tree": "c63d6a333da6971745055f34bb9467a87c665850", "parents": [ "65f31eb6e7da19a39b082635d20f730059aac8cb" ], "author": { "name": "Bharath Kumarasubramanian", "email": "bharathkk@apache.org", "time": "Tue Nov 21 11:56:08 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Nov 21 11:56:08 2023 -0800" }, "message": "SAMZA-2796: Introduce config knob for framework thread sub DAG execution (#1691)\n\nDescription\r\nAs part of SAMZA-2781, we use framework thread pool to execute hand-offs and sub-DAG execution. We want to add a config knob to enable users opt-in to the feature as opposed to enable it by default.\r\n\r\nChanges\r\nIntroduce config knob to use the framework executor\r\n\r\nTests\r\nAdded unit tests\r\n\r\nUsage Instructions\r\nRefer to the configuration documentation. To enable framework thread pool for sub-DAG execution and message hand off, set job.operator.framework.executor.enabled to true\r\n" }, { "commit": "65f31eb6e7da19a39b082635d20f730059aac8cb", "tree": "41fd38cb6caef0c0a3ba8ec18dfb353d0f404826", "parents": [ "2e91dfeff642f75192f28be33071fcb9ce443bb4" ], "author": { "name": "Bharath Kumarasubramanian", "email": "bharathkk@apache.org", "time": "Mon Nov 20 14:02:51 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Nov 20 14:02:51 2023 -0800" }, "message": "SAMZA-2763: Support worker JVM opts for Samza Beam portable mode (#1689)\n\nSummary: Support JVM options for worker process in Samza Beam portable mode\r\nDescription: With portable mode support for Samza Beam, we want to tune and configure the JVM options for worker process. In this PR, we add support by introducing worker.opts configuration and autosizing integration support.\r\n\r\nChanges:\r\n- Added worker.opts configuration\r\n- Add autosizing integration support for Xmx\r\n- Updated configuration table and website\r\n\r\nAPI Changes: None\r\n\r\nUsage Instructions: worker.opts can be used similar to other samza application configuration although it only applies to Samza Beam portable execution mode and is ignored otherwise.\r\n\r\nUpgrade Instructions: None" }, { "commit": "2e91dfeff642f75192f28be33071fcb9ce443bb4", "tree": "df0511bc2449cb5d687c652cb02013db689fc4b7", "parents": [ "24e530d7ec498f2cbb971e7b4a01a6768fd43198" ], "author": { "name": "Bharath Kumarasubramanian", "email": "bharathkk@apache.org", "time": "Wed Nov 08 10:27:55 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 08 10:27:55 2023 -0800" }, "message": "SAMZA-2795: Set thread to daemon thread for operator executor service (#1690)\n\nDescription\r\nAs part of SAMZA-2781, we introduced operator executors to manage operator handoff execution. However, the threads created by the executor service are non-daemon and hence prevent the JVM from shutting down.\r\n\r\nFor context, we don\u0027t have a clean way to shutdown the executor due to lack of clean lifecycle management of the factory. Hence shutting down the executor service within TaskInstance is not an option and the fix is to make it daemon threads.\r\n\r\nChanges\r\nMake the threads spawned by the operator executor to be non-daemon\r\n\r\nTests\r\nNone\r\n\r\nAPI Changes\r\nNone\r\n\r\nUpgrade Instructions\r\nNone\r\n\r\nUsage Instructions\r\nNone" }, { "commit": "24e530d7ec498f2cbb971e7b4a01a6768fd43198", "tree": "0db6ba2222e34e1957c6267c05b1705626785517", "parents": [ "c43f423356e8cb0f487b5211be00896c08a5caca" ], "author": { "name": "Jun Guan", "email": "guanjun0215@gmail.com", "time": "Mon Sep 11 16:48:58 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 11 16:48:58 2023 -0700" }, "message": "Unique System.exit in ContainerLaunchUtil (#1686)\n\n" }, { "commit": "c43f423356e8cb0f487b5211be00896c08a5caca", "tree": "78c88b7cca5d2b1718d0739284b1770b5625f31e", "parents": [ "65bf4ae058d4d674cc63c3f172f3f348a4513640" ], "author": { "name": "Jon Bringhurst", "email": "jon@bringhurst.org", "time": "Wed Aug 30 18:01:15 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 30 18:01:15 2023 -0700" }, "message": "SAMZA-2774: Resolve classpath symlinks (#1653)\n\nManifest generation (application-level) is pointing to container-level symlinks instead of application-level files.\r\n\r\nThis will cause issues when multiple containers for an application land on a single host and the container the manifest is pointing to gets cleaned up.\r\n\r\n[yyy@zzz filecache]$ cat /data/appcache/application_1668550466222_0989/filecache/10/xxx-0.0.672.tgz/classpath_workspace/manifest.txt|tail -n 2\r\n /data/appcache/application_1668550466222_0989/container_e114_1668550466222_0989_01_000003/__package/lib/zstd-jni-1.5.2-3.jar\r\n\r\n[yyy@zzz filecache]$\r\nThis patch is to resolve those symlinks to their application level paths." }, { "commit": "65bf4ae058d4d674cc63c3f172f3f348a4513640", "tree": "457652cb01988237492d8e30ac2ced71fd1d291d", "parents": [ "aeeaf0b9782d03932da4ee3da9e15ef9e28ed313" ], "author": { "name": "Bharath Kumarasubramanian", "email": "bharathkk@apache.org", "time": "Tue Aug 29 14:49:50 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 29 14:49:50 2023 -0700" }, "message": "SAMZA-2791: Introduce callback timeout specific to watermark messages (#1681)\n\nDescription:\r\nCurrently, watermark is implemented as a special message within Samza. However, in terms of processing semantics, it shares similar behavior to normal messages processed by the task. i.e., task.callback.timeout.ms, a configuration to tune the time until which runloop waits for a message to be processed applies to both watermark and normal messages.\r\n\r\nHowever, this tie up constrains watermark processing logic to be bounded by the processing messages time bound. For Beam on Samza, we use watermark as a trigger to execute event timers which can take a long time depending on the number of timers accumulated. Especially, when the application is down, the timers accumulated could be too large and users will have to tune this configuration which will also impact fault tolerance behavior in case of failures/delays during processing messages.\r\n\r\nChanges:\r\n- Introduce callback timeout configuration specific to watermark\r\n- Update configuration documentation\r\n- Consolidate overload methods for TaskCallbackManager\r\n- Always use watermark specific timeout even when run loop is in draining mode\r\n\r\nAPI Changes:\r\n- Internal change to constructor\r\n\r\nUpgrade Instructions: None\r\n\r\nUsage Instructions:\r\n- Users can configure the timeout for watermark messages using task.callback.watermark.timeout.ms\r\n- Refer to the configuration documentation for more details and defaults." }, { "commit": "aeeaf0b9782d03932da4ee3da9e15ef9e28ed313", "tree": "d664670a3d73dc63a680f6499c1749c3e8c41eae", "parents": [ "fa4008ee9c3f6b9499dfdceca1427b76a665420c" ], "author": { "name": "Bharath Kumarasubramanian", "email": "bharathkk@apache.org", "time": "Tue Aug 29 08:49:32 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 29 08:49:32 2023 -0700" }, "message": "SAMZA-2790: Cleanup RunLoop constructor explosion (#1680)\n\nDescription:\r\nRunloop currently takes in lot of parameters and the constructor has grown to the point where it is unmanageable with multiple overloads. Introducing new configuration requires lot of updates to existing tests and components even if the parameters have no effect on all of the usages.\r\n\r\nWith this PR, we should be able to decouple different users of RunLoop and enable these components to have their own scoped config. e.g., SideInputManager can now have its own set of runloop parameters without having to tie itself with TaskConfig.\r\n\r\nChanges:\r\nIntroduce RunLoopConfig, a container object to hold all required parameters for runloop from Config.\r\nRemove existing overloads of constructor\r\nSimplify the constructor to take RunLoopConfig and initialize the necessary components and fields\r\nIntroduce SideInputManagerRunLoopConfig, an overload of RunLoopConfig to be used within SideInputManager\r\nModify RunLoopFactory create method signature\r\nClean up ApplicationUtil and moved the method to ApplicationConfig and added unit tests\r\n\r\nAPI Changes:\r\nNo external API change" }, { "commit": "fa4008ee9c3f6b9499dfdceca1427b76a665420c", "tree": "895a93f4b4ab3c508e99cec8b1ab813b764270c6", "parents": [ "0e8ac2e6d7ae4ee88538c508a162c5e855a36135" ], "author": { "name": "Shekhar Sharma", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Wed Aug 23 15:45:23 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 23 15:45:23 2023 -0700" }, "message": "Ignore file owners comparison on restore when config is set (#1684)\n\n* Ignore file owners comparison on restore when config is set\r\n\r\n* Minor updates\r\n\r\n* Style fix in tests\r\n\r\n---------\r\n\r\nCo-authored-by: Shekhar Sharma \u003cshekhar.sharmaa@gmail.com\u003e" }, { "commit": "0e8ac2e6d7ae4ee88538c508a162c5e855a36135", "tree": "9b815e4eb14958aacbab2c2852ced88cb525090d", "parents": [ "1373db238683179123a15b246272cb0955f9b37c" ], "author": { "name": "shekhars-li", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Fri Aug 18 11:25:51 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 18 11:25:51 2023 -0700" }, "message": "Close TaskRestoreManager only after all restores are complete (#1682)\n\nClose TaskRestoreManager only after all restores are complete" }, { "commit": "1373db238683179123a15b246272cb0955f9b37c", "tree": "1a4878a5cac480e965f6a103196fc43c484a95e3", "parents": [ "3ce67acb0494a47c167f938323917c0bbca32b7d" ], "author": { "name": "shekhars-li", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Wed Aug 09 18:42:46 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 09 18:42:46 2023 -0700" }, "message": "[SAMZA-2787] GetDeleted API and Recover from DeletedException (#1676)\n\n* GetDeleted API and Recover from DeletedException in commit for BlobStoreBackendFactory\r\n\r\n* Fix style issues, fix breaking test cases\r\n\r\n* Fixed test failure\r\n\r\n* Fix failing integration test - move storeConsumer start after init() in ContainerStorageManager\r\n\r\n* Style check fix\r\n\r\n* Bug fixes, integration test\r\n\r\n* Delete all blob from deleted SnapshotIndex\r\n\r\n* clean up for final review\r\n\r\n* Add integ test - delete snapshotindex and recover\r\n\r\n* Fix failing integration tests related to init()\r\n\r\n* Cleanup - remove unused code and move some code to util methods\r\n\r\n* Unit test, minor refractoring\r\n\r\n* Review comments - 1st round\r\n\r\n* Refractor code\r\nRemove taskcheckpoints mutation\r\n\r\n* Address review comment\r\n\r\n* Address review comment part 2\r\n\r\n* Fix failing CSM test\r\n\r\n* Addressed final review comments - more logs, consistent naming\r\n\r\n* Checkstyle fix for test class\r\n\r\n---------\r\n\r\nCo-authored-by: Shekhar Sharma \u003cshesharm@shesharm-mn1.linkedin.biz\u003e\r\nCo-authored-by: Shekhar Sharma \u003cshekhar.sharmaa@gmail.com\u003e" }, { "commit": "3ce67acb0494a47c167f938323917c0bbca32b7d", "tree": "d7b14c030290f09453b8f6722ea345708ccc8761", "parents": [ "9357a9101c7000296a3fb1cac576e4988db92169" ], "author": { "name": "Bharath Kumarasubramanian", "email": "bharathkk@apache.org", "time": "Fri Aug 04 14:44:10 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 04 14:44:10 2023 -0700" }, "message": "SAMZA-2789: Remove cap on intermediate stream partition count for stream mode (#1679)\n\nProblem: Intermediate stream partition count inference logic caps the partition size to 256 resulting in imbalances in work assignments to tasks\r\n\r\nDescription: As part of the intermediate partition size inference logic, we currently employ the following algorithm.\r\n\r\npartitionCount \u003d Math.max(maxPartitionSize(inputStreams), maxPartitionSize(outputStreams))\r\ncap the partitionCount to MAX_INFERRED_PARTITIONS defined in the IntermediateStreamManager which is 256\r\napply the inferred partition count to intermediate streams whose partition count is uninitialized\r\nThe logic above always caps the partition size of intermediate streams to 256 for all auto-created intermediate streams. This can prevent the job from scaling up uniformly as the intermediate partition assignment is capped to 256 tasks thereby rendering other tasks imbalanced in case of number tasks \u003e 256.\r\n\r\nChanges:\r\n\r\nApply the cap only for batch mode as 256 limit was introduced for batch mode where number of files (partition) could be large\r\nAdd unit tests for IntermediateStreamManager\r\nMinor java doc fix for DefaultTaskExecutorFactory\r\nTests: Added unit tests for the code changes\r\n\r\nAPI Changes: None\r\n\r\nUpgrade Instructions:\r\n\r\nJobs that are temporarily worked around this constraint by setting job.intermediate.stream.partitions should remove the configuration in order for samza to infer and apply the partition count as described above\r\nJobs that don\u0027t use job.intermediate.stream.partitions need no changes.\r\nUsage Instructions: Refer to upgrade instruction." }, { "commit": "9357a9101c7000296a3fb1cac576e4988db92169", "tree": "60f9107fb0dfe2bb987770eddaa10fff2c7f74ec", "parents": [ "7dee9977a4cc689d80376c1f53817af90aa5accb", "b78b66c540a7a2dc17808c501cfa5bc621e5f13f" ], "author": { "name": "Boris Shkolnik", "email": "boryas@apache.org", "time": "Fri Jul 28 11:52:16 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 28 11:52:16 2023 -0700" }, "message": "Merge pull request #1678 from Zhangyx39/master\n\nSAMZA-2788: Add new metric to emit 1 when containers are processing" }, { "commit": "b78b66c540a7a2dc17808c501cfa5bc621e5f13f", "tree": "60f9107fb0dfe2bb987770eddaa10fff2c7f74ec", "parents": [ "ec7d105eef0125e52e328f1c073e01235d758d06" ], "author": { "name": "Yixing Zhang", "email": "tommykuaidlaa@gmail.com", "time": "Thu Jul 27 15:45:20 2023 -0700" }, "committer": { "name": "Yixing Zhang", "email": "tommykuaidlaa@gmail.com", "time": "Thu Jul 27 15:45:20 2023 -0700" }, "message": "add comment\n" }, { "commit": "ec7d105eef0125e52e328f1c073e01235d758d06", "tree": "751b1c812a290b348ee04a4c0f2c7c5a8751ccd5", "parents": [ "f53c6f9e3c013b3f930afc4670e976f64b3428b6" ], "author": { "name": "Yixing Zhang", "email": "tommykuaidlaa@gmail.com", "time": "Thu Jul 27 14:46:51 2023 -0700" }, "committer": { "name": "Yixing Zhang", "email": "tommykuaidlaa@gmail.com", "time": "Thu Jul 27 14:46:51 2023 -0700" }, "message": "fix unit test\n\nThe containerRunning metric is set by the callback function\n" }, { "commit": "f53c6f9e3c013b3f930afc4670e976f64b3428b6", "tree": "661641e878aba6d071b20fc6a2ba25cd4d439b38", "parents": [ "7dee9977a4cc689d80376c1f53817af90aa5accb" ], "author": { "name": "Yixing Zhang", "email": "tommykuaidlaa@gmail.com", "time": "Thu Jul 27 11:17:27 2023 -0700" }, "committer": { "name": "Yixing Zhang", "email": "tommykuaidlaa@gmail.com", "time": "Thu Jul 27 11:17:27 2023 -0700" }, "message": "Add new metric to emit 1 when containers are processing\n\nThis allows dashboards to calculate how many containers are running / not running based on this metric.\n" }, { "commit": "7dee9977a4cc689d80376c1f53817af90aa5accb", "tree": "17eb78d5bb811c277550ae614dbcde61d35e34b8", "parents": [ "f8560047677c7661c4a69b49541464f73148f1b4" ], "author": { "name": "jia-gao", "email": "94939653+jia-gao@users.noreply.github.com", "time": "Tue Jul 18 13:13:59 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 18 13:13:59 2023 -0700" }, "message": "Add new SamzaApplicationMaster metric to track allocated containers buffered in AM (#1677)\n\n* Add new SamzaApplicationMaster metric to track containers allocated by RM and buffered in AM\r\n\r\n* update TestApplicationMasterRestClient\r\n\r\n* Add allocated-containers-in-buffer in metrics doc" }, { "commit": "f8560047677c7661c4a69b49541464f73148f1b4", "tree": "5044759353e31fc9745ae0c34f9b55f7df92cb99", "parents": [ "aa5db44e25e87e84d7fc32dc50ba51347290acd8" ], "author": { "name": "Clay Johnson", "email": "cjohnson@gradle.com", "time": "Tue Jun 20 13:09:08 2023 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jun 20 11:09:08 2023 -0700" }, "message": "Authenticate GitHub Actions builds to ge.apache.org (#1672)\n\nThis change allows GitHub Actions builds to submit build scans to ge.apache.org by authenticating those builds. The access key has been stored as an organizational secret by the ASF Infrastructure team in the Apache GitHub organization. The access key is not available to workflows triggered from forks.\r\n\r\nThis builds on the changes in https://github.com/apache/samza/pull/1665" }, { "commit": "aa5db44e25e87e84d7fc32dc50ba51347290acd8", "tree": "737c848088909753eea4a974667d993f6f146307", "parents": [ "cc21b6ebebf829b8c7a47256d2717fc9250ba9c7" ], "author": { "name": "Eric Honer", "email": "ehoner@linkedin.com", "time": "Fri Jun 16 19:20:58 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jun 16 19:20:58 2023 -0700" }, "message": "SAMZA-2778: Make AzureBlobOutputStream buffer initialization size configurable. (#1662)\n\nSymptom:\r\nJVM crashes due to OOM exceptions. When the system has a large number of AzureBlobWriterObjects, the memory becomes heavily fragmented and can be susceptible to crashing when a new AzureBlobOutputStream is created.\r\n\r\nCause:\r\nThe crashes are caused by inefficient memory management when using the G1GC. (The default garbage collector in Java 11+.) What causes (code paths) this issue and why this is a problem for the G1GC are examined separately.\r\n\r\nWhat causes this issue?\r\nThe underlying issue is caused by the AzureBlobOutputStream. When a new instance is created, it creates a new ByteArrayOutputStream initialized to maxFlushThresholdSize.12 The ByteArrayOutputStream is the buffer used by the parent to accumulate messages between flush intervals. It requires 10MB (current default) of memory to initialize. This allows the buffer to accumulate messages without resize operations, however, it does not prevent resizing. The maxFlushThresholdSize is enforced in the parent during write() - see AzureBlobAvroWriter.\r\n\r\nWhy this is a problem for the G1GC?\r\nThe focus here is on the G1GC and humongous objects (G1 specific).3 The G1 GC introduced a new memory management strategy that divides the heap into regions, -XX:G1HeapRegionSize\u003dn. The GC can operate on regions concurrently and copies live objects between regions during full GC to reclaim/empty regions.4 The default behavior creates ~2048 regions, sized to a factor of 2 between 1MB and 32MB. Any object larger than half of a region size, is considered a humongous object.\r\nHumongous objects are allocated an entire region (or consecutive regions if larger than a single region) and are not copied between regions during full GC. Any space not used by the object within a region is non-addressable for the life of the object.5 A JVM heap size of 31GB, -Xmx31G, will default to 16MB regions. Considering the current default size is 10MB, each buffer requires an entire region and prevents the use of 6MB, regardless of the how much data is in the buffer. For a heap smaller than 16GB, each buffer would require multiple regions.\r\nThe 10MB buffer size can exhaust the regions and cause OOMs or create fragmentation that causes OOMs. A fragmentation caused OOM occurs in the following sequence. On new, the JVM attempts to create the object in Eden. If there is insufficient space in Eden a minor GC is performed. If there is insufficient space after minor GC, the object is immediately promoted Old Gen. If there is insufficient space in Old Gen, a full GC is performed. If a full GC cannot allocate memory or region(s) for a humongous object the JVM will exit with OOM.\r\n\r\nChanges:\r\nThe javadocs, where appropriate, have been updated to reflect changes or describe new behaviors. No public APIs were removed, they were marked deprecated and migrated to the new default initialization value. All of the changes are itemized below.\r\n\r\nAzureBlobConfig\r\nAdding two new public fields and one public method. The new configuration is made accessible in the same manner as existing configs (see #configs SEP-26), also consistent the coding-guide. There is a new public config key: SYSTEM_INIT_BUFFER_SIZE - named initBufferSize.bytes. The default value is public field SYSTEM_INIT_BUFFER_SIZE_DEFAULT. The user provided configuration value is accessible with new public method getInitBufferSizeBytes(..). The method returns the configuration value between 32 and getMaxFlushThresholdSize(..) inclusive. 32 is the default initialization size of a ByteArrayOutputStream in the parameterless constructor.\r\n\r\nAzureBlobWriterFactory\r\nThere are two changes to this interface, both to the method, getWriterInstance(..). The existing implementation is marked @Deprecated and a new method with an additional parameter is added. The new parameter is an int that is expected to be the _ initBufferSize_.\r\n\r\nAzureBlobAvroWriterFactory\r\nThe modifications here are consistent with the changes to interface AzureBlobWriterFactory. However, the deprecated implementation uses the new field AzureBlobConfig.SYSTEM_INIT_BUFFER_SIZE_DEFAULT when calling the new public API. This will migrate users to the new initialization behavior.\r\n\r\nAzureBlobAvroWriter\r\nThere are two public, one package-private, and two private changes. Both public changes are to constructors. The existing public constructor is marked deprecated and invokes the new public constructor with the new field AzureBlobConfig.SYSTEM_INIT_BUFFER_SIZE_DEFAULT. The new constructor sets the new private field initBufferSize. The package-private constructor is modified with an additional int parameter and the tests were changed accordingly. The remaining private change modifies creation of new AzureBlobOutputStream instances to include the additional private field initBufferSize.\r\n\r\nAzureBlobOutputStream\r\nThe existing public API is marked @Deprecated and the ByteArrayOutputStream is initialized with the new field AzureBlobConfig.SYSTEM_INIT_BUFFER_SIZE_DEFAULT. The new public constructor includes the new parameter initBufferSize and initializes the ByteArrayOutputStream to that size.\r\n\r\n" }, { "commit": "cc21b6ebebf829b8c7a47256d2717fc9250ba9c7", "tree": "54fb0e2e1d69fed9c68bdaf529c5f978ffb3d67c", "parents": [ "d138757f7e163673fe9f80dd39131d4b1d3219f8" ], "author": { "name": "Andy Sautins", "email": "github@sautins.com", "time": "Thu Jun 15 10:24:41 2023 -0600" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jun 15 09:24:41 2023 -0700" }, "message": "Fix scope of caches for DirDiffUtil.areSameFile (#1671)\n\nSummary\r\nFix scope of owner and group name caches in DirDiffUtil.areSameFile\r\n\r\nDetail\r\nIn #1669 the caches were scoped incorrectly and will be re-created each time areSameFile called. Scoping the group and name caches outside of the lambda to cache as expected.\r\n\r\nTest\r\nAdded unit test TestDirDiffUtilAreSameFile.testAreSameFile_Cache to verify the caches are working as expected.\r\n./gradlew check" }, { "commit": "d138757f7e163673fe9f80dd39131d4b1d3219f8", "tree": "7b2a803ecb515bd10c5ce0316160a02b923c2f82", "parents": [ "03d70a5a19398b8087ab9e2de5319d027438e56f" ], "author": { "name": "Andy Sautins", "email": "github@sautins.com", "time": "Wed Jun 14 13:11:58 2023 -0600" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jun 14 12:11:58 2023 -0700" }, "message": "Re-factor DirDiffUtil.getDirDiff to minimize calls to sun.nio.fs.* (#1669)\n\nRe-factor DirDiffUtil.getDirDiff to minimize calls to sun.nio.fs.*" }, { "commit": "03d70a5a19398b8087ab9e2de5319d027438e56f", "tree": "110d500662d420b09b3452381924fa71dccebe29", "parents": [ "9111bb744a9c27cd4a568576eaa0eadd7dc0cb88" ], "author": { "name": "Eric Honer", "email": "ehoner@linkedin.com", "time": "Wed Jun 14 07:48:46 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jun 14 07:48:46 2023 -0700" }, "message": "- Fix parameter ReceiverOptions parameters usaged (#1670)\n\n- Change powermock parameters" }, { "commit": "9111bb744a9c27cd4a568576eaa0eadd7dc0cb88", "tree": "a9e77ad779029f460ab9e2809ffa93100837cd3a", "parents": [ "4600dd3dd0497fbffacbea21e5b7b51b2b0209bb" ], "author": { "name": "Bharath Kumarasubramanian", "email": "bharathkk@apache.org", "time": "Tue Jun 13 16:57:17 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jun 13 16:57:17 2023 -0700" }, "message": "SAMZA-2781: Use framework thread to execute hand-offs and sub-DAG execution (#1667)\n\nDescription:\r\nCurrently, the operator implementation chains the future using synchronous APIs (thenCompose, thenApply) which results in execution of these method calls on the future completion thread which happens to be the user thread in case of asynchronous operators in application DAG.\r\n\r\nChanges:\r\nUse thread pool inject through job.container.task.executor.factory\r\nExtend the task executor factory to return operator executor \r\nWire in the task executor in TaskContext\r\nDefault implementation which uses #getTaskExecutor if enabled and job.container.thread.pool.size \u003e 1 or fallback to single threaded executor otherwise.\r\n\r\nTesting:\r\nUnit tests for the factory\r\nTest updates for existing OperatorImpl tests.\r\n\r\nAPI Changes:\r\nAdd getOperatorExecutor to TaskExecutorFactory\r\nProvide a default implementation that reuses getTaskExecutor\r\n\r\nUsage Instructions:\r\nRefer to the config documentation to enable operator thread pool and about the task executor factory\r\n\r\nUpgrade Instructions: \r\nNone" }, { "commit": "4600dd3dd0497fbffacbea21e5b7b51b2b0209bb", "tree": "59949cfd752b66d7ef7a727705c1391a5d05cd44", "parents": [ "7020e3a8d4579d88b5faf923ce542423e1837b01" ], "author": { "name": "Bharath Kumarasubramanian", "email": "bharathkk@apache.org", "time": "Tue Jun 13 13:17:47 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jun 13 13:17:47 2023 -0700" }, "message": "SAMZA-2782: Clean up container thread pool DI into StreamOperatorTask (#1668)\n\nDescription:\r\nPrior to TaskContext and ContainerContext, the container thread pool was injected into StreamOperatorTask using TaskFactoryUtil and round about ways.\r\nNow that we have these contexts, it is easier to inject and simplify the DI.\r\n\r\nChanges:\r\n\r\nAdd container thread pool to ContainerContext\r\nClean up TaskFactoryUtil and other remnants of previous DI\r\nDeprecate StreamOperatorTaskFactory\r\nTests:\r\n\r\nUnit tests\r\nUpdated existing tests\r\nAPI Changes:\r\nAdded getContainerThreadPool to ContainerContext\r\n\r\nUpgrade Instructions: None\r\n\r\nUsage Instructions: None" }, { "commit": "7020e3a8d4579d88b5faf923ce542423e1837b01", "tree": "cfa3bd1c8b1ffb4e61fec0835d47535f2f266208", "parents": [ "e75a7e6e056a5d8c141f34743079cf57b98e5267" ], "author": { "name": "Eric Honer", "email": "ehoner@gmail.com", "time": "Mon Jun 12 15:53:46 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 12 15:53:46 2023 -0700" }, "message": "SAMZA-2776: Upgrade Azure libraries in module samza-azure. (#1660)\n\n* Upgrade azure libraries" }, { "commit": "e75a7e6e056a5d8c141f34743079cf57b98e5267", "tree": "78d7b4cff266e9578918e797b62477fc2cb17575", "parents": [ "524d68705c23b94d9eecd16c83915e83e95ec67d" ], "author": { "name": "Eric Honer", "email": "ehoner@gmail.com", "time": "Mon Jun 12 15:52:44 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 12 15:52:44 2023 -0700" }, "message": "SAMZA-2777: Add support for GenericFixed to AzureBlobAvroWriter (#1661)\n\n* Add support for GenericFixed to AzureBlobAvroWriter\r\nUpdate comments" }, { "commit": "524d68705c23b94d9eecd16c83915e83e95ec67d", "tree": "d34f66d2e254e51a285504ebb1065443d825d757", "parents": [ "815dfc6386d02316b63595c734b0eb19eb0c1e5d" ], "author": { "name": "Clay Johnson", "email": "cjohnson@gradle.com", "time": "Tue May 09 19:31:45 2023 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue May 09 10:31:45 2023 -0700" }, "message": "Capture build scans on ge.apache.org to benefit from deep build insights (#1665)\n\nThis PR publishes a build scan for every CI build on Jenkins and GitHub Actions and for every local build from an authenticated Apache committer. The build will not fail if publishing fails.\r\n\r\nThe build scans of the Apache Samza project are published to the Gradle Enterprise instance at ge.apache.org, hosted by the Apache Software Foundation and run in partnership between the ASF and Gradle. This Gradle Enterprise instance has all features and extensions enabled and is freely available for use by the Apache Samza project and all other Apache projects.\r\n\r\nThis pull request enhances the functionality of publishing build scans to the publicly available scans.gradle.com by instead publishing build scans to ge.apache.org. On this Gradle Enterprise instance, Apache Samza will have access not only to all of the published build scans but other aggregate data features such as:\r\n\r\nDashboards to view all historical build scans, along with performance trends over time\r\nBuild failure analytics for enhanced investigation and diagnosis of build failures\r\nTest failure analytics to better understand trends and causes around slow, failing, and flaky tests\r\nIf interested in exploring a fully populated Gradle Enterprise instance, please explore the builds already connected to ge.apache.org, the Spring project’s instance, or any number of other OSS projects for which we sponsor instances of Gradle Enterprise.\r\n" }, { "commit": "815dfc6386d02316b63595c734b0eb19eb0c1e5d", "tree": "b1fcdf84586f073b9042e8d9a65cfed8b75a84ce", "parents": [ "03b187a6de0e123568f3ce3af94c946e6380fc8d" ], "author": { "name": "Prateek Maheshwari", "email": "prateekm@utexas.edu", "time": "Mon Apr 10 10:34:44 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Apr 10 10:34:44 2023 -0700" }, "message": "Removed unnecessary logging for configs (#1664)\n\n" }, { "commit": "03b187a6de0e123568f3ce3af94c946e6380fc8d", "tree": "9f7e3b2a1d585ffa960fb050d66fd795366da695", "parents": [ "985a6048c1e589d0b3af5d0040f26b64741c9260" ], "author": { "name": "Prateek Maheshwari", "email": "prateekm@utexas.edu", "time": "Thu Mar 23 09:28:24 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Mar 23 09:28:24 2023 -0700" }, "message": "Only verify permissions for file owner after blob store restore. (#1659)\n\n" }, { "commit": "985a6048c1e589d0b3af5d0040f26b64741c9260", "tree": "7788d2c6f3d2954de0fa152e08da6871501ed8b0", "parents": [ "d261a0b6962f4f733c742821b4dbf87f64038979" ], "author": { "name": "Prateek Maheshwari", "email": "prateekm@utexas.edu", "time": "Wed Mar 22 10:03:31 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Mar 22 10:03:31 2023 -0700" }, "message": "Changed uses of Apache Commons IO FileUtils.deleteDirectory to PathUtils.deleteDirectory (#1658)\n\nSymptom: Blob store state restore fails when verifying restored directory contents due to a mismatch between permissions of local vs remote files.\r\n\r\nCause: Apache Commons 2.11 (at least, more versions may be affected) FileUtils#deleteDirectory() has a bug where it changes the permissions of symlink/hardlink files to read-only after deletion. For more details, see https://issues.apache.org/jira/browse/IO-751. \r\n\r\nChanges: This PR changes all uses of FileUtils#deleteDirectory to PathUtils#deleteDirectory() which is not affected." }, { "commit": "d261a0b6962f4f733c742821b4dbf87f64038979", "tree": "2d0ea37044fecf84a18d29b49f3c7b7f018b66b4", "parents": [ "325e463db898f586e3570b7b713199a697400022" ], "author": { "name": "Prateek Maheshwari", "email": "prateekm@utexas.edu", "time": "Tue Mar 21 12:33:30 2023 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Mar 21 12:33:30 2023 -0700" }, "message": "Side Inputs + Blob Store Backups - Part 2 - Allow bulk restore from blob store for side input stores (#1657)\n\nAllow bulk restore from blob store for side input stores. If a blob store state backend is available, side input store restores can be sped up significantly by bulk-restoring initial state from the blob store, and only using the Kafka side input topic to catch up on the delta since last checkpoint.\r\n\r\nFixed a bug found during additional testing: in-memory stores configured with a changelog are not being restored correctly. The store instance created initially is retained but not restored, instead a new store instance is created and restored from changelog, and then thrown away." }, { "commit": "325e463db898f586e3570b7b713199a697400022", "tree": "136d3154fe50bd1f4a9dce69f05a7729eb3ab1ea", "parents": [ "cfae3e47ebfb3dffd379807e29dc0f2cde6a7d2a", "542cd48c75c6b4e697ee460e5f0724a90c0f0400" ], "author": { "name": "Prateek Maheshwari", "email": "prateekm@utexas.edu", "time": "Fri Mar 10 13:14:07 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Mar 10 13:14:07 2023 -0800" }, "message": "Merge pull request #1656 from prateekm/side-inputs-blob-store-restore-part-2\n\nFixed line-length related formatting issues" }, { "commit": "542cd48c75c6b4e697ee460e5f0724a90c0f0400", "tree": "136d3154fe50bd1f4a9dce69f05a7729eb3ab1ea", "parents": [ "cfae3e47ebfb3dffd379807e29dc0f2cde6a7d2a" ], "author": { "name": "Prateek Maheshwari", "email": "pmaheshw@linkedin.com", "time": "Fri Mar 10 12:48:29 2023 -0800" }, "committer": { "name": "Prateek Maheshwari", "email": "pmaheshw@linkedin.com", "time": "Fri Mar 10 12:48:29 2023 -0800" }, "message": "Fixed line-length related formatting issues.\n" }, { "commit": "cfae3e47ebfb3dffd379807e29dc0f2cde6a7d2a", "tree": "664d2740f0b24f8e1f0e823b9d81aea66c900d2c", "parents": [ "3b54392833efda4906a961349d99eff569f69184", "69851fbb0e3894c1d7defe995fc9263ac9680b19" ], "author": { "name": "Prateek Maheshwari", "email": "prateekm@utexas.edu", "time": "Wed Mar 08 14:18:59 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Mar 08 14:18:59 2023 -0800" }, "message": "Merge pull request #1655 from prateekm/side-inputs-blob-store-restore\n\nRefactored ContainerStorageManager for readability and manageability" }, { "commit": "69851fbb0e3894c1d7defe995fc9263ac9680b19", "tree": "664d2740f0b24f8e1f0e823b9d81aea66c900d2c", "parents": [ "5c439628022b0a35d96409f866389d77388be5df" ], "author": { "name": "Prateek Maheshwari", "email": "pmaheshw@linkedin.com", "time": "Wed Mar 08 09:15:02 2023 -0800" }, "committer": { "name": "Prateek Maheshwari", "email": "pmaheshw@linkedin.com", "time": "Wed Mar 08 09:15:02 2023 -0800" }, "message": "Updated based on PR feedback.\n" }, { "commit": "5c439628022b0a35d96409f866389d77388be5df", "tree": "f6149053250aedbda1fb2036b10af5735c4cf107", "parents": [ "9041b3a4d4d6dda1e16e509448aafcc503ddd6b2" ], "author": { "name": "Prateek Maheshwari", "email": "pmaheshw@linkedin.com", "time": "Mon Mar 06 22:42:46 2023 -0800" }, "committer": { "name": "Prateek Maheshwari", "email": "pmaheshw@linkedin.com", "time": "Mon Mar 06 22:42:46 2023 -0800" }, "message": "Refactoring ContainerStorageManager for readability and manageability.\n" }, { "commit": "3b54392833efda4906a961349d99eff569f69184", "tree": "b7144565f94f9d52b855b0b6a9994ba1599d7e36", "parents": [ "77dec99be3d40a0d111dbf22d8b49a3f102d3ba3", "9041b3a4d4d6dda1e16e509448aafcc503ddd6b2" ], "author": { "name": "Prateek Maheshwari", "email": "prateekm@utexas.edu", "time": "Mon Mar 06 13:55:20 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Mar 06 13:55:20 2023 -0800" }, "message": "Merge pull request #1654 from prateekm/master\n\nSide Inputs + Blob Store Backups - Part 1 - Fix duplicate uploads and broken metrics" }, { "commit": "9041b3a4d4d6dda1e16e509448aafcc503ddd6b2", "tree": "b7144565f94f9d52b855b0b6a9994ba1599d7e36", "parents": [ "77dec99be3d40a0d111dbf22d8b49a3f102d3ba3" ], "author": { "name": "Prateek Maheshwari", "email": "pmaheshw@linkedin.com", "time": "Mon Mar 06 11:48:42 2023 -0800" }, "committer": { "name": "Prateek Maheshwari", "email": "pmaheshw@linkedin.com", "time": "Mon Mar 06 11:48:42 2023 -0800" }, "message": "Side Inputs + Blob Store Backups - Fix duplicate uploads and broken metrics.\n" }, { "commit": "77dec99be3d40a0d111dbf22d8b49a3f102d3ba3", "tree": "f6ab40bba379b470347d3a818b4b74e226034412", "parents": [ "19e61abf59f33fd7bafee2ffa5b61aa0bf11d4d0" ], "author": { "name": "jia-gao", "email": "94939653+jia-gao@users.noreply.github.com", "time": "Wed Feb 01 13:33:05 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Feb 01 13:33:05 2023 -0800" }, "message": "Fix deadlock in StreamAppender (#1652)\n\nIssue:\r\nIn StreamAppender, there is a synchronized block around setupSystem(), which should be called the first time (after loggingContext config is set up) any logger logs something.\r\nIt could lead to a deadlock situation. Because if during the setupSystem(), any other threads try to do LOG.xxx(), they will be blocked, and if the system setup depends on those threads, it leads to deadlock\r\n\r\nChange:\r\nReplace the synchronized block with a tryLock(long time, TimeUnit unit)\r\nso that other threads won\u0027t be blocked forever if setupSystem() stuck or timeout\r\n\r\nUsage Change:\r\nIf any thread try to log during the time that another thread is setting up the stream, it will wait for SET_UP_SYSTEM_TIMEOUT_MILLI_SECONDS. If timeout, StreamAppender will skip sending its log event\r\n\r\nNote that before the change, the thread will block and wait until the stream setting up is done. This behavior opens the chance of deadlock and needs to be changed" }, { "commit": "19e61abf59f33fd7bafee2ffa5b61aa0bf11d4d0", "tree": "a3015472d9f7c6c26b77b166e02de3f5caa360a6", "parents": [ "b15b4f86bba868dec9bb20b16482f4119f2b5907", "083c85d9c16858f825841a51f8c625b2e287f187" ], "author": { "name": "Prateek Maheshwari", "email": "prateekm@utexas.edu", "time": "Wed Feb 01 11:18:43 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Feb 01 11:18:43 2023 -0800" }, "message": "Merge pull request #1650 from shekhars-li/master\n\nMake blob store related retry tunables configurable" }, { "commit": "083c85d9c16858f825841a51f8c625b2e287f187", "tree": "6f37ae5fc44fe1d21b2956fbce1acea3a1196a13", "parents": [ "dac13337d25d439249351a940dcb9357e20a6f27" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 17:27:51 2023 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 17:27:51 2023 -0800" }, "message": "Fix comment, remove accessors\n" }, { "commit": "dac13337d25d439249351a940dcb9357e20a6f27", "tree": "34aaf003c916bbf0d2820c54e40b054d4f35e2a8", "parents": [ "b8fc2db773732bc3958b7c2224a3800a28df31b0" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 16:32:13 2023 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 16:32:13 2023 -0800" }, "message": "style fix\n" }, { "commit": "b8fc2db773732bc3958b7c2224a3800a28df31b0", "tree": "8744ce0433e64ec5459e28d0c3fd70de33241802", "parents": [ "773029b1382e4133ee2243ab33e507a2c67a56ee" ], "author": { "name": "shekhars-li", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Tue Jan 31 16:28:08 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jan 31 16:28:08 2023 -0800" }, "message": "Delete twoEnter.txt" }, { "commit": "773029b1382e4133ee2243ab33e507a2c67a56ee", "tree": "67ffab5bca0a5a6dc5e9558ab1b31bc387a81e46", "parents": [ "bb3194b2e193130239413f8175a3e723e7df9b0f" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 16:26:12 2023 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 16:26:44 2023 -0800" }, "message": "Add license to new file\n" }, { "commit": "bb3194b2e193130239413f8175a3e723e7df9b0f", "tree": "e7590830829eb8d3b1a4fee16e4462051522493b", "parents": [ "ba0a756cd84d616337db110b4fa1229340951e2d" ], "author": { "name": "shekhars-li", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Tue Jan 31 16:22:39 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jan 31 16:22:39 2023 -0800" }, "message": "Delete empty.txt" }, { "commit": "ba0a756cd84d616337db110b4fa1229340951e2d", "tree": "519a841fb49ac9608ecf9c057ca46fb572f59822", "parents": [ "2a4365ec4f4d8f993a0616493f11969af377750d" ], "author": { "name": "shekhars-li", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Tue Jan 31 16:22:30 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jan 31 16:22:30 2023 -0800" }, "message": "Delete oneEnter.txt" }, { "commit": "2a4365ec4f4d8f993a0616493f11969af377750d", "tree": "adc735c983b09128254845bcad70ea0966837a34", "parents": [ "a39e2a4da464bcd99eb3378769628f918d5f282f" ], "author": { "name": "shekhars-li", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Tue Jan 31 16:22:18 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jan 31 16:22:18 2023 -0800" }, "message": "Delete noEnter.txt" }, { "commit": "a39e2a4da464bcd99eb3378769628f918d5f282f", "tree": "7607b9d93d7ee4704a8aa64bad1597f52ba65868", "parents": [ "9a95f759b5a3d1aa5085cc81f489e36674eff82d" ], "author": { "name": "shekhars-li", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Tue Jan 31 16:21:59 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jan 31 16:21:59 2023 -0800" }, "message": "remove file added by mistake" }, { "commit": "9a95f759b5a3d1aa5085cc81f489e36674eff82d", "tree": "b2ef70f3956b449bf910a5d9c0d9211b556a6d92", "parents": [ "653ae51c7bea0c1e49e7fe5c5b836ec5abca9166" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 16:20:16 2023 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 16:20:16 2023 -0800" }, "message": "Move configs to retry into RetryPolicyConfig class\n" }, { "commit": "653ae51c7bea0c1e49e7fe5c5b836ec5abca9166", "tree": "21af05b886881d47f782761b00150377245cf3c0", "parents": [ "70819db46cc73d75e66fc04758488cb164b0eabd" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 15:13:14 2023 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 15:25:25 2023 -0800" }, "message": "Fix style check - unused import\n" }, { "commit": "70819db46cc73d75e66fc04758488cb164b0eabd", "tree": "73c6db8124eadbf93f69bd086e27f0539fc84b7e", "parents": [ "21724089c3f1575b304b1abb6bd589d94802643b" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 15:13:14 2023 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 15:13:14 2023 -0800" }, "message": "Fix style check - unused import\n" }, { "commit": "21724089c3f1575b304b1abb6bd589d94802643b", "tree": "8f18f3b7dee56ecaf61a1b8d3b2da5d0b6c73375", "parents": [ "931d963cf9407142c0af9c28e0d82591ede3b594" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 15:07:59 2023 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 31 15:07:59 2023 -0800" }, "message": "Fix issues based on PR comments\n" }, { "commit": "b15b4f86bba868dec9bb20b16482f4119f2b5907", "tree": "c28e8afd7523dfa9f7fdaa59a630e13a24454cc7", "parents": [ "d102d208e3bc356fbdd6027643661e7768ab2ed3" ], "author": { "name": "Katie Liu", "email": "katiliu@linkedin.com", "time": "Mon Jan 30 10:18:10 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jan 30 10:18:10 2023 -0800" }, "message": "[SAMZA-2773] Migrate zkclient library (#1651)\n\n* Migrate zkclient library\r\nI0Itec zkclient has not been maintained for a while. Moving to the apache helix copy.\r\n* Remove unused imports\r\n* PR comments: version constant, remove NonNull" }, { "commit": "931d963cf9407142c0af9c28e0d82591ede3b594", "tree": "44d4763d7b512e9b261eeab132a35c15d73f03d4", "parents": [ "617dd934624b275e809d47a8efdd0be4b30c7529" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 24 16:05:12 2023 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 24 16:05:12 2023 -0800" }, "message": "Fix style issues\n" }, { "commit": "617dd934624b275e809d47a8efdd0be4b30c7529", "tree": "a9e9d232997905d96264d67049636c2bdec85124", "parents": [ "344b318b653eba2b665911c45882b224906fdd4b" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 24 14:16:12 2023 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Tue Jan 24 14:16:25 2023 -0800" }, "message": "Move Blob Store retry related tunables behind configs\n" }, { "commit": "344b318b653eba2b665911c45882b224906fdd4b", "tree": "24339fb2cffd41fc3a9947679a6ec202d90e8427", "parents": [ "8b4beb6ef00717e0691680fea0f2522fd1439375", "d102d208e3bc356fbdd6027643661e7768ab2ed3" ], "author": { "name": "shekhars-li", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Tue Jan 24 13:57:05 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jan 24 13:57:05 2023 -0800" }, "message": "Merge branch \u0027apache:master\u0027 into master\n" }, { "commit": "d102d208e3bc356fbdd6027643661e7768ab2ed3", "tree": "24339fb2cffd41fc3a9947679a6ec202d90e8427", "parents": [ "1f8033b06b20f647e2213d941102aee3ef2d4ff1" ], "author": { "name": "Adam Faris", "email": "afaris@linkedin.com", "time": "Mon Jan 23 12:34:46 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jan 23 12:34:46 2023 -0800" }, "message": "[SAMZA-2772] Manage Jekyll dependencies with Gradle (#1647)\n\n* Update Jekyll \u0026 swap redcarpet for kramdown\r\n\r\n* Add jruby \u0026 jekyll dependencies to build.gradle\r\n\r\n* buildscript.gradle update\r\n\r\n* buildscript update\r\n\r\n* downgrade jruby-gradle plugin so it is compatible with JDK8\r\n\r\n* update documentation and finalize changes\r\n\r\n* Update Apache RAT excludes for clean build\r\n\r\n* This is a empty commit to trigger remote build tests\r\n\r\n* trigger build tests by adding new line to build.gradle\r\n\r\n* pin jekyll dependency\r\n\r\n* remove jekyll incremental build flag\r\n\r\n* Downgrade Jruby \u0026 Rouge so it works on RHEL/CentOS 7\r\n\r\n* Replace \u0027jekyll build\u0027 with gradle task in publish-site.sh" }, { "commit": "1f8033b06b20f647e2213d941102aee3ef2d4ff1", "tree": "07d2a41341a945755d2ae60fa37a9d7891c81a3c", "parents": [ "a0d12da6eba7279fe0882a92ac282e935810dee1", "790afd3cc028725aeb7932c8ebf149b4b9d1d77b" ], "author": { "name": "ajo thomas", "email": "ajo.thomas24@gmail.com", "time": "Wed Jan 18 10:56:00 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jan 18 10:56:00 2023 -0800" }, "message": "Merge pull request #1649 from ajothomas/DocUpdate\n\n[Docs] Samza 1.8.0 release website updates" }, { "commit": "790afd3cc028725aeb7932c8ebf149b4b9d1d77b", "tree": "07d2a41341a945755d2ae60fa37a9d7891c81a3c", "parents": [ "a0d12da6eba7279fe0882a92ac282e935810dee1" ], "author": { "name": "Ajo Thomas", "email": "ajthomas@ajthomas-mn2.linkedin.biz", "time": "Tue Jan 17 14:55:40 2023 -0800" }, "committer": { "name": "Ajo Thomas", "email": "ajthomas@ajthomas-mn2.linkedin.biz", "time": "Tue Jan 17 16:28:11 2023 -0800" }, "message": "Samza 1.8.0 release website updates\n" }, { "commit": "a0d12da6eba7279fe0882a92ac282e935810dee1", "tree": "325fc8b0ee4d2e1d36af7cb5162475b48fef0a56", "parents": [ "a0bac341bd159a67d652601c5b7f5739ebb32f48" ], "author": { "name": "shekhars-li", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Mon Jan 09 16:30:21 2023 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jan 09 16:30:21 2023 -0800" }, "message": "Update RocksDB to version 7.8.3 to reduce write amplification (#1648)\n\n" }, { "commit": "8b4beb6ef00717e0691680fea0f2522fd1439375", "tree": "62360248d3b1e8f8aad30a18f09a14a7cb017912", "parents": [ "4cfcdf54489cdbe1ca3cb34017b00834d735f45a" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Mon Jan 09 14:34:00 2023 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Mon Jan 09 14:34:00 2023 -0800" }, "message": "Update RocksDB to 7.8.3\n" }, { "commit": "4cfcdf54489cdbe1ca3cb34017b00834d735f45a", "tree": "9d16242f3d8ac2b642c6a698fc069e5e0eb50ab2", "parents": [ "5aec2c09366073f1cc5afc04f73c052f1923a04d" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Mon Jan 09 14:33:43 2023 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Mon Jan 09 14:33:43 2023 -0800" }, "message": "Update RocksDB to 7.8.3\n" }, { "commit": "5aec2c09366073f1cc5afc04f73c052f1923a04d", "tree": "62360248d3b1e8f8aad30a18f09a14a7cb017912", "parents": [ "d9e139a8666c260cbb4e167da0bccb02b9cb268b" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Fri Jan 06 14:31:29 2023 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Fri Jan 06 14:31:49 2023 -0800" }, "message": "Update RocksDB to 7.8.3\n" }, { "commit": "a0bac341bd159a67d652601c5b7f5739ebb32f48", "tree": "c3c8b30717df176cb9c4653b3089c9f91c0eeebd", "parents": [ "ad8af0c2d431646c00a773c63e671d4560b573b3" ], "author": { "name": "ajo thomas", "email": "ajthomas@linkedin.com", "time": "Fri Dec 16 09:35:34 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Dec 16 09:35:34 2022 -0800" }, "message": "Adding ajothomas to samza committers and gpg keys (#1646)\n\n" }, { "commit": "d9e139a8666c260cbb4e167da0bccb02b9cb268b", "tree": "1b6331d3cbc11c683b69925c04d8a82098423cb0", "parents": [ "7a6a5bf7a617d82dd655848cff5fccc7992adbb7", "ad8af0c2d431646c00a773c63e671d4560b573b3" ], "author": { "name": "shekhars-li", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Thu Dec 15 11:47:25 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Dec 15 11:47:25 2022 -0800" }, "message": "Merge branch \u0027apache:master\u0027 into master\n" }, { "commit": "ad8af0c2d431646c00a773c63e671d4560b573b3", "tree": "1b6331d3cbc11c683b69925c04d8a82098423cb0", "parents": [ "2688cc11ea2cd4f5a91325a7161f869510e2a361" ], "author": { "name": "Daniel Chen", "email": "xrchen@uwaterloo.ca", "time": "Thu Dec 15 10:46:12 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Dec 15 10:46:12 2022 -0800" }, "message": "Fix restore factory not configured log (#1645)\n\n" }, { "commit": "2688cc11ea2cd4f5a91325a7161f869510e2a361", "tree": "fa37f86bc7636b8c0156f69707eda0327bca7b93", "parents": [ "d5d3d989b0b97b55e573b1c895156238b4599d29" ], "author": { "name": "shekhars-li", "email": "72765053+shekhars-li@users.noreply.github.com", "time": "Mon Nov 28 12:59:35 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Nov 28 12:59:35 2022 -0800" }, "message": "Fix perferomance bug with async commit (#1644)\n\n" }, { "commit": "7a6a5bf7a617d82dd655848cff5fccc7992adbb7", "tree": "342fd864aa2fe0c3e8f1c3cde8e9ed11c5ff3c94", "parents": [ "c1bc4426ba426f2f16aa2831140fd32d38211994" ], "author": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Mon Nov 28 10:53:17 2022 -0800" }, "committer": { "name": "Shekhar Sharma", "email": "shesharma@linkedin.com", "time": "Mon Nov 28 10:53:17 2022 -0800" }, "message": "Fix perferomance bug with async commit\n" }, { "commit": "d5d3d989b0b97b55e573b1c895156238b4599d29", "tree": "b0e7a986b22f1c1638d3777159fd87a3814c548a", "parents": [ "d41284226430cbbef3377040f1a5e183628856ae" ], "author": { "name": "jia-gao", "email": "94939653+jia-gao@users.noreply.github.com", "time": "Wed Nov 16 19:48:25 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 16 19:48:25 2022 -0800" }, "message": "LISAMZA-27395 removing the current recursive call prevention logic (#1641)\n\n* LISAMZA-27395 removing the current recursive call prevention logic since it doesn’t work as expected\r\n\r\n* Add unit tests for streamappender change" }, { "commit": "d41284226430cbbef3377040f1a5e183628856ae", "tree": "ef47321cbce78d8caac42c549684ec6bd0d89277", "parents": [ "6a1652f9a5d0e675df434d41189b9cc60793a8de" ], "author": { "name": "ajo thomas", "email": "ajthomas@linkedin.com", "time": "Wed Nov 16 16:51:20 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 16 16:51:20 2022 -0800" }, "message": "SAMZA-2769: [PipelineDrain] Introduce enum in DrainNotification to allow for different drain modes (#1642)\n\n" }, { "commit": "6a1652f9a5d0e675df434d41189b9cc60793a8de", "tree": "a5ed53ee9f7d2d5d57c1a9356786590bd9f2a660", "parents": [ "86d9b51cd76bd4f43b49fbcc4d2aa051a8e23ccf" ], "author": { "name": "ajo thomas", "email": "ajthomas@linkedin.com", "time": "Wed Nov 09 22:27:47 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 09 22:27:47 2022 -0800" }, "message": "SAMZA-2765: [Pipeline Drain] Adding config for task callback timeout during drain (#1637)\n\nAdd a new config task.callback.drain.timeout.ms in TaskConfig\r\nAdded logic to override TaskCallbackManager\u0027s timeout parameter in TaskWorkers on drain." }, { "commit": "86d9b51cd76bd4f43b49fbcc4d2aa051a8e23ccf", "tree": "80a2db1c29700028008408e1420ebd3e51e5b399", "parents": [ "dd8ecf185152ec7ff0739a10c700dca6802f8a37" ], "author": { "name": "ajo thomas", "email": "ajthomas@linkedin.com", "time": "Tue Nov 08 17:50:00 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Nov 08 17:50:00 2022 -0800" }, "message": "SAMZA-2741: [Pipeline Drain] Fix ApplicationUtil.isHighLevelApiJob to work for anonymous SamzaApplication (#1640)\n\n" }, { "commit": "dd8ecf185152ec7ff0739a10c700dca6802f8a37", "tree": "bc46229d55e05cee3300b30283bfcb1ded696b3e", "parents": [ "95a0c3334fbb701145e2cca922422f46af1544ae" ], "author": { "name": "Alan Zhang", "email": "shuai.xyz@gmail.com", "time": "Mon Nov 07 15:35:17 2022 -0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Nov 07 15:35:17 2022 -0800" }, "message": "SAMZA-2762: new cpu usage metric which counts child processes usage (#1636)\n\nSymptom\r\nWe have observed that some use cases used quasar(TensorFlow framework) to do model inference and this framework spawn child processes(non-JVM) to run TensorFlow serving. These child processes were using high CPU usage(200%) however their CPU usage can\u0027t be captured by the existing CPU usage metric process-cpu-usage\r\n\r\nCause\r\nThe existing metric process-cpu-usage metric was designed for capturing the CPU usage for the JVM process only, it can\u0027t count the child processes(especially for non-JVM processes) usage.\r\n\r\nChanges\r\nReply on oshi framwork to capture the CPU usage for the JVM process and all its child processes, and create a new metric to display the total CPU usage.\r\nThe CPU usage percentage is calculated based on top of the logical CPU count on the system\r\n\r\nAPI Changes\r\nAdded a new metric total-process-cpu-usage in SamzaContainerMetrics which is similar with how we provided physical-memory-mb metric" }, { "commit": "95a0c3334fbb701145e2cca922422f46af1544ae", "tree": "d332b209f7f5decb4ddff8b14680b350c3cbe690", "parents": [ "3d4747c5c2cfc2aadff42bca562f1fbfb4acad0b" ], "author": { "name": "Alan Zhang", "email": "shuai.xyz@gmail.com", "time": "Thu Nov 03 15:09:14 2022 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Nov 03 15:09:14 2022 -0700" }, "message": "add the metric \"container-active-threads\" back (#1638)\n\nSymptom\r\nNo data was emitted for the metric container-active-threads\r\n\r\nCause\r\nThis PR(#1501 ) removed the logic to emit data for metric accidently: https://github.com/apache/samza/pull/1501/files#diff-f79781ad4c55ae7860829b06fd9dfd15e8069c37e64f8854d8f27ca2cd1f3ee5L637\r\n\r\nChanges\r\nAdd the deleted logic back in a new classSamzaContainerMonitorListener" }, { "commit": "3d4747c5c2cfc2aadff42bca562f1fbfb4acad0b", "tree": "d869f8f37b2bf3e19032aad87f3b2424f18647b0", "parents": [ "c1bc4426ba426f2f16aa2831140fd32d38211994" ], "author": { "name": "ajo thomas", "email": "ajthomas@linkedin.com", "time": "Thu Nov 03 13:31:24 2022 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Nov 03 13:31:24 2022 -0700" }, "message": "SAMZA-2741: Pipeline Drain- Fix processing of Drain messages for both low-level and high level API (#1639)\n\n" }, { "commit": "c1bc4426ba426f2f16aa2831140fd32d38211994", "tree": "bcf7d1959b57b218adfb2e6eb8a7e735ffbe34ec", "parents": [ "5580bcfac68768ea3e6c1871a03a50361df10889", "40e06084bca89b6d886499ae40f9889452957ac0" ], "author": { "name": "Boris Shkolnik", "email": "boryas@apache.org", "time": "Mon Oct 17 13:45:37 2022 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 17 13:45:37 2022 -0700" }, "message": "Merge pull request #1634 from PawasChhokra/ZK\n\nUpdate zookeeper and kazoo version" }, { "commit": "40e06084bca89b6d886499ae40f9889452957ac0", "tree": "bcf7d1959b57b218adfb2e6eb8a7e735ffbe34ec", "parents": [ "5580bcfac68768ea3e6c1871a03a50361df10889" ], "author": { "name": "Pawas Chhokra", "email": "pchhokra@linkedin.com", "time": "Thu Oct 13 08:33:38 2022 -0700" }, "committer": { "name": "Pawas Chhokra", "email": "pchhokra@linkedin.com", "time": "Thu Oct 13 08:33:38 2022 -0700" }, "message": "Update zookeeper and kazoo version\n" }, { "commit": "5580bcfac68768ea3e6c1871a03a50361df10889", "tree": "09ecf0800e1013542a5e04c3484e771bb155e747", "parents": [ "010d14879f401cc01f083cf84f32eb8741cc8398" ], "author": { "name": "Jamie DeMichele", "email": "james.demichele@redfin.com", "time": "Mon Oct 10 13:45:05 2022 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 10 11:45:05 2022 -0700" }, "message": "SAMZA-2757: Make Samza Compatible with Java 11 (#1628)\n\nFeature: Make Samza build artifacts runnable in JDK11\r\n \r\nChanges:\r\n - introduced new build modules samza-yarn3 and samza-shell-yarn3 to build Samza w/ YARN 3.3.4, which is runtime compatible w/ JDK11\r\n \r\n\u2028Tests:\r\n - local test and hello-samza succeeded with Java11" }, { "commit": "010d14879f401cc01f083cf84f32eb8741cc8398", "tree": "5f0a11c93a9c5942d11b25bc7cee1e0cc00a5f24", "parents": [ "4e4e477bf14d86a9784399d44fd7ae70b7705b69" ], "author": { "name": "jia-gao", "email": "94939653+jia-gao@users.noreply.github.com", "time": "Tue Sep 27 18:44:07 2022 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 27 18:44:07 2022 -0700" }, "message": "Emit to diagnostic kafka topic in round robin fashion to support increasing this topic partition count (#1631)\n\nSymptom: For some Samza jobs, the size of the diagnostic stream partition grows to be large and can cause issues in system streams like Kafka\r\n\r\nCause:\r\nCurrently, diagnostics topics for Samza jobs are created with a single partition.\r\nBecause it is created in DiagnosticsUtil and uses the default partition count, which is 1.\r\nThere could be a need to increase the partition count of the diagnostic stream to resolve the issue above.\r\nHowever, Samza currently uses hostnames as partition key to distribute diagnostic msgs. The host names can be very similar, the hash of these in the partition key will not evenly distribute the msgs across partitions.\r\n\r\nChange:\r\nUse the “null” partition key instead of using the hostname as the partition key, and leverage on Kafka default partitioner to evenly distributed the events across partitions.\r\nNote that this change is backward compatible with the default partition count (1) since every msg goes to the same partition regardless of the partition key.\r\n\r\nAPI changes: None\r\n\r\nBackward compatible: No. Note that the change will be backwards incompatible only when the diagnostic stream has more than one partition. The default creation for diagnostic stream inside samza is with 1 partition, but there are cases that the stream pre-exists or partition count increased after creation.\r\n\r\nTest Done:\r\n./gradlew build" }, { "commit": "4e4e477bf14d86a9784399d44fd7ae70b7705b69", "tree": "80f923bcc47676238af4e846da3d27d4b1421792", "parents": [ "1e8745d6780b13afc7b7efd7edd614aa4ef1147a" ], "author": { "name": "jia-gao", "email": "94939653+jia-gao@users.noreply.github.com", "time": "Mon Sep 19 10:25:28 2022 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 19 10:25:28 2022 -0700" }, "message": "Make StreamAppender extensible for sending event to SystemProducer (#1627)\n\nIssues:\r\nCurrently, StreamAppender is not extensible in terms of setting systemProducer and exposing sendEventToSystemProducer to child classes. It also makes it hard for child classes out of package to unit test any logic related to systemProducer\r\n\r\nChanges: Make StreamAppender extensible\r\n1. Change method sendEventToSystemProducer from private to protected\r\n2. Change member variable systemProducer from private to protected" } ], "next": "1e8745d6780b13afc7b7efd7edd614aa4ef1147a" }