| # Qualification Tool |
| |
| The Qualification Tool analyzes Spark event log files to determine the compatibility and performance of SQL workloads with Gluten. |
| |
| ## Build |
| |
| To compile and package the Qualification Tool, run the following Maven command: |
| |
| ```bash |
| mvn clean package |
| ``` |
| |
| This will create a jar file in the `target` directory. |
| |
| ## Run |
| |
| To execute the tool, use the following command: |
| |
| ```bash |
| java -jar target/qualification-tool-1.3.0-SNAPSHOT-jar-with-dependencies.jar -f <eventFile> |
| ``` |
| |
| ### Parameters: |
| - **`-f <eventFile>`**: Path to the Spark event log file(s). This can be: |
| - A single event log file. |
| - A folder containing multiple event log files. |
| - Deeply nested folders of event log files. |
| - Compressed event log files. |
| - Rolling event log files. |
| - Comma separated files |
| - **`-k <gcsKey>`**: (Optional) Path to Google Cloud Storage service account keys. |
| - **`-o <output>`**: (Optional) Path to the directory where output will be written. Defaults to a temporary directory. |
| - **`-t <threads>`**: (Optional) Number of processing threads. Defaults to 4. |
| - **`-v`**: (Optional) Enable non verbose output. Omit this flag for verbose mode. |
| - **`-p <project>`**: (Optional) Project ID for the run. |
| - **`-d <dateFilter>`**: (Optional) Analyze only files created after this date (format: YYYY-MM-DD). Defaults to the last 90 days. |
| |
| ### Example Usage: |
| ```bash |
| java -jar target/qualification-tool-1.3.0-SNAPSHOT-jar-with-dependencies.jar -f /path/to/eventlog |
| ``` |
| |
| ### Advanced Example: |
| ```bash |
| java -jar target/qualification-tool-1.3.0-SNAPSHOT-jar-with-dependencies.jar -f /path/to/folder -o /output/path -t 8 -d 2023-01-01 -k /path/to/gcs_keys.json -p my_project |
| ``` |
| |
| ## Features |
| |
| - Analyzes Spark SQL execution plans for compatibility with Gluten. |
| - Supports single files, folders, deeply nested folders, compressed files, and rolling event logs. |
| - Provides detailed reports on supported and unsupported operations. |
| - Generates metrics on SQL execution times and operator impact. |
| - Configurable verbosity and threading. |
| |
| ## How It Works |
| |
| The Qualification Tool analyzes a Spark plan to determine the compatibility of its nodes / operators and clusters with Gluten. Here's a step-by-step explanation of the process: |
| |
| ### Example Spark Plan |
| |
| Consider the following Spark plan: |
| |
| ``` |
| G |
| / \ |
| G[2] G[3] |
| | | |
| S[2] G[3] |
| | | |
| G S |
| / \ |
| G[1] G |
| | |
| G[1] |
| | |
| G |
| ``` |
| |
| - **G**: Represents a plan supported by Gluten. |
| - **S**: Represents a plan not supported by Gluten. |
| - **[1], [2], [3]**: Indicates the node belongs to a Whole Stage Code Gen Block (Cluster). |
| |
| ### 1. NodeSupportVisitor |
| |
| The first step is marking each node as supported (`*`) or not supported (`!`) by Gluten: |
| |
| ``` |
| *G |
| / \ |
| *G[2] *G[3] |
| | | |
| !S[2] *G[3] |
| | | |
| *G !S |
| / \ |
| *G[1] *G |
| | |
| *G[1] |
| | |
| *G |
| ``` |
| |
| - All supported nodes are marked with `*`. |
| - All unsupported nodes are marked with `!`. |
| |
| ### 2. ClusterSupportVisitor |
| |
| The second step marks entire clusters as not supported (`!`) if any node in the cluster is unsupported: |
| |
| ``` |
| *G |
| / \ |
| !G[2] *G[3] |
| | | |
| !S[2] *G[3] |
| | | |
| *G !S |
| / \ |
| *G[1] *G |
| | |
| *G[1] |
| | |
| *G |
| ``` |
| |
| #### Reasoning: |
| Although Gluten supports these operators, breaking Whole Stage Code Gen (WSCG) boundaries introduces row-to-columnar and columnar-to-row conversions, degrading performance. Hence, we pessimistically mark the entire cluster as not supported. |
| |
| ### 3. ChildSupportVisitor |
| |
| The final step marks nodes and their parents as not supported if their children are unsupported: |
| |
| ``` |
| !G |
| / \ |
| !G[2] !G[3] |
| | | |
| !S[2] !G[3] |
| | | |
| *G !S |
| / \ |
| *G[1] *G |
| | |
| *G[1] |
| | |
| *G |
| ``` |
| |
| #### Reasoning: |
| If a child node is not supported by Gluten, row-to-columnar and columnar-to-row conversions are added, degrading performance. Therefore, we pessimistically mark such nodes as not supported. |
| |
| ### Sample Output |
| #### AppsRecommendedForBoost.tsv |
| | applicationId | applicationName | batchUuid | rddPercentage | unsupportedSqlPercentage | totalTaskTime | supportedTaskTime | supportedSqlPercentage | recommendedForBoost | expectedRuntimeReduction | |
| |-----------------------------------|-----------------------------------|-----------------------------------------|---------------|--------------------------|---------------|-------------------|-------------------------|---------------------|--------------------------| |
| | app-20241001064609-0000 | POC - StoreItem | a56bee42 | 0.0% | 5.2% | 3244484900 | 3074633672 | 94.7% | true | 28.4% | |
| | application_1729530422325_0001 | job-audience-generation-21 | UNKNOWN | 0.3% | 0.6% | 1378942742 | 1365259621 | 99.0% | true | 29.7% | |
| | application_1729410290443_0001 | job-audience-generation-27 | UNKNOWN | 0.0% | 0.7% | 1212612881 | 1203690936 | 99.2% | true | 29.7% | |
| | app-20241001080158-0000 | POC - StoreItem | 94088807 | 0.0% | 12.5% | 668601513 | 584812056 | 87.4% | true | 26.2% | |
| | application_1729991008434_0001 | DailyLoad | UNKNOWN | 0.0% | 12.6% | 17134675 | 14963535 | 87.3% | true | 26.1% | |
| | application_1730097715348_0003 | Spark shell | UNKNOWN | 0.0% | 0.0% | 4680 | 4680 | 100.0% | true | 30.0% | |
| | application_1728526688221_0001 | job-audience-generation-27 | UNKNOWN | 0.4% | 59.4% | 805991113 | 323274568 | 40.1% | false | 12.0% | |
| | application_1726070403340_0450 | driver.py | UNKNOWN | 0.0% | 81.1% | 398992332 | 75187879 | 18.8% | false | 5.6% | |
| | application_1727842554841_0001 | driver.py | UNKNOWN | 0.0% | 58.3% | 166686890 | 69492539 | 41.6% | false | 12.5% | |
| | application_1726070403340_0474 | driver.py | UNKNOWN | 0.0% | 81.6% | 325389669 | 59687634 | 18.3% | false | 5.5% | |
| | application_1729133595704_0001 | job-audience-generation-54 | UNKNOWN | 0.6% | 99.3% | 472621872 | 20205 | 0.0% | false | 0.0% | |
| | application_1730097715348_0002 | Spark shell | UNKNOWN | 33.8% | 16.6% | 4197 | 2077 | 49.5% | false | 14.8% | |
| | application_1730097715348_0001 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% | |
| | application_1712155629437_0011 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% | |
| | application_1712155629437_0012 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% | |
| | application_1712155629437_0007 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% | |
| | app-20241120163343-0000 | Config Test | c2884285 | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% | |
| | application_1712155629437_0008 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% | |
| | application_1712155629437_0010 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% | |
| | application_1730097715348_0004 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% | |
| | application_1718075857669_0261 | UnregisteredAsset | UNKNOWN | 0.0% | 100.0% | 12205458 | 0 | 0.0% | false | 0.0% | |
| | application_1712155629437_0009 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% | |
| | application_1712155629437_0013 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% | |
| |
| #### UnsupportedOperators.tsv |
| | unsupportedOperator | cumulativeCpuMs | count | |
| |--------------------------------------------------------|-----------------|---------| |
| | Scan with format JSON not supported | 1722606716687 | 3105 | |
| | Parquet with struct | 1722606716687 | 3105 | |
| | Execute InsertIntoHadoopFsRelationCommand not supported| 37884897400 | 405732 | |
| | BHJ Not Supported | 20120117622 | 132164 | |
| | SerializeFromObject not supported | 13856742899 | 16098 | |
| | MapPartitions not supported | 13856742899 | 16098 | |
| | FlatMapGroupsInPandas not supported | 257970400 | 224 | |
| | Sample not supported | 157615298 | 432 | |
| | DeserializeToObject not supported | 157610379 | 6721 | |
| | MapElements not supported | 5261628 | 694 | |
| | Parquet with struct and map | 29020 | 44 | |
| | Execute CreateViewCommand not supported | 0 | 157 | |
| | Execute CreateTableCommand not supported | 0 | 42 | |
| | WriteFiles not supported | 0 | 20 | |
| | Execute SetCommand not supported | 0 | 12 | |
| | Execute SaveIntoDataSourceCommand not supported | 0 | 10 | |
| | CreateTable not supported | 0 | 4 | |
| | Execute RefreshTableCommand not supported | 0 | 3 | |
| | Execute DeltaDynamicPartitionOverwriteCommand not supported | 0 | 2 | |
| | Execute TruncateTableCommand not supported | 0 | 2 | |
| | Execute AlterTableAddPartitionCommand not supported | 0 | 1 | |
| | CreateNamespace not supported | 0 | 1 | |
| |
| |
| ### Summary |
| |
| The tool ensures that the Spark plan optimizes performance by: |
| 1. Identifying individual node compatibility. |
| 2. Accounting for cluster boundaries and WSCG optimizations. |
| 3. Considering child dependencies and their impact on parent nodes. |
| |
| ## Requirements |
| |
| - **Java**: Ensure you have JDK 11 or later installed. |
