| <!DOCTYPE html> |
| <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <meta name="author" content="Apache Software Foundation"> |
| <link rel="shortcut icon" href="../../img/favicon.ico"> |
| <title>Job Execution History Store - Apache Gobblin</title> |
| <link href='https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700' rel='stylesheet' type='text/css'> |
| |
| <link rel="stylesheet" href="../../css/theme.css" type="text/css" /> |
| <link rel="stylesheet" href="../../css/theme_extra.css" type="text/css" /> |
| <link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/github.min.css"> |
| <link href="../../css/extra.css" rel="stylesheet"> |
| |
| <script> |
| // Current page data |
| var mkdocs_page_name = "Job Execution History Store"; |
| var mkdocs_page_input_path = "user-guide/Job-Execution-History-Store.md"; |
| var mkdocs_page_url = null; |
| </script> |
| |
| <script src="../../js/jquery-2.1.1.min.js" defer></script> |
| <script src="../../js/modernizr-2.8.3.min.js" defer></script> |
| <script src="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script> |
| <script>hljs.initHighlightingOnLoad();</script> |
| |
| |
| </head> |
| |
| <body class="wy-body-for-nav" role="document"> |
| |
| <div class="wy-grid-for-nav"> |
| |
| |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side stickynav"> |
| <div class="wy-side-nav-search"> |
| <a href="../.." class="icon icon-home"> Apache Gobblin</a> |
| <div role="search"> |
| <form id ="rtd-search-form" class="wy-form" action="../../search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" title="Type search term here" /> |
| </form> |
| </div> |
| </div> |
| |
| <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> |
| <ul class="current"> |
| |
| |
| <li class="toctree-l1"> |
| |
| <a class="" href="/">Home</a> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <a class="" href="../../Powered-By/">Companies Powered By Gobblin</a> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <a class="" href="../../Getting-Started/">Getting Started</a> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <a class="" href="../../Gobblin-Architecture/">Architecture</a> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">User Guide</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../Working-with-Job-Configuration-Files/">Job Configuration Files</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-Deployment/">Deployment</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-as-a-Library/">Gobblin as a Library</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-CLI/">Gobblin CLI</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-Compliance/">Gobblin Compliance</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-on-Yarn/">Gobblin on Yarn</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Compaction/">Compaction</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../State-Management-and-Watermarks/">State Management and Watermarks</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Working-with-the-ForkOperator/">Fork Operator</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Configuration-Properties-Glossary/">Configuration Glossary</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Source-schema-and-Converters/">Source schema and Converters</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Partitioned-Writers/">Partitioned Writers</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Monitoring/">Monitoring</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-template/">Template</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-Schedulers/">Schedulers</a> |
| </li> |
| <li class=" current"> |
| |
| <a class="current" href="./">Job Execution History Store</a> |
| <ul class="subnav"> |
| |
| <li class="toctree-l3"><a href="#table-of-contents">Table of Contents</a></li> |
| |
| |
| <li class="toctree-l3"><a href="#overview">Overview</a></li> |
| |
| |
| <li class="toctree-l3"><a href="#information-recorded">Information Recorded</a></li> |
| |
| |
| <li class="toctree-l3"><a href="#job-execution-information">Job Execution Information</a></li> |
| |
| |
| <li class="toctree-l3"><a href="#task-execution-information">Task Execution Information</a></li> |
| |
| |
| <li class="toctree-l3"><a href="#default-implementation">Default Implementation</a></li> |
| |
| |
| <li class="toctree-l3"><a href="#rest-query-api">Rest Query API</a></li> |
| |
| <ul> |
| |
| <li><a class="toctree-l4" href="#example-queries">Example Queries</a></li> |
| |
| </ul> |
| |
| |
| <li class="toctree-l3"><a href="#job-execution-history-server">Job Execution History Server</a></li> |
| |
| |
| </ul> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Building-Gobblin/">Building Gobblin</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-genericLoad/">Generic Configuration Loading</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Hive-Registration/">Hive Registration</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Config-Management/">Config Management</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Docker-Integration/">Docker Integration</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Troubleshooting/">Troubleshooting</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../FAQs/">FAQs</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Sources</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../sources/AvroFileSource/">Avro files</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/CopySource/">File copy</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/QueryBasedSource/">Query based</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/RestApiSource/">Rest Api</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/GoogleAnalyticsSource/">Google Analytics</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/GoogleDriveSource/">Google Drive</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/GoogleWebmaster/">Google Webmaster</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/HadoopTextInputSource/">Hadoop Text Input</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/HelloWorldSource/">Hello World</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/HiveAvroToOrcSource/">Hive Avro-to-ORC</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/HivePurgerSource/">Hive compliance purging</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/SimpleJsonSource/">JSON</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/KafkaSource/">Kafka</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/MySQLSource/">MySQL</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/OracleSource/">Oracle</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/SalesforceSource/">Salesforce</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/SftpSource/">SFTP</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/SqlServerSource/">SQL Server</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/TeradataSource/">Teradata</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/WikipediaSource/">Wikipedia</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Sinks (Writers)</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../sinks/AvroHdfsDataWriter/">Avro HDFS</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/ParquetHdfsDataWriter/">Parquet HDFS</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/SimpleBytesWriter/">HDFS Byte array</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/ConsoleWriter/">Console</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/CouchbaseWriter/">Couchbase</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/Http/">HTTP</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/Gobblin-JDBC-Writer/">JDBC</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/Kafka/">Kafka</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Gobblin Adaptors</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../adaptors/Gobblin-Distcp/">Gobblin Distcp</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../adaptors/Hive-Avro-To-ORC-Converter/">Hive Avro-To-Orc Converter</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Case Studies</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../case-studies/Kafka-HDFS-Ingestion/">Kafka-HDFS Ingestion</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../case-studies/Publishing-Data-to-S3/">Publishing Data to S3</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../case-studies/Writing-ORC-Data/">Writing ORC Data</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../case-studies/Hive-Distcp/">Hive Distcp</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Gobblin Data Management</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../data-management/Gobblin-Retention/">Retention</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../data-management/DistcpNgEvents/">Distcp-NG events</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Gobblin Metrics</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../metrics/Gobblin-Metrics/">Quick Start</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../metrics/Existing-Reporters/">Existing Reporters</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../metrics/Metrics-for-Gobblin-ETL/">Metrics for Gobblin ETL</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../metrics/Gobblin-Metrics-Architecture/">Gobblin Metrics Architecture</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../metrics/Implementing-New-Reporters/">Implementing New Reporters</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../metrics/Gobblin-Metrics-Performance/">Gobblin Metrics Performance</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Developer Guide</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/Customization-for-New-Source/">Customization for New Source</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/Customization-for-Converter-and-Operator/">Customization for Converter and Operator</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/CodingStyle/">Code Style Guide</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/Gobblin-Compliance-Design/">Gobblin Compliance Design</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/IDE-setup/">IDE setup</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/Monitoring-Design/">Monitoring Design</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/Documentation-Architecture/">Documentation Architecture</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/Contributing/">Contributing</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/GobblinModules/">Gobblin Modules</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/HighLevelConsumer/">High Level Consumer</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Project</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../project/Feature-List/">Feature List</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="/people">Contributors and Team</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../project/Talks-and-Tech-Blogs/">Talks and Tech Blog Posts</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../project/Posts/">Posts</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Miscellaneous</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../miscellaneous/Camus-to-Gobblin-Migration/">Camus to Gobblin Migration</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../miscellaneous/Exactly-Once-Support/">Exactly Once Support</a> |
| </li> |
| </ul> |
| </li> |
| |
| </ul> |
| </div> |
| |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> |
| |
| |
| <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="../..">Apache Gobblin</a> |
| </nav> |
| |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| <div role="navigation" aria-label="breadcrumbs navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="../..">Docs</a> »</li> |
| |
| |
| |
| <li>User Guide »</li> |
| |
| |
| |
| <li>Job Execution History Store</li> |
| <li class="wy-breadcrumbs-aside"> |
| |
| <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Job-Execution-History-Store.md" rel="nofollow"> Edit on Gobblin</a> |
| |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main"> |
| <div class="section"> |
| |
| <h2 id="table-of-contents">Table of Contents</h2> |
| <div class="toc"> |
| <ul> |
| <li><a href="#table-of-contents">Table of Contents</a></li> |
| <li><a href="#overview">Overview</a></li> |
| <li><a href="#information-recorded">Information Recorded</a></li> |
| <li><a href="#job-execution-information">Job Execution Information</a></li> |
| <li><a href="#task-execution-information">Task Execution Information</a></li> |
| <li><a href="#default-implementation">Default Implementation</a></li> |
| <li><a href="#rest-query-api">Rest Query API</a><ul> |
| <li><a href="#example-queries">Example Queries</a></li> |
| </ul> |
| </li> |
| <li><a href="#job-execution-history-server">Job Execution History Server</a></li> |
| </ul> |
| </div> |
| <h2 id="overview">Overview</h2> |
| <p>Gobblin provides the users a way of keeping tracking of executions of their jobs through the Job Execution History Store, which can be queried either directly if the implementation supports queries directly or through a Rest API. Note that using the Rest API needs the Job Execution History Server to be up and running. The Job Execution History Server will be discussed later. By default, writing to the Job Execution History Store is not enabled. To enable it, set configuration property <code>job.history.store.enabled</code> to <code>true</code>.</p> |
| <h2 id="information-recorded">Information Recorded</h2> |
| <p>The Job Execution History Store stores various pieces of information of a job execution, including both job-level and task-level stats and measurements that are summarized below.</p> |
| <h2 id="job-execution-information">Job Execution Information</h2> |
| <p>The following table summarizes job-level execution information the Job Execution History Store stores. </p> |
| <table> |
| <thead> |
| <tr> |
| <th>Information</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>Job name</td> |
| <td>Gobblin job name.</td> |
| </tr> |
| <tr> |
| <td>Job ID</td> |
| <td>Gobblin job ID.</td> |
| </tr> |
| <tr> |
| <td>Start time</td> |
| <td>Start time in epoch time (of unit milliseconds) of the job in the local time zone.</td> |
| </tr> |
| <tr> |
| <td>End time</td> |
| <td>End time in epoch time (of unit milliseconds) of the job in the local time zone.</td> |
| </tr> |
| <tr> |
| <td>Duration</td> |
| <td>Duration of the job in milliseconds.</td> |
| </tr> |
| <tr> |
| <td>Job state</td> |
| <td>Running state of the job. Possible values are <code>PENDING</code>, <code>RUNNING</code>, <code>SUCCESSFUL</code>, <code>COMMITTED</code>, <code>FAILED</code>, <code>CANCELLED</code>.</td> |
| </tr> |
| <tr> |
| <td>Launched tasks</td> |
| <td>Number of launched tasks of the job.</td> |
| </tr> |
| <tr> |
| <td>Completed tasks</td> |
| <td>Number of tasks of the job that completed.</td> |
| </tr> |
| <tr> |
| <td>Launcher type</td> |
| <td>The type of the launcher used to launch and run the task.</td> |
| </tr> |
| <tr> |
| <td>Job tracking URL</td> |
| <td>This will be set to the MapReduce job URL if the Gobblin job is running on Hadoop MapReduce. This may also be set to the Azkaban job execution tracking URL if the job is running through Azkaban but not on Hadoop MapReduce. Otherwise, this will be empty.</td> |
| </tr> |
| <tr> |
| <td>Job-level metrics</td> |
| <td>Values of job-level metrics. Note that this data is not time-series based so the values will be overwritten on every update.</td> |
| </tr> |
| <tr> |
| <td>Job configuration properties</td> |
| <td>Job configuration properties used at runtime for job execution. Note that it may include changes made at runtime by the job.</td> |
| </tr> |
| </tbody> |
| </table> |
| <h2 id="task-execution-information">Task Execution Information</h2> |
| <p>The following table summarizes task-level execution information the Job Execution History Store stores. </p> |
| <table> |
| <thead> |
| <tr> |
| <th>Information</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>Task ID</td> |
| <td>Gobblin task ID.</td> |
| </tr> |
| <tr> |
| <td>Job ID</td> |
| <td>Gobblin job ID.</td> |
| </tr> |
| <tr> |
| <td>Start time</td> |
| <td>Start time in epoch time (of unit milliseconds) of the task in the local time zone.</td> |
| </tr> |
| <tr> |
| <td>End time</td> |
| <td>End time in epoch time (of unit milliseconds) of the task in the local time zone.</td> |
| </tr> |
| <tr> |
| <td>Duration</td> |
| <td>Duration of the task in milliseconds.</td> |
| </tr> |
| <tr> |
| <td>Task state</td> |
| <td>Running state of the task. Possible values are <code>PENDING</code>, <code>RUNNING</code>, <code>SUCCESSFUL</code>, <code>COMMITTED</code>, <code>FAILED</code>, <code>CANCELLED</code>.</td> |
| </tr> |
| <tr> |
| <td>Task failure exception</td> |
| <td>Exception message in case of task failure.</td> |
| </tr> |
| <tr> |
| <td>Low watermark</td> |
| <td>The low watermark of the task if avaialble.</td> |
| </tr> |
| <tr> |
| <td>High watermark</td> |
| <td>The high watermark of the task if available.</td> |
| </tr> |
| <tr> |
| <td>Extract namespace</td> |
| <td>The namespace of the <code>Extract</code>. An <code>Extract</code> is a concept describing the ingestion work of a job. This stores the value specified through the configuration property <code>extract.namespace</code>.</td> |
| </tr> |
| <tr> |
| <td>Extract name</td> |
| <td>The name of the <code>Extract</code>. This stores the value specified through the configuration property <code>extract.table.name</code>.</td> |
| </tr> |
| <tr> |
| <td>Extract type</td> |
| <td>The type of the <code>Extract</code>. This stores the value specified through the configuration property <code>extract.table.type</code>.</td> |
| </tr> |
| <tr> |
| <td>Task-level metrics</td> |
| <td>Values of task-level metrics. Note that this data is not time-series based so the values will be overwritten on every update.</td> |
| </tr> |
| <tr> |
| <td>Task configuration properties</td> |
| <td>Task configuration properties used at runtime for task execution. Note that it may include changes made at runtime by the task.</td> |
| </tr> |
| </tbody> |
| </table> |
| <h2 id="default-implementation">Default Implementation</h2> |
| <p>The default implementation of the Job Execution History Store stores job execution information into a MySQL database in a few different tables. Specifically, the following tables are used and should be created before writing to the store is enabled. Checkout the MySQL <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration" rel="nofollow">DDLs</a> of the tables for detailed columns of each table.</p> |
| <ul> |
| <li>Table <code>gobblin_job_executions</code> stores basic information about a job execution including the start and end times, job running state, number of launched and completed tasks, etc. </li> |
| <li>Table <code>gobblin_task_executions</code> stores basic information on task executions of a job, including the start and end times, task running state, task failure message if any, etc, of each task. </li> |
| <li>Table <code>gobblin_job_metrics</code> stores values of job-level metrics collected through the <code>JobMetrics</code> class. Note that this data is not time-series based and values of metrics are overwritten on every update to the job execution information. </li> |
| <li>Table <code>gobblin_task_metrics</code> stores values of task-level metrics collected through the <code>TaskMetrics</code> class. Again, this data is not time-series based and values of metrics are overwritten on updates.</li> |
| <li>Table <code>gobblin_job_properties</code> stores the job configuration properties used at runtime for the job execution, which may include changes made at runtime by the job.</li> |
| <li>Table <code>gobblin_task_properties</code> stores the task configuration properties used at runtime for task executions, which also may include changes made at runtime by the tasks.</li> |
| </ul> |
| <p>To enable writing to the MySQL-backed Job Execution History Store, the following configuration properties (with sample values) need to be set:</p> |
| <pre><code class="properties">job.history.store.url=jdbc:mysql://localhost/gobblin |
| job.history.store.jdbc.driver=com.mysql.jdbc.Driver |
| job.history.store.user=gobblin |
| job.history.store.password=gobblin |
| </code></pre> |
| |
| <h2 id="rest-query-api">Rest Query API</h2> |
| <p>The Job Execution History Store Rest API supports three types of queries: query by job name, query by job ID, or query by extract name. The query type can be specified using the field <code>idType</code> in the query json object and can have one of the values <code>JOB_NAME</code>, <code>JOB_ID</code>, or <code>TABLE</code>. All three query types require the field <code>id</code> in the query json object, which should have a proper value as documented in the following table. </p> |
| <table> |
| <thead> |
| <tr> |
| <th>Query type</th> |
| <th>Query ID</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>JOB_NAME</td> |
| <td>Gobblin job name.</td> |
| </tr> |
| <tr> |
| <td>JOB_ID</td> |
| <td>Gobblin job ID.</td> |
| </tr> |
| <tr> |
| <td>TABLE</td> |
| <td>A json object following the <code>TABLE</code> schema shown below.</td> |
| </tr> |
| </tbody> |
| </table> |
| <pre><code class="json">{ |
| "type": "record", |
| "name": "Table", |
| "namespace": "gobblin.rest", |
| "doc": "Gobblin table definition", |
| "fields": [ |
| { |
| "name": "namespace", |
| "type": "string", |
| "optional": true, |
| "doc": "Table namespace" |
| }, |
| { |
| "name": "name", |
| "type": "string", |
| "doc": "Table name" |
| }, |
| { |
| "name": "type", |
| "type": { |
| "name": "TableTypeEnum", |
| "type": "enum", |
| "symbols" : [ "SNAPSHOT_ONLY", "SNAPSHOT_APPEND", "APPEND_ONLY" ] |
| }, |
| "optional": true, |
| "doc": "Table type" |
| } |
| ] |
| } |
| </code></pre> |
| |
| <p>For each query type, there are also some option fields that can be used to control the number of records returned and what should be included in the query result. The optional fields are summarized in the following table.</p> |
| <table> |
| <thead> |
| <tr> |
| <th>Optional field</th> |
| <th>Type</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><code>limit</code></td> |
| <td><code>int</code></td> |
| <td>Limit on the number of records returned.</td> |
| </tr> |
| <tr> |
| <td><code>timeRange</code></td> |
| <td><code>TimeRange</code></td> |
| <td>The query time range. The schema of <code>TimeRange</code> is shown below.</td> |
| </tr> |
| <tr> |
| <td><code>jobProperties</code></td> |
| <td><code>boolean</code></td> |
| <td>This controls whether the returned record should include the job configuration properties.</td> |
| </tr> |
| <tr> |
| <td><code>taskProperties</code></td> |
| <td><code>boolean</code></td> |
| <td>This controls whether the returned record should include the task configuration properties.</td> |
| </tr> |
| </tbody> |
| </table> |
| <pre><code class="json">{ |
| "type": "record", |
| "name": "TimeRange", |
| "namespace": "gobblin.rest", |
| "doc": "Query time range", |
| "fields": [ |
| { |
| "name": "startTime", |
| "type": "string", |
| "optional": true, |
| "doc": "Start time of the query range" |
| }, |
| { |
| "name": "endTime", |
| "type": "string", |
| "optional": true, |
| "doc": "End time of the query range" |
| }, |
| { |
| "name": "timeFormat", |
| "type": "string", |
| "doc": "Date/time format used to parse the start time and end time" |
| } |
| ] |
| } |
| </code></pre> |
| |
| <p>The API is built with <a href="http://www.rest.li" rel="nofollow">rest.li</a>, which generates documentation on compilation and can be found at <code>http://<hostname:port>/restli/docs</code>.</p> |
| <h3 id="example-queries">Example Queries</h3> |
| <p><em>Fetch the 10 most recent job executions with a job name <code>TestJobName</code></em></p> |
| <pre><code class="bash">curl "http://<hostname:port>/jobExecutions/idType=JOB_NAME&id.string=TestJobName&limit=10" |
| </code></pre> |
| |
| <h2 id="job-execution-history-server">Job Execution History Server</h2> |
| <p>The Job Execution History Server is a Rest server for serving queries on the Job Execution History Store through the Rest API described above. The Rest endpoint URL is configurable through the following configuration properties (with their default values):</p> |
| <pre><code class="properties">rest.server.host=localhost |
| rest.server.port=8080 |
| </code></pre> |
| |
| <p><strong>Note:</strong> This server is started in the standalone deployment if configuration property <code>job.execinfo.server.enabled</code> is set to <code>true</code>.</p> |
| |
| </div> |
| </div> |
| <footer> |
| |
| <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> |
| |
| <a href="../Building-Gobblin/" class="btn btn-neutral float-right" title="Building Gobblin">Next <span class="icon icon-circle-arrow-right"></span></a> |
| |
| |
| <a href="../Gobblin-Schedulers/" class="btn btn-neutral" title="Schedulers"><span class="icon icon-circle-arrow-left"></span> Previous</a> |
| |
| </div> |
| |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <!-- Copyright etc --> |
| |
| </div> |
| |
| Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>. |
| </footer> |
| |
| </div> |
| </div> |
| |
| </section> |
| |
| </div> |
| |
| <div class="rst-versions" role="note" style="cursor: pointer"> |
| <span class="rst-current-version" data-toggle="rst-current-version"> |
| |
| |
| <span><a href="../Gobblin-Schedulers/" style="color: #fcfcfc;">« Previous</a></span> |
| |
| |
| <span style="margin-left: 15px"><a href="../Building-Gobblin/" style="color: #fcfcfc">Next »</a></span> |
| |
| </span> |
| </div> |
| <script>var base_url = '../..';</script> |
| <script src="../../js/theme.js" defer></script> |
| <script src="../../js/extra.js" defer></script> |
| <script src="../../search/main.js" defer></script> |
| |
| </body> |
| </html> |
