[GOBBLIN-2269] Surface snapshot/partition copy metrics in observability (#4206) * [GOBBLIN-XXXX] Surface snapshot/partition copy metrics in observability Add per-dataset snapshotsCommitted and partitionsCommitted counts to both the GaaSJobObservabilityEvent DatasetMetric and the job-status JobStatistics REST surface, so file/blob movement jobs (e.g. DDM Iceberg snapshot replication) can report table snapshots and partitions committed. DatasetMetric (GaaSJobObservabilityEvent.avsc) gains snapshotsCommitted and partitionsCommitted (long, default -1 = unsupported/unknown). DatasetTaskSummary carries the two counts as non-@NonNull fields so the existing 5-arg RequiredArgsConstructor used by AbstractJobLauncher is preserved; they populate via JSON deserialization for events that supply them, and toDatasetMetric maps them through. JobStatistics (executionStatistics) gains optional bytesWritten, recordsWritten, filesCommitted, snapshotsCommitted and partitionsCommitted. JobStatusRetriever reads them from the flat ddm.* copy-metric keys the JobSummary event persists into the job state, and FlowExecutionResource sets each only when reported, so jobs that do not emit them are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Drop the unrelated flow-level 'issues' field from the flowexecutions snapshot FlowExecution.pdl already declares 'issues' on master; this branch never touched that file. The snapshot json was simply stale, so regenerating it picked the field up as an incidental diff. Removing it keeps the PR scoped to the copy metrics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix findbugs UWF_NULL_FIELD on the new DatasetTaskSummary fields The 'Run static checks' job failed on :gobblin-runtime:findbugsMain with two UWF_NULL_FIELD violations: snapshotsCommitted and partitionsCommitted were declared with an explicit '= null' initializer and no in-repo writer, so FindBugs correctly concluded every read returns null. They are normally populated reflectively during JSON deserialization, which FindBugs cannot see. Add @AllArgsConstructor to give native producers a real write path and drop the redundant '= null' initializers (fields already default to null). The 5-arg @RequiredArgsConstructor is untouched, so existing call sites still compile. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apache Gobblin is a highly scalable data management solution for structured and byte-oriented data in heterogeneous data ecosystems.
If building the distribution with tests turned on:
If you are going to build Gobblin from the source distribution, run the following command for downloading the gradle-wrapper.jar from Gobblin git repository to gradle/wrapper directory (replace GOBBLIN_VERSION in the URL with the version you downloaded).
wget --no-check-certificate -P gradle/wrapper https://github.com/apache/gobblin/raw/${GOBBLIN_VERSION}/gradle/wrapper/gradle-wrapper.jar
(or)
curl --insecure -L https://github.com/apache/gobblin/raw/${GOBBLIN_VERSION}/gradle/wrapper/gradle-wrapper.jar > gradle/wrapper/gradle-wrapper.jar
Alternatively, you can download it manually from: https://github.com/apache/gobblin/blob/${GOBBLIN_VERSION}/gradle/wrapper/gradle-wrapper.jar
Make sure that you download it to gradle/wrapper directory.
./gradlew rat. Report will be generated under build/rat/rat-report.html./gradlew build -x findbugsMain -x test -x rat -x checkstyleMain The distribution will be created in build/gobblin-distribution/distributions directory. (or)./gradlew build The distribution will be created in build/gobblin-distribution/distributions directory.