[GOBBLIN-2269] Surface snapshot/partition copy metrics in observability (#4206)

* [GOBBLIN-XXXX] Surface snapshot/partition copy metrics in observability

Add per-dataset snapshotsCommitted and partitionsCommitted counts to both the
GaaSJobObservabilityEvent DatasetMetric and the job-status JobStatistics REST
surface, so file/blob movement jobs (e.g. DDM Iceberg snapshot replication) can
report table snapshots and partitions committed.

DatasetMetric (GaaSJobObservabilityEvent.avsc) gains snapshotsCommitted and
partitionsCommitted (long, default -1 = unsupported/unknown). DatasetTaskSummary
carries the two counts as non-@NonNull fields so the existing 5-arg
RequiredArgsConstructor used by AbstractJobLauncher is preserved; they populate
via JSON deserialization for events that supply them, and toDatasetMetric maps
them through.

JobStatistics (executionStatistics) gains optional bytesWritten, recordsWritten,
filesCommitted, snapshotsCommitted and partitionsCommitted. JobStatusRetriever
reads them from the flat ddm.* copy-metric keys the JobSummary event persists
into the job state, and FlowExecutionResource sets each only when reported, so
jobs that do not emit them are unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Drop the unrelated flow-level 'issues' field from the flowexecutions snapshot

FlowExecution.pdl already declares 'issues' on master; this branch never touched
that file. The snapshot json was simply stale, so regenerating it picked the field
up as an incidental diff. Removing it keeps the PR scoped to the copy metrics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix findbugs UWF_NULL_FIELD on the new DatasetTaskSummary fields

The 'Run static checks' job failed on :gobblin-runtime:findbugsMain with two
UWF_NULL_FIELD violations: snapshotsCommitted and partitionsCommitted were
declared with an explicit '= null' initializer and no in-repo writer, so
FindBugs correctly concluded every read returns null. They are normally
populated reflectively during JSON deserialization, which FindBugs cannot see.

Add @AllArgsConstructor to give native producers a real write path and drop the
redundant '= null' initializers (fields already default to null). The 5-arg
@RequiredArgsConstructor is untouched, so existing call sites still compile.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
8 files changed
tree: cc2a83792ab48ae26f10553b9cbd5c9d9faf6004
  1. .github/
  2. bin/
  3. buildSrc/
  4. conf/
  5. config/
  6. dev/
  7. gobblin-admin/
  8. gobblin-all/
  9. gobblin-api/
  10. gobblin-audit/
  11. gobblin-aws/
  12. gobblin-binary-management/
  13. gobblin-cluster/
  14. gobblin-compaction/
  15. gobblin-completeness/
  16. gobblin-config-management/
  17. gobblin-core/
  18. gobblin-core-base/
  19. gobblin-data-management/
  20. gobblin-distribution/
  21. gobblin-docker/
  22. gobblin-docs/
  23. gobblin-example/
  24. gobblin-hive-registration/
  25. gobblin-iceberg/
  26. gobblin-kubernetes/
  27. gobblin-metastore/
  28. gobblin-metrics-libs/
  29. gobblin-modules/
  30. gobblin-oozie/
  31. gobblin-rest-service/
  32. gobblin-restli/
  33. gobblin-runtime/
  34. gobblin-runtime-hadoop/
  35. gobblin-salesforce/
  36. gobblin-service/
  37. gobblin-temporal/
  38. gobblin-test/
  39. gobblin-test-harness/
  40. gobblin-test-utils/
  41. gobblin-tunnel/
  42. gobblin-utility/
  43. gobblin-yarn/
  44. gradle/
  45. ligradle/
  46. maven-nexus/
  47. maven-sonatype/
  48. .asf.yaml
  49. .codecov_bash
  50. .dockerignore
  51. .gitignore
  52. build.gradle
  53. CHANGELOG.md
  54. defaultEnvironment.gradle
  55. FlowTriggerHandlerTest.java
  56. gobblin-flavored-build.gradle
  57. gradle.properties
  58. gradlew
  59. gradlew.bat
  60. HEADER
  61. LICENSE
  62. mkdocs.yml
  63. NOTICE
  64. query_github_issues.py
  65. README.md
  66. readthedocs.yml
  67. settings.gradle
README.md

Apache Gobblin

Build Status Documentation Status Maven Central Stack Overflow Join us on Slack codecov.io

Apache Gobblin is a highly scalable data management solution for structured and byte-oriented data in heterogeneous data ecosystems.

Capabilities

  • Ingestion and export of data from a variety of sources and sinks into and out of the data lake. Gobblin is optimized and designed for ELT patterns with inline transformations on ingest (small t).
  • Data Organization within the lake (e.g. compaction, partitioning, deduplication)
  • Lifecycle Management of data within the lake (e.g. data retention)
  • Compliance Management of data across the ecosystem (e.g. fine-grain data deletions)

Highlights

  • Battle tested at scale: Runs in production at petabyte-scale at companies like LinkedIn, PayPal, Verizon etc.
  • Feature rich: Supports task partitioning, state management for incremental processing, atomic data publishing, data quality checking, job scheduling, fault tolerance etc.
  • Supports stream and batch execution modes
  • Control Plane (Gobblin-as-a-service) supports programmatic triggering and orchestration of data plane operations.

Common Patterns used in production

  • Stream / Batch ingestion of Kafka to Data Lake (HDFS, S3, ADLS)
  • Bulk-loading serving stores from the Data Lake (e.g. HDFS -> Couchbase)
  • Support for data sync across Federated Data Lake (HDFS <-> HDFS, HDFS <-> S3, S3 <-> ADLS)
  • Integrate external vendor API-s (e.g. Salesforce, Dynamics etc.) with data store (HDFS, Couchbase etc)
  • Enforcing Data retention policies and GDPR deletion on HDFS / ADLS

Apache Gobblin is NOT

  • A general purpose data transformation engine like Spark or Flink. Gobblin can delegate complex-data processing tasks to Spark, Hive etc.
  • A data storage system like Apache Kafka or HDFS. Gobblin integrates with these systems as sources or sinks.
  • A general-purpose workflow execution system like Airflow, Azkaban, Dagster, Luigi.

Requirements

  • Java >= 1.8

If building the distribution with tests turned on:

  • Maven version 3.5.3

Instructions to download gradle wrapper

If you are going to build Gobblin from the source distribution, run the following command for downloading the gradle-wrapper.jar from Gobblin git repository to gradle/wrapper directory (replace GOBBLIN_VERSION in the URL with the version you downloaded).

wget --no-check-certificate -P gradle/wrapper https://github.com/apache/gobblin/raw/${GOBBLIN_VERSION}/gradle/wrapper/gradle-wrapper.jar

(or)

curl --insecure -L https://github.com/apache/gobblin/raw/${GOBBLIN_VERSION}/gradle/wrapper/gradle-wrapper.jar > gradle/wrapper/gradle-wrapper.jar

Alternatively, you can download it manually from: https://github.com/apache/gobblin/blob/${GOBBLIN_VERSION}/gradle/wrapper/gradle-wrapper.jar

Make sure that you download it to gradle/wrapper directory.

Instructions to run Apache RAT (Release Audit Tool)

  1. Extract the archive file to your local directory.
  2. Run ./gradlew rat. Report will be generated under build/rat/rat-report.html

Instructions to build the distribution

  1. Extract the archive file to your local directory.
  2. Skip tests and build the distribution: Run ./gradlew build -x findbugsMain -x test -x rat -x checkstyleMain The distribution will be created in build/gobblin-distribution/distributions directory. (or)
  3. Run tests and build the distribution (requires Maven): Run ./gradlew build The distribution will be created in build/gobblin-distribution/distributions directory.

Quick Links