| --- |
| layout: default |
| |
| # Top navigation |
| top-nav-group: admin-guide |
| top-nav-pos: 4 |
| top-nav-title: Monitoring |
| |
| # Sub-level navigation |
| sub-nav-group: admin-guide |
| sub-nav-parent: admin-guide |
| sub-nav-id: monitoring |
| sub-nav-pos: 4 |
| sub-nav-title: Monitoring |
| --- |
| |
| .. contents:: Monitoring |
| |
| Monitoring |
| ========== |
| |
| DistributedLog uses the stats library provided by Apache BookKeeper for reporting metrics in |
| both the server and the client. This can be configured to report stats using pluggable stats |
| provider to integrate with your monitoring system. |
| |
| Stats Provider |
| ~~~~~~~~~~~~~~ |
| |
| `StatsProvider` is a provider that provides different kinds of stats logger for different scopes. |
| The provider is also responsible for reporting its managed metrics. |
| |
| :: |
| |
| // Create the stats provider |
| StatsProvider statsProvider = ...; |
| // Start the stats provider |
| statsProvider.start(conf); |
| // Stop the stats provider |
| statsProvider.stop(); |
| |
| Stats Logger |
| ____________ |
| |
| A scoped `StatsLogger` is a stats logger that records 3 kinds of statistics |
| under a given `scope`. |
| |
| A `StatsLogger` could be either created by obtaining from stats provider with |
| the scope name: |
| |
| :: |
| |
| StatsProvider statsProvider = ...; |
| StatsLogger statsLogger = statsProvider.scope("test-scope"); |
| |
| Or created by obtaining from a stats logger with a sub scope name: |
| |
| :: |
| |
| StatsLogger rootStatsLogger = ...; |
| StatsLogger subStatsLogger = rootStatsLogger.scope("sub-scope"); |
| |
| All the metrics in a stats provider are managed in a hierarchical of scopes. |
| |
| :: |
| |
| // all stats recorded by `rootStatsLogger` are under 'root' |
| StatsLogger rootStatsLogger = statsProvider.scope("root"); |
| // all stats recorded by 'subStatsLogger1` are under 'root/scope1' |
| StatsLogger subStatsLogger1 = statsProvider.scope("scope1"); |
| // all stats recorded by 'subStatsLogger2` are under 'root/scope2' |
| StatsLogger subStatsLogger2 = statsProvider.scope("scope2"); |
| |
| Counters |
| ++++++++ |
| |
| A `Counter` is a cumulative metric that represents a single numerical value. A **counter** |
| is typically used to count requests served, tasks completed, errors occurred, etc. Counters |
| should not be used to expose current counts of items whose number can also go down, e.g. |
| the number of currently running tasks. Use `Gauges` for this use case. |
| |
| To change a counter, use: |
| |
| :: |
| |
| StatsLogger statsLogger = ...; |
| Counter births = statsLogger.getCounter("births"); |
| // increment the counter |
| births.inc(); |
| // decrement the counter |
| births.dec(); |
| // change the counter by delta |
| births.add(-10); |
| // reset the counter |
| births.reset(); |
| |
| Gauges |
| ++++++ |
| |
| A `Gauge` is a metric that represents a single numerical value that can arbitrarily go up and down. |
| |
| Gauges are typically used for measured values like temperatures or current memory usage, but also |
| "counts" that can go up and down, like the number of running tasks. |
| |
| To define a gauge, stick the following code somewhere in the initialization: |
| |
| :: |
| |
| final AtomicLong numPendingRequests = new AtomicLong(0L); |
| StatsLogger statsLogger = ...; |
| statsLogger.registerGauge( |
| "num_pending_requests", |
| new Gauge<Number>() { |
| @Override |
| public Number getDefaultValue() { |
| return 0; |
| } |
| @Override |
| public Number getSample() { |
| return numPendingRequests.get(); |
| } |
| }); |
| |
| The gauge must always return a numerical value when sampling. |
| |
| Metrics (OpStats) |
| +++++++++++++++++ |
| |
| A `OpStats` is a set of metrics that represents the statistics of an `operation`. Those metrics |
| include `success` or `failure` of the operations and its distribution (also known as `Histogram`). |
| It is usually used for timing. |
| |
| :: |
| |
| StatsLogger statsLogger = ...; |
| OpStatsLogger writeStats = statsLogger.getOpStatsLogger("writes"); |
| long writeLatency = ...; |
| |
| // register success op |
| writeStats.registerSuccessfulEvent(writeLatency); |
| |
| // register failure op |
| writeStats.registerFailedEvent(writeLatency); |
| |
| Available Stats Providers |
| ~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| All the available stats providers are listed as below: |
| |
| * Twitter Science Stats (deprecated) |
| * Twitter Ostrich Stats (deprecated) |
| * Twitter Finagle Stats |
| * Codahale Stats |
| |
| Twitter Science Stats |
| _____________________ |
| |
| Use following dependency to enable Twitter science stats provider. |
| |
| :: |
| |
| <dependency> |
| <groupId>org.apache.bookkeeper.stats</groupId> |
| <artifactId>twitter-science-provider</artifactId> |
| <version>${bookkeeper.version}</version> |
| </dependency> |
| |
| Construct the stats provider for clients. |
| |
| :: |
| |
| StatsProvider statsProvider = new TwitterStatsProvider(); |
| DistributedLogConfiguration conf = ...; |
| |
| // starts the stats provider (optional) |
| statsProvider.start(conf); |
| |
| // all the dl related stats are exposed under "dlog" |
| StatsLogger statsLogger = statsProvider.getStatsLogger("dlog"); |
| DistributedLogNamespace namespace = DistributedLogNamespaceBuilder.newBuilder() |
| .uri(...) |
| .conf(conf) |
| .statsLogger(statsLogger) |
| .build(); |
| |
| ... |
| |
| // stop the stats provider (optional) |
| statsProvider.stop(); |
| |
| |
| Expose the stats collected by the stats provider by configuring following settings: |
| |
| :: |
| |
| // enable exporting the stats |
| statsExport=true |
| // exporting the stats at port 8080 |
| statsHttpPort=8080 |
| |
| |
| If exporting stats is enabled, all the stats are exported by the http endpoint. |
| You could curl the http endpoint to check the stats. |
| |
| :: |
| |
| curl -s <host>:8080/vars |
| |
| |
| check ScienceStats_ for more details. |
| |
| .. _ScienceStats: https://github.com/twitter/commons/tree/master/src/java/com/twitter/common/stats |
| |
| Twitter Ostrich Stats |
| _____________________ |
| |
| Use following dependency to enable Twitter ostrich stats provider. |
| |
| :: |
| |
| <dependency> |
| <groupId>org.apache.bookkeeper.stats</groupId> |
| <artifactId>twitter-ostrich-provider</artifactId> |
| <version>${bookkeeper.version}</version> |
| </dependency> |
| |
| Construct the stats provider for clients. |
| |
| :: |
| |
| StatsProvider statsProvider = new TwitterOstrichProvider(); |
| DistributedLogConfiguration conf = ...; |
| |
| // starts the stats provider (optional) |
| statsProvider.start(conf); |
| |
| // all the dl related stats are exposed under "dlog" |
| StatsLogger statsLogger = statsProvider.getStatsLogger("dlog"); |
| DistributedLogNamespace namespace = DistributedLogNamespaceBuilder.newBuilder() |
| .uri(...) |
| .conf(conf) |
| .statsLogger(statsLogger) |
| .build(); |
| |
| ... |
| |
| // stop the stats provider (optional) |
| statsProvider.stop(); |
| |
| |
| Expose the stats collected by the stats provider by configuring following settings: |
| |
| :: |
| |
| // enable exporting the stats |
| statsExport=true |
| // exporting the stats at port 8080 |
| statsHttpPort=8080 |
| |
| |
| If exporting stats is enabled, all the stats are exported by the http endpoint. |
| You could curl the http endpoint to check the stats. |
| |
| :: |
| |
| curl -s <host>:8080/stats.txt |
| |
| |
| check Ostrich_ for more details. |
| |
| .. _Ostrich: https://github.com/twitter/ostrich |
| |
| Twitter Finagle Metrics |
| _______________________ |
| |
| Use following dependency to enable bridging finagle stats receiver to bookkeeper's stats provider. |
| All the stats exposed by the stats provider will be collected by finagle stats receiver and exposed |
| by Twitter's admin service. |
| |
| :: |
| |
| <dependency> |
| <groupId>org.apache.bookkeeper.stats</groupId> |
| <artifactId>twitter-finagle-provider</artifactId> |
| <version>${bookkeeper.version}</version> |
| </dependency> |
| |
| Construct the stats provider for clients. |
| |
| :: |
| |
| StatsReceiver statsReceiver = ...; // finagle stats receiver |
| StatsProvider statsProvider = new FinagleStatsProvider(statsReceiver); |
| DistributedLogConfiguration conf = ...; |
| |
| // the stats provider does nothing on start. |
| statsProvider.start(conf); |
| |
| // all the dl related stats are exposed under "dlog" |
| StatsLogger statsLogger = statsProvider.getStatsLogger("dlog"); |
| DistributedLogNamespace namespace = DistributedLogNamespaceBuilder.newBuilder() |
| .uri(...) |
| .conf(conf) |
| .statsLogger(statsLogger) |
| .build(); |
| |
| ... |
| |
| // the stats provider does nothing on stop. |
| statsProvider.stop(); |
| |
| |
| check `finagle metrics library`__ for more details on how to expose the stats. |
| |
| .. _TwitterServer: https://twitter.github.io/twitter-server/Migration.html |
| |
| __ TwitterServer_ |
| |
| Codahale Metrics |
| ________________ |
| |
| Use following dependency to enable Twitter ostrich stats provider. |
| |
| :: |
| |
| <dependency> |
| <groupId>org.apache.bookkeeper.stats</groupId> |
| <artifactId>codahale-metrics-provider</artifactId> |
| <version>${bookkeeper.version}</version> |
| </dependency> |
| |
| Construct the stats provider for clients. |
| |
| :: |
| |
| StatsProvider statsProvider = new CodahaleMetricsProvider(); |
| DistributedLogConfiguration conf = ...; |
| |
| // starts the stats provider (optional) |
| statsProvider.start(conf); |
| |
| // all the dl related stats are exposed under "dlog" |
| StatsLogger statsLogger = statsProvider.getStatsLogger("dlog"); |
| DistributedLogNamespace namespace = DistributedLogNamespaceBuilder.newBuilder() |
| .uri(...) |
| .conf(conf) |
| .statsLogger(statsLogger) |
| .build(); |
| |
| ... |
| |
| // stop the stats provider (optional) |
| statsProvider.stop(); |
| |
| |
| Expose the stats collected by the stats provider in different ways by configuring following settings. |
| Check Codehale_ on how to configuring report endpoints. |
| |
| :: |
| |
| // How frequent report the stats |
| codahaleStatsOutputFrequencySeconds=... |
| // The prefix string of codahale stats |
| codahaleStatsPrefix=... |
| |
| // |
| // Report Endpoints |
| // |
| |
| // expose the stats to Graphite |
| codahaleStatsGraphiteEndpoint=... |
| // expose the stats to CSV files |
| codahaleStatsCSVEndpoint=... |
| // expose the stats to Slf4j logging |
| codahaleStatsSlf4jEndpoint=... |
| // expose the stats to JMX endpoint |
| codahaleStatsJmxEndpoint=... |
| |
| |
| check Codehale_ for more details. |
| |
| .. _Codehale: https://dropwizard.github.io/metrics/3.1.0/ |
| |
| Enable Stats Provider on Bookie Servers |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| The stats provider used by *Bookie Servers* is configured by setting the following option. |
| |
| :: |
| |
| // class of stats provider |
| statsProviderClass="org.apache.bookkeeper.stats.CodahaleMetricsProvider" |
| |
| Metrics |
| ~~~~~~~ |
| |
| Check the Metrics_ reference page for the metrics exposed by DistributedLog. |
| |
| .. _Metrics: ../user_guide/references/metrics |