GEODE-7390: Add Micrometer metrics example (#90)
Add example to demonstrate publishing metrics from Geode to a monitoring
system comprised of Prometheus and Grafana.
Co-authored-by: Aaron Lindsey <alindsey@pivotal.io>
Co-authored-by: Dale Emery <demery@pivotal.io>
Co-authored-by: Mark Hanson <mhanson@pivotal.io>
Co-authored-by: Nick Vallely <nvallely@pivotal.io>
Authored-by: Aaron Lindsey <alindsey@pivotal.io>
diff --git a/README.md b/README.md
index 00fade2..0c8a667 100644
--- a/README.md
+++ b/README.md
@@ -89,6 +89,7 @@
* [Lucene Spatial Indexing](luceneSpatial/README.md)
* [WAN Gateway](wan/README.md)
* [Durable Messaging for Subscriptions](durableMessaging/README.md)
+* [Micrometer Metrics](micrometerMetrics/README.md)
* Delta propagation
* Network partition detection
* D-lock
diff --git a/gradle.properties b/gradle.properties
index 4bbd66b..4328334 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -29,3 +29,4 @@
mockitocoreVersion = 2.19.1
log4jVersion = 2.11.0
systemrulesVersion = 1.16.1
+micrometerVersion = 1.2.0
diff --git a/gradle/rat.gradle b/gradle/rat.gradle
index a37298a..2aedfe0 100644
--- a/gradle/rat.gradle
+++ b/gradle/rat.gradle
@@ -74,7 +74,9 @@
'**/server-ln-2/**',
'**/locator-ny/**',
'**/server-ny-1/**',
- '**/server-ny-2/**'
+ '**/server-ny-2/**',
+
+ '**/META-INF/**'
]
}
diff --git a/micrometerMetrics/README.md b/micrometerMetrics/README.md
new file mode 100644
index 0000000..225ca4e
--- /dev/null
+++ b/micrometerMetrics/README.md
@@ -0,0 +1,87 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Geode Micrometer Metrics Example
+
+This example demonstrates publishing metrics from Geode to a monitoring system comprised of
+Prometheus and Grafana. For more details about how this works, see [Publishing Geode Metrics to
+External Monitoring
+Systems](https://cwiki.apache.org/confluence/display/GEODE/Publishing+Geode+Metrics+to+External+Monitoring+Systems).
+
+## Steps to Run and Validate the Example
+
+1. From the `geode-examples/micrometerMetrics` directory, run the `start` task to build the example
+ and start a cluster. The cluster will have one locator, one server, and a single region. The
+ locator and server will expose HTTP endpoints for Prometheus to scrape.
+
+ $ ../gradlew start
+
+1. Run the example to put entries into the region and verify that the HTTP endpoints are working.
+
+ $ ../gradlew run
+
+ The previous command should produce output like the following:
+
+ The entry count for region example-region on the server is 10.
+ A Prometheus endpoint is running at http://localhost:9914.
+ A Prometheus endpoint is running at http://localhost:9915.
+
+1. To validate the example, navigate to the endpoints above in a browser. You should see text
+ resembling the [Prometheus exposition
+ format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md).
+
+## (Optional) Viewing Metrics Using Prometheus and Grafana
+
+1. Download and run Prometheus. Go to <https://prometheus.io/download> to download the
+ latest release. Then, run the commands below to extract and run it:
+
+ $ tar xvfz prometheus-*.tar.gz
+ $ cd prometheus-*
+ $ ./prometheus --config.file=../prometheus.yml
+
+ The provided `prometheus.yml` file configures Prometheus to scrape Geode's Prometheus endpoints
+ at two-second intervals.
+
+1. In a browser, navigate to <http://localhost:9090/targets>. This page shows Geode's Prometheus
+ endpoints, and their state should be "UP" as shown below:
+
+ 
+
+1. Navigate to
+ <http://localhost:9090/graph?g0.range_input=30m&g0.expr=geode_cache_entries&g0.tab=0>. This page
+ shows the result of the PromQL query `geode_cache_entries` over the last 30 minutes. It should
+ show a value of "10" as the entry count for region `example-region` on the server:
+
+ 
+
+1. Download and run Grafana. Add a new Prometheus data source with the address of the Prometheus
+ server from the previous step. For detailed instructions, see Grafana's [Getting Started
+ Guide](https://grafana.com/docs/guides/getting_started/).
+
+1. Once you have added the Prometheus data source, you can import [this
+ dashboard](https://grafana.com/grafana/dashboards/11060) from Grafana.com.
+
+ To import, select the "plus" icon on the left-hand side and choose the "Import" option. Use the
+ ID from the link above for the dashboard and specify the Prometheus data source you created in
+ the last step.
+
+## Clean Up
+1. Shut down the cluster.
+
+ $ cd ..
+ $ ../gradlew stop
+
diff --git a/micrometerMetrics/build.gradle b/micrometerMetrics/build.gradle
new file mode 100644
index 0000000..45ebcd4
--- /dev/null
+++ b/micrometerMetrics/build.gradle
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+configurations {
+ dependenciesToIncludeInEndpointJar
+}
+
+dependencies {
+ dependenciesToIncludeInEndpointJar "io.micrometer:micrometer-registry-prometheus:$micrometerVersion"
+ configurations.compile.extendsFrom(configurations.dependenciesToIncludeInEndpointJar)
+}
+
+jar {
+ from {
+ configurations.dependenciesToIncludeInEndpointJar.collect { it.isDirectory() ? it : zipTree(it) }
+ }
+}
diff --git a/micrometerMetrics/prometheus-graph.png b/micrometerMetrics/prometheus-graph.png
new file mode 100644
index 0000000..ac3122c
--- /dev/null
+++ b/micrometerMetrics/prometheus-graph.png
Binary files differ
diff --git a/micrometerMetrics/prometheus-targets.png b/micrometerMetrics/prometheus-targets.png
new file mode 100644
index 0000000..5eb7dbc
--- /dev/null
+++ b/micrometerMetrics/prometheus-targets.png
Binary files differ
diff --git a/micrometerMetrics/prometheus.yml b/micrometerMetrics/prometheus.yml
new file mode 100644
index 0000000..100ad44
--- /dev/null
+++ b/micrometerMetrics/prometheus.yml
@@ -0,0 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+# agreements. See the NOTICE file distributed with this work for additional information regarding
+# copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance with the License. You may obtain a
+# copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+# or implied. See the License for the specific language governing permissions and limitations under
+# the License.
+
+global:
+ scrape_interval: 2s
+scrape_configs:
+ - job_name: 'geode-examples'
+ metrics_path: /
+ static_configs:
+ - targets: ['localhost:9914']
+ - targets: ['localhost:9915']
diff --git a/micrometerMetrics/scripts/start.gfsh b/micrometerMetrics/scripts/start.gfsh
new file mode 100644
index 0000000..1bf3a2e
--- /dev/null
+++ b/micrometerMetrics/scripts/start.gfsh
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# ports 9914 and 9915 chosen at random
+
+start locator --name=locator --bind-address=127.0.0.1 --classpath=../build/libs/micrometerMetrics.jar --J=-Dprometheus.metrics.port=9914
+
+start server --name=server --locators=127.0.0.1[10334] --server-port=0 --classpath=../build/libs/micrometerMetrics.jar --J=-Dprometheus.metrics.port=9915
+
+create region --name=example-region --type=REPLICATE
diff --git a/micrometerMetrics/scripts/stop.gfsh b/micrometerMetrics/scripts/stop.gfsh
new file mode 100644
index 0000000..15cd93c
--- /dev/null
+++ b/micrometerMetrics/scripts/stop.gfsh
@@ -0,0 +1,19 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+connect --locator=127.0.0.1[10334]
+shutdown --include-locators=true
diff --git a/micrometerMetrics/src/main/java/org/apache/geode_examples/micrometerMetrics/Example.java b/micrometerMetrics/src/main/java/org/apache/geode_examples/micrometerMetrics/Example.java
new file mode 100644
index 0000000..079ff81
--- /dev/null
+++ b/micrometerMetrics/src/main/java/org/apache/geode_examples/micrometerMetrics/Example.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode_examples.micrometerMetrics;
+
+import java.io.IOException;
+import java.net.HttpURLConnection;
+import java.net.URL;
+import java.util.stream.IntStream;
+
+import org.apache.geode.cache.Region;
+import org.apache.geode.cache.client.ClientCache;
+import org.apache.geode.cache.client.ClientCacheFactory;
+import org.apache.geode.cache.client.ClientRegionShortcut;
+
+public class Example {
+ public static void main(String[] args) {
+ addCacheEntries();
+ verifyPrometheusEndpointsAreRunning();
+ }
+
+ private static void addCacheEntries() {
+ // connect to the locator using default port
+ ClientCache cache = new ClientCacheFactory().addPoolLocator("localhost", 10334)
+ .set("log-level", "WARN").create();
+
+ // create a local region that matches the server region
+ Region<Integer, String> region =
+ cache.<Integer, String>createClientRegionFactory(ClientRegionShortcut.PROXY)
+ .create("example-region");
+
+ // add entries to the region
+ IntStream.rangeClosed(1, 10).forEach(i -> region.put(i, "value" + i));
+
+ System.out.println(String.format("The entry count for region %s on the server is %d.",
+ region.getName(), region.sizeOnServer()));
+
+ cache.close();
+ }
+
+ private static void verifyPrometheusEndpointsAreRunning() {
+ String[] endpoints = {"http://localhost:9914", "http://localhost:9915"};
+
+ for (String endpoint : endpoints) {
+ try {
+ URL url = new URL(endpoint);
+ HttpURLConnection connection = (HttpURLConnection) url.openConnection();
+ connection.connect();
+
+ if (HttpURLConnection.HTTP_OK != connection.getResponseCode()) {
+ throw new IllegalStateException(
+ "Prometheus endpoint returned status code " + connection.getResponseCode());
+ }
+ } catch (IOException e) {
+ throw new IllegalStateException("Failed to connect to Prometheus endpoint", e);
+ }
+
+ System.out.println("A Prometheus endpoint is running at " + endpoint + ".");
+ }
+ }
+}
diff --git a/micrometerMetrics/src/main/java/org/apache/geode_examples/micrometerMetrics/SimpleMetricsPublishingService.java b/micrometerMetrics/src/main/java/org/apache/geode_examples/micrometerMetrics/SimpleMetricsPublishingService.java
new file mode 100644
index 0000000..1880ce2
--- /dev/null
+++ b/micrometerMetrics/src/main/java/org/apache/geode_examples/micrometerMetrics/SimpleMetricsPublishingService.java
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode_examples.micrometerMetrics;
+
+import static io.micrometer.prometheus.PrometheusConfig.DEFAULT;
+import static java.lang.Integer.getInteger;
+import static org.slf4j.LoggerFactory.getLogger;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+
+import com.sun.net.httpserver.HttpContext;
+import com.sun.net.httpserver.HttpExchange;
+import com.sun.net.httpserver.HttpServer;
+import io.micrometer.prometheus.PrometheusMeterRegistry;
+import org.slf4j.Logger;
+
+import org.apache.geode.metrics.MetricsPublishingService;
+import org.apache.geode.metrics.MetricsSession;
+
+public class SimpleMetricsPublishingService implements MetricsPublishingService {
+ private static final String PORT_PROPERTY = "prometheus.metrics.port";
+ private static final int DEFAULT_PORT = 0; // If no port specified, use any port
+ private static final String HOSTNAME = "localhost";
+ private static final int PORT = getInteger(PORT_PROPERTY, DEFAULT_PORT);
+
+ private static Logger LOG = getLogger(SimpleMetricsPublishingService.class);
+
+ private final int port;
+ private PrometheusMeterRegistry registry;
+ private HttpServer server;
+
+ public SimpleMetricsPublishingService() {
+ this(PORT);
+ }
+
+ public SimpleMetricsPublishingService(int port) {
+ this.port = port;
+ }
+
+ @Override
+ public void start(MetricsSession session) {
+ registry = new PrometheusMeterRegistry(DEFAULT);
+ session.addSubregistry(registry);
+
+ InetSocketAddress address = new InetSocketAddress(HOSTNAME, port);
+ server = null;
+ try {
+ server = HttpServer.create(address, 0);
+
+ HttpContext context = server.createContext("/");
+ context.setHandler(this::requestHandler);
+ server.start();
+
+ int boundPort = server.getAddress().getPort();
+ LOG.info("Started {} http://{}:{}/", getClass().getSimpleName(), HOSTNAME, boundPort);
+ } catch (IOException thrown) {
+ LOG.error("Exception while starting " + getClass().getSimpleName(), thrown);
+ }
+ }
+
+ private void requestHandler(HttpExchange httpExchange) throws IOException {
+ final byte[] scrapeBytes = registry.scrape().getBytes();
+ httpExchange.sendResponseHeaders(200, scrapeBytes.length);
+ final OutputStream responseBody = httpExchange.getResponseBody();
+ responseBody.write(scrapeBytes);
+ responseBody.close();
+ }
+
+ @Override
+ public void stop(MetricsSession session) {
+ session.removeSubregistry(registry);
+ registry = null;
+ server.stop(0);
+ }
+}
diff --git a/micrometerMetrics/src/main/resources/META-INF/services/org.apache.geode.metrics.MetricsPublishingService b/micrometerMetrics/src/main/resources/META-INF/services/org.apache.geode.metrics.MetricsPublishingService
new file mode 100644
index 0000000..0bfa3ce
--- /dev/null
+++ b/micrometerMetrics/src/main/resources/META-INF/services/org.apache.geode.metrics.MetricsPublishingService
@@ -0,0 +1 @@
+org.apache.geode_examples.micrometerMetrics.SimpleMetricsPublishingService
\ No newline at end of file
diff --git a/micrometerMetrics/src/test/java/org/apache/geode_examples/micrometerMetrics/SimpleMetricsPublishingServiceTest.java b/micrometerMetrics/src/test/java/org/apache/geode_examples/micrometerMetrics/SimpleMetricsPublishingServiceTest.java
new file mode 100644
index 0000000..39acfac
--- /dev/null
+++ b/micrometerMetrics/src/test/java/org/apache/geode_examples/micrometerMetrics/SimpleMetricsPublishingServiceTest.java
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.geode_examples.micrometerMetrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.catchThrowable;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.verify;
+import static org.mockito.quality.Strictness.STRICT_STUBS;
+
+import java.io.IOException;
+
+import io.micrometer.prometheus.PrometheusMeterRegistry;
+import org.apache.http.HttpResponse;
+import org.apache.http.HttpStatus;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.conn.HttpHostConnectException;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.mockito.Mock;
+import org.mockito.junit.MockitoJUnit;
+import org.mockito.junit.MockitoRule;
+
+import org.apache.geode.metrics.MetricsPublishingService;
+import org.apache.geode.metrics.MetricsSession;
+
+public class SimpleMetricsPublishingServiceTest {
+ @Rule
+ public MockitoRule mockitoRule = MockitoJUnit.rule().strictness(STRICT_STUBS);
+
+ @Mock
+ public MetricsSession metricsSession;
+
+ private MetricsPublishingService subject;
+
+ @Before
+ public void setUp() {
+ subject = new SimpleMetricsPublishingService(9000);
+ }
+
+ @Test
+ public void start_addsRegistryToMetricsSession() {
+ subject.start(metricsSession);
+
+ verify(metricsSession).addSubregistry(any(PrometheusMeterRegistry.class));
+
+ subject.stop(metricsSession);
+ }
+
+ @Test
+ public void start_addsAnHttpEndpointThatReturnsStatusOK() throws IOException {
+ subject.start(metricsSession);
+
+ HttpGet request = new HttpGet("http://localhost:9000/");
+ HttpResponse response = HttpClientBuilder.create().build().execute(request);
+
+ assertThat(response.getStatusLine().getStatusCode()).isEqualTo(HttpStatus.SC_OK);
+
+ subject.stop(metricsSession);
+ }
+
+ @Test
+ public void start_addsAnHttpEndpointThatContainsRegistryData() throws IOException {
+ subject.start(metricsSession);
+
+ HttpGet request = new HttpGet("http://localhost:9000/");
+ HttpResponse response = HttpClientBuilder.create().build().execute(request);
+
+ String responseBody = EntityUtils.toString(response.getEntity());
+ assertThat(responseBody).isEmpty();
+
+ subject.stop(metricsSession);
+ }
+
+ @Test
+ public void stop_removesRegistryFromMetricsSession() {
+ subject.start(metricsSession);
+ subject.stop(metricsSession);
+
+ verify(metricsSession).removeSubregistry(any(PrometheusMeterRegistry.class));
+ }
+
+ @Test
+ public void stop_hasNoHttpEndpointRunning() {
+ subject.start(metricsSession);
+ subject.stop(metricsSession);
+
+ HttpGet request = new HttpGet("http://localhost:9000/");
+
+ Throwable thrown = catchThrowable(() -> HttpClientBuilder.create().build().execute(request));
+
+ assertThat(thrown).isInstanceOf(HttpHostConnectException.class);
+ }
+}
diff --git a/settings.gradle b/settings.gradle
index 57d108d..19ac5c0 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -40,6 +40,7 @@
include 'jdbc'
include 'sessionState'
include 'colocation'
+include 'micrometerMetrics'
// Logic for defining a custom Geode clone for integration with this project
// Define `-PgeodeCompositeDirectory` to your geode root, default `../geode`
