stormcrawler-opensearch-java

A collection of resources for OpenSearch built on the OpenSearch Java Client 3.x and Apache HttpClient 5:

This module is functionally equivalent to the legacy external/opensearch module (which is based on the deprecated RestHighLevelClient and HttpClient 4), but uses the typed OpenSearchClient and the ApacheHttpClient5TransportBuilder transport. Unlike the legacy client, the Java Client 3.x no longer ships a sniffer nor a built-in BulkProcessor; this module provides an internal AsyncBulkProcessor that preserves the same semantics (size/count/time based flushing, back-pressure, listener callbacks).

Getting started

Add the dependency to your crawler project:

<dependency>
    <groupId>org.apache.stormcrawler</groupId>
    <artifactId>stormcrawler-opensearch-java</artifactId>
    <version>${stormcrawler.version}</version>
</dependency>

You will of course need to have both Storm and OpenSearch installed. For the latter, see the OpenSearch documentation for Docker-based setups.

Schemas are automatically created by the bolts on first use; you can override them by providing your own index definitions before starting the topology.

Configuration and dashboards

For a ready-to-use crawler configuration, example Flux topologies, index initialization scripts and OpenSearch Dashboards exports, refer to the external/opensearch module: all of those resources are compatible with this module and have not been duplicated here.

Differences from the legacy external/opensearch module

  • opensearch.<bolt>.responseBufferSize is no longer supported. The legacy module used the HC4-based low-level REST client and set a heap response buffer via HeapBufferedResponseConsumerFactory. The HC5-based async transport used here does not expose an equivalent per-request override, so the key is ignored. A WARN is logged at startup if it is found in the configuration; remove it when migrating.
  • opensearch.<bolt>.sniff is no longer supported. The legacy module enabled node auto-discovery by default via the low-level REST client Sniffer. The OpenSearch Java Client 3.x does not ship a sniffer equivalent, so this feature is dropped. Keep the addresses list up to date manually or put a load balancer in front of the cluster. A WARN is logged at startup if the key is found in the configuration; remove it when migrating.
  • Date fields (nextFetchDate in the status index and timestamp in the metrics index) are serialized as ISO-8601 strings produced by Instant#toString() (for example 2026-01-01T00:00:00Z). The legacy module serializes the same fields through XContentBuilder#timeField, which emits ISO-8601 strings via ISODateTimeFormat.dateTime() (for example 2026-01-01T00:00:00.000Z). Both representations are accepted by the default OpenSearch date mapping and by the date_optional_time / strict_date_optional_time formats used in the example mappings under src/test/resources/. If you run a custom mapping that restricts the field format to epoch_millis, update it to accept ISO-8601 (for example strict_date_optional_time||epoch_millis) before writing with this module.