A collection of resources for OpenSearch:
as well as resources for building basic real-time monitoring dashboards for the crawls, see below.
This module is ported from the Elasticsearch one.
The easiest way is currently to use the archetype for OpenSearch with:
mvn archetype:generate -DarchetypeGroupId=org.apache.stormcrawler -DarchetypeArtifactId=stormcrawler-opensearch-archetype -DarchetypeVersion=<CURRENT_VERSION>
Be sure to replace <CURRENT_VERSION> with the latest released version, which you can find on search.maven.org.
You'll be asked to enter a groupId (e.g. com.mycompany.crawler), an artefactId (e.g. stormcrawler), a version, a package name and details about the user agent to use.
This will not only create a fully formed project containing a POM with the dependency above but also a set of resources, configuration files and a topology class. Enter the directory you just created (should be the same as the artefactId you specified earlier) and follow the instructions on the README file.
You will of course need to have both Storm and OpenSearch installed. For the latter, the OpenSearch documentation contains resources for Docker.
Unlike in the Elastic module, the schemas are automatically created by the bolts. You can of course override them by using the script ‘OS_IndexInit.sh’ generated by the archetype, the index definitions are located in src/main/resources.
To import the dashboards into a local instance of OpenSearch Dashboard, go into the folder dashboards and run the script importDashboards.sh.
You should see something like
Importing status dashboard into OpenSearch Dashboards
{"successCount":4,"success":true,"successResults":[{"type":"index-pattern","id":"7445c390-7339-11e9-9289-ffa3ee6775e4","meta":{"title":"status","icon":"indexPatternApp"}},{"type":"visualization","id":"status-count","meta":{"title":"status count","icon":"visualizeApp"}},{"type":"visualization","id":"Top-Hosts","meta":{"title":"Top Hosts","icon":"visualizeApp"}},{"type":"dashboard","id":"Crawl-status","meta":{"title":"Crawl status","icon":"dashboardApp"}}]}
Importing metrics dashboard into OpenSearch Dashboards
{"successCount":9,"success":true,"successResults":[{"type":"index-pattern","id":"b5c3bbd0-7337-11e9-9289-ffa3ee6775e4","meta":{"title":"metrics","icon":"indexPatternApp"}},{"type":"visualization","id":"Fetcher-:-#-active-threads","meta":{"title":"Fetcher : # active threads","icon":"visualizeApp"}},{"type":"visualization","id":"Fetcher-:-num-queues","meta":{"title":"Fetcher : num queues","icon":"visualizeApp"}},{"type":"visualization","id":"Fetcher-:-pages-fetched","meta":{"title":"Fetcher : pages fetched","icon":"visualizeApp"}},{"type":"visualization","id":"Fetcher-:-URLs-waiting-in-queues","meta":{"title":"Fetcher : URLs waiting in queues","icon":"visualizeApp"}},{"type":"visualization","id":"Fetcher-:-average-bytes-per-second","meta":{"title":"Fetcher : average bytes per second","icon":"visualizeApp"}},{"type":"visualization","id":"Fetcher-:-average-pages-per-second","meta":{"title":"Fetcher : average pages per second","icon":"visualizeApp"}},{"type":"visualization","id":"Total-bytes-fetched","meta":{"title":"Total bytes fetched","icon":"visualizeApp"}},{"type":"dashboard","id":"Crawl-metrics","meta":{"title":"Crawl metrics","icon":"dashboardApp"}}]}
The dashboard screen should show both the status and metrics dashboards. If you click on Crawl Status, you should see 2 tables containing the count of URLs per status and the top hostnames per URL count. The Metrics dashboard can be used to monitor the progress of the crawl.
The file storm.ndjson is used to display some of Storm's internal metrics and is not added by default.
The metrics index can be configured per time period. This best practice is discussed on the Elastic website.
The crawler config YAML must be updated to use an optional argument as shown below to have one index per day:
#Metrics consumers:
topology.metrics.consumer.register:
- class: "org.apache.stormcrawler.opensearch.metrics.MetricsConsumer"
parallelism.hint: 1
argument: "yyyy-MM-dd"