Queue stream (#1974) * Opensearch added QueueBolt Signed-off-by: Julien Nioche <julien@digitalpebble.com> * added config for queues index Signed-off-by: Julien Nioche <julien@digitalpebble.com> * follow up merge master Signed-off-by: Julien Nioche <julien@digitalpebble.com> * minor changes Signed-off-by: Julien Nioche <julien@digitalpebble.com> * Fix post merge Signed-off-by: Julien Nioche <julien@digitalpebble.com> * Fix licenses for QueueBolt, add it to crawler.flux and fix its package name in injection.flux Signed-off-by: Julien Nioche <julien@digitalpebble.com> * Add metadata to queue stream + added tests Signed-off-by: Julien Nioche <julien@digitalpebble.com> * #990 - QueueBolt: handle bulk failures, configurable cache, java.time Address review comments on #1974: - use the BulkProcessor.Listener variant of getConnection and only cache keys once the bulk write succeeded, so failed writes are logged and retried when the key is next seen - treat version conflicts (entry already exists, e.g. after a worker restart or cache eviction) as success, keeping create-only semantics - make the known-queues cache configurable via opensearch.queues.cache.spec (default maximumSize=10000) - replace org.joda.time.Instant with java.time.Instant - rename ESIndexNameParamName to OSQueuesIndexNameParamName - presence-only cache values --------- Signed-off-by: Julien Nioche <julien@digitalpebble.com> Co-authored-by: Richard Zowalla <rzo1@apache.org>
Apache StormCrawler is an open source collection of resources for building low-latency, scalable web crawlers on Apache Storm. It is provided under Apache License and is written mostly in Java.
NOTE: These instructions assume that you have Apache Maven installed. You will need to install Apache Storm 2.8.8 to run the crawler.
StormCrawler requires Java 17 or above. To execute tests, it requires you to have a locally installed and working Docker environment.
Once Storm is installed, the easiest way to get started is to generate a new StormCrawler project following the instructions below:
mvn archetype:generate -DarchetypeGroupId=org.apache.stormcrawler -DarchetypeArtifactId=stormcrawler-archetype -DarchetypeVersion=<CURRENT_VERSION>
Be sure to replace <CURRENT_VERSION> with the latest released version, which you can find on search.maven.org.
You'll be asked to enter a groupId (e.g. com.mycompany.crawler), an artefactId (e.g. stormcrawler), a version, a package name and details about the user agent to use.
This will not only create a fully formed project containing a POM with the dependency above but also the default resource files, a default CrawlTopology class and a configuration file. Enter the directory you just created (should be the same as the artefactId you specified earlier) and follow the instructions on the README file.
Alternatively if you can‘t or don’t want to use the Maven archetype above, you can simply copy the files from archetype-resources.
Have a look at crawler.flux, the crawler-conf.yaml file as well as the files in src/main/resources/, they are all that is needed to run a crawl topology : all the other components come from the core module.
The documentation is a good place to start your investigations but if you are stuck please use the tag stormcrawler on StackOverflow or ask a question in the discussions section.
The project website has a page listing companies providing commercial support for Apache StormCrawler.
Please format your code before submitting a PR with
mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false
You can enable pre-commit format hooks by running:
mvn clean install -Dskip.format.code=false
The requirements for building from source are as follows
The build itself is straightforward:
mvn clean install
Note: We use some binary files for testing advanced crawler functionality. These files are located exclusively in the src/test directories of the respective modules.
YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.