Playwright

Protocol implementation for Apache StormCrawler based on Playwright

Standalone Chrome

Playwright installs the browsers it needs locally, which can take more than 1 GB disk space and takes some time. Instead, you can install Chrome separately and get Playwright to connect to its CDP.

Chrome for Testing is the recommended way of installing a standalone version of Chrome.

You can start it like so

 ./chrome --remote-debugging-port=9222 --user-data-dir=remote-profile --headless &

NOTE this is how ChromeDriver launches Chrome

chrome --allow-pre-commit-input --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-dev-shm-usage --disable-gpu --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-software-rasterizer --disable-sync --dns-prefetch-disable --enable-automation --enable-logging --headless --log-level=0 --mute-audio --no-first-run --no-sandbox --no-service-autorun --password-store=basic --remote-debugging-port=9222 --use-mock-keychain --user-data-dir=/tmp/.org.chromium.Chromium.CPTbw3 data:,

Then URL for the CDP connection is configured with playwright.cdp.url.

Alternatively, you can still rely on what Playwright installs but do it manually and just for the browser type you want

mvn exec:java -e -Dexec.mainClass=com.microsoft.playwright.CLI -Dexec.args="install chromium"

The setting playwright.skip.download to true in the configuration will assume that the browser has been installed and will not trigger the installation of all the different browsers.

Configuration

Per-URL metadata triggers:

Page actions

Custom post-navigate behaviour is added by implementing PageAction and listing the implementation in the JSON file referenced by playwright.page.actions.config.file. Actions follow the same Configurable pattern as URL/parse filters and are loaded as an ordered chain. A failure in one action is logged and swallowed so the rest of the chain still runs.

{
  "org.apache.stormcrawler.protocol.playwright.PageActions": [
    {
      "class": "org.apache.stormcrawler.protocol.playwright.actions.ExpandClickablesAction",
      "name": "tabs",
      "params": {
        "selectors": [".tab-widget .tab-header"],
        "root": ".tab-widget",
        "body": ".tab-widget-body",
        "waitMs": 300
      }
    }
  ]
}

Built-in actions

Writing your own action

Extend PageAction (which extends AbstractConfigurable) and implement apply(Page, url, sourceMetadata, responseMetadata). Read parameters from the supplied JsonNode in configure() and validate at load time — anything thrown there propagates through PageActions.fromConf() and stops the topology from starting with a misconfigured chain.

public class MyAction extends PageAction {

    private String selector;

    @Override
    public void configure(final Map<String, Object> stormConf, final JsonNode params) {
        if (!params.has("selector")) {
            throw new IllegalArgumentException("MyAction requires 'selector'");
        }
        this.selector = params.get("selector").asText();
    }

    @Override
    public void apply(final Page page, final String url,
                      final Metadata sourceMetadata, final Metadata responseMetadata) {
        page.locator(selector).click();
    }
}

Reference it from the chain JSON via its fully-qualified class name. Per-action failures are logged and swallowed by the chain wrapper so one bad action cannot abort the rest — if you need a hard failure, raise it from configure(), not from apply().

Tests

The module has three test classes:

• PageActionsTest — JSON loader / chain construction (no browser).
• actions/ActionConfigureTest — configure() validation for every built-in (no browser).
• PageActionsLiveTest — end-to-end browser tests for the chain and individual actions.

The live tests use the same assumeTrue("false".equals(System.getProperty("CI_ENV", "false"))) gate as ProtocolTest, so they run locally (mvn test) but skip on CI runners launched with -DCI_ENV=true. They expect a usable Chromium — either via mvn exec:java -e -Dexec.mainClass=com.microsoft.playwright.CLI -Dexec.args="install chromium" or by pointing playwright.cdp.url at an existing browser.

JS rendering detection

Browser-based fetching is expensive — typically 10–50× slower than a plain HTTP fetch and limited by how many browsers a host can run concurrently. Most operators only want Playwright on the URLs that actually need it. The JsRenderingDetector parse filter solves the routing question without adding new infrastructure: it inspects the parsed page from a cheap fetch and, when the content looks JS-rendered, sets a metadata flag that DelegatorProtocol (already part of core) routes on.

How detection works

The filter applies four heuristics, cheapest-first, and short-circuits on the first hit:

1. SPA framework fingerprints in raw HTML — data-reactroot, ng-version=, __NEXT_DATA__, window.__NUXT__, data-svelte-h=, data-vue-app, data-astro-cid, <router-outlet. Defaults are overridable via the fingerprints parameter.
2. <noscript> blocks that explicitly request JavaScript — match patterns like “enable JavaScript”, “requires JavaScript”, “JavaScript is disabled”.
3. Empty SPA hydration roots — <div id="root"></div> / #app / #__next / #__nuxt with no children. IDs override­able via emptyRootIds.
4. Outcome-based fallback — when at least one <script> is present and both text.length < minTextLength (default 200) and outlinks.size() < minOutlinks (default 2), the URL is flagged as a thin SPA. The <script> gate keeps the filter from flagging static error stubs.

What the filter sets

Loop guards

• Detection is skipped when playwright.protocol.end is already present on the URL — i.e. the URL was just fetched by Playwright; reapplying the heuristic would just reflag it. Override the watch key via skipIfMetadataPresent.
• Detection is also skipped when the routing key is already set, so the filter is idempotent and safe to leave permanently in parsefilters.json.

Parameters

Wiring

Add the filter to your parsefilters.json:

{
  "class": "org.apache.stormcrawler.protocol.playwright.parsefilter.JsRenderingDetector",
  "name": "js-rendering-detector",
  "params": { "minTextLength": 200, "minOutlinks": 2 }
}

Route on the metadata key it sets via DelegatorProtocol:

http.protocol.implementation:  "org.apache.stormcrawler.protocol.DelegatorProtocol"
https.protocol.implementation: "org.apache.stormcrawler.protocol.DelegatorProtocol"
protocol.delegator.config:
  - className: "org.apache.stormcrawler.protocol.playwright.HttpProtocol"
    filters:
      "fetch.with": "playwright"
  - className: "org.apache.stormcrawler.protocol.okhttp.HttpProtocol"

A few wiring notes:

• The dotted metadata key (fetch.with) is quoted in the YAML above to make it unambiguous to a human reader; SnakeYAML treats unquoted fetch.with: "playwright" as the same single-key scalar, so either parses correctly.
• DelegatorProtocol requires the last entry in protocol.delegator.config to have no filters: — it acts as the fallback. Keep OkHttp (or whichever cheap protocol you pick) at the bottom of the list.
• The filter alone does not trigger an immediate refetch. It only sets the metadata; the URL is rescheduled by DefaultScheduler according to the FETCHED interval (fetchInterval.default, 24h by default), and DelegatorProtocol picks Playwright on the next scheduled fetch. To get faster turnaround, either drop in the JsRenderingRedirectionBolt described below, or add a per-metadata-key fetch interval to your YAML: fetchInterval.fetch.with=playwright: 5 (refetch flagged URLs in 5 minutes instead of 24 hours).
• Sibling URLs on the same host don't inherit the flag — that requires a host-keyed metadata transfer scheme and is intentionally out of scope.

Forcing an immediate refetch — JsRenderingRedirectionBolt

The detector flags URLs but doesn‘t, on its own, prevent the cheap fetch’s stub document from flowing downstream into the parser, indexer, and outlink emission. For most crawls that's fine — the next scheduled fetch replaces the stub with the rendered version. If you want the stub to be discarded and the URL refetched immediately through Playwright, drop JsRenderingRedirectionBolt between the parser and the indexer. The bolt:

• reads the routing flag set by the detector (or any other upstream component),
• on hit, emits only to StatusStreamName with Status.FETCHED so the URL is rescheduled and the stub never reaches the index,
• on miss, passes the tuple through unchanged,
• short-circuits when playwright.protocol.end is already on the URL — the loop guard.

The bolt has no detection logic of its own; it just acts on the metadata flag. That keeps the heuristics in one place (the parse filter) and lets you swap or extend the bolt independently.

Topology fragment:

... -> JSoupParserBolt -> JsRenderingRedirectionBolt -> IndexerBolt -> ...
                                                    \-> StatusStream

YAML:

# refetch flagged URLs in 5 minutes rather than 24 hours
fetchInterval.fetch.with=playwright: 5

Configuration keys:

When not to use it

• Operator allowlist suffices. If you already know which hosts need a browser, add them as a urlPatterns rule on the Playwright leg of DelegatorProtocol and skip the filter.
• Anti-bot / WAF challenge pages. Cloudflare, DataDome, and Akamai challenge fingerprints aren't covered here; those usually need a stealth-mode browser, not just rendering.
• Aggressively first-fetch-sensitive crawls. The first fetch on an unknown SPA host is always wasted (you get a stub document) before the filter learns about the host. If that's unacceptable, prefer the operator allowlist.