tree: 8ef6173a7f41a7510e326cbd504c88294d6e1a7a

tika-grpc/README.md

Tika Pipes GRPC Server

The following is the Tika Pipes GRPC Server.

This server will manage a pool of Tika Pipes clients.

Tika Pipes Fetcher CRUD operations
- Create
- Read
- Update
- Delete
Fetch + Parse a given Fetch Item

Security note: runtime fetcher/iterator management — mutations (Create/Update/Delete) and reading stored configs back (Read), which may contain secrets — plus per-request parse configuration are disabled by default. Enable them explicitly via allowComponentManagement / allowPerRequestConfig in the grpc section of your tika-config; with management off, the Read RPCs return only component id and class, never the config. See the Tika gRPC security configuration docs.

Distribution and Maven Artifact

tika-grpc is designed to be run via Docker — it is not a standalone runnable artifact published to Maven Central.

The Maven artifact for tika-grpc is a thin JAR (~238KB) containing only the compiled classes and resources. It does not include a bundled distribution ZIP or any plugin ZIPs. This was an intentional change (TIKA-4723) to avoid uploading hundreds of megabytes of native libraries and plugin bundles to Nexus on every release.

The runnable Docker image is built using tika-grpc/docker-build/docker-build.sh, which assembles the full classpath and plugins at Docker build time — not during mvn package.
The tika-pipes-plugins (fetchers, emitters, iterators) are also not attached as Maven artifacts. They are packaged as pf4j ZIP plugins and included in the Docker image.
To build the full distribution ZIP locally (e.g. for debugging outside Docker), use the docker Maven profile:
```
mvn package -Pdocker -pl tika-grpc
```
This produces tika-grpc/target/tika-grpc-<version>.zip but does not deploy it to Nexus.

Concurrency

fetchAndParse runs on a pool of forked worker JVMs sized by pipes.numClients (when unset, derived from host cores, at most 4). A call that cannot get a worker within pipes.maxWaitForClientMillis (default 60s) returns the in-band status CLIENT_UNAVAILABLE_WITHIN_MS: the server is at capacity, not failing. With pipes.useSharedServer: true the workers share one JVM instead.

Quick Start - Development Mode

The fastest way to run tika-grpc in development mode with plugin hot-reloading:

# 1. Build Tika and all plugins (from tika project root)
./mvnw clean install -DskipTests

# 2. Run in development mode (from tika-grpc directory)
cd tika-grpc
./run-dev.sh

This will:

Automatically enable tika.plugin.dev.mode=true
Load plugins from ../tika-pipes/tika-pipes-plugins/*/target/classes directories
Start gRPC server on port 50052
Use the dev-tika-config.json configuration

You can also specify a custom config file:

./run-dev.sh my-custom-config.json

Plugin Development Mode

When developing plugins, you can use pf4j's development mode to load plugins directly from their target/classes directories without needing to package them as ZIP files. This significantly speeds up the development cycle.

Enabling Development Mode

Set one of the following:

System Property:

-Dtika.plugin.dev.mode=true

Environment Variable:

export TIKA_PLUGIN_DEV_MODE=true

Maven Dev Profile: (Recommended)

./mvnw exec:java -Pdev -Dconfig.file=dev-tika-config.json

Configuration Example

The dev-tika-config.json file shows how to configure plugin-roots with relative paths:

{
  "plugin-roots": [
    "../tika-pipes/tika-pipes-plugins/tika-pipes-file-system/target/classes",
    "../tika-pipes/tika-pipes-plugins/tika-pipes-http/target/classes",
    "../tika-pipes/tika-pipes-plugins/tika-pipes-s3/target/classes"
  ],
  "fetchers": [
    {
      "fs": {
        "myFetcher": {
          "basePath": "/tmp/input"
        }
      }
    }
  ]
}

Note: Paths are relative to the tika-grpc directory (where you run the server from). You can also use absolute paths if preferred:

{
  "plugin-roots": [
    "/home/user/tika/tika-pipes/tika-pipes-plugins/tika-pipes-s3/target/classes",
    "/home/user/tika/tika-pipes/tika-pipes-plugins/tika-pipes-file-system/target/classes"
  ]
}

Development Workflow

Build the plugin modules (only needed once or when dependencies change):
```
cd tika-pipes/tika-pipes-plugins
./mvnw clean compile
```
Run in development mode using the convenience script:
```
cd tika-grpc
./run-dev.sh
```
Make code changes to your plugin
Recompile just the changed plugin (much faster than full rebuild):
```
# From the project root
./mvnw compile -pl :tika-pipes-s3
```
Restart the server - changes are immediately picked up

What Happens in Development Mode

ZIP extraction is skipped - TikaPluginManager doesn't try to unzip plugins
Plugins loaded from directories - pf4j loads classes directly from target/classes
Each plugin directory must contain plugin.properties in the root (automatically present after ./mvnw compile)
Dependencies are available - The dev profile includes all plugin modules as dependencies

Expected Directory Structure

When pointing to target/classes, pf4j expects this structure:

tika-pipes-s3/target/classes/
├── plugin.properties          # Required: plugin metadata
├── META-INF/
│   └── extensions.idx         # Generated by pf4j annotation processor
└── org/
    └── apache/
        └── tika/
            └── pipes/
                └── fetcher/
                    └── s3/
                        ├── S3Fetcher.class
                        └── S3FetcherFactory.class

IntelliJ IDEA Setup

For IntelliJ IDEA development, here's a complete workflow for developing plugins:

Initial Setup

Import the Tika project as a Maven project in IntelliJ
Build all plugins once (required before first run):
- Open the Maven tool window: View → Tool Windows → Maven
- Navigate to: tika-pipes-plugins (the parent module)
- Right-click → Lifecycle → compile
- Or use terminal:
```
cd tika-pipes/tika-pipes-plugins
./mvnw clean compile
```
Create a Run Configuration for tika-grpc:
- Go to: Run → Edit Configurations
- Click + → Application
- Name: Tika gRPC Server (Dev Mode)
- Main class: org.apache.tika.pipes.grpc.TikaGrpcServer
- VM options: -Dtika.plugin.dev.mode=true
- Program arguments: --config dev-tika-config.json
- Working directory: $PROJECT_DIR$/tika-grpc
- Use classpath of module: tika-grpc
- Click OK

Run the configuration - You should see output like:

INFO  TikaPluginManager running in DEVELOPMENT mode
INFO  PF4J version 3.14.0 in 'development' mode
INFO  Plugin 'tika-pipes-file-system-plugin@4.0.0-SNAPSHOT' resolved
INFO  Plugin 'tika-pipes-s3-plugin@4.0.0-SNAPSHOT' resolved
...
INFO  Server started, listening on 50052

Development Workflow: Making Plugin Changes

Scenario: You want to modify the S3 Fetcher plugin

Navigate to the plugin code:

tika-pipes/tika-pipes-plugins/tika-pipes-s3/src/main/java/

Make your code changes in the plugin source files
- Edit fetcher, emitter, or iterator classes
- Modify configuration handling
- Add new features
Build just the modified plugin module:
- In Project view, right-click on tika-pipes-s3 module
- Select Build Module ‘tika-pipes-s3.main’
- Or use terminal:
```
# From project root
./mvnw compile -pl :tika-pipes-s3
```
- Build time: ~5-10 seconds (much faster than full rebuild!)
Restart the tika-grpc server:
- You MUST restart for changes to take effect
- Click the stop button (red square) in IntelliJ
- Click run button (green arrow) to restart
- PF4J loads plugins at startup - no hot reload without restart
Verify your changes:
- Check the server logs for successful plugin loading
- Test your changes via gRPC client
- Make additional changes and repeat steps 3-4 as needed

Tips for Faster Development

Use IntelliJ's Build shortcuts:

Ctrl+F9 (Windows/Linux) or Cmd+F9 (Mac) - Build project/module
Configure to build only changed files for even faster iteration

Debug mode:

Use “Debug” instead of “Run” to set breakpoints in plugin code
Step through plugin initialization and execution
Inspect variables and plugin state

Multiple terminal windows:

Terminal 1: Run ./run-dev.sh
Terminal 2: Quick builds with ./mvnw compile -pl :plugin-name from project root
Restart server with Ctrl+C and up-arrow to re-run

Keyboard shortcut for restart:

Set up: Preferences → Keymap → Search for “Rerun”
Assign a hotkey to “Rerun ‘Tika gRPC Server (Dev Mode)’”
Quick restart: One keystroke instead of mouse clicking

Common Issues and Solutions

“ClassNotFoundException” after changes:

Make sure you ran Build Module on the changed plugin
Check that target/classes was updated (look at file timestamps)
Do a clean compile: ./mvnw clean compile

Changes not visible after restart:

Verify you built the correct module (check module name in IntelliJ)
Ensure the config file points to the right target/classes directory
Check that development mode is enabled (look for “DEVELOPMENT mode” in logs)

Server won't start:

Build ALL plugins first: ./mvnw compile -pl tika-pipes/tika-pipes-plugins
Check that dev-tika-config.json exists in the working directory
Verify working directory is set to $PROJECT_DIR$/tika-grpc

Want to see what changed:

Before restarting, add a log statement to verify your change
Check the plugin's version in startup logs
Use debugger to set breakpoint in modified code

Alternative: Maven Exec from IntelliJ

Instead of creating an Application run configuration, you can also run the Maven goal directly:

Open Maven tool window
Navigate to: tika-grpc → Profiles → dev (check the checkbox)
Navigate to: tika-grpc → Plugins → exec → exec:java
Right-click exec:java → Run
To use custom config: Edit the run configuration and add:
```
-Dconfig.file=my-config.json
```

Same workflow applies: make changes → build module → restart Maven goal

Why You Must Restart

PF4J plugin loading happens at server startup:

TikaPluginManager.loadPlugins() discovers and loads plugins from directories
Creates PluginClassLoader instances for each plugin
Instantiates plugin classes and calls start() methods
This happens once when the server starts

No hot reload support (yet):

PF4J doesn't automatically detect changed class files
ClassLoaders are not refreshed during runtime
Server must be restarted to reload plugin code

Fast iteration cycle:

Compile plugin: ~5-10 seconds
Restart server: ~10-15 seconds
Total: ~20-25 seconds per change
Much faster than packaging ZIPs (2-3 minutes) or full rebuild (5+ minutes)

Switching Back to Production Mode

For production deployments, use packaged ZIP files:

Remove or set development mode to false:

unset TIKA_PLUGIN_DEV_MODE
# OR
export TIKA_PLUGIN_DEV_MODE=false

Build plugin ZIPs:

./mvnw clean package -pl tika-pipes/tika-pipes-plugins

Update plugin-roots to point to the directory containing ZIP files:

{
  "plugin-roots": [
    "/opt/tika/plugins"
  ]
}

Place plugin ZIPs in the configured directory:

cp tika-pipes-plugins/*/target/*.zip /opt/tika/plugins/

Troubleshooting

Plugin not loading?

Ensure ./mvnw compile was run on the plugin module
Check that plugin.properties exists in target/classes/
Verify development mode is enabled
Look for “DEVELOPMENT mode” in the logs on startup

Changes not picked up?

Recompile the plugin module: ./mvnw compile -pl :plugin-name
Restart the application
Check that you're editing the correct plugin module

ClassNotFoundException errors?

Make sure you built all plugins first with ./mvnw clean install -DskipTests
The dev profile includes all plugin dependencies, but they must be compiled first

References

pf4j Development Mode Documentation
JIRA TIKA-4587 - Development mode implementation

Docker and Kubernetes Deployment

Docker Image

The official Tika gRPC Docker images are published to Docker Hub at apache/tika-grpc.

Pull the latest image:

docker pull apache/tika-grpc:latest

Run with default configuration:

docker run -p 50052:50052 apache/tika-grpc:latest

Run with custom configuration:

docker run -p 50052:50052 \
  -v $(pwd)/my-config.json:/config/tika-config.json \
  apache/tika-grpc:latest --config /config/tika-config.json

Kubernetes Deployment

Single Instance Deployment

For simple deployments without distributed configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tika-grpc
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tika-grpc
  template:
    metadata:
      labels:
        app: tika-grpc
    spec:
      containers:
      - name: tika-grpc
        image: apache/tika-grpc:latest
        ports:
        - containerPort: 50052
          name: grpc
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
        volumeMounts:
        - name: config
          mountPath: /config
      volumes:
      - name: config
        configMap:
          name: tika-grpc-config
---
apiVersion: v1
kind: Service
metadata:
  name: tika-grpc
spec:
  selector:
    app: tika-grpc
  ports:
  - port: 50052
    targetPort: 50052
    name: grpc
  type: ClusterIP

Deployment with Apache Ignite ConfigStore

Not available in the published apache/tika-grpc image. tika-grpc no longer ships the Ignite jars — they pulled in ~196 transitive artifacts for a store that, in practice, only ever talked to an embedded node in the same JVM (IgniteConfigStore.init() hardcodes 127.0.0.1:10800). Setting configStoreType: ignite without those jars fails at startup with an explanatory error. To use it, build an image that adds tika-pipes-config-store-ignite and its Ignite dependencies to the classpath (the dev Maven profile wires them up for local runs). memory and file are the config stores available out of the box.

For distributed deployments with shared configuration using Apache Ignite:

1. Create ConfigMap with Tika configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tika-grpc-config
data:
  tika-config.json: |
    {
      "pipes": {
        "configStoreType": "ignite",
        "configStoreParams": "{\"cacheName\":\"tika-config-cache\",\"cacheMode\":\"REPLICATED\",\"igniteInstanceName\":\"TikaCluster\"}"
      },
      "fetchers": [
        {
          "s3": {
            "myS3Fetcher": {
              "region": "us-east-1",
              "bucket": "my-bucket"
            }
          }
        }
      ],
      "emitters": [
        {
          "s3": {
            "myS3Emitter": {
              "region": "us-east-1",
              "bucket": "my-output-bucket"
            }
          }
        }
      ]
    }

2. Create StatefulSet for Ignite cluster discovery:

apiVersion: v1
kind: Service
metadata:
  name: tika-grpc-ignite
  labels:
    app: tika-grpc
spec:
  clusterIP: None  # Headless service for Ignite discovery
  selector:
    app: tika-grpc
  ports:
  - port: 47100
    name: ignite-comm
  - port: 47500
    name: ignite-disco
  - port: 50052
    name: grpc
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: tika-grpc
spec:
  serviceName: tika-grpc-ignite
  replicas: 3
  selector:
    matchLabels:
      app: tika-grpc
  template:
    metadata:
      labels:
        app: tika-grpc
    spec:
      containers:
      - name: tika-grpc
        image: apache/tika-grpc:latest
        ports:
        - containerPort: 50052
          name: grpc
        - containerPort: 47100
          name: ignite-comm
        - containerPort: 47500
          name: ignite-disco
        env:
        - name: JAVA_OPTS
          value: >-
            -Xmx2g
            -Xms2g
            --add-opens=java.base/java.nio=ALL-UNNAMED
            --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
            --add-opens=java.base/java.lang=ALL-UNNAMED
            --add-opens=java.base/java.util=ALL-UNNAMED
            --add-opens=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED
        - name: IGNITE_KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: IGNITE_KUBERNETES_SERVICE_NAME
          value: "tika-grpc-ignite"
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
        volumeMounts:
        - name: config
          mountPath: /config
        readinessProbe:
          exec:
            command:
            - grpc_health_probe
            - -addr=:50052
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          exec:
            command:
            - grpc_health_probe
            - -addr=:50052
          initialDelaySeconds: 30
          periodSeconds: 10
      volumes:
      - name: config
        configMap:
          name: tika-grpc-config
---
apiVersion: v1
kind: Service
metadata:
  name: tika-grpc
spec:
  selector:
    app: tika-grpc
  ports:
  - port: 50052
    targetPort: 50052
    name: grpc
  type: LoadBalancer  # Or ClusterIP for internal-only access

3. RBAC permissions for Kubernetes API access (Ignite discovery):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tika-grpc
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: tika-grpc
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tika-grpc
subjects:
- kind: ServiceAccount
  name: tika-grpc
roleRef:
  kind: Role
  name: tika-grpc
  apiGroup: rbac.authorization.k8s.io

Then update the StatefulSet to use the service account:

spec:
  template:
    spec:
      serviceAccountName: tika-grpc
      containers:
      # ... rest of container spec

Ignite Cluster Configuration

When using Apache Ignite ConfigStore in Kubernetes, the Ignite nodes automatically discover each other using Kubernetes API. Here's how it works:

Headless Service (clusterIP: None) allows each pod to have its own DNS entry
StatefulSet ensures predictable pod names: tika-grpc-0, tika-grpc-1, tika-grpc-2
Environment Variables tell Ignite how to discover other nodes:
- IGNITE_KUBERNETES_NAMESPACE: Current namespace
- IGNITE_KUBERNETES_SERVICE_NAME: Service name for discovery
RBAC grants permission to query Kubernetes API for pod discovery

Key Ports:

50052: gRPC API for Tika operations
47100: Ignite communication port
47500: Ignite discovery port

Benefits of Ignite ConfigStore in Kubernetes:

Shared Configuration: All pods share fetcher/emitter configurations
Configuration Updates: Update config on one pod, all pods see the change
High Availability: Config survives pod restarts
Automatic Discovery: Pods automatically find each other

Horizontal Pod Autoscaling

Scale based on CPU/memory or custom metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: tika-grpc
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: tika-grpc
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Monitoring and Observability

Prometheus metrics endpoint:

Add to the StatefulSet:

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8081"
  prometheus.io/path: "/metrics"

Logging:

Configure structured JSON logging for better observability:

env:
- name: LOG_LEVEL
  value: "INFO"
- name: LOG_FORMAT
  value: "json"

Best Practices

Resource Limits: Always set memory/CPU requests and limits
Readiness/Liveness Probes: Use gRPC health checks
Pod Disruption Budget: Ensure high availability during updates
Network Policies: Restrict access to gRPC and Ignite ports
Persistent Storage: For file-based fetchers/emitters, use PersistentVolumeClaims
Configuration Management: Use ConfigMaps/Secrets for sensitive data
Ignite Memory: Allocate sufficient heap for Ignite cache (typically 2-4GB)

Troubleshooting

Pods can't discover each other:

Check RBAC permissions
Verify headless service exists
Check environment variables are set
Review Ignite logs: kubectl logs tika-grpc-0 | grep Ignite

Out of Memory errors:

Increase JVM heap size via JAVA_OPTS
Increase pod memory limits
Adjust Ignite data region sizes

Slow gRPC responses:

Scale horizontally with HPA
Check resource utilization
Review parser performance
Consider adding caching layer