Apache Airavata

Apache Airavata is a software framework for executing and managing computational jobs on distributed computing resources including local clusters, supercomputers, national grids, academic and commercial clouds. Airavata builds on general concepts of service oriented computing, distributed messaging, and workflow composition and orchestration. Airavata bundles a server package with an API, client software development Kits and a general purpose reference UI implementation.

Key Features

• 🔧 Service-oriented architecture with distributed messaging
• 🔄 Fully-managed task lifecycle (environment setup, data staging, execution, and output retrieval)
• ☁️ Multi-cloud and hybrid cloud support
• 🖥️ Comprehensive API and SDK ecosystem
• 🌐 Reference UI Implementations

Using Airavata

If you’re a researcher, Airavata offers several ways to streamline your workflows:

1. Submit Batch Jobs via the Airavata Application Portal -- Example
2. Launch Interactive Experiments through the Airavata Research Portal -- Example
3. Explore and run published experiments from the Airavata Research Catalog -- Example
4. Run interactive computational jobs directly from your IDE using Airavata Python SDK -- PyPI

🧱 The Airavata Ecosystem

Apache Airavata is composed of modular components spanning core services, data management, user interfaces, and developer tooling.

🔧 Core Services

• airavata – Main resource management and task orchestration middleware
• airavata-custos – Identity and access management framework
• airavata-mft – Managed file transfer services
• airavata-portals – All frontends for airavata

📦 Data & Metadata Services

• airavata-data-lake – Data lake and storage backend
• airavata-data-catalog – Metadata and search services

📚 Documentation & Branding

• airavata-docs – Developer documentation
• airavata-user-docs – End-user guides
• airavata-admin-user-docs – Admin-focused documentation
• airavata-custos-docs – Custos documentation
• airavata-site – Project website

🧪 Experimental & Research

• airavata-sandbox – Prototypes and early-stage work
• airavata-labs – Experimental projects
• airavata-jupyter-kernel – Jupyter integration
• airavata-cerebrum – Airavata for Neuroscience

🔄 How Airavata Works

Airavata runs as a single unified JVM (Spring Boot + Armeria) that serves gRPC and REST (HTTP/JSON transcoding) on a single port, plus background workers.

graph TB
    subgraph "Docker Infrastructure (compose.yml)"
        DB[(MariaDB<br/>:13306)]
        RMQ[RabbitMQ<br/>:5672]
        ZK[ZooKeeper<br/>:2181]
        KFK[Kafka<br/>:9092]
        KC[Keycloak<br/>:18080]
    end

    subgraph "Unified Airavata Server (single JVM)"
        direction TB
        ARMERIA["Armeria :9090<br/>gRPC + REST transcoding"]
        SVC["Service Layer<br/>(airavata-api module)"]
        BG["Background Services<br/>12 workers"]
    end

    SDK[Python SDK] -->|gRPC| ARMERIA
    Portal[Web Portal] -->|REST| ARMERIA
    ARMERIA --> SVC
    SVC --> DB
    SVC --> RMQ
    BG --> ZK
    BG --> KFK
    BG --> RMQ
    SVC -.->|auth| KC

1. Unified Airavata Server (airavata-server)

Entry point: org.apache.airavata.server.AiravataServerMain Started via: tilt up or java -jar airavata-server/target/airavata-server-0.21-SNAPSHOT.jar

The unified server hosts gRPC and REST (via Armeria HTTP/JSON transcoding) on a single port, Spring Boot services (Research, Agent, File Server), and background workers in a single JVM. On startup it:

1. Initializes the unified airavata database
2. Starts Armeria server on port 9090 (gRPC + REST transcoding + DocService at /docs)
3. Starts background IServer workers

Architecture: Service Layer + gRPC Transport

All business logic lives in a transport-agnostic service layer (org.apache.airavata.service.*). gRPC service implementations delegate to services. Proto definitions with google.api.http annotations provide automatic REST transcoding via Armeria.

Shared utilities: SharingHelper (sharing operations), EventPublisher (messaging), RequestContext (transport-agnostic identity).

Background Services (IServer lifecycle, same JVM)

stateDiagram-v2
    [*] --> CREATED
    CREATED --> VALIDATED : validateExperiment
    VALIDATED --> LAUNCHED : launchExperiment
    LAUNCHED --> EXECUTING : PreWorkflowManager
    EXECUTING --> MONITORING : job submitted
    MONITORING --> COMPLETED : job finished
    MONITORING --> FAILED : job error
    EXECUTING --> CANCELLED : terminateExperiment
    COMPLETED --> [*]
    FAILED --> [*]
    CANCELLED --> [*]

graph LR
    subgraph "Startup Sequence"
        direction TB
        A[1. DB Init] --> B[2. Armeria Server :9090]
        B --> C[3. DBEventManager]
        C --> D[4. HelixController]
        D --> E{waitForHelixCluster}
        E --> F[5. HelixParticipant]
        F --> G[6. Pre/Post/Parser WF Managers]
        G --> H[7. Email/Realtime Monitors]
    end

Embedded Spring Boot Services

The following services are embedded in the unified server (not run separately):

Additional service modules in airavata-api/:

🏗️ Getting Started

Prerequisites

Quick Start (Tilt)

Tilt orchestrates the full development stack — infrastructure, build, and all services:

git clone https://github.com/apache/airavata.git
cd airavata
tilt up

This starts:

• Infrastructure: MariaDB, RabbitMQ, ZooKeeper, Kafka, Keycloak (via compose.yml)
• Unified Airavata Server: gRPC + REST on port 9090, API docs at /docs

Open the Tilt UI at http://localhost:10350 to monitor all resources. Integration tests can be triggered manually from the Tilt UI.

Manual Start

docker compose up -d                          # Infrastructure
mvn clean package -DskipTests -T4             # Build
java -jar airavata-server/target/airavata-server-0.21-SNAPSHOT.jar  # Start server

Or use the helper scripts:

./dev-tools/scripts/setup.sh    # Infrastructure + build
./dev-tools/scripts/start.sh    # Start server

Health Monitoring

The Armeria server on port 9090 exposes:

Configuration

The main configuration file is airavata-server/src/main/resources/application.yml. Key settings:

• spring.datasource.url — JDBC URL (default: jdbc:mariadb://localhost:13306/airavata)
• armeria.server.port — Armeria server port (default: 9090)
• airavata.security.openid-url — Keycloak realm URL

Integration Tests

mvn test -pl airavata-api -Dgroups=runtime

Server Ports

🤝 Contributing

We welcome contributions from the community! Here's how you can help:

1. 🍴 Fork the repository
2. 🌿 Create a feature branch
3. ✨ Make your changes
4. 🧪 Add tests if applicable
5. 📝 Submit a pull request

Learn More:

• Contributing Guidelines
• Code of Conduct
• Developer Resources

Setting up your IDE

Use tilt up to start the full stack, then attach your IDE debugger. For remote debugging, add -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 to the serve_cmd in the Tiltfile.

💬 Community & Support

Get Help:

• 📧 User Mailing List: users@airavata.apache.org
• 👨‍💻 Developer Mailing List: dev@airavata.apache.org
• 🔗 All Mailing Lists: airavata.apache.org/mailing-list

📄 License

Licensed to the Apache Software Foundation (ASF) under one or more contributor
license agreements. See the NOTICE file distributed with this work for
additional information regarding copyright ownership.

The ASF licenses this file to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License at:

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

See the LICENSE file for complete license details.