refactor: reorganize airavata-api into domain modules (Phase 1+2) Squash merge of Phase 1 (common, messaging, security, sharing) and Phase 2 (credential, storage, compute) domain module reorganization. See #597 and #598 for details.
Apache Airavata is a software framework for executing and managing computational jobs on distributed computing resources including local clusters, supercomputers, national grids, academic and commercial clouds. Airavata builds on general concepts of service oriented computing, distributed messaging, and workflow composition and orchestration. Airavata bundles a server package with an API, client software development Kits and a general purpose reference UI implementation.
If you’re a researcher, Airavata offers several ways to streamline your workflows:
Apache Airavata is composed of modular components spanning core services, data management, user interfaces, and developer tooling.
airavata – Main resource management and task orchestration middlewareairavata-custos – Identity and access management frameworkairavata-mft – Managed file transfer servicesairavata-portals – All frontends for airavataairavata-data-lake – Data lake and storage backendairavata-data-catalog – Metadata and search servicesairavata-docs – Developer documentationairavata-user-docs – End-user guidesairavata-admin-user-docs – Admin-focused documentationairavata-custos-docs – Custos documentationairavata-site – Project websiteairavata-sandbox – Prototypes and early-stage workairavata-labs – Experimental projectsairavata-jupyter-kernel – Jupyter integrationairavata-cerebrum – Airavata for NeuroscienceAiravata is composed of a consolidated JVM server plus three Spring Boot microservices that together facilitate the full lifecycle of computational jobs.
graph TB subgraph "Docker Infrastructure (compose.yml)" DB[(MariaDB<br/>:13306)] RMQ[RabbitMQ<br/>:5672] ZK[ZooKeeper<br/>:2181] KFK[Kafka<br/>:9092] KC[Keycloak<br/>:18080] end subgraph "AiravataServer (single JVM)" direction TB MUX["TMultiplexedProcessor :8930<br/>9 Thrift services<br/>(airavata-thrift-server module)"] SVC["Service Layer<br/>(airavata-api module)"] BG["Background Services<br/>8 IServer workers<br/>(ServiceRegistry)"] MON["MonitoringServer :9097<br/>/metrics /health/services<br/>/admin/restart"] end subgraph "Spring Boot Services" RS[Research Service<br/>:18889 / gRPC :19908] AS[Agent Service<br/>:18880 / gRPC :19900] FS[File Server<br/>:8050] RP[REST Proxy<br/>:8082] end SDK[Python SDK] -->|TMultiplexedProtocol| MUX Portal[Web Portal] -->|REST| RP MUX --> SVC SVC --> DB SVC --> RMQ BG --> ZK BG --> KFK BG --> RMQ AS --> DB RS --> DB SVC -.->|auth| KC MON -.->|tracks| BG
(airavata-thrift-server + airavata-api)Entry point:
org.apache.airavata.api.server.AiravataServer(inairavata-thrift-servermodule) Started via:tilt upor manually withjava -cp ...
AiravataServer is the single JVM that hosts all Thrift services and background workers. On startup it:
airavata database via FlywayTMultiplexedProcessor (port 8930)IServer workers after the Thrift server is readyAll business logic lives in a transport-agnostic service layer (org.apache.airavata.service.*). Thrift handlers are thin adapters that delegate to services via ThriftAdapter, translating between Thrift types and service exceptions. This decoupling enables future REST/gRPC transports without duplicating logic.
| Service | Responsibility |
|---|---|
ExperimentService | Experiment lifecycle (create, launch, clone, terminate, intermediate outputs) |
ProjectService | Project CRUD and search |
GatewayService | Gateway management and sharing domain setup |
ApplicationCatalogService | App modules, deployments, and interfaces |
ResourceService | Compute/storage resources, job submission, data movement, storage info |
CredentialService | SSH/password credential management with sharing |
ResourceSharingService | Resource sharing and access control |
GroupResourceProfileService | Group resource profiles and policies |
GatewayResourceProfileService | Gateway resource preferences |
UserResourceProfileService | User resource preferences |
SSHAccountService | SSH account provisioning and validation |
DataProductService | Data products and replicas |
NotificationService | Notification CRUD |
ParserService | Parsers and parsing templates |
Shared utilities: SharingHelper (sharing operations), EventPublisher (messaging), RequestContext (transport-agnostic identity).
| Service name | Handler | Responsibility |
|---|---|---|
Airavata | AiravataServerHandler | Public API — delegates to service layer |
RegistryService | RegistryServerHandler | Metadata and definitions for tasks and applications |
CredentialStore | CredentialStoreServerHandler | Secure storage and retrieval of compute credentials |
SharingRegistry | SharingRegistryServerHandler | Sharing and permissioning of Airavata resources |
UserProfile | UserProfileServiceHandler | User profile management |
TenantProfile | TenantProfileServiceHandler | Tenant/gateway management |
IamAdminServices | IamAdminServicesHandler | IAM administration |
GroupManager | GroupManagerServiceHandler | Group and role management |
Orchestrator | OrchestratorServerHandler | Workflow DAG construction and task dispatch |
| Service | Responsibility |
|---|---|
DBEventManagerRunner | Syncs task execution events to the DB via pub/sub |
MonitoringServer | Prometheus metrics endpoint (optional) |
ComputationalResourceMonitoringService | Polls compute resource queue status |
DataInterpreterService | Metadata analysis for submitted jobs |
ProcessReschedulingService | Retries and reschedules failed processes |
HelixController | Manages Helix cluster state transitions |
GlobalParticipant | Executes task steps (EnvSetupTask, InputDataStagingTask, OutputDataStagingTask, JobVerificationTask, CompletingTask, ForkJobSubmissionTask, DefaultJobSubmissionTask, LocalJobSubmissionTask, ArchiveTask, WorkflowCancellationTask, RemoteJobCancellationTask, CancelCompletingTask, DataParsingTask, ParsingTriggeringTask) |
PreWorkflowManager | Handles pre-execution phase (env setup, data staging) |
PostWorkflowManager | Handles post-execution phase (cleanup, output fetching) |
ParserWorkflowManager | Handles parsing/data-interpretation workflow phase |
EmailBasedMonitor | Polls email inbox for job status updates (optional) |
RealtimeMonitor | Listens on Kafka for real-time job state changes |
stateDiagram-v2 [*] --> CREATED CREATED --> VALIDATED : validateExperiment VALIDATED --> LAUNCHED : launchExperiment LAUNCHED --> EXECUTING : PreWorkflowManager EXECUTING --> MONITORING : job submitted MONITORING --> COMPLETED : job finished MONITORING --> FAILED : job error EXECUTING --> CANCELLED : terminateExperiment COMPLETED --> [*] FAILED --> [*] CANCELLED --> [*]
graph LR subgraph "Startup Sequence" direction TB A[1. DB Init] --> B[2. Thrift Handlers] B --> C[3. TMultiplexedProcessor :8930] C --> D[4. DBEventManager] D --> E[5. MonitoringServer :9097] E --> F[6. HelixController] F --> G{waitForHelixCluster} G --> H[7. HelixParticipant] H --> I[8. Pre/Post/Parser WF Managers] I --> J[9. Email/Realtime Monitors] end
(apache-airavata-file-server)Class Name:
org.apache.airavata.file.server.FileServerApplicationCommand: `bin/
The Airavata File Server is a lightweight SFTP wrapper running on storage nodes integrated with Airavata. It lets users securely access storage via SFTP, using Airavata authentication tokens as ephemeral passwords.
(apache-airavata-agent-service) [NEW]Class Name:
org.apache.airavata.agent.connection.service.AgentServiceApplication
The Airavata Agent Service is the backend for launching interactive jobs using Airavata. It provide constructs to launch a custom “Agent” on a compute resource, that connects back to the Agent Service through a bi-directional gRPC channel. The Airavata Python SDK primarily utilizes the Agent Service (gRPC) and the Airavata API (Thrift) to submit and execute interactive jobs, spawn subprocesses, and create network tunnels to subprocesses, even if they are behind NAT.
(apache-airavata-research-service) [NEW]Class Name:
org.apache.airavata.research.service.ResearchServiceApplication
The Airavata Research Service is the backend for the research catalog in Airavata. It provides the API to add, list, modify, and publish notebooks, repositories, datasets, and computational models in cybershuttle, and launch interactive remote sessions to utilize them in a research setting.
| Requirement | Version | Check Using |
|---|---|---|
| Java SDK | 17+ | java --version |
| Apache Maven | 3.8+ | mvn -v |
| Docker Engine | 20.10+ | docker -v |
| Docker Compose | 2.0+ | docker compose version |
| Tilt | 0.33+ | tilt version |
Tilt orchestrates the full development stack — infrastructure, build, and all services:
git clone https://github.com/apache/airavata.git cd airavata tilt up
This starts:
compose.yml)Open the Tilt UI at http://localhost:10350 to monitor all resources. Integration tests can be triggered manually from the Tilt UI.
docker compose up -d # Infrastructure mvn clean package -DskipTests -T4 # Build java -cp "airavata-thrift-server/target/classes:airavata-api/target/classes:airavata-api/target/dependency/*" \ -Dairavata.config.dir=airavata-api/src/main/resources \ org.apache.airavata.api.server.AiravataServer # Thrift server mvn spring-boot:run -pl modules/research-framework/research-service # Research Service mvn spring-boot:run -pl modules/agent-framework/agent-service # Agent Service mvn spring-boot:run -pl modules/file-server # File Server mvn spring-boot:run -pl modules/restproxy # REST Proxy
The monitoring server on port 9097 exposes:
| Endpoint | Method | Description |
|---|---|---|
/metrics | GET | Prometheus metrics |
/health/services | GET | JSON status of all background services (UP/DOWN/DISABLED) |
/admin/restart/{name} | POST | Restart a background service by name |
Spring Boot services expose /actuator/health on their respective ports.
The main configuration file is airavata-api/src/main/resources/airavata-server.properties. Key settings:
apiserver.port — Thrift port (default: 8930)airavata.jdbc.url — JDBC URL for the unified airavata databaseemail.based.monitoring.enabled — Enable/disable email-based job monitorapi.server.monitoring.enabled — Enable/disable Prometheus metrics endpointmvn test -pl integration-tests -Dgroups=integration
Tests verify: all 9 thrift services, database tables, RabbitMQ exchanges, Kafka broker, Keycloak realm, Prometheus metrics, and the /health/services endpoint.
| Service | Port | gRPC | Purpose |
|---|---|---|---|
| Research Service | 18889 | 19908 | Research catalog API |
| Agent Service | 18880 | 19900 | Backend for interactive jobs |
| File Server | 8050 | — | SFTP wrapper for storage nodes |
| REST Proxy | 8082 | — | REST-to-Kafka bridge |
We welcome contributions from the community! Here's how you can help:
Learn More:
Use tilt up to start the full stack, then attach your IDE debugger to the running services. For the Airavata Thrift Server, use remote JVM debug (add -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 to the serve_cmd in the Tiltfile). For Spring Boot services, use mvn spring-boot:run -Dspring-boot.run.jvmArguments="-agentlib:jdwp=..." directly.
Get Help:
Licensed to the Apache Software Foundation (ASF) under one or more contributor
license agreements. See the NOTICE file distributed with this work for
additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
See the LICENSE file for complete license details.