Apache Airavata Custos Security

Clone this repo:
  1. f358501 Fix portal session drops (#755) by Yasith Jayawardana · 3 days ago master
  2. dd3cfc6 Sort admin users list (#753) by Yasith Jayawardana · 4 days ago v2.0.0.pre-20261117
  3. 811b970 Configurable TTL for ssh cert (#752) by Lahiru Jayathilake · 4 days ago
  4. 20b8a0b Move storage definitions into core (#748) by Yasith Jayawardana · 4 days ago
  5. 26ee483 Publish a revocation list the login nodes read (#751) by Lahiru Jayathilake · 4 days ago

Apache Airavata Custos

License GitHub closed pull requests

Custos is account-lifecycle middleware for HPC resource providers. It keeps one record per user, allocation, membership and cluster account. Connectors apply each record change to the site's accounts, scheduler, storage and email. Allocations arrive from ACCESS or from staff in the portal. Users sign in through an OpenID Connect provider.

Project website

Left: seven manual tasks across five systems for one new account. Right: with Custos, access starts from an ACCESS AMIE packet that an admin approves, or from an admin creating the account, which skips approval. Custos notifies the user, the user signs in and accepts the terms of use, and Custos creates the Unix account, adds it to the allocation's Unix group and to its scheduler account.

Who it helps

RoleWhat changes
Account staffOne approval per account replaces seven manual tasks in five systems. Removal takes one account delete and one approval per marked top-level storage folder.
ResearcherSigns in, picks a username, and sees each setup step and the login command in the portal.
PISees credits used and left, by resource and by member. With the core:allocations:write privilege, requests allocation changes in the portal.
Security staffEach approval or denial has an audit row with the reviewer, user, cluster and username.
Gateway operatorGets short-lived SSH certificates per user. Creates temporary accounts through the API.

Problems it addresses

Problem at a siteWhat Custos does
One account lives in several systems, and they drift apartCustos holds the single record, and connectors push each change out. Edit accounts in Custos, not in the account registry. Registry edits are not read back.
A downstream system is down when an account is approvedThe change and its events are saved in one transaction, so an outage loses no event. Each delivery makes up to 10 attempts, backing off from 30 seconds to 1 hour. An admin can retry a failed one.
Scheduler access drifts from what allocations grantA sweep runs every 5 minutes by default. It writes missing or changed associations and removes those of departed members. It removes associations only on scheduler accounts named after Custos allocations.
Two people get the same username or groupThe database allows one local_username and one posix_group per cluster.
SSH keys to issue, rotate and revokeUsers log in with their institutional identity through the OAuth2 device flow. Gateways get short-lived SSH certificates.
A departed member keeps accessEnding a membership removes the scheduler association and the group membership. Deleting the cluster account removes it from the account registry. The next device-flow SSH login then fails. Key logins fail once the login node's SSSD cache expires, 90 minutes by default.
Storage is deleted by mistakeRemoving an account or allocation only marks its storage directories for deletion. An admin approves each deletion, and the folders go to the storage trash.

Account lifecycle

With every connector enabled, an admin's approval is the only staff task for a new account. A removal needs two staff steps: delete the cluster account through the API (DELETE /compute-cluster-users/{id}) or the allocation in the portal, then approve each marked top-level storage folder in the portal. Folders under a top-level folder go with it.

StepOperation on the site's systemDone by
Allocation arrivesApply the AMIE packet, create each member's pending cluster account, reply to ACCESS for the project. Account replies wait for approvalAllocation connector (AMIE Processor), or staff entry in the portal
Sign-inResearcher signs in over OIDC, picks a username, accepts the user agreement, before or after approvalPortal
ApprovalAdmin approves the cluster account, alone or per allocationPortal
AccountsCreate the person, POSIX uid, gid and groups in the account registryAccounts connector (COmanage)
SchedulerCreate the scheduler account and one association per active member, with limitsScheduler connector (SLURM)
StorageCreate user and project directories, quotas and viewsStorage connector (VAST)
NoticeEmail on request, decision and account readyNotification connector (Email)
RemovalRemove associations and group membership. Mark storage directories for deletion, which an admin approvesScheduler, accounts and storage connectors

Removal, step by step:

When a member leaves: the scheduler connector removes the association, the accounts connector removes the group membership, an admin deleting the account has Custos mark its storage folders for deletion, the PAM module refuses that account's next device-flow login, and an admin approves each deletion.

Every connector is optional and enabled by a block in config/custos.yaml. To support another system, write a connector. docs/architecture.md covers the core, the event bus and the connector contract.

SSH sign-in

ExtensionWhat it does
CILogon PAM modulepam_oauth2_device authenticates SSH logins with the OAuth2 device flow against the site's OIDC provider. The node enrollment playbook keeps SSH key logins, and password logins for accounts with a local password, unless the site disables them.
SSH Certificate SignerAn SSH CA for science gateways. Issues short-lived OpenSSH user certificates bound to the user's OIDC identity.

Portal

A researcher picks a username and accepts the agreement:

Jane Doe sees custos-jdoe available, accepts the user agreement, four setup steps complete, and the portal shows the command ssh custos-jdoe@login.cluster.example.edu.

The setup steps finish once an admin approves the request. Approval can come before or after the researcher signs in. Admins approve requests grouped by allocation:

On Account Approvals, an admin approves Jane Doe's request for the Genomic Sequencing allocation on ClusterA as custos-jdoe, and the request leaves the queue.

Admins approve each marked storage folder of a deleted account:

On the admin Resources page, the Storage tab lists storage folders. Three of custos-jdoe's folders are marked for deletion. The admin deletes /home/custos-jdoe, a notice reports it deleted, and the row leaves the list.

PIs track usage:

The Analytics page for a PI, showing credits left, credits used, days remaining and daily usage by resource.

Documentation

DocCovers
INSTALL.mdRun Custos locally
CONFIG.mdServer configuration, with links to each connector's keys
docs/architecture.mdHow core and connectors work
CONTRIBUTING.mdBuild, test and conventions
docs/publications.mdPapers

Questions or Need Help?

Acknowledgment

This project is funded by the National Science Foundation (NSF).

We are grateful to Trusted CI for conducting the First Principles Vulnerability Assessment (FPVA) for this software and providing security architecture guidance and improvements.