DATA_SOURCES.md

# ASF Data Sources — Reference

Where does the authoritative data live?

## The Hierarchy

```
                    ┌─────────────────────┐
                    │   LDAP (id.apache.org)│  ← THE source of truth for people,
                    │   + committee-info.txt│     groups, PMC membership
                    └──────────┬──────────┘
                               │
              ┌────────────────┼────────────────┐
              │                │                │
    ┌─────────▼──────┐  ┌─────▼─────┐  ┌──────▼──────┐
    │  Whimsy/Roster │  │ Incubator │  │  Board      │
    │  (public JSON) │  │podlings.xml│  │  minutes    │
    └─────────┬──────┘  └─────┬─────┘  └──────┬──────┘
              │                │                │
              └────────────────┼────────────────┘
                               │
                    ┌──────────▼──────────┐
                    │ projects.apache.org  │  ← SECONDARY: aggregates from above
                    │ (DOAP + cronjobs)    │     + project-maintained DOAP files
                    └─────────────────────┘

```

## Primary/Authoritative Sources

| Data | Authoritative Source | Access | Notes |
| --- | --- | --- | --- |
| **PMC membership** | `committee-info.txt` (private SVN) | Via Whimsy JSON | Updated by Secretary |
| **Committer list** | LDAP (`id.apache.org`) | Via Whimsy `public_ldap_projects.json` |  |
| **Podling status** | `incubator/.../podlings.xml` | Via Whimsy `public_podling_status.json` | Maintained by VP Incubator |
| **Project metadata** | DOAP files (maintained by each PMC) | Listed in `data/projects.xml` on SVN |  |
| **Retired projects** | `committee-info.yaml` + Attic | Via Whimsy `committee-retired.json` |  |
| **Board actions** | Board minutes | SVN: `private/committers/board/` |  |

## Whimsy Public JSON (our best bet for programmatic access)

Base URL: `https://whimsy.apache.org/public/`

| File | Contents | Useful for |
| --- | --- | --- |
| `committee-info.json` | PMC names, members, chairs | Community events (PMC additions) |
| `committee-retired.json` | Retired committees | Project lifecycle |
| `public_ldap_projects.json` | PMC/podling owners + members | Committer additions |
| `public_podling_status.json` | Podling incubation status | Project lifecycle |
| `public_ldap_people.json` | Person names + disabled status | People data |
| `icla-info.json` | ICLA signers with committer IDs |  |

These are regenerated hourly by Whimsy cron jobs **only when underlying data changes**.

## What projects.apache.org adds

- **DOAP aggregation**: Project descriptions, categories, programming languages, homepage URLs
- **Release data**: Scraped from various sources
- **Cronjob Python scripts**: Run daily, cache to JSON in SVN

Source: `https://svn.apache.org/repos/asf/comdev/projects.apache.org/trunk/`

## Our Strategy

### Phase 1 (MVP — DONE — use what's publicly available without auth)

- **Mailing lists**: Pony Mail API (`lists.apache.org/api/`) — no auth needed- Aggressive caching: past months immutable, only current month refreshed
- **Git metrics**: GitHub API (`api.github.com/orgs/apache`) — public, rate-limited (use token from `.secrets`)- Also supports SVN via `svn log --xml` for projects configured with `vcs: svn` or auto-detected from `repositories.json`
- Per-project VCS override in `config.yml` → `project_overrides:`
- Per-repo monthly breakdown with aggressive caching (same-day = skip, next-day = incremental)
- Rate-limit tracking with automatic pause-and-wait at 50 remaining calls
- **Project roster changes**: Whimsy public JSON (diff `committee-info.json` between runs)
- **Project list**: projects.apache.org `committees.json` + `podlings.json` (TLPs + active podlings)
- **Repo inventory**: GitHub org listing → `_project_map.json` (auto-run on first use)
- **Dashboard**: 12-month rolling window with linear regression trend lines- Current month extrapolated to full-month estimate for trend accuracy

### Phase 2 (richer data, may need ASF Infra coordination)

- **Release history**: projects.apache.org JSON cache or direct DOAP parsing
- **Podling lifecycle**: Whimsy `public_podling_status.json`
- **Retired projects**: Whimsy `committee-retired.json`
- **Board report status**: Would need access to board minutes (private SVN)

### Phase 3 (if we get service account access)

- **LDAP direct queries**: Real-time committer/PMC additions (requires Infra ticket)
- **Next Committer integration**: PMC-only access via LDAP auth

## Key Insight

**projects.apache.org is convenient but not authoritative.** It aggregates from:

1. DOAP files (maintained by PMCs, varying quality/currency)
2. Whimsy/LDAP (the real source of truth for people/committee data)
3. Its own cronjob caches (may lag)

For community activity metrics, **Whimsy public JSON + Pony Mail + GitHub API** gives us everything we need for Phase 1 without any special access. We only need to involve ASF Infra if we want LDAP direct access (for Next Committer) or private SVN (for board data).

## URLs Quick Reference

```
# Whimsy public data
https://whimsy.apache.org/public/committee-info.json
https://whimsy.apache.org/public/public_ldap_projects.json
https://whimsy.apache.org/public/public_podling_status.json
https://whimsy.apache.org/public/committee-retired.json

# Pony Mail
https://lists.apache.org/api/preferences.json   (list discovery, POST)
https://lists.apache.org/api/stats.json         (per-list stats, POST + JSON body)

# GitHub
https://api.github.com/orgs/apache/repos        (repo listing)

# projects.apache.org data (secondary)
https://projects.apache.org/json/projects.json  (all projects metadata)
https://projects.apache.org/json/releases.json  (release data)

```