site/about.html - comdev-metrics - Git at Google

 <!DOCTYPE html>
 <html lang="en">
 <head>
     <meta charset="UTF-8">
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
     <title>About the Data — ASF Community Metrics</title>
     <style>
         * { box-sizing: border-box; margin: 0; padding: 0; }
         body {
             font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
             line-height: 1.8;
             color: #333;
             max-width: 800px;
             margin: 0 auto;
             padding: 2rem;
             background: #f8f9fa;
         }
         h1 { margin-bottom: 0.25rem; color: #1a1a2e; }
         h2 { margin-top: 2rem; margin-bottom: 0.5rem; color: #1a1a2e; }
         h3 { margin-top: 1.5rem; margin-bottom: 0.3rem; color: #444; }
         a { color: #d45500; text-decoration: none; }
         a:hover { text-decoration: underline; }
         .breadcrumb { color: #666; margin-bottom: 1.5rem; font-size: 0.9rem; }
         .section {
             background: white; border-radius: 8px; padding: 1.5rem;
             margin-bottom: 1.5rem; box-shadow: 0 1px 3px rgba(0,0,0,0.1);
         }
         table { width: 100%; border-collapse: collapse; margin: 1rem 0; }
         th, td { text-align: left; padding: 0.5rem 0.8rem; border-bottom: 1px solid #eee; }
         th { font-weight: 600; color: #666; font-size: 0.85rem; }
         code { background: #f0f0f0; padding: 0.1rem 0.4rem; border-radius: 3px; font-size: 0.9em; }
         ul { margin-left: 1.5rem; margin-bottom: 1rem; }
         li { margin-bottom: 0.3rem; }
     </style>
 </head>
 <body>
     <div class="breadcrumb"><a href="index.html">← Dashboard</a></div>
     <h1>About the Data</h1>

     <div class="section">
         <h2>Data Sources</h2>
         <p>This dashboard collects data from multiple public sources. No authentication
         or special access is required — everything shown here is derived from
         publicly available information.</p>

         <table>
             <thead><tr><th>What</th><th>Source</th><th>How</th></tr></thead>
             <tbody>
                 <tr>
                     <td><strong>Mailing list activity</strong></td>
                     <td><a href="https://lists.apache.org/">Pony Mail</a></td>
                     <td>POST to <code>/api/stats.json</code> — message counts per month, per list</td>
                 </tr>
                 <tr>
                     <td><strong>Git commits &amp; PRs</strong></td>
                     <td><a href="https://github.com/apache">GitHub API</a></td>
                     <td>Commits and pull request activity per repo, aggregated monthly</td>
                 </tr>
                 <tr>
                     <td><strong>SVN commits</strong></td>
                     <td><a href="https://svn.apache.org/">ASF Subversion</a></td>
                     <td><code>svn log --xml</code> against remote repos (no checkout needed)</td>
                 </tr>
                 <tr>
                     <td><strong>Project roster &amp; PMC data</strong></td>
                     <td><a href="https://projects.apache.org/">projects.apache.org</a></td>
                     <td>Daily-generated JSON from LDAP, Whimsy, and committee-info.txt</td>
                 </tr>
                 <tr>
                     <td><strong>Releases</strong></td>
                     <td><a href="https://projects.apache.org/">projects.apache.org</a></td>
                     <td>Release history JSON (scraped from dist archives)</td>
                 </tr>
                 <tr>
                     <td><strong>Repository inventory</strong></td>
                     <td><a href="https://github.com/apache">GitHub API</a></td>
                     <td>Full org listing — classifies 3000+ repos into projects</td>
                 </tr>
                 <tr>
                     <td><strong>New committer dates</strong></td>
                     <td><a href="https://whimsy.apache.org/public/public_ldap_people.json">Whimsy LDAP</a></td>
                     <td>Account creation timestamps from <code>public_ldap_people.json</code>, cross-referenced with project group membership</td>
                 </tr>
             </tbody>
         </table>
     </div>

     <div class="section">
         <h2>How It Works</h2>
         <ul>
             <li>A collector runs weekly, fetching data from the sources above.</li>
             <li>Data is cached aggressively — past months are immutable and never re-fetched.</li>
             <li>Only the current month's data is refreshed on each run.</li>
             <li>After collection, an activity classification pass categorizes projects by trend
             (Sharp Decline / Declining / Dormant) from the collected data using fixed thresholds.</li>
             <li>The output is static HTML + JSON — no live server or database.</li>
             <li>Trend lines use linear regression over a 12-month rolling window.</li>
             <li>The current (incomplete) month is extrapolated to a full-month estimate for trend accuracy.</li>
             <li>Each project has a dedicated page showing all its metrics in one place —
             mailing lists (linked to <a href="https://lists.apache.org/">lists.apache.org</a> archives),
             git repos, releases, community growth, and activity trends.</li>
         </ul>
     </div>

     <div class="section">
         <h2>What's Measured</h2>

         <h3>Mailing Lists</h3>
         <p>Monthly message volume per list. All lists with traffic in the past 12 months
         are shown. Covers dev@, user@, issues@, commits@, and other project-specific lists.</p>

         <h3>Git/VCS Activity</h3>
         <p>Per-repository commit counts and unique committers (monthly). For GitHub repos,
         also tracks pull requests opened, merged, and closed. SVN projects use remote
         <code>svn log</code> — no checkout required.</p>

         <h3>Community Growth</h3>
         <p>Tracks both new committers and new PMC members:</p>
         <ul>
             <li><strong>New committers</strong> — detected via LDAP <code>createTimestamp</code> from
             <a href="https://whimsy.apache.org/public/public_ldap_people.json">Whimsy</a>,
             cross-referenced with <code>people.json</code> group membership to determine project affiliation.</li>
             <li><strong>New PMC members</strong> — from roster dates in the committees data on projects.apache.org.</li>
         </ul>
         <p>The Community Growth tab on the overview page shows both in a two-color chart
         (blue = committers, orange = PMC members). Per-project pages list individual names and dates.</p>

         <h3>Releases</h3>
         <p>Release publications in the past 12 months, as recorded by projects.apache.org.</p>

         <h3>Activity Trends</h3>
         <p>A deterministic classification of quarter-over-quarter trends. Compares the two
         most recent full quarters (excluding the current partial month) for both mailing list
         discussion activity and git commit volume.</p>
         <ul>
             <li><strong>🔴 Sharp Decline</strong> — both mailing list and git activity down ≥ 30% quarter-over-quarter.</li>
             <li><strong>🟡 Declining</strong> — one metric down ≥ 30%, with meaningful prior activity (≥ 30 messages or commits in the prior quarter). If the other metric is growing ≥ 30%, the project is not flagged.</li>
             <li><strong>⚫ Dormant</strong> — fewer than 5 commits and 5 mailing list messages in the recent quarter.</li>
         </ul>
         <p>Only human discussion lists (dev, user, general, discuss) are counted for mailing list
         trends — automated lists (notifications, commits, issues) are excluded. The thresholds are
         fixed constants, not heuristic or ML-based.</p>
     </div>

     <div class="section">
         <h2>Limitations</h2>
         <ul>
             <li>Committer detection relies on LDAP <code>createTimestamp</code> — this reflects
             when the ASF account was created, which is typically the committer grant date but
             may differ slightly for people who had accounts before being invited to a project.</li>
             <li>Release counts may include multiple artifacts per release for some projects.</li>
             <li>Bot activity is not yet filtered from commit/PR counts.</li>
             <li>Projects that are SVN-only but not yet detected may show zero git activity.</li>
             <li>GitHub API rate limits (5000/hour) mean large initial runs may take 1–2 hours.</li>
             <li>Some projects use Slack, Discord, or GitHub Discussions as primary discussion
             channels — mailing list declines may not reflect actual community decline for those projects.</li>
             <li>Quarter-over-quarter comparison can be noisy for projects with bursty release
             cycles or seasonal patterns. A single quarter's decline does not necessarily indicate
             a long-term trend.</li>
         </ul>
     </div>

     <div class="section">
         <h2>Source Code</h2>
         <p>The collector and dashboard are open source:</p>
         <ul>
             <li><a href="https://github.com/rbowen/comdev-metrics">github.com/rbowen/comdev-metrics</a></li>
         </ul>
         <p>Feedback and contributions welcome. See <code>README.md</code> for setup instructions.</p>
     </div>

 </body>
 </html>
	<!DOCTYPE html>
	<html lang="en">
	<head>
	<meta charset="UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<title>About the Data — ASF Community Metrics</title>
	<style>
	* { box-sizing: border-box; margin: 0; padding: 0; }
	body {
	font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
	line-height: 1.8;
	color: #333;
	max-width: 800px;
	margin: 0 auto;
	padding: 2rem;
	background: #f8f9fa;
	}
	h1 { margin-bottom: 0.25rem; color: #1a1a2e; }
	h2 { margin-top: 2rem; margin-bottom: 0.5rem; color: #1a1a2e; }
	h3 { margin-top: 1.5rem; margin-bottom: 0.3rem; color: #444; }
	a { color: #d45500; text-decoration: none; }
	a:hover { text-decoration: underline; }
	.breadcrumb { color: #666; margin-bottom: 1.5rem; font-size: 0.9rem; }
	.section {
	background: white; border-radius: 8px; padding: 1.5rem;
	margin-bottom: 1.5rem; box-shadow: 0 1px 3px rgba(0,0,0,0.1);
	}
	table { width: 100%; border-collapse: collapse; margin: 1rem 0; }
	th, td { text-align: left; padding: 0.5rem 0.8rem; border-bottom: 1px solid #eee; }
	th { font-weight: 600; color: #666; font-size: 0.85rem; }
	code { background: #f0f0f0; padding: 0.1rem 0.4rem; border-radius: 3px; font-size: 0.9em; }
	ul { margin-left: 1.5rem; margin-bottom: 1rem; }
	li { margin-bottom: 0.3rem; }
	</style>
	</head>
	<body>
	<div class="breadcrumb"><a href="index.html">← Dashboard</a></div>
	<h1>About the Data</h1>

	<div class="section">
	<h2>Data Sources</h2>
	<p>This dashboard collects data from multiple public sources. No authentication
	or special access is required — everything shown here is derived from
	publicly available information.</p>

	<table>
	<thead><tr><th>What</th><th>Source</th><th>How</th></tr></thead>
	<tbody>
	<tr>
	<td><strong>Mailing list activity</strong></td>
	<td><a href="https://lists.apache.org/">Pony Mail</a></td>
	<td>POST to <code>/api/stats.json</code> — message counts per month, per list</td>
	</tr>
	<tr>
	<td><strong>Git commits & PRs</strong></td>
	<td><a href="https://github.com/apache">GitHub API</a></td>
	<td>Commits and pull request activity per repo, aggregated monthly</td>
	</tr>
	<tr>
	<td><strong>SVN commits</strong></td>
	<td><a href="https://svn.apache.org/">ASF Subversion</a></td>
	<td><code>svn log --xml</code> against remote repos (no checkout needed)</td>
	</tr>
	<tr>
	<td><strong>Project roster & PMC data</strong></td>
	<td><a href="https://projects.apache.org/">projects.apache.org</a></td>
	<td>Daily-generated JSON from LDAP, Whimsy, and committee-info.txt</td>
	</tr>
	<tr>
	<td><strong>Releases</strong></td>
	<td><a href="https://projects.apache.org/">projects.apache.org</a></td>
	<td>Release history JSON (scraped from dist archives)</td>
	</tr>
	<tr>
	<td><strong>Repository inventory</strong></td>
	<td><a href="https://github.com/apache">GitHub API</a></td>
	<td>Full org listing — classifies 3000+ repos into projects</td>
	</tr>
	<tr>
	<td><strong>New committer dates</strong></td>
	<td><a href="https://whimsy.apache.org/public/public_ldap_people.json">Whimsy LDAP</a></td>
	<td>Account creation timestamps from <code>public_ldap_people.json</code>, cross-referenced with project group membership</td>
	</tr>
	</tbody>
	</table>
	</div>

	<div class="section">
	<h2>How It Works</h2>
	<ul>
	<li>A collector runs weekly, fetching data from the sources above.</li>
	<li>Data is cached aggressively — past months are immutable and never re-fetched.</li>
	<li>Only the current month's data is refreshed on each run.</li>
	<li>After collection, an activity classification pass categorizes projects by trend
	(Sharp Decline / Declining / Dormant) from the collected data using fixed thresholds.</li>
	<li>The output is static HTML + JSON — no live server or database.</li>
	<li>Trend lines use linear regression over a 12-month rolling window.</li>
	<li>The current (incomplete) month is extrapolated to a full-month estimate for trend accuracy.</li>
	<li>Each project has a dedicated page showing all its metrics in one place —
	mailing lists (linked to <a href="https://lists.apache.org/">lists.apache.org</a> archives),
	git repos, releases, community growth, and activity trends.</li>
	</ul>
	</div>

	<div class="section">
	<h2>What's Measured</h2>

	<h3>Mailing Lists</h3>
	<p>Monthly message volume per list. All lists with traffic in the past 12 months
	are shown. Covers dev@, user@, issues@, commits@, and other project-specific lists.</p>

	<h3>Git/VCS Activity</h3>
	<p>Per-repository commit counts and unique committers (monthly). For GitHub repos,
	also tracks pull requests opened, merged, and closed. SVN projects use remote
	<code>svn log</code> — no checkout required.</p>

	<h3>Community Growth</h3>
	<p>Tracks both new committers and new PMC members:</p>
	<ul>
	<li><strong>New committers</strong> — detected via LDAP <code>createTimestamp</code> from
	<a href="https://whimsy.apache.org/public/public_ldap_people.json">Whimsy</a>,
	cross-referenced with <code>people.json</code> group membership to determine project affiliation.</li>
	<li><strong>New PMC members</strong> — from roster dates in the committees data on projects.apache.org.</li>
	</ul>
	<p>The Community Growth tab on the overview page shows both in a two-color chart
	(blue = committers, orange = PMC members). Per-project pages list individual names and dates.</p>

	<h3>Releases</h3>
	<p>Release publications in the past 12 months, as recorded by projects.apache.org.</p>

	<h3>Activity Trends</h3>
	<p>A deterministic classification of quarter-over-quarter trends. Compares the two
	most recent full quarters (excluding the current partial month) for both mailing list
	discussion activity and git commit volume.</p>
	<ul>
	<li><strong>🔴 Sharp Decline</strong> — both mailing list and git activity down ≥ 30% quarter-over-quarter.</li>
	<li><strong>🟡 Declining</strong> — one metric down ≥ 30%, with meaningful prior activity (≥ 30 messages or commits in the prior quarter). If the other metric is growing ≥ 30%, the project is not flagged.</li>
	<li><strong>⚫ Dormant</strong> — fewer than 5 commits and 5 mailing list messages in the recent quarter.</li>
	</ul>
	<p>Only human discussion lists (dev, user, general, discuss) are counted for mailing list
	trends — automated lists (notifications, commits, issues) are excluded. The thresholds are
	fixed constants, not heuristic or ML-based.</p>
	</div>

	<div class="section">
	<h2>Limitations</h2>
	<ul>
	<li>Committer detection relies on LDAP <code>createTimestamp</code> — this reflects
	when the ASF account was created, which is typically the committer grant date but
	may differ slightly for people who had accounts before being invited to a project.</li>
	<li>Release counts may include multiple artifacts per release for some projects.</li>
	<li>Bot activity is not yet filtered from commit/PR counts.</li>
	<li>Projects that are SVN-only but not yet detected may show zero git activity.</li>
	<li>GitHub API rate limits (5000/hour) mean large initial runs may take 1–2 hours.</li>
	<li>Some projects use Slack, Discord, or GitHub Discussions as primary discussion
	channels — mailing list declines may not reflect actual community decline for those projects.</li>
	<li>Quarter-over-quarter comparison can be noisy for projects with bursty release
	cycles or seasonal patterns. A single quarter's decline does not necessarily indicate
	a long-term trend.</li>
	</ul>
	</div>

	<div class="section">
	<h2>Source Code</h2>
	<p>The collector and dashboard are open source:</p>
	<ul>
	<li><a href="https://github.com/rbowen/comdev-metrics">github.com/rbowen/comdev-metrics</a></li>
	</ul>
	<p>Feedback and contributions welcome. See <code>README.md</code> for setup instructions.</p>
	</div>

	</body>
	</html>