| Item | Value |
|---|---|
| Platform | macOS (Apple Silicon), GCC/Clang, Release -O2 |
| Build | cmake -DENABLE_MEM_STAT=ON -DBUILD_TEST=OFF |
| Encoding | PLAIN (value columns) + PLAIN (timestamp, via global config override) |
| Compression | UNCOMPRESSED |
| Table model | Aligned (shared timestamp via TimeChunkWriter) |
| Symbol | Formula | Description |
|---|---|---|
| $s_{data}$ | $8 + \sum \text{sizeof}(\text{field_type}_j)$ | Per-row data in ChunkWriter (shared timestamp) |
| $b$ | $n_{field} \times 104 + 96$ | Meta bytes per device per flush |
| $M_{init}$ | ~900 KB | Fixed writer overhead |
Goal: Validate that calculate_mem_size_for_all_group() matches $s_{data} \times F$.
| Layer | What it measures | Example (5000 rows, 1 DOUBLE field) |
|---|---|---|
| Formula ($s \times F$) | Raw data bytes | 80,000 bytes (16 bytes/row) |
| Estimate API | Encoded data in ChunkWriter | 73,767 bytes (14.8 bytes/row) |
| ModStat | Actually allocated memory | 131,512 bytes (26.3 bytes/row) |
With PLAIN+UNCOMPRESSED (1 DOUBLE field):
With SNAPPY+TS_2DIFF (8 mixed fields, default encoding):
write_var_int (variable-length), not fixed 4 bytes| batch_size | Direct (KB) | Formula (KB) | Error |
|---|---|---|---|
| 5,000 | 136 | 273 | 50.1% |
| 8,000 | 213 | 437 | 51.2% |
| 16,000 | 155 | 875 | 82.2% |
| 32,000 | 814 | 1,750 | 53.5% |
| 65,536 | 1,647 | 3,584 | 54.0% |
Conclusion: The formula gives a conservative upper bound. Direct monitoring gives the actual (post-encoding) value. This aligns with the thesis Section 3.5 design: formula for pre-planning, direct monitoring for runtime control.
Goal: Validate $M_{read} \approx M_{fixed} + \text{batch_size} \times s_{row} + N_{cols} \times C_{page}$.
| N_cols | batch_size | Peak (KB) | Formula (KB) | Error |
|---|---|---|---|---|
| 2 | 16,384 | 1,045 | 1,024 | 2.1% |
| 4 | 16,384 | 1,366 | 1,536 | 11.1% |
| 6 | 16,384 | 1,709 | 1,920 | 11.0% |
| 8 | 16,384 | 2,300 | 2,432 | 5.4% |
| 8 | 65,536 | 6,974 | 5,888 | 18.5% |
Best accuracy at batch_size=16384 (2%~11% error), consistent with the Page size (128 KB) being aligned with the batch data.
Goal: Prove the U-shaped memory curve $M_{peak} = M_{init} + s \cdot F + K \cdot b$ and that minimum occurs at $F_{opt} = \sqrt{R \cdot D \cdot b / s}$.
| Parameter | Value |
|---|---|
| D (devices) | 20 |
| R (rows/device) | 2,000,000 |
| n_field | 8 DOUBLE |
| s_data | 72 bytes/row |
| b | 928 bytes/device/flush |
| F_opt (formula) | 22,705 rows |
| M_min (formula) | 4.0 MB |
| F/F_opt | F | Peak (MB) | Trend |
|---|---|---|---|
| 0.12 | 2,838 | 17.0 | M_meta dominates (too many flushes) |
| 0.25 | 5,676 | 8.7 | |
| 0.50 | 11,352 | 4.6 | |
| 1.00 | 22,705 | 3.3 | Near minimum |
| 1.41 | 32,109 | 3.1 | Minimum |
| 2.00 | 45,410 | 3.4 | |
| 4.00 | 90,820 | 5.5 | |
| 8.00 | 181,640 | 10.7 | M_data dominates (too few flushes) |
The minimum occurs at F/F_opt = 1.0~1.4, confirming the EOQ model prediction.
The measured minimum at F/F_opt ≈ 1.4 rather than exactly 1.0 is explained by the formula constants' deviation from actual values:
| Constant | Formula Value | Measured Ratio (actual/formula) | Source of deviation |
|---|---|---|---|
| $s$ (data/row) | 72 bytes | ×0.92 (formula overestimates 8%) | Page header + statistics overhead amortized over rows |
| $b$ (meta/flush) | 928 bytes | ×1.36 (formula underestimates 36%) | Object headers, pointer overhead, allocator alignment |
Per-F detailed measurements:
| F/F_opt | F | s ratio | b ratio | Peak (MB) |
|---|---|---|---|---|
| 0.12 | 2,838 | 1.027 | 1.347 | 16.95 |
| 0.50 | 11,352 | 0.761 | 1.353 | 4.58 |
| 1.00 | 22,705 | 0.943 | 1.361 | 3.28 |
| 1.41 | 32,109 | 0.926 | 1.367 | 3.13 |
| 2.00 | 45,410 | 0.908 | 1.376 | 3.36 |
| 8.00 | 181,640 | 0.856 | 1.471 | 10.67 |
Correcting for the actual constants:
$$F_{opt,corrected} = \sqrt{\frac{b_{actual}}{s_{actual}}} \times F_{opt} = \sqrt{\frac{1.36}{0.92}} \times F_{opt} \approx 1.22 \times F_{opt}$$
The corrected theoretical optimum (1.22×) is close to the measured minimum (1.4×). The remaining gap is within the flat bottom of the U-curve: peak memory between F/F_opt = 1.0 and 2.0 differs by only 6% (3.13 MB vs 3.28 MB at 1.0×, 3.36 MB at 2.0×), making the exact position of the minimum practically insignificant.
Key takeaway: The formula parameter $b$ underestimates actual meta overhead by ~35% due to memory allocator alignment and object management overhead beyond the serialization size. This causes the actual optimal $F$ to shift rightward (fewer flushes needed). Since the formula yields a conservative $F_{opt}$ (slightly smaller than actual optimal), it is safe for pre-planning: the system flushes slightly more often than necessary, which is preferable to flushing too infrequently and risking memory overflow.
Goal: Validate that MemConstrainedWriter keeps memory within budget and triggers file rotation when meta accumulates.
| M_limit | peak_total | Within budget? | Flushes | Rotations | Files | Throughput |
|---|---|---|---|---|---|---|
| 2 MB | 1.16 MB | OK | 2441 | 5 | 6 | 1.77 M/s |
| 4 MB | 2.71 MB | OK | 918 | 0 | 1 | 1.69 M/s |
| 8 MB | 4.14 MB | OK | 409 | 0 | 1 | 1.72 M/s |
| 16 MB | 7.90 MB | OK | 194 | 0 | 1 | 1.94 M/s |
| 32 MB | 15.79 MB | OK | 95 | 0 | 1 | 2.01 M/s |
Memory constraint satisfied: peak_total never exceeds M_limit in all configurations. The two-level control (flush for M_data + rotation for M_meta) works correctly.
File rotation triggered at 2 MB: With only 0.6 MB meta budget, the accumulated meta (~1.3 KB/flush) triggers rotation after approximately 420 flushes per file. 50M rows across 6 files = ~8.3M rows/file, demonstrating continuous writing across file boundaries.
Data budget overshoot: At 2 and 4 MB, peak_data slightly exceeds data_budget (1.2~1.3x) because flush is checked after each batch write (4096 rows ≈ 230 KB per batch). This batch-granularity overshoot is bounded and does not cause total memory to exceed M_limit since the meta budget absorbs the slack.
Throughput vs budget: Throughput degrades slightly at very small budgets (1.77 M/s at 2 MB vs 2.01 M/s at 32 MB) due to frequent flush I/O overhead. The degradation is modest (~12%) even with a 16x budget reduction.
Flush count scales inversely: Flush count approximately halves when budget doubles (2441 → 918 → 409 → 194 → 95), consistent with $F \propto M_{avail}$.
Formula accuracy (PLAIN+UNCOMPRESSED): With no encoding/compression and aligned timestamp, Estimate API matches formula within ~8%. Remaining gap comes from page headers and statistics overhead.
Formula as conservative upper bound (with encoding): With SNAPPY+TS_2DIFF, formula overestimates $M_{data}$ by ~50%, which is by design. Direct monitoring provides exact runtime control. The thesis's two-tier design (formula for pre-planning, direct monitoring for runtime) is validated.
EOQ U-shape confirmed: The measured peak memory curve matches the theoretical U-shape. Minimum occurs at F/F_opt ≈ 1.0~1.4, close to the formula prediction. The shift is explained by $b$ being underestimated by ~35% (allocator alignment and object overhead beyond serialization size). After correcting $b$, the theoretical optimum is at 1.22× F_opt, consistent with the measurement.
Flat optimum region: Peak memory between F/F_opt = 1.0 and 2.0 differs by only 6%, meaning the exact value of F has little practical impact near the optimum. The formula's conservative F_opt (slightly smaller than actual) is safe: the system flushes slightly more often than necessary, avoiding memory overflow.
Two-level control works: Direct monitoring achieves stable throughput (2.0 M rows/s) across memory budgets (8128 MB). Flush frequency inversely proportional to budget ($F \propto M_{avail}$). No file rotation triggered at 20M rows.
Aligned mode formula: For aligned tables, $s_{data} = 8 + \sum \text{sizeof}(\text{field_type}j)$ (shared timestamp), differing from non-aligned $s{data} = \sum (8 + \text{sizeof}(\text{field_type}_j))$ by ~2x in the timestamp component.
Three layers of memory measurement: Formula ($s \times F$, upper bound) > Estimate API (encoded actual, for runtime control) > formula with actual encoding. ModStat allocation (includes ByteStream page granularity at 64 KB) > all of the above. Each layer serves a different purpose.