| # Benchmark Guide |
| |
| ## Generating Benchmark Data |
| |
| Use the CLI to generate a comprehensive benchmark suite: |
| |
| ```bash |
| otava-gen generate --output-dir ./benchmark --lengths 50 500 --seed 42 |
| ``` |
| |
| This creates: |
| - CSV files for each test case |
| - `manifest.json` with metadata about each file |
| - `summary.json` with overall statistics |
| |
| ## Running Otava |
| |
| ```bash |
| # Example Otava invocation (adjust based on Otava's actual CLI) |
| otava analyze --input ./benchmark/0001_step_function_L500.csv |
| ``` |
| |
| ## Comparing Algorithms |
| |
| The manifest.json file contains ground truth for each test case: |
| |
| ```python |
| import json |
| |
| with open("benchmark/manifest.json") as f: |
| manifest = json.load(f) |
| |
| for entry in manifest: |
| print(f"{entry['filename']}: {entry['n_change_points']} change points") |
| print(f" Expected indices: {entry['change_point_indices']}") |
| ``` |
| |
| ## Metrics |
| |
| When comparing algorithms, consider: |
| |
| 1. **True Positive Rate**: % of actual change points detected |
| 2. **False Positive Rate**: % of non-change-points flagged |
| 3. **Location Accuracy**: How close detected points are to actual |
| 4. **Latency**: How many points after change before detection |
