OpenDAL C++ Test Framework

This directory contains a comprehensive test framework for the OpenDAL C++ bindings, designed to match the sophistication and organization of test frameworks in other language bindings (Python, Go, Node.js).

Structure

tests/
├── framework/              # Test framework utilities and base classes
│   └── test_framework.hpp  # Core framework header
├── behavior/               # Behavior-based tests (similar to Go tests)
│   ├── read_test.cpp       # Read operation tests
│   ├── write_test.cpp      # Write operation tests
│   ├── delete_test.cpp     # Delete operation tests
│   ├── list_test.cpp       # List operation tests
│   └── async_read_test.cpp # Async read tests (if OPENDAL_ENABLE_ASYNC)
├── benchmark/              # Performance benchmarks
│   └── benchmark_test.cpp  # Comprehensive benchmarks
├── test_main.cpp           # Main test runner
├── basic_test.cpp          # Legacy basic tests (for compatibility)
├── async_test.cpp          # Legacy async tests (for compatibility)
└── README.md               # This file

Features

🚀 Comprehensive Test Coverage

• Behavior-based testing: Tests organized by functionality (read, write, delete, list, etc.)
• Async testing: Full support for async operations when enabled
• Performance benchmarks: Detailed performance measurement and analysis
• Error handling: Comprehensive error condition testing
• Concurrency testing: Multi-threaded operation validation

🔧 Test Framework Features

• Environment configuration: Automatic test environment setup from environment variables
• Random data generation: Utilities for generating test data
• Performance measurement: Built-in timing and benchmarking utilities
• Service abstraction: Tests work with any OpenDAL service (memory, fs, s3, etc.)
• Capability-based testing: Skip tests based on service capabilities

📊 Test Categories

Behavior Tests

• Read operations: File reading, error handling, edge cases
• Write operations: File writing, overwriting, binary data
• Delete operations: File/directory deletion, recursive operations
• List operations: Directory listing, metadata validation
• Async operations: All above operations in async mode

Benchmark Tests

• Performance measurement: Latency and throughput testing
• Memory usage: Memory allocation pattern analysis
• Concurrent operations: Multi-threaded performance testing
• Different file sizes: Performance across various data sizes

Usage

Building and Running Tests

# Configure with testing enabled
cmake -DOPENDAL_ENABLE_TESTING=ON -DOPENDAL_DEV=ON ..

# Build tests
make opendal_cpp_test

# Run all tests
./opendal_cpp_test

# Run specific test suites
./opendal_cpp_test --gtest_filter="ReadBehaviorTest.*"
./opendal_cpp_test --gtest_filter="*Benchmark*"

Environment Configuration

The test framework supports configuration through environment variables:

# Set the service to test (default: memory)
export OPENDAL_TEST=memory

# Service-specific configuration
export OPENDAL_MEMORY_ROOT=/tmp/opendal_test

# Disable random root (for deterministic paths)
export OPENDAL_DISABLE_RANDOM_ROOT=true

# For other services (example with filesystem)
export OPENDAL_TEST=fs
export OPENDAL_FS_ROOT=/tmp/opendal_fs_test

Async Testing

When built with async support (-DOPENDAL_ENABLE_ASYNC=ON):

# Build with async support
cmake -DOPENDAL_ENABLE_TESTING=ON -DOPENDAL_ENABLE_ASYNC=ON -DOPENDAL_DEV=ON ..
make opendal_cpp_test

# Run async tests
./opendal_cpp_test --gtest_filter="Async*"

Writing New Tests

Basic Test Structure

#include "../framework/test_framework.hpp"

namespace opendal::test {

class MyBehaviorTest : public OpenDALTest {
protected:
    void SetUp() override {
        OpenDALTest::SetUp();
        // Additional setup if needed
    }
};

OPENDAL_TEST_F(MyBehaviorTest, TestSomething) {
    auto path = random_path();
    auto content = random_string(100);
    
    // Your test logic here
    op_.write(path, content);
    auto result = op_.read(path);
    EXPECT_EQ(result, content);
}

} // namespace opendal::test

Async Test Structure

#ifdef OPENDAL_ENABLE_ASYNC

class MyAsyncBehaviorTest : public AsyncOpenDALTest {
    // Similar structure but using co_await
};

OPENDAL_TEST_F(MyAsyncBehaviorTest, TestAsyncSomething) {
    cppcoro::sync_wait([&]() -> cppcoro::task<void> {
        auto path = random_path();
        auto content = random_bytes(100);
        
        co_await op_->write(path, content);
        auto result = co_await op_->read(path);
        EXPECT_EQ(result, content);
        
        co_return;
    }());
}

#endif

Benchmark Test Structure

class MyBenchmarkTest : public BenchmarkTest {
    // Inherit from BenchmarkTest
};

OPENDAL_TEST_F(MyBenchmarkTest, BenchmarkMyOperation) {
    benchmark_operation(
        "My operation description",
        [&]() {
            // Operation to benchmark
            auto path = random_path();
            op_.write(path, random_string(1024));
        },
        100 // iterations
    );
}

Test Framework API

Base Classes

• OpenDALTest: Base class for sync tests
• AsyncOpenDALTest: Base class for async tests (when enabled)
• BenchmarkTest: Base class for performance tests

Utility Functions

• random_path(): Generate random file path
• random_dir_path(): Generate random directory path
• random_string(size): Generate random string data
• random_bytes(size): Generate random binary data

Performance Utilities

• PerformanceTimer: High-resolution timing
• benchmark_operation(): Automated benchmarking with statistics

Environment

• TestConfig::instance(): Access test configuration
• initialize_test_framework(): Initialize framework (called by main)

Integration with CI/CD

GitHub Actions Example

- name: Run C++ Tests
  run: |
    cd bindings/cpp
    mkdir build && cd build
    cmake -DOPENDAL_ENABLE_TESTING=ON -DOPENDAL_DEV=ON ..
    make opendal_cpp_test
    ./opendal_cpp_test --gtest_output=xml:test_results.xml

Test Selection

# Run only basic functionality tests
./opendal_cpp_test --gtest_filter="*BehaviorTest*" --gtest_filter="-*Benchmark*"

# Run only benchmarks
./opendal_cpp_test --gtest_filter="*Benchmark*"

# Run only async tests
./opendal_cpp_test --gtest_filter="*Async*"

# Run with verbose output
./opendal_cpp_test --gtest_verbose

Comparison with Other Language Bindings

This C++ test framework provides equivalent functionality to:

• Python: pytest-based testing with fixtures and parametrization
• Go: behavior-based testing with comprehensive coverage
• Node.js: Vitest-based testing with async/sync separation
• All: Environment-based configuration and capability testing

Contributing

When adding new tests:

1. Organize by behavior: Group related tests in behavior files
2. Include async variants: When adding sync tests, consider async equivalents
3. Add benchmarks: For performance-critical operations
4. Document thoroughly: Include clear test descriptions
5. Follow patterns: Use existing test patterns and utilities

Troubleshooting

Common Issues

1. Tests fail with “No service configured”

   • Set OPENDAL_TEST environment variable
   • Ensure the service is properly configured

2. Async tests don't compile

   • Build with -DOPENDAL_ENABLE_ASYNC=ON
   • Ensure C++20 support and clang compiler

3. Permission errors

   • Check file system permissions
   • Ensure test directories are writable

4. Memory issues

   • Build with address sanitizer: -DOPENDAL_ENABLE_ADDRESS_SANITIZER=ON
   • Check for memory leaks in custom tests