TSFile Viewer API Documentation - Enhanced Edition

Table of Contents

  1. Model Compatibility
  2. Performance Optimization
  3. API Endpoints
  4. UI Integration Guide
  5. Error Handling
  6. Best Practices

Model Compatibility

Table Model (V4) vs Tree Model (V3/Legacy)

Table Model (V4)

  • Structure: Uses TableSchema with TAG and FIELD columns
  • TAG Columns: Metadata dimensions (device_id, location, sensor_type)
  • FIELD Columns: Actual measurement data (temperature, humidity, pressure)
  • Device Identifier: tablename.tagvalue1.tagvalue2...
    • Example: sensor_data.device_001.room_A.floor_3
  • Use Cases:
    • Multi-dimensional IoT data
    • Relational-style queries
    • Tag-based filtering and grouping
    • Multiple tables in single file

Tree Model (V3/Legacy)

  • Structure: Traditional device-measurement hierarchy
  • Device Identifier: Same as table name
    • Example: root.building1.floor2.sensor3
  • Columns: Only FIELD columns (measurements)
  • Use Cases:
    • Time-series data organized by device paths
    • Legacy TSFile compatibility
    • Simple device-measurement relationships

API Unification

All endpoints work transparently with both models:

  • devices parameter filters by device identifiers (constructed from TAG values in Table Model)
  • measurements parameter filters by FIELD columns in both models
  • TAG columns appear in measurements map for Table Model (for display and filtering)
  • Response format is identical regardless of model type

Performance Optimization

Query Optimization Strategies

1. Pagination

Always use pagination for large datasets

{
  "fileId": "abc123",
  "limit": 100,      // Recommended: 100-500
  "offset": 0
}
  • Recommended page sizes: 100-500 rows
  • Maximum: 1000 rows per request
  • Use hasMore field in response to determine if more pages exist

2. Time Range Filtering

Most efficient filter - uses chunk-level optimization

{
  "fileId": "abc123",
  "startTime": 1705478400000,
  "endTime": 1705564800000  // 24-hour window recommended
}
  • Backend skips entire chunks outside the time range
  • Best Practice: Query 1-24 hour windows for optimal performance
  • Avoid unbounded time ranges on large files

3. Device/Measurement Filtering

Reduces data transfer and processing

{
  "fileId": "abc123",
  "devices": ["sensor_table.device_001.room_A"],
  "measurements": ["temperature", "humidity"]
}
  • Filter at query time, not in frontend
  • Specify only needed measurements to reduce payload size
  • For Table Model: Filter by constructed device IDs

4. Visualization Queries

Use aggregation and downsampling for charts

{
  "fileId": "abc123",
  "measurements": ["temperature"],
  "aggregation": "AVG",
  "windowSize": 60000,      // 1-minute windows
  "maxPoints": 1000         // LTTB downsampling
}
  • Aggregation types: AVG, MIN, MAX, COUNT
  • Window-based aggregation: Groups data into time windows (reduces points by 10-100x)
  • LTTB downsampling: Automatically applied when raw points > maxPoints (preserves trends)
  • Recommended maxPoints: 500-2000 for smooth charts without lag

Cache Utilization

The backend implements two-tier caching:

  1. Reader Cache (30 min TTL, 100 files)

    • Caches TSFile readers to avoid repeated file opening
    • Automatic eviction when capacity reached
    • Shared across all query types

  2. Metadata Cache (60 min TTL, 1000 entries)

    • Caches parsed metadata DTOs
    • Significantly faster metadata retrieval on repeated access
    • Cleared automatically after TTL expires

Best Practices:

  • Reuse same fileId for multiple queries to benefit from caching
  • Metadata queries are nearly instant after first load
  • Large files benefit most from reader caching

Query Timeout

  • Default timeout: 30 seconds
  • Configurable: tsfile.query.timeout-seconds in application.yml
  • Queries exceeding timeout return 504 Gateway Timeout
  • Mitigation: Use smaller time ranges, pagination, or increase timeout

API Endpoints

File Management

GET /api/files/tree

Used in: File Selection View

Browse server directory tree with lazy loading.

Parameters:

  • root (required): Whitelisted root directory
  • path (optional): Subdirectory for lazy loading

Example:

GET /api/files/tree?root=/data/tsfiles&path=sensors/2024

Response:

{
  "name": "2024",
  "path": "/data/tsfiles/sensors/2024",
  "isDirectory": true,
  "children": [
    {
      "name": "january.tsfile",
      "path": "/data/tsfiles/sensors/2024/january.tsfile",
      "isDirectory": false,
      "size": 10485760
    }
  ]
}

UI Usage:

  • FileSelectionView.vue: Directory tree navigation
  • FileTree.vue: Lazy-loaded tree component
  • Enable users to browse server directories and select TSFiles

POST /api/files/upload

Used in: File Selection View

Upload a TSFile to server.

Request: multipart/form-data with file Max size: 100MB (configurable via spring.servlet.multipart.max-file-size)

Response:

{
  "fileId": "f7a3b2c1-4d5e-6f7g-8h9i-0j1k2l3m4n5o",
  "fileName": "sensor_data.tsfile",
  "size": 10485760
}

Validation:

  • Must have .tsfile extension
  • Size must not exceed limit
  • File must be valid TSFile format (validated on access)

Metadata

GET /api/meta/{fileId}

Used in: Metadata View, Filter Panel

Retrieve comprehensive TSFile metadata.

Example:

GET /api/meta/f7a3b2c1-4d5e-6f7g-8h9i-0j1k2l3m4n5o

Table Model Response:

{
  "fileId": "f7a3b2c1-4d5e-6f7g-8h9i-0j1k2l3m4n5o",
  "fileName": "sensor_data.tsfile",
  "fileSize": 10485760,
  "version": "v4",
  "timeRange": {
    "startTime": 1705478400000,
    "endTime": 1705564800000
  },
  "deviceCount": 50,
  "measurementCount": 12,
  "tables": [
    {
      "tableName": "sensor_table",
      "tagColumns": [
        {
          "name": "device_id",
          "dataType": "STRING",
          "category": "TAG"
        },
        {
          "name": "location",
          "dataType": "STRING",
          "category": "TAG"
        }
      ],
      "fieldColumns": [
        {
          "name": "temperature",
          "dataType": "FLOAT",
          "encoding": "GORILLA",
          "compression": "SNAPPY",
          "category": "FIELD"
        },
        {
          "name": "humidity",
          "dataType": "FLOAT",
          "encoding": "GORILLA",
          "compression": "SNAPPY",
          "category": "FIELD"
        }
      ]
    }
  ],
  "rowGroups": [
    {
      "index": 0,
      "device": "sensor_table.device_001.room_A",
      "startTime": 1705478400000,
      "endTime": 1705492800000,
      "chunkCount": 8
    }
  ]
}

Tree Model Response:

{
  "fileId": "xyz789",
  "fileName": "legacy_sensors.tsfile",
  "version": "v3",
  "deviceCount": 10,
  "measurementCount": 5,
  "measurements": [
    {
      "name": "s1",
      "dataType": "FLOAT",
      "encoding": "GORILLA",
      "compression": "SNAPPY"
    },
    {
      "name": "s2",
      "dataType": "INT64",
      "encoding": "RLE",
      "compression": "SNAPPY"
    }
  ],
  "rowGroups": [
    {
      "index": 0,
      "device": "root.sensor1",
      "startTime": 1705478400000,
      "endTime": 1705564800000,
      "chunkCount": 5
    }
  ]
}

UI Usage:

  • MetadataView.vue: Display file metadata, tables, measurements, row groups
  • FilterPanel.vue: Populate device/measurement selection dropdowns
  • MetaCards.vue: Show statistics cards (device count, measurement count, time range)

Performance: Cached for 60 minutes after first load

Data Query

POST /api/data/preview

Used in: Data Preview View

Query and preview TSFile data with filtering and pagination.

Table Model Example:

{
  "fileId": "abc123",
  "devices": ["sensor_table.device_001.room_A"],
  "measurements": ["temperature", "humidity"],
  "startTime": 1705478400000,
  "endTime": 1705564800000,
  "limit": 100,
  "offset": 0
}

Table Model Response:

{
  "data": [
    {
      "timestamp": 1705478400000,
      "device": "sensor_table.device_001.room_A",
      "measurements": {
        "device_id": "device_001",
        "location": "room_A",
        "temperature": 25.5,
        "humidity": 60.0
      }
    }
  ],
  "total": 8640,
  "offset": 0,
  "limit": 100,
  "hasMore": true,
  "columnNames": ["Time", "Device", "device_id", "location", "temperature", "humidity"]
}

Tree Model Example:

{
  "fileId": "xyz789",
  "devices": ["root.sensor1"],
  "measurements": ["s1", "s2"],
  "startTime": 1705478400000,
  "endTime": 1705564800000,
  "limit": 100,
  "offset": 0
}

Tree Model Response:

{
  "data": [
    {
      "timestamp": 1705478400000,
      "device": "root.sensor1",
      "measurements": {
        "s1": 10.5,
        "s2": 100
      }
    }
  ],
  "total": 8640,
  "offset": 0,
  "limit": 100,
  "hasMore": true,
  "columnNames": ["Time", "Device", "s1", "s2"]
}

Value Range Filtering:

{
  "fileId": "abc123",
  "measurements": ["temperature"],
  "valueRange": {
    "min": 20.0,
    "max": 30.0
  }
}

Note: Value range filter requires ALL numeric measurements in a row to fall within the range (strictest filtering mode).

Multi-Table Query (Table Model):

{
  "fileId": "abc123",
  "devices": [
    "table1.device_001.room_A",
    "table2.device_002.room_B"
  ],
  "measurements": ["temperature"]
}

Backend automatically handles queries spanning multiple tables by device identifier prefix.

UI Usage:

  • DataPreviewView.vue: Main data browsing interface
  • FilterPanel.vue: Build query from user selections
  • DataTable.vue: Display paginated results with column headers
  • Workflow:
    1. User selects filters in FilterPanel
    2. View sends POST request with filter parameters
    3. DataTable displays results with pagination controls
    4. User clicks next/previous or changes limit to navigate

Performance Tips:

  • Use time range filters for best performance (chunk-level optimization)
  • Limit to 100-500 rows per page for smooth UI
  • Filter by specific devices/measurements to reduce payload
  • Avoid value range filters on large datasets (scans all data)

POST /api/data/query

Used in: Chart Visualization View

Query data optimized for visualization with aggregation and downsampling.

Basic Visualization Query:

{
  "fileId": "abc123",
  "devices": ["sensor_table.device_001.room_A"],
  "measurements": ["temperature", "humidity"],
  "startTime": 1705478400000,
  "endTime": 1705564800000,
  "maxPoints": 1000
}

Response:

{
  "series": [
    {
      "name": "temperature",
      "device": "sensor_table.device_001.room_A",
      "data": [
        {"timestamp": 1705478400000, "value": 25.5},
        {"timestamp": 1705478460000, "value": 25.6}
      ],
      "unit": "°C"
    }
  ],
  "timeRange": {
    "start": 1705478400000,
    "end": 1705564800000
  },
  "downsampled": true,
  "originalPoints": 86400,
  "returnedPoints": 1000
}

Time Window Aggregation:

{
  "fileId": "abc123",
  "measurements": ["temperature"],
  "aggregation": "AVG",
  "windowSize": 300000,  // 5-minute windows
  "startTime": 1705478400000,
  "endTime": 1705564800000
}

Response with Aggregation:

{
  "series": [
    {
      "name": "temperature_AVG",
      "device": "sensor_table.device_001.room_A",
      "data": [
        {"timestamp": 1705478400000, "value": 25.3, "count": 300},
        {"timestamp": 1705478700000, "value": 25.5, "count": 300}
      ],
      "aggregation": "AVG",
      "windowSize": 300000
    }
  ]
}

Aggregation Types:

  • AVG: Average value per window
  • MIN: Minimum value per window
  • MAX: Maximum value per window
  • COUNT: Count of points per window

Downsampling (LTTB):

  • Automatically applied when raw data points exceed maxPoints
  • Preserves visual trends and patterns
  • Reduces data transfer and chart rendering time
  • Algorithm: Largest Triangle Three Buckets (LTTB)

UI Usage:

  • ChartVisualizationView.vue: Interactive chart interface
  • ChartPanel.vue: ECharts visualization component
  • Workflow:
    1. User selects measurements and time range
    2. Optionally select aggregation type and window size
    3. Set maxPoints for downsampling (default: 1000)
    4. View sends POST request
    5. ChartPanel renders time-series line chart
    6. User can zoom, pan, and explore data points

Performance Tips:

  • Use aggregation for long time ranges (reduces points by 10-100x)
  • Set maxPoints to 500-2000 for responsive charts
  • Window size recommendations:
    • 1 hour range: 10-60 second windows
    • 24 hours: 1-5 minute windows
    • 1 week: 15-60 minute windows
    • 1 month: 1-6 hour windows

UI Integration Guide

View-to-API Mapping

1. File Selection View (FileSelectionView.vue)

Primary APIs:

  • GET /api/files/tree - Browse server directories
  • POST /api/files/upload - Upload local TSFiles

Integration:

// Browse directory tree
const response = await fetch(`/api/files/tree?root=${encodedRoot}&path=${encodedPath}`)
const treeNode = await response.json()

// Upload file
const formData = new FormData()
formData.append('file', file)
const uploadResponse = await fetch('/api/files/upload', {
  method: 'POST',
  body: formData
})
const { fileId } = await uploadResponse.json()
router.push(`/meta/${fileId}`)

User Flow:

  1. User browses directories via tree navigation (lazy-loaded)
  2. User selects .tsfile → Navigate to Metadata View
  3. OR User uploads file → Auto-navigate to Metadata View with new fileId

2. Metadata View (MetadataView.vue)

Primary APIs:

  • GET /api/meta/{fileId} - Load metadata

Integration:

const metadata = await metadataApi.getMetadata(fileId)

// Display metadata cards
displayMetaCards({
  deviceCount: metadata.deviceCount,
  measurementCount: metadata.measurementCount,
  timeRange: metadata.timeRange,
  fileSize: metadata.fileSize
})

// Show tables (Table Model) or measurements (Tree Model)
if (metadata.tables) {
  displayTables(metadata.tables)  // Table Model
} else {
  displayMeasurements(metadata.measurements)  // Tree Model
}

// Display row groups and chunks
displayRowGroups(metadata.rowGroups)

User Flow:

  1. View loads metadata automatically on mount
  2. User sees file statistics, structure, and metadata
  3. User clicks “View Data” → Navigate to Data Preview
  4. User clicks “Visualize” → Navigate to Chart View

3. Data Preview View (DataPreviewView.vue)

Primary APIs:

  • GET /api/meta/{fileId} - Populate filters (via FilterPanel)
  • POST /api/data/preview - Query data with filters

Integration:

// FilterPanel populates dropdowns from metadata
const metadata = await metadataApi.getMetadata(fileId)
const devices = metadata.rowGroups.map(rg => rg.device)
const measurements = metadata.tables 
  ? metadata.tables[0].fieldColumns.map(fc => fc.name)
  : metadata.measurements.map(m => m.name)

// Query data when filters change
async function handleFilterChange(filters) {
  const request = {
    fileId,
    ...filters,
    limit: 100,
    offset: 0
  }
  const response = await dataApi.previewData(request)
  displayData(response.data, response.columnNames)
  updatePagination(response.total, response.offset, response.hasMore)
}

// Handle pagination
async function handlePageChange(direction) {
  const newOffset = direction === 'next' 
    ? currentOffset + limit 
    : currentOffset - limit
  const request = { fileId, ...currentFilters, limit, offset: newOffset }
  const response = await dataApi.previewData(request)
  // Update display...
}

User Flow:

  1. FilterPanel loads metadata and populates dropdowns
  2. User selects table (Table Model only), devices, measurements, time range
  3. DataTable displays filtered results
  4. User pagina tes through data or changes page size
  5. User exports data as CSV/JSON

4. Chart Visualization View (ChartVisualizationView.vue)

Primary APIs:

  • POST /api/data/query - Query visualization data

Integration:

async function loadChartData() {
  const measurements = measurementsInput.split(',').map(m => m.trim())
  
  const request = {
    fileId,
    measurements,
    startTime: startTime ? new Date(startTime).getTime() : undefined,
    endTime: endTime ? new Date(endTime).getTime() : undefined,
    aggregation: aggregationType || undefined,
    windowSize: aggregationType ? windowSize : undefined,
    maxPoints: 1000
  }
  
  const response = await dataApi.queryChartData(request)
  
  // Configure ECharts
  const chartConfig = {
    series: response.series.map(s => ({
      name: s.name,
      type: 'line',
      data: s.data.map(d => [d.timestamp, d.value]),
      smooth: true
    })),
    xAxis: { type: 'time' },
    yAxis: { type: 'value' }
  }
  
  chartInstance.setOption(chartConfig)
}

User Flow:

  1. User enters measurements (comma-separated)
  2. User sets time range, aggregation, window size, maxPoints
  3. User clicks “Apply” → Chart loads
  4. User interacts with chart (zoom, pan, hover for values)
  5. User changes parameters and reloads chart

Error Handling

Common Error Responses

400 Bad Request

{
  "status": 400,
  "error": "Bad Request",
  "message": "Invalid file ID format",
  "timestamp": "2024-01-17T10:30:00Z",
  "path": "/api/data/preview"
}

Common Causes:

  • Invalid fileId format
  • Validation errors (limit < 1, offset < 0)
  • Invalid time range (startTime > endTime)
  • Missing required fields

403 Forbidden

{
  "status": 403,
  "error": "Forbidden",
  "message": "Directory path outside whitelist",
  "path": "/api/files/tree"
}

Cause: Attempted to access directory not in tsfile.allowed-directories

404 Not Found

{
  "status": 404,
  "error": "Not Found",
  "message": "TSFile not found for fileId: abc123",
  "path": "/api/meta/abc123"
}

Common Causes:

  • File has been deleted or moved
  • fileId not in upload cache
  • Reader cache evicted and file path invalid

413 Payload Too Large

{
  "status": 413,
  "error": "Payload Too Large",
  "message": "File size exceeds maximum allowed size of 100MB"
}

Cause: Upload file exceeds spring.servlet.multipart.max-file-size

504 Gateway Timeout

{
  "status": 504,
  "error": "Gateway Timeout",
  "message": "Query execution exceeded timeout of 30 seconds"
}

Causes:

  • Large file with unbounded time range
  • Complex value range filtering
  • Too many devices/measurements requested

Mitigation:

  • Add time range filters
  • Reduce number of devices/measurements
  • Use pagination
  • Increase timeout in configuration

Best Practices

For Frontend Developers

  1. Always handle loading states

    const loading = ref(false)
const error = ref(null)

try {
  loading.value = true
  const response = await dataApi.previewData(request)
  // Handle success...
} catch (e) {
  error.value = e.message
  showErrorToast(e.message)
} finally {
  loading.value = false
}

  2. Use TypeScript types from API

    import type { DataRow, DataPreviewRequest, TSFileMetadata } from '@/api/types'

  3. Implement pagination properly

    • Start with reasonable page size (100)
    • Disable “Previous” when offset === 0
    • Disable “Next” when !hasMore
    • Update offset correctly on page changes

  4. Cache fileId in stores

    const fileStore = useFileStore()
fileStore.setCurrentFile(fileId, fileName)

  5. Provide user feedback

    • Show loading spinners during API calls
    • Display error messages with clear context
    • Use toast notifications for actions (upload success, export complete)
    • Show empty states when no data found

For Backend Configuration

  1. Whitelist directories

    tsfile:
  allowed-directories:
    - /data/tsfiles
    - /mnt/storage/sensors

  2. Tune cache settings based on usage

    tsfile:
  cache:
    metadata:
      max-size: 1000  # Increase for many files
      ttl-minutes: 60
    reader:
      max-size: 100   # Increase for concurrent users
      ttl-minutes: 30

  3. Adjust upload limits

    spring:
  servlet:
    multipart:
      max-file-size: 500MB  # For large TSFiles
      max-request-size: 500MB

  4. Set appropriate timeout

    tsfile:
  query:
    timeout-seconds: 60  # For large files

Performance Checklist

  • [ ] Use time range filters whenever possible
  • [ ] Implement pagination (100-500 rows per page)
  • [ ] Use aggregation for visualization queries
  • [ ] Set reasonable maxPoints (1000-2000)
  • [ ] Filter by specific devices/measurements
  • [ ] Cache metadata responses in frontend
  • [ ] Debounce user input in filter forms
  • [ ] Show loading indicators for all API calls
  • [ ] Handle errors gracefully with user-friendly messages
  • [ ] Use virtualization for large data tables

API Versioning

Current API version: v1 (implicit in /api path)

Future versions will use explicit versioning:

  • /api/v2/data/preview
  • Backward compatibility maintained for v1 endpoints

Support

For issues, feature requests, or questions: