rtu_v5/.planning/phases/02-data-persistence-file-management/02-RESEARCH.md

Phase 2: Data Persistence & File Management - Research

Researched: 2026-03-12 Phase: 02-data-persistence-file-management

Domain Overview

Phase 2 implements CSV data logging to flash memory with file management UI. This builds on Phase 1's Flask backend by adding:

• File system operations (list, read, delete)
• CSV file generation/management
• tidEDA formatted export
• File management UI page

Requirements Analysis

Requirement	Description	Implementation Approach
FILE-01	List CSV files in flash memory	Flask route returning JSON array of file metadata
FILE-02	View CSV file contents	Flask route serving file content with pagination
FILE-03	Navigate file list (scroll)	Frontend JavaScript with pagination or virtual scroll
FILE-04	Delete CSV files	Flask DELETE endpoint with confirmation
FILE-05	Generate tidEDA formatted files	Backend transformation endpoint
BACK-04	File management API	All above endpoints combined

Technical Implementation

Storage Location

• Path: /myvscada/logger (per CONTEXT.md)
• Note: On Pi, this may need to be relative to project or use a configurable path

File Naming Convention

• Format: SP{station_id}_{timestamp}.csv
• Example: SP001_20260312_143022.csv

API Endpoints Needed

GET  /api/files              - List all CSV files with metadata
GET  /api/files/<filename>   - Get file contents
DELETE /api/files/<filename> - Delete a file
POST /api/files/export       - Generate tidEDA format from CSV

Response Formats

GET /api/files

{
  "files": [
    {
      "name": "SP001_20260312_143022.csv",
      "size": 1234,
      "created": "2026-03-12T14:30:22Z"
    }
  ],
  "total": 10
}

GET /api/files/

{
  "name": "SP001_20260312_143022.csv",
  "content": "timestamp,rainfall,solar,battery\n...",
  "lines": 50
}

tidEDA Format

Based on standard environmental data transmission formats. The tidEDA format should contain:

• Station identification
• Timestamp
• Sensor readings (rainfall, voltage)
• Data quality indicators

Suggested structure:

$STATION,SP001
$DATETIME,2026-03-12T14:30:00Z
$DATA
timestamp,rainfall_mm,solar_v,battery_v
2026-03-12T14:30:00,0.2,12.4,12.8
...
$END

Frontend Implementation

New Page: Files (/files)

• Bootstrap 5.3 template (reuse existing patterns)
• File list with:
  • Filename, size, date
  • View button
  • Delete button with confirmation
  • Export to tidEDA button
• Scroll navigation (per FILE-03)
• Touch-optimized 48px+ targets

UI Components

• File list table/card layout
• Modal for file content viewing
• Confirmation dialogs for delete
• Loading states during API calls

Dependencies

No new Python packages required beyond existing Flask:

• os, datetime, json (stdlib)
• Flask already installed

Integration Points

1. Main Navigation: Add "Files" menu item
2. Settings API reuse: Get station ID from settings
3. Socket.IO: Optional - file list refresh on changes
4. Template base: Extend existing Bootstrap 5.3 layout

Validation Architecture

Success Criteria Validation:

1. User can view list of CSV files stored in flash memory

   • GET /api/files returns file array
   • Files page renders list correctly

2. User can navigate file list (scroll up/down)

   • Frontend implements scroll or pagination
   • Touch targets work on 7" display

3. User can view contents of selected CSV files

   • GET /api/files/<filename> returns content
   • Modal/displays shows formatted content

4. User can delete unwanted CSV files from flash memory

   • DELETE /api/files/<filename> removes file
   • UI updates after deletion

5. User can generate tidEDA formatted files

   • POST /api/files/export generates format
   • Returns downloadable content

Risks & Mitigations

Risk	Mitigation
File path not accessible	Use configurable path, default to project data dir
Large files crash display	Paginate content, limit preview lines
Concurrent file access	Handle file locked errors gracefully
tidEDA format incompatibility	Define format clearly, add version field

Implementation Order

1. Backend API routes first (file operations)
2. Frontend file list page
3. File viewing modal
4. Delete functionality with confirmation
5. tidEDA export endpoint
6. Integration testing

---

Research complete - ready for planning