# Phase 2: Data Persistence & File Management - Research

**Researched:** 2026-03-12
**Phase:** 02-data-persistence-file-management

## Domain Overview

Phase 2 implements CSV data logging to flash memory with file management UI. This builds on Phase 1's Flask backend by adding:
- File system operations (list, read, delete)
- CSV file generation/management
- tidEDA formatted export
- File management UI page

## Requirements Analysis

| Requirement | Description | Implementation Approach |
|-------------|-------------|------------------------|
| FILE-01 | List CSV files in flash memory | Flask route returning JSON array of file metadata |
| FILE-02 | View CSV file contents | Flask route serving file content with pagination |
| FILE-03 | Navigate file list (scroll) | Frontend JavaScript with pagination or virtual scroll |
| FILE-04 | Delete CSV files | Flask DELETE endpoint with confirmation |
| FILE-05 | Generate tidEDA formatted files | Backend transformation endpoint |
| BACK-04 | File management API | All above endpoints combined |

## Technical Implementation

### Storage Location
- **Path:** `/myvscada/logger` (per CONTEXT.md)
- **Note:** On Pi, this may need to be relative to project or use a configurable path

### File Naming Convention
- Format: `SP{station_id}_{timestamp}.csv`
- Example: `SP001_20260312_143022.csv`

### API Endpoints Needed

```
GET  /api/files              - List all CSV files with metadata
GET  /api/files/<filename>   - Get file contents
DELETE /api/files/<filename> - Delete a file
POST /api/files/export       - Generate tidEDA format from CSV
```

### Response Formats

**GET /api/files**
```json
{
  "files": [
    {
      "name": "SP001_20260312_143022.csv",
      "size": 1234,
      "created": "2026-03-12T14:30:22Z"
    }
  ],
  "total": 10
}
```

**GET /api/files/<filename>**
```json
{
  "name": "SP001_20260312_143022.csv",
  "content": "timestamp,rainfall,solar,battery\n...",
  "lines": 50
}
```

### tidEDA Format

Based on standard environmental data transmission formats. The tidEDA format should contain:
- Station identification
- Timestamp
- Sensor readings (rainfall, voltage)
- Data quality indicators

**Suggested structure:**
```
$STATION,SP001
$DATETIME,2026-03-12T14:30:00Z
$DATA
timestamp,rainfall_mm,solar_v,battery_v
2026-03-12T14:30:00,0.2,12.4,12.8
...
$END
```

## Frontend Implementation

### New Page: Files (`/files`)
- Bootstrap 5.3 template (reuse existing patterns)
- File list with:
  - Filename, size, date
  - View button
  - Delete button with confirmation
  - Export to tidEDA button
- Scroll navigation (per FILE-03)
- Touch-optimized 48px+ targets

### UI Components
- File list table/card layout
- Modal for file content viewing
- Confirmation dialogs for delete
- Loading states during API calls

## Dependencies

No new Python packages required beyond existing Flask:
- `os`, `datetime`, `json` (stdlib)
- Flask already installed

## Integration Points

1. **Main Navigation:** Add "Files" menu item
2. **Settings API reuse:** Get station ID from settings
3. **Socket.IO:** Optional - file list refresh on changes
4. **Template base:** Extend existing Bootstrap 5.3 layout

## Validation Architecture

**Success Criteria Validation:**

1. User can view list of CSV files stored in flash memory
   - `GET /api/files` returns file array
   - Files page renders list correctly

2. User can navigate file list (scroll up/down)
   - Frontend implements scroll or pagination
   - Touch targets work on 7" display

3. User can view contents of selected CSV files
   - `GET /api/files/<filename>` returns content
   - Modal/displays shows formatted content

4. User can delete unwanted CSV files from flash memory
   - `DELETE /api/files/<filename>` removes file
   - UI updates after deletion

5. User can generate tidEDA formatted files
   - `POST /api/files/export` generates format
   - Returns downloadable content

## Risks & Mitigations

| Risk | Mitigation |
|------|------------|
| File path not accessible | Use configurable path, default to project data dir |
| Large files crash display | Paginate content, limit preview lines |
| Concurrent file access | Handle file locked errors gracefully |
| tidEDA format incompatibility | Define format clearly, add version field |

## Implementation Order

1. Backend API routes first (file operations)
2. Frontend file list page
3. File viewing modal
4. Delete functionality with confirmation
5. tidEDA export endpoint
6. Integration testing

---

*Research complete - ready for planning*