164 lines
4.4 KiB
Markdown
164 lines
4.4 KiB
Markdown
# Phase 2: Data Persistence & File Management - Research
|
|
|
|
**Researched:** 2026-03-12
|
|
**Phase:** 02-data-persistence-file-management
|
|
|
|
## Domain Overview
|
|
|
|
Phase 2 implements CSV data logging to flash memory with file management UI. This builds on Phase 1's Flask backend by adding:
|
|
- File system operations (list, read, delete)
|
|
- CSV file generation/management
|
|
- tidEDA formatted export
|
|
- File management UI page
|
|
|
|
## Requirements Analysis
|
|
|
|
| Requirement | Description | Implementation Approach |
|
|
|-------------|-------------|------------------------|
|
|
| FILE-01 | List CSV files in flash memory | Flask route returning JSON array of file metadata |
|
|
| FILE-02 | View CSV file contents | Flask route serving file content with pagination |
|
|
| FILE-03 | Navigate file list (scroll) | Frontend JavaScript with pagination or virtual scroll |
|
|
| FILE-04 | Delete CSV files | Flask DELETE endpoint with confirmation |
|
|
| FILE-05 | Generate tidEDA formatted files | Backend transformation endpoint |
|
|
| BACK-04 | File management API | All above endpoints combined |
|
|
|
|
## Technical Implementation
|
|
|
|
### Storage Location
|
|
- **Path:** `/myvscada/logger` (per CONTEXT.md)
|
|
- **Note:** On Pi, this may need to be relative to project or use a configurable path
|
|
|
|
### File Naming Convention
|
|
- Format: `SP{station_id}_{timestamp}.csv`
|
|
- Example: `SP001_20260312_143022.csv`
|
|
|
|
### API Endpoints Needed
|
|
|
|
```
|
|
GET /api/files - List all CSV files with metadata
|
|
GET /api/files/<filename> - Get file contents
|
|
DELETE /api/files/<filename> - Delete a file
|
|
POST /api/files/export - Generate tidEDA format from CSV
|
|
```
|
|
|
|
### Response Formats
|
|
|
|
**GET /api/files**
|
|
```json
|
|
{
|
|
"files": [
|
|
{
|
|
"name": "SP001_20260312_143022.csv",
|
|
"size": 1234,
|
|
"created": "2026-03-12T14:30:22Z"
|
|
}
|
|
],
|
|
"total": 10
|
|
}
|
|
```
|
|
|
|
**GET /api/files/<filename>**
|
|
```json
|
|
{
|
|
"name": "SP001_20260312_143022.csv",
|
|
"content": "timestamp,rainfall,solar,battery\n...",
|
|
"lines": 50
|
|
}
|
|
```
|
|
|
|
### tidEDA Format
|
|
|
|
Based on standard environmental data transmission formats. The tidEDA format should contain:
|
|
- Station identification
|
|
- Timestamp
|
|
- Sensor readings (rainfall, voltage)
|
|
- Data quality indicators
|
|
|
|
**Suggested structure:**
|
|
```
|
|
$STATION,SP001
|
|
$DATETIME,2026-03-12T14:30:00Z
|
|
$DATA
|
|
timestamp,rainfall_mm,solar_v,battery_v
|
|
2026-03-12T14:30:00,0.2,12.4,12.8
|
|
...
|
|
$END
|
|
```
|
|
|
|
## Frontend Implementation
|
|
|
|
### New Page: Files (`/files`)
|
|
- Bootstrap 5.3 template (reuse existing patterns)
|
|
- File list with:
|
|
- Filename, size, date
|
|
- View button
|
|
- Delete button with confirmation
|
|
- Export to tidEDA button
|
|
- Scroll navigation (per FILE-03)
|
|
- Touch-optimized 48px+ targets
|
|
|
|
### UI Components
|
|
- File list table/card layout
|
|
- Modal for file content viewing
|
|
- Confirmation dialogs for delete
|
|
- Loading states during API calls
|
|
|
|
## Dependencies
|
|
|
|
No new Python packages required beyond existing Flask:
|
|
- `os`, `datetime`, `json` (stdlib)
|
|
- Flask already installed
|
|
|
|
## Integration Points
|
|
|
|
1. **Main Navigation:** Add "Files" menu item
|
|
2. **Settings API reuse:** Get station ID from settings
|
|
3. **Socket.IO:** Optional - file list refresh on changes
|
|
4. **Template base:** Extend existing Bootstrap 5.3 layout
|
|
|
|
## Validation Architecture
|
|
|
|
**Success Criteria Validation:**
|
|
|
|
1. User can view list of CSV files stored in flash memory
|
|
- `GET /api/files` returns file array
|
|
- Files page renders list correctly
|
|
|
|
2. User can navigate file list (scroll up/down)
|
|
- Frontend implements scroll or pagination
|
|
- Touch targets work on 7" display
|
|
|
|
3. User can view contents of selected CSV files
|
|
- `GET /api/files/<filename>` returns content
|
|
- Modal/displays shows formatted content
|
|
|
|
4. User can delete unwanted CSV files from flash memory
|
|
- `DELETE /api/files/<filename>` removes file
|
|
- UI updates after deletion
|
|
|
|
5. User can generate tidEDA formatted files
|
|
- `POST /api/files/export` generates format
|
|
- Returns downloadable content
|
|
|
|
## Risks & Mitigations
|
|
|
|
| Risk | Mitigation |
|
|
|------|------------|
|
|
| File path not accessible | Use configurable path, default to project data dir |
|
|
| Large files crash display | Paginate content, limit preview lines |
|
|
| Concurrent file access | Handle file locked errors gracefully |
|
|
| tidEDA format incompatibility | Define format clearly, add version field |
|
|
|
|
## Implementation Order
|
|
|
|
1. Backend API routes first (file operations)
|
|
2. Frontend file list page
|
|
3. File viewing modal
|
|
4. Delete functionality with confirmation
|
|
5. tidEDA export endpoint
|
|
6. Integration testing
|
|
|
|
---
|
|
|
|
*Research complete - ready for planning*
|