admin/rtu_v5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(phase-02): create phase plans for file management

Browse Source
This commit is contained in:
TCK e-Solutions Sdn. Bhd.
2026-03-12 06:46:29 +08:00
parent ef9eeef95f
commit bcdcbc4a78
4 changed files with 628 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

181
.planning/phases/02-data-persistence-file-management/02-01-PLAN.md Normal file
View File

@@ -0,0 +1,181 @@
---
phase: 02-data-persistence-file-management
plan: "01"
type: execute
wave: 1
depends_on: []
files_modified:
- src/app.py
- src/templates/files.html
autonomous: true
requirements:
- FILE-01
- FILE-02
- FILE-04
- BACK-04
must_haves:
truths:
- "User can view list of CSV files stored in flash memory"
- "User can view contents of selected CSV files"
- "User can delete unwanted CSV files from flash memory"
artifacts:
- path: "src/app.py"
provides: "File management API endpoints"
exports: ["GET /api/files", "GET /api/files/<filename>", "DELETE /api/files/<filename>"]
- path: "src/templates/files.html"
provides: "File management UI page"
routes: ["/files"]
key_links:
- from: "src/templates/files.html"
to: "/api/files"
via: "fetch in useEffect"
pattern: "fetch.*api/files"
---
<objective>
Create file management API endpoints and basic UI for listing/viewing/deleting CSV files.
</objective>
<context>
@./.planning/phases/02-data-persistence-file-management/02-CONTEXT.md
@./.planning/phases/02-data-persistence-file-management/02-RESEARCH.md
@./.planning/ROADMAP.md
@./.planning/REQUIREMENTS.md
@./src/app.py
</context>
<tasks>
<task type="auto">
<name>task 1: Add file management API endpoints to Flask backend</name>
<files>src/app.py</files>
<action>
Add the following API endpoints to src/app.py:
1. GET /api/files - List all CSV files in the logger directory
- Returns JSON: {files: [{name, size, created}], total: count}
- Storage path: /myvscada/logger (configurable, default to src/data/logger)
- Handle missing directory gracefully
2. GET /api/files/&lt;filename&gt; - Get file contents
- Returns JSON: {name, content: "full file text", lines: count}
- Limit preview to first 100 lines for large files
3. DELETE /api/files/&lt;filename&gt; - Delete a file
- Returns JSON: {success: true/false}
- Confirm file exists before delete
- Handle errors gracefully
Use the existing Flask patterns from src/app.py:
- Use jsonify for responses
- Use request.get_json() for POST data
- Add error handling with try/except
- Follow existing route structure
</action>
<verify>
<automated>curl -s http://localhost:8080/api/files | python3 -c "import sys,json; d=json.load(sys.stdin); print('OK' if 'files' in d else 'FAIL')"</automated>
</verify>
<done>API endpoints return valid JSON, file list shows files, delete removes files</done>
</task>
<task type="auto">
<name>task 2: Create files.html template</name>
<files>src/templates/files.html</files>
<action>
Create src/templates/files.html with:
1. Main navigation menu (reuse from other templates)
2. Page title: "Flash Memory Files"
3. File list display:
- Table or card layout showing: filename, size, date
- "View" button for each file (opens modal)
- "Delete" button with confirmation
4. Modal for viewing file contents:
- File name header
- Scrollable content area (pre/code block)
- Close button
5. JavaScript:
- Fetch file list on page load
- Handle view button click -> fetch file content
- Handle delete button click -> confirm -> delete -> refresh list
- Use Bootstrap 5.3 + Bootstrap Icons (like existing templates)
- Touch-optimized 48px+ buttons
Base on the existing templates (settings.html, calibration.html) for:
- Bootstrap 5.3 structure
- Navigation menu
- CSS variables and touch targets
- Socket.IO integration (optional)
</action>
<verify>
<automated>curl -s http://localhost:8080/files | grep -q "Flash Memory" && echo "OK" || echo "FAIL"</automated>
</verify>
<done>Files page loads, shows file list, view modal works, delete works</done>
</task>
<task type="auto">
<name>task 3: Add /files route to Flask app</name>
<files>src/app.py</files>
<action>
Add the route for the files page in src/app.py:
```python
@app.route('/files')
def files():
"""File management page"""
return render_template('files.html',
page='files',
version=SOFTWARE_VERSION,
build_date=BUILD_DATE)
```
Add this route near the other page routes (index, settings, calibration).
</action>
<verify>
<automated>curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/files</automated>
</verify>
<done>Route returns 200 OK</done>
</task>
<task type="auto">
<name>task 4: Add Files menu item to navigation</name>
<files>src/templates/dashboard.html, src/templates/settings.html, src/templates/calibration.html</files>
<action>
Add "Files" navigation button to main menu in:
- dashboard.html
- settings.html
- calibration.html
The navigation should link to /files route. Follow the existing nav structure with nav-btn class.
Example (add after calibration button):
```html
<a href="/files" class="nav-btn btn btn-outline-primary">
<i class="bi bi-folder"></i> Files
</a>
```
</action>
<verify>
<automated>grep -l "Files" src/templates/*.html | wc -l</automated>
</verify>
<done>All pages have Files link in navigation</done>
</task>
</tasks>
<verification>
- API endpoints return valid JSON
- Files page loads with file list
- View modal shows file contents
- Delete removes files from storage
- Navigation works across all pages
</verification>
<success_criteria>
User can view list of CSV files, view file contents, and delete files from flash memory via web UI
</success_criteria>
<output>
After completion, create `.planning/phases/02-data-persistence-file-management/02-01-SUMMARY.md`
</output>

195
.planning/phases/02-data-persistence-file-management/02-02-PLAN.md Normal file
View File

@@ -0,0 +1,195 @@
---
phase: 02-data-persistence-file-management
plan: "02"
type: execute
wave: 1
depends_on: []
files_modified:
- src/app.py
- src/templates/files.html
autonomous: true
requirements:
- FILE-03
- FILE-05
must_haves:
truths:
- "User can navigate file list (scroll up/down) to find specific files"
- "User can generate tidEDA formatted files for data transmission"
artifacts:
- path: "src/app.py"
provides: "tidEDA export API endpoint"
exports: ["POST /api/files/export"]
- path: "src/templates/files.html"
provides: "Scroll navigation and export functionality"
key_links:
- from: "src/templates/files.html"
to: "/api/files/export"
via: "fetch POST"
pattern: "fetch.*api/files/export"
---
<objective>
Add scroll navigation to file list and tidEDA export functionality.
</objective>
<context>
@./.planning/phases/02-data-persistence-file-management/02-CONTEXT.md
@./.planning/phases/02-data-persistence-file-management/02-RESEARCH.md
@./.planning/phases/02-data-persistence-file-management/02-01-PLAN.md
@./src/app.py
</context>
<tasks>
<task type="auto">
<name>task 1: Add scroll navigation to file list</name>
<files>src/templates/files.html</files>
<action>
Enhance src/templates/files.html with scroll navigation:
1. If file list exceeds viewport:
- Enable native scroll (overflow-y: auto)
- Or add pagination controls (Previous/Next buttons)
2. For touch-optimized scrolling on 7" display:
- Ensure smooth scrolling with -webkit-overflow-scrolling: touch
- Minimum touch target size: 48px for all interactive elements
3. Add file count display:
- "Showing X of Y files"
- Update after scroll/filter
4. Optional enhancements:
- Sort by date (newest first)
- Filter by filename search
- These are "OpenCode's Discretion" - implement if straightforward
The key requirement is FILE-03: User can navigate file list (scroll up/down) to find specific files.
</action>
<verify>
<automated>Test by loading files page with multiple files and verifying scroll works</automated>
</verify>
<done>User can scroll through file list to find specific files</done>
</task>
<task type="auto">
<name>task 2: Add tidEDA export API endpoint</name>
<files>src/app.py</files>
<action>
Add POST /api/files/export endpoint to src/app.py:
```python
@app.route('/api/files/export', methods=['POST'])
def api_files_export():
"""Generate tidEDA formatted export from CSV file"""
data = request.get_json() or {}
filename = data.get('filename')
if not filename:
return jsonify({"error": "filename required"}), 400
# Validate filename (security: prevent path traversal)
if '..' in filename or '/' in filename or '\\' in filename:
return jsonify({"error": "invalid filename"}), 400
# Ensure .csv extension
if not filename.endswith('.csv'):
filename = filename + '.csv'
# Read the CSV file
file_path = os.path.join(LOGGER_DIR, filename)
if not os.path.exists(file_path):
return jsonify({"error": "file not found"}), 404
# Generate tidEDA format
# Structure:
# $STATION,{station_id}
# $DATETIME,{timestamp}
# $DATA
# {original_csv_content}
# $END
settings = load_settings()
station_id = settings.get("station", {}).get("id", "UNKNOWN")
with open(file_path, 'r') as f:
csv_content = f.read()
tideda_content = f"$STATION,{station_id}\n"
tideda_content += f"$DATETIME,{datetime.now().isoformat()}\n"
tideda_content += "$DATA\n"
tideda_content += csv_content
tideda_content += "$END\n"
return jsonify({
"filename": filename.replace('.csv', '.tideda'),
"content": tideda_content,
"format": "tidEDA v1.0"
})
```
Add LOGGER_DIR constant at top of file:
```python
LOGGER_DIR = os.path.join(os.path.dirname(__file__), 'data', 'logger')
os.makedirs(LOGGER_DIR, exist_ok=True)
```
</action>
<verify>
<automated>curl -s -X POST -H "Content-Type: application/json" -d '{"filename":"test.csv"}' http://localhost:8080/api/files/export</automated>
</verify>
<done>API returns tidEDA formatted content with $STATION, $DATETIME, $DATA, $END structure</done>
</task>
<task type="auto">
<name>task 3: Add export button to files UI</name>
<files>src/templates/files.html</files>
<action>
Enhance src/templates/files.html with export functionality:
1. Add "Export" button next to each file or as a bulk action:
- Button with download icon
- Calls POST /api/files/export
2. Handle export response:
- Receive tidEDA formatted content
- Trigger browser download as .tideda file
3. Add JavaScript download helper:
```javascript
function downloadFile(filename, content) {
const blob = new Blob([content], {type: 'text/plain'});
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = filename;
a.click();
URL.revokeObjectURL(url);
}
```
4. Add "Generate tidEDA" button at top of page for exporting selected/all files
This satisfies FILE-05: User can generate tidEDA formatted files for data transmission.
</action>
<verify>
<automated>Verify export button exists in UI and triggers download</automated>
</verify>
<done>User can generate and download tidEDA formatted files</done>
</task>
</tasks>
<verification>
- Scroll navigation works smoothly
- tidEDA export generates correct format
- Download works in browser
</verification>
<success_criteria>
User can navigate file list with scrolling and generate tidEDA formatted files for transmission
</success_criteria>
<output>
After completion, create `.planning/phases/02-data-persistence-file-management/02-02-SUMMARY.md`
</output>

163
.planning/phases/02-data-persistence-file-management/02-RESEARCH.md Normal file
View File

@@ -0,0 +1,163 @@
# Phase 2: Data Persistence & File Management - Research
**Researched:** 2026-03-12
**Phase:** 02-data-persistence-file-management
## Domain Overview
Phase 2 implements CSV data logging to flash memory with file management UI. This builds on Phase 1's Flask backend by adding:
- File system operations (list, read, delete)
- CSV file generation/management
- tidEDA formatted export
- File management UI page
## Requirements Analysis
| Requirement | Description | Implementation Approach |
|-------------|-------------|------------------------|
| FILE-01 | List CSV files in flash memory | Flask route returning JSON array of file metadata |
| FILE-02 | View CSV file contents | Flask route serving file content with pagination |
| FILE-03 | Navigate file list (scroll) | Frontend JavaScript with pagination or virtual scroll |
| FILE-04 | Delete CSV files | Flask DELETE endpoint with confirmation |
| FILE-05 | Generate tidEDA formatted files | Backend transformation endpoint |
| BACK-04 | File management API | All above endpoints combined |
## Technical Implementation
### Storage Location
- **Path:** `/myvscada/logger` (per CONTEXT.md)
- **Note:** On Pi, this may need to be relative to project or use a configurable path
### File Naming Convention
- Format: `SP{station_id}_{timestamp}.csv`
- Example: `SP001_20260312_143022.csv`
### API Endpoints Needed
```
GET /api/files - List all CSV files with metadata
GET /api/files/<filename> - Get file contents
DELETE /api/files/<filename> - Delete a file
POST /api/files/export - Generate tidEDA format from CSV
```
### Response Formats
**GET /api/files**
```json
{
"files": [
{
"name": "SP001_20260312_143022.csv",
"size": 1234,
"created": "2026-03-12T14:30:22Z"
}
],
"total": 10
}
```
**GET /api/files/<filename>**
```json
{
"name": "SP001_20260312_143022.csv",
"content": "timestamp,rainfall,solar,battery\n...",
"lines": 50
}
```
### tidEDA Format
Based on standard environmental data transmission formats. The tidEDA format should contain:
- Station identification
- Timestamp
- Sensor readings (rainfall, voltage)
- Data quality indicators
**Suggested structure:**
```
$STATION,SP001
$DATETIME,2026-03-12T14:30:00Z
$DATA
timestamp,rainfall_mm,solar_v,battery_v
2026-03-12T14:30:00,0.2,12.4,12.8
...
$END
```
## Frontend Implementation
### New Page: Files (`/files`)
- Bootstrap 5.3 template (reuse existing patterns)
- File list with:
- Filename, size, date
- View button
- Delete button with confirmation
- Export to tidEDA button
- Scroll navigation (per FILE-03)
- Touch-optimized 48px+ targets
### UI Components
- File list table/card layout
- Modal for file content viewing
- Confirmation dialogs for delete
- Loading states during API calls
## Dependencies
No new Python packages required beyond existing Flask:
- `os`, `datetime`, `json` (stdlib)
- Flask already installed
## Integration Points
1. **Main Navigation:** Add "Files" menu item
2. **Settings API reuse:** Get station ID from settings
3. **Socket.IO:** Optional - file list refresh on changes
4. **Template base:** Extend existing Bootstrap 5.3 layout
## Validation Architecture
**Success Criteria Validation:**
1. User can view list of CSV files stored in flash memory
- `GET /api/files` returns file array
- Files page renders list correctly
2. User can navigate file list (scroll up/down)
- Frontend implements scroll or pagination
- Touch targets work on 7" display
3. User can view contents of selected CSV files
- `GET /api/files/<filename>` returns content
- Modal/displays shows formatted content
4. User can delete unwanted CSV files from flash memory
- `DELETE /api/files/<filename>` removes file
- UI updates after deletion
5. User can generate tidEDA formatted files
- `POST /api/files/export` generates format
- Returns downloadable content
## Risks & Mitigations
| Risk | Mitigation |
|------|------------|
| File path not accessible | Use configurable path, default to project data dir |
| Large files crash display | Paginate content, limit preview lines |
| Concurrent file access | Handle file locked errors gracefully |
| tidEDA format incompatibility | Define format clearly, add version field |
## Implementation Order
1. Backend API routes first (file operations)
2. Frontend file list page
3. File viewing modal
4. Delete functionality with confirmation
5. tidEDA export endpoint
6. Integration testing
---
*Research complete - ready for planning*

89
.planning/phases/02-data-persistence-file-management/02-VALIDATION.md Normal file
View File

@@ -0,0 +1,89 @@
---
phase: 02
slug: data-persistence-file-management
status: draft
nyquist_compliant: false
wave_0_complete: false
created: 2026-03-12
---
# Phase 2 — Validation Strategy
> Per-phase validation contract for feedback sampling during execution.
---
## Test Infrastructure
| Property | Value |
|----------|-------|
| **Framework** | pytest (Python) + Flask test client |
| **Config file** | tests/ (Wave 0 creates) |
| **Quick run command** | `pytest tests/test_api.py -v` |
| **Full suite command** | `pytest tests/ -v` |
| **Estimated runtime** | ~15 seconds |
---
## Sampling Rate
- **After every task commit:** Run `pytest tests/test_api.py -v`
- **After every plan wave:** Run `pytest tests/ -v`
- **Before `/gsd-verify-work`:** Full suite must be green
- **Max feedback latency:** 30 seconds
---
## Per-task Verification Map
| task ID | Plan | Wave | Requirement | Test Type | Automated Command | File Exists | Status |
|---------|------|------|-------------|-----------|-------------------|-------------|--------|
| 02-01-01 | 01 | 1 | BACK-04 | API | `pytest tests/test_api.py::test_list_files -v` | ⚠️ W0 | ⬜ pending |
| 02-01-02 | 01 | 1 | FILE-01 | API | `pytest tests/test_api.py::test_get_file_contents -v` | ⚠️ W0 | ⬜ pending |
| 02-01-03 | 01 | 1 | FILE-04 | API | `pytest tests/test_api.py::test_delete_file -v` | ⚠️ W0 | ⬜ pending |
| 02-02-01 | 02 | 1 | FILE-02 | UI | Browser test or manual | - | ⬜ pending |
| 02-02-02 | 02 | 1 | FILE-03 | UI | Manual (touch scroll) | - | ⬜ pending |
| 02-02-03 | 02 | 2 | FILE-05 | API | `pytest tests/test_api.py::test_export_tideda -v` | ⚠️ W0 | ⬜ pending |
*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*
---
## Wave 0 Requirements
- [ ] `tests/conftest.py` — Flask test client fixture
- [ ] `tests/test_api.py` — API endpoint stubs for:
- `test_list_files`
- `test_get_file_contents`
- `test_delete_file`
- `test_export_tideda`
- [ ] `tests/__init__.py` — Package marker
*Existing Flask test client pattern from src/app.py can be reused.*
---
## Manual-Only Verifications
| Behavior | Requirement | Why Manual | Test Instructions |
|----------|-------------|------------|-------------------|
| File list scroll navigation | FILE-03 | Touch UI interaction | Visit /files, scroll through list |
| File content display formatting | FILE-02 | Visual verification | Click file, verify content renders |
| Delete confirmation dialog | FILE-04 | UX flow | Click delete, verify confirmation |
| tidEDA export download | FILE-05 | File download | Click export, verify file downloads |
*These require browser/UI interaction that automated tests don't reliably capture.*
---
## Validation Sign-Off
- [ ] All tasks have `<automated>` verify or Wave 0 dependencies
- [ ] Sampling continuity: no 3 consecutive tasks without automated verify
- [ ] Wave 0 covers all MISSING references
- [ ] No watch-mode flags
- [ ] Feedback latency < 30s
- [ ] `nyquist_compliant: true` set in frontmatter
**Approval:** pending