219 lines
11 KiB
Markdown
219 lines
11 KiB
Markdown
# Project Research Summary
|
|
|
|
**Project:** TCKRTUIYO - Rainfall Station RTU Web Interface
|
|
**Domain:** Embedded IoT/Web Monitoring Systems
|
|
**Researched:** 2026-03-12
|
|
**Confidence:** HIGH
|
|
|
|
## Executive Summary
|
|
|
|
This is a **Raspberry Pi-based web monitoring interface for rainfall telemetry**. The system reads tipping bucket sensors, displays real-time data on a 7-inch touchscreen (kiosk mode), and transmits CSV files to a remote myvscada server. The target hardware is Pi Zero 2 W with ~512MB RAM — significantly constrained compared to typical web servers.
|
|
|
|
**Recommended approach:** Build a vertical slice first (sensor → API → basic UI), then layer features. Use Flask with Flask-SocketIO for real-time updates, Bootstrap for responsive touchscreen layouts, and Chart.js for future historical graphs. The architecture follows a clear three-tier pattern: Web Layer (dual ports 8080/9090), API Layer (blueprint-based Flask routes), and Service Layer (Sensor, Data, Network, Config services).
|
|
|
|
**Key risks to mitigate:**
|
|
1. **Performance on Pi Zero 2 W** — Flask dev server will be unusable; must use eventlet workers and optimize imports
|
|
2. **SD card corruption** — Continuous CSV writes destroy SD cards; use SQLite or buffered writes with tmpfs
|
|
3. **Chromium kiosk instability** — Updates break kiosk mode; pin version and add watchdog script
|
|
4. **Silent data transmission failures** — FTP/SFTP failures must be visible in UI, not just logged
|
|
|
|
---
|
|
|
|
## Key Findings
|
|
|
|
### Recommended Stack
|
|
|
|
**Core technologies:**
|
|
- **Flask 3.1.0** — Lightweight web framework, ~15MB RAM vs Django's 80MB+
|
|
- **Flask-SocketIO 5.6.0** — Real-time server-push for live sensor updates
|
|
- **Chart.js 4.4.x** — Canvas-based charting, GPU-accelerated on Pi
|
|
- **Bootstrap 5.3.x** — Touchscreen-friendly grid system, ~200KB
|
|
- **Python 3.11+** — Raspberry Pi OS Bookworm ships with 3.11
|
|
|
|
**Supporting libraries:**
|
|
- **eventlet 0.40+** — Required for Flask-SocketIO production (not gunicorn threading)
|
|
- **paramiko 3.5+** — SFTP/SCP file transmission
|
|
- **psutil 6.1.0** — System health monitoring
|
|
- **pyserial 3.5** — Sensor communication
|
|
|
|
**What NOT to use:** Django (too heavy), FastAPI (unnecessary Pydantic overhead), React/Vue/Angular (build step + memory intensive), Node.js (breaks Python sensor stack), InfluxDB+Grafana (too heavy for Pi Zero).
|
|
|
|
### Expected Features
|
|
|
|
**Must have (table stakes):**
|
|
- Real-time rainfall display (Today, Hourly, MAR Acc, Yearly Acc)
|
|
- Voltage monitoring (battery and solar)
|
|
- Date/time and station identification
|
|
- Communication status (RSSI, connection state)
|
|
- Main menu navigation for 7" touchscreen
|
|
- Settings configuration (thresholds, network, calibration)
|
|
- Calibration view (live sensor readings)
|
|
- File management (CSV navigation/deletion)
|
|
|
|
**Should have (differentiators):**
|
|
- Historical data graphs (Chart.js on Pi Zero is viable with limited data points)
|
|
- Configurable alarm thresholds with visual indicators
|
|
- Touch-optimized UI (minimum 48px touch targets, 64px for primary)
|
|
- Dual-mode display (single codebase serves kiosk 1024x600 and remote HD)
|
|
- Real-time Socket.IO push (live updates without page refresh)
|
|
- CSV transmission status (last sync time, pending count)
|
|
|
|
**Defer (v2+):**
|
|
- Multi-station dashboard (server-side aggregation)
|
|
- Cloud data storage (direct CSV transmission is the model)
|
|
- Video streaming (not in spec)
|
|
- SMS/email alerts (future server-side feature)
|
|
- Data export from browser (low priority)
|
|
|
|
### Architecture Approach
|
|
|
|
The system follows a **three-tier architecture** with clear component boundaries:
|
|
|
|
1. **Web Layer (Dual Port)** — Port 8080 for kiosk (no auth, fixed 1024x600), Port 9090 for remote (auth required, responsive HD)
|
|
2. **API Layer** — Flask blueprints for routes: `/api/status`, `/api/sensors`, `/api/settings`, `/api/files`, `/api/calibration`
|
|
3. **Service Layer** — SensorService (GPIO/ADC polling), DataLogger (CSV write), NetworkService (FTP/SFTP), ConfigService (JSON files)
|
|
|
|
**Data flow:** Timer → Sensor Service reads hardware → Update in-memory state → DataLogger writes CSV → API returns JSON → Web layer displays
|
|
|
|
**Build order:** Foundation (sensor→API→basic UI) → Local Interface (settings, calibration) → Data Persistence (CSV logging, transmission) → Remote Access (auth, responsive)
|
|
|
|
### Critical Pitfalls
|
|
|
|
1. **Flask performance on Pi Zero** — Dev server takes 60+ seconds to start, blocks on every request. **Avoid:** Use eventlet workers, pre-compile templates, cache static assets, implement SSE instead of polling.
|
|
|
|
2. **Chromium kiosk instability** — Updates replace version, Wayland/X11 differences break flags, session restore prompts interrupt. **Avoid:** Pin Chromium version (`apt-mark hold`), use X11 instead of Wayland, add watchdog script, test after every system update.
|
|
|
|
3. **SD card corruption** — Continuous CSV writes cause wear, power loss corrupts filesystem. **Avoid:** Mount `/var/log` and `/tmp` as tmpfs, write to memory buffer and flush every 5 minutes, use SQLite with WAL mode, implement shutdown button, disable swap.
|
|
|
|
4. **Touchscreen unresponsiveness** — 7-inch display has 33Hz polling rate, causing lag. **Avoid:** Set `gpu_mem=128`, design 64px touch targets, provide immediate visual feedback, avoid rapid-fire interactions.
|
|
|
|
5. **Network transmission failures silent** — FTP/SFTP fails but only logged, not surfaced to UI. **Avoid:** Implement retry queue with exponential backoff, verify server receipt, show transmission status prominently, dead-letter alert after N failures.
|
|
|
|
6. **Stale data without notification** — Dashboard shows old readings with no indication. **Avoid:** Always display "last updated" timestamp, use Socket.IO for live updates, add backend health check, show warning if sensor reader hasn't updated in X minutes.
|
|
|
|
---
|
|
|
|
## Implications for Roadmap
|
|
|
|
Based on research, suggested phase structure:
|
|
|
|
### Phase 1: Foundation & Kiosk UI
|
|
**Rationale:** Must prove sensor-to-display works before adding persistence or networking. This is a vertical slice — each layer works end-to-end before adding features.
|
|
|
|
**Delivers:**
|
|
- Hardware interface (GPIO/ADC)
|
|
- Sensor service with live readings
|
|
- Basic API (status, sensors endpoints)
|
|
- Minimal dashboard UI (rainfall, voltage, time, station ID)
|
|
- Port 8080 kiosk server with Chromium autostart
|
|
|
|
**Addresses:** FEATURES table stakes — dashboard display, voltage monitoring, date/time, station ID, main menu
|
|
|
|
**Avoids:** Pitfall 1 (Flask performance) — use eventlet from start, not as afterthought; Pitfall 4 (touchscreen lag) — test touch responsiveness early; Pitfall 6 (stale data) — show timestamps from day one
|
|
|
|
**Research flag:** Complex integration with sensor hardware — may need `/gsd-research-phase` for GPIO/ADC specifics if not already available from existing RTU.
|
|
|
|
### Phase 2: Data Persistence
|
|
**Rationale:** CSV logging is the primary data workflow. Must establish proper storage before transmission. Avoids Pitfall 3 (SD card corruption) by designing correctly from the start.
|
|
|
|
**Delivers:**
|
|
- Data logger service with buffered writes
|
|
- SQLite or structured CSV storage
|
|
- tmpfs mounts for logs/tmp
|
|
- Settings UI and configuration persistence
|
|
- Calibration view UI
|
|
|
|
**Addresses:** FEATURES settings configuration, calibration view
|
|
|
|
**Avoids:** Pitfall 3 (SD card corruption) — implement tmpfs, buffered writes, proper shutdown
|
|
|
|
**Research flag:** Standard patterns — SQLite and CSV logging are well-documented. Skip deep research.
|
|
|
|
### Phase 3: Network Transmission
|
|
**Rationale:** Data must reach myvscada server. Requires proven persistence layer first. Avoids Pitfall 5 (silent transmission failures) by building verification upfront.
|
|
|
|
**Delivers:**
|
|
- Network service with FTP/SFTP/SCP
|
|
- Transmission queue with retry logic
|
|
- Server receipt verification
|
|
- Transmission status UI (last sync, pending count)
|
|
- File management UI
|
|
|
|
**Addresses:** FEATURES file management, transmission status
|
|
|
|
**Avoids:** Pitfall 5 (silent failures) — queue with backoff, verification, UI visibility
|
|
|
|
**Research flag:** May need deeper research on myvscada server integration specifics (API, expected file format).
|
|
|
|
### Phase 4: Remote Access
|
|
**Rationale:** Full HD remote interface is the "pro" experience. Can reuse most backend. Focus on responsive design and authentication.
|
|
|
|
**Delivers:**
|
|
- Port 9090 server with authentication
|
|
- Responsive templates for Full HD
|
|
- Historical graphs (Chart.js)
|
|
- Configurable alarm thresholds with visual indicators
|
|
|
|
**Addresses:** FEATURES historical graphs, threshold indicators, alarm threshold config, dual-mode display
|
|
|
|
**Research flag:** Standard Flask + Bootstrap patterns — skip deep research.
|
|
|
|
### Phase Ordering Rationale
|
|
|
|
- **Vertical slice first:** Each layer works end-to-end before adding complexity
|
|
- **Data before network:** Can't transmit what hasn't been stored correctly
|
|
- **Kiosk before remote:** Local display is primary use case; remote is secondary
|
|
- **Pitfall prevention:** Each phase addresses specific pitfalls identified in research
|
|
|
|
### Research Flags
|
|
|
|
Phases likely needing deeper research during planning:
|
|
- **Phase 1:** Sensor hardware integration — need specifics on existing RTU GPIO/ADC interface
|
|
- **Phase 3:** myvscada server API — format expectations, authentication, transmission protocol
|
|
|
|
Phases with standard patterns (skip research-phase):
|
|
- **Phase 2:** SQLite/CSV logging is well-documented
|
|
- **Phase 4:** Flask + Bootstrap responsive design is standard
|
|
|
|
---
|
|
|
|
## Confidence Assessment
|
|
|
|
| Area | Confidence | Notes |
|
|
|------|------------|-------|
|
|
| Stack | HIGH | Verified with multiple GitHub projects using exact Flask+SocketIO+Chart.js pattern on Pi |
|
|
| Features | HIGH | Based on competitive analysis of Rainwise, Weather Display, Davis WeatherLink |
|
|
| Architecture | HIGH | Matches proven IoT/SCADA three-tier patterns, clear component boundaries |
|
|
| Pitfalls | HIGH | Documented from Raspberry Pi forums, Stack Exchange, community reports |
|
|
|
|
**Overall confidence:** HIGH
|
|
|
|
### Gaps to Address
|
|
|
|
- **Sensor hardware specifics:** Research assumes GPIO/ADC interface available. Need to verify existing RTU sensor connection method.
|
|
- **myvscada integration:** Research assumes FTP/SFTP transmission but need to verify server requirements (path, authentication, file format).
|
|
- **Touchscreen calibration:** 33Hz polling rate is documented but may need tuning per unit.
|
|
|
|
---
|
|
|
|
## Sources
|
|
|
|
### Primary (HIGH confidence)
|
|
- GitHub g1forfun/Pi-Monitor — Active project (Feb 2026), Flask+SocketIO+Chart.js pattern
|
|
- Cytron.io tutorial — "Create a Real-Time Raspberry Pi Dashboard" (Dec 2025)
|
|
- Flask-SocketIO PyPI — Verified 5.6.0 release (Dec 2025), Python 3.8+ compatibility
|
|
- piwheels.org — Verified packages available for Raspberry Pi OS
|
|
|
|
### Secondary (MEDIUM confidence)
|
|
- Raspberry Pi Forums — Kiosk mode issues, performance discussions
|
|
- Raspberry Pi Stack Exchange — Flask on Pi Zero W performance
|
|
- Multiple community tutorials — Confirms Flask is de facto standard
|
|
|
|
### Tertiary (LOW confidence)
|
|
- GitHub Issue #3777 (raspberrypi/linux) — 7" touchscreen 33Hz polling (needs verification)
|
|
- Community reports on SD card corruption — patterns consistent but may vary by card quality
|
|
|
|
---
|
|
|
|
*Research completed: 2026-03-12*
|
|
*Ready for roadmap: yes* |