rtu_v5/.planning/research/SUMMARY.md

Project Research Summary

Project: TCKRTUIYO - Rainfall Station RTU Web Interface Domain: Embedded IoT/Web Monitoring Systems Researched: 2026-03-12 Confidence: HIGH

Executive Summary

This is a Raspberry Pi-based web monitoring interface for rainfall telemetry. The system reads tipping bucket sensors, displays real-time data on a 7-inch touchscreen (kiosk mode), and transmits CSV files to a remote myvscada server. The target hardware is Pi Zero 2 W with ~512MB RAM — significantly constrained compared to typical web servers.

Recommended approach: Build a vertical slice first (sensor → API → basic UI), then layer features. Use Flask with Flask-SocketIO for real-time updates, Bootstrap for responsive touchscreen layouts, and Chart.js for future historical graphs. The architecture follows a clear three-tier pattern: Web Layer (dual ports 8080/9090), API Layer (blueprint-based Flask routes), and Service Layer (Sensor, Data, Network, Config services).

Key risks to mitigate:

1. Performance on Pi Zero 2 W — Flask dev server will be unusable; must use eventlet workers and optimize imports
2. SD card corruption — Continuous CSV writes destroy SD cards; use SQLite or buffered writes with tmpfs
3. Chromium kiosk instability — Updates break kiosk mode; pin version and add watchdog script
4. Silent data transmission failures — FTP/SFTP failures must be visible in UI, not just logged

---

Key Findings

Recommended Stack

Core technologies:

• Flask 3.1.0 — Lightweight web framework, ~15MB RAM vs Django's 80MB+
• Flask-SocketIO 5.6.0 — Real-time server-push for live sensor updates
• Chart.js 4.4.x — Canvas-based charting, GPU-accelerated on Pi
• Bootstrap 5.3.x — Touchscreen-friendly grid system, ~200KB
• Python 3.11+ — Raspberry Pi OS Bookworm ships with 3.11

Supporting libraries:

• eventlet 0.40+ — Required for Flask-SocketIO production (not gunicorn threading)
• paramiko 3.5+ — SFTP/SCP file transmission
• psutil 6.1.0 — System health monitoring
• pyserial 3.5 — Sensor communication

What NOT to use: Django (too heavy), FastAPI (unnecessary Pydantic overhead), React/Vue/Angular (build step + memory intensive), Node.js (breaks Python sensor stack), InfluxDB+Grafana (too heavy for Pi Zero).

Expected Features

Must have (table stakes):

• Real-time rainfall display (Today, Hourly, MAR Acc, Yearly Acc)
• Voltage monitoring (battery and solar)
• Date/time and station identification
• Communication status (RSSI, connection state)
• Main menu navigation for 7" touchscreen
• Settings configuration (thresholds, network, calibration)
• Calibration view (live sensor readings)
• File management (CSV navigation/deletion)

Should have (differentiators):

• Historical data graphs (Chart.js on Pi Zero is viable with limited data points)
• Configurable alarm thresholds with visual indicators
• Touch-optimized UI (minimum 48px touch targets, 64px for primary)
• Dual-mode display (single codebase serves kiosk 1024x600 and remote HD)
• Real-time Socket.IO push (live updates without page refresh)
• CSV transmission status (last sync time, pending count)

Defer (v2+):

• Multi-station dashboard (server-side aggregation)
• Cloud data storage (direct CSV transmission is the model)
• Video streaming (not in spec)
• SMS/email alerts (future server-side feature)
• Data export from browser (low priority)

Architecture Approach

The system follows a three-tier architecture with clear component boundaries:

1. Web Layer (Dual Port) — Port 8080 for kiosk (no auth, fixed 1024x600), Port 9090 for remote (auth required, responsive HD)
2. API Layer — Flask blueprints for routes: /api/status, /api/sensors, /api/settings, /api/files, /api/calibration
3. Service Layer — SensorService (GPIO/ADC polling), DataLogger (CSV write), NetworkService (FTP/SFTP), ConfigService (JSON files)

Data flow: Timer → Sensor Service reads hardware → Update in-memory state → DataLogger writes CSV → API returns JSON → Web layer displays

Build order: Foundation (sensor→API→basic UI) → Local Interface (settings, calibration) → Data Persistence (CSV logging, transmission) → Remote Access (auth, responsive)

Critical Pitfalls

1. Flask performance on Pi Zero — Dev server takes 60+ seconds to start, blocks on every request. Avoid: Use eventlet workers, pre-compile templates, cache static assets, implement SSE instead of polling.

2. Chromium kiosk instability — Updates replace version, Wayland/X11 differences break flags, session restore prompts interrupt. Avoid: Pin Chromium version (apt-mark hold), use X11 instead of Wayland, add watchdog script, test after every system update.

3. SD card corruption — Continuous CSV writes cause wear, power loss corrupts filesystem. Avoid: Mount /var/log and /tmp as tmpfs, write to memory buffer and flush every 5 minutes, use SQLite with WAL mode, implement shutdown button, disable swap.

4. Touchscreen unresponsiveness — 7-inch display has 33Hz polling rate, causing lag. Avoid: Set gpu_mem=128, design 64px touch targets, provide immediate visual feedback, avoid rapid-fire interactions.

5. Network transmission failures silent — FTP/SFTP fails but only logged, not surfaced to UI. Avoid: Implement retry queue with exponential backoff, verify server receipt, show transmission status prominently, dead-letter alert after N failures.

6. Stale data without notification — Dashboard shows old readings with no indication. Avoid: Always display "last updated" timestamp, use Socket.IO for live updates, add backend health check, show warning if sensor reader hasn't updated in X minutes.

---

Implications for Roadmap

Based on research, suggested phase structure:

Phase 1: Foundation & Kiosk UI

Rationale: Must prove sensor-to-display works before adding persistence or networking. This is a vertical slice — each layer works end-to-end before adding features.

Delivers:

• Hardware interface (GPIO/ADC)
• Sensor service with live readings
• Basic API (status, sensors endpoints)
• Minimal dashboard UI (rainfall, voltage, time, station ID)
• Port 8080 kiosk server with Chromium autostart

Addresses: FEATURES table stakes — dashboard display, voltage monitoring, date/time, station ID, main menu

Avoids: Pitfall 1 (Flask performance) — use eventlet from start, not as afterthought; Pitfall 4 (touchscreen lag) — test touch responsiveness early; Pitfall 6 (stale data) — show timestamps from day one

Research flag: Complex integration with sensor hardware — may need /gsd-research-phase for GPIO/ADC specifics if not already available from existing RTU.

Phase 2: Data Persistence

Rationale: CSV logging is the primary data workflow. Must establish proper storage before transmission. Avoids Pitfall 3 (SD card corruption) by designing correctly from the start.

Delivers:

• Data logger service with buffered writes
• SQLite or structured CSV storage
• tmpfs mounts for logs/tmp
• Settings UI and configuration persistence
• Calibration view UI

Addresses: FEATURES settings configuration, calibration view

Avoids: Pitfall 3 (SD card corruption) — implement tmpfs, buffered writes, proper shutdown

Research flag: Standard patterns — SQLite and CSV logging are well-documented. Skip deep research.

Phase 3: Network Transmission

Rationale: Data must reach myvscada server. Requires proven persistence layer first. Avoids Pitfall 5 (silent transmission failures) by building verification upfront.

Delivers:

• Network service with FTP/SFTP/SCP
• Transmission queue with retry logic
• Server receipt verification
• Transmission status UI (last sync, pending count)
• File management UI

Addresses: FEATURES file management, transmission status

Avoids: Pitfall 5 (silent failures) — queue with backoff, verification, UI visibility

Research flag: May need deeper research on myvscada server integration specifics (API, expected file format).

Phase 4: Remote Access

Rationale: Full HD remote interface is the "pro" experience. Can reuse most backend. Focus on responsive design and authentication.

Delivers:

• Port 9090 server with authentication
• Responsive templates for Full HD
• Historical graphs (Chart.js)
• Configurable alarm thresholds with visual indicators

Addresses: FEATURES historical graphs, threshold indicators, alarm threshold config, dual-mode display

Research flag: Standard Flask + Bootstrap patterns — skip deep research.

Phase Ordering Rationale

• Vertical slice first: Each layer works end-to-end before adding complexity
• Data before network: Can't transmit what hasn't been stored correctly
• Kiosk before remote: Local display is primary use case; remote is secondary
• Pitfall prevention: Each phase addresses specific pitfalls identified in research

Research Flags

Phases likely needing deeper research during planning:

• Phase 1: Sensor hardware integration — need specifics on existing RTU GPIO/ADC interface
• Phase 3: myvscada server API — format expectations, authentication, transmission protocol

Phases with standard patterns (skip research-phase):

• Phase 2: SQLite/CSV logging is well-documented
• Phase 4: Flask + Bootstrap responsive design is standard

---

Confidence Assessment

Area	Confidence	Notes
Stack	HIGH	Verified with multiple GitHub projects using exact Flask+SocketIO+Chart.js pattern on Pi
Features	HIGH	Based on competitive analysis of Rainwise, Weather Display, Davis WeatherLink
Architecture	HIGH	Matches proven IoT/SCADA three-tier patterns, clear component boundaries
Pitfalls	HIGH	Documented from Raspberry Pi forums, Stack Exchange, community reports

Overall confidence: HIGH

Gaps to Address

• Sensor hardware specifics: Research assumes GPIO/ADC interface available. Need to verify existing RTU sensor connection method.
• myvscada integration: Research assumes FTP/SFTP transmission but need to verify server requirements (path, authentication, file format).
• Touchscreen calibration: 33Hz polling rate is documented but may need tuning per unit.

---

Sources

Primary (HIGH confidence)

• GitHub g1forfun/Pi-Monitor — Active project (Feb 2026), Flask+SocketIO+Chart.js pattern
• Cytron.io tutorial — "Create a Real-Time Raspberry Pi Dashboard" (Dec 2025)
• Flask-SocketIO PyPI — Verified 5.6.0 release (Dec 2025), Python 3.8+ compatibility
• piwheels.org — Verified packages available for Raspberry Pi OS

Secondary (MEDIUM confidence)

• Raspberry Pi Forums — Kiosk mode issues, performance discussions
• Raspberry Pi Stack Exchange — Flask on Pi Zero W performance
• Multiple community tutorials — Confirms Flask is de facto standard

Tertiary (LOW confidence)

• GitHub Issue #3777 (raspberrypi/linux) — 7" touchscreen 33Hz polling (needs verification)
• Community reports on SD card corruption — patterns consistent but may vary by card quality

---

Research completed: 2026-03-12 Ready for roadmap: yes