diff --git a/AUDIT_REPORT.md b/AUDIT_REPORT.md new file mode 100644 index 00000000..3922983e --- /dev/null +++ b/AUDIT_REPORT.md @@ -0,0 +1,312 @@ +# AUDIT REPORT — SIDES (tckdev) + +**Date**: 2026-05-20 +**Auditor**: Automated Code Review +**Scope**: Full brownfield codebase audit + +--- + +## Executive Summary + +SIDES is a flood early warning system built on Laravel 12 + PostgreSQL + Docker. The application is **functional for its purpose** but has significant security vulnerabilities, code quality issues, and architectural concerns that should be addressed before any further development or production deployment. + +**Risk Level: HIGH** + +--- + +## 1. CRITICAL — Security Vulnerabilities + +### 1.1 SQL Injection (HIGH) + +**Multiple controllers** build raw SQL queries with string interpolation instead of parameterized queries. + +**`RainfallController::index()` (line 46):** +```php +$stationCondition = " AND s.stationid = '{$stationFilter}'"; +``` + +**`WaterLevelController::index()` (lines 30, 37):** +```php +$stationCondition = " WHERE s.stationid = '{$stationFilter}' "; +$dateCondition = " AND w.datetime = '{$sqlDate}' "; +``` + +**`RainfallController::index()` (lines 55-106):** +```php +$rainfallData = collect(DB::select(" + SELECT ... CAST('$displayDate' AS timestamp) ... + ... CAST('$dateFilter' as date) - INTERVAL '6 days' ... + $stationCondition +")); +``` + +User input (`$stationFilter`, `$dateFilter`, `$displayDate`, `$sqlDate`) is interpolated directly into SQL strings. An attacker could inject SQL through these parameters. + +**Impact**: Full database compromise (read, modify, delete data) +**Files affected**: `RainfallController.php`, `WaterLevelController.php` +**Fix**: Use parameterized queries with `?` placeholders or named bindings (`:param`) consistently. + +### 1.2 Hardcoded Credentials in Source Code (HIGH) + +**`autoscript/sidesdecode.py` (lines 22-31):** +```python +ftp_server = "myvscada.com" +ftp_username = "tck" +ftp_password = "tck6789" +pg_host = "192.168.0.211" +pg_database = "sides_db" +pg_user = "tck" +pg_password = "projectdev##1" +``` + +**`src/.env` (committed to git):** +``` +MAIL_PASSWORD="ipmu zifw bpmf fsyp" +POSTGRES_PASSWORD="projectdev##1" +PGADMIN_PASSWORD="projectdev##1" +``` + +**`src/storage/app/firebase/sides-b4abb-3604a7cf7584.json`** — Firebase service account key is committed to the repository. + +**Impact**: Anyone with repository access has all credentials +**Fix**: Move to environment variables, add `.env` to `.gitignore`, rotate all exposed credentials + +### 1.3 Default Admin Credentials (HIGH) + +**`DatabaseSeeder.php` and migration `2025_12_11_124201`:** +```php +'name' => 'admin', +'password' => Hash::make('password123'), +``` + +**Impact**: Anyone who discovers this gets admin access +**Fix**: Remove hardcoded credentials, require admin setup on first run, use strong passwords + +### 1.4 API Endpoints Unauthenticated (MEDIUM) + +All API endpoints except `/api/login` have **no authentication**: +```php +Route::get('/station/current', [StationController::class, 'getCurrentData']); +Route::get('/station/rainfall', [StationController::class, 'getRainfallData']); +// ... all publicly accessible +``` + +**`/api/alert`** is publicly accessible — anyone can trigger push notifications: +```php +Route::post('/alert',[AlertController::class,'send']); +``` + +**Impact**: Data exfiltration, unauthorized push notification spam +**Fix**: Add API authentication (sanctum/passport tokens) to all endpoints + +### 1.5 API Login Returns Password Hash (LOW) + +**`Api\AuthController::login()` (line 23):** +```php +$user = DB::select("SELECT id, name, email, access_level, password FROM users ..."); +``` + +While the full `$user` object isn't returned, the query fetches the password hash unnecessarily. + +**Fix**: Remove `password` from the SELECT query. + +--- + +## 2. HIGH — Code Quality Issues + +### 2.1 No Eloquent ORM Usage + +The entire application uses `DB::table()` and `DB::select()` (query builder / raw SQL) instead of Eloquent models. Only the `User` model exists. + +**Impact**: +- No type safety, no attribute casting, no relationship definitions +- Difficulty maintaining and extending the codebase +- No model-level validation or events + +**Fix**: Create Eloquent models for `Station`, `Rainfall`, `WaterLevel`, `Siren`, `Notification` with proper relationships. + +### 2.2 No Database Foreign Keys + +All table relationships (`stationid` references) are maintained at the application level with no database constraints: + +```php +// Migration: no foreign key +$table->string('stationid'); +// vs what it should be: +$table->foreign('stationid')->references('stationid')->on('station'); +``` + +**Impact**: Data integrity cannot be guaranteed at the database level. Orphaned records possible. + +### 2.3 Missing Database Indexes + +No indexes exist on frequently queried columns: +- `rainfall.stationid` — used in every rainfall query +- `rainfall.timestamp` — used for date filtering and max() queries +- `waterlevel.stationid` — used in every water level query +- `waterlevel.datetime` — used for date filtering +- `siren.stationid` — used in every siren query +- `siren.active_time` — used for max() and ordering +- `notification.stationid` — used in every notification query +- `notification.timestamp` — used for date filtering + +**Impact**: Query performance degrades significantly as data grows +**Fix**: Add composite indexes on `(stationid, timestamp)` for data tables + +### 2.4 Duplicate Admin User Creation + +Admin user is created in **both** the DatabaseSeeder AND a migration: + +- `DatabaseSeeder.php` line 22: `User::create(['name' => 'Admin', ...])` +- Migration `2025_12_11_124201`: `DB::table('users')->insert(['name' => 'admin', ...])` + +**Impact**: Running `migrate:fresh --seed` will create two admin users with different names ("Admin" and "admin") +**Fix**: Remove one — keep only the seeder. + +### 2.5 Inconsistent Naming Conventions + +| Issue | Example | +|-------|---------| +| Case inconsistency | `cctvController` (lowercase), `SirenHistory` (PascalCase) | +| Mixed route naming | `sirenhistory` vs `historicalrf` vs `historicalwl` | +| Mixed table naming | `waterlevel.datetime` vs `rainfall.timestamp` (same concept, different column names) | +| Variable typos | `$dateTImeFilter` (capital I), `$dateFilter` | + +### 2.6 No Input Validation on Some Endpoints + +`MapController::getStations()` and `MapController::getCurrentData()` accept no parameters but other controllers accept GET/POST parameters with inconsistent validation. + +### 2.7 Commented-Out Code Blocks + +Multiple large blocks of commented-out code throughout: +- `RainfallController.php` lines 250-263 (alternative graph query) +- `WaterLevelController.php` date conditions +- `sidesdecode.py` file management functions + +--- + +## 3. MEDIUM — Architectural Concerns + +### 3.1 N+1 Query Pattern in MapController + +`MapController::getCurrentData()` uses subqueries in JOINs for latest data: + +```php +->whereRaw('r.timestamp = (SELECT MAX(timestamp) FROM rainfall WHERE stationid = s.stationid)') +``` + +This executes a subquery for every station for every table join (3 subqueries per station). Should use a CTE or window function. + +### 3.2 No Caching + +Real-time data queries (dashboard, map, station listings) run complex SQL on every page load with no caching layer despite data only updating when the Python script runs. + +### 3.3 No Form Request Classes + +Validation is done inline in controllers despite Laravel's Form Request feature. Only `LoginRequest` and `ProfileUpdateRequest` exist from Breeze scaffolding. + +### 3.4 No API Token Authentication + +The API login endpoint validates credentials but returns no token. This makes the API unusable for stateless/mobile clients that need persistent authentication. + +### 3.5 FCM Topic Fixed + +`AlertController::send()` always sends to `FCM_TOPIC_RAINFALL_WARNING` regardless of alert type: + +```php +$topic = env('FCM_TOPIC_RAINFALL_WARNING'); +``` + +Water level and siren alerts go to the rainfall topic, not their respective topics. + +### 3.6 Water Level Alert Bug + +**`sidesdecode.py` line 378:** +```python +send_alert_to_laravel(station_id, level, 1) # stationtype=1 (rainfall) instead of 2 (water level) +``` + +Water level threshold alerts are sent with `stationtype=1` instead of `stationtype=2`, causing them to be classified as rainfall alerts. + +### 3.7 `is_blocked` Not Enforced + +The `is_blocked` column exists on users but no middleware checks it. Blocked users can still log in and use the system. + +### 3.8 No Rate Limiting + +No rate limiting on login, API endpoints, or form submissions. + +--- + +## 4. LOW — Minor Issues + +### 4.1 `APP_DEBUG=true` in `.env` + +Debug mode should be `false` in any non-local environment. + +### 4.2 `database.db` and `database.sqlite` Committed + +SQLite database files are in the repository root and `src/database/`. + +### 4.3 Unused Packages + +- `laravel/sail` — installed but Docker Compose is used directly +- `laravel/pail` — installed but no log streaming is used +- `nunomaduro/collision` — dev dependency, standard + +### 4.4 `.env` Committed + +Both `.env` files (root and `src/`) are committed to the repository. They should be in `.gitignore`. + +### 4.5 `.editorconfig` Duplication + +`.editorconfig` exists in both root and `src/` directories. + +### 4.6 No `.gitignore` for Autoscript Logs + +`autoscript/sidesdecode.log` and `sidesdecode_error.log` are tracked in git. + +### 4.7 Mixed CSS Frameworks + +Both Tailwind CSS and Bootstrap 5 are loaded on pages, creating potential style conflicts and unnecessary bundle size. + +### 4.8 No Tests for Domain Logic + +Only Breeze default tests exist (`AuthenticationTest`, `EmailVerificationTest`, etc.). No tests for domain controllers, data pipeline, or export logic. + +--- + +## 5. Summary Table + +| # | Issue | Severity | File(s) | Effort | +|---|-------|----------|---------|--------| +| 1.1 | SQL Injection | CRITICAL | RainfallController, WaterLevelController | Medium | +| 1.2 | Hardcoded credentials | CRITICAL | sidesdecode.py, .env | Low | +| 1.3 | Default admin password | CRITICAL | DatabaseSeeder, migration | Low | +| 1.4 | Unauthenticated API | HIGH | routes/api.php | Medium | +| 1.5 | Password hash in query | LOW | AuthController | Low | +| 2.1 | No Eloquent models | HIGH | All controllers | High | +| 2.2 | No foreign keys | HIGH | All migrations | Medium | +| 2.3 | Missing indexes | HIGH | All data tables | Medium | +| 2.4 | Duplicate admin creation | MEDIUM | Seeder + migration | Low | +| 2.5 | Inconsistent naming | LOW | Throughout | Medium | +| 3.1 | N+1 query pattern | MEDIUM | MapController | Medium | +| 3.2 | No caching | MEDIUM | Throughout | Medium | +| 3.5 | FCM topic hardcoded | MEDIUM | AlertController | Low | +| 3.6 | Wrong stationtype for WL | HIGH | sidesdecode.py:378 | Low | +| 3.7 | Blocked users not enforced | MEDIUM | Auth flow | Low | +| 3.8 | No rate limiting | MEDIUM | Throughout | Low | +| 4.1 | Debug mode on | LOW | .env | Low | +| 4.2 | DB files committed | LOW | Root | Low | +| 4.4 | .env committed | LOW | Root | Low | +| 4.7 | Mixed CSS frameworks | LOW | Blade templates | Medium | +| 4.8 | No domain tests | LOW | tests/ | High | + +--- + +## 6. Recommended Priority + +1. **Immediate** (before any further changes): Fix SQL injection, rotate credentials, remove `.env` from git +2. **Short-term**: Fix water level stationtype bug, add API auth, add database indexes, fix FCM topic routing +3. **Medium-term**: Create Eloquent models, add foreign keys, implement caching, add input validation +4. **Long-term**: Full test coverage, clean up naming conventions, choose single CSS framework diff --git a/autoscript/sidesdecode.py b/autoscript/sidesdecode.py new file mode 100644 index 00000000..b44ed1a1 --- /dev/null +++ b/autoscript/sidesdecode.py @@ -0,0 +1,530 @@ +from ftplib import FTP +from datetime import datetime +from datetime import date +from io import BytesIO +import os +import time +import psycopg2 +import requests + +today = date.today() +day = f"{today.day:02}" +month = f"{today.month:02}" +year = str(today.year) + +today_str_file = datetime.today().strftime('%d%m%Y') + +ftp_server = os.environ.get("FTP_SERVER", "myvscada.com") +ftp_username = os.environ.get("FTP_USERNAME", "tck") +ftp_password = os.environ.get("FTP_PASSWORD", "") +ftp_root_folder = f"files/SIDES/SUCCESS/{year}/{month}/{day}" + +pg_host = os.environ.get("PG_HOST", "postgres") +pg_database = os.environ.get("PG_DATABASE", "sides_db") +pg_user = os.environ.get("PG_USER", "tck") +pg_password = os.environ.get("PG_PASSWORD", "") + +# Connect to PostgreSQL +conn = psycopg2.connect( + host=pg_host, + database=pg_database, + user=pg_user, + password=pg_password +) + + +cursor = conn.cursor() + + + +# Connect to FTP +ftp = FTP(ftp_server) +ftp.login(user=ftp_username, passwd=ftp_password) +ftp.cwd(ftp_root_folder) + +# Get today's date in yymmdd format +today_str = datetime.today().strftime('%y%m%d') + +# List all subfolders in /files/JKR +folders = ftp.nlst() + +#Move File to Error Folder Function +def move_to_error_folder(filename,filecontent,error_folder="/"+ftp_root_folder+"/ERROR/"): + with FTP(ftp_server) as ftp: + ftp.login(ftp_username, ftp_password) + parts = error_folder.strip("/").split("/") + current_path = '' + for part in parts : + current_path += "/" + part + try : + ftp.mkd(current_path) + except Exception : + pass + ftp.cwd(error_folder) + if isinstance(filecontent,str) : + filecontent = filecontent.encode("utf-8") + + bio = BytesIO(filecontent) + ftp.storbinary(f"STOR {filename}",bio) + bio.close() + print(f"⚠️ Moved {file_name} into {error_folder}") + + + try: + ftp.cwd('..') + ftp.delete(filename) + print(f"🗑️ Removed {file_name} from {ftp_root_folder}") + except Exception as e: + print(f"⚠️ Could not remove {file_name} from {ftp_root_folder}: {e}") + +#Move File to Success Folder Function +def move_to_success_folder(filename,filecontent,success_folder="/"+ftp_root_folder+"/SUCCESS/"): + with FTP(ftp_server) as ftp : + ftp.login(ftp_username,ftp_password) + + + parts = success_folder.strip("/").split("/") + current_path ="" + for part in parts : + current_path += "/" + part + try : + ftp.mkd(current_path) + print(f"Created Folder : {current_path}") + except Exception: + pass + + ftp.cwd(success_folder) + + if isinstance(filecontent,str): + filecontent = filecontent.encode("utf-8") + + with open("temp_upload.tmp", "wb") as temp_file : + temp_file.write(filecontent) + + with open("temp_upload.tmp", "rb") as temp_file: + ftp.storbinary(f"STOR {filename}",temp_file) + + os.remove("temp_upload.tmp") + print(f"Uploaded {filename} to {success_folder}") + + try : + ftp.cwd(f"/{ftp_root_folder}") + ftp.delete(filename) + print(f"🗑️ Removed {file_name} from {ftp_root_folder}") + except Exception as e: + print(f"⚠️ Could not remove {file_name} from {ftp_root_folder}: {e}") + +#Send Alert Function +def send_alert_to_laravel(stationid,level,stationtype): + + #Each parameter variable is from decoded process in Function process_line() + #Data will send to SIDES API in src/app/Http/Controllers/Api/AlertController.php + payload = { + "stationid" : stationid, + "level" : level, + "stationtype" :stationtype, + } + + #Request and send the payload data to API URL + try: + response = requests.post("https://sides.tck.com.my/api/alert",json=payload,timeout=5) + print("Alert Sent:" ,response.status_code) + except Exception as e: + print("Failed to send Alert",e) + + +# Define processing function once +def process_line(line,filename=None): + + # Strip each ',' seperator in file line + columns = line.strip().split(",") + + #Check Columns Length each line + if len(columns) < 25: + print(f"Skipping malformed line: {line}") + + #Uncomment If needed + # try : + # #Move decode folder into ERROR folder files/SIDES/ERROR + # #move_to_error_folder(filename,line,error_folder="/"+ftp_root_folder+"/ERROR") + + # except Exception as e : + # print(f"⚠️ Failed to move malformed line to ERROR folder: {e}") + return + + + # Variable Declaration based on CSV Column Data + + #Common Data Variable that needed + station_id = columns[1] + timestamp = columns[4] + + #Siren Data Variable + sirenid = columns[18] + siren = columns[19] + + #Rainfall Variable + anncumm = float(columns[21]) if columns[21] else None + dailycumm = float(columns[22]) if columns[22] else None + hourlycumm = float(columns[23]) if columns[23] else None + currrf = float(columns[24]) if columns[24] else None + + #Water Level Variable + waterlevel = float(columns[36]) if columns[36] else None + + #Water Level Threshold Variable + wldgr = float(columns[17]) if columns[17] else None + wlwarn = float(columns[16]) if columns[16] else None + wlalert = float(columns[15]) if columns[15] else None + + #Battery Variable + battery = float(columns[6]) if columns[6] else None + + + + + try: + + #Timestamp Format + datetime_object = datetime.strptime(timestamp, "%y%m%d%H%M%S") + + #Uncomment If Using Separate Date And Time Column + # date_part = datetime_object.strftime("%Y-%m-%d") + # time_part = datetime_object.strftime("%H:%M:%S") + + # date_part2 = datetime_object.date() + + + #Rainfall Data Insertion Into Database + #Check dailycumm variable Or hourtycumm variable is Not Null + #If either dailycumm or hourlycum, is Null. Skip this process + + if dailycumm != None or hourlycumm != None : + + #Execute Query for check data is already available or not in rainfall Table + #TABLE : RAINFALL + #COLUMN : stationid,timestamp + #INPUT : station_id,datetime_object + #OUTPUT : - + + cursor.execute(""" + SELECT COUNT(*) + FROM rainfall + WHERE stationid = %s AND timestamp = %s + """, (station_id, datetime_object)) + + #Delcarable Variable Record Exists Condition + record_exists = cursor.fetchone()[0] > 0 + + + # Check If Record is exist + # If record not exist execute INSERT rainfall data into rainfall Table + if not record_exists: + + #Execute Insert Query into rainfall table + #TABLE : RAINFALL + #COLUMN : stationid,timestamp,anncum,daily,hourly,currentrf,battery + #INPUT : station_id ,date_object,anncumm,dailycumm,hourlycumm,currrf,battery + #OUTPUT : - + + cursor.execute(""" + INSERT INTO rainfall (stationid,timestamp, anncum, daily, hourly, currentrf,battery) + VALUES (%s, %s, %s, %s, %s, %s, %s) + """, (station_id, datetime_object,anncumm, dailycumm, hourlycumm, currrf,battery)) + conn.commit() + + # If Rainfall Threshold Warning Threshold + if hourlycumm >= 30 : + + #Threshold Level Declarion + if hourlycumm >= 30 and hourlycumm < 60 : + level = 'Warning' + elif hourlycumm >= 60 : + level = 'Danger' + else : + level = 'Normal' + + #Execute Query for check data is already available or not in notification Table + #TABLE : NOTIFICATION + #COLUMN : stationid,timestamp,stationtype + #INPUT : station_id,datetime_object,1 + #OUTPUT : - + + cursor.execute(""" + SELECT COUNT(*) + FROM notification + WHERE stationid = %s AND timestamp = %s AND stationtype = %s + """, (station_id, datetime_object,1)) + + record_exists = cursor.fetchone()[0] > 0 + + # Check If Record is exist + # If record not exist execute INSERT rainfall data that trigger threshold into notification Table + if not record_exists: + + #Execute Insert Query into notification table + #stationtype = 1 because rainfall data + #TABLE : NOTIFICATION + #COLUMN : stationid,timestamp.stationtype,level,active_time + #INPUT : station_id,datetime_object,1,level,datetime_object + #OUTPUT : - + + cursor.execute(""" + INSERT INTO notification (stationid,timestamp, stationtype, level,active_time) + VALUES (%s, %s, %s, %s, %s) + """, (station_id, datetime_object,1, level,datetime_object)) + conn.commit() + + #Call Alert Function with station id , level , stationtype as parameter + #Send Status Station Detail to laravel API for Alert Notification + send_alert_to_laravel(station_id,level,1) + + #Print Rainfall Details + print(f"Station ID {station_id}") + print(f"Timestamp {timestamp}") + print(f"Daily Cumm : {dailycumm}") + print(f"Hourly : {hourlycumm}\n") + + + #Water Level Data Insertion Into Database + #Check waterlevel is Not Null + #If water level is Null. Skip this process + + if waterlevel != None : + + #Execute Query for check data is already available or not in waterlevel Table + #TABLE : WATERLEVEL + #COLUMN : stationid,datetime + #INPUT : station_id,datetime_object + #OUTPUT : - + + cursor.execute(""" + SELECT COUNT(*) + FROM waterlevel + WHERE stationid = %s AND datetime = %s + """, (station_id, datetime_object)) + record_exists = cursor.fetchone()[0] > 0 + + + # Check If Record is exist + # If record not exist execute INSERT waterlevel data into waterlevel Table + if not record_exists: + + #Execute Insert Query into waterlevel table + #TABLE : WATERLEVEL + #COLUMN : stationid,datetime,waterlevel,alert,warning,danger + #INPUT : station_id,datetime_object,waterlevel,wlalert,wlwarn,wldgr + #OUTPUT : - + cursor.execute(""" + INSERT INTO waterlevel (stationid,datetime, waterlevel, alert, warning,danger) + VALUES (%s, %s, %s, %s, %s, %s) + """, (station_id, datetime_object,waterlevel, wlalert, wlwarn, wldgr)) + conn.commit() + + # If Water Level Threshold Trigger + if waterlevel >= wlalert : + + #Water Level Level Declaration + if waterlevel >= wlalert and waterlevel < wlwarn : + level = 'Alert' + elif waterlevel >= wlwarn and waterlevel < wldgr : + level = 'Warning' + elif waterlevel >= wldgr: + level = 'Danger' + else : + level = 'Normal' + + #Execute Query for check data is already available or not in notification Table + #TABLE : NOTIFICATION + #COLUMN : stationid,timestamp,stationtype + #INPUT : station_id,datetime_object,2 + #OUTPUT : - + + cursor.execute(""" + SELECT COUNT(*) + FROM notification + WHERE stationid = %s AND timestamp = %s AND stationtype = %s + """, (station_id, datetime_object,2)) + record_exists = cursor.fetchone()[0] > 0 + + + # Check If Record is exist + # If record not exist execute INSERT waterlevel data that trigger threshold notification Table + if not record_exists: + + #Execute Insert Query into notification table + #stationtype = 2 because waterlevel data + + #TABLE : NOTIFICATION + #COLUMN : stationid,timestamp,stationtype,level,active_time + #INPUT : station_id,datetime_object,2,level,datetime_object + #OUTPUT : - + + cursor.execute(""" + INSERT INTO notification (stationid,timestamp, stationtype, level,active_time) + VALUES (%s, %s, %s, %s, %s) + """, (station_id, datetime_object,2, level,datetime_object)) + conn.commit() + + + #Call Alert Function with station id , level , stationtype as parameter + #Send Status Station Detail to laravel API for Alert Notification + send_alert_to_laravel(station_id,level,2) + print(f"Station ID {station_id}") + print(f"Timestamp {timestamp}") + print(f"Water Level : {waterlevel}") + print(f"Danger : {wldgr}") + print(f"Warning : {wlwarn}\n") + + + #Siren Data Insertion Into Database + #Check sirenid variable is Not Null + #If sirenid is Null. Skip this process + + if sirenid != None: + + #Execute Query for check data is already available or not in siren Table + #TABLE : SIREN + #COLUMN : stationid,active_time + #INPUT : station_id,datetime_object + #OUTPUT : - + + cursor.execute(""" + SELECT COUNT(*) + FROM siren + WHERE stationid = %s AND active_time = %s + """, (station_id, datetime_object)) + record_exists = cursor.fetchone()[0] > 0 + + + #Siren level Declaration + if siren == 'H' : + level = 'Danger' + elif siren == 'L': + level = 'Warning' + else : + level = 'Normal' + + + # check siren condition + # If Siren is Normal + + if siren == 'N': + + # Check If Record is exist + # If record not exist execute INSERT siren data into siren Table + + if not record_exists: + + #TABLE : SIREN + #COLUMN : stationid,active_time,level + #INPUT : station_id,datetime_object,siren + #OUTPUT : - + + cursor.execute(""" + INSERT INTO siren (stationid, stationtype, active_time, level) + VALUES (%s, %s, %s, %s) + """, (station_id, 3, datetime_object, siren)) + conn.commit() + + + #Print Siren Details + print(f"Station ID {station_id}") + print(f"SRID {sirenid}") + print(f"Timestamp {timestamp}") + print(f"Alarm : {siren}\n") + print(f"Level : {level}\n") + + #If siren not Normal + elif siren != 'N' : + + # Check If Record is exist + # If record not exist execute INSERT siren data into siren Table + + if not record_exists: + + + #Execute Insert Query into siren table + #TABLE : SIREN + #COLUMN : stationid,active_time,level + #INPUT : station_id,datetime_object,siren + #OUTPUT : - + + cursor.execute(""" + INSERT INTO siren (stationid, stationtype, active_time, level) + VALUES (%s, %s, %s, %s) + """, (station_id, 3, datetime_object, siren)) + conn.commit() + + + + #Print Siren Table + print(f"Station ID {station_id}") + print(f"SRID {sirenid}") + print(f"Timestamp {timestamp}") + print(f"Alarm : {siren}\n") + + + + #Send Station Alert To API When Siren Level is not normal + #Call Alert Function with station id , level , stationtype as parameter + #Send Status Station Detail to laravel API for Alert Notification + + if level != 'Normal' : + send_alert_to_laravel(station_id,level,3) + + except Exception as e: + + conn.rollback() + + #Move decode folder into ERROR folder files/SIDES/ERROR + #move_to_error_folder(filename,line,error_folder="/"+ftp_root_folder+"/ERROR") + print(f"Error processing line: {e}") + + +#Start Process FTP File +try: + + #Folder File List Variable Declaration + files = ftp.nlst() + + #Loop Through the Folder/Files + for file_name in files: + + #Skip Tideda File + if "rf" in file_name.lower(): + #print(f"Skipping file: {file_name} (contains 'rf')") + continue + + #Skip Others Day File + if today_str not in file_name: + #print(f"Skipping file: {file_name} (not from today {today_str})") + continue + + print(f"Processing file: {file_name}") + all_lines = [] + + #Retrieve the files as a text , line by line + ftp.retrlines( + f"RETR {file_name}", + lambda line: (all_lines.append(line), process_line(line, file_name)) + ) + file_content = "\n".join(all_lines) + + #Uncomment if needed + #Move decode folder into success folder files/SIDES/SUCCESS/year/month/day + + #move_to_success_folder(file_name,file_content,success_folder="/"+ftp_root_folder+"/SUCCESS/"+year+"/"+month+"/"+day) + +except Exception as e: + print(f"Failed to process folder: {e}") + +#Uncomment this for test Alert Notification +# send_alert_to_laravel('KBLG0026','Warning',1) +# send_alert_to_laravel('KBLG0026','Danger',2) +# send_alert_to_laravel('KBLG0031','Warning',3) + +# Close connections +ftp.quit() +cursor.close() +conn.close() diff --git a/backup/sides_20260120 b/backup/sides_20260120 new file mode 100644 index 00000000..63d2ea36 Binary files /dev/null and b/backup/sides_20260120 differ diff --git a/docs/01-OVERVIEW.md b/docs/01-OVERVIEW.md new file mode 100644 index 00000000..5a4b8828 --- /dev/null +++ b/docs/01-OVERVIEW.md @@ -0,0 +1,71 @@ +# SIDES - Project Overview + +## What Is SIDES? + +**Sabo Integrated Debris Flow Monitoring and Early Warning System (SIDES)** is a flood and debris flow early warning system for **Sungai Kupang**, Malaysia. It is operated under **Jabatan Pengairan dan Saliran (JPS)** — the Malaysian Department of Irrigation and Drainage. + +The system monitors rainfall, water levels, and siren activations at remote telemetry stations and presents real-time data on a web dashboard with an interactive map. + +## Purpose + +SIDES provides: + +- **Real-time monitoring** of hydrological and meteorological data from field stations +- **Early warning alerts** when rainfall or water levels exceed danger thresholds +- **Siren status tracking** for stations equipped with warning sirens +- **Historical data analysis** with exportable reports (Excel, PDF) +- **Push notifications** to mobile devices via Firebase Cloud Messaging (FCM) +- **Admin management** of stations, users, and access control + +## Domain Context + +- **Location**: Sungai Kupang, Malaysia (Kedah region based on station IDs) +- **Operator**: JPS (Department of Irrigation and Drainage), Malaysia +- **Data source**: Remote telemetry stations transmit CSV data via FTP to a central server +- **Data pipeline**: Python script (`autoscript/sidesdecode.py`) fetches FTP files, parses CSV, inserts into PostgreSQL, and triggers push notifications + +## User Roles + +| Role | Access Level | Description | +|------|-------------|-------------| +| **Public** | Unauthenticated | View dashboard map with station data | +| **User** | `access_level = 2` | View all monitoring pages, historical data, export reports | +| **Admin** | `access_level = 1` | Full access including station management and user management | + +## High-Level Architecture + +``` +[Telemetry Stations] --CSV/FTP--> [FTP Server (myvscada.com)] + | + [Python Script (cron)] + | + v +[PostgreSQL Database] <--data-- [sidesdecode.py] --alert--> [Laravel API] + | + v + [Firebase FCM] + | + v + [Mobile Devices] + +[Laravel Web App] <--reads-- [PostgreSQL Database] + | + v +[Web Browser] <--Leaflet Map, Charts, Tables--> +``` + +## Application Name + +The application is registered as **SIDES** in Laravel config (`APP_NAME=SIDES`). + +## Bilingual Support + +The application supports: +- **English** (en) — default +- **Bahasa Malaysia** (bm) + +Language files are stored in `src/lang/{en,bm}/` and toggled via `LocaleController` with session-based persistence. + +## Project Name Origin + +The repository is named `tckdev` (likely **TCK Development**), referencing the organization running the SIDES system. diff --git a/docs/02-ARCHITECTURE.md b/docs/02-ARCHITECTURE.md new file mode 100644 index 00000000..2eaf62b1 --- /dev/null +++ b/docs/02-ARCHITECTURE.md @@ -0,0 +1,165 @@ +# Architecture + +## Technology Stack + +| Layer | Technology | Version | +|-------|-----------|---------| +| Backend | Laravel (PHP) | 12.x | +| PHP | PHP-FPM | 8.2 | +| Frontend | Blade Templates + Alpine.js | 3.x | +| CSS | Tailwind CSS + Bootstrap 5.3 | 3.x / 5.3 | +| Charts | Custom JS (Chart.js via `graph.js`) | — | +| Maps | Leaflet.js | 1.9.4 | +| Build Tool | Vite | 7.x | +| Database | PostgreSQL | 15 | +| Cache/Session | Database (via Laravel) | — | +| Queue | Database (via Laravel) | — | +| Containerization | Docker + Docker Compose | 3.9 | +| Web Server | Nginx | stable-alpine | +| PDF Generation | barryvdh/laravel-dompdf | 3.1 | +| Excel Export | maatwebsite/excel | 3.1 | +| Push Notifications | Firebase Cloud Messaging (FCM) via Google Auth | 1.49 | +| Date Picker | Flatpickr | 4.6 | +| Auth Scaffold | Laravel Breeze | 2.3 | +| Data Pipeline | Python (psycopg2, ftplib) | 3.x | + +## Container Architecture + +The application runs in 5 Docker containers defined in `docker-compose.yml`: + +``` +┌──────────────────────────────────────────────────────────┐ +│ Docker Network: tckdev_net │ +│ │ +│ ┌─────────┐ ┌─────────┐ ┌──────────┐ │ +│ │ web │ │ app │ │ postgres │ │ +│ │ (nginx) │──>│ (PHP-FPM│──>│ (DB) │ │ +│ │ :80 │ │ :9000) │ │ :5432 │ │ +│ └─────────┘ └─────────┘ └──────────┘ │ +│ │ +│ ┌─────────┐ ┌─────────┐ │ +│ │ pgAdmin │ │ adminer │ │ +│ │ :5050 │ │ :6060 │ (DB management tools) │ +│ └─────────┘ └─────────┘ │ +└──────────────────────────────────────────────────────────┘ +``` + +### Container Details + +| Container | Image | Port | Purpose | +|-----------|-------|------|---------| +| `tckdev-app` | Custom (PHP 8.2-FPM + Composer 2.3 + Node.js) | 9000 (internal) | Laravel application | +| `tckdev-web` | nginx:stable-alpine | 80 | Reverse proxy & static files | +| `tckdev-db` | postgres:15 | 5432 | PostgreSQL database | +| `tckdev-pgAdmin` | dpage/pgadmin4 | 5050 | Database management UI | +| `tckdev-adminer` | adminer | 6060 | Lightweight DB management UI | + +## Request Flow + +``` +Browser --> Nginx (:80) --> PHP-FPM (:9000) --> Laravel Router + | + ┌─────────────┼─────────────────┐ + v v v + Web Routes API Routes Auth Routes + (web.php) (api.php) (auth.php) + | | | + v v v + Controllers Api/Controllers Breeze Controllers + | | + v v + DB::raw() SQL DB::raw() SQL + | | + v v + PostgreSQL PostgreSQL + | + v + Blade Views --> HTML Response +``` + +## Middleware Stack + +| Middleware | Route Group | Purpose | +|-----------|------------|---------| +| `auth` | Web (protected) | Require authenticated session | +| `admin` | Admin routes | Require `access_level = 1` | +| `LocalizationMiddleware` | All web routes | Set locale from session (`en`/`bm`) | +| `guest` | Auth routes (login/register) | Redirect away if already authenticated | + +## Database Design + +### Core Tables + +``` +┌─────────────┐ ┌──────────────┐ +│ station │ │ users │ +│─────────────│ │──────────────│ +│ stationid* │ │ id* │ +│ name │ │ name │ +│ district │ │ email │ +│ lng, lat │ │ password │ +│ mainriverbasin│ │ access_level │ +│ subriverbasin│ │ is_blocked │ +│ rainfall │ │ login_attempts│ +│ waterlevel │ └──────────────┘ +│ siren │ +│ cctv_link │ +└──────┬──────┘ + │ + │ (stationid FK via application logic, not DB constraint) + │ + ┌────┼────────────┬──────────────┐ + v v v v +┌──────────┐ ┌────────────┐ ┌───────────┐ +│ rainfall │ │ waterlevel │ │ siren │ +│──────────│ │────────────│ │───────────│ +│ id* │ │ id* │ │ id* │ +│ stationid│ │ stationid │ │ stationid │ +│ timestamp│ │ datetime │ │ stationtype│ +│ anncum │ │ waterlevel │ │ active_time│ +│ daily │ │ alert │ │ level │ +│ hourly │ │ warning │ └───────────┘ +│ currentrf│ │ danger │ +│ battery │ └────────────┘ +└──────────┘ + │ + v +┌──────────────┐ +│ notification │ +│──────────────│ +│ id* │ +│ stationid │ +│ timestamp │ +│ stationtype │ (1=rainfall, 2=waterlevel, 3=siren) +│ level │ (Normal, Alert, Warning, Danger) +│ active_time │ +└──────────────┘ +``` + +### Station Types + +The `station` table uses boolean flags to indicate which monitoring types each station supports: + +- `rainfall = 1` — Station has rainfall monitoring +- `waterlevel = 1` — Station has water level monitoring +- `siren = 1` — Station has a warning siren + +### Foreign Key Note + +There are **no database-level foreign keys** between `station` and the data tables. Relationships are maintained at the application level via `stationid` column matching. + +### Laravel Standard Tables + +The application also uses standard Laravel tables: +- `users` — Authentication and authorization +- `password_reset_tokens` — Password reset flow +- `sessions` — Database-backed sessions +- `cache`, `cache_locks` — Database cache store +- `jobs`, `job_batches`, `failed_jobs` — Database queue + +## Authentication & Authorization + +- **Web auth**: Laravel Breeze (session-based, Blade views) +- **API auth**: Custom token-less login via `Api\AuthController` (returns user info, no token generation) +- **Admin access**: Controlled by `AdminMiddleware` checking `access_level === 1` +- **User blocking**: Users can be blocked via `is_blocked` flag (though no middleware enforces it currently) diff --git a/docs/03-DEPLOYMENT.md b/docs/03-DEPLOYMENT.md new file mode 100644 index 00000000..8e410595 --- /dev/null +++ b/docs/03-DEPLOYMENT.md @@ -0,0 +1,122 @@ +# Deployment & Infrastructure + +## Prerequisites + +- Docker ^19.* +- Docker Compose +- Make (optional, for Makefile commands) + +## Environment Configuration + +### Root `.env` (Docker-level) + +Located at `/root/sides/tckdev/.env`: + +``` +POSTGRES_DB="tckdev" +POSTGRES_USER="tck" +POSTGRES_PASSWORD="projectdev##1" +PGADMIN_EMAIL="tck68000@gmail.com" +PGADMIN_PASSWORD="projectdev##1" +``` + +### Application `.env` (Laravel-level) + +Located at `/root/sides/tckdev/src/.env`: + +Key configuration: +- `APP_URL=https://sides.tck.com.my` +- `DB_CONNECTION=pgsql` → `DB_HOST=192.168.0.211` (direct IP, not Docker service name) +- `SESSION_DRIVER=database` +- `CACHE_STORE=database` +- `QUEUE_CONNECTION=database` +- `MAIL_MAILER=smtp` via Gmail SMTP (`sideskupang@gmail.com`) +- Firebase FCM credentials at `storage/app/firebase/sides-b4abb-3604a7cf7584.json` + +## Initial Setup + +### From Scratch (New Project) + +```bash +make create-project +``` + +This runs: +1. Creates `src/` directory +2. Builds Docker images +3. Starts containers +4. Runs `composer create-project laravel/laravel .` +5. Generates application key +6. Creates storage link +7. Sets permissions +8. Installs npm dependencies + +### From Existing Code (Clone) + +```bash +make init +``` + +This runs: +1. Builds and starts containers +2. `composer install` +3. Copies `.env.example` to `.env` +4. Generates application key +5. Creates storage link +6. Sets permissions +7. `npm install` +8. `php artisan migrate:fresh --seed` + +## Makefile Commands Reference + +| Command | Description | +|---------|-------------| +| `make up` | Start containers (detached) | +| `make down` | Stop containers, remove orphans | +| `make down-v` | Stop containers, remove volumes | +| `make build` | Build Docker images | +| `make remake` | Full destroy + reinit | +| `make destroy` | Remove all containers, images, volumes | +| `make stop` | Stop containers | +| `make restart` | Down + up | +| `make ps` | Show container status | +| `make logs` | Show all container logs | +| `make logs-watch` | Follow all logs | +| `make log-app` | Show app container logs | +| `make log-db` | Show database logs | +| `make app` | Shell into app container | +| `make web` | Shell into web container | +| `make migrate` | Run migrations | +| `make fresh` | Fresh migration with seed | +| `make seed` | Run seeders | +| `make tinker` | Open Laravel tinker | +| `make test` | Run PHPUnit tests | +| `make cache` | Optimize autoload + cache | +| `make cache-clear` | Clear all caches | + +## Access Points + +| Service | URL | +|---------|-----| +| Application | http://localhost:80 | +| pgAdmin | http://localhost:5050 | +| Adminer | http://localhost:6060 | +| PostgreSQL | localhost:5432 | + +## Database Connection Discrepancy + +**Important**: The `docker-compose.yml` defines a `postgres` service, but `src/.env` connects to `192.168.0.211` — an external PostgreSQL host, not the Docker container. This means: + +- In **development/local**: The app may connect to an external database instead of the Docker postgres container +- The Docker postgres service is still available at `postgres:5432` for inter-container communication +- The Python autoscript also connects to `192.168.0.211` directly + +To use the Docker PostgreSQL, change `DB_HOST=postgres` in `src/.env`. + +## Production Notes + +- `APP_DEBUG=true` is set in `.env` — should be `false` in production +- `URL::forceScheme('https')` is enabled globally in `AppServiceProvider` +- The app URL is `https://sides.tck.com.my` +- Gmail SMTP is used for password reset emails +- Firebase credentials JSON file is stored in `storage/app/firebase/` diff --git a/docs/04-DATA-PIPELINE.md b/docs/04-DATA-PIPELINE.md new file mode 100644 index 00000000..05af5470 --- /dev/null +++ b/docs/04-DATA-PIPELINE.md @@ -0,0 +1,143 @@ +# Data Pipeline: Python Autoscript + +## Overview + +The file `autoscript/sidesdecode.py` is the data ingestion pipeline that: + +1. Connects to an FTP server where telemetry stations upload CSV data +2. Downloads and parses CSV files for the current day +3. Inserts rainfall, water level, and siren data into PostgreSQL +4. Triggers push notifications when thresholds are exceeded + +## How It Runs + +The script is designed to be run on a **schedule** (likely cron job), processing new data files uploaded by remote telemetry stations throughout the day. + +## FTP Connection + +``` +Server: myvscada.com +Username: tck +Password: tck6789 +Path: files/SIDES/SUCCESS/{year}/{month}/{day}/ +``` + +The script navigates to today's date folder and lists all files. + +### File Filtering + +- Skips files containing "rf" in the filename (Tideda format files) +- Only processes files with today's date (`yymmdd` format) in the filename + +## CSV Format + +Each line in the CSV file contains 37+ comma-separated columns. Key columns extracted: + +| Column Index | Field | Description | +|-------------|-------|-------------| +| 1 | `station_id` | Station identifier (e.g., KBLG0026) | +| 4 | `timestamp` | Timestamp in `yymmddHHMMSS` format | +| 6 | `battery` | Battery voltage | +| 15 | `wlalert` | Water level alert threshold | +| 16 | `wlwarn` | Water level warning threshold | +| 17 | `wldgr` | Water level danger threshold | +| 18 | `sirenid` | Siren identifier | +| 19 | `siren` | Siren status (`H`=Danger/High, `L`=Warning/Low, `N`=Normal) | +| 21 | `anncumm` | Annual cumulative rainfall | +| 22 | `dailycumm` | Daily cumulative rainfall | +| 23 | `hourlycumm` | Hourly rainfall | +| 24 | `currrf` | Current rainfall | +| 36 | `waterlevel` | Current water level reading | + +## Data Processing Logic + +### Rainfall Data + +1. Check if `dailycumm` or `hourlycumm` is not null +2. Check if record already exists for this station+timestamp +3. If new, INSERT into `rainfall` table +4. **Threshold check**: If `hourlycumm >= 30`: + - `30 <= hourly < 60` → **Warning** level + - `hourly >= 60` → **Danger** level + - INSERT into `notification` table + - Send push notification via Laravel API + +### Water Level Data + +1. Check if `waterlevel` is not null +2. Check if record already exists for this station+datetime +3. If new, INSERT into `waterlevel` table (with alert/warning/danger thresholds) +4. **Threshold check**: If `waterlevel >= alert`: + - `alert <= wl < warning` → **Alert** level + - `warning <= wl < danger` → **Warning** level + - `wl >= danger` → **Danger** level + - INSERT into `notification` table + - Send push notification via Laravel API + +### Siren Data + +1. Check if `sirenid` is not null +2. Check if record already exists for this station+active_time +3. Determine level from siren status: + - `H` → **Danger** + - `L` → **Warning** + - `N` → **Normal** +4. INSERT into `siren` table +5. If level is not Normal, send push notification via Laravel API + +## Alert Notification Flow + +When a threshold is triggered, the script calls `send_alert_to_laravel()`: + +```python +def send_alert_to_laravel(stationid, level, stationtype): + payload = { + "stationid": stationid, + "level": level, + "stationtype": stationtype, # 1=rainfall, 2=waterlevel, 3=siren + } + response = requests.post("https://sides.tck.com.my/api/alert", json=payload, timeout=5) +``` + +This hits the Laravel `AlertController` which: +1. Builds notification title/body based on station type and level +2. Calls `FcmService::sendToTopic()` which: + - Reads Firebase service account credentials + - Gets an OAuth2 access token from Google + - Sends FCM message to topic (e.g., `rainfall_warning`) + - Push notification arrives on subscribed mobile devices + +## PostgreSQL Connection + +The script connects directly to PostgreSQL: + +```python +pg_host = "192.168.0.211" +pg_database = "sides_db" +pg_user = "tck" +pg_password = "projectdev##1" +``` + +**Note**: This is a hardcoded external IP, not using the Docker container. The database name is `sides_db` (different from the Docker `.env` which uses `tckdev`). + +## File Management (Commented Out) + +The script contains (commented out) functions for: +- `move_to_error_folder()` — Move malformed files to an FTP error folder +- `move_to_success_folder()` — Move processed files to a success archive folder + +These are currently disabled — files remain in the source folder after processing. + +## Log Files + +- `autoscript/sidesdecode.log` — Processing output +- `autoscript/sidesdecode_error.log` — Error output + +## Known Issues + +1. **Hardcoded credentials** — FTP and PostgreSQL credentials are embedded in the script +2. **No deduplication beyond same-timestamp** — If the script runs twice, it skips exact duplicates but has no broader deduplication +3. **Commented out file management** — Processed files are not moved/archived +4. **Water level alert sends `stationtype=1`** instead of `2` (likely a bug at line 378) +5. **No error recovery** — If the script crashes mid-processing, some data may be partially inserted +6. **No connection pooling** — New FTP and database connections each run diff --git a/docs/05-FEATURES.md b/docs/05-FEATURES.md new file mode 100644 index 00000000..5c889126 --- /dev/null +++ b/docs/05-FEATURES.md @@ -0,0 +1,203 @@ +# Features & User Guide + +## Feature Overview + +SIDES provides the following feature areas: + +### 1. Public Dashboard (Map View) + +**Route**: `/` or `/dashboard` +**Access**: Public (no auth required) +**Controller**: `MapController::getCurrentData()` + +Displays an interactive Leaflet.js map showing all monitoring stations with their current readings: +- Station markers color-coded by status +- Popup showing latest rainfall, water level, and siren data +- Real-time station data from PostgreSQL + +### 2. Rainfall Monitoring + +**Route**: `/rainfall` +**Access**: Authenticated users +**Controller**: `RainfallController::index()` + +- Table showing rainfall data for all rainfall-enabled stations +- Filter by station and date +- Displays: hourly, daily, and 7-day historical values +- Last data update timestamp + +### 3. Rainfall Graph + +**Route**: `/rainfall/graph/{stationid}` +**Access**: Authenticated users +**Controller**: `RainfallController::rainfallGraph()` + +- Interactive chart showing hourly rainfall for the selected station today +- X-axis: time (HH:MM), Y-axis: rainfall value + +### 4. Early Warning Threshold (IDF) + +**Route**: `/threshold` +**Access**: Authenticated users +**Controller**: `RainfallController::rainfallSum()` + +- 6-hour cumulative rainfall data for early warning +- Shows hourly values and timestamps for each of the last 6 hours +- Filter by station and date +- Threshold graph data available via API + +### 5. Historical Rainfall + +**Route**: `/rainfall/historical` +**Access**: Authenticated users +**Controller**: `RainfallController::historicalRainfall()` + +- Detailed hourly rainfall breakdown (24 hours per row) +- Filter by station and date range +- Shows total 24h rainfall per day +- **Export to Excel**: `/rainfall/historical/export` + +### 6. Water Level Monitoring + +**Route**: `/waterlevel` +**Access**: Authenticated users +**Controller**: `WaterLevelController::index()` + +- Table showing current water level readings +- Displays: water level value, alert/warning/danger thresholds +- Filter by station and date +- Interactive water level graph + +### 7. Historical Water Level + +**Route**: `/waterlevel/historical` +**Access**: Authenticated users +**Controller**: `WaterLevelController::wlHistory()` + +- Hourly water level data for a selected station and date +- Shows threshold levels alongside actual readings +- **Export to Excel**: `/waterlevel/historical/export` + +### 8. Siren Monitoring + +**Route**: `/siren` +**Access**: Authenticated users +**Controller**: `SirenController::index()` + +- Current siren status for all siren-equipped stations +- Shows last 3 days of siren activations +- Siren levels: Normal (N), Warning (L), Danger (H) + +### 9. Siren History + +**Route**: `/sirenhistory` +**Access**: Authenticated users +**Controller**: `SirenController::SirenHistory()` + +- Paginated history of all siren activations (excluding Normal) +- **Export to PDF**: `/export/siren-history/pdf` + +### 10. Notifications + +**Rainfall Notifications**: `/notificationrf` +**Water Level Notifications**: `/notificationwl` +**Siren Notifications**: `/notificationsiren` +**Access**: Authenticated users +**Controller**: `NotificationController` + +- Shows today's threshold-triggered notifications +- Grouped by type (rainfall, water level, siren) + +### 11. Notification History + +**Rainfall History**: `/historyrf` +**Water Level History**: `/historywl` +**Access**: Authenticated users +**Controller**: `NotificationController` + +- Paginated history of all notifications +- **Export to PDF**: + - `/export/rainfall-history/pdf` + - `/export/waterlevel-history/pdf` + +### 12. CCTV Links + +**Route**: `/cctv` +**Access**: Authenticated users +**Controller**: `cctvController::index()` + +- Lists stations with CCTV links (where `cctv_link` is not null and `waterlevel = 1`) +- Links open external CCTV feeds + +### 13. Admin: Station Management + +**Route**: `/stationmanagement` +**Access**: Admin only (`access_level = 1`) +**Controller**: `AdminController` + +- List all stations (paginated, 5 per page) +- Add new station +- Edit station details (name, district, coordinates, type flags, CCTV link) +- Delete station +- Shows station type counts (rainfall, water level, siren) + +### 14. Admin: User Management + +**Route**: `/usermgmt` +**Access**: Admin only +**Controller**: `AdminController` + +- List all users (paginated, 5 per page) +- Add new user +- Edit user details (name, email, access level, block/unblock) +- Reset user password +- Delete user +- Shows user counts by role + +### 15. API Endpoints + +**Prefix**: `/api/` +**Auth**: Custom login (no token-based auth) + +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/station/current` | GET | All stations with current data | +| `/api/station/rainfall` | GET | Current rainfall data per station | +| `/api/station/waterlevel` | GET | Current water level data per station | +| `/api/station/notification` | GET | Today's notifications per station | +| `/api/station/history` | GET | 3-day notification history | +| `/api/station/siren` | GET | Current siren status | +| `/api/station/siren/history` | GET | 3-day siren history | +| `/api/login` | POST | Login (username + password) | +| `/api/alert` | POST | Send FCM push notification | + +### 16. Profile Management + +**Route**: `/profile` +**Access**: Authenticated users +**Controller**: `ProfileController` (Breeze scaffold) + +- Update name and email +- Update password +- Delete account + +### 17. Localization + +**Route**: `/locale/{locale}` +**Access**: Public +**Controller**: `LocaleController` + +- Switch between English (`en`) and Bahasa Malaysia (`bm`) +- Persisted in session +- Applied via `LocalizationMiddleware` on every request + +### 18. Authentication + +**Routes**: `/login`, `/register`, `/forgot-password`, `/reset-password` +**Access**: Guest only (for login/register) +**Scaffold**: Laravel Breeze + +- Email/password authentication +- Password reset via email (Gmail SMTP) +- Email verification +- Session-based auth diff --git a/docs/06-DATABASE.md b/docs/06-DATABASE.md new file mode 100644 index 00000000..3fc3f611 --- /dev/null +++ b/docs/06-DATABASE.md @@ -0,0 +1,131 @@ +# Database Schema + +## Tables + +### `station` + +Primary table storing telemetry station metadata. + +| Column | Type | Description | +|--------|------|-------------| +| `stationid` | `varchar` (PK) | Unique station identifier (e.g., "KBLG0026") | +| `name` | `varchar(255)` | Station display name | +| `district` | `varchar(255)` | District location | +| `lng` | `float` | Longitude coordinate | +| `lat` | `float` | Latitude coordinate | +| `mainriverbasin` | `varchar(255)` | Main river basin name | +| `subriverbasin` | `varchar(255)` | Sub river basin name | +| `rainfall` | `integer` | Has rainfall sensor (1=yes, 0=no) | +| `waterlevel` | `integer` | Has water level sensor (1=yes, 0=no) | +| `siren` | `integer` (nullable) | Has siren (1=yes, 0=no) | +| `cctv_link` | `varchar(500)` (nullable) | URL to CCTV feed | + +### `rainfall` + +Stores rainfall readings from telemetry stations. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | `bigint` (PK, auto) | Auto-increment ID | +| `stationid` | `varchar` | Station identifier (FK to station) | +| `timestamp` | `timestamp` | Reading timestamp | +| `anncum` | `double` | Annual cumulative rainfall | +| `daily` | `double` | Daily cumulative rainfall | +| `hourly` | `double` | Hourly rainfall | +| `currentrf` | `double` | Current rainfall | +| `battery` | `double` | Battery voltage | + +### `waterlevel` + +Stores water level readings with threshold values. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | `bigint` (PK, auto) | Auto-increment ID | +| `stationid` | `varchar` | Station identifier (FK to station) | +| `datetime` | `timestamp` | Reading timestamp | +| `waterlevel` | `double` | Current water level (meters) | +| `alert` | `double` | Alert threshold level | +| `warning` | `double` | Warning threshold level | +| `danger` | `double` | Danger threshold level | + +### `siren` + +Stores siren activation records. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | `bigint` (PK, auto) | Auto-increment ID | +| `stationid` | `varchar` | Station identifier (FK to station) | +| `stationtype` | `varchar` | Station type identifier | +| `active_time` | `timestamp` | Siren activation time | +| `level` | `varchar` | Siren level (`H`=Danger, `L`=Warning, `N`=Normal) | + +### `notification` + +Stores threshold-exceeded alert records. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | `bigint` (PK, auto) | Auto-increment ID | +| `stationid` | `varchar` | Station identifier (FK to station) | +| `timestamp` | `timestamp` | Alert timestamp | +| `stationtype` | `integer` | Type: 1=rainfall, 2=waterlevel, 3=siren | +| `level` | `varchar` | Alert level (Alert, Warning, Danger) | +| `active_time` | `timestamp` (nullable) | Activation time | + +### `users` + +Stores application users. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | `bigint` (PK, auto) | Auto-increment ID | +| `name` | `varchar(255)` | Username | +| `email` | `varchar(255)` (unique) | Email address | +| `email_verified_at` | `timestamp` (nullable) | Email verification timestamp | +| `password` | `varchar(255)` | Bcrypt-hashed password | +| `access_level` | `integer` | 1=Admin, 2=User | +| `is_blocked` | `boolean` | Account blocked status | +| `login_attempts` | `integer` | Failed login attempt count | +| `remember_token` | `varchar` | Remember me token | +| `created_at`, `updated_at` | `timestamp` | Laravel timestamps | + +### Laravel Standard Tables + +- `password_reset_tokens` — Password reset tokens +- `sessions` — Database-backed sessions +- `cache` / `cache_locks` — Cache store +- `jobs` / `job_batches` / `failed_jobs` — Queue system +- `migrations` — Migration tracking + +## Relationships + +``` +station (1) ──< (many) rainfall via stationid +station (1) ──< (many) waterlevel via stationid +station (1) ──< (many) siren via stationid +station (1) ──< (many) notification via stationid +``` + +**Note**: No database-level foreign keys or constraints exist. All relationships are maintained at the application level. + +## Indexes + +- `users.email` — unique index +- `sessions.last_activity` — index +- `sessions.user_id` — index +- No additional indexes on data tables (potential performance concern) + +## Default Data + +### Default Admin User + +Created via `DatabaseSeeder` and migration `2025_12_11_124201_add_default_user_to_users_table.php`: + +- **Username**: `admin` (seeder) / `admin` (migration) +- **Email**: `admin@example.com` +- **Password**: `password123` +- **Access Level**: 1 (Admin) + +**Note**: The admin user is created in both the seeder AND a migration, which would cause a duplicate key error if both run. diff --git a/docs/07-API.md b/docs/07-API.md new file mode 100644 index 00000000..dc69a284 --- /dev/null +++ b/docs/07-API.md @@ -0,0 +1,139 @@ +# API Reference + +## Authentication + +### POST `/api/login` + +Login via API (returns user info, no token). + +**Request Body:** +```json +{ + "username": "admin", + "password": "password123" +} +``` + +**Success Response (200):** +```json +{ + "error": false, + "id": 1, + "username": "admin", + "email": "admin@example.com", + "acc_lvl": 1 +} +``` + +**Error Response (200):** +```json +{ + "error": true, + "message": "Wrong Password/Username" +} +``` + +**Notes:** +- No authentication token is generated — this endpoint only validates credentials +- No session is created — API endpoints are stateless +- All other API endpoints have **no authentication** — they are publicly accessible + +--- + +## Station Data + +### GET `/api/station/current` + +Returns all stations with their latest rainfall, water level, and siren data. + +**Response:** +```json +[ + { + "stationid": "KBLG0026", + "name": "Stesen Kupang", + "district": "Baling", + "lng": 100.7521, + "lat": 5.7879, + "rainfall": 1, + "waterlevel": 1, + "siren": 1, + "rainfall_value": 12.5, + "rainfall_time": "2025-11-06T14:00:00", + "waterlevel_value": 3.2, + "waterlevel_time": "2025-11-06T14:00:00", + "siren_level": "N", + "siren_time": "2025-11-06T14:00:00" + } +] +``` + +### GET `/api/station/rainfall` + +Returns latest rainfall data for rainfall-enabled stations. + +### GET `/api/station/waterlevel` + +Returns latest water level data for waterlevel-enabled stations. + +### GET `/api/station/notification` + +Returns today's latest notification per station. + +### GET `/api/station/history` + +Returns notification history for the last 3 days (latest per station per day). + +### GET `/api/station/siren` + +Returns current siren status for siren-equipped stations (last 3 days). + +### GET `/api/station/siren/history` + +Returns siren history for the last 3 days (excluding Normal level). + +--- + +## Alert (Push Notification) + +### POST `/api/alert` + +Sends an FCM push notification. Called by the Python autoscript. + +**Request Body:** +```json +{ + "stationid": "KBLG0026", + "level": "Warning", + "stationtype": 1 +} +``` + +**Parameters:** + +| Field | Type | Description | +|-------|------|-------------| +| `stationid` | string | Station identifier | +| `level` | string | Alert level: "Alert", "Warning", "Danger" | +| `stationtype` | integer | 1=Rainfall, 2=Water Level, 3=Siren | + +**Response:** +```json +{ + "status": 200 +} +``` + +The `status` is the HTTP status code returned by the FCM API (200 = success). + +--- + +## Public Web API + +### GET `/stations` + +Returns all stations with coordinates as JSON (used by the public map). + +### GET `/locale/{locale}` + +Switches language. Valid values: `en`, `bm`. Redirects back to previous page. diff --git a/docs/08-CONFIGURATION.md b/docs/08-CONFIGURATION.md new file mode 100644 index 00000000..f3bfbcb0 --- /dev/null +++ b/docs/08-CONFIGURATION.md @@ -0,0 +1,119 @@ +# Configuration Reference + +## Environment Variables + +### Application Settings + +| Variable | Default | Description | +|----------|---------|-------------| +| `APP_NAME` | `SIDES` | Application name | +| `APP_ENV` | `local` | Environment (local/production) | +| `APP_KEY` | (generated) | Encryption key | +| `APP_DEBUG` | `true` | Show debug errors | +| `APP_URL` | `https://sides.tck.com.my` | Application URL | +| `APP_LOCALE` | `en` | Default language | +| `APP_TIMEZONE` | `Asia/Kuala_Lumpur` | Timezone (set in config/app.php) | + +### Database + +| Variable | Value | Description | +|----------|-------|-------------| +| `DB_CONNECTION` | `pgsql` | Database driver | +| `DB_HOST` | `192.168.0.211` | Database host (external IP) | +| `DB_PORT` | `5432` | PostgreSQL port | +| `DB_DATABASE` | `sides_db` | Database name (via `POSTGRES_DB`) | +| `DB_USERNAME` | `tck` | Database user (via `POSTGRES_USER`) | +| `DB_PASSWORD` | `projectdev##1` | Database password (via `POSTGRES_PASSWORD`) | + +### Docker-level Database + +| Variable | Value | Description | +|----------|-------|-------------| +| `POSTGRES_DB` | `tckdev` | Docker postgres database name | +| `POSTGRES_USER` | `tck` | Docker postgres user | +| `POSTGRES_PASSWORD` | `projectdev##1` | Docker postgres password | + +**Note**: There is a discrepancy — Docker `.env` creates database `tckdev` while Laravel `.env` connects to `sides_db` on an external host. + +### Firebase / FCM + +| Variable | Description | +|----------|-------------| +| `FIREBASE_PROJECT_ID` | `sides-b4abb` | +| `FIREBASE_CREDENTIALS` | `storage/app/firebase/sides-b4abb-3604a7cf7584.json` | +| `FCM_TOPIC_RAINFALL_WARNING` | `rainfall_warning` | +| `FCM_TOPIC_RAINFALL_DANGER` | `rainfall_danger` | +| `FCM_TOPIC_WATERLEVEL_ALERT` | `waterlevel_alert` | +| `FCM_TOPIC_WATERLEVEL_DANGER` | `waterlevel_danger` | + +### Mail (SMTP) + +| Variable | Value | Description | +|----------|-------|-------------| +| `MAIL_MAILER` | `smtp` | Mail driver | +| `MAIL_HOST` | `smtp.gmail.com` | Gmail SMTP | +| `MAIL_PORT` | `587` | SMTP port (TLS) | +| `MAIL_USERNAME` | `sideskupang@gmail.com` | Gmail account | +| `MAIL_PASSWORD` | `ipmu zifw bpmf fsyp` | Gmail App Password | +| `MAIL_FROM_ADDRESS` | `sideskupang@gmail.com` | From address | +| `MAIL_FROM_NAME` | `${APP_NAME}` | From name | + +### Session & Cache + +| Variable | Value | Description | +|----------|-------|-------------| +| `SESSION_DRIVER` | `database` | Session storage | +| `SESSION_LIFETIME` | `120` | Session timeout (minutes) | +| `CACHE_STORE` | `database` | Cache storage | +| `QUEUE_CONNECTION` | `database` | Queue driver | + +## Docker Compose Configuration + +### Networks + +- `tckdev_net` — Custom bridge network for all containers + +### Volumes + +| Volume | Container | Purpose | +|--------|-----------|---------| +| `./src:/var/www/html` | app, web | Laravel application code | +| `./docker/postgres/data:/var/lib/postgres/data` | postgres | PostgreSQL data | +| `./docker/nginx/default.conf:/etc/nginx/conf.d/default.conf` | web | Nginx config | +| `./backup:/var/lib/pgadmin/storage/...` | pgadmin | pgAdmin backup storage | + +### Ports + +| Host Port | Container Port | Service | +|-----------|---------------|---------| +| 80 | 80 | Nginx (web) | +| 5432 | 5432 | PostgreSQL | +| 5050 | 80 | pgAdmin | +| 6060 | 8080 | Adminer | + +## Nginx Configuration + +Located at `docker/nginx/default.conf`: + +- Root: `/var/www/html/public` +- PHP processing via FastCGI to `app:9000` +- Security headers: X-Frame-Options, X-XSS-Protection, X-Content-Type-Options +- Laravel URL rewriting: `try_files $uri $uri/ /index.php?$query_string` +- Hidden file access denied (except `.well-known`) + +## Laravel Configuration + +### Middleware Registration + +Registered in `bootstrap/app.php`: + +- `admin` → `AdminMiddleware::class` (alias) +- `LocalizationMiddleware` appended to web middleware group + +### HTTPS Forcing + +`AppServiceProvider::boot()` calls `URL::forceScheme('https')` globally, making all generated URLs use HTTPS. + +### Timezone + +Set to `Asia/Kuala_Lumpur` in `config/app.php` (UTC+8, Malaysia Time).