Mera — The Digital Health Companion

Team Syntax Error (Lead: Yao Xiang) · Synapxe × IMDA AI Innovation Challenge · Mar 2026

Next.js 16 FastAPI SEA-LION v3.5 MERaLiON ASR/SER MediaPipe WASM WebRTC SQLite Docker Telegram Bot API

View Repo 💬 Try on Telegram (@Meramerarabot)

Mera is a fully functional AI digital health companion deployed entirely inside Telegram, targeting aged and chronically ill patients. Instead of waiting for a reactive medical crisis, Mera conducts continuous background multimodal screening — fusing voice biomarkers, gamified clinical assessments, and vision-language document analysis — powered exclusively by Singapore's sovereign AI models: SEA-LION and MERaLiON.

100%

Smile Checker Precision
(0 False Positives)

92–98%

Mobility FSM Accuracy
(Sit-Stand, March, Shoulder)

3.70

rPPG Heart Rate MAE
(0.93 Pearson Correlation)

478-pt

Facial Mesh Density
(Zero Server Upload)

🎯 Problem Statement

Tackling PS2: AI for Multimodal Remote Health and Wellness Monitoring

Conventional healthcare apps fail aged or chronically ill patients due to UI friction — installations, logins, unfamiliar navigation. Mera solves this by deploying entirely inside Telegram, an app patients already use daily. By intelligently fusing Audio & Linguistic analysis (passive tracking) with Visual Pose & Facial tracking (active assessment), we form a comprehensive, frictionless health trendline for clinicians.

⚙️ The "Open Claw" Sovereign AI Stack

The entire intelligence layer runs on Singapore's own sovereign LLMs — no foreign cloud inference, no data leaving the host.

Category	Technology	Usage
Core Framework	Next.js 16 (React 19), FastAPI, Python	Full-stack orchestration and bot back-end
Intelligence (Chat)	SEA-LION v3.5-8B-R (Ollama)	SEA-context reasoning, health synthesis, Q&A
Intelligence (Speech)	MERaLiON ASR 2-3B + SER v1	Voice transcription & acoustic biomarker extraction
Intelligence (Vision)	SEA-LION v4-4B-VL (Gemma)	Photo & PDF medical document analysis
Clinical CV (Edge)	MediaPipe, WASM, WebRTC	Zero-leakage client-side 478-pt facial & 33-pt skeletal tracking
Storage & Infra	SQLite, Docker, FFmpeg, Ngrok	Secure local logging & sandboxed orchestration

🏗️ System Architecture

All inference runs locally behind the "Open Claw" Dockerized sandbox. The LLM has tool-calling capabilities but is strictly containerized to prevent rogue access to the host system.

🎮 The 3 Clinical Edge-Games (Symptom Checkers)

When anomalies are detected — elevated fatigue score, negative sentiment drift, or explicit user report — Mera deploys WebRTC Telegram Mini-Apps that execute entirely on the edge (WASM) for zero data leakage. Raw camera frames never leave the patient's device.

😊 Smile Checker — Stroke / Bell's Palsy Screening

Tracks a dense 478-point facial mesh in-browser. Computes real-time Euclidean distance ratios between lip corners relative to the nose tip to measure mouth symmetry. A clinically safe FSM enforces a sustained 3-second symmetry window at ≥70% expansion before logging a valid reading — biasing hard toward false negatives to prevent medical panic.

✅ Empirical Result: 100% Precision — Zero False Positives

🏋️ Mobility Workout — Motor Decline / Fall Risk

Maps 33 3D posture landmarks using three rigorous Finite State Machine exercises: Sit-to-Stand (lower-body strength), Shoulder Raise & Extension (rotator cuff flexibility, frozen shoulder detection), and Standing March (balance & cross-lateral motor control). Joint angles calculated via 3D dot-product vector math with strict angular thresholds.

✅ Empirical Results: 92% Sit-Stand · 96% Shoulder Raise · 98% Standing March

❤️ Heart Rate Game — Remote Photoplethysmography (rPPG)

Extracts resting BPM non-invasively from subtle skin color variations captured by the device camera. Samples red channel pixel intensity over a 20-second window, applies bandpass filtering (0.5–4 Hz) to isolate cardiac frequencies, and counts peaks using actual elapsed time (not assumed FPS) to calculate BPM. Rejects readings outside 30–220 BPM or with insufficient variance.

✅ Empirical Result: 3.70 MAE vs clinical oximeter · 0.93 Pearson Correlation

🔄 The Agentic 4-State Flow (FSM)

Mera operates as a continuous state machine, escalating from passive monitoring to active clinical intervention and back — fully autonomously.

stateDiagram-v2 direction LR state "State 0: Passive Monitoring" as S0 state "State 1: Action - Mini-App Deployed" as S1 state "State 2: Analysis & Data Wrangling" as S2 state "State 3: Feedback & Intervention" as S3 [*] --> S0 S0 --> S1 : Anomaly Detected / Scheduled Timer S1 --> S2 : Metrics POSTed to FastAPI S2 --> S3 : SEA-LION Analysis Complete S3 --> S0 : Feedback Sent to Patient

State 0: Empathic Surface

Default companion mode. Processes text and voice notes continuously via MERaLiON ASR/SER. If inactivity is detected, the scheduler triggers a proactive warm check-in message.

State 1: Gamified Assessment

An anomaly (e.g., fatigue spike in voice, negative sentiment trend) triggers a Telegram Mini-App deployment. Voice-guided FSM transitions ensure eyes-free accessibility for elderly users.

State 2: Trend-Aware Wrangling

Scores are POSTed to FastAPI. SQLite returns a rolling N=20 window of historical stats (mean, median, std, min/max) which are injected alongside current scores into the SEA-LION prompt.

State 3: 4-Part Clinical Review

SEA-LION generates a structured report: (1) Past vs. current comparison, (2) Medical information & awareness, (3) Factual actionable suggestions, (4) Grounded CDC/NIH/AHA source links.

🔊 Voice Pipeline: MERaLiON → SEA-LION

Voice notes flow through a local acoustic ML pipeline before SEA-LION synthesizes a final empathetic response fusing what was said (ASR transcript) with how they sound (SER biomarkers).

sequenceDiagram participant TG as Telegram participant BOT as bot.py participant VPIPE as ml_bridge.py participant FFM as FFmpeg participant ASR as MERaLiON ASR (2-3B) participant SER as MERaLiON SER (v1) participant LLM as SEA-LION (Voice Fusion) TG->>BOT: voice note (.ogg) BOT->>VPIPE: analyze_voice(file) VPIPE->>FFM: .ogg → 16kHz mono .wav FFM-->>VPIPE: wav file VPIPE->>ASR: wav → transcript ASR-->>VPIPE: transcript text VPIPE->>SER: wav → VAD + fatigue_score SER-->>VPIPE: biomarker scores VPIPE-->>BOT: {transcript, fatigue_score, vad} BOT->>LLM: fuse(transcript + biomarkers) LLM-->>BOT: empathetic reply BOT-->>TG: send reply

🎥 Visual Demonstrations (Clinical Demos)

Live gameplay and real-time inference snapshots from the Mera clinical edge-games.

❤️ Heart Rate Extraction (rPPG)

Non-invasive BPM detection via subtle skin color variations.

🏋️ Mobility Workout (FSM)

33-point skeletal tracking for motor decline assessment.

😊 Smile Check (Positive)

Successful symmetry detection (Stroke/Palsy screening).

😟 Smile Check (Negative/Baseline)

Baseline capture and facial asymmetry detection.

🚀 Scalability: The Innovation Blueprint

Modular Skill Architecture

The "Open Claw" tooling lets the LLM infinitely extend without changing the conversation engine. New tools — hospital booking APIs, Parkinson's spiral-draw tests, IoT wearable telemetry — plug in as new skills.

Cloud-Native Transition

The Dockerized Next.js / FastAPI / LLM architecture is enterprise-ready. Transitioning from local SQLite + Ollama to a SYNAPXE-hosted GPU cluster requires no architectural changes — only config updates.

Zero Friction Adoption

No app install. No account. No onboarding. Patients message @Meramerarabot in the Telegram app they already use — the entire clinical engagement surface lives in one conversation window.

Try Mera Live

The bot runs on our local inference server, guaranteed active 7–9 PM daily till 5th Apr.

💬 Open @Meramerarabot View GitHub Repo →