03 · Case Study Repository: zephyr-scripts/sgbiz-sentiment-tracker

Internal Product / Production Pipeline Automated Intelligence

SGBiz Pulse

Automated Business Intelligence Workflow

Automated research combining deterministic heuristic scoring with LLM-assisted synthesis, engineered for auditable executive monitoring across fragmented Singapore commercial sectors.

Core Discipline Deterministic + AI Pipeline Engineering

Verification Paradigm Zero Subjective LLM Scoring / Full Audit Trace

Output Cadence Scheduled Daily Digest · Material Change Triggers

“From noisy source material to structured, auditable intelligence.”

Architecture & Pipeline Blueprint

Deterministic + AI Intelligence Pipeline

Ingestion Deterministic Gate Synthesis

Lane 01 · Ingestion & Enrichment input

Multi-Source Pull Composio MCP

Automated fetch across Reddit forums (r/singapore, r/singaporefi etc) & targeted regional business web indices.

arrow_downward

Raw Candidate Ingestion Raw Capture

Unprocessed payload serialization, capture timestamps, metadata headers, and author footprints stored in immutable stage tables.

arrow_downward

External Link Scraping Apify Crawler

Dereferencing shared news URLs, resolving canonical payloads, bypassing soft paywalls, and extracting primary body copy.

Lane 02 · Deterministic Gate tune

Relevance & Noise Filter Rule-Based

Keyword-vector heuristic elimination of generic chit-chat, personal rants, spam, and non-commercial local discourse.

arrow_downward

Event Clustering & Dedupe Text Distance

Collapses multiple threads of community posts discussing the exact same corporate merger or restructuring into 1 distinct event entity.

arrow_downward

Deterministic Scorecard Mathematically Proven

Engagement-weighted velocity, independent source counts, and corporate entity mentions. LLMs are strictly forbidden from generating scores.

Lane 03 · Synthesis & Distribution send_and_archive

AI-Assisted Brief & Watchlist Constrained LLM

Grounded factual extraction restricted strictly to verified candidate tokens. Signals classified: New, Watch, or Easing.

arrow_downward

Scheduled Owner Digest Daily Dispatch

Clean executive dispatch delivered via email / webhook only when novel or materially altered enterprise signals exceed threshold.

arrow_downward

Target Distribution Portal & Feeds

SGBiz Pulse executive console, queryable raw SQL warehouse tables, and automated alerting channels.

dns

Underlying Cross-Cutting Infrastructure Ensuring zero hallucination, repeatable execution, and continuous monitoring

Full Decision Trace Log Postgres pg_cron Engine Offline Replay Test Rig Health & Observability Probe

01

The Challenge

Recurring business intelligence in high-tempo markets is severely undermined by signals scattered across disorganized digital forums, regional aggregators, and industry news outlets. For strategy and transformation leaders, attempting to monitor market shifts through manual tracking or simplistic social listening tools fails due to three systemic data-quality barriers:

filter_none Redundant Echo Chambers

Dozens of social posts and syndications reiterate the exact same singular press release or local rumor, distorting raw frequency calculations into artificial urgency.

trending_down Deceptive Metrics

Public engagement (upvotes, emotional forum commentary) is frequently mistaken for independent commercial evidence, conflating consumer controversy with actual business impact.

history_edu Hallucinatory Summaries

Standard GenAI wrappers summarize unvetted collections without citations, inventing subjective scores and leaving strategic decision-makers with zero evidentiary auditability.

02

Thinking & Rationale

The foundational philosophy of SGBiz Pulse is deliberate architectural decoupling. Rather than feeding raw forum text directly into an LLM and asking it to both analyze and score the situation, the system bifurcates qualitative synthesis from deterministic calculation.

Decoupled Processing Architecture

Step 01 Collection Raw Candidates

Step 02 Evidence Article Scraping

Step 03 Deterministic Formulaic Scorecard

Step 04 AI Synthesis Grounded Briefing

Step 05 Publication Material Change Only

Immutable Evidence Ledger Retain every candidate payload and query response in raw format. Any business intelligence brief can be audited backward to the precise timestamp, original URL, and scraper snapshot.

Zero Hallucinated Metrics Velocity, volatility, and sentiment metrics are calculated purely using statistical algorithms and lexical models—never by prompting a model with “Rate this signal from 1-10.”

03

Implemented Pipeline

The production pipeline executes continuously in Supabase, orchestrated by automated chronologies that transition unprocessed web signals into boardroom-ready intelligence.

STAGE 01

Tracked Entity & Topic Sweeper

Composio MCP integration runs targeted querying across Singapore corporate ecosystems, tracking regulatory announcements, corporate restructuring, real estate occupancy rates, and banking sentiment.

Hourly Run

STAGE 02

Deep Link Dereferencing (Apify)

If an ingested candidate points to an external primary investigative article (e.g. Business Times, Straits Times, SGX filings), a headless crawler scrapes the target content, validating that the forum discussion reflects actual reportage.

Event Triggered

STAGE 03

Heuristic Deduplication & Event Clustering

Applies strict string-distance and canonical URL deduplication. When multiple posts aggregate around one occurrence, they are gathered into a single Event Container, preventing repetitive alert spam.

Automated

STAGE 04

Audit-Logged Novelty Decision Gate

Before triggering any notification, the pipeline checks historical vectors. A signal is published strictly if it establishes a brand-new corporate development or alters an existing position. Every dismissal decision is preserved in the audit log.

Gate Keeper

04

Signal Taxonomy & Scorecard

All detected intelligence items are categorized into one of three strict state machines. This ensures readers immediately understand the operational relevance without reading redundant updates.

New / Changed fiber_new

Emerging Structural Shifts

Brand-new regulatory filings, executive departures, unconfirmed institutional acquisitions, or sudden sector-wide pricing shifts that require primary evaluation.

Action: Immediate Digest Priority 1

Continuing Watch visibility

Ongoing Situation Monitoring

Known developments with steady metric accretion. Updates are bundled into daily periodic wraps rather than interruptive ad-hoc alerts.

Action: Scheduled Wrap Priority 2

Easing / Contradicted check_circle

Resolution or False Rumor

Prior concerns refuted by verified regulatory releases, official corporate responses, or diminishing community velocity. Automatically de-escalated.

Action: Archive with Note Resolved

analytics Deterministic Scorecard Output Sample

EVENT ID: SG-2024-089A

Singtel · Regional Data Centre Consortium Expansion Clustered from 8 distinct forum nodes + 2 verified press releases

Status: New / Changed

Raw Candidates 14 Posts

Unique Domains 3 Sources

Velocity Score 4.8x baseline

Audit Verification Deterministic 100%

Synthesized Briefing: Joint venture discussions verified regarding AI-tier colocation facilities in Tuas. Initial forum leaks corroborated by SGX announcement 14:00 SGT. Low commercial risk, high strategic relevance for infrastructure allocators.

05

System & Technical Specification

Constructed with a lean, reliable systems architecture designed for zero maintenance overhead, automated recovery, and strict data governance.

Frontend Application React · TypeScript · Vite

Zero-latency dashboard with real-time signal stream filters, cluster drill-down explorer, and raw payload audit modal.

Backend & Database Supabase Postgres · Edge Functions · RLS

Relational schema isolating raw candidates from verified events. Strict Row Level Security policies governing enterprise multi-tenancy.

Scheduling Engine Postgres pg_cron

In-database job dispatch triggering ingestion, deduplication sweeps, and digest compilation without separate external cron servers.

External Connectivity Composio MCP & Apify Gateway

Standardized Model Context Protocol connections for authenticated forum access and distributed scraping actors.

science Offline Replay Test Rig & Decision Audit Log

To verify pipeline integrity when updating prompt templates or heuristic thresholds, SGBiz Pulse includes an offline test harness. Historical candidate snapshots can be played back deterministically to assert that deduplication logic and LLM synthesis outputs meet strict accuracy bounds without invoking live internet sources.

check Deterministic Replay

check Prompt Drift Guard

check Traceable Decisions

06

Outcomes & Strategic Value

A Repeatable Intelligence Workflow

SGBiz Pulse successfully converts high-volume, fragmented, and duplicated social source material into a structured, owner-level monitoring surface. Strategic stakeholders receive distilled, high-signal alerts while retaining absolute transparency into the underlying evidence chain.

Core Capabilities Demonstrated

account_tree

Beyond Simple API Chaining Orchestrating stateful, multi-tier data workflows with intermediate persistence rather than fragile single-shot completions.

calculate

Combining AI with Deterministic Logic Restricting LLM responsibilities to qualitative text synthesis while assigning scoring and filtering entirely to verifiable algorithms.

verified_user

Evidence Quality & Auditability Building enterprise confidence by ensuring that every summarized claim links back directly to timestamped, scraped proof.

inventory_2

Productising Recurring Research Replacing manual, recurring analyst sweeps with continuous autonomous operations governed by precise health checks.