Cross-Border Scraping Dispatch & Notification Hub
Centralized scraping task management with Microsoft Teams webhook orchestration
An internal cross-team automation hub linking international stakeholders (Japan) with technical scraping engineers (Indonesia), providing structured URL ingestion, automated Microsoft Teams alerts, and artifact delivery tracking.
Problem Statement
Scraping tasks requested by overseas teams were fragmented across emails and chat messages, leading to missed URLs, unclear requirements, and untracked deliverable delivery statuses.
Architecture & Implementation
Designed and deployed a centralized request queue system with structured input schemas, automated Microsoft Teams webhook notifications, task status lifecycle management, and secure result repository.
How The System Operates
Structured Request Ingestion
Japan-side requester submits target websites, extraction fields, and scheduling criteria into a structured form.
Instant Webhook Broadcast
System formats the request payload and triggers an immediate alert directly into the Indonesia engineering team's Microsoft Teams channel.
Task Claim & Execution
Engineers pick up the ticket, run targeted scraping scripts, and validate extracted dataset outputs.
Result Upload & Status Closure
Scraped datasets are uploaded back to the central portal, notifying the requester and archiving the task lifecycle.
Key Capabilities & Highlights
- •Eliminated ad-hoc communication overhead between international business teams
- •Real-time Microsoft Teams webhook notifications with deep links to request details
- •Audited repository of completed scraping deliverables and historic schemas