Architecture & design

Service-oriented layout for the SuperSet Telegram Notification Bot. Data comes in from SuperSet, email groups, and official pages. Notifications go out on Telegram and Web Push. Daemon mode for production, FastAPI when something external needs to poke the system.

The repository follows a layered, feature-based organization:

• Core: configuration and daemon utilities
• Clients: low-level integrations (MongoDB, Telegram API, Google Groups IMAP)
• Services: domain services implementing business logic (notifications, placement processing, official data scraping)
• Servers: HTTP servers (Telegram bot, webhook, scheduler)
• Runners: orchestration modules for update and notification dispatch
• Data: local JSON datasets and MongoDB storage artifacts

graph TB
subgraph "Core"
CFG["Config (Settings)"]
DAE["Daemon Utils"]
end
subgraph "Clients"
DB["DBClient (Mongo)"]
TG["TelegramClient (HTTP)"]
GG["GoogleGroupsClient (IMAP)"]
end
subgraph "Services"
DBS["DatabaseService"]
NTF["NotificationService"]
TGS["TelegramService"]
PLS["PlacementService"]
ENS["EmailNoticeService"]
OPS["OfficialPlacementService"]
end
subgraph "Servers"
BOT["BotServer (Telegram)"]
WHS["WebhookServer (FastAPI)"]
SCH["SchedulerServer (APScheduler)"]
end
subgraph "Runners"
UPR["UpdateRunner"]
NOR["NotificationRunner"]
end
CFG --> BOT
CFG --> WHS
CFG --> SCH
DAE --> BOT
DAE --> SCH
DB --> DBS
TG --> TGS
GG --> PLS
GG --> ENS
DBS --> NTF
TGS --> NTF
PLS --> DBS
ENS --> DBS
OPS --> DBS
UPR --> PLS
UPR --> ENS
NOR --> NTF
BOT --> NTF
WHS --> NTF
SCH --> UPR
SCH --> NOR

• Configuration and Environment: centralized settings with typed validation and logging initialization
• Daemon Utilities: process forking, PID management, and graceful shutdown
• Clients:
• DBClient: MongoDB connection and collection access
• TelegramClient: Telegram Bot API wrapper with retries and rate-limit handling
• GoogleGroupsClient: IMAP-based email fetching with forwarded metadata extraction
• Services:
• DatabaseService: MongoDB operations for notices, jobs, placement offers, users, policies, and official data
• NotificationService: channel-agnostic orchestrator for broadcasting to Telegram and Web Push
• TelegramService: Telegram channel implementation with formatting and broadcasting
• PlacementService: LLM-powered placement offer extraction pipeline with privacy sanitization
• EmailNoticeService: LLM-powered notice classification and extraction with policy detection
• OfficialPlacementService: web scraping of official placement data with deduplication
• Servers:
• BotServer: Telegram bot with commands and user management
• WebhookServer: FastAPI endpoints for push subscriptions, notifications, and stats
• SchedulerServer: APScheduler-based automation for periodic updates and official data scraping
• Runners:
• UpdateRunner: orchestrates fetching and processing from SuperSet and email sources
• NotificationRunner: dispatches unsent notices to channels

SOA with DI. Clients wrap SuperSet, email, and MongoDB. Services hold business logic. CLI starts servers and one-shot scripts; daemon helpers keep processes in the background.

graph TB
CLI["CLI (main.py)"]
CFG["Settings (config.py)"]
DAE["Daemon (daemon.py)"]
subgraph "Data Sources"
SS["SuperSet Portal"]
EM["Google Groups (IMAP)"]
OW["Official Website (JIIT)"]
end
subgraph "Processing"
PLS["PlacementService"]
ENS["EmailNoticeService"]
OPS["OfficialPlacementService"]
DBS["DatabaseService"]
end
subgraph "Delivery"
TGS["TelegramService"]
NTF["NotificationService"]
WHS["WebhookServer"]
end
CLI --> CFG
CLI --> DAE
CLI --> BOT["BotServer"]
CLI --> SCH["SchedulerServer"]
CLI --> WHS
SS --> PLS
EM --> PLS
EM --> ENS
OW --> OPS
PLS --> DBS
ENS --> DBS
OPS --> DBS
DBS --> NTF
TGS --> NTF
NTF --> TGS
NTF --> WHS

DatabaseService#

• Responsibilities: CRUD operations for notices, jobs, placement offers, users, policies, and official placement data; statistical queries; MongoDB connection lifecycle
• Patterns: dependency injection via DBClient; collection delegation; transaction-like operations with error handling and logging
• Persistence: MongoDB collections for each domain entity; deduplication and indexing considerations for notices and placement offers

classDiagram
class DBClient {
+connect()
+close_connection()
+notices_collection
+jobs_collection
+placement_offers_collection
+users_collection
+policies_collection
+official_placement_data_collection
}
class DatabaseService {
+notice_exists(id)
+save_notice(notice)
+get_unsent_notices()
+mark_as_sent(post_id)
+save_placement_offers(offers)
+save_official_placement_data(data)
+get_placement_stats()
+add_user(...)
+get_active_users()
+get_user_by_id(user_id)
+get_users_stats()
+get_policy_by_year(year)
+upsert_policy(policy)
}
DatabaseService --> DBClient : "uses"

NotificationService and TelegramService#

• NotificationService: channel-agnostic orchestrator; broadcasts to enabled channels; tracks unsent notices and marks them after successful delivery
• TelegramService: channel implementation with message formatting (HTML/Markdown), long message splitting, rate limiting, and user broadcasting

classDiagram
class NotificationService {
+channels
+broadcast(message, channels)
+send_to_channel(message, channel_name)
+send_unsent_notices(telegram, web)
+send_new_posts_to_all_users(telegram, web)
}
class TelegramService {
+channel_name
+send_message(message, parse_mode)
+send_to_user(user_id, message, parse_mode)
+broadcast_to_all_users(message, parse_mode)
+convert_markdown_to_html(text)
+convert_markdown_to_telegram(text)
}
NotificationService --> TelegramService : "routes to"

PlacementService (LLM-powered)#

• Implements a LangGraph pipeline for placement offer extraction:
• Classification: keyword-based confidence scoring
• Extraction: LLM-based JSON schema extraction with retry
• Validation: schema validation and enrichment
• Privacy: sanitization of headers and forwarded metadata
• Integrates with GoogleGroupsClient for email ingestion and DatabaseService for persistence

flowchart TD
Start(["Email Received"]) --> Classify["Classify with Keywords"]
Classify --> Confidence{"Confidence ≥ 0.6?"}
Confidence --> |No| Reject["Reject as Non-Relevant"]
Confidence --> |Yes| Extract["LLM Extraction (JSON)"]
Extract --> Validate["Validate Schema & Enrich"]
Validate --> Privacy["Sanitize Privacy"]
Privacy --> Store["Save to DB & Emit Events"]
Reject --> End(["Done"])
Store --> End

EmailNoticeService (LLM-powered)#

• Classifies and extracts structured notices from Google Groups emails
• Detects placement policy updates and delegates to PlacementPolicyService
• Formats notices for consistent delivery and persists to database

sequenceDiagram
participant GG as "GoogleGroupsClient"
participant ENS as "EmailNoticeService"
participant LLM as "LLM (LangGraph)"
participant DB as "DatabaseService"
GG->>ENS : "Unread email IDs"
ENS->>GG : "Fetch email content"
ENS->>LLM : "Classify & Extract"
LLM-->>ENS : "Structured notice or policy update"
ENS->>DB : "Save notice"
ENS-->>GG : "Mark as read"

OfficialPlacementService#

• Scrapes official JIIT placement page, parses tabs and tables, and stores deduplicated results
• Uses DatabaseService for persistence and timestamp updates

flowchart TD
Start(["Scrape Official Page"]) --> Fetch["GET HTML"]
Fetch --> Parse["Parse Tabs & Tables"]
Parse --> Model["Build Pydantic Models"]
Model --> Save["Save to DB (Deduplicate)"]
Save --> End(["Done"])

Servers and orchestration#

• BotServer: Telegram bot with commands and user management; DI for services
• WebhookServer: FastAPI app exposing health, push subscription, notification, and stats endpoints; DI for services
  • SchedulerServer: APScheduler-based automation for periodic updates and official data scraping

sequenceDiagram
participant User as "User"
participant Bot as "BotServer"
participant Ntf as "NotificationService"
participant Tgs as "TelegramService"
participant DB as "DatabaseService"
User->>Bot : "/start"
Bot->>DB : "Add/Activate User"
User->>Bot : "/stats"
Bot->>DB : "Get Placement Stats"
Bot-->>User : "Stats Response"
Note over Bot,Ntf : "Background scheduling triggers updates"
Ntf->>DB : "Get Unsented Notices"
Ntf->>Tgs : "Broadcast to Users"
Tgs-->>Ntf : "Results"
Ntf->>DB : "Mark as Sent"

• Configuration and daemon utilities are foundational and consumed by all servers and CLI commands
• Clients encapsulate external integrations and are injected into services
• Services depend on clients and each other minimally, so tests stay small and modules stay swappable
• Servers orchestrate services and expose APIs; they rely on DI factories to wire dependencies
• Runners coordinate update and notification dispatch, mirroring CLI commands

graph LR
CFG["config.py"] --> BOT["bot_server.py"]
CFG --> WHS["webhook_server.py"]
CFG --> SCH["scheduler_server.py"]
DAE["daemon.py"] --> BOT
DAE --> SCH
DB["db_client.py"] --> DBS["database_service.py"]
TG["telegram_client.py"] --> TGS["telegram_service.py"]
GG["google_groups_client.py"] --> PLS["placement_service.py"]
GG --> ENS["email_notice_service.py"]
DBS --> NTF["notification_service.py"]
TGS --> NTF
PLS --> DBS
ENS --> DBS
OPS["official_placement_service.py"] --> DBS
BOT --> NTF
WHS --> NTF
SCH --> UPR["update_runner.py"]
SCH --> NOR["notification_runner.py"]

• Asynchronous processing: Telegram bot and webhook server use async frameworks; scheduler uses APScheduler for non-blocking jobs
• Retry and backoff: TelegramClient implements exponential backoff for rate-limited responses; PlacementService and EmailNoticeService include retry logic for LLM extraction
• Connection pooling and lifecycle: DBClient manages MongoDB connections; services close connections after operations to prevent leaks
• Rate limiting: TelegramService applies small delays between broadcasts to avoid throttling
• Scheduling cadence: SchedulerServer runs at midnight and hourly 8 AM through 11 PM IST, plus an official scrape at noon IST. Overnight 1 AM through 7 AM is idle.
• Scalability: Modular design allows horizontal scaling of servers and runners; database sharding and indexing can be introduced at the MongoDB layer

[No sources needed since this section provides general guidance]

• Configuration issues: verify environment variables for MongoDB, Telegram, Google API, and VAPID keys; Settings validates and loads from .env
• Daemon mode: PID files and logging are managed by daemon utilities; use status/stop commands to inspect and terminate processes
• Email fetching: GoogleGroupsClient requires proper credentials and app password; IMAP connectivity and folder selection are handled internally
• Telegram delivery: TelegramClient logs failures and retries; check rate limits and parse modes; fallback to plain text if formatted messages fail
• Database connectivity: DBClient tests connection with ping; ensure connection string and network access to MongoDB
• Scheduler jobs: SchedulerServer logs errors and continues; review logs for specific job failures and adjust schedules as needed

Clients fetch, services process, runners orchestrate, servers talk to the outside world. Daemon mode, FastAPI, and APScheduler cover how it actually runs. DI is why you can test a service without standing up Telegram.