This is a high-performance, asynchronous file storage backend designed to leverage Telegram’s cloud infrastructure as a persistent storage layer. Unlike simple wrapper scripts, this system utilizes a service-oriented architecture (SOA) to decouple metadata management from physical storage, enabling features such as virtual file systems, two-way synchronization, and high-concurrency streaming.

The system is designed for self-hosted deployment on Virtual Private Servers (VPS), utilizing a containerized environment to ensure reproducibility and scalability.

The application is architected as a modular monolith. It serves client requests via a REST API while offloading heavy I/O operations (uploads, downloads, synchronization) to background workers.

High-Level Data Flow#

graph TD
    Client[Client Application] -->|HTTPS/REST| Nginx[Reverse Proxy]
    Nginx -->|Proxy| API[FastAPI Server]
    
    subgraph "Data Persistence Layer"
        API -->|Read/Write Metadata| DB[(PostgreSQL)]
        API -->|Cache/Broker| Redis[(Redis)]
        API -->|Search Index| Meili[Meilisearch]
    end
    
    subgraph "Background Processing"
        API -->|Enqueue Task| Redis
        Redis -->|Consume Task| Worker[Celery Worker]
        Redis -->|Consume Task| Sync[Sync Service]
    end
    
    subgraph "Storage Layer (Telegram)"
        Worker -->|MTProto Upload/Stream| TG[Telegram Data Centers]
        Sync -->|Long Poll Events| TG
    end
    
    Sync -->|Write Orphan Files| DB
    Sync -->|Update Index| Meili

Component Description#

1. FastAPI Backend: Handles authentication, file system logic, and metadata management. It does not handle heavy file I/O directly to avoid blocking the event loop.
2. PostgreSQL (Async): Stores the virtual file system structure (using ltree), user data, and the mapping between logical files and Telegram message IDs.
3. Redis: Acts as the message broker for Celery and a caching layer for user sessions and upload progress tracking.
4. Celery Workers: Handle long-running tasks such as file uploads, thumbnail generation, and metadata extraction.
5. Sync Service: A specialized daemon that maintains persistent connections to Telegram channels to detect external file additions (two-way sync).
6. Telegram Client Pool: A rotation mechanism managing multiple bot tokens and user sessions to distribute load and mitigate Telegram’s API rate limits.

The database schema is designed to separate the logical file existence from its physical location.

Entity Relationship Diagram#

erDiagram
    USERS ||--o{ FILES : owns
    USERS ||--o{ FOLDERS : owns
    
    FOLDERS {
        uuid id PK
        string name
        ltree path "Materialized Path"
        uuid parent_id FK
    }
    
    FILES {
        uuid id PK
        string name
        bigint size
        string mime_type
        boolean is_orphan "True if unsorted"
        uuid folder_id FK
        uuid owner_id FK
        timestamp created_at
    }
    
    TELEGRAM_METADATA {
        uuid id PK
        bigint message_id
        bigint channel_id
        string file_unique_id "Deduplication Key"
        uuid file_id FK
    }

    FOLDERS ||--o{ FOLDERS : hierarchy
    FOLDERS ||--o{ FILES : contains
    FILES ||--|| TELEGRAM_METADATA : maps_to

Key Design Decisions#

• ltree Extension: Used for the folders table to enable recursive queries (e.g., “select all children of folder X”) in sub-millisecond time.
• Orphan State: Files detected via the Sync Service are inserted into files with is_orphan=True and folder_id=NULL. This allows them to exist in the system before the user organizes them.
• Physical Separation: The telegram_metadata table allows for potential future migrations (e.g., splitting a large file across multiple messages) without changing the files table ID.

• Language: Python 3.11+
• Web Framework: FastAPI (0.109+)
• Database: PostgreSQL 16
• ORM: SQLAlchemy 2.0 (Async) + Alembic
• Asynchronous Task Queue: Celery 5.3 + Redis 7
• Telegram Interface: Pyrogram (Async MTProto Client)
• Search Engine: Meilisearch
• Containerization: Docker & Docker Compose

This plan breaks down the development into five distinct phases to ensure a robust delivery.

Phase 1: Foundation & Authentication (Week 1)#

Goal: Establish the project structure, database connectivity, and user management.

1. Project Skeleton: Initialize Poetry project, Docker Compose configuration (Postgres, Redis, Web containers).
2. Database Modeling: Implement SQLAlchemy models for User, Folder, and File. Configure Alembic for migrations.
3. Authentication Module:

• Implement POST /auth/register and /auth/login.
• Implement JWT issuance and validation middleware.
• Create get_current_user dependency.

4. Folder Logic:

• Enable ltree extension in Postgres.
• Implement POST /folders (Create) and GET /folders/{id} (List contents).

Phase 2: The Telegram Engine (Week 2)#

Goal: Implement the core logic for interacting with Telegram’s MTProto API.

1. Client Pool Manager: Create the TelegramClientPool class to initialize multiple Pyrogram.Client instances from the configuration.
2. Upload Logic:

• Implement the upload_to_telegram Celery task.
• Logic: Receive file stream -> Chunk -> Upload to Storage Channel -> Return message_id.

3. Download/Stream Logic:

• Implement ByteStreamer class.
• Logic: Accept Range header -> Calculate offset -> Request chunk from Telegram -> Yield bytes to response.

4. File CRUD: Connect the Telegram Engine to the API.

• POST /files/upload: Handling multipart uploads.
• GET /files/{id}/download: Streaming response.

Phase 3: Synchronization Service (Week 3)#

Goal: Implement the “Two-Way Sync” to detect files added directly to Telegram.

1. Event Listener: Create a standalone service (app/services/sync.py) that launches Pyrogram listeners on configured SYNC_CHANNEL_IDS.
2. Ingestion Pipeline:

• On Message event: Extract metadata (Name, Size, MIME).
• Check telegram_metadata for file_unique_id (Deduplication).
• If new: Insert into files as is_orphan=True.

3. Orphan Management API:

• GET /files/orphans: List unsorted files.
• PATCH /files/{id}/move: Logic to assign a folder and remove the orphan flag.

Phase 4: Search & Metadata (Week 4)#

Goal: Integrate Meilisearch for full-text capabilities.

1. Search Infrastructure: Add Meilisearch to Docker Compose.
2. Indexing Service:

• Create signals (SQLAlchemy event listeners) on the File model.
• On INSERT/UPDATE: Push document to Meilisearch index.
• On DELETE: Remove from index.

3. Search API: Implement GET /search with query parameters and filters (type, size range).

Phase 5: Production Hardening (Week 5)#

Goal: Prepare for deployment.

1. Nginx/Traefik Configuration: Set up reverse proxy, SSL termination, and rate limiting.
2. Error Handling: Global exception handlers for standardizing API error responses.
3. Documentation: Generate OpenAPI/Swagger documentation.
4. Optimization: Add database indexes on owner_id, folder_id, and is_orphan.

Prerequisites#

• Ubuntu/Debian VPS (2GB RAM minimum recommended).
• Docker Engine & Docker Compose plugin.
• Domain name pointing to the VPS IP.

Configuration (.env)#

Create a .env file in the root directory:

# Application Secrets
SECRET_KEY=change_this_to_a_secure_random_string
ALGORITHM=HS256
ACCESS_TOKEN_EXPIRE_MINUTES=1440

# Database & Cache
POSTGRES_USER=tgdrive
POSTGRES_PASSWORD=secure_db_password
POSTGRES_DB=tgdrive
REDIS_URL=redis://redis:6379/0

# Telegram Configuration
API_ID=12345
API_HASH=your_api_hash
# Comma-separated bot tokens
BOT_TOKENS=1234:Token1,5678:Token2
# Comma-separated channel IDs (negative integers)
SYNC_CHANNEL_IDS=-10012345678,-10087654321

# Search Configuration
MEILI_MASTER_KEY=secure_master_key
MEILI_HOST=http://meilisearch:7700

Docker Compose Orchestration#

Run the entire stack with a single command:

docker-compose up -d --build

System Services#

1. API Security: All endpoints (except auth) are protected via JWT.
2. Bot Permissions: Bots used in the pool must be administrators in the Storage Channels to read/write messages.
3. Data Privacy: While the metadata is on your VPS, the physical files reside on Telegram servers. Do not store sensitive, unencrypted data if you do not trust Telegram’s storage guarantees.
4. Rate Limiting: The client pool handles distribution, but excessive concurrent uploads may still trigger Telegram’s flood waits. The system is designed to catch FloodWait exceptions and retry automatically via Celery.