Job view

Great Stories Processing Flows

"The Great Stories" system has two main processing flows: Asynchronous pipeline (Kafka + workers): For full job processing with webhooks, polling, and API-key quotas. The worker processes jobs, segments text, generates images and audio, and stores assets in S3. Synchronous agents (gRPC + MCP): A separate agents service exposes segmentation, audio, and image generation over gRPC and MCP (Model Context Protocol) with API key auth. External systems can call these directly: gRPC provides all agents (segmentation, audio narration + TTS, image prompt + generation) with protobuf contracts.

MCP API with S3 Asset Handling

MCP (JSON-RPC 2.0 over HTTP) exposes segmenttext, generateimageprompt, and generateimage as tools with schema discovery. Large assets (audio, images) are automatically uploaded to S3 with user-scoped paths (agents/<user_uid>/audio/...) and returned as URLs to avoid message size limits.

Great Stories System Architecture

To understand how "The Great Stories System" functions, we look at the flow of data through three main internal layers: the API Layer, the Processing Layer, and the Data Layer, all interacting with external users and AI services. 1. Job Ingestion and Validation The process begins when an external User or Client sends a POST request to the /v1/jobs endpoint. The API Service acts as the gatekeeper.

First, it validates the request and checks usage quotas.
Next, it creates a job record in the PostgreSQL database and may query for necessary assets.
Once validated and stored, the API Service publishes the task to the Message Queue (running on Kafka or Redpanda).
Finally, it returns a specific Job ID to the user. The user can use this ID to poll the system for status updates while the job processes in the background.

Asynchronous Job Processing and Notification

2. Asynchronous Processing The heavy lifting happens in the Processing Layer. The Worker Service consumes the new message from the queue.

It updates the job status in the database.
To generate the actual content, the Worker Service calls out externally to the Gemini API (Google AI) for LLM generation.
Resulting assets are stored in S3 Storage (which handles Signed URLs).
The worker manages metadata, handles markup, and eventually marks the job as complete in both the message queue and the PostgreSQL database.

3. Completion and Notification Once the processing is finished, the system handles delivery via the Webhook Dispatcher.

The completion event triggers the dispatcher to retrieve the full job details from the database.
The dispatcher then sends a POST Callback to the external User Webhook, proactively notifying the client that the work is done.
Finally, the delivery status of that callback is recorded back into the Data Layer.