How I built a fully automated pipeline for iartshorts — with a custom Python orchestrator.

🔒 100% offline, zero cloud. Everything runs locally: LLMs, image generation, video, TTS, Wikipedia search. No external API calls, no data leaves my server.

The Project: Bringing Paintings to Life

iartshorts (and aiartshorts in English) is a vertical shorts channel that transforms portrait paintings into talking videos. The person in the painting animates, tells their story, and references the artist who painted them — all in a slightly cheeky tone, spoken in first person.

The goal: automate everything. From historical research to uploading on YouTube Shorts, TikTok, and Instagram Reels. One single human intervention during processing : validation.

But for that to work, you need hardware.

🖥️ Homelab AI Workstation

The Software Stack

Everything runs on Proxmox since it’s also my homelab. A single Ubuntu LXC container hosts everything that needs GPU access.

The LLMs

Models are managed by llama-swap + a custom Python script that generates the config from my model storage folder. Services run in Docker containers (vLLM, SGLANG, llama.cpp).

ComfyUI

All image/video generation goes through ComfyUI, with:

• Flux Klein 9b for painting restoration and realistic transformation
• Qwen TTS for speech synthesis
• LTX2.3 for video lip-sync generation (AudioSync)
• Wan2.2 for transitions and animations

The Pipeline:

The Batch Processing Trick

Instead of processing each painting end-to-end, I process Step 1 for ALL paintings, then Step 2 for all, etc.

This drastically reduces GPU loading times: one warmup per step instead of 10 × N. It’s a big gain on total processing time.

The State Machine

Each painting is tracked in a SQLite database with a precise state: PENDING → PROCESSING → COMPLETE / FAILED / ABORTED / REDO.

The system supports automatic retries (×2) and resuming from any step. A crash at step 7 on 50 videos doesn’t mean redoing everything.

Custom ComfyUI Client

My pipeline uses a custom HTTP + WebSocket client to interact with ComfyUI, with real-time progress tracking and polling fallback. Between batches, ComfyUI is automatically restarted via SSH (systemctl restart comfyui) to prevent memory leaks.

The Human-in-the-Loop: Honest Numbers

Contrary to what you might think, the pipeline doesn’t produce « plug-and-play » content. Every day, I manually review each video, one by one.

Rejection Rates

Out of 30,000 paintings in my database (free-rights photos), a random draw picks ~20 paintings from a predefined list meant to correspond to portraits each evening.

In the morning: ~10 validated videos out of ~20 started.

💡 Pre-filtering is as critical as the pipeline itself. Better to reject 30% before wasting GPU time. Vision AI analysis covers part of the filtering (nudity, child, single portrait), but some criteria (blur, character size, head orientation) still require the human eye.

Why Local? The Energy Argument

Cloud (closed) models are better. But here, I’m only limited by electricity costs.

In France:

• ✅ Electricity is cheap
• ✅ Low carbon (nuclear)
• ✅ GPUs run at night when energy is even cheaper
• ✅ Full independence: no API credits, no request limits
• ✅ Zero marginal cost per generated video

💰 Average Cost Per Video: €0.0193 (~1.9 ct) — at off-peak rate of €0.1579/kWh (2.9 ct if we take into account the 30% reject rate)

Lessons Learned

1. SQLite state management is essential. Without it, a crash = start over. With it, you resume where you left off.
2. Batch-by-step = big speedup. One GPU warmup per step instead of one warmup per painting per step.
3. Automatic ComfyUI restart. Via SSH between batches — it’s the only reliable way to avoid memory leaks over multi-hour runs.
4. Offline search via ZIM. No network dependency for Wikipedia data. Faster, more reliable, more private.
5. 30% rejection is the current reality. Video models still generate too many artifacts for 100% automation. Human-in-the-loop is non-negotiable for quality content.

Licenses