Tomorrowland Video Proxy Pipeline
A production-grade, distributed video processing pipeline built to process 500 TB of RAW festival footage across 16 stages with zero job loss.
Designed and built an event-driven, distributed video processing architecture to handle over 500 TB of RAW footage captured across 16 stages during Tomorrowland. The system decouples workload management into an Orchestrator, Worker fleet, and Monitoring Dashboard via RabbitMQ, allowing horizontal scaling tailored to match live festival demands.
To support live editing workflows, the pipeline features intelligent multi-tier prioritization: lightweight H.264 proxies generate immediately for rapid review, followed by high-quality Apple ProRes 422 proxies delivered within 5 minutes. A format normalization layer standardizes incoming codecs from dozens of camera types, eliminating processing failures at scale.
Reliability was engineered using a transactional outbox pattern to guarantee zero job loss, complemented by live worker heartbeats and hardware telemetry. In internal benchmarking, the pipeline outperformed the existing third-party solution by ~55% in processing speeds and is currently being evaluated for full production adoption.
