BroadcastOps: DevOps in media
How broadcasters can harness the power of change
Tech stacks at TV broadcasters used to rely on custom hardware, often monolithic in architecture, supplied by a few vendors locking down the market. But that has changed: Performant COTS hardware and convergence on IP for media transport have initiated the move to define many services in software rather than specialized hardware.
With the use of Agile methods and DevOps practices, software release cycles are becoming increasingly shorter. By adapting to these changing conditions, broadcasters are taking advantage of the profound technological change.
I. The need
Broadcasting is an industry driven by long lifecycles and almost no downtime. It largely operates with monolithic, tightly coupled architectures, most of which are on-premises. In addition to operational security, data security needs are high – the personal data of whistleblowers that may be stored could turn out to be life-threatening. In this environment it is a challenge to stay ahead of the curve. After all, integrations into decades-old systems must be maintained in tightly regulated spaces.
Broadcasters are hard-pressed to keep up with the increasing pace of software lifecycles imposed by manufacturers pivoting to DevOps methods – i.e., the collaborative or shared approach to the tasks performed by the company’s application development (dev) and IT operations (ops) teams. In its most narrow interpretation, DevOps describes the adoption of iterative software development, automation, and programmable infrastructure deployment and maintenance.
The increase in COTS hardware performance along a growing maturity of software implementations has made available a large class of "software defined" products, solving problems that previously required expensive custom hardware. Software defined vision mixing, audio mixing, video routing, graphics playout, channel playout – there is a piece of software that does it. Thus, broadcasters are not simply having to increase adoption speed for software that they already have – they also face the challenge of replacing traditional hardware with "COTS plus software-defined" solutions.
II. Defining BroadcastOps
Let's define what we'll call BroadcastOps. The idea is to iterate through given methods of the Agile framework as well as DevOps best practices and consider how they might apply in a broadcast environment to help adapt to the increasing speed of the software landscape.
1. Continuous Integration and Development
Continuous Integration and Continuous Deployment (or CI and CD), the on-going task of largely automatic testing of design changes and their deployment to production environments, is something that is almost never done in top-down systems design at broadcasters. Strict Requirements Engineering is the norm, disregarding the shortcomings of this model, especially in organizations where stakeholders are as numerous and diverse as in media publishing houses.
Changes must be continuously integrated into the broadcast production chain. This move goes away from having to flick one big switch at the end, replacing and integrating components on a smaller level. Think "Microservices at the infrastructure level".
Clearly, to make this viable, systems must be automated to the point where version up-and-down steps can be run at least overnight, as opposed to time-consuming year-long projects for version changes.
Here are a few specific ideas to where this can apply at broadcasters:
- Automated testing for network configurations, possibly using a digital twin such as GNS-3 or EVE-NG (network simulators)
- Automated route-checker to test if video streams can reach their target, again digital-twining for video routing
- Essentially, running a network simulation and testing changes against it after having defined which endpoint must talk to whom – like a digital twin
2. Infrastructure as Code
Systems are almost never defined in machine-readable manner. CAD drawings created manually, excel sheets maintained manually, maybe there are some config databases, but the data is rarely used operationally. This data must be made usable for orchestration.
- Automatic documentation diagrams using declarative methods that could result from Infrastructure-as-Code (IaC) definitions (e.g., Mermaid.js diagrams)
- Manage and use machine-readable data about endpoints to provision and configure cloud resources, when needed, based on IaC best-practice.
