BE1024519B1 - Modular software-based video production server, video production server operation method, and distributed video production system - Google Patents

Abstract

A video production server including at least one processor and a storage store is suggested. Software modules composed of executable program code are loaded into a working memory of the at least one processor. Each software module, when executed by said at least one processor, provides a basic service. Concatenation of basic services provides functionality involving the processing of video and / or audio signals required to produce a broadcast program. The video production server includes a set of software components that runs on conventional hardware. Each feature of the video production server is realized using specific unit software that is assembled from reusable functional software blocks and can run on any compatible hardware platform. In addition, a method of operating the video production server and a distributed video production system including the video production server is suggested.

Description

(73) Holder (s):

EVS BROADCAST EQUIPMENT SA

4102, SERAING

Belgium (72) Inventor (s):

BARN ICH Olivier 4000 LIEGE Belgium

BASTINGS Michael 4420 SAINT NICOLAS Belgium

VOUNCKX Johan 3210 LINDEN Belgium (54) Video production server based on modular software, operating method of the video production server and distributed video production system

208 (57) A video production server comprising at least one processor and a storage store is suggested. Software modules composed of an executable program code are loaded into a working memory of said at least one processor. Each software module, when executed by said at least one processor, provides a basic service. A concatenation of basic services provides functionality involving the processing of video and / or audio signals required in order to produce a broadcast program. The video production server includes a set of software components that runs on conventional hardware. Each functionality of the video production server is carried out using specific unitary software which is assembled from reusable functional software blocks and which can run on any compatible hardware platform. In addition, a method of operating the video production server and a distributed video production system including the video production server is suggested.

206205—

204— 203 — L

User applications

User applications

Service Service service | Service Service _n ingestion stccksge encoder j decoder XX

Fig. 2

201 □ □ ...

207

BELGIAN INVENTION PATENT

FPS Economy, SMEs, Middle Classes & Energy

Publication number: 1024519 Deposit number: BE2016 / 5252

Intellectual Property Office International Classification: H04N 21/854 H04N 5/265 H04N 21/226 H04N 21/2343 H04N 21/241 Date of issue: 03/28/2018

The Minister of the Economy,

Having regard to the Paris Convention of March 20, 1883 for the Protection of Industrial Property;

Considering the law of March 28, 1984 on patents for invention, article 22, for patent applications introduced before September 22, 2014;

Given Title 1 “Patents for invention” of Book XI of the Code of Economic Law, article XI.24, for patent applications introduced from September 22, 2014;

Having regard to the Royal Decree of 2 December 1986 relating to the request, the issue and the maintenance in force of invention patents, article 28;

Given the patent application received by the Intellectual Property Office on 12/04/2016.

Whereas for patent applications falling within the scope of Title 1, Book XI of the Code of Economic Law (hereinafter CDE), in accordance with article XI. 19, §4, paragraph 2, of the CDE, if the patent application has been the subject of a search report mentioning a lack of unity of invention within the meaning of the §ler of article XI.19 cited above and in the event that the applicant does not limit or file a divisional application in accordance with the results of the search report, the granted patent will be limited to the claims for which the search report has been drawn up.

Stopped :

First article. - It is issued to

EVS BROADCAST EQUIPMENT SA, Rue Bois Saint-Jean 13, 4102 SERAING Belgium;

represented by

SPEICH Stéphane, Rue des Bruyères 55, 1274, HOWALD;

a 20-year Belgian invention patent, subject to payment of the annual fees referred to in article XI.48, §1 of the Code of Economic Law, for: Video production server based on modular software, operating method of the video production server and distributed video production system.

INVENTOR (S):

BARNICH Olivier, Boulevard d'Avroy 200, 4000, LIEGE;

BASTINGS Michael, Rue de Tilleur 275, 4420, SAINT NICOLAS;

VOUNCKX Johan, Tempelstraat 30, 3210, LINDEN;

PRIORITY (S):

DIVISION:

divided from the basic application: filing date of the basic application:

Article 2. - This patent is granted without prior examination of the patentability of the invention, without guarantee of the merit of the invention or of the accuracy of the description thereof and at the risk and peril of the applicant (s) ( s).

Brussels, 03/28/2018, By special delegation:

DESCRIPTION ^D

Modular software-based video production server, video production server operating method, and distributed video production system

Technical area

The present disclosure relates to a software-based video production server, a method of operating the video production server, and a distributed video production system. The video production server supports different types of functionality that are required for broadcast production.

Context

For live TV production in today's video production environment, very expensive specialized equipment is used for the acquisition, processing and playback of video and / or audio sources or streams from acquisition devices, for example cameras and microphones. Sometimes the video and / or audio sources are combined with associated metadata, for example audio / video metadata or metadata associated with the captured event, for example the speed of athletes, or situational data provided by experts, for example comments on game facts such as goals, fouls, etc. All incoming signals or incoming data must be processed to produce one or more final program output signals. A processing unit that is part of the production equipment implements signal processing. Treatment including:

the ingestion of signals from acquisition devices into processing equipment;

encoding raw signals from the acquisition devices into low bit rate signals;

the derivation of lower resolution copies of the original signal, for example for surveillance or other purposes;

decoding low bit rate signals into the original raw signal; signal transcoding;

recording a signal for later use; applying video effects to signals;

mixing different signals into a single signal; reading signals;

BE2016 / 5252 display of signals.

Much of the processing required is implemented by video mixers 5 and production servers. Video mixers are available, for example, from companies like EVS, Grass Valley, Sony, SAM and Ross. In a typical broadcast production, video mixers and production servers are connected to other devices such as cameras, video storage stores, video effects generators, and other devices. All of these devices are commercially available. Today, all of these devices are made from specialized hardware components. Obviously, the audio signals that accompany the video signals must also be processed during broadcast production.

Dedicated devices operate on these specialized hardware devices in order to implement the specific functions mentioned above. The probability of using all dedicated devices and their maximum power at the same time is very low. This results in an excessive provisioning of the processing power. At the same time, this also causes a great lack of flexibility of the processing infrastructure in a conventional broadcast studio, since different types of productions, for example live sports versus live news, cannot be produced on the same infrastructure unless all the devices necessary for both types of production are available.

Another reason also contributes to the ineffectiveness of conventional broadcast studios. In a conventional broadcast environment, most of the communication between the broadcast equipment devices is accomplished through baseband transmission of video / audio signals using the proprietary SDI format. The devices operate synchronously, according to a shared timing signal (called “Genlock” or “black burst”, or even “reference signal”) which is routed together with the SDI signals. This timing signal is typically a quadratic signal whose rising edges correspond to the arrival of new video frames in the system. As a result, video / audio frames must be processed internally during a fixed time interval, which results in “worst case based” sizing of the infrastructure, since all processing steps must be finalized before the arrival of the next frame. Typical hardware systems require an even finer level of synchronization, down to a line level or even a sub-line level.

US Patent 2009/0187826 A1 describes a data display and control system which controls all production devices in a video production system. The production devices are conventional, but the control system consolidates the functionality of the connected production devices. The control system simplifies the operation of the video production system.

A more recent concept for broadcast productions is distributed video production. This concept attempts to overcome at least some of the limitations imposed by conventional broadcast equipment. The concept of distributed video production is described, for example, in the article by Boutaba et ass. : “Distributed Video Production: Tasks, Architecture and QoS Provisioning”, published in the book “Multimedia Tools and Applications”, Kluwer Academie Publishers, Boston, United States, Volume 16, number 1 to 2.1 ^January 2002, pages 99 to 136. Boutaba and ass. address the problem of delay, or delay, fluctuations in delay and requirements in terms of asymmetry, or obliquity, between supports. Boutaba et al. explicitly state that delay performance is measured on the basis of delay fluctuation or "jitter". The jitter corresponds to a measure of the difference in terms of delay encountered by different packets in a network, under the effect of a variation in terms of buffer memory occupation in the intermediate switching nodes. Another form of jitter relates to inter-stream or “asymmetry” jitter, also called “skew”, which measures the difference in terms of delay as perceived by separate streams belonging to the same application (for example audio and video). To ensure proper intra-flow synchronization, a small delay fluctuation is often required. Boutaba et ass. suggest compensating for jitter by buffering data streams. This requires the provision of sufficient memory capable of storing sufficiently long intervals of video and audio data to compensate for jitter. In the case of high definition video data, this requires a large storage capacity.

Boutaba et ass. rely on different types of servers to provide the different functionality necessary for broadcast production. For each feature, a specific type of server is used. Each specific server is either built exclusively with proprietary hardware, or with standard hardware and acceleration cards

BE2016 / 5252 taking this as a starting loei video, the present disclosure suggests a server offering increased flexibility in production

summary

According to a first aspect, the present disclosure suggests a video production server comprising at least one processor and a storage store. Software modules composed of an executable program code are loaded into a working memory of said at least one processor. Each software module, when executed by said at least one processor, provides a basic service. A concatenation of basic services provides functionality involving processing of video and / or audio signals required in order to produce a broadcast program.

According to one embodiment of the video production server, the software modules are matched on the hardware of the video production server.

In an advantageous embodiment, the video production server is configured as an ingestion server ingesting signals, in particular video and / or audio signals, in the form of streams of data frames from digital devices. acquisition in synchronicity with the output of acquisition devices.

According to one embodiment, the ingestion server is capable of simultaneously ingesting multiple signal streams of different formats.

If the video production server is configured as an ingestion server, it is incoming data.

In another embodiment, the video production server is configured as a playback server outputting video and / or audio signals in the form of data frames in synchronicity with downstream devices receiving the output signals from the video production server. Acquisition devices and downstream devices do not need to have the same synchronicity.

Advantageously, said at least one processor of the video production server implements an asynchronous internal processing of the video and / or audio signals.

In an advantageous embodiment, said at least one processor is configured so as to process the data frames according to the sequence of their timestamps affected by the ingestion modules.

Advantageously, each software module of the video production server has defined data interfaces (API) describing a unitary data format for video data, audio data, metadata and / or control data, at the input level. and the output of software modules, allowing a "user-defined" sequence of elementary sendees. In this case, the "user-defined" sequence of basic services forms at least one processing pipeline for video and / or audio signals and / or metadata.

In an advantageous embodiment, an elementary sendee can be deleted from said at least one processing pipeline, or replaced or inserted into said at least one processing pipeline, without affecting the operation of the rest of the processing pipeline.

In one embodiment, the video production server executes an algorithm that detects hardware failures and automatically matches the software modules so that the software modules are no longer matched on the failing hardware.

In a further embodiment, the video production server includes a storage store allowing direct access to any grain after the grain has been stored, in which the term "grain" is a general term for the smallest item processed in the video production server. For example, a grain is a video frame or an audio frame.

In yet another embodiment, the video production server includes multiple server nodes connected in communication over a network. According to a development of this embodiment, the software modules are dynamically matched on the multiple server nodes with a view to improving the performance of said at least one processing pipeline composed of elementary services provided by the software modules.

According to a second aspect, the present disclosure suggests a distributed video production system comprising at least two broadcast production devices, which are interconnected by a network and are mutually synchronized. At least one of the broadcast production devices is a video production server according to this disclosure.

According to a third aspect, the present disclosure suggests a method of operating this video production server, in which the method comprises the steps below

BE2016 / 5252 consisting of:

defining a sequence of elementary services applied to data frames of received signal streams such as video and audio streams;

process data frames according to the elementary service sequence defined asynchronously, and simultaneously maintain the time relationship of time-stamped data frames of audio and / or video streams.

In an additional embodiment, the method further comprises the step below consisting in:

dynamically modify the sequence of elementary services, from a data frame to the successive data frame, of a received signal stream, by deleting an elementary service from the sequence of elementary services, or by replacing an elementary service in the sequence of elementary services, or by inserting an elementary service in the sequence of elementary services.

In another embodiment, the method further comprises the step below consisting in:

dynamically match the software modules on the hardware platform composed of multiple server nodes in order to improve the performance of the sequence of basic services.

According to a development of this embodiment, the dynamic mapping varies from a data frame to the successive data frame of a received signal stream.

In a final aspect, the present disclosure suggests software containing program code, which, when executed by a processor, implements a method according to the third aspect of the present disclosure.

Brief description of the drawings

In the drawings, exemplary embodiments of the present disclosure are illustrated. Identical or similar elements in the figures are indicated by identical or similar reference numbers. In which figures:

Figure 1 illustrates a schematic structure of a

Ί

BE2016 / 5252 distributed broadcasting production;

Figure 2 illustrates the architecture of a video production server according to

Figure 3 illustrates a visualization of the conceptual relationship between sources, basic services and functionalities;

Figure 4 illustrates the mapping of basic services on a hardware platform;

Figure 5A illustrates one. first basic service pipeline for live production;

FIGS. 5B and 5C represent a schematic illustration of elementary services and of corresponding API interfaces;

Figure 5D illustrates a second pipeline of elementary outputs for live production;

Figure 6 illustrates a pipeline reconfiguration mechanism;

FIG. 7 represents an illustration of the principles of synchronization of the video production server;

FIGS. 8A and 8C show illustrations of an application with integrated security, in other words fault-tolerant, on the video production server;

Figures 9A and 9B illustrate basic service pipelines which are fault tolerant;

FIG. 10 illustrates a flow diagram of the operating method of the video production server;

FIG. 11 illustrates a video production server composed of nodes of interconnected servers: and

Figures 12A and 12B illustrate a mapping of pipelines on the video production server of Figure 11.

; embodiments

Any reference herein to "a single embodiment" or to "an embodiment" means that a specific function, structure or characteristic described in connection with the embodiment may be included in at least one implementation of this solution. The occurrences of the expression “in an embodiment” in different passages of this specification do not necessarily all refer to the same embodiment, and do not necessarily constitute distinct or alternative embodiments, mutually exclusive of other embodiments.

While the present solution may be subject to various modifications and take alternative forms, specific embodiments have been illustrated by way of example in the accompanying drawings and will be described in detail below. However, it should be understood that the present solution is not intended to be limited to the specific forms disclosed.

One or more specific embodiments of the present solution will be described below. In order to provide a concise description of these embodiments, all of the functions of an implementation in progress will not necessarily be described in the specification. It should be appreciated that in the development of any such implementation in progress, for example in any design or engineering project, many decisions specific to the implementation must be made to achieve the specific objectives. developers, for example compliance with system and business constraints, which may vary from implementation to implementation. Furthermore, it should be appreciated that such a development effort, although potentially complex and time-consuming, would nevertheless represent a systematic design and manufacturing activity for the skilled person benefiting from this disclosure.

Certain aspects comparable, in terms of scope, to the disclosed embodiments, are described below. It should be understood that these aspects are presented simply for the purpose of providing the reader with a brief summary of certain forms that the solution might take and that these aspects are not intended to limit its scope. Indeed, the present solution may encompass a variety of aspects which may not be defined below.

Figure 1 shows a schematic illustration of the architecture of a distributed video production system (DVP). The directors and editors work in a local 101 studio, where they create a provisional model of a broadcast program. Some studios are equipped with all of the required high quality broadcast devices that are necessary in order to produce broadcast programs, including complex programs. E§ ^E2 ° ^{16/5252 on the} other hand, the local studio 101 in FIG. 1 is connected, via a broadband broadcasting network 102, to two video production servers only, namely a video server 103 and a server 104. The labels “video server” and “production server” refer to the tasks in the arrangement illustrated in FIG. L In the following, the term “video production server” is used as a generic term for servers performing various tasks in a video production system. The video server 103 stores a database of video sequences, in the form of video streams which are available to the director. The production server 104 activates various functionalities, such as cutting, erasing, attenuation, overlay, etc., making it possible to compose the program output stream according to the creative concept of the director. Local studio 101 is also equipped with cameras and microphones (not shown in Figure 1) that produce video and audio data, for example from a moderator interviewing a guest from the studio. In the case of a previously produced program, these streams are sent from the local studio 101 to the video server 103. This is symbolized, in FIG. 1, by the arrows 105 and 106. In the case of a live program, the video stream produced in the local studio 1 ö 1 is sent directly to the production server 104 (arrows 105, 107), with a view to including it in an edited program stream 108 output by the production server 104. The stream 108 is transferred via the network 102 to the local studio 101 in order to be monitored by the director and to be broadcast. The edited program stream 108 generally also includes video streams stored on the video server 103, which sends these streams (arrow 109) via the network 102 to the production server 104. The director regulates the operation of the video server 103 and the production server 104 by sending instruction signals 110 from the local studio 101.

In the exemplary system illustrated in FIG. 1, only two servers 103, 104 are present. However, other systems may have one or more than two servers, depending on the type of functionality required by the developer in order to execute the program. In the DVP production system according to this disclosure, video production servers can be integrated as required. This gives the DVP production system more flexibility than that of conventional broadcast studios, which need to be sized for larger and more complicated production. Certainly, many of the equipment in such a conventional broadcast studio is on standby when only simple programs are produced.

This disclosure primarily refers to a broadcast studio? However, this disclosure is also applicable to TV studios, TV production vehicles or any other live TV production environment.

Figure 2 illustrates the architecture of the video production server according to this disclosure. The server as a whole is referenced by the reference number 201. As illustrated in FIG. 2, the architecture is organized in layers. The lowest layer, in Figure 2, is a hardware layer 202. For illustrative purposes, some hardware components are illustrated, for example a central processing unit (CPU), a digital disk recorder (DDR), a network interface card (NIC), graphics processing unit (GPU) and input / output cards (I / O), As indicated by the dot streaks on hardware layer 202, there may be other hardware components, in greater or lesser quantity, in the other embodiments. All hardware components share the same clock.

A bus 203, for example a PCI or QPI interface bus, connects the hardware components together. An operating system 204 controls the operation of the hardware. The operating system can be a Linux-based system. The video production server 201 described so far hosts, on a service layer 205, software modules comprising a program code which is executable by Γ CPU unit of the server 201. Each software module is designed so as to provide a said service particular elementary. Basic services are built from threads. Threads are the smallest units that are eligible for hardware mapping. For example, the elementary service in the context of the ingestion of a stream of format “ST2022-5 / 6/7” includes three execution threads, namely: i) a thread for receiving the UDP protocol packets “ ST2022-5 / 6/7 ", ii) a wire to decode the flow of packets" ST2022-5 / 6/7 "into images, and iii) a wire to copy the images into a working memory. The data stream coded in the format "ST2022-5 / 6/7" is just one example. The concept described also applies to other standards, for example to the standards "AES67" and "RFC41.75".

A single image is also called a "grain", where "grain" is a general term for the smallest item processed in the video production server 201. A grain can also represent a video frame or an audio frame.

In the description of the embodiments, reference is most often made to video signal processing. However, it should be understood that, within the framework of the present patent application, the processing of the video signal also involves processing ^2016/5252 corresponding to an accompanying audio signal and associated metadata, if applicable. In order to make the description of the present solution more intelligible, the audio signals and the metadata are not always mentioned in conjunction with the video signals.

FIG. 2 illustrates four distinct types of elementary sendees on service layer 205, namely an ingestion service, a storage service, an encoder sendee and a decoder service. Furthermore, FIG. 2 graphically indicates that the video production server 201 can also host additional services. On an application layer 206, several elementary services are combined in order to obtain the functionality necessary for broadcasting production.

An exemplary application of the video production server 201 should be illustrated with reference again to FIG. 1. In the local studio 101, an interview, in other words an interview, is recorded. A video production server (not shown in Figure 1) ingests uncompressed audio and video streams from cameras and microphones in the studio, encodes them, and reads data through network 102 over the video server 103, where interview data can be recalled for news delivery later today. As a result, the video production server in the local studio uses ingestion, encoding, and playback of basic services.

The architecture of the video production server 201 makes a complete abstraction of the underlying hardware. In different embodiments of the video production server, the hardware can correspond to a single machine or to several machines, as well as to a distributed and networked environment. Likewise, the software architecture of the video production server 201 has a modular composition. The basic services of the video production server 201 can be combined in order to create new instances and new functional entities. For example, instances can be intermediate results in complex production, such as an edited feed that needs to be inserted into another feed for the final broadcast program. In another embodiment, a first elementary service ingests raw uncompressed signals from acquisition devices, for example cameras and microphones, which are intended to be coded into a low bit rate signal by the elementary sendee. successive.

This signal is stored in a data store by the successive elementary sendee for later use. The following elementary sendees read the data and decode the signal to the original raw signal before it is finally transcoded by the last elementary service.

Although the architecture of the video production server 201 is not limited to a specific type of hardware, it goes without saying that, whatever the type of hardware chosen, it must have sufficient performance to perform the necessary processing. the realization of basic services and functionalities.

From the description of FIG. 2, it becomes obvious that the video production server makes it possible to distinguish five basic functions in broadcasting production, namely, the functions of acquisition, storage, calculation, playback and control. The acquisition of video and / or audio signals is implemented by acquisition devices outside and separately from the video production server. The basic function, the "control" function, refers to the operation of the video production server using the control panel or the control application. Only the calculation is performed inside the video production server, or inside a cluster of several video production servers sharing the calculation tasks. At the hardware level, these basic functions are separated, while the hardware components communicate via network connections, for example an IP network.

A second server 207 is shown in Figure 2. Communication between server 201 and server 207 is established at the corresponding service layers 205 of servers 201 and 207, respectively, through a network connection 208. The network connection 208 can be a local area network connection as well as a wide area network connection, depending on the architecture of the entire DVP production system.

It should be noted that the input and output of the video production server 201 is synchronized with the other hardware components within a specific DVP production system. The video production server hardware modules are synchronized with other devices in the digital video production system, for example, using the PTP protocol, to ensure synchronization of the hardware modules over the IP network. However, the execution of the basic services within the video production server 201 is implemented asynchronously.

The video production server corresponds to a set of software components running on hardware of commercial products (COTS). Each of its functionalities is carried out using specific unitary software which is assembled from reusable functional software blocks and which can operate on any compatible hardware platform. New functionalities can be provided quickly by dialing del ^E2 ° ^16/5252 existing functional software blocks.

Relying exclusively on one or more software provides unprecedented flexibility:

The hardware components are the subject of a total abstractioit and become a basic product that can be reused and reconfigured according to specific functional needs.

The hardware platform becomes dynamically scalable since the software can be deployed on any specific machine, which avoids excessive provisioning of hardware.

Sizing can be tight, since all available processing power can be used at any time.

The hardware platform can be virtualized and implemented in a cloud.

All inter-equipment communication of commands and signals is carried out via IP links, which offers flexibility and scalability on the one hand for the routing of all types of signals, including baseband signals and compressed signals. On the other hand, inter-equipment communication becomes asynchronous. Synchronous communication is maintained only at the system boundaries, to ensure interoperability with conventional equipment. Asynchronous communication leads to a more flexible dimensioning, since it allows a statistical dimensioning of the input / output interfaces and the computing power, instead of a dimensioning based on a worst case scenario.

Figure 3 visually illustrates, in a simple graph, the architecture of the hierarchical software. The small circles 301 represent input sources or flows, for example raw signals from acquisition devices requiring resources for ingestion, compression, etc., in the form of the aforementioned elementary services, which are illustrated in FIG. 3 by ellipses 302a and 302b. This concept can be cascaded so that the output of elementary sendee 302a becomes the input of elementary sendee 302b. The sequence of elementary services is expressed by a functionality or a functional service 303. The functional service 303 can be designed in a flexible manner by modifying the type and / or the number of the underlying elementary services so that they adapt to the needs. from the director.

BE2016 / 5252

Figure 4 symbolizes, by different graphic elements, the abstraction of the software architecture of the underlying hardware. As mentioned previously, the software modules implementing elementary sendees can operate on any hardware provided that its performance is sufficient to execute the software modules with the required quality of service, namely to provide sufficient processing, storage and connectivity. Basic services 302 are made up of wires

401 having a pattern, in Figure 4, which corresponds to the pattern of associated elementary services 302 and associated hardware nodes 402. The blank portions of the hardware node symbols indicate that the capacity of the corresponding hardware node is not fully used. The lower part of Figure 4 illustrates how specific hardware nodes

402 representing processors, memories and input / output interfaces form a hardware platform. The basic services 302 are mapped on the hardware nodes 402. The same software instance can be mapped on several hardware topologies. This mapping is implemented, for example, by an algorithm allocating the computing resources, storage and connectivity resources necessary for the hardware resources, which are available on a specific hardware platform. The matching algorithm allows the mapping of elementary services on different hardware topologies. As a result, the software architecture is completely detached from the hardware topology. As a result, the hardware platform can be a single video production server or a cluster of multiple video production servers.

In general terms, a specific functionality or a functional sendee offered by the suggested video production server is conceived as a pipeline. At the entrance to the pipeline, the video production server ingests a stream generated by an acquisition device. At the end of the pipeline, a stream is read. In between, basic services are applied to the flow to provide the desired functionality.

FIG. 5A illustrates an example of such a pipeline which includes the elementary services “ingestion” 501, “storage” 502 and “reading” 503. Between the various elementary sendees, application programming interfaces (APIs) are defined ( Figure 5B). The API interfaces prescribe a unitary data format for video, audio and / or metadata at the input and output of software modules, which allows a sequence of basic "user-defined" services. The API interfaces also include a unitary format for ordering data, making it possible to order the basic services and

BE2016 / 5252 the video production server as a whole. These API interfaces are fundamental since they allow, in conjunction with the pipeline mechanism, the flexibility of the video production server according to this disclosure.

In one embodiment of the suggested video production server, which is suitable for live broadcast productions, the pipeline mechanism requires some optimization. In the context of live broadcast productions, very large amounts of data must be transferred and processed. An HD video stream consists of 24 to 60 frames per second, and each HD image is up to 1,920 x 1,080 pixels. This leads to a data rate of 50 to 440 Mbit / s for a compressed video stream, in production, and up to 3 Gbit / s if the video stream is uncompressed. “Ultra High Definition” (UHD), “High Dynamic Range” (HDR) and “High Frame Rate” (HFR) video signals lead to even higher data rates. Therefore, it is necessary to properly manage the load on the hardware platform and not to overload the pipeline with unnecessary copies of video images. This is achieved by implementing a source / collector mechanism between each elementary sendee. The source / collector mechanism is instantiated as part of the mapping of each elementary service on a given hardware platform. In case the mapping is done on a single machine, a protected shared memory mechanism is used. In the case where the mapping is performed on a cluster of multiple video production servers, communication based on Internet protocol (IP) is used. For trusted and short distance communication, the TCP protocol can be used, while for longer distance communication, an improved UDP protocol is preferred.

Figure 5B illustrates the relationship and interaction between API interfaces and basic services. Each elementary service 504 has, at its input, a source API interface 506, and at its output, a collector API interface 507. The source and collector API interfaces 506, 507 subtract the elementary sendee 504 allowing a concatenation of sendees elementary 504 as illustrated in FIG. 5C. In FIG. 5C, the arrow 508 symbolizes the data flow from the elementary sendee 504 to the elementary service 504 ’, via the collector API interface 507 and the source API interface 506 ′.

API interfaces allow intercession, that is, additional elementary sendables are executable between elementary services of an existing pipeline. This is illustrated in Figure 5D. Figure 5D still contains the pipeline from the first example shown in Figure 5A. However, there are other elementary sendees, such as the "coding" service 509 and the "decoding" service 510, placed between the elementary sendees 501 to 503 originally existing. The additional elementary outputs 509 and 510 are only exemplary. Any other elementary send could also be inserted, since the API interfaces allow any combination of elementary sendees producing the specific functionality required by the producer.

Figure 6 shows a diagram to illustrate the pipeline reconfiguration mechanism. On the left side of Figure 6, the timeline 601 is illustrated in which the time increases from top to bottom in Figure 6. At time T1, the original pipeline exists in which signals pass from elementary service A at elementary sendee B. At time Tl + delta, a new elementary service C is announced, then active at time T2, while, simultaneously, elementary service B is deactivated. At time T2 + delta, the new pipeline with a signal flow from elementary send A to elementary service C is established. The active signal flow, from left to right in FIG. 6, is illustrated by arrows in solid line 602, while a potential signal flow for an elementary service announced, or even announced, is illustrated by dotted arrows 603. On the right side, Figure 6 also illustrates a flowchart showing the steps from the establishment of the original pipeline to the establishment of the new pipeline.

In addition, API interfaces allow the pipeline to be changed grain by grain. This flexibility leads to interesting applications, since it is possible to manage very different needs in production outfits, with a single system, without requiring any reconfiguration overload. A typical example is consecutive reading in news apps.

News apps must combine several different video sources. These different video sources have different properties, for example streams from different sources have different codings, and therefore require different decoders. This means that the pipeline contains different decoders for the different video sources.

The suggested video production server, based on its "pipeline" approach to sequenced elementary services, with API interfaces defined between them, allows the use of a different decoder for each grain. The API interfaces and the pipeline remain unchanged, while only the internal processing of basic services is modified.

The pipeline and API interfaces allow the modification of the internal processing of a

E2016 / 5252 basic service with no impact on the pipeline. This offers the possibility, for example, of modifying the video format (resolution, color matrix, or the video protocol (ST2022-5 / 6/7, RFC4175,.,.) Without modifying the rest of the pipeline. This allows rapid upgrades of components of the video production server, as well as extreme versatility and reconfiguration "on a grain scale" of the pipeline.

Technically, this amounts to replacing or modifying the wires that make up the elementary sendee. However, regardless of these changes, the relationships between the elementary services remain unchanged, so that the pipeline itself is not altered.

By way of illustration, a simple instance of video production server is envisaged which stores video and / or audio streams in a storage store. A first ingestion service ingests "10801" type flows according to the ST2022-5 / 6/7 protocol. This requires simple configuration with a basic ingestion service and a storage service. If, instead, an audio stream must be stored, then only the internal processing of the basic ingestion service must be modified to ingest, for example, an "AES67" audio stream. Likewise, the ingestion feed is adaptable to other video and audio formats. Finally, it should be noted that the ingestion module is able to simultaneously ingest several sources of distinct content, for example audio content, video content and metadata, and different formats, notably different encodings. Sources can be received from different technical interfaces, for example: IP multicast, SDI, ASI, TCP IP unicast, etc. The acquisition devices from which sources are received need not be in the same time domain.

The video production server is based on a unique synchronization mechanism. Rather than relying on synchronization by the execution pipeline itself, as is the case in hardware-oriented implementations, the present video production server is inherently asynchronous internally. Synchronicity between different devices in a broadcast studio, which is essential in broadcast applications, is achieved by assigning timestamps to the grains. The ingestion module of the video production server assigns a time stamp to each packet or "grain". Time stamps are based on information within the data stream. In addition, time stamps are based on a single clock that is made unique across multiple video production servers using the PTP timing protocol. The individual packages or grains are identified by time and stamp

Π _Λ , ... ΒΕ2016 / 5252 of acquisition.

temporal. The ingestion module itself is synchronized with the di In this way, the video production server allows real-time processing as part of a live production.

Each "grain" is processed, for example by coding, storage, etc., within the video production server, independently, while maintaining the processing order. This implies that the temporal relationship between different grains is not guaranteed within the video production server.

When the video production server outputs the grains again, the correct sequence of the grains must be restored. Note that the clock frequency of the output system may be different from the clock frequency of the ingestion module, for example, input image frames are received from a camera at one point. frequency of 240 frames per second, while the output image frames are output at a frequency of 50 Hz or 60 Hz.

Figure 7 illustrates the operating principle of the video production server. In FIG. 7, two incoming serial flows 701, 702 are ingested by a video production server 703. The ingestion module of the video production server is synchronized with the acquisition devices. The processing of data frames within the video production server 703 is implemented asynchronously. The output module of the video production server is again synchronized with downstream processing devices and outputs two streams 704, 705. The synchronicity of the output frames of streams 704, 705 is guaranteed by the clock of a output module requesting the corresponding grains: for each clock tick of the output module, the system searches for the grain which corresponds to this tick, that is to say the grain whose tick is located in the period [time stamp , time stamp plus duration]. The clock ticks of the output module are displayed as circles 706. The clock ticks 706 occur simultaneously for the two output streams 704 and 705.

For this reason, it is important that different hardware modules share the same clock, while the software layers of the video production server architecture are extracted from the underlying hardware. The PTP protocol is used to guarantee the synchronization of the hardware modules on IP networks. Given the frame rates or frequencies currently in use (one frame every 8 to 50 ms), the PTP protocol is the

The combination of "timestamp-based" stream synchronization and the interconnection capabilities of the video production server pipeline using unitary API interfaces enable an innovative approach to system-level fault resolution.

A so-called "referee" application constitutes an application of the video production server presented. The referee app allows the display of frames of images captured by different cameras from different angles. The displayed frames are taken at the same time by the different cameras, and an arbiter is placed in a position to correctly assess a complex situation in a ball game like basketball. There may also be other applications where browsing in multiple stored video streams is desirable.

Figure 8A illustrates the normal operation of the video production server. The basic services 801-803 and the wires 805 are matched on a given hardware platform 806. FIG. 8B illustrates a different situation in which an additional elementary coding service 804 is introduced into the pipeline running simultaneously to provide a warm recovery function in the event of the failure of any equipment supporting one of the two basic coding services 802, 804. This situation is illustrated in Figure 8C, where the basic coding service 802 has failed without affecting the processing flow or the pipeline as a whole, since in this situation all packets are routed through basic coding sendee

The time-stamped nature of the grains allows successive elementary service, namely the storage service 803, to easily cope with any differences in latency between the two coding modules 802, 804, and to store only a single instance of a processed frame. In the event of a hardware module failure, the system continues to be operational. Obviously, thanks to the possibility of "grain-scale" reconfiguration of the pipeline, this approach can also be used in an intermediate recovery configuration.

This recovery approach, or continuity of services, requires significantly less redundant resources, since it relies on system knowledge and the exploitation of the properties of the video production server pipeline, rather than duplication of the data. '' complete installation by reproducing all of the underlying material,

The dynamic modification capacity of the pipeline not only allows fail-safe operation, in other words fault-tolerant operation, but also allows ^2016/5252 to dynamically reconfigure the same video production server hardware, in order to switch from a server ingestion to a read or storage server. These modifications can be implemented from a grain, or data frame limit, to the successive grain of a received data stream.

Figure 9A illustrates fault tolerance in more detail. Elementary service A 901 sends an identical grain "Ti" 902 to elementary sendee Bl 903 and to elementary service B2 904 at the same time. The grain Ti is processed by both elementary services Bl and B2, 903, 904, which send their output to the source API interface 506 of the elementary sendee C 906. The source API interface 506 of the elementary service C 906 removes the second grain Ti if the identical grain TI has already been received. Otherwise dif, if the Ti grain processed by the elementary service B2 904 arrives as a second Ti grain at the source API interface 506, it is abandoned. This configuration guarantees processing without interruption, even in the event of failure of one of the elementary sendees Bl, B2, 903, 904.

Figure 9B illustrates a section of an additional embodiment of a fault tolerant pipeline. In this embodiment, the elementary service A 901 sends the same grain Ti 902 to three elementary services B1, B2 and B3 903,904 and 905 implementing the same type of processing on the grain Ti. The output of each elementary service Bl, B2 and B3 903, 904 and 905 is transmitted to the source API interface 506 of the elementary service C 906. If the whole of the material functions correctly, then the grain Ti is received three times at level of the source API interface 506 of the elementary service C 906. As soon as two identical grains Ti are received at the level of the interface API source 506, one of the identical grains Ti is routed towards the elementary sendee C 906 for 'be treated. If the source API interface 506 does not detect at least two identical Ti grains, an error message is generated. This mechanism allows the detection of failures at the hardware level.

Finally, it should be noted that the suggested video production server provides a scalable software storage component that provides instant fine-grained IP-based access to grains as soon as they are recorded.

FIG. 10 visually illustrates the concept of the operating method of the video production server. A sequence of elementary services is initially defined, 1001, which must be applied to video and / or audio signal frames. Then, the processing of video signals and / or audio according to the sequence defined basic services is implemented asynchronously 1002. Finally, it is important to maintain the relatioîP ^E2 1003 ° ^16/5252 temporal frames stamped flows associated video and / or audio. The associated video and audio streams relate to the same application.

FIG. 11 represents an overall view of a production installation involving a plurality of servers 1101A to 1101F. In Figure 11, the individual servers are labeled as server nodes to emphasize that the server nodes 1101A to 1101F are individual stand-alone machines. Server nodes 1101A to 1101F are linked together through an internal network 1102 to form a cluster of servers implementing processing tasks as part of broadcast production.

Since the basic services are extracted from the underlying material, as far as the user is concerned, it does not matter whether the processing for the production of broadcasting is carried out by an individual server node or a cluster of servers comprising a plurality server nodes. The only difference for the user is that a cluster of server nodes provides more processing power, which is scalable in that additional server nodes can be added to or removed from the cluster.

The server nodes receive video data as input 1103, and provide processed video data as output 1104. The video data is communicated via the network 1102. The server nodes 1101D and 1101E are connected to local disk storage 1105 and 1106, the server nodes 1101A to 1101C are connected to a system area network (SAN) î 107 connecting the server cluster illustrated in FIG. 11 to other server clusters, in order to exchange video data, for example, during a broadcast production. Server nodes 1101C and 1101D are connected to client devices 1108 to receive input and / or provide output. One of the 1108 client devices can run an application that controls the server cluster.

FIG. 12A represents a schematic illustration of an example showing how the pipeline already illustrated in FIG. 5 is implemented on a cluster of servers.

The pipeline includes elementary services, 501, 502, 503, 509 and 510. The arrows in Figure 12A indicate on which server node each of the elementary services is mapped. In other embodiments, the. matching could be different. In addition, in some embodiments, the mapping is dynamically adapted to improve the performance of the server cluster. Where appropriate, the mapping changes occur from one data frame, or grain, of a signal stream, to the successive data frame. These quick changes are also relevant to the fault tolerance of the video production server.

BE2016 / 5252

Figure 12B shows an illustration of the mapping of the two pipelines. The first pipeline includes basic services 501, 502, 503, 509 and 510. The second pipeline includes basic services 50Γ, 502 ’, 503’, 509 ’and 510’.

In the examples illustrated in Figures 12A and 12B, the elementary service pipelines are mapped on a plurality of server nodes. However, it should be noted that, in an alternative approach, each server node in the cluster performs all of the basic services of a complete pipeline, but only a fraction of all of the video signals. The two approaches have in common that the processing required for a specific broadcast production is shared among the server nodes in the cluster.

List of reference numbers

101 Local studio

BE2016 / 5252

509,510 services

102 network 601 timeline 103 video server 602 effective signal flow 104 production server 603 potential signal flow 201 video production server 701,702 input streams 202 material layer 703 video production server 203 bus layer 704, 705 output stream 204 operating system layer 706 clock ticks 205 service layer 801 - 804 basic services 206 application layer 805 son 207 second video production server 806 hardware platform 208 network connection 901 sendee elementary 301 source 902 grain 302 basic service 903 - 906 basic services 303 functionality 1001 - 1003 process steps 401 source, input flow 1101A-1101F server nodes 402 hardware nodes 1102 internal server network 501 -  504 basic services 1103 video input 506 source API interface 1104 video output 507 API collector interface 1105, 1106 disk storage 508 data flow 1107 system zone network 1108 client device

Claims (15)

Claims BE2016 / 5252 1. Video production server comprising at least one processor and a storage magazine, in which software modules composed of an executable program code are loaded into a working memory of said at least one processor, in which each software module, when 'it is executed by said at least one processor, provides one. elementary service (501-504); in which a concatenation of elementary services provides a functionality involving the processing of video and / or audio signals required in order to produce a broadcasting program, in which the video production server is configured as an ingestion server ingesting signals, in particular video and / or audio signals, received in the form of data frame streams from acquisition devices in synchronicity with the output of acquisition devices; in which the video production server is equipped with an ingestion module assigning time stamps to each incoming data frame, and in which the video production server includes a storage magazine allowing direct access to any data frame after that the grain has been stored The video production server according to claim 1, wherein the software modules are mapped onto the hardware (401) of the video production server. 3. Video production server according to claim 1, in which the ingestion server is capable of simultaneously ingesting multiple signal streams of different tonnais. 4. Video production server according to claim 1, in which the video production server is also configured as a playback server delivering video and / or audio signals in the form of data frames in synchronicity with devices in downstream receiving the output signals from the video production server. BE2016 / 5252 5. Video production server according to claim 1, in which said at least one processor of the video production server implements an asynchronous internal processing of the video and / or audio and / or metadata signals. 6, Video production server according to claim 1, wherein said at least one processor is configured so as to process the data frames according to the sequence of their timestamps affected by the ingestion module. 10 7. Video production server according to claim 1, in which each software module of the video production server has defined API data interfaces (506, 507) prescribing a unitary data format for video data, audio data, metadata and / or control data, at the input and output of the software modules, allowing a sequence 15 "user-defined" basic services. 8. A video production server according to claim 7, in which the “user-defined” sequence of basic services forms at least one processing pipeline for video and / or audio signals and / or metadata. 9. Production, video server according to claim 8, in which a basic service can be deleted from said at least one processing pipeline, or replaced or inserted into said at least one processing pipeline without affecting the operation of the rest of the processing pipeline. . The video production server according to claim 2, wherein the video production server executes an algorithm which detects hardware failures and automatically re-matches the software modules, so that the software modules are no longer mapped to 3rd hardware. failing. BE2016 / 5252 11. Video production server according to one or more of the preceding claims, comprising multiple server nodes (1101 A-l 101F) connected in communication by a network (1102). 12. Video production server according to claim 11, in which the software modules are dynamically matched on the multiple server nodes (1101A-1101F) in order to improve the performance of said at least one processing pipeline composed of services. elements provided by software modules. 13. Distributed video production system comprising at least two broadcast production devices (103, 104), which are interconnected by a network (102) and are mutually synchronized, in which at least one device from one of the production devices broadcast is a video production server according to one or more of the preceding claims. 14. Operating method of a video production server, in which the method comprises the steps below consisting in: defining a sequence of elementary services applied to data frames of received signal streams such as video and audio streams; process data frames according to the elementary service sequence defined asynchronously, and simultaneously maintain the time order of time-stamped data frames and the synchronicity between associated streams, for example a video stream, and an associated audio stream, dynamically match the software modules on a hardware platform composed of multiple server nodes (1101 Al 101F) with a view to improving the performance of the sequence of basic services, and - in which the dynamic mapping varies from a data frame to the successive data frame of a received signal stream. BE2016 / 5252 15. The method according to claim 14, in which the method further comprises the step below consisting in: dynamically modify the sequence of elementary services, from a data frame to the successive data frame, of a received signal stream, by deleting an elementary service from the sequence of services 10 elementary, or by replacing an elementary service in the sequence of elementary services, or by inserting an elementary service in the sequence of elementary services. 16. Software containing program code, which when executed by 15 a processor, implements a method according to claims 14 to 15. BE2016 / 5252 CTt Server ® C o 3 ü Φ 3> X? Φ S ex. BE2016 / 5252 BE2016 / 5252 BE2016 / 5252 503