Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
  • H04N21/23602—Multiplexing isochronously with the video sync, e.g. according to bit-parallel or bit-serial interface formats, as SDI
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/222—Studio circuitry; Studio devices; Studio equipment
  • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
  • H04N5/268—Signal distribution or switching
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
  • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
  • H04N21/234309—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/233—Processing of audio elementary streams
  • H04N21/2335—Processing of audio elementary streams involving reformatting operations of audio signals, e.g. by converting from one coding standard to another
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
  • H04N21/2381—Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
  • H04N21/2385—Channel allocation; Bandwidth allocation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/242—Synchronization processes, e.g. processing of PCR [Program Clock References]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/41—Structure of client; Structure of client peripherals
  • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
  • H04N21/4223—Cameras
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
  • H04N21/4305—Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
  • H04N21/4342—Demultiplexing isochronously with video sync, e.g. according to bit-parallel or bit-serial interface formats, as SDI
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
  • H04N21/4346—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream involving stuffing data, e.g. packets or bytes
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/438—Interfacing the downstream path of the transmission network originating from a server, e.g. retrieving encoded video stream packets from an IP network
  • H04N21/4381—Recovering the multiplex stream from a specific network, e.g. recovering MPEG packets from ATM cells
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
  • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
  • H04N21/643—Communication protocols
  • H04N21/64322—IP
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
  • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
  • H04N21/643—Communication protocols
  • H04N21/6437—Real-time Transport Protocol [RTP]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
  • H04N23/661—Transmitting camera control signals through networks, e.g. control via the Internet
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04H—BROADCAST COMMUNICATION
  • H04H60/00—Arrangements for broadcast applications with a direct linking to broadcast information or broadcast space-time; Broadcast-related systems
  • H04H60/02—Arrangements for generating broadcast information; Arrangements for generating broadcast-related information with a direct linking to broadcast information or to broadcast space-time; Arrangements for simultaneous generation of broadcast information and broadcast-related information
  • H04H60/04—Studio equipment; Interconnection of studios
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04H—BROADCAST COMMUNICATION
  • H04H60/00—Arrangements for broadcast applications with a direct linking to broadcast information or broadcast space-time; Broadcast-related systems
  • H04H60/76—Arrangements characterised by transmission systems other than for broadcast, e.g. the Internet
  • H04H60/81—Arrangements characterised by transmission systems other than for broadcast, e.g. the Internet characterised by the transmission system itself
  • H04H60/82—Arrangements characterised by transmission systems other than for broadcast, e.g. the Internet characterised by the transmission system itself the transmission system being the Internet

Definitions

• This invention relates to conversion and transmission of audio-video and contro) signals between cameras and studio equipment.
• the present invention provides an encoding/ decoding method, an encoder/ decoder and transmitter or receiver.
• the invention also provides a device that may be provided as an addition to a camera or to studio equipment.
• the invention provides a device that converts signals used in a broadcast environment from multiple existing standards to Internet Protocol (IP) and also from IP to such existing standards.
• IP Internet Protocol
• the IP signal provides broadcast quality of audio video signals as well as signalling required in a studio environment.
• the signalling required in the studio environment may be referred to as "control" signalling, in the sense that it controls devices and displays, such as providing information to studio operators, or to control equipment.
• control signals include indications such as which camera is live, where to move a camera and so on.
• the invention provides apparatus for converting between synchronous audio, video and control signals and asynchronous packaged data streams for an IP network, comprising, a first interface for audio and video signals; a second interface for control signals; and a processor arranged to convert between synchronous audio, video and control signals and asynchronous packaged data streams, wherein each packaged data stream is according to one of multiple IP standards, each standard being selected according to the nature of the signal to be transmitted.
• This has the advantage that the nature of the signal (e.g. whether audio, video, control or type of control) may be used to determine the type of IP standard used for that signal.
• the apparatus is bidirectional in the sense that the packaged data streams are sent and received over an IP network and then converted to and from IP standards to synchronous audio, video and control signals.
• the IP streams are thus for an IP network in the sense that they may be transmitted or received over such a network.
• the standard selected is the lowest bandwidth such standard for the selected signal.
• a lower bandwidth protocol is used for the control signals than the audio video signals.
• the audio and video are converted to RTP.
• RTP This has the advantage of a being packet format which enables reliable transmission and guarantees order of delivery as well as potential for forward error correction.
• the control signals are converted to UDP. This allows the most efficient packetisation giving appropriate speed of delivery and lower bandwidth than RTP.
• the protocols are as set out in the table at Figure 4 herein.
• the apparatus includes a processor for receiving control signals in an IP standard and for asserting a control output at a camera.
• the control output is preferably a tally visual or audio indicator, such as a tally light or a sound generated in operator's headphone.
• the control output is preferably a camera control signal, such as RS232, RS422, LANC or similar for controlling aspects of a camera, such as focus, zoom, white balance and so on.
• the control output is preferably a talkback signal, namely a bidirectional audio feed between camera operator and a controller.
• the apparatus comprises an input arranged to receive the multiple IP video streams over the IP network from other camera sources and a processor arranged to output video for presentation to a camera operator.
• the apparatus includes switching to allow a camera operator to switch between these video streams.
• the apparatus comprises a device connectable to a video camera having connections to the interfaces, typically in the form of a separate box with attachment to the camera.
• the processor is arranged to convert from native audio-video signals of the camera to asynchronous packaged data streams for transmission to studio equipment.
• the processor is also arranged to convert control signals from asynchronous packaged data streams received from studio equipment to native signalling required by the camera or by ancillary devices coupled to the camera, such as tally lights, headphones or the like.
• the apparatus comprises a device connectable to studio equipment.
• the processor is arranged to convert from asynchronous packaged data streams received from cameras to native audio- video signals required by the studio equipment.
• the processor is also arranged to convert control signals from the studio equipment to asynchronous packaged data streams for transmission to one or more cameras.
• a single device is connectable to either a camera or to studio equipment to provide the appropriate conversion.
• the invention may also be delivered by way of a method of operating any of the functionality described above, and as a system encorporating multiple cameras, studio equiepment and apparatus as described above.
• Fig. 1 is a an image of a device embodying the invention
• Fig. 2 is a block diagram of the main components of the device of Figure 1 ;
• Fig. 3 is a table showing the preferred protocols as used in a device embodying the invention.
• Fig.4 is a block diagram showing the main hardward components of a device embodying the invention.
• Fig 5. shows a process diagram for a controller algorithm.
• An embodiment of the invention comprises a device that is connectable to a camera to provide conversion from signalling required by the camera to IP data streams and from IP data streams to signalling for the camera.
• the same device may also be used at studio equipment for converting IP streams received from cameras for use by the studio equipment.
• a single type of device may be deployed at existing items of television production equipment such that transmission between devices may use IP.
• An advantage of an embodiment of the invention is that it allows camera equipment of the type used in a studio environment or remotely but in conjunction with a production facility to take advantage of transmission of data to and from the device over packet based networks.
• Such a system may include multiple cameras, studio equipment and potentially one or more central servers for control, each with a device embodying the invention.
• the embodiment may additionally provide functionality for example coders converting will be automatically set depending upon connectivity factors such as how many cameras are detected in the system, what editing system is used and so on.
• the server within a system can send instructions back to each device to change various settings using return packets.
• the cameras may be anywhere in the world and the instructions may include corrective information or other control data such as a "tally light".
• the device may be implemented as an integral part to future cameras and studio equipment.
• the main embodiment that will be described, though, is a separate device that may be used as an add-on to existing equipment such as cameras, mixing desks and other studio equipment.
• a “Stage Box” as described in the following technical note description.
• a camera may be attached to a so called “stage box" for conversion of its output to an IP stream, and a remote control remote from the camera may be attached to a second such stage box for converting between IP and control signals.
• stage box for conversion of its output to an IP stream
• remote control remote from the camera may be attached to a second such stage box for converting between IP and control signals.
• Each of the camera and the remote control need to be unaware of the intermediary IP network and to send and receive appropriate timing signals in the manner of a synchronous network, although the intermediary is an sychronous open standard IP network.
• each device attached to an IP network requires functionality to provide timing. For this purpose a timing arrangement is provided.
• the timing arrangement comprises use of a timestamp in a field within IP packets sent from each device, the timestamp being derived from a local clock within each device.
• the timestamps within the packets received by each device are then processed according to a function and used relative to a local clock to ensure each device has a common concept of time, in particular a lock on frequency and preferably also a lock of phase, (n the embodiment, the function includes deriving network latency and setting a local time accordingly.
• the function includes controlling the local clock for frequency and/ or phase.
• the majority of IP packets are RTP.
• RTP is used to transport video and audio data from one box to another.
• the RTP packets are timestamped using a clock which is being synchronised via PTP.
• PTP is used to synchronise the clocks between multiple devices, and to establish a choice of best master.
• the timing functionality may also include a smoothing function to ensure that any packets arriving do not cause any sudden changes in comparison to the local clock.
• the timing arrangement may also include functionality within each device to determine whether it should act as a master clock to other devices, or as a slave to other devices. Using this functionality, a network of such devices may self organise when connecting to an IP network.
• SDI Serial Digital Interface
• IP Internet Protocol
• RTP Real Time Protocol
• the Stage Box builds on the concept of sending and receiving video and audio across broadcast centres, and looks at the tools required by camera operators and in studios. Based lower down the 'food chain', the Stage Box aims to commoditise IT equipment and standards in the professional broadcast arena.
• a primary aim of the Stage Box is to produce an open-standard device, where possible using the Industry IT standards. This will allow further integration in the future to what-ever the industry may develop.
• FIG. 1 shows an example of a device embodying the invention, the so called “Stage Box”.
• the main interfaces can be seen, gold BNC connectors for the video in and out (HD SDI), and the long silver SFP cage for the network adaptor.
• the block diagram of Figure 2 shows the different interfaces included in the design. It also shows the core processor elements.
• the Stage Box technical design is based around a Field Programmable Gate Array (FPGA), which has two main roles, the first, a supervisory role.
• the diagram shows how all the different interfaces are routed by the FPGA to the different functional blocks. Its second role is to provide the real-time video encoder and decoder.
• FPGA Field Programmable Gate Array
• the blocks on the left of the diagram are all resources available to either the FPGA, or the ARM processor, for example DDR3 memory.
• the all-encompassing idea for the Stage Box is to take the many different production formats and move them from traditional linear signals to a single, bidirectional data feed over standardised internet Protocols (IP), running on an Ethernet iayer two network.
• IP internet Protocol
• the Ethernet component is arguably the most intrinsic part of the Stage Box, and it is here we find the greatest challenges.
• IP signals can contain any number of discrete data lines, however the big difference is that the traffic can 'flow' inborn directions.
• IP infrastructures have a very limited bandwidth, which is significantly less that that of uncompressed HD.
• RTP Real Time Media Protocol
• RTSP Real Time Media Protocol
• UDP User Datagram Protocol
• TCP Transmission Control Protocol
• the final part of the Ethernet block is the physical layer.
• the Stage Box is supporting the use of Small Form Protocol blocks, SFPs. These are a physical cage in which the user manually fits a module either for a standard networking cable (RJ45 CAT 5e) or a fibre optic link.
• HD-SDI is defined by SMPTE 292M, and contains three main elements, video, audio and ancillary data.
• the Stage Box fully supports the standard with regards to it's different frame rates, and resolutions for video.
• the Stage Box also handles it's main elements.
• the diagram at Figure 2 shows how HD-SDl enters the Stage Box, and is converted to IP.
• SD1 is a digital signal, and so the A to D process is handled outside of the Stage Box.
• Process 1 The SDI is received and split into its constituent parts, the audio and ancillary data are stored in RAM, for retrieval later.
• Process 2- The video is encoded to AVC-1 100
• Process 3- as the encoding is achieved; the resultant stream is packaged and along with the audio and ancillary data is made ready for transmission over the IP protocol.
• the audio is added to the RAM as before, and the pulled out (FIFO buffer process) by the FPGA as required by the IP packager.
• Process 5- HD-SDl synchroniser pulls the audio, video and ancillary data as required. Audio
• the Stage Box supports the two of the most common methods
• HD-SDI As an analogue signal 'broken' out of the HD-SDI stream HD-SDI carries 16 discrete audio channels as part of its signal, and the
• Stage Box correctly handles this. This requires some delaying of the audio, to compensate for the video encoding delay and still ensure synchronised video and audio, when they are both packaged for the IP stream.
• the extra addition of analogue audio break-out gives productions an incredibly useful feature, in that additional microphones can be added at will to a soundscape, or can be used for monitoring (receiving programme audio down the line).
• Having analogue audio presents a series of technical challenges, as professional broadcast audio requires a large amount of headroom, relatively high voltage, and is very sensitive to electromagnetic interface on printed circuit boards (PCB) with fast data transmissions. The interference has been mitigated in the Stage Box, by having a separate PCB for the audio.
• PCB printed circuit boards
• the Stage Box needs to be configurable (patchable). Patching is achieved through the web interface, managed on the ARM processor.
• the Stage Box includes a talkback stream, over IP-, which is in effect a common VOIP (Voice Over IP) application. This has the added benefit of being easily supported by IT professionals.
• VOIP Voice Over IP
• the Stage Box In addition to the VOIP application, the Stage Box also has Bluetooth capabilities, and will stream the talkback over Bluetooth, thus giving the production teams, wireless talkback with out any additional equipment or cost, to that of the Stage Box.
• VOIP ad-hoc network signal
• the VOIP needs to be bi-directional, i.e. a microphone signal needs to be sent from the Stage Box.
• the Tally is a simple light that is triggered in multi-camera shoots when the vision mixer has selected a specific camera. I.E. Camera 1 's Tally will light, when the vision mixer has selected Camera 1 to go live. Floor Managers, and On Screen Talent often use this in order to know which camera to look at.
• the information is easily sent over IP, and is decoded by a simple application running on the ARM core.
• the application will also generate an audio signal over the talkback system for the operator.
• the Stage Box can also provide an IP video stream, at low bitrate, over Wifi for remote monitoring via a simple web interface. This will be based around HTML5 and will be supported by all the major browers.
• Configuration of the Stage Box is possible over Wifi, as the configuration web page is served to all HTTP requests, and the Wifi chip within the Stage Box, is set to work as an Ad-hoc network point.
• HD-SDI has a bitrate of ⁇ 15Q0Mb/s as apposed to most networks maximum bitrate of 1000Mb/s. As a production is likely to have multiple cameras on a single network, the maximum realistic bitrate one could network is lOOMb/s.
• H.264 High Level encoding, or Advanced Video Coding (AVC) as it's known has a specific sub-standard; AVC-I 100, which is a very rigid encoding profile, that limits the bandwidth to IQOMbfe.
• the Stage Box is using an AVC-I encoder and decoder developed by
• ZeroConf ZeroConf is a networking protocol, which allows a network device to automatically announce itself on a network and get the necessary IP details to work alongside other devices with out manual configuration. It achieves this by using Multicast Domain Name Services (mDNS). mDNS is a very useful tool, which is widely used by Apple, called their Bonjour system.
• mDNS Multicast Domain Name Services
• the Stage Box implements an open-source version of ZeroConf on the ARM hardware, which allows automatic configuration of the device's IP settings. It is also used for the recorder and control application to run the 'Workflow Toolset', a suit of tools, which allow the user to dynamically draw the production network as they see fit. Timing Information
• a device embodying the invention incorporates a new arrangement for providing timing.
• the "Stagebox” device can operate as an SDI to IP and IP to SDI bridge on a local network, and may be used as part of the wider IP Studio environment.
• This disclosure describes concepts addressing the problems of timing synchronisation in an IP network environment.
• AV material is captured, translated into an on-the-wire format, and then transmitted to receiving device, which then translates it back to the original format.
• the media data arrive with the same timing relationship as they are sent, so the signals themselves effectively carry their own timing.
• any point-to-point IP Audio-visual (AV) link the receiving end must employ a buffer of data which is written to as data arrive and read from at a fixed frequency for content output.
• the transmitter will transmit data at a fixed frequency, and except in cases of extreme network congestion the frequency at which the data arrives will, when averaged out over time, be equal to the frequency at which the transmitter sends it.
• the receive buffer will either start to fid faster than it is emptied or empty faster than it is filled. If, over time, the rate of reception averages out to be the same as the rate of processing at the receive end then this will be a temporary effect, if the two frequencies are notably different, however, then the buffer will eventually either empty entirely or overflow, causing disruptions in the stream of media. To avoid this, a mechanism is needed to keep the oscillators running on the transmitter and the receiver synchronised to each other. For this purpose, a new arrangement is provided as shown in Figure 4.
• Figure 4 shows a simplified version of the timing, networking, and control subsystems of the stagebox circuitry. For clarity this diagram shows the connections necessary for understanding the functionality and leaves off various further connections that may be provided. The diagram also omits the existence of an additional counter, the "Fixed Local Clock" (FLC) which runs from 125MHz ethernet oscillator, and as such is unaffected by any changes made to the frequency of a 27MHz crystal oscillator.
• FLC Fixed Local Clock
• the function performed by the arrangement of Figure 4 is to provide a local clock that is in frequency lock with a clock provided by a network source (which may be another "stagebox") and is preferably also in phase lock with such a network clock.
• the frequency lock is provided for reasons discussed above in relation to rate of arrival and buffering of packets.
• the phase lock allows devices to switch between multiple different such sources without suffering sequencing problems.
• the arrangement comprises a main module in the form of an FPGA 50 arranged to receive and send packets from and to a network 5, and a timing processor or module 24 coupled to the FPGA and having logic to control the provision of local clock signals in relation to received packets.
• the timing processor 24 implements functionality later referred to as a PTP stack under control of a software module referred to as a PTP daemon. This receives packets and implements routines to determine how to control local clocks to ensure frequency and phase lock.
• the functionality of the FPGA 50 will be described first. IP packets are sent to and received from network 5 via a tri-mode ethernet block 10 and a FIFO buffer 26.
• the packets are provided to and from the ARM processor, via a communication module here shown as EMBus 20 that provides the packets to other units within the main module 50, but also to the timing processor 24
• EMBus 20 that provides the packets to other units within the main module 50, but also to the timing processor 24
• a problem is to ensure that the local device to which the circuit is connected (or within which it is embedded) operates at a frequency locked with the frequency with which packets were sent such that the FIFO 26 neither empties nor overflows. For this reason, a Genlock output 3 is arranged so that it is frequency locked to a local clock which may be driven by a local input, allowed to run free, or driven to match a remote clock.
• a clock module here LMH1983 dock module 2
• LMH1983 dock module 2 is provided having a 27MHz output. This is provided to a black and burst generator 4 which feeds a DAC 6 to provide a genlock out signal to a camera.
• the input to the clock module 2 takes the form of three signals, F, V, and H, which are expected to be such that H has a falling edge at the start of every video line, and V has a falling edge at the start of every video field, F is intended to be high during the odd fields and low during the even ones, (f there is a genlock input attached to the device, and the device is in a master mode (described later), then a signal from a synch separator 8, here LMH1981 sync separator, may take this from an external device and feed this directly into the clock module 2. If no genlock input is connected to the device, then the devise is in a slave mode (described later) and these signals are then synthesized by a Sync Pulser module 18.
• the Sync Pulser module 18 is designed to operate alongside a Variable Local Clock (VLC) 16 module. These two modules both take a frequency control signal controlled by one of the registers settable in the EMBus module 20 (in the form of a 32-bit unsigned integer), and can both be reset to a specified value by setting other registers.
• the Sync Pulser 18 receives a fine number and a number of nanoseconds through the line in order to be set, whilst the variable local clock 16 requires a number of seconds, frames, and sub-frame nanoseconds. In all cases these are specified assuming a 50Hz European refresh rate (but may be modified if a 60/1 .001 Hz American refresh rate is to be used).
• variable local clock 16 and Sync Pulser 18 will be initially set to values which correspond to each other according to the following relationship:
• the frequency control value is a 32-bit unsigned integer specified such that the variable local clock 16 counter will gain a number of nanoseconds equal to the frequency control value every 228 cycles of a received nominally 125MHz ethernet clock, with the addition of these nanoseconds evenly distributed across this period. As such a value of 0x80000000 in the frequency control variable will ensure that the VLC counts at the same rate as the Fixed Local Clock (FLC), a second and nanosecond counter which runs off the ethernet clock and adds 8ns every tick.
• FLC Fixed Local Clock
• a Phase-lock- loop Counter (PLL Counter) 22 is a nanoseconds, frames, and seconds counter which runs from the generated 27MHz video clock, and so when the Sync Pulser 18 is being used to drive the dock module it should in general maintain the same frequency as the variable local clock, however near the time when the frequency of the variable local clock changes there may be some delay in the response of the analogue PLL 22 in the clock module, and so the PLL Counter 22 would fall out of phase with the variable local clock counter. To avoid this, the PLL Counter 22 can be set to update its current time value once per frame so that it matches the variable local clock at that point, and this is the mode of operation normally used when the Sync Pulser is being used to drive the clock module.
• the stagebox device When the clock module 2 is driven from the Sync Separator 8 then the stagebox device is running with a Genlock input. In such circumstances it is highly likely that there is also a Linear Time Code (LTC) input to the box, and so the PLL Counter may be set to adjust its time of day to match the LTC input once per frame.
• LTC Linear Time Code
• the black and burst generator 4 also takes its synchronisation from the clock module 2 and the PLL Counter 20, and so will either generate a time-shifted version of the original genlock input (if running with a genlock input) or a black and burst output which has the frequency and phase specified for the Sync Pulser 18 (if the Sync Pulser is being used).
• the PLL Counter 20 is used to drive three slave counters which are kept in phase with it.
• One is a PTP seconds and nanoseconds counter used to generate PTP timestamps for outgoing packets, the second is a 32-bit counter which always obeys the following relationship with the PLL Counter.
• RTP' 90 is a 32-bit value which can be set in a register controllable from the processor board.
• this counter is a nominal 90kH2 32 -bit counter as required for the video profile of RTP.
• the third counter is another 32-bit counter which always obeys the following relationship with the PLL Counter:
• RTP' 48 is a 32-bit value which can be set in a register controllable from the processor board, this counter actually runs off the nominal 24.576MHz (512 times the nominal 48kHz audio sample rate) clock output from the clcok module 2 and so is suitable for use when tagging audio data sampled using that clock.
• Stagebox Core which performs packetisation of the RTP streams used to transmit the stagebox's payload data.
• the device hardware described may have a number of local oscillators which are used for different purposes.
• the ones which matter for this disclosure are a 125MHz crystal oscillator used to time ethernet packets, and the 27MHz voltage controlled oscillator used for audio and video signals.
• the 27MHz oscillator is managed by a hardware clock management chip, the LMH1983 clcok module 2 which is used in many traditional video devices.
• This module serves several purposes, most notably including a phase-lock-loop (PLL) designed to match the frequency of the local oscillator to that of an incoming reference signal generated from an incoming genlock signal via a sync separator chip.
• PLL phase-lock-loop
• the LMH1983 chip also provides additional PLLs which multiply and divide the frequency of the 27MHz oscillator giving a variety of clock output frequencies, all locked as multiples of the controllable frequency of the oscillator.
• clock module has the following outputs:
• These clocks may be used by the device's other functions as their reference frequencies, as such it is possible to ensure that the audio and video sampling and playback performed by the stagebox hardware will be at the same frequency as that of another device by ensuring that the frequency of the 27MHz voltage controlled oscillator (here termed F) is the same between the two devices. Since the value of F is controlled by the input reference signals to the LMH1983 clock module controlling the clock is achieved by controlling these signals. In the example design these signals are not connected directly to the output of the LMH1981 sync separator. Instead they are connected to controllable outputs on a Virtex 6 field-programmable-gate-array (FPGA) on the board. The outputs of the LMH1981 are similarly connected to controllable inputs of the FPGA.
• F 27MHz voltage controlled oscillator
• the signals can be routed directly through the FPGA from the LMH1981 synch separator to the LMH1983 clock module, but it is also possible for the LMH1983 input signals to be driven by another source generating an artificially constructed series of synchronisation pulses synthesised based on a mathematical model of the remote clock.
• the device uses the PTPv2 protocol, which enables high precision clock synchronisation over a packet-switched network.
• the PTP protocol relies for its precision on the ability to timestamp network packets in hardware at point of reception and transmission.
• stagebox architecture all packets received by the box's 1000Mb/s ethernet interface are processed through the working of an SFP module 12, then passed back to the Xilinx Tri-Mode Ethernet MAG core 10 via the 1000-baseX PCS/PMA protocol.
• the Tri-Mode Ethernet Mac then passes these packets to the other components via an AXI-Steam interface. Since some of these packets will be video and audio which the stagebox will need to decode in hardware all packets are passed to a core processor, here shown as Stagebox Core 14 for filtering, processing, and decoding. In addition all packets are also passed into a series of hardware block RAMs as part of the FIFO and Packet Filter Block.
• the values of the VIC, the FLC, and the PLL Counter are all sampled at the time that the first octet of the packet leaves the MAC 10 and these values are stored with the packet, ready to be passed back to the processor. Not all packets, however, are passed back to the processor, instead each packet is examined according to the following rules: • IF is_unicast (pkt . address) AND pfet .address 1 ⁇ 2 self .address THEM DROP
• mcast_addr_hashes is a set of hash values of ethemet multicast addresses which can be set via the EMBus registers
• port_whitelist is similarly a list of udp port numbers.
• the hash function is such that it generates only 64 different hashes
• the port whitelist can be set using bitmasks to allow for certain patterns to be allowed through.
• no port filtering is performed on non-UDP-in-IPv4 traffic directed to the box, so it would be possible to perform a denial of service attack on a stagebox by ooding it with large amounts of IPv6 or TCP traffic. In practice this is unlikely to happen unless done intentionally.
• the timing processor 24 receives packets from the FIFO 26 via incoming bus line 7 and sends packets to the FIFO via outgoing bus line 9, connected via the EMBus 20.
• the transmit side there are three streams of packets which are switched together before being handed to the MAC for transmission.
• One is the stream of hardware generated packets emerging from the Stagebox Core 14, the second is the stream of software generated packets passed in via the EMBus 20, and the third is a second stream passed in via the EMBus 20.
• This last stream will only store one packet at a time prior to transmission, and records the values of the FLC, the VLC, and the PLL Counter at the time at which the first octet of the packet enters the MAC. These values are then conveyed back to the processor board via the EMBus.
• the software implementing the timing processor 24 may choose to mark a specific packet as requiring a hardware transmission time stamp.
• That packet is then sent preferentially (with higher priority than either the hardware or other software generated packets) and the timestamp is returned and made available to the software.
• the hardware timestamping of certain received and transmitted packets is a feature provided to implement a PTP stack in the timing processor 24.
• the fact that multiple timestamps off different counters are generated allows a more complex algorithm for clock reconstruction.
• the use of packet filtering is important because the EMBus has only limited bandwidth (approximately 150Mb/s when running continuously with no overhead, in practice often less than this) and the RTP streams generated by other AV streaming devices on the same network (such as other stageboxes) would swamp this connection very quickly if all sent to the processor.
• the PTP stack implemented by the timing processor 24 on the stagebox is not maintained purely in hardware, rather hardware timestamping and clock control are managed by a software daemon excecuting on the timing processor 24 which operates the actual PTP state machine.
• the PTP daemon can operate in two different modes: Master-only, and Best-master mode.
• the best-master mode mode is automatically triggered whenever the device detects that it does not have a valid 50Hz black and burst signal on the genlock input port on the board.
• the software implementing the timing processor 24 will advertise itself as a PTP Master to the network 5, but will defer to other masters and switch to the SLAVE state as described in the PTP specification if it receives messages from another clock which comes higher in the rankings of the PTP Best Master Algorithm.
• the software instructs the hardware to use the incoming reference from the Sync Separator 8 to run the Clock Module 2, and does not contro) the VLC at all, if there is no reference from the sync separator then this results in the 27MHz oscillator free-running.
• the hardware When acting as a slave the hardware instead uses the Sync Pulser 18 as the source of synchronisation signals for the LMH1983 Clock Module 2 and the VLC as the source of timing values for the PLL, and the software in the timing processor 24 steers the oscillator by controlling the frequency control of the Sync Pulser 18 and VLC 16.
• the stagebox When advertising itself as a master the stagebox provides the following information in its PTP Announce messages.
• _ clocKCIass is set to 13 if there is a valid 50Hz black and burst genlock input, and 248 otherwise.
• _ clockAccuracy is set to 0x2C if there is a valid 50Hz black and burst genlock input and a valid linear timecode input, and OxFE otherwise.
• _ offsetScaledLog Variance is currently set to -4000, though a future implementation may measure this in hardware.
• Clockldentity is set to an EUI-64 derived from the ethernet MAC address of the stagebox treated as an EUl-48 rather than a MAC-48.
• stageboxes will, for preference, use non-stagebox masters (since most masters are set with a Priorityl value of less than 248), will favour stageboxes with a genlock input over those without, and will favour those with an LTC input over those without.
• a tie is broken by the value of the stagebox's MAC address, which is essentially arbitrary.
• the actual synchronisation of the clocks is achieved via the exchange of packets described in the PTP specification.
• this implementation uses the IPv4 encapsulation of PTPv2, and acts as a two-step end-to-end ordinary clock capable of operating in both master and slave states.
• the master implementation is relatively simple, using the PLL Counter in the hardware as the source for timestamps on both the transmitted and received packets. Since this counter is driven from the 27MHz oscillator, and is set based on incoming linear time-code this means that the master essentially distributes a PTP clock which is driven from the incoming genlock for phase alignment, and the incoming LTC for time of day, or runs freely from system start up time. In either case since no date information is being conveyed to the box by any means the master defaults to the 1 st of January 1970, with the startup time treated as midnight if there is no LTC input to provide time of day information.
• the slave implementation is more complex.
• Incoming packets are timestamped using the VLC 16 and FLC (not shown) as well as the PLL Counter 22, and these values are used in the steering of the clock.
• VLC 16 and FLC not shown
• the PLL Counter 22 the PLL Counter 22
• Incoming Sync packets received by the daemon in the timing processor in the slave state originating from its master are processed and their Remote Clock (RC) timestamp is stored along with the FLC and VLC timestamps for their time of reception.
• the FLC/RC timestamp pairs are filtered to discard erroneous measurements: in particular packets which have been delayed in a switch before being transmitted on to the slave will have an FLC timestamp which is higher than one would expect given their RC (transmission) timestamp and the apparent frequency of the clock based on the other measurements.
• These packets are marked as bad (though their value is retained as future data may indicate that they were not in fact bad packets) and ignored when performing further statistical analysis triggered by the receipt of this particular Sync packet.
• the further analysis takes the form of a Least-Mean-Squares (LMS) regression on the data, which is a simple statistical tool used to generate a line of best-fit from data with non-systematic error.
• LMS regression requires a level of precision in arithmetic which is beyond the capabilities of the 64-bit arithmetic primitives provided by the operating system and processor, for that reason the daemon contains its own limited implementation of 128-bit integer arithmetic.
• the LMS regression attempts to construct the gradient of the line of best fit for the graph of FLC timestamp vs. RC timestamp, which is to say the difference in frequency between the remote clock on the master (a multiple of the 27MHz voltage controlled oscillator if the master is another stagebox) and a multiple of the ethernet clock on the local device (chosen because it is unaffected by the local oscillator steering, and because timestamps applied using this clock can be extremely accurate due to it being the same clock used for the actual transmit and receive architecture). To do so it selects the line which minimises the mean of the square of the difference between the line of best fit and the actual RC value at each FLC measurement.
• F[n] is the nth filtered value
• D[n ⁇ is the run raw delay measurement
• s[n] is a filter stiffness value which is such that:
• D ⁇ - ⁇ ⁇ is set to be equal to D(0] to avoid a discontinuity in the filter at 0.
• the daemon then hands control over to the "Precise lock” algorithm, which is the traditional PI controller with frequency change restrictions. If the error ever reaches more than one quarter of a line of video then control is passed over to the "Slow Lock” algorithm, which is a P2 controller with change restrictions, and when the error falls back below the one quarter of a line threshold the "Precise Lock” is invoked again. Only if the error reaches more than one line of video is another "crash” triggered and the "Fast Lock” algorithm reinvoked.
• the "Precise lock” algorithm which is the traditional PI controller with frequency change restrictions.
• the stagebox software build will, at start up, search for a DHCP server on the local network, and use an IPv4 address provided by one if there is one. If no address can be acquired via DHCP it falls back to automatic configuration of a link-local address. It also automatically configures IPv6 addresses in the same way, but these are not currently used. This behaviour ensures that stageboxes can operate correctly even if the only devices on the network are a number of stageboxes connected to switches. It even aiiows the stageboxes to operate correctly when connected using a point-to-point network cable between two boxes.
• the design contains a Stagebox Core which can generate two streams of RTP packets, a video stream and an audio stream which contains raw 24-bit PCM audio. These packets also contain RTP header extensions in compliance with specifications for RTP streams for IP Studio.
• the hardware generating these streams requires certain parameters (such as source and destination addresses, ports, payload types, and ttl values) to be set in the registers made available to the processor, and also generates certain counters which report back data required in order to generate the accompanying RTCP packets to go with the streams.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Computer Networks & Wireless Communication (AREA)
• Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
• Time-Division Multiplex Systems (AREA)
• Studio Devices (AREA)
• Studio Circuits (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (2)

Family

ID=45930046

Family Applications (1)

Country Status (7)

Cited By (3)

Families Citing this family (13)

Family Cites Families (18)

• 2012
  • 2012-02-10 GB GB1202472.5A patent/GB2499261B/en not_active Expired - Fee Related
• 2013
  • 2013-02-11 EP EP13707022.3A patent/EP2813084A2/en not_active Withdrawn
  • 2013-02-11 IN IN6735DEN2014 patent/IN2014DN06735A/en unknown
  • 2013-02-11 AU AU2013217470A patent/AU2013217470A1/en not_active Abandoned
  • 2013-02-11 WO PCT/GB2013/000054 patent/WO2013117889A2/en not_active Ceased
  • 2013-02-11 JP JP2014556131A patent/JP2015510349A/ja active Pending
  • 2013-02-11 US US14/377,697 patent/US20160029052A1/en not_active Abandoned

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events