EP3507987A1

EP3507987A1 - Method for transmitting real-time-based digital video signals in networks

Info

Publication number: EP3507987A1
Application number: EP17761866.7A
Authority: EP
Inventors: Ulrich Pflueger; Oliver LIETZ
Original assignee: Nanocosmos Informationstechnologien GmbH
Current assignee: Nanocosmos Informationstechnologien GmbH
Priority date: 2016-09-05
Filing date: 2017-09-04
Publication date: 2019-07-10
Also published as: US20190191195A1; WO2018042036A1; DE102016116555A1

Abstract

The invention relates to a method for transmitting video signals from a video signal source to a reproduction device. The signal is emitted by the video signal source into a stream and subsequently correspondingly fragmented in packets into a known format, the size thereof corresponding to at least one video image with associated audio information. The packets are transmitted to the reproduction device, the content of the packets being displayed using said device.

Description

Method for transmitting real-time-based digital video signals in networks

description

The invention relates to the transmission of digital video and audio signals from a video signal source via a server to a display device, on which the data provided by the video signal source are displayed.

State of the art

The transmission of real-time video data, also called live streaming, generally takes place in networks via several intermediate stations, starting from the camera via a processing or coding unit at the camera to a server which makes the forwarding to the receiver or the playback device , The distances in the networks can lead to delays in the transmission. Additional delays arise through

Media technology required interference in the data stream, which are required on the basis of current standards and technologies. All delays in the network add up to a total delay.

The transmission of video data in the highest possible quality is a high

A technical challenge that poses several problems that result in limited image quality and delays, often in the range of several seconds.

The quality of the transmission results on the one hand from the image quality itself, but also from the liquid of the reproduction. The image quality is related to the available bandwidth in the network, the liquid at the frame rate measured in frames per second or Hz. It is not possible with previous methods to transmit video data in everywhere available networks without loss of quality. The video data must be compressed (encoded) to match the available ones

Bandwidth in the network not to be exceeded. In order to adapt the amount of data of the generated video signal to the available bandwidth, lossy compression methods are required which allow as few visible errors as possible. The processing and compression of video and audio data is usually done independently in separate modules.

Compression of video data uses hybrid techniques that produce a large amount of data per frame by using different types of images (keyframes = I-frames, difference images = P / B frames). The grouping of groups of keyframe groups is referred to as the "group of pictures" (GOP), and the GOP length often ranges from 1-10 seconds or more, the longer the GOP length, the better the average video quality achieved But the longer they are necessary

Frame buffer.

To fluctuations in the data rate of the encoder as well as the available

To compensate for network bandwidth, buffer memories are used, which should compensate for the fluctuations over certain time periods. In addition, long intervals of "key frames" (I-frames) are selected during the compression, which are also in the range of 2-10 seconds.With multiple buffering in all components involved (transmitter, server, receiver), buffer times or buffer times often occur Latencies of more than 30 seconds.

Significantly lower latency is required for video communication between different subscribers, particularly when they are signals that are to be close to "real-time" (e.g., videoconferencing, live events).

There is already a method for generating a file format for the

Multimedia streaming is known as well as a device for the Multan

Multimedia streaming is known using this format. This generates fragments of data. In each fragment will be a variety of

Multimedia data boxes are arranged one behind the other, and at the end of the fragment, a box is arranged, which contains on the multimedia data related metadata (US 201 1/0276662 A1).

Packing / Multiplex The signal output from the video signal source in a stream is divided into packets. A packet may contain video data, audio data, or both (multiplexing).

The term packet in this context refers to the aggregation, multiplexing of video and optionally audio data in the output format, not to the size of packets in the network transport, which subsequently takes place on a lower system level. The consideration of the network layer is not the subject of this device.

Streaming differs in concept usually in the application of video communication by the number of viewers. Streaming should be possible regardless of the number of viewers.

In video-on-demand streaming, which retrieves certain prerecorded video data, the problem of the above-mentioned latencies is irrelevant, in contrast to the real-time live streaming with which the invention is concerned.

In real-time live streaming, the material is not pre-produced but produced in real time in the present. Sources are usually live cameras, either as a separate device or built into a device (mobile device, laptop, surveillance camera, action cam, stationary or attached to mobile devices, etc.). However, the source can also be an artificially generated signal that does not have to come from a camera, e.g. for presentations or games.

There are already protocols and standards for streaming.

Real-time Messaging Protocol (RTMP): Developed by Adobe as part of the "Flash" technology, the process is not officially standardized and now outdated, and is based on a continuous stream of data sent from a server to a client (player) can be

Playback devices are only able to play RTMP with the help of an additional software module ("Flash-Plugin") .The "Flash-Plugin" has been around for many years

Component of browser applications on the desktop ("desktop") and becomes from 2017 probably no longer supported by any browser. On mobile devices and most TVs and embedded devices like loT, this technique is not supported. http live streaming (HLS and DASH)

HLS was invented by Apple and is based on a buffered transmission of portions of the livestream. DASH is an ISO standard based on the same principle. The pieces of the livestream need a certain amount

Minimum size, so that a faultless transmission is guaranteed.

Streaming / playback with HTML5

HLS and DASH are part of the "HTMLS" standard, used by most

Supported by browser manufacturers.

With HTML5 it is possible to embed video data directly on a web page and present it in multimedia environments. HTML only allows certain video formats and protocols for embedding video data.

The state of the art in HTML5 is the ISO-MPEG4 video compression method, ITU-H264 with the MP4 file format and the proprietary, less widely used VP8 or VP9 with the WebM file format. The file-based formats like mp4 are not designed for real-time playback.

For transmission and playback via real-time signals ("live") via

Networks additionally exist the protocols HLS (proprietary Apple for iPhone and MacOS based on the MPEG-TS format) and MPEG DASH. However, in these cases "live" does not mean "real time" but delays of often 30 seconds or more. These can be reduced by fine-tuning all the state-of-the-art components to the minimum 6-0 second range.

The transmission of video signals with short latencies so far require other methods than the o.g. HTML / HTTP protocols. State of the art for this are the protocols UDP and RTP, which are e.g. in the applications Skype or the

Web technology WebRTC are used. Real-time communication, in contrast to streaming, is not standardized in terms of international standards and is available on a few devices, so that end devices such as TV and mobile devices do not have a uniform interface for this.

Short latency transmission with these protocols often results in non-disruptive transmission, which limits the application experience. Video communication applications are designed for the transmission of point-to-point connections similar to telephony (one-to-one).

The video communication and chat protocols (Skype, WebRTC) are not compatible with streaming standards (HTML5, HLS, DASH).

Main lack of the past HTML / http solutions is the long delay of the picture transmission over several seconds. This has the great disadvantage that these methods are not suitable for communication applications that require a short latency between sender and receiver. (e.g., for audio-telephony or interactive inclusion of viewers with a return channel.)

Application areas such as video conferencing or auctions are therefore not possible.

In the prior art, an HTML 5 video element is capable of playing either a complete file or a video fragment or segment, which may be provided in the form of a file or as part of a data stream. For the formats DASH and HLS segments are used, which in turn are divided into fragments. The state of the art is the ISO MP4 file format in the variant fMP4. A segment length corresponds to at least one GOP length, ie 1 to 10 seconds per segment. The additional inserted latency is a length of a segment. In the prior art, a segment may contain one or more complete GOPs. The minimum latency thus corresponds to the GOP length. Multiple buffering creates a threefold latency in existing devices.

The fragmentation can be carried out, for example, on the basis of the ISO standard "fMP4". For the fMP4 Formai package is synonymous with MP4 Fragment. The temporal size of a fragment corresponds to several video images in the prior art. According to the prior art, a fragment contains at least the number of video pictures of a GOP length.

The fragments consist of different type designations ("atoms") .The packet fragments are divided into headers and payloads, so there is one between the individual payloads of the packet fragments

Space, which consists only of the header information for the "atoms" moof and mdat, which can be used for synchronization.

The transmission usually takes place via the IP protocol TCP, so that a disturbance on the transmission link at this protocol level is excluded. If the connection is lost, it is necessary and possible to re-connect to the live stream to continue a real-time transmission.

Both the coding and the server and the playback device have buffer memory.

It can be provided that each packet is provided with a time stamp. Timestamps are a common means of synchronizing A / V packets. For each time of recording with a live source, there is a timestamp that can be synchronized with, for example, the real time. The playback page can then determine how late or early the packet is relative to real time and other packets.

A data stream according to the fMP4 format consists of an introductory data structure "ftyp" and "moov" followed by in an example 5 packet fragments.

Each package fragment consists of 2 parts, namely a part "moof der

Information includes the number of video and audio frames in the package, the timing or duration of the video and audio frames, the byte size of the video and audio frames, and the byte position of the video and audio frames. This atom "moof is then followed by an atom" mdat ", in which the actual video and audio are included. The individual parts of this exemplary stream are immediately adjacent to each other.

For segmentation and fragmentation, the HLS format can also be used instead of fMP4. The HLS format consists of 2 parts: several

Segments in the format TS (ISO-MPEG transport stream), each comprising at least one GOP length and playable independently of each other, and the index data (playlist) in the format m3u8, each pointing to the segments.

Usually, 3 segments per index are used, which shift in time during the transmission. By way of example, at 10 seconds per segment a minimum latency of 3 × 10 = 30 seconds results.

The playback device in the prior art includes a dedicated buffer which generates additional latency. The buffer is set automatically in the playback device. The automatic setting usually takes place on the basis of the set playing time of the data stream, which corresponds at least to the segment length.

The data generated by a signal source are in practice

Time variations of the systems and other system influences are not always

generated continuously. Example: theoretically, a camera has a frame rate of 25 frames per second. An image corresponds to a duration of 40 ms. The images generated by the signal source can in practice in different

Follow time intervals. In the example, a gap of 3 images could result, which corresponds to 3x40 = 120ms duration. In the example 5 pictures per second results with 200 ms per picture and for 3 pictures 600 ms. These gaps lead to latencies on the playback side in the prior art.

Description of the invention

The invention is based on the object to provide a method for transmitting real-time-based digital video signals in networks, which can also find application where it is a quick response on the part of the Receiver arrives, for example, in videoconferencing, auctions or interactive involvement of the audience.

To solve this problem, the invention proposes a method with the features mentioned in claim 1. Further developments of the invention are the subject of dependent claims.

The signal output by the video signal source in a stream is thus fragmented into packets, with a packet fragment corresponding to at least one video picture with associated audio information. Using just one video frame allows playback with the least possible delay between video capture and playback. When using multiple video images in a packet fragment, the delay is still significantly less than in the prior art, as long as the number of times in the

Packet fragment contained in the known in the prior art number in a GOP (Group of Pictures,) remains.

The temporal size of a fragment corresponds to the length of one or more video images that is smaller than a GOP. The data size corresponds to one or more video images and possibly the corresponding time

Audio data plus multiplexed data.

There may be buffer memory in the processing unit and the playback device, but according to the invention the packetizing unit keeps the buffer as small as possible, since the filling of a buffer is usually associated with latencies which the invention wishes to keep as small as possible.

In a further development of the invention it can be provided that the packaging is made into fragments in the area of the video source.

However, it has been found to be especially useful if fragmentation is done using a separate server from the video source. However, it is also possible in individual cases and wins within the scope of the invention that the packet fragmentation takes place in the playback device, for example in the browser. This can be done at the script level, ie using a browser-supported program module. Common is Java script. This type of program control with Java script is common for a variety of applications and state of the art. This program control occurs under browser-controlled environments and does not allow direct hardware access.

In a development of the invention, it can be provided that the packet fragments are present in the fragmented MP4 format (fMP4). In this format, an initialization segment is provided, followed by a repeating group of a fragment header (moof) and a fragment data segment (mdat).

Other features, details and benefits will be apparent from the following

Description of a preferred embodiment and with reference to the drawing. Hereby show:

Figure 1 is a schematic overview of the various stages of the invention; Figure 2 is a schematic representation of one consisting of 5 fragments

streams;

Fig. 3 is a flowchart showing a processing unit for processing the incoming stream;

FIG. 4 shows a flowchart of the processing in the one mentioned in FIG

packet assembling unit;

Figure 5 is a parent diagram.

FIG. 1 shows, in a highly simplified schematic form, the structure of a

Transmission system on which the proposed method of the invention proceeds. The video signal is generated by a video signal source 1, for example a video camera. The video signal source 1 is connected via a transmission path 2 with a Paketierungseinnchtung 3. This Paketierungseinnchtung 3 may be, for example, a server. Via the transmission link 2, the signal of the video source 1 is transmitted to the Paketierungseignchtung. In Paketierung 3 3 the video signal is fragmented into packets, which in the The following will be explained in more detail. The packetizer 3 is connected via a further transmission path or channel 4 to a display device 5 on which a user can see what the source is transmitting.

Channel 4 may be a continuous channel with back and forth

Trade return data. However, it can also be a static channel, which only works in the direction from the server 3 to the reproduction device 5, without there being a return channel.

In the packaging device 3 are adaptations of the incoming

Data stream carried out, namely on the one hand, a packaging and segmentation of the income data stream and on the other an adaptation of the data stream to a suitable format for the playback device 5 format.

FIG. 2 shows, by way of example, the data stream on the basis of the fMP4 format (state of the art). As a result, the stream is in a standard-compliant form. The stream starts with a ftyp box followed by a moov box. This is followed by a continuous sequence

alternating metadata boxes moof and media data boxes mdat.

Each fragment consists of a moof and a mdat box. Moof contains

Information about the number of video and audio frames in the package, the timing or duration of the video and audio frames, the byte size of the video and audio frames, and the byte position of the video and audio frames. The mdat box contains the actual video and audio data.

Since a continuous stream is sent, only one request is required.

FIG. 3 shows in simplified form the procedure or the sequence of the method within a processing unit in which the input stream is processed. The process begins in block 1 1, where the stream arrives. Block 1 1 is followed by processing block 12, which may also be referred to as a demultiplex block. In this block, the incoming stream in video, audio and

Split metadata packages. In the subsequent block 13, the data in the internal

Package structure of the media data converted. These media data include the type of packet, namely video, audio or metadata, time information

Sync point, video and audio configuration data, data buffer and data size.

In the then subsequent block 14, the output is made to the

Packaging according to the invention admitting device, which is explained in more detail in Figure 4.

From the block 14 then enter the data in the query block 21 of Figure 4. There is queried from the incoming data, whether it is configuration data. If it is actually configuration data, it will be stored in block 22.

If it is not configuration data, flow proceeds to block 23 where it is queried if it is metadata. If so, they are stored in block 24. If not, the flow proceeds to block 25, where it is queried whether the stored configuration data is available. If you are

Configuration data was not stored as a result of the query in block 25, the data packet is discarded in block 26.

In block 27 following a positive query, it is checked whether a keyframe is contained in this packet or whether a keyframe has already been received. If not, in block 28 the data packet is discarded.

In the block 29 following a positive query, a query is made as to whether it is an audio packet. If so, the audio packet is stored in block 30.

In the subsequent block 31 is a query whether it is a

Video package is trading. If so, the next block 33 queries whether it is the start of a video frame. If this is not the case, the video packet is stored in block 34. In the abrage block 35 following on a positive ablation, it is checked whether the number of video frames in the fragment is buffered. If not, the video packet is stored in block 36.

If it is the first fragment to be sent, which is determined in the query block 37, the initialization of a header ftyp and moov is prepared and carried out in block 38, see the information on FIG.

Subsequently, even if the question is answered negatively in the query block 37, in block 39, the header of a fragment is prepared and created moof. Subsequently, in block 40, the data part of the fragment is created, namely the element mdat, which was explained in FIG.

Block 40b stores the current video packet.

If it is the first fragment to send, as determined in query block 41, block 42 outputs the initialization header, the fragment header, and the fragment data.

If it is not the first fragment to be sent, that is, the query block 41 provides a negative response, the fragment header and the fragment data are output in block 43. With the output in block 44 is the activity of

Packing unit finished.

FIG. 5 shows again in a superordinate representation the structure of the method as proposed by the invention. The control of the stream is done in such a way that at the beginning there is "source", which is the data stream

provides. The video, audio and metadata contained in the stream are unpacked and forwarded via connection 52 to the packetizer or multiplexer component 53. The multiplexer component 53 does what was explained in detail in FIG. In this multiplexer component 53 is generated in an HTMLS capable Filestream format. For example, it can be fMP4 for Chrome, Firefox, or IE 11. For Safari US X and iOS it will Format m3u8 / ts (HLS) preferred. Subsequently, the forwarding takes place via the connection 54 to the output group 55. From there, the forwarding takes place to the outputs, which are no longer shown in detail.

Total end-to-end latency is the sum of network transport latency, format-related latencies, and buffers latency in the playback device.

The network transport latencies are made up of the transmission of the encoder to the server, the transfer from the server to the packaging unit Player / Transmux Server (53 in FIG. 5) and the transfer therefrom to the server

Reproduction device.

Grouping-related timing dependencies on the delivery of a stream lead to additional latency. The beginning of a segment or fragment can not be delivered until all contained samples have been received. The additional latency for fMP4 formatting is the length of a fMP4 fragment. In the methods used so far, a fragment contains one or more complete GOPs (group of pictures). The minimum latency in the known methods thus corresponds to the GOP length. The method of fragmentation per frame proposed by the invention shortens the format-related latency on the frame length.

For segmentation and fragmentation, the HLS format can also be used instead of fMP4. When using the HLS format, in the prior art the format-related additional latency is length times the number of HLS segments, for example 3 ^* 3 seconds = 9 seconds. Due to the measures according to the invention, the HLS playlist contains only one segment with a short nominal playing time (m3u8 tags).

The buffer in the display device according to the prior art corresponds to at least one segment length, which can also lead to a latency. In a development of the invention, it may be provided to set the nominal playing time of the segments to low values. This is controlled by customization by the device in the fMP4 header and / or in the HLS playlist.

The device also monitors and controls the buffer of the playback unit.

In the prior art, a segment or fragment corresponding to a GOP is transmitted on request to the player, who considers this alone

playable segment views. After playing the segment, the player requests a new segment. This method has technical limitations in the minimum length of the GOP and the access and request times between the units.

According to the invention, the GOP limit is canceled. Many small frames are transmitted and received as a data stream.

In a development of the invention, it may be provided that time information (time stamp, duration, seasons) is changed.

In development of the invention can be provided that audio packets

transmitted together or separately from video packets.

In development of the invention can be provided that packets are omitted or added.

Claims

claims:

1 . Method for transmitting video signals from a video signal source (1) to a display device (5), comprising the following method steps:

the signal is output from the video signal source (1) in a stream,

the streamed signal is fragmented into packets according to a known format,

wherein the packet size corresponds to at least one video image with associated audio information,

the packages are sent to the

Reproducing device (5),

the contents of the packets are displayed using the playback device (5).

2. The method of claim 1, wherein the packet fragmentation at the

Video source (1) takes place.

3. The method of claim 1, wherein the packet fragmentation using a separate from the video source (1) server (3).

4. The method according to any one of the preceding claims, wherein the

Package fragmentation in the playback device (5) takes place.

5. The method according to any one of the preceding claims, wherein each packet is provided with a time stamp.

6. The method according to any one of the preceding claims, wherein in a format incompatible for the playback device of the signal source (1) streams this is converted by the packaging unit in one of the playback device (5) supported format.

7. The method according to any one of the preceding claims, wherein the

Timestamp of the incoming real-time data to that for the correct one

Play required dimensions to be adjusted.

8. The method according to any one of the preceding claims, wherein the

Fragmentation is independent of keyframes and GOP length.

9. The method according to any one of the preceding claims, wherein the

Packet fragmentation in fragmented MP4 format.

10. The method according to any one of claims 1 to 8, wherein the

Package fragmentation in the HLS format is present.

1 1. A method according to any one of the preceding claims, wherein the

Time information (time stamp, time duration, game times) can be changed.

12. The method according to any one of the preceding claims, wherein the audio packets are transmitted together or separately from video packets.

13. The method according to any one of the preceding claims, wherein the packages

omitted or added.