DE102020100695A1 - Method and device for the transmission of multi-view video data - Google Patents

Abstract

A device (250) for coding N image sequences, with N> 1, is described, an image sequence comprising a sequence of images (115) for a corresponding sequence of points in time. The device (250) is set up to process at least part of the N image sequences as a function of a target bit rate (117) in order to provide N preprocessed image sequences, a preprocessed image sequence being a sequence of preprocessed images (205) for the corresponding sequence of Time includes. The device (250) is further configured to multiplex N preprocessed images (205) for a respective point in time to form an overall image (116) for the sequence of points in time, in order to provide a sequence of overall images (116) for the sequence of points in time. Furthermore, the device (250) is set up to encode the sequence of overall images (116) by means of an encoding module (114) as a function of the target bit rate (117) in an encoded bit stream (118).

Description

The invention relates to methods and devices for the transmission of multi-view video data over a bandwidth-limited transmission path.

A vehicle can include a plurality of image cameras, each of which is designed to capture video data relating to the surroundings of the vehicle. The video data can in each case reproduce at least partially different views and / or perspectives of the surroundings.

For remote control of the vehicle by means of a remote control unit, it may be necessary to send the video data from the plurality of cameras, i.e. the multi-view video data, to the remote control unit via a transmission link, e.g. via a wireless communication link. The multi-view video data should be transmitted with the smallest possible time delay and / or with the highest possible image quality, even if the transmission path has a limited bandwidth.

To reduce the bit rate of video data, a hardware-based coding module, in English "hardware-based encoder", is typically used, which is set up to convert the video data into an encoded bit stream using a video compression method, such as H.264 bandwidth-limited transmission path is transmitted. The provision of a hardware-based coding module is associated with relatively high costs and with additional weight and space requirements in a vehicle.

The present document deals with the technical task of transmitting multi-view video data in the most efficient, high-quality and flexible manner possible over a bandwidth-limited transmission path.

The problem is solved by each of the independent claims. Advantageous embodiments are described, inter alia, in the dependent claims. It is pointed out that additional features of a patent claim dependent on an independent patent claim without the features of the independent patent claim or only in combination with a subset of the features of the independent patent claim can form a separate invention independent of the combination of all features of the independent patent claim, which can be made the subject of an independent claim, a divisional application or a subsequent application. This applies equally to technical teachings described in the description, which can form an invention that is independent of the features of the independent patent claims.

According to one aspect, an apparatus for coding N image sequences, with N> 1, is described. The N image sequences can, for example, have been recorded by corresponding N cameras. For example, the N image sequences can correspond to N different views of the surroundings of a vehicle. The N image sequences can thus be multi-view video data. The device can be part of a transmission unit, which in turn can be part of a vehicle (the vehicle comprising the N cameras). The device can encode the N image sequences into an encoded bit stream, wherein the encoded bit stream can be sent to a receiving unit (e.g. to a remote control unit for remote control of a vehicle) via a (bandwidth-limited) transmission path.

An image sequence comprises a sequence of images for a corresponding sequence of points in time. There can therefore be N different sequences of images that are to be coded together in a coded bit stream (e.g. to provide a teleoperator with N different views of the surroundings of a vehicle). The individual image sequences can have a certain image rate, e.g. of 20 images or frames / second or more, or of 30 images or frames / second or more.

The device is set up to process at least part of the N image sequences as a function of a target bit rate for the encoded bit stream in order to provide N preprocessed image sequences, a preprocessed image sequence comprising a sequence of preprocessed images for the corresponding sequence of times. In other words, at least some of the N image sequences can each be preprocessed individually in order to provide N preprocessed image sequences, of which at least a part differs from the original N image sequences. On the other hand, a complementary part of the N preprocessed image sequences can be identical to the corresponding original N image sequences.

The preprocessing can take place taking into account the target bit rate for the coded bit stream in order to enable a subsequent coding module to encode the N preprocessed image sequences into an encoded bit stream that has the target bit rate and / or does not exceed it. The preprocessing can take place in such a way that the (perceptible) loss of information in the coding of the N preprocessed image sequences is smaller than the (perceptible) loss of information in the case of the N original Image sequences are encoded directly into the encoded bit stream. As a result of the preprocessing, information can be lost in one or more of the N image sequences in an image sequence-selective manner. The loss of information in an image sequence can be brought about, for example, by reducing the spatial resolution and / or the temporal resolution of the image sequence and / or by spatial and / or temporal low-pass filtering of the image sequence.

The device is further configured to multiplex N preprocessed images for a respective point in time to form an overall image for the sequence of points in time, in order to provide a sequence of overall images for the sequence of points in time. In other words, corresponding images of the N pre-processed image sequences for a specific point in time can be combined to form an overall image for the specific point in time (e.g. by arranging the individual images next to one another). This multiplexing process can be repeated for the sequence of points in time in order to provide a sequence of total images for the sequence of points in time, i.e. to provide a total image sequence.

Furthermore, the device is set up to encode the sequence of overall images by means of a (possibly hardware-based) encoding module as a function of the target bit rate in an encoded bit stream. In particular, the overall image sequence can be coded as a unit (by means of a single coding module). The encoded bit stream can then be sent to a receiving unit (e.g. to a remote control unit for a vehicle) via a transmission link.

A device is thus described which is designed to process one or more image sequences selectively and / or independently of one another in order to reduce the information content of one or more individual image sequences even before the common coding of the N image sequences. In this way (even when using a single coding module) the total available target bit rate can be flexibly distributed to the different image sequences.

The device can be set up to determine N quality targets for the N image sequences on the basis of the target bit rate for the encoded bit stream. In particular, the device can be set up to divide the target bit rate over the N image sequences. The bit rate available for the respective image sequence can then be displayed by the respective quality target for the respective image sequence.

The device can be set up to determine context information with regard to a relative priority of the N image sequences to one another. The context information can in particular indicate which one or more image sequences have a higher priority and / or are relatively important, and which one or more image sequences have a lower priority and / or are relatively unimportant.

The context information can at least partially be provided by the receiving unit (e.g. by a remote control unit) for which the coded bit stream is intended and / or to which the coded bit stream is sent. As an alternative or in addition, the context information can be provided at least partially by a vehicle that includes the N cameras for capturing the N image sequences.

The N quality goals for the N image sequences can then be determined in a precise manner on the basis of the context information.

As an alternative or in addition, the device can be set up to determine image information relating to the complexity of the N image sequences, in particular by means of a statistical analysis of the images of the N image sequences. The N quality goals for the N image sequences can then be determined on the basis of the image information. The quality target (in particular the proportion of the target bit rate) for a relatively complex image sequence can possibly be higher than for a relatively less complex image sequence.

The individual image sequences can then be processed as a function of the quality target for the respective image sequence in order to determine the preprocessed image sequence for the respective image sequence. The quality target for an image sequence can indicate how much the image sequence should be processed and / or how high the quality of the image sequence should be after coding (relative to the one or more other image sequences) and / or which one or more channels of an image sequence should be processed. A particularly flexible and precise coding of individual images can thus be made possible by a common coding module.

• • die räumliche Auflösung der vorbearbeiteten Bilder der entsprechenden vorbearbeiteten Bildsequenz;
• • die räumliche und/oder zeitliche Grenzfrequenz der (tiefpassgefilterten) vorbearbeiteten Bildsequenz;
• • die zeitliche Auflösung der vorbearbeiteten Bildsequenz;
• • die gewünschte Bit-Rate für die vorbearbeiteten Bildsequenz; und/oder
• • die ein oder mehreren Kanäle, der Bildsequenz, die bearbeitet bzw. die beibehalten werden sollen.

In particular, the quality target for an image sequence can indicate

• The spatial resolution of the preprocessed images of the corresponding preprocessed image sequence;
• • the spatial and / or temporal limit frequency of the (low-pass filtered) preprocessed image sequence;
• • the temporal resolution of the pre-processed image sequence;
• • the desired bit rate for the preprocessed image sequence; and or
• • the one or more channels of the image sequence that are to be edited or retained.

The device can be set up to change the quality goals for the individual image sequences over time. By changing the quality target for an image sequence, the type of processing of this image sequence is typically also changed. By adapting the quality goals for the individual image sequences, the focus can be flexibly adapted to different image sequences.

The device can be set up to reduce the temporal resolution of a specific image sequence in order to provide a corresponding specific preprocessed image sequence. The temporal resolution can be reduced in such a way that the frame rate remains unchanged. For example, an image of a specific image sequence for a specific point in time can be replaced by an image of the specific image sequence for a directly preceding point in time in order to provide the corresponding specific pre-processed image sequence. This process of replacing images of the specific image sequence can be repeated at a specific rate, the rate depending on the desired temporal resolution of the preprocessed image sequence.

The temporal resolution of the specific preprocessed image sequence and / or the rate at which images of the specific image sequence are replaced can depend on the target bit rate, in particular on the quality target for the specific image sequence. By reducing the temporal resolution of one or more image sequences, the information content of individual image sequences can be reduced in a particularly effective manner.

The device can be set up to filter at least one image of a specific image sequence by means of a spatial low-pass filter in order to provide a corresponding specific pre-processed image sequence. As an alternative or in addition, the device can be set up to filter a specific image sequence by means of a temporal low-pass filter along the sequence of points in time in order to provide the corresponding specific pre-processed image sequence. Spatial and / or temporal low-pass filtering of one or more image sequences can thus take place. The number of pixels and / or the frame rate of the individual image sequences typically remain unchanged.

The spatial and / or temporal low-pass filter, in particular the limit frequency of the spatial and / or temporal low-pass filter, can depend on the target bit rate, in particular on the quality target for the specific image sequence. The information content of individual image sequences can be selectively reduced in a particularly effective manner by means of a spatial and / or temporal low-pass filter.

The device can be set up to reduce the spatial resolution of at least one image of a specific image sequence in order to provide a corresponding preprocessed image of a corresponding specific preprocessed image sequence. For this purpose, if necessary, the values of pixels can be copied to neighboring pixels in order to effectively reduce the spatial resolution while leaving the number of pixels of an image unchanged. This enables the preprocessed images to be efficiently multiplexed to form an overall image.

In a preferred example, the number of pixels of at least one image of a specific image sequence can be reduced in order to provide a corresponding preprocessed image of the corresponding specific preprocessed image sequence.

The spatial resolution of the preprocessed image and / or the number of pixels in the preprocessed image can depend on the target bit rate, in particular on the quality target for the particular image sequence. By reducing the spatial resolution of the images of one or more image sequences, the information content of individual image sequences can be selectively reduced in a particularly effective manner.

As a result of the reduction in the spatial resolution, in particular as a result of the reduction in the number of pixels, of individual image sequences, a situation can arise in which the preprocessed images of different preprocessed image sequences are of different sizes. For example, a preprocessed image i, with i = 1,... N, Q _i × R _{i can have} pixels. Here, Q _i , R _i and / or Q _i × R _i can be different for at least some of the N preprocessed images.

The device can be set up to select a scheme from a plurality of different (predefined) schemes for an overall image. The scheme to be used can be selected as a function of the target bit rate and / or as a function of the N quality goals for the N image sequences. The N preprocessed images can then be multiplexed as a function of the selected scheme. Reliable multiplexing into an overall image can thus take place even with pre-processed images of different dimensions.

Diese Bedingung kann für alle Schemata der Mehrzahl von unterschiedlichen (vordefinierten) Schemata erfüllt sein. Dabei können sich unterschiedliche Schemata der Mehrzahl von unterschiedlichen Schemata in Bezug auf die Anordnung und/oder die Bildpunkt-Größe von Teilbereichen für die N vorbearbeiteten Bilder unterscheiden.A (predefined) scheme can comprise N sub-areas for the N preprocessed images for an overall image, the sub-area i for the preprocessed image i _{corresponding to a section with Q i} × R _i pixels of the overall image, with i = 1, ..., N The N partial areas of a scheme can be arranged in such a way that an overall image with Q × R image points results and so that ${\displaystyle {\sum }_{i=0}^N{Q}_i}\times {\mathrm{R.}}_i=Q\times \mathrm{R.}.$

This condition can be met for all schemes of the plurality of different (predefined) schemes. In this case, different schemes of the plurality of different schemes can differ with regard to the arrangement and / or the pixel size of partial areas for the N preprocessed images.

The device can be set up to effect the spatial resolution of the preprocessed images of the N preprocessed image sequences as a function of the selected scheme for the overall image. In particular, it can be achieved that the spatial resolution of one or more image sequences is reduced in such a way that the N pre-processed image sequences resulting therefrom can be combined to form an overall image sequence in accordance with the selected scheme. Reliable multiplexing of the N image sequences can thus be effected.

The device can also be set up to introduce information relating to the selected scheme as metadata into the coded bit stream, in particular to inform a corresponding decoding device about how the individual image sequences are arranged within the overall image sequence.

At least one image of a specific image sequence can comprise a plurality of channel images for a corresponding plurality of channels. In particular, the images of at least one specific image sequence can each include a plurality of channel images for a corresponding plurality of channels. In other words, at least one specific image sequence can comprise a plurality of channel image sequences for a corresponding plurality of channels. The plurality of channels can include, for example, a luminance channel, one or more chrominance channels, and / or one or more color channels. Exemplary channels of a picture sequence are the channels Y (luminance) as well as U and V (each chrominance), or Y (luminance) as well as Cb and Cr (each chrominance)

The device can be set up to process at least one channel image of the plurality of channel images in order to provide a corresponding preprocessed image of the corresponding specific preprocessed image sequence. In particular, the channel images can be processed for at least one channel of the specific image sequence in order to provide the corresponding specific pre-processed image sequence. By processing the channel images of one or more channels of at least one image sequence, the information content of the respective image sequence can be selectively reduced in a particularly effective manner.

The processing of one or more channel images of a channel of a specific image sequence can include at least one of the processing methods described in this document. In particular, the spatial resolution of at least one channel image can be reduced in the course of processing (as generally described in this document for an image). Alternatively or in addition, at least one channel image can be filtered by means of a spatial low-pass filter (as described in general for an image in this document). As an alternative or in addition, the sequence of channel images of a channel of the specific image sequence can be filtered with a temporal low-pass filter (as generally described in this document for a sequence of images). As an alternative or in addition, the time resolution of the sequence of channel images of a channel of the specific image sequence can be reduced (as generally described in this document for a sequence of images). The processing methods described in this document for the pre-processing of images can thus be applied to the channel images of one or more channels of an image sequence. The information content of an image sequence can thus be reduced in a particularly selective manner.

As an alternative or in addition, the values of pixels of the respective channel image can be set to zero within the framework of the processing of the channel images of a channel of a specific image sequence. For example, the channel images of the one or more (in particular the two or all) chrominance channels of the specific image sequence can be set to zero. In this way, the information content of the specific image sequence can be reduced in a particularly effective manner.

• • die ein oder mehreren Bildsequenzen auszuwählen, für die eine Bearbeitung von jeweils ein oder mehreren Kanälen erfolgen soll;
• • den Kanal einer bestimmten Bildsequenz zu ermitteln, der bearbeitet werden soll;
• • die Anzahl von Kanälen zu ermitteln, die bearbeitet werden soll; und/oder
• • die Form bzw. die Methode der Bearbeitung des jeweiligen Kanals zu ermitteln.

The device can be set up as a function of the target bit rate, in particular as a function of the N quality targets for the N image sequences,

• • select the one or more image sequences for which one or more channels are to be processed in each case;
• • determine the channel of a particular image sequence to be processed;
• • determine the number of channels to be processed; and or
• • to determine the form or the method of processing the respective channel.

In this way, the information content of individual image sequences can be selectively reduced in a particularly effective manner.

According to a further aspect, a device for decoding an encoded bit stream for N image sequences, with N> 1, is described, an image sequence comprising a sequence of images for a corresponding sequence of points in time. The device can be part of a receiving unit.

The device can be set up to decode the coded bit stream into a sequence of decoded overall images by means of a (single) decoding module. Furthermore, the device can be set up to determine a scheme according to which N decoded images for the N image sequences are multiplexed into a decoded overall image. For example, the device can be designed to extract information relating to the scheme from the bitstream. The device can also be set up, depending on the determined scheme, to extract N sequences of decoded images from the sequence of decoded overall images.

According to a further aspect, a (road) motor vehicle (in particular a passenger car or a truck or a bus) is described which comprises the (coding) device and / or transmission unit described in this document.

According to a further aspect, a method for coding N image sequences, with N> 1, is described, an image sequence comprising a sequence of images for a corresponding sequence of points in time. The method comprises the processing of at least part of the N image sequences as a function of a target bit rate in order to provide N preprocessed image sequences, a preprocessed image sequence comprising a sequence of preprocessed images for the corresponding sequence of points in time. Furthermore, the method comprises multiplexing, for the sequence of points in time, N preprocessed images in each case for a respective point in time to form an overall image in order to provide a sequence of overall images for the sequence of points in time. The method further comprises encoding the sequence of overall images as a function of the target bit rate in an encoded bit stream.

According to a further aspect, a method for decoding an encoded bit stream for N image sequences, with N> 1, is described, an image sequence comprising a sequence of images for a corresponding sequence of points in time. The method comprises decoding the encoded bit stream into a sequence of decoded overall images. Furthermore, the method comprises the determination of a scheme according to which N decoded images for the N image sequences are multiplexed into a decoded overall image. In addition, the method comprises the extraction of N sequences of decoded pictures from the sequence of decoded overall pictures as a function of the determined scheme.

According to a further aspect, a software (SW) program is described. The software program can be set up to be executed on a processor (e.g. on a control unit of a vehicle) and thereby to execute one of the methods described in this document.

According to a further aspect, a storage medium is described. The storage medium can comprise a software program which is set up to be executed on a processor and thereby to execute one of the methods described in this document.

It should be noted that the methods, devices and systems described in this document can be used both alone and in combination with other methods, devices and systems described in this document. Furthermore, any aspects of the methods, devices and systems described in this document can be combined with one another in diverse ways. In particular, the features of the claims can be combined with one another in diverse ways.

• 1a ein beispielhaftes System zur Übertragung von Multi-View-Videodaten;
• 1b ein beispielhaftes Gesamtbild mit einer Mehrzahl von Bildern von einer entsprechenden Mehrzahl von Kameras;
• 2a ein beispielhaftes System zur Übertragung von Multi-View-Videodaten mit Vorbearbeitungsmodulen zur Vorbearbeitung der Videodaten der einzelnen Kameras;
• 2b eine beispielhafte Vorrichtung zur Kodierung von Multi-View-Videodaten;
• 3a ein Ablaufdiagramm eines beispielhaften Verfahrens zur Kodierung von Multi-View-Videodaten; und
• 3b ein Ablaufdiagramm eines beispielhaften Verfahrens zur Dekodierung eines Bitstroms mit Multi-View-Videodaten.

The invention is described in more detail below on the basis of exemplary embodiments. Show it

• 1a an exemplary system for transmitting multi-view video data;
• 1b an exemplary overall image with a plurality of images from a corresponding plurality of cameras;
• 2a an exemplary system for the transmission of multi-view video data with preprocessing modules for preprocessing the video data of the individual cameras;
• 2 B an exemplary apparatus for encoding multi-view video data;
• 3a a flow diagram of an exemplary method for encoding multi-view video data; and
• 3b a flowchart of an exemplary method for decoding a bit stream with multi-view video data.

As stated at the outset, the present document deals with the efficient and flexible transmission of multi-view video data, for example for the transmission of multi-view video data from a vehicle to a remote control unit which is designed for remote control of the vehicle. In this context shows 1a an exemplary system 100 with a transmitter unit 110 that is set up a bit stream 118 , 128 with multi-view video data via a transmission link 101 to a receiving unit 120 to send. The transmitter unit 110 can be part of a vehicle, for example. The receiving unit 120 can be part of a remote control unit that makes it a teleoperator 130 enables the vehicle to be controlled remotely.

The sending unit 110 includes a plurality of cameras 111 , with each camera 111 is set up, video data, ie an image sequence or a temporal sequence of images 115 , capture. The individual cameras 111 can each have detection areas that at least partially differ from one another. For example, the individual cameras 111 each provide different perspectives of the surroundings and / or different views of the surroundings of a vehicle.

The sequence of images from a camera 111 can be provided with a specific frame rate, in particular a specific frame rate. Example frame rates are 30fps (frames per second) or 60fps. The image sequence (ie the video data) from a camera 111 can thus be a sequence of images 115 for a corresponding sequence of points in time, with directly successive points in time from the sequence of points in time according to the frame rate of the camera 111 are spaced apart in time.

The transmitter unit 110 can use a multiplexer 112 include, which is set up, at a specific point in time of the sequence of points in time, the images 115 of the different cameras 111 to an overall picture 116 summarize. The big picture 116 can be referred to as "superframe" in English.

Consequently, the multiplexer 112 be designed, a sequence of total images for the sequence of points in time 116 provide, with the bigger picture 116 the plurality of images for a specific point in time 115 the majority of cameras 111 for the specific point in time.

1b shows an exemplary overall picture 116 that the pictures 115 of the individual cameras 111 as part of the overall picture 116 includes. The big picture 116 , especially the sequence of complete images 116 , can through a coding module 114 , in particular by a hardware-based coding module, to be coded to the bit stream 118 , 128 provide. Thereby the sequence of total images 116 from the coding module 114 processed as if it were the sequence of complete images 116 be video data or a sequence of images from a single camera. The multiplexing of the video data from the individual cameras 111 into a (single) sequence of total images 116 , ie in a (single) image sequence, thus enables multi-view video data in an efficient manner using a single (hardware-based) coding module 114 to encode.

The coding module 114 can be set up, the bit rate of the generated bit stream 118 , 128 to change. In particular, the coding module 114 be set up, the bit rate of the generated bit stream to a current bandwidth situation on the transmission link 101 adapt. To this end, the degree of compression of the sequence of total images 116 can be changed, which in turn affects the perceptual quality or image quality (in English "perceptual quality") of the sequence of compressed overall images 116 Has. As the bit rate falls, the quality of perception of the sequence of compressed overall images typically falls 116 , typically evenly for all images 115 within a compressed overall image 116 .

The sending unit 110 can be a bit rate module 113 which is set up based on state information 102 with regard to the condition, in particular with regard to the available bandwidth, of the transmission link 101 a target bit rate 117 for the encoded bit stream 118 , 128 to determine. The coding module 114 can then depending on the target bit rate 117 operated to the sequence of complete images 116 to compress, and so the encoded bit stream 118 , 128 for the multi-view video data. The coding module 114 can be operated in such a way that the bit stream 118 , 128 (at least on average over time) the target bit rate 117 has or at least does not exceed. The coding module 114 can be designed for this purpose to carry out a video compression method (e.g. H.264).

The receiving unit 120 typically comprises a (possibly hardware-based) decoding module 124 that is set up to receive a Bitstream 128 for multi-view video data to decode to a sequence of decoded complete images 126 provide. The individual decoded overall images 126 can in a de-multiplexer 122 be demultiplexed to a plurality of decoded pictures 125 provide. The majority of sequences of decoded pictures 125 can for example on a corresponding plurality of screens 121 be reproduced (e.g. to make it a teleoperator 130 to make it possible to see the different views of the surroundings of a vehicle).

As already stated above, the reduction of the target bit rate results 117 in the in 1a illustrated system 100 typically to a uniform degradation of the quality of the decoded images 125 for all cameras 111 or views. The different views can at least partially have a different degree of importance, in particular for teleoperation by a teleoperator 130 . The degree of importance of the different views can change over time, for example depending on the traffic situation in which a remote-controlled vehicle is located.

2 B shows a system 100 for the transmission of multi-view video data, which allows flexible adaptation of the bit rates and thus the quality of perception of the different images 115 or views of the multi-view video data. The sending unit 110 of the system 100 includes preprocessing modules 220 for the pictures 115 of the different cameras 111 . A preprocessing module 220 for a camera 111 can be set up a picture 115 or a sequence of images 115 the camera 111 edit to a preprocessed image 205 or a sequence of pre-processed images 205 (ie a preprocessed image sequence). The majority of sequences of preprocessed images 205 (ie the plurality of preprocessed image sequences) for the plurality of cameras 111 can then in the multiplexer 112 to a sequence of complete images 116 be summarized.

• • die angestrebte zeitliche Auflösung, insbesondere die Bildrate, der vorbearbeiteten Bildsequenz;
• • die angestrebte räumliche Auflösung, insbesondere die Anzahl von Bildpunkten, eines vorbearbeiteten Bildes 115;
• • die angestrebte Bitrate für die vorbearbeitete Bildsequenz; und/oder
• • die ein oder mehreren Kanäle (z.B. der Luminanz-Kanal oder die ein oder mehreren Chrominanz-Kanäle) der Bildsequenz, die vorbearbeitet werden sollen.

The preprocessing of an image 115 or an image sequence, depending on a quality target 204 for the picture 115 or for the image sequence. The quality goal 204 can for example show

• • the desired temporal resolution, in particular the frame rate, of the preprocessed image sequence;
• • the desired spatial resolution, in particular the number of pixels, of a preprocessed image 115 ;
• • the desired bit rate for the preprocessed image sequence; and or
• • the one or more channels (eg the luminance channel or the one or more chrominance channels) of the image sequence that are to be preprocessed.

The quality goals 204 for the different image sequences from the different cameras 111 can in a distribution module 210 be determined, which is set up, the total available target bit rate 117 to divide up the different image sequences in order to achieve the quality goals 204 to determine for the different image sequences. Context information 201 are taken into account, the context information 201 eg shows which view, ie which image sequence, for the teleoperator 130 is important and which view, ie which image sequence, for the teleoperator 130 is less important. Alternatively or in addition, the context information 201 show the status and / or the current traffic situation of the remote-controlled vehicle.

The distribution module 210 can also be set up, the quality objectives 204 for the different sequences of images 115 based on image information 203 in terms of the pictures 115 of the different cameras 111 to determine. The image information 203 in terms of the pictures 115 a camera 111 can, for example, the complexity of the images 115 in terms of image compression 115 Show. Alternatively or in addition, the image information 203 one or more statistical parameters of the images 115 show or include.

The transmitter unit 110 can thus be set up for the different cameras 111 each a sequence of preprocessed images 205 (ie a preprocessed image sequence in each case) to be provided, each of which has been individually adapted in order to bring about that the respective sequence of preprocessed images 205 in the context of coding the sequence of overall images 116 in the following coding module 114 an individually adjustable quality and / or an individually adjustable proportion of the target bit rate 117 having. Through the individual pre-processing of the image sequences from the different cameras 111 This enables flexible adaptation of the compression quality and / or the bit rate for the image sequences of the different cameras 111 be effected, even if the image sequences are shared by a common coding module 114 be coded.

2 B shows an exemplary device 250 for coding a plurality of sequences of images 115 , e.g. as part of the transmitter unit 110 out 2a . The preprocessing module 220 for the sequence of images 115 a camera 111 can be a first sub-module 221 to adjust the temporal resolution, a second sub-module 222 for customization the spatial resolution, a third sub-module 223 to adapt the bit rate and / or a fourth sub-module 224 to adjust the channels of an image 115 a camera 111 include.

The first sub-module 201 (to adjust the temporal resolution) can be set up, the frame rate of the sequence of images 115 to reduce. For this purpose, for example, individual images 115 deleted, and through the previous images 115 replaced so that the appropriate sequence of preprocessed images 205 at least intermittently identical images at successive points in time 205 having. For example, halving the frame rate can result in pairs of consecutive frames 205 the sequence of preprocessed images 205 are identical. The provided sequence of preprocessed images 205 then has the same number of images as the original sequence of images 115 so that the multiplexer 112 furthermore a preprocessed image for each point in time of the sequence of points in time 205 to be provided. However, pictures 205 be identical at directly consecutive points in time, so that the sequence of preprocessed images 205 can typically be compressed at a lower bit rate than the original sequence of images 115 . For example, by the coding module 114 a so-called skip mode can be brought about when it is recognized that a partial area of an overall image to be compressed at a current point in time 116 identical to the corresponding part of the overall picture 116 is at the immediately preceding point in time.

The second sub-module 222 (to adapt the spatial resolution) can be set up, the number of pixels and / or the spatial resolution of the images 115 to reduce. The sequence of preprocessed images 205 can thus have a lower pixel resolution than the original sequence of images 115 . Reducing the spatial resolution can reduce the bit rate required to compress a sequence of images 115 be reduced.

It can thus through the different preprocessing modules 220 for the different cameras 111 or sequences of pre-processed images for the different views 205 are provided that have at least partially different spatial resolutions. The multiplexer 112 can be set up to include the plurality of preprocessed images 205 for a point in time that have at least partially different spatial resolutions, to form a rectangular overall image 116 summarized, as exemplified in 2 B shown.

The multi-view video data to be transmitted can, for example, have N different views, that is to say N different sequences of images 115 , include, with N> 1, eg N = 2, 3, 4, or 5. One or more schemes can be defined that allow the N preprocessed images 205 for the N views to a rectangular overall picture 116 summarize. The N preprocessed images 205 have at least partially different pixel resolutions. Adjusting the spatial resolution of the different images 115 for the different views can then be done in a coordinated manner, so that the N preprocessed images 205 for the N views according to a scheme of the one or more predefined schemes to form a rectangular overall image 116 can be summarized.

The third sub-module 223 to reduce the bit rate of a sequence of images 115 The sequence of images can be designed for a view 115 to filter spatially (ie along at least one spatial axis) and / or temporally (ie along the time axis) (without changing the spatial and / or temporal resolution). In this case, in particular, low-pass filtering can take place in order to remove the high-frequency spatial and / or temporal component in the sequence of images 115 to reduce. The cut-off frequency of the low-pass filter can be adapted in order to adapt the bit rate that is used for the coding of the provided sequence of preprocessed images 205 is needed.

The one or more pictures 115 a camera 111 can have several different channels. Exemplary channels are a luminance channel that determines the brightness of the individual pixels of one or more images 115 indicates, and one or more (in particular two) chrominance channels, which the chromaticity or the color of the individual pixels of the one or more images 115 Show. Alternatively or in addition, a picture 115 have different channels for different basic colors (e.g. R (red), G (green) and B (blue)). A picture 115 a camera 111 can thus also put together several channel images for several channels.

The fourth sub-module 224 can be set up to reduce the coding complexity of one or more channels (ie of one or more channel pictures) of the one or more pictures 115 , especially the sequence of images 115 , a camera 111 to reduce. For example, depending on the situation, one or more color channels can be used for a teleoperator 130 be less important than another color channel. In particular, it may possibly be possible (at least temporarily) the one or more (in particular the two or all) chrominance channels of one or more images 115 to set to zero, and / or just the luminance channel of the one or more images 115 to maintain. One or more pre-processed images can thus be used 205 are provided in each of which one or more channels (in particular the one or more, in particular all, chrominance channels) have been set to zero or removed.

As a result, the fourth sub-module 224 one or more pre-processed images 205 are provided within the corresponding one or more overall images 116 with a reduced bit rate 117 through the coding module 114 can be coded (e.g. because in the partial area of the one or more overall images 116 for the one or more pre-processed images 205 only the luminance channel, but not the one or more chrominance channels, need to be coded).

2 B also shows details of the distribution module 210 who is trained, the quality goals 204 for the different views, ie for the different image sequences, to be determined. The quality goals 204 for the different image sequences can be adapted or changed over time. The distribution module 210 can be a sub-module 211 include that is set up to identify one or more views from the plurality of views that are relatively important at a particular point in time. The fraction of the target bit rate available 117 for the identified one or more views can then be increased (e.g. in the sub-modules 212 , 213 ). Furthermore, a higher-level video quality management 214 a coding mode 215 the one in the coding module 114 The video compression method used is selected and the target bit rate is distributed 117 on the N image sequences or when determining the N quality goals 204 must be taken into account.

3a shows a flow diagram of an exemplary (computer-implemented) method 300 for coding N image sequences, with N> 1, where an image sequence is a sequence of images 115 for a corresponding sequence of times. The procedure 300 can, for example, by a transmitter unit 110 and / or by a coding device 250 are executed.

The procedure 300 includes editing 301 at least some of the N image sequences as a function of a target bit rate 117 to provide N preprocessed image sequences. A preprocessed image sequence has a sequence of preprocessed images 205 for the corresponding sequence of times.

• • die vorbearbeitete Bildsequenz eine niedrigere informationstheoretische Entropie aufweist als die ursprüngliche Bildsequenz; und/oder
• • die vorbearbeitete Bildsequenz mit einer niedrigeren Bitrate kodiert werden kann als die ursprüngliche Bildsequenz.

The target bit rate 117 can, for example, depend on the available bandwidth of the transmission link 101 for the transmission of the bit stream 118 for the N image sequences depend. The editing 301 can be designed to provide a corresponding preprocessed image sequence for at least one of the N image sequences, such that

• • the preprocessed image sequence has a lower information-theoretical entropy than the original image sequence; and or
• • the preprocessed image sequence can be encoded with a lower bit rate than the original image sequence.

The editing 301 can, for example, reduce the spatial resolution of the images 115 an image sequence, the reduction of the temporal resolution of an image sequence, the low-pass filtering of the images 115 an image sequence, and / or the processing of one or more channels of an image 115 comprise an image sequence.

It should be pointed out that only part of the N image sequences may be processed, so that only part of the N pre-processed image sequences may differ from the corresponding original image sequences. In other words, one or more of the preprocessed image sequences can be identical to the corresponding one or more original image sequences. On the other hand, at least one preprocessed image sequence differs from the corresponding original image sequence.

The procedure 300 further comprises multiplexing 302 , for the sequence of points in time, each of N preprocessed images 205 for a given point in time for an overall picture 116 to create a sequence of complete images 116 for the sequence of points in time to be provided. In other words, for each point in time of the sequence of points in time there can be N preprocessed images 205 of the N preprocessed image sequences to form an overall image 116 be summarized. For this purpose the N (rectangular) pre-processed images 205 can be arranged side by side to form a (rectangular) overall picture 116 provide.

The method also includes 300 coding 303 the sequence of total images 116 depending on the target bit rate 117 into an encoded bit stream 118 . Thereby the sequence of total images 116 can be viewed as an overall picture sequence, which is processed as a whole by a (single) coding module 114 is encoded.

3b shows a flow diagram of an exemplary (computer-implemented) method 310 for decoding an encoded bit stream 128 for N image sequences, with N> 1, where an image sequence is a sequence of images 115 for a corresponding sequence of times. The encoded bit stream 128 can in particular represent N decoded image sequences, it being possible for the N decoded image sequences to have been combined in a (decoded) overall image sequence. The procedure 310 can through a receiving unit 120 are executed.

The procedure 310 includes decoding 311 of the encoded bit stream 128 into a sequence of decoded total images 126 (ie in a decoded overall picture sequence). This can be done using a decoding module 124 respectively.

The method also includes 310 determining 312 of a scheme, according to the N decoded pictures 125 for the N image sequences into a decoded overall image 126 have been multiplexed. In particular, the scheme can indicate how the individual images 115 when multiplexing have been arranged side by side to form an overall picture 116 provide that then in the bitstream 128 was coded. Information relating to the scheme used can be, for example, metadata within the bit stream 128 to be provided.

The method also includes 310 extracting 313 of N sequences of decoded pictures 125 from the sequence of decoded total images 126 depending on the determined scheme.

The measures described in this document allow multi-view video data (that is to say N image sequences) over a bandwidth-limited transmission path in a particularly efficient and flexible manner 101 be transmitted.

The present invention is not restricted to the exemplary embodiments shown. In particular, it should be noted that the description and the figures are only intended to illustrate the principle of the proposed methods, devices and systems by way of example.

Claims (23)

Apparatus (250) for coding N image sequences, with N> 1; wherein an image sequence comprises a sequence of images (115) for a corresponding sequence of points in time; wherein the device (250) is set up, - to process at least part of the N image sequences as a function of a target bit rate (117) in order to provide N pre-processed image sequences; wherein a preprocessed image sequence comprises a sequence of preprocessed images (205) for the corresponding sequence of points in time; - for the sequence of points in time, multiplexing N pre-processed images (205) for a respective point in time to form an overall image (116) in order to provide a sequence of overall images (116) for the sequence of points in time; and - to encode the sequence of overall images (116) by means of an encoding module (114) as a function of the target bit rate (117) in an encoded bit stream (118). Device (250) according to Claim 1 wherein the device (250) is set up to determine N quality targets (204) for the N image sequences on the basis of the target bit rate (117) for the coded bit stream (118); and - to process an image sequence as a function of the quality target (204) for the image sequence in order to determine the preprocessed image sequence for the image sequence. Device (250) according to Claim 2 wherein the device (250) is set up to determine context information (201) with regard to a relative priority of the N image sequences to one another; and - to determine the N quality goals (204) for the N image sequences on the basis of the context information (201). Device (250) according to Claim 3 wherein - the context information (201) is at least partially provided by a receiving unit (120) for which the coded bit stream (118) is intended and / or to which the coded bit stream (118) is sent; and / or - the context information (201) is provided at least partially by a vehicle which comprises N cameras (111) for capturing the N image sequences. Device (250) according to one of the Claims 2 to 4th wherein the device (250) is set up to determine image information (203) in relation to a complexity of the N image sequences, in particular by means of a statistical analysis of the images (115) of the N image sequences; and - to determine the N quality targets (204) for the N image sequences on the basis of the image information (203). Device (250) according to one of the preceding claims, wherein the device (250) is set up - to reduce a temporal resolution of a specific image sequence in order to provide a corresponding specific pre-processed image sequence; and or to replace an image of a specific image sequence for a specific point in time with an image of the specific image sequence for a directly preceding point in time in order to provide a corresponding specific pre-processed image sequence. Device (250) according to Claim 6 , with the temporal resolution of the particular preprocessed image sequence and / or a rate at which images of the particular image sequence are replaced depend on the target bit rate (117), in particular on a quality target (204) for the particular image sequence that is dependent on the target bit rate (117). Device (250) according to one of the preceding claims, wherein the device (250) is set up - to filter at least one image (115) of a specific image sequence by means of a spatial low-pass filter in order to provide a corresponding specific pre-processed image sequence; and or to filter a specific image sequence by means of a temporal low-pass filter along the sequence of points in time in order to provide a corresponding specific pre-processed image sequence. Device (250) according to Claim 8 wherein the spatial and / or temporal low-pass filter, in particular a cutoff frequency of the spatial and / or temporal low-pass filter, depends on the target bit rate (117), in particular on a quality target (204) for the particular image sequence that is dependent on the target bit rate (117) , depend. Device (250) according to one of the preceding claims, wherein the device (250) is set up to reduce a spatial resolution of at least one image of a specific image sequence in order to provide a corresponding preprocessed image of a corresponding specific preprocessed image sequence; and or to reduce a number of pixels of at least one image of a specific image sequence in order to provide a corresponding preprocessed image of a corresponding specific preprocessed image sequence. Device (250) according to Claim 10 , wherein the spatial resolution of the preprocessed image and / or the number of pixels of the preprocessed image depend on the target bit rate (117), in particular on a quality target (204) for the particular image sequence that is dependent on the target bit rate (117). Device (250) according to one of the Claims 10 to 11 wherein - the device (250) is set up to select a scheme from a plurality of different schemes for an overall image (116); - A scheme for an overall image (116) comprises N partial areas for the N preprocessed images (205); - has a preprocessed image i (205), with i = 1, ..., N, Q _i × R _i pixels; - the sub-area i for the preprocessed image i (205 _{) corresponds to a section with Q i} × R _i image points of the overall image (116); and - the N sub-areas of a scheme are arranged in such a way that an overall image (116) with Q × R pixels results and so that ${\displaystyle {\sum }_{i=1}^N{Q}_i\times {\mathrm{R.}}_i}=Q\times \mathrm{R.}.$

Device (250) according to Claim 12 , where Q _i , R _i and / or Q _i × R _i are different for at least some of the N preprocessed images (205). Device (250) according to one of the Claims 12 to 13th wherein different schemes of the plurality of different schemes differ with regard to the arrangement and / or the pixel size of partial areas for the N preprocessed images (205). Device (250) according to one of the Claims 12 to 14th wherein the device (250) is set up to effect the spatial resolution of the preprocessed images (205) of the N preprocessed image sequences as a function of the selected scheme for the overall image (116); and / or - to select the scheme as a function of the target bit rate (117), in particular as a function of N quality goals (204) for the N image sequences. Device (250) according to one of the preceding claims, wherein - At least one image (115) of a specific image sequence comprises a plurality of channel images for a corresponding plurality of channels; and - the device (250) is set up to process at least one channel image of the plurality of channel images in order to provide a corresponding preprocessed image (205) of a corresponding specific preprocessed image sequence. Device (250) according to Claim 16 , wherein the device (250) is set up to - reduce a spatial resolution of the channel image; filter the channel image using a spatial low-pass filter; - to filter a sequence of channel images of the specific image sequence with a temporal low-pass filter, - to reduce a temporal resolution of the sequence of channel images; and / or - to set values of pixels of the channel image to zero. Device (250) according to one of the Claims 16 to 17th wherein the plurality of channels comprises - a luminance channel; - At least one chrominance channel, in particular two chrominance channels; and / or - one or more color channels. Device (250) according to one of the Claims 16 to 18th , wherein the device (250) is set up, as a function of the target bit rate (117), in particular as a function of N quality targets (204) for the N image sequences, - the channel that is processed, - a number of channels that are processed, and / or - a form of the Processing of the channel. Device (250) according to one of the preceding claims, wherein the device (250) is set up to send the coded bit stream (118) via a transmission link (101) to a receiving unit (120). Apparatus (120) for decoding an encoded bit stream (128) for N image sequences, with N> 1; wherein an image sequence comprises a sequence of images (115) for a corresponding sequence of points in time; wherein the device (120) is set up, - to decode the encoded bit stream (128) into a sequence of decoded overall images (126) by means of a decoding module (124); - to determine a scheme according to which N decoded images (125) for the N image sequences are multiplexed into a decoded overall image (126); and - as a function of the determined scheme, to extract N sequences of decoded images (125) from the sequence of decoded overall images (126). Method (300) for coding N image sequences, with N> 1; wherein an image sequence comprises a sequence of images (115) for a corresponding sequence of points in time; wherein the method comprises (300), Processing (301) at least part of the N image sequences as a function of a target bit rate (117) in order to provide N pre-processed image sequences; wherein a preprocessed image sequence comprises a sequence of preprocessed images (205) for the corresponding sequence of points in time; - multiplexing (302), for the sequence of points in time, of in each case N preprocessed images (205) for a respective point in time to form an overall image (116) in order to provide a sequence of overall images (116) for the sequence of points in time; and - Coding (303) the sequence of total images (116) as a function of the target bit rate (117) in an encoded bit stream (118). Method (310) for decoding an encoded bit stream (128) for N image sequences, with N> 1; wherein an image sequence comprises a sequence of images (115) for a corresponding sequence of points in time; wherein the method comprises (310), - Decoding (311) the coded bit stream (128) into a sequence of decoded total images (126); - determining (312) a scheme according to which N decoded images (125) for the N image sequences are multiplexed into a decoded overall image (126); and - Extracting (313) N sequences of decoded images (125) from the sequence of decoded overall images (126) as a function of the determined scheme.

Images