LU102424B1 - Method for playing a video stream by a client - Google Patents

Abstract

The invention relates to a method for playing back a video stream by a client (C), the video stream comprising frames from a camera in relation to a moving object from different positions, comprising the steps of -receiving a video stream (VB) from an encoder (SRV ), -Decoding of the obtained video stream (VB) using camera parameters (CP) and geometry data (GD), - Playback of the processed video stream (AVB).

Description

Our reference: THA 47446 P DE 12.01.2021 Application No.: NN LU102424 Title: Method for playing a video stream by a client -1- Method for playing a video stream by a client Background Digital coding of video signals is in many forms these days to find. For example, digital video streams are stored on data carriers such as DVDs, Blu-ray or as a download or

Video stream (e.g. also for video communication) provided. The aim of video coding is not only to send representations of the images to be transmitted, but also to keep data consumption low at the same time. On the one hand, this allows more content to be stored on media with limited storage space, such as DVDs, or to enable several users to transport (different) video streams at the same time.

A distinction is made between lossless and lossy coding.

What all approaches have in common is that information for subsequent images is predetermined from previously transmitted images.

Current analysis estimates that such encoded video will account for 82% of all network traffic by 2022 (up from 75% in 2017), see Cisco Visual Networking Index: Forecast and trends, 2017-2022 (white paper ), Cisco, Feb.

2019

253 From this it can be seen that any saving that can be achieved here will lead to a large saving in data volume and thus to a saving in electrical energy for transport. as a rule, a coder, a carrier medium, e.g. a transmission channel, and a decoder are required. The encoder processes raw video data. As a rule, a single image is referred to as a frame. Again, a frame can be thought of as a collection of pixels. A pixel represents a point in the frame and indicates its color value and/or its brightness.

Our reference: THA 47446 P DE 12.01.2021 Application no.: NN LU102424 Title: Method for playing back a video stream by a client -2- The amount of data for a subsequent frame can be reduced, for example, if a large part of the information is already in one or more previously encoded frame(s). Then it would be sufficient, for example, if only the difference is transmitted. In doing so, one makes use of the knowledge that a lot of the same content can often be seen in consecutive frames.

This is the case, for example, when a camera captures a particular scene from one point of view and few things change, or when a camera slowly moves or rotates through the scene (translation and/or affine movement of the camera).

However, this concept reaches its limits when a high proportion changes between frames, e.g. when the camera moves (rapidly) within a scene or when objects move within the scene. In this case, in the worst case, every pixel of two frames could be different.

OBJECT Proceeding from this, it is an object of the invention to provide an improvement.

Summary of the invention The object is achieved by a method according to claim 1. Further advantageous configurations are the subject matter of the dependent claims, the description and the figures.

1 shows schematic flow charts according to aspects of the invention, FIG. 2 shows schematic flow charts according to further aspects of the invention, FIG. 3 shows schematic flow charts according to further aspects of the invention, and FIG Questions.

Detailed presentation of the invention

Our reference: THA 47446 P DE 12.01.2021 Application no.: NN LU102424 Title: Method for playing back a video stream by a client -3- The invention will be presented in more detail below with reference to the figures.

It should be noted that different aspects are described, which can be used individually or in combination.

That is, each aspect can be used with different embodiments of the invention, unless explicitly presented as purely alternative.

Furthermore, for the sake of simplicity, only one entity will generally be referred to below.

Unless explicitly noted, however, the invention can also have several of the entities concerned.

In this respect, the use of the words “a”, “an” and “an” is only to be understood as an indication that at least one entity is used in a simple embodiment.

Insofar as methods are described below, the individual steps of a method can be arranged and/or combined in any order, unless something different is explicitly stated in the context.

Furthermore, the processes can be combined with one another, unless otherwise expressly indicated.

In the following we will discuss various aspects of the invention in the context of a complete system of encoder and decoder.

Errors that can occur between encoding and decoding are not discussed below because they are not relevant to understanding (de)coding.

In the popular video delivery systems, the encoder is based on prediction/prediction.

prediction). This means that the better you can predict a frame to be coded from a previously decoded frame, the less information (bit(s)) has to be transmitted.

In previous approaches, the approach followed was to predict frames based on similarities between the frames in a two-dimensional model.

However, it should be noted that the recording of videos mostly takes place in three-dimensional space.

Our reference: THA 47446 P DE 12.01.2021 Application no.: NN LU102424 Title: Method for playing back a video stream by a client -4- With computing power now available, it is possible to determine/estimate depth information on the encoder and/or decoder side. A three-dimensional motion model can therefore also be made available 3 within the invention. Without restricting the generality of the invention, it is also possible to use the invention with all current video (de)coders as long as they are upgraded accordingly. In particular, Versatile Video Coding ITU-T H.266 / ISO/IEC 23090-3 can be added to the invention.

The invention is based on the idea of motion-compensated prediction. To motivate this, we refer to Figure 4 below. Here, a video consisting of (consecutive) encoded images of two-dimensional frames (i.e. a sequence} is considered. A frame is also referred to as a two-dimensional representation. Due to the temporal redundancy between consecutive frames, a frame to be (encoded) at time t can be predicted from previously (t-1, t-2 ... (shown t-4) without being limited to four previous frames) encoded frames. These previous frames are also used as reference (reference frames , reference images). It should be noted that the frame order does not necessarily have to be a chronological order, but that the order shown and the (de)coded order can be different. I.e. for the (de)coding not only information can be used from chronologically preceding frames, but also information from chronologically subsequent (in the representation ung / chronological sequence of future) frames are used. If the motion-compensated prediction is precise enough, it is sufficient to just calculate the difference between the prediction and the frame to be coded, the so-called prediction error. prediction error). The better the prediction, the fewer prediction errors have to be transmitted, i.e. the less data has to be transmitted or stored between encoder and decoder.

That is, from the coder's point of view, efficiency increases.

Our reference: THA 47446 P DE 01/12/2021 Application no.: NN LU102424 Title: Method for playing back a video stream by a client -5- The conventional coders are based on the similarity of frames in a two-dimensional model, i.e. only transitions and/or affine movements are considered. However, there are a number of movements that cannot be easily expressed as a 2D model. Therefore, this invention uses an approach based on the three-dimensional environment in which the sequence is recorded and from which a 3D motion model can be displayed. In practical terms, video recording is analogous to projecting a three-dimensional scene into the camera's two-dimensional plane. However, since the depth information is lost during the projection, the invention enables a different provision.

In the example of the flowchart according to FIG. 1, the 3D information is reconstructed on the decoder side, while in the example of the flowchart according to FIG. 2 the coder makes the 3D information available (compressed) and the decoder only uses it. In the example of the flow chart according to FIG. 3, a mixed form is provided in which the encoder makes the (rough) 3D information available (compressed) and the decoder further processes the 3D information in order to improve it. It is obvious that in the first case the required bandwidth/storage capacity can be less than in the second or third case. On the other hand, in the first case, the computational requirements are high for the encoder and the decoder, while in the second. If the computing power requirements are lower for the decoder and highest for the encoder. This means that different scenarios can be operated on the basis of the options available. In the case of a query for a video stream in particular, provision can therefore be made for a decoder, for example, to make its properties known to the coder, so that the coder may be able to dispense with the provision of (more precise) 3D information because the decoder uses a method according to FIG 1 or 3 provides. In the following we assume that the camera is any camera and is not fixed to a specific type. 30 . In the following, reference will be made to a monocular camera with unknown camera parameters as the most difficult application, but without thereby using other types of cameras, such as. e.g. light field, stereo camera, etc.

Our reference: THA 47446 P DE 12.01.2021 Application no.: NN LU102424 Title: Method for playing back a video stream by a client -6- The camera parameters CP and geometry data GD can be inferred. The camera parameters CP can be inferred, for example, by methods such as Structure from Motion, Simultaneous Localization and Mapping or sensors.

If such data is known from certain camera types, e.g. stereo cameras and/or additional sensors, such as LIDAR sensors, gyroscopes, etc., these can be alternatively or additionally transmitted or processed and thus reduce the computing effort or make it obsolete. The camera parameters CP can typically be determined from sensor data from gyroscopes, inertial measurement units (IMU), location data from a global positioning system (GPS), etc., while geometry data GD from sensor data from a LIDAR sensor, stereo cameras, depth sensors , light field sensors, etc. are determined. If both camera parameters CP and geometry data GD are available, the (de-)coding becomes easier and usually of better quality. | The encoder SRV can receive a conventional video signal Input Video in step 301, for example. Advantageously, this video signal can be monitored for movement, i.e. relative movement of the camera. If a relative movement of the camera is detected, the input video signal Input Video can be subjected to a coding according to the invention, otherwise, if no relative movement of the camera is detected, the signal can be subjected to a usual coding as before and as indicated in step 303, 403, 503 Decoder C are provided. Camera movement may be detected by the encoder in embodiments, e.g., by visual data processing of the video signal and/or by sensors such as an IMU (Inertial Measurement Unit), a GPS (Global Positioning System), etc. If, on the other hand, a movement is detected, a corresponding flag Flag_3D or other signaling can be used to signal the presence of inventive content according to step 304, 404, 504 if it is not already per se recognizable from the data stream.

Our reference: THA 47446 P DE 12.01.2021 Application no.: NN LU102424 Title: Method for playing back a video stream by a client -7- If camera movement is detected, as indicated in step 305, 405, 505, the {intrinsic and extrinsic) camera parameters CP can be estimated/determined in step 306, 406, 506.

Techniques such as visual data processing such as Structure-from-Motion (SfM), Simultaneous Localization and Mapping (SLAM), Visual Odometry (V.0.), or any other suitable method can be used for this.

The camera parameters CP can of course also be estimated/determined/taken over as a known value by other sensors. Without restricting the generality of the invention, these camera parameters CP can be processed and encoded in step 307, 407, 507 and made available to the decoder C separately or embedded in the video stream VB. % Co The geometry in three-dimensional space can be estimated/determined in step 310, 410, 510. In particular, in step 310 the geometry in three-dimensional space can be estimated from one or more previously encoded frames (step 309). The previously determined camera parameters CP can be included in step 308 for this purpose. In the embodiments of FIGS. 2 and 3, the 3D geometry data can be estimated/determined from "raw" data. In the embodiment of Figure 1, this data can be estimated/determined from the encoded data. Typically, the visual quality in the Figures 2 and 3 embodiments will be better than in the Figure 1 embodiment, so that these embodiments can provide higher quality 3D geometry data.

In order to estimate the geometry in three-dimensional space, one can use so-called multi-view computer vision techniques without using other techniques, such as possibly! Existing depth sensors, such as LiDAR, or other image sensors that allow depth detection, such as stereo cameras, RGB+D sensors, light field sensors, etc. should be excluded.

Our reference: THA 47446 P DE 01/12/2021 Application no.: NN LU102424 Title: Method for playing back a video stream by a client -8- The geometry determined in this way in three-dimensional space can be replaced by a suitable data structure, e.g. a 3D model, a 3D network , 2D depth maps, point clouds (sparse or dense), etc. can be represented. The video signal VB can now be encoded in the three-dimensional space in steps 312, 412, 512 on the basis of the determined geometry. The novel motion-based model can now be applied to the reproduced three-dimensional information.

For example, a reference image can be determined/selected in step 311 for this purpose. This can then be presented to the standard video encoder in step 312. Obviously, the encoding that follows can be applied to one, several or all frames of a predetermined set. Correspondingly, of course, the coding can also be based on one, several or all previous frames of a predetermined set. Provision can also be made for the coder SRV to process only some spatial regions within a frame in the specified manner according to the invention and others in a conventional manner. As already stated, a standard video encoder can be used. An additional reference can be added to the list of reference images (in step 311) or an existing reference image can be replaced. Likewise, as already indicated, only a specific spatial area can be overwritten with the new reference. This enables the standard video coder to independently select the reference image that has a favorable property, e.g. high compression with low distortions (rate-distortion optimization), based on the available reference images.

The standard video encoder can thus encode the video stream using the synthesized reference and make it available to the decoder C in step 313, 413, 513.

Our reference: THA 47446 P DE 12.01.2021 Application no.: NN LU102424 Title: Method for playing back a video stream by a client -9- As in previous methods, the coder SRV can start again with a recognition according to step 301 at appropriate re-entry points and go through the procedure again. Restart points can be set at set time intervals based on channel characteristics, video frame rate, application, etc. The 3D geometry can be reconstructed in three-dimensional space or an existing one can be further developed. The 3D geometry will continue to grow as new frames are added until it restarts at the next respawn point.

The decoder side C can be operated in a corresponding manner, with the coder SRV and decoder C in FIGS. 1 to 3 being arranged horizontally at approximately the same height in their functionally corresponding components. The decoder C can first check whether a corresponding flag Flag_3D or another signaling was used. If such a signaling (Flag_3D is e.g. 0) is not present, the video stream can be treated in step 316 by default. Otherwise the video stream can be treated in the new way according to the invention. First, camera parameters CP can be obtained in steps 317, 417, 517. The obtained camera parameters CP can be processed and/or decoded in optional steps 318 . These camera parameters CP can be used e.g. for a depth estimation as well as for the generation of the geometry in the three-dimensional space in step 320 on the basis of previous frames 319. . Overall, the same strategy can be used with regard to the reference images as for the encoder (steps

309..312, 409...412, 509...512) can be used in corresponding steps 319...332, 419...432, 519...532. For example, it is possible to render the synthesized reference image in step 321 by transforming the previously decoded frame {step 319) into the frame to be decoded, below

Our reference: THA 47446 P DE 12.01.2021 Application no.: NN LU102424 Title: Method for rendering a video stream by a client -10- Guide the decoded camera parameters CP (step 318) and the geometry in three-dimensional space (step 320). Finally, in step 323, 423, 523, the video stream processed according to the invention can be decoded by a standard video encoder and output as a decoded video stream 324, 424, 524. Usually, the decoder should be synchronous with the encoder in terms of settings, so that the decoder C uses the same settings (especially for depth determination, reference creation, etc.) as the encoder SRV. In the embodiment of FIG. 2, in contrast to the embodiment of FIG. 1, the geometry in three-dimensional space can be estimated from raw video frames in step 405. An (additional) bit stream 410.1 can be generated from the data, which, for example, is the subject of further processing, e.g. decimation, (lossless/lossy) compression and coding, which can be made available to the decoder C. This bit stream 2.2 that has been made available can now also—as in the decoder C—be converted back in step 410.2 (to ensure the congruence of the data) and made available for processing in step 411.

Likewise, the geometry in three-dimensional space can also be retained beyond a re-entry point. However, the process also allows the geometry in three-dimensional space to be continuously improved on the basis of previous and current frames. This geometry in three-dimensional space can suitably be subject to further processing, e.g. decimation (e.g. mesh decimation), (lossy/lossy) compression/encoding. In a corresponding manner, the decoder C can receive and decode the bit stream 2.2 obtained in step 419.1 with the data relating to the geometry in three-dimensional space. The decoded geometry in three-dimensional space can then be used in step 420. ;

Our reference: THA 47446 P DE 01/12/2021 Application no.: NN LU102424 Title: Method for playing back a video stream by a client -11- The decoder can obviously work faster in this variant, since the decoding requires less effort than the reconstruction of the geometry in the three-dimensional Space (Figure 1) requires.

While a very efficient method with regard to the bit rate reduction is presented in the embodiment in FIG. 1, a smaller bit rate reduction can be achieved with the embodiment in FIG. The embodiment of FIG. 3 combines aspects of the embodiments of FIG. 1 and FIG. 2, spreading the complexity and allowing a flexible and efficient method.

Essentially, the concept of the embodiment in Figure 3 differs from the embodiment in Figure 2 in that the geometry in three-dimensional space, i.e. the 3D data in step 510.1 roughly represents the original geometry in three-dimensional space, i.e. in a slimmed-down version, so that the bit rate required for this decreases. Any suitable data reduction technique such as, but not limited to, sub-sampling, coarse quantization, transform coding, and dimensionality reduction, etc. can be used for this purpose. The 3D data minimized in this way can be coded and made available to the decoder C as before in step 510.2. The bit stream 510.1/510.2 may be subject to further processing, e.g., decimation, compression (lossless/lossy) and encoding, which may be provided to the decoder C. This bit stream 2.2 that has been made available can now also—as in the decoder C—be converted back in step 510.3 (to ensure the congruence of the data) and made available for processing in step 511. In this case, the previously encoded frames 509 and the camera parameters 506 can be used for finer processing of the 3D data. Correspondingly, the decoder C can receive the encoded and minimized 3D data in step 519.1 and decode them in step 519.2 and—corresponding to the encoder SRV—the processing can be made available. The previously encoded frames 519.3 and camera parameters 518 can be used for finer processing of the 3D data. |

Our reference: THA 47446 P DE 01/12/2021 Application no.: NN LU102424 Title: Method for playing back a video stream by a client -12- I.e. in all embodiments of the decoder C, a video stream VB from the encoder SRV, e.g. a streaming server, In a first Step 315, 415, 515 get. The client C decodes the received video stream VB using camera parameters CP and geometry data GD, and then reproduces this as a prepared video stream AVB in step 324, 424, 524. As shown in Figures 1 to 3, in embodiments of the invention the camera parameters CP can be obtained from the encoder SRV in step 317, 417, 517 (e.g. as bitstream 2.1) or can be determined from the obtained video stream VB. In embodiments of the invention, geometry data GD can be obtained from the encoder SRV (e.g. as bitstream 2.2) or determined from the obtained video stream VB.

In particular, it can be provided that before receiving the video stream vB, the decoder C signals to the encoder SRV that it is able to process it. A set of processing options can also be supplied so that the coder can provide the appropriate format. For this purpose, the format made available can have a corresponding coding with regard to setting data.

In one embodiment of the invention, the geometry data includes depth data. In summary, it should again be pointed out that in the case of FIG. 1, a 3D reconstruction is used both on the encoder and on the decoder side. In the example in FIG. 2, a 3D reconstruction is carried out only by the encoder and made available to the decoder. This means that the decoder does not have to carry out a 3D reconstruction, but can use the 3D geometry data provided by the encoder. Estimating the 3D geometry data on the part of the encoder is generally simpler than on the part of the decoder. The configuration according to FIG. 2 is advantageous in particular when the computing power is limited on the part of the decoder. In the case of FIG. 3, as in FIG. 2, a 3D reconstruction is carried out by the coder and made available to the decoder. However, only a rough version of the 3D geometry data is provided here. This allows the

Our reference: THA 47446 P DE 01/12/2021 Application no.: NN LU102424 Title: Method for playing back a video stream by a client -13- bit rate for the 3D geometry data is reduced. At the same time, however, the decoder now has to complete/post-process (refine) the 3D geometry data. The choice of method (e.g. according to Figure 1, Figure 2 or Figure 3) can be negotiated between the encoder and the decoder. This can be done, for example, on the basis of previous knowledge (e.g. computing power) or via a control/return channel (adaptive) in order to also take changes in the transmission capacity into account, for example. In a broadcast scenario that is aimed at a number of decoders, the configuration according to FIG. 2 will generally be preferred.

Even if the invention is described in relation to methods, it is clear to a person skilled in the art that the invention can also be made available in hardware, in particular hardware set up by software. Common (de-)coding units, special computing units such as GPUs and DSPs as well as ASICs or FPGAs-based solutions can be used for this, without thereby excluding the applicability of general microprocessors.

In particular, the invention can accordingly also be embodied in computer program products for setting up a data processing system for carrying out a method.

With the invention, it is possible to achieve significant savings of several percent in the bit rate if there are scenes that can be encoded accordingly.

Claims (15)

Our reference: THA 47446 P DE 01/12/2021 Application no.: NN LU102424 Title: Method for playing a video stream by a client -14- claims 1. Method for playing back a video stream by a client (C), the video stream comprising frames from a camera in relation to a moving object from different positions, comprising the steps ® obtaining a video stream (VB) from an encoder (SRV), * Decoding of the obtained video stream (VB) using camera parameters (CP) and geometry data (GD), * Playback of the processed video stream (AVB). 2. Method according to claim 1, characterized in that the camera parameters {CP) are obtained from the encoder (SRV) or are determined from the obtained video stream (VB). 3. Method according to claim 1 or 2, characterized in that the geometry data (GD) are obtained from the coder (SRV) or are determined from the obtained video stream (VB). 4. Method according to one of the preceding claims, characterized in that before receiving the video stream (VB), the client {C) signals to the coder (SRV} that it is able to process it. 5. The method according to any one of the preceding claims, characterized in that the geometry data have depth data. ; 6. Device for carrying out a method according to any one of claims 1 to 5. 7. Computer program product for setting up a data processing system for carrying out a method according to one of the preceding claims 1 to 5. 8. Method for providing a video stream by an encoder (SRV) to a client (C), the video stream having frames from a camera in relation to a moving object from different positions, comprising the steps of * obtaining frames from a camera in Reference to an object from different positions in the form of a video stream (eVB) ® Encoding of the obtained video stream (eVB) using camera parameters (CP) and geometry data (GD), e streaming of the video stream (VB). ‘ Our reference: THA 47446 P DE 01/12/2021 Application no.: NN LU102424 Title: Method for playing a video stream by a client -15- 9. The method as claimed in claim 8, characterized in that the camera parameters (CP) are obtained from an external source or are determined from the video stream (VB) obtained. 10. The method as claimed in claim 8 or 9, characterized in that the geometry data (GD) are obtained from an external source or are determined from the video stream (VB) obtained. 11. The method according to any one of the preceding claims 9 or 10, characterized in that the external source is a sensor and/or the cameras. 12. The method as claimed in one of the preceding claims 8 to 11, characterized in that before the video stream (VB) is streamed, the coder {SRV} is signaled by the client (C) that it is capable of processing. 13. The method according to any one of the preceding claims 1 to 12, characterized in that the geometry data have depth data. 14. Device for carrying out a method according to any one of claims 8 to 13. 15. Computer program product for setting up a data processing system for carrying out a method according to one of the preceding claims 8 to 13.