WO2022179681A1

WO2022179681A1 - Method for operating a display device of a vehicle for displaying video frames of a video stream at a predefined frame rate and corresponding display device and vehicle

Info

Publication number: WO2022179681A1
Application number: PCT/EP2021/054545
Authority: WO
Inventors: Uwe Hackl
Original assignee: Cariad Se
Priority date: 2021-02-24
Filing date: 2021-02-24
Publication date: 2022-09-01

Abstract

The invention is concerned with a method for operating a display device (14) of a vehicle for displaying video frames (31) of a video stream (23) at a predefined frame rate. In a receiving unit (26) a ping-pong buffer module (28) is operated that comprises two buffers (32) and a locking logic (33) for selectively locking one of the buffers (32) at a time. For filling the ping-pong buffer module (28) a parser module (27) parses the received video stream (23) for detecting a respective new video frame (31) and, if the corresponding data pattern (30) is detected, the detected new video frame (31) is provided to a currently un-locked buffer of the ping-pong buffer module (28), and a scheduler module (29) provides the frame data of the new video frame (31) from one of the buffers for displaying the corresponding single video frame (31), wherein the buffer that is currently accessed by the scheduler module (29) is locked by the scheduler module (29) using the locking logic (33).

Description

Method for operating a display device of a vehicle for displaying video frames of a video stream at a predefined frame rate and corresponding display device and vehicle DESCRIPTION:

The invention is concerned with a method for operating a display device of a vehicle for displaying video frames of a video stream at a predefined frame rate. The invention is also concerned with a corresponding display device that can be operated according to the method and vehicle comprising the display device.

In a real-time-based camera system, the decoding of a transmitted video stream should be as free of temporal jitter as possible, i.e. the decoding should be performed at a constant frame rate. The corresponding display device must address the mutually contradictory requirements of real-time conditions (reduction of delay) and jitter compensation (increase of delay).

In today's motor vehicles, various camera systems and display systems (e.g. rear view camera, top view, virtual exterior mirror) can be used. The camera and the display device are separated from each other and connected via an electrical link (or a communication network) to transport the video image or the video stream from the camera to the display unit. The transmission of the video signal causes a delay from the real world to the display of this world in the display unit. This latency depends on the selected transmission method and, especially in the case of compressed transmission (e.g. H.264, H.265) via network topologies (e.g. Ethernet), on a wide variety of influencing factors (e.g. compression method, bandwidth, buffer management). The resulting latency quickly reaches a critical value for controlling the vehicle in real time. For this reason, it is necessary to keep this latency as low as possible. On the other hand, the encoding, transmission and decoding of a video stream (especially in compressed transmission, e.g. with H.264, H.265) does not take place uniformly over time across network topologies (e.g. in an Ethernet network). Some video frames are compressed and transmitted faster (e.g., P- frames according to the above mentions standard), while other video frames are compressed and transmitted slower (e.g., l-frames). The resulting disparity leads to a temporal change in the representation of a moving video image, which is noticeable to the viewer as "jitter". Fig. 5 shows how, for example, a longer transmission time of l-frames 1 , 5 at GOP = 4 (GOP - Group of Pictures), e.g. due to the size of an encoded video frame at constant bandwidth, affects the display time of frames 1 , 2 and 4 when the data is decoded as soon as it is received. Over time t, a source clock SRC_CLCK triggers a source rendering SRC_RNDR in a video source device, e.g. a video camera (requiring scan time), which results in raw video frames that are identified here by order numbers or sequence numbers 1 , 2,... , 6. An Encoding ENC is performed on the raw video frames (requiring encoding time) and the encoded video frames send over a communication network NETW (requiring transmission time). At a receiving display device, a decoding DEC of the received encoded video frames is performed (requiring decoding time). If each decoded video frame is displayed on a display screen by applying a rendering DIS_RNDR, the resulting display duration of the displayed video frames may vary, as is illustrated by the varying lengths of the displayed video frames over time t.

Document US 2006/0161676 A1 describes a video source device that may send a video stream out into an IP-based communication network (IP - internet protocol). For providing a smooth or regular output of stream data, the sending unit uses a two-buffer module.

US 6,901 ,270 B1 describes an audio processing device that transfers audio data over a Wi-Fi network. For splitting up the continuous audio stream into packets, two buffers are used. Document US 6,989,835 B2 describes a sending device for generating video streams. Before sending out the video streams, a microprocessor may analyze the stream data of the video streams for detecting specific pixel values.

In the prior art no solution is presented for ensuring a constant or fixed frame rate for displaying of video frames in a display device that receives a video stream over a communication network.

It is an object of the present invention to present video frames of a video stream with a constant frame rate in a display device of a vehicle.

The object is accomplished by the subject matter of the independent claims. Advantageous developments with convenient and non-trivial further embodiments of the invention are specified in the following description, the dependent claims and the figures.

The invention provides a method for operating a display device of a vehicle for displaying video frames from a video stream at a predefined frame rate, that is preferably constant or, for example, varies by less than 5 %. Such a display device may be part of the dashboard or the electronic instrument cluster of the vehicle. The video frames may be displayed or presented, e.g. on an OLED display (OLED - Organic Light-Emitting Diode) and/or a Micro-LED display and/or a TFT display (TFT - Thin Film Transistor). Exemplary values for the frame rate can be in the range of, for example, 10 ms to 50 ms per video frame. A receiving unit of the display device receives the video stream over a communication network and provides frame data sets to a decoder unit and/or to a Tenderer unit of the display device. Each frame data set is for displaying a respective single video frame of the video stream. In other words the receiving unit provides the video frames to the decoder and/or to the Tenderer on a frame-by-frame basis. Those stream data that constitute or define one video frame, are combined or organized as one “data frame set”. It should be noted, though, that the final video frame may depend not only on the one frame data set, but also on at least one previously decoded frame data set, as is the case for a frame data set of a P-frame that may depend on a frame data set of a preceding l-frame (P-frame - predicted frame; l-frame - intra-coded frame). In other words a frame data set may either be self-contained, if it comprises all graphical information for displaying one video frame (e.g. an l-frame), or a frame data set may depend on at least one preceding frame data set for deriving the complete graphical information of the corresponding video frame. The decoder unit may be present, if the video stream is encoded, as is the case for an H.264 video stream or an H.265 video stream, where in particular definitions for l-frames and P-frames can be found. A Tenderer unit may set pixel values of single pixels that are displayed by e.g. an OLED-panel of the display device.

For obtaining a constant frame rate for displaying the video frames, it has been discovered that it is sufficient to present the single “frame data sets” of the video frames to the decoder unit and/or the Tenderer unit at that frame rate. In order to present the single frame data sets at the predefined frame rate to the decoder unit and/or the Tenderer unit, the receiving unit may comprise three components: a ping-pong buffer module and a parser module and a scheduler module.

For storing individual video frames separately, the ping-pong buffer module is operated which comprises two buffers and a locking logic for selectively locking one of the buffers at a time, wherein locking a buffer indicates that writing operations are forbidden. A ping-pong buffer is a double buffer, where by the locking logic it is ensured that new incoming data is stored in one buffer, while the other buffer is locked for reading such that the reading process will not be disturbed or interrupted by overwriting the locked buffer with new data. The locking can be implemented by setting a locked-state flag. The buffers can both be in an un-locked state. Each one of the two buffers may be locked, but only one at a time.

For filling the ping-pong buffer module at its input side, a parser module parses the received video stream for detecting at least one predefined data pattern that indicates the beginning or the end of a respective (next or) new video frame and, if the respective data pattern is detected, the detected new video frame is provided to the currently un-locked buffer of the ping-pong buffer module. The parser module parses or analyses the incoming or received video stream by comparing the stream data with at least one predefined data pattern and whenever stream data correspond to one data pattern, this indicates that the beginning or the end of a new video frame is detected, depending on the type of data pattern. This newly detected video frame then becomes the content of the currently un-locked buffer, i.e. the stream data of that new video frame makes up the next “frame data set” in that un-locked buffer. Putting the new video frame into the un-locked buffer can be done by copying the corresponding stream data into the un-locked buffer or by setting a pointer indicating the beginning of the un-locked buffer to the beginning of the new video frame in the video stream such that no copying of data is required.

For reading out the content in the buffers of the ping-pong buffer at its output side, the scheduler module is provided. The scheduler module provides the respective stream data from one of the buffers (i.e. a single video frame) as a respective frame data set to the decoder / Tenderer unit, wherein the buffer that is currently accessed by the scheduler module is locked by the scheduler module using the locking logic. While on the input side the un-locked buffer is provided with a new video frame by the parser module, the locked buffer is accessed by the scheduler module for forwarding its content to the decoder units and/or Tenderer unit. While the scheduler module needs access or accesses or uses the frame data set in that buffer, this buffer is set to the locked-state by the scheduler module. Again, providing the frame data set in the locked buffer to the decoder unit and/or Tenderer unit can be done by copying the frame data set or by setting a pointer to the locked buffer such that the decoding unit and/or Tenderer unit may access the locked buffer using the pointer.

By switching between the buffers at a predefined frame rate, the individual frame data sets of single video frames stored in the buffers are forwarded or provided to the decoder unit and/or to the Tenderer unit at that frame rate such that the video frames are processed or output or displayed at that frame rate. Any temporal jitter caused by the video source device, e.g. a video camera, and/or by the communication network, can be compensated in the receiving unit to a degree that is only limited by the duration of each displayed video frame, i.e. the inverse value of the predefined frame rate.

An alternative implementation comprises that the scheduler acts on the already decoded video frames instead of the (undecoded or encoded) video frame (e.g. l-frames and P-frames). Then it is not the (raw) parser that is writing to the ping-pong buffer and the decoder reading from the ping-pong buffer, but rather the decoder is receiving the encoded video data from the parser or in the parser and the decoder is writing the single decoded video frames to the ping pong buffer and the Tenderer reads them from the ping-pong buffer. This is an alternative way of handling I/P- Streams (l-frame / P-frame streams) with a different architecture.

Accordingly, according to one aspect, the parser module provides encoded frame data (e.g. an l-frame or a P-frame) of the detected new video frame to the currently un-locked buffer of the ping-pong buffer module and the scheduler module provides the encoded frame data of the respective new video frame to the decoder unit for decoding.

And according to the alternative aspect, the parser module comprises the decoder and the decoder decodes the frame data of the detected new video frame and provides the decoded frame data to the currently un-locked buffer of the ping-pong buffer module and, correspondingly, the scheduler module provides the decoded frame data of the respective new video frame from the ping-pong buffer to the Tenderer unit (as no further decoding is necessary). Thus, in the buffers, complete or decoded video images are stored. This provides the advantage that no dependency remains between the content of the buffers in the ping-pong buffer as opposed to encoded P-frames that rely on information from a preceding l-frame.

The invention also comprises embodiments that provide features which afford additional technical advantages. Several embodiment are concerned with a switching pattern for switching between the buffer of the ping-pong buffer module.

One embodiment comprises that the scheduler module periodically operates a timer that is set to the predefined frame rate for displaying the video images and, whenever the timer runs out, the scheduler module switches to the respective other one of the buffers for providing the new video frame in that buffer as the next frame data set to the decoder unit and/or Tenderer unit. However, the switching is preferably only performed, if a predefined availability criterion is fulfilled by that other buffer. If the availability criterion is fulfilled, the scheduler module can unlock the currently locked or used buffer and lock the other buffer for switching to that buffer and providing its content (i.e. a frame data set of a new video frame) to the decoder unit and/or Tenderer unit. If the availability criterion is not fulfilled, it means that in the other buffer a valid or complete frame data set is missing, i.e. no new video frame is available yet. This is a buffer underrun situation or buffer underrun event. By only switching to the other buffer after having verified the availability criterion is fulfilled, it is prevented that the display device displays or outputs incomplete video frames.

One embodiment comprises that said availability criterion comprises that for compensating the buffer underrun event the scheduler module only switches to the respective other buffer, when a confirmation signal is available from the parser module that indicates that the video frame stored in the other buffer is newer than the video frame stored in the currently used locked buffer, and otherwise the scheduler module provides the video frame from the currently used locked buffer again as the next video frame set. In other words, the scheduler repeatedly displays the same video frame of the currently locked or used buffer, while the availability criterion is not fulfilled. This ensures a continuous output of video frames in the display device.

For parsing the video stream, one embodiment comprises that the parser module uses a data pattern that indicates the beginning of a new video frame and the parser module verifies that a detected new video frame is fully contained in the already received part of the video stream and, as long as the new video frame is not completely received, the confirmation signal is suppressed. When a data pattern for detecting the beginning of a new video frame in the video stream is used, it is therefore also verified that the whole new video frame is available or has already been received and is available for providing it to the currently un-locked buffer. Thus, also a data pattern for detecting the beginning of a new video frame may be used or applied without accidentally delivering a not-yet fully received video frame to a buffer.

One embodiment comprises that stream data of the received video stream is stored in a receiving buffer where the parser module searches for the at least one data pattern. In other words the stream data received from the communication network may be written continuously or directly into the receiving buffer which improves the reduction of temporal jitter. The two buffers of the ping-pong buffer module can be implemented in the receiving buffer itself by using pointers that indicated that region or part or section of the receiving buffer that is to be used as a respective buffer of the ping-pong buffer module. Alternatively, the buffers of the ping-pong buffer module may be implemented as separate buffers and a copy function may transfer stream data from the receiving buffer to the buffers of the ping-pong buffer module.

One embodiment comprises that for compensating a buffer overrun event the parser module re-uses the currently used un-locked buffer for the currently detected new video frame, such that the previous new video frame that is currently in the currently used un-locked buffer will not be provided to the decoder and/or Tenderer, wherein the currently used un-locked buffer is re used, until it is detected that the other buffer is un-locked. In other words, the parser module keeps writing or providing consecutive new video frames to the same buffer and thus overwrites the respective previous frame data set of the previously provided new video frame until the currently locked buffer becomes available or un-locked. Although the overwritten frame data sets in the re-used buffer are lost or deleted or overwritten, this generally results in the loss of only one single video frame which, in contrast to a temporal jitter in the resulting sequence of video frames displayed by the display device, has only little impact on the spectator or less impact than discarding the same amount of data at an arbitrary position within the stream. This embodiment is especially suitable for video streams only using l-Frames (GOP=1 ). This approach is applicable to l-Frame-Only Streams. There is a benefit handling P-Frames differently.

Flowever, overwriting the current content of a buffer in an overrun event may not always be beneficial, for example, it has been discovered that an l-frame should not override a P-frame, it depends on. One embodiment comprises that the currently used un-locked buffer is re-used, for one of the following combinations: the currently detected video frame is an l-frame; both the currently detected new video frame and the previous video frame that is stored in the buffer are P-frames. As an l-frame is a self-contained frame data set that also may form the basis for following P-frames, securing or providing an I- frame in the buffer ensures that all the following P-frames belonging to the same group of pictures GOP may also be displayed. A P-frame may be replaced by the next P-frame such that the displayed video stream may be updated or continued.

One embodiment comprises that the currently detected new video frame is ignored, such that the currently detected new video frame will not be provided to the decoder and/or Tenderer, if the previous video frame that is currently stored in the buffer is an l-frame and the currently detected video frame is a P- frame. Overwriting an l-frame by a P-frame renders also the P-frame useless, as the information from the overwritten l-frame is missing. Therefore protecting an l-frame ensures that the latest self-contained frame data set is available.

One embodiment comprises that the at least one data pattern comprises a data pattern of a NAL unit (NAL - network abstraction layer) according to a FI.264 standard and/or a NAL unit according to a H.265 standard. It is proven that an NAL unit provides a data pattern that can be recognized or detected reliably by a parser module.

The described modules (parser module, ping-pong buffer module) may be implemented as a respective programming code or software. To this end, the receiving unit may comprise at least one microprocessor and/or at least one DSP (Digital Signal Processor). The respective module may also be partly or fully implemented as an electronic circuit, e. g. on the basis of an integrated circuit, e. g. an ASIC (Application-Specific Integrated Circuit) and/or a FPGA (Field-Programmable Gate Array). The decoder unit and the Tenderer unit may be based on an electronic circuit and/or a software as is available from the prior art.

The scheduler module, ping-pong buffer module, and parser module may be based on software and/or hardware of the receiving unit. The decoder unit and the Tenderer unit may be taken from the prior are, where corresponding hardware and/or software is available. The parser module may be based on a parser software, as it is known from the prior art for detecting a predefined data pattern in a stream of data.

The invention also provides a display device for a vehicle, wherein the display device comprises: a decoder unit and/or a Tenderer unit for consecutively displaying video frames of a video stream and a receiving unit for receiving the video stream over a communication network and for providing separate frame data sets to the decoder unit and/or to the Tenderer unit, wherein each frame data set is for displaying a respective single video frame. The receiving unit is designed to perform an embodiment of the inventive method. The display device may present or output video frames at the predefined frame rate and may, according to some embodiments, compensate for underrun events and/or overrun events.

The invention also provides a motor vehicle comprising an embodiment of the inventive display device, a video source device for generating a video stream, and a communication network that links the video source device to the display device. The inventive vehicle is preferably designed as a motor vehicle, in particular as a passenger vehicle or a truck, or as a bus or a motorcycle. The video source device can be provided in the form of a video camera, for example a rear view camera or an electronic side mirror. The communication network can be based on a switched network and/or an Ethernet network. Note that an Ethernet network can also be a switched network (and today typically it is), but also non-switched Ethernet Bus Systems exists (e.g. Coax ).The stream data of the video stream may be transferred by the communication network on the basis of data packets. For example, the IP may be used as a protocol.

The invention also comprises the combinations of the features of the different embodiments. In the following an exemplary implementation of the invention is described. The figures show:

Fig. 1 a schematic illustration of an embodiment of the inventive motor vehicle;

Fig. 2 a diagram illustrating a method that may be performed by a display device in the motor vehicle of Fig. 1 ;

Fig. 3 a diagram illustrating a situation where a frame rate of the display device is slower than a frame rate of a video source device that provides a video stream to the display device;

Fig. 4 a diagram illustrating the compensation of a buffer overrun caused by the situation according to Fig. 3;

Fig. 5 a diagram illustrating temporal jitter that may occur while video frames are displayed.

The embodiment explained in the following is a preferred embodiment of the invention. Flowever, in the embodiment, the described components of the embodiment each represent individual features of the invention which are to be considered independently of each other and which each develop the invention also independently of each other and thereby are also to be regarded as a component of the invention in individual manner or in another than the shown combination. Furthermore, the described embodiment can also be supplemented by further features of the invention already described.

In the figures identical reference signs indicate elements that provide the same function.

Fig. 1 shows a motor vehicle that can be designed for example as a passenger vehicle. The motor vehicle 10 may comprise a video source device 11 that may generate a video stream 12 comprising of stream data 13. The stream data 13 may describe a sequence of single video images or video frames 31 .

The video stream 12 may be transmitted to a display device 14 over a communication network 15. A user of the motor vehicle 10 may be presented or provided with video images or video frames 16 on a display panel 17 of the display device 14. For example, the video source device 11 may comprise a camera 18, e. g. an electronic rear view camera or an electronic side mirror camera and the camera images may be used for generating the video stream 12 such that the user may be presented with the camera images or the camera video stream as the video frames 16 on the display panel 17. The user may therefore use the display device 14 as an electronic rear mirror or an electronic side mirror.

The camera 18 may generate image raw data 19 that may be encoded using an encoder unit 20 that may be based, for example, on the FI.264 standard and/or the FI.265 standard. The compressed or encoded image data together with the communication protocol may constitute the stream data 13 of the video stream 12. The display device 11 may be connected to the communication network 15 on the basis of a port 21 that may be, for example, part of an NIC (Network Interface Controller). The communication network 15 may be based, for example, on an Ethernet network. The video source device 11 may be connected to the communication network 15 on the basis of a port 22 that may be part of an NIC of the display device 14. The display device 14 may be a monitor or display positioned in the dashboard and/or electronic instrument cluster of the vehicle 10. The display device 14 and the video source device 11 may be positioned in the vehicle 10 at a distance of, for example, more than 1 m or more than 1.5 m. The received video stream 23, i. e. the received stream data 24, may be stored in a receiving buffer 25 of a receiving unit 26 in the display device 14. The receiving buffer 25 may be based on a volatile data storage e.g. an RAM (Random Access Memory). The receiving unit 26 may operate a parser module 27, a ping-pong buffer module 28, and a scheduler module 29. The parser module 27 may search and/or detect at least one predefined data pattern 30 in the received video stream 23 stored in the receiving buffer 25.

By means of the data pattern 30 the parser module 27 may detect the beginning and/or end of the respective single video frames 31 and may provide the corresponding stream data 24 in an available or un-locked buffer 32 of the buffer module 28. The buffer module 28 may comprise two buffers and a locking logic 33 for setting one buffer 32 in a locked state at a time. Thus, the parser module 27 will provide the detected video frame 31 to the other, un locked buffer 32. The two ping-pong buffers 32 of the ping-pong buffer module are also called ping buffer B0 and pong buffer B1 . One of the buffers 32 may be locked at a time by the scheduler module 29 which may provide the frame data set 34 of the corresponding video frame 31 stored in that buffer 32 to a decoder unit 35 that may decode the frame data set or the received frame data set 34 from that buffer and may thus generate a decoded video frame 36 which may be provided to a Tenderer unit 37 that drives or runs the panel 17. By rendering or setting pixel values on the basis of the decoded video frame 36, the corresponding video frame 16 is displayed on the panel 17. The decoder unit 35 and the Tenderer unit 37 may be based on a technology from the prior art.

The scheduler module 29 may switch between the buffer 32 of the buffer module 28 based on a timer 40 that may be set to a time value T1 that is the inverse value of a predefined frame rate at which the video frame 16 shall be refreshed on the panel 17. Alternatively, the parser module may comprise the decoder and the decoder may decode the frame data of the detected new video frame and provide the decoded frame data to the currently un-locked buffer of the ping-pong buffer module and, correspondingly, the scheduler module provides the already decoded frame data of the respective new video frame from the ping-pong buffer to the Tenderer unit.

Fig. 2 illustrates the resulting presentation of the video frames 16 on the panel 17, i. e. the refreshment at every T1 time period. T1 can be in the range of 20 ms to 50 ms. Whenever the corresponding frame data set is one of the buffers 32, i. e. the ping buffer B0 or the pong buffer B1 , and the buffer 32 is in the locked-state, as set by the locking logic 33, a hatching is shown in Fig. 2 for that time period or time interval.

Fig. 3 and Fig. 4 show one continuous situation starting from the transmission of video frame 1 and continuing a succession of video frames until a video frame D. Fig. 3 and Fig. 4 show one continuous process over time t.

Fig. 3 shows an example where the video source device 11 generates the video frames 31 in the video stream 12 at a source clock rate or source frame rate that results in time intervals or time values TO. Fig. 3 and Fig. 4 show an example where the clocks in the video source device 11 and the timer 40 may differ such that the timer 40 runs at a slower rate such that T1 is greater than TO, for example T1 may be 40 ms and TO may be 33 ms.

Fig. 4 illustrates how this will result in a buffer overrun event. The video frame A is provided in the pong buffer B1 but may not be accessed by the scheduler module 29 as the frame B arrives before the scheduler module 29 may switch to the pong buffer B0 and the content of the pong buffer B0 is overwritten by the video frame B before video frame A can be provided to the decoder unit 35. A preferred embodiment comprises an arrangement and method which, on the one hand, reduces the effect of jitter and, on the other hand, uses only minimal latency to reduce jitter.

The arrangement (see Fig. 1 ) may consist of a receiver, which may comprise following components:

- A camera system that encodes a camera image at a constant time frame rate as video frames (e.g. H.264, H.265) and provides the compressed video stream on a network (e.g. Ethernet).

- A network (e.g. Ethernet) that transports the compressed video data from the camera system to a display system.

- a display system that preferably comprises:

- an input buffer to receive the compressed video data from a network (e.g. Ethernet).

- a parser that receives the compressed video data from the input buffer and splits it into video frames by pre-processing (parsing).

- a ping-pong buffer, which is filled by the parser with the individual video frames in turn.

- a scheduler that reads the video frames from the ping-pong at a constant time frame rate and passes them to the decoder.

- a decoder that decodes compressed video frames.

- a Tenderer, which displays the decoded video frames in a video buffer (graphics card).

In a preferred embodiment, the method describes within the display system the receiving of video data from the network, the splitting into video frames, the filling of the ping-pong buffer as well as the scheduling. Optional components of the method described here are the camera system, the network, the decoder and the Tenderer.

The video data is received by the display system from the network and may first be written to an receiving buffer or input buffer. This serves only to compensate for runtime fluctuations of the parser. The parser may split the video data into individual video frames. For this purpose, the NAL units of the H.264 or H.265 video may be analyzed and the beginning of a video frame may be determined. As soon as the end of the video frame or the beginning of the next video frame is determined, the data of the video frame may be written into or provided to one of the two buffers of the ping-pong buffer. The scheduler may be started when the ping-pong buffer is written for the first time.

The scheduler may read the buffer and pass it to the decoder. The scheduler may now read the ping-pong buffer at a specified constant frame rate. The frame rate can be e.g. the target frame rate of the camera system (e.g. clock recovery by PCR), or defined otherwise. The ping-pong buffer that is currently being processed by the scheduler is preferably locked so that it cannot be overwritten by the parser. This may serve to prevent overwriting of the data currently being processed.

The decoder may decode the video frame specified by the scheduler at the time specified by the scheduler and provides it to the Tenderer in uncompressed form. The Tenderer may then display the video frame in a graphics card, for example. Fig. 2 shows the constant-time decoding of the video frames.

If the frame rates of the camera system and the scheduler are not synchronized, over a longer runtime the deviation becomes an "underrun" or "overrun". In the case of an "overrun" the parser must overwrite the last described ping-pong buffer, because the other buffer has not yet been released by the scheduler (see Fig. 3 and Fig. 4).

An essential advantage of this arrangement and method is that video frames which are received by the display system at a distance which is not constant in time (typical e.g. for H.264/FI.265 with GOP- Size greater than 1 ) may be brought to the display at a constant distance in time by using a scheduler and a ping-pong buffer. This avoids that a uniform movement in a video is perceived as "jerky" by the viewer. Another significant advantage of this invention is that the latency induced by the scheduler is reduced to a necessary minimum (maximum 1 frame) by using a ping-pong buffer.

Another significant advantage of this arrangement is that a parser is used to compensate for the deviation of the frame rates of the camera system and the display system. This means that in the event of an overflow, data from the input buffer is not simply overwritten incoherently, which usually leads to visible artifacts in the decoding of the video, but instead only entire video frames are ever dropped/deleted due to the locked ping-pong buffer.

Another advantage of this arrangement is that the frame rates of the camera system and the display system are synchronized with minimal effect on the smooth playback of the video. Fig. 3 and Fig. 4 show an example where the display system clock differs from the camera system clock. Flere, frame A is overwritten by frame B from the parser because the other buffer is still locked by the scheduler with frame 9 (see Fig. 4). This measure has an effect on the temporally constant display, but is unavoidable. By using the ping-pong buffer and the parser, however, only a complete video frame is overwritten and not random data in the input buffer, which would corrupt the video stream with unpredictable consequences.

Overall, the example shows how a constant decoding rate of a real-time video stream can be provided.

Claims

CLAIMS:

Method for operating a display device (14) of a vehicle for displaying video frames (31 ) of a video stream (23) at a predefined frame rate, wherein a receiving unit (26) of the display device (14) receives the video stream (23) over a communication network (15) and provides frame data sets (34) to a decoder unit (35) and/or to a Tenderer unit (37) of the display device (14), wherein each frame data set (34) is for displaying a respective single video frame (31 ) of the video stream (23); characterized in that in the receiving unit (26) a ping-pong buffer module (28) is operated that comprises two buffers (32) and a locking logic (33) for selectively locking one of the buffers (32) at a time, wherein locking a buffer (32) indicates that writing operations are forbidden, and for filling the ping-pong buffer module (28) a parser module (27) parses the received video stream (23) for detecting at least one predefined data pattern (30) that indicates the beginning or the end of a respective new video frame (31 ) and, if the respective data pattern (30) is detected, the detected new video frame (31 ) is provided to the currently un-locked buffer of the ping-pong buffer module (28) and a scheduler module (29) provides the frame data of the respective new video frame (31 ) from one of the buffers as a respective frame data set (34) for displaying the corresponding single video frame (31 ), wherein the buffer that is currently accessed by the scheduler module (29) is locked by the scheduler module (29) using the locking logic (33).

Method according to claim 1 , wherein the scheduler module (29) periodically operates a timer (40) that is set to the predefined frame rate for displaying the video images and, whenever the timer (40) runs out, the scheduler module (29) switches to the respective other one of the buffers for providing the new video frame (31 ) from that buffer as the next frame data set (34) to the decoder unit (35) and/or Tenderer unit (37), if a predefined availability criterion is fulfilled by that other buffer.

3. Method according to claim 2, wherein the availability criterion comprises that for compensating a buffer underrun event the scheduler module (29) only switches to the respective other buffer, when a confirmation signal is available from the parser module (27) that indicates that the video frame (31 ) stored in the other buffer is newer than the video frame (31 ) stored in the currently used locked buffer, and otherwise the scheduler module (29) provides the video frame (31 ) from the currently used locked buffer again as the next video frame (31) set. 4. Method according to claim 3, wherein the parser module (27) uses a data pattern (30) that indicates the beginning of a new video frame (31) and the parser module (27) verifies that a detected new video frame (31) is fully contained in the already received part of the video stream (23) and, as long as the new video frame (31) is not completely received, the confirmation signal is suppressed.

5. Method according to any of the preceding claims, wherein stream data (24) of the received video stream (23) is stored in a receiving buffer (25) where the parser module (27) searches for the at least one data pattern (30).

6. Method according to any of the preceding claims, wherein for compensating a buffer overrun event the parser module (27) re-uses the currently used un-locked buffer for the currently detected new video frame (31 ), such that the previous new video frame (31 ) that is currently stored in the currently used un-locked buffer will not be provided to the decoder and/or Tenderer, wherein the currently used un-locked buffer is re-used for storing respective new video frames, until it is detected that the other buffer is un-locked.

7. Method according to claim 6, wherein the currently used un-locked buffer is re-used, for one of the following combinations:

- the currently detected video frame (31 ) is an l-frame, - both the currently detected new video frame (31) and the previous video frame (31) that is stored in the buffer are P-frames,

- and/or the currently detected new video frame (31) is ignored, such that the currently detected new video frame (31 ) will not be provided to the decoder and/or Tenderer, if the previous video frame (31 ) that is currently stored in the buffer is an l-frame and the currently detected video frame (31) is a P-frame.

8. Method according to any of the preceding claims, wherein the at least one data pattern (30) comprises a data pattern (30) of a NAL unit according to a H.264 standard and/or a NAL unit according to a H.265 standard.

9. Method according to any of the preceding claims, wherein the parser module (27) provides encoded frame data of the detected new video frame (31) to the currently un-locked buffer of the ping-pong buffer module (28) and the scheduler module (29) provides the encoded frame data of the respective new video frame (31 ) to the decoder unit (35).

10. Method according to any of the claims 1 to 8, wherein the parser module (27) comprises the decoder (35) and the decoder (35) decodes the frame data of the detected new video frame (31) and provides the decoded frame data to the currently un-locked buffer of the ping-pong buffer module (28) and the scheduler module (29) provides the decoded frame data of the respective new video frame (31 ) to the Tenderer unit (37).

11. Display device (14) for a vehicle, wherein the display device (14) comprises:

- a decoder unit (35) and/or a Tenderer unit (37) for consecutively decoding and/or displaying video frames (31 ) of a video stream (23),

- a receiving unit (26) for receiving the video stream (23) over a communication network (15) and for providing separate frame data sets (34) to the decoder unit (35) and/or to the Tenderer unit (37), wherein each frame data set (34) is for displaying a respective single video frame (31), characterized in that the receiving unit (26) is designed to perform a method according to one of the preceding claims.

12. Motor vehicle (10) comprising

- a display device (14) according to claim 11 , - a video source device (11 ) for generating a video stream (23), and

- a communication network (15) that links the video source device (11 ) to the display device (14).