WO2019092392A1 - Procédé et système de traitement de données d'affichage - Google Patents

Procédé et système de traitement de données d'affichage Download PDF

Info

Publication number
WO2019092392A1
WO2019092392A1 PCT/GB2018/052966 GB2018052966W WO2019092392A1 WO 2019092392 A1 WO2019092392 A1 WO 2019092392A1 GB 2018052966 W GB2018052966 W GB 2018052966W WO 2019092392 A1 WO2019092392 A1 WO 2019092392A1
Authority
WO
WIPO (PCT)
Prior art keywords
frame
colour
frames
display data
signature
Prior art date
Application number
PCT/GB2018/052966
Other languages
English (en)
Inventor
Prince JOVELURO
Patrick David COOPER
Original Assignee
Displaylink (Uk) Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Displaylink (Uk) Limited filed Critical Displaylink (Uk) Limited
Publication of WO2019092392A1 publication Critical patent/WO2019092392A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/60—Memory management
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/14—Coding unit complexity, e.g. amount of activity or edge presence estimation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/18—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a set of transform coefficients
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/507—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction using conditional replenishment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/587—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/625—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding using discrete cosine transform [DCT]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/174—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a slice, e.g. a line of blocks or a group of blocks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock

Definitions

  • the present invention relates to a method and system for processing display data, and in particular to a method and system for processing display data for use in virtual reality systems.
  • Virtual reality systems typically comprise a host computer for generating display data, and a display device, such as a virtual reality headset, for displaying the display data to a user.
  • the host computer may compress and encode the display data for sending to the display device.
  • the resolution of the display data must be high and must have fast refresh and update rates.
  • virtual reality is currently one of the most bandwidth intensive technologies, typically requiring around 90 frames per second to be sent between the host computer and display device. Attempts have been made to try and reduce the bandwidth requirements, but typically only by compressing and/or encoding the display data at the host computer before the display data is sent to the display device. Any way in which the bandwidth requirements could be reduced would clearly be beneficial for allowing the virtual reality technology to develop further.
  • the applicant has identified a method for being able to rapidly process display data, and so a method that is suitable for such virtual reality systems, to identify when frames are similar enough to be dropped, and so to reduce bandwidth requirements without compromising on the quality of the user experience.
  • a method of processing display data for a system comprising a host device for generating display data, and a display device for displaying the generated display data to a user, wherein the generated display data comprises frames of display data for displaying consecutively to a user, the method comprising:
  • the method further comprises identifying the second frame, or portion of the second frame, as a candidate for dropping.
  • This method may be particularly efficient since the frames typically need to be compressed or processed before being sent to the display device in any case, and this method can take advantage of values that would be generated during compression/processing. This method therefore requires very little additional processing, but has the potential to vastly reduce the bandwidth requirements of the system.
  • Each frame or portion of a frame will typically have thousands or millions of pixels and will always have at least 1000 pixels. Corresponding portions of frames may be located at the same position, or location, within a frame.
  • Determining a colour signature for a frame, or portion of a frame may comprise compressing the frame, or portion of the frame, respectively. Determining a colour signature for a frame, or portion of a frame, may comprise determining an average colour value for the frame, or portion of the frame, respectively. Determining a colour signature for a frame, or portion of a frame, may comprise applying a transformation to the frame, or portion of the frame, respectively, and wherein the colour signature corresponds to a generated DC value for the frame, or portion of the frame, respectively, following the transform.
  • the transform may comprise Haar encoding.
  • the transform may comprise a discrete cosine transformation (DCT).
  • the colour signature for a frame, or portion of a frame may correspond to any one or more of a red, green, blue, luma and chroma signature. Use of certain colour signatures may be more suitable for certain applications.
  • the method may further comprise dropping the second frame, or portion of the second frame.
  • dropping the second frame, or portion of the second frame Given that it has been identified that the frames are unchanged for the purpose of perception by the human eye, it may be more efficient to just drop the frame or portion at this stage. This may be beneficial where it would be particularly helpful to reduce the bandwidth.
  • the method may further comprise determining whether to drop the second frame, or portion of the second frame, or whether to instead send the second frame, or portion of the second frame, to the display device. This second check may be performed to prevent longer term divergence, and error propagation in the system.
  • the memory may be configured to store a predetermined number of colour signatures, and wherein the method may further comprise tagging the colour signatures stored in the memory to indicate whether the corresponding frames, or portions of frames, were dropped.
  • the memory may be configured to store just the previous colour signature.
  • a counter may be increased when a frame, or portion of a frame, is dropped, and on identifying the second frame, or portion of the second frame, as a candidate for dropping, the method may further comprise comparing the counter value to a second threshold value to determine whether to drop the second frame, or portion of the second frame, wherein if the counter value exceeds the second threshold value, the method further comprises dropping the second frame, or portion of the second frame.
  • the method may further comprise determining a number of previous frames, or portions of frames, immediately preceding the second frame, or portion of the second frame, that were dropped, and comparing this number to a third threshold value, wherein if this number does not exceed this threshold value, the method comprises dropping the second frame, or portion of the second frame, and wherein if this number of previous frames, or portions of frames, does exceed this third threshold value, the method comprises sending the second frame, or portion of the second frame, to the display device for displaying to the user.
  • the method thereby reduces the chance of any longer- term divergence, and error propagation in the system.
  • the method ensures that the current frame is then sent to the display device for display to the user.
  • the third threshold value may be any one of 5, 6, 7, 8, 9 or 10.
  • the third threshold may also be below 5 or above 10. It may be possible to alter the third threshold depending on the application. Particular thresholds may be more suited to particular applications.
  • the method may further comprise detecting movement of the display device, and if movement of the display device is detected, the first threshold may be reduced to provide for a higher similarity requirement. This may help reduce the chance of error propagation in the system.
  • the first threshold may be reduced further where detected movement is faster.
  • the generated display data may be video data and a frame of the video data may correspond to image data.
  • the frames of display data may be generated, processed and sent to the display device at a rate of approximately at least 50 frames per second, or at least 60 frames per second, or at least 90 frames per second, or at least 120 frames per second for displaying to a user.
  • the system may be a virtual reality system. This method may be particularly advantageous in such systems.
  • the host device and the display device may be wirelessly connected. Having a method that allows for reduced bandwidth requirements may provide for greater ease in use of wireless display devices, and provide for a greater ability in further developing such systems.
  • the wireless connection may use any one or more or Wi-Fi, Radio, Internet or any other suitable technology.
  • the display device may be a head mounted display, and/or may comprise augmented reality glasses.
  • the host device and the display device may be contained within a housing.
  • the housing may be the casing for any one of a mobile phone, a PDA, a tablet or any other handheld portable device.
  • the memory may be a buffer. This may be particularly suitable. Reduced storage requirements may free up the processing capacity of the system for other uses. Any other suitable memory or memory unit could be used.
  • a system for processing display data comprising:
  • a host device for generating and processing display data, wherein the generated display data comprises frames of display data for displaying consecutively to a user, the host device comprising a processor and a memory;
  • a display device connected to the host device configured to receive generated display data from the host device and to display the generated display data to a user; wherein the processor is configured to:
  • the processor is further configured to identify the second frame, or portion of the second frame, as a candidate for dropping.
  • This system may be particularly efficient since the frames typically need to be compressed or processed before being sent to the display device in any case, and this system can take advantage of values that would be generated during compression/processing. This system therefore requires very little additional processing, but has the potential to vastly reduce the bandwidth requirements of the system.
  • the processor may be configured to compress the frame or portion of the frame, respectively.
  • the processor may be configured to determine an average colour value for the frame, or portion of the frame, respectively.
  • the processor may be configured to apply a transform to the frame, or portion of the frame, respectively, and the colour signature may correspond to a generated DC value for the frame, or portion of the frame, respectively, following the transform.
  • the transform may comprise Haar encoding.
  • the transform may comprise a discrete cosine transformation (DCT).
  • the colour signature for a frame, or portion of a frame may correspond to any one or more of a red, green, blue, luma and chroma signature.
  • the processor may be further configured to drop the second frame, or portion of the second frame.
  • the processor may be further configured to determine whether to drop the second frame, or portion of the second frame, or whether to instead send the second frame, or portion of the second frame, to the display device.
  • the memory may be configured to store a predetermined number of colour signatures
  • the processor may be configured to tag the colour signatures stored in the memory to indicate whether the corresponding frames, or portions of frames, were dropped.
  • the memory may be configured to store just the previous colour signature.
  • the system may include a counter, and the processor may be configured to increase the counter when a frame, or portion of a frame, is dropped, and on identifying the second frame, or portion of the second frame, as a candidate for dropping, the processor may be further configured to compare the counter value to a second threshold value to determine whether to drop the second frame, or portion of the second frame, wherein if the counter value exceeds the second threshold value, the processor is further configured to drop the second frame, or portion of the second frame.
  • the processor may be configured to determine the number of previous frames, or portions of frames, immediately preceding the second frame, or portion of the second frame, that were dropped, and to compare this number to a third threshold value, wherein if this number does not exceed this threshold value, the processor may be configured to drop the second frame, or portion of the second frame, and wherein if this number does exceed this third threshold value, the processor may be configured to send the second frame, or portion of the second frame, to the display device for displaying to the user.
  • the third threshold value may be any one of 5, 6, 7, 8, 9 or 10.
  • the third threshold may also be below 5 or above 10. It may be possible to alter the third threshold depending on the application.
  • the system may comprise means for detecting movement of the display device, wherein the means are in communication with the processor, and wherein when movement of the display device is detected, the processor is configured to reduce the first threshold to provide for a higher similarity requirement.
  • the means may comprise an accelerometer or any other suitable device. The first threshold may be reduced further where detected movement is faster.
  • the generated display data may be video data and a frame of the video data corresponds to image data.
  • the frames of display data may be generated, processed and sent to the display device at a rate of approximately at least 50 frames per second, or at least 60 frames per second, or at least 90 frames per second, or at least 120 frames per second for displaying to a user.
  • the system may be a virtual reality system.
  • the host device and the display device may be wirelessly connected.
  • the display device may be a head mounted display, and/or may comprise augmented reality glasses.
  • the host device and the display device may be contained within a housing.
  • the housing may be the casing for any one of a mobile phone, a PDA, a tablet or any other handheld portable device.
  • the memory may be a buffer. This may be particularly suitable. Reduced storage requirements may free up the processing capacity of the system for other uses. Any other suitable memory or memory unit could be used.
  • Figure 1 shows a block diagram overview of a system for processing display data
  • Figure 2 shows a block diagram overview of a system for processing display data wherein the display device is a headset;
  • Figure 3 shows a block diagram overview of a system for processing display data, wherein the host device and display device are contained within a single casing, for example in a smartphone or other such mobile computing device;
  • Figure 4 shows a method for processing display data in accordance with an embodiment of the present invention
  • Figure 5 shows an exemplary Haar encoding process.
  • Figure 1 shows a block diagram overview of a system according to the current art.
  • a host computer [1 1] is connected to a display control device [12], which is in turn connected to a display device [13].
  • the host [1 1] contains an application [14], which produces display data.
  • the display data may be produced and sent for compression either as complete frames or as canvasses, which may, for example, be separate application windows. In either case, they are made up of tiles of pixels, where each tile is a geometrically-shaped collection of one or more pixels.
  • the display data is sent to a compression engine [15], which may comprise software running in a processor or an appropriate hardware engine.
  • the compression engine [15] may perform an encoding of the data, to convert the data into a format that may then be further compressed, minimising data loss.
  • the compression engine [15] may then further compress the data and thereafter send the compressed data to an output engine [16].
  • the output engine [16] manages the connection with the display control device [12] and may, for example, include a socket for a cable to be plugged into for a wired connection or a radio transmitter for a wireless connection. In either case, it is connected to a corresponding input engine [17] on the display control device [12].
  • the input engine [17] is connected to a decompression engine [18]. When it receives compressed data, it sends it to the decompression engine [18] or to a memory from which the decompression engine [18] can fetch it according to the operation of a decompression algorithm.
  • the decompression engine [18] may decompress the data, if necessary, and performs a decoding operation.
  • the decompressed data is then sent to a scaler [19].
  • the display data was produced and compressed as multiple canvasses, it may be composed into a frame at this point.
  • scaling it is preferable for it to be carried out on a display control device [12] as this minimises the volume of data to be transmitted from the host [1 1] to the display control device [12], and the scaler [19] operates to convert the received display data to the correct dimensions for display on the display device [13].
  • the scaler may be omitted or may be implemented as part of the decompression engine.
  • the data is then sent to an output engine [1 10] for transmission to the display device [13]. This may include, for example, converting the display data to a display-specific format such as VGA, HDMI, etc.
  • the display device is a virtual reality headset [21], as illustrated in Figure 2, connected to a host device [22], which may be a computing device, gaming station, etc.
  • the virtual reality headset [21] incorporates two display panels [23], which may be embodied as a single panel split by optical elements. In use, one display is presented to each of a viewer's eyes.
  • the host device [22] generates image data for display on these panels [23] and transmits the image data to the virtual reality headset [21].
  • the headset is a set of augmented reality glasses.
  • the headset is a set of augmented reality glasses.
  • the host device [22] may be a static computing device such as a computer, gaming console, etc., or may be a mobile computing device such as a smartphone or smartwatch. As previously described, it generates image data and transmits it to the augmented reality glasses or virtual reality headset [21] for display.
  • the display device may be connected to the host device [1 1 , 22] or display control device [12] if one is present by a wired or wireless connection. While a wired connection minimises latency in transmission of data from the host to the display, wireless connections give the user much greater freedom of movement within range of the wireless connection and are therefore preferable. A balance must be struck between high compression of data, in particular video data, which can be used to enable larger amounts of data (e.g. higher resolution video) to be transmitted between the host and display, and the latency that will be introduced by processing of the data.
  • the end-to-end latency between sensing a user's head movement, generating the pixels in the next frame of the VR (virtual reality) scene and streaming the video should be kept below 20ms, preferably below 10ms, further preferably below 5ms.
  • the wireless link should be implemented as a high bandwidth short-range wireless link, for example at least 1 Gbit/s, preferably at least 2 Gbit/s, preferably at least 3 Gbit/s.
  • An "extremely high frequency (EHF)" radio connection such as a 60GHz radio connection is suitable for providing such high-bandwidth connections over short-range links.
  • Such a radio connection can implement the WiFi standard IEEE 802.1 1 ad.
  • the 71 - 76, 81 -86 and 92-95 GHz bands may also be used in some implementations.
  • the wireless links described above can provide transmission between the host and the display of more than 50 frames per second, preferably more than 60 frames per second, further preferably more than 90 frames per second, or even as high as 120 frames per second.
  • Figure 3 shows a system which is similar in operation to the example shown in
  • FIG. 2 In this case, however, there is no separate host device [22].
  • the entire system is contained in a single casing [31], for example in a smartphone or other such mobile computing device.
  • the device contains a processor [33], which generates display data for display on the integral display panel [32].
  • the mobile computing device may be mounted such that the screen is held in front of the user's eyes as if it were the screen of a virtual reality headset.
  • a system and method for processing display data may comprise any or all of the features of the systems described above in relation to Figures 1 -3.
  • a computing device, or host contains an application, which generates image, or display, data, for playing back to a user as video.
  • each frame comprises a plurality of pixels, wherein each pixel may comprise values for the levels of red (R), green (G), and blue (B) therein.
  • RGB red
  • the pixel data can be processed as separate, or as a combination of, R, G, and B values, and/or the pixel data can be converted to luma (Y) and chroma ( ⁇ , ⁇ ) values where luma indicates the luminescence of the pixel and chroma indicates its colour.
  • the display data is sent for processing/compression as complete frames, or as tiles of pixels; the frames are made up of tiles of pixels, where each tile is a geometrically-shaped collection of one or more pixels.
  • Processing/compression of a frame (or tile) in one embodiment involves performing a transform on the pixel values for that frame. This is preferably a Haar encoding process, but may also be a Discrete Cosine Transformation (DCT). This results in a single DC value being generated for a frame (or tile), associated with a particular pixel for that frame (or tile), and which provides a colour signature for that frame (or tile), and represents the average colour of the pixels in that frame (or tile). It will be appreciated that this could also be performed on individual tiles in the frames, as well as or instead.
  • a transform on the pixel values for that frame. This is preferably a Haar encoding process, but may also be a Discrete Cosine Transformation (DCT). This results in a single DC value being generated for a frame (or tile), associated with
  • performing such a transform also results in generation of N-1 AC values, one for each of the remaining pixels in the frame, these AC values representing the colour change between the pixels, across the frame. This may be performed on the display data in the RGB colour space and/or in a colour space related to luma and chroma.
  • These and/or other display data values may then be used in further processing of the image data, and for sending to an output engine in the host, which manages the connection with a display control device, and so display device.
  • the output engine is connected to a corresponding input engine on the display control device, and so display device.
  • the DC value for a frame Before being sent to the output engine, the DC value for a frame, and in some cases the corresponding AC values, are stored in a memory unit of the host.
  • the memory unit may be a short term or temporary memory, for instance a buffer.
  • the DC value of the current frame being processed is compared to the DC value of the previous frame that is stored in the buffer. This could be a DC value for each colour component, or for a combination. Either way, when the current DC value is within a predetermined range of DC (or colour) values of the corresponding previous DC value, wherein the predetermined range of DC (or colour) values is centred on the previous DC value for the previous frame, it can be assumed that the frame is unchanged (at least according to the perception of the human eye), and so the frame is identified as a candidate for dropping. In some embodiments, the current frame can just be dropped at this stage. If the frame is dropped, it is not forwarded to the display control device or display device for displaying to a user.
  • the DC value corresponding to the dropped frame may be tagged to indicate that the frame was dropped. This current DC value is then stored in the buffer for comparison with the corresponding DC value for the next frame. In some embodiments, a further check is performed on the candidate before deciding whether the frame should be dropped.
  • the buffer may store a predetermined number of the DC values, for instance x DC values (where x may for instance be 1 , 2, 3, 4, 5, 6, 7, 8, 9, 10 or more).
  • the DC values may be tagged or marked with whether the associated frames were dropped. Therefore, once a candidate for dropping is determined, before dropping the frame associated with the current DC value, it is determined whether a predetermined number of the previous frames, preceding the current frame, for instance whether the previous y frames (where y may for instance be 5, 6, 7, 8, 9, 10 or more), were sent for further processing and to the display device, using the previous y DC values stored in the buffer. If the previous y frames were not sent for further processing and to the display device, then the current frame is nonetheless sent for further processing for sending to the display device.
  • This may be carried out by determining the number of DC values in the buffer, preceding the DC value associated with the current frame, that are tagged to indicate that their associated frame was dropped, and if the number of previous frames that were dropped exceeds a predetermined threshold value, then the current frame is nonetheless sent for further processing and/or for sending to the display device. This prevents the propagation of errors in the system. Otherwise, the current frame is dropped, not sent for further processing, and not sent to the display device. Being able to process the data quickly in this way, and to determine whether to drop frames quickly in this way therefore enables a reduction in the bandwidth required, while still providing the user of the display device with an acceptable level of visual quality.
  • a counter that counts the frames as they are dropped, and then re-sets when a frame is not dropped. This counter could therefore instead be compared to a threshold value to determine whether a candidate for dropping should be dropped.
  • the memory may only need to store a single colour signature, or colour value or DC value.
  • the display device may comprise an accelerometer, or other means for detecting and/or measuring motion of the display device.
  • the host device may be configured to monitor movement of the display device, in particular whether the display device is moving and how fast the display device may be moving.
  • the frame may be sent regardless.
  • the similarity requirements for the DC values, or other colour values may be stricter when it is determined that the display device is moving and/or when it is determined that the display device is moving at a velocity above a particular threshold. For instance, when the DC value of the current frame is compared to the DC value of the previous frame that is stored in the buffer, the predetermined range of colour values may be narrower when the display device is moving. This is to ensure that it can still be assumed that the frame is unchanged, at least according to the perception of the human eye, and so the user.
  • Haar transform takes place on the host [1 1], specifically in the compression engine [15].
  • Decompression takes place on the display control device [12], specifically in the decompression engine [18], where the data is put through an inverse Haar transform to return it to its original form.
  • each tile [41] comprises one pixel, but of course may be much larger.
  • Each pixel [41] has a value indicating its colour, here represented by the pattern of hatching.
  • the first pixel [41 A] is marked with dots and considered to have the lightest colour.
  • the second pixel [41 B] is marked with diagonal hatching and is considered to have the darkest colour.
  • the third pixel [41 C] is marked with vertical hatching and is considered to have a light colour
  • the fourth pixel [41 D] is marked with horizontal hatching and is considered to have a dark colour.
  • the values of the four pixels [41] are combined using the formulae [44] shown to the right of Figure 5 to produce a single pixel value [42], referred to as "W", which is shaded in grey to indicate that its value is derived from the original four pixels [41], as well as a set of coefficients [43] referred to in Figure 5 as "x, y, z".
  • the pixel value [42] is generated from a sum of the values of all four pixels: ((A+B)+(C+D)).
  • the three coefficients [43] are generated using the other three formulae [44] as follows:
  • any or all of these values may then be quantised: divided by a constant in order to produce a smaller number which will be less accurate but can be more effectively compressed and rounded.
  • W may be used as a colour value for the tile, or frame where appropriate, alone and/or in combination with any or all of the coefficients.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Discrete Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

L'invention concerne un procédé de traitement de données d'affichage pour un système, comprenant les étapes consistant à : déterminer une première signature de couleur pour une première trame, ou une partie d'une première trame, de données d'affichage générées; stocker la première signature de couleur dans une mémoire; à déterminer une seconde signature de couleur pour une seconde trame, ou une partie d'une seconde trame, de données d'affichage générées, la seconde trame ou partie de la seconde trame étant destinée à afficher consécutivement à l'utilisateur suivant la première trame, ou une partie de la première trame respectivement, et la partie de la seconde trame correspondant à la partie de la première trame; à comparer la seconde signature de couleur à la première signature de couleur pour déterminer une différence dans les signatures de couleur; et à comparer la différence des signatures de couleurs à un premier seuil, si la différence dans les signatures de couleur est inférieure au premier seuil, le procédé consiste en outre à identifier la seconde trame, ou une partie de la seconde trame, en tant que candidat pour abandon.
PCT/GB2018/052966 2017-11-07 2018-10-15 Procédé et système de traitement de données d'affichage WO2019092392A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB1718421.9A GB2568112B (en) 2017-11-07 2017-11-07 Method and system for processing display data
GB1718421.9 2017-11-07

Publications (1)

Publication Number Publication Date
WO2019092392A1 true WO2019092392A1 (fr) 2019-05-16

Family

ID=60664900

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB2018/052966 WO2019092392A1 (fr) 2017-11-07 2018-10-15 Procédé et système de traitement de données d'affichage

Country Status (2)

Country Link
GB (1) GB2568112B (fr)
WO (1) WO2019092392A1 (fr)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130266073A1 (en) * 2012-04-08 2013-10-10 Broadcom Corporation Power saving techniques for wireless delivery of video

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110032984A1 (en) * 2008-07-17 2011-02-10 Guy Dorman Methods circuits and systems for transmission of video
GB201105716D0 (en) * 2011-04-04 2011-05-18 Advanced Risc Mach Ltd Method of and apparatus for displaying windows on a display
US20140152891A1 (en) * 2012-12-05 2014-06-05 Silicon Image, Inc. Method and Apparatus for Reducing Digital Video Image Data
GB2528265B (en) * 2014-07-15 2021-03-10 Advanced Risc Mach Ltd Method of and apparatus for generating an output frame
GB2531358B (en) * 2014-10-17 2019-03-27 Advanced Risc Mach Ltd Method of and apparatus for processing a frame

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130266073A1 (en) * 2012-04-08 2013-10-10 Broadcom Corporation Power saving techniques for wireless delivery of video

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
JUNG WOO LEE ET AL: "Hash aided Motion Estimation: Various Approaches", 11 March 2005 (2005-03-11), XP055300213, Retrieved from the Internet <URL:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.467.5002&rep=rep1&type=pdf> *
ZHU WEIJIA ET AL: "Hash-Based Block Matching for Screen Content Coding", IEEE TRANSACTIONS ON MULTIMEDIA, IEEE SERVICE CENTER, PISCATAWAY, NJ, US, vol. 17, no. 7, 1 July 2015 (2015-07-01), pages 935 - 944, XP011584980, ISSN: 1520-9210, [retrieved on 20150615], DOI: 10.1109/TMM.2015.2428171 *

Also Published As

Publication number Publication date
GB201718421D0 (en) 2017-12-20
GB2568112A (en) 2019-05-08
GB2568112B (en) 2022-06-29

Similar Documents

Publication Publication Date Title
US11553222B2 (en) Low latency wireless virtual reality systems and methods
US11217201B2 (en) Video frame interfaces for logically-defined pixels
US12096031B2 (en) Method and apparatus for digital content stream compression and decompression
US11615734B2 (en) Method and apparatus for colour imaging
KR102617258B1 (ko) 이미지 프로세싱 방법 및 장치
WO2015167313A1 (fr) Procédé et dispositif de compression adaptative de données d&#39;image
EP4046382A1 (fr) Procédé et appareil de codage vidéo pour des machines
US20220382053A1 (en) Image processing method and apparatus for head-mounted display device as well as electronic device
EP3300368A1 (fr) Procédé d&#39;encodage et de décodage vidéo, et encodeur/décodeur vidéo
US9123090B2 (en) Image data compression device, image data decompression device, display device, image processing system, image data compression method, and image data decompression method
US20240054623A1 (en) Image processing method and system, and device
WO2019092392A1 (fr) Procédé et système de traitement de données d&#39;affichage
CN109413445B (zh) 一种视频传输方法及装置
US20230395041A1 (en) Content Display Process
US20240121406A1 (en) Content Compression for Network Transmission
CN101065760B (zh) 处理图像数据的系统和方法
US11233999B2 (en) Transmission of a reverse video feed
US20210058616A1 (en) Systems and Methods for Selective Transmission of Media Content
CN112929703A (zh) 码流数据的处理方法和装置
US9571844B2 (en) Image processor
WO2022246653A1 (fr) Système de traitement d&#39;image, extrémité de service en nuage et procédé
CN100428269C (zh) 处理图像数据的方法
CN113721877A (zh) 推屏方法、装置、设备和存储介质
CN114630097A (zh) 图像处理方法、装置、系统及计算机可读存储介质
KR20100119088A (ko) 영상데이터 처리 방법 및 그 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18792451

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18792451

Country of ref document: EP

Kind code of ref document: A1