JP2015520406A - Display power reduction using SEI information - Google Patents

Abstract

The video segment is a payload unit with a supplemental enhancement information (SEI) message that contains display adjustment information that can be used to control display brightness and reduce power consumption between each segment display. It is transmitted separately. The display adjustment information scales the pixel brightness to the maximum, thereby reducing backlighting for the liquid crystal display or at least the maximum pixel brightness that can be used to adjust the supply voltage for the OLED display. Including. Display adjustment information may optionally include an indicator of minimum pixel brightness, histogram step size, and scaling method.

Description

  The present invention relates to reducing energy consumption of a wireless mobile communication device, and more particularly to content-based display adjustment control for video content displayed on a wireless communication device.

  Recently, the display resolution on mobile devices has become the mainstream or mainstream of 720p or higher super liquid crystal display (LCD) or organic light emitting diode (OLED) displays. It has greatly improved to the point. However, such a high display resolution requires more energy for rendering, especially for videos where high frequency frame buffering and display panel refresh are essential.

  For LCD displays, power consumption is a monotonic function of backlighting brightness level, and for OLED displays, power consumption is controlled by the supply voltage and the display content itself. Brightness control is already implemented on some mobile devices, and such control usually has to be applied before new work is given, ie before video playback starts. For example, the brightness can be set to 100%, 50%, or 25% before watching the video, but cannot be changed dynamically without disturbing the video playback. Also, the current implementation does not provide a mechanism for adjusting the voltage because power consumption is determined by the supply voltage and input image to the OLED.

  Therefore, there is a need in the art to improve mobile device displays by greatly reducing display energy by allowing LCD display backlighting brightness or OLED supply voltage to be adjusted according to the content being displayed.

  The present invention provides a method and apparatus for efficiently reducing the display power, in order to solve the above-mentioned problems that occur in the related art.

  According to one aspect of the present invention, receiving video segment data, the received data being used to adjust display parameters for adjusting one or more display parameters that affect display brightness and power consumption. Including a supplemental enhancement information (SEI) message including information, and adjusting at least one display parameter for display based on the display adjustment information from the SEI message during display of the video segment. A method is provided.

  According to another aspect of the present invention, a receiver configured to receive video segment data, and the received data is one or more display parameters that affect display brightness and power consumption. At least one display parameter for display based on the display adjustment information from the SEI message during display of the video segment, including a SEI (supplemental enhancement information) message including display adjustment information for adjusting the display And a processor configured to regulate the system.

  According to yet another aspect of the present invention, the method includes receiving data of a video segment to transmit, wherein the received data adjusts one or more display parameters that affect display brightness and power consumption. There is provided a method including a supplemental enhancement information (SEI) message including display adjustment information for displaying, and one or more payload units including data for displaying the video segment.

  In accordance with yet another aspect of the present invention, a video server configured to format data for a transmitted video segment, wherein the formatted data affects display brightness and power consumption1. A system is provided that includes a supplemental enhancement information (SEI) message including display adjustment information for adjusting one or more display parameters, and one or more payload units including data for displaying the video segment.

  According to yet another aspect of the present invention, receiving data of a video segment to transmit, said data for adjusting one or more display parameters that affect display brightness and power consumption. Including a supplemental enhancement information (SEI) message including display adjustment information and one or more payload units including data for displaying the video segment, and transmitting the data to a device on which the video segment is to be displayed. Is provided.

  A more complete understanding of the present invention and the advantages associated therewith can be more readily understood with reference to the following detailed description when considered in conjunction with the accompanying drawings. In the drawings, the same reference numerals denote the same or similar components.

FIG. 6 is a high-level diagram illustrating a network that enables internal devices to achieve dynamic content-based display power reduction according to one or more embodiments of the present invention. 1B is a front view of a wireless device from the network of FIG. 1A in which dynamic content-based display adjustment and corresponding power reduction may be implemented according to an embodiment of the present invention. FIG. FIG. 1B is a high-level block diagram for functional components of the wireless device shown in FIG. 1B. FIG. 6 is a diagram illustrating a display adjustment that maintains brightness using display adjustment information embedded in a SEI message for dynamic content-based display adjustment and corresponding power reduction according to an embodiment of the present invention. FIG. 6 is a diagram illustrating a display adjustment that maintains brightness using display adjustment information embedded in a SEI message for dynamic content-based display adjustment and corresponding power reduction according to an embodiment of the present invention. FIG. 6 illustrates SEI message insertion in a video data bitstream for dynamic, content-based display adjustment and corresponding power reduction according to one embodiment of the invention. FIG. 6 illustrates SEI message insertion in a video data bitstream for dynamic, content-based display adjustment and corresponding power reduction according to one embodiment of the invention. FIG. 5 is a high level flow diagram of a process for encoding video utilizing SEI message insertion for dynamic content-based display adjustment and corresponding power reduction according to an embodiment of the present invention. FIG. 6 is a high-level flow diagram of a video decoding and display process based on SEI messages inserted for dynamic content-based display adjustment and corresponding power reduction according to an embodiment of the present invention.

  Video segments have payloads with supplemental enhancement information (SEI) messages that incorporate display adjustment information that can be used to control display brightness and reduce power consumption between each segment display. Sent by unit. The display adjustment information includes at least the maximum pixel brightness that can be used to scale the pixel brightness to the maximum, thereby reducing backlighting for the liquid crystal display or adjusting the supply voltage for the OLED display. The display adjustment information may optionally include an indicator of minimum pixel brightness pixel histogram step size and scaling method.

  Prior to describing the present invention in detail, it is desirable to disclose definitions of specific words and phrases used throughout the specification. Not only the phrases “include” and “comprise”, but also the derivatives thereof are meant to include, but not be limited to. The term “or” encompasses the meaning of “and / or”. Not only the terms “associated with” and “associated thewith”, but their derivatives include “include”, “be included with”, “concatenated with each other”. “Interconnect with”, “contain”, “be connected with”, “connect to or with”, “couple to or with”, “communication” “Becommunicable with”, “cooperate with”, “interleave”, “jumptapose”, “proximity” That (be proximate to) "," contact (be bound to or with) "," has (have) ", and" has the characteristics can be and the like (have a property of) ". A controller refers to a device, system, or part thereof that controls at least one operation, and can be implemented in hardware, firmware, software, or a combination of two or more thereof. It should be noted that the functions associated with a particular controller may be centralized or distributed over short or long distances. Such definitions for specific words and phrases are set forth throughout this specification, and those skilled in the art will find such definitions in many, if not most, cases. Obviously, this applies to the future use as well as the prior use of the words and phrases defined in.

  The various embodiments used to describe the principles of the invention of FIGS. 1-5 and this patent document discussed below are exemplary only and limit the scope of the invention in any way. Should not be interpreted. Those skilled in the art will appreciate that the principles of the present invention may be implemented in a suitably configured wireless communication system.

  Metadata used for display adjustment may be embedded in the video stream as an extended SEI (supplemental enhancement information) message.

  In the present invention, display adaptation is embedded in the video content information using SEI messages that are parsed by the decoder to help reduce display power. In the case of an LCD display, the brightness of the display is adjusted, and in the case of an OLED display, the display supply voltage is adjusted. Elements in such an extended SEI message can be derived from the encoder during video encoding.

  Display adjustments allow SEI messages (ie display_adaptation ()) that can be inserted by frame in the stream, by GOP (group of pictures), by GOP scene, or by time interval, depending on the basic application and hardware capabilities. Is defined by Compared to frame level solutions, GOP, scene or time interval based processing schemes require less overhead for message insertion. For processors that cannot support high frequency display adjustments, eg 33 ms per 30 Hz video, GOP, scene or time interval based methods are better than frame based solutions. Nevertheless, the concept is basically explained in this document using a frame level solution.

  FIG. 1A is a high-level diagram illustrating a network that enables internal devices to achieve dynamic content-based display adjustment and corresponding power reduction in accordance with one or more embodiments of the present invention. The network 100 includes existing procedures, however, an encoder controller configured to encode video content with display adjustment information embedded within the NALU header (s) as described in more detail below. An encoder data processing system 101 is included. Content encoder 101 is communicatively coupled (or otherwise merged) with a content server data processing system 102 that communicates video content to a user device. The content server 102 is connected by a wireless communication system including a communication network such as the Internet 103 and a base station (BS) 104 for video content transmission to a user equipment 105 called a user device (UE) or mobile station (MS). Is done. As described above, the user device 105 may be a “smart” phone or tablet device capable of functions including wireless audio communication and at least video content playback. Unlike this, the user device 105 has an LCD or OLED display and is at least a dynamic content base during video playback, as is the case with certain devices that basically receive power from the battery during at least typical operation. It can be a laptop computer or other wireless device that benefits from reduced display power.

  FIG. 1B is a front view of a wireless device from a network in which dynamic content-based display adjustment and corresponding power reduction may be implemented according to one embodiment of the present invention, and FIG. 1C is a functional component of the wireless device. It is a high-level block diagram. User device 105 is a mobile phone and includes a backlit LCD (including the selective luminance light source shown in FIG. 1C) and OLED display 106. A processor 107 coupled to the display 106 controls the content displayed on the display. The processor 107 and other components in the user device 105 are charged by an external power source (not shown) or powered by a battery (not shown) that can be powered by the external power source. A memory 108 coupled to the processor 107 can store or buffer video content played by the processor 107 and displayed on the display 106, and a video playback application (or to perform such video playback) (or "App") 109 can also be saved. The video content to be played can be received through the transceiver 110 connected to the antenna 111 either concurrently (eg, overlapping in time) or prior to playback of the video content. As described above, video content can be received through wireless communication from the base station 104. In an exemplary embodiment, the video content received by the mobile device 105 for internal playback and display on the display 106 includes display adjustment information embedded within the SEI message (s). Display adjustment information is used by processor 107 to set display controls 112 for the selective luminance light source and display 106.

  ITU-T (International Telecommunication Union (ITU) Telecommunication Standardization Section) VCEG (Video Coding Experts Group) H. ITU-T (International Telecommunication Union (ITU) Telecommunication Standardization Section) H.264 high-end video coding (also referred to as AVC, MPEG-4 Part 10 (Motion Picture Experts Group 4 Part 10)) and its extensions, each SEI message (s) as described in more detail below. Inserted into the bitstream. The new SEI message with payloadType = 47 is as shown in Table 1 below. (The selection of payloadType = 47 is illustrative in this example, and an unspecified payloadType value can be used instead). Each time it meets an SEI message in the bitstream, the decoder parses the SEI message and performs a frame level, GOP level, scene level or time interval level display adjustment as defined in Table 2.

  The current definition of SEI messages is modified through extensions to support the inclusion of display adjustment related information. <Table 1> 3 shows an extended SEI message for H.264 / AVC and its extensions (modifications shown in italics in Table 1).

  <Table 2> The display adjustment SEI message in H.264 / AVC shows the syntax (the correction items shown in italics in <Table 2>).

  As obvious from Table 2, the following three different types of display adjustments (“display_scaling_method”) are considered: Display adjustments that maintain pixel brightness (“BRIGHTNESS_PRESVERVED”); Maintain display adjustment (“CONTRAST_PRESSERVED”); and cognitive lossless display adjustment (“PERCEPTUAL_LOSSLESS”). Display adjustment to maintain brightness takes one value as a parameter-the maximum pixel brightness value ("max_pixel_value") within the pixel brightness value for the reconstructed frame encoded with each NALU header. . Display adjustments that maintain contrast take both the maximum and minimum pixel brightness values (“max_pixel_value”) as parameters in the histogram of pixel brightness values for the reconstructed frame. Cognitive lossless display adjustments that maintain both brightness and contrast are three parameters: the maximum and minimum pixel brightness values in the histogram (“max_pixel_value”) and the pixel brightness value used when generating the histogram. Step size (“pixel_hist_stepsize”).

  ITU VCEG and ISO (International Standards Organization) / IEC (International Electro-technical Commission) MPEG (Motion Pictures Expert Cobalt Standard) JCT-VC (Joint CollaboLab. Standard) With H.265 High Efficiency Video Coding (HVEC), both SEI and video usability information (VUI) metadata are allowed. Thus, those skilled in the art will know how the techniques described above can be easily adjusted for use with HVEC streams.

  2A and 2B illustrate a display adjustment that maintains brightness using display adjustment information embedded in a SEI message for dynamic content-based display adjustment and corresponding power reduction according to an embodiment of the present invention. Illustrate. Prior to decoding all frames, the SEI message is parsed to extract the maximum pixel value used to extend the current reconstructed frame by 255 / max_pixel_value). Let p (i) denote the original brightness of the i th pixel value (in raster scan order) in the pixel brightness histogram for the reconstructed frame, as shown in FIG. 2A. , The pixel brightness pnew (i) scaled for that pixel in the scaled frame histogram shown in FIG. 2B is (in the 8-bit pixel brightness value):

  Here, Y = (255 / max_pixel_value), and max_pixel_value is a parameter specified by the SEI message as described above. As is obvious from a comparison of FIGS. 2A and 2B, the histogram moves through linear scaling.

  On the other hand, through increasing pixel brightness, lower brightness backlighting (for LCD displays) and lower supply voltage (for OLED displays) can be used for net reduction in energy. That is, for LCD displays, scaled pixel brightness is used with reduced backlighting brightness. The scaled value can be set at a ratio of (max_pixel_value / 255) * 100%. That is, the scaled backlighting brightness bnew is as follows:

  Here, b is the original backlighting brightness, and the scaled supply voltage is as follows:

  Here, V is the original supply voltage. Due to the additional reduction in energy, some pixel distortion (ie, saturated after scaling) is allowed, but the maximum pixel value may be further modified so that there is no cognitive difference.

  The parameter min_pixel_value can be used together with max_pixel_value for adjustment when scaling in CONTRAST_PRESVERD mode. The range between maximum and minimum pixel brightness can be adjusted to maintain contrast. Similarly, when scaling in CONSTRAST_PRESVERD mode, the parameters min_pixel_value and pixel_hist_stepsize can be used together with max_pixel_value for adjustment. The range between the maximum and minimum pixel brightness and the distribution of pixel brightness can all be adjusted. While the above assumes linear scaling of backlight brightness and supply voltage, in actual embodiments such scaling is also non-linear. Either linear or non-linear adjustment can be implemented through a look-up table that can be generated by measuring display power at various levels of backlight brightness and supply voltage.

  3A and 3B illustrate SEI message insertion in a video data bitstream for dynamic, content-based display adjustment and corresponding power reduction according to one embodiment of the present invention. FIG. 53A illustrates frame-based SEI message insertion, and FIG. 3B illustrates GOP-based SEI message insertion. Similar insertion schemes can be used for scene-based or time interval-based SEI message insertion.

  In an LCD display with individual backlighting for each of the red (R), green (G), and blue (B) color channels, pixel brightness scaling and backlighting brightness reduction as described above can be achieved for each individual RGB color pixel and backlight. Each can be implemented for lighting brightness. To the extent that individual supply voltages are used for red, green and blue LEDs in an OLED display, pixel brightness scaling and supply voltage reduction as described above can be implemented individually for each RGB color. In such a manner, various color components can be individually adjusted.

  FIG. 4 is a high-level flow diagram of a process for encoding video using SEI message insertion for dynamic content-based display adjustment and corresponding power reduction according to one embodiment of the present invention. This process is performed by the encoder controller within the encoder 101. Devices that do not support display adjustment can easily ignore the display adjustment information embedded in the SEI message, so video encoding is independent of whether transmission to a device that supports display adjustment is intended. The same process can be used for. Process 400 begins with receiving pixel data of a frame, GOP, scene, or time interval segment of a video to be encoded (step 401).

  A pixel brightness histogram of the video data of the segment being processed is determined (step 402), including determination of at least max_pixel_value and optionally min_pixel_value and pixel_hist_stepsize. For the segment of video data being processed, an SEI message containing the scaling method and appropriate parameters is generated (step 403). At this time, the SEI message is inserted into the payload stream associated with the segment data, and the encoded video data is transmitted (step 4044). If the video encoding is incomplete (step 405), then another iteration of the process is performed on the pixel data for the next frame, GOP, scene or time interval segment of the video being encoded.

  FIG. 5 is a high-level flow diagram of a video decoding and display process based on SEI messages inserted for dynamic content-based display adjustment and corresponding power reduction according to an embodiment of the present invention. The above process is executed by the user device 105. Process 500 begins with receiving an SEI message and associated payload for a frame, GOP, scene, or time interval segment of a video to be decoded (step 501). A scaling method and parameter (s) are extracted from the SEI message (step 502), and pixel brightness and supply voltage are adjusted (for OLED display) based on the scaling method and parameter (s), The backlighting brightness is adjusted (for LCD display) (step 503). Video content decoded from the payload of the frame, GOP, scene, or time interval segment is displayed through the adjusted display settings (step 504). If video decoding is incomplete (step 505), then another iteration of the process is performed on the next frame, GOP, scene or time interval segment of the video being decoded.

  Display adjustment using SEI messages based on the brightness maintenance algorithm is described by way of example in the above disclosure. Such an algorithm requires that the maximum pixel value be kept the same as the embedded information. However, the disclosed principles are not limited to such embodiments. In other embodiments, certain information derived from the video encoder to help display adjustments, such as both minimum and maximum pixel brightness values or histogram distributions, may be incorporated as part of the SEI message.

  Because the present invention is more power efficient while reducing data costs, products such as smartphones and tablets can be made that can improve the user experience for mobile streaming applications.

  Each process flow and / or signal sequence illustrated in the drawings and described above describes a sequence of steps and / or signals made in series or in a row, but not explicitly mentioned or obvious (e.g. , The signal cannot be received before transmission), the specific order of execution from such a sequence, the execution of steps or parts or the transmission of the signal is concurrent or sequential not in overlapping order Neither should an assumption be made that execution of the illustrated steps or signal transmission is complete without intervention or intermediate steps or signal generation. Moreover, those skilled in the art will appreciate that complete processes and signal sequences are illustrated or not described. Instead, for simplicity and clarity, only individual processes and signal sequences to the extent that are unique to the invention or necessary for an understanding of the invention are shown and described.

  Although the present invention has been described in connection with specific embodiments, the present invention has been described in detail without departing from the scope and spirit of the invention which is defined based on the description of the claims and equivalents thereof. It will be apparent to those skilled in the art that various changes in the details can be made.

101 Encoder 102 Content Server Data Processing System 103 Internet 104 Base Station 105 User Device 106 Display 107 Processor 108 Memory 109 Video Playback Application 110 Transceiver 111 Antenna 112 Display Control

Claims (17)

A method,
Receiving video segment data, wherein the received data includes display enhancement information for adjusting one or more display parameters that affect display brightness and power consumption; SEI (supplemental enhancement information); Including messages,
Adjusting at least one display parameter for display during display of the video segment based on the display adjustment information from the SEI message.   Scaling the pixel brightness and backlighting brightness of the pixels based on a maximum pixel brightness value from the display adjustment information during display of the video segment on a backlit liquid crystal display (LCD). The method of claim 1, comprising:   During the display of the video segment on the backlit LCD, the pixel brightness and backlighting brightness are scaled based on the maximum pixel brightness value and the minimum pixel brightness value from the display adjustment information. The method of claim 2 further comprising a step.   During display of the video segment on the LCD, the pixel brightness and backlight brightness based on the maximum pixel brightness value, the minimum pixel brightness value, and a pixel histogram step size value from the display adjustment information. The method of claim 3, further comprising the step of scaling the length.   The method of claim 1, further comprising scaling a supply voltage based on a maximum pixel brightness value from the display adjustment information during display of the video segment on an organic light emitting diode (OLED) display. The method described. A system,
A receiver configured to receive data of a video segment, and the received data includes display adjustment information for adjusting one or more display parameters that affect display brightness and power consumption. Configured to include SEI (supplemental enhancement information) messages,
And a processor configured to adjust at least one display parameter for display based on the display adjustment information from the SEI message during display of the video segment.   The processor scales pixel pixel brightness and backlighting brightness based on a maximum pixel brightness value from the display adjustment information during display of the video segment on a backlit liquid crystal display (LCD). The system of claim 6, wherein the system is configured to: The processor is
During the display of the video segment on the backlit LCD, the pixel brightness and backlighting brightness are scaled based on the maximum pixel brightness value and the minimum pixel brightness value from the display adjustment information. 8. The system of claim 7, wherein the system is configured as follows. The processor is
During display of the video segment on the backlit LCD, the pixel brightness based on the maximum pixel brightness value, the minimum pixel brightness value, and a pixel histogram step size value from the display adjustment information. 9. The system of claim 8, wherein the system is configured to scale the backlighting brightness. The processor is
7. The display voltage of the video segment on an organic light emitting diode (OLED) display is configured to scale a supply voltage based on a maximum pixel brightness value from the display adjustment information. The system described in. A mobile communication device comprising the system of claim 6, comprising:
The mobile communication device is configured to receive video segment data from a base station through wireless communication. A tablet comprising the system of claim 6,
The tablet is configured to receive video segment data from a network through wireless communication. A method,
Formatting the data of the video segment to be transmitted, the formatted data comprising:
One or more including a supplemental enhancement information (SEI) message including display adjustment information for adjusting one or more display parameters that affect display brightness and power consumption, and data for displaying the video segment. A method comprising a payload unit. A system,
Including a video server configured to format data of a video segment for transmission, wherein the formatted data comprises:
One or more including a supplemental enhancement information (SEI) message including display adjustment information for adjusting one or more display parameters that affect display brightness and power consumption, and data for displaying the video segment. A system comprising a payload unit. A method,
Receiving data of a video segment to transmit, the data comprising a supplemental enhancement information (SEI) message including display adjustment information for adjusting one or more display parameters that affect display brightness and power consumption. And one or more payload units containing data for displaying the video segment,
Transmitting the data to a device on which the video segment is to be displayed. The display adjustment information is
The method of claim 11, the system of claim 6, and the system of claim 6, comprising one or more of a maximum pixel brightness value, a minimum pixel brightness value, and a pixel brightness histogram step size value. 16. The method of claim 14, the system of claim 14, or the method of claim 15. The display adjustment information is
12. The method of claim 11, comprising an identifier indicating a scaling method selected from a scaling method for maintaining brightness, a scaling method for maintaining contrast, and a cognitive lossless scaling method. The system of claim 13, the method of claim 13, the system of claim 14, or the method of claim 15.

Images