JP2016536815A - Systems and methods for reducing visual artifacts in the display of compression and decompression digital images and video - Google Patents

Abstract

An encoding method and system are disclosed. The processing device receives the image buffer. The processing device converts one or more pixels of the image buffer from a native color space to one or more perceptually uniform color spaces. The processing device multiplies the lightness channel of one or more pixels by a first value. The processing device multiplies one or more color channels of one or more pixels by a second value. The processing device converts the image buffer from one or more perceptually uniform color spaces to a native color space. The processing device transmits the image buffer to the downstream device. [Selection] Figure 2

Description

  Embodiments of the present disclosure relate to image processing, and more particularly to improving perceived quality and / or efficiency of existing image and video compression methods.

  In order to achieve suitable image and video compression efficiency, it is necessary to select a symbol representation of image color and brightness that closely approximates the difference sensitivity of the human visual system (hereinafter referred to as HVS), otherwise Joan L., incorporated herein by reference. “MPEG Video Compression Standard” by Mitchell et al., Chapman & Hall, Ltd. The coding rate is wasted as described in London, UK, 1996. Since the perceptual mechanism of HVS is complex and non-linear, the color sequence and color space design for video and image compression, transmission and display trades off accuracy and precision with system complexity and realistic implementation. It becomes.

  The International Lighting Commission (Commission international l'eclaage) is a color space (hereinafter referred to as “CIE1931XYZ”) based on rough measurement of human color sensation. Indicates the first attempt to generate (ie, 1931 indicates the year of publication). CIE1931XYZ is used in codec systems based on JPEG2000, such as those used by the Digital Cinema Package “Digital Cinema System Specification Version 1.2 with Errata” DCI, LLC, 10 October 2012 However, otherwise, it is not widely used in video transmission systems due to its complexity and the need for standard luminance and tristimulus values to be provided by the system implementer (or end user). Despite its good approximation to the color perception sensitivity of HVS, CIE 1931XYZ is far from a complete representation of HVS sensitivity to color differences, and even if the required tristimulus and luminance values are held constant, Knowledge gained as a result of the study and careful measurement of HVS, as well as the International Color Consortium “Specification ICC. 1: 2004-10 (Profile version 4.2.0.0) Image technology color management— It is far from its perceptual sensitivity described in Architecture, Profile format and data structure (2006).

  After the CIE1931XYZ, due to the advent and standardization of color television, the luma (Y), in-phase (I), quadrature (Q) color space (hereinafter referred to as “YIQ color space”) is mainly used in a transmission / reception system in which the band is strictly limited It was developed in 1953 as a way to encode color signals. YIQ is certainly a better approximation of the sensory trend of human visual perception than the RGB color space representation required by RGB fluorescent cathode-ray tube color televisions in their inventions and in widespread use, Not at all ideal. The YIQ color space retains backward compatibility with existing "black and white" television transmission standards, primarily for realistic implementation purposes of cost-effective analog radio frequency components that were available at the same time It was thought to do.

  The YCbCr and Y'CbCr color spaces are very effective, yet are only derivatives of the YIQ color space aimed at a very coarse approximation of human visual perceptual color processing and perceptual uniformity. While these color spaces were realistic when designed in the early 1980s, designs using YCbCr and Y'CbCr color spaces are simple digital with very limited processing power and digital memory transfer bandwidth. Limited to circuits and systems. The YCbCr and Y'CbCr color space representations form the basis of early and current video compression codec systems that use the JPEG and MPEG standards for compression. Despite their realities, the representation of the YCbCr and Y′CbCr color spaces is not efficient because they assign symbol and bit rates with high luminance and color depth to color differences that are not perceptually important.

More recently, “ISO 11664-4: 2008 (E) / CIE S 014-4 / E: 2007: Joint ISO / CIE Standard: Colorimetry-Part4: CIE 1976 L ^* a ^* b ^* Color Space” (hereinafter, CIELB) A framework of color spaces and perceptual differences that represent a more faithful approximation of HSV perceptual uniformity. CIELAB takes into account perception sensitivity in the lightness and color dimensions and CIEAM02 described in “CIE159: 2004: A Color Appearance Model for Color Management System: CIEAM02” (hereinafter referred to as CIECAM02). H. Incorporates the aforementioned CIELAB dimensions with the well-known spatial center peripheral Retinex effect observed in Land, “The retinex theory of color vision”, Scientific American, 1997.

Even these advanced efforts in defining a perceptually uniform color space are frustrated by certain observational exceptions. An example is known as the “blue-purple homeostasis” problem, and Moroney “Assessing hue constitutive using gradients” Color Imaging: Device-Independent Color, Color Hard copy, and Graphic Vir. As described in detail in Marcu, Editors, Proceedings of the SPIE, Volume 3963, pp. 294-300 (2000), the color of the blue hue is when the lightness shifts from dark to bright. Does not follow a completely linear path. CIELAB's more specific color constancy exceptions and related color spaces are also described in Braun et al., 1998, “Color Gamut Mapping in a Hue-Linearized CIELAB Color Space”, IS & T / SID6 ^th Color Imaging Con, Carefully measured and mapped as taught on pages 163-168, it is more popular than just a blue-purple homeostasis problem. Specifically, a commonly observed form of this problem is known as the Bezold-Brucke (u-umrant) shift, where the apparent hue changes with luminance (and vice versa), and this effect is Among other applications, efforts have been frustrated to find perceptually efficient color representations for video and image transmission.

There have been a number of attempts to develop or modify CIELAB and related color space representations, for example, Takamura and Kobayashi, 2002, “Practical extension to teach alternating transform matrix coefficients for CIELV that improve perceptual linearity. CIELUV color space to improve uniformity "IEE ICIP002, and Behrens" ^{Deficiencies of the CIE-L * a} * b * color space and introduction of the SRLAB2 color model "," www.magnetkern.de/srlab2.tex "(hereinafter, the SRLAB2 Good, incorporated here by reference). SRLAB2 proposes an overall new color space representation using the CIECAM02 color adaptation model, which is blue-purple (color) constancy for reduced hue angle uniformity, especially in the skin color region of the color space, There is a trade-off between hue-lightness interval length uniformity.

  Clearly, there is no completely perceptually uniform color space that exhibits all of the properties of brightness or brightness uniformity, color constancy, hue angle uniformity, and hue-lightness interval length uniformity. Are all required for the ideal color space representation for video and image coding and transmission.

  Since the introduction of the YCbCr and Y'CbCr color spaces, most video coding systems now use the YCbCr color space mainly for simplicity and practicality of implementation and then for reasons of backward compatibility. However, it is standardized in response to the use of more complex but more perceptually uniform color spaces and color representations based on perceptual differences rather than recent ones.

  The inefficiency of using YCbCr as the basis of color space for video compression extends not only to color symbol representation but also to the realization of luminance. For example, CIELAB introduces a non-exponential perceptual curve that is non-linear not only for color but also for lightness perception, which is a simplified exponential gamma function in many image and video coding and display schemes. Anything other than that by is not considered. As taught by CIELAB, actual observer measurements have proven that simple exponential or logarithmic relationships are not sufficient to represent perceptual luminance differences, especially in regions of low luminance range.

  Furthermore, many current video and image coding schemes, such as the HEVC video coding standard proposed at the time of this disclosure, despite the disadvantages such as reduced reconstruction quality and wasted code rate. YCbCr continues to be used as the basis of color space.

  The color space fundamentals of current video coding schemes and the disadvantages of choosing symbolic representations are well known in the art, and several attempts have been made to correct or at least mitigate the negative effects of these codec inefficiencies. Has been made. Moroney and Fairchild, "Color space selection for JPEG image compression", Journal of Electrical Imaging 4 (4), 373-381 (October 1995), and Drukarev, "recomp-splay". Early attempts to use a perceptually uniform color space representation as the basis for image and video compression, such as those taught in the volume, are their applications for real-time encoding and decoding complicated? Or frustrated by their lack of benefits. Also, the CIE 1931XYZ color space widely used in the related art is recognized as a better approximation of human visual system sensitivity to color differences when compared to RGB, YIQ or YCbCr, but both are incorporated herein by reference. , Charles, “A Technical Introduction to Digital Video”, John Wiley & Sons, 1996 and accompanying “www.poynton.com/ColorFAQ.html”. Thus, this also becomes significantly non-uniform with ratios as high as 80: 1. Although CIELAB improves this ratio to about 6: 1, Poynton points out that at the time of his writing, the CIELAB transform is computationally very expensive for video and not suitable for real-time processing. . Uniformity is a key concept for realizing many efficient encodings, but even CIELAB is far from a perceptually uniform ideal color space, as shown in the prior art.

  Other methods, such as those described in US Patent Application Publication US2012 / 0314943A1 (hereinafter “Guerrero”), are applied to a color space optimized for a human visual system model, A quantization factor that reduces spatial entropy and redundancy, or (2) system space transfer bandwidth because it reduces color space entropy and redundancy but requires a table lookup for each pixel in the previous encoding stage Try to achieve the solution by using any of the color table lookup steps that increase the requirement. Both of these methods are effective in slightly reducing color space entropy, provided that a proper HVS model and a perceptually uniform color space are utilized. Although Guerrero uses CIE1931XYZ, which is not a perceptually uniform color space, and CIELAB is briefly referred to, CIELAB is not an ideal color space for this purpose for the reasons described above. . Most importantly, the method disclosed by Guerrero reduces the color entropy at the expense of increasing the spatial entropy, and most of the benefits of implementations using encoders based on standard DCT such as JPEG and MPEG. Attenuate. Although it may be possible to reduce both spatial and color space entropy and redundancy by combining the disclosed quantization factor or color table reference with a histogram compression function, decoding to expand the histogram Subsequent processing is required at the device end, and in-band or out-of-band communication is required to adjust such histogram companding parameters. Guerrero does not disclose these concepts and actually teaches the opposite. Also, as taught in U.S. Pat. No. 8,451,384, when histogram calculations are performed at high definition and real-time throughput rates, real-time 4K, 5K and 8K resolution images on mass-market computer systems or end-user devices are exponential functions. It becomes difficult.

  Such in-band or out-of-band communication methods for adjusting the pre-filtering and post-filtering of the encoder and decoder, respectively, are described in the art as described in US Pat. No. 6,195,394 (hereinafter referred to as the '394 patent). It is well known. The pre-filtering and post-filtering processes described in the '394 patent relate to spatial bandwidth reduction and subsequent recovery, not perceptual color bandwidth reduction and recovery, but the' 394 patent Demonstrate that out-of-band communication methods for signaling the presence and configuration of pre- and post-filtering processes are required to ensure proper reconstruction of images and video in the vicinity of the decoder. Yes.

  What is needed but not provided does not place undue burden on processing complexity or memory transfer bandwidth requirements on either the encoder or decoder device or system, and does not provide existing images and video A high throughput system and method that improves the perceived quality and / or transmission or storage efficiency of the current compression or transmission system and method. The system and method does not impose the requirement to replace the encoder or decoder. The system and method will signal the presence and configuration of the system and method by synchronizing the pre-filtering of the encoder and the post-filtering of the decoder.

  The above problems are addressed and technical solutions are achieved in the art by providing an encoding method and system. The processing device receives the image buffer. The processing device converts one or more pixels of the image buffer from a native color space to one or more perceptually uniform color spaces. The processing device multiplies the lightness channel of one or more pixels of the image buffer by a first value. The processing device multiplies one or more pixels of one or more color channels by a second value. The processing device converts one or more pixels of the image buffer from one or more perceptually uniform color space to a native color space. The processing device transmits the image buffer to the downstream device.

  A perceptually uniform color space is a color space in which two colors that are at the same Cartesian distance in the color space are perceptually substantially equal to the human visual system.

  In one example, the first value may be a constant value, and the second value may be a constant value. In one example, the processing device transmits the first value and the second value to the second processing device. The processing device can receive the first value and the second value from one of the upstream devices or the memory buffer. The processing device can transmit the converted image buffer to the encoder, which encodes the image buffer before the transmitter sends the encoded image buffer to the downstream device.

  In one example, the processing device can receive one or more white point values. The processing device can convert the pixels of the image buffer from the intrinsic color space to one or more perceptually uniform color spaces based on the one or more white point values. The processing device can transmit one or more white point values to the downstream device.

  In one example, the first value, the second value, and one or more white point values are transmitted as metadata along with the image buffer to the downstream device in the band. In one example, the first value, the second value, and the one or more white point values are transmitted to the downstream device out of band as metadata along with the image buffer.

  In one example, converting the image buffer from the native color space to one or more perceptually uniform color spaces may be based on measuring hue. In one example, measuring the hue includes evaluating the hue of each pixel in the image buffer and selecting from one or more of a plurality of perceptually uniform color spaces using the measured hue. obtain.

  The above problems are addressed and technical solutions are achieved in the art by providing decoding methods and systems. The first processing device receives the image buffer and the first value from the second processing device. The first processing device converts one or more pixels of the image buffer from the intrinsic color space to one or more perceptually uniform color spaces. The first processing device multiplies the brightness channel of one or more pixels by the reciprocal of the first value. The first processing device receives the second value from the second processing device and multiplies one or more pixels of the one or more color channels by the reciprocal of the second value. The first processing device converts one or more pixels of the image buffer from one or more perceptually uniform color spaces to a native color space. The first processing device transmits the image buffer to the downstream device. The first value may be a constant value, and the second value may be a constant value.

  In one example, the first processing device receives one or more white point values from the second treatment device, and based on the one or more white point values, the pixels of the image buffer are one or more perceptually uniform colors. Convert from space to intrinsic color space. In one embodiment, the first value, the second value, and one or more white point values are received in-band from the second processing device along with the image buffer. In one example, the first value, the second value, and the one or more white point values are received out-of-band from the first processing device along with the image buffer.

  In one example, the unique color space associated with the first processing device is different from the unique color space associated with the second processing device. In one example, the one or more perceptually uniform color spaces associated with the first processing device are different from the one or more perceptually uniform color spaces associated with the second processing device.

  In one example, the downstream device may be one or more of a display, an encoder, or an image processing pipeline.

FIG. 1 is a block diagram illustrating an example of an encoded computing system in which examples of this disclosure may operate. FIG. 2 is a flow diagram illustrating an example of an encoding method associated with the encoding computing system of FIG. FIG. 3 is a block diagram illustrating an example of a decoding computing system in which each example of the present disclosure may operate. FIG. 4 is a flow diagram illustrating an example of a decoding method associated with the encoded computing system of FIG. FIG. 5 shows the YCbCr total area volume plotted in the CIELAB color space. FIG. 6 shows three cross sections of the YCbCr color space normalized with Y = 0, Y = 0.5 and Y = 1. FIG. 7 shows the cross section of FIG. 6 converted to CIELAB color space after quantization to 8-bit precision. FIG. 8 shows the cross section of FIG. 7 converted to the CIELAB color space and shows the quantized contour boundaries. FIG. 9 shows a cross section of the normalized CIE 1931XYZ color space at Z = 1. FIG. 10 shows the cross section of FIG. 9 converted to CIELAB color after quantization to 8-bit precision. FIG. 11 shows the cross section of FIG. 8 converted to CIELAB color space, quantized to show quantized contour boundaries. FIG. 12 is a block diagram of an embodiment of the computing system of FIGS. 1 and 3 operating in conjunction. FIG. 13 is a block diagram of a video transmission system in the related art. FIG. 14 is a block diagram of a video transmission system that reduces perceptual color entropy in the related art. FIG. 15 illustrates the disadvantage of applying a quantization function to color space to reduce color entropy, where color entropy is replaced with space entropy. FIG. 16 illustrates a schematic representation of a machine in an exemplary form of a computer system in which a set of instructions for causing the machine to perform any one or more of the methods described herein may be executed.

  Embodiments of the present disclosure provide that existing image and video transceivers or codec systems and methods allow brightness and resolution without adding excessive processing complexity or excessive memory bandwidth requirements to the encoder or decoder. High throughput systems and methods are provided that make it possible to take advantage of the latest and future discoveries regarding the perception of color in the human visual system. Embodiments of the present disclosure provide additional perceptual quality or transmission coding rate or storage size efficiency without requiring replacement or modification of the encoder or decoder. A method for synchronizing encoder pre-filtering and decoder post-filtering signaling the presence and configuration of an exemplary system of the present disclosure is described.

  More particularly, a high-throughput method and system for improving the perceived quality and / or efficiency of image and video compression methods is disclosed. A system and method according to embodiments of the present disclosure includes (1) converting one or more of each pixel of an input image or video into one or more perceptually uniform color and lightness representations, and (2) the lightness of each pixel. Multiplying the channel by a first value and multiplying each color channel of each pixel by a second value; (3) inversely transforming each pixel of the image into a color and luminance representation suitable for a compression encoder; Or encoding the video, and (4) transmitting or storing the first and second non-zero values with the image or video data as in-band or out-of-band metadata.

  A system and method according to embodiments of the present disclosure includes (5) reading or decoding a received image or video with received metadata, and (6) one or more perceptually uniform pixels of the decoded image or video. Converting to color and lightness representations, (7) multiplying each pixel in the lightness channel by the reciprocal of the first value, multiplying each pixel in the color channel by the reciprocal of the second value, and (8) display Performs the function of inverse transforming the image into a suitable color and brightness representation.

  In the following description, numerous details are described. However, it should be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the embodiments of the present disclosure.

  FIG. 1 is a block diagram of an example encoded computing system 100 that improves the perceptual quality and / or efficiency of image and video compression methods in which examples of this disclosure may operate. By way of non-limiting example, computing system 100 receives data from one or more data sources 105, such as a video camera, online storage device, or transmission medium. The computing system 100 may also include a digital video or image imaging system 110 and a computing platform 115. The digital video or image imaging system 110 processes one or more images, digital video streams, etc., and converts the analog video to digital video into a format that can be processed by the computing platform 115 as one or more data sources 105. . The computing platform 115 comprises a host system 120 that may comprise a processing device 125, such as one or more central processing devices 130a-130n, for example. Processing unit 125 is coupled to host memory 135. The host memory 135 can store digital image or video data received from one or more data sources 105 in the image data buffer 150.

  The processing device may further implement a graphics processing device 140 (GPU). Those skilled in the art will recognize that other cooperating processor architectures may be utilized in addition to the GPU, such as, but not limited to, DPS, FPGA or ASIC, or the attached fixed functional configuration of the processing unit 125 itself. . Those skilled in the art will also appreciate that the GPU 140 can be located on the same physical chip or logical unit as the central processing units 130a-130n, also known as "APU" as found on mobile phones and tablets. Individual GPU and CPU functions are found in computer server systems where the GPU is a physical expansion card, personal computer system and laptop. The GPU 140 may include a GPU memory 137. One skilled in the art will appreciate that host memory 135 and GPU memory 137 may also be located on the same physical chip or logical device, such as on an APU.

  The processing device 125 is configured to implement a color space converter and processor 145 that receives data from the data source 105 and receives the image data buffer 150 (hereinafter referred to as “color space processor 145”). Is transferred to the GPU memory 137 as an image buffer 155. In one example, the processing unit 125 can implement the color processor 145 as a component of the GPU 140. The color space processor 145 is configured to convert one or more of the pixels of the image data buffer 155 from a native color space to one or more perceptually uniform color spaces.

  The color space processor 145 is configured to multiply the lightness channel of each pixel of the converted image buffer 155 by a first value and multiply the color channel of each pixel of the converted image buffer 155 by a second value. . The color space processor 145 further converts one or more pixels of the image buffer 155 from one or more perceptually uniform color spaces into an original native input color space or a color space suitable for display output or transmission to the encoder 180. Configured to do. In one example, the converted image data can be displayed on the display 170. In other examples, the color space processor 145 can transmit the converted image data to the encoder 180. In one example, the encoder 180 can encode the transformed image data using encoding methods known in the art. The encoder 180 conveys the encoded data to the transmitter 185, which transmits the encoded data to one or more downstream devices 190 directly or via the network 195. In one example, one or both of encoder 180 or transmitter 185 may be external to processing device 125 or computing platform 115. In other examples, one or both of encoder 180 or transmitter 185 may be integrated into processing unit 125 or computing platform 115.

  FIG. 2 is a flow diagram illustrating an example of an encoding method 200 that improves the perceptual quality and / or efficiency of existing image and video compression methods. The method 200 is performed by the computer system 100 of FIG. 1 and may comprise hardware (eg, circuitry, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or a combination thereof. . In one example, the method 200 is performed primarily by the color space processor 145 of the computing system 100 of FIG.

  As shown in FIG. 2, the color space processor 145 receives the image buffer 155 at block 210 to cause the computing system 100 to encode the image data. At block 220, the color space processor 145 converts one or more pixels of the image buffer 155 from the native color space to one or more perceptually uniform color spaces. A perceptually uniform color space is a color space in which two colors that are at the same Cartesian distance in the color space are perceptually substantially equal to the human visual system.

  One or more pixels of the converted image buffer 155 may consist of a plurality of pixels, each pixel having a brightness channel and one or more color channels. At block 230, the color space processor 145 multiplies the lightness channel of one or more pixels of the converted image buffer 155 by a first value. In one example, the first value is a constant value. At block 240, the color space processor 145 multiplies the one or more color channels of the one or more pixels of the converted image buffer 155 by a second value. In one example, the second value is a constant value.

  At block 250, the color space processor 145 converts one or more pixels of the image buffer 155 from one or more perceptually uniform color spaces to a native color space. In block 260, the color space processor 145 transmits the image buffer 155, the first value and the second value to the encoder 180 or multiplexer or wrapper. The wrapper used here is a term used in the meaning of the SMPTE definition of the video file format, “essence” describes the codec payload and its contents, and “wrapper” is for file format or transmission format or essence. Describe other payload packages. As will be appreciated by those skilled in the art, a wrapper may also be interpreted to mean a file format for a single image, such as JFIF, which describes a standard mode JPEG image encapsulated in a file. To do. In block 270, the encoder 180 encodes the image buffer 155. In block 280, encoder 180 transmits image buffer 155 to transmitter 185. In block 290, the transmitter 185 transmits the image buffer 155 to one or more of the display 170 or one or more downstream devices 190. In one example, the transmitter 185 transmits the image buffer 155 to one or more downstream devices 190 (eg, a second processing device) via the network 195.

In one example, the color space processor 145 further includes: “ISO 11664-4: 2008 (E) / CIE S 014-4 / E: 2007: Joint ISO / CIE Standard: Colorimetry-Part 4: CIE 1976 L ^* a ^* b ^* One or more reference white point tristimulus values as described in "Color Space" (hereinafter referred to as CIELAB, incorporated herein by reference) can be received. The color space processor 145 can convert one or more pixels of the image buffer from the intrinsic color space to one or more perceptually uniform color spaces based on the one or more white point values. The transmitter can further transmit one or more white point values to one or more downstream devices 190. The first value, the second value, and one or more white point values may be transmitted to the one or more downstream devices 190 in-band or out-of-band with the image buffer 155.

  FIG. 3 is a block diagram of an exemplary decoding computing system 300 that improves the perceptual quality and / or efficiency of image and video compression methods in which examples of this disclosure may operate. By way of non-limiting example, computing system 300 is configured to receive encoded data from one or more data sources 305. The one or more data sources 305 may be the encoded computing system 100 of FIG. Computing system 300 may also include a computing platform 315. The computing platform 315 comprises a host system 320 that may comprise a processing unit 325, such as one or more central processing units 330a-330n, for example. Processing unit 325 is coupled to host memory 335. Host memory 335 can store encoded digital image or video data received from one or more data sources 305 in image data buffer 350. The encoded data is received by the receiver 360, decoded by the decoder 365, and sent to the image data buffer 350. Receiver 360 can receive encoded data directed from one or more data sources 305 or via network 310. In one example, one or both of receiver 360 or decoder 365 may be external to processing device 325 or computing platform 315. In other examples, one or both of receiver 360 or decoder 365 may be integrated into processing unit 325 or computing platform 315.

  The processing device 325 can further implement a graphics processing device 340 (GPU). Those skilled in the art will appreciate that other cooperating processor architectures may be utilized in addition to the GPU, such as, but not limited to, DPS, FPGA or ASIC, or the attached fixed function configuration of the processing unit 325 itself. . One skilled in the art will also appreciate that the GPU 340 can be located on the same physical chip or logical unit as the central processing units 330a-330n, also known as "APU" as found on mobile phones and tablets. Individual GPU and CPU functions can be found in computer server systems where the GPU is a physical expansion card, personal computer system and laptop. The GPU 340 may include a GPU memory 337. One skilled in the art will appreciate that host memory 335 and GPU memory 337 may also be located on the same physical chip or logical device, such as on an APU. It should also be understood by those skilled in the art that the decoding processor 325 may be partially or wholly integrated with the encoding processor 125 in the computing system 100 of FIG. 1 to provide both encoding and decoding functions. Should be understood.

  The processing device 325 receives the encoded image data, the first value and the second value (for example, the first value and the second value used in the encoding system 100 of FIG. 1) from the data source 305. 360 for receiving. The processing device 325 transfers the first value and the second value, generates the image data buffer 350 based on the received encoded image data for the decoder 365, the image buffer 350, the first value and the second value. Configured to decode the value of. The decoder 365 is configured to transfer the image buffer 350, the first value, and the second value that is transferred to the GPU memory 337 as the image buffer 355.

  The processing unit 325 implements an image buffer 355, a color space converter and processor 345 (hereinafter referred to as "color space processor 345") that receives the first and second values from the decoder, demultiplexer or unwrapper. Configured. In one example, the processing device 325 can implement the color processor 345 as a component of the GPU 340.

  Color space processor 345 is configured to convert one or more pixels of image buffer 355 from a native color space to one or more perceptually uniform color spaces. The color space processor 345 multiplies the lightness channel of one or more pixels of the converted image buffer by the reciprocal of the first value and the color channel of one or more pixels of the converted image buffer by the reciprocal of the second value. Configured to multiply. The color space processor 345 further moves one or more pixels of the image buffer 355 from one or more perceptually uniform color spaces to the original native input color space, or the display output of the display 370 or one or more downstream devices 375 ( For example, it is configured to convert to a color space suitable for transmission to an encoder.

  FIG. 4 is a flow diagram illustrating an example of a decoding method 400 for improving the perceptual quality and / or efficiency of existing image and video compression methods. The method 400 is performed by the computer system 300 of FIG. 3 and may comprise hardware (eg, circuitry, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or a combination thereof. . In one example, the method 300 is performed primarily by the color space processor 345 of the computing system 300 of FIG.

  As shown in FIG. 4, to cause the computing system 300 to decode the image data, at block 410, the receiver 360 receives the encoded image data, the first value and the second value from the data source 305. . At block 420, the decoder 365 decodes the image buffer 355, the first value and the second value, and places the decoded data into the image data buffer 350 of the processing device 325. At block 430, the processing unit 325 transfers the decoded image buffer 355, the first value and the second value to the GPU image buffer 355 of the GPU memory 337. At block 440, the color space processor 345 receives the image buffer 355, the first value and the second value from the GPU memory 337.

  The converted image buffer 355 may consist of a plurality of pixels, each pixel having a brightness channel and one or more color channels. At block 450, the color space processor 345 converts one or more pixels of the image buffer 355 from the native color space to one or more perceptually uniform color spaces. At block 460, the color space processor 345 multiplies the lightness channel of one or more pixels of the transformed image buffer 355 by a reciprocal of the first value. In one example, the first value is a constant value. At block 470, the color space processor 345 multiplies one or more color channels of one or more pixels of the transformed image buffer 355 by a reciprocal of the second value. In one example, the second value is a constant value.

  At block 480, the color space processor 345 moves the image buffer 355 from one or more perceptually uniform color spaces to a native color space, or a display output on the display 370 or one or more downstream devices 375 (eg, network 310 To a color space suitable for transmission to an encoder). At block 390, color space processor 345 outputs the image buffer to display 370 or downstream device 375.

  In one example, the color space processor 345 can further receive one or more white point values from the receiver 360. The color space processor 345 can use one or more white point values in converting one or more pixels of the image buffer from one or more perceptually uniform color spaces to a native color space. In one example, a first value, a second value, and one or more white point values are received from data source 305 (eg, computing system 100 of FIG. 1) with image buffer 255 in-band or out-of-band.

  In one example, the native color space associated with processing device 325 may be different from the native color space associated with data source 305 (eg, processing device 125 of FIG. 1). In one example, the perceptually uniform color space associated with processing device 325 may be different from the perceptually uniform color space associated with data source 305 (eg, processing device 125 of FIG. 1).

  Embodiments of the present disclosure provide better perceived quality or reduced encoding by utilizing the latest or future discoveries in perceptual color science while retaining compatibility with compression based on YCbCr as the transmission medium and color space. It is operable to provide a high-throughput efficient system and method for achieving rates in existing suitable YCbCr based compression systems. As a result, entropy and redundancy can be reduced in both the luminance and chrominance channels without requiring a direct modification to either the encoder or decoder or the transmitter or receiver.

  The YCbCr color space has great perceptual redundancy, especially at the upper and lower boundaries of the color space volume along the luminance or Y-channel axis. It has long been recognized that an ideal image or video compression system should select a perceptually uniform color space as the basis for symbolic representation. As used herein, perceptual uniformity means that two colors that are at the same Cartesian distance in the color space volume are also perceptually equally separated.

  FIG. 5 is a three-dimensional projection rendering of the CIELAB color space, where the boundaries of the YCbCr color space are outlined in wireframe shape. The CIELAB color space is a much larger color space than the YCbCr color space. Note that the volume of the entire YCbCr section within the CIELAB color space has an irregular shape. The volume of the entire area of the YCbCr color space within the CIELAB color space is shown “stretched”. These stretched regions represent regions of volume such that YCbCr based coding and compression systems devote “excessive” coding rates to representing perceived color differences relative to other colors. If the YCbCr color space was exactly perceptually uniform for CIELAN, the volume of the entire area would be a cube. In other words, the reduced area represents a volume area where the encoding system based on YCbCr devotes an “underly” encoding rate to representing perceived color differences relative to other colors.

  FIG. 6 shows three cross sections of the normalized YCbCr color space at Y = 0, Y = 0.5 and Y = 1.

  FIG. 7 shows the cross section of FIG. 6 converted to CIELAB color space, which has a more uniform approximation of HVS perceptual sensitivity, which is quantization to 8-bit precision. Depending on the color of the original YCbCr space, it occupies more area than other colors.

  FIG. 8 shows the quantized contour boundary of FIG. The region of FIG. 7 shows a non-uniform distribution, and the large region shows a “waste” encoding rate whenever a video or image encoding or compression system uses the YCbCr color space. Assuming that the YCbCr color space was completely efficient, the areas shown would all have the same approximate area. In FIG. 8, wasted encoding rates are concentrated along the Cb axis for low luminance values where Cb is smaller than 0, and concentrated along the Cb axis for high luminance values where Cb is larger than 0.

  JPEG, MPEG / H. 262 / H. 263, AVC / H. H.264 and HEVC / H. In examples of systems based on YCbCr, such as H.265, it was discovered that the problem does not only extend to the color channel—the luminance assumption of YCbCr does not exactly match the HVS sensitivity to lightness. For this reason, the Y channel in the YCbCr color space also has redundancy that can contribute to artifacts and / or transmission inefficiencies, especially in the dark areas of the luminance range.

  More specifically, artifacts are seen when viewing a decoded high saturation dark scene, such as a concert video that tends to have high contrast and bright colored light. In such situations, color banding or posterization artifacts are particularly apparent in standard coding systems, especially in situations where the coding rate is limited or in transmission network congestion. These quality defects can be prevented or mitigated when the examples of this disclosure are used in the context of YCbCr-based encoding or compression systems.

  The examples of this disclosure provide a high-throughput efficient system and method for achieving better perceived quality or reduced code rate in existing suitable CIE 1931XYZ based compression systems. The example of this disclosure uses the latest or future discoveries in perceptual color science, while retaining the CIE 1931XYZ color space as an assumption of transmission media and color space, either the encoder or decoder or transmitter or receiver. The entropy and redundancy in the luminance channel can be reduced without requiring direct modification.

  In examples of systems based on CIE 1931XYZ, such as JPEG2000, the problem becomes more serious with the assumption of brightness that CIE 1931XYZ is not an exact representation known to match HVS sensitivity to lightness. For this reason, the Y channel in the CIE 1931 XYZ color space has redundancy that can contribute to artifacts and / or transmission inefficiencies.

  FIG. 9 shows a cross section of the CIE 1931 XYZ space normalized at Z = 1.

  FIG. 10 shows the cross section of FIG. 9 converted to the CIELAB color space, showing a more uniform approximation of HVS perceptual sensitivity quantized to 8-bit accuracy. It can be seen that depending on the color of the original CIE 1931 XYZ space, it occupies a larger area than other colors.

  FIG. 11 shows the quantized contour boundary of FIG. The region of FIG. 10 shows a non-uniform distribution, and the large region shows a “waste” encoding rate whenever the video or image encoding or compression system uses the CIE 1931XYZ color space. Assuming that the CIE 1931XYZ color space was completely efficient, the regions shown would all have the same approximate region.

  As a practical matter, this means that even modern JPEG 2000 based systems such as digital cinema packages can be made more efficient without significant perceptual differences.

  Furthermore, embodiments of the present disclosure can provide high throughput systems and methods that can take advantage of future improvements and standards for color representation as more accurate data, and perceive HVS color and lightness. As well as processing standards. Future color space representations such as those issued by the CIE can add complexity, but are perceptually more uniform. Embodiments of the present disclosure rely only on the resulting perceptually uniform color space transformation (and vice versa) that is uniform with respect to the difference in HVS perception, where the system and method is a color lookup table or any The embodiment described herein as being valid is possible because it does not depend on the non-linear arithmetic processing.

  FIG. 12 is a block diagram of an embodiment 1200 of the computing systems 100 and 300 of FIGS. 1 and 3 operating in conjunction. 1 and 3 embodiment 1200 of computer systems 100 and 300 includes hardware (eg, circuitry, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or combinations thereof. Prepare.

As shown in FIG. 12, the original digital image or video input 1205 is input to a first color space conversion process 1210. The first color space conversion processing 1205 is, as a non-limiting example, from a specific color space such as YCbCr color space or R′G′B ′ color space to a perceptually linear color space such as CIELAB, CIELV, or SRLAB2. Convert to The conversion of the R′G′B ′ color space to the L ^* a ^* b ^* color space is expressed by Equation 1:
Is a two-step process including a first conversion to the CIE 1931XYZ color space as shown in FIG.

The next step is Equation 2:
_{^{L * = 116f (Y / Y}} n) -16
_{^{a * = 500 [f (X}} / X n) -f (Y / Y n)]
_{^{b * = 200 [f (Y}} / Y n) -f (Z / Z n)]
Assuming that there are a set of three assumed reference white point tristimulus values as shown in FIG. 4, the color space is converted into the L ^* a ^* b ^* color space. Here, the normalized relative luminance tristimulus value for the CIE standard D65 white point used in Equation 2 is Equation 3:
_{X n = 95.047, Y n =} 1000, Z n = 108.883
Shown in Here, the function f () of Expression 2 is expressed by Expression 4:
If t> (6/29) ³ , then f (t) = t ^1/3
Otherwise, f (t) = (1/3) (29/6) ² +4/29
Defined by

Numerous methods can be used to validate the direct conversion from the native color space format to the L ^* a ^* b ^* color space, and the above equation is a non-limiting example and is convenient and clear for explanation. Those skilled in the art will recognize that this is provided. Furthermore, those skilled in the art will recognize that the selection of the white spot is affected by the lighting, the starting camera or imaging system configuration data, and can vary depending on the input content and its associated colorimetric metadata. Should do.

Next, each channel of each pixel of the digital image is converted by the brightness and color pre-processor 1215 into Equation 5:
_Is multiplied by the values D _L and D _ab provided by the two users. The standard value of _{D L} is between 0.8 and _0.95, the standard value of _{D ab} is between 0.5 and 0.8.

Next, the correction values L ₁ ^* , a ₁ ^*, and b ₁ ^* are converted back to the original inherent color space such as the YCbCr color space or the R′G′B ′ color space by the first reverse color conversion processing 1220. Is done. This is usually a two-step process where the first step is Equation 6:
As shown in FIG. 4, the conversion is inverse to the CIE 1931XYZ color space.
Here, f ⁻¹ (t) is expressed by Equation 7:
When t> 6/29, f ⁻¹ (t) = t ³
Otherwise, f ⁻¹ (t) = 3 (6/29) ² (t−4 / 29)
Given in.

Then, the CIE 1931XYZ image buffer is converted back to the original inherent color space, and in the case of the R′G′B ′ color space, Equation 8:
Given in.

Again, a number of methods can be used to enable direct conversion from the L ^* a ^* b ^* color space to the native color space format, and the above equation is a non-limiting example, Those skilled in the art will appreciate that this is provided for convenience and clarity of explanation.

As noted above, attempts have been made to achieve a “sufficiently good” approximation, but there is no completely perceptually uniform color space representation. All modern color space representations such as CIELAB, CIECAM02 and SRLAB2 have certain uniformity tradeoffs such as hue uniformity or hue linearity along the dimension of brightness or brightness. As a result of these imperfections, the magnitude of the values of D _L and D _ab are limited before visible artifacts become apparent on display 370. This limitation has a direct impact on the coding efficiency gain that can be achieved since the smaller the magnitudes of D _L and D _ab, the less entropy the encoder (and decoder) needs to encode.

Another example of a color space processor 345 and 145 is that this efficiency is evaluated by evaluating the hue of each pixel in the image data buffer inputs 355 and 155 and using this hue value, between magenta, red, yellow and green. Improved by determining whether to convert a given pixel using CIELAB with good hue uniformity characteristics for values of or SRLAB2 with good hue uniformity characteristics for values between green, cyan, blue and magenta To do. The hue described here is from RGB, for example, Equation 9:
Maximum RGB = Maximum (R, G, B)
Minimum RGB = Minimum (R, G, B)
Color difference = Maximum RGB-Minimum RGB
When Color Difference = 0, Hue = None When Maximum RGB = R, Hue = (GB) / Color Difference mod6
In the case of maximum RGB = G, hue = (BR) / color difference + 2
In the case of maximum RGB = B, hue = (RG) / color difference + 4
Calculated by

In one example, if the previously calculated hue is greater than a first hue cutoff point, eg, 3.0 (green in the numerical representation of the hue above), or the hue is a second hue cutoff point, eg , 0.0 (magenta), alternate color space conversion can be applied. As a non-limiting example, these cutoff points represent the green-cyan-blue range according to Equation 9 for RGB sources, which is a color representation that has been tried many times by CIELAB in terms of color constancy. , SRLAB2 can be used as an alternate color conversion variant. Alternatively, CIELAB can be utilized within a color space conversion. As a non-limiting example, one of ordinary skill in the art will recognize that other approximate perceptually uniform color transformations such as CIELV, HunterLAB, or CIECAM02 can be substituted. Still further, the choice of color space, such as CIELAB, CIEUV, CIECAM02, SRLAB2, or a future representation of a substantially perceptually uniform color space for one or more of the aforementioned hue ranges, is valid for each As long as a hue range is selected and each color space representation is perceptually uniform within the utilized hue range, it is not important in the effective function of this example. By utilizing this method, the previous minimum values of D _L and D _ab are 0.8 and 0.5, with the attendant benefit to encoder efficiency, without any visible degradation of the decoded and rendered quality. To 0.5 and 0.11 respectively. The hue determination process is local to both the sending color processor 1215 and the receiving color processor 1255. Whether a given pixel has been processed by a given color space is a 4-channel image between color space transformation 1210 and inverse color space transformation 1220 or between color space transformation 1250 and inverse color space transformation 1260. Can be communicated by out-of-band or in-band means, such as through the alpha channel of the buffer or a separate data buffer, or through any other means suitable for carrying information for each pixel for which color conversion is utilized . It should be noted that as long as color constancy is observed with respect to the color space conversion processes 1210 and 1260, the color space determination information need not be carried along the communication channel only locally.

First inverse color conversion processing 1220, the modified image data is output to the encoder 1225, _D values of _L and _{D ab,} and selectively, the white point is utilized by the first color conversion processing 1210 X _n , Y _n and Z _n are carried directly to the encoder 1225 or to the transmitter / multiplexer / writer 1230. One skilled in the art will appreciate that in many cases, the file format encapsulation or stream multiplexing process may be located in the encoder 1225. In addition, these values are communicated to the downstream receiver / demultiplexer / reader process 1240, where the encoder 1225, decoder 1245, transmitter / multiplexer / writer 1230 or receiver / demultiplexer / reader 1240 It can be conveyed to encoder 1225 or transmitter / multiplexer / writer 1230 so that it does not require a custom implementation of processing. As is well known in the art, file and stream formats provide various ways to accomplish this.

As a non-limiting example, the JPEG file exchange format (hereinafter referred to as JFIF) is commonly used to encapsulate JPEG compressed images, and JFIF is a well-known meta-data in a number of formats including EXIF, ICC profile, and Picture Info. defining a data expansion can be used to carry information about the presence and value as well as selective X _n, Y _n and Z _n value of these both D _L and D _ab.

As a non-limiting example, H264 encoded bitstream supports containing optical supplemental reinforcement Information (SEI) _{header D L} and _{D ab} and selective _X n, _{Y n} and _{Z n} value may be embedded . These methods do not require changes to the encoder (or decoder) processes 1225 and 1245, but output these values to either the encoder 1225 or the transmitter / multiplexer / writer 1230, associated with them. Only the ability to query the downstream decoder 1245 or receiver / demultiplexer / reader 1240 for the value of.

  Other ways to carry metadata are not tied to the encoder or codec format at all, but to the transport stream itself. As a non-limiting example, these include an MPEG transport stream ES descriptor that may also be used via an RTP extension header for the purposes described herein.

Numerous transport methods are possible for D _L and D _ab and optionally X _n , Y _n and Z _n values, and the desired property required for the present invention is the independence of the encoder and decoder implementations And access to these carrier metadata via APIs or other methods by processes external to the encoders, decoders, transmitters, receivers, multiplexers, demultiplexers, readers and writers described above. The contractor should understand. This also, D _L and D _ab value and selectively X _n, transporting of a fully out-of-band due to standardization of a particular encoder and decoder in accordance with the Y _n and Z _n values, or non-limiting Examples may include communication of these values over other channels and contexts, such as a manifest file of an MPEG-DASH or HLS stream or a separate file or protocol.

  Along with the compressed image or video data itself, the metadata decapsulates the image or video data from the upstream transmitter / multiplexer / writer 1230 via the transmission channel 1235 or storage medium into a format suitable for the decoder 1245. The data is selectively transmitted or communicated to a downstream receiver / demultiplexer / reader 1240 that outputs the data to the second color space conversion process 1250. The second color space conversion processing 1250 converts the intrinsic color space to one or more perceptually uniform color spaces in the same manner as the first color space conversion processing 1205 described in Equations 1 to 4. As a non-limiting example, optionally, a particular perceptually uniform color space is selected for each pixel by Equation 9. Those skilled in the art will appreciate that the eigencolor space provided by the decoder 1245 may be different from that provided to the encoder 1225 and requires a different conversion process. The perceptually uniform color space utilized by the first lightness and color pre-processor 1215 may be different from the one or more perceptually uniform color spaces utilized by the decoder 1245 and is required. It can further be seen that the only requirement is that each is substantially perceptually uniform.

The converted image data is output to the brightness and color post-processor 1255. In addition, either the receiver / demultiplexer / reader 1240 or the decoder 1245 may selectively transmit D _L and D _ab and optionally to the lightness and color post-processor 1255 by either the metadata communication method described above or a corresponding method. X _n, and supplies the _{Y n} and _{Z n} value.

The lightness and color post-processor 1255 is represented by Equation 10:
In this way, each channel of each pixel of the digital image is multiplied by the two supplied values D _L and D _ab .

After applying the reciprocal multiplication operation, the brightness and color post-processor 1255 outputs the L ^* a ^* b ^* color space value to the second inverse color conversion process 1260.

  Although the perceptually uniform color space utilized by the second inverse color conversion process 1260 may be different, the second inverse color conversion process 1260 is a first inverse conversion process 1220 as shown by equations 5-8. The output unique color space will be suitable for later transcoders or displays and may be different from the unique color space supplied as input to the first color conversion process 1210. This image data is, as a non-limiting example, output 1265 to a display, a subsequent encoder, or image processing pipeline.

  The present invention has several advantages over prior art methods that enhance the perceptual quality and encoding efficiency of existing image and video compression systems.

  FIG. 13 shows a standard encoder and decoder chain, which becomes inefficient due to the sub-optimal selection of the color space and symbol representation described above.

  FIG. 14 illustrates a prior art attempt according to Guerrero to address the aforementioned inefficiencies. As shown in FIG. 14, the original digital image or video input is input to the color space conversion process. In this color space conversion process, the intrinsic color space such as the YCbCr color space or the R′G′B ′ color space is converted into a color space modeled in accordance with the HVS such as the CIE 1931XYZ color space or the CIELAB color space. Then either the color lookup table or the equidistant quantization formula is applied to each color channel in the color space. Lightness, brightness or Y channel is not handled. As a result, this process does not take advantage of the great redundancy of the luminance or Y channel, and as is known to those skilled in the art, a number of simultaneous video and image compression systems and methods are compared to the chrominance channel. Thus, it is particularly important to eliminate redundancy in the luminance channel since it directs more computational complexity and channel transmission bandwidth.

  It is also assumed that the processing according to Guerrero is sufficient to perform equidistant quantization of the color channel for any color space modeled according to HVS, but is demonstrated by FIGS. 9, 10 and 11 and by Poynton. As has been done, the CIE 1931XYZ color space has a large non-uniformity. The CIE 1931XYZ color space is the most preferred of those disclosed to Guerrero, but the CIE 1931XYZ color space is (of course, all color spaces) based on the HVS model, while it is not perceptually uniform, so the equidistant quantization process Will produce visible artifacts in the resulting images and videos.

  To deal with this, Guerrero discloses a color mapping function, which effectively acts as a color lookup table (hereinafter referred to as “LUT”). There are two disadvantages of this method over the embodiments of the present disclosure. First, the implementation of the present disclosure for each combination of native input color space and perceptually modeled color space of the output HSV. The need for a separate LUT that the form does not require, and second, the use of the LUT brings memory transfer bandwidth costs to the encoding system, so that each pixel is executed accordingly with the reading, LUT and It must have additional indirect reference memory readings for subsequent write operations for at least two color channels, which again, but is not required by embodiments of the present disclosure. Guerrero discloses a color range that requires multiple polynomial solutions and comparison / branch operations when the LUT is replaced by an expression, which is performed iteratively for each pixel of the converted video stream. It is disclosed that it is not ideal considering the need to

  As a comparison, embodiments of the present disclosure require only a single value multiplication operation for each pixel of each channel of transformed image data.

  Guerrero expects the systems and methods disclosed herein to utilize a perceptually uniform color space, such as the CIELAB color space, instead of the CIE 1931 XYZ color space. However, the aforementioned disadvantages with respect to color mapping functions and / or LUT implementations apply. In the disclosed quantization function example, the advantage of doing so is that when used in any DCT-based coding scheme, the color entropy is simply replaced with spatial entropy, eliminating the total gain of coding rate efficiency. Be killed in

  FIG. 15 illustrates this, where the original image patch containing a smooth 8-bit gradation is quantized to 4 bits. Color entropy is reduced, but the newly generated boundary edges are sharply bounded, plus spatial entropy increases due to boundary condition errors, and spotty spatial features that represent significantly increased spatial energy and entropy Was brought. For DCT-based coding techniques, such as either JPEG or MPEG variants including HEVC, this will extend the extra entropy with DCT coefficients at higher frequencies. This can be mitigated with a custom encoder that anticipates this, but embodiments of the present disclosure can provide encoding and decoding quality and / or efficiency without requiring changes or replacement of the encoder and decoder implementation. Systems and methods for increasing the cost are provided.

  The spatial entropy problem described above is due to the proper use of histogram functions (such as those disclosed in the '384 patent) in the vicinity of both the encoder and the decoder when using the quantization function disclosed in Guerrero. Can be mitigated by mitigating the inherent features and problems of the process as disclosed in, but this problem is neither recognized nor mentioned and teaches the opposite of Guerrero. Embodiments of the present disclosure obviate the need for such processing.

  The pre-encoding process of the embodiment of the present disclosure is performed in the vicinity of real-time for 5K and 8K, and in the vicinity of real-time and real-time for 4K video resolution video at 30 fps on computer hardware that is mass-marketed in the real-time and simultaneous markets. It is efficient enough to run a large number of images and videos at various resolutions. The post-encoding process of the embodiment of the present disclosure includes a GPU, CPU or APU at full HD resolution for a single instance of video and images, such as feature phones, smartphones, tablets, laptops, PCs, set-top boxes and televisions. It is efficient enough to run on any end-user device used.

  This combination of efficiency both near the encoder and near the decoder according to embodiments of the present disclosure opens up new applications. These applications include, but are not limited to, real-time enhancement of video encoders for over-the-top video distribution, cost-effective real-time reduction of public radio access network congestion, video and images from mobile devices Increase real-time passband television distribution capabilities when uploading and downloading data, increase satellite transponder capabilities, reduce storage costs for content management systems and network DVR architectures, high throughput processing of images and video in distributed network cores Including.

  FIG. 16 illustrates a schematic representation of a machine in an exemplary form of a computer system 1600 in which a set of instructions for causing the machine to perform one or more of the methods described herein may be executed. In one example, the machine may be connected (eg, networked) to other machines in a LAN, intranet, extranet or the Internet. The machine can operate at the capabilities of the server machine in a client-server network environment. The machine executes a personal computer (PC), set top box (STB), server, network router, switch or bridge, or a set of instructions (sequential or other) that specify the action to be taken by the machine Any machine can be used. Also, although only a single machine is shown, the term “machine” may be used individually or in cooperation to execute a set (or sets) of instructions to perform any one or more of the methods described herein. It should also be interpreted as including any set of machines to execute.

  The exemplary computer system 1600 includes a processing unit (processor) 1602, a main memory 1604 (eg, dynamic random access memory (DRAM) such as read only memory (ROM), flash memory, synchronous DRAM (SDRAM)), static memory 1606 ( For example, flash memory, static random access memory (SRAM), and data storage device 1616, which communicate with each other via bus 1608.

  The processor 1602 represents one or more general-purpose processing devices such as a microprocessor, a central processing unit, and the like. More particularly, processor 1602 is a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or processor that executes other instruction sets. Alternatively, it may be a processor that executes a combination of instruction sets. The processor 1602 may also be one or more application specific processing devices such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, and the like. The color space processors 145 and 345 shown in FIGS. 1 and 3, respectively, can be performed by a processor 1602 that is configured to perform the operations and steps described herein.

  Computer system 1600 can further include a network interface device 1622. The computer system 1600 also includes a video display 1610 (eg, liquid crystal display (LCD) or cathode ray tube (CRT)), alphanumeric input device 1612 (eg, keyboard), cursor control device 1614 (eg, mouse), and signal generator. 1620 (eg, a speaker) may be included.

  The driver 1616 can include a computer readable medium 1624 that stores one or more sets of instructions (eg, instructions of the color space processors 145 and 345) that use any one or more of the methods or functions described herein. The instructions of color space processors 145 and 345 may also be fully or at least partially within main memory 1604 and / or during execution by computer system 1600, main memory 1604 and processor 1602 comprising a computer readable medium. May exist within 1602. The instructions of color space processors 145 and 345 may also be sent or received over the network via network interface device 1622.

  In one example, computer readable storage medium 1624 is shown as being a single medium, but the term “computer readable storage medium” refers to a single non-transitory medium or media that stores one or more sets of instructions. Non-transitory media (e.g., centralized or distributed databases and / or associated caches and servers). The term “computer-readable storage medium” also refers to any medium that can store, encode, or carry a set of instructions for execution by a machine and that causes the machine to perform any one or more of the disclosed methods. Should be construed as including. Thus, the term “computer-readable storage medium” should be interpreted as including, but not limited to, semiconductor memory, optical media, and magnetic media.

  In the above description, numerous details are set forth. However, it will be apparent to one skilled in the art having the benefit of this disclosure that the disclosed examples may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the embodiments of the present description.

  Some portions of the detailed description are presented in terms of algorithms and symbolic representations of processing on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to those skilled in the art. Here, and generally, the algorithm is considered a self-consistent sequence of steps leading to the desired result. Steps are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, or manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

  However, all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels attached to these quantities. As is apparent from the above, unless otherwise noted, throughout the description, the use of terms such as “receive”, “write”, “hold”, etc. refer to the computer system registers and Data represented as physical (eg, electronic) quantities in memory to other data that is similarly represented as physical quantities in computer system memory or registers or other such information storage, transmission or display devices The operation and processing of a computer system or similar electronic computing device that operates and converts.

  The disclosed example also relates to an apparatus for performing the process herein. This apparatus can be specially constructed for the required purposes, or can comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. The high-throughput systems and methods disclosed herein that improve the perceived quality and / or transmission or storage efficiency of existing image and video compression or transmission systems and methods are several examples, but over-the-top video Improving real-time efficiency of video encoder for distribution, cost-effective real-time reduction of public radio access network congestion, real-time passband TV distribution when uploading and downloading video and image data from mobile devices It solves problems in a number of areas, such as increased capacity, increased satellite transponder capacity, reduced storage costs for content management systems and network DVR architectures, and high-throughput processing of images and video in a distributed network core.

  Such computer programs include, but are not limited to, any type of disk including floppy disks, optical disks, CD-ROMs and magneto-optical disks, read only memory (ROM), random access memory (RAM), EPROM, It can be stored on a computer readable storage medium such as an EEPROM, a magnetic or optical card, or any type of medium suitable for storing electronic instructions.

  The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used in the program according to the teachings herein to construct a more specialized device to perform the necessary method steps. Exemplary structures for these various systems are apparent from the description herein. In addition, the present disclosure is not described with reference to any particular programming language. It should be appreciated that a variety of programming languages may be used to implement the teachings according to the disclosure described herein.

  It should be understood that the above description is illustrative and not restrictive. Many other examples will be apparent to those of skill in the art upon reading and understanding the above description. Accordingly, the scope of the disclosure should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims (27)

Receiving at the processing device an image buffer;
Converting one or more pixels of the image buffer from a native color space to one or more perceptually uniform color space using the processing device;
Multiplying a lightness channel of the one or more pixels of the image buffer by a first value using the processing device;
Using the processing device to convert the one or more pixels of the image buffer from the one or more perceptually uniform color spaces to the intrinsic color space; and transmitting the image buffer to a downstream device. A method comprising:   The one or more perceptually uniform color spaces are color spaces such that two colors at equal Cartesian distances in the color space are substantially equally separated from the human visual system. the method of.   The method of claim 1, further comprising multiplying one or more pixels of one or more color channels of the image buffer by a second value using the processing device.   4. The method of claim 3, further comprising transmitting the first value and the second value to the downstream device.   4. The method of claim 3, further comprising receiving at the processing device the first value and the second value from an upstream device or a memory buffer.   The method of claim 1, further comprising the step of transmitting the transformed image buffer to an encoder by the processing device, the encoder being operable to encode the image buffer. Receiving at least one white point value in the processing device;
Converting one or more pixels of the image buffer from the intrinsic color space to one or more perceptually uniform color spaces based on the one or more white point values; and The method of claim 3, further comprising the step of transmitting to the downstream device.   The method of claim 7, wherein the first value, the second value, and the one or more white point values are transmitted to the downstream device in-band with the image buffer.   The method of claim 7, wherein the first value, the second value, and the one or more white point values are transmitted out of band with the image buffer to the downstream device.   Converting one or more pixels of the image buffer from a native color space to one or more perceptually uniform color spaces, evaluating the hue of each pixel of the image buffer and using the evaluated hue The method of claim 1 based on selecting from a plurality of perceptually uniform color spaces.   The method of claim 10, wherein converting one or more pixels of the image buffer from one or more perceptually uniform color spaces to the native color space is based on the previously evaluated hue of each pixel. A memory of the processing device for receiving the image buffer;
A color space processor of the processing device coupled to the memory and using the memory,
Converting one or more pixels of the image buffer from a native color space to one or more perceptually uniform color spaces;
Multiplying the brightness channel of the one or more pixels of the image buffer by a first value;
Converting the one or more pixels of the image buffer from the one or more perceptually uniform color spaces to the native color space;
A system comprising the color space processor for transmitting the image buffer to a downstream device;   The one or more perceptually uniform color spaces are color spaces such that two colors that are at an equal Cartesian distance in the color space are substantially equally separated from the human visual system. System.   Converting one or more pixels of the image buffer from a native color space to one or more perceptually uniform color spaces, evaluating the hue of each pixel of the image buffer and using the evaluated hue The system of claim 12, based on selecting from a plurality of perceptually uniform color spaces.   15. The system of claim 14, wherein the color processor is implemented as one of a microprocessor, microcontroller, graphics processing unit, digital signal processor, field programmable gate array or application specific integrated circuit. Receiving an image buffer and a first value from a second processing device at a first processing device;
Using the first processing device to convert one or more pixels of the image buffer from a native color space to one or more perceptually uniform color spaces;
Multiplying the brightness channel of the one or more pixels of the image buffer by a reciprocal of the first value using the first processing unit;
Using the first processing device to convert the one or more pixels of the image buffer from the one or more perceptually uniform color spaces to the intrinsic color space; and using the first processing device. And transmitting the image buffer to a downstream device.   The one or more perceptually uniform color spaces are color spaces such that two colors that are at an equal Cartesian distance in the color space are substantially equally separated from the human visual system. the method of. Receiving a second value from the second processing device in the first processing device; and using the first processing device to one or more color channels of the one or more pixels of the image buffer. The method of claim 16, further comprising multiplying a reciprocal of the second value. Receiving at least one white point value from the second processing unit in the first processing unit; and using the first processing unit, based on the at least one white point value, The method of claim 18, further comprising converting the one or more pixels from the one or more perceptually uniform color spaces to the native color space.   The method of claim 19, wherein the first value, the second value, and the one or more white point values are received from the second processing device in-band with the image buffer.   20. The method of claim 19, wherein the first value, the second value, and the one or more white point values are received from the first processing device out of band with the image buffer.   The method of claim 16, wherein at least one unique color space associated with the first processing device is different from at least one unique color space associated with the second processing device.   The one or more perceptually uniform color spaces associated with the first processing device are different from the one or more perceptually uniform color spaces associated with the second processing device. the method of.   The method of claim 16, wherein the downstream device is one or more of a display, an encoder, or an image processing pipeline. A memory of the first processing device for receiving the image buffer and the first value from the second processing device;
A color space processor of the first processing unit coupled to the memory and using the memory,
Converting one or more pixels of the image buffer from a native color space to one or more perceptually uniform color spaces;
Multiplying the brightness channel of the one or more pixels of the image buffer by a reciprocal of a first value;
Converting the one or more pixels of the image buffer from the one or more perceptually uniform color spaces to the native color space;
A system comprising the color space processor for transmitting the image buffer to a downstream device;   26. The one or more perceptually uniform color spaces are color spaces such that two colors that are at an equal Cartesian distance in the color space are substantially equally separated from the human visual system. System.   26. The system of claim 25, wherein the color processor is implemented as one of a microprocessor, microcontroller, graphics processing unit, digital signal processor, field programmable gate array or application specific integrated circuit.