JP5214144B2 - Strategies for processing image information using color information data structures - Google Patents

Description

  This application is directed to Stephen J. et al. It claims the priority of US Provisional Application No. 60 / 49,920, filed August 1, 2003, entitled “Bandwidth-Efficient Processing of Video Images,” filed on August 1, 2003, with Estrop as the sole inventor. US Provisional Patent Application No. 60 / 49,029 is incorporated herein by reference in its entirety.

  This application is directed to Stephen J. et al. This is also a continuation-in-part of US Provisional Application No. 10/694144, filed Oct. 27, 2003, entitled “Bandwidth-Efficient Processing of Video Images”, filed October 27, 2003, with Estrop as the sole inventor. US Provisional Application No. 10/694144 is incorporated herein by reference in its entirety.

  Finally, this application refers to Stephen J. et al. Related to US Provisional Patent Application No. 10/273505, filed Oct. 18, 2002, with the title “Methods and Apparatuses for Facilitating Processing of Interlaced Images for Progressive Video Displays,” with Estrop as the sole inventor. . US Provisional Application No. 10/273505 is hereby incorporated by reference in its entirety.

  This subject matter relates to a strategy for processing image information, and in a more specific embodiment relates to a strategy for processing image information using improved techniques to ensure that the color content of the image information is correctly replicated.

  Human vision relies on rod photoreceptor cells that respond to very low levels of light and cone photoreceptor cells that react on color. Cone cells generally have three parts of the visible electromagnetic spectrum: a long wavelength (eg, generally corresponding to red), a medium wavelength (eg, generally corresponding to green), and a short wavelength (eg, generally corresponding to blue). React). Thus, all colors can be represented as different combinations of at least three different color components. In general, color itself is a complex phenomenon that arises from both the physical aspects of electromagnetic radiation in the visible portion of the spectrum and the visual-related “mechanisms” and cerebral “mechanisms” used to process such information. It is. For example, human vision is more responsive to light intensity than light color (chroma) components.

  Electronic devices that replicate color images supplement the nature of the three-color color vision of human vision by providing three types of light sources. The three types of light sources produce different spectral responses that are perceived as different colors by the human observer. For example, cathode ray tubes (CRTs) provide red, green, and blue phosphors to create different colors. Other technologies do not use phosphors, but use light sources that emit at least three types of light to replicate colors in other ways.

  The International Commission on Illumination (CIE) has shown a comprehensive system for mapping the spectral characteristics of light into different perceived colors. In this regard, the term “matching function” refers to the “average” viewer's statistically tabulated response curve (usually short, medium, and For long wavelengths). For red, green, and blue, this function is denoted r (w), g (w), and b (w), respectively, and “w” represents wavelength. Such a reference lamp (or primary color) defines a light source (usually a monitor phosphor) used by a device that replicates image information having color content. The term “color space” refers to a specification defined by a set of primary colors and matching functions.

  The absolute color specification can mathematically map chromaticity tuples to different colors in the manner described above. However, several specific coding systems have been developed to ensure a more efficient coding scheme that can be applied to real-world applications, such as transmission and presentation of color image information. The first real-world application that the industry faced was broadcasting and presenting analog television signals. More recent applications include the transmission and presentation of digital video information over a network such as a TCP / IP network (eg, the Internet). In addition, the industry addresses the transmission and presentation of high definition (HD) video information in addition to standard definition (SD) video information. Thus, the features of coding systems can often be traced back to certain problems that the industry has faced at some time.

  Whatever the approach, the coding system addresses a common set of problems that arise when replicating image information with color content. The following discussion outlines common problems that are likely to address some form of coding system (in terms of terminology, the term “image information” refers to all information that can be displayed to the user in this disclosure. (This term is used broadly to include both still image information and animated video information).

• Considerations associated with color space Colors can be specified using three components. An image stream that relies on transmission of color content using discrete color components is called component video. One common coding technique uses red, green, and blue (RGB) components to specify colors. More formally, the RGB component describes the proportional intensity of a reference lamp that produces a color that is perceptually equivalent to a given spectrum. For example, the R component is

Where L (w) corresponds to a given spectrum and r (w) corresponds to the color space matching function r (w). In general, the RGB color space can be specified by color values associated with its primary color and white point. The white point refers to the chromaticity associated with the reference white.

  Computer monitors typically use RGB models to present color content to the user. However, the RGB coding model can be an inefficient option for the transmission of image information. Thus, image information is typically transmitted to the target device using some coding model other than RGB. Upon receipt, the image information can be converted to RGB color space for display using, for example, a 3 × 3 affine matrix. As described in the heading “Gamma Considerations” below, each R, G, or B component data can also be expressed in terms of a pre-gamma correction form called R ′, G ′, and B ′ values. (In general, by convention, accent marks represent non-linear information in this disclosure).

  A general strategy in this regard is to define color by reference to luminance related components (Y) and chroma related components. Luminance generally refers to the perceived intensity (brightness) of light. Luminance can be expressed in the form before gamma correction (as described in “Gamma Considerations” below) to create a non-linear counterpart called “Luma” (Y ′). The chroma component defines the color content of image information about luma. For example, in the digital domain, the symbol “Cb” corresponds to an n-bit integer scaled representation of the difference B′−Y ′ (usually from the range of −127. “Cr” corresponds to a scaled representation of an n-bit integer of the difference R′−Y ′. The symbol “Pb” refers to the analog counterpart of Cb, and the symbol “Pr” refers to the analog counterpart of Cr. The symbols “Pb” and “Pr” have a nominal range [−0.5. . . 0.5] can also be referred to as the digital normalized form of Cb or Cr. Component image information defined by CbCr and PbPr can be formally accented (eg, Cb'Cr 'and Pb'Pr') when representing non-linear information.

  Color content can also be communicated as composite video (rather than the component video described above). In the composite signal, luma information and chroma information are combined into one signal. For example, in the coding system Y'UV, U represents a scaled version of BY and V represents a scaled version of RY. The luma and chroma components are processed to provide a single signal (eg, in the form shown in NTSC (National Television System Committee) or PAL (Phase Alternate Line) format). Coding system Y'IQ defines another composite coding system formed by transforming the U and V components in a specified manner. In general, industry has historically encouraged Y-related color spaces (Y'CbCr, Y'PbPr, YUV, YIQ, etc.). This is because reducing the color image information in these color spaces can be performed more easily than the image information expressed in the RGB color space.

  In general, it is possible to convert color content from one color space to another by using one or more matrix affine transformations. More formally, the condition-like color characteristics allow a set of color space coefficients to be expressed for another set of matching functions ("conditional color-representing colors" Refers to two spectra that map to the same set and therefore appear perceptually identical, ie, appear to be the same color).

• Gamma considerations Cathode ray tubes (CRTs) do not have a linear response transfer function. In other words, the relationship between the voltage applied to the CRT and the resulting luminance produced by the CRT does not define a linear function. More specifically, the predicted theoretical response of a CRT has a response that is proportional to 5/2, ie, for a given input voltage “V”, the luminance “L” of the CRT result is , L = V ^2.5 .

  In applications, a source of image information (such as a video camera) typically compensates for image information in advance by applying a transfer function to the image information. The “transfer function” is generally an inverse function of the CRT luminance response. This transfer function (commonly referred to as an encoding transfer function) applied at the source creates “gamma corrected” nonlinear image information. As this non-linear signal passes through the display device, a linear luminance is created. According to the notation described above, non-linear (or pre-compensated) image information is represented by, for example, Y'Cb'Cr 'by accenting its components.

  It is common to transmit image information in a non-linear (compensated) form. The receiving device's presentation device (eg, CRT) supplements the encoding transfer function due to its inherent non-linearity to provide color content appropriately converted to consumption.

  It is common to adjust the exponent of the encoding transfer function to take into account the conditions under which image information is likely to be seen. For example, video information displayed on a normal television is usually presented in a dim viewing environment typical of home settings, whereas image information displayed on a normal computer monitor is displayed in an office setting. Presented in a typical bright viewing environment. Different transfer function adjustments are appropriate for these different viewing environments. For this reason, television video sources typically use transfer functions based on the built-in assumption that image information is presented in a dim viewing environment. This means that the transfer function applied by the source generally compensates less for the inherent non-linearity of the CRT.

  As another special consideration, encoding image information using a transfer function generally applies a special approximation function to the low voltage portion of the function. That is, encoding techniques generally provide a linear segment in this portion to reduce the effects of imaging sensor noise. This segment is referred to as a “linear tail” with a defined “toe slope”.

• Chroma information sampling and alignment for luma information As noted above, human vision is more sensitive to light intensity than light color components. The coding system takes advantage of this fact to reduce the amount of chroma (Cb′Cr ′) information that is coded with respect to the amount of luma information (Y ′). This technique is called chroma subsampling. A numerical notation generally expressed as L: M: N can be used to represent this sampling strategy, where “L” represents the sampling reference factor for the luma component (Y ′) and “M” And “N” refer to chroma sampling (eg, Cb and Cr, respectively) for luma sampling (Y ′). For example, the notation 4: 4: 4 can represent Y′CbCr data with one chroma sample per luma sample. The notation 4: 2: 2 can represent Y′CbCr data (in the horizontal direction) with one chroma sample for every two luma samples. The notation 4: 2: 0 can represent Y′CbCr data with one chroma sample for every 2 × 2 cluster of luma samples. The notation 4: 1: 1 can represent Y′CbCr data with one chroma sample for every four luma samples (in the horizontal direction).

  In situations where the coding strategy provides more luma information than chroma information, the decoder can reconstruct “missing” chroma information by performing interpolation based on the supplied chroma information. More generally speaking, downsampling refers to any technique that produces a smaller number of image samples compared to the first set of image samples. Upsampling refers to any technique that produces a larger number of image samples compared to the first set of image samples. Thus, the interpolation described above defines a type of upsampling.

  The coding strategy also specifies the form in which chroma samples are “aligned” spatially with the corresponding luma samples. The coding strategy is different in this regard. One strategy aligns the chroma sample to the luma sample so that the chroma sample is positioned directly “above” the luma sample. This is called “cositing”. In another strategy, the chroma sample is positioned in a gap space within the two-dimensional array of luma samples. Figures 10-12 (described in turn below) illustrate different sampling and alignment strategies that present luma and chroma information.

Quantization considerations Quantization refers to a methodology for assigning discrete values to signal amplitudes of color components. In the digital domain, the numerical value spans the specified range (color gamut) of the color space value for the specified number of steps. For example, it is common to use 255 steps to describe each component value, and each component can take a value from 0 to 255. Although it is common to use 8 bits to represent each color value, colors can also be represented with higher accuracy (eg, 10 bits) as well as with lower accuracy.

  Coding strategies often assign portions at the ends of a range of quantization levels to represent the back level and white level, respectively. That is, coding strategies often define a reference black level and a reference white level, but also assign coding levels above these reference levels to represent values that swing beyond the reference black level and the reference white level. For example, an 8-bit coding strategy can assign level 16 to black and level 235 to white. The remaining levels below 16 define the so-called “toe room” and the remaining levels above 235 define the so-called “head room”.

• Considerations for interlaced versus progressive representation Traditional television signals are scanned in an interlaced manner. In interlacing, the first field of the video frame is captured and immediately followed by the second field of the video frame (eg, 1/50 second or 1/60 second later). The second field is offset perpendicular to the first field by a small amount so that the second field captures information in the gap space between the scan lines of the first field. So-called Bob interlace is one known type of interleaving strategy. A complete video frame is constructed by presenting the first and second fields in quick succession so that they are perceived by a human viewer as a single frame of information.

  However, computer monitors and other presentation devices display image information in a progressive form, not in an interleaved form. Therefore, in order for a device to present interlaced information to a computer monitor, it must display progressive frames at an interlaced field rate by interpolating data from the opposite field (a process called “deinterlacing”). ). For example, to display an interlaced field, the “missing” data in the spatial position between the scan lines must be interpolated by examining the opposite field. A non-interlaced image format is referred to as a “progressive” format.

  Additional information regarding each of the above topics is described in Non-Patent Document 1.

  To double the complexity described above, the industry adapts to a number of different format standards that represent image information. Standards include International Telecommunication Union (ITU), European Broadcasting Union (EBU) (DVB or Digital Video Broadcasting Encouraging), AES (Audio Engineering Society), ATSC (Advanced Television System Systems Films) Has been promulgated by several organizations and committees, including the Association of Persons (SMPTE), SECAM (Sequential couleur ave mEmoire), NTSC (National Television System Committee), and the like.

  Each of these organizations cut specific combinations of coding features from the area of possible coding options described above. Thus, as the inventor understands, standards generally include definitions of primary colors; transfer functions; intended viewing conditions; transfer matrices; towroom and headroom specifications; chroma subsampling and alignment strategies, and so on. Different in application. The primary color (along with the white point criterion) defines the basic color space of the standard. The transfer function determines how the standard translates between linear image information and nonlinear information. The intended viewing conditions define the assumptions that the standard makes regarding viewing environments where image information is likely to be consumed (such as the assumption that the television is viewed in a dimly lit home setting). Viewing conditions change the effective gamma and brightness (black level) and contrast (white level) of the image information. The transfer matrix determines how the standard translates between different color spaces (eg, from Y'YbYr color space to RGB color space). The headroom and toeroom specifications determine the quantization level that the standard assigns to represent the black and white range. Chroma subsampling and alignment strategies specify the way in which chroma information is subsampled and positioned with respect to luma information.

The requirements for each standard are detailed and detailed in existing standards documents. Typical standards include the following:
ITU-R recommendation BT. 470 is an international standard that provides specifications for analog monochrome television devices.
ITU-R recommendation BT. Reference numeral 601 denotes an international standard that defines studio digital coding of image information. In this standard, Y′CbCr coding of image information is used.
ITU-R recommendation BT. 709 is an international standard that defines studio coding of high-definition video information. High definition (HD) content represents video content that is higher than standard definition (SD) and is typically 1920 × 1080, 1280 × 720, and the like.
SMPTE 170M is a standard that defines the coding of composite analog video information (eg NTSC).
SMPTE 240M is a standard that defines the coding of analog high definition video information.
IEC 61966-2-1 (sRGB) is a standard for coding image information to 255 levels using an 8-bit quantization scheme.
IEC 61966-2-2 (scRGB) is a standard that defines the linear format of sRGB and greatly expands the sRGB color gamut.
ISO / IEC 13818 (MPEG-2) is a standard for coding audio and video signals in a compressed format.
ISO 10918-1 (JPEG) is a standard for lossy compression of still image information.

  The high diversity of currently used coding standards contributes to multiple problems of coding, transmitting and processing image information. As an overview, video processing pipelines associated with specific devices are often designed to process specific types of signals with defined formatting, and in this limited role, these devices are reliable. Such image information can be processed correctly in the form. However, in the context of the wider world of image information that is currently in use, these devices can reliably format this formatting information through mechanisms and pipelines that interpret the color formatting of other types of image information. There is a possibility of lacking a mechanism to propagate in. More precisely, video pipelines can receive information defining certain aspects of color formatting applied to the received image information, but for the purpose of the inventor, these video pipelines May lack appropriate mechanisms to reliably propagate this color information through the pipeline to downstream components in the pipeline. As a result, such formatting information is “lost” or “discarded”. The downstream component can address the lack of information regarding color formatting by “guessing” the formatting information. When a component incorrectly guesses, the pipeline processes the image information in a suboptimal or incorrect way.

  FIG. 1 is presented as a medium to further illustrate the above potential problems. A high level representation of the video processing pipeline 100 is shown in FIG. Pipeline 100 includes a conventional processing stage defined by an input stage 102, a processing stage 104, and an output stage 106. At input stage 102, input source 108 represents any source of image information. The source 108 typically has either newly captured image information (eg, created by a camera or scanner) or previously captured image information (eg, from a disc) that is presented to the input stage 102 via a channel. Received via an IP network, etc.). In the former case, the capture processing functionality 110 can perform all kinds of preliminary processing on the image information received from the source 108. In the latter case, the decoder functionality 112 performs all types of stream-based information extraction and decompression to create image data. In general, such processing can include separating image information of received information from audio information, decompressing information, and the like. With respect to processing stage 104, processing functionality 114 performs all kinds of processing on the resulting image information, such as mixing multiple streams of image information together into a composite signal. With respect to the output stage, the output processing functionality 116 represents all types of processing performed on the processed image information in preparation for output to the output device 118. Output device 118 may represent a television, a computer monitor, or the like. The output device can also represent a storage device. Further, the output “device” (or output processing functionality 116) can provide compression and formatting functionality (such as a multiplexer) that prepares the information for storage to the device or distribution over the network.

  The bottom row of FIG. 1 summarizes the defects described above for known systems. Block 120 is that the pipeline functionality (110, 112, 114, 116) cannot accurately interpret the color formatting applied to the input signal and / or can trust the color information to downstream components through the pipeline. Indicates that it cannot propagate in the form. For example, pipeline 100 may receive image information coded using a specified format. The received information may include fields that identify the formatting features used, or these features can be deduced based on other telltale properties of the received information. However, due to the large number of standards used, the first stage of the pipeline 100 lacks the functionality to correctly interpret this information and pass it to downstream components in the video pipeline 100. As a result, this coding information is lost immediately. This can lead to situations where the image information is passed to the downstream pipeline component without guidelines on how the downstream component should interpret this image information (essentially only 1s and 0s).

  Block 122 represents how the video pipeline 100 handles the above problem. That is, functional components that do not have a guideline on how to interpret color content in image information often “guesses” how to interpret it. The guess may or may not be accurate. For example, the video pipeline may have a transfer function applied to the image information (possibly based on the image size), a lighting condition assumption specific to the image information, and a chroma subsampling scheme used by the image information (based on the data format). May make assumptions regarding

  Block 124 represents the potential consequences of incorrect guessing. That is, incorrect guesses can result in suboptimal display quality or incorrect display quality. An image presentation may appear to have “unnatural” colors or motion artifacts. Or it may be unduly “contrast strong”, distorted, or cropped incorrectly.

Charles Poyton's well-regarded Digital Video and HDTV

  Thus, there is a need for a more satisfactory technique for processing image information having color content.

  According to one exemplary embodiment, a method for processing image information having color content represented in a specified format is described. The method includes (a) providing a color information data structure that includes color information defining at least one aspect of the specified format, and (b) used to process the image information. Passing the color information data structure together with the image information to at least one functional component; and (c) based on the color information in the color information data structure, the image information in the at least one functional component. Including processing.

  According to another exemplary feature, a video that conveys information about a transfer function used to convert (a) linear image information to nonlinear image information and / or vice versa in a color information data structure. A transfer function field; (b) a primary color field conveying a primary color associated with the image information; (c) an image light field conveying an intended viewing condition associated with the image information; A transfer matrix field that can be used to change the color space; (e) a nominal range field that conveys information about potential towrooms and headrooms associated with the image information; and (f) chroma samples within the image information. Video chroma showing the sampled and aligned shape with respect to the associated luma sample Pump includes a ring field.

  According to another exemplary feature, the at least one functional component described above belongs to a group of functional components that define a video processing pipeline, and the color information data structure is for use by downstream functional components. Passed through the video processing pipeline.

  According to another exemplary feature, the at least one functional component further determines at least one other aspect of the specified format and determines the at least one other aspect as the color information data. Add to at least one unknown field in the structure.

  According to another exemplary feature, the above-described passing of the color information data structure includes passing the color information data structure in an unused field of the existing data structure.

  According to another exemplary feature, the method further includes setting a flag indicating that the at least one functional component has the ability to process the color information data structure.

  (A) providing a main video stream of image information; (b) providing a video substream of image information; and (c) specified based on color information specified in a color information data structure. Another example of processing image information comprising performing at least two tasks using the main video stream and the video substream to produce output video information represented in a specific color space A method will be described.

  (A) converting input image information in a luma-related color space into linear image information in an RGB-related color space; and (b) the linear image information in the RGB-related color space to produce processed information. And (c) converting the processed information into non-linear image information in a luma-related color space, another exemplary method for processing image information is described.

  According to another exemplary feature of the above-described method, the conversion is performed by converting the input image information from an interlace format to a progressive format before the processing operation, and after the processing, Converting information from a progressive format to an interlaced format.

Additional exemplary embodiments will now be described.
Throughout this disclosure and the drawings, the same reference numerals are used to refer to the same components and features. The numbers in the 100 series refer to the features that first appear in FIG. 1, the numbers in the 200 series refer to the features that appear first in FIG. 2, and the numbers in the 300 series refer to the features that appear first in FIG. The same applies hereinafter.

  Next, exemplary mechanisms and procedures for improving the processing of image information having color content are described. As a broad overview, this mechanism and procedure provides a data structure that captures information about color formatting applied to image information. This information is called “color information”, and the data structure itself is called “color information (CI) data structure”. In one exemplary case, the color information may include fields that convey in particular the following information related to image information: transfer function information; primary color information; video writing information; transfer matrix information; nominal range information; And chroma subsampling information. This grouping of fields is intended to be illustrative rather than restrictive. Other embodiments may include additional fields, omit certain fields identified above, and so forth.

  Video processing pipelines use color information to help facilitate the processing of image information. For example, color information can specify the form in which functional components in the pipeline must convert the image information. In addition, in sharp contrast to known systems, the video pipeline includes functionality to pass a CI data structure through the pipeline so that downstream components in the pipeline can be passed from the CI data structure. Information can be extracted. This provision reduces the likelihood that the video pipeline will make false guesses about the nature of the image information, so that this provision has the potential to improve the quality of the displayed image.

  According to another exemplary feature, components in the pipeline can include functionality to independently interpret image information. If a component can reliably determine color-related aspects that were not previously specified for color formatting, the component can add this information to the CI data structure. This supplemental color information is made available from downstream components in the video pipeline, which can supplement the CI data structure by adding new information. Therefore, the color information data structure becomes more “knowledge” when gathering information from components in the pipeline that can pass through the pipeline and deduct different formatting characteristics applied to the image information. Can do.

  According to another exemplary feature, the color information is intended to provide a “receptacle” that conveys information about a number of possible formats that can be used to code image information. Thus, the CI data structure allows certain types of image information to be accepted by the video pipeline and allows it to be processed correctly by interpreting the color information conveyed by the CI data structure. It can be regarded as a general-purpose interface.

  According to another exemplary feature, the CI data structure can pack a wide variety of information regarding different format standards into a small number of bits of information using a particular efficient scheme. In one exemplary case, this data structure packs a great variety of color information into a small number of words (eg, 16-bit structure, 24-bit structure, etc.).

  According to another exemplary feature, the system communicates the CI data structure by using unused fields provided in one or more existing data structures used in the video pipeline. Can do. This allows systems that use such existing data structures to make use of color information without expensive and complex retooling of the system. In addition, if these systems are not configured to process CI data structures, but the CI data structures are present in the input information provided to such systems, these systems CI data structures can be safely ignored without "destruction".

  According to another exemplary feature, interlaced image information (eg, Y'Cb'Cr ') in the luma-related color space can be converted to a linear progressive RGB color space. Various processing (such as rescaling) can then be performed on the image information in a linear / progressive RGB color space. The image information can then be converted to a luma-related color space (eg, Y'Pb'Pr ') for output (eg, display on a television). The luma related information processing 104 in the linear / progressive RGB space deviates from other approaches that do not perform the conversion 104 of the image information to the intermediate linear RGB color space for processing in the form summarized above.

  Other features and associated benefits are described in detail in the following detailed specification.

  With respect to terminology, the term “image information” is intended to include all kinds of information that can be consumed by the user in some visual form. The image information can represent information expressed in all formats, such as an analog format, a digital format, or a combination of an analog format and a digital format. The image information can represent still image information (eg, digital photos) and / or moving information (eg, video information). Further variations are contemplated by the use of the term image information.

  The term “color information” refers to all information describing the color content of image information. For example, when the image information represents the color content of a specified format, the color information can convey information regarding the format. The term “color information data structure (CI data structure)” refers to the way color information is coded and communicated within a video pipeline.

  The term “video pipeline” refers to all functionality that processes image information. The pipeline includes at least two functional components that manipulate the image information in a continuous manner, ie one after another.

  This disclosure includes the following sections: Section A outlines an exemplary color information data structure and the manner in which it is used in the video pipeline. Section B describes exemplary conversion operations that can be performed within the video pipeline of Section A. Section C describes an exemplary configuration of the color information data structure defined in Section A. Section D describes one particular exemplary embodiment of a video pipeline that implements the principles shown in Sections A-C. Section E describes an exemplary computing environment that implements aspects of the features described in Sections AD.

  In general, for the structural aspects of this subject, all of the functions described herein are implemented using software, firmware (eg, fixed logic), manual processing, or a combination of these embodiments. be able to. The terms “module”, “functionality”, and “logic” as used herein generally refer to software, firmware, or a combination of software and firmware. In the case of a software embodiment, the term module, functionality, or logic represents program code that performs a particular task when executed on one or more processing devices (eg, one or more CPUs). The program code can be stored in one or more fixed and / or removable computer readable memory devices.

  With respect to the procedural aspects of this subject matter, certain operations are described as separate configuration steps that are performed in a certain order. Such embodiments are exemplary and non-limiting. Certain steps described herein can be grouped together and performed in a single operation, and certain steps can be performed in an order different from that used in the examples presented in this disclosure. it can.

A. Exemplary Color Information (CI) Data Structure and Its Application Overview FIG. 2 shows an exemplary overview of a video processing pipeline 200 that uses a color information (CI) data structure 202. Information describing the formatting used to represent the colors in the image information 204 and / or how the image information 204 must be later processed by the pipeline 200 in the CI data structure 202. Is included. CI data structure 202 provides a uniform “receptacle” that receives color information for a number of different color coding formats. The CI data structure 202 thereby provides some kind of general purpose interface and extends the types of image information that the pipeline 200 can successfully process in an anomalous manner. According to another advantageous feature, all functional components in this video pipeline can be derived from the CI data structure 202, potentially with new information about the image information 204 that these components can be deduced. The CI data structure 202 can be supplemented.

  The video pipeline 200 itself includes a series of stages including an input stage 206, a processing stage 208, and an output stage 210. With respect to input stage 206, input source 212 represents all sources of image information 204. Source 212 generally receives newly captured image information (eg, captured by a camera or scanner) or received via any type of network, eg, received via broadcast transmission (eg, satellite transmission or cable transmission). Captured before being presented to the input stage 206 via some route, such as being received via a local storage of image information (such as a TCP / IP digital network such as the Internet), etc. (video disc, local database, etc.) Image information can be included. The input processing functionality 214 is configured to perform all types of preliminary processing on the image information 204, which may vary depending on the nature of the input source 212 (eg, the image information 204 is Depending on whether it is newly captured or input from a pre-captured source of information, when entering previously captured / stored information, the image information in the received signal is processed. Separation from audio information, decompression of image information, etc. With respect to processing stage 208, processing functionality 216 mixes video information into a composite signal of multiple streams, Perform all types of processing on captured image information 204, such as performing color space conversion With respect to the output stage 210, the output processing functionality 218 is configured to perform all processing of the image information 204 in preparation for output to the output device 220. The output device It can represent a television, a computer monitor, a storage device (both remote and local), all network accessible target locations, etc. Each of the functional components (214, 216, 218) can perform a single task to perform a task. It can be physically implemented as a single device or a plurality of devices coupled together in series or parallel.In one instance, components in pipeline 200 are dynamically configured logic modules ( For example, software modules) Kill.

  Each of the functional components (214, 216, 218) includes a respective functionality (222, 224, 226) that is configured to process the CI data structure 202. This functionality (222, 224, 226) includes the logic for reading information in the CI data structure 202, the logic for interpreting the information in the CI data structure 202, and the interpreted information in the CI data structure 202 as image information 204. Can include logic that applies to the process. Logic (222, 224, 226) is deduced to provide additional information related to color formatting used in the image information 204, and the deduced information is provided to the CI data structure 202 to provide the CI data structure Logic that fills an unknown field before the field 202 can also be included. For example, the functionality (222, 224, 226) can determine that a certain format is being used. Based on this, the functionality (222, 224, 226) can be deduced that the standard has certain coding characteristics (eg, by accessing a predetermined look-up table). The functionality (222, 224, 226) can then write information about this deduced property into a previously unknown field of the CI data structure 202. Specifically, in one embodiment, a component in pipeline 200 can automatically set an unknown value in CI data structure 202 to a specified value, such as zero. This operation constitutes initialization of the CI data structure. Subsequent components are alerted to the unknown nature of certain values in the CI data structure by a value set to zero. These components can then do so if they can supply the missing value. Arrow 228 generally indicates that functionality (222, 224, 226) can pull information from CI data structure 202. Arrow 230 generally indicates that functionality (222, 224, 226) can be added to CI data structure 202. FIG. 2 specifically illustrates that the functional component 214 interacts with the CI data structure 202. However, as mentioned above, this same CI data structure 202 is passed through the video pipeline 200 so that none of the other functional components (214, 216, 218) are shown. Can interact with the CI data structure 202 in any way.

  Different functional components (214, 216, 218) have different CI data depending on a number of factors such as the processing stage used (206, 208, 210), the details of the particular application, the color format being processed, etc. The structure 202 can be processed.

  For example, the functionality 222 associated with the source 212 and input processing functionality 214 determines information about the formatting standards used in the received image information 104 and that information for use by downstream components of the pipeline 200. To the CI data structure 202 can be performed. As a general rule, in the case of a new capture, the capture device usually “knows” implicitly the formatting applied to the image information it produces. On the other hand, devices that receive image information captured by other sources apply to the image information by examining format-related information related to the received image information or by making a logical and reliable guess. In some cases, the determined color formatting can be determined. Continue with specific examples. For example, analog capture devices generally know the video standard and color space into which data is captured. As another example, the DVD navigator knows the color space of the image information 204 it is dealing with by the fact that it is analyzing a DVD with NTSC content. As another example, a capture device that receives an MPEG-2 elementary video stream deducts the formatting characteristics being used, since MPEG-2 explicitly lists color information in its sequence display extension header. For example, this header indicates the primary color, transfer function, and transfer matrix associated with the image information 204. As another example, a high definition (HD) digital tuner must know that it is streaming HD data in the 709 color space, and so forth. In all of these cases, functionality 222 may provide CI data structure 202 with “already known” information about the color formatting being used for use by downstream functional components of pipeline 200. it can. In known systems, this information was quickly discarded and therefore lost.

  The color space converter in the pipeline 200 uses the CI data structure 202 to ensure that correct conversion operations are performed on the received image information 204. In other words, for example, the color space converter uses information collected from the CI data structure 202 to determine the conversion algorithm it applies to, or to determine the settings / parameters used in that algorithm, etc. Can do.

  The mixers in pipeline 200 serve the purpose of mixing different types of image information together, and such information is potentially represented using different color formats. For example, an application may want to combine a digital photograph with superimposed graphical information. Pipeline 200 can use the extended formatting information provided in CI data structure 202 to ensure that all of the combined information has a common format (eg, a common color space). . This operation can be performed, for example, by an upstream component (with respect to the mixer in the pipeline) before the information is received by the mixer.

  Alternatively, this action can be assigned to graphical processing hardware. The graphical processing hardware can include one or more graphical processing units (GPUs) provided by, for example, a video processing card (described below with respect to FIG. 13). In this case, the CI data structure 202 associated with the combined information can be passed through the pipeline to the graphical processing hardware, which uses the CI data structure 202 to Image information can be converted to a common format, and this information can then be combined. In this embodiment, the supplemental color information provided by the CI data structure 202 eliminates the need for drivers in the video pipeline to infer the intended color space or chroma scheme, and thus the driver (or graphics). The processing hardware) guesses wrong and produces bad output results.

  In summary, the bottom row of the block of FIG. 2 summarizes exemplary benefits of using the CI data structure 202 applied to the video pipeline 200. Block 232 indicates that the CI data structure 202 has a uniform structure that is passed through the pipeline 200. Block 234 indicates that each functional component (214, 216, 218) of pipeline 200 can retrieve information from CI data structure 202 and supply values for unknown fields in CI data structure 202. This means that compared to known systems, the functional components (214, 216, 218) can reduce or eliminate the amount of unjustified guesses that need to be made to process the image information 204. To do. Block 236 shows that the final result of this strategy is to provide improved quality of color reproduction. That is, by reducing some of the inaccurate guesses, the video pipeline 200 has an image with low contrast, images with unnatural colors, images with various distortions (motion artifacts, clipping, etc.). The possibility of providing output information having various abnormalities is reduced.

  According to another benefit, as described in more detail in Section D (below), the video pipeline 200 can communicate the CI data structure 202 using existing data structures. For example, CI data structure 202 can be “resident” in unused field (s) of an existing data structure used to transmit information through pipeline 200. A flag can be provided to inform whether or not the CI data structure 202 is included in an unused field (or whether it contains meaningless unspecified information or possibly a series of default information such as zero). This provision has at least two benefits. First, the use of pre-existing fields of pre-existing size requires a complete re-engineering of existing functional components (214, 216, 218) to address the use of the new CI data structure 202. Means no. Further, these functional components (214, 216, 218), which may not have the ability to understand or interpret the CI data structure 202, need not process this information. In other words, components that do not understand the CI data structure 202 are simply not affected by it. At the same time, the unobtrusive nature of providing color information allows these components to automatically pass color information along the pipeline 200. In other words, color information is usually not lost because it exists in an existing data field. This allows downstream components that may be configured to use color information to receive unchanged color information from such non-CI compliant upstream components.

  In the above discussion, it was assumed that the image information 204 is immediately displayed on the display device based on the color information in the CI data structure 202. However, the CI data structure 202 also provides an efficient technique for packing color information associated with the image information 204 and thus stores the image information 204 and the associated CI data structure 202 in a space efficient manner. It can be used as an archiving technique (while retaining a large amount of formatting information applied to the image information 204).

  Specifically, the component can be configured to automatically store the CI information 202 along with the image information 204. When the image information 204 is later retrieved, the respective fields containing the CI information 202 can be unpacked and passed to the pipeline 200, and therefore, for the benefit of downstream components, a large amount of image information 204 color structure. Is saved.

  Furthermore, in the above discussion, it was assumed that the components in pipeline 200 could provide missing color information for the benefit of only the downstream components of the pipeline. However, components in the pipeline can provide color information for use by upstream components in subsequent processing of image information. For example, consider the case where a video pipeline is applied to the task of processing and presenting a movie from an unknown video source. One of the first components in the pipeline may not be able to determine certain features of the formatting applied to this video information, and therefore may not initially process this information in an optimal manner. . However, later components in the pipeline can then deduct formatting applied to the video information by an unknown video source. These downstream components can communicate with the upstream component to inform the upstream component that subsequent signals it receives from the same source have the specified formatting. The upstream component can then process this video information in a more error free manner.

  Further, the term “downstream” does not necessarily indicate a fixed order of operation within the system. In general, pipelines can be created in a dynamic manner to provide desired behavior for image information 204 by stitching together different functional components. In this regard, color information can be provided in a media type data structure that is used for negotiation between components. When the pipeline is first created from the “output end”, the color information flows in the “reverse direction” when the components are connected.

  FIG. 3 illustrates the operation of the pipeline 200 of FIG. 2 in the form of a flowchart. Step 302 includes first capturing / inputting image information 204 from one or more sources 212. Step 304 includes processing the image information 204 according to the color information collected from the CI data structure 202 by the appropriate functional components (214, 216, 218) of the pipeline 200. At step 306, optionally, color information deduced by the appropriate functional components (214, 216, 218) of the pipeline 200 is provided to the CI data structure 202 for use by downstream components of the video pipeline 200. For example. For example, steps 304 and 306 correspond to the processing performed by input stage 206 in the first iteration of the loop defined by this flowchart. At step 308, it is determined whether steps 304 and 306 must be repeated for the next functional component (214, 216, 218) of video pipeline 200. The process of FIG. 3 ends with the final output of the image information 204 to an output device (eg, television, computer monitor, archive device, network destination location, etc.) based on the CI data structure 202.

B. Exemplary Conversion Operation FIG. 4 illustrates a conversion operation 400 that converts the image information 204 using the video pipeline 200 of FIG. 2 or in the context of another type of pipeline. Accordingly, in FIG. 4, the use of color information 202 in the video pipeline is further developed. Another feature shown in FIG. 4 represents an advance in video processing technology even when color information 202 is not used. That is, as an overview, FIG. 4 shows conversion of image information represented by a luma-related color space (for example, Y′Cb′Cr ′) to a linear progressive RGB space, and subsequent execution of processing for linear progressive RGB data. Techniques are shown. The technique then converts this information into luma related space (eg, Y'Pb'Pr ') for output devices (television devices, storage devices, etc. that display image information 204 of luma related color space). Can be output. The processing of luma related information 204 in a linear / progressive RGB color space deviates from other approaches that do not convert the image information 204 to an intermediate linear RGB color space for processing in the form summarized above. Linear RGB processing is advantageous for several reasons. For example, linear RGB processing removes brightness and color shift artifacts that can be caused by processing image information in a non-linear RGB or non-linear YUV color space.

  The conversion operation 400 of FIG. 4 includes an exemplary series of blocks describing the conversion steps described above. Specifically, the top row of the block in this figure shows the conversion of Y'Cb'Cr 'image information to a designated color space for processing. The bottom row of the block shows the conversion of the processed data into Y'Pb'Pr data (Y'Pb'Pr can define the analog counterpart of Y'Cb'Cr ').

  With respect to the upper row, block 402 indicates that Y'Cb'Cr'4: 2: 0 image information is received and upscaled to Y'Cb'Cr'4: 4: 4 image information. The 4: 2: 0 notation indicates that the chroma information (Cb′Cr ′) is subsampled with respect to the luma information (Y ′). The representative sample shown above block 402 in FIG. 4 shows that different coding strategies can place chroma samples with respect to luma samples in different messengages. The upsampling operation interpolates chroma information to produce the same amount of chroma information as luma information (eg, provides a 4: 4: 4: representation of input image information).

  Block 404 applies matrix transformation to the Y'Cb'Cr'4: 4: 4 image information to convert it to another color space, namely the R'G'B 'color space.

  Block 406 converts the non-linear R'G'B 'image information to a linear form by applying a transfer matrix. As noted above, the accent mark (') associated with R'G'B' image information indicates that it is in a non-linear form, and the absence of an accent mark (eg, RGB) is usually Represents linear data (except that it is also common to remove accent marks when it is commonly understood that the referenced signal represents non-linear information). The model transfer function shown above block 406 in FIG. 4 shows the general shape of the function used, which is also the general shape of the transfer function specific to the CRT (not shown). This model transfer function also shows that a linear tail can be used near the V = 0 portion of the curve.

  Block 408 optionally performs deinterlacing of the content to convert from interlaced format to progressive format in order to correctly perform the image rescaling operation.

  In block 410, optionally, the primary colors of the linear RGB information are converted for representation in another color space. For this conversion, whatever primary color is desired for processing performed in the downstream components of the video processing pipeline, matrix conversion is applied to the RGB information to change the primary color to correspond to that primary color. Can be included. In one example, block 410 may include converting the image information from one RGB related color space to another RGB related color space (eg, scRGB).

  Block 412 generally represents all types of processing of image information having a transformed color space. This may constitute, for example, execution of processing on image information in a linear progressive RGB related color space.

  The lower row of FIG. 4 is generally the reverse of the operations described above. That is, after processing at block 412, image information is converted to another color space, such as optionally returning to RGB color space, at block 414. At block 416, a motion reinterlacing of the content is applied if an interlaced form of store or display is required. At block 418, a transfer function is applied to convert the RGB image information back to a non-linear form (R'G'B '). At block 420, the color space of the R'G'B 'image information is changed to a format that separates its luma component (Y') from its chroma component (Pb'Pr '). Finally, at block 422, the Y'Pb'Pr 'image information is optionally subsampled to reduce the amount of chroma samples (Pb'Pr') relative to the amount of luma samples (Y '). That is, this block 422 converts Y′Pb′Pr′4: 4: 4 image information (there is a chroma sample for each luma sample) to Y′Pb′Pr′4: 2: 0 image information (a smaller number than the luma sample). Chroma sample).

  The CI data structure 202 can play a role in the context of FIG. 4 by providing instructions on how each of the processing blocks processes the image information 204. For example, it is possible to supply color information that specifies the form in which the pipeline converts from the luma-related color space to the RGB-related color space.

C. Exemplary Configuration of Color Information Data Structure FIG. 5 illustrates one exemplary embodiment of the CI data structure 202 introduced in FIG. CI data structure 202 includes a plurality of fields that define various aspects of coding standards that can potentially be applied to image information 204. The fields shown in FIG. 5 are representative rather than a limitation on the type of information that can be packed into the CI data structure 202. In other embodiments, certain fields shown in FIG. 5 can be omitted, or other fields not shown in FIG. 5 can be added.

  This section provides a general overview of the fields shown in FIG. 5, followed by a detailed description of one exemplary embodiment of the CI data structure 202.

  First, the first field 502 defines transfer function information. This information is used to define how (non-linear) R'G'B 'image information 204 is converted to (linear) RGB information (and / or vice versa).

  The second field 504 defines the primary color used for encoding the image information 204. For example, this field 504 can specify an RGB response function associated with the RGB image information 204.

  The third field 506 defines which video lighting assumptions are applied to the image information 204. For example, this field 506 may indicate that the image information 204 is a bright office environment (on a computer monitor) rather than a dimly lit home environment (usually for information intended for a home television presentation). It can be specified whether it has been coded for presentation at (usually the information intended for the presentation).

  A fourth field 508 defines a transfer matrix applicable to the image information 204. For example, the matrix may define a method for converting between the Y'Cb'Cr 'color space and the R'G'B' color space.

  The fifth field 510 defines nominal range information associated with the image information 204. Specifically, this field defines whether the image information 204 provides a specified toe room below the reference black level and / or a specified headroom above the reference white level.

  Finally, the sixth field 512 defines chroma subsampling information that defines how chroma information is sampled and positioned with respect to luma information.

  Different coding schemes can be used to represent the six fields shown in FIG. According to one exemplary non-limiting approach, a collection of bits is allocated to represent the CI data structure 202. This collection of bits is placed in a designated data structure with predefined fields (or slots that receive color information). In one exemplary case, 16 bits can be allocated to represent the CI data structure 202. In another exemplary case, 24 bits can be allocated to represent the CI data structure 202. Other embodiments provide further different CI data structure sizes, including sizes less than 16 bits and sizes greater than 24 bits.

  In an exemplary, non-limiting case where 24 bits are used, the fields (502-512) can be organized as follows. The first “enum” (integer enumeration of values) VideoTransferFunction can be used to represent the first field 502, which can be assigned 5 bits, and thus the bits of the 24-bit CI data structure 202 23-19 can be occupied. A second enum VideoPrimaries can be used to represent the second field 504, which can be allocated 5 bits, and thus can occupy bits 18-14 of the CI data structure 202. A third enum VideoLighting can be used to represent the third field 506, which can be assigned 4 bits, and thus can occupy bits 13-10 of the CI data structure 202. A fourth enum TransferMatrix can be used to represent the fourth field 508, which can be assigned 3 bits, and can therefore occupy bits 9-7 of the CI data structure 202. The fifth enum NominalRange can be used to represent the fifth field 512, which can be assigned 3 bits, and thus occupy bits 6-4 of the CI data structure 202. Finally, a sixth enum VideoChromaSubsampling can be used to represent the sixth field 512, which can be assigned 4 bits, and thus can occupy bits 3-0 of the CI data structure 202. .

  The remainder of this section provides additional details regarding one exemplary embodiment of the six enums described above. The details provided are exemplary rather than limiting, and other data structures having different syntax can be used.

・ DXVA_VideoTransferFunction
DXVA_VideoTransferFunction enum indicates a conversion function from (non-linear) R′G′B ′ to (linear) RGB. This generally corresponds to the gamma function of the image data. Some transfer functions have corrections to account for 8-bit integer quantization effects. In one exemplary embodiment, 5 bits may be used to represent this enum (eg, bits 23-19 of the 24-bit CI data structure 202).

  An exemplary syntax for the DXVA_VideoTransferFunction enum is specified as follows:

  The first member of this enum indicates that this field is unknown. This field can be set to a predefined value when necessary to proceed with the calculation.

  The member represented by the suffix “10” identifies a linear RGB with gamma = 1.0.

  The members represented by the suffix “18”, “20”, “22”, “28” are, for example, L = 0. . . Represents true gammas of 1.8, 2.0, 2.2, and 2.8, such as L '= pow (L, 1 / gamma) for 1. In the standard BT470-2 SysM, gamma 2.2 is used.

  The member represented by the suffix “22_709” relates to a gamma 2.2 curve having a low and linear range suitable for a format defined by BT1361, BT709, SMPTE296M, SMPTE170M, BT470, SMPTE274M, and the like.

  The member represented by the suffix “22_204M” relates to a gamma 2.2 curve having a low and linear range suitable for SMPTE 240M, interim 274M, etc.

  The member represented by the suffix “22_8bit_sRGB” relates to a gamma 2.4 curve with a low and linear range that can match an exact 2.2 gamma 8-bit curve.

  FIG. 6 shows further exemplary details suitable for the DXVA_VideoTransferFunction enum.

・ DXVA_VideoPrimaries
The DXVA_VideoPrimaries enum lists the primary colors that identify which RGB basis functions are used in the image information 204. In one exemplary embodiment, 5 bits may be used to represent this enum (eg, bits 18-14 of the 24-bit CI data structure 202).

  An exemplary syntax for the DXVA_VideoPrimaries enum is specified as follows:

  The first member of this enum indicates that this field is unknown. A pre-defined value can be set in this field if necessary to proceed with the calculation (eg, the default value can be set to the primary color specified in standard BT709).

  Members with the suffix “BT709” define the primary colors associated with the BT709 standard (this is also applicable to the standards sRGB, scRGB, etc.).

  Members with the suffix “BT470_2_SysM” define the original NTSC primaries.

  Members with suffixes “BT601”, “BT470_2_SysBG”, “SMPTE240M”, and “EBU3213” define the various primary colors associated with these standards.

  Members with the suffix “SMPTE170M” define the analog NTSC primaries (which are currently not used much).

  Members with the suffix “SMPTE_C” define the analog '79 NTSC primaries.

  FIG. 7 provides further exemplary details suitable for the DXVA_VideoPrimaries enum.

・ DXVA_VideoLighting
DXVA_VideoLighting enum describes the desired viewing lighting conditions. This information can also be used to change gamma to produce comparable experiences with different lighting conditions. In one exemplary embodiment, 4 bits can be used to represent this enum (eg, bits 13-10 of the 24-bit CI data structure 202).

  An exemplary syntax for the DXVA_VideoLighting enum is specified as follows:

  The first member of this enum indicates that this field is unknown. This field can be set to a predefined value if necessary to proceed with the calculation (eg, the default value can be set assuming that a dim viewing condition is intended). .

  Members with the suffix “bright” can correspond to outdoor lighting conditions.

  Members with the suffix “office” can correspond to moderate brightness levels associated with home office conditions.

  A member having the suffix “dim” can correspond to a dim brightness level associated with a dimly illuminated viewing condition.

  Members with the suffix “dark” can correspond to dark brightness levels associated with theater viewing conditions.

・ DXVA_VideoTransferMatrix
The DXVA_VideoTransferMatrix enum describes a transformation matrix used to transform the image information 204 from the Y'Cb'Cr 'color space to the (studio) R'G'B' color space. In one exemplary embodiment, 3 bits can be used to represent this enum (eg, bits 9-7 of the 24-bit CI data structure 202).

  An exemplary syntax for the DXVA_VideoTransferMatrix enum is specified as follows:

  The first member of this enum indicates that this field is unknown. A pre-defined value can be set in this field when necessary to proceed with the calculation (eg, the default value includes the standard BT601 of the standard definition image information 204 and the standard BT709 of the high definition image information 204). Can be used to set the transfer matrix used).

  Members with the suffix “BT709” define the transfer matrix used in the BT709 standard.

  Members with the suffix “BT601” define the transfer matrix used in the BT601 standard.

  Members with the suffix “SMPTE204M” define the transfer matrix used in the SMPTE204M standard (this is a high quality standard that is not currently commonly used).

  8 and 9 provide further exemplary details suitable for the DXVA_VideoTransferMatrix enum.

・ DXVA_NominalRange
The DXVA_NominalRange enum describes whether the data includes headroom (value above the standard 1.0 white) and toe room ("super black" below the standard 0.0 black). For example, in order to ensure correct interpretation of the image information 204, it is useful to distinguish between a wide color gamut R′G′B ′ (black dots 16, 16, 16, white dots 235, 235, 235) and normal sRGB. is there. In one exemplary embodiment, 3 bits may be used to represent this enum (eg, bits 6-4 of the 24-bit CI data structure 202).

  An exemplary syntax for the DXVA_NominalRange enum is specified as follows:

  The first member of this enum indicates that this field is unknown. This field can be set to a predefined value when necessary to proceed with the calculation.

  Members with the suffix “Normal” . . 255 (8 bits) or 0. . . 1023 (10 bits) of normalized chroma [0. . . 1] is defined.

  Members with the suffix “Wide” . . 235 (8 bits) or 64. . . Normalized chroma mapped to 940 (10 bits) [0. . . 1] is defined.

・ DXVA_VideoChromaSubsampling
DXVA_VideoChromaSubsampling enum describes a chroma encoding scheme applied to Y′Cb′Cr ′ data. A “cosite” variation indicates that the chroma sample is aligned with the luma sample. 4: 2: 0 data typically has chromas aligned to luma data in one or more directions. 4: 4: 4, 4: 2: 2, and 4: 1: 1 data are co-sited in both directions. In one exemplary embodiment, 3 bits can be used to represent this enum (eg, bits 3-0 of the 24-bit CI data structure 202).

  An exemplary syntax for the DXVA_VideoChromaSubsampling enum is specified as follows:

  The first member of this enum indicates that this field is unknown. This field can be set to a predefined value when necessary to proceed with the calculation.

  Members with the suffix “Progressive Chroma” define chroma samples that are temporarily interpreted as progressive content (eg, from the same frame, not from two temporally offset fields).

  A member with the suffix “Horizontally_Cosited” defines a chroma sample that is horizontally aligned to a plurality of luma samples.

  A member with the suffix “Vertically_Costed” defines a chroma sample that is vertically aligned to a plurality of luma samples.

  Members with the suffix “Aligned ChromaPlanes” define the Pb and Pr (or Cb and Cr) plains as having the same phase alignment. This flag is 0 if the data is co-sited vertically.

  FIGS. 10-12 provide additional exemplary details suitable for the DXVA_VideoChromaSubsampling enum. That is, these figures provide a summary of the location of the chroma sample relative to the luma sample in the normal Y'Cb'Cr 'image information 204.

D. One exemplary application of the color information data structure There are many applications of the coding strategy described in the previous section. FIGS. 13-16 illustrate one exemplary application using an application programming interface (API) called DeinterlaceBltEx.

  In summary, the DeinterlaceBltEx functionality provides logic that allows multiple operations to be performed together as a single operation, such as performing a compositing operation with a deinterlace operation. The composition operation indicates a combination of main image stream information and image substream information. Image stream information refers to the main sequence of image frames. Image substream information refers to auxiliary image information that can be presented together with the image stream of the main image stream information. In one example, the image substream information may correspond to close caption data. Close captioning information is combined with main image stream information to form composite image information for display. Deinterlacing refers to a technique that combines successive image fields created by interlaced operation to provide a progressive (non-interlaced) representation of an image frame. DeinterlaceBltEx functionality also allows other types of operations to be performed simultaneously (instead, the so-called DeinterlaceBlt functionality can be used, which performs the operations identified above in succession. Is configured to).

  In one exemplary embodiment, the DeinterlaceBltEx functionality provides a data structure having an unused portion (or a partially unused portion) that can be used to convey the CI data structure 202. That is, in one exemplary ratio-limiting embodiment, this part can define a 32-bit word. The 24-bit portion of this word can be used to convey the six fields of information shown in FIG. Another part of the existing structure (eg, the remaining 8 bits) may be used to convey other aspects of the image information 204, such as whether the image information 204 is coded in interlaced or progressive format. it can.

  Specifically, color information can serve multiple roles in the context of DeinterlaceBltEx operation. Regarding color space processing issues, the color information specified in the CI data structure 202 must: (a) perform any color space conversion (if any) on the input image information supplied to the DeinterlaceBltEx functionality. (B) which color space must be used to perform various processing tasks on the image information, and (c) which color space is used to supply the output image information. You can specify whether you have to. Possible color spaces that can be selected for these tasks can include all types of RGB related color spaces, all types of luma related color spaces (eg, YUV), and the like. Further, possible color spaces can be in either a linear form or a non-linear (eg, gamma corrected) form.

  For example, in one scenario, one or more streams of image information are converted from a luma-related color space to a linear RGB-related color space. Next, several operations are performed on the image information in the linear RGB related color space to provide an output also in the linear RGB related color space. Thus, this sequence of operations can implement the technique shown in FIG. 4 (described previously). In another scenario, one or more streams of image information may be received and processed in a luma related color space to provide an output also in the luma related color space. In another scenario, one or more streams of image information are transferred from one type of RGB related color space to another type of RGB color space, or from one type of luma related color space to another type of luma related color. Can be converted to space. These are merely illustrative examples of the various processing options enabled by the DeinterlaceBltEx functionality. In any of these cases, the color information in the CI data structure 202 can provide an indication that determines the color space selection applied by the DeinterlaceBltEx functionality.

  Furthermore, using the unique function of the DeinterlaceBltEx functionality, color space conversion can be performed in the same operation as other operations (such as deinterlacing and composition) performed on image information. For example, image information can be converted from a luma-related color space to an RGB-related color space with the same operations as deinterlacing and compositing with a video substream. Alternatively, certain operations performed on the image information can be performed sequentially. For example, in one example fully described below, de-interlacing and compositing can be performed in a single operation within the YUV color space, followed by conversion of the output result to an RGB-related color space. .

  As a final preface comment, DinterlaceBltEx functionality acts on separate image information streams (such as one or more main video streams and / or one or more video substreams) represented in different color spaces. Can do. This image information stream can be associated with its own CI data structure 202. This CI data structure 202 includes information defining how different streams can be converted to a common color space in order to perform operations (deinterlacing, compositing, etc.) on the different streams. Can do.

  With regard to the above introduction, FIG. 13 outlines an exemplary system 1300 that can be used to implement aspects of the video pipeline 200 shown in FIG. The system 1300 can rely on the DirectX® family of technologies created by Microsoft® Corporation of Redmond, Washington, USA. The DirectX family includes DirectX Video Acceleration (DirectX-VA), Direct3D, DirectDraw, and the like. However, the principles described above can be implemented using other types of rendering technologies that operate on other types of technical platforms. System 1300 may represent a personal computer, a gaming machine (such as an Xbox ™ gaming machine from Microsoft® Corporation), or other type of device.

  First, the system 1300 accepts image information from any one of a plurality of sources. For example, the device 1300 may include a network 1302 (such as a remote source coupled to the Internet), all types of databases 1304, all types of computer readable disk media 1306 (optical disks, DVDs, etc.), or other sources 1308 (FIG. 204). Image information from the source 212 associated with the source 212 shown in FIG. In any case, the received information may include a combination of image information and audio information. A demultiplexer unit 1310 separates audio information from image information. Audio processing functionality 1312 processes the audio information.

  An image decoder 1314 processes the image information. Image decoder 1314 can convert the compressed image information from its received format to some other format and can perform an initial resize operation or other operation on this information. The output of the image decoder 1314 can include so-called pure image information as well as image substream information. Pure image information constitutes the main image stream that is rendered on the display device. Image substream information relates to pure image information, such as closed captioning information, all types of graphical overlay information (such as various graphical editing controls), various types of subimages presented by DVD players, etc.) All supplementary information can be configured.

  In one exemplary embodiment, the Video Mixing Glendala (VMR) module 1316 performs the central role of processing image information received in this manner. In overview, the VMR module 1316 interacts with the graphics interface 1318 and the display driver 1320, which controls the graphics module 1322. As will be described in detail below, this interaction includes probing the functionality of the graphics module 1322. This interaction also includes coordinating the processing of image information by graphics interface 1318, display driver 1320, and graphics module 1322. In one embodiment, the graphics interface 1318 can be implemented using DirectDraw functionality provided by DirectX from Microsoft Corporation. DirectDraw acts in this context as a message conduit that communicatively couples VMR module 1316 to graphics module 1322. The graphics module 1322 itself may constitute a fixed module in a computer or similar device, or may constitute a removable unit such as a graphics card. The vertical chain of functionality represented by the VMR module 1316, the graphics interface 1318, the display driver 1320, and the graphics module 1322 can be divided into user mode and kernel mode. User mode refers to an aspect of programming functionality that can be operated by a user through various interfaces. Kernel mode represents an aspect of programming functionality that cannot be manipulated directly by the user.

  The graphics module 1322 itself includes one or more graphics processing units (GPUs) 1324. The GPU 1324 is generally a processing device similar to a CPU. The GPU 1324 is generally assigned to an information intensive rendering task that is repeatedly performed by the performing device 1300. By assigning these repetitive or information intensive tasks to GPU 1324, a CPU (not shown) is freed up to perform other tasks, thus improving the performance of device 1300. Two exemplary tasks that this embodiment assigns to GPU 1324 are deinterlacing and rate conversion. These functions are represented by deinterlace logic 1326 and rate converter logic 1328. Deinterlace logic 1326 combines multiple fields of image information together to form a frame of image information. Rate converter logic 1328 changes the frame rate of the sequence of image frames. The GPU 1324 can perform a number of additional processing tasks.

  GPU 1324 can interact with local memory 1330 associated with graphics module 1322. This local memory 1330 can serve multiple storage related purposes. For example, the memory 1330 can store the final image surface that is transferred to the display device 1332.

  FIG. 13 shows that the CI data structure 202 can be passed along the set of functional components shown in FIG. 13 in a manner similar to that described with respect to FIG. That is, the CI data structure 202 has a uniform structure when passed through the pipeline defined by the system 1300. Individual functional components in system 1300 can use the color information in CI data structure 202, or potentially use the color information in CI data structure 202 for the benefit of downstream functional components. Can be supplemented. For example, with respect to color space processing issues, the color information includes: (a) which color space conversion (if any) must be performed (eg, YUV to RGB) and how this conversion can be performed. (B) which color space must be used to perform various processing tasks on the image information, and (c) which color space must be used to provide the output image information. Or can be defined. FIG. 13 illustrates that multiple CI data structures 202 play a role in the processing performed by system 1300, for example, each stream of image information processed by system 1300 is its own. A data structure 202 can be associated. These CI data structures 202 can coordinate the conversion of different streams of image information to a common color space.

  Further details regarding exemplary forms in which the CI data structure 202 can be integrated into the DeinterlaceBltEx framework are provided below.

  First, FIG. 14 shows an outline 1400 of the operation of the apparatus 1300 of FIG. At step 1402, the VMR module 1316 queries the display driver 1320 and the graphics module 1322 as to which processing mode is supported. After receiving the response, at step 1404, the VMR module 1316 sends another question to find more specific information about the capabilities of the display driver 1320 and the associated graphics module 1322. Steps 1402 and 1404 are described in more detail below under the heading “Preliminary Information Processing”.

  After investigating the functionality of the connected hardware and associated interfaces, the VMR module 1316 opens the image stream object (at step 1408) so that the image information and control information can be transferred to the hardware. . Thereafter, at step 1408, the VMR module 1316 coordinates the execution of one or more image processing functions by hardware (eg, by the graphics module 1322). One such function is deinterlacing. Another such function is substream synthesis. Depending on the information received, deinterlacing can be combined with compositing as described above, or either function can be performed separately. For example, when progressive image information is received, there is no need to perform deinterlacing, in which case the VMR module 1316 simply changes the size of the object and adds image substream information or other functions. Or a combination of functions can be performed. In addition to deinterlacing and synthesis, a number of other functions are implemented.

  Finally, in step 1410, the VMR module closes the image stream opened in step 1406. This step 1410 may be in response to a command provided by the user or in response to running out of stream image information.

  In the next discussion, more detailed information about the selected steps referenced above will be presented.

Initialization Step In step 1402 described above, the VMR module 1316 asks the display driver 1320 what processing functionality is supported for the input image format. When display driver 1320 responds, VMR module 1316 sends a request for more specific information regarding the requirements of display driver 1320 for a particular mode. Display driver 1320 responds by specifying various information in the information structure. Such information identifies the number of required forward reference samples, the number of required reverse reference samples, the format of the output frame, and so forth. This information structure also includes a flag that indicates whether support for a combination of deinterlacing and compositing is supported by the graphics module 1322 and the associated interface. This flag is referred to as the DXVA_ImageProcess_Sub-streams flag in one exemplary embodiment.

  In addition, to properly support combined deinterlacing and compositing, graphics module 1322 and associated interfaces and drivers can independently stretch (horizontally) both the deinterlaced image frames and the supplied image subframes. And / or vertically). This is required in one embodiment because the pixel aspect ratio of the main image and the video substream are different and may not be square in nature. In addition to the DXVA_VideoProcess_Sub-streams flag, the display driver 1320 can communicate the ability to handle this functionality by returning the DXVA_VideoProcess_StretchX flag and the DXVA_VideoProcess_StretchY flag that convey the ability to stretch the image.

  In addition, DeinterlaceBltEx DDI supports extended color information for each of the source and destination surfaces (where “source surface” defines the input image information and “destination surface” or “target surface” is created by the DeinterlaceBltEx operation. Define output image information). The display driver 1320 can indicate the level of support it has for this new color information via various color related flags, such as the next flag.

  Support for the DXVA_VideoProcess_SubStreamsExtended flag indicates that the system 1300 can perform color adjustments on the source video information and substream information when the video information is deinterlaced, combined with the substream information, and written to the destination surface.

  Support for the DXVA_VideoProcess_YUV2RGBExtended flag allows the system 1300 to use color space conversion options (eg, , YUV to RGB).

  Support for the DXVA_VideoProcess_AlphaBlendExtended flag indicates that the system 1300 can perform an alpha blending operation with the destination surface when the deinterlaced and synthesized pixels of the image information are written to the destination surface.

Deinterlacing step The VMR module 1316 coordinates the deinterlacing and compositing execution by the graphics module 1322 using the DeinterlaceBltEx functionality described above. Specifically, this DeinterlaceBltEx functionality is implemented as a single call to the display driver 1320, even when technically involved with multiple basic operations (color space conversion, deinterlacing, compositing, etc.) can do. The DeinterlaceBltEx functionality writes the output of the operation to the specified destination surface.

  Specifically, the VMR module 1316 transfers the following data structure to the display driver 1320 to implement the DeinterlaceBltEx functionality.

  In this structure, the rtTargetFrame parameter identifies the temporal position of the output frame within the sequence of input frames. If only deinterlacing is being performed, the target time must match one of the reference sample rtStart time or the midpoint time (rtStart + rtEnd) / 2. If frame rate conversion is required, the rtTargetFrame time may be different from either the rtStart or midpoint time of the reference sample.

  The prcTargetRect parameter identifies the location in the destination surface to which the DeinterlaceBltEx operation writes. In one embodiment, the output must be limited to pixels within this rectangle. That is, in any way, all pixels in prcTargetRect must be written, and pixels outside prcTargetRect must not be changed.

  The BackgroundColor parameter identifies the background color on which all video streams and substreams are composited.

  The DestinationFormat parameter includes extended color information about the destination surface.

  The Destination Flags parameter includes a set of flags indicating changes in destination related parameters from a previous call to DeinterlaceBltEx. These flags reflect changes to the background color, extended color information, target rectangle, or planar alpha parameters and are provided to help optimize driver code.

  The destination surface can be an off-screen plane surface located in video memory (eg, local memory 1330). The pixel format of the destination surface may be that indicated by the d3dOutputFormat field of the data structure returned to the VMR module 1316 during the initialization step. In one exemplary embodiment, the destination surface specified by this structure may be in a Y-related color space (eg, YUV).

  The lpDDSrcSurfaces parameter points to an array of DXVA_VideoSample2 structures. The SampleFormat field of this structure indicates whether the sample is a reference for deinterlacing operation or a video substream sample that needs to be combined with a deinterlaced video frame. A video substream sample must have a DXVA_SampleSub-stream value for its sample format.

  Specifically, an exemplary VideoSample2 structure is identified below.

  In addition to indicating whether the sample is interlaced or progressive, the SampleFormat field of the data structure identified above defines where the CI data structure 202 can be represented. Adding the CI data structure 202 to the SampleFormat field does not increase the size of the VideoSample2 data structure and does not require reengineering of the DeinterlaceBltEx functionality in other ways. Thus, “pushing” the CI data structure 202 into this data structure represents a particularly efficient way of conveying color information through the video pipeline that includes the graphics module 1322.

  For the reference video sample, the rtStart field and the rtEnd field indicate the temporal position of the sample. For video substream samples, these fields are cleared to zero.

  The source rectangle and the destination rectangle are used for sub-rectangular deinterlacing or sub-rectangular stretching. Note that the stretching of the video substream is independent of the video stream and that support for stretching is mandatory in one embodiment. For the paletted video substream pixel format, the PAL field of the DXVA_VideoSample2 structure contains 16 palette entries that can be used when synthesizing the substream samples. For non-paletted pixel formats, these palettes are cleared to 0 and can be ignored.

  Each input sample includes a set of flags indicating changes in the current sample from the previous sample. These flags reflect changes to each sample's palette, color information, source rectangle, and destination rectangle and are provided to help optimize driver code.

  Continuing the description of the DeinterlaceBltEx structure, the dwNumSurfaces parameter indicates the number of elements of the lpDDSrcSurface array. The video reference sample is the head of the array, followed by the video substream in Z order. In one exemplary embodiment, the number of video substreams passed to the driver can range from 0 to 15. Most often, 0 or 1 video substream is passed to the driver when DeinterlaceBltEx is called.

  Finally, the Alpha parameter indicates the planar transparency value that can be applied to the composite background color, video stream, and substream images when written to the destination surface.

  Note that the VMR module 1316 can invoke the DeinterlaceBltEx functionality when progressive video and multiple image substreams are received. This can occur, for example, when the VMR module 1316 is used for DVD playback including a mixture of progressive and interlaced video. In this case, the display driver 1320 does not attempt to deinterlace the video stream (since it is already in a progressive format), and the VMR module 1316 combines the video stream with a given substream, each as desired or required. It can be configured to change the size of the stream (if a deinterlace mode requiring multiple reference samples is used with progressive video, multiple reference samples are sent to the display driver 1320. However, each reference sample refers to the same progressive video frame).

  To conclude the discussion of DeinterlaceBltEx functionality, FIGS. 15 and 16 illustrate how this functionality combines synthesis and deinterlacing operations together into one operation.

  FIG. 15 is an overview of certain operations that can be provided by the system 1300 of FIG. 13 according to one exemplary, non-limiting scenario. This figure shows that compressed image information 1502 can be supplied to image decoder 1504 to create a current frame of uncompressed image information along with CI information. Further, the current frame and CI information of uncompressed image information 1506 is sent to deinterlace logic 1508 along with one or more previous samples of the uncompressed image (eg, previous uncompressed images 1510 and 1512). The logic 1508 of FIG. 15 also functions to add image substream information (eg, from substream information 1514, 1516, etc.) to the deinterlaced image information. In other words, logic 1508 effectively combines the deinterlacing operation with the substream combining operation. Further, as described in detail below, logic 1508 performs the two functions so that the two functions can be performed in a single memory read / write operation rather than multiple passes. That is, the system 1500 of FIG. 15 requires only a single storage (eg, a single memory read / write transaction) (ie, in one exemplary case, only a single read from memory). I need).

  The bandwidth efficiencies described above can be used to image information (eg, image information 1506, 1510, and 1512) and image substream information (eg, 1514, 1516, etc.) different textures used by the GPU module 1322. This can be achieved by assigning to units. In the most common application, the texturing unit is assigned an image surface that is manipulated in the process of 3D rendering application. For example, “texture” generally refers to an image that is “pasted” to the surface of geometric primitives (eg, triangles) that form part of the rendered three-dimensional scene. These different texture surfaces are assigned to different so-called texturing units. The system shown in FIG. 15 assigns image information (eg, 1506, 1510 and 1512, etc.) and image substream information (1514, 1516, etc.) to each texturing unit, thus deinterlacing and compositing operations. Achieve similar efficient performance. Specifically, GPU logic essentially reads and processes information from each of the collections of texturing units simultaneously, rather than in a staggered serial form. This reduces bus congestion at the implementing device and better enables the implementing device to present image information and associated image substream information at an optimal frame rate.

  In other embodiments, the logic 1508 can perform other operations, such as color space conversion. For example, logic 1508 can convert one or more streams of image information from a luma-related color space (eg, YUV) to an RGB-related color space (eg, linear RGB or non-linear RGB). Instead, logic 1508 performs a conversion, such as from one type of luma-related color space to another type of luma-related color space, or from one type of RGB-related color space to another type of RGB-related color section. Can do. In one exemplary scenario, logic 1508 can be used to implement aspects of the features shown in FIG. 4 (top), where image information is converted from a luma-related color space to a linear RGB. Transformed, and then processing is performed on the image information in the linear RGB color space (deinterlacing and other processing in the linear RGB color space thereafter or combined with it). In other words, the output surface of logic 1508 can be represented in an RGB-related color space (such as linear RGB).

  In another scenario, the output of logic 1508 processes the image information in a luma-related color space (eg, YUV), and the combined de-interlaced image information 1518 in the luma-related color space instead of the RGB color space. Can be made. Logic 1508 or other module (not shown) can then convert the luma related color information to RGB related information, which is then rendered on a display to render rendered image information 1520. Can be made. Delaying the conversion from the luma-related color space to the RGB-related color space has potential bandwidth savings. For example, in one embodiment, luma related color information may be presented using 12 bits. In contrast, in one embodiment, RGB information requires 32 bits for representation. Thus, by performing tasks such as compositing using luma related information rather than RGB information, bus traffic associated with this transaction is reduced.

  The logic 1508 can perform other types of processing functions, such as image resizing. Nevertheless, the logic 1508 need not perform each of the processing operations described above. For example, if the image information is already in progressive format, logic 1508 is dedicated to simply changing the size of the image information or performing other desired operations on the image information (such as color space conversion). be able to.

  Furthermore, in other embodiments, one or more operations may be performed sequentially, for example as separate operations, rather than a single operation. The alternative logic 1508 'of FIG. 15 represents such an embodiment. For example, the DeinterlaceBlt functionality is adapted to perform image processing in this manner.

  In any case, the color information contained in the CI data structure 202 provides the necessary instructions for converting the image information in the form desired for the particular application. Specifically, logic 1508 can examine the CI data structure 202 associated with each of the streams of image information when processing multiple streams of image information. Using such per-stream CI data structures 202, different streams of image information (which may be represented in different respective color spaces) are put into a common color space, such as a linear RGB color space. Can be converted. This feature is beneficial because it eliminates or reduces the need to store temporary image information represented in one or more intermediate color spaces. In the techniques described above, logic 1508 can receive, transform, and write input image information to, for example, information that is displayed by an output device (eg, a desktop provided by a personal computer). There is no need to provide additional synthesized YUV image information before.

  The emphasis on the use of CI information 202 to provide color space related indications is exemplary only. CI information 202 provides a variety of other information that can be used to determine the operations shown in FIG. For example, the CI information 202 can be used in a deinterlacing operation to correctly interpolate the corresponding progressive image.

  FIG. 16 illustrates one exemplary technique that can be used to achieve the joint processing functionality enabled by logic 1508. That is, FIG. 16 illustrates one technique for performing deinterlacing and combining in a single operation.

  FIG. 16 shows an exemplary organization 1600 of texturing units and associated memory. These elements are used to simultaneously process the main image information in the image substream information. The memory and texturing unit may be implemented using local memory 1330 and / or shared memory 1334 associated with graphics module 1322 shown in FIG.

  As noted above, interleaving and compositing joint operations can be performed while the image information is still in a Y-related color space, such as YUV format. Thus, in this exemplary non-limiting scenario, VMR module 1316 allocates a portion of memory to store this Y related information. A first block 1602 can be allocated to store Y information, a second block 1604 can be allocated to store a first chroma component (eg, U information), and a third block 1606 can be A second chroma component (eg, V information) can be assigned to store. More bits than the two chroma components are assigned to the Y information. For example, for an image containing 720 × 480 pixels, a 720 × 480 byte block is assigned to the Y information store, a 360 × 240 byte block is assigned to the first chroma component store, and a 360 × 240 byte block is assigned. , Can be assigned to the second chroma component store. Finally, the block of memory 1608 can be allocated to a store of substream information (close captioning information, DVD subimage display information, various types of graphical icons, etc.).

  In the exemplary organization 1600 shown in FIG. 16, only four texturing units are shown (1610, 1612, 1614, and 1616). However, other embodiments include more than four units. Texturing unit 1610 is assigned to processing the image input surface associated with memory block 1602 (eg, Y information), and texturing unit 1612 is responsible for image input surface associated with memory block 1604 (eg, the first chroma component, For example, U) is assigned to processing, and texturing unit 1614 is assigned to processing the image input surface associated with memory block 1606 (eg, the second chroma component, eg, V). Texturing unit 1616 is assigned to processing the image input surface (eg, substream information) associated with memory block 1608. The memory blocks (Y, chroma 1, chroma 2, and substream information) are separate but need not be contiguous in memory. Additional memory blocks and texturing units can be provided to process additional image reference samples and / or additional image substreams. For example, an application that includes two previous reference streams should require at least nine texturing units (eg, three units for the current sample, six units for two reference samples).

  Finally, FIG. 16 generally illustrates GPU processing logic 1618 associated with GPU 1324 of FIG. GPU processing logic 1618 interacts with the texturing unit. GPU processing logic 1618 may perform deinterlacing, frame rate conversion, and / or other tasks.

  The use of YUV related information to describe the operation of FIG. 16 is merely exemplary. Similar benefits can be obtained by performing processing operations in an RGB-related space, such as linear RGB (eg, in the form shown in FIG. 4).

E. Exemplary Computer Environment In one exemplary embodiment, the aspects of the processing illustrated in the previous figures may be performed by a computing device. In this case, FIG. 17 provides information about an exemplary computer environment 1700 that can be used to implement aspects of the processing shown in the previous drawings.

  The computing environment 1700 includes a general purpose type computer 1702 and a display device 1704. However, the computing environment 1700 can include other types of computing devices. For example, although not shown, processing functionality integrated into computing environment 1700 in handheld devices, laptop devices, set-top boxes, game consoles, video processing / presentation devices (eg, television, DVR, etc.) , Mainframe computers and the like can be included. Further, FIG. 17 illustrates elements of a computer environment 1700 grouped together for ease of discussion. However, the computing environment 1700 can use a distributed processing configuration. In a distributed computing environment, computing resources can be physically distributed throughout the environment.

  Exemplary computer 1702 includes one or more processors or processing units 1706, system memory 1708, and bus 1710. Bus 1710 connects various system components together. For example, bus 1710 connects processor 1706 to system memory 1708. The bus 1710 uses any type of bus structure or combination of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor bus or a local bus, using any of a variety of bus architectures. Can be implemented.

  Computer 1702 can also include a variety of computer readable media, including various types of volatile and non-volatile media, each of which can be removable or non-removable. For example, system memory 1708 includes computer readable media in the form of volatile memory, such as random access memory (RAM) 1712, and non-volatile memory, such as read only memory (ROM) 1714. ROM 1714 includes an input / output system (BIOS) 1716 that includes basic routines that help transfer information between elements within computer 1702 such as during startup. The RAM 1712 typically includes data and / or program modules in a form that can be quickly accessed by the processing unit 1706.

  Magnetics that read from and write to non-removable non-volatile magnetic media to other types of computer storage media, hard disk drive 1718, removable non-volatile magnetic disk 1722 (eg, "floppy disk"), and write to it Included is an optical disk drive 1724 that reads from and / or writes to a removable non-volatile optical disk 1726, such as a disk drive 1720, CD-ROM, DVD-ROM, or other optical media. The hard disk drive 1718, magnetic disk drive 1720, and optical disk drive 1724 are each connected to the system bus 1710 by one or more data media interfaces 1728. Alternatively, hard disk drive 1718, magnetic disk drive 1720, and optical disk drive 1724 may be connected to system bus 1710 by a SCSI interface (not shown) or other coupling mechanism. Although not shown, computer 1702 includes a magnetic cassette, or other magnetic storage device, flash memory card, CD-ROM, digital versatile disk (DVD), or other optical storage, electrically erasable programmable read-only memory. Other types of computer readable media such as (EEPROM) may be included.

  In general, the computer-readable media identified above provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for use by computer 1702. For example, operating system 1730, application module 1732, other program modules 1734, and program data 1736 may be stored on a readable medium.

  The computer environment 1700 can include a variety of input devices. For example, the computer environment 1700 includes a keyboard 1738 and a pointing device 1740 (eg, a “mouse”) that enter commands and information into the computer 1702. The computer environment 1700 may include other input devices (not shown) such as a microphone, joystick, game pad, satellite dish, serial port, scanner, card reading device, digital camera, video camera, and the like. An input / output interface 1742 couples the input device to the processing unit 1706. More generally, input devices can be coupled to the computer 1702 via any type of interface and bus structure, such as a parallel port, serial port, game port, USB (universal serial bus) port, and the like.

  The computer environment 1700 also includes a display device 1704. Video adapter 1744 couples display device 1704 to bus 1710. In addition to the display device 1704, the computer environment 1700 can include other output peripheral devices such as speakers (not shown), printers (not shown), and the like.

  Computer 1702 operates in a networked environment using logical connections to one or more remote computers, such as remote computing device 1746. Remote computing device 1746 may include all types of computing equipment including general purpose personal computers, portable computers, servers, game consoles, network expansion devices, and the like. The remote computing device 1746 can include all or a subset of the features described above with respect to the computer 1702.

  Any type of network 1748 may be used to couple the computer 1702 to the remote computing device 1746, such as a LAN, WAN, or the like. Computer 1702 is coupled to network 1748 via network interface 1750, which may use broadband connectivity, modem connectivity, DSL connectivity, or other connectivity strategies. Although not shown, the computing environment 1700 can provide wireless communication functionality for connecting a remote computing device 1746 and a computer 1702 (eg, via a modulated radio signal, a modulated infrared signal, etc.). ).

  Finally, examples were presented as alternatives in this disclosure (eg, Case A or Case B). Furthermore, this disclosure includes cases (eg, Case A and Case B) that combine alternatives in a single embodiment, although not always explicitly mentioned in all cases.

  In addition, multiple features have been described herein by first identifying exemplary problems that these features can address. The form of this description does not constitute an admission that others have reviewed and / or expressed the problem in the manner specified herein. The understanding and representation of problems that exist in video coding techniques must be understood as part of the present invention.

  More generally, although the invention has been described in language specific to structural features and / or methodological operations, the invention as defined in the claims is not necessarily limited to the specific features or operations described. I want you to understand. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed invention.

FIG. 2 illustrates an example video pipeline and associated problems. FIG. 3 illustrates an improved exemplary video pipeline with functionality for processing data structures that provide color information. 3 is a flow diagram illustrating an exemplary method for processing image information using the data structure introduced in FIG. FIG. 2 is a block diagram illustrating a novel technique for converting image information into a progressive linear RGB format and then performing processing on the image information in that format. FIG. 3 shows an overview of an exemplary data structure used in the system of FIG. FIG. 6 shows various exemplary formatting information that can be referenced by the data structure of FIG. FIG. 6 shows various exemplary formatting information that can be referenced by the data structure of FIG. FIG. 6 shows various exemplary formatting information that can be referenced by the data structure of FIG. FIG. 6 shows various exemplary formatting information that can be referenced by the data structure of FIG. FIG. 6 shows various exemplary formatting information that can be referenced by the data structure of FIG. FIG. 6 shows various exemplary formatting information that can be referenced by the data structure of FIG. FIG. 6 shows various exemplary formatting information that can be referenced by the data structure of FIG. FIG. 3 illustrates one exemplary system that implements the general features of FIG. 14 is a flow diagram illustrating an exemplary method of operation of the system of FIG. FIG. 14 illustrates details of exemplary composition functionality that can be used in the system of FIG. FIG. 14 illustrates details of exemplary composition functionality that can be used in the system of FIG. FIG. 3 illustrates an exemplary computing environment for implementing aspects of the system illustrated in FIG. 2 and the like.

Claims (25)

A method of processing image information having color content represented in a specified format, comprising:
Providing a color information data structure including color information defining at least one aspect of the specified format;
Passing the color information data structure along with the image information to at least one functional component used to process the image information;
Processing the image information with the at least one functional component based on the color information in the color information data structure; and
It said color information data structure, the related image information, gamma, look including an image light field to convey information including the brightness, and contrast,
The at least one functional component determines at least one other aspect of the specified format and adds the at least one other aspect to at least one unknown value in the color information data structure ; Feature method.   The at least one functional component belongs to a group of functional components that define a video processing pipeline, and the color information data structure is passed through the video processing pipeline for use by downstream functional components. The method of claim 1, characterized in that: The video processing pipeline is
An operation of converting input image information in the luma-related color space into linear image information in the RGB-related color space;
The method of claim 2, comprising performing an operation on the linear image information in the RGB-related color space to produce processed information.   The method of claim 3, wherein the converting comprises converting the input image information from an interlaced format to a progressive format prior to the processing operation. The video processing pipeline further includes:
The method of claim 3, comprising performing an operation of converting the processed information into non-linear image information in a luma-related color space.   6. The method of claim 5, wherein the conversion of the processed information comprises converting the processed information from a progressive format to an interlace format.   The method of claim 2, wherein the video processing pipeline performs color conversion, and the color information in the color information data structure determines the color conversion. The method of claim 1, further comprising the step of initializing the specified defaults the unknown value in the color information data structure.   The method of claim 1, wherein passing the color information data structure comprises passing the color information data structure in an unused field of an existing data structure.   The method of claim 1, further comprising setting a flag indicating that the at least one functional component is capable of processing the color information data structure.   The color information data structure includes a video transfer function field that conveys information about a transfer function used to convert linear image information to non-linear image information and / or vice versa. The method according to 1.   The method of claim 1, wherein the color information data structure includes a primary color field conveying a primary color associated with the image information.   The method of claim 1, wherein the color information data structure includes a transfer matrix field that conveys information about a transfer matrix that can be used to change the color space of the image information.   The method of claim 1, wherein the color information data structure includes a nominal range field that conveys information about potential toerooms and headrooms associated with the image information.   The method of claim 1, wherein the color information data structure includes a video chroma sampling field that indicates how chroma samples in the image information are sampled and aligned with respect to associated luma samples. The color information data structure is:
A video transfer function field that conveys information about the transfer function used to convert linear image information to non-linear image information and / or vice versa;
A primary color field for transmitting primary colors related to the image information;
A transfer matrix field that can be used to change the color space of the image information;
A nominal range field that conveys information about potential towrooms and headrooms associated with the image information;
The method of claim 1, further comprising: a video chroma sampling field that indicates how chroma samples in the image information are sampled and aligned with respect to associated luma samples.   One or more computer-readable recording media having stored thereon computer-executable instructions for causing a processor to perform the method of claim 1. A processor;
A computer-readable recording medium storing computer-executable instructions for causing the processor to execute each of the providing step, the passing step, and the processing step according to claim 1. . A processor;
17. A computer readable storing computer executable instructions for causing the processor to perform each of the providing, passing and processing steps using the color information data structure of claim 16. And a recording medium. A system for processing image information having color content represented in a specified format,
Includes multiple functional components that define the video processing pipeline,
Each of the plurality of functional components is configured to process a color information data structure including color information defining at least one aspect of the specified format;
It said color information data structure, the related image information, gamma, look including an image light field to convey information including the brightness, and contrast,
Each of the plurality of functional components determines at least one other aspect of the specified format and adds the at least one other aspect to at least one unknown value in the color information data structure. A system characterized by The color information data structure is:
A video transfer function field that conveys information about the transfer function used to convert linear image information to non-linear image information and / or vice versa;
A primary color field for transmitting primary colors related to the image information;
A transfer matrix field that can be used to change the color space of the image information;
A nominal range field that conveys information about potential towrooms and headrooms associated with the image information;
21. The system of claim 20 , further comprising at least one of a video chroma sampling field that indicates how chroma samples in the image information are sampled and aligned with respect to associated luma samples. A method of processing image information having color content represented in a specified format, comprising:
Providing a color information data structure including color information defining at least one aspect of the specified format;
Passing the color information data structure along with the image information to at least one functional component used to process the image information;
Converting input image information in a luma-related color space into linear image information in an RGB-related color space;
Performing processing on the linear image information in the RGB-related color space to produce processed information;
Converting the processed information into non-linear image information in a luma-related color space;
The color information data structure includes an image light field that conveys information related to the image information, including gamma, brightness, and contrast ;
The at least one functional component determines at least one other aspect of the specified format and adds the at least one other aspect to at least one unknown value in the color information data structure;
Using the color information specified in the color information data structure, the at least one functional component converts to the linear image information, performs the process, or converts to the non-linear image information Determining at least one of: The converting step includes:
Converting the input image information from an interlaced format to a progressive format before performing the process;
23. The method of claim 22 , comprising, after the processing step, converting the processed information from a progressive format to an interlaced format. 23. One or more computer-readable media having stored thereon computer-executable instructions for causing a processor to perform the method of claim 22 . A processor;
23. A computer-readable record storing computer-executable instructions for causing the processor to execute each of the step of converting the input information according to claim 22 , the step of executing the processing, and the step of converting to the non-linear image information. A device comprising: a medium.

Images