RU2402811C2 - Image processing using linear parametres of lighting apparatus and other image processng enhancements - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: processing involves scaling, creation of composite images, alpha blending, edge detection etc. A more specific realisation describes strategies for processing image information which: a) is linear; b) lies in the colour space RGB; c) has high accuracy (e.g. provided through floating-point representation); d) is row-wise; and e) is full-channel. Other enhancements provide strategies for: a) processing image information in pseudolinear space in order to increase rate of processing; b) realising an improved error variance technique; c) dynamic calculation and use of filtration nuclei; d) optimising pipelined code generation; and e) realising different processing tasks using novel pixel shader techniques.

EFFECT: reduced image distortion.

20 cl, 21 dwg

Description

Cross reference to related applications

This application is a partial continuation of U.S. Patent Application Serial Number 10/902325 ('325 Application), registered July 29, 2004 and entitled "Strategies for Processing Image Information Using a Color Information Data Structure" by Glenn F. Evans and Stephen J. Estrop. Application '325, in turn, claims the priority of Provisional application for US patent number 60/492029 (Application' 029), filed August 1, 2003. Applications '325 and' 029 are fully incorporated herein by reference.

FIELD OF THE INVENTION

The subject of this application is related to image information processing strategies, and in a more specific implementation, to video image information processing strategies using a video processing pipeline.

State of the art

Sample video processing issues

The video processing technology used today can be explained by tracking the evolution of this technology over time. Separate attributes were added at various stages of evolution to solve the problems that the industry was facing at that moment. To maintain compatibility and consistency, new technologies may have retained some of these attributes, even if the problems for which the attributes were developed have since disappeared. As a result, modern technology can be considered as the accumulation of similar signs that appear successively, reflecting the system of previous problems that the industry faced at different times, agreements between standardization groups, changes in technological limitations and opportunities, etc.

One consequence of the above-described nature of video processing technology is that those skilled in the art have created ingrained thinking stereotypes regarding certain aspects of video processing technology. There are well-established beliefs regarding how to interpret the video information, and well-established beliefs about how to "process" this video information correctly. The authors of this invention believe that many of these established norms are not reasonable and should be reviewed.

The main one of these established notions is that video information should usually be processed in the form in which it is received from a broadcast source, storage medium (for example, a DVD-ROM) or other medium. Nevertheless, many video standards were developed without considering that video information should be processed before display. For example, traditional television sets still do not use sophisticated processing functionality; these devices simply receive and display video information. In fact, the form in which video information is received may not provide the possibility of simple and effective processing of this information.

As a result, the direct application of standard processing algorithms to many generally accepted forms of video information creates various image distortions. Specialists in the art in some cases paid attention to these distortions. However, instead of exploring the basic premises of the techniques used, professionals often resorted to local fixes to fix problems. These solutions can hide problems in certain application-specific situations, but do not solve problems in general.

For example, video information is often received through a video processing pipeline in a form that is non-linear, interlaced, color-downsampled, and is expressed in a specific embodiment associated with the brightness of the color space (for example, Y'U'V information). (The term “non-linear” means here that there is a non-linear relationship between the input signal and the resulting output brightness generated from this signal; other terms in the previous sentence are fully construed below). Specialists may try to apply various linear-type processing algorithms to this information in order to modify it in a predetermined way, for example, by resizing video information, combining video information with other information (for example, creating composite images), etc. The authors of this invention believe that many of these algorithms do not provide optimal and even correct results when processing video information of this nature. Working only with interlaced, color-subsampled 4: 2: 2 or 4: 2: 0 information (which is described below) leads to negative results. For example, processing information in 4: 2: 2 or 4: 2: 0 can lead to the spread of errors through various stages of the video processing pipeline.

Defects in the processed results are expressed in various hindrances, which may be obvious (or not) to the naked eye. In addition, those skilled in the art probably noticed poor results, but did not identify the causes. In some cases, this may be due to the inability of specialists to fully understand the complex nature of many video coding standards. In other cases, specialists may not be aware that they use linear algorithms in order to process non-linear information; in fact, in some cases, specialists may mistakenly believe that they are dealing with linear information. In addition, the general basis of the field of video processing technology is aimed at generating image information, and not necessarily intermediate processing and adjustment of this information.

The application of linear algorithms to non-linear information is just one example of the above-rooted ingrained thinking stereotype in the field of video processing technology. As described below, many other techniques that do not generate optimal results have become standard, such as in the case of smoothing.

For example, those skilled in the art may try to eliminate the interference caused by some smoothing quantization algorithms by adding a small amount of random noise to the input image information and then quantizing the resulting image with noise. These techniques estimate the quantization error by subsequently calculating the difference between the noise image and the quantized result. This may have the effect of eliminating anti-aliasing noise, but at the cost of the output image having a higher noise content relative to the amount of random noise added to the original image information.

There are many other examples of established concepts in the field of video processing technology that continue to be used due to habit and acquaintance without recognizing their significant but unobvious flaws. The general idea of the enhancements described in this document includes a review of these inflexible ideas along with the implementation of alternative solutions.

The field of video processing technology has rich terminology. And a brief introduction to specific sections of the video processing technique is outlined to assist readers. For example, some of the terms already used above (linear, interlaced, brightness, color downsampling, etc.) are defined below. As a general subject of terminology, the term “image information” is used herein to represent a wide class of information that can be prepared for viewing as any type of visual output, including, but not limited to, video image information.

The concept of the current level of technology

- Color space and related issues

Colors can be set using three components. The image stream, which is based on the transmission of color content using discrete color components, is referred to as a component video signal. One standard specification defines color using the red, green, and cyan (RGB) components. More formally, the RGB components describe the proportional intensity of the reference lamps, which create a perceived equivalent color for a given spectrum. In general, an RGB color space can be defined by color values associated with color constituents and a white dot. The white dot means the color associated with the reference white color.

Electronic devices that reproduce color images complement the tri-color nature of human vision by providing three types of light sources. Three types of light sources generate different spectral characteristics, which are perceived as different colors by a person. For example, a cathode ray tube (CRT) provides red, green, and cyan phosphors to create different colors, thus constituting a specific variation of the RGB color space described above. Other technologies do not use phosphors, but otherwise reproduce color using light sources that emit at least three types of light.

However, the RGB coding model is not an effective option for transmitting image information and does not fully comply with some outdated standards. Therefore, image information is typically transmitted to the target device using some kind of encoding model other than RGB. Upon receipt, the image information may be internally converted by the display device into an RGB-linked color space for presentation. As described further in the “Matters Related to the Gamma Index” section, the data of each component R, G, and B can be expressed in relation to the pre-gamma-corrected form, referred to as the values of R ′, G ′, and B ′.

(In general, according to the conventions, an apostrophe means non-linear information in this invention.)

A common tactic in this regard is to set the color by referencing the luminance-related component (Y) and the luminance-related components. Brightness generally means the perceived intensity (illumination) of light. Luminance can be expressed in a pre-gamma-corrected form (as described in the section “Questions Related to the Gamma Index”) to obtain its non-linear equivalent, referred to as the “luminance signal” (Y '). The color components define the color content of the image information regarding the luminance signal. For example, in the digital domain, the symbol Cb corresponds to an n-bit integer scaled representation of the difference B'-Y '(typically in the range 127 ... 128 in 8-bit values), and the symbol Cr corresponds to an n-bit integer scaled representation of the difference R'-Y '. The symbol Pb means the analogue equivalent of Cb, and the symbol Pr means the analogue equivalent of Cr. The symbols Pb and Pr can also mean a digital normalized form of Cb and Cr in the nominal range [-0.5 ... 0.5]. Information of component images specified by CbCr and PbPr can be formally provided with an apostrophe (for example, Cb'Cr 'and Pb'Pr') if they represent non-linear information. However, since Pb, Pr, Cr, and Cr always mean non-linear data, the notation with an apostrophe of frequencies is omitted for convenience and uniformity (for example, the notation Y'PbPr is used instead of Y'Pb'Pr ').

Color content can also be transmitted as a composite video signal (instead of the above component video signal). Composite signals combine luminance and color information into a single signal. For example, in the coding system, Y'UV U represents a scaled version of B-Y, and V represents a scaled version of R-Y. These luminance and chrominance signal components are then processed to provide a single signal. The Y'IQ coding system defines another composite coding system formed by converting the U and V components in the manner described above. One reason that the industry has historically promoted the use of Y-linked color spaces (Y'CbCr, Y'PbPr, YUV, YIQ, etc.) is that reducing color information in these color spaces can be made easier compared to image information expressed in RGB color space. These color spaces are also backward compatible with older standards designed for black and white image information. The term “luminance-related information” generally means any color space that has a luminance-related component and color-related components, and includes at least all of the aforementioned color spaces.

In general, it is possible to convert color content from one color space to another color space using one or more matrix affine transforms. More formally, the metamerism property makes it possible to express one set of color space coefficients with respect to another set of correspondence functions (in this case, “metamer excitations” mean two spectra that correspond to one set of color space coefficients, and, therefore, are considered to be perceivedly identical, i.e. look like one color).

- Questions related to the gamma indicator

Electron beam tubes (CRTs) do not have a linear characteristic conversion function. In other words, the ratio of the voltage applied to the CRT and the resulting brightness generated by the CRT do not define a linear function. More specifically, the predicted theoretical characteristic of a CRT has a characteristic proportional to a power law of 5/2; those. for a given input voltage V, the resulting CRT brightness L can be calculated as L = V ^2.5 . The conversion function is also referred to as a “gamma function”, and the exponent of the voltage signal is referred to as “gamma”.

On the other hand, when information of video images is captured by a camera or generated by a three-dimensional preparation system for viewing, image information is expressed in a linear RGB color space, which means that there is a linear relationship between the input signal and the output brightness. To resolve the mismatch between the linearity of the camera and the non-linearity of the display, cameras traditionally perform a preliminary adjustment of the signal that they generated by applying gamma inversion. In other words, the camera conversion function (sometimes referred to as the encoding conversion function) is approximately the inverse of the CRT brightness characteristic. The result of applying the transform coding function (or inverse gamma) is to generate gamma-corrected image information that is non-linear. When a non-linear signal is transmitted by the display device, near-linear brightness is generated. In addition, as described above, non-linear (or pre-adjusted) image information is indicated by adding an apostrophe to the components, for example, R'G'B 'or Y'CbCr (where apostrophes for the components Cb and Cr are implied).

Thus, it has become a common and standard practice to store image information in a non-linear (corrected) form of brightness-color. To maintain compatibility, any source that generates a signal that must be displayed on a CRT must also first apply the inverse function to the signal.

We emphasize that the encoding of image information using the conversion function, as a rule, applies a special approximation function to the low voltage part of the function. Those. coding techniques typically provide a linear segment in this part to attenuate the noise effect in the image forming sensor. This segment is referred to as a “linear tail” having a predetermined “rounded slope”. This segment improves the quality of the information in images displayed on actual CRTs because these devices have linear brightness-voltage characteristics of about 0 due to the physical structure of these devices.

- Discretization and synchronization of color information regarding luminance information

Human vision responds more to changes in light intensity than to the chromatic components of light. Coding systems take advantage of this to reduce the amount of chroma information (CbCr), which is encoded relative to the amount of luminance information (Y '). This technique is referred to as chroma subsampling. A numerical designation, usually represented as L: M: N, can be used to express this sampling strategy, where L represents the reference sampling coefficient of the luminance component (V), and M and N mean color sampling (for example, Cb and Cr, respectively) with respect to the discretization of brightness (V). For example, a 4: 4: 4 designation may denote Y'CbCr data in which one chroma sample is provided for each luma sample. A 4: 2: 2 designation can denote Y'CbCr data that provides one color sample for every two brightness samples (horizontal). A 4: 2: 0 designation may denote Y'CbCr data, in which one chroma sample is provided for each cluster of two-on-two brightness samples. A 4: 1: 1 designation can denote Y'CbCr data that provides one color sample for four brightness samples (horizontal).

In cases where the encoding strategy provides more luminance information than chroma information, the decoder can recover the “missing” chroma information by performing interpolation based on the chroma information that is provided. Summarizing, downsampling means any technique that generates fewer image samples compared to the original set of image samples. Upsampling means any technique that generates a larger number of image samples compared to the original set of image samples. Thus, the above interpolation defines the type of upsampling.

Coding strategies also define the way in which chroma samples are spatially synchronized with luma samples. Coding strategies vary in this regard. The color samples are synchronized with the brightness samples so that the color samples are placed directly "above" the brightness samples. This is referred to as spatial alignment. Other strategies place color samples in voids between samples in a two-dimensional lattice of color samples.

- Questions related to quantization

Quantization refers to a methodology by which discrete numerical values are assigned to signal amplitudes of color components (or black and white information). In the digital domain, numerical values encompass a predetermined range (color gamut) of color space values in a predetermined number of steps. Often 255 steps are used, for example, to describe each component value so that each component can tolerate a value from 0 to 255. Each color value is often expressed using 8 bits.

Converting from a high precision number to a lower precision number can sometimes generate various image distortions. Various error dispersion algorithms have been developed to solve this problem, such as the Floyd-Steinberg algorithm. Error dispersion algorithms make it possible to distribute errors generated by rounding during quantization to neighboring pixel points. Additional background information regarding the Floyd-Steinberg algorithm is presented in the detailed description below.

- Matters related to comparing interlaced and row-wise representations

Initially, television sets only displayed black and white information in progressive scan from top to bottom. Today, traditional television signals are interlaced. When interlaced, the first field of the video frame is captured, and then after a short period of time - the second field of the video frame (for example, after 1/50 or 1/60 of a second). The second field is vertically offset slightly from the first field so that the second field captures information in the voids between the scan lines of the first field. Video information is presented by quickly displaying the first and second fields one after another, so that in most cases video information is perceived by a person as one continuous stream of information.

However, computer monitors and other presentation equipment display image information line by line rather than interlaced. Thus, in order for the device to present interlaced information on a computer monitor, it must display progressive frames at the speed of an odd field by interpolating data for the opposite field (the process is referred to as “interleaving”). For example, to display an odd field, it is necessary to interpolate the “missing” data for the spatial position between the lines by analyzing the fields on the other side. The term "line-by-line format" generally refers to all progressive scan image formats.

Image information (for example, from a video camera) is typically stored in interlaced form, for example, when the first field is stored separately (semantically) from the second field. If the image information needs to be simply displayed on an interlaced television display, its Y'UV interlaced information can be transmitted directly to the CRT. A CRT internally converts information into R'G'B 'information and controls the output guns with this signal.

Interlacing is more advantageous because it doubles the actual vertical resolution of the image information. However, interlacing also leads to image distortion. This is due to the fact that objects can move at 60 Hz, but when interlaced, only half of the image is displayed every 30 Hz. The resulting distortion generated as a result of this phenomenon is sometimes referred to as “stretching”. Image distortion is detected to a large extent when displaying rapidly changing video images, when it seems that objects are breaking up into even and odd lines.

Additional information regarding each of the above topics can be found in many introductory publications, for example, in the well-known material Digital Video and HDTV (Morgan Kaufmann Publishers, 2003) by Charles Poyton.

SUMMARY OF THE INVENTION

Strategies for processing image information in a linear form are described in order to reduce the amount of image distortion (compared to processing data in a non-linear form). Exemplary types of processing operations may include scaling, composite image creation, alpha blending, edge selection, etc. In a more specific implementation, strategies are described for processing image information that is: a) linear; b) is in the RGB color space; c) has high accuracy (for example, provided through a floating point representation); d) is line-by-line and e) is full-channel. Other enhancements provide strategies for: a) processing image information in pseudo-linear space to increase processing speed; b) implementing an improved error dispersion technique; c) dynamic calculation and application of filter cores; d) optimizing the generation of pipelined code; and e) implementing various processing tasks using new pixel shader techniques.

Brief Description of the Drawings

Figure 1 shows an example image processing pipeline including processing image information in a linear format.

FIG. 2 shows an example procedure for processing image information in a linear format using the image processing pipeline of FIG. 1.

Figure 3 shows a selection of 4: 2: 0 image information as a subject of discussion of some difficulties in processing this information without introducing distortion into the images.

4 shows an exemplary image processing pipeline module that relates to processing image information in a pseudo-linear format.

FIG. 5 shows an example procedure for processing image information in a pseudo-linear format using the image processing pipeline module shown in FIG. 4.

6 and 7 together explain well-known approaches to performing anti-aliasing and error dispersion.

FIG. 8 shows an exemplary system for providing error smoothing and dispersion that provides excellent results for the models described in connection with FIGS. 6 and 7.

Fig.9 shows an exemplary procedure for smoothing and variance errors using the system of Fig.8.

10 shows an exemplary filtering providing system when performing scaling, including dynamically computing filter cores.

11 shows an exemplary filtering procedure using the system of FIG. 10.

12 shows an example image processing pipeline that may include all of the enhancements described herein.

FIG. 13 shows an example efficient code generating system used to implement the image processing pipeline of FIG. 12, for example, by eliminating code modules that are not required for a particular image processing application.

Fig. 14 shows an exemplary procedure for applying the code optimization system shown in Fig. 13.

FIG. 15 shows an example system for implementing the image processing pipeline of FIG. 12 by using a graphics processing unit (GPU) comprising pixel shader functionality.

Fig. 16 shows a known pixel shader structure.

17 shows the general principles associated with applying texture information to polygons in the context of typical graphical applications.

FIG. 18 shows an exemplary implementation of a 4-tap filter using a pixel shader.

Fig.19 shows an exemplary procedure that describes the operation of the filter of Fig.19.

FIG. 20 shows a more general implementation of aspects of the image processing pipeline of FIG. 1 using a pixel shader.

21 shows an exemplary computing environment for implementing aspects of various features shown in the above-described drawings.

The same numbers are used throughout this specification and in the drawings to refer to like components or features. Sequence numbers 100 mean signs originally laid down in FIG. 1, sequence numbers 200 mean signs originally laid down in FIG. 2, sequence numbers 300 mean signs originally laid down in FIG. 3, etc.

Detailed Description of a Preferred Embodiment

The following description sets forth various strategies for improving the image processing pipeline. Strategies offer new ways of processing image information based on the revision of entrenched beliefs in the art. The first class of enhancements (described in Section A) can abstractly apply to all image processing pipelines. The second class of enhancements (described in Section B) applies more to certain technology-specific or implementation-specific applications for image processing pipelines.

According to one exemplary strategy, image information received by the image processing pipeline is converted to a linear form and then processed in that form. In one exemplary implementation, for example, received luminance-related image information (eg, Y'CbCr) is converted to a linear RGB shape and processed in that form. Exemplary processing tasks may include interleaving, resizing (scaling), composing composite images, alpha blending, selecting edges, sharpening, etc. Processing image information in linear space (as opposed to non-linear space) has clear advantages, since it usually generates output results that have less image distortion.

According to another exemplary strategy, the image information is converted to a full-channel (4: 4: 4) high-precision format and processed in this format, after which the information is quantized to a lower accuracy. Higher precision information can be expressed in floating point format. Processing high-precision full-channel information is advantageous because it generates output results that have less image distortion and can reduce the color resolution drop as the information is processed in the pipeline.

According to another exemplary strategy, special measures are provided for processing 4: 2: 0 interlaced image information. These special measures provide ways to integrate the upsampling operation with the interleaving operation. This strategy has advantages for several reasons. For example, this strategy ensures that 4: 2: 0 information will be processed correctly, for example, without generating a so-called “color defect” processing distortion.

According to another exemplary strategy, special transformation functions can be applied to convert information into a pseudo-linear space instead of theoretically adjusting the linear space. This is followed by the processing of image information in this pseudo-linear space using linear-type algorithms (without the need to modify these algorithms to take into account their application to non-linear signals). This strategy has advantages because it entails mathematical operations that can be performed faster using available processing hardware. This strategy also eliminates the need to modify processing algorithms by including error correction mechanisms in these algorithms. The purpose of error correction algorithms was to minimize negative effects when using linear algorithms in order to work with non-linear data. However, the execution and application of these error correction mechanisms has often become more complex than the original processing algorithms.

According to another exemplary strategy, a unique error dispersion algorithm is used. For each pixel in the information of the source images, the algorithm quantizes the amount specified by the source image, the noise information, and the error vector. This results in a quantized value for this particular pixel. After that, the algorithm calculates the error vector for the next pixel to be processed by calculating the difference between the quantized value and the original value. This strategy is superior to the strategies known in the art, which essentially add noise to the source image information and then quantize the image information with noise. These known techniques further calculate an error vector with reference to image information with noise instead of the original image information. Thus, in contrast to the strategy disclosed in this document, known techniques degrade the information of the original images in proportion to the information about the noise added to it. As an additional improvement, the strategies described in this document use a noise generator having a relative repetition period and pseudorandom characteristics sufficiently similar to noise, so that the noise generator itself does not distort the information in the target images.

According to another strategy, the Katmull-Rum filter option is used to perform scaling operations. The strategy includes calculating the number of filter cores required to resize image information and the number of taps required by each core, and then pre-calculating the filter cores. These filter cores are then applied to image information. More specifically, kernels can be cyclically applied to rows and columns of image information. This strategy has advantages for several reasons. For example, the use of dynamically calculated filter cores reduces some image distortion known in the used applications. Pre-computing filter cores has advantages as it speeds up the actual use of the filter. The wise use of a minimum set of cyclically applied cores has additional performance benefits.

According to another strategy, code generation functionality for implementing an image processing pipeline is described. Functionality accepts requirements that specify what types of operations the pipeline is supposed to perform, and then selectively assemble code modules from a library of such modules in order to implement these functions. Functionality does not contain modules from the library that are not required. This strategy has advantages for many reasons. For example, it helps generate code that is more efficient and thus potentially runs faster.

Finally, another strategy uses a graphics processing unit (GPU) to implement certain aspects of the image processing pipeline or all functions in the image processing pipeline. Specific implementations use a pixel shader (digital signal processor (DSP) in the GPU) to filter images. In one case, one or more pixel shader texture units may be assigned to filter core weights, and one or more other pixel shader texture units may be assigned to other delta-shifted versions of the same input image information (associated with core taps). This strategy has advantages because it potentially provides greater processing speed and processing throughput compared to the implementation in the CPU due to the vector processing capabilities of pixel shaders.

Additional features and concomitant advantages of strategies are described in this description.

With regard to terminology, the term "image information" is used to include all types of information that can be consumed by the user in any visual form. The image information may represent information expressed in any format, for example, an analog format, a digital format, or any combination of analog and digital formats. The image information may represent still image information (e.g., digital photographs) and / or moving image information (e.g., video image information). Additional variations are allowed when using the term "image information".

The term "image processing pipeline" refers to any functionality for processing image information. The conveyor includes at least two functional components that work with image information in a sequential manner, i.e. in sequence.

The term "linearization" means the conversion of image information from a non-linear pre-adjusted form to a linear form. The term "non-linearization" means the operation, the inverse of linearization.

The term “luminance-related image information” means image information having a luminance-related component (eg, Y ′) and color components. The term “luminance-related color space” means any of several standards for generating luminance-related image information (eg, Y'CbCr, etc.).

In general, with respect to the structural aspects of the described entity, any of the functions described in this document can be implemented using software, firmware (for example, fixed logic circuits), manual processing, or a combination of these implementations. The terms “module,” “functionality,” and “logic,” as used herein, typically represent software, firmware, or a combination of software and firmware. If implemented in software, the terms module, functionality, or logic represent program code that performs specified tasks when executed on a processor device or devices (for example, a CPU or multiple CPUs). The program code may be stored on one or more fixed or removable computer-readable memory devices.

Regarding the procedural aspects of this entity, certain operations are described as constituting separate steps, performed in a specific order. These implementations are exemplary and non-limiting. Certain steps described herein may be grouped together and performed in a single operation, and certain steps may be performed in an order that differs from the order used in the examples set forth herein.

This description includes the following contents:

A. Exemplary General Video Processing Pipeline Enhancements

A1. Image processing in RGB color (linear) space

A2. Using full-channel floating-point image information in a pipeline

A3. Special measures for image information 4: 2: 0

A4. Processing image information in pseudo-linear space

A5. Improvements to Error Dispersion Algorithms

A6. Dynamic calculation of filter cores

B. Exemplary Implementation-related Improvements in the Video Processing Pipeline

B1. Optimal Pipeline Generation

B2. General principles of using GPU for image processing

B3. Pixel Shader Enhancements

C. Exemplary computing environment

A. Exemplary General Video Processing Pipeline Enhancements

A1. Image processing in RGB (linear) color space

FIG. 1 shows an example image processing pipeline 100 for processing image information (e.g., video information). As a general representation, the topmost line of modules receives image information from any of various sources (camera, scanner, disk, storage medium, digital network, etc.) and then converts the image information into a processing form. A large module in the middle of the drawing represents this processing, which leads to processed image information. The bottom line of modules converts the processed image information into the form that is suitable for output to the destination node (for example, a television receiver, computer monitor, storage medium, digital network, etc.).

Several unique and advantageous aspects are provided for the image processing conveyor 100 identified in separate subsections of the present invention. This subsection focuses on processing image information in a linear fashion. Other improvements result from additional processing of image information in interlaced (interleaved) form within the RGB color space. In contrast, a typical approach in the art is to process interlaced Y'UV information of 4: 4: 4 or 4: 2: 2. As recognized by the authors of this invention, the processing of image information in non-linear space leads to various image distortions. The approach taken in the conveyor 100 of FIG. 1 solves these problems.

Each of the modules shown in FIG. 1 is described in turn.

Regarding the top row of modules, anti-aliasing module 102 converts received Y'CbCr image information having a first accuracy into image information having a second accuracy, with the second accuracy being higher than the first accuracy. Module 102 performs this task by performing anti-aliasing processing. Those. a smoothing operation (not shown) could be used to quantize the input image information that arrives at the pipeline 100 to the first precision. Module 102 effectively applies the inverse of these quantization operations to obtain a second, higher accuracy.

The upsampling module 104 converts input image information having a first number of samples into output image information having a second number of samples, the second number of samples being greater than the first number of samples. In a typical application, the image information was received by the conveyor in a form in which there are more samples of luminance information (Y ') compared to the color information (Cb or Cr); this is because the eye is more sensitive to luminance information, which makes it possible to reduce the amount of color information with respect to luminance information without noticeable deterioration in quality. The upsampling module 104 typically operates to increase the number of color samples so that the color samples are equal to the brightness samples. For example, upsampling module 104 may convert Y'CbCr 4: 2: 0 or 4: 2: 2 image information into Y'PbPr 4: 4: 4 image information. The upsampling operation performed by module 104 increases the number of samples by interpolating existing samples (thereby effectively performing the scaling operation). These interpolation techniques often specifically take into account how different standards place color information regarding luminance information to provide accurate results.

Module 106 applies the matrix transform to Y'CbCr 4: 4: 4 image information to convert it to another color space, i.e. color space R'G'B '.

Module 108 then converts the non-linear information R'G'B 'into a linear form by applying a transform function to it. This imposes a repetition of the fact that the apostrophe characters (') associated with the image information R'G'B' indicate that it is in a non-linear form. The absence of apostrophes (for example, RGB) usually means linear data (except that it is also common to delete apostrophes, if it is traditionally understood that the signals mentioned represent non-linear information, as described above). The model transform function shown in FIG. 1 under module 108 in FIG. 1 illustrates the general form of the transform function that is used, which is also the general form of the CRT's own transform function (not shown). This model transform function also indicates that it can use a linear tail near the V = 0 part of its curve to take into account special considerations that apply in this rounded region.

The module 110 optionally performs an interleaving operation for the image information in order to convert them from interlaced to a progressive format. In performing this task, the module 110 can be controlled using either the RGB image information or the original Y'CbCr image information (for example, for the special case of 4: 2: 0 information). However, the interleaving operations should preferably be performed in linear linear space RGB. This is because interleaving is usually a form of image processing (including, for example, interpolating existing samples). Therefore, performing this processing in linear space gives more accurate results.

Module 112 optionally converts the composite colors of linear RGB information to express information in a different color space. This transform may include applying a matrix transform to the RGB information to change its composite colors to match those composite colors that are necessary for processing performed in the downstream components of the video processing pipeline. In one example, module 112 may convert various types of image information into a common color space, which facilitates the mixing of information in a subsequent operation.

Module 114, in general, represents any type of information processing of images having a converted color space. In this aspect, processing is performed for image information, i.e. in this specific example: a) in a linear form; b) in the RGB color space; c) in 4: 4: 4 (full-channel) form; and d) in line-by-line format. Although it is preferable that the processing be carried out in a linear color space in order to reduce image distortion, all aspects listed in this list should not be present in the image processing pipeline to give advantages.

In any case, module 114 may perform any type of image information processing. In fact, the processing contains any conversion of image information, for example, output image = some function (input image), including any type of image filter operation. A representative and non-limiting list of these processing operations includes: a) creation of composite images; b) alpha conjugation (for example, various attenuation and fading); c) selection of edges; d) sharpening; f) resizing (scaling to larger or smaller image sizes); f) de-allocation (if not already done), etc. Composite image creation entails mixing one type of image information with another type of image information. For example, module 114 may be used to combine graphics (e.g., textual information) on top of video information captured from a DVD disc. Alpha blending entails smooth blending of colors based on the alpha coefficient (which determines the degree to which one color mates with another). For example, a pairing operation is defined by the following equation: Final color = source color × source pairing coefficient + destination color × destination pairing coefficient. In this equation, the color of the destination pixel represents the color of the pixel in the pre-existing scene, and the color of the source pixel represents the new color of the pixel that the interface intends to add to the destination pixel. Conjugation coefficients can vary from 0 to 1 and are used to control how much of the color of the source and destination pixels in the final color value. Once again, representative examples of a large number of potential processing applications are provided. In general, many of the video processing applications described above can apply affine weighting to the transition between source and destination image information. In these types of applications, the use of nonlinear weighting can cause the overall brightness of the image information to decrease too quickly. This is just one example of image distortion that the present solution can eliminate or mitigate by processing non-linear image information.

The bottom row of the modules in figure 1, in General, performs operations inverse to the above operations in the top row. Those. after processing in module 114, module 116 optionally converts the image information into another color space, for example, back into an RGB color space. Module 118 applies an optional re-interleaving of image information if it is to be stored or displayed in interlaced form. Module 120 employs a conversion function to again convert RGB image information to non-linear form (R'G'B ') (if required). Module 122 changes the color space of the image information R'G'B 'back in a form that separates the luminance component (Y') from the color components (e.g., PbPr). Module 124 optionally sub-samples the image information of Y'PbPr to reduce the size of the color samples (PbPr) relative to the size of the brightness samples (Y '). Those. this module 124 can convert Y'PbPr 4: 4: 4 image information (where chroma samples are provided for each brightness sample) to Y'PbPr 4: 2: 2 or 4: 2: 0 image information (where fewer chroma samples are provided relative to samples brightness). Finally, module 126 applies the quantization operation to image information. The quantization information has the effect of converting the image information to a lower accuracy in order to facilitate efficient transmission through a network, storage device, display, and the like. The quantization operation can be combined with a smoothing operation to reduce distortion that might otherwise occur due to rounding errors generated by quantization. As described in more detail in the next subsection, the smoothing operation has the effect of propagating these errors to neighboring pixels as image information is processed, thereby mitigating the effects of image distortion.

2 shows a procedure 200 that summarizes the above operations in the form of a flowchart of a method. Step 202 includes converting the image information into a linear color space, such as RGB. The information is also preferably interleaved in a line-by-line form and up-sampled to 4: 4: 4 (full channel) format. Thus, step 202 corresponds to the top row of modules in FIG.

Step 204 entails the implementation of any kind (and combination) of tasks for processing image information in a linear manner.

Step 206 represents the conversion of linear image information into any output format requested by a particular application. This may entail the conversion of the processed linear image information into a nonlinear color space, its repeated interleaving, downsampling, quantization, etc. Step 206 corresponds to the bottom row of the modules in FIG.

A2. Using full-channel floating-point image information in a pipeline

Returning to FIG. 1, the pipeline 100 also provides excellent results for known strategies since it uses (a) full-channel image information (i.e. 4: 4: 4) having (b) relatively high processing efficiency with (c) optionally linear semantics. More specifically, as described above, the conveyor 100 converts the image information that it receives to a greater accuracy than the original form. The conveyor 100 also converts the image information into full-channel (4: 4: 4) form by increasing the size of the color samples in the information regarding the brightness samples. Then various processing tasks are performed with more accurate full-channel image information. After the processing is completed, the pipeline 100 may again optionally convert the processed image information into a sub-sampled form of lower accuracy.

In general, the conversion of image information into a more accurate and full-channel form effectively enhances the signal component with respect to the noise component of the information, thereby enabling the processing that is performed with this information to obtain more accurate results. In contrast, well-known strategies that do not use high-precision or full-channel image information in various intermediate pipeline connections can propagate errors along the pipeline and can reduce the resolution of image information (through significant loss of color content). Figure 1, in General, illustrates the processing of high-precision full-channel image information through an information block, designated 128, which transmits data to the processing module 128 (although other modules in this drawing can also take advantage of the processing of information in high-precision and full-channel form, for example, interleaver 110 )

The following is a more specific illustrative example. Let's consider the advantage of working with high-precision linear information of RGB images in comparison with Y'UV 4: 2: 0 or 4: 2: 2 image information of usual accuracy. Recall that 4: 2: 0 and 4: 2: 2 image information provides half and one fourth (respectively) the amount of color information relative to the amount of luminance information. Typically, the information of high-definition (HD) images (in particular, 1920 × 1080i, 2.07 megapixels) is scaled down to 1280 × 720p (0.92 megapixels) or 720 × 480p (0.35 megapixels). If the pipeline performs a downscaling operation in this context to a 4: 2: 0 downsampling format (for example, from 1920 × 1080 to 1280 × 720), the pipeline scales down 2.07 megapixels of luminance information and 0.52 megapixels of color information to 0 , 92 megapixels of luminance information and 0.23 megapixels of luminance information. Nevertheless, if the pipeline performs upsampling to 4: 4: 4, then converts to high-precision RGB (for example, with a floating point), then scales it down and then converts it to YUV 4: 4: 4, then the pipeline effectively saves 0.92 megapixels of luminance information and 0.52 megapixels of luminance information. In other words, the use of high-precision processing in this context enables the conveyor to save almost all the original color information.

One way to achieve high-precision intermediate image information is to use the floating point format in order to represent the information. Floating point numbers have mantissa components and exponent components. The exponent component defines how the mantissa component should be offset. Using this format, floating point information can express values from very small to very high by correspondingly biasing the presentation of information (for example, by changing the component of the exponent). Any type of floating point representation having any number of bits can be used, including a 16-bit floating point representation, a 32-bit floating point representation, etc. A particularly suitable form of floating point for use in the image processing pipeline 100 is the so-called FP16 floating point format, although other floating point representations may be used.

A3. Special measures for image information 4: 2: 0

As a review, as described in the Background section, 4: 4: 4 image information includes four samples of Cb and Cr for each of four samples of Y '. The 4: 2: 2 image information includes two samples Cb and two samples Cr for every four samples Y '. Information 4: 2: 0 includes two times less samples of Cb and Cr for each scan line and two times less scan lines of Cb and Cr in comparison with Y '. In other words, the resolution of the color information is half the resolution of the brightness information in the horizontal and vertical directions. For example, if the full resolution of the image is 720 × 480, then the color information is stored only at 360 × 240. At 4: 2: 0, not only missing samples should be interpolated in each scan line, but also all scan lines of color information should be interpolated from the scan lines above and below. Figure 3 shows an example of interlaced information of 4: 2: 0 MPEG2 images. Additional information regarding 4: 2: 0 image information processing can be obtained from Don Munsil and Stacey Spears, "The Chroma Up-sampling Error and the 4: 2: 0 Interlaced Chroma Problem", DVD Benchmark, April 2001 (December 2002, updated January 2003).

The insufficient volume and positional alignment of the color information in the 4: 2: 0 image information creates a number of difficulties in processing this information. Generally speaking, problems may arise because one stage in the processing pipeline 100 requires an analysis of another stage in the processing pipeline 100 in order to correctly interpret the 4: 2: 0 image information. Consider an example of upsampling module 104 and conversion module 106 of FIG. 1. In order to correctly interpret 4: 2: 0 image information, upsampling module 104 optimally requires knowledge of how color information is expressed in image information, which is information that is provided by interleaver 110. But, since the interleaver 110 is located lower with respect to the upsample module 104, this knowledge is not available to the upsample module 104. If modules 104 and 106 process 4: 2: 0 information without the benefits of interleave analysis, they can generate errors in image information that may be difficult or impossible to correct. This difficulty ultimately stems from the fact that 4: 2: 0 scan lines cannot be processed independently of one another in interleave analysis (as opposed to, for example, 4: 2: 2 image information).

To resolve this complexity, the pipeline 100 of FIG. 1 may include special measures for processing 4: 2: 0 image information. As a general measure, the pipeline 100 can be modified so that the analysis performed at various stages (and the information available at various stages) is more commonly shared with other stages in the pipeline instead of the need for each stage to automatically analyze only the information that provided to her through the previous stage.

With particular regard to the problem caused by the position of the interleaving unit 110 in the pipeline 100, the conveyor 100 may be modified so that the interleaving analysis (performed by the module 110) is performed earlier, for example, together with the upsampling analysis (performed by the module 104). FIG. 1 illustrates this tactic with a dashed line 130. Alternatively, the interleaver 110 may perform an interleave analysis by also examining the input source image information that is provided to the upsampler 104. Those. the interleaver 110 may perform an analysis to output the module 108 in combination with the source image information to gain a better understanding of how the interleaving of the image information should be performed.

In addition, module integration measures can be developed to share the analysis performed separately by the various modules in the pipeline 100, thereby thereby more effectively resolving the lack of color information in the 4: 2: 0 image information.

A4. Processing image information in pseudo-linear space

The previous sections have resolved the goal of developing an image processing pipeline that eliminates some image distortion. However, image processing pipelines should also provide output results in an efficient way. The image processing pipeline generates results in an efficient way when it implements it with sufficient speed and with an acceptable amount of processing resources. To this end, this section offers various approximations that can be applied to the conveyor 100 shown in FIG. 1 in order to reduce the complexity of the conveyor 100.

Those. this subsection proposes modifications to the conveyor 100 shown in FIG. 1 so that it provides image information that is almost linear, but not completely linear. As a result of these modifications, it is believed that they should transform the image information in pseudo-linear space instead of the theoretical adjustment of linear space. The advantage is that the hardware implementation of the mathematical operations required by the conversion functions (in modules 108 and 120) can be greatly simplified, leading to faster and more efficient processing of image information. More specifically, the selected operations typically execute an order of magnitude faster than the original accompanying functions (which are mathematically more accurate). The negative influence of image processing in pseudo-linear space is minimal, since the information of pseudo-linear images is almost linear. Thus, any linear algorithm can be applied to this information without modification to provide satisfactory output for most applications.

4 provides additional information regarding the above approximation. More specifically, FIG. 4 is a modification of only certain modules of the processing conveyor 100 of FIG. 1 and, therefore, contains only an abbreviation for the complete processing conveyor 100 shown in FIG.

The conveyor shutter speed 400 shown in FIG. 4 includes a conversion function module 402 that replaces the conversion function module 108, FIG. 8. Instead of applying mathematically accurate transformations of image information to convert it to linear space (as module 108 does), module 402 uses the approximation conversion function (g_approx (x)), which converts the image information into the above pseudo-linear space. This module resorts to the conversion function g_approx (x) because it is “simpler” than the more accurate conversion function required by the module 108, FIG. 1.

Module 404 represents the equivalent of processing module 114 of FIG. Module 404 performs any number of processing tasks with pseudo-linear image information in a pseudo-linear processing space.

Modules 406, 408, and 410 then perform the task of converting the processed image information into a format suitable for output (for example, a display, a storage device, a network target, etc.). Module 406 is assigned a specific task of being responsible for eliminating the effects of module 402 by applying an inversion of its operation, i.e. g_approx '(x). Module 408 then linearizes the output of module 406 by applying the transform function g_in (x); this operation is performed because the next step 410 is intended to receive line input. Module 410 then applies the inverse transform function (g_out '(x)) to obtain the final gamma-corrected output image information to suit the format that is required. In an actual implementation, modules 406, 408, and 410 can be integrated into a single transformation represented by F (x) performed by module 412.

Importantly, the algorithms that apply to the pseudo-linear color space of the module 404 are the same as the algorithms that apply to the correct linear color space of the module 114, FIG. 1. In other words, the processing strategy presented by FIG. 4 does not require special adjustment of the image processing algorithms themselves. In contrast, the approach used by the well-known previous strategies is to perform processing in the non-linear space in which the image information is received, and then apply various special or suitable correction factors for the algorithms to correct all image distortions . However, these adjustments were often inaccurate and typically complex in any case. Summarizing, the preceding methods do not apply a comprehensive and generally applicable approach to converting information into a pseudo-linear form, after which this pseudo-linear form is deleted in the manner described above.

The remainder of the description in this subsection is devoted to approximate approximations that can be used to implement the above modules in FIG. To begin with, we note that the conversion functions used to perform gamma processing are typically power functions (for example, x ^0.45 or inversion x ^{1 / 0.45} ≈ x ^2.222 ) or a combination of several functions. Power functions are typically very resource intensive in pixel-based computing using traditional hardware.

However, note that x ^{0.45 is} approximately equal to the square root of x or, in other words, x ^0.45 ≈ x ^0.50 . In addition, x ^{1 / 0.45 is} approximately equal to x ² or, in other words, x ^{1 / 0.45} ≈ x · x. There are optimized hardware for computing x, which is faster than the operations required by a common power function (often an order of magnitude). This hardware allows you to square x very quickly, and it's just a multiplication operation. Therefore, these approximations can significantly reduce the computational resources associated with the implementation of transformation functions.

More formally, the function g_approx (x) performed by the module 402 may take the form x ² · sign (x), while the inverse transform function performed by the module 406 may take the form (abs (x)) · sign (x). (We emphasize that transform functions and inverse transform functions can apply a linear tail segment of approximately zero.)

The above approach can be generalized to various input and output scales represented by the functions g_in (x) and g_out (x) using the following exemplary sequence of operations:

1) Application of the inverse linear approximation g_approx (x) = x ² = x · x to the image information. Module 402 may perform this step.

2) Data processing in pseudo-linear space. Module 404 may perform this step.

3) Application of the linear approximation g_approx '(x). Module 406 may perform this operation.

4) Applying the conversion function g_in (x) to linearize the output of module 406. Module 408 can perform this operation.

5) Applying the conversion function g_out '(x) to the output of module 408 to generate information in the desired g_out space.

Steps (3) - (5) can be combined into one function F (x), i.e.: F (x) = g_out '(g_in (g_approx' (x))).

If the conveyor converts from gamma 2.222 to gamma 2.4, then the function F (x) can be expressed as:

F (x) - (((x ^0.5 ) ^{l / 0.45} ) ^{l / 2.6} ) = x ^0.426 .

The resources in the above algorithm include only the very “uncomplicated” dialog required by the g_approx (x) function, after which the information is processed in pseudo-linear space, then the potentially resource-intensive dialog required by the F (x) function follows. But, since the equation F (x) simply requires the multiplication of exponentials, the cost of this function is no more than the original inverse function. Therefore, the solution shown in FIG. 4 has the effect of potentially halving the costs of a full linearization / non-linearization cycle by about half. As indicated above, this solution has advantages in that it requires special tuning of the algorithms executed by the 404 module.

FIG. 5 shows a procedure 500 that summarizes the above operations, and is equivalent to the linearization procedure 200 shown in FIG. 2. Step 502 entails converting the input image information into pseudo-linear space. Step 504 entails performing image information processing in the pseudo-linear space using module 404. Also, step 506 entails converting the processed information into a corresponding non-linear color space for output.

A5. Improvements to Error Dispersion Algorithms

The quantization of high-precision image information into a lower-precision format can lead to errors, since the approximations required by quantization can lead to image distortions if they are not processed correctly. Various error dispersion algorithms serve to disperse the errors caused by quantization, and thereby minimize image distortion. This variance of errors is a form of smoothing.

6 shows a known application of the Floyd-Steinberg algorithm to disperse errors caused by quantization. Consider the point X in the image, which represents a pixel (or, more generally, an image element) in the image. The quantization of this image element X generates an error difference, calculated as the difference between the high-precision representation of the image element and the value into which it is quantized. The Floyd-Steinberg algorithm divides the difference into various components and then distributes these components into image elements that are located next to element X (and which have not yet been processed by the algorithm). When the processing moves to this neighboring image element, the error components previously scattered by this image element are added to it, and then this element is processed in the manner described above, for example, by quantizing it, calculating another error difference and scattering this error into neighboring elements Images. Processing the entire image can continue in various ways, for example, from left to right and from bottom to top.

The Floyd-Steinberg algorithm assigns different weights to the error components, which it scatters to adjacent pixels. 6 shows the weights 7, 1, 5, and 3 assigned to image elements that are placed next to element X (clockwise). More specifically, the weighting coefficients applied to the error components are 7/16, 1/16, 5/15 and 3/16, where the denominator 16 reflects the sum of the weights assigned to adjacent pixels.

However, error dispersion algorithms such as Floyd-Steinberg sometimes generate destination images having different image distortions. These image distortions can manifest themselves in visually distinguishable ornaments, which are especially noticeable with certain image content. Known strategies solve this problem by adding random noise to the original image to obscure the patterns caused by error dispersion algorithms. But this elimination in itself can have various disadvantages. First, well-known strategies add noise to the original image and then quantize the original image with noise as a basis (in a manner described in more detail below). Although this technique has the effect of masking image distortions caused by the error dispersion algorithm, it also degrades the quality of the final destination image. Those. the noise in the destination image increases in proportion to the noise added to the original image. Secondly, well-known strategies can add noise using noise generators having a relatively short repetition period. Therefore, noise generators can generate intrinsic distortions of ornamental images, which can be visually distinguishable in the destination image (perhaps, along with some signs of error dispersion distortions that the noise generator is designed to hide).

7 shows a procedure 700 that more accurately explains the first of the above problems. The purpose of procedure 700 is to quantize the source image information (“Source”) to generate destination image information (“Final”) using an error distribution algorithm. Step 702 entails generating “Temporary” image information by adding noise information to the original image information for the image element i (for example, a pixel). Step 704 entails generating information of the final images by quantizing the sum given by the time-image information and the image information Error_Term (i). Error_Term represents an error component added to the image element i based on the previous quantization of the previously processed image element or elements. Step 706 entails setting the next Error_Term component (which should be applied to the future image element to be processed) as thus calculated information of the final images minus the information of temporary images. The application of the algorithm, in fact, has the pure effect of quantizing an image with noise with the following negative sides.

FIG. 8 shows a system 800 that solves the above problems of the related art. The smoothing module 802 converts the information 804 of the source images into the information 806 of the destination images using the error dispersion algorithm. The error dispersion algorithm contains, but is not limited to, the Floyd-Steinberg algorithm described above. A noise generator 808 inserts arbitrary noise into the quantization process to help hide some of the image distortions generated by the error dispersion algorithm.

System 800 differs from known strategies in at least two respects. First, system 800 essentially adds noise information to the quantization process, not information 804 of the original images. In other words, the source image information 804 maintains the basis from which the error vectors are measured, instead of (in previous techniques) the original image plus noise information. Thus, an increase in the amount of noise in the system 800 does not necessarily have the effect of gradually adding noise to the information 806 of the destination image for the eye. This is because the error vectors that propagate to adjacent image elements store a deviation rate of the destination image information 806 relative to the original image information 804, and thus the error dispersion algorithm constantly tries to correct the effects of random noise added to the quantization process.

9 shows a procedure 900 that explains the above processing. Step 902 entails generating final image information for the image element i by quantizing the sum of the original image information, the noise information, and the error vector for the image element i. Step 904 entails calculating an error vector that must be scattered to an adjacent image element (s) by calculating the difference between the final image information and the original image information for the image element i.

The following is a more specific illustrative example. Suppose that the task is to quantize an image element having a value of 23.3 to the nearest integer. Assume (in the well-known strategy shown in Fig. 7) that +0.4 of the amount of noise information is added to the value, making it equal to 23.7. The final image value for this image element must be 24, and the error vector that propagates must be -0.3 (the difference between the final output image value and the image value with noise 23.7). In the improved strategy of FIG. 9, in contrast, the error vector that propagates should be equal to −0.7 (the difference between the value of the final output image and the value of the original image).

A more formal description of the above methodology is described below based on the application of the Floyd-Steinberg error dispersion algorithm. For each pixel in the image, the task of the algorithm is to quantize the high-precision value of V and store it in the pixel position P (x, y) of the matrix of pixels P, where P (x, y) has less accuracy than V. Error vectors generated algorithm, are stored in a high-precision matrix of pixels E having the same size as the matrix of pixels P. The error vector for the pixel P (x, y) is stored in the corresponding location in E (x, y). The round () function in the algorithm rounds a high-precision number to the nearest low-current number. The random () function in the algorithm returns a random high-precision number in the range {-0.5 ... 0.5}. Temporary Desired and Error values are high-precision values. Actual error propagation coefficients are taken from the Floyd-Steinberg error propagation algorithm.

Algorithm 1: Advanced Error Dispersion

For each pixel x, y:

Desired = V + E (x, y)

P (x, y) = round (Desired + random ())

Error = Desired-P (x, y)

E (x + 1, y) = Error7 / 16

E (x-1, y + 1) = Error · l / 16

E (x, y + 1) = Error5 / 16

E (x + 1, y + 1) = Error3 / 16

End

As a second enhancement, the noise generator 808 uses a relatively long snooze pattern, in contrast to the known approaches. The phrase “relatively long” can be interpreted in relative terms by confirming that the noise generator 808 does not repeat its processing numbers for the entire frame of image information or at least does not repeat itself to the extent that leads to noticeable distortions in the destination image information 806 . One specific random number generator that meets this requirement is the so-called R250 random noise generator.

A6. Dynamic calculation of filter cores

Scaling filters can be used in various connections of the pipeline 100 shown in FIG. 1 when the size of the image information changes from the original size to the destination size. For example, the processing performed in module 114 may entail scaling of the image information up or down depending on the application and / or depending on the control signals input by the user. Scaling may also be performed in other connections of the conveyor 100, for example, in sampling units 104 and 124.

A number of filtering algorithms can be used in conjunction with scaling algorithms. One particular known type of filter is the so-called Kathmullah-Roma filter. Application of this type of filter to the task of changing the size of image information leads to some increase in the sharpness of the edges of the image information.

This subsection describes a filter used for scaling (such as a Katmull-Rum filter, but not just this type of filter), which can be calculated and applied on a pixel by pixel basis to image information. This can be conceptualized as dynamic calculation and applying a separate filter for each pixel and image information. Each filter may include one or more cores, and each core may have one or more taps. The kernel sets the weights that are applied to the pixels in the image information. Bends define samples of image information for which the kernel is executed. Failure to compute and apply filter cores in this way can lead to various distortions in the destination image, such as beat patterns.

However, calculating the individual filter cores for each pixel can be unnecessarily costly in terms of the time required to compute the cores. This may prevent real-time image information from being displayed on some hardware. To solve this problem, this subsection defines various strategies for dynamically calculating phase filter cores for all rows or columns of input image information and subsequent application of these cores. As a review of the strategies, they entail calculating the number of cores required by the row (or column) of information in the source images, calculating the number of taps required by the cores, and computing and storing the required number of cores in the matrix. After these steps, the filter cores stored in the matrix are applied to the image information.

The algorithm is efficient because kernels are computed before application. In addition, savings in computing resources are achieved, since in certain cases a relatively small number of calculated cores can be cyclically applied to a much larger row or column in the image information. In addition, the same kernels applied to a particular row or column can be applied to other rows or columns (respectively) in the image information. In other words, depending on the size requirements of a particular application, it is not necessary to calculate a separate filter for each individual pixel in the image information.

FIG. 10 shows an overview of a system 1000 for implementing a filtering module 1002 of the above-described embodiment. Filtering module 1002 includes a dynamic core calculation module 1004. The purpose of this module 1004 is to first calculate the number of cores required for a particular scaling operation and the number of taps required for cores. The number of cores and taps depends on the way in which the size of the image information is changed (by the method described below). Then, the preliminary calculation module 1004 pre-calculates the required number of cores and stores the cores in the storage module 1006. Next, the kernel application module 1008 applies the kernels in the kernel storage module 1006 to the image information to achieve the desired filtering mode. As mentioned above, calculating the cores before applying them helps you perform the filtering process faster. Moreover, the complexity of calculating filters is significantly reduced for those scaling operations that require preliminary calculation of only a few cores.

The right side of FIG. 10 shows how kernels 1010 can be iteratively applied to a particular line 1012 of image information 1014. The cores 1010 are applied sequentially and cyclically (where in one example case each core is applied to one output pixel). For example, if there is a small number of cores for a relatively long line of image information, then these cores are applied one after another and are repeated many times while the image line is being processed. Those. previously saved kernels form a list; they are applied one after the other to the line, and when the last core appears in the list, processing continues by looping back and retrieving from the beginning of the list. The iterative nature of the application methodology is represented through a loop 1016 shown in FIG. 10. Kernels 1010 can be applied to other strings in a similar way. In addition, similar processing may be repeated for cores that apply to columns of image information 1014.

An example algorithm that can be used by module 1004 to calculate the number of cores and taps is attached below. Module 1004 applies the algorithm separately to scaling in the x and y dimensions of the image information 1014. Thus, although the algorithm is specified to scale in dimension x, the algorithm is also applied to dimension y, for example, by changing the links from width to height, etc. In dimension x, the ScaleRatio used in the algorithm sets the ratio of the width of the source image to the width of the desired final (target) image. The ceiling function used in the algorithm rounds the real number to the next larger integer. The gcd function in the algorithm calculates the largest common denominator of two integers.

Algorithm 2: Calculation of the number of cores and taps

ScaleRatio = SourceWidth / DestinationWidth

if ScaleRatio <1.0

ScaleRatio = 1.0

Taps = ceiling (ScaleRatio4)

if Taps is odd

Taps = Taps + 1

Kernels = DestinationWidth / gcd (SourceWidth, DestinationWidth)

As an example, consider vertically scaling information of source images having a weight of 720 pixels to information of destination images having a weight of 480 pixels. Applying the above algorithm to this scenario yields:

ScaleRatio = 720/480 = 1.5

Taps = ceiling (ScaleRatio4) = 6

Kernels = 480 / gcd (720, 480) = 480/240 = 2

Thus, module 1004 must pre-compute and pre-allocate 2 cores of 6 taps each. The zoom operation is performed in alternating alternating mode between two cores as it moves along a row of pixels.

As another example, consider vertical scaling of source image information having a weight of 721 pixels to destination image information having a weight of 480 pixels. Applying the above algorithm to this scenario yields:

ScaleRatio = 721/480 = 1.5021

Taps = ceiling (ScaleRatio4) = 7

The number of taps is even, so add one and get 8

Kernels = 480 / gcd (721, 480) = 480/1 = 480

Thus, the module 1004 must pre-calculate and pre-allocate 480 cores of 8 taps each. The resize operation uses a unique core for each of the 480 output pixels. However, significant savings are still achieved since one set of 480 cores can be used in each vertical column of pixels. In addition, in real cases, in practice, the ratio of the sizes of the destination and the source tends to be a fairly simple ratio, which makes the resulting number of necessary cores adjustable. Special restrictions may also be imposed that prevent users from entering resize requests that require a particularly large number of filter cores (for example, if a predetermined threshold is exceeded).

11 shows a procedure 1100 that summarizes the above operations. Step 1102 entails determining the number of cores (and taps per core) required to achieve the necessary image resizing in horizontal and vertical dimensions. The precomputing unit 1004 can accomplish this task using the algorithm provided above. Step 1104 entails allocating space in the core storage unit 1006 to store the number of cores calculated in step 1004. Step 1106 entails actually pre-computing and storing the cores. Next, step 1108 involves applying the pre-computed kernels stored in the storage unit 1006 to the image information. The application continues through cyclic sequential passage through the cores during the processing of all data in rows or columns.

B. Exemplary Implementation-related Improvements in the Video Processing Pipeline

B1. Optimal Pipeline Generation

The next section provides a number of implementations of the video processing pipeline 100 of FIG. 1 for a more technology-specific nature compared to section A. To begin with, FIG. 12 shows a high-level overview of the image processing pipeline 1200, which will serve as a basis for explaining the improvements in this section. .

The conveyor 1200 shown in FIG. 12 includes the processing steps defined by the input stage 1202, the processing stage 1204, and the output stage 1206. As for the input stage 1202, the input source 1208 represents any image information source. The source 1208 as a whole may contain new captured image information (for example, created by a camera or scanner) or previously captured image information that is presented to input stage 1202 through a specific channel (for example, received from disk, over an IP network, etc. .). In the first case, the capture processing functionality 1210 can perform any kind of preprocessing of the image information received from the source 1208. In the second case, the decoder functionality 1212 performs any kind of streaming extraction and decompression of the images to generate image data. In general, this processing may include separating image information from audio information in the received information, decompressing the information, etc. With regard to processing step 1204, processing functionality 1214 performs any kind of processing with the resulting image information, for example, mixing several streams of image information together into a composite signal. As for the output stage, the output processing functionality 1216 represents any kind of processing performed on the processed image information in preparation for output to the output device 1218. The output device 1218 may represent a television receiver, computer monitor, etc. Output devices may also represent storage devices. Additionally, the output device (or output functionality 1216) may provide compression and formatting functionality (eg, multiplexers) that prepare information for storage on the device, distribution over the network.

In general, the processing operations set forth in FIG. 1 can be distributed in stages (1202, 1204, 1206) by any method. For example, processing step 1204 typically implements the processing module 114 shown in FIG. Each of the stages (1202, 1204, 1206) can be physically implemented as a separate device for performing assigned tasks or several devices connected in a serial or parallel way. Functions can be implemented through any combination of software and hardware.

FIG. 13 shows an example system 1300 for configuring the image processing pipeline 1200 shown in FIG. 12. More specifically, the exemplary system 1300 may include functionality for automatically generating computational code to implement the conveyor 1200 so that it performs any combination of image processing operations shown in FIG.

In order to function as described above, the system 1300 includes a configuration module 1302. The configuration module 1302 receives pipeline requirements information 1304, which sets the requirements that the configured pipeline must satisfy. Requirement information 1304 may have several components. The input requirement component 1306 specifies the characteristics of the image information that the pipeline is expected to receive. Input requirements component 1306 may specify one type of image information that may be processed, or may specify a set of several permitted types of image information that may be processed. One way to conveniently specify several characteristics of the allowed input information is to set the used video coding standards, which may imply the full set of features that the image information possesses, such as allowed color spaces, color subsampling schemes, gamma conversion functions, etc. For example, ITU-R Recommendation BT.601 is an international standard that defines studio digital encoding of image information. This standard uses Y'CbCr encoding of image information. ITU-R Recommendation BT.709 is an international standard that defines studio encoding of high definition video information. High definition content (HD) represents video content that is higher than standard definition (SD), typically 1920 × 1080, 1280 × 720, and the like. These are just two of the many video coding standards that the image processing pipeline can handle.

The output requirement component 1308 specifies the characteristics of the image information that the pipeline is expected to output. Those. output requirement component 1308 may specify one kind of image information that may be generated to correspond to a particular output device, or may define a set of several permitted types of image information that may be generated to correspond to various types of output devices. In addition, one way to conveniently specify several characteristics of the allowed output information is to set the video coding standards used.

The intermediate component 1310 processing requirements sets the nature of the processing tasks that the pipeline must perform for information output images. Any number of processing tasks can be set, including, but not limited to, resizing (scaling), composing composite images, alpha blending, selecting edges, etc.

In general, an operator can manually select conveyor requirements 1304. Alternatively, one or more requirements 1304 may be automatically removed from the environment in which the conveyor is to be used.

Given the input requirements 1304 for the pipeline, the configuration module 1302 performs the task of using static analysis in order to interact with the library of code modules 1312 to compose its own set of code modules that meets the requirements 1304 for the pipeline. One way of doing this is to generate a basic equation that converts any type of input to any type of output, including any type of intermediate processing. This basic equation includes a number of components. The components are associated with the corresponding code modules stored in the library 1312. In this implementation, the configuration module 1302 performs the task of compiling its own set of code modules by eliminating all components that are not required in terms of pipeline input requirements 1304. This results in the selection of specific code modules from the 1312 library and the passage of the remaining modules.

The result of the processing performed by the configuration module 1302 is an optimized pipeline code 1314, which can then be applied to process image information. This code 1314 is upgraded to perform only those functions that are required of it. As a result, the pipelined assembly based on this code has the potential to perform its operations faster than, say, a voluminous universal program that has various associated routines for processing many separate tasks that may never be used in a particular application.

As one example, let’s say that the task of the video conveyor in a particular application is to convert the interlaced information of Y'CbCr 4: 2: 2 images to an intermediate linear line-by-line RGB 4: 4: 4 color space by creating composite images in the color space RGB and then outputting the resulting processed information to the television receiver. This sequence of operations includes certain operations (for example, upsampling, transforming the transfer matrix, applying the transform function, creating composite images, etc.), but not other operations. Therefore, the configuration module 1302 should only generate the code necessary to perform the calculations that are required, and nothing more.

The configuration operation is described above in the context of the inclusion or exclusion of modules. However, the advanced configuration functionality may perform other optimization operations, such as combining selected code modules with a specific effective method, removing redundant code that is common to selected code modules, etc.

A number of methods are provided in order to implement configuration module 1302. One way to implement this functionality is to use the resources of existing configuration tools (for example, the traditional C ++ compiler), which already have a certain ability to analyze the source code and remove redundant code, and apply this functionality to the current task of optimizing pipeline code by specifying and selecting the appropriate modules.

FIG. 14 shows a procedure 1400 that summarizes the above description in the form of a flowchart of a method. Step 1402 entails inputting requirements for a video pipeline. Step 1404 entails determining an optimal code that satisfies the input requirements. Step 1406 entails the conclusion and execution of the optimal code.

IN 2. General principles of using GPU for image processing

FIG. 15 shows an overview of a system 1500 that can be used to implement aspects of the image pipeline shown in FIG. 12 (and, more abstractly, the video processing operations 100 illustrated in FIG. 1). System 1500 may represent a computer (eg, a personal computer) containing one or more CPUs. System 1500 assigns certain image processing tasks (or all image processing tasks) shown in FIG. 1 to the functionality of a graphics module. The graphics processing functionality may comprise one or more graphics processors (referred to as GPUs in the art). In general, FIG. 15 includes a dashed line to demarcate the functions performed by the CPU of the system 1500 from the functions that can be performed by the graphics module of the system 1500 (although this separation is merely exemplary, other CPU / GPU assignments are possible).

As initial data, a GPU is, in general, a processing device similar to a CPU, but, as a rule, with less ability to execute branch decisions. Systems typically use a GPU in order to perform information-rich preparation tasks for viewing, which are performed periodically, for example, preparing to view information using a three-dimensional processing pipeline (including vertex shaders, pixel shaders, etc.). Thus, the main area of GPU use is gaming technology and simulation, which uses the GPU to prepare various scenes, characters, special effects, etc. for viewing. The assignment of periodic or information-intensive GPU tasks frees up the system CPU in order to perform other high-level control tasks, and thereby increases the performance of such a system. In this case, instead of generating information for playing games, the system 1500 uses the functionality of the graphics module in order to change the received image information (e.g., video information) before output to any type of output device. For example, one application of the 1500 system is to receive video information from a DVD, create composite images from video information in graphic functionality (for example, by combining text labels on video information), and then output the resulting signal to a television receiver.

Using the above general presentation, each of the exemplary components is described below alternately. The following section (B3) provides more specific information regarding how the functionality of the graphics module can be used to implement aspects of the pipeline 100 shown in FIG. 1.

To begin with, the 1500 system receives video information from any of a number of sources. For example, system 1500 may receive video information from a network 1502 (such as a remote source connected to the Internet), any type of database 1504, any type of computer-readable disk medium 1506 (such as an optical disk, DVD, etc.) or any another source 1508. In any case, the received information may comprise a combination of image information and audio information. A demultiplexer unit 1510 separates audio information from image information. The audio processing functionality 1512 processes the audio information.

The image decoder 1514 processes the image information. The decoder 1514 images can convert the compressed information of the images from the received format to any other format, etc. The output of the image decoder 1514 may include so-called ideal image information, as well as image substream information. Ideal image information constitutes the main image stream that must be prepared for viewing on a display device. The image substream information may constitute any additional information associated with the ideal image information, for example, information of coded captions between frames, all kinds of graphic overlay information (e.g., various graphic editing controls), various types of sub-images represented by DVD players, and so on. .d. (In another implementation, certain aspects of a video decoder may be assigned to the functionality of a graphics module.)

In one exemplary implementation, the video mixing renderer (VMR) module 1516 plays a central role in processing the image information thus obtained. As an overview, the VMR module 1516 interacts with the graphical interface 1518 and the display driver 1520, which in turn controls the graphics module 1522. This interaction may entail exploring the capabilities of the graphics module 1522. This interaction also entails the coordination of information processing. images through a graphical interface 1518, a display driver 1520, and a graphics module 1522. In one implementation, a graphical interface 1318 can be implemented using DirectDraw functionality by providing available through Microsoft® Corporation DirectX. DirectDraw acts in this context as a messaging tool for the data connection of the VMR module 1516 with the graphics module 1522. The graphics module 1522 itself can constitute a fixed module in a computer or similar device, or it can constitute a removable unit, such as a graphics card. (In general, in most cases, the designated applications mentioned in the first section of this application provide information regarding sample data structures that the VMR module 1516 can use to interact with the graphics module 1522. Since this interaction is not important for the present applications, information about this interaction are not repeated in this document.)

The graphics module 1522 includes one or more graphic processors (GPUs) 1524. As mentioned above, the system can assign any combination of the processing operations shown in FIG. 1 to the GPU 1524. The GPU 1524 performs these tasks using the pixel shader 1526. The pixel shader means functionality that allows you to perform various types of operations with image information on a pixel-by-pixel basis. Subsection B3 provides additional information regarding the architecture of a typical pixel shader, as well as how this technology can be used to perform the operations shown in FIG.

The GPU 1524 can interact with a local storage 1528 associated with the graphics module 1522. This local storage 1528 can serve any number of storage related tasks. For example, this storage device 1528 may store the final image surface, which is then redirected to the output device 1530 (such as a display monitor, television receiver, storage device, network destination, etc.).

IN 3. Pixel Shader Enhancements

As raw data, FIG. 16 shows the architecture of a pixel shader 1600, which is typically used in 3D processing pipelines. Each pixel in a typical application of a pixel shader can be represented by a vector of four floating point values, for example, RGBA (red, green, blue, alpha), where each value corresponds to a separate channel. The architecture of the pixel shader 1600 includes a series of input / output registers (1602, 1604, 1606, 1608) and an arithmetic logic unit (ALU) 1610 for performing operations on input data. More specifically, these registers include color registers 1602. These registers 1602 route iterative vertex color data from a vertex shader (not shown) to the pixel shader 1600. Permanent registers 1604 provide user-defined constants to the pixel shader 1600. The output / temporary registers 1606 provide a temporary storage of intermediate calculations. In this set of registers, register r0 also receives the output of pixel shader 1600. Texture registers 1608 provide texture data ALU 1610 of the pixel shader. The ALU 1610 pixel shader executes program-based arithmetic and texture addressing instructions. The program includes a set of instructions selected from the set of allowed pixel shader commands.

The following description is specifically devoted to the use of texture registers 1608 (hereinafter referred to generally as “blocks”) to provide image information and filter weights of the ALU 1610 pixel shader. Therefore, additional background information is provided regarding the concept of textures in the context of FIG.

Speaking approximately, in gaming applications, a texture defines an image that is inserted onto polygonal surfaces that define characters, scenes, etc. 17 shows a texturing application operation 1700 in which texture 1702 is applied to polygon 1704. Polygon 1704 consists of two triangular non-derivative elements arranged to form a rectangle. Polygon 1704 includes four peaks V1, V2, V3 and V4. Each vertex includes texture coordinates. The texture coordinates are relative to the traditional coordinate system U and V. In this coordinate system, the U coordinate generally corresponds to the X axis, and the V coordinate generally corresponds to the Y axis. Values along the U axis are fixed to vary in the range from 0.0 to 1.0, and the values along the V axis are fixed to vary in the range from 0.0 to 1.0.

The texture coordinates associated with the vertices specify how texture 1702 should be placed on polygon 1704. In the example case of FIG. 17, vertex V1 has texture coordinates 0,0, 0,0, which corresponds to the upper left corner of texture 1702. Vertex V2 has texture coordinates 1,0, 0,0, which corresponds to the upper right corner of the surface 1702. Vertex V3 has texture coordinates 0,0, 0,5, which corresponds to the center of the left edge of texture 1702. And vertex V4 has texture coordinates 1, 0, 0.5, which corresponds to the center of the right edge of the texture 1702. Therefore, when texture 1702 is displayed at the range 1704 in accordance with the texture coordinates, only the upper half of the texture applied to a polygon 1702 1704 1702 The result of applying texture to the polygon 1704 is shown in a textured surface 1706.

Three-dimensional processing pipelines typically provide a number of special texture processing operations designed in the context of generating game information. One special operation is referred to as a wrapping mode. In wrapping mode, the 3D processing pipeline repeats textures several times, for example, in one case in order to generate a row or matrix of textures that have the same content. The mirroring mode also duplicates the neighboring structure, but transposes (reflects) the texture like a mirror.

Additional information regarding general issues related to any pixel shader can be found in a number of paid publications, such as Wolfgang F. Engel, "Direct3D ShaderX: Vertex and Pixel Shader Tips and Tricks," Wordware Publishing, Inc., 2002.

Using the foregoing introduction, the rest of the description sets forth exemplary new techniques for using the pixel shader in order to implement aspects of the image processing pipeline shown in FIG.

To begin with, FIG. 18 shows an exemplary application of the 1800 pixel shader 1526 (FIG. 15) to implement an image processing filter having a four-tap core. Since the type of processing shown in FIG. 18 is also fundamental to many other operations performed by the pipeline 100, FIG. 1, the principles set forth in relation to FIG. 18 apply to other types of processing that may be performed by the pixel shader 1526. Case with The four-tap filter core, of course, is merely illustrative. The following examples explain how this model can be extended to other filter structures.

In general, the CPU 1524 may be configured to scan one or more input images, extract pixel values from each, apply input-based computation, and output one pixel. This operation can be expressed as follows: output (x, y) = function (input ₁ (x, y), input ₂ (x, y), ..., input _p (x, y), matrix constant [m]). In other words, this general expression means that the result (output (x, y)) mathematically depends on some function of the input signals (input ₁ (x, y), input ₂ (x, y), ..., input _p (x, y )) and, optionally, various given constants (matrix constants [m]).

In the specific context of image processing using filters, the pixel shader 1526 requires the input of one or more input images along with the associated filter weights to be applied to the images. More formally, the filter that generates the output (Out [x]) associated with the output pixel can be defined as follows:

Out [x] = sum (in [x-taps / 2 + i] · kernel [i], i = 0 ... taps-1).

In other words, the output for the pixel (Out [x]) represents a weighted summation of the various input terms. The kernel represents the weighting information that should be applied to the input members. Different input members, in turn, can represent biased versions of the same sample of input image information.

FIG. 18 shows how a pixel shader 1526 can implement the above equation. The illustrated four-tap example 1800 shows a sequence of texture units 1802 that provide input to an ALU 1804 pixel shader. The approach shown should assign filter weights to the first texture block and assign four different delta-shifted versions of the same image information to the next four texture blocks (representing four core taps). ALU 1804 pixel shader reads information stored in texture units (in one read operation) and provides one output for a specific pixel (in one write operation). This procedure is repeated many times to generate a complete image. This approach defines a one-dimensional convolutional filter with a base window of filter taps equal to taps.

The operation of example 1800 shown in FIG. 18 can be more formally expressed as follows. The approximate used one-dimensional core has pixels w wide and core weights w [-1], w [0], w [2] and w [3]. The texture of the weights is calculated by calculating the four weights of the filter core for each pixel. The one-dimensional kernel also includes four inputs defined as in [-1], in [0], in [1] and in [2]. The symbol Δ is specified as 1 / w. Based on the foregoing, the following information is assigned to texture units 1-5:

Texture 1: "texture of scales" with coordinates 0, ..., 1;

Texture 2: in [-1], input image with coordinates (0, ..., 1) + (- 1) · Δ (ie from 0-Δ to 1-Δ);

Texture 3: in [0], input image with coordinates (0, ..., 1) + (0) · Δ;

Texture 4: in [1], input image with coordinates (0, ..., 1) + (1) · Δ;

Texture 5: in [2], input image with coordinates (0, ..., 1) + (2) · Δ delta (ie 0 + 2 · Δ to 1 + 2 · Δ).

If tn represents a pixel extracted from texture unit n, then the calculations performed by the ALU 1804 pixel shader can be expressed using the following program:

Algorithm 3: Implementing a filter using a pixel shader

Let w [0] = t1.red

Let w [l] = t1.green

Let w [2] = t1.blue

Let w [3] = t1.alpha

Out.rgba = t2.rgba · w [0] + t3.rgba · w [1] + t4.rgba · w [2] + t5.rgba · w [3]

In other words, the matrix w is first assigned weighting information values that are stored in the first texture block (t1). Then, the output result (Out.rgba) is generated by modifying the information of the offset images in the texture blocks t2-t5 by the weight w. Suffixes attached to register information specify channel information. Therefore, the output of Out.rgba represents a vector of four floating point values stored in the green, red, blue, and alpha channels. You can see that the above algorithm requires, for example, taps + 1 input texture units, since one unit is assigned to store weighting information.

In general, if the input image information is expressed in RGB format, then texture units can store the same values of the red, green, and blue components. However, if a pixel shader is used to process luminance-related image information (such as YUV), then texture units can store more luminance information regarding color information (U, V). This measure takes advantage of the fact that the human eye is more sensitive to luminance information than to chroma information, so it is not necessary to store and process as much chroma information as luminance information in order to achieve acceptable output results.

A number of variations and optimizations of the above approach are allowed.

According to one embodiment, the above-described wrapping mode can be used to process any texture as an endless element of image information. In this mode, one approach is to set the coordinates of the input weighting texture from 0 to 1.0 / gcd (source width, destination width) instead of 0, ..., 1. When applying this texture, the texture prefetch unit will automatically “wrap” to the next copy of the weighing information stored in the texture. This measure enables the designer to reduce kernel storage requirements while ensuring that information is used with duplication if necessary.

In another embodiment, the core may have more than four taps.

To resolve this situation, an implementation can break down the kernel weighting information sets into groups of four values and assign each an additional input weighting texture.

Consider the case where six taps are provided. In this case, the implementation can use two weighting textures (the first texture with four values, and the second texture with the remaining two values, leaving the two segments unused). This implementation also requires six input image textures. Thus, to implement this structure, an eight-texture GPU can be used to filter with six taps in a single pass.

In another embodiment, the number of taps may exceed the number of texture units. Note that the calculation of the output image information is a summation of the taps members. Therefore, a larger number of taps can be calculated by splitting the summation calculation into several processing passes. For example, if the filter core has 12 taps, then one implementation can calculate Out [x] = sum (in [x + i] · w [i], i = 0 ... 11) as:

Sum1 [x] = sum (in [x + i] · w [i], i = 0 ... 3)

Sum2 [x] = sum (in [x + i] · w [i], i = 4 ... 8)

Sum3 [x] = sum (in [x + i] · w [i], i = 9 ... 11)

This implementation can then combine the results using a final pass:

Out [x] = sum1 [x] + sum2 [x] + sum3 [x] (three reads, one record)

Another possible implementation may generate an aggregated result Out [x] through the following series of operations:

Out [x] = sum1 [x] (write)

Out [x] = out [x] + sum2 [x] (read, read, write)

Out [x] = out [x] + sum3 [x] (read, read, write)

The second approach requires significantly less memory than the first approach, but also requires twice as many memory accesses as compared with the first approach (for example, four reads and three records). This feature of the second strategy may make it unacceptable, because the read-modify-write cycles in the GPU are very expensive or even forbidden.

In another embodiment, the implementation may provide a special mode (“critical pixel”) that interprets all pixels outside the specified image information as black (or some other default value). This measure can be applied in order to automatically hide boundary conditions in the information that should be displayed.

According to another embodiment, the implementation may apply the aforementioned mirroring mode when processing image information. This mode reflects image information horizontally or vertically when accessing image information outside the boundaries of image information.

FIG. 19 shows a procedure 1900 that summarizes many of the above features. Step 1902 entails the assignment of various input information to the various input units of the pixel shader. This information may include image information and weighting information applied to various texture blocks, constants applied to various constant blocks, and the like. Step 1904 entails computing the output information of the images on a pixel-by-pixel basis based on program instructions provided in the ALU 1804 pixel shader. Step 1906 determines whether additional passes are required. If so, procedure 1900 repeats one or more of the operations shown in FIG. 19 one or more times. Step 1908 gives the final output.

Finally, FIG. 20 shows an embodiment of a pixel shader 1526 that performs some of the operations presented in the context of FIG. 1. FIG. 20 specifically addresses one exemplary embodiment in which the pipeline receives YUV 4: 2: 2 image information, upsambles it, converts it to a linear RGB shape, and performs scaling of the resulting linear information. The processing shown in FIG. 20 is based on the concepts developed above with respect to FIGS. 18 and 19.

Note that 4: 2: 0 image information expressed in a hybrid flat format (such as NV12) can be interpreted as containing a brightness plane and a color plane. Therefore, certain operations with this information can be performed in parallel.

One approach is to divide the texture units into two groups, one group for use in the interpolation of the color component. The upsampling operation activates the scaling enlargement operation, which can be used to generate 4: 4: 4 color information. Then, the pixel shader 1526 can use the luminance component and the interpolated chrominance component in order to calculate the corresponding R'G'B 'information on a pixel basis using the transfer matrix. Next, the pixel shader 1526 may apply a transform function in order to linearize the image information. After that, the pixel shader 1526 can be used to perform additional scaling in RGB space using the second group of texture units.

More formally, the following is an exemplary sequence of steps that can be used to convert image information in the manner described above.

1) Setting the textures 1-5 by the above method (shown in FIG. 18) in order to perform the first scaling operation.

2) Using the above-described scaling operation in order to calculate the СbСr information at a sampling rate twice as large as for the brightness information.

3) Loading the matrix for converting the color space from Y'CbCr to R'G'B 'into an array of 16 constraints as a matrix M.

4) Calculation of R'G'B 'information as follows:

Calculation R '= dotProd4 (M [0], aY'CbCr)

Calculation G '= dotProd4 (M [1], aY'CbCr)

Calculation B '= dotProd4 (M [2], aY'CbCr)

Calculation A = dotProd4 (M [3], aY'CbCr)

5) Compute RGB from R'G'B 'using the conversion function.

6) Performing a second scaling operation by computing scaled linear RGB data using the above algorithms (with reference to FIG. 18), but for textures 6-12.

7) After the horizontal scaling is done, apply the vertical scaling to the RGB information.

20 shows a procedure 2000 that describes the above algorithm in the form of a flowchart of a method. Step 2002 entails the assignment of information to the respective texture units (for example, the first set of texture units). Stage 2004 entails the use of a first set of texture units to perform up-sampling of color information. Step 2006 entails loading constraints to use when performing color space conversion in the pixel shader 1526. Step 2008 entails using constraints to convert image information to non-linear R'G'B '. Stage 2010 entails converting the R'G'B 'information into a linear RGB form. In addition, stage 2012 entails scaling of RGB information.

C. Exemplary computing environment

In one exemplary implementation, various processing aspects shown in the previous drawings may be performed by computing equipment. In this case, FIG. 21 provides information regarding an exemplary computing environment 2100 that can be used to implement the processing aspects shown in the preceding drawings. For example, a computer may be used to implement part or all of the image processing pipeline 100 shown in FIG.

Computing environment 2100 includes a general purpose computing machine 2102 and a display device 2104. However, computing environment 2100 may include other types of computing equipment. For example, although not shown, computing environment 2100 may include handheld or travel devices, set top boxes, game consoles, processing functionality integrated into video processing / presentation devices (e.g., television receivers, digital video recorders, etc.) , mainframes, etc. Additionally, FIG. 21 shows elements of a computing environment 2100 grouped to simplify an explanation. However, computing environment 2100 may use a distributed processing configuration. In a distributed computing environment, computing resources can be physically distributed around the environment.

An exemplary computer 2102 includes one or more processors 2106, a system storage device 2108, and a bus 2110. A bus 2110 connects various system components. For example, bus 2110 connects processor 2106 to system storage 2108. Bus 2110 can be implemented using any type of bus structure or combination of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any from many bus architectures. Computing machine 2102 can also connect to one or more GPU devices (not shown) in the manner described above.

Computing machine 2102 may also include a variety of computer-readable media, including many types of volatile and non-volatile media, each of which may be removable or stationary. For example, system memory 2108 includes a computer-readable medium in the form of a volatile memory, such as random access memory (RAM) 2112, and non-volatile memory, such as read-only memory (ROM) 2114. ROM 2114 includes an input- output (BIOS) 2116, which contains basic procedures that help transfer information between elements of the computer 2102, for example, during boot. RAM 2112 typically comprises data and / or program modules in a form that can be quickly accessed by processor 2106.

Other types of computing storage media include a hard disk drive 2118 for reading and writing to a stationary volatile magnetic medium, a magnetic disk drive 2120 for reading or writing to a removable non-volatile magnetic disk 2122 (for example, a flexible disk), and an optical drive 2124 disks for reading or writing to a removable non-volatile optical disk 2126, such as a CD-ROM, DVD-ROM or other optical media. A hard disk drive 2118, a magnetic disk drive 2120, and an optical disk drive 2124 are connected to the system bus 2110 via one or more storage media interfaces 2128. Alternatively, the hard disk drive 2118, the magnetic disk drive 2120, and the optical disk drive 2124 may be connected to the system bus 2110 via a SCSI interface (not shown) or other connection mechanism. Although not shown, computer 2102 may include other types of computer-readable media, such as magnetic tape cassettes or other magnetic storage devices, flash memory cards, CD-ROMs, universal digital disks (DVDs), or other optical storage devices that are electrically erasable programmable read-only memory (EEPROM), etc.

In general, the above computer-readable media provides non-volatile storage of computer-readable instructions, program modules, and other data for use by computing machine 2102. For example, readable media may store operating system 2130, application modules 2132, other program modules 2134, and program data 2136.

Computing environment 2100 may include multiple input devices. For example, computing environment 2100 includes a keyboard 2138 and a pointing device 2140 (e.g., a mouse) for entering commands and information into computing machine 2102. Computing environment 2100 may include other input devices (not illustrated), such as a microphone, joystick , gaming tablet, satellite dish, serial port, scanner, card readers, digital or video cameras, etc. Input / output interfaces 2142 connect input devices to a processor 2106. Generally, input devices can be connected to a computer 2102 via any type of interface and bus structures such as a parallel port, a serial port, a game port, a universal serial bus (USB) port, and etc.

Computing environment 2100 also includes a display device 2104. Video adapter 2144 connects the display device 2104 to bus 2110. In addition to display device 2104, computing environment 2100 may include other peripheral output devices, such as speakers (not shown), a printer (not shown) etc.

Computing machine 2102 operates in a networked environment using logical connections with one or more remote computing machines, such as remote computing device 2146. Remote computing device 2146 may comprise any type of computing equipment, including a general purpose personal computer, a portable computing machine, server, game console, network expansion device, etc. Remote computing device 2146 may include all of the features described above with respect to personal computing machine 2102, or any subset thereof.

Any type of network 2148 can be used to connect a computing machine 2102 to a remote computing device 2146, for example, a WAN, LAN, etc. Computing machine 2102 connects to network 2148 via network interface 2150, which can use broadband, modem, DSL, or another connection strategy. Although not illustrated, computing environment 2100 can provide wireless functionality for connecting computing machine 2102 to remote computing device 2146 (e.g., via modulated radio signals, modulated infrared signals, etc.).

In conclusion, in this application a number of examples are presented in the alternative (for example, case A or case B). In addition, this application contains cases that combine alternatives in one implementation (for example, case A and case B), even if this application may not explicitly mention these combined cases in each example.

Moreover, a number of symptoms are described herein by initially identifying exemplary problems that these symptoms can solve. This manner of explanation is not an acceptance of others recognizing or formulating problems in the manner specified in this document. Recognition and formulation of problems in the field of video processing technology should be understood as part of the present invention.

Although the invention has been described in a language characteristic of structural features and / or methodological actions, it should be understood that the invention defined in the appended claims is not necessarily limited to the described characteristic features or actions. More precisely, characteristic features and actions are disclosed as exemplary forms of implementation of the claimed invention.

Claims (20)

1. A method of processing image information in a linear form, the method comprising the steps of:
receive image information in interlaced nonlinear form;
converting image information into a linear form by applying a conversion function to image information;
converting image information into line-by-line form by interleaving image information;
performing image information processing in a linear linear form and performing at least one scaling operation, wherein
the scaling operation entails the preliminary calculation of a number of filter cores and the subsequent application of pre-calculated filter cores in order to process image information. 2. The method according to claim 1, in which the preliminary calculation also entails calculating the number of filter cores that are needed, and determining how many taps each core should have. 3. The method according to claim 2, in which the use of filter cores entails cyclic repetition of a sequence of pre-calculated filter cores when processing a row or column of image information. 4. The method according to any one of the preceding paragraphs, in which the received image information is in a color space associated with brightness. 5. The method according to claim 4, wherein the method further comprises the steps of converting the image information into a non-linear color space R'G'B 'before converting the image information into a linear form by applying the transformation matrix to the image information. 6. The method according to claim 1, wherein the processing comprises the step of resizing the image information. 7. The method according to claim 1, in which the processing comprises the step of performing the operation of creating composite images with image information. 8. The method according to claim 1, wherein the processing comprises the step of performing alpha-conjugation of the image information. 9. The method according to claim 1, in which the processing comprises the step of performing the extraction of the edges of the image information. 10. The method according to claim 1, in which the received image information is in a color space associated with brightness, and in which the received image information has a smaller number of color samples in comparison with the brightness samples. 11. The method according to claim 10, wherein the method further comprises a step of up-sampling the image information in order to increase the number of color samples in comparison with the number of brightness samples. 12. The method according to claim 1, wherein the method further comprises quantizing and smoothing the image information after it is processed in order to reduce its accuracy, thereby providing information of the quantized images. 13. The method according to item 12, in which quantization and smoothing apply the error dispersion algorithm to reduce the amount of distortion in the information of the quantized images, the error dispersion algorithm distributes the error vector to adjacent image elements relative to the image element that is being processed. 14. The method according to claim 1, in which the linear form contains a pseudo-linear form. 15. The method according to 14, the method further comprises the step of performing an operation to remove the pseudo-linear form of image information after processing. 16. The method according to claim 1, wherein the method comprises at least one scaling operation. 17. The method according to claim 1, wherein the method defines an image processing pipeline, and wherein the optimized code program is used to implement the image processing pipeline. 18. The method according to 17, in which the optimized code is generated through the steps in which:
introduce image processing template requirements and
assembling code modules from a library of code modules that can be used to implement requirements, while code modules that are not required are not used in optimized code. 19. An image processing conveyor comprising:
means for receiving image information in interlaced nonlinear form;
means for converting image information into a linear form by applying a conversion function to image information;
means for converting image information into line-by-line form by interleaving image information;
means for processing image information in a linear linear form; and
means for performing at least one scaling operation, wherein
the scaling operation entails the preliminary calculation of a number of filter cores and the subsequent application of pre-calculated filter cores in order to process image information. 20. A machine-readable medium storing computer-readable instructions configured to implement the method of claim 1.

Images