KR20010006040A - Method and apparatus for assessing the visibility of differences between two signal sequences - Google Patents

Abstract

Two input signal sequences are disclosed, for example, a method and apparatus 110 for evaluating visibility differences between image sequences. The apparatus includes a perceptual metric generator 112 having an input signal processor 210, a luminance processor 220, a color processor 230, and a perceptual metric generator 240, 250, 260.

Description

METHOD AND APPARATUS FOR ASSESSING THE VISIBILITY OF DIFFERENCES BETWEEN TWO SIGNAL SEQUENCES

Signal Processing Systems For example, imaging system designers often evaluate the performance of their designs with physical parameters such as contrast, resolution, and / or bit-ratio efficiency during compression / decompression (codec) processes. While these parameters can be easily measured, there may be no accurate measurement equipment to evaluate performance. The reason is that the end-user of the imaging system may not be able to see the visibility of artifacts or distortions and, in some cases, the presence of tumors in an image, such as a magnetic resonance imaging (MRI) image or a computer-assisted tomography (CAT) scan image. It is generally more relevant to subjective visual performance, such as improvements in imaging features that can expose information.

For example, two different codec algorithms for generating two different codec images may be used to process the input image. If the measurement of codec image fidelity is based solely on parameters such as the implementation of the squared mean error (MSE) operation for both codec images without considering the psychophysical characteristics of human vision, the codec image with a low MSE value Has substantially larger visual distortion than codec images with high MSE values.

Accordingly, there is a need in the art for a method and apparatus for evaluating the influence of physical parameters on a signal processing system such as the subjective performance of an imaging system.

This application was filed on April 4, 1997. No. 60 / 043,050 and filed Feb. 2, 1998. 60 / 073,435 claims the rights of provisional applications in the United States and is hereby incorporated by reference.

An apparatus and an ancillary method for evaluating and improving the performance of a signal processing system. In particular, the present invention relates to a method and apparatus for evaluating the difference in visibility between sequences of two signals.

1 is a diagram illustrating a signal processing system of the present invention;

2 is a block diagram of a perceptual metric generator;

3 is a block diagram of an input signal processor;

4 is a block diagram showing a luminance processing unit;

5 is a block diagram of a color processing unit;

6 is a diagram showing a detailed block diagram of the luminance processing section;

7 is a block diagram showing a luminance metric generating unit;

8 is a block diagram showing a color processing unit;

9 is a block diagram showing a color metric generating unit;

10 is a graph showing luminance spatial sensitivity data;

11 is a graph showing luminance time sensitivity data;

12 is a graph showing luminance contrast determination data;

13 is a graph showing disk detection data;

14 is a graph illustrating checkerboard detection data;

15 is a graph showing edge sharpness determination data;

16 is a graph depicting color space sensitivity data;

17 is a graph showing color contrast determination data;

18 is a graph illustrating grade prediction data;

19 is a block diagram of an optional embodiment of a luminance processing unit;

20 is a detailed block diagram of an alternative embodiment of the luminance processor of FIG. 19;

FIG. 21 is a detailed block diagram of an alternative embodiment of the luminance metric generator of FIG. 19; FIG.

FIG. 22 is a block diagram showing a luminance processor for processing 1 / 2-height images; FIG.

FIG. 23 is a block diagram showing a luminance metric generating unit for processing 1 / 2-height images; FIG.

24 is a detailed block diagram of an alternative embodiment of the color processing unit;

25 is a detailed block diagram of an alternative embodiment of the color metric generating unit;

FIG. 26 is a block diagram of a color processor for processing 1 / 2-height images; FIG. And

FIG. 27 is a block diagram illustrating a color metric generation unit for processing 1 / 2-height images.

The present invention is a method and apparatus for evaluating the difference in visibility between two input signal sequences, for example image signals. The apparatus includes a perceptual metric generator composed of an input signal processor, a luminance processor, a color processor, and a perceptual metric generator.

The input signal processor converts the input signals into psychophysically defined quantities, eg, luminance components and color components. The luminance processor processes the luminance components to generate a luminance perceptual metric, while the color processor processes the color components to generate a color perceptual metric. Finally, the perceptual metric generating unit correlates the luminance perceptual metric with the color perceptual metric to generate a unified perceptual metric such as a just-noticeable-difference map.

1 is a diagram illustrating a signal processing system 100 utilizing the present invention. The signal processing system includes a signal receiver 130, a signal processor 110, input / output devices 120, and a system under test 140.

Signal receiver 130 is provided for receiving input data signals such as image sequences from video devices or other time-varying signals such as microphones or audio signals from a recording medium. Thus, although the invention will be described below with respect to an image, it should be understood that the invention can be applied to other input signals as mentioned above.

The signal receiver 130 includes a data receiver 132 and a data storage unit 134. The data receiver 130 may include a plurality of devices such as a modem and an analog-to-digital converter. While analog-to-digital converters convert analog signals to digital signals, it is known that modems include modulators and demodulators for transmitting and receiving binary data over telephone lines or other communication systems. Accordingly, the signal receiver 130 may receive input signals "on-line" or in real time, and may convert them into digital form if necessary. The signal receiver 130 itself may receive signals from one or more devices, such as a computer, a camera, a video recorder, or various medical imaging devices.

The data storage unit 134 is provided for storing input signals received by the data receiving unit 132. The data storage unit 134 includes one or more devices such as a disk drive, semiconductor memory, or other storage medium. Such storage devices are provided for a method for delaying the input signals or for simply storing the input signal for the next procedure.

In a preferred embodiment, signal processor 110 is known as a perceptual metric generator (or visual discrimination measure (VDM)), central processing unit 114 (CPU), and video. A general purpose computer having a memory 116 to assist with processing. Perceptual metric generator 112 may be a processor coupled to the CPU through a physical device or communication channel configured from various filters. Optionally, the perceptual metric generator 112 may be implemented as a software application called from an input / output device 120 and executed by the CPU of the signal processor. Accordingly, the perceptual metric generator 112 of the present invention may be stored in a computer-readable medium.

The signal processor 110 is also not limited to keyboards, mice, video monitors, or magnetic and optical devices, but can also include a plurality of input / output devices, such as storage devices including diskettes or tapes such as hard disk drives or compact disk drives. Coupler 120 with each other. The input devices are provided for providing inputs (control signals and data) to the signal processor for processing an input image, the output devices displaying or storing the results, eg displaying a perceptual metric on a display device. Is provided for.

The signal processing system 100 using the perceptual metric generator 112 can predict the perceptual grades that a human subject will assign to two signal sequences, eg, attenuated color-image sequences associated with an undamped copy. have. The perceptual metric generator 112 evaluates the visibility difference between two sequences or streams of the input image, generating several difference estimates, including a single metric of the perceptual differences between the sequences. These differences are quantized in units of modified human JND metrics. This metric can be represented as a JND value, JND map, or probability prediction. In turn, the CPU can utilize JND image metrics to optimize various processing processes, including, but not limited to, digital image compression. To illustrate, the input image sequence passes through two different paths or channels to reach the signal processing system 100. The input image sequence directly arrives at the signal processing unit without (reference channel or reference image sequence) processing on one path, and the same input image sequence is processed in some form of the image sequence (inspected channel or inspection image). Sequence) passes through another path through the system 140 being inspected. The signal processing system 100 generates a perceptual metric measuring the difference between the two image sequences. The inspected system 140 may be many devices or systems, such as a decoder, a transport channel itself, an audio or video recorder, a scanner, a display device, or a transmitter. Accordingly, the signal processing system 100 can be employed to evaluate the subjective quality of the inspection image sequence. Through the use of the perceptual metric generator 112, weekly query evaluation of the test image associated with the reference sequence can be performed without a human observer.

Finally, the perceptual metric can be used to modify or control the parameters of the system under inspection via path 150. For example, the parameters of the encoder can be modified to produce an improved perceptual ratio, eg, an encoded image having a smaller visual distortion when the encoded image is decoded. In addition, although the system 140 to be inspected is described as a separate device, it will be understood by those skilled in the art that the system to be inspected may be implemented by a software implementation resident in the memory 166 of the signal processor, for example, by a video encoding method.

2 illustrates a simplified block diagram of the perceptual metric generator 112. In a preferred embodiment, the perceptual metric generator is an input signal processor 210, a luminance processor 220, a color processor 230, a luminance matrix generator 240, a color matrix generator 250, and a perceptual matrix It consists of a generator 260.

The input signal processor converts the input signal 205 into psychophysically defined quantities, eg, luminance and color components for the image signal. The input signals are two image sequences of arbitrary length. Although only one signal is shown in FIG. 2, it will be appreciated that the input signal processor can process one or more input signals simultaneously. The purpose of the input signal processor 210 is to convert the input image signals into light outputs, and to convert these light outputs into luminance and color separations into characterized psychophysically defined quantities.

In particular, for each field of each input sequence, there are three data sets labeled Y ', C _b ', and C _r 'at the top of FIG. 2 derived from the D1 tape. In turn, Y, C _b , and C _r data are converted into R ′, G ′, B ′ electron-total voltages resulting in display pixel values. In the input signal processor, the R ', G', and B 'voltages are also subjected to a process of being converted into two colored images and luminance which is subjected to the next processing step or parts.

The luminance processor 220 receives two images (inspection and reference) of luminance Y expressed as fractions of the maximum luminance of the display device. The outputs from the luminance processor 220 may be broadly defined as spatial and / or temporal responses due to various temporal and spatial processing, as described below. These outputs are provided to a luminance metric generator 240 that generates a luminance JND map. The JMD map is an image whose gray is proportional to the number of JNDs between the inspection and reference images at the corresponding pixel location.

Similarly, the color processor 230 processes the color components of the input signals to generate a color perceptual metric. That is, the color processor 230 is based on the CIE L * u * v * uniform-color space (which is generated for each of the color images u * and v *) and fractions of the maximum brightness of the display device. Accept two images of color (inspection and reference). Again, outputs from the color processor 230 may be broadly defined as spatial and / or temporal responses due to various temporal and spatial processing, as described below. In turn, the outputs of u * and v * processing are received and synthesized by the color metric generator 250 for generating a color JND map.

In addition, the two color and luminance processes are affected by the inputs from the so-called " masking " channel through the path 225, which makes the sensory difference appear more or less clear depending on the structure of the luminance image. Receive. Masking (self or vice versa) generally refers to a decrease in sensitivity in current information in a channel or neighboring channel.

The color, luminance, and composite luminance-color JND maps are available as outputs to the perceptual metric generator 260, respectively, with a small number of summary measurements derived from these maps. Since a single JND value (JND summaries) output is useful for modeling the overall rating of an eye of test sequence distortion, the JMD maps give a more detailed view of the stringency and location of the artifacts. In turn, the perceptual metric generator 260 correlates the luminance perceptual metric with the color perceptual metric to form a single perceptual image metric 270, for example, the entire just-noticeable-difference (JND). Generate a map.

It should be noted that two basic assumptions are the basis of the present invention. First, we "square" each pixel and set a clock angle of .03 degrees. This number is derived from the screen height of 480 pixels and the viewing distance of four screen-heights (the closest viewing distance defined in the "Rec 500" standard). When comparing the perceptual metric generator to human perception at distant viewing distances at four screen heights, the perceptual metric generator may overestimate human senses in spatial detail. In the absence of strict limitations of the viewing distance, the perceptual metric generator is adjusted to be as sensitive as possible within the recommendation of "Rec 500". However, the sensitivity of the perceptual metric generator can be adjusted for specific applications.

Secondly, the perceptual metric generator has the highest accuracy at about 20 ft-L (all construction frequencies measured) and is applied to .01-100 ft-L screen brightness (total sensitivity measured). It is also assumed that the varying luminance occurs in proportion to the sensory change at all construction frequencies, and this assumption is less important near 20 ft-L where additional measurements have occurred. Measurement and empirical data are presented below.

Hereinafter, the processing unit shown in FIG. 2 will be described in more detail with reference to FIGS. 3 to 7.

3 is a block diagram of the input signal processor 210. In the preferred embodiment, each input signal is processed into a set of four fields 305. Thus, Figure 3 the upper part of the Y ', C _b' a stack of four fields as shown, C _r 'represents a set of four successive fields from the test or reference video sequence. However, the present invention is not limited to such an implementation, and other field grouping methods may be used.

Multiple transformations are included in the input signal processor 210. Briefly, the input signal processor 210 first converts Y ', C _b ', C _r 'video input signals into electron-total voltages, and then converts them into luminance values of three phosphors, and finally Convert to psychophysical variables that separate luminance and color components. The triple stimulus value Y to be calculated below replaces the "model intensity value" used before color processing. In addition, color components u * and v * per pixel are generated according to the CIE uniform-color specifications.

It should be noted that if the input signal is always in an acceptable color space, the input signal processor 210 can be optimally implemented. For example, the input signal may have been previously processed in a suitable format and stored in a storage device such as a magnetic or optical drive and disk. Furthermore, although the invention is implemented with CIELUV, pixels mapped to the international standard uniform-color space, the invention can be employed and implemented to process input signals mapped to other spaces.

The first processing step 310 converts the Y ', C _b ', C _r 'data into R', G ', B' total voltages. In particular, the steps outlined below describe the conversion from Y ', _Cb ', _Cr 'image frames to R', G ', B' voltage signals driving a CRT device. Here, the astroper indicates that the input signals are gamma-precorrected at the encoder. That is, these signals can drive a CRT display device after conversion, whose voltage-current conversion function can be approximated by gamma nonlinearity.

It is assumed in two formats: the input digital image are 4:02 brightness correlation Y 'at the maximum resolution, and color correlation C' and C _{b 'r} in horizontally 1/2-resolution, wherein, Y', C _'b , C _'r data are assumed to be stored in the order set forth in ANSI / SMPTE standard 125M-1992. For example,

C ' _b0 , Y' ₀ , C ' _r0 , C' _b1 , Y ' ₁ , C' _r1 , Y ' _{3 ,.} , C ' _{bn / 2-1} , Y' _n-1 , C ' _{rn / 2-1} , Y' _n-2 ,... .

Matrix transformation (matrix convesion) in the listed steps, to the color of the up-sampling (upsampling) the two embodiments or substitute for, and Y 'C' C _{b 'r} from the R'G'B' under the There are three embodiments or alternatives for this. Such alternatives cover various common requirements, such as decoding requirements, which can be considered in various applications.

In particular, in the first color upsampling embodiment, the Y 'C' _b C ' _r arrays from a single frame extend to the maximum resolution of the Y' image. The C ′ _b and C ′ _r arrays are initially half resolution horizontally, then upsampled to produce maximum resolution fields. That is, alternating C _'b, C' _r pixels in the column direction are assigned to the even-numbered Y _i of the data stream. The C ′ _b , C ′ _r pairs associated with even Y _i are then computed by one of (i) replication and (ii) an average with its neighbors.

In a second color upsampling embodiment, the maximum resolution Y 'C' _b C ' _r arrays are divided into two fields. In the case of Y ', the first field contains odd lines of the Y' array and the second field contains even lines of the Y 'array. The same process is performed on the C ' _b and C' _r arrays to generate the first and second C ' _b and C' _r fields.

In the matrix transformation from Y 'C' _b C ' _r to R'G'B', the corresponding Y 'C' _b C ' _r values are the total input values R'G for each pixel of each of the two fields. Is converted to 'B'. The Y 'C' _b C ' _r values are related to the R'G'B' values by one of the following three optional equations. The first two equations were described in 1996 by Video Demystified Ch. By San Diego Hightext, Keith Jack. 3, p. Found at 40-42. Equation (3) is 1996 Wylie, Po. a. Corresponds to Equation 9.9 in A Technical Introduction to Digital Video by CA Poynton (where C _b is replaced by U and C _r is replaced by V). (2) is selected as the default and otherwise used unless the measurement of the display of interest is indicated.

The R ', G', and B 'arrays are received by the second processing step 320 in the input signal processor 210. The second processing step 320 applies point non-linearity to each of the R ', G', and B 'images. This second processing step models the conversion (fractions at maximum brightness) of the R ', G', B 'total voltages to the intensities R, G, and B of the display device. The nonlinearity also performs clipping at low luminance in each plane by the display device.

In particular, the transformation between (R ', G', B ') and (R, G, B) comprises two parts, one of which transforms each pixel value independently and the other of the converted pixel values Is to perform spatial filtering on. The two parts are described below.

Pixel-value transformations

First, a fraction for the maximum luminance R corresponding to the input R 'is computed for each pixel. Similarly, fractional luminances G and B are computed from inputs G ', B'. It is assumed that the maximum luminance from each gun corresponds to the input value 255. The following equations represent the transformation from (R ', G', B ') to (R, G, B).

The default threshold t _d is selected to be 16, corresponding to the black level of the display device, and γ is defaulted to 2.5.

The value 16 is selected for t _d to provide the display device with an active range of (255/16) ^2.5 which is nearly 1000: 1. This dynamic range is relatively wide and may not be needed where the ambient illuminance is about 1% of the maximum display white. Thus, the physical fidelity can be maintained even when the perceptual generator adopts a value 40 which provides a dynamic range of 100: 1 as the black level instead of the value 16. In fact, a smaller dynamic range makes it possible to save on computational cycles, for example one or two bits during processing.

Two observations on the display are described below. The first observation includes a dependency on absolute screen brightness. The predictions of the perceptual metric generator implicitly apply only to the luminance levels generated by measuring the perceptual metric generator.

For typical measurement data (JJ Koenderink and AJ van Doorn, "The Spatiotemporal Contrast Detection Critical Surfaces Are Bimoses," Optical Characters 4, 32-34 (1979), the retinal illuminance was 200 trolands using a default pupil of 2 mm diameter. This means a screen brightness of 63.66 cd / m 2, or 18.58 ft-L. The measured luminance can be compared with the luminance of the displays used for subjective grading checking. For example, although the maximum-white luminance of both experiments is 71 and 97 ft-L, the luminance at pixel value 128 is 15 and 21 ft-L, respectively. Considering these values and the fact that the overall sensitivity of the perceptual metric generator was measured from .01 to 100 ft-L, it can be concluded that the perceptual metric generator was applied to screen luminances from about 20 to 100 ft-L. Can be.

The second observation involves the relationship of equation (4) in other models. The offset voltage t _d (eg from the grid setting between the cathode and the TV screen) is the model presented by Poynton, R = k [R '+ b] ^γ (and similar for G and B). Can be used to transform equation (4) (CA Poynton, "Gamma" and its disguises: The nonlinear mappings of intensity in perception, CRTs, Film, and Video, SMPTE Journal, 1993, 12, pp. 1099). -1108). By defining a new voltage R " R " -t _d , a model of Shepton can be obtained. Thus R = k [R " + t _d ] ^γ is similar for G and B. By writing equation (4) rather than Shepton's equation, it is assumed that the offset voltage is -t _d . It is also assumed that there is no ambient illuminance.

In the case of ambient illuminance, the voltage offset can be neglected, and Equation (4) is based on the model proposed by Meyer ("The importance of gun balancing in monitor calibration," in Perceiving, Measuring, and Using Color (M). Brill, ed.), Proc. SPIE, Vol. 1250, pp. 69-79 (1990)), ie, R = kR ′ ^γ + c. Similar expressions are given for G and B. If the ambient illuminance is given, equation (4) is replaced by a Meyer model with k = (1/255) ^γ + c and c = .01.

This perceptual metric generator provides three options for specifying a vertical representation of (R, G, B) images, each frame (in sequential images), and odd and even fields.

Optional 1. Frame

The image is full-height and contains one sequentially scanned image.

Option 2. Maximum-Height Interlace

Since half-height images are in an interlaced screen, they are dotted with blank lines to achieve maximum-height. The blank lines are subsequently filled by interpolation as described below.

Option 3. Maximum-Height Interlace

Process half height images directly.

The first two options are more faithful to the video image structure, with the third option having the advantage of reducing processing time and memory requirements by 50%. Since options 1 and 2 operate on the maximum-height image, the luminance and color processing for options 1 and 2 are the same. The three options are described in detail below.

Spatial Pre-Filtering

Options 1 and 3 above do not require spatial pre-processing. However, there is spatial pre-filtering associated with maximum-height interlace option 2.

In order to accommodate the spread of light from a line to an inter-line pixel in a field, the R, G, and B field images must also undergo a line interpolation process. Four interpolation methods are described below, but the present invention is not limited by these interpolation methods. In each method, the entire frame is read and then each pixel of the lines belonging to the inactive field is replaced with values computed from the pixels directly above and below. For methods (3) and (4), the operation also uses pixel values from the inactive field.

When marking inactive line pixels to be interpolated P _inactive and P _inactive up and down active line pixels as P _above and P _below , respectively, the above four methods are:

Method (1) Average is the default.

Returning to FIG. 3, after the nonlinear processing, the third processing step 330 replaces the interline values of the R, G, and B fields with interpolated values from up and down, thereby spreading the vertical electrons spread at the interline location. Model the beam spot. The vector (R, G, B) at each pixel in the field is then linearly transformed (depending on the display phosphors) to CIE 1931 tristimulus coordinates (X, Y, Z). As mentioned above, the luminance component Y of the vector is provided to the luminance processing unit 220.

In particular, the CIE 1931 Tristimulus coordinates X, Y, and Z are computed for the luminance values R, G, B, respective pixels of a given fraction. This process requires the following inputs depending on the display device: color coordinates (x _r , y _r ), (x _g , y _g ), (x _b , y _b ) and monitor white point of the three phosphors Chromaticity of (x _w , y _w ).

The white point is selected as corresponding to roughness D65 such that (x _w , y _w ) = (. 3128, .3292) (G. Wyszecki and WS Stile, Color Science, 2nd ed., Wiley, 1982, p. 761). (X _r , y _r ) = (. 6245, .3581), (x _g , y _g ) = (. 2032, .716), and (x _b , y _b ) for red, green, and blue phosphors = (. 1465, .0549) corresponds to the currently available phosphors very close to the NTSC phosphors. However, Table 1 below shows the different display phosphor coordinate (phosphor primary chromosome) options. ITU-R BT.709) is the default.

Table 1. Display illuminant coordinate options

Using the above parameter values, the pixel's X, Y, Z values are given by the following equations:

Where z _r = (1-x _r -y _r ), z _g = (1-x _g -y _g ), z _b = (1-x _b -y _b ), Y _0r , Y _0g , Y _0b Is given by the following equation:

Where z _W = (1-x _W -y) (D.Post, Colorimetric measurement, calibration, and characterization of self-luminous displays, in Color in Electronic Displays, H. Widdel and DLPost (eds), Plenum Press, 1992, p.306).

Tristimulus values of the white point of the device are also needed. These values correspond to color (x _W , y _W ) _and are at maximum phosphor activity (R '= G' = B '= 255), thus Y = 1. Thus, the tristimulus values for the white point are (X _n , Y _n , Z _n ) = (x _W / y _W , 1, z _W / y _W ).

As an optional final step in deriving the X, Y, and Z values, adjustments can be made to accommodate the assumed ambient light resulting from hiding the reflection from the display screen. This control takes the following form:

Here, two user-definable parameters, L _max and L _n, are introduced and default values are assigned. The display maximum brightness, L _max, is set at 100 cd / m 2 to correspond to commercial displays. The veiling luminance L _n is set to 5 cd / m 2, consistent with the screen values measured under Rec 500 conditions.

Assume that the color of the ambient light is the same as the color of the display white point. In luminance-only implementation options that do not compute the neutral point (X _n , Y _n , Z _n ), the adjustment is instead of equation (6a):

It should be noted that the Since Y _n is always 1, this is the same as the Y component of equation (6a). It should also be noted that the amount L _max * Y is the luminance of the display of cd / m 2.

Returning to FIG. 3 again, in order to ascertain the approximate perceptual uniformity for the isoluminant color differences in the color space (at each pixel), individual pixels were removed in a fourth processing step 340 by CIELUM, International. Map to standard uniform-color space. The color components u * and v * of this space are provided to the color processor 230.

In particular, transform the X, Y, Z, values per pixel into the 1976 CIELUV uniform-color system (Wyszecki and Stiles, 1982, p. 165):

here,

It should be noted that the coordinate L * is not sent to the luminance processor 220. L * is used only when calculating color coordinates u * and v *. Thus, only u * and v * images are stored for other processing.

4 is a block diagram of the luminance processor 220. 4 may be understood as a flow diagram of luminance processing steps or a block diagram of a number of hardware components such as filters, various circuit components and / or an application specific integrated circuit (ASIC) for implementing such luminance processing steps.

Referring to FIG. 4, each luminance field is filtered and a four-level Gaussian pyramid, in order to model the psychophysical and physiological observed analysis of the input visual signal into different spatial-frequency bands 412-418. Down-sampled at 410 (pyramid generator). After the analysis, subsequent selective processing, eg adaptive filtering, may be implemented at each pyramid level (resolution level).

Then, nonlinear operation 430 is performed immediately after the pyramid analysis. This step is a gain-setting operation (standardization) based on the time-dependent window average of the maximum luminance within the coarsest pyramid level.

It should be noted that the intermediate standardization process 420 is implemented to derive the intermediate value I _norm . As described below, an I _norm value is employed to scale each of the four pyramid levels.

After normalization, the lowest resolution pyramid image 418 is subjected to temporal filtering (temporal filter) and contrast operation 450, and the other three levels 412-416 are subjected to spatial filtering and contrast operation 440. In each case, the contrast is the local difference of pixel values divided by the local sum, which is almost scaled. In the formula of the perceptual metric generator, this has established the definition of "1 JND" and proceeds to the next steps of the perceptual metric generator. (Measurement is repeated to calibrate the 1-JND analysis in an intermediate perceptual metric generator as described below). In each case, the contrast is squared to yield the contrast energy, as known. The algebraic sign of contrast is preserved to add back immediately before image comparison (JND map operation).

The following steps 460 and 470 (contrast-energy masking) form a gain-setting operation that divides each orientated response (contrast energy) as a function of all contrast energies. This synthesized attenuation of each response by other local responses is included to model visual "masking" effects, such as a decrease in sensitivity to distortions in "busy" image regions. In this step of the perceptual metric generator, a temporal structure (flicker) is also formed to mask the temporal difference. As described below, luminance masking is also applied to the color side.

Masked contrast energies (along with contrast codes) are used to generate the luminance JND map 408. In brief, the luminance JND map comprises: 1) classifying each image into each image with positive and negative components (half-wave rectification); 2) implementing local pooling (mean and downsampling to model local spatial summations observed in mental physics experience); 3) evaluating the channel of absolute image differences by channel; 4) coring; 5) generate cored image differences for power; And 6) up-sampling at the same resolution (which results in half the resolution of the original image by the pulling step).

19 is a block diagram of an optional embodiment of the luminance processor 220. In particular, the normalization steps 420 and 430 of FIG. 4 are replaced by the luminance compression step 1900. In sum, each luminance value of the input signal is first subjected to nonlinear fixation of compression, which will be described in detail below. Other steps of FIG. 19 are similar to FIG. 4. These similar steps are described above. A detailed description of the luminance processor of FIG. 19 for dissimilar steps is given below with reference to FIG. 20.

In general, the luminance processor of Fig. 19 is a preferred embodiment. However, since the two embodiments have different features, their performance may be different under different applications. For example, the luminance processor of FIG. 4 operates well on higher dynamic ranges, eg, 10-bit input images, compared to lower dynamic ranges.

5 is a block diagram of the color processor 230. 5 may be considered as a number of hardware components, such as a filter, various circuit components and / or an application specific integrated circuit (ASIC), for implementing color processing steps or such color processing steps. Color processing is parallel to luminance processing in many ways. Intra-image differences of the colors (u * 502 and v * 504) of the CIELV space are used to define the detection threshold in the luminance processor. In addition, in analogy of the luminance operation, the "contrasts" of the color defined by the u * and v * differences are subjected to the masking operation. Transducer nonlinearity makes the determination of the contrast increment between one image and another image dependent on the contrast energy common to both images.

In particular, FIG. 5 shows that each of the color components u * 502 and v * 504 passes through the pyramid analysis process 510, as in the luminance processor. However, while the luminance process performs four pyramid level analysis in the preferred embodiment, the color process is implemented in seven levels. This implementation points to the empirical fact that color channels are sensitive to spatial frequencies much lower than luminance channels (KT Mullen, "The contrast sensitivity of human color vision to red-green and blue-yellow chromatic gratings," J. Physiol. 359, 381-400, 1985). In addition, such an analysis takes into account the intuitive fact that color differences can be seen in wide and even areas.

Then, to reflect the natural insensitivity of the color channels to the flicker, a time process 520 is achieved by averaging over four image fields.

Subsequently, spatial filtering by the Laplacian kernel (530) is performed for u * and v *. This operation produces color differences in u *, v * that are quantitatively connected (by definition of a uniform color space) to JND. A value of 1 in this step means that a single JND was obtained from the analogy to Weber's-law-based contrast in the luminance channel. (As with luminance processing, 1-JND color units must undergo reinterpretation during measurement.)

This chrominance value is weighted, squared, and proceeds to contrast-energy masking process step 540 (with contrast logarithmic sign). The masking step performs the same function as in the luminance processor. The operation is rather simple because the difference only receives input from the luminance and color channels being evaluated. Finally, in step 50 the masked contrast energies are processed exactly as in the luminance processing section to generate a color JND map.

For each field in the video-sequence comparison, the luminance and color JND maps must first be converted to single-number summaries, ie, luminance and color JND values. In each case, the conversion from the map to the channel is performed by summing all pixel values through Minkowski addition. Subsequently, the luminance and color JND numbers are synthesized again through Minkoski addition to generate a JND evaluation value for the field processed by the perceptual metric generator 260. A single performance measure 270 for multiple fields of the video sequence is determined by adding, in Minkoski's sense, a JND estimate for each field.

6 is a block diagram illustrating a detailed block diagram of the luminance processor 220 of FIG. 4. The input check and reference field images are represented by I _k and I ^ref k (k = 0, 1, 2, 3), respectively. Pixel values at I _k and I ^ref k are _denoted by I _k (i, j) and I ^ref k (i, j), respectively. These values are Y tristimulus values 605 calculated by the input signal processor 210. Hereinafter, the operation procedure by the same I ^ref k and I _k, I _k field is only illustrative only. Note that k = 3 indicates the most recent field in the four-field sequence.

Spatial analysis at four resolution levels is achieved through pyramid processing or pyramid analysis computationally efficient methods of smearing and downsampling the image with a factor 2 at each of the successive worsening levels of resolution. When the original maximum resolution image is referred to as the zeroth level (level 0) of the pyramid, G ₀ = I ₃ (i, j). At lower resolutions, successive levels can be obtained by a so-called reduce (REDUCE) operation. That is, a three-tap low pass filter 610 with weights (1,2,1) / 4 is applied to G ₀ continuously in each direction of the image to produce a blurred image. The resulting image is subsampled by factor 2 (all other pixels are removed) to produce the next level G ₁ .

When expressed as fds1 () as an operator for filtering and downsampling by one pyramid level, a REDUCE process can be expressed as Equation (14).

The reduce process is applied to each new level cyclically. (P. J. Burt and E. H. Adelson, "The Laplacian pyramid as a compact image code," IEEE Transactions on Communication, COM-31, 532-540 (1983)).

Conversely, an operator EXPAND is defined that upsamples and filters by the same 3x3 kernel. This operator is denoted by usf1 () and is shown below.

The fds1 () and usf1 () filter kernels in each direction (horizontal and vertical) are k _d [1,2,1] and k _u [1,2,1], respectively, where the uniform-field values will be preserved. So that k _d and k _u are selected. For fds1, the constant k _d = 0.25 and for ufs1, k _u = 0.5 (due to zeros in the upsampled image). To perform usf1 as a proper operation, the kernel is replaced by linear interpolation of equality to replace zero values. However, for conceptual simplicity, it may be referred to as an "upsample-filter."

Normalization is then applied, where the intermediate value (denoted by I _{| v | 3} ) is averaged by four values, the maximum pixel values at level 3 for each field (k = 0, 1, 2, 3). Is calculated. This step weakens the influence of outliers in the maximum resolution (level 0) image by inherent smoothing in the pyramid analysis process. Then I _{| v | 3} is compared with the attenuated value of the normalization factor, I _norm , used in the previous band (k = 2). I _norm for the current band (k = 3) is set equal to the larger of these two values. Images for all four pyramid levels of the last field are _rescaled using the new value of I _norm to undergo saturation nonlinearization.

The following equations illustrate this process. If the pyramid levels from above are I _{3, l} (i, j) and 3 and l are the last field and pyramid levels, respectively,

Where I _norm = [αI ^(') _norm , I _{| v | 3} ] (615), I ^(') _norm is field-3 pyramid levels, m defaults for 2, and

The value of I _norm used in the previous band to standardize is, Δt is the inverse of the field frequency, and t _half = 1/2 is related to the rate of adaptation of human vision following the removal of bright stimulus . The values for 50 and 60 Hz are 0.9727 and 0.9772. The constant L _D represents the defaults in the value of 0.01 and the excess visual response (noise) present in the absence of light. Saturation nonlinearity in equation (14) is derived from biological basic models (see Shapley and Enroth-Cugell, 1984).

Directional spatial filters (center and periphery) are applied to level 0, 1, and 2 images for field 3. In contrast, a time filter is applied at the lowest level (level 3). That is, the pairs of the first and last fields are linearly synthesized into early and later images, respectively.

Center and peripheral filters 625 and 627 are separable 3x3 filters, providing all combinations of orientation: center vertical (CV), center horizontal (CH), peripheral vertical (SV), and peripheral horizontal (SH).

The filter kernels are as follows:

Level 3 initial 630 and later 632 images, respectively,

to be.

The constants t _e and t ₁ for 60 Hz are 0.5161 and 0.4848 respectively and 0.70 and 0.30 respectively for 50 Hz. Inputs for contrast computation are the center and peripheral images CV _i , CH _i , SV _i , and SH _i ; i = 0,1,2 for pyramid levels 0, 1, and 2, and the initial and Later images (E ₃ and L ₃ ; for pyramid level 3). The equation used to calculate the contrast ratio is similar to Michelson contrasts. For horizontal and vertical orientations, by one pixel, the contrasts are

to be.

Similarly, the contrast ratio for the time component

to be.

As determined by measurement with psychophysical test data, the wi values for i = 0,1,2,3 are 0.015, 0.0022, 0.0015, and 0.003, respectively.

The horizontal and vertical contrast-energy images 640 and 642 are computed by squaring the pixel values defined by the two preceding equations, thus obtaining equation (21) below.

Similarly, temporal contrast-energy image 650 is computed by square the pixel values:

Before squaring, the algebraic sign of each contrast ratio pixel value is reserved for later use.

Contrast-energy masking is a nonlinear function applied to each of the contrast-energys or images computed with equations (21) and (22). The masking operation models the influence of the constructional structure on the reference image sequence in determining the distortion of the inspection image.

For example, the inspection image and the reference image differ by as much as a low-amplitude spatial sine wave. This difference is known to be more visible when the two images have a common mid-contrast sipa of the same spatial frequency, rather than when the two images contain a uniform field (Nachmias and Sansbury, 1974). However, if the contrast of the common sine wave is too large, the difference in the image is less visible. It is also the case that sinusoids of other spatial frequencies may affect the visibility of the contrast difference. This action can be modeled by an increasing power function for nonlinearity and high contrast energies that are curved at low contrast energies. In addition, the following features can be seen approximately with human vision. Each channel masks itself, high spatial frequencies mask low frequencies (but not vice versa), and temporal flicker masks spatial contrast sensitivity (and vice versa).

In response to these characteristics of human vision, the following nonlinear form 660 (applied by one pixel) is applied:

Where y is the masked contrast energy: spatial, Hi or Vi (equation (21)) or temporal (T ₃ ) (equation (22)). The amount D _i is related (by one pixel) to the image depending on the pyramid level i to which y belongs. Perceptual metric generator operations found that the quantities β, σ, a, and c were 1.17, 1.00, 0.0757, and 0.4753, respectively, where d _y is the original algebraic sign of contrast y before squared.

The computation of D _i requires pyramid structure (filtering and downsampling) and pyramid reconstruction (filtering and upsampling). This D _i operation is shown in FIG. 6. First, E ₀ is calculated as the sum of H ₀ and V ₀ . The summation is downsampled by filtering, step 652, which is added to H ₁ + V ₁ to be E ₁ . Next, E ₁ is also filtered and downsampled and added to H ₂ + V ₂ to become E ₂ . In turn, E ₂ is also filtered and downsampled to E ₃ . In the meantime, the image of time contrasts T ₃ is multiplied by m _t and added to mftE ₃ to generate a sum represented by D ₃ .

In turn, D ₃ is repeatedly upsampled and filtered by step 654 to generate T ₂ , T ₁ , and T ₀ . Finally, the image D _i is defined as D _i = m _f E _i + T _i , i = 0, 1, 2. Where m _f is determined by calculating to be 0.05, mft is set to 0.0005, and mt is set to 0.05. The filtering, downsampling, and upsampling steps are the same as described above.

The above procedure explains that higher spatial frequencies mask lower frequencies (since D _i is affected by a pyramid level less than or equal to i), and the temporal channel is masked by all spatial channels. This masking operation is generally in harmony with biological observation. The amounts D _i , i = 0,1,2 also mask color contrasts as described below (but not vice versa).

FIG. 20 is a detailed block diagram illustrating an exemplary embodiment of the luminance processor 220 of FIG. 19. Since the luminance processor of FIG. 19 includes many similar steps as the luminance processor of FIG. 6, only the other steps will be described below.

One important difference is the replacement of the normalization steps of FIG. 20 to FIG. 6 with a luminance compression (nonlinearity of compression) step 2000. That is, nonlinearity includes a deceleration power function that is offset by contrast. Let the relative-luminance array from the last field be Y ₃ (i, j), where 3 refers to the latest field.

Set the maximum brightness L _max of the display to 100 cd / m 2. The function is tested with contrast-sensitive data at 8 cd / deg. It can be seen that the adjustable parameters m and L _D are 0.65 and 7.5, respectively. That is, L _d and m values are selected to match the contrast detection data to 0.001 to 100 ft-L (van Des and Bouman, 1967) luminance levels. In other words, equation 23a is verified against absolute luminance. For example, the maximum brightness change of the display will affect the total brightness output. Another way of looking at equation 23a is that the perceptual metric generator cooperates with the luminance-dependent contrast-sensing function.

Optionally, an additional luminance compression step 2000 (indicated by a box of hidden lines in FIG. 20) may be inserted at each pyramid level to allow the present perceptual metric generator to model contrast sensitivity as a function of luminance and spatial frequency. have. If not, performing one luminance compression step 2000 with only two parameters would be insufficient to model other spatial frequencies.

In particular, after pyramid analysis of each luminance image, nonlinearity is applied to each pyramid level k. At that time, compression non-forming for pyramid level k

Where again m (k) and L _D (k) are selected to match the contrast detection at luminance levels from 0.01 to 100 ft-L (van Nes et al. 1967). The value L _a is an offset for the ambient screen brightness (set to 5 cd / m 2 based on screen measurements) and L _max is the maximum brightness of the display (generally about 100 cd / m 2).

The data for verifying equation (23b) is tabulated below.

Each contrast modulation C _m in the table above is an empirical value that caused the just-discriminable contrast of the sine wave of the spatial frequency f _s and the luminance I ₀ of the retina. Since 2 mm artificial pupils are used for verification, the luminance values of the retina (I ₀ in trolands) are used to derive the luminance values (L cd / m 2). Multiply by A good starting point for the validation of all the m (k) and _D L (k) is to use the default values for the appropriate indices of 0.65 m is 8c / deg sine wave is detected, the appropriate value of _D L is 7.5 cd / M 2.

The spatial and temporal filtering of luminance for the two perceptual metric generators of FIGS. 6 and 20 are identical. However, the luminance contrast operation of the perceptual metric generator of FIG. 20 is obtained without the square operation. Steps 640, 642, and 650 are replaced with steps 2040, 2042, and 2050 in FIG. 20.

In particular, the contrast-responsive images are computed as clipped versions of the positive absolute values defined by equations (19) and (20) above. These quantities are calculated as equations (23c) and (23d).

The algebraic sign of each contrast ratio pixel value before the absolute-value operation must also be preserved for later use.

Another important difference between the perceptual metric generators of FIGS. 6 and 20 is the contrast energy masking. Unlike FIG. 6, the perceptual metric generators of FIG. 20 perform contrast energy masking 2060 in two separate steps: cross masking step and shelf masking step for each of the horizontal and vertical channels (see FIG. 20). ). Shelf masking reduces the sensitivity in the presence of information in the current channel. Cross masking, on the other hand, reduces the sensitivity when information appears in neighboring channels. In fact, the order of these two separate masking steps can be reversed. These contrast energy masking steps take the following form:

Where y is the contrast to be masked: spatial, Hi or Vi (equation (23c)) or temporal (T ₃ ) (equation (24d)). The amount D _i is associated with the image (by one pixel) that depends on the pyramid level i to which y belongs. The quantities b, a, c, mf, and mt are 1.4, 3/32, 5/32. Model validation was made to be 10/1024, and 50. dy is the algebraic sign of the preserved contrast before taking the absolute value.

The _Di operation is similar to that of FIG. 6 as described above. That is, fds () represents 3x3 filtering followed by downsampling per one pyramid, and usf () represents upsampling per one pyramid level following 3x3 filtering. First, the array E ₀ is calculated as follows:

Then, for i = 1,2, the arrays E _i are computed iteratively:

It said array E _i is followed by a contrast analyzer arrays time contrast image, to apply D _i T ₃ and T _i and the image is synthesized in the following manner:

Where the parameter mft = 3/64 modulates the intensity masking the temporal (flicker) luminance-channel along with all the spatial-luminance channels; The parameter mt = 50 modulates the intensity masking each of the spatial-luminance channels together with the temporal (flickr) luminance-channel.

FIG. 7 is a diagram illustrating a detailed block diagram of the luminance metric generating unit 240. Again, FIG. 7 is a flow chart of luminance metric generating steps or a luminance metric generator having a plurality of hardware components, such as filters, various circuit components and / or an application specific integrated circuit (ASIC) for performing such luminance metric generating steps. It can be considered as a block diagram. The structure described below applies to all masked-contrast images generated in FIG. 6 above:

Images in Paramides H and V (eg, Images H ₀ , V ₀ , H ₁ , V ₁ , and H ₂ , V ₂ ), Image T ₃ (with resolution at level 3), and reference sequence Corresponding images derived from DE (shown with superscript ref in FIGS. 6 and 7).

In the following processing, the first four steps are individually applied to the above images. In the following description, X represents any of these images derived from the test sequence, and X ^ref represents the corresponding image derived from the reference sequence. When giving this notation, the steps are as follows:

In step 710, the image X is separated into two half-wave rectified images, one for positive contrasts 712 and the other for negative contrasts 714. In the positive-contrast (so-called X ₊ ), the signs from the X contrast are used to assign zero to all the pixels in X ₊ with negative contrasts. The opposite process occurs in the negative-contrast image X ₋ .

In step 720, for each image X ₊ and X ₋ , a local fling operation is implemented by applying a 3 × 3 filter to wind the image with a filter kernel of 0.25 (1,2,1) horizontally and vertically.

Further, in step 720, the result images are downsampled by factor 2 in each direction to remove duplicate results from the fling operation. The same processing as that applied to X is performed for the corresponding reference image X ^ref .

In step 730, the absolute-difference image the | X ₊ - X ^ref ₊ | and | X _- - X _- ^ref | is calculated for each pixel. The resulting images are JND maps.

In step 740, a coring operation is performed on the JND maps. That is, all values below the threshold tc are set to zero. In a preferred embodiment, tc defaults to a value of 0.5.

In step 750, the Q-th power of this image is determined. In a preferred embodiment, Q is a positive integer defaulted to the value 2.

After this process is performed for all X, X ^ref pairs, the composite measurements are determined by iteratively upsampling, filtering and adding all the images up to the required level. This is accomplished as follows:

In step 760, the upsampling and filtering process is applied to the level-3 images derived from T ₃ , T ₃ ^ref to derive the level-2 images.

In step 761, the upsampling and filtering process is applied to the sum of the level-2 images from step 760 and the level-2 images derived from H ₂ , H ₂ ^ref , V ₂ , and V ₂ ^ref .

In step 762, the upsampling and filtering process is applied to the summation of the level-2 images from step 761 and the level-2 images derived from H ₁ , H ₁ ^ref , V ₁ , and V ₁ ^ref .

In step 763, the upsampling and filtering process is applied to the sum of the level-2 images from step 762 and the level-2 images derived from H ₀ , H ₀ ^ref , V ₀ , and V ₀ ^ref . The output on path 765 from step 763 is the luminance JND map.

It should be noted that before the final processing step 763, the resulting image is half the original image resolution. Similarly, each pyramid-level index in the present processing section is originally related to the pyramid level from which it is derived and is twice the image associated with the level after filtering / downsampling.

All images generated by the above repeated upsampling, filtering, and addition processes are Q-th-power-JND images. The level-0 image is used in two forms, where it is either sent directly to the summation process on path 764, or upsampled and filtered in step 763 with the original image resolution for display use.

21 is a detailed block diagram of an alternative embodiment of the luminance metric generator 240. Since the luminance metric generation process of FIG. 21 includes many similar steps as the luminance metric generation process of FIG. 7, only the different steps will be described below.

In particular, the " coring "step 740 and the " rising to Q ^th power " step 750 are replaced by a number of max and summation steps that maintain the running sum and running maximum of the channel outputs. The process shown in Figure 21 by the same up to step 730, the process of Figure 21 is absolute-difference image the | will be described from the end to determine the | X ₊ - X ^ref ₊ | and | ^ref _- X _- - X.

Then, after the process for all pairs of X and X ^ref is completed, the running-sum image is initialized in step 2140 to include the summation of the level-3 images derived from T ₃ , T ₃ ^ref . Likewise, the running-up image is | is initialized at step 2142 to include the maximum image-up as the running of each one point of | T ₃₊ - - ₃₊ X ^ref | | X and T _{3- 3-} ^ref.

The running-sum and running-maximum images are then upsampled and filtered by steps 2140a and 2142a, respectively, to include two level-2 images. The running-sum image is then updated by step 2144 by adding the level-2 images derived from H ₂ , H ₂ ^ref , V ₂ , and V ₂ ^ref . Similarly, the running-maximum image is updated by step 2146 compared to the level-2 images derived from H ₂ , H ₂ ^ref , V ₂ , and V ₂ ^ref .

The running-sum and running-maximum images are then upsampled and filtered by steps 2144a and 2146a, respectively, to include two level-1 images. The running-sum image is updated by step 2148 by adding the level-1 images derived from H ₁ , H ₁ ^ref , V ₁ , and V ₁ ^ref . Similarly, the running-maximum image is updated by step 2150 compared to level-1 images derived from H ₁ , H ₁ ^ref , V ₁ , and V ₁ ^ref .

The running-sum and running-maximum images are then upsampled and filtered by steps 2148a and 2150a, respectively, to include two level-0 images. The running-sum image is updated by step 2152 by adding the level-0 images derived from H ₀ , H ₀ ^ref , V ₀ , and V ₀ ^ref . Similarly, the running-maximum image is updated by step 2154 compared to the level-0 images derived from H ₀ , H ₀ ^ref , V ₀ , and V ₀ ^ref .

Finally, linear synthesis by one point of the running-sum and running-maximum images is performed in step 2160 to generate a luminance JND map by the following equation (23k):

Where kL = 0.783. The k value is determined by the Minkowski Q-norm approximation. Given the value of Q and the number N of images to be brought together, the approximate measurement is exactly identical to the Q-norm if all the compared trees (in one pixel) are equal or only one entry is non-zero. Confirm that value kL = [N-N ^{1 / Q} ] / [N-1]. In this case, N = 14 (number of channels), and Q = 2.

It should be noted that after this processing, the resulting image is half the original image resolution. Similarly, in this process each pyramid-level index is for the original pyride level, which is an index derived and twice the resolution associated with that level after filtering / downsampling.

Finally, it should be noted that the weights kL and 1-kL may be added to all the images generated by the repeated filtering / downsampling and addition / combining process to generate JND images. The level-0 image can be processed in bivalent form, where the level-0 image is sent directly to the JND summation process via path 2161 or the original image resolution for display use by step 2170. Up-sampled and filtered.

Generally, the luminance metric generator of FIG. 21 is a preferred embodiment, whereas the luminance metric generator of FIG. 7 is an optional embodiment. One reason is that the compounding-summing method is computationally cheaper. Therefore, if a dynamic range of constant setting is desired, the luminance metric generating portion of Fig. 21 is preferable. Alternatively, if a floating point processor is employed, the luminance metric generator of FIG. 7 may also be used.

1 / 2-high luminance processing

Since storage requirements and computational cycles are an important task of the processing process, the present invention provides an alternative embodiment of a perceptual metric generator capable of filling half-height images, such as top and bottom of an interlaced image. This embodiment reduces the amount of storage space needed to store the maximum-height images, while at the same time reducing the number of computation cycles.

If the 1/2 height images have to pass at the real image height without zero-filtering, the luminance processor 220 above must be modulated to reflect that the original vertical resolution is only half of the original horizontal resolution. do. 22 and 23 are block diagrams illustrating a luminance processing unit and a luminance metric generating unit for processing a 1 / 2-height image.

Comparing these diagrams (FIGS. 22 and 23) and the corresponding diagrams for the maximum-height interlaced images (FIGS. 20 and 21), it can be seen that many of the steps are the same. As such, the following description of FIGS. 22 and 23 limits the differences between the two understandings.

First, the highest-resolution horizontal channel H0 is removed. Secondly, the highest-resolution image is lowpass-filtered vertically (ie along a column) with a 3x1 "Kell" filter (vertical filter). The filter is an anti-aliasing filter in the vertical dimension to eliminate the effect of sampling on the half of the spatial frequency. That is, it is a low pass filter that is blurred vertically. Subsequently, the resulting vertically filtered image L0 is horizontally filtered by the 1 × 3 filter 2220 (kernel 0.25 [1,21]). The resulting image LP0 is a horizontally low-passed version of L0.

L0 and LP0 are then synthesized to generate a separated band pass (LP0-L0) by a lowpass (LP0) directional response similar to the (S-C) / (S + C) responses of the other directional channels.

In turn, image LP0 (720x240 pixels) is downsampled in the horizontal direction to the maximum height 1 / 2-resolution image 360x240. In this regard, the aspect ratio may allow the processing of this image and the remaining three pyramid levels to continue as in the maximum-height options.

The downsampling and upsampling between half-height images from level 0 and the maximum height images of level 1 are then performed by step 2232 (leveled 1x3 filter & ds) by 1x3 filtering / horizontal downsampling and step 2234. Each is implemented as horizontal upsampling (hus) / 1x3 filtering. Horizontal upsampling applies deciamtion by two factors in the horizontal dimension, i.e. by throwing every other column of the image. Horizontal upsampling inserts a column of zeros between each two columns present. The filter kernel after upsampling is defined by 0.5 [1, 2, 1] for reasons noted above.

FIG. 23 is a diagram illustrating a luminance metric generator for processing 1 / 2-height images. First, the highest-resolution horizontal channel H0 is not presented. For the V0 channel, a 1x3 filtering and horizontal downsampling step 2300 is provided to replace the 3x3 filtering and downsampling step, as used in other channels.

As the H0 channel is missed, the paths of various parameters and running-maximum and running-sum are modified. For example, the value of N that determines k varies from 14 to 12. The same value k = 0.783 is used for both maximum-height and half-height processing and is the average of the maximum-height and half-height constants calculated from the equation given above.

Finally, as in the maximum-height embodiment, the luminance map for summation measurements must be taken at the maximum image resolution before being displayed. Just before the display, the final JND map is at full resolution in the horizontal direction by upsampling following 1x3 filtering (kernel 0.5 [1,2,1]) in step 2310. In the vertical direction, line-doubling is performed at step 2320.

Since each spatial filter has horizontal and vertical spatial dependencies, there are some differences in the 1 / 2-height embodiment, as compared to its maximum-height copy. However, the half-height embodiment will only express some confusion in correlation with the subjective grade. Thus, non-interlaced options can be employed as viable and time-saving alternatives to interlaced options.

8 is a detailed block diagram of the color processor 230. Again, FIG. 8 can be understood as a block diagram having a flow chart of color processing steps or a number of hardware components for implementing the color processing steps, such as a filter, various circuit components and / or an application specific integrated circuit (ASIC). Apart from the pyramid having levels 0, 1, 2, the color processor 230 computes pyramids having levels 0, 1, 2 for u * 802 and v * 804.

Since the resolution is derived not by inter-receptor space but by inter-pix space, the spatial resolution of the color channels (i.e., the pyramid-level resolution of the maximum height) is determined by the luminance. It is selected to be the same as the resolution. The inter-receptor space is 0.0007 degrees of viewing angle and the inter-pixel space is 0.03 degrees protruding from the screen with 480 pixels in height when viewed at 4 times its height. On the other hand, in 1985 Morgan and Aiba, red-green vernier acuity had three factors in isoluminacne, three interfaces for different kinds of acuity. We found that it is reduced by the factor that needs to be equalized with the receptor spaces. In addition, the resolution of the blue-yellow color channel is limited by the fact that it is tricolor blind (blue blind) for light subtending less than about 2 '(or .033 degrees) visible angle (Wyszecki and Stiles, 1982, p. 571). A pixel resolution of 0.03 degrees of viewing angle is very close to the largest value of these values, so it is used to equalize the pixel resolutions of the luminance and color channels.

The color pyramid extends to level six. This supports the evidence that the observer witnesses the difference between the wide and spatially uniform fields of color. This effect can be presented by using a spatially extended JND map. However, quantitative evidence for the contribution to JND by low spatial frequencies has been presented by Mullen (1985).

Returning to FIG. 8 again, similar to luminance processing, spatial analysis at seven resolution levels is achieved through smearing and downsampling pyramid analysis of an image by factor 2 at each successive lower level of resolution. do. The original maximum resolution image is called the zeroth level (level 0) of the pyramid.

Subsequent levels at lower resolutions are obtained with an operation called REDUCE. That is, a three-tap lowpass filter 805 with weights (1, 2,1) / 4 is applied to level 0 continuously in each direction of the image to produce a blurred image. The resulting image is then subsampled by factor 2 (all other pixels are removed) to produce the next level, level 1.

In step 810, a four-field average is performed on u * images and v * images for each resolution level with tap weights (0.25, 0.25, 0.25, 0.25). In other words:

(231)

Where j is the field index. This averaging reflects the natural low pass time filtering of the color channels and replaces the "early-late" processing of the temporal luminance channel.

In step 820, a non-directional Laplacian spatial filter 820 is applied to each of the u * and v * images. The filter has the following 3x3 kernel:

(24)

Equation (24) is chosen to respond to the edge between two uniform areas with unit values with a maximum length of 1 and to have a total weight of zero (the maximum response is obtained by the horizontal and vertical edges). v * Change the images into maps of color difference by evaluation in JND units.

In step 830, a contrast operation is directly performed on the u * and v * images by step 820 as a color contrast pyramid, so as to be continuously interpreted as Michelson contrasts computed by the luminance processor. Similarly, color contrasts are computed through intra-image comparison influenced by Laplacian pyramids. The Laplacian pyramid, which operates on u * and v * images, represents the Mitchellson contrast, which assumes a constant value at the 1-JND level (detection threshold) through the Webber's law, as the Laplacian difference divided by the spatial square. Has a JND interpretation. Similarly, this interpretation is modulated according to the verification procedure. The modulation reflects the fact that the inner workings of all parts of the present invention and the stimuli that lead to the 1-JND response are not simple in the equation of the perceptual metric generator.

Further, in step 830, the contrast pyramid images for each level are divided by seven constants qi (i = 0, ..., 6), and the values are 1000, 125, 40, 12.5, 10, 10, 10, respectively. Is determined by verification. These constants are similar to the quantities wi (i = 0, .., 3) in the luminance processor.

In step 840, the squares of all u * and v * contrasts are determined and the algebraic sign is again preserved for future use. The preservation of the sign prevents the possibility of writing 0 JNDs between two different images due to the confusion of loss of sign in the square operation. The result is two color squared-constant pyramids Cu, Cv.

In step 850, contrast energy masking is performed. First, the determinant pyramid level D _m (m = 0, 1, 2) is employed directly from the luminance processing unit 220 without incident. However, for levels 3, ..., 6, the sequence filtering and downsampling of D ₂ is implemented using the same method as in the luminance processing method without the addition of a new term. These D _m values are used in step 840 based on the perturbation principle, assuming that the luminance effect on color is more important than the color effect on luminance since luminance is a more important factor of the JNDs. That is, in most cases, the luminance effect is expected to be predominant over the color effect, so the color processor can be regarded as the first-order perturbation on the luminance processor. Thus, the effects of luminance D _m are modeled as masking of color but not vice versa.

The masked color contrast pyramid is generated by using the same functional form used in the luminance converter to mask the color squared-contrast pyramid for luminance-channel denominator pyramid D _m and all pyramid levels m = 0, 1, 2. do:

Note that the algebraic sign removed in step 830 is reattached via the s _um and s _vm factors. This operation results in masked contrast pyramids for u * and v *. The verification determines the values a _c = 0.15, c _c = 0.3, k = 0.7, σ _c = 1.0, and β _c = 1.17. In addition, setting the value of m _c to 1 provides sufficient performance in all verifications and expectations.

24 is a detailed block diagram of an alternative embodiment of the color processing unit 230. Since the color processor of FIG. 24 includes many similar steps to those of the color processor of FIG. 8, only dissimilar steps will be described below.

In particular, in step 830, the contrast pyramid images for each level are divided by seven constants qi (i = 0, ..., 6), the values of which are 384, 60, 24, 6, 4, 3, 3, respectively. Is determined by verification. It should be noted that these constants are different from the constants of FIG. 8. These constants are similar to the quantities wi (i = 0, .., 3) in the luminance processor.

The clipped absolute values of all u * and v * contrasts, where clip (x) = max (0, x−e), are then calculated, where e = 0.75. Again, the algebraic sign is preserved and given again for later use. This preservation of the sign prevents the possibility of writing 0 JNDs between two different images due to the confusion of loss of sign in the square operation. The result is two color squared-constant pyramids Cu, Cv.

Another important difference between the perceptual metric generators of FIGS. 8 and 24 lies in the implementation of the contrast energy masking. Unlike FIG. 8, the perceptual metric generators of FIG. 24 implement contrast energy masking 2410 in two separate steps: masking step and shelf masking step (FIG. 24) for each of the horizontal and vertical channels. Reference). Shelf masking reduces the sensitivity at the appearance of information into the current channel, and cross masking reduces the sensitivity at the appearance of information of neighboring channels. In fact, the order of these two separate masking steps can be reversed.

For the luminance-channel denominator pyramid D _m and all pyramid levels m = 0, 1, 2, use the same functional form used in the luminance converter to mask the color squared-contrast pyramids:

Here, D _i is a filtered and downsampled version of D ₂ when i> 2. similarly,

Note that the algebraic sign removed above is reattached via the s _um and s _vm arguments. This operation results in masked contrast pyramids for u _i and v _i . The verification determines values a _c = 1/2, c _c = 1/2, k = 0.7, β _c = 1.4, and m _c = m _f = 10/1024. In general, the color processor of FIG. 24 is a preferred embodiment, whereas the color processor of FIG. 8 is an optional embodiment.

9 is a block diagram illustrating the color metric generating unit 250. Again, FIG. 9 can be understood as a block diagram with a flow diagram of color metric generation steps or a number of hardware components for implementing the luminance metric generation steps, eg, a filter, various circuit components and / or an application specific integrated circuit (ASIC). have. The structure of the color JND map is similar to that of the luminance JND map as described above with respect to FIG. In the case of color, the process is applied to all masked-contrast color images generated by step 840 above: i.e. C _u0 , C _v0 , ..., C _u6 , C _v6 images and Corresponding images derived from the reference sequence (indicated by superscript ref in FIGS. 8 and 9).

The first four steps in the following process are individually applied to the above images. X to be mentioned next refers to one of the images derived from the test sequence and the corresponding image derived from the reference sequence by X ^ref . Given this notation, the steps are as follows:

In step 910, image X is separated into two half-wave rectified images. One is for positive contrast 912 and the other is for negative contrast 914. In a positive-contrast image (so-called X ₊ ), the symbols from the X contrast (stored separately as mentioned above) are used to assign zero to all pixels of X ₊ with negative contrasts. The opposite process occurs in the negative-contrast image X ₋ .

In step 920, for each image X ₊ and X ₋ , a local fling operation is implemented by applying a 3 × 3 filter that winds up the image with a filter kernel of 0.5 (1,2,1) vertically and horizontally.

Further, in step 920, the resulting image is downsampled by factor 2 in each direction to remove duplicate results from the fling operation. As applied to X, the same process is applied to the corresponding reference image X ^ref .

In step 930, absolute-differential images | X ₊ -X ₊ ^ref | and | X _-- X _- ^ref | are computed per pixel. The resulting images are JND maps.

In step 940, a coring operation is performed on the JND map. That is, all values below the threshold t _c are set to zero. In a preferred embodiment, t _c defaults to a value of 0.5.

In step 950, the Q-th power of these images is determined. In a preferred embodiment, Q is a positive integer defaulted to the value 2.

After this process is implemented for all X, X ^ref pairs, the summation measurement is determined by iteratively upsampling, filtering, and adding all the images up to the required level. This is accomplished as follows:

In step 960, upsampling and filtering is applied to level-6 images derived from C _u6 , C _u6 ^ref , C _v6 , C _v6 ^ref to derive a level-5 image.

In a next step, upsampling and filtering is applied to the sum of the level-5 images derived from C _u5 , C _u5 ^ref , C _v5 , C _v5 ^ref and the level-5 images derived from step 960. This process continues to level zero.

Similar to the luminance processing, it should be noted that before the final processing step 963, the resulting image is half the resolution of the original image. Similarly, it should be noted that each pyramid-level index in the present processing section is twice the resolution of the image associated with the level after filtering / downsampling, and relates to the pyramid level derived directionally.

It should also be noted that all images generated by the repeated upsampling, filtering, and addition process are Q-th-power-JND images. The level-0 image is either sent directly to the summing process on path 964 or used in two forms, upsampling and filtering in step 930 to the original image resolution for display purposes.

As described above, the luminance and chrominance (JND) map passed in the output summary step is a Q-order power JND image, representing half the resolution of the original image. This uses inherent redundancy in performing pooling at each masked contrast stage. Each of these half resolution images can be reduced to a single JND performance measure by averaging all the pixels through the Minkowski addition.

N _P is the number of pixels in each JND map, JND _luminance and JND _chrominance are outline measurements, and L _JND ^Q and C _JND ^Q are half resolution maps from the luminance and chrominance map configuration, respectively. In each case, the sum is through all the pixels of the image. As mentioned above, the value of the Minkowski index Q defaults to two.

From the luminance and chrominance scheme measurements, a single performance measure for the field is calculated by the Minkowski addition. In other words,

Where Q is again defaulted to two.

Again in the sense of Minkowski Q, by adding the JND value for each field, the single performance measure JND _field for the N field of the video sequence is defaulted to two.

25 is a detailed block diagram of another embodiment of the color difference metric generator 250. Since the color difference metric generation in FIG. 25 includes stages that are much similar to the color difference metric generation in FIG. 9, only the dissimilar stages will be described in detail.

More specifically, the "coring" step 940 and the "rise to Q order power" step 950 are replaced by a number of map and summation stages that maintain the execution summation and execution maxima of the channel output. Since the process shown in FIG. 25 is the same as FIG. 9 until step 930, the process of FIG. 25 is an absolute value-difference image. And Describes starting from the point where is determined.

Then, after the process is completed for all X, X ^ref pairs, the execution sum image is stage 2540 to include the sum of the level-6 images derived from C _u6 , C _u6 ^ref , C _v6 , and C _v6 ^ref . Initialized at Similarly, the execution maximum image is initialized in step 2542 as a point-by-point maximum of the same image.

The run sum and run max images are then unsampled and filtered by steps 2540a and 2542a, respectively, to construct two level-5 images. The execution summation image is updated by step 2544 by adding to it the level-5 derived from C _u5 , C _u5 ^ref , C _v5 , and C _v5 ^ref . Similarly, the execution maximum image is updated by step 2546 by comparing it with a level-5 image derived from C _u5 , C _u5 ^ref , C _v5 , and C _v5 ^ref . This process is repeated down to pyramid-level zero.

Finally, by performing the above steps, a point-by-point combination of the run sum and the run maximum image is performed to generate the chrominance JND map:

Where k _c is 0.836. The value for k _c is determined by approximating the Minkowski Q-standard. Assuming that the Q value and multiple images N are combined, the value k _{c =} [NN ^{1 / Q} ] / [N-1] is approximated when all compared entries (in pixels) are equal and only one It guarantees an exact match to the Q-standard for nonzero entries. In this case, N = 28 (number of channels) and Q = 2.

In luminance processing, after these operations the resulting image is half of the original resolution. Note that the pyramid level index in this process quotes the first derived pyramid level, which is twice the resolution associated with the level after filtering / downsampling.

It should also be noted that all images generated by the repeated upsampling / filtering and addition / maximization can be added to the weights k _c and 1-k ^c to produce a JND image. Level-0 images are used in two ways, either sent directly to the Gyayo process or upsampled to the original image resolution and filtered for display purposes.

In general, the chrominance metric generator of FIG. 25 is a preferred embodiment, while the luminance metric generator of FIG. 9 is a modified embodiment. One reason is that the maximum summation method is computationally less expensive. Therefore, when the dynamic range of the generator of FIG. 25 is appropriate, the color difference metric generator of FIG. 25 is preferable. Alternatively, when the floating point processor is used, the luminance metric generator of FIG. 9 may be used.

Half-height color difference processing

If the half-height image is passed directly through the net image height without zero-filtering, the chrominance processing unit 230 should be modified to reflect that the intrinsic vertical resolution is half the intrinsic horizontal resolution.

In the perceptual meter generator of the present invention, it has been observed that the border reflections at each stage can propagate the artifacts to the luminescence and chrominance JND maps, so it is necessary to cut the JND maps to prevent contamination from these artifacts Done. To represent this threshold, a method of substituting screen borders with an infinite range of gray bezels has been developed, but this does not improve the actual image size beyond six pixels on the face. Using this "virtual bezel" eliminates the need to truncate the JND map to avoid border artifacts. Infinite gray bezels are considered non-artificial by shaping observational conditions. By this interpretation, the entire JND map is not contaminated by artificial results and can be displayed by the photo quality analyzer.

In the detailed description below, an image in which six pixels are padded on all sides will be referred to as a "pad image" and an unpadded image or a rule in the padded image will be referred to as an "image proper". .

Since image manipulation is local, a virtual infinite bezel can be used effectively. If the image prop is far enough away an infinite bezel results in a set of ideal and constant values at a given stage. The effect of image manipulation, for example filtering done in this area, can be calculated as having a priori. As such, a narrow border (6 pixels in the current configuration) can provide an appropriate transition from the image propeller to the infinite bezel.

At the input, the basel is given by Y '= 90 and U' = V = 0. (Y '= 90 corresponds to half of the Rec 500 background value of 15% of the maximum screen emission.) However, because the spatial interaction extending beyond the image border does not occur at this stage, the bezel is front-end It is not necessary until after processing. In the light emitting channel, no borders (and bezel values) are added until after light compression. In the chrominance channel, the border is added after the front end processing.

In the light emitting channel, the first bezel value after light emission compression is as follows.

(30c)

In u * and v * channels, the first bezel value is both zero.

These values propagate through the following three stages of processing in three cases:

1) Pixel by pixel The pixel-by-pixel function manipulates to generate a new bezel value from an old bezel value. For example, the bezel value from the 1.4 power function is

(30d)

2) A 3x3 spatial filter with a row and column sum of P sets the output bezel value to the input bezel times P.

3) The contrast function calculator and 4-field time filter (with zero tap sum) set the output bezel value to zero.

In the contrast stage and subsequent stages, the logic result manipulated with a zero-sum linear kernel on a spatially constant array of values of the emission and chrominance channels is given to the bezel.

The present invention for generating a virtual bezel is disclosed in US Patent Application 08 / 997,267, "Method for Generating Image Pyramid Border," filed December 23,1997. This US application 08 / 997,267 is incorporated by reference.

Integrate image and bezel

Starting with the pyramid stage model, a border needs to be provided. The first border operation on the N × M input image pads the image to 6 pixels (on all sides) with the appropriate bezel values (first_emission_bezel for compressed light emission images and zeros for u * and v * images). will be. The padded image has (N + 12) × (M + 12) dimensions. For k ^th pyramid levels, where k can be 0 to 7, the padded image Has the dimension of "[x]" where x represents the largest integer.

Images of all pyramid levels are registered respectively in the upper left hand corner of the image prop. The index of the image prop has a range of 0 ≦ y ≦ height and 0 ≦ x ≦ width. The upper left hand corner of the image prop always has an index (0,0). The index of the bezel pixel takes a value less than or equal to zero in height and width values. For example, the upper left hand bezel pixel is (-6, -6). When looking along the x dimension starting at the top left hand corner of the width w (the sum of the bezel width w + 12 and the image), the bezel pixels are indexed with x = (-6, -5, ...,-1) The actual image is indexed as (0.1, ..., w-1) and the right hand bezel index is the value of (w, w + 1, ..., w + 5).

Given a padded image, four things can happen depending on the subsequent processing stage. In describing these manipulations below, it is possible for a single image line to be used to summarize spatial processing (it should be understood that similar cases may occur in the vertical direction).

(a) One pixel One pixel operation. If the next operation is one pixel one pixel operation (eg with nonlinearity), the padded image simply goes through the operation and the output image dimension is the same as the input image dimension. The same result occurs when an operation is made between corresponding pixels of different fields or different color bands.

(b) 3 × 3 spatial filter. Assume that the unpadded input image (in one dimension) has N _k +12 dimensions. The padded input image then has N _k +12 dimensions, and the padded output image also has N _k +12 dimensions. The output bezel values are calculated first and written to at least these bezel pixels that are not otherwise filled by subsequent image manipulations. Different, starting from one pixel away from the left edge of the padded input image, the 3x3 kernel starts to manipulate on the input image and overwrites the bezel value of the output image, and falls away from the right (or bottom) edge of the image Terminate at 1 pixel (where the initial bezel value remains). The prewritten bezel values make it unnecessary for kernel operations to escape outside the original (padded) image to calculate these values.

(c) Regarding filtering and down sampling at REDUCE. Given an N _k +12 dimension in the input padding image, an [N _k / 2] +12 dimension is assigned to the output array. The bezel values are written to at least these bezels that are not otherwise filled by subsequent filter and downsample operations. Next, the input images are filtered according to (b), but the filter is applied to the pixels (-4, -2, 0, 2, 4) until the input image is ejected, and the output values are added to these output images. It is written into successive pixels (-2, -1, 0, 1, 2, ...) until not arranged. Note that the position of pixel 0 in the new image 7 is seven pixels from the left end of the new image. The final pixel application of the filter is N _k is an odd number if the input pixel (N _k +3) for output pixel _{([N k / 2] +2} ) for taking, N _k is an even number if the input pixel (N _k +4) for Take an output pixel [N _k / 2] +2 (here, the input pixel of the filter is referred to as the pixel corresponding to the center of the three-pixel kernel).

Luminance Adjustment and Prediction

The psychophysical data serves two purposes: 1) to adjust the luminance processor (e.g., to determine a value for a particular processing parameter), and 2) to form a prediction of the luminance processing section when the section is adjusted. Used to In all cases, stimuli are injected into the metric generator by perception as Y-value images immediately before luminance processing.

adjustment

The luminance processor 220 may be repeatedly adjusted using two data sets. One data set is used to adjust the premasking constants w _i , t _e and tl in steps 640, 642 and 650 of the luminance processor. The other data set is used to adjust the masking step constants σ, β, a and c in step 660 of the luminance processor. Since the JND value is always evaluated after step 660, constant adjustment of step 660 with the second data set requires reconditioning of the constants 640, 642 and 650 with the first data set. The readjustment of this constant lasts until no further change is observed from one iteration to another. Although the iterative process is performed by interpreting unit values of unmasked contrast (steps 640, 642 and 650) as the JND of one virtual output, the masking process disturbs this interpretation. Details of these adjustments are described in the subsections below.

Adjusting Contrast-Normalization Constants (Steps 640, 642, and 650)

The perceptual metric generator predicts spatial and temporal contrast sensitivity prior to masking that matches the contrast-sensitivity data for the sine wave presented by coendlink and van Dorn (1979). To generate a position on a curve based on the perceptual metric generator, a low amplitude sine wave is provided as a test image to the perceptual metric generator (in space or time), and the contrast threshold for one JND output is evaluated. In each case the reference image implicitly has a uniform field with the same average luminance as the test field.

Spatial contrast sensitivity to the data (see FIG. 10 for the final fit) was used to adjust the contrast-pyramid sensitivity parameters w0, w1 and w2 in the metric generator's steps 640, 642 and 650 by perception. The dashed line in FIG. 10 represents the sensitivity of the individual pyramid channel including the overall sensitivity (solid line). The spatial model fit of FIG. 10 does not exceed 15 cysles / deg and is compatible with the observation-distance constraints already disclosed, ie the viewing distances of four screen heights. Similar adjustments of w0, w1 and w2 can be performed to accommodate slightly different viewing distances, with much larger viewing distances probably requiring lower resolution pyramid levels, which can be easily implemented at low computational cost.

(See Fig. 11 for end feet) feet of the temporary contrast sensitivity for the data, a temporary filter was used to adjust the parameters tab _(e t and tl) as well as a contrast pyramid sensitivity parameter (w3). The method used to fit the parameters is similar to the space-contrast adjustment. At various temporal frequencies, the lowest spatial frequency data of the half-row and the co-end link were matched against the sensitivity calculated for the spatially uniform temporal sine wave. In each case, the vision-model field ratio samples the temporal sine wave from 50 to 60 Hz, which is given by the separate parameter values already mentioned.

Adjusting Masking Constants (Step 660)

The masking parameter values σ, β, a and c (at step 660 of the perceptual metric generator) were fitted by comparing the prediction for masked contrast determination with the data obtained by Carlson and Cohen (1978). The result of the final pit comparison is shown in FIG. From the Carlson-Cohen study, the data of a single observer are chosen to be dependent on the representative criteria and to have enough data points. In this case, the perceptual metric generator steamy consists of a test and a spatial sine wave of a predetermined pedestal contrast of the reference field, and additionally a contrast increment of the test field sine wave. Contrast increments are needed to obtain 1 JND, which is determined from the perceptual metric generator for each contrast pedestal value, and is depicted in FIG. 12.

prediction

After perceptual metric generator adjustment, perceptual metric generator prediction is computed by detecting and discriminating data from stimuli that are not sinusoidal. This is done to check the transmission of sinusoidal results for more general steamy. In Figures 13, 14 and 15, it can be seen that the prediction does not apply to patterns with nominal spatial frequencies above 10 cycles / deg. This pattern will have significant energy at spatial frequencies above 15 cycles / deg and will be referred to as the pixel sampling rate (30 samples / degree-see discussion above).

In the first study (see FIG. 13), the low contrast disk of the test field is detected for a uniform reference field. Empirical data comes from Blackwell and Blackwell (1971). In the implementation of the perceptual metric generator for this particular study, the data is needed to replace the Q-Norma schema size with the maximum value. Otherwise the JND result is sensitive to the size of the background of the disc (eg image size).

In a second study (FIG. 14) on small size checkerboard detection, data is obtained from Sarnov's unpublished paper.

The third study (data from Carlson and Cohen, 1980) is somewhat different from the first two studies. A blurry edge provided by erf (ax) is present in the reference image, and an identification has been attempted against the edge provided by erf (a'x) in the inspection image. x is the retinal distance at the viewing angle (a = πf / [ln (2)] ^0.5 , a '= π (f + Δf / [ln (2)] ^0.5 ), and f is the cycle / angle, where Δf is This is the change in f required for one JND, Figure 15 is Δf / f vs. f.

Perceptual metric generator predictions show good agreement with data for the spatial frequency characteristic range of the display at four screen height viewing distances.

Color difference calibration

As a luminance parameter calibration, physical data is used to calibrate the luminance parameter (ie, adjust its value for the best model). In all cases, stimuli are four identical fields projected onto the perceptual metric generator as CIE X, Y and Z before being converted to CIELUV.

Adjusting Contrast Standardization Constants (Step 830)

Perceptual metric generator predictions for color constant sensitivity prior to masking are matched with the contrast sensitivity data provided by Mullen (1985). The test sequence used is four identical fields, each of which has a horizontal variable spatial sine wave grating introduced as a (X, Y, Z) value. The data used for calibration is shown in Figure 6 of Mullen, where each inspection image corresponds to a cyan isoluminescent sine wave. In pixel i, the lyrics image sine wave has a tristimulus value provided by the following equation:

_{X (i) = (Y 0} /2) {xr / yr + xg / yg) + cos (2πfai) Δm (xr / yr - xg / yg)}

Y (i) = Y ₀ (31)

_{Z (i) = (Y 0} /2) {zr / yr + zg / yg) + cos (2πfai) Δm (zr / yr - zg / yg)}

Where Δm is the critical increase discrimination contrast value, (xr, yr) = (.636, .364) is the chromaticity of the red interference filter (at 602 nm), and (xg, yg) = (.122, .823) is The chromaticity of the green filter (at 526 nm), zr = 1-xr-yr, zg = 1-xg-yg, and a = .03 degrees / pixel. The reference image is a uniform field represented by equation 28 but has Δm = 0. For perceptual metric generators, setting Y ₀ = 1 is sufficient.

In order to generate points on the model ground curve, the stimulus appears at several f values and a constant threshold Δm for 1 JND output is evaluated. Installing the modeled color contrast sensitivity in the data (see FIG. 16 for final matching) is used to adjust the parameters qi (i = 0, ..., 6) in the perceptual metric generator.

Adjusting Masking Constants (Step 840)

Perceptual metric generator predictions for color masking match the data provided by Switkes et al. (1988). The test sequence used is four identical fields, each with a horizontal variable space sine wave grating introduced as a (X, Y, Z) value. In order to correspond to FIG. 4 of the above operation (color difference masking of chrominance), in pixel i, the inspection image sine wave has a Tristimulus value provided by the following equation:

_{X (i) = (Y 0} /2) {(xr / yr + xg / yg) + cos (2πfai) [(m + Δm) (xr / yr-xg / yg)]}

Y (i) = Y ₀ (32)

_{Z (i) = (Y 0} /2) {(zr / yr + zg / yg) + cos (2πfai) [(m + Δm) (zr / yr-zg / yg)]}

Where Δm is the critical increase determination constant, (xr, yr) = (. 580, .362) is the color difference of red phosphorus, (xg, yg) = (. 301 ,. 589) is the color difference of the red phosphor, zr = 1-xr-yr, zg = 1-xg-yg and fa = 2c / eh * .03 deg / pixel = .06. The reference image sine wave is the same as the test image sine wave but has Δm = 0. For perceptual metric generators, it is satisfactory to set Y ₀ = 1.

Compare with rating data

For four image sequences, image sizing with varying degrees of distortion is used to compare the DSCQS rating data with the current perceptual metric generator. The result is shown in FIG. 18 and shows the correlation .9474 between the perceptual metric generator and the data. For each sequence, the perceptual metric generator processes 30 fields (fields opposite to the four fields used to check for previous releases).

Some data points are removed from the figure shown in the previous release. These points are deleted for two reasons:

(1) Five points are deleted to correspond to a "warm up" check on all subjects. Rec 500 suggests that the first five checks of the sequence should be deleted because they indicate subject decision stabilization.

(2) For one of the "Gwen" sequences, there is a small shift in the inspection sequence with respect to the reference sequence that occurs between the images of the background tree even when the foreground is correctly aligned between the inspection and the reference. Blue screen video is introduced respectively for inspection and reference with temporary alignment errors in this particular case.

JND Map Correlation

The JND map is in a form suitable for further processing to determine the JND in any spatial or temporary window. As described above, the value of the map is a JND unit that is raised to the Q th power than a simple JND unit. To obtain a single JND value for a given spatial time region of a video stream, one only needs to sum the values from the JND maps in that region and obtain the Q root.

Some examples clarify this process. In order to retrieve the 1 JND value (the most appropriate output) for each pixel, the Q th root of each pixel of the JND map is obtained.

However, in the field of typical MPEG-2 encoder analysis, it is useful to have a single JND for each 16x16 pixel microblock rather than each pixel. To get 1 JND per microblock, all JND map outputs within each microblock are summed first and then the Q th root is obtained. The result is a microblock-resolution map of JND values.

Pyramid Structure: Image Size & Boundary Requirements

The implementation of the current pyramid method does not cause an image-dimension problem if the large image dimension N and the small image dimension M satisfy the following conditions.

1) M must be at least 128

2) M should be divided by 2 by the same number of times as searching for coefficients less than 64

3) N must be divided P times by 2

The perceptual metric generator determines abnormally an image that does not satisfy the conditions. As an example of how these rules work, consider image dimensions N = 720 and M = 480. Since M> 128, condition (a) is satisfied. The condition (b) is satisfied because M is divided by 2 three times and divided by three times to be 64 or less (where P = 3). Finally, condition (c) is satisfied that N can be divided by two three times to obtain an integer.

Interlaced

The purpose of the following description is to clarify the processing of field interlaces (particularly inter-line space) in the perceptual metric generator of the present invention. The inter-line space is not visible by the human viewing display, but when they are modeled as block values, produces a distinct effect in the perceptual metric generator. As a result of the line's visibility by the perceptual metric generator, the most image distortion at a given spatial frequency is masked by the high frequency line structure. In addition, the display of the line structure is a primary cause of JND artifacts when the interlaced sequence is compared to the uninterlaced sequence.

The solution to this problem is to change the display model, which combines the known mean in space with the time that occurs on the display itself. This average makes the inter-line space appear less. The first step is to define the magnitude of these effects to determine the appropriate model.

Since the phosphor has a finite delay, a temporal average occurs on the display. Thus, for example, the remainder may be issued after the erase of the odd line of field N-1 at the primary emission time from the even line of field N. However, compared to the inter-field interval (16500 microseconds), the phosphor scavenging time is generally quite short, for example 70 microseconds for the blue phosphor, 100 microseconds for the green phosphor and 700 microseconds for the red phosphor. . Thus, the temporal average in the display model does not contribute to inter-line smoothing.

Since the emission from the pixels spreads beyond the normal pixel boundaries, a spatial average occurs in the display. In an interlaced display, the electron beam spot structure is designed to combine with the interlaced architecture. As a result, pixel diffusion becomes clearer in the vertical direction, filling the inter-line space so that they are less visible. Diffusion is particularly severe at high beam currents, which corresponds to high luminance values and thus is most noticeable in the image. Thus, from display observations, the spatial mean is a good physical model for inter-line smoothing.

Optionally, some temporal mean may be used for inter-line smoothing. The visual system itself performs a sufficient temporal averaging so that the inter-line space is not visible. However, as can be seen in the following description, the lack of eye movement in the perceptual metric generator of the present invention causes the perceptual metric generator to deviate from the temporal average operation.

Human vision is aided by a mechanism with two distinct space-time response forms: high spatial resolution "maintains" at low temporal resolution and "transfers" to high temporal resolution low spatial resolution.

One realization of this perceptual metric generator is to use a spatial / temporal filter that is separable to form the response of the two channels. The direct importance of this modeling choice is the temporal filter on the low pass hold channel compared to the 60Hz time sampling rate typical for displays. The propagation response is not sensitive even at 60Hz sampling rate. However, one element that does not enter the maintenance / transfer model is the eye movement effect, in particular the eye's ability to track moving objects in the image. This tracking improves the visual sensitivity of the details of the objects shown, which is prevented from being captured by the perceptual metric generator, which is faithful to the psychophysical experience of limited stimuli.

The effect of the shift on the distortion measure in the image sequence can be considered. If the eye does not track moving objects in the image, the blur in the image resulting from the retained temporal response is accurately reflected in the perceptual metric generator by many temporal averages in one channel. However, the eye does not track the moving object, so the image is not blurred. Without the ability to track moving objects, perceptual metric generators to improve temporal visual response should display moving blur. However, such blurring prevents the generation of accurate JND maps.

To solve this difficulty without a tracking model, a compromise is made that represents the spatial channel (having the role of a "maintain" channel sensitive to spatial detail) by operating the last field rather than the time average of the field. As a result of this method, the spatial channel represents a well-focused JND map in the case of the eye tracking the movement of the object shown in the image sequence.

While maintaining the spirit involved above, the "specious-present" property of the spatial channel is relaxed to be averaged over two fields and thus averaged over one frame. This method reduces the visibility of the blank lines in the interlace field and is more physical and physiological than "extroverted" solutions. However, the disadvantage is that it saves the temporal averaging of the two fields, which has the shape of a "comb" which should be a smooth moving edge.

To understand why combs should appear in a model with two-field averaging, it is sufficient to visualize moving objects in the time interval between even fields (called field N) and odd fields (called field N + 1). Do. Assume that an object has a vertical edge that moves five pixels horizontally between fields. Also assume that the object edge is located at pixel n of the even line of field N. This edge would then represent pixel n +% of the odd lines in field N + 1. If there is no "fill" between the raster lines of a particular field, the averaging of field N and field N + 1 is no longer vertical, but produces an edge that changes between pixels N and n + 5. This is a "comb" effect.

To understand why the real visual system does not exhibit this comb effect, assume that the object is sufficient for the eye to track the object. This means that the object is fixed to the retina, because the retina expects to move the object into the next field. If the edge of the object is located at pixel n of the even line of field N, it will also be located at pixel n of the odd line of field N + 1, simply because the eye tracking on the object is almost perfect.

To avoid combs and other interlacing shortcomings, the perpendicular force meter generator performs space filling between each field of the display. This vertical averaging prevents the comb effect because it provides translation of temporal spatial edges (no averaging occurs). In addition, vertical averaging solves the inherent problem of visibility of interlaced line structures and is compatible with the known spatial dispersion of these electron beam spot structures.

In the above, a novel apparatus and method for accessing visibility with differences between two input images for improving image fidelity and visualization applications have been shown and described. However, those skilled in the art will appreciate that many modifications, changes, and the like are possible in view of the described embodiments and the accompanying drawings.

Claims (10)

A method for evaluating a difference in visibility between two input image sequences each having a plurality of images comprising a color component, (a) analyzing a color component of each of the input images to generate a pyramid having a plurality of resolution levels of the color component; (b) applying temporal filtering of at least one of said levels of said pyramid of said color component into a color time response: and (c) generating an image metric from the color time response. The method of claim 1, wherein each of the input images is one half height. The method of claim 1, wherein each of the input images further comprises a luminance component. (a ') analyzing a luminance component of each of the input images to generate a pyramid having a plurality of resolution levels of the luminance component; (b) applying temporal filtering of the luminance component to a luminance time response of at least one of the levels of the pyramid; and Wherein said generating step (c) generates an image metric from said time responses from said luminance and said color response. The method of claim 1, wherein the method is (a ") transforming the color component of the input image into a CIELUV uniform-color space prior to the analyzing step (a) and converting the color component of the input image before the analyzing step (a) Transforming to CIELUV uniform-color space The method of claim 3, wherein the method (a ") transforming said luminance component of said input image to a CIE 1931 tristimulus value Y prior to said analyzing step (a). The method of claim 3, wherein the generating step (c), (c ') generating a luminance metric separate from said luminance time response; (c ") generating a color metric separate from said color time response; and (c '") generating an image metric from said luminance metric and said color metric. The method of claim 3, wherein the applying step (b ') applies time filtering to the lowest level of the pyramid of the luminance component in a luminance time response, The method further includes (b ") applying spatial filtering to the remaining levels of the pyramid of the luminance component in the luminance spatial response. 7. The method of claim 6, wherein said generating step (c ") generates a color metric separate from said color time response, and said color time response is masked by said luminance time response. In the apparatus 112 for evaluating the difference in visibility between two input image sequences each having a plurality of images comprising a color component, A pyramid generator (410, 510) for analyzing a color component of each of the input images to generate a pyramid having a plurality of resolution levels of the color component; A time filter 520, coupled to the pyramid generator, for time filtering at least one of the levels of the pyramid of the color component with a color time response; and An image metric generator (250, 550) coupled to the time filter for generating an image metric from the color time response. A computer-readable medium storing a plurality of instructions consisting of instructions, the method comprising: (a) analyzing a color component of each of the input images to generate a pyramid having a plurality of resolution levels of the color component; (b) applying temporal filtering of at least one of said levels of said pyramid of said color component into a color time response: and (c) generating an image metric from the color time response, wherein the processor performs the steps when the processor is executed.