JP2011216967A - Image processor and method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress image quality degradation when performing IP conversion and scaling processing.SOLUTION: An IP conversion scaling part 121 IP converts input interlaced SD images to progressive SD images based on a specified method, and then converts the SD images to HD middle images. Based on a motion vector V, an MC block differential value BD and the positional relationship L between input image pixels and middle image pixels, a cyclic coefficient setting part 133 sets a cyclic coefficient K for every pixel. A product sum operation part 135 composites the middle image and a motion compensation image at a rate that is indicated by the cyclic coefficient K, and generates output images that are progressive HD images. This method can apply to image converters which perform IP conversion and scaling processing.

Description

  The present invention relates to an image processing apparatus, method, and program, and more particularly, to an image processing apparatus, method, and program that can suppress a reduction in image quality when performing IP conversion and scaling processing.

  When shooting a subject moving in the vertical direction or shooting with the shooting device moved in the vertical direction, the interlaced image (hereinafter also referred to as an interlaced image) has insufficient bandwidth in the vertical direction. Aliasing distortion (hereinafter, referred to as vertical aliasing distortion) occurs, and a phenomenon occurs in which the thickness of the line in the horizontal direction varies and the strength of the edge varies. Further, the vertical aliasing distortion has a feature that is more conspicuous when an interlaced image is converted into a progressive image (hereinafter also referred to as a progressive image) (hereinafter referred to as IP conversion).

  By the way, there is a case where scaling processing for converting the image size is performed after the IP conversion (see, for example, Patent Document 1 and Patent Document 2).

  In the method described in Patent Document 1, as shown in FIG. 1, for example, a 720 × 480 interlaced image is converted into a progressive image (720 × 480p) by IP conversion processing, and is further subjected to super-resolution scaling processing 1440. It is converted into a × 960p image.

  In the method described in Patent Document 2, as shown in FIG. 2, for example, a 720 × 480 interlaced image is converted into a progressive image (720 × 480p) by a temporary IP conversion process. Thereafter, the past output subjected to motion compensation by the motion vector is added by the cyclic coefficient by cyclic processing, and vertical aliasing distortion and noise are removed. Then, it is scaled and converted into an image of 1440 × 960p.

JP 2008-252701 A JP 2007-251686 A

  However, in the case of the method described in Patent Document 1, since the scaling process is performed on the IP conversion result, the amount of motion of the input image cannot be used, and the reduction in image quality due to the scaling process cannot be suppressed. There was a fear. In the case of the super-resolution algorithm, iterative calculation is necessary because it uses an iterative method. As a result, the processing load may increase or the delay time may increase. Therefore, it is not suitable for immediate processing such as outputting an input image while converting it.

  In the case of the method described in the cited document 2, although good IP conversion using input motion information can be performed, since the input motion information is not used in the scaling process, the image quality is reduced by the scaling process. There was a fear that it could not be suppressed.

  The present invention has been proposed in view of the above-described circumstances, and an object thereof is to suppress a reduction in image quality when performing IP conversion and scaling processing using motion information of an input image. .

  One aspect of the present invention is an image processing apparatus that converts an interlaced input image into a scaled progressive output image by converting the input image into a progressive image and further converting the image size. An IP conversion scaling means for converting to a scaled progressive intermediate image, a motion vector detecting means for detecting a motion vector of the input image having a distance shorter than an interval between pixels of the input image as a minimum unit, and the motion Based on the motion vector detected by the vector detection unit, and the positional relationship between the pixel of the input image and the pixel of the intermediate image obtained by converting the input image by the IP conversion scaling unit, a cycle is performed for each pixel. A cyclic coefficient setting means for setting a coefficient and the motion vector detecting means. On the basis of the detected motion vector, the input image is converted by a motion compensation unit that generates a motion compensated image, which is an image obtained by performing motion compensation on the past output image, and the IP conversion scaling unit. Further, the pixel value of the pixel of the intermediate image and the pixel value of the pixel at the corresponding position of the motion compensated image generated by the motion compensation unit are used by the cyclic coefficient set by the cyclic coefficient setting unit. And an output image generation means for generating the output image by weighted addition.

  The cyclic coefficient setting means can set the value of the cyclic coefficient to be smaller as the distance between the target pixel and the pixel of the input image closest to the target image is shorter in the current frame of the intermediate image.

  The cyclic coefficient setting means determines the value of the cyclic coefficient as the distance between the position moved by the motion vector from the pixel of interest and the pixel of the input image closest to the position is shorter in the past frame of the intermediate image. Can be set large.

  The apparatus may further include reliability detection means for detecting the reliability of the motion vector and correction means for correcting the cyclic coefficient based on the reliability of the motion vector.

  The reliability detection means detects, as the reliability of the motion vector, a first motion variance indicating a degree of variation of each motion vector from the surrounding motion vectors, and the correction means The larger the motion variance of 1, the smaller the cyclic coefficient can be corrected.

  The reliability detection means further determines, as the reliability of the motion vector, 1 for each of the motion vectors around the position where the pixel corresponding to the motion vector is moved by the direction and distance indicated by the motion vector. The second motion variance indicating the degree of variation with the motion vector before the frame is detected, and the correction means further corrects the cyclic coefficient so as to decrease as the second motion variance increases. it can.

  The motion vector detection means detects the motion vector for each block of a predetermined size, and the reliability detection means indicates a pixel value in the block of the input image and the block by the motion vector. A difference value with a pixel value in a block of the past output image moved by a direction and a distance may be detected, and the correction unit may correct the cyclic coefficient so as to decrease as the difference value increases. it can.

  The output image generation means can increase the ratio of the motion compensated image in the output image as the cyclic coefficient increases, and increase the ratio of the intermediate image in the output image as the cyclic coefficient decreases.

  One aspect of the present invention is also an image processing method of an image processing apparatus that converts an interlaced input image into a scaled progressive output image, wherein the IP conversion scaling unit of the image processing apparatus includes The input image is converted into a progressive image, and further converted into a scaled intermediate image by converting the image size, and the motion vector detecting means of the image processing device detects the pixel interval of the input image. A motion vector of the input image having a shorter distance as a minimum unit is detected, and a cyclic coefficient setting unit of the image processing apparatus converts the detected motion vector, the pixel of the input image, and the input image. A cyclic coefficient is set for each pixel based on the positional relationship with the pixels of the intermediate image, and the image processing device Based on the detected motion vector, generates a motion compensated image that is a motion compensated image of the past output image, and the output image generating means of the image processing device includes: Weighting and adding the pixel value of the pixel of the intermediate image converted from the input image and the pixel value of the pixel at the corresponding position of the generated motion compensated image using the set cyclic coefficient This is an image processing method for generating the output image.

  According to an aspect of the present invention, a computer that converts an interlaced input image into a scaled progressive output image is further converted into a progressive image, and the image size is further converted. An IP conversion scaling means for converting to a scaled progressive intermediate image, a motion vector detecting means for detecting a motion vector of the input image having a distance shorter than an interval between pixels of the input image as a minimum unit, and the motion Based on the motion vector detected by the vector detection unit, and the positional relationship between the pixel of the input image and the pixel of the intermediate image obtained by converting the input image by the IP conversion scaling unit, a cycle is performed for each pixel. A cyclic coefficient setting means for setting a coefficient and the motion vector detecting means; Based on the detected motion vector, the input image is converted by a motion compensation unit that generates a motion compensated image that is an image obtained by performing motion compensation on the past output image, and the IP conversion scaling unit. Further, the pixel value of the pixel of the intermediate image and the pixel value of the pixel at the corresponding position of the motion compensated image generated by the motion compensation unit are used by the cyclic coefficient set by the cyclic coefficient setting unit. This is a program that functions as output image generation means for generating the output image by weighted addition.

  In one aspect of the present invention, an input image is converted into a progressive image, and the image size is converted into a scaled progressive intermediate image, which is shorter than the pixel interval of the input image. The motion vector of the input image having the minimum unit of distance is detected, and the detected motion vector and the positional relationship between the pixel of the input image and the pixel of the intermediate image obtained by converting the input image are determined for each pixel. Based on the detected motion vector, a cyclic coefficient is set, a motion compensated image that is a motion compensated image for the past output image is generated, and the pixel value of the pixel of the intermediate image obtained by converting the input image And the pixel value of the pixel at the corresponding position of the generated motion compensated image is weighted and added using the set cyclic coefficient to output an image It is generated.

  According to the present invention, an image can be processed. In particular, it is possible to suppress a reduction in image quality when performing IP conversion and scaling processing.

It is a figure explaining the method of the conventional IP conversion and scaling. It is a figure explaining the method of the conventional IP conversion and scaling. It is a block diagram which shows the main structural examples of the image converter to which this invention is applied. It is a block diagram which shows the main structural examples of a cyclic IP conversion scaling part. It is a block diagram which shows the main structural examples of an IP conversion scaling part. It is a block diagram which shows the other structural example of an IP conversion scaling part. It is a figure explaining the example of the mode of IP conversion scaling. It is a figure explaining the example of the mode of IP conversion scaling. It is a figure explaining the example of the mode of IP conversion scaling. It is a figure explaining the example of the mode of IP conversion scaling. It is a block diagram which shows the main structural examples of a motion vector detection part. It is a block diagram which shows the main structural examples of the cyclic coefficient setting part. It is a flowchart explaining the example of the flow of an image process. It is a flowchart explaining the example of the flow of cyclic type IP conversion scaling processing. It is a flowchart explaining the example of the flow of cyclic type conversion processing. It is a figure for demonstrating the detection method of MC block difference value BD. It is a figure for demonstrating the detection method of MC block difference value BD. It is a figure explaining the example of the mode of a basic cyclic coefficient setting. It is a figure explaining the example of the mode of a basic cyclic coefficient setting. It is a figure explaining the example of the mode of a basic cyclic coefficient setting. It is a figure which shows the example of distribution of a motion vector. It is a figure which shows the other example of distribution of a motion vector. It is a figure for demonstrating the calculation method of motion dispersion | distribution MD0. It is a figure for demonstrating the calculation method of motion dispersion | distribution MD1. It is a flowchart explaining the example of the flow of a motion vector detection process. It is a figure for demonstrating the method to detect the motion vector of pixel precision. It is a block diagram which shows the other structural example of the cyclic | annular IP conversion scaling part. FIG. 10 is a block diagram illustrating still another configuration example of the cyclic IP conversion scaling unit. It is a block diagram which shows the main structural examples of the personal computer to which this invention is applied.

Hereinafter, modes for carrying out the invention (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. First embodiment (image conversion apparatus)
2. Second embodiment (image conversion apparatus)
3. Third Embodiment (Image Conversion Device)
4). Fourth embodiment (personal computer)

<1. First Embodiment>
[Configuration of Image Conversion Device]
FIG. 3 is a block diagram showing a main configuration example of an image processing apparatus to which the present invention is applied.

  An image conversion apparatus 100 shown in FIG. 3 is an apparatus that converts an interlaced SD (Standard Definition) image into an IP (High Definition) image by performing IP conversion and further scaling processing.

  As illustrated in FIG. 3, the image conversion apparatus 100 includes a cyclic IP conversion scaling unit 111, a natural image prediction unit 113, an artificial image prediction unit 114, a natural image / artificial image determination unit 115, and a synthesis unit 116.

  The cyclic IP conversion scaling unit 111 is a processing unit that performs IP conversion and scaling processing, and suppresses reduction in image quality when performing IP conversion and scaling processing using motion information of an input image. The cyclic IP conversion scaling unit 111 includes an IP conversion scaling unit 121 and a cyclic conversion unit 122.

  Based on a predetermined method, the IP conversion scaling unit 121 IP-converts an input interlaced SD image (hereinafter also referred to as an input image) into a progressive SD image, and further converts the SD image into an HD image (hereinafter referred to as an HD image). , Also referred to as an intermediate image). The IP conversion scaling unit 121 supplies the progressive HD image after conversion to the cyclic conversion unit 122.

  The cyclic conversion unit 122 obtains a motion vector between an input image and an SD image generated from a progressive HD image (hereinafter also referred to as an output image) output from the cyclic conversion unit 122 one frame before. .

  The cyclic conversion unit 122 weights and adds the pixel value of the input image and the pixel value of the image that has been subjected to motion compensation to the output image based on the obtained motion vector, using the cyclic coefficient. Is converted to an output image that is a progressive HD image with higher image quality.

  The cyclic conversion unit 122 supplies the output image to the natural image prediction unit 113, the artificial image prediction unit 114, and the natural image / artificial image determination unit 115. Note that the cyclic coefficient is set for each pixel of the intermediate image based on the positional relationship between the input image and the pixel of the intermediate image and the reliability probability indicating the likelihood of the motion vector.

  The natural image prediction unit 113 predicts, from the HD image supplied from the cyclic conversion unit 122, an HD image (hereinafter referred to as a natural high-quality image) in which the natural image of the HD image is of high quality. Specifically, the natural image prediction unit 113 classifies the target pixel, which is a pixel of a natural high-quality image obtained from the HD image, into a class suitable for the feature of the natural image according to the feature of the HD image.

  Then, the natural image prediction unit 113 performs an operation using a prediction coefficient for predicting a natural high-quality image corresponding to the class and the HD image, so that the HD image supplied from the output phase conversion unit 112 is obtained. Predict natural high-quality images. The natural image prediction unit 113 supplies the natural high quality image to the synthesis unit 116.

  Here, the natural image is an image that is not an artificial image, which will be described later, and is an image obtained by directly capturing an image existing in the natural world.

  Similar to the natural image prediction unit 113, the artificial image prediction unit 114 uses an HD image (hereinafter referred to as artificial high quality) obtained from the HD image supplied from the cyclic conversion unit 122 with a high quality of the artificial image of the HD image. (Referred to as image). Specifically, the artificial image prediction unit 114 classifies the target pixel, which is a pixel of an artificial high quality image obtained from the HD image, into a class suitable for the feature of the artificial image according to the feature of the HD image.

  Then, the artificial image prediction unit 114 calculates the HD image supplied from the output phase conversion unit 112 by using the prediction coefficient for predicting the artificial high quality image corresponding to the class and the HD image. To predict artificial high quality images. The artificial image prediction unit 114 outputs the artificial high quality image to the synthesis unit 116.

  Here, the artificial image is an artificial image such as a character or a simple figure with little gradation and with clear edge phase information which is phase information, that is, with many flat portions. In other words, the artificial image is an image in which there are not many gradations such as characters and simple figures, and phase information such as contours is dominant.

  The natural image / artificial image determination unit 115 determines whether each pixel of the HD image supplied from the cyclic conversion unit 122 belongs to an area classified as an artificial image or an area classified as a natural image. Judgment is performed, and the determination result is output to the synthesis unit 116 as an artificial image degree. That is, the artificial degree of art is a value of 0 to 1 indicating the ratio of the artificial image in the natural image in the region classified between the artificial image and the natural image.

  Based on the determination result supplied from the natural-image / artificial-image determination unit 115, the synthesis unit 116 supplies the pixel value of each pixel of the natural high-quality image supplied from the natural-image prediction unit 113 and the artificial-image prediction unit 114. A pixel value of each pixel of the artificial high-quality image to be synthesized is synthesized at a ratio corresponding to the artificial picture degree, and an HD image obtained as a result of the synthesis is output.

[Configuration of cyclic IP conversion scaling unit]
FIG. 4 is a block diagram illustrating a main configuration example of the cyclic IP conversion scaling unit 111.

  As shown in FIG. 4, the cyclic conversion unit 122 of the cyclic IP conversion scaling unit 111 includes a motion vector detection unit 131, an MC (Motion Compensation) block difference detection unit 132, a cyclic coefficient setting unit 133, and a motion compensation unit 134. A product-sum operation unit 135, a frame memory 136, and a scaling unit 137.

  Hereinafter, in the coordinate system representing the pixel position of each image processed by the cyclic IP conversion scaling unit 111, the horizontal (horizontal) direction is the x-axis direction and the vertical (vertical) direction is the y-axis direction. That is, the coordinates of each pixel are represented by (x, y). Hereinafter, an input image that is an interlaced SD image input to the IP conversion scaling unit 121 from the outside is also referred to as an image I1, and an intermediate image that is a progressive HD image output from the IP conversion scaling unit 121 is referred to as an image. Also referred to as P1.

  The motion vector detection unit 131 detects a motion vector of each pixel of the image I1. An output image that is a progressive HD image output from the recursive conversion unit 122 is stored in the frame memory 136. Thus, the image one frame before (hereinafter also referred to as image P3) stored in the frame memory 136 is read from the frame memory 136, scaled in the scaling unit 137, and the same image size as the input image I1. To an SD image. The SD image converted in this way (hereinafter also referred to as image P <b> 3 ′) is supplied to the motion vector detection unit 131.

  The motion vector detection unit 131 divides the image I1 into blocks having a predetermined size, that is, N × M pixels (N and M are arbitrary positive integers), and detects a motion vector for each block.

  Hereinafter, in order to distinguish the coordinate system representing the position of a predetermined block in each image such as a block from the coordinate system representing the position of the pixel, the horizontal (horizontal) direction is taken as the X-axis direction, and the vertical (vertical) The direction is the Y-axis direction. That is, the coordinates of each block are represented by (X, Y). Further, hereinafter, the motion vector of the block of coordinates (X, Y) is represented as V (X, Y). The pixel in the upper left corner of each block is referred to as a reference pixel, and the coordinates of the reference pixel are referred to as reference coordinates.

  The motion vector detection unit 131 supplies information indicating the detected motion vector V to the MC block difference detection unit 132, the cyclic coefficient setting unit 133, and the motion compensation unit 134. In addition, the motion vector detection unit 131 supplies the image I1 and the image P3 ′ used for detection of the motion vector to the MC block difference detection unit 132.

  The MC block difference detection unit 132 for each block of the image I1, the pixel value of each block and the block of the image P3 ′ corresponding to the block, that is, the direction and distance indicated by the motion vector V for each block of the image I1. The MC block difference value, which is a difference value with the pixel value of the block of the image P3 ′ at the position moved by only the position, is detected. The MC block difference detection unit 132 supplies information indicating the detected MC block difference value to the cyclic coefficient setting unit 133.

  Hereinafter, the block difference value corresponding to the block of the coordinates (X, Y) of the image I1 is represented as BD (X, Y).

  The cyclic coefficient setting unit 133 performs pixel-by-pixel based on the motion vector V, the MC block difference value BD, and the positional relationship L between the pixels of the input image I1 and the pixels of the intermediate image P1 supplied from the IP conversion scaling unit 121. The cyclic coefficient K is set to. Hereinafter, the cyclic coefficient K for the pixel at the coordinates (x, y) is represented as K (x, y). The cyclic coefficient setting unit 133 supplies information indicating the set cyclic coefficient K to the product-sum operation unit 135.

  The motion compensation unit 134 reads the image P3 from the frame memory 136. Based on the motion vector V, the motion compensation unit 134 generates an image P4 obtained by performing motion compensation on the image P3. Since the motion vector V is detected in the SD image, the motion compensation unit 134 performs motion compensation after the motion vector V is made to correspond to the HD image (after changing to the representation on the HD image). Do. The motion compensation unit 134 supplies the image P4 to the product-sum operation unit 135.

  The product-sum operation unit 135 combines the images P1 and P4 at a ratio indicated by the cyclic coefficient K (weighted addition), and generates an output image (hereinafter also referred to as an image P2) that is a progressive HD image. . The product-sum operation unit 135 supplies the image P2 to the natural image prediction unit 113, the population image prediction unit 114, and the natural image population image determination unit 115 and stores the image P2 in the frame memory 136.

[Configuration of IP conversion scaling unit]
An IP conversion scaling method for converting an input image into an intermediate image is arbitrary. Therefore, the configuration of the IP conversion scaling unit 121 is arbitrary. For example, as illustrated in FIG. 5, the IP conversion scaling unit 121 may perform IP conversion and scaling processing using class classification adaptation processing.

  In the case of the example in FIG. 5, the IP conversion scaling unit 121 includes a class classification adaptation processing unit 141. The class classification adaptive processing unit 141 obtains a class code from tap data corresponding to each pixel (target pixel) in the unit pixel block constituting the output image, which is selectively extracted from the input image. The class classification adaptive processing unit 141 generates coefficient data of each class based on the coefficient seed data of each class and the phase information of the target pixel. Then, the class classification adaptive processing unit 141 calculates the data of the target pixel from the tap data corresponding to the target pixel and the coefficient data read by the class code using the estimation formula.

  In this way, the class classification adaptation processing unit 141 performs IP conversion and scaling processing on the input image I1, generates an intermediate image P1, and supplies it to the product-sum operation unit 135. In addition, the class classification adaptive processing unit 141 supplies the cyclic coefficient setting unit 133 with the positional relationship L between the pixels of the input image I1 and the pixels of the intermediate image P1.

  An example of the details of the class classification adaptive processing is described in Japanese Patent Laid-Open No. 2002-196737.

  The IP conversion scaling unit 121 may sequentially perform IP conversion and scaling processing as shown in FIG. In the case of the example in FIG. 6, the IP conversion scaling unit 121 includes a motion adaptive IP conversion unit 142 and a linear interpolation scaling unit 143.

  The motion-adaptive IP conversion unit 142 determines that a pixel of interest is newly generated when the positional change in the same field as the pixel of interest is larger than the temporal change of the pixel at the same position as the pixel of interest. Are considered to be included in a still region where there is little motion of the image. In this case, the motion adaptive IP conversion unit 142 uses the average value of the pixel values at the same position in the preceding and following fields as the pixel value of the target pixel.

  In addition, when the temporal change of the pixel at the same position as the target pixel is greater than the positional change of the same field as the target pixel, the motion adaptive IP conversion unit 142 The pixel of interest is considered to be included in an operation area where the motion of the image is large. In this case, the motion adaptive IP conversion unit 142 sets the average value of the pixel values adjacent to the target pixel as the pixel value of the target pixel.

  The linear interpolation scaling unit 143 performs image enlargement or reduction using linear interpolation on the IP conversion result image P0 generated by the motion adaptive IP conversion unit 142 to generate an intermediate image P1. The linear interpolation scaling unit 143 supplies the generated intermediate image P1 to the product-sum operation unit 135 and supplies the positional relationship L between the pixels of the input image I1 and the pixels of the intermediate image P1 to the cyclic coefficient setting unit 133.

[Positional relationship between input image and intermediate image]
Next, an example of the positional relationship L between the pixels of the input image and the pixels of the intermediate image will be described. 7 and 8 show an example of the positional relationship when the IP conversion is performed and the horizontal direction and the vertical direction are each doubled. FIG. 7 shows the positional relationship between the horizontal direction and the vertical direction, and FIG. 8 shows the positional relationship between the time direction and the vertical direction. Black circles indicate pixels of the intermediate image, and white circles indicate pixels of the input image. The solid white circle and the dotted white circle indicate the difference between fields.

  FIG. 9 and FIG. 10 show examples of the positional relationship when the horizontal and vertical directions are each expanded by 8/3 after IP conversion. FIG. 9 shows the positional relationship between the horizontal direction and the vertical direction, and FIG. 10 shows the positional relationship between the time direction and the vertical direction. Thus, scaling is not limited to integer multiples.

[Configuration of motion vector detection unit]
FIG. 11 is a block diagram illustrating a main configuration example of the motion vector detection unit 131.

  As illustrated in FIG. 11, the motion vector detection unit 131 includes a motion evaluation value detection unit 201, a pixel accuracy motion vector detection unit 202, a tap extraction unit 203, and a tap extraction unit 204. The motion vector detection unit 131 includes an ADRC (Adaptive Dynamic Range Coding) processing unit 205, a class classification unit 206, a coefficient memory 207, a prediction calculation unit 208, and a sub-pixel accuracy motion vector detection unit 209.

  The motion evaluation value detection unit 201 acquires an image I1 input from the outside, and acquires an image P3 from the frame memory 136. The motion evaluation value detection unit 201 divides the image I1 into blocks of a predetermined size, and sequentially sets each block as a target block. Further, the motion evaluation value detection unit 201 sequentially sets pixels within a predetermined range of the image P3 as a target pixel, and a block having the same size as the target block having the target pixel as a reference pixel (hereinafter referred to as a comparison target block). ) And the pixel value of the pixel at the position corresponding to the target block, and the sum of the absolute values of the difference values is obtained as a motion evaluation value for the target pixel. The motion evaluation value detection unit 201 supplies information indicating the detected motion evaluation value to the pixel accuracy motion vector detection unit 202, the tap extraction unit 203, and the tap extraction unit 204. Further, the motion evaluation value detection unit 201 supplies the image I1 and the image P3 used for detection of the motion vector to the MC block difference detection unit 132.

  Hereinafter, the motion evaluation value for the pixel of interest at coordinates (x, y) is represented as M (x, y).

  The pixel accuracy motion vector detection unit 202 uses a vector connecting the coordinates of the pixel having the smallest motion evaluation value M (hereinafter referred to as the minimum evaluation value pixel) and the coordinates of the reference pixel of the target block, as a pixel accuracy for the target block. That is, it is detected as a motion vector whose minimum unit is a distance equal to the pixel interval in the progressive image. The pixel accuracy motion vector detection unit 202 supplies information indicating the detected pixel accuracy motion vector to the tap extraction unit 203 and the tap extraction unit 204.

  The tap extraction unit 203 is used to predict a motion evaluation value at a position between the minimum evaluation value pixel and a pixel adjacent to the minimum evaluation value pixel, that is, a position with a pixel accuracy or less (hereinafter also referred to as a predicted position). A motion evaluation value M corresponding to a pixel of the image P3 that is a pixel and is adjacent to the minimum evaluation value pixel including the minimum evaluation value pixel is extracted as a prediction tap. The tap extraction unit 203 supplies the extracted prediction tap to the prediction calculation unit 208.

  The tap extraction unit 204 classifies the motion evaluation values M corresponding to some pixels of the image P3 used for classifying the minimum evaluation value pixels into any of several classes. Extract as The tap extraction unit 204 supplies the extracted class tap to the ADRC processing unit 205.

  The ADRC processing unit 205 performs ADRC processing on the motion evaluation value M of the pixels constituting the class tap, and supplies information indicating the resulting ADRC code to the class classification unit 206.

  In the K-bit ADRC, for example, the maximum value MAX and the minimum value MIN of the motion evaluation value M of the pixels constituting the class tap are detected, and DR = MAX−MIN is set to the motion evaluation value M constituting the class tap. Based on the dynamic range DR, the motion evaluation value M constituting the class tap is requantized to K bits based on the local dynamic range of the set. That is, the minimum value MIN is subtracted from the motion evaluation value M of each pixel constituting the class tap, and the subtracted value is divided (quantized) by DR / 2K. Then, a bit string in which the motion evaluation values M of the K-bit pixels constituting the class tap obtained in the above manner are arranged in a predetermined order is output as an ADRC code.

  The class classification unit 206 classifies the minimum evaluation value pixels based on the ADRC code from the ADRC processing unit 205, and supplies information indicating the class code corresponding to the class obtained as a result to the coefficient memory 207.

  The coefficient memory 207 stores a set of tap coefficients for each class obtained in advance by learning. The coefficient memory 207 is supplied from the tap coefficient stored in the address corresponding to the class code supplied from the class classification unit 206 in the stored tap coefficient set, that is, supplied from the class classification unit 206. The tap coefficient of the class represented by the class code is supplied to the prediction calculation unit 208.

  The prediction calculation unit 208 acquires the prediction tap output from the tap extraction unit 203 and the tap coefficient output from the coefficient memory 207, and uses the prediction tap and the tap coefficient to calculate the true value of the motion evaluation value M at the prediction position. A predetermined prediction calculation for obtaining a predicted value is performed. Thereby, the prediction calculation unit 208 obtains a predicted value of the motion evaluation value M ′ at the predicted position, and supplies it to the sub-pixel accuracy motion vector detection unit 209.

  The sub-pixel accuracy motion vector detection unit 209 detects the pixel or position where the motion evaluation value is minimum from the minimum evaluation value pixel and the predicted position, and the coordinates of the detected pixel or position and the coordinates of the reference pixel of the target block Are detected as motion vectors V with subpixel accuracy for the block of interest. The sub-pixel accuracy motion vector detection unit 209 supplies information indicating the detected motion vector V to the MC block difference detection unit 132, the cyclic coefficient setting unit 133, and the motion compensation unit 134.

[Configuration of cyclic coefficient setting unit]
FIG. 12 is a block diagram illustrating a main configuration example of the cyclic coefficient setting unit 133.

  As illustrated in FIG. 12, the cyclic coefficient setting unit 133 includes a basic cyclic coefficient setting unit 251, a motion variance detection unit 252, and a cyclic coefficient calculation unit 253. The cyclic coefficient calculation unit 253 includes a motion variance subtraction amount calculation unit 261, an MC block difference subtraction amount calculation unit 262, a calculation unit 263, and a calculation unit 264.

  The basic cyclic coefficient setting unit 251 sets the basic cyclic coefficient K0 based on the motion vector V and the positional relationship L between the pixels of the input image and the intermediate image. Hereinafter, the basic cyclic coefficient K0 for the pixel at the coordinates (x, y) is represented as K0 (x, y). The basic cyclic coefficient setting unit 251 supplies the basic cyclic coefficient K0 to the calculation unit 263.

  The motion variance detection unit 252 detects motion variance MD indicating the degree of variation with the surrounding motion vector V as the reliability of the motion vector V detected by the motion vector detection unit 131. Hereinafter, the motion variance MD for the motion vector V (X, Y) is represented as MD (X, Y). The motion variance detection unit 252 supplies information indicating the motion variance MD to the motion variance subtraction amount calculation unit 261.

  The motion variance subtraction amount calculation unit 261 calculates a motion variance subtraction amount KM1 that is a correction value for correcting the basic cyclic coefficient K0 based on the motion variance MD. Hereinafter, the motion dispersion subtraction amount KM1 for the pixel at the coordinates (x, y) of the image P1 is represented as KM1 (x, y). The motion variance subtraction amount calculation unit 261 supplies information indicating the motion variance subtraction amount KM1 to the calculation unit 263.

  The MC block difference subtraction amount calculation unit 262 calculates an MC block difference subtraction amount KM2 that is a correction value for correcting the basic cyclic coefficient K0 based on the MC block difference value BD. Hereinafter, the MC block difference subtraction amount KM2 for the pixel at the coordinates (x, y) of the image P1 is represented as KM2 (x, y). The MC block difference subtraction amount calculation unit 262 supplies information indicating the MC block difference subtraction amount KM2 to the calculation unit 264.

  The calculation unit 263 subtracts the motion variance subtraction amount KM1 from the basic cyclic coefficient K0. The calculation unit 264 subtracts the MC block difference subtraction amount KM2 from the value calculated by the calculation unit 263. Thereby, the cyclic coefficient K is calculated. Hereinafter, the cyclic coefficient K for the pixel at the coordinates (x, y) of the image P1 is represented as K (x, y). The calculation unit 264 supplies information indicating the cyclic coefficient K to the product-sum calculation unit 135.

[Image processing flow]
Next, an example of the flow of image processing executed by the image conversion apparatus 100 will be described with reference to the flowchart of FIG. The image processing is started, for example, when input of the image I1 is started from the outside.

  When the image processing is started, the cyclic IP conversion scaling unit 111 performs cyclic IP conversion scaling processing in step S101.

  In step S102, the natural image prediction unit 113 performs natural image prediction processing. Specifically, a natural high-quality image in which the natural image of the HD image is of high quality is predicted from the HD image supplied from the cyclic IP conversion scaling unit 111. That is, the natural image prediction unit 113 classifies the target pixel, which is a pixel of a natural high quality image obtained from the HD image, into a class suitable for the feature of the natural image, according to the feature of the HD image.

  Then, the natural image prediction unit 113 is supplied from the cyclic IP conversion scaling unit 111 by calculating using a prediction coefficient for predicting a natural high quality image corresponding to the class and the HD image. Predict natural high quality images from HD images. The natural image prediction unit 113 supplies the natural high quality image to the synthesis unit 116.

  In step S103, the artificial image prediction unit 114 performs an artificial image prediction process. Specifically, similarly to the natural image prediction unit 113, the artificial image prediction unit 114 is an artificial image obtained by improving the quality of an artificial image of the HD image from the HD image supplied from the cyclic IP conversion scaling unit 111. Predict high quality images. That is, the artificial image prediction unit 114 classifies the target pixel, which is a pixel of an artificial high quality image obtained from the HD image, into a class suitable for the feature of the artificial image according to the feature of the HD image.

  Then, the artificial image prediction unit 114 is supplied from the cyclic IP conversion scaling unit 111 by calculating using a prediction coefficient for predicting the artificial high quality image corresponding to the class and the HD image. An artificial high quality image is obtained from the HD image. The artificial image prediction unit 114 outputs the artificial high quality image to the synthesis unit 116.

  In step S104, the natural-image / artificial-image determination unit 115 performs a natural-image / artificial-image determination process. Specifically, the natural-image / artificial-image determination unit 115 determines, for each pixel of the HD image supplied from the cyclic IP conversion scaling unit 111, an area classified as an artificial image or an area classified as a natural image. It is determined to which region it belongs, and the determination result is output to the synthesis unit 116 as an artificial image degree.

  In step S105, the synthesizing unit 116 synthesizes the images. Specifically, based on the determination result supplied from the natural-image / artificial-image determination unit 115, the composition unit 116 determines the pixel value of each pixel of the natural high-quality image supplied from the natural-image prediction unit 113, and the artificial image. The pixel values of the respective pixels of the artificial high quality image supplied from the prediction unit 114 are combined at a ratio corresponding to the artificial image quality. The combining unit 116 outputs the combined image to a subsequent apparatus.

  In addition, when performing image conversion of a plurality of images continuously, the processing of steps S101 to S105 described above is repeatedly executed.

[Flow of cyclic IP conversion scaling processing]
Next, an example of the flow of the cyclic IP conversion scaling process executed in step S101 of FIG. 13 will be described with reference to the flowchart of FIG.

  When the cyclic IP conversion scaling process is started, the IP conversion scaling unit 121 performs the IP conversion scaling process in step S121 to generate the intermediate image P1. In step S122, the cyclic conversion unit 122 performs cyclic conversion processing to generate an output image P2.

  When the process of step S102 ends, the cyclic IP conversion scaling unit 111 ends the cyclic IP conversion scaling process, returns the process to step S101 of FIG. 13, and advances the process to step S102.

[Flow of cyclic conversion processing]
Next, an example of the flow of the cyclic conversion process executed in step S122 of FIG. 14 will be described with reference to the flowchart of FIG.

  When the cyclic conversion process is started, the motion vector detection unit 131 performs a motion vector detection process in step S141. By this processing, a motion vector V with a precision equal to or lower than that of each pixel of the image I1 is detected.

  In step S142, the MC block difference detection unit 132 detects the MC block difference value. Here, an MC block difference value detection method will be described with reference to FIGS. 16 and 17.

  First, as shown in FIG. 16, the block of the image P3 'located at the position corresponding to the block B0 of the image I1 is defined as B0'. A block obtained by moving the block B0 'by the direction and distance indicated by the motion vector V of the block B0 is referred to as a block B1.

  Since the motion vector V has subpixel accuracy, the position of the pixel in the block B1 does not coincide with the actual pixel position in the image P3 'when there is a fractional part in the horizontal motion amount or the vertical motion amount. Here, a method for calculating the pixel value of each pixel of the block B1 when the block B1 does not coincide with the actual pixel position in the image P3 'will be described with reference to FIG.

  A pixel z indicated by a black circle in FIG. 17 is one pixel in the block B1 and is located at a position that does not actually exist in the image P3 '. Pixels a to d indicated by white circles are pixels that exist in the image P <b> 3 ′ adjacent to (ie closest to) the pixel z. The pixel values of the pixels a to d and the pixel z are a to d and z, respectively, and the distance in the x-axis direction between the pixel a and the pixel z diagonally above and to the left of the pixel z is xd, and the distance in the y-axis direction is In the case of yd, the pixel value z of the pixel z is calculated based on the following formula (1).

  z = (1−yd) × ((1−xd) × a + xd × b) + yd × ((1−xd) × c + xd × d) (1)

  When the pixel position of the block B1 matches the pixel position actually present in the image P3 ', the pixel value of the image P3' is directly used as the pixel value of each pixel of the block B1.

  After calculating the pixel value of each pixel of the block B1, the MC block difference detection unit 132 calculates the MC block difference value BD based on the following equation (2).

  B0 (i, j) indicates the pixel value of the pixel in the block B0 at the position (i, j) with the reference pixel being the pixel in the upper left corner of the block B0 as the origin (0,0), and B1 (I, j) indicates the pixel value of the pixel in the block B1 at the position (i, j) with the reference pixel of the block B1 as the origin (0,0). That is, the MC block difference value BD is the sum of the absolute values of the difference values of the pixels at the corresponding positions in the block B0 and the block B1.

  Therefore, the greater the change in the image in the block corresponding to the current frame and the previous frame, the greater the MC block difference value BD. That is, a block having a large MC block difference value BD has a motion of the image due to a subject being deformed or complicated, a plurality of small subjects, or including an edge portion of the subject. Is likely to be a complex block. Therefore, there is a high possibility that the motion vector V detected for the block is not accurately detected, and it can be said that the reliability of the motion vector V is low.

  The MC block difference detection unit 132 detects the MC block difference value BD for each block of the image I1, and supplies information indicating the detected MC block difference value BD to the MC block difference subtraction amount calculation unit 262.

  Returning to FIG. 15, in step S143, the basic cyclic coefficient setting unit 251 sets the basic cyclic coefficient K0.

  In the conventional method, it is possible to obtain a high-definition IP conversion result that cannot be created from only one field of the input image by obtaining pixel information from the past by using a motion amount with sub-pixel accuracy of the input image. did it.

  At this time, as shown in FIG. 18, there are two types of cyclic coefficients depending on whether or not the pixel of the intermediate image whose pixel value is corrected using the cyclic coefficient is a pixel existing in the input image (ka and kb) Prepared.

  On the other hand, in the case of the present invention, pixel information is obtained from the past by using the amount of motion with sub-pixel accuracy of the input image, which cannot be created from only one field of the input image or the IP conversion result. A fine IP conversion scaling result can be obtained.

  In this case, since scaling can be arbitrarily performed in the scaling process, it is necessary to classify whether or not the pixel exists in the input image. In this case, since the scaling process is performed, it is necessary to consider the movement in the horizontal direction.

  Therefore, in this case, as shown in FIG. 19, the cyclic coefficient is obtained for each pixel according to the positional relationship L between the pixel of the input image and the pixel of the intermediate image.

  In FIG. 19, for convenience of explanation, it is shown that the pixels of the intermediate image are generated using a plurality of input fields. However, in practice, only the past output 1 frame is referred to and the cyclic coefficient is set. By using and correcting, pixels of the intermediate image are generated equivalently.

  Therefore, as shown in FIG. 20, the basic cyclic coefficient setting unit 251 determines the distance d0 between the target pixel of the intermediate image and the pixel of the input image closest to the target pixel, and the target pixel in the previous frame. Then, the distance d between the pixel moved by the motion vector and the pixel of the input image closest to the pixel is obtained.

  For these two variables d0 and d, the basic cyclic coefficient setting unit 251 generates the input image of the current frame when the distance d0 is small, that is, when the target pixel is close to the pixel of the input image of the current frame. Weight intermediate images. On the contrary, when the distance d is small, that is, when the target pixel is close to the pixel of the (past) input image of the previous frame, the basic cyclic coefficient setting unit 251 generates the intermediate image generated from the past input image. Weight the.

  That is, the basic cyclic coefficient setting unit 251 calculates the basic cyclic coefficient K0 as shown in the following formula (3).

  K0 = d0 / (d0 + d) (3)

  In this way, by setting the basic cyclic coefficient K0 according to the positional relationship L between the pixels of the input image and the pixels of the intermediate image, the image P1 can be kept in good resolution, that is, without blurring the image. It is possible to remove vertical folding distortion and noise.

  In step S144, the motion variance detector 252 detects motion variance. Here, motion dispersion will be described with reference to FIGS. 21 and 22.

  21 and 22 are diagrams showing examples of motion vector distribution. As shown in FIG. 21, when the motion vector V0a of the block indicated by diagonal lines is substantially the same as the peripheral motion vectors V1a to V8a, that is, the motion vector V0a and the motion vectors V1a to V8a of the peripheral blocks When the similarity is high, it is highly possible that the motion vector V0a is accurately obtained, and it can be said that the reliability of the motion vector V0a is high.

  On the other hand, as shown in FIG. 22, when the size and direction of the motion vector V0b of the block indicated by hatching and the motion vectors V1b to V8b of the peripheral blocks are greatly different, that is, the motion vector V0b and the peripheral motion vectors V1b to When the similarity with V8b is low, it is highly possible that the motion vector V0b is not accurately obtained, and it can be said that the reliability of the motion vector V0b is low.

  Therefore, the motion variance detection unit 252 detects, as the reliability of the motion vector V detected by the motion vector detection unit 131, a motion variance MD0 indicating the degree of variation from the surrounding motion vector V for each motion vector V. .

  Specifically, for example, the motion variance MD0 (X, Y) with respect to the motion vector V (X, Y) of the block Bp at the coordinates (X, Y) indicated by the oblique lines in FIG. The calculation is performed based on the following equation (4) for a region D0 of Nb0 × horizontal Mb0 block (vertical Np0 × horizontal Mp0 pixel when expressed in the number of pixels). Note that the coordinates of the pixel in the upper left corner of the region D0 are (x0, y0).

  Note that vx0 (x, y) indicates the horizontal motion amount of the block including the pixel at the coordinate (x, y), and vy0 (x, y) indicates the block including the pixel at the coordinate (x, y). Indicates the amount of vertical movement.

  That is, the motion variance MD0 represents an average value of the squares of the distances between the motion vector V of the block Bp and the motion vector V of each pixel in the region D0.

  In addition, as the reliability of the motion vector V, the motion variance detection unit 252 moves, for each motion vector V, a pixel (or block) corresponding to the motion vector V by the direction and distance indicated by the motion vector V. Motion variance MD1 indicating a variation with the motion vector V one frame before in the vicinity of is detected.

  Specifically, for example, first, a block of an image one frame before at a position corresponding to the block Bp in FIG. 23 is referred to as a block Bp ′ in FIG. Further, the block Bp ′ is in the direction and the distance indicated by the values rounded off to the nearest decimal point of the horizontal motion amount VX (X, Y) and the vertical motion amount VY (X, Y) of the motion vector V (X, Y) of the block Bp. The block to which is moved is referred to as block Bm. The motion variance detection unit 252 represents the motion variance MD1 (X, Y) for the motion vector V (X, Y) of the block Bp as a vertical Nb1 × horizontal Mb1 block centered on the block Bm. Np1 × horizontal Mp1 pixel) is calculated based on the following equation (5). Note that the coordinates of the pixel at the upper left corner of the region D1 are (x1, y1).

  Note that vx1 (x, y) indicates the horizontal motion amount of the block including the pixel at the coordinate (x, y) of the previous frame, and vy1 (x, y) indicates the coordinate (x, y) of the previous frame. ) Indicates the vertical motion amount of the block including the pixel.

  That is, the motion variance MD1 represents an average value of the square of the distance between the motion vector V of the block Bp and the motion vector V of each pixel in the region D1.

  Furthermore, the motion variance detection unit 252 calculates motion variance MD (X, Y) based on the following equation (6).

  MD (X, Y) = MD1 (X, Y) + MD2 (X, Y) (6)

  Accordingly, the greater the variation between the target motion vector V and the surrounding motion vector V, the greater the motion variance MD (X, Y). In other words, the pixel corresponding to the motion vector V (X, Y) and its vicinity have image movement due to the subject being deformed or complicated, or the presence of a plurality of small subjects. It is likely to be a complex area. Therefore, there is a high possibility that the motion vector V is not accurately detected, and it can be said that the reliability of the motion vector V (X, Y) is low.

  The motion variance detection unit 252 calculates the motion variance MD for each motion vector V in accordance with the method described above, and supplies information indicating the motion variance MD to the motion variance subtraction amount calculation unit 261.

  In step S145, the motion variance subtraction amount calculation unit 261 calculates a motion variance subtraction amount. Specifically, the motion variance subtraction amount calculation unit 261 calculates the motion variance subtraction amount KM1 for each pixel of the image P1 based on the following equation (7) using the motion variance MD of the block including the pixel. calculate.

  KM1 (x, y) = a1 × MD (X, Y) + b1 (7)

  Note that a1 and b1 are constants having predetermined values. However, the constant a1 is a positive value. Further, when the value on the right side of Expression (7) is smaller than 0, KM1 (x, y) is corrected to 0.

  In step S146, the MC block difference subtraction amount calculation unit 262 calculates the MC block difference subtraction amount. Specifically, the MC block difference subtraction amount calculation unit 262 uses the MC block difference subtraction amount KM2 for each pixel of the image P1 as the following formula (8) using the MC block difference value BD of the block including the pixel. ).

  KM2 (x, y) = a2 * BD (X, Y) + b2 (8)

  Note that a2 and b2 are predetermined constants. However, the constant a2 is a positive value. Further, when the value on the right side of Expression (8) becomes smaller than 0, KM2 (x, y) is corrected to 0.

  In step S147, the calculation unit 263 and the calculation unit 264 calculate a cyclic coefficient. Specifically, the calculation unit 263 subtracts the motion variance subtraction amount KM1 from the basic cyclic coefficient K0. The calculation unit 264 subtracts the MC block difference subtraction amount KM2 from the calculation result in the calculation unit 263. The subtraction result is the cyclic coefficient K. That is, the cyclic coefficient K (x, y) for the pixel at the coordinates (x, y) of the image P1 is calculated as in the following equation (9).

  K (x, y) = K0 (x, y) −KM1 (x, y) −KM2 (x, y) (9)

  That is, the cyclic coefficient K (x, y) increases as the motion variance subtraction amount KM1 (x, y) or the MC block difference subtraction amount KM2 (x, y) increases, that is, the motion variance MD (x, y) or MC. The larger the block difference value BD (x, y), the smaller the correction.

  In step S148, the motion compensation unit 134 performs motion compensation on the output image one frame before. Specifically, the motion compensation unit 134 reads the image P3 of the previous frame from the frame memory 136. The motion compensation unit 134 uses the motion vector V to generate an image P4 obtained by performing motion compensation on the image P3.

  Note that the pixel value of each pixel of the image P4 is the pixel value of the pixel of the image P3 at the position moved by the direction and distance indicated by the motion vector V of the pixel of the image P1 at the position corresponding to that pixel. . However, when there is a fractional part in the horizontal motion amount or the vertical motion amount of the motion vector V and there is no actual pixel at the position moved by the motion vector V of the image P3, the same method as described above with reference to FIG. By this method, the pixel value of the image P4 is calculated.

  In step S149, the product-sum operation unit 135 combines the images, and the cyclic conversion process ends. Specifically, the product-sum operation unit 135 weights and adds the pixel values of the pixels at the corresponding positions in the image P1 and the image P4 using the cyclic coefficient K. That is, the product-sum operation unit 135 uses the cyclic coefficient K to calculate the pixel value of the image P2 based on the following equation (10).

  Output image P2 = (1−K) × intermediate image P1 + K × motion compensation image P4 (10)

  In general, as the value of the cyclic coefficient K increases, the proportion of the component of the motion compensated image P4 in the output image P2 increases. Therefore, the effect of suppressing the occurrence of vertical aliasing distortion in the output image P2 increases. The blur is likely to occur.

  Conversely, the smaller the value of the cyclic coefficient K, the smaller the proportion of the motion compensated image P4 component in the output image P2, so the effect of suppressing the occurrence of vertical aliasing distortion in the output image P2 is reduced, while the blurring of the image is reduced. The occurrence of is suppressed.

  In the embodiment of the present invention, as described above, the cyclic coefficient K decreases as the motion variance subtraction amount KM1 or the MC block difference subtraction amount KM2 increases, that is, as the motion variance MD or MC block difference value BD increases. .

  That is, when the reliability of the motion vector V (X, Y) of the pixel at the coordinate (x, y) is low and the possibility that the correlation between the pixel of the intermediate image P1 to be weighted and the pixel of the motion compensated image P4 is low is high. Or, if the pixel is likely to be included in a region where the aliasing distortion or random noise is not noticeable and the motion of the image is complicated, the pixel value P1 (x, y) of the pixel value P2 (x, y) The proportion of components increases.

  On the other hand, the reliability of the motion vector V (X, Y) of the pixel at the coordinate (x, y) is high, and the correlation between the pixel of the intermediate image P1 to be weighted and the pixel of the motion compensated image P4 may be high. Is high, or when the pixel is likely to be included in a region where the motion of the image is small over a wide range in which aliasing distortion or random noise is conspicuous, the pixel value P1 ( The proportion of x, y) components increases.

  In addition, as the vertical folding distortion increases, the value of the cyclic coefficient K increases, and the ratio of the pixel value P4 (x, y) component to the pixel value P2 (x, y) increases. On the other hand, the smaller the vertical folding distortion, the smaller the value of the cyclic coefficient K, and the greater the proportion of the pixel value P1 (x, y) component in the pixel value P2 (x, y).

[Flow of cyclic conversion processing]
Next, the details of the motion vector detection processing in step S141 in FIG. 15 will be described with reference to the flowchart in FIG.

  In step S161, the motion evaluation value detection unit 201 acquires an image one frame before. Specifically, the motion evaluation value detection unit 201 acquires an image P3 ′, which is an output image one frame before, from the scaling unit 137.

  In step S162, the motion evaluation value detection unit 201 calculates a motion evaluation value of the pixel position. Specifically, the motion evaluation value detection unit 201 selects one block that has not yet detected a motion vector from among the blocks in the image I1, and sets it as a target block. The motion evaluation value detection unit 201 sequentially sets pixels within a predetermined range of the image P3 ′ as a target pixel, and obtains a motion evaluation value M (x, y) at each target pixel by the following equation (11).

  In equation (11), the coordinates of the pixel of interest are (x, y), and the coordinates of the reference pixel (upper left corner) of the block of interest are (xb, yb).

  That is, the motion evaluation value M is a value obtained by taking the difference between the pixel values of the pixels at the corresponding positions of the comparison target block and the target block with the target pixel as a reference pixel, and summing the absolute values of the difference values. It can be said that the smaller the motion evaluation value M, the closer the image in the comparison target block is to the image in the target block.

  In step S163, the pixel accuracy motion vector detection unit 202 detects a pixel accuracy motion vector. Specifically, the pixel accuracy motion vector detection unit 202 detects a minimum evaluation value pixel that minimizes the motion evaluation value M. The pixel accuracy motion vector detection unit 202 detects a vector connecting the coordinates of the minimum evaluation value pixel and the coordinates of the reference pixel of the target block as a pixel accuracy motion vector of the target block. The pixel accuracy motion vector detection unit 202 supplies information indicating the detected pixel accuracy motion vector to the tap extraction unit 203 and the tap extraction unit 204.

  In step S164, the tap extraction unit 203 extracts a prediction tap. Specifically, the tap extraction unit 203 extracts a motion evaluation value M corresponding to a pixel of the image P3 ′ in the vicinity of the minimum evaluation value pixel including the minimum evaluation value pixel as a prediction tap.

  FIG. 26 is a diagram illustrating a part of the image P3 '. In FIG. 26, the minimum evaluation value pixel is a pixel P0, and the motion evaluation values corresponding to the pixels P0 to P24 are the motion evaluation values M0 to M24. For example, the tap extraction unit 203 performs motion evaluation values corresponding to the pixel P0 that is the minimum evaluation value pixel, the pixels P1 to P8 adjacent to the outer periphery of the pixel P0, and the pixels P9 to P24 adjacent to the outer periphery of the pixels P1 to P8. M0 to M24 are extracted as prediction taps. The tap extraction unit 203 supplies the extracted prediction tap to the prediction calculation unit 208.

  In step S165, the tap extraction unit 204 extracts class taps. For example, in the example illustrated in FIG. 26, the tap extraction unit 204 extracts the motion evaluation values M0 to M8 corresponding to the pixel P0 that is the minimum evaluation value pixel and the pixels P1 to P8 adjacent to the pixel P0 as class taps. . The tap extraction unit 204 supplies the extracted class tap to the ADRC processing unit 205.

  In step S166, the ADRC processing unit 205 performs ADRC processing. Specifically, the ADRC processing unit 205 performs ADRC processing on the motion evaluation value M of the pixels constituting the class tap, and supplies information indicating the ADRC code obtained as a result to the class classification unit 206.

  In step S167, the class classification unit 206 performs class classification. Specifically, the class classification unit 206 classifies the minimum evaluation value pixel based on the ADRC code from the ADRC processing unit 205, and stores information indicating the class code corresponding to the resulting class in the coefficient memory 207. Supply.

  In step S168, the coefficient memory 207 supplies tap coefficients. Specifically, the coefficient memory 207 acquires a tap coefficient corresponding to the class code of the minimum evaluation value pixel from the set of tap coefficients stored therein. The coefficient memory 207 supplies the acquired tap coefficient to the prediction calculation unit 208.

  In step S169, the prediction calculation unit 208 performs prediction calculation. Specifically, for example, in the example illustrated in FIG. 26, when the position on the straight line connecting the pixel P0 and the pixels P1 to P8 and the intermediate positions P1 ′ to P8 ′ are set as the predicted positions, the prediction is performed. The computing unit 208 calculates motion evaluation values M1 ′ to M8 ′ at the positions P1 ′ to P8 ′ based on the following equation (12).

ただし、m＝1乃至8である。

However, m = 1 to 8.

Note that w _mn (n = 0 to 24) is a tap coefficient used to calculate the motion evaluation value Mm ′ (m = 1 to 8) at the position Pm ′ (m = 1 to 8).

  The prediction calculation unit 208 supplies the motion evaluation value Mm ′ at each prediction position to the sub-pixel accuracy motion vector detection unit 209.

  In step S170, the sub-pixel accuracy motion vector detection unit 209 detects a sub-pixel accuracy motion vector. Specifically, for example, in the case of the example shown in FIG. 26, the sub-pixel precision motion vector detection unit 209 detects the smallest one of the motion evaluation values of the pixel P0 and the positions P1 ′ to P8 ′. The sub-pixel accuracy motion vector detection unit 209 detects a vector connecting the coordinates of the pixel or position having the smallest motion evaluation value and the coordinates of the reference pixel of the target block as a sub-pixel accuracy motion vector V for the target block. That is, in the example of FIG. 26, a motion vector is detected with an accuracy of one half of the pixel interval.

  Note that by increasing the number of predicted positions set between pixels, it is possible to detect a motion vector with higher accuracy, that is, in smaller units.

  The sub-pixel accuracy motion vector detection unit 209 supplies information indicating the detected motion vector V to the MC block difference detection unit 132, the cyclic coefficient setting unit 133, and the motion compensation unit 134.

  In step S171, the motion evaluation value detection unit 201 determines whether all motion vectors have been detected. If there is still a block for which no motion vector has been detected, the motion evaluation value detection unit 201 determines that all motion vectors have not been detected, and the process returns to step S162. Thereafter, the processes in steps S162 to S171 are repeatedly executed until it is determined in step S171 that all motion vectors have been detected.

  If it is determined in step S171 that all motion vectors have been detected, the process proceeds to step S172.

  In step S172, the motion evaluation value detection unit 201 supplies the image used for the motion vector detection, and the sub-pixel precision motion vector detection process ends. Specifically, the motion evaluation value detection unit 201 supplies the image I1 and the image P3 ′ used for detection of the motion vector to the MC block difference detection unit 132.

  Next, learning of tap coefficients stored in the coefficient memory 207 will be described.

  First, learning of tap coefficients in the class classification adaptation process will be described based on a more general example. Specifically, a pixel value y of a pixel constituting an HD image (hereinafter, appropriately referred to as HD pixel) is used to predict an HD pixel from a pixel constituting the SD image (hereinafter appropriately referred to as SD pixel). Learning of tap coefficients will be described based on an example of a linear linear combination model obtained by linear combination of the following linear linear expressions using a plurality of SD pixels extracted as prediction taps and tap coefficients.

However, in Expression (13), x _n represents the pixel value of the pixel of the n th SD image data constituting the prediction tap for the HD pixel y, and w _n represents the pixel value of the n th SD pixel. Represents the nth tap coefficient to be multiplied. In Equation (13), the prediction tap is assumed to be composed of N SD pixels x1, x2,..., XN.

  Now, when the true value of the pixel value of the HD pixel of the k-th sample is expressed as yk and the predicted value of the true value yk obtained by the equation (13) is expressed as yk ′, the prediction error ek is expressed by the following equation: expressed.

  Since the predicted value yk ′ of equation (14) is obtained according to equation (13), the following equation is obtained by replacing yk ′ of equation (14) according to equation (13).

  In equation (15), xn, k represents the nth SD pixel constituting the prediction tap for the HD pixel of the kth sample.

Tap coefficient w _n of the prediction error ek and 0 of the formula (15) is the optimal for predicting the HD pixel, for all the HD pixels, to determine such tap coefficients w _n is Generally difficult.

Therefore, as the standard for indicating that the tap coefficient w _n is optimal, for example, when adopting the method of least squares, optimal tap coefficient w _n, as statistical errors, for example, by the following formula It can be obtained by minimizing the sum E of the squared errors represented.

  However, in Expression (16), K is the number of samples in the set of the HD pixel yk and the SD pixels x1, k, x2, k,..., XN, k constituting the prediction tap for the HD pixel yk. Represents.

The tap coefficient w _n that minimizes the sum E of square errors in the equation (16) is a value obtained by partially differentiating the sum E by the tap coefficient w _n , and therefore, it is necessary to satisfy the following equation: .

Therefore, when partial differentiation of above equation (17) by the tap coefficients w _n, the following equation is obtained.

  From the equations (17) and (18), the following equation is obtained.

  By substituting equation (15) into ek in equation (19), equation (19) can be expressed by the normal equation shown in equation (20).

Normal equation of Equation (20), HD pixel yk and SD pixels xn, a set of k, by preparing only a certain number, it is possible to formulate the same number as the number of tap coefficients w _n to be determined, thus by solving equation (20) it is possible to find the optimum tap coefficients w _n. In solving the equation (20), for example, a sweeping method (Gauss-Jordan elimination method) or the like can be employed. However, in order to solve the equation (20), in the equation (20), the left side matrix related to the tap coefficient w _n needs to be regular.

As described above, a large number of HD pixels y1, y2,..., YK are used as teacher data that serves as a teacher for learning tap coefficients, and SD pixels x1, k constituting the prediction tap for each HD pixel yk. , x2, k, · · ·, xN, a k, as student data serving as a student of the tap coefficient learning, by solving equation (20), it is possible to find the optimal tap coefficient w _n.

  Here, as the teacher data y, an image (hereinafter referred to as a teacher image) composed of the motion evaluation value M detected for each pixel of an image having a pixel density higher than that of the image P3 ′, which also has a pixel at the predicted position. By adopting an image obtained by thinning out pixels from the teacher image so that the student data x has the same pixel density as the image P3 ′, the tap coefficient used in the equation (12) can be obtained.

  Note that details of a method for detecting a motion vector with sub-pixel accuracy using the class classification adaptive processing are disclosed in, for example, Japanese Patent Application Laid-Open No. 9-187003 filed earlier by the present applicant.

  As described above, it is possible to suppress a reduction in image quality when performing IP conversion and scaling processing.

  Further, for example, the cyclic IP conversion scaling unit 111 outputs an image from which vertical aliasing distortion and noise are removed, so that a better image quality can be obtained in the subsequent image processing apparatus. For example, in the process of freely adjusting the image quality on a plurality of axes using the class classification adaptive process described in Japanese Patent Application Laid-Open No. 2002-218413 filed earlier by the present applicant, the resolution is increased more strongly. It becomes possible to adjust, and better image quality can be obtained.

<2. Second Embodiment>
[Configuration of cyclic IP conversion scaling unit]
In the above description, the example in which the image that is the target for detecting the motion vector V of the image I1 is the image P3 ′ that is the output image one frame before has been described. For example, instead of the image P3 ′, 1 You may make it use the image I2 which is the input image before a field.

  FIG. 27 is a block diagram illustrating a main configuration example of the cyclic IP conversion scaling unit 111 in the case where the image I2 that is the input image one field before is used as the target image for detecting the motion vector V of the image I1. is there.

  The cyclic IP conversion scaling unit 111 in this case basically has the same configuration as that of FIG. 4, but includes a cyclic conversion unit 322 instead of the cyclic conversion unit 122 of FIG.

  The cyclic conversion unit 322 basically has the same configuration as the cyclic conversion unit 122 of FIG. 4, but includes a frame memory 341 instead of the scaling unit 137. The cyclic conversion unit 322 includes a motion vector detection unit 331 instead of the motion vector detection unit 131, and includes an MC block difference detection unit 332 instead of the MC block difference detection unit 132.

  The frame memory 341 stores the image I1 input from the outside, delays it by one field, and supplies it to the motion vector detection unit 331. That is, an image I2, which is an image I1 delayed by one field after being input to the cyclic conversion unit 322, is supplied to the motion vector detection unit 331.

  The motion vector detection unit 331 detects the motion vector V of the image I1 with respect to the image I2 by the same method as the motion vector detection unit 131 described above. The motion vector detection unit 331 supplies the detected motion vector V to the MC block difference detection unit 332, the cyclic coefficient setting unit 133, and the motion compensation unit 134. In addition, the motion vector detection unit 331 supplies the image I1 and the image I2 used for the detection of the motion vector V to the MC block difference detection unit 332.

  For each block of the image I1, the MC block difference detection unit 332 has the pixel value of each block and the block of the image I2 corresponding to the block, that is, each block of the image I1 in the direction and distance indicated by the motion vector V. An MC block difference value that is a difference value from the pixel value of the block of the image I2 at the moved position is detected. The MC block difference detection unit 332 supplies information indicating the detected MC block difference value BD to the cyclic coefficient setting unit 133.

<3. Third Embodiment>
[Configuration of cyclic IP conversion scaling unit]
For example, instead of the image P3 ′, an image I3 that is an input image two fields before may be used.

  FIG. 28 is a block diagram illustrating a main configuration example of the cyclic IP conversion scaling unit 111 in the case where the image I3 that is the input image two fields before is used as the target image for detecting the motion vector V of the image I1. is there.

  The cyclic IP conversion scaling unit 111 in this case also basically has the same configuration as that of FIG. 4, but includes a cyclic conversion unit 422 instead of the cyclic conversion unit 122 of FIG.

  The cyclic conversion unit 422 basically has the same configuration as the cyclic conversion unit 122 of FIG. 4, but includes a frame memory 341 and a frame memory 441 instead of the scaling unit 137. The cyclic conversion unit 422 includes a motion vector detection unit 431 instead of the motion vector detection unit 131, and includes an MC block difference detection unit 432 instead of the MC block difference detection unit 132.

  The frame memory 341 stores the image I1 input from the outside, delays it by one field, and supplies it to the frame memory 441. That is, an image I2 that is an image I1 delayed by one field after being input to the cyclic IP conversion scaling unit 111 is supplied to the frame memory 441.

  The frame memory 441 stores the image I2, delays it by one field, and supplies it to the motion vector detection unit 431. That is, an image I3 that is an image I1 delayed by two fields after being input to the cyclic IP conversion scaling unit 111 is supplied to the motion vector detection unit 431.

  The motion vector detection unit 431 detects the motion vector V of the image I1 with respect to the image I3 by the same method as the motion vector detection unit 131 described above. Note that the detected motion vector V is a motion vector that is twice as long as the field interval at which the image I1 is input and the frame interval at which the image P2 is output, that is, a motion vector between two fields. Supplies the MC block difference detection unit 432, the cyclic coefficient setting unit 133, and the motion compensation unit 134 with information indicating a vector obtained by halving the size of the detected motion vector in order to ensure consistency. In addition, the motion vector detection unit 431 supplies the image I1 and the image I3 used for detection of the motion vector V to the MC block difference detection unit 432.

  For each block of the image I1, the MC block difference detection unit 432 converts the pixel value of each block and the block of the image I3 corresponding to each block, that is, each block of the image I1 to a vector that is a half of the motion vector V. The MC block difference value, which is a difference value from the pixel value of the block of the image I3 at the position moved by the direction and distance indicated by. The MC block difference detection unit 432 supplies information indicating the detected MC block difference value BD to the cyclic coefficient setting unit 133.

  In addition, the motion vector detection method with subpixel accuracy is not limited to the above-described method, and other methods may be used.

  Furthermore, in the above description, an example in which IP conversion processing and scaling processing are performed on an SD image has been described. Of course, the present invention performs IP conversion processing and scaling processing on a high-resolution image such as an HD image. It can also be applied to cases.

<4. Fourth Embodiment>
[Personal computer]
The series of processes described above can be executed by hardware or can be executed by software. In this case, for example, a personal computer as shown in FIG. 29 may be configured.

  In FIG. 28, the CPU 501 of the personal computer 500 executes various processes according to a program stored in a ROM (Read Only Memory) 502 or a program loaded from a storage unit 513 to a RAM (Random Access Memory) 503. The RAM 503 also appropriately stores data necessary for the CPU 501 to execute various processes.

  The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input / output interface 510 is also connected to the bus 504.

  The input / output interface 510 includes an input unit 511 including a keyboard and a mouse, a display including a CRT (Cathode Ray Tube) and an LCD (Liquid Crystal Display), an output unit 512 including a speaker, and a hard disk. A communication unit 514 including a storage unit 513 and a modem is connected. The communication unit 514 performs communication processing via a network including the Internet.

  A drive 515 is connected to the input / output interface 510 as necessary, and a removable medium 521 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted, and a computer program read from them is It is installed in the storage unit 513 as necessary.

  When the above-described series of processing is executed by software, a program constituting the software is installed from a network or a recording medium.

  For example, as shown in FIG. 29, the recording medium is distributed to distribute the program to the user separately from the apparatus main body, and includes a magnetic disk (including a flexible disk) on which the program is recorded, an optical disk ( It only consists of removable media 521 consisting of CD-ROM (compact disc-read only memory), DVD (including digital versatile disc), magneto-optical disc (including MD (mini disc)), or semiconductor memory. Rather, it is composed of a ROM 502 on which a program is recorded and a hard disk included in the storage unit 513, which is distributed to the user in a state of being pre-installed in the apparatus main body.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  Further, in the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but may be performed in parallel or It also includes processes that are executed individually.

  Further, in this specification, the system represents the entire apparatus composed of a plurality of devices (apparatuses).

  In addition, in the above description, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, the configurations described above as a plurality of devices (or processing units) may be combined into a single device (or processing unit). Of course, a configuration other than that described above may be added to the configuration of each device (or each processing unit). Furthermore, if the configuration and operation of the entire system are substantially the same, a part of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or other processing unit). . That is, the embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

  The present invention can be applied to an image processing apparatus that converts an interlaced image into a progressive image, such as a television receiver.

  100 image converter, 111 cyclic IP conversion scaling unit, 121 IP conversion scaling unit, 122 cyclic conversion unit, 131 motion vector detection unit, 132 MC block difference detection unit, 133 cyclic coefficient setting unit, 134 motion compensation unit, 135 Product-sum operation unit, 136 frame memory, 137 scaling unit, 251 basic cyclic coefficient setting unit, 252 motion variance detection unit, 253 cyclic coefficient calculation unit, 261 motion variance subtraction amount calculation unit, 262 MC block difference subtraction amount calculation unit, 263 Calculation unit, 264 calculation unit, 331 motion vector detection unit, 332 MC block difference detection unit, 341 frame memory, 431 motion vector detection unit, 432 MC block difference detection unit, 441 frame memory

Claims (10)

In an image processing apparatus that converts an interlaced input image into a scaled progressive output image,
IP conversion scaling means for converting the input image into a progressive image, and further converting the image size into a scaled intermediate intermediate image;
A motion vector detecting means for detecting a motion vector of the input image having a distance shorter than an interval between pixels of the input image as a minimum unit;
Based on the positional relationship between the motion vector detected by the motion vector detection means and the pixels of the input image and the pixels of the intermediate image obtained by converting the input image by the IP conversion scaling means. Cyclic coefficient setting means for setting a cyclic coefficient in
Motion compensation means for generating a motion compensated image, which is an image obtained by performing motion compensation on the past output image, based on the motion vector detected by the motion vector detection means;
A pixel value of a pixel of the intermediate image obtained by converting the input image by the IP conversion scaling unit and a pixel value of a pixel at a corresponding position of the motion compensation image generated by the motion compensation unit An image processing apparatus comprising: output image generation means for generating the output image by weighted addition using the cyclic coefficient set by the coefficient setting means. The cyclic coefficient setting means sets the value of the cyclic coefficient to be smaller as the distance between the target pixel and the pixel of the input image closest to the target image is shorter in the current frame of the intermediate image. The image processing apparatus described. The cyclic coefficient setting means determines the value of the cyclic coefficient as the distance between the position moved by the motion vector from the pixel of interest and the pixel of the input image closest to the position is shorter in the past frame of the intermediate image. The image processing apparatus according to claim 1, wherein a large value is set. Reliability detection means for detecting the reliability of the motion vector;
The image processing apparatus according to claim 1, further comprising: a correcting unit that corrects the cyclic coefficient based on the reliability of the motion vector. The reliability detection means detects, as the reliability of the motion vector, a first motion variance indicating a degree of variation from the surrounding motion vector for each of the motion vectors,
The image processing apparatus according to claim 4, wherein the correction unit corrects the cyclic coefficient to be smaller as the first motion variance is larger. The reliability detection means further determines, as the reliability of the motion vector, 1 for each of the motion vectors around the position where the pixel corresponding to the motion vector is moved by the direction and distance indicated by the motion vector. Detecting a second motion variance indicating the degree of variation with the motion vector before the frame;
The image processing apparatus according to claim 5, wherein the correction unit further corrects the cyclic coefficient so as to decrease as the second motion variance increases. The motion vector detection means detects the motion vector for each block of a predetermined size,
The reliability detection means is a difference value between a pixel value in the block of the input image and a pixel value in the block of the past output image obtained by moving the block by a direction and a distance indicated by the motion vector. Detect
The image processing apparatus according to claim 4, wherein the correction unit corrects the cyclic coefficient so as to decrease as the difference value increases. The output image generation means increases the ratio of the motion compensated image in the output image as the cyclic coefficient increases, and increases the ratio of the intermediate image in the output image as the cyclic coefficient decreases. Image processing apparatus. An image processing method of an image processing apparatus for converting an interlaced input image into a scaled progressive output image,
The IP conversion scaling means of the image processing device converts the input image into a progressive image, and further converts the image size into a scaled progressive intermediate image,
The motion vector detection means of the image processing device detects a motion vector of the input image having a minimum unit of a distance shorter than an interval of pixels of the input image;
The cyclic coefficient setting means of the image processing device, for each pixel, based on the detected motion vector and the positional relationship between the pixel of the input image and the pixel of the intermediate image obtained by converting the input image. Set the cyclic coefficient
The motion compensation unit of the image processing device generates a motion compensated image that is an image obtained by performing motion compensation on the past output image based on the detected motion vector,
The output image generation means of the image processing device sets the pixel value of the pixel of the intermediate image obtained by converting the input image and the pixel value of the pixel at the corresponding position of the generated motion compensation image. An image processing method for generating the output image by performing weighted addition using the cyclic coefficient. A computer that converts interlaced input images to scaled progressive output images.
IP conversion scaling means for converting the input image into a progressive image, and further converting the image size into a scaled intermediate intermediate image;
A motion vector detecting means for detecting a motion vector of the input image having a distance shorter than an interval between pixels of the input image as a minimum unit;
Based on the positional relationship between the motion vector detected by the motion vector detection means and the pixels of the input image and the pixels of the intermediate image obtained by converting the input image by the IP conversion scaling means. Cyclic coefficient setting means for setting a cyclic coefficient in
Motion compensation means for generating a motion compensated image, which is an image obtained by performing motion compensation on the past output image, based on the motion vector detected by the motion vector detection means;
A pixel value of a pixel of the intermediate image obtained by converting the input image by the IP conversion scaling unit and a pixel value of a pixel at a corresponding position of the motion compensation image generated by the motion compensation unit A program that functions as an output image generation unit that generates the output image by performing weighted addition using the cyclic coefficient set by a coefficient setting unit.

Images