JP2011217011A - Coefficient learning apparatus and method, image processing apparatus and method, program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve image quality, and to perform image quality conversion processing capable of coping with various conversion patterns by a simple constitution.SOLUTION: In a regression prediction operation and a discrimination prediction operation, a plurality of pixel values corresponding to the attention pixel of an input image or feature amounts obtained from the values are given as parameters. Four extraction values which are the plurality of pixel values corresponding to the attention pixel of the input image and the three feature amounts obtained from the values are used. The four extraction values are the pixel value extracted according to a moving direction, the maximum value and minimum value of the extracted pixel value, the absolute value of a differential feature amount according to the moving direction of the extracted pixel value, and the maximum value of the absolute value of the differential feature amount according to the moving direction of the extracted pixel value.

Description

  The present invention relates to a coefficient learning apparatus and method, an image processing apparatus and method, a program, and a recording medium, and more particularly, a coefficient learning apparatus and method that enables various motion blur removal to be realized efficiently and at low cost. The present invention relates to an image processing apparatus and method, a program, and a recording medium.

  In order to predict an image without noise from an input image including degradation such as noise, or to convert an SD signal into a high-resolution HD signal, a method using class classification adaptive processing has been proposed.

  When an SD signal is converted into an HD signal by class classification adaptive processing, first, the characteristics of the class tap composed of the input SD signal are obtained using ADRC (adaptive dynamic range coding) or the like, and the class tap obtained is converted. Classify based on features. Then, an HD signal is obtained by calculating a prediction coefficient prepared for each class and a prediction tap composed of an input SD signal.

  In addition, by applying the classification adaptation process, it is possible to remove the motion blur included in the image and restore the original image. Alternatively, the amount of blur is estimated using information on how much the imaged object is moving, the frequency characteristic is calculated based on the amount of blur, and the inverse filter of the calculated frequency characteristic is filtered. There has also been proposed a technique for performing correction according to (see, for example, Patent Document 1). However, when the original image is restored by removing the motion blur included in the image, in general, there is an adverse effect that noise is amplified or ringing occurs.

  In order to eliminate such adverse effects, for example, many variations of motion blur removal processing are prepared in advance, various motion blur removal processing is performed on the input image, and those that have not deteriorated are selected. There is a method to do.

  In addition, there is a method in which an optimization criterion that does not cause ringing is used to solve it in an iterative operation.

Japanese Patent Laid-Open No. 2006-081150

  However, when various types of motion blur removal processing are performed in advance, it is necessary to prepare many variations according to the direction and size of the motion. Also, when performing iterative calculations, the processing is repeated many times, and as a result, the circuit scale tends to increase and the processing time increases.

  In addition, it is difficult to uniformly determine the standard for selecting one of the variations in motion blur removal processing and the standard for stopping the iterative calculation, often resulting in deterioration and loss of image detail. .

  The present invention has been made in view of such a situation, and enables various motion blurs to be removed efficiently and at low cost.

  The first aspect of the present invention is a feature amount specified based on a motion vector from an image of the first signal, and is configured as a plurality of feature amounts obtained from pixel values of a target pixel and surrounding pixels. Regression that obtains a tap and calculates the regression coefficient of the regression prediction calculation that calculates the value of the pixel corresponding to the target pixel in the image of the second signal by the product-sum operation of each of the elements of the tap and the regression coefficient A coefficient calculation means, a regression prediction value calculation means for calculating a regression prediction value by performing the regression prediction calculation based on the calculated regression coefficient and the tap obtained from the image of the first signal; Based on a comparison result between the calculated regression prediction value and a pixel value corresponding to the target pixel in the image of the second signal, the target pixel is a pixel belonging to a first discrimination class, or A discriminating information providing means for providing discriminating information for discriminating whether the pixel belongs to two discriminating classes, and acquiring the tap from the image of the first signal based on the given discriminating information; A discriminant coefficient calculating means for calculating the discriminant coefficient of the discriminant prediction calculation for obtaining a discriminant prediction value for specifying the discriminant class to which the pixel of interest belongs by multiplying and summing each of the elements of the tap and the discriminant coefficient; A discriminant prediction value calculating means for calculating a discriminant prediction value by performing the discriminant prediction calculation based on the discriminant coefficient and the tap obtained from the image of the first signal; and the calculated discriminant prediction Classification means for classifying each pixel of the image of the first signal into either the first discrimination class or the second discrimination class based on a value, and the regression coefficient calculating unit Further calculates the regression coefficient using only the pixels classified into the first discrimination class, and further calculates the regression coefficient using only the pixels classified into the second discrimination class It is.

  Based on the regression prediction value calculated by the regression prediction value calculation unit for each of the discrimination classes by the regression coefficient calculated by the regression coefficient calculation unit for each of the discrimination classes, the discrimination information providing unit provides discrimination information. The process, the process in which the discrimination coefficient calculating unit calculates the discrimination coefficient, and the process in which the discrimination prediction value calculation unit calculates the discrimination prediction value can be repeatedly executed.

  If the difference between the regression prediction value and the value of the pixel corresponding to the target pixel in the image of the second signal is 0 or more, the target pixel is determined to be a pixel belonging to the first determination class; When the difference between the regression prediction value and the value of the pixel corresponding to the target pixel in the image of the second signal is less than 0, the target pixel is determined to be a pixel belonging to the first determination class. Can be.

  When the regression prediction value, the value of the pixel corresponding to the target pixel in the image of the second signal, and the absolute difference value are greater than or equal to a preset threshold value, the target pixel is a pixel belonging to the first discrimination class If the regression prediction value, the value of the pixel corresponding to the pixel of interest in the image of the second signal, and the absolute difference value are less than the threshold, the pixel of interest is a second discrimination class. It can be determined that the pixel belongs to the pixel.

  The image of the first signal may be an image obtained by adding motion blur to the image of the second signal.

  The tap is based on the motion direction and the amount of motion specified by the motion vector, the pixel value extracted according to the motion direction centered on the pixel of interest, the maximum and minimum values of the extracted pixel value, and the extracted pixel value Each of the absolute value of the differential feature quantity according to the movement direction and the maximum value of the absolute value of the differential feature quantity according to the movement direction of the extracted pixel value can be configured as an element.

  According to a first aspect of the present invention, the regression coefficient calculation means is a feature amount specified based on a motion vector from an image of the first signal, and includes a plurality of pixel values obtained from pixel values of a target pixel and peripheral pixels. The regression prediction calculation that obtains a tap configured as a feature quantity and obtains a value of a pixel corresponding to the target pixel in the image of the second signal by a product-sum operation of each of the elements of the tap and a regression coefficient Regression coefficient is calculated, and the regression prediction value calculation means calculates the regression prediction value by performing the regression prediction calculation based on the calculated regression coefficient and the tap obtained from the image of the first signal. Then, the determination information providing unit performs first determination on the target pixel based on a comparison result between the calculated regression prediction value and a pixel value corresponding to the target pixel in the image of the second signal. Belongs to class Discrimination information for discriminating whether the pixel is a pixel or a pixel belonging to the second discrimination class is provided, and the discrimination coefficient calculation unit is configured to calculate from the image of the first signal based on the provided discrimination information. Obtaining the tap, calculating the discrimination coefficient of the discrimination prediction calculation for obtaining a discrimination prediction value for specifying the discrimination class to which the pixel of interest belongs by multiplying and calculating each of the elements of the tap and the discrimination coefficient; A discriminant prediction value calculating unit calculates the discriminant prediction value by performing the discriminant prediction calculation based on the calculated discriminant coefficient and the tap obtained from the image of the first signal. Based on the calculated discrimination prediction value, each pixel of the image of the first signal is classified into one of the first discrimination class and the second discrimination class, and the first discrimination is performed. Minutes to class By which only said regression coefficient is further calculated by using pixels, which is the second coefficient learning method comprising the regression coefficients using only the classified pixels to determine class is further calculated.

  According to a first aspect of the present invention, a computer is a feature amount specified based on a motion vector from an image of a first signal, and is a plurality of feature amounts obtained from pixel values of a target pixel and peripheral pixels. Obtaining the configured tap, and calculating the regression coefficient of the regression prediction calculation for obtaining the value of the pixel corresponding to the pixel of interest in the image of the second signal by multiply-and-accumulate each of the elements of the tap and the regression coefficient. Regression prediction value calculation means for calculating a regression prediction value by performing the regression prediction calculation based on the calculated regression coefficient calculation means, the calculated regression coefficient, and the tap obtained from the image of the first signal And an image belonging to the first discrimination class for the target pixel based on a comparison result between the calculated regression prediction value and the value of the pixel corresponding to the target pixel in the second signal image. Discriminating information adding means for adding discriminating information for discriminating whether the pixel is a pixel belonging to the second discriminating class or the tap from the image of the first signal based on the given discriminating information And calculating the discriminant coefficient of the discriminant prediction calculation for obtaining the discriminant prediction value for specifying the discriminant class to which the pixel of interest belongs by multiplying and summing each of the elements of the tap and the discriminant coefficient Means, a discrimination prediction value calculation means for calculating a discrimination prediction value by performing the discrimination prediction calculation based on the calculated discrimination coefficient and the tap obtained from the image of the first signal, and the calculation Classification means for classifying each pixel of the image of the first signal into either the first discrimination class or the second discrimination class based on the determined discrimination prediction value, The regression coefficient calculation means further calculates the regression coefficient using only the pixels classified into the first discrimination class, and further calculates the regression coefficient using only the pixels classified into the second discrimination class This is a program that functions as a coefficient learning device.

  In the first aspect of the present invention, the feature amount is specified based on the motion vector from the first signal image, and is configured as a plurality of feature amounts obtained from the pixel values of the target pixel and the surrounding pixels. The regression coefficient of the regression prediction calculation for calculating the pixel value corresponding to the target pixel in the image of the second signal is calculated by multiplying and summing each of the elements of the tap and the regression coefficient. , The regression prediction value is calculated by performing the regression prediction calculation based on the calculated regression coefficient and the tap obtained from the image of the first signal, and the calculated regression prediction value, Based on the comparison result with the value of the pixel corresponding to the target pixel in the image of the second signal, the target pixel is a pixel belonging to the first determination class or a pixel belonging to the second determination class. Discriminating information for discriminating whether or not the tap is obtained from the image of the first signal based on the given discriminating information, and the product sum of each of the elements of the tap and the discriminant coefficient The discriminant coefficient of the discriminant prediction calculation for obtaining the discriminant prediction value for specifying the discriminant class to which the pixel of interest belongs is calculated, and is obtained from the calculated discriminant coefficient and the image of the first signal. A discrimination prediction value is calculated by performing the discrimination prediction calculation based on the tap, and each pixel of the image of the first signal is converted into the first discrimination class based on the calculated discrimination prediction value. And the regression coefficient is further calculated using only the pixels classified into the first discrimination class, and only the pixels classified into the second discrimination class. The regression coefficient is further calculated are.

  The second aspect of the present invention is a feature quantity specified from the first signal image based on the motion vector, and is configured as a plurality of feature quantities obtained from the pixel values of the target pixel and the surrounding pixels. A discriminant prediction means for obtaining a tap and performing a discriminant prediction operation for obtaining a discriminant prediction value for specifying a class to which the pixel of interest belongs by performing a product-sum operation of each of the elements of the tap and a discriminant coefficient; Based on the value, the classification means for classifying each pixel of the image of the first signal into either the first discrimination class or the second discrimination class, and from the image of the first signal, Regression prediction means that obtains the tap and predicts the pixel value of the pixel corresponding to the pixel of interest in the image of the second signal by calculating the regression prediction value by the product-sum operation of the tap and the regression coefficient. And An image processing apparatus to obtain.

  It is possible to repeatedly execute a process in which the discrimination prediction unit performs the discrimination prediction calculation and a classification unit in which the classification unit classifies each pixel of the image of the first signal.

  The image of the first signal may be an image obtained by adding motion blur to the image of the second signal.

  The tap is based on the motion direction and the amount of motion specified by the motion vector, the pixel value extracted according to the motion direction centered on the pixel of interest, the maximum and minimum values of the extracted pixel value, and the extracted pixel value Each of the absolute value of the differential feature quantity according to the movement direction and the maximum value of the absolute value of the differential feature quantity according to the movement direction of the extracted pixel value can be configured as an element.

  According to a second aspect of the present invention, the discriminating / predicting means is a feature amount specified based on a motion vector from the first signal image, and a plurality of features obtained from pixel values of the target pixel and the surrounding pixels. Classifying means that obtains a tap configured as a quantity and performs a discrimination prediction calculation to obtain a discrimination prediction value for specifying a class to which the pixel of interest belongs by performing a product-sum operation of each of the elements of the tap and a discrimination coefficient; Classifying each pixel of the image of the first signal into either the first discrimination class or the second discrimination class based on the discrimination prediction value; The pixel of the pixel corresponding to the pixel of interest in the second signal image is obtained by obtaining the tap from the image of the first signal and calculating the regression prediction value by the product-sum operation of the tap and the regression coefficient. Predict value Step is an image processing method comprising.

  According to a second aspect of the present invention, the computer is a feature amount specified based on a motion vector from an image of the first signal, and is a plurality of feature amounts obtained from pixel values of a target pixel and peripheral pixels. A discrimination prediction means for obtaining a configured tap and performing a discrimination prediction calculation for obtaining a discrimination prediction value for specifying a class to which the pixel of interest belongs by a product-sum operation of each of the elements of the tap and a discrimination coefficient; Classification means for classifying each pixel of the image of the first signal into either the first discrimination class or the second discrimination class based on the discrimination prediction value; and The pixel value of the pixel corresponding to the pixel of interest in the image of the second signal is predicted by acquiring the tap from the image and calculating the regression prediction value by the product-sum operation of the tap and the regression coefficient. A program to function as an image processing apparatus and a return prediction means.

  In the second aspect of the present invention, the feature amount is specified based on the motion vector from the first signal image, and is configured as a plurality of feature amounts obtained from the pixel values of the target pixel and the surrounding pixels. A discriminant prediction calculation for obtaining a discriminant prediction value for specifying a class to which the pixel of interest belongs is performed by a product-sum operation of each of the elements of the tap and the discriminant coefficient. Each of the pixels of the image of the first signal is classified into one of the first discrimination class and the second discrimination class, and the tap is acquired from the image of the first signal. Thus, by calculating the regression prediction value by the product-sum operation of the tap and the regression coefficient, the pixel value of the pixel corresponding to the target pixel in the image of the second signal is predicted.

  According to the present invention, various motion blurs can be removed efficiently and at low cost.

It is a block diagram which shows the structural example of the learning apparatus which concerns on one embodiment of this invention. It is a block diagram which shows the structural example of a learning pair production | generation apparatus. It is a figure explaining the pixel value extracted according to a motion direction. It is a figure explaining the complementation of the value of a pixel. It is a figure explaining the example of the system which calculates | requires the absolute value of the differential feature-value according to the motion direction of the extracted pixel value. It is a figure explaining another example of the system which calculates | requires the absolute value of the differential feature-value according to the motion direction of the extracted pixel value. It is a figure explaining another example of the system which calculates | requires the absolute value of the differential feature-value according to the motion direction of the extracted pixel value. It is a figure explaining another example of the system which calculates | requires the absolute value of the differential feature-value according to the motion direction of the extracted pixel value. It is a histogram explaining the process of the labeling part of FIG. It is a figure explaining learning of the discriminant coefficient performed repeatedly. It is a figure explaining learning of the discriminant coefficient performed repeatedly. It is a figure explaining the example in the case of classifying an input image using a binary tree structure. It is a block diagram which shows the structural example of the image processing apparatus corresponding to the learning apparatus of FIG. It is a flowchart explaining the example of the discrimination coefficient regression coefficient learning process by the learning apparatus of FIG. It is a flowchart explaining the example of a labeling process. It is a flowchart explaining the example of a regression coefficient calculation process. It is a flowchart explaining the example of a discrimination coefficient calculation process. It is a flowchart explaining the example of the discrimination regression prediction process by the image processing apparatus of FIG. It is a flowchart explaining the example of a discrimination | determination process. It is a figure explaining the effect of the image quality improvement process using the learning apparatus and image processing apparatus of this invention. It is a figure explaining the effect of the image quality improvement process using the learning apparatus and image processing apparatus of this invention. It is a figure explaining the effect of the image quality improvement process using the learning apparatus and image processing apparatus of this invention. It is a block diagram which shows the structural example of the television receiver which mounts the image processing apparatus of this invention. And FIG. 16 is a block diagram illustrating a configuration example of a personal computer.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration example of a learning device according to an embodiment of the present invention.

  The learning device 10 is a learning device used for image quality enhancement processing, and generates coefficients used in the image quality enhancement processing based on input student image and teacher image (or teacher signal) data. It is made like that.

  Here, the high image quality processing is, for example, processing that removes motion blur included in an image and restores the original image, or removes noise included in the image.

  The learning device 10 learns a regression coefficient that is a coefficient for generating a student image as an input image and a high-quality image close to a teacher image as an output image. Although the details will be described later, the regression coefficient calculates the value of the pixel corresponding to the target pixel in the high-quality image using the characteristic amount obtained from the value of the plurality of pixels corresponding to the target pixel of the input image as a parameter. The coefficients are used for the linear linear expression. The regression coefficient is learned for each class number described later.

  In addition, the learning device 10 classifies the target pixel into one of a plurality of classes based on the values of a plurality of pixels corresponding to the target pixel of the input image or the feature amount obtained from those values. . That is, the learning device 10 is configured to learn a discrimination coefficient for specifying which class of each pixel of interest in the input image belongs to which class for high image quality processing. Although details will be described later, the discrimination coefficient is a coefficient used in a linear linear expression using a value of a plurality of pixels corresponding to the target pixel of the input image or a feature amount obtained from these values as a parameter.

  That is, using the discriminant coefficient learned by the learning device 10, a linear linear expression is repeatedly executed using the values of a plurality of pixels corresponding to the target pixel of the input image or the feature amount obtained from these values as parameters. Thus, a class for high image quality processing is specified. Then, using a regression coefficient corresponding to the identified class, a linear linear calculation is performed using the values of a plurality of pixels corresponding to the target pixel of the input image or a feature amount obtained from these values as parameters. Thus, the pixel value of the image with high image quality is calculated.

  In the learning device 10, for example, an image without motion blur is input as a teacher image, and an image with motion blur is input as a student image. A teacher image and a student image are input one by one (referred to as a learning pair), and learning is performed by the learning device 10.

  FIG. 2 is a block diagram illustrating a configuration example of the learning pair generation device 30. As shown in the figure, the learning pair generation device 30 is configured to include a motion blur adding unit 31.

  In this example, a still image without motion blur is input to the learning pair generation device 30, and the input image is directly output as a teacher image. On the other hand, an image obtained by performing the process of the motion blur adding unit 31 on the input image (still image) is output as a student image.

  The motion blur adding unit 31 is a functional block that mainly adds motion blur to an input image. For example, the process of the motion blur adding unit 31 generates an image in which motion blur of a predetermined direction and size has occurred in the input image.

  Note that the learning pair generation device 30 may generate a student image by adding noise to the input image along with motion blur, for example.

  For example, a plurality of still images are supplied to the learning pair generation device 30, and learning pairs are generated as described above. Then, the generated learning pair is supplied as a student image and a teacher image to the learning apparatus 10 in FIG.

  Returning to FIG. 1, the student image data is supplied to the regression coefficient learning unit 21, the regression prediction unit 23, the discrimination coefficient learning unit 25, the discrimination prediction unit 27, and the motion vector detection unit 29.

  The motion vector detection unit detects a motion vector of the student image using, for example, a block matching method, a gradient method, or the like. The motion vector detected by the motion vector detection unit represents the direction and magnitude of motion blur of the student image. The regression coefficient learning unit 21, the regression prediction unit 23, the discrimination coefficient learning unit 25, and the discrimination prediction. Supplied to the unit 27.

  The regression coefficient learning unit 21 sets a predetermined pixel as a pixel of interest from among pixels constituting the student image. Then, the regression coefficient learning unit 21 calculates the coefficient of the regression prediction calculation formula for predicting the teacher image pixel value corresponding to the target pixel from the target pixel of the student image and the surrounding pixel values, for example, the least square method. Use to learn.

  Although details will be described later, in the present invention, in the above-described regression prediction calculation, it is assumed that the predicted value is a linear model using a regression coefficient learned by the learning device 10. At this time, in the regression prediction calculation, a value of a plurality of pixels corresponding to the target pixel of the input image or a feature amount obtained from these values is given as a parameter. In the present invention, values of a plurality of pixels corresponding to the target pixel of the input image and three feature amounts obtained from these values are used as this parameter. The values of a plurality of pixels corresponding to the target pixel of the input image extracted from the input image (student image) and the three feature amounts obtained from these values will be referred to as four extracted values.

  The above four extracted values are the pixel value extracted according to the movement direction, the maximum and minimum values of the extracted pixel value, the absolute value of the differential feature quantity according to the movement direction of the extracted pixel value, and the extracted pixel value, respectively. The absolute value of the differential feature quantity according to the movement direction of

  FIG. 3 is a diagram for explaining pixel values extracted according to the motion direction, which is the first extracted value of the four extracted values described above. In the example of FIG. 3, the pixels of the student image are indicated by circles arranged on the xy plane. Now, it is assumed that the target pixel corresponding to the phase (coordinates) of the pixel of the teacher image whose pixel value is to be predicted is a pixel indicated by a black (hatched) circle in the center of the drawing.

  The motion direction is specified based on the motion vector detected by the motion vector detection unit 29. In FIG. 3, “vertical movement”, “horizontal movement”, and “oblique movement” are shown as examples of movement directions.

  The pixel value extracted according to the motion direction that is the first extracted value extracts the value of the number of pixels corresponding to the motion amount specified based on the motion vector according to the motion direction specified based on the motion vector. Can be obtained. The amount of motion is the magnitude of the motion vector. For example, when the magnitude of the motion vector is represented by mv, the value of a pixel existing within a distance of 3 mv to 4 mv from the target pixel is extracted.

  Note that the motion vector is given by, for example, a two-dimensional vector (mx, my), and the magnitude mv of the motion vector is obtained as the square root of the sum of squares of the x and y components of the motion vector.

  For example, when the motion direction specified based on the motion vector is “vertical motion”, the value of a pixel that exists within a distance of 3 mv to 4 mv on the straight line in the vertical vertical direction centering on the target pixel in FIG. Is extracted. When the motion direction specified based on the motion vector is “horizontal motion”, the values of pixels existing within a distance of 3 mv to 4 mv on the straight line in the horizontal horizontal direction with respect to the target pixel in FIG. 3 are extracted. Is done.

  For example, when the motion direction specified based on the motion vector is “oblique motion”, pixels existing within a distance of 3 mv to 4 mv on a straight line in the oblique direction that is symmetric about the target pixel in FIG. The value of is extracted. In the example of FIG. 3, the direction of 45 ° to the right is an example of “oblique movement”, but actually there are more various “oblique movements”.

  In addition, when the motion direction specified based on the motion vector is “oblique motion”, there is a case where the pixel does not exist on an oblique straight line that is symmetric with respect to the target pixel. In that case, the pixel value extracted according to the movement direction is obtained by complementing the value of the nonexistent pixel based on the value of the existing pixel.

  FIG. 4 is a diagram for explaining pixel value complementation. In the figure, on the xy plane, pixels that exist are indicated by “x” in the figure, and pixels that do not exist are indicated by circles in the figure. For example, as shown in FIG. 4, when an oblique straight line 51 that is symmetric about the target pixel is obtained, it is necessary to extract the value of a pixel that does not exist at the position indicated by a circle in the drawing.

For example, the value x _ij of a pixel (non-existing pixel) indicated by a circle at the second position from the bottom in the figure is calculated by the equation (1) using the actual pixel values x _{0 to} x ₃ .

  As a result, the value of the nonexistent pixel is complemented based on the value of the existing pixel. In addition, in Formula (1), although the example complemented by the bilinear method was shown, you may make it complement by another system.

The maximum value and the minimum value of the extracted pixel values, which are the second extraction values among the four extraction values described above, are obtained by Expressions (2) and (3). In Expressions (2) and (3), x _ij represents each of the pixel values extracted according to the movement direction. For example, j pixel values corresponding to the i-th target pixel are extracted. Shall.

X _i ^(max) obtained by Expression (2) is the maximum value of the extracted pixel values, and x _i ^(min) obtained by Expression (3 ⁾ is the minimum value of the extracted pixel values.

  The absolute value of the differential feature amount according to the movement direction of the extracted pixel value, which is the third extraction value of the four extraction values described above, is, for example, two of the pixel values extracted according to the movement direction. It is obtained by a difference in pixel values, a dynamic range of surrounding pixel values, and the like.

  5 to 7 are diagrams illustrating an example of a method for obtaining the absolute value of the differential feature amount according to the movement direction of the extracted pixel value.

  5 and 6 show an example of a method for obtaining the absolute value of the differential feature quantity according to the movement direction of the pixel value extracted by the difference between two pixel values extracted from the pixel values according to the movement direction. 5 and 6, it is assumed that the motion direction specified based on the motion vector is “horizontal motion”.

In FIG. 5, the absolute value (| x _ij ^(mv) |) of the differential feature quantity according to the movement direction of the pixel value is obtained from the difference absolute value (| x _{ij + 1} −x _ij |) of the adjacent pixel values. An example of the case is shown. In this way, the absolute value of the differential feature value is obtained for each pixel value extracted according to the movement direction. In the case of FIG. 5, to be precise, it means the absolute value of the differential feature quantity at the coordinates indicated by “x” in the figure, but corresponds to, for example, the circle shown on the left side of “x” in the figure. This is assumed as the absolute value of the differential feature value of the pixel value.

FIG. 6 shows the absolute value (| x _ij ^(mv) |) of the differential feature quantity according to the movement direction of the pixel value by the difference absolute value (| x _{ij + 1} −x _ij−1 |) of the values of pixels not adjacent to each other. This shows an example in which is required. In this way, the absolute value of the differential feature value is obtained for each pixel value extracted according to the movement direction. In the case of FIG. 6, this means the absolute value of the differential feature value of the pixel value corresponding to the circle marked with “x” in the figure.

FIG. 7 shows an example of a method for obtaining the absolute value of the differential feature amount according to the movement direction of the extracted pixel value by the dynamic range (DR) of the peripheral pixel value. In the case of FIG. 7, it is assumed that pixels extracted according to the movement direction exist at the coordinates indicated by “x” in the drawing. The maximum value (p ^(max) ) of the values of pixels (including non-existing pixels) within a predetermined range from the coordinates indicated by “x” in the figure regardless of the motion direction specified based on the motion vector. And the minimum value (p ^(min) ) is the absolute value of the differential feature quantity according to the movement direction of the extracted pixel value.

  In addition, for example, the absolute value of the differential feature amount according to the movement direction of the pixel value extracted using the Sobel operator may be obtained.

  FIGS. 5 to 7 illustrate examples of methods for obtaining the absolute value of the differential feature amount according to the movement direction for each of all extracted pixel values. For example, in order to reduce the calculation scale, for example. The absolute value of the differential feature amount according to the movement direction may be obtained only for the main term. Here, the main term means a portion that is particularly effective for blur removal among the respective pixels corresponding to all the extracted pixel values.

  FIG. 8 is a diagram illustrating an example of a method for obtaining the absolute value of the differential feature amount according to the movement direction only for the main term. In FIG. 8, it is assumed that the motion direction specified based on the motion vector is “horizontal motion”. In the case of the example of FIG. 8, as in FIG. 5, the absolute value of the differential feature amount according to the movement direction of the pixel value is obtained from the absolute difference value of the adjacent pixel values.

  In the example of FIG. 8, the main term is a position that is separated by ½ mV on the left and right, respectively, and a position that is further separated by 1 mV from each position, with the pixel of interest indicated by a black circle at the center in the figure as the center. ing. That is, in the example of FIG. 8, the pixel indicated by the leftmost circle (referred to as the 0th pixel), the 5th pixel, the 10th pixel, and the 15th pixel are the main terms. ing.

  By obtaining the absolute value of the differential feature amount as shown in FIG. 8, for example, even if 17 pixel values are extracted as the first extraction value, the extracted pixel obtained as the third extraction is obtained. The absolute value of the differential feature amount according to the movement direction of the value can be four. By doing so, for example, the operation scale can be reduced, leading to cost reduction.

The maximum value of the absolute value of the differential feature quantity according to the movement direction of the extracted pixel value, which is the fourth extracted value of the four extracted values described above, is obtained by Expression (4). In Expression (4), x ^(mv) _ij represents each absolute value of the differential feature amount according to the moving direction of the extracted pixel value. That is, for example, j + 1 pixel values corresponding to the i-th target pixel are extracted, and the absolute value of the differential feature amount according to the j motion directions is obtained for each of the pixel values. To do.

| X _i ^(mv) | ^(max) obtained by Expression (4) is the maximum value of the absolute value of the differential feature quantity according to the movement direction of the extracted pixel value.

  In this way, four extracted values are obtained.

Next, learning of the above-described regression coefficient will be described. In the regression prediction calculation formula for predicting the teacher image pixel value described above, for example, the pixel value t _i (i = 1, 2,... N) of the teacher image is used, and the predicted value y _i (i = 1, 2, (N), equation (5) is established. Here, N represents the total number of samples of the pixel of the student image and the pixel of the teacher image.

Here, ε _i (i = 1, 2,... N) is an error term.

Assuming a linear model using the regression coefficient w, the predicted value y _i can be expressed as Equation (6) using the four extracted values extracted from the student image as parameters (also referred to as taps).

Note that w ^T represents a transposed matrix of w expressed as a determinant. w _o is a bias parameter and is a constant term, and it is possible that the bias parameter w _o is not included in equation (6). Note that the value of M corresponds to the number of tap elements described later.

In Expression (6), x _ij used as a parameter (tap) is a pixel value extracted according to the movement direction obtained corresponding to the target pixel of the student image, and a maximum value and a minimum value of the extracted pixel value. The absolute value of the differential feature quantity according to the movement direction of the pixel value and the maximum value of the absolute value of the differential feature quantity according to the movement direction of the extracted pixel value. That is, the tap x _ij is a vector having the above-described four extracted values as elements.

  When learning the coefficient of the regression prediction formula using the least square method, the predicted value obtained as described above is substituted into formula (5), and the sum of squares for all samples of the error term in formula (5). Is calculated by Equation (7).

Then, a regression coefficient w (including a bias parameter w _o as necessary) is derived so that the sum of squares E of all the samples of the error term in Expression (7) is minimized.

  Returning to FIG. 1, the regression coefficient learning unit 21 obtains the regression coefficient in this way. The regression coefficient obtained by the regression coefficient learning unit 21 is a coefficient used for calculation for predicting a pixel value of an image whose image quality is improved by regression prediction.

  The regression coefficient obtained by the regression coefficient learning unit 21 is stored in the regression coefficient storage unit 22.

  The regression prediction unit 23 sets a predetermined pixel as a target pixel from among the pixels constituting the student image. Then, the regression prediction unit 23 calculates the tap (four extracted values) described above.

The regression prediction unit 23 calculates the predicted value y _i by substituting the tap and the regression coefficient w into Equation (6).

The labeling unit 24 compares the predicted value y _i calculated by the regression prediction unit 23 with a true value t _i that is a pixel value of the teacher image. For example, the labeling unit 24 labels the target pixel whose predicted value y _i is equal to or greater than the true value t _i as the discrimination class A, and sets the target pixel whose predicted value y _i is less than the true value t _i as the discrimination class B. Label it. That is, the labeling unit 24 classifies each pixel of the student image into a discrimination class A and a discrimination class B based on the calculation result of the regression prediction unit 23.

FIG. 9 is a histogram for explaining the processing of the labeling unit 24. In the figure, the horizontal axis, (the combination of the pixel of the teacher image pixel and the learner image) from the predicted value y _i represents a difference value obtained by subtracting the true value t _i, the vertical axis, the sample difference value is obtained Represents the relative frequency.

As shown in the figure, the frequency of the sample in which the difference value obtained by subtracting the true value t _i from the predicted value y _i is 0 by calculation of the regression prediction unit 23 is the highest. When the difference value is 0, an accurate predicted value (= true value) is calculated by the regression predicting unit 23, and an image quality improvement process is appropriately performed. That is, since the regression coefficient is learned by the regression coefficient learning unit 21, it can be said that there is a high possibility that an accurate predicted value is calculated according to the equation (6).

  However, it cannot be said that an accurate regression prediction was made for a difference value other than zero. If so, there is room to learn more appropriate regression coefficients.

In the present invention, for example, if a regression coefficient is learned only for a target pixel whose predicted value y _i is equal to or greater than the true value t _i , a more appropriate regression coefficient can be learned for those target pixels. It is assumed that if a regression coefficient is learned only for a target pixel whose predicted value y _i is less than the true value t _i , a more appropriate regression coefficient can be learned for those target pixels. For this reason, the labeling unit 24 classifies each pixel of the student image into a discrimination class A and a discrimination class B based on the calculation result of the regression prediction unit 23.

  Thereafter, the coefficient used for the prediction calculation for classifying each pixel into the discrimination class A and the discrimination class B is learned based on the pixel value of the student image by the process of the discrimination coefficient learning unit 25. . That is, in the present invention, even if the true value is unknown, each pixel can be classified into the discrimination class A and the discrimination class B based on the pixel value of the input image.

Here, it has been described that the labeling unit 24 labels each pixel of the student image. However, the unit of labeling is precisely for each tap obtained from the student image corresponding to the true value t _i which is the pixel value of the teacher image. It will be labeled one by one.

Further, here, the pixel of interest predicted value y _i is equal to or greater than the true value t _i, but prediction value y _i has been described an example of labeling to determine the pixel of interest is less than the true value t _i, You may make it label by another system. For example, the pixel of interest whose absolute difference between the predicted value y _i and the true value t _i is less than a preset threshold is labeled as a discrimination class A, and the absolute difference between the predicted value y _i and the true value t _i A pixel of interest having a value equal to or greater than a preset threshold value may be labeled as a discrimination class B. Furthermore, the pixel of interest may be labeled into discrimination class A and discrimination class B by other methods. In the following, an example will be described in which a target pixel whose predicted value y _i is equal to or greater than the true value t _i and a target pixel whose predicted value y _i is less than the true value t _i are discriminated and labeled.

  Returning to FIG. 1, the discrimination coefficient learning unit 25 sets a predetermined pixel as a target pixel from among pixels constituting the student image. And the discrimination coefficient learning part 25 learns the coefficient used for the calculation of the predicted value for judging discrimination class A and discrimination class B from the attention pixel of a student image, and the surrounding pixel value.

In learning of the discriminant coefficient, a predicted value y _i for determining the discriminant class A and discriminant class B is obtained by equation (8) based on the feature amount obtained from the target pixel of the student image and the surrounding pixel values. Shall be.

Z ^T represents a transposed matrix of z expressed as a determinant. z _o is a bias parameter and is a constant term. It should be noted that the bias parameter z _o which is a constant term in the equation (8) can be excluded.

In equation (8), x _i used as a parameter is a tap composed of the four extracted values described above.

The discrimination coefficient learning unit 25 learns the coefficient z and the bias parameter z _o of the equation (8) and stores them in the discrimination coefficient storage unit 26.

  The coefficient of the discriminant prediction formula is derived by, for example, discriminant analysis. Or you may make it learn using a least squares method.

The coefficient z of the discriminant prediction formula thus obtained is a vector having the same number of elements as the number of tap elements described above. The coefficient z obtained by the discrimination coefficient learning unit 25 is a coefficient used for calculation for predicting whether a predetermined target pixel belongs to the discrimination class A or the discrimination class B, and is referred to as a discrimination coefficient z. To do. The bias parameter z _{o is} also a discrimination coefficient in a broad sense, and is stored in association with the discrimination coefficient z as necessary.

The prediction value is calculated by the discrimination prediction unit 27 using the coefficient z learned in this way, and it is determined whether the target pixel of the student image belongs to the discrimination class A or the discrimination class B. be able to. The discrimination prediction unit 27 calculates the predicted value y _i by substituting the tap and the discrimination coefficient z (including the bias parameter z _o as necessary) into the equation (8).

Then, the pixel of interest of the tap whose predicted value y _i is 0 or more as a result of the calculation by the discrimination prediction unit 27 is a pixel belonging to the discrimination class A, and the pixel of interest of the tap whose predicted value y _i is less than 0. Can be estimated to be pixels belonging to the discrimination class B.

However, the estimation based on the calculation result by the discrimination prediction unit 27 is not necessarily true. That is, the predicted value y _i calculated by substituting the tap and the discrimination coefficient z into the equation (8) is a result of prediction from the pixel value of the student image regardless of the pixel value (true value) of the teacher image. Actually, a pixel belonging to the discrimination class A may be estimated as a pixel belonging to the discrimination class B, or a pixel belonging to the discrimination class B may actually be estimated as a pixel belonging to the discrimination class A.

  Therefore, in the present invention, it is possible to perform prediction with higher accuracy by repeatedly learning the discrimination coefficient.

  That is, the class dividing unit 28 divides each pixel constituting the student image into a pixel belonging to the discrimination class A and a pixel belonging to the discrimination class B based on the prediction result of the discrimination prediction unit 27.

  Then, the regression coefficient learning unit 21 learns the regression coefficient for only the pixels belonging to the discrimination class A by the class dividing unit 28 and stores it in the regression coefficient storage unit 22 as described above. The regression prediction unit 23 calculates the prediction value based on the regression prediction as described above for only the pixels determined to belong to the discrimination class A by the class division unit 28.

  In this way, the obtained predicted value is compared with the true value, and the labeling unit 24 labels the pixels determined to belong to the discrimination class A by the class division unit 28 to the discrimination class A and the discrimination class B. .

  Further, the regression coefficient learning unit 21 learns the regression coefficient in the same manner as described above for only the pixels determined to belong to the discrimination class B by the class dividing unit 28. The regression prediction unit 23 calculates a prediction value based on the regression prediction in the same manner as described above for only the pixels that are determined to belong to the discrimination class B by the class division unit 28.

  In this way, the obtained predicted value is compared with the true value, and the labeling unit 24 labels the pixels determined to belong to the discrimination class B by the class dividing unit 28 to the discrimination class A and the discrimination class B. .

  That is, the pixel of the student image is divided into four sets. The first set is a set of pixels that are determined to belong to the discrimination class A by the class division unit 28 and are labeled to the discrimination class A by the labeling unit 24. The second set is a set of pixels that are determined to belong to the discrimination class A by the class division unit 28 and are labeled to the discrimination class B by the labeling unit 24. The third set is a set of pixels that are determined to belong to the discrimination class B by the class dividing unit 28 and are labeled to the discrimination class A by the labeling unit 24. The fourth set is a set of pixels that are determined to belong to the discrimination class B by the class dividing unit 28 and are labeled to the discrimination class B by the labeling unit 24.

  Thereafter, the discrimination coefficient learning unit 25 learns the discrimination coefficient again in the same manner as described above based on the first set and the second set among the four sets described above. Also, the discrimination coefficient learning unit 25 learns the discrimination coefficient again in the same manner as described above, based on the third set and the fourth set among the above-described four sets.

  FIG. 10 and FIG. 11 are diagrams for explaining learning of the discrimination coefficient that is performed iteratively.

  FIG. 10 is a diagram showing the tap values obtained from the student images and showing the spaces representing the taps of the student images with the tap value 1 as the horizontal axis and the tap value 2 as the vertical axis. That is, in order to simplify the description, all taps that can exist in the student image are virtually represented in the two-dimensional space with the number of tap elements being two. Therefore, in the figure, it is assumed that the tap is a vector composed of two elements.

  A circle 71 shown in the figure represents a set of taps corresponding to the pixels that the labeling unit 24 first labeled with the discrimination class A, and a circle 72 represents a pixel that the labeling unit 24 first labeled with the discrimination class B. Represents a set of taps corresponding to. The symbol 73 shown in the circle 71 represents the position of the average value of the tap elements included in the circle 71, and the symbol 74 shown in the circle 71 represents the average of the values of the tap elements included in the circle 72. It represents the position of the value.

  As shown in the figure, since the circle 71 and the circle 72 overlap each other, the tap corresponding to the pixel that is correctly labeled with the discrimination class A based only on the value of the tap element obtained from the student image. It is impossible to discriminate between the discrimination class B and the tap corresponding to the labeled pixel.

However, it is possible to specify the boundary line 75 for discriminating roughly two classes based on the symbols 73 and 74. Here, the process of specifying the boundary line 75 corresponds to the process of the discrimination prediction of the discrimination prediction unit 27 using the discrimination coefficient obtained by the first learning performed by the discrimination coefficient learning unit 25. In addition, the tap located on the boundary line 75 is a tap in which the predicted value y _i calculated by Expression (8) is zero.

  In order to identify a set of taps located on the right side of the boundary line 75 in the figure, the class dividing unit 28 assigns class code bits 1 to the pixels corresponding to those taps. In addition, in order to identify a set of taps located on the left side of the boundary line 75 in the figure, the class dividing unit 28 in FIG. 1 gives class code bits 0 to the pixels corresponding to those taps.

  The discriminant coefficient obtained by the first learning is stored in the discriminant coefficient storage unit 26 in FIG. 1 in association with a code representing the discriminant coefficient used for the first discriminant prediction. The Further, based on the result of the first discrimination prediction, the regression coefficient is learned again based on only the pixel to which the class code bit 1 is assigned, and the regression prediction is performed. Similarly, based on the result of the first discrimination prediction, the regression coefficient is learned again based on only the pixel to which the class code bit 0 is assigned, and the regression prediction is performed.

  Then, learning of the discrimination coefficient is repeated based on each of the pixel group to which the class code bit 1 is assigned and the pixel group to which the class code bit 0 is assigned. As a result, the pixel group to which the class code bit 1 is assigned is further divided into two, and the pixel group to which the class code bit 2 is assigned is further divided into two. The division at this time is performed by the discrimination prediction of the discrimination prediction unit 27 using the discrimination coefficient obtained by the second learning performed by the discrimination coefficient learning unit 25.

  The discriminant coefficient obtained by the second learning is stored in the discriminant coefficient storage unit 26 in FIG. 1 in association with a code representing the discriminant coefficient used for the second discriminant prediction. The The discriminant coefficients obtained by the second learning are the pixel group to which the class code bit 1 is given by the first discriminant prediction and the pixel group to which the class code bit 0 is given by the first discriminant prediction, respectively. 1 is stored in the discrimination coefficient storage unit 26 of FIG. 1 in association with a code indicating which pixel group is used for discrimination prediction. The That is, two types of discrimination coefficients used for the second discrimination prediction are stored.

  In addition, based on the results of the first and second discrimination predictions, the regression coefficient is learned again based on only the pixels to which the class code bits 11 are assigned, and regression prediction is performed. Similarly, based on the results of the first and second discrimination predictions, the regression coefficient is learned anew and regression prediction is performed based only on the pixels to which the class code bit 10 is assigned. Further, based on the results of the first and second discrimination predictions, the regression coefficient is learned again based on only the pixels to which the class code bit 01 is assigned, and the regression prediction is performed. Based on only the assigned pixels, the regression coefficient is learned again and regression prediction is performed.

  By repeating such processing, the space shown in FIG. 10 is divided as shown in FIG.

  FIG. 11 is a diagram showing a space representing each tap of a student image with the tap value 1 as the horizontal axis and the tap value 2 as the vertical axis, as in FIG. 10. In the figure, an example is shown in which the discrimination coefficient learning unit 25 repeats learning of the discrimination coefficient three times. That is, the boundary line 75 is specified by the discrimination prediction using the discrimination coefficient obtained by the first learning, and the boundary line 76-1 and the boundary are determined by the discrimination prediction using the discrimination coefficient obtained by the second learning. Line 76-2 is identified. The boundary lines 77-1 to 77-4 are specified by the discrimination prediction using the discrimination coefficient obtained by the third learning.

  The class dividing unit 28 in FIG. 1 assigns a first class code bit to identify a set of taps divided by the boundary line 75, and divides it by the boundary line 76-1 and the boundary line 76-2. In order to identify the set of generated taps, a second class code bit is provided, and the third bit is used to identify the set of taps divided by the boundary lines 77-1 to 77-4. Gives the eye class code bit.

  Accordingly, as shown in FIG. 11, each tap obtained from the student image is divided (classified) into eight classes of class numbers C0 to C7 specified based on the 3-bit class code. Become.

  When classification is performed as shown in FIG. 11, one type of discriminant coefficient used for the first discriminant prediction is stored in the discriminant coefficient storage unit 26 of FIG. 1, and the discriminant used for the second discriminant prediction is stored. Two types of coefficients are stored, and four types of discrimination coefficients used for the third discrimination prediction are stored.

  Further, when classification is performed as shown in FIG. 11, the regression coefficient storage unit 22 of FIG. 1 stores eight types of regression coefficients corresponding to the class numbers C0 to C7. Here, the eight types of regression coefficients corresponding to the class numbers C0 to C7 are the taps of the target pixels of the student images classified into the class numbers C0 to C7 as a result of the third discrimination prediction, and the attention Using the pixel value of the teacher image corresponding to the pixel as a sample, the regression coefficient is learned again for each class number and stored.

  As described above, if the discrimination coefficient is learned in advance using the student image and the teacher image and the discrimination prediction is repeated for the input image, the pixels of the input image are classified into eight classes having class numbers C0 to C7. It becomes possible to classify. If the regression prediction is performed using the taps corresponding to the pixels classified into the eight classes and the regression coefficients corresponding to the respective classes, it is possible to perform appropriate image quality improvement processing.

  FIG. 12 is a diagram illustrating an example of classifying an input image as shown in FIG. 11 using a binary tree structure. Each pixel of the input image is classified into a pixel to which the class code bit 1 or 0 of the first bit is given by the first discrimination prediction. At this time, it is assumed that the discrimination coefficient used for discrimination prediction is stored in the discrimination coefficient storage unit 26 of FIG. 1 as the discrimination coefficient corresponding to the iteration code 1.

  The pixels to which the first bit class code bit 1 is assigned are further classified into the pixels to which the second bit class code bit 1 or 0 is assigned. At this time, it is assumed that the discrimination coefficient used for discrimination prediction is stored in the discrimination coefficient storage unit 26 of FIG. 1 as the discrimination coefficient corresponding to the repetition code 21. Similarly, the pixels to which the class code bit 0 of the first bit is assigned are further classified into the pixels to which the class code bit 1 or 0 of the second bit is assigned. At this time, it is assumed that the discrimination coefficient used for discrimination prediction is stored in the discrimination coefficient storage unit 26 of FIG. 1 as the discrimination coefficient corresponding to the repetition code 22.

  Pixels to which the class code bit 11 of the first bit and the second bit is assigned are further classified into pixels to which the class code bit 1 or 0 of the third bit is assigned. At this time, it is assumed that the discrimination coefficient used for discrimination prediction is stored in the discrimination coefficient storage unit 26 of FIG. 1 as a discrimination coefficient corresponding to the repetitive code 31. The pixels to which the class code bit 10 of the first bit and the second bit is assigned are further classified into pixels to which the class code bit 1 or 0 of the third bit is assigned. At this time, it is assumed that the discrimination coefficient used for discrimination prediction is stored in the discrimination coefficient storage unit 26 of FIG. 1 as the discrimination coefficient corresponding to the repetition code 32.

  Similarly, pixels to which the first and second bit class code bits 01 or 00 are assigned are further classified into pixels to which the third bit class code bit 1 or 0 is assigned. Then, it is assumed that the discrimination coefficient corresponding to the repetition code 33 or the repetition code 34 is stored in the discrimination coefficient storage unit 26 of FIG.

  In this way, by repeating the determination three times, a class code consisting of 3 bits is set for each pixel of the input image, and the class number is specified. A regression coefficient corresponding to the specified class number is also specified.

  In this example, a value obtained by connecting the class code bits from the upper bit to the lower bit in order of the number of repetitions corresponds to the class number. Therefore, the class number Ck corresponding to the final class code is specified as shown in Equation (9), for example.

  Also, as shown in FIG. 12, the relationship between the number of iterations p and the final class number Nc is expressed by equation (10).

  Note that the final class number Nc is equal to the total number Nm of regression coefficients to be finally used.

  The total number Nd of discrimination coefficients is expressed by equation (11).

  Note that, in the discrimination prediction in the image quality improvement processing using the image processing apparatus to be described later, it is possible to increase the number of iterations adaptively, thereby increasing the processing speed and speed. In such a case, since the regression coefficients used in the respective branches in FIG. 12 are also required, the total number Nm of the regression coefficients is expressed by Expression (12).

  Although an example in which learning of the discrimination coefficient is performed three times repeatedly has been mainly described here, the number of repetitions may be one. That is, after the first discrimination coefficient learning is completed, the discrimination coefficient calculation by the discrimination coefficient learning unit 25 and the discrimination prediction by the discrimination prediction unit 27 may not be repeatedly executed.

FIG. 13 is a block diagram illustrating a configuration example of an image processing apparatus according to an embodiment of the present invention.
The image processing apparatus 100 in FIG. 1 is an image processing apparatus corresponding to the learning apparatus 10 in FIG. That is, the image processing apparatus 100 determines the class of each pixel of the input image using the determination coefficient learned by the learning apparatus 10. Then, the image processing apparatus 100 performs a regression prediction calculation of taps obtained from the input image using the regression coefficients corresponding to the determined class and learned by the learning apparatus 10, and increases the input image. Image processing for improving image quality is performed.

  That is, the discrimination coefficient stored in the discrimination coefficient storage unit 26 of the learning device 10 is stored in advance in the discrimination coefficient storage unit 122 of the image processing apparatus 100. The regression coefficient storage unit 124 of the image processing apparatus 100 stores in advance the regression coefficient stored in the regression coefficient storage unit 22 of the learning apparatus 10.

  In addition, the image processing apparatus 100 is provided with a motion vector detection unit 126. The motion vector detection unit 126 detects a motion vector of a student image using, for example, a block matching method, a gradient method, or the like. The motion vector detected by the motion vector detection unit represents the direction and magnitude of motion blur of the student image, and is supplied to the discrimination prediction unit 121 and the regression prediction unit 125. The motion vector is used when acquiring a tap as described above.

  The discrimination prediction unit 121 in the figure sets a target pixel in the input image, obtains a tap corresponding to the target pixel, and performs a calculation predicted with reference to Expression (8). The taps here are also the above four extracted values. That is, the pixel value extracted according to the movement direction, the maximum and minimum values of the extracted pixel value, the absolute value of the differential feature amount according to the movement direction of the extracted pixel value, and the differentiation according to the movement direction of the extracted pixel value The absolute value of the feature value is the maximum value.

  At this time, the discrimination prediction unit 121 identifies the iteration code based on the number of iterations and the pixel group that is the subject of discrimination prediction, and reads the discrimination coefficient corresponding to the iteration code from the discrimination coefficient storage unit 122.

The class division unit 123 divides the pixels of the input image into two sets by adding class code bits to the target pixel based on the prediction result of the discrimination prediction unit 121. At this time, as described above, for example, the predicted value y _i calculated by Expression (8) is compared with 0, and the class code bit is given to the target pixel.

  Through the processing of the class division unit 123, the discrimination prediction unit 121 repeatedly performs discrimination prediction, and further division is performed by the class division unit 123. The discrimination prediction is repeated by a preset number of times. For example, when the discrimination prediction is performed three times, the input image is classified into a pixel group corresponding to the class number of the 3-bit class code as described above with reference to FIG. 11 or FIG. It will be.

  Note that the number of iterations of discrimination prediction in the image processing apparatus 100 is set to be the same as the number of iterations of learning of the discrimination coefficient by the learning device 10.

  The class division unit 123 associates information for specifying each pixel of the input image with the class number of the pixel and supplies the information to the regression coefficient storage unit 124.

  The regression prediction unit 125 sets a target pixel in the input image, acquires a tap corresponding to the target pixel, and performs a calculation predicted with reference to Expression (6). At this time, the regression prediction unit 125 supplies information specifying the target pixel to the regression coefficient storage unit 124, and reads the regression coefficient corresponding to the class number of the target pixel from the regression coefficient storage unit 124. Yes.

  Then, an output image is generated in which the predicted value obtained by the calculation of the regression prediction unit 125 is the value of the pixel corresponding to the target pixel. As a result, an output image in which the input image is improved in image quality is obtained.

  Thus, according to the present invention, by performing discrimination prediction on the input image, each pixel constituting the input image (actually, a tap corresponding to the target pixel) is suitable for the high image quality processing. Can be classified into classes.

  For example, when various types of motion blur removal processing are performed in advance as in the conventional technology, it is necessary to prepare a large number of variations according to the direction and size of the motion. Also, when performing iterative calculations, the processing is repeated many times, and as a result, the circuit scale tends to increase and the processing time increases.

  In addition, it is difficult to uniformly determine the standard for selecting one of the variations in motion blur removal processing and the standard for stopping the iterative calculation, often resulting in deterioration and loss of image detail. .

  On the other hand, in the present invention, it is not necessary to change the processing according to the direction and size of the movement, and the circuit scale can be made appropriate. In addition, there is no need for a criterion for selecting a process or a criterion for stopping an iterative calculation, so that the image does not remain deteriorated or the details of the image are not impaired.

  Furthermore, in the present invention, it is possible to classify more appropriately by repeatedly performing discrimination prediction. In addition, it is not necessary to generate intermediate data or the like obtained by performing processing on the pixel values of the input image in the middle of the repetitive discrimination prediction processing, so that the processing speed can be increased. That is, when predicting an output image, class classification and regression prediction can be performed by calculating the prediction formula at most (p + 1) times for any pixel, so that high-speed processing is possible. In addition, when class classification and regression prediction are performed, it is possible to use the pipeline structure in the implementation because it is always completed only by the operation for the input without using the intermediate data of the tap operation. .

  Next, details of the discrimination coefficient regression coefficient learning process will be described with reference to the flowchart of FIG. This process is executed by the learning device 10 of FIG.

  In step S101, the discrimination coefficient learning unit 25 identifies a repetitive code. In this case, since the first learning process, the repetitive code is specified as 1.

  In step S102, the regression coefficient learning unit 21 to the labeling unit 24 execute a labeling process to be described later with reference to FIG. Here, a detailed example of the labeling process in step S102 in FIG. 14 will be described with reference to the flowchart in FIG.

  In step S131, the regression coefficient learning unit 21 executes a regression coefficient calculation process which will be described later with reference to FIG. Thereby, the regression coefficient used for the calculation for predicting the pixel value of the teacher image based on the pixel value of the student image is obtained.

In step S132, the regression prediction unit 23 calculates a regression prediction value using the regression coefficient obtained by the process of step S131. At this time, for example, the calculation of Expression (6) is performed to obtain the predicted value y _i .

In step S133, the labeling unit 24 compares the predicted value y _i obtained by the process of step S132 with the true value t _i that is the pixel value of the teacher image.

  In step S134, the labeling unit 24 labels the target pixel (actually, a tap corresponding to the target pixel) into the discrimination class A or the discrimination class B based on the comparison result in step S133. Thereby, for example, as described above with reference to FIG. 9, the discrimination class A or the discrimination class B is labeled.

  Note that the processing in steps S132 to S134 is performed for each pixel to be processed that is determined in accordance with the repetition code.

  In this way, the labeling process is executed.

  Next, a detailed example of the regression coefficient calculation process in step S131 in FIG. 15 will be described with reference to the flowchart in FIG.

  In step S151, the regression coefficient learning unit 21 specifies a sample corresponding to the repetitive code specified in the process of step S101. Here, the sample means a combination of a tap corresponding to the target pixel of the student image and a pixel of the teacher image corresponding to the target pixel. The tap is acquired based on the motion vector detected by the motion vector detection unit 29.

  For example, if the repetitive code is 1, it is the first learning process, and therefore, the sample is specified with each of the pixels of the student image as the target pixel. For example, if the repetitive code is 21, it is a part of the second learning process, so that each pixel of the student image to which the class code bit 1 is assigned in the first learning process is selected as the target pixel. As the sample is identified. For example, if the repetitive code is 34, it is a part of the third learning process, so that the class code bit 0 is given in the first learning process among the pixels of the student image, and the second learning process. Thus, a sample is specified using each pixel to which the class code bit 0 is assigned as a target pixel.

  In step S152, the regression coefficient learning unit 21 adds the samples specified in the process of step S151. At this time, for example, the sample tap and the pixel value of the teacher image are added to Expression (5).

  In step S153, the regression coefficient learning unit 21 determines whether all samples have been added, and the process of step S152 is repeatedly executed until it is determined that all samples have been added.

  In step S154, the regression coefficient learning unit 21 performs, for example, the calculation of Expression (7), and derives the regression coefficient using the least square method.

  In this way, the regression coefficient calculation process is executed.

  Thus, the labeling process in step S102 in FIG. 14 is completed, and the process proceeds to the discrimination coefficient calculation process in step S103 in FIG.

  In step S103, the discrimination coefficient learning unit 25 executes a discrimination coefficient calculation process described later with reference to FIG. Here, a detailed example of the discrimination coefficient calculation process in step S103 in FIG. 14 will be described with reference to the flowchart in FIG.

  In step S171, the discrimination coefficient learning unit 25 specifies a sample corresponding to the repetitive code specified in the process of step S101. Here, the sample means a combination of the tap corresponding to the target pixel of the student image and the result of labeling of the discrimination class A or the discrimination class B for the target pixel.

  For example, if the repetitive code is 1, it is the first learning process, and therefore, the sample is specified with each of the pixels of the student image as the target pixel. For example, if the repetitive code is 21, it is a part of the second learning process, so that each pixel of the student image to which the class code bit 1 is assigned in the first learning process is selected as the target pixel. As the sample is identified. For example, if the repetitive code is 34, it is a part of the third learning process, so that the class code bit 0 is given in the first learning process among the pixels of the student image, and the second learning process. Thus, a sample is specified using each pixel to which the class code bit 0 is assigned as a target pixel.

  In step S172, the discrimination coefficient learning unit 25 adds the samples specified in the process of step S171.

  In step S173, the discrimination coefficient learning unit 25 determines whether or not all samples have been added, and the process of step S172 is repeatedly executed until it is determined that all samples have been added.

  In step S174, the discrimination coefficient learning unit 25 derives a discrimination coefficient by, for example, discriminant analysis (the least square method may be used).

  In this way, the discrimination coefficient calculation process is executed.

Returning to FIG. 14, in step S <b> 104, the discrimination prediction unit 27 calculates a discrimination prediction value by using the coefficient obtained by the processing in step S <b> 103 and the tap obtained from the student image. At this time, for example, the calculation of Expression (8) is performed, and the predicted value y _i (discriminated predicted value) is obtained.

  In step S105, the class dividing unit 28 determines whether or not the discrimination prediction value obtained by the process of step S104 is 0 or more.

  If it is determined in step S105 that the discrimination prediction value is 0 or more, the process proceeds to step S106, and the class code bit 1 is set to the target pixel (actually a tap). On the other hand, when it is determined in step S105 that the discrimination prediction value is less than 0, the process proceeds to step S107, and the class code bit 0 is set to the target pixel (actually a tap).

  Note that the processing from step S104 to step S107 is performed for each pixel to be processed that is determined corresponding to the repetition code.

  After the process of step S106 or step S107, the process proceeds to step S108, and the discrimination coefficient storage unit 26 associates the discrimination coefficient obtained in the process of step S103 with the repetition code specified in step S101. Remember.

  In step S109, the learning device 10 determines whether or not the iteration has ended. For example, when learning by repeating three times is set in advance, it is determined that the iteration has not been completed yet, and the process returns to step S101.

  In step S101, a repetitive code is specified again. In this case, since it is the first process of the second learning, the repetitive code is identified as 21.

  Similarly, the processes of steps S102 to S108 are executed. At this time, as described above, in the processing of step S102 and the processing of step S103, each of the pixels of the student image to which the class code bit 1 is assigned in the first learning processing is sampled as a target pixel. Will be specified.

  Then, in step S109, it is determined whether or not the iteration has been completed.

  As described above, the processes in steps S101 to S108 are repeatedly executed until it is determined in step S109 that the repetition has been completed. If it is preset that learning is repeated three times, it is determined in step S101 that the iteration code is 34, and then the processing of steps S102 to S108 is executed. In step S109, the iteration is completed. Will be judged.

  As described above with reference to FIG. 9, the seven types of discrimination coefficients are associated with the iteration codes in the discrimination coefficient storage unit 26 as a result of the processes of steps S101 to S109 being repeatedly executed as described above. It will be remembered.

  If it is determined in step S109 that the iteration has been completed, the process proceeds to step S110.

  In step S110, the regression coefficient learning unit 21 executes a regression coefficient calculation process. Since this process is the same as that described above with reference to the flowchart of FIG. 16, a detailed description thereof is omitted. In this case, in step S151, a sample corresponding to the repetitive code is not specified. Each sample corresponding to the class number is identified.

  That is, as the processing of steps S101 to S109 is repeatedly executed, each pixel of the student image is classified into one of the class numbers C0 to C7 as described above with reference to FIG. Become. Therefore, a sample is identified using the pixel of class number C0 of the student image as the pixel of interest, and the first regression coefficient is derived. Also, a sample is identified with the pixel of class number C1 of the student image as the pixel of interest, a second regression coefficient is derived, a sample is identified with the pixel of class number C2 of the student image as the pixel of interest, and the first A regression coefficient of 3 is derived, a sample is specified with a pixel of class number C7 of the student image as a target pixel, and an eighth regression coefficient is derived.

  That is, in the regression coefficient calculation process in step S110, eight types of regression coefficients corresponding to the class numbers C0 to C7 are obtained.

  In step S111, the regression coefficient storage unit 22 stores each of the eight types of regression coefficients obtained by the process of step S110 in association with the class number.

  In this way, the discriminant regression coefficient learning process is executed.

  Although an example has been described here in which learning of the discrimination coefficient is repeatedly performed three times, the number of repetitions may be one. That is, after the first discrimination coefficient learning is completed, the discrimination coefficient calculation by the discrimination coefficient learning unit 25 and the discrimination prediction by the discrimination prediction unit 27 may not be repeatedly executed.

  Next, an example of the discriminant regression prediction process will be described with reference to the flowchart of FIG. This process is executed by the image processing apparatus 100 of FIG. Prior to the execution of the processing, each of the discrimination coefficient storage unit 122 and the regression coefficient storage unit 124 of the image processing apparatus 100 is stored in the discrimination coefficient storage unit 26 by the discrimination regression coefficient learning process of FIG. It is assumed that the types of discrimination coefficients and the 8 types of regression coefficients stored in the regression coefficient storage unit 22 are stored.

  In step S191, the discrimination prediction unit 121 identifies a repetitive code. In this case, since it is the first determination process, the repetitive code is specified as 1.

  In step S192, the discrimination prediction unit 121 executes discrimination processing described later with reference to FIG. Here, a detailed example of the determination processing in step S192 in FIG. 18 will be described with reference to the flowchart in FIG.

  In step S211, the discrimination prediction unit 121 sets a pixel of interest corresponding to the repetition code. For example, if the repetitive code is 1, since this is the first determination process, all the pixels of the input image are set as the target pixel. For example, if the repetitive code is 21, it is a part of the second determination process, and therefore each pixel to which the class code bit 1 is assigned in the first determination process is the pixel of interest among the pixels of the input image. Set as For example, if the repetitive code is 34, it is a part of the third discrimination process, and therefore, the class code bit 0 is given in the first discrimination process among the pixels of the input image, and the second discrimination process. Thus, each pixel to which the class code bit 0 is assigned is set as a target pixel.

  In step S212, the discrimination prediction unit 121 acquires a tap corresponding to the target pixel set in step S211. At this time, a tap is acquired based on the motion vector detected by the motion vector detection unit 126.

  In step S213, the discrimination prediction unit 121 specifies a discrimination coefficient corresponding to the repetition code specified in the process of step S211 and reads it from the discrimination coefficient storage unit 122.

  In step S214, the discrimination prediction unit 121 calculates a discrimination prediction value. At this time, for example, the above-described calculation of Expression (8) is performed.

In step S215, the class dividing unit 123 sets (applies) a class code bit to the pixel of interest based on the determined prediction value calculated in the process of step S214. At this time, as described above, for example, the predicted value y _i calculated by Expression (8) is compared with 0, and the class code bit is given to the target pixel.

  Note that the processing in steps S211 to S215 is performed for each pixel to be processed that is determined corresponding to the repetition code.

  In this way, the determination process is executed.

  Returning to FIG. 18, after the process of step S <b> 192, in step S <b> 193, the discrimination prediction unit 121 determines whether or not the iteration has ended. For example, when it is preset that learning is repeated three times, it is determined that the iteration has not been completed yet, and the process returns to step S191.

  Thereafter, in step S191, the repetitive code is specified as 21, and similarly, the process of step S192 is executed. At this time, as described above, in the process of step S192, each of the pixels of the input image to which the class code bit 1 is assigned in the first determination process is set as the target pixel.

  Then, in step S193, it is determined whether or not the iteration has been completed.

  As described above, the processes in steps S191 to S193 are repeatedly executed until it is determined in step S193 that the iteration has been completed. If it is preset that learning is repeated three times, it is determined in step S191 that the repetitive code is 34, and then the process of step S192 is executed. In step S193, it is determined that the iteration has ended. Will be.

  If it is determined in step S193 that the iteration has been completed, the process proceeds to step S194. By the processing so far, the input image is classified into the pixel group corresponding to the class number of the 3-bit class code as described above with reference to FIG. 11 or FIG. Further, as described above, the class dividing unit 123 associates the information specifying each pixel of the input image with the class number of the pixel and supplies the information to the regression coefficient storage unit 124.

  In step S194, the regression prediction unit 125 sets a target pixel in the input image.

  In step S195, the regression prediction unit 125 acquires a tap corresponding to the target pixel set in step S194. At this time, a tap is acquired based on the motion vector detected by the motion vector detection unit 126.

  In step S196, the regression prediction unit 125 supplies information specifying the target pixel set in step S194 to the regression coefficient storage unit 124, specifies the regression coefficient corresponding to the class number of the target pixel, and sets the regression coefficient. Read from the storage unit 124.

  In step S197, the regression prediction unit 125 calculates the regression prediction value by calculating Equation (6) using the tap acquired in step S195 and the regression coefficient specified and read in step S196.

  Note that the processing from step S191 to step S197 is performed for each pixel of the input image.

  Then, an output image is generated in which the predicted value obtained by the calculation of the regression prediction unit 125 is the value of the pixel corresponding to the target pixel. As a result, an output image in which the input image is improved in image quality is obtained.

  In this way, the discrimination prediction process is executed. In this way, the image quality enhancement process can be performed more efficiently and at high speed.

  Next, with reference to FIG. 20 to FIG. 22, the effect of the high image quality processing using the learning device 10 and the image processing device 100 of the present invention will be described.

  FIG. 20 shows an example of an image with motion blur. In the example in the figure, motion blur occurs in the image of the building (tower).

  FIG. 21 shows an image obtained as a result of performing a process of removing motion blur by a general inverse filter method on the image shown in FIG. In the image shown in FIG. 20, although the blur is removed to some extent as compared with the image of FIG. 20, the image is deteriorated due to ringing.

  FIG. 22 shows an image from which motion blur has been removed by performing the discrimination regression prediction process using the image shown in FIG. 20 as an input image and using the image processing apparatus 100 shown in FIG. In the image shown in the figure, the blur is removed as compared with the image of FIG. 20, and the image is not deteriorated due to ringing unlike the image shown in FIG.

  Thus, motion blur can be effectively removed by performing the image quality enhancement processing according to the present invention.

  The image processing apparatus described above with reference to FIG. 13 can be mounted on a television receiver as an image quality improving circuit, for example. FIG. 23 is a block diagram showing a configuration example of a television receiver 511 equipped with the image processing apparatus described above with reference to FIG.

  The television receiver 511 in the figure includes a controlled unit 531 and a control unit 532. The controlled unit 531 implements various functions of the television receiver 511 under the control of the control unit 532.

  The controlled unit 531 includes a digital tuner 553, a demultiplexer (Demux) 554, an MPEG (Moving Picture Expert Group) decoder 555, a video / graphic processing circuit 556, a panel driving circuit 557, a display panel 558, an audio processing circuit 559, and an audio amplification. A circuit 560, a speaker 561, and a receiving unit 562 are provided. The control unit 532 includes a CPU (Central Processing Unit) 563, a flash ROM 564, a DRAM (Dynamic Random Access Memory) 565, and an internal bus 566.

  The digital tuner 553 processes a television broadcast signal input from an antenna terminal (not shown) and supplies a predetermined TS (Transport Stream) corresponding to the channel selected by the user to the demultiplexer 554.

  The demultiplexer 554 extracts the partial TS (the TS packet of the video signal and the TS packet of the audio signal) corresponding to the channel selected by the user from the TS supplied from the digital tuner 553, and supplies it to the MPEG decoder 555.

  The demultiplexer 554 extracts PSI / SI (Program Specific Information / Service Information) from the TS supplied from the digital tuner 553 and supplies the PSI / SI to the CPU 563. A plurality of channels are multiplexed in the TS supplied from the digital tuner 553. The process in which the demultiplexer 554 extracts a partial TS of an arbitrary channel from the TS can be performed by obtaining packet ID (PID) information of an arbitrary channel from PSI / SI (PAT / PMT).

  The MPEG decoder 555 decodes a video PES (Packetized Elementary Stream) packet composed of TS packets of the video signal supplied from the demultiplexer 554 and converts the resulting video signal into a video / graphic processing circuit. 556. Also, the MPEG decoder 555 performs a decoding process on the audio PES packet configured by the TS packet of the audio signal supplied from the demultiplexer 554, and supplies the audio signal obtained as a result to the audio processing circuit 559.

  The video / graphic processing circuit 556 performs scaling processing, graphics data superimposition processing, and the like on the video signal supplied from the MPEG decoder 555 as necessary, and supplies the result to the panel drive circuit 557.

  An image quality improving circuit 570 is connected to the video / graphic processing circuit 556, and an image quality improving process is executed prior to supplying a video signal to the panel driving circuit 557.

  The image quality improving circuit 570 has the same configuration as that of the image processing apparatus described above with reference to FIG. 13, and with respect to image data obtained from the video signal supplied from the MPEG decoder 555, with reference to FIG. The discriminant regression prediction process described above is executed as an image quality enhancement process.

  The panel drive circuit 557 drives the display panel 558 based on the video signal supplied from the video / graphic processing circuit 556 to display the video. The display panel 558 is configured by, for example, an LCD (Liquid Crystal Display) or a PDP (Plasma Display Panel).

  The audio processing circuit 559 performs necessary processing such as D / A (Digital to Analog) conversion on the audio signal supplied from the MPEG decoder 555 and supplies the audio signal to the audio amplification circuit 560.

  The audio amplifier circuit 560 amplifies the analog audio signal supplied from the audio processing circuit 559 and supplies the amplified analog audio signal to the speaker 561. The speaker 561 outputs sound corresponding to the analog sound signal from the sound amplifier circuit 560.

  The receiving unit 562 receives, for example, an infrared remote control signal transmitted from the remote controller 567 and supplies it to the CPU 563. The user can operate the television receiver 511 by operating the remote controller 567.

  The CPU 563, the flash ROM 564, and the DRAM 565 are connected via an internal bus 566. The CPU 563 controls the operation of each unit of the television receiver 11. The flash ROM 564 stores control software and data. The DRAM 565 constitutes a work area of the CPU 563 and the like. That is, the CPU 563 develops software and data read from the flash ROM 564 on the DRAM 565, starts up the software, and controls each unit of the television receiver 511.

  As described above, the present invention can be applied to a television receiver.

  The series of processes described above can be executed by hardware, or can be executed by software. When the above-described series of processing is executed by software, a program constituting the software is installed from a network or a recording medium into a computer incorporated in dedicated hardware. In addition, by installing various programs, it is installed from a network or a recording medium in a general-purpose personal computer 700 as shown in FIG. 24 that can execute various functions.

  In FIG. 24, a CPU (Central Processing Unit) 701 executes various processes according to a program stored in a ROM (Read Only Memory) 702 or a program loaded from a storage unit 708 to a RAM (Random Access Memory) 703. To do. The RAM 703 also appropriately stores data necessary for the CPU 701 to execute various processes.

  The CPU 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output interface 705 is also connected to the bus 704.

  The input / output interface 705 is connected to an input unit 706 composed of a keyboard, a mouse, etc., a display composed of an LCD (Liquid Crystal display), etc., and an output unit 707 composed of a speaker. The input / output interface 705 is connected to a storage unit 708 composed of a hard disk and a communication unit 709 composed of a network interface card such as a modem and a LAN card. The communication unit 709 performs communication processing via a network including the Internet.

  A drive 710 is also connected to the input / output interface 705 as necessary, and a removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted. Then, the computer program read from these removable media is installed in the storage unit 708 as necessary.

  When the above-described series of processing is executed by software, a program constituting the software is installed from a network such as the Internet or a recording medium such as a removable medium 711.

  Note that this recording medium is a magnetic disk (including a floppy disk (registered trademark)) on which a program is recorded, which is distributed to distribute the program to the user separately from the apparatus main body shown in FIG. Removable media consisting of optical disks (including CD-ROM (compact disk-read only memory), DVD (digital versatile disk)), magneto-optical disks (including MD (mini-disk) (registered trademark)), or semiconductor memory It includes not only those configured by 711 but also those configured by a ROM 702 in which a program is recorded, a hard disk included in the storage unit 708, and the like distributed to the user in a state of being incorporated in the apparatus main body in advance.

  Note that the series of processes described above in this specification includes processes that are performed in parallel or individually even if they are not necessarily processed in time series, as well as processes that are performed in time series in the order described. Is also included.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

  DESCRIPTION OF SYMBOLS 10 Learning apparatus, 21 Regression coefficient learning part, 22 Regression coefficient memory | storage part, 23 Regression prediction part, 24 Labeling part, 25 Discriminant coefficient learning part, 26 Discriminant coefficient memory | storage part, 27 Discriminant prediction part, 28 Class division part, 29 Motion vector Detection unit, 100 image processing device, 121 discrimination prediction unit, 122 discrimination coefficient storage unit, 123 class division unit, 124 regression coefficient storage unit, 125 regression prediction unit, 126 motion vector detection unit, 511 television receiver, 570 high image quality Circuit, 701 CPU, 702 ROM, 711 removable media

Claims (15)

From the image of the first signal, a feature amount specified based on a motion vector, which is a tap configured as a plurality of feature amounts obtained from pixel values of a target pixel and surrounding pixels, is acquired, and the tap Regression coefficient calculating means for calculating the regression coefficient of the regression prediction calculation for obtaining the value of the pixel corresponding to the pixel of interest in the image of the second signal by the product-sum operation of each of the elements and the regression coefficient;
A regression prediction value calculating means for calculating a regression prediction value by performing the regression prediction calculation based on the calculated regression coefficient and the tap obtained from the image of the first signal;
Based on a comparison result between the calculated regression prediction value and a pixel value corresponding to the target pixel in the image of the second signal, the target pixel is a pixel belonging to a first discrimination class, or Discrimination information giving means for giving discrimination information for discriminating whether the pixel belongs to the second discrimination class;
Based on the given discrimination information, to acquire the tap from the first signal image, and to identify the discrimination class to which the pixel of interest belongs by multiply-accumulate each of the elements of the tap and the discrimination coefficient Discriminant coefficient calculating means for calculating the discriminant coefficient of the discriminant prediction calculation for obtaining the discriminant predicted value of
A discrimination prediction value calculation means for calculating a discrimination prediction value by performing the discrimination prediction calculation based on the calculated discrimination coefficient and the tap obtained from the image of the first signal;
Classification means for classifying each pixel of the image of the first signal into either the first discrimination class or the second discrimination class based on the calculated discrimination prediction value;
The regression coefficient calculating means further calculates the regression coefficient using only the pixels classified into the first discrimination class, and further calculates the regression coefficient using only the pixels classified into the second discrimination class. Coefficient learning device to calculate. Based on the regression prediction value calculated by the regression prediction value calculation unit for each of the discrimination classes by the regression coefficient calculated by the regression coefficient calculation unit for each of the discrimination classes, the discrimination information providing unit provides discrimination information. The coefficient learning apparatus according to claim 1, wherein the processing, the determination coefficient calculation unit repeatedly calculates the determination coefficient, and the determination prediction value calculation unit repeatedly calculates the determination prediction value. If the difference between the regression prediction value and the value of the pixel corresponding to the target pixel in the image of the second signal is 0 or more, the target pixel is determined to be a pixel belonging to the first determination class;
When the difference between the regression prediction value and the value of the pixel corresponding to the target pixel in the image of the second signal is less than 0, the target pixel is determined to be a pixel belonging to the first determination class. The coefficient learning apparatus according to claim 1. When the regression prediction value, the value of the pixel corresponding to the target pixel in the image of the second signal, and the absolute difference value are greater than or equal to a preset threshold value, the target pixel is a pixel belonging to the first discrimination class Is determined to be
When the regression prediction value, the value of the pixel corresponding to the target pixel in the image of the second signal, and the absolute difference value are less than the threshold value, the target pixel is a pixel belonging to the second discrimination class. The coefficient learning device according to claim 1, wherein the coefficient learning device is discriminated. The coefficient learning apparatus according to claim 1, wherein the image of the first signal is an image obtained by adding motion blur to the image of the second signal. The tap is based on the motion direction and the amount of motion specified by the motion vector, the pixel value extracted according to the motion direction centered on the pixel of interest, the maximum and minimum values of the extracted pixel value, and the extracted pixel value The coefficient learning device according to claim 1, wherein each of the absolute values of the differential feature amounts according to the movement direction and the maximum absolute value of the differential feature amounts according to the movement direction of the extracted pixel value are configured as elements. The regression coefficient calculation means obtains taps configured as a plurality of feature amounts obtained from the pixel values of the target pixel and the surrounding pixels, which are feature amounts specified based on the motion vector, from the first signal image. Then, the regression coefficient of the regression prediction calculation for obtaining the value of the pixel corresponding to the target pixel in the image of the second signal by the product-sum operation of each of the elements of the tap and the regression coefficient,
A regression prediction value calculation means calculates the regression prediction value by performing the regression prediction calculation based on the calculated regression coefficient and the tap obtained from the image of the first signal,
The discriminating information providing means assigns the target pixel to the first discriminating class based on a comparison result between the calculated regression prediction value and the value of the pixel corresponding to the target pixel in the second signal image. Providing discrimination information for discriminating whether the pixel belongs to or belongs to the second discrimination class;
Discrimination coefficient calculation means acquires the tap from the image of the first signal based on the given discrimination information, and the pixel of interest belongs by a product-sum operation of each element of the tap and the discrimination coefficient Calculating the discriminant coefficient of the discriminant prediction calculation for obtaining a discriminant prediction value for specifying the discriminant class;
A discriminant prediction value calculating unit calculates the discriminant prediction value by performing the discriminant prediction calculation based on the calculated discriminant coefficient and the tap obtained from the image of the first signal,
Classification means classifies each pixel of the image of the first signal into either the first discrimination class or the second discrimination class based on the calculated discrimination prediction value,
A coefficient that further includes calculating the regression coefficient using only the pixels classified into the first discrimination class, and further calculating the regression coefficient using only the pixels classified into the second discrimination class. Learning method. Computer
From the image of the first signal, a feature amount specified based on a motion vector, which is a tap configured as a plurality of feature amounts obtained from pixel values of a target pixel and surrounding pixels, is acquired, and the tap Regression coefficient calculating means for calculating the regression coefficient of the regression prediction calculation for obtaining the value of the pixel corresponding to the pixel of interest in the image of the second signal by the product-sum operation of each of the elements and the regression coefficient;
A regression prediction value calculating means for calculating a regression prediction value by performing the regression prediction calculation based on the calculated regression coefficient and the tap obtained from the image of the first signal;
Based on a comparison result between the calculated regression prediction value and a pixel value corresponding to the target pixel in the image of the second signal, the target pixel is a pixel belonging to a first discrimination class, or Discrimination information giving means for giving discrimination information for discriminating whether the pixel belongs to the second discrimination class;
Based on the given discrimination information, to acquire the tap from the first signal image, and to identify the discrimination class to which the pixel of interest belongs by multiply-accumulate each of the elements of the tap and the discrimination coefficient Discriminant coefficient calculating means for calculating the discriminant coefficient of the discriminant prediction calculation for obtaining the discriminant predicted value of
A discrimination prediction value calculation means for calculating a discrimination prediction value by performing the discrimination prediction calculation based on the calculated discrimination coefficient and the tap obtained from the image of the first signal;
Classification means for classifying each pixel of the image of the first signal into either the first discrimination class or the second discrimination class based on the calculated discrimination prediction value;
The regression coefficient calculating means further calculates the regression coefficient using only the pixels classified into the first discrimination class, and further calculates the regression coefficient using only the pixels classified into the second discrimination class. A program that functions as a coefficient learning device to calculate. From the image of the first signal, a feature amount specified based on a motion vector, which is a tap configured as a plurality of feature amounts obtained from pixel values of a target pixel and surrounding pixels, is acquired, and the tap Discriminant prediction means for performing a discriminant prediction operation for obtaining a discriminant prediction value for specifying a class to which the pixel of interest belongs by multiply-accumulate each element and a discrimination coefficient
Classification means for classifying each pixel of the image of the first signal into either the first discrimination class or the second discrimination class based on the discrimination prediction value;
A pixel corresponding to the target pixel in the second signal image by acquiring the tap from the first signal image and calculating a regression prediction value by a product-sum operation of the tap and the regression coefficient. An image processing apparatus comprising regression prediction means for predicting the pixel value of the image. The image processing apparatus according to claim 9, wherein the discrimination prediction unit repeatedly executes a process of performing the discrimination prediction calculation, and the classification unit repeatedly performs a process of classifying each pixel of the image of the first signal. The image processing apparatus according to claim 9, wherein the image of the first signal is an image obtained by adding motion blur to the image of the second signal. The tap is based on the motion direction and the amount of motion specified by the motion vector, the pixel value extracted according to the motion direction centered on the pixel of interest, the maximum and minimum values of the extracted pixel value, and the extracted pixel value The image processing apparatus according to claim 9, wherein each of the absolute value of the differential feature quantity according to the movement direction and the maximum value of the absolute value of the differential feature quantity according to the movement direction of the extracted pixel value are configured as elements. The discriminating / predicting means acquires, from the first signal image, taps configured as a plurality of feature amounts that are specified based on the motion vector and obtained from the pixel values of the target pixel and the surrounding pixels. Performing a discrimination prediction calculation to obtain a discrimination prediction value for identifying the class to which the pixel of interest belongs by multiply-accumulate each of the elements of the tap and the discrimination coefficient,
Classification means classifies each pixel of the image of the first signal into either the first discrimination class or the second discrimination class based on the discrimination prediction value,
The regression prediction means acquires the tap from the image of the first signal, and calculates a regression prediction value by a product-sum operation of the tap and the regression coefficient, whereby the attention in the image of the second signal is obtained. An image processing method comprising: predicting a pixel value of a pixel corresponding to a pixel. Computer
From the image of the first signal, a feature amount specified based on a motion vector, which is a tap configured as a plurality of feature amounts obtained from pixel values of a target pixel and surrounding pixels, is acquired, and the tap Discriminant prediction means for performing a discriminant prediction operation for obtaining a discriminant prediction value for specifying a class to which the pixel of interest belongs by multiply-accumulate each element and a discrimination coefficient
Classification means for classifying each pixel of the image of the first signal into either the first discrimination class or the second discrimination class based on the discrimination prediction value;
A pixel corresponding to the target pixel in the second signal image by acquiring the tap from the first signal image and calculating a regression prediction value by a product-sum operation of the tap and the regression coefficient. A program that functions as an image processing apparatus that includes regression prediction means for predicting the pixel value of an image.   A recording medium on which the program according to claim 8 or 14 is recorded.

Images