JP2008124955A - Image processor and image processing method, learning device and learning method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain image data of high quality by effectively removing noise. <P>SOLUTION: A prediction tap extracting unit 123 extracts a pixel for predicting the high quality image data from output image data as a prediction tap, and a class tap extracting unit 124 extracts a pixel for classifying a target pixel into classes as a class tap. A class classifying unit 125 classifies the target pixel into classes on the basis of the class tap, and a coefficient storage unit 126 outputs a coefficient corresponding to the class of the target pixel. A predictive operation unit 127 finds the high quality image data by using the coefficient from the coefficient storage unit 126 and the prediction tap from the prediction tap extracting unit 123. A control unit 128 controls processing blocks of the output image data so that image conversion processing for finding the high quality image data is applied to each processing blocks. The present invention is applicable to, for example, a television receiver. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing device, an image processing method, a learning device, a learning method, and a program, and more particularly, for example, an image processing device and an image processing method that allow a user to enjoy high-quality image data. The present invention relates to a learning apparatus, a learning method, and a program.

For example, a decoded image obtained by encoding and decoding a predetermined image for each predetermined block by an encoding and decoding method such as MPEG-2 (Moving Picture Experts Group phase 2) is block distortion or mosquito noise. It becomes a noise image including noise such as. As a noise removal method for removing noise from such a noise image, a technique for performing noise removal processing using an 8 × 8 pixel (horizontal × vertical) block, which is a processing unit for encoding and decoding, as a processing unit for noise removal, It is disclosed in Patent Documents 1 and 2.
JP 2003-324738 A JP 2004-7607 A

  By the way, in terrestrial digital broadcasting, as shown in FIG. 1, on the broadcast station 31 side, the original image of 1920 × 1080 pixels (horizontal × vertical) to be broadcast is reduced by 3/4 times in the horizontal direction. Then, the reduced image of 1440 × 1080 pixels obtained as a result is encoded by an encoding method such as MPEG-2 for each 8 × 8 pixel block, and transmitted to each home as an MPEG encoded image. .

  On the other hand, each home side receives the MPEG encoded image transmitted from the broadcasting station 31, and the decoder 32 installed in each home uses a code such as MPEG-2 for each 8 × 8 pixel block. By performing decoding corresponding to the encoding method, the MPEG encoded image is decoded into an MPEG decoded image of 1440 × 1080 pixels.

  In addition, the decoder 32 enlarges the 1440 × 1080 pixel MPEG decoded image by 4/3 times in the horizontal direction to expand the 1920 × 1080 pixel enlarged MPEG decoded image having the same number of pixels as the original image, The enlarged MPEG decoded image is displayed on a television receiver (not shown).

  In the enlarged MPEG decoded image, a block of 8 × 8 pixels in horizontal × vertical which is a processing unit of encoding and decoding by MPEG-2 or the like is multiplied by 4/3 in the horizontal direction to be 10.666 (to be exact, 10.666 ...) × 8 pixels (horizontal × vertical).

  Therefore, the expanded MPEG decoded image has mosquito noise and block distortion at the boundary (near the block) of 10.666 × 8 pixels, but the noise removal method for removing the noise generated in the expanded MPEG decoded image For example, when noise removal processing using the techniques disclosed in Patent Documents 1 and 2 described above is employed, noise may not be effectively removed.

  That is, in the techniques disclosed in Patent Documents 1 and 2, for example, noise removal processing is performed for each 8 × 8 pixel block that matches the MPEG-2 encoding and decoding processing units. Noise removal processing is not performed for each block of 10.666 × 8 pixels.

  Then, an 8 × 8 pixel block as a processing unit of noise removal processing using the techniques disclosed in Patent Documents 1 and 2 and 10.666 × 8 pixel as a unit in which noise is generated on an enlarged MPEG decoded image If the block does not match, or if there is a phase shift, it is more effective than the case where the noise removal processing unit and the unit that generates noise match, that is, if there is no phase shift. In some cases, noise cannot be removed.

  The present invention has been made in view of such a situation, and makes it possible to effectively remove noise and obtain high-quality image data.

  An image processing apparatus according to a first aspect of the present invention provides encoded data obtained by encoding converted image data obtained by converting the number of pixels of original image data by a predetermined number of times for each predetermined block. Image data obtained by decoding the encoded data, and outputting the number of pixels equal to the number of pixels of the original image data by multiplying the number of pixels by a reciprocal of the predetermined number. In the image processing apparatus that processes the output image data output by the decoder that converts to image data, the image quality that is higher than the output image data corresponding to the target pixel that is the target pixel of the output image data A prediction tap extracting means for extracting a plurality of pixels used for predicting a pixel of image data as a prediction tap from the output image data; Based on the class tap extracting means for extracting a plurality of pixels used for class classification as a class tap from the output image data, and the class tap extracted by the class tap extracting means. Predicting teacher image data corresponding to the high-quality image data obtained by a prediction operation using class classification means for classifying the pixel of interest and student image data corresponding to the output image data and a predetermined coefficient A coefficient output for outputting a coefficient corresponding to the class of the target pixel from the coefficients corresponding to each of the plurality of classes, which is obtained by learning that minimizes an error between the predicted value and the teacher image data Means, the coefficient output by the coefficient output means, and the prediction tap extraction means An image for converting the output image data into the high-quality image data, the prediction calculation means for obtaining a pixel of the high-quality image data corresponding to the target pixel by the prediction calculation using the prediction tap; The image conversion means for performing conversion processing and the image conversion means are controlled so that the image conversion processing is performed for each processing block of the output image data having a block size determined based on the predetermined number of times. Control means.

  In the image processing apparatus according to the first aspect, a pixel on the boundary, which is a pixel on the boundary of an inverse block obtained by multiplying the block size of the predetermined block by an inverse multiple of the predetermined number, is included in the output image data. Pixel determining means for determining whether or not it is included can be further provided. In the image converting means, when it is determined by the pixel determining means that the output image data includes the pixel on the boundary, The output image data can be converted into the high-quality image data.

  The prediction tap extraction unit can extract the prediction tap from the processing block including the target pixel.

  The class tap extraction unit can extract the class tap from the processing block including the target pixel.

  The image processing apparatus according to the first aspect may further include threshold value calculation means for calculating a threshold value used for the class classification of the target pixel using a pixel of the output image data, and the class classification means includes Based on the class tap extracted by the class tap extraction unit and the threshold value calculated by the threshold value calculation unit, the pixel of interest can be classified.

  The threshold value calculation means can calculate the threshold value using a pixel of the processing block including the target pixel.

  In the image processing apparatus according to the first aspect, a pixel on the boundary, which is a pixel of the output image data, is located on a boundary of a reciprocal multiple block in which the block size of the predetermined block is a reciprocal multiple of the predetermined number. , A boundary pixel determination unit that determines whether or not the pixel is included in the processing block may be further provided. The threshold calculation unit includes the boundary pixel determination unit that includes the boundary pixel determination unit. When it is determined that the threshold value is included, the threshold value can be calculated using a pixel obtained by excluding the pixel on the boundary from the pixel of the processing block.

  The image processing apparatus according to the first aspect may further include pixel position mode determination means for determining the pixel position mode of the pixel of interest using information representing the position of each pixel in the processing block as a pixel position mode. The coefficient output means can output a coefficient corresponding to the class of pixel of interest and the pixel position mode.

  In the pixels of the processing block, the same value can be assigned as the pixel position mode to pixels that are in a line-symmetrical or point-symmetrical positional relationship.

  In the image processing device of the first aspect, an area determination that determines whether the target pixel is located in any of the plurality of areas when the processing block is divided into the plurality of areas. Means can be further provided, and the prediction calculation means can obtain the pixel of the high-quality image data corresponding to the target pixel by the prediction calculation corresponding to the region where the target pixel is located. .

  If the processing block includes pixels on the boundary, which are pixels of the output image data, on the boundary of the inverse block obtained by making the block size of the predetermined block an inverse multiple of the predetermined number, the processing block In the pixels excluding the pixels on the boundary from the block, the same value as the pixel position mode can be assigned to the pixels having a line-symmetrical or point-symmetrical positional relationship.

  A value different from pixels other than the pixel on the boundary can be assigned to the pixel on the boundary as a pixel position mode.

  The image processing method or program according to the first aspect of the present invention provides encoded data obtained by encoding converted image data obtained by converting the number of pixels of original image data by a predetermined number of times for each predetermined block, The image data obtained by decoding every predetermined block and decoding the encoded data is obtained by multiplying the number of pixels by a reciprocal of the predetermined number to obtain the same number of pixels as the original image data. An image processing method for processing the output image data output by the decoder for converting the output image data into a number of output image data, or encoding the converted image data obtained by converting the number of pixels of the original image data by a predetermined number of times for each predetermined block The encoded data obtained is decoded for each of the predetermined blocks, and the image data obtained by decoding the encoded data is converted to the inverse of the predetermined number of pixels. In the program for causing the computer to execute image processing for processing the output image data output by the decoder that converts the output image data to the same number of pixels as the number of pixels of the original image data by doubling the output image data, A plurality of pixels used for predicting pixels of high-quality image data having higher image quality than the output image data corresponding to the target pixel that is a pixel of interest are extracted as prediction taps from the output image data A prediction tap extraction step, a class tap extraction step of extracting a plurality of pixels used for class classification that classifies the pixel of interest into any one of a plurality of classes as a class tap from the output image data, and Based on the class tap extracted by the class tap extraction step, the attention image is displayed. A class classification step for classifying the image data, a predicted value obtained by predicting the teacher image data corresponding to the high-quality image data, which is obtained by a prediction calculation using student image data corresponding to the output image data and a predetermined coefficient; A coefficient output step of outputting a coefficient corresponding to the class of the target pixel from the coefficients corresponding to each of the plurality of classes, obtained by learning that minimizes an error from the teacher image data; Prediction calculation for obtaining a pixel of the high-quality image data corresponding to the target pixel by the prediction calculation using the coefficient output by the coefficient output step and the prediction tap extracted by the prediction tap extraction step An image conversion step for performing image conversion processing for converting the output image data into the high-quality image data. And a control step for controlling the image conversion step so that the image conversion processing is performed for each processing block of the output image data having a block size determined based on the predetermined number of times. .

  According to the first aspect of the present invention, the encoded data obtained by encoding the converted image data obtained by converting the number of pixels of the original image data by a predetermined number of times for each predetermined block is obtained for each predetermined block. The image data obtained by decoding and decoding the encoded data is output image data having the same number of pixels as the number of pixels of the original image data by multiplying the number of pixels by a reciprocal of the predetermined number. A plurality of pixels used for predicting pixels of high-quality image data with higher image quality than the output image data, corresponding to the target pixel that is a target pixel of the output image data output by the decoder that converts to , And extracted as a prediction tap from the output image data. Further, a plurality of pixels used for class classification for classifying the target pixel into any one of a plurality of classes are extracted as class taps from the output image data, and based on the extracted class taps The predicted value obtained by predicting the teacher image data corresponding to the high-quality image data obtained by the prediction calculation using the student image data corresponding to the output image data and a predetermined coefficient, wherein the target pixel is classified Then, a coefficient corresponding to the class of the target pixel is output from the coefficients corresponding to each of the plurality of classes obtained by learning that minimizes an error from the teacher image data. Furthermore, a pixel of the high-quality image data corresponding to the target pixel is obtained by the prediction calculation using the output coefficient and the extracted prediction tap. In addition, for each processing block of the output image data having a block size determined based on the predetermined number of times, various types of image conversion processing are performed to convert the output image data into the high-quality image data. Be controlled.

  The learning device according to the second aspect of the present invention provides encoded data obtained by encoding the converted image data obtained by converting the number of pixels of the original image data by a predetermined number of times for each predetermined block. And outputting the number of pixels equal to the number of pixels of the original image data obtained by multiplying the number of pixels of the image data obtained by decoding the encoded data by a reciprocal multiple of the predetermined number The coefficient used for the prediction calculation for converting the image data into the high-quality image data having higher image quality than the output image data is the result of the prediction calculation using the student image data which is an image corresponding to the output image data. In the learning device that is obtained by learning that minimizes an error from the teacher image data that is an image corresponding to the high-quality image data, the attention is a pixel of interest of the student image data A prediction tap extracting means for extracting, as prediction taps, from the student image data, a plurality of pixels used in the prediction calculation for converting a prime into a pixel of the teacher image data corresponding to the pixel of interest; and a plurality of the pixels of interest Class tap extracting means for extracting a plurality of pixels used for class classification as class taps from the student image data, and the class tap extracted by the class tap extracting means. Classifying means for classifying the pixel of interest on the basis of, a result of the prediction calculation using the prediction tap for each class of the pixel of interest classified by the class classification unit, and the pixel of interest Corresponding coefficient calculation means for calculating the coefficient that minimizes an error from the pixel of the teacher image data. The learning process is performed for each processing block of the student image data having a block size determined based on the learning unit for performing the learning process for obtaining the coefficient for each class and the predetermined multiple. And control means for controlling the learning means.

  The prediction tap extraction unit can extract the prediction tap from the processing block including the target pixel.

  The class tap extraction unit can extract the class tap from the processing block including the target pixel.

  The learning device according to the second aspect may further include a threshold value calculation unit that calculates a threshold value used for the class classification of the pixel of interest using a pixel of the student image data, and the class classification unit includes: Based on the class tap extracted by the class tap extraction unit and the threshold value calculated by the threshold value calculation unit, the target pixel can be classified.

  The threshold value calculation means can calculate the threshold value using a pixel of the processing block including the target pixel.

  In the learning device according to the second aspect, pixels on the boundary, which are pixels of the student image data, on the boundary of the reciprocal multiple block in which the block size of the predetermined block is the reciprocal multiple of the predetermined number, A boundary pixel determination unit that determines whether or not the image is included in the processing block may be further provided, and the threshold calculation unit includes the pixel on the pixel in the processing block by the pixel determination unit on the boundary. If it is determined that, the threshold value can be calculated using a pixel obtained by excluding the pixel on the boundary from the pixel of the processing block.

  The learning device according to the second aspect may further include pixel position mode determination means for determining the pixel position mode of the pixel of interest using information indicating the position of each pixel in the processing block as a pixel position mode, The coefficient calculation means calculates the coefficient that minimizes an error between the prediction calculation result using the prediction tap and the pixel of the teacher image data for each class of pixel of interest and pixel position mode. Can do.

  In the learning device of the second aspect, the same value as the pixel position mode can be assigned to the pixels of the processing block that are in a line symmetric or point symmetric positional relationship.

  When the processing block includes pixels on the boundary, which are pixels of the student image data, on the boundary of the reciprocal multiple block obtained by making the block size of the predetermined block the reciprocal multiple of the predetermined number, the processing In the pixels excluding the pixels on the boundary from the block, the same value as the pixel position mode can be assigned to the pixels having a line-symmetrical or point-symmetrical positional relationship.

  A value different from pixels other than the pixel on the boundary can be assigned to the pixel on the boundary as a pixel position mode.

  The learning method or program according to the second aspect of the present invention provides encoded data obtained by encoding converted image data obtained by converting the number of pixels of original image data by a predetermined number of times for each predetermined block. The same number of pixels as the number of pixels of the original image data obtained by decoding each block of the image data and by multiplying the number of pixels of the image data obtained by decoding the encoded data by a reciprocal multiple of the predetermined number The prediction calculation using the student image data, which is an image corresponding to the output image data, with a coefficient used for the prediction calculation for converting the number of output image data into high-quality image data with higher image quality than the output image data And a learning method that is obtained by learning that minimizes an error between the result and the teacher image data that is an image corresponding to the high-quality image data, or the number of pixels of the original image data is multiplied by a predetermined number The encoded data obtained by encoding the converted image data after conversion for each predetermined block is decoded for each predetermined block, and the number of pixels of the image data obtained by decoding the encoded data is set to the predetermined data. For predictive computation that converts output image data having the same number of pixels as the original image data obtained by multiplying the reciprocal of the number into high-quality image data with higher image quality than the output image data. Learning that minimizes an error between the prediction coefficient using the student image data that is an image corresponding to the output image data and the teacher image data that is an image corresponding to the high-quality image data. In the program for causing the computer to execute the processing obtained by the above, a target pixel that is a target pixel of the student image data corresponds to the target pixel. A prediction tap extracting step of extracting a plurality of pixels used for the prediction calculation to be converted into pixels of the teacher image data as prediction taps from the student image data; and the target pixel is set to any one of a plurality of classes. A class tap extracting step of extracting a plurality of pixels used for class classification as class taps from the student image data, and classifying the pixel of interest based on the class tap extracted by the class tap extracting step A class classification step, a prediction result using the prediction tap for each class of the target pixel classified by the class classification step, a pixel of the teacher image data corresponding to the target pixel, and A coefficient calculation step for calculating the coefficient that minimizes the error of A learning step for performing a learning process for obtaining the coefficient for each class, and the learning process is performed for each processing block of the student image data having a block size determined based on the predetermined number of times. And a control step for controlling the learning step.

  According to the second aspect of the present invention, the encoded data obtained by encoding the converted image data obtained by converting the number of pixels of the original image data by a predetermined number of times for each predetermined block is obtained for each predetermined block. An output image having the same number of pixels as the number of pixels of the original image data obtained by decoding and multiplying the number of pixels of the image data obtained by decoding the encoded data by a reciprocal of the predetermined number The coefficient used for the prediction calculation for converting the data into the high-quality image data having higher image quality than the output image data is the result of the prediction calculation using the student image data that is an image corresponding to the output image data; It is obtained by learning that minimizes an error from teacher image data that is an image corresponding to the high-quality image data. In addition, a plurality of pixels used for the prediction calculation for converting a target pixel that is a target pixel of the student image data into a pixel of the teacher image data corresponding to the target pixel is predicted from the student image data. A plurality of pixels that are extracted as taps and used for class classification that classifies the pixel of interest into any one of a plurality of classes are extracted as class taps from the student image data. Further, the pixel of interest is classified based on the extracted class tap, and the result of the prediction calculation using the prediction tap for each class of the pixel of interest classified, and the pixel of interest The coefficient that minimizes the error with the pixel of the teacher image data corresponding to is calculated. Further, various control is performed so that the learning process for obtaining the coefficient for each class is performed for each processing block of the student image data having a block size determined based on the predetermined number of times.

  As described above, according to the first aspect of the present invention, for example, noise can be effectively removed and high-quality image data can be obtained.

  Further, according to the second aspect of the present invention, for example, a coefficient for generating high-quality image data can be obtained.

  Embodiments of the present invention will be described below. Correspondences between the constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

The image processing apparatus according to the first aspect of the present invention provides:
Encoded data (for example, encoded image data (for example, the reduced image in FIG. 1) obtained by encoding the converted image data (for example, the reduced image in FIG. 1) obtained by converting the number of pixels of the original image data (for example, the original image in FIG. 1) by a predetermined number times. For example, the MPEG encoded image of FIG. 1) is decoded for each of the predetermined blocks,
The image data obtained by decoding the encoded data (for example, the MPEG decoded image in FIG. 1) is the same as the number of pixels of the original image data by making the number of pixels the inverse of the predetermined number. The image processing apparatus (for example, FIG. 2) that processes the output image data output by the decoder (for example, the decoder 32 of FIG. 1) that converts the output image data to the number of pixels of the output image data (for example, the enlarged MPEG decoded image of FIG. 1). In the image conversion unit 61),
A plurality of pixels used for predicting pixels of high-quality image data with higher image quality than the output image data corresponding to the target pixel that is a pixel of interest of the output image data are predicted from the output image data Prediction tap extraction means (for example, prediction tap extraction unit 123 in FIG. 2) that extracts as taps;
Class tap extraction means (for example, class tap extraction in FIG. 2) that extracts a plurality of pixels used for class classification that classifies the target pixel into one of a plurality of classes from the output image data. Part 124),
Class classification means (for example, the class classification unit 125 in FIG. 2) for classifying the pixel of interest based on the class tap extracted by the class tap extraction means;
An error between the predicted value obtained by predicting the teacher image data corresponding to the high-quality image data and the teacher image data obtained by a prediction calculation using the student image data corresponding to the output image data and a predetermined coefficient. Coefficient output means (for example, coefficient storage unit 126 in FIG. 2) for outputting a coefficient corresponding to the class of the target pixel from the coefficients corresponding to each of the plurality of classes obtained by learning to be minimized; ,
Prediction for obtaining a pixel of the high-quality image data corresponding to the pixel of interest by the prediction calculation using the coefficient output by the coefficient output unit and the prediction tap extracted by the prediction tap extraction unit And an image conversion means (for example, the image conversion unit 61 in FIG. 2) that performs an image conversion process for converting the output image data into the high-quality image data. When,
Control means (for example, the control shown in FIG. 2) that controls the image conversion means so that the image conversion processing is performed for each processing block of the output image data having a block size determined based on the predetermined multiple. 128).

Pixel determination means for determining whether or not a pixel on the boundary, which is a pixel on the boundary of the reciprocal block obtained by reversing the block size of the predetermined block, is included in the output image data (For example, a block size determination unit 402 in FIG. 17),
The image conversion unit converts the output image data into the high-quality image data when the pixel determination unit determines that the output image data includes the pixel on the boundary.

A threshold value calculation means (for example, a threshold value calculation unit 122 in FIG. 2) that calculates a threshold value used for the class classification of the target pixel by using a pixel of the output image data;
The class classification unit classifies the pixel of interest based on the class tap extracted by the class tap extraction unit and the threshold value calculated by the threshold value calculation unit.

It is determined whether or not a pixel on the boundary, which is a pixel of the output image data, on the boundary of the reciprocal multiple block obtained by making the block size of the predetermined block the reciprocal multiple of the predetermined number is included in the processing block. It further comprises a boundary pixel determination means for determining (for example, the threshold value calculation unit 122 in FIG. 2 that executes the process of step S33 in FIG. 7),
The threshold calculation means, when the boundary pixel determination means determines that the boundary pixel is included in the processing block, the threshold value is a pixel obtained by excluding the boundary pixel from the pixel of the processing block. Calculate using.

It further includes pixel position mode determination means (for example, a pixel position mode determination unit 275 in FIG. 10) that determines the pixel position mode of the pixel of interest using information representing the position of each pixel in the processing block as a pixel position mode.
The coefficient output means outputs a coefficient corresponding to the class of pixel of interest and the pixel position mode.

Region determination means for determining whether the target pixel is located in any of the plurality of regions when the processing block is divided into a plurality of regions (for example, the region determination unit 277 in FIG. 10). )
The prediction calculation means obtains a pixel of the high-quality image data corresponding to the target pixel by the prediction calculation corresponding to a region where the target pixel is located.

An image processing method or program according to the first aspect of the present invention includes:
Encoded data (for example, encoded image data (for example, the reduced image in FIG. 1) obtained by encoding the converted image data (for example, the reduced image in FIG. 1) obtained by converting the number of pixels of the original image data (for example, the original image in FIG. 1) by a predetermined number times. For example, the MPEG encoded image of FIG. 1) is decoded for each of the predetermined blocks,
The image data obtained by decoding the encoded data (for example, the MPEG decoded image in FIG. 1) is the same as the number of pixels of the original image data by making the number of pixels the inverse of the predetermined number. An image processing method for processing the output image data output by a decoder (for example, the decoder 32 of FIG. 2), or an original image data The encoded data obtained by encoding the converted image data obtained by converting the number of pixels by a predetermined number of times for each predetermined block is decoded for each predetermined block,
Image data obtained by decoding the encoded data is converted to output image data having the same number of pixels as the number of pixels of the original image data by multiplying the number of pixels by a reciprocal of the predetermined number. In a program for causing a computer to execute image processing for processing the output image data output by a decoder,
A plurality of pixels used for predicting pixels of high-quality image data with higher image quality than the output image data corresponding to the target pixel that is a pixel of interest of the output image data are predicted from the output image data A prediction tap extraction step (for example, step S37 in FIG. 7) for extracting as a tap;
A class tap extraction step (eg, step S38 in FIG. 7) that extracts a plurality of pixels used for class classification that classifies the target pixel into any one of a plurality of classes from the output image data. When,
A class classification step (for example, step S39 in FIG. 7) for classifying the pixel of interest based on the class tap extracted in the class tap extraction step;
An error between the predicted value obtained by predicting the teacher image data corresponding to the high-quality image data and the teacher image data obtained by a prediction calculation using the student image data corresponding to the output image data and a predetermined coefficient. A coefficient output step (for example, step S40 in FIG. 7) for outputting a coefficient corresponding to the class of the target pixel from the coefficients corresponding to each of the plurality of classes obtained by learning to minimize;
Prediction for obtaining a pixel of the high-quality image data corresponding to the target pixel by the prediction calculation using the coefficient output by the coefficient output step and the prediction tap extracted by the prediction tap extraction step And an image conversion step (for example, step S31 to step S43 in FIG. 7) for performing an image conversion process for converting the output image data into the high-quality image data. ,
A control step for controlling the image conversion step so that the image conversion process is performed for each processing block of the output image data having a block size determined based on the predetermined number of times (for example, the step of FIG. 7) And S31).

The learning device according to the second aspect of the present invention provides:
Encoded data (for example, encoded image data (for example, the reduced image in FIG. 1) obtained by encoding the converted image data (for example, the reduced image in FIG. 1) obtained by converting the number of pixels of the original image data (for example, the original image in FIG. 1) by a predetermined number times. For example, the MPEG encoded image of FIG. 1) is decoded for each of the predetermined blocks,
The number of pixels of the original image data obtained by multiplying the number of pixels of image data (for example, MPEG decoded image of FIG. 1) obtained by decoding the encoded data by a reciprocal multiple of the predetermined number; A coefficient used for a prediction calculation for converting output image data having the same number of pixels (for example, the enlarged MPEG decoded image in FIG. 1) into high-quality image data having higher image quality than the output image data is used as the output image data. A learning device (for example, FIG. 8) that obtains by learning that minimizes an error between the result of the prediction calculation using student image data that is a corresponding image and teacher image data that is an image corresponding to the high-quality image data. In the learning device 201)
A plurality of pixels used in the prediction calculation for converting a target pixel that is a target pixel of the student image data into a pixel of the teacher image data corresponding to the target pixel is used as a prediction tap from the student image data. A prediction tap extraction means (for example, a prediction tap extraction unit 235 in FIG. 8) to extract;
Class tap extraction means for extracting, as class taps, a plurality of pixels used for class classification that classifies the pixel of interest into one of a plurality of classes (for example, class tap extraction in FIG. 8). Part 236),
A class classification unit (for example, a class classification unit 237 in FIG. 8) that classifies the target pixel based on the class tap extracted by the class tap extraction unit;
For each class of the target pixel classified by the class classification unit, an error between the prediction calculation result using the prediction tap and the pixel of the teacher image data corresponding to the target pixel is minimized. A learning unit (for example, the learning apparatus 201 in FIG. 8) that performs a learning process for obtaining the coefficient for each class, and a coefficient calculation unit (for example, the coefficient calculation unit 239 in FIG. 8) that calculates the coefficient.
Control means (for example, the control unit 240 in FIG. 8) that controls the learning means so that the learning process is performed for each processing block of the student image data having a block size determined based on the predetermined multiple. ) And.

A threshold value calculation means (for example, a threshold value calculation unit 234 in FIG. 8) for calculating a threshold value used for the class classification of the target pixel by using a pixel of the student image data;
The class classification unit classifies the pixel of interest based on the class tap extracted by the class tap extraction unit and the threshold value calculated by the threshold value calculation unit.

It is determined whether or not a pixel on the boundary, which is a pixel of the student image data, is included in the processing block on a boundary of an inverse multiple block obtained by making the block size of the predetermined block an inverse multiple of the predetermined number. It further includes a pixel determination unit for determination (for example, the threshold value calculation unit 234 in FIG. 8 that executes the process of step S73 in FIG. 9),
The threshold calculation means, when the boundary pixel determination means determines that the pixel on the pixel is included in the processing block, the pixel obtained by excluding the pixel on the boundary from the pixel of the processing block Calculate using.

It further comprises pixel position mode determination means (for example, a pixel position mode determination unit 339 in FIG. 15) that determines information indicating the position of each pixel in the processing block as a pixel position mode and determines the pixel position mode of the target pixel.
The coefficient calculation means calculates the coefficient that minimizes an error between the prediction calculation result using the prediction tap and the pixel of the teacher image data for each class of pixel of interest and pixel position mode.

The learning method or program according to the second aspect of the present invention includes:
Encoded data (for example, encoded image data (for example, the reduced image in FIG. 1) obtained by encoding the converted image data (for example, the reduced image in FIG. 1) obtained by converting the number of pixels of the original image data (for example, the original image in FIG. 1) by a predetermined number times. For example, the MPEG encoded image of FIG. 1) is decoded for each of the predetermined blocks,
The number of pixels of the original image data obtained by multiplying the number of pixels of image data (for example, MPEG decoded image of FIG. 1) obtained by decoding the encoded data by a reciprocal multiple of the predetermined number; A coefficient used for a prediction calculation for converting output image data having the same number of pixels (for example, the enlarged MPEG decoded image in FIG. 1) into high-quality image data having higher image quality than the output image data is used as the output image data. A learning method obtained by learning that minimizes an error between the result of the prediction calculation using student image data that is an image corresponding to the teacher image data that is an image corresponding to the high-quality image data, or the original image data The encoded data obtained by encoding the converted image data obtained by converting the number of pixels by a predetermined number of times for each predetermined block is decoded for each predetermined block,
Output image data having the same number of pixels as the number of pixels of the original image data, obtained by multiplying the number of pixels of the image data obtained by decoding the encoded data by a reciprocal of the predetermined number, The coefficient used for the prediction calculation for converting the high-quality image data with higher image quality than the output image data, the result of the prediction calculation using the student image data that is the image corresponding to the output image data, and the image quality In a program that causes a computer to execute processing to be obtained by learning that minimizes an error from teacher image data that is an image corresponding to image data,
A plurality of pixels used in the prediction calculation for converting a target pixel that is a target pixel of the student image data into a pixel of the teacher image data corresponding to the target pixel is used as a prediction tap from the student image data. A prediction tap extraction step (for example, step S77 in FIG. 9) to be extracted;
A class tap extraction step (eg, step S78 in FIG. 9) that extracts a plurality of pixels used for class classification that classifies the target pixel into any one of a plurality of classes as class taps from the student image data. When,
A class classification step (for example, step S79 in FIG. 9) for classifying the pixel of interest based on the class tap extracted by the class tap extraction step;
For each class of the target pixel classified by the class classification step, an error between the prediction calculation result using the prediction tap and the pixel of the teacher image data corresponding to the target pixel is minimized. A coefficient calculating step (for example, step S83 in FIG. 9) for calculating the coefficient, and a learning step (for example, step S71 to step S83 in FIG. 9) for performing a learning process for obtaining the coefficient for each class;
A control step (for example, step S71 in FIG. 9) for controlling the learning step so that the learning process is performed for each processing block of the student image data having a block size determined based on the predetermined number of times. Includes and.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 2 is a block diagram showing a configuration example of an embodiment of an image conversion unit to which the present invention is applied.

  The image conversion unit 61 in FIG. 2 performs, for example, an image conversion process for converting the output image data output from the decoder 32 in FIG. 1 into high-quality image data with higher image quality than the output image data. That is, the image conversion unit 61 uses, for example, an 8 × 8 pixel block by MPEG-2 from the expanded MPEG decoded image as described in FIG. 1 as the output image data output from the decoder 32 of FIG. A noise removal process is performed to remove block distortion and mosquito noise caused by encoding and decoding in units.

  Note that the output image data here is an expanded MPEG decoded image of 1920 × 1080 pixels obtained by multiplying the MPEG decoded image of 1440 × 1080 pixels directly obtained by MPEG-2 decoding by 4/3 in the horizontal direction. In the image data, a 16 × 16 pixel (horizontal × vertical) macroblock of the MPEG decoded image is 21.333 (more precisely, 21.333...) Obtained by multiplying the macroblock by 4/3 in the horizontal direction. It becomes an enlarged macroblock of × 16 pixels (horizontal × vertical). In the output image data, an 8 × 8 pixel block constituting a macroblock of the MPEG decoded image is an enlarged block of 10.666 × 8 pixels (horizontal × vertical) obtained by multiplying the block by 4/3 in the horizontal direction. It becomes.

  Here, in order to simplify the description, only the luminance signal is considered below. Accordingly, the macroblock of the MPEG decoded image is composed of 2 × 2 (horizontal × vertical) luminance signal blocks, and the enlarged macroblock of the enlarged MPEG decoded image is 2 × 2 (horizontal × vertical) luminance signal. It is assumed that it is composed of an enlarged block.

  The image conversion unit 61 includes a block extraction unit 121, a threshold calculation unit 122, a prediction tap extraction unit 123, a class tap extraction unit 124, a class classification unit 125, a coefficient storage unit 126, a prediction calculation unit 127, and a control unit 128. The

  The block extraction unit 121 sequentially extracts, from the output image data supplied from the decoder 32 of FIG. 1, 64 × 16 pixel extraction target blocks to be described later, for example, in a so-called raster scan order. Further, the block extraction unit 121 divides the extraction target block into a plurality of processing target blocks to be described later, and sequentially supplies them to the threshold value calculation unit 122.

  The threshold value calculation unit 122 divides the processing target block supplied from the block extraction unit 121 into a plurality of processing blocks described later, and sequentially sets the processing blocks as the target block in the raster scan order, for example.

  The threshold calculation unit 122 calculates, from the target block, a threshold used for class classification that classifies a target pixel, which is a target pixel of the prediction tap extraction unit 123 described later, into one of a plurality of classes. The data is supplied to the class classification unit 125.

  The prediction tap extraction unit 123 selects pixels included in the target block sequentially as the target pixel, for example, in the raster scan order, and further converts the output image data corresponding to the target pixel to obtain high image quality. A plurality of pixels used for predicting the pixel values of the pixels of the image data are extracted from the output image data as prediction taps.

  That is, for example, the prediction tap extraction unit 123 extracts, from the output image data, the horizontal tap 71 that is a plurality of pixels arranged in the horizontal direction around the target pixel as a prediction tap. Here, in FIG. 2, four pixels indicated by black circles adjacent to the right of the pixel of interest indicated by white circles in the upper right of FIG. 2, four pixels indicated by black circles adjacent to the left of the pixel of interest, and pixels of interest Nine pixels arranged in the horizontal direction are shown as horizontal taps 71.

  The prediction tap extraction unit 123 supplies the prediction tap extracted from the output image data to the prediction calculation unit 127.

  The class tap extraction unit 124 extracts a plurality of pixels used for classifying the target pixel as class taps from the output image data.

  That is, for example, the class tap extraction unit 124 extracts, from the output image data, horizontal taps 71 that are a plurality of pixels arranged in the horizontal direction around the target pixel as class taps.

  Here, in order to simplify the description, the prediction tap and the class tap are a plurality of pixels having the same tap structure (positional relationship with respect to the target pixel). However, the prediction tap and the class tap can be a plurality of pixels having different tap structures.

  The class tap extraction unit 124 supplies the class tap extracted from the output image data to the class classification unit 125.

  The class classification unit 125 classifies the pixel of interest based on the threshold value supplied from the threshold value calculation unit 122 and the class tap supplied from the clap tap extraction unit 124, and calculates a class code corresponding to the resulting class as a coefficient. The data is supplied to the storage unit 126.

  The coefficient storage unit 126 stores a coefficient table in which tap coefficients obtained by performing a learning process to be described later are registered, and tap coefficients associated with the class codes supplied from the class classification unit 125 (attention) The pixel class tap coefficient) is supplied to the prediction calculation unit 127.

  The prediction calculation unit 127 uses the prediction tap supplied from the prediction tap extraction unit 123 and the tap coefficient supplied from the coefficient storage unit 126, and the pixel value of the pixel of the high-quality image data corresponding to the target pixel ( Predetermined calculation for obtaining (predicted value) is performed. Thereby, the prediction calculation unit 127 calculates and outputs the pixel value of the pixel of the high-quality image data corresponding to the target pixel.

  The control unit 128 controls the block extraction unit 121, the threshold calculation unit 122, the prediction tap extraction unit 123, the class tap extraction unit 124, the class classification unit 125, the coefficient storage unit 126, and the prediction calculation unit 127.

  That is, the control unit 128 controls each block so that the image conversion process in the image conversion unit 61 is performed for each processing block.

  Next, with reference to FIGS. 3 and 4, processing in the block extraction unit 121 in FIG. 2 will be described.

  FIG. 3 is a diagram illustrating the order in which the block extraction unit 121 extracts extraction target blocks.

  The block extraction unit 121 sequentially extracts N × M pixel (horizontal × vertical) (N and M are natural numbers) extraction target blocks composed of a plurality of enlarged macroblocks from the output image data, for example, in raster scan order. Divide into a plurality of processing target blocks.

  In other words, the block extraction unit 121 sequentially extracts 64 × 16 pixel (horizontal × vertical) extraction target blocks from the 1920 × 1080 pixel (horizontal × vertical) output image data, for example, in raster scan order, and performs processing. Divide into blocks.

  Next, FIG. 4 is a diagram illustrating a method in which the block extraction unit 121 divides the extraction target block into a plurality of processing target blocks.

  The upper side of FIG. 4 shows an extraction target block of 64 × 16 pixels extracted by the block extraction unit 121, and the extraction target blocks are three enlarged macroblocks of 21.333 × 16 pixels arranged in the horizontal direction. Consists of.

  The enlarged macroblock is composed of 2 × 2 enlarged blocks of 10.666 × 8 pixels.

  In the lower part of FIG. 4, three processing target blocks 131 to 133 obtained by dividing the extraction target block are shown.

  The extraction target blocks of 64 × 16 pixels are partitioned in the vertical direction, so that the processing target block 131 of 21 × 16 pixels, the processing target block 132 of 22 × 16 pixels, and the 21 × are arranged in the horizontal direction sequentially from the left. The block is divided into 16-pixel processing target blocks 133.

  The 21 × 16 pixel processing target block 131 includes an 11 × 8 pixel (horizontal × vertical) upper left and lower left processing block and a 10 × 8 pixel (horizontal × vertical) upper right and lower right processing block. It is divided by a total of 4 processing blocks.

  The 22 × 16 pixel processing target block 132 is divided into 11 × 8 pixel upper left, lower left, upper right, and lower right processing blocks.

  Further, the 21 × 16 pixel processing target block 133 is divided into a total of four processing blocks, that is, 10 × 8 pixel upper left and lower left processing blocks and 11 × 8 pixel upper right and lower right processing blocks. .

  Here, in the 11 × 8 pixel processing block on the lower side of FIG. 4, the hatched portion indicates pixels on the boundary including the boundary of the enlarged block indicated by the white rectangle on the upper side of FIG. 4. The pixel on the boundary is included only in the processing block of 11 × 8 pixels and is not included in the processing block of 10 × 8 pixels.

  Note that the block size of the processing block is determined based on the magnification (3/4 times) when the original image is converted into the reduced image, which is performed on the broadcasting station 31 side in FIG.

  In other words, the block size of the processing block is a reciprocal multiple of the magnification (3/4 times) when converting the 8 × 8 pixel block of the MPEG decoded image in the horizontal direction into the reduced image. This is a block size obtained by multiplying by 3 and rounding or rounding up the number of pixels in the horizontal direction of the 10.666 × 8 pixel block obtained as a result. That is, the block size of the processing block is determined to be 10 × 8 pixels by truncating the portion below the first decimal point of the number of pixels in the horizontal direction of the 10.666 × 8 pixel block. By rounding up the portion of the number of pixels in the horizontal direction of the block of 666 × 8 pixels below the first decimal place, the block size of the processing block is determined to be 11 × 8 pixels.

  Also, the processing target block 131, processing target block 132, and processing target block 133 on the lower side of FIG. 4 correspond to the first, second, and third enlarged macroblocks from the left on the upper side of FIG.

  The block extraction unit 121 classifies the extraction target block shown on the upper side of FIG. 4 into the processing target blocks 131 to 133 shown on the lower side of FIG. 4, and first supplies the processing target block 131 to the threshold value calculation unit 122. To do.

  Then, when all the pixels of the processing target block 131 are the target pixel, the block extraction unit 121 supplies the processing target block 132 to the threshold value calculation unit 122.

  Further, when all the pixels of the processing target block 132 are set as the target pixel, the block extraction unit 121 supplies the processing target block 133 to the threshold calculation unit 122.

  When the processing target block is supplied from the block extraction unit 121, the threshold calculation unit 122 displays the processing target block as a 10 × 8 pixel processing block or an 11 × 8 pixel processing block shown on the lower side of FIG. For example, the threshold value is calculated as the target block sequentially in the raster scan order.

  Next, FIG. 5 is a diagram illustrating processing in which the threshold value calculation unit 122 calculates a threshold value from the processing blocks supplied from the block extraction unit 121.

  The processing target block 131 (FIG. 4) supplied to the threshold value calculation unit 121 is shown on the upper side of FIG. 5, and the enlarged macroblock corresponding to the processing target block 131 (before expansion) is shown on the right side of FIG. The block to be processed 131 is shown when the block structure of (macroblock) is a frame structure, that is, when the DCT type is a frame DCT. Further, on the left side of FIG. 5, when the block structure of the expanded macroblock (the macroblock before expansion) corresponding to the processing target block 131 is a field structure, that is, the processing target block when the DCT type is the field DCT. 131 is shown.

  The hatched portion shown in the block on the right side of FIG. 5 and the block on the left side of FIG. 5 represents an even line (bottom field) in the processing target block 131, and the white portion represents an odd number in the processing target block 131. Represents a line (top field).

  Further, on the lower side of FIG. 5, a processing block of 11 × 8 pixels at the upper left or lower left of the processing target block 131 is shown. In addition, the shaded portion in the processing block of 11 × 8 pixels shown on the lower side of FIG. 5 indicates the pixel on the boundary.

  For example, the threshold value calculation unit 122 determines whether the block structure of the enlarged macroblock corresponding to the processing target block 131 supplied from the block extraction unit 121 is a frame structure or a field structure. For example, the threshold value calculation unit 122 compares the intra-field autocorrelation and the inter-field correlation of the expanded macroblock corresponding to the processing target block 131, thereby obtaining the block structure of the expanded macroblock corresponding to the processing target block 131. Can be determined.

  When it is determined that the block structure of the enlarged macroblock corresponding to the processing target block 131 is a frame structure, the threshold value calculation unit 122 determines that the upper left 11 × of the processing target blocks 131 illustrated on the right side of FIG. A threshold value is calculated using a processing block of 8 pixels as a target block.

  After all the pixels of the upper left processing block as the target block have been set as the target pixels, the threshold calculation unit 121 performs processing of the upper right 10 × 8 pixel processing block and lower left 11 × 8 pixel processing. The threshold value is calculated as the target block sequentially in the order of the block and the processing block of 10 × 8 pixels at the lower right.

  On the other hand, when it is determined that the block structure of the enlarged macroblock corresponding to the processing target block 131 is a field structure, the threshold value calculation unit 122 determines that the upper left of the processing target blocks 131 shown on the right side of the center of FIG. The threshold value is calculated using the processing block of 11 × 8 pixels in the upper left as the target block.

  After all of the pixels of the upper left processing block that is the target block are set as the target pixels, the threshold value calculation unit 121 now processes the upper right 10 × 8 pixel processing block and the lower left 11 × 8 pixel. Threshold values are sequentially calculated as the target block in the order of the processing block and the lower right 10 × 8 pixel processing block.

  The threshold calculation unit 121 calculates the threshold from the block of interest as follows, for example.

  That is, as shown in the lower side of FIG. 5, the threshold value calculation unit 121 is a pixel-to-pixel difference absolute value of adjacent pixel values among pixels arranged in the horizontal direction (horizontal direction) of the target block. Find the difference.

  However, the inter-pixel difference is obtained only for pixels excluding the pixels on the boundary.

  Accordingly, when the target block is a 10 × 8 pixel processing block that does not include the pixels on the boundary, 9 × 8 = 72 inter-pixel differences are obtained for all the pixels constituting the processing block. It is done.

  When calculating the 72 inter-pixel differences for the block of interest, the threshold calculation unit 121 calculates the maximum value MAX and the minimum value MIN among the 72 inter-pixel differences, and sets DR = MAX−MIN between the pixels. Calculated as the dynamic range DR of the difference. Further, the threshold value calculation unit 121 calculates a value obtained by adding the minimum value MIN to a value obtained by dividing the dynamic range DR by 4 as a threshold value Th = (DR / 4) + MIN.

  On the other hand, when the block of interest is an 11 × 8 pixel processing block including pixels on the boundary shown on the lower side of FIG. 5, the lower side of FIG. 5 among the 11 × 8 pixels constituting the processing block Again, 72 inter-pixel differences are obtained for pixels excluding the 1 × 8 pixels on the boundary indicated by the shaded area shown in FIG.

  When the threshold calculation unit 122 calculates the threshold from the block of interest, the threshold calculation unit 122 supplies the threshold to the class classification unit 125. The class classification unit 125 classifies the target pixel based on the class tap supplied from the class tap extraction unit 124 and the threshold supplied from the threshold calculation unit 122.

  FIG. 6 is a diagram for explaining class classification performed by the class classification unit 125 of FIG.

  Nine white circles arranged in the horizontal direction on the upper side of FIG. 6 indicate horizontal taps 71 as class taps supplied from the class tap extraction unit 124, and below the nine white circles arranged in the horizontal direction. The numerical value shown in each indicates the pixel value of each pixel constituting the class tap represented by a white circle.

  Also, the numerical value in the center of FIG. 6 is an inter-pixel difference that is an absolute value of a difference between adjacent pixels in the horizontal direction (horizontal direction) in the drawing among the horizontal taps 71 that are nine pixels arranged in the horizontal direction. Is shown. Further, the numerical values on the lower side of FIG. 6 indicate class codes output by the class classification unit 125.

  The class classification unit 125 obtains eight inter-pixel differences between adjacent pixels among the horizontal taps 71 that are nine pixels arranged in the horizontal direction shown in the upper side of FIG.

  Next, the class classification unit 125 determines whether or not the eight inter-pixel differences shown in the center of FIG. 6 are equal to or greater than the threshold value Th supplied from the threshold value calculation unit 122.

  When the inter-pixel difference is equal to or greater than the threshold value Th, the class classification unit 125 assigns a 1-bit size flag in which 1 representing 1 or more is set to the inter-pixel difference. If the inter-pixel difference is not greater than or equal to the threshold Th, the class classification unit 125 assigns a magnitude flag set to 0 indicating that the inter-pixel difference is not greater than Th. Then, the class classification unit 125 outputs, to the coefficient storage unit 126, a sequence of eight large and small flags set to 1 or 0 shown in the lower part of FIG. 6 as class codes.

  Next, image conversion processing in the image conversion unit 61 in FIG. 2 will be described with reference to the flowchart in FIG.

  This image conversion process is started, for example, when output image data is supplied to the image conversion unit 61 by the decoder 32 of FIG. At this time, output image data is supplied to each of the block extraction unit 121, the prediction tap extraction unit 123, and the class tap extraction unit 124.

  In step S31, the control unit 128 starts control of each block so that the image conversion processing in the image conversion unit 61 is performed for each processing block.

  Specifically, the control unit 128 controls the block extraction unit 121, and the extraction target block (FIG. 4) of 64 × 16 pixels is shown in FIG. 3 from the output image data output from the decoder 32 of FIG. As illustrated in FIG. 4, for example, the extraction target blocks are sequentially extracted in the raster scan order, and the extracted extraction target blocks are divided into processing target blocks 131, 132, or 133 as illustrated in FIG. 4.

  In addition, the control unit 128 controls the block extraction unit 121 and causes the threshold calculation unit 122 to supply the divided processing target blocks as appropriate. Further, the control unit 128 controls the threshold value calculation unit 122 to divide the processing target block into processing blocks of 11 × 8 pixels or 10 × 8 pixels. Then, the control unit 128 controls the threshold calculation unit 122, the prediction tap extraction unit 123, the class tap extraction unit 124, the class classification unit 125, the coefficient storage unit 126, or the prediction calculation unit 127, so that an image is processed for each processing block. Let the conversion process.

  The threshold calculation unit 122, the prediction tap extraction unit 123, the class tap extraction unit 124, the class classification unit 125, the coefficient storage unit 126, or the prediction calculation unit 127 performs image conversion processing for each processing block in accordance with the control of the control unit 128. .

  That is, after the process of step S31, the process proceeds to step S32, and the threshold value calculation unit 122 converts 2 × 2 processing blocks included in the processing target block supplied from the block extraction unit 121, for example, in raster scan order. The target block is sequentially selected, and the process proceeds to step S33.

  In step S33, the threshold value calculation unit 122 determines whether the block of interest includes a pixel on the boundary.

  If it is determined in step S33 that the block of interest includes a pixel on the boundary, the process proceeds to step S34, and the threshold calculation unit 122 starts from the pixel of the block of interest to the boundary as illustrated in the lower side of FIG. The threshold value is calculated using the pixels excluding the upper pixel, supplied to the class classification unit 125, and the process proceeds to step S36.

  On the other hand, when it is determined in step S33 that the pixel of interest does not include the pixel on the boundary, the process proceeds to step S35, and the threshold value calculation unit 122 calculates the threshold value from all the n pixels of the target block, and class The data is supplied to the classification unit 125, and the process proceeds to step S36.

  In step S36, the prediction tap extraction unit 123 extracts pixels included in the target block sequentially from the output image data supplied from the decoder 32 in FIG. 1 in the raster scan order, selects the target pixel, and performs processing. Advances to step S37.

  In step S <b> 37, the prediction tap extraction unit 123 predicts a plurality of pixels used for predicting the pixel value of the high-quality image data to be obtained by converting the output image data corresponding to the target pixel. Are extracted from the output image data and supplied to the prediction calculation unit 127.

  Thereafter, the process proceeds from step S37 to step S38, and the class tap extraction unit 124 extracts a plurality of pixels used for classifying the target pixel as class taps from the output image data supplied from the decoder 32 of FIG. Then, the data is supplied to the class classification unit 125, and the process proceeds to step S39.

  In step S39, the class classification unit 125 performs the processing described with reference to FIG. 6 on the basis of the threshold supplied from the threshold calculation unit 122 and the class tap supplied from the clap extraction unit 124. And class code corresponding to the resulting class is supplied to the coefficient storage unit 126, and the process proceeds to step S40.

  The coefficient storage unit 126 stores a coefficient table in which tap coefficients obtained by performing a learning process to be described later are registered. In step S40, the coefficient storage unit 126 is associated with the class code supplied from the class classification unit 125. The tap coefficient is supplied to the prediction calculation unit 127, and the process proceeds to step S41.

  In step S41, the prediction calculation unit 127 uses the prediction tap supplied from the prediction tap extraction unit 123 and the tap coefficient supplied from the coefficient storage unit 126 to calculate the pixel of the high-quality image data corresponding to the target pixel. A predetermined prediction calculation for obtaining a pixel value (predicted value) is performed, and the process proceeds to step S42.

  In step S <b> 42, the prediction tap extraction unit 123 determines whether all the pixels of the target block are the target pixels.

  If it is determined in step S42 that all the pixels of the block of interest are not yet the pixel of interest, the process returns to step S36, and the prediction tap extraction unit 123 selects pixels of the block of interest that are not yet the pixel of interest. The process proceeds to step S37, and the same process is repeated thereafter.

  On the other hand, when it is determined in step S42 that all the pixels of the target block are the target pixels, the process proceeds to step S43, and the control unit 128 sets all the processing blocks of the output image data as the target block. It is determined whether or not.

  If it is determined in step S43 that all the processing blocks of the output image data are not the target block, the processing returns to step S32, and thereafter the same processing is repeated.

  On the other hand, when it is determined in step S43 that all the processing blocks of the output image data are the target blocks, the image conversion processing in FIG. 7 is ended.

  In the image conversion processing of FIG. 7 as described above, when a pixel on the boundary is included in the processing block, a threshold is calculated using a pixel obtained by excluding the pixel on the boundary from the pixel in the processing block, and the pixel on the boundary is included in the processing block. When it is not included, the threshold value is calculated from all pixels of the processing block. Therefore, the threshold value is calculated from the pixel of the processing block both when the pixel on the boundary is included and when it is not included in the processing block. Compared with the case where the same process is performed, a more appropriate threshold value can be calculated.

  Therefore, in the image conversion process of FIG. 7, the pixel of interest can be classified into an appropriate class, and as a result, pixel values of high-quality image data with little error (high image quality) can be obtained. That is, it is possible to effectively remove mosquito noise and block distortion generated at the boundary (near the enlarged block) of 10.666 × 8 pixels of the output image data.

  Next, prediction calculation in the prediction calculation unit 127 in FIG. 2 and learning of tap coefficients stored in the coefficient storage unit 126 will be described.

  Now, as image conversion processing, a prediction tap is extracted from output image data, and pixels of high-quality image data free of mosquito noise and block distortion (hereinafter, referred to as high-quality pixels as appropriate) using the prediction tap and tap coefficients. It is assumed that the pixel value is determined (predicted) by a predetermined prediction calculation.

  For example, when a linear primary prediction calculation is adopted as the predetermined prediction calculation, the pixel value y of the high-quality pixel is obtained by the following linear linear expression.

・・・（１）

... (1)

In Equation (1), x _n represents a pixel value of an n-th pixel of output image data (hereinafter, appropriately referred to as an output pixel) constituting a prediction tap for the high-quality pixel y. Here, for example, if the horizontal tap 71 that is 9 pixels arranged in the horizontal direction shown in the upper right side of FIG. 2 is taken as an example of the prediction tap, the pixel value of the nth output pixel is the horizontal tap. This is the pixel value of the output pixel corresponding to the number n assigned to 71.

Also, w _n denotes the n-th tap coefficient multiplied with the n-th output pixel (pixel value of). In equation (1), the prediction tap is assumed to be composed of N output pixels x ₁ , x ₂ ,..., X _N.

  Here, the pixel value y of the high-quality pixel can be obtained not by the linear primary expression shown in Expression (1) but by a higher-order expression of the second or higher order.

Now, when the true value of the pixel value of the high-quality pixel of the k-th sample is expressed as y _k and the predicted value of the true value y _k obtained by the equation (1) is expressed as y _k ′, the prediction error e _k Is expressed by the following equation.

・・・（２）

... (2)

Now, since the predicted value y _k ′ of Equation (2) is obtained according to Equation (1), the following equation is obtained by replacing y _k ′ of Equation (2) according to Equation (1).

・・・（３）

... (3)

In Equation (3), x _{n, k} represents the n-th output pixel constituting the prediction tap for the high-quality pixel of the k-th sample.

Tap coefficient w _n for the prediction error e _k 0 of the formula (3) (or Equation (2)) is, is the optimal for predicting the high-quality pixel, for all the high-quality pixel, such In general, it is difficult to obtain a large tap coefficient w _n .

Therefore, as the standard for indicating that the tap coefficient w _n is optimal, for example, when adopting the method of least squares, optimal tap coefficient w _n, the sum E of square errors expressed by the following formula It can be obtained by minimizing.

・・・（４）

... (4)

However, in Equation (4), K is the high-quality pixel y _k and the output pixels x _{1, k} , x _{2, k} ,..., X _N, constituting the prediction tap for the high-quality pixel y _{k . This represents the} number of samples in the set with _k (the number of samples for learning).

The minimum value of the sum E of square errors of Equation (4) (minimum value), as shown in Equation (5), given that by partially differentiating the sum E with the tap coefficient w _n by w _n to 0.

・・・（５）

... (5)

On the other hand, when the partial differentiation of the above equation (3) with the tap coefficient w _n, the following equation is obtained.

・・・（６）

... (6)

  From the equations (5) and (6), the following equation is obtained.

・・・（７）

... (7)

By substituting equation (3) into e _k in equation (7), equation (7) can be expressed by the normal equation shown in equation (8).

・・・（８）

... (8)

Normal equation of Equation (8), for example, by using a like sweeping-out method (Gauss-Jordan elimination method) can be solved for the tap coefficient w _n.

The normal equation of Equation (8), by solving for each class, the optimal tap coefficient (here, the tap coefficient that minimizes the sum E of square errors) to w _n, can be found for each class .

  In the image conversion unit 61 of FIG. 2, the output image data is converted into high-quality image data by performing the calculation of Expression (1) using the tap coefficient for each class as described above.

Next, FIG. 8 shows an example of the configuration of a learning device 201 that performs a learning process for determining the tap coefficient w _n by solving the normal equations in equation (8) for each class.

  8 includes a learning image storage unit 231, a noise addition unit 232, a block extraction unit 233, a threshold calculation unit 234, a prediction tap extraction unit 235, a class tap extraction unit 236, a class classification unit 237, and an addition unit. 238, a coefficient calculation unit 239, and a control unit 240.

In addition, a large number of 1920 × 1080 pixel image data corresponding to the original image (FIG. 1) obtained at the broadcast station 31 is input to the learning device 201 as learning image data used for learning the tap coefficient w _n. It has become so.

  In the learning device 201, the learning image data is supplied to and stored in the learning image storage unit 231.

  The noise adding unit 232 reads out the learning image data from the learning image storage unit 231, and performs the same processing as the processing performed in FIG. 1 on the learning image data, thereby obtaining an image corresponding to the output image data. A certain student image data is generated.

  That is, the noise adding unit 232 obtains image data of 1440 × 1080 pixels by multiplying the learning image data of 1920 × 1080 pixels stored in the learning image storage unit 231 by 3/4 in the horizontal direction, MPEG encoding. Further, the noise adding unit 232 decodes the encoded data obtained by the MPEG encoding into image data of 1440 × 1080 pixels, and the image data is multiplied by 4/3 in the horizontal direction to be output image data. Student image data, which is a corresponding image, is generated.

  The noise adding unit 232 supplies the generated student image data to the block extracting unit 233, the prediction tap extracting unit 235, and the class tap extracting unit 236.

  The block extraction unit 233 sequentially extracts the extraction target blocks from the student image data supplied from the noise addition unit 232, for example, in the raster scan order, in the same manner as the block extraction unit 121 in FIG.

  Further, the block extraction unit 233 divides the extraction target block into a plurality of processing target blocks 131 to 133 in the same manner as the block extraction unit 121 of FIG. Supply.

  The threshold calculation unit 234 divides the processing target block supplied from the block extraction unit 233 into 2 × 2 processing blocks in the same manner as the threshold calculation unit 122 in FIG. Let it be a block.

  Furthermore, the threshold calculation unit 234 performs the same processing as the threshold calculation unit 122 in FIG. 2, thereby calculating a threshold used for class classification from the target block and supplies the threshold to the class classification unit 237.

  The prediction tap extraction unit 235 extracts the pixels included in the block of interest, for example, from the student image data supplied from the noise addition unit 232 sequentially in the raster scan order, and sets it as the pixel of interest. Further, the prediction tap extraction unit 235 extracts a predetermined one of the pixels constituting the student image data supplied from the noise addition unit 232, so that the prediction tap extraction unit 123 of FIG. Obtaining a prediction tap having the same tap structure as the tap (a prediction tap made up of pixels of student image data having the same positional relationship as the pixels constituting the prediction tap obtained by the prediction tap extraction unit 123 of FIG. 2 for the target pixel), It supplies to the addition part 238.

  The class tap extraction unit 236 extracts a predetermined one of the pixels constituting the student image data supplied from the noise addition unit 232, so that the class tap extraction unit 124 of FIG. Class taps having the same tap structure (class taps composed of pixels of student image data having the same positional relationship as the pixels constituting the class tap obtained by the class tap extraction unit 124 of FIG. To the unit 237.

  The class classification unit 237 performs the same class classification as the class classification unit 125 of FIG. 2 based on the threshold value supplied from the threshold value calculation unit 234 and the class tap supplied from the class tap extraction unit 236, and obtains the result. The class code corresponding to the class to be added is supplied to the adding unit 238.

  The adding unit 238 reads out the pixel (its pixel value) of the teacher image data (learning image data) corresponding to the target pixel from the learning image storage unit 231, and supplies the pixel and the prediction tap extraction unit 235. For each class code supplied from the class classification unit 237, addition is performed on the pixel of the student image data constituting the prediction tap for the target pixel.

That is, the adder 238, (among the pixel, the pixel values of pixels corresponding to the pixel of interest) teacher image data stored in the learning image storage unit 231 y _k, the prediction tap extracting unit 235 outputs The prediction tap (the pixel value of the pixel of the student image data constituting the _{xn, k} ) and the class code representing the class of the pixel of interest output from the class classification unit 237 are supplied.

Then, the adding unit 238 uses the prediction tap (student image data) x _{n, k} for each class corresponding to the class code supplied from the class classifying unit 237 _, and uses the student image in the matrix on the left side of Expression (8). An operation corresponding to multiplication (x _{n, k} x _{n ′, k} ) between data and summation (Σ) is performed.

Furthermore, the adding unit 238 also uses the prediction tap (student image data) x _{n, k} and the teacher image data y _k for each class corresponding to the class code supplied from the class classification unit 237, and uses the equation (8 ) Is multiplied by the student image data x _{n, k} and the teacher image data y _k (x _{n, k} y _k ) in the vector on the right side of), and an operation corresponding to summation (Σ) is performed.

In other words, the adding unit 238 performs the left-side matrix component (Σx _{n, k} x _{n ′, k} ) and the right-side vector component in Equation (8) previously obtained for the student image data set as the target pixel. (Σx _{n, k} y _k ) is stored in its built-in memory (not shown), and the matrix component (Σx _{n, k} x _{n ′, k} ) or vector component (Σx _{n, k} y) _k ), the corresponding component x _{n, k} calculated using the student image data x _{n, k + 1} and the teacher image data y _{k + 1} for the student image data newly set as the pixel of interest Add ₊₁ x _{n ′, k + 1} or x _{n, k + 1} y _{k + 1} (addition represented by the summation of equation (8) is performed).

  Then, the adding unit 238 performs the above-described addition using all the student image data generated by the noise adding unit 232 as the target pixel, thereby forming the normal equation shown in Expression (8) for each class. Then, the normal equation is supplied to the coefficient calculation unit 239.

The coefficient calculation unit 239 obtains and outputs an optimum tap coefficient w _n for each class by solving the normal equation for each class supplied from the addition unit 238.

  The control unit 240 is a learning image storage unit 231, a noise addition unit 232, a block extraction unit 233, a threshold calculation unit 234, a prediction tap extraction unit 235, a class tap extraction unit 236, a class classification unit 237, an addition unit 238, or The coefficient calculation unit 239 is controlled.

  That is, the control unit 240 controls each block so that the learning process in the learning device 201 is performed for each processing block of the student image data generated by the noise adding unit 232.

  Next, the learning process in the learning apparatus 201 in FIG. 8 will be described with reference to the flowchart in FIG. 9.

  Note that a large number of 1920 × 1080 pixel learning image data is already stored in the learning image storage unit 231.

  In step S71, the control unit 240 starts control of each block so that the learning process in the learning device 201 is performed for each processing block of the student image data.

  Specifically, the control unit 240 controls the noise adding unit 232 to read out the learning image data from the learning image storage unit 231 and causes the learning image data to be the same as the processing performed in FIG. By performing the process, student image data, which is an image corresponding to the output image data, is generated and supplied to the block extraction unit 233.

  The control unit 240 also controls the block extraction unit 233 to supply the processing target blocks to the threshold value calculation unit 234 sequentially, for example, in raster scan order. Specifically, the block extraction unit 233 controls the extraction target block from the student image data supplied from the noise addition unit 232, for example, in the same manner as the block extraction unit 121 of FIG. The data are sequentially extracted in the raster scan order. Further, under the control of the control unit 240, the block extraction unit 233 divides the extraction target blocks into processing target blocks 131 to 133 in the same manner as the block extraction unit 121 of FIG. This is supplied to the threshold value calculation unit 234.

  Further, the control unit 240 controls the threshold value calculation unit 234 to divide the processing target block into processing blocks of 11 × 8 pixels or 10 × 8 pixels, similarly to the threshold value calculation unit 122 of FIG. Then, the control unit 240 controls the threshold calculation unit 234, the prediction tap extraction unit 235, the class tap extraction unit 236, the class classification unit 237, the addition unit 238, or the coefficient calculation unit 239, so that the noise addition unit 232 A learning process is performed for each processing block of generated student image data.

  The threshold calculation unit 234, the prediction tap extraction unit 235, the class tap extraction unit 236, the class classification unit 237, the addition unit 238, or the coefficient calculation unit 239 performs learning processing for each processing block according to the control of the control unit 240.

  That is, after the process of step S71, the process proceeds to step S72, and the threshold value calculation unit 234 receives 2 × 2 processing target blocks supplied from the block extraction unit 233 in the same manner as the threshold value calculation unit 122 of FIG. For example, the target block is sequentially set in the raster scan order, and the process proceeds to step S72.

  In step S <b> 73, the threshold calculation unit 234 determines whether the block of interest includes a pixel on the boundary in the same manner as the threshold calculation unit 122 in FIG. 2.

  If it is determined in step S73 that the target block includes a pixel on the boundary, the process proceeds to step S74, and the threshold value calculation unit 234 performs the same process as the threshold value calculation unit 122 in FIG. The threshold value is calculated using a pixel obtained by excluding the pixel on the boundary from the pixel of the target block, and is supplied to the class classification unit 237, and the process proceeds to step S76.

  On the other hand, if it is determined in step S73 that the pixel of interest does not include the pixel on the boundary, the process proceeds to step S75, and the threshold value calculation unit 234 performs the same process as the threshold value calculation unit 122 in FIG. Thus, the threshold value is calculated from all the pixels of the block of interest, supplied to the class classification unit 237, and the process proceeds to step S76.

  In step S76, the prediction tap extraction unit 235 sequentially extracts pixels included in the target block from the student image data supplied from the noise adding unit 232 in the raster scan order, selects the target pixel, and performs processing. Advances to step S76.

  In step S77, the prediction tap extraction unit 235 extracts a predetermined one of the pixels constituting the student image data supplied from the noise addition unit 232, so that the prediction tap extraction unit 123 of FIG. Obtains a prediction tap having the same tap structure as the prediction tap obtained for the target pixel, and supplies the prediction tap to the adding unit 238.

  Thereafter, the process proceeds from step S77 to step S78, and the class tap extraction unit 236 extracts a predetermined one of the pixels constituting the student image data supplied from the noise addition unit 232, so that the pixel of interest is extracted. The class tap extraction unit 124 in FIG. 2 obtains a class tap having the same tap structure as the class tap obtained for the target pixel, and supplies it to the class classification unit 237.

  Further, the process proceeds from step S78 to step S79, where the class classification unit 237 is based on the threshold supplied from the threshold calculation unit 234 and the class tap supplied from the class tap extraction unit 236. The same class classification as 125 is performed, and the class code corresponding to the resulting class is supplied to the adding unit 238, and the process proceeds to step S80.

  In step S80, the adding unit 238 reads the pixel (the pixel value) of the teacher image data corresponding to the target pixel from the learning image storage unit 231 and is supplied from the pixel and the prediction tap extraction unit 235. Addition is performed for each class code supplied from the class classification unit 237, and the process proceeds to step S81.

  In step S <b> 81, the prediction tap extraction unit 235 determines whether or not all the pixels of the target block are the target pixels.

  If it is determined in step S81 that all the pixels of the target block have not been set as the target pixel, the process returns to step S76, and the prediction tap extraction unit 235 selects pixels in the target block that have not been set as the target pixel yet. The process proceeds to step S77, and thereafter the same process is repeated.

On the other hand, if it is determined in step S81 that all the pixels of the target block are the target pixels, the control unit 240 proceeds to step S82, and the control unit 240 processes all the processing blocks of the student image data supplied from the noise adding unit 232. Then, it is determined whether or not the block is the target block.

  In step S82, when it is determined that all the processing blocks of the student image data supplied from the noise adding unit 232 are not the target block, the processing returns to step S72, and thereafter the same processing is repeated.

On the other hand, when it is determined in step S82 that all the processing blocks of the student image data supplied from the noise adding unit 232 are the target blocks, that is, the student generated by the noise adding unit 232 in the adding unit 238. When the above-described addition is performed using all image data as the target pixel to obtain the normal equation shown in Expression (8) for each class, the addition unit 238 adds the normal equation to the coefficient calculation unit 239. The process proceeds to step S83. The coefficient calculation unit 239 obtains and outputs an optimum tap coefficient w _n for each class by solving the normal equation for each class supplied from the addition unit 238.

  Thereafter, the learning process of FIG. 9 is terminated.

  Next, FIG. 10 is a block diagram showing a configuration example of another embodiment of an image conversion unit to which the present invention is applied.

  In the figure, portions corresponding to those in FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  That is, the image conversion unit 61 of FIG. 10 includes a coefficient storage unit 276, a prediction calculation unit 278, or a control unit 279, respectively, instead of the coefficient storage unit 126, the prediction calculation unit 127, or the control unit 128 of FIG. 2 except that a pixel position mode determination unit 275 and a region determination unit 277 are newly provided.

  In addition, in FIG. 10, the prediction tap extraction unit 123 supplies the prediction tap to the prediction calculation unit 278 and, in addition, the position information of the pixel that is the current pixel of interest (for example, the coordinates of the target pixel in the target block), The pixel position mode determination unit 275 and the region determination unit 277 are supplied.

  The pixel position mode determination unit 275 determines the pixel position mode of the target pixel based on the position information of the target pixel supplied from the prediction tap extraction unit 123 and supplies the determined pixel position mode to the coefficient storage unit 276.

  Here, the pixel position mode is information representing the position of each pixel in the processing block, and is a value used to determine the tap coefficient output from the coefficient storage unit 276, as with the class code from the class classification unit 125. It is. Details of the pixel position mode will be described later.

  The coefficient storage unit 276 stores a coefficient table in which tap coefficients obtained by performing learning processing as described below are registered, and the class code supplied from the class classification unit 125 and the pixel position mode determination unit The tap coefficient associated with the pixel position mode supplied from 275 is supplied to the prediction calculation unit 278.

  The region determination unit 277 determines region information of the target pixel based on the position information supplied from the prediction tap extraction unit 123 and supplies the region information to the prediction calculation unit 278.

  Here, the area information is information indicating whether the target pixel is located in any of the plurality of areas when the processing block is divided into a plurality of areas.

  The prediction calculation unit 278 uses the prediction tap supplied from the prediction tap extraction unit 123 and the tap coefficient supplied from the coefficient storage unit 276 to perform a predetermined prediction corresponding to the region information supplied from the region determination unit 277. By performing the calculation, the pixel value (predicted value) of the pixel of the high-quality image data corresponding to the target pixel is obtained and output. That is, the prediction calculation unit 278 obtains and outputs the pixel value of the pixel of the high-quality image data corresponding to the target pixel. The predetermined prediction calculation corresponding to the area information will be described later.

  The control unit 279 includes a block extraction unit 121, a threshold calculation unit 122, a prediction tap extraction unit 123, a class tap extraction unit 124, a class classification unit 125, a pixel position mode determination unit 275, a coefficient storage unit 276, a region determination unit 277, or The prediction calculation unit 278 is controlled.

  That is, the control unit 279 controls each block so that the image conversion processing in the image conversion unit 61 is performed for each processing block of output image data.

  Next, FIG. 11 is a diagram for explaining a predetermined prediction calculation corresponding to the pixel position mode and the region information using an 8 × 8 pixel block as a processing block.

  On the upper side of FIG. 11, an 8 × 8 pixel block, which is a processing unit used in encoding and decoding by MPEG-2, is shown as a processing block. In addition, on the upper side of FIG. 11, a horizontal line represented by a thick line extending in the left-right direction (horizontal direction) and a vertical direction (vertical direction) extending through the center of an 8 × 8 pixel block which is a processing unit. A vertical line represented by a thick line is shown.

  Note that the 8 × 8 pixel block shown on the upper side of FIG. 11 is a block subjected to DCT conversion / inverse DCT conversion in the process of encoding and decoding by MPEG-2.

  In the upper center of FIG. 11, taps 311 that are nine pixels arranged in the horizontal direction around the target pixel indicated by a white circle are shown. Further, on the lower side of the center of FIG. 11, a tap 312 that is reversed left and right with the target pixel of the tap 311 as the starting point (center) is shown.

  On the lower left side of FIG. 11, a tap 313 that is nine pixels arranged in the vertical direction around the target pixel indicated by a white circle is shown. Furthermore, on the lower right side of FIG. 11, a tap 314 that is inverted up and down with the target pixel of the tap 313 as the starting point (center) is shown.

  For example, an 8 × 8 pixel block is divided into an upper left area, which is an upper left block of 4 × 4 pixels, an upper right area, which is an upper right block of 4 × 4 pixels, and a lower left area of 4 × 4 pixels. The pixel position modes 0 to 15 are assigned to each pixel in the upper left area, for example, in the raster scan order, and is divided into a lower left area that is a block and a lower right area that is a lower right block of 4 × 4 pixels.

  Then, in each pixel in the upper left area, a pixel having a line-symmetric positional relationship with respect to a vertical line or a horizontal line and a pixel having a point-symmetrical positional relationship with respect to the center of an 8 × 8 pixel block are The same value is assigned as the position mode.

  The pixel position mode determination unit 275 determines the pixel position mode of the target pixel based on the position information of the target pixel supplied from the prediction tap extraction unit 123 and supplies the determined pixel position mode to the coefficient storage unit 276.

  The coefficient storage unit 276 outputs the tap coefficient corresponding to the pixel position mode supplied from the pixel position mode determination unit 275 and the class code supplied from the class classification unit 125 to the prediction calculation unit 278.

  Based on the position information of the target pixel supplied from the prediction tap extraction unit 123, the region determination unit 277 sets the target pixel in any one of the upper left region, the upper right region, the lower left region, and the lower right region. The area information indicating whether the position is present is determined and supplied to the prediction calculation unit 278.

  The prediction calculation unit 278 performs a predetermined prediction calculation corresponding to the region information from the region determination unit 277.

That is, when the region information from the region determination unit 277 is information indicating that the target pixel is located in the upper left region, the prediction calculation unit 278 supplies the prediction taps x ₁ and x ₂ supplied from the prediction tap extraction unit 123. ,..., X ₉ and tap coefficients w ₁ , w _2, ..., W ₉ supplied from the coefficient storage unit 276 are multiplied in order, whereby high-quality image data corresponding to the target pixel is obtained. The pixel value (predicted value) of w ₁ x ₁ + w ₂ x ₂ +... + W ₉ x ₉ is obtained and output.

  When the region information from the region determination unit 277 is information indicating that the target pixel is located in the upper right region or the lower right region, the prediction calculation unit 278 displays the prediction tap supplied from the prediction tap extraction unit 123. 11 As shown in the center, the left and right are reversed from the target pixel as a starting point. Then, the prediction calculation unit 278 sequentially multiplies the prediction tap that is reversed left and right with the pixel of interest as the starting point and the tap coefficient supplied from the coefficient storage unit 276, thereby corresponding to the pixel of interest. The pixel value of the data pixel is obtained and output.

Specifically, the prediction tap supplied from the prediction tap extracting unit 123, a tap 311, x ₁ is the pixel value of the pixel corresponding to the number assigned to the tap _{311, x 2, ···, x} 9 In this case, the prediction calculation unit 278 sets the tap 311 to the tap 312 by reversing the left and right from the target pixel, and the pixel values x ₉ , x ₈ ,... Of the pixels corresponding to the numbers assigned to the tap 312. .., X ₁ and tap coefficients w ₁ , w _2, ..., W ₉ supplied from the coefficient storage unit 276 are sequentially multiplied to obtain the pixel of the high-quality image data corresponding to the target pixel. Pixel value (predicted value) w ₁ x ₉ + w ₂ x ₈ +... + W ₉ x ₁ is obtained and output.

  When the region information from the region determination unit 277 is information indicating that the target pixel is located in the lower left region or the lower right region, the prediction calculation unit 278 selects the prediction tap supplied from the prediction tap extraction unit 123. As shown in the lower side of FIG. 11, the image is inverted up and down starting from the target pixel. Then, the prediction calculation unit 278 sequentially multiplies the prediction tap that is turned upside down with the pixel of interest as the starting point and the tap coefficient supplied from the coefficient storage unit 276, thereby corresponding to the pixel of interest. The pixel value of the data pixel is obtained and output.

Specifically, the prediction tap supplied from the prediction tap extraction unit 123 is the lower left tap 313 in FIG. 11, and the pixel values of the pixels corresponding to the numbers assigned to the tap 313 are x ₁ , x ₂ ,. .., X ₉ , the prediction calculation unit 278 sets the tap 313 as a tap 314 by vertically inverting the pixel of interest as a starting point, and the pixel value x ₉ of the pixel corresponding to the number assigned to the tap 314. x _8, · · ·, and x _1, the tap coefficients supplied from the coefficient storage unit 276 w _1, w ₂ ···, and w _9, by multiplying in turn, corresponds to the pixel of interest, high-quality The pixel value (predicted value) w ₁ x ₉ + w ₂ x ₈ +... + W ₉ x ₁ of the pixel of the image data is obtained and output.

  Next, FIG. 12 is a diagram illustrating an example in which a pixel position mode is assigned to pixels of a processing block of 10 × 8 pixels.

  FIG. 12 shows a processing block of 10 × 8 pixels (horizontal × vertical).

  FIG. 12 shows a horizontal line represented by a thick line extending in the left-right direction (horizontal direction) and a thick line extending in the vertical direction (vertical direction) passing through the center of the processing block of 10 × 8 pixels. Vertical lines are shown.

  In the pixel position mode, a block of 10 × 8 pixels is divided into an upper left region which is an upper left block of 5 × 4 pixels and an upper right region which is an upper right block of 5 × 4 pixels by a vertical line and a horizontal line. A pixel is divided into a lower left area which is a lower left block of pixels and a lower right area which is a lower right block of 5 × 4 pixels, and pixel position modes 0 to 19 are assigned to each pixel in the upper left area, for example, in raster scan order. .

  Then, in each pixel in the upper left area, a pixel having a line-symmetric positional relationship with respect to a vertical line or a horizontal line, and a pixel having a point-symmetrical positional relationship with respect to the center of a 10 × 8 pixel block are The same value is assigned as the position mode.

That is, when the region information from the region determination unit 277 is information indicating that the target pixel is located in the upper left region, the prediction calculation unit 278 is supplied from the prediction tap extraction unit 123 as in the case of FIG. prediction tap x _1, x ₂ and, ..., and x _9, the tap coefficients supplied from the coefficient storage unit 276 w _1, w _2, ..., and w _9, is multiplied in order, the pixel of interest The pixel values (predicted values) w ₁ x ₁ + w ₂ x ₂ +... + W ₉ x ₉ of the pixels of the high-quality image data corresponding to are obtained and output.

  When the region information from the region determination unit 277 is information indicating that the pixel of interest is located in the upper right region or the lower right region, the prediction calculation unit 278 performs the same operation as in the case of FIG. The supplied prediction tap is reversed left and right starting from the target pixel. Then, the prediction calculation unit 278 sequentially multiplies the prediction tap that is reversed left and right with the pixel of interest as the starting point and the tap coefficient supplied from the coefficient storage unit 276, thereby corresponding to the pixel of interest. The pixel value of the data pixel is obtained and output.

  When the region information from the region determination unit 277 is information indicating that the target pixel is located in the lower left region or the lower right region, the prediction calculation unit 278, like the case of FIG. The prediction tap supplied from 123 is flipped up and down starting from the target pixel. Then, the prediction calculation unit 278 sequentially multiplies the prediction tap that is turned upside down with the pixel of interest as the starting point and the tap coefficient supplied from the coefficient storage unit 276, thereby corresponding to the pixel of interest. The pixel value of the data pixel is obtained and output.

  FIG. 13 is a diagram illustrating an example in which a pixel position mode is assigned to pixels of a processing block of 11 × 8 pixels (horizontal × vertical).

  FIG. 13 shows a processing block of 11 × 8 pixels, and a portion surrounded by a bold rectangle on the right side of FIG. 13 indicates a pixel on the boundary.

  Further, in FIG. 13, the processing block of 11 × 8 pixels is represented by a thick line extending in the horizontal direction (horizontal direction) passing through the center of the 10 × 8 pixel block excluding the pixels on the boundary. And a vertical line represented by a thick line extending in the vertical direction (vertical direction).

  For example, among the processing blocks of 11 × 8 pixels, a block of 10 × 8 pixels excluding the pixels on the boundary is an upper left region that is a 5 × 4 pixel upper left region by a vertical line and a horizontal line, and 5 × 4 The pixel is divided into an upper right area, which is an upper right block of pixels, a lower left area, which is a lower left block of 5 × 4 pixels, and a lower right area, which is a lower right block of 5 × 4 pixels. Pixel position modes 0 to 19 are assigned in the raster scan order.

  Then, in each pixel in the upper left area, a pixel having a line-symmetric positional relationship with respect to a vertical line or a horizontal line, and a pixel having a point-symmetrical positional relationship with respect to the center of a 10 × 8 pixel block are The same value is assigned as the position mode.

  Among the 11 × 8 pixel processing blocks, the 1 × 8 pixel block that is the upper boundary pixel is a 10 × 8 pixel block that excludes the upper boundary pixel from the 11 × 8 pixel processing block. A pixel position mode value different from the value taken by the pixel position mode assigned to each pixel is assigned.

Here, when the region information from the region determination unit 277 is information indicating that the target pixel is located in the upper left region, the prediction calculation unit 278 supplies the prediction tap extraction unit 123 as in the case of FIG. By multiplying the predicted taps x ₁ , x ₂ ,..., X ₉ and the tap coefficients w ₁ , w ₂ ..., W ₉ supplied from the coefficient storage unit 276 in order. The pixel value (predicted value) w ₁ x ₁ + w ₂ x ₂ +... + W ₉ x ₉ of the pixel of the high-quality image data corresponding to the pixel is obtained and output. Is output.

  When the region information from the region determination unit 277 is information indicating that the pixel of interest is located in the upper right region or the lower right region, the prediction calculation unit 278 performs the same operation as in the case of FIG. The supplied prediction tap is reversed left and right starting from the target pixel. Then, the prediction calculation unit 278 sequentially multiplies the prediction tap that is reversed left and right with the pixel of interest as the starting point and the tap coefficient supplied from the coefficient storage unit 276, thereby corresponding to the pixel of interest. The pixel value of the data pixel is obtained and output.

  When the region information from the region determination unit 277 is information indicating that the target pixel is located in the lower left region or the lower right region, the prediction calculation unit 278, like the case of FIG. The prediction tap supplied from 123 is flipped up and down starting from the target pixel. Then, the prediction calculation unit 278 sequentially multiplies the prediction tap that is turned upside down with the pixel of interest as the starting point and the tap coefficient supplied from the coefficient storage unit 276, thereby corresponding to the pixel of interest. The pixel value of the data pixel is obtained and output.

  In addition to the 11 × 8 pixel processing block shown in FIG. 13 where the upper boundary pixel exists on the right side, there is a processing block where the upper boundary pixel exists on the left side. Can be assigned a pixel position mode.

  In FIGS. 12 and 13, the value of the pixel position mode of the pixel adjacent to the vertical line is set to 4. For example, the pixel position mode of the pixel having the pixel position mode of 4 is set to 4 and the pixel position mode of 4 is set. It may be set to 3 which is the same as the pixel position mode of the pixel adjacent to this pixel. In this case, since the number of pixel position mode modes can be saved, the tap coefficient stored in the coefficient storage unit 276 can be reduced by the tap coefficient corresponding to the pixel position mode to which the value 4 is assigned, The memory capacity of the coefficient storage unit 276 can be saved.

  Next, image conversion processing in the image conversion unit 61 in FIG. 10 will be described with reference to the flowchart in FIG.

  This image conversion processing is started when, for example, output image data of 1920 × 1080 pixels is supplied to the image conversion unit 61 by the decoder 32 of FIG. At this time, output image data is supplied to each of the block extraction unit 121, the prediction tap extraction unit 123, and the class tap extraction unit 124.

  In step S131, the control unit 279 starts control of each block so that the image conversion processing in the image conversion unit 61 is performed for each processing block of output image data.

  Specifically, the control unit 279 controls the block extraction unit 121, and an extraction target block (FIG. 4) of 64 × 16 pixels is shown in FIG. 3 from the output image data output from the decoder 32 of FIG. As shown in FIG. 4, the extracted extraction target blocks are divided into processing target blocks 131, 132, or 133 as shown in FIG.

  In addition, the control unit 279 controls the block extraction unit 121 and causes the threshold calculation unit 122 to supply the divided processing target blocks as appropriate. Further, the control unit 279 controls the threshold value calculation unit 122 to divide the processing target block into processing blocks of 11 × 8 pixels or 10 × 8 pixels. The control unit 279 includes a threshold value calculation unit 122, a prediction tap extraction unit 123, a class tap extraction unit 124, a class classification unit 125, a pixel position mode determination unit 275, a coefficient storage unit 276, a region determination unit 277, or a prediction calculation unit 278. By controlling, image conversion processing is performed for each processing block of output image data.

  The threshold calculation unit 122, the prediction tap extraction unit 123, the class tap extraction unit 124, the class classification unit 125, the pixel position mode determination unit 275, the coefficient storage unit 276, the region determination unit 277, or the prediction calculation unit 278 is controlled by the control unit 279. The image conversion process is performed for each processing block of the output image data.

  That is, after the process of step S131, the process proceeds to step S132. In steps S132 to S139, the same processes as in steps S32 to S39 of FIG. 7 are performed, and the process proceeds to step S140.

  In step S140, the pixel position mode determination unit 275 determines the pixel position mode of the target pixel based on the position information supplied from the prediction tap extraction unit 123, supplies the pixel position mode to the coefficient storage unit 276, and proceeds to step S141. . Note that in step S137, the prediction tap extraction unit 123 extracts the prediction tap and uses the position information of the pixel that is currently the target pixel (for example, the coordinates of the target pixel in the target block) as the pixel position mode determination unit. H.275 is supplied.

  In step S141, the coefficient storage unit 276 stores a coefficient table in which tap coefficients obtained by performing learning processing as described below are registered, and the class code supplied from the class classification unit 125 and the pixel The tap coefficient associated with the pixel position mode supplied from the position mode determination unit 275 is supplied to the prediction calculation unit 278. In step S139, the class classification unit 125 supplies a class code corresponding to the class obtained by the class classification to the coefficient storage unit 276.

  After the process of step S141, the process proceeds to step S142, and the region determination unit 277 determines the region information of the target pixel based on the position information supplied from the prediction tap extraction unit 123 and supplies the region information to the prediction calculation unit 278. Then, the process proceeds to step S143. In step S137, the prediction tap extraction unit 123 extracts the prediction tap and supplies the position determination unit 277 with the position information of the pixel that is the current pixel of interest.

  In step S143, the prediction calculation unit 278 uses the prediction tap supplied from the prediction tap extraction unit 123 and the tap coefficient supplied from the coefficient storage unit 276 to correspond to the region information supplied from the region determination unit 277. By performing the predetermined prediction calculation, the pixel value (predicted value) of the pixel of the high-quality image data corresponding to the target pixel is obtained and output, and the process proceeds to step S144 and is the same as step S42 of FIG. Perform the process.

  Thereafter, the process proceeds to step S145, and the control unit 279 determines whether or not all the processing blocks of the output image data are the target block.

  If it is determined in step S144 that all the processing blocks of the output image data are not the target block, the processing returns to step S132, and the same processing is repeated thereafter.

  On the other hand, if it is determined in step S145 that all the processing blocks of the output image data are the blocks of interest, the image conversion process in FIG. 14 is terminated.

  In the image conversion process of FIG. 14 as described above, the target pixel is classified, and the pixel position mode of the target pixel is determined, so that the tap coefficient is determined based on the class of the target pixel and the pixel position mode. Therefore, a more appropriate tap coefficient is selected as compared with the case where the tap coefficient is selected based on the class of the pixel of interest, so that a pixel value of high-quality image data with less error can be obtained. . That is, mosquito noise and block distortion added to the output image data can be more effectively removed.

FIG. 15 shows a configuration example of a learning device 201 that performs a learning process for obtaining the tap coefficient w _n by solving the normal equation of the equation (8) for each class and pixel position mode (set). .

  In the figure, portions corresponding to those in FIG. 8 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  That is, the learning apparatus 201 in FIG. 15 is provided with an addition unit 340, a coefficient calculation unit 341, or a control unit 342 in place of the addition unit 238, the coefficient calculation unit 239, or the control unit 240 in FIG. Furthermore, the configuration is the same as in the case of FIG. 8 except that a pixel position mode determination unit 339 is newly provided.

  Note that the prediction tap extraction unit 235 supplies pixel position information (for example, the coordinates of the pixel of interest in the block of interest) of the pixel currently being the pixel of interest to the pixel position mode determination unit 339.

  The pixel position mode determination unit 339 determines the pixel position mode of the target pixel by performing the same determination as the pixel position mode determination unit 275 of FIG. 10 based on the position information supplied from the prediction tap extraction unit 235. , And supplied to the adding portion 340.

  The adding unit 340 reads out the pixel (its pixel value) of the teacher image data (learning image data) corresponding to the target pixel from the learning image storage unit 231, and supplies the pixel and the prediction tap extraction unit 235. The class code supplied from the class classification unit 237 and the pixel position mode supplied from the pixel position mode determination unit 339 are added to the pixel of the student image data that forms the prediction tap for the target pixel that has been processed. Do it every time.

That is, the adder 340, (among the pixel, the pixel values of pixels corresponding to the pixel of interest) teacher image data stored in the learning image storage unit 231 y _k, the prediction tap extracting unit 235 outputs Predictive tap (pixel value of pixel of student image data) x _{n, k} , class code indicating class of target pixel output by class classification unit 237, and target pixel output by pixel position mode determination unit 339 Pixel position modes are provided.

Then, the adding unit 340 generates a prediction tap (student image data) x _n, for each class corresponding to the class code supplied from the class classification unit 237 and each pixel position mode supplied from the pixel position mode determination unit 339 _{. Using k} , an operation corresponding to multiplication (x _{n, k} x _{n ′, k} ) between student image data in the matrix on the left side of equation (8) and summation (Σ) is performed.

Furthermore, the adding unit 340 also performs a prediction tap (student image data) x for each class corresponding to the class code supplied from the class classification unit 237 and each pixel position mode supplied from the pixel position mode determination unit 339. _n, using _k and the teacher image data y _k, student image data x _n, multiplication of _k and the teacher image data y _k (x _{n, k} y _k) in the vector of the right side of equation (8) and, summation (sigma ) Is performed.

In other words, the adding unit 340 includes the left-side matrix component (Σx _{n, k} x _{n ′, k} ) and the right-side vector component in Equation (8) previously obtained for the student image data set as the target pixel. (Σx _{n, k} y _k ) is stored in its built-in memory (not shown), and the matrix component (Σx _{n, k} x _{n ′, k} ) or vector component (Σx _{n, k} y) _k ), the corresponding component x _{n, k} calculated using the student image data x _{n, k + 1} and the teacher image data y _{k + 1} for the student image data newly set as the pixel of interest Add ₊₁ x _{n ′, k + 1} or x _{n, k + 1} y _{k + 1} (addition represented by the summation of equation (8) is performed).

  Then, the adding unit 340 performs the above-described addition using all the student image data generated by the noise adding unit 232 as the target pixel, so that each class and pixel position mode is expressed by Expression (8). When the normal equation is established, the normal equation is supplied to the coefficient calculation unit 341.

The coefficient calculation unit 341 obtains the optimum tap coefficient w _n for each class and pixel position mode by solving the normal equation for each class and pixel position mode supplied from the addition unit 340. Output.

  The control unit 342 includes a learning image storage unit 231, a noise addition unit 232, a block extraction unit 233, a threshold calculation unit 234, a prediction tap extraction unit 235, a class tap extraction unit 236, a class classification unit 237, and a pixel position mode determination unit 339. The addition unit 340 or the coefficient calculation unit 341 is controlled.

  That is, the control unit 342 controls each block so that the learning process in the learning device 201 is performed for each processing block of the student image data.

  Next, the learning process in the learning apparatus 201 in FIG. 15 will be described with reference to the flowchart in FIG.

The learning process is, for example, by the user, the image data corresponding to the original image obtained by the broadcast station 31 (FIG. 1), as the learning image data used for learning of the tap coefficient w _n, is supplied to the learning apparatus 201 Will start when. At this time, in the learning apparatus 201, the learning image data is supplied to and stored in the learning image storage unit 231.

  In step S171, the control unit 342 controls each block so that the learning process in the learning device 201 is performed for each processing block of the student image data.

  Specifically, the control unit 342 controls the noise adding unit 232 to read the learning image data from the learning image storage unit 231 and causes the learning image data to be the same as the process performed in FIG. By performing the processing, student image data that is an image corresponding to the output image data is generated.

  That is, the noise adding unit 232 converts the learning image data, which is image data corresponding to the original image obtained at the broadcasting station 31 shown in FIG. 1, into the enlarged MPEG decoded image from the original image in FIG. By performing the same processing, student image data that is an image corresponding to the output image data is generated.

  Further, the control unit 342 controls the block extraction unit 233 to supply the processing target blocks to the threshold value calculation unit 234 sequentially in the raster scan order, for example. In other words, the block extraction unit 233 sequentially extracts predetermined blocks from the student image data supplied from the noise addition unit 232, for example, in the raster scan order, under the control of the control unit 342, so that the blocks shown in FIG. The same extraction target block as the extraction target block extracted by the extraction unit 121 is obtained. In addition, the block extraction unit 233 divides the extracted extraction target block into a plurality of blocks according to the control of the control unit 342, so that the same plurality of processing target blocks divided by the block extraction unit 121 of FIG. For example, are sequentially supplied to the threshold value calculation unit 234 in the raster scan order.

  Thereafter, the control unit 342 controls the threshold calculation unit 234, the prediction tap extraction unit 235, the class tap extraction unit 236, the class classification unit 237, the pixel position mode determination unit 339, the addition unit 340, or the coefficient calculation unit 341. Thus, the learning process is performed for each processing block of the student image data generated by the noise adding unit 232.

  At this time, the threshold calculation unit 234, the prediction tap extraction unit 235, the class tap extraction unit 236, the class classification unit 237, the pixel position mode determination unit 339, the addition unit 340, or the coefficient calculation unit 341 follows the control of the control unit 342. The learning process is performed for each processing block of the student image data generated by the noise adding unit 232.

  That is, the process proceeds from step S171 to step S172. In steps S172 to S179, the same processes as in steps S72 to S79 in FIG. 9 are performed, and the process proceeds to step S180. The pixel position mode determination unit 339 performs the prediction. Based on the position information supplied from the tap extraction unit 235, the pixel position mode of the target pixel is determined by performing the same determination as that of the pixel position mode determination unit 275 in FIG. 10 and supplied to the adding unit 340. In step S177, the prediction tap extraction unit 235 extracts the prediction tap and supplies the pixel position mode determination unit 339 with the position information of the pixel that is the current pixel of interest.

  The process proceeds from step S180 to step S181, and the adding unit 340 reads out the pixel (its pixel value) of the teacher image data, which is the learning image, corresponding to the target pixel from the learning image storage unit 231. The class code supplied from the class classifying unit 237, the pixel position mode, and the pixel and the pixel of the student image data constituting the prediction tap for the target pixel supplied from the prediction tap extracting unit 235 The process is performed for each pixel position mode supplied from the determination unit 339, and the process proceeds to step S182.

  In step S <b> 182, the prediction tap extraction unit 235 determines whether all the pixels of the target block are the target pixels.

  If it is determined in step S182 that all the pixels of the block of interest are not yet the pixels of interest, the process returns to step S176, and the prediction tap extraction unit 235 selects pixels of the block of interest that are not yet the pixels of interest. The process proceeds to step S177, and the same process is repeated thereafter.

On the other hand, if it is determined in step S182 that all the pixels of the target block are the target pixels, the process proceeds to step S183, and the control unit 342 controls all of the student image data generated by the noise adding unit 232. It is determined whether or not the processing block is the target block.

  If it is determined in step S183 that all the processing blocks of the student image data generated by the noise adding unit 232 are not the target block, the processing returns to step S172, and the same processing is repeated thereafter.

On the other hand, when it is determined in step S183 that all the processing blocks of the student image data generated by the noise adding unit 232 are the target block, that is, in the adding unit 340, the noise adding unit 232 generates the processing block. By performing the above-described addition using all student image data as the target pixel, the normal equation shown in Expression (8) is established for each class and pixel position mode, and the normal equation is converted into a coefficient calculation unit. In the case of supplying to 341, the process proceeds to step S 184, and the coefficient calculation unit 341 solves each class and pixel position mode by supplying a normal equation for each class and pixel position mode. and the pixel position mode, determines and outputs the optimum tap coefficients w _n.

  Thereafter, the learning process of FIG. 16 is terminated.

  In the learning process of FIG. 16 as described above, the target pixel of the student image data is classified, the pixel position mode of the target pixel of the student image data is determined, and the class of the target pixel and the pixel position mode are set ) Is learned, so that more variation tap coefficients can be learned as compared to the case where the tap coefficient is learned without determining the pixel position mode of the target pixel. .

  FIG. 17 is a block diagram showing an image processing device 371 that can selectively use the image conversion processing in the image conversion unit 61 of FIG. 2 and the image conversion processing in Patent Documents 1 and 2 described above.

  The image processing device 371 includes a block size detection unit 401, a block size determination unit 402, an image conversion unit 61 in FIG. 2, and an image conversion unit 403.

  In addition, the image processing device 371 outputs, for example, output image data such as an enlarged MPEG decoded image output from the decoder 32 of FIG. 1 or each block of 8 × 8 pixels output from another decoder, for example. Image data obtained by encoding and decoding with MPEG-2 or the like is supplied as input image data. At this time, input image data is supplied to the block size detection unit 401 of the image processing device 371.

  The block size detection unit 401 detects the block size of the input image data supplied thereto, and supplies the block size determination unit 402 together with the input image data.

  Here, the block size of the input image data means, for example, a block size of 10.666 × 8 pixels when the input image data is output image data, and 8 × when the input image data is image data. The size of a block of 8 pixels.

  The block size determination unit 402 determines whether or not the block size supplied from the block size detection unit 401 is a predetermined block size such as 8 × 8 pixels, for example. That is, the block size determination unit 402 determines whether the input image data supplied from the block size detection unit 401 is an image including pixels on the boundary.

  When it is determined that the block size supplied from the block size detection unit 401 is a predetermined block size such as 8 × 8 pixels, that is, the input image data supplied from the block size detection unit 401 determines pixels on the boundary. When it is determined that the image is not included, the block size determination unit 402 supplies the input image data from the block size detection unit 401 to the image conversion unit 403.

  On the other hand, when it is determined that the block size supplied from the block size detection unit 401 is not a predetermined block size such as 8 × 8 pixels (a block size such as 10.666 × 8 pixels), that is, the block size When it is determined that the input image data supplied from the detection unit 401 is an image including pixels on the boundary, the block size determination unit 402 supplies the input image data from the block size detection unit 401 to the image conversion unit 61. To do.

  The image conversion unit 403 performs image conversion processing on the input image data from the block size determination unit 402 using the techniques disclosed in Patent Documents 1 and 2.

  The image conversion unit 61 performs the image conversion process described with reference to FIG. 2 on the input image data from the block size determination unit 402.

  Next, image conversion processing in the image processing apparatus 371 in FIG. 18 will be described with reference to the flowchart in FIG.

  This image conversion processing is performed, for example, on output image data such as an enlarged MPEG decoded image output from the decoder 32 of FIG. 1, or on an MPEG-2 basis for each block of 8 × 8 pixels output from another decoder, for example. This is started when image data obtained by encoding and decoding is supplied to the image processing device 371 as input image data. At this time, input image data is supplied to the block size detection unit 401 of the image processing device 371.

  In step S191, the block size detection unit 401 detects the block size of the input image data supplied thereto, supplies the input image data together with the input image data to the block size determination unit 402, proceeds to step S192, and proceeds to step S192. 402 determines whether the block size supplied from the block size detection unit 401 is a predetermined block size such as 8 × 8 pixels, for example. That is, in step S192, the block size determination unit 402 determines whether or not the input image data supplied from the block size detection unit 401 is an image including pixels on the boundary.

  When it is determined in step S192 that the block size supplied from the block size detection unit 401 is a predetermined block size such as 8 × 8 pixels, that is, the input image data supplied from the block size detection unit 401 is When it is determined that the image does not include the pixel on the boundary, the block size determination unit 402 supplies the input image data from the block size detection unit 401 to the image conversion unit 403, and proceeds to step S193, where the image conversion unit Step 403 performs image conversion processing using the techniques disclosed in Patent Documents 1 and 2 on the input image data from the block size determination unit 402, and the image conversion processing in the image processing device 371 ends.

  On the other hand, when it is determined that the block size supplied from the block size detection unit 401 is not a predetermined block size such as 8 × 8 pixels (a block size such as 10.666 × 8 pixels), that is, the block size When it is determined that the input image data supplied from the detection unit 401 is an image including pixels on the boundary, the block size determination unit 402 supplies the input image data from the block size detection unit 401 to the image conversion unit 61. In step S194, the image conversion unit 61 performs the image conversion process described with reference to FIG. 2 on the input image data from the block size determination unit 402, and the image conversion process in FIG.

  In the image conversion process of FIG. 18 as described above, the block size of the input image data is determined, and an optimal image conversion process is performed according to the block size. Therefore, the input image data having different block sizes is the same. For example, mosquito noise and block distortion added to the input image data are effectively removed as compared with the case where the image conversion process is performed.

  Next, referring to FIG. 19 to FIG. 25, the processing result when the noise removal processing of Patent Document 1 is performed on the output image data having a block size of 10.666 × 8 pixels, and the present invention is applied. The noise removal effect by this invention is demonstrated by showing the process result at the time of performing the noise removal process of patent document 1 which performed.

  FIG. 19 is a diagram illustrating an image obtained when the noise removal process of Patent Document 1 is performed on output image data having a block size of 10.666 × 8 pixels.

  FIG. 20 is a diagram illustrating a waveform of a pixel value of a pixel at a predetermined position of an image obtained when the noise removal process of Patent Document 1 is performed.

  That is, the upper side of FIG. 20 shows an image obtained when the noise removal processing of Patent Document 1 is performed, and the image has a horizontal direction orthogonal to the boundary of the 10.666 × 8 pixel block. An extended arrow is shown.

Further, on the lower side of FIG. 20, a graph represented by a horizontal axis indicating the position of the arrow shown on the upper side of FIG. 20 and a vertical axis indicating the pixel value of the pixel corresponding to the position of the arrow is shown. Yes.
Furthermore, the vicinity of the symbol Y shown in the lower graph of FIG. 20 indicates the vicinity of the boundary of the block of 10.666 × 8 pixels.

  FIG. 21 is a diagram illustrating an image obtained when the noise removal processing of Patent Document 1 to which the present invention is applied is performed on output image data having a block size of 10.666 × 8 pixels.

  FIG. 22 is a diagram illustrating a waveform of a pixel value of a pixel at a predetermined position of an image obtained when the noise removal process of Patent Document 1 to which the present invention is applied is performed.

  That is, on the upper side of FIG. 22, an image obtained when the noise removal processing of Patent Document 1 to which the present invention is applied is shown. The image includes a 10.666 × 8 pixel block boundary. An arrow extending in the left-right direction perpendicular to the arrow is shown.

Further, on the lower side of FIG. 22, a graph represented by a horizontal axis indicating the position of the arrow shown on the upper side of FIG. 22 and a vertical axis indicating the pixel value of the pixel corresponding to the position of the arrow is shown. Yes.
Further, the vicinity of the symbol Y shown in the graph shown in the lower part of FIG. 22 indicates the vicinity of the boundary of the block of 10.666 × 8 pixels.

  FIG. 23 is a diagram in which the graph shown on the lower side of FIG. 20 and the graph shown on the lower side of FIG. 22 are superimposed.

  In FIG. 23, the graph shown in the lower part of FIG. 20 as the graph old (the graph obtained from the image obtained when the noise removal processing of Patent Document 1 is performed) is shown in the lower part of FIG. (A graph obtained from an image obtained when the noise removal processing of Patent Document 1 to which the present invention is applied) is performed.

  In the vicinity of the symbol Y shown in FIG. 23, the graph “old” draws a vertically curved curve, whereas the graph “new” draws a smooth curve. From this, it can be seen that the block distortion due to the block of 10.666 × 8 pixels can be reduced when the noise removal of Patent Document 1 to which the present invention is applied is performed.

  FIG. 24 shows the SNR (signal to noise ratio) when the noise removal process of Patent Document 1 is performed on the output image data, and the SNR when the noise removal process of Patent Document 1 to which the present invention is applied. FIG.

  The first column (column) from the left side of the table shown in FIG. 24 is a column in which output image data having a bit rate of 12 Mbps (megabit per second), which is a target of noise removal processing, is shown.

  Also, the second column from the left side of the table shown in FIG. 24 is a column that shows the SNR when the noise removal processing of Patent Document 1 is performed on the output image data shown in the first column. It is. Further, the third column from the left side of the table shown in FIG. 24 shows the SNR when the noise removal processing of Patent Document 1 to which the present invention is applied is performed on the output image data shown in the first column. Is a column in which is shown.

  Referring to the table shown in FIG. 24, it can be seen that in any case, the SNR of the noise removal process of Patent Document 1 to which the present invention is applied is good.

  FIG. 25 shows another example in which the SNR when the noise removal process of Patent Document 1 is performed on the output image data and the SNR when the noise removal process of Patent Document 1 to which the present invention is applied are compared. FIG.

  The first column (column) from the left side of the table shown in FIG. 25 is a column in which output image data with a bit rate of 15 Mbps, which is a target of noise removal processing, is shown.

  Also, the second column from the left side of the table shown in FIG. 25 is a column that shows the SNR when the noise removal processing of Patent Document 1 is performed on the output image data shown in the first column. It is. Further, the third column from the left side of the table shown in FIG. 25 shows the SNR when the noise removal processing of Patent Document 1 to which the present invention is applied is performed on the output image data shown in the first column. Is a column in which is shown.

  Referring to the table shown in FIG. 25, it can be seen that the SNR of the noise removal process of Patent Document 1 to which the present invention is applied is good in any case as in the case of FIG.

  Next, the series of processing performed by the image conversion unit 61 in FIGS. 2 and 10 described above, the learning device 201 in FIGS. 8 and 15, and the image processing device 371 in FIG. 17 may be executed by dedicated hardware. It can be executed by software. When a series of processing is executed by software, a program constituting the software can execute various functions by installing a so-called embedded computer or various programs. For example, it is installed from a program storage medium in a general-purpose computer or the like.

  FIG. 26 is a block diagram illustrating a configuration example of a computer that executes the above-described series of processing by a program.

  A CPU (Central Processing Unit) 701 executes various processes according to a program stored in a ROM (Read Only Memory) 702 or a storage unit 708. A RAM (Random Access Memory) 703 appropriately stores programs executed by the CPU 701, data, and the like. These CPU 701, ROM 702, and RAM 703 are connected to each other by a bus 704.

  An input / output interface 705 is also connected to the CPU 701 via a bus 704. Connected to the input / output interface 705 are an input unit 706 composed of a keyboard, mouse, microphone, and the like, and an output unit 707 composed of a display, a speaker, and the like. The CPU 701 executes various processes in response to commands input from the input unit 706. Then, the CPU 701 outputs the processing result to the output unit 707.

  A storage unit 708 connected to the input / output interface 705 includes, for example, a hard disk, and stores programs executed by the CPU 701 and various data. A communication unit 709 communicates with an external device via a network such as the Internet or a local area network.

  A program may be acquired via the communication unit 709 and stored in the storage unit 708.

  A drive 710 connected to the input / output interface 705 drives a removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the program or data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 708 as necessary.

  As shown in FIG. 26, a program storage medium that stores a program that is installed in a computer and can be executed by the computer includes a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory), DVD (Digital Versatile Disc, etc.), magneto-optical disc (including MD (Mini-Disc)), or removable media 711, which is a package media composed of semiconductor memory, or the program is temporarily or permanently It is configured by a ROM 702 to be stored, a hard disk constituting the storage unit 708, or the like. The program is stored in the program storage medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 709 that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program storage medium is not limited to the processing performed in time series according to the described order, but is not necessarily performed in time series. Or the process performed separately is also included.

  In the present embodiment, the image conversion unit 61 in FIG. 2, the image conversion unit 61 in FIG. 10, and the image processing apparatus 371 in FIG. 17 have been described as performing the image conversion process. As a specific example of the apparatus to be performed, for example, a television receiver, a hard disk recorder, a DVD recorder, or the like can be employed.

  The present embodiment is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a block diagram explaining the process performed in terrestrial digital broadcasting. It is a block diagram which shows the structural example of one Embodiment of the image conversion part to which this invention is applied. It is a figure which shows the order which the block extraction part 121 of FIG. 2 extracts an extraction object block. FIG. 4 is a diagram illustrating a method in which the block extraction unit 121 in FIG. 2 divides the extraction target block shown in FIG. 3 into processing target blocks. It is a figure explaining the process which the threshold value calculation part 122 of FIG. 2 calculates a threshold value from a process block. It is a figure explaining the class classification performed by the class classification part 125 of FIG. It is a flowchart explaining the image conversion process in the image conversion part 61 of FIG. It is a block diagram which shows the structural example of the learning apparatus 201 which performs the learning process which calculates | requires the tap coefficient used by the image conversion part 61 of FIG. It is a flowchart explaining the learning process in the learning apparatus 201 of FIG. It is a block diagram which shows the other structural example of one Embodiment of the image conversion part to which this invention is applied. It is a figure explaining predetermined prediction calculation corresponding to pixel position mode and area information, using an 8 × 8 pixel block as a processing block. It is a figure which shows an example which allocated pixel position mode to the pixel of the processing block of 10x8 pixel. It is a figure which shows an example which allocated pixel position mode to the pixel of the processing block of 11x8 pixel. It is a flowchart explaining the image conversion process in the image conversion part 61 of FIG. It is a block diagram which shows the structural example of the learning apparatus 201 which performs the learning process which calculates | requires the tap coefficient used in the image conversion part 61 of FIG. It is a flowchart explaining the learning process in the learning apparatus 201 of FIG. FIG. 3 is a block diagram showing an image processing device 371 that can selectively use the image conversion processing in the image conversion unit 61 of FIG. 2 and the image conversion processing in Patent Documents 1 and 2 described above. 19 is a flowchart for describing image conversion processing in the image processing apparatus 371 in FIG. 18. It is a figure which shows the image obtained when the noise removal process of patent document 1 is performed with respect to the output image data whose block size is 10.666x8 pixel. It is a figure which shows the waveform of the pixel value of the pixel in the predetermined position of the image obtained when the noise removal process of patent document 1 was performed. It is a figure which shows the image obtained when the noise removal process of patent document 1 to which this invention is applied with respect to output image data whose block size is 10.666x8 pixel. It is a figure which shows the waveform of the pixel value of the pixel in the predetermined position of the image obtained when performing the noise removal process of patent document 1 to which this invention is applied. FIG. 23 is a diagram in which the graph shown on the lower side of FIG. 20 and the graph shown on the lower side of FIG. 22 are superimposed. It is the figure which compared SNR at the time of performing the noise removal process of patent document 1 with respect to output image data, and SNR at the time of performing the noise removal process of patent document 1 to which this invention is applied. FIG. 25 shows another example in which the SNR when the noise removal process of Patent Document 1 is performed on the output image data and the SNR when the noise removal process of Patent Document 1 to which the present invention is applied are compared. FIG. It is a block diagram which shows the structural example of a computer.

Explanation of symbols

  61 image conversion unit, 121 block extraction unit, 122 threshold calculation unit, 123 prediction tap extraction unit, 124 class tap extraction unit, 125 class classification unit, 126 coefficient storage unit, 127 prediction calculation unit, 201 learning device, 231 learning image Storage unit, 232 noise addition unit, 233 block extraction unit, 234 threshold calculation unit, 235 prediction tap extraction unit, 236 class tap extraction unit, 237 class classification unit, 238 addition unit, 239 coefficient calculation unit, 240 control unit, 275 Pixel position mode determination unit, 276 coefficient storage unit, 277 region determination unit, 278 prediction calculation unit, 279 control unit, 339 pixel position mode determination unit, 340 addition unit, 341 coefficient calculation unit, 342 control unit, 371 image processing apparatus 401 block Size detection unit, 402 block size determination unit, 403 image conversion unit

Claims (26)

The encoded data obtained by encoding the converted image data obtained by converting the number of pixels of the original image data by a predetermined number of times for each predetermined block is decoded for each predetermined block,
Image data obtained by decoding the encoded data is converted to output image data having the same number of pixels as the number of pixels of the original image data by multiplying the number of pixels by a reciprocal of the predetermined number. In an image processing apparatus that processes the output image data output by a decoder,
A plurality of pixels used for predicting pixels of high-quality image data with higher image quality than the output image data corresponding to the target pixel that is a pixel of interest of the output image data are predicted from the output image data A predictive tap extracting means for extracting as a tap;
Class tap extracting means for extracting, as class taps, a plurality of pixels used for class classification that classifies the target pixel into any one of a plurality of classes;
Class classification means for classifying the pixel of interest based on the class tap extracted by the class tap extraction means;
An error between the predicted value obtained by predicting the teacher image data corresponding to the high-quality image data and the teacher image data obtained by a prediction calculation using the student image data corresponding to the output image data and a predetermined coefficient. Coefficient output means for outputting a coefficient corresponding to the class of the target pixel from among the coefficients corresponding to each of the plurality of classes obtained by learning to minimize,
Prediction for obtaining a pixel of the high-quality image data corresponding to the pixel of interest by the prediction calculation using the coefficient output by the coefficient output unit and the prediction tap extracted by the prediction tap extraction unit Image conversion means for performing image conversion processing for converting the output image data into the high-quality image data;
An image processing apparatus comprising: a control unit that controls the image conversion unit so that the image conversion process is performed for each processing block of the output image data having a block size determined based on the predetermined multiple. Pixel determination means for determining whether or not a pixel on the boundary, which is a pixel on the boundary of the reciprocal block obtained by reversing the block size of the predetermined block, is included in the output image data Further comprising
The said image conversion means converts the said output image data into the said high quality image data, when it determines with the said pixel on a boundary being contained in the said output image data by the said pixel determination means. Image processing device. The image processing apparatus according to claim 1, wherein the prediction tap extraction unit extracts the prediction tap from the processing block including the target pixel. The image processing apparatus according to claim 1, wherein the class tap extraction unit extracts the class tap from the processing block including the target pixel. A threshold value calculating means for calculating a threshold value used for the class classification of the target pixel by using a pixel of the output image data;
The image processing according to claim 1, wherein the class classification unit classifies the target pixel based on the class tap extracted by the class tap extraction unit and the threshold calculated by the threshold calculation unit. apparatus. The image processing apparatus according to claim 5, wherein the threshold calculation unit calculates the threshold using a pixel of the processing block including the target pixel. It is determined whether or not a pixel on the boundary, which is a pixel of the output image data, on the boundary of the reciprocal multiple block obtained by making the block size of the predetermined block the reciprocal multiple of the predetermined number is included in the processing block. It further comprises a pixel determination unit on the boundary for determining,
The threshold calculation means, when the boundary pixel determination means determines that the boundary pixel is included in the processing block, the threshold value is a pixel obtained by excluding the boundary pixel from the pixel of the processing block. The image processing apparatus according to claim 6, wherein calculation is performed using Pixel position mode determination means for determining the pixel position mode of the pixel of interest as information indicating the position of each pixel in the processing block as a pixel position mode;
The image processing apparatus according to claim 1, wherein the coefficient output unit outputs a coefficient corresponding to a class of the target pixel and a pixel position mode. The image processing apparatus according to claim 8, wherein, in the pixels of the processing block, the same value is assigned as a pixel position mode to pixels that are in a line-symmetrical or point-symmetrical positional relationship. A region determination unit that determines whether the target pixel is located in any of the plurality of regions when the processing block is divided into a plurality of regions;
The image processing apparatus according to claim 9, wherein the prediction calculation unit obtains a pixel of the high-quality image data corresponding to the target pixel by the prediction calculation corresponding to a region where the target pixel is located. If the processing block includes pixels on the boundary, which are pixels of the output image data, on the boundary of the inverse block obtained by making the block size of the predetermined block an inverse multiple of the predetermined number, the processing block The image processing apparatus according to claim 9, wherein the same value as the pixel position mode is assigned to pixels that are in a line-symmetrical or point-symmetrical positional relationship in the pixels excluding the pixels on the boundary from the block. The image processing device according to claim 11, wherein a value different from pixels other than the pixels on the boundary is assigned to the pixels on the boundary as a pixel position mode. The encoded data obtained by encoding the converted image data obtained by converting the number of pixels of the original image data by a predetermined number of times for each predetermined block is decoded for each predetermined block,
Image data obtained by decoding the encoded data is converted to output image data having the same number of pixels as the number of pixels of the original image data by multiplying the number of pixels by a reciprocal of the predetermined number. In an image processing method for processing the output image data output by a decoder,
A plurality of pixels used for predicting pixels of high-quality image data with higher image quality than the output image data corresponding to the target pixel that is a pixel of interest of the output image data are predicted from the output image data A predictive tap extraction step for extracting as taps;
A class tap extracting step of extracting a plurality of pixels used for class classification that classifies the target pixel into any one of a plurality of classes as a class tap from the output image data;
A class classification step for classifying the pixel of interest based on the class tap extracted by the class tap extraction step;
An error between the predicted value obtained by predicting the teacher image data corresponding to the high-quality image data and the teacher image data obtained by a prediction calculation using the student image data corresponding to the output image data and a predetermined coefficient. A coefficient output step of outputting a coefficient corresponding to the class of the target pixel from the coefficients corresponding to each of the plurality of classes obtained by learning to minimize;
Prediction for obtaining a pixel of the high-quality image data corresponding to the target pixel by the prediction calculation using the coefficient output by the coefficient output step and the prediction tap extracted by the prediction tap extraction step An image conversion step for performing an image conversion process for converting the output image data into the high-quality image data,
And a control step of controlling the image conversion step so that the image conversion processing is performed for each processing block of the output image data having a block size determined based on the predetermined number of times. The encoded data obtained by encoding the converted image data obtained by converting the number of pixels of the original image data by a predetermined number of times for each predetermined block is decoded for each predetermined block,
Image data obtained by decoding the encoded data is converted to output image data having the same number of pixels as the number of pixels of the original image data by multiplying the number of pixels by a reciprocal of the predetermined number. In a program for causing a computer to execute image processing for processing the output image data output by a decoder,
A plurality of pixels used for predicting pixels of high-quality image data with higher image quality than the output image data corresponding to the target pixel that is a pixel of interest of the output image data are predicted from the output image data A predictive tap extraction step for extracting as taps;
A class tap extracting step of extracting a plurality of pixels used for class classification that classifies the target pixel into any one of a plurality of classes as a class tap from the output image data;
A class classification step for classifying the pixel of interest based on the class tap extracted by the class tap extraction step;
An error between the predicted value obtained by predicting the teacher image data corresponding to the high-quality image data and the teacher image data obtained by a prediction calculation using the student image data corresponding to the output image data and a predetermined coefficient. A coefficient output step of outputting a coefficient corresponding to the class of the target pixel from the coefficients corresponding to each of the plurality of classes obtained by learning to minimize;
Prediction for obtaining a pixel of the high-quality image data corresponding to the target pixel by the prediction calculation using the coefficient output by the coefficient output step and the prediction tap extracted by the prediction tap extraction step An image conversion step for performing an image conversion process for converting the output image data into the high-quality image data,
Control processing for controlling the image conversion step so that the image conversion processing is performed for each processing block of the output image data having a block size determined based on the predetermined number of times. A program to be executed. The encoded data obtained by encoding the converted image data obtained by converting the number of pixels of the original image data by a predetermined number of times for each predetermined block is decoded for each predetermined block,
Output image data having the same number of pixels as the number of pixels of the original image data, obtained by multiplying the number of pixels of the image data obtained by decoding the encoded data by a reciprocal of the predetermined number, The coefficient used for the prediction calculation for converting the high-quality image data with higher image quality than the output image data, the result of the prediction calculation using the student image data that is the image corresponding to the output image data, and the image quality In a learning device that is obtained by learning that minimizes an error from teacher image data, which is an image corresponding to image data,
A plurality of pixels used in the prediction calculation for converting a target pixel that is a target pixel of the student image data into a pixel of the teacher image data corresponding to the target pixel is used as a prediction tap from the student image data. A prediction tap extracting means for extracting;
Class tap extracting means for extracting, as class taps, a plurality of pixels used in class classification for classifying the target pixel into any one of a plurality of classes;
Class classification means for classifying the pixel of interest based on the class tap extracted by the class tap extraction means;
For each class of the target pixel classified by the class classification unit, an error between the prediction calculation result using the prediction tap and the pixel of the teacher image data corresponding to the target pixel is minimized. A learning means for performing a learning process for obtaining the coefficient for each class;
A learning apparatus comprising: a control unit that controls the learning unit such that the learning process is performed for each processing block of the student image data having a block size determined based on the predetermined number of times. The learning device according to claim 15, wherein the prediction tap extraction unit extracts the prediction tap from the processing block including the target pixel. The learning device according to claim 15, wherein the class tap extraction unit extracts the class tap from the processing block including the target pixel. A threshold value calculation means for calculating a threshold value used for the class classification of the target pixel by using a pixel of the student image data;
The learning device according to claim 15, wherein the class classification unit classifies the pixel of interest based on the class tap extracted by the class tap extraction unit and the threshold calculated by the threshold calculation unit. . The learning device according to claim 18, wherein the threshold value calculation unit calculates the threshold value using a pixel of the processing block including the target pixel. It is determined whether or not a pixel on the boundary, which is a pixel of the student image data, is included in the processing block on a boundary of an inverse multiple block obtained by making the block size of the predetermined block an inverse multiple of the predetermined number. It further comprises a pixel determination unit on the boundary for determining,
The threshold calculation means, when the boundary pixel determination means determines that the pixel on the pixel is included in the processing block, the pixel obtained by excluding the pixel on the boundary from the pixel of the processing block The learning device according to claim 19, wherein the learning device is calculated using Pixel position mode determination means for determining the pixel position mode of the pixel of interest as information indicating the position of each pixel in the processing block as a pixel position mode;
The coefficient calculation means calculates the coefficient that minimizes an error between a result of the prediction calculation using the prediction tap and a pixel of the teacher image data for each class of pixel of interest and pixel position mode. Item 16. The learning device according to Item 15. The learning device according to claim 21, wherein, in the pixels of the processing block, the same value is assigned as the pixel position mode to pixels that are in a line symmetric or point symmetric positional relationship. When the processing block includes pixels on the boundary, which are pixels of the student image data, on the boundary of the reciprocal multiple block obtained by making the block size of the predetermined block the reciprocal multiple of the predetermined number, the processing The learning device according to claim 22, wherein in the pixel excluding the pixel on the boundary from the block, the same value as the pixel position mode is assigned to a pixel having a line-symmetrical or point-symmetrical positional relationship. The learning device according to claim 23, wherein a value different from pixels other than the pixel on the boundary is assigned to the pixel on the boundary as a pixel position mode. The encoded data obtained by encoding the converted image data obtained by converting the number of pixels of the original image data by a predetermined number of times for each predetermined block is decoded for each predetermined block,
Output image data having the same number of pixels as the number of pixels of the original image data, obtained by multiplying the number of pixels of the image data obtained by decoding the encoded data by a reciprocal of the predetermined number, The coefficient used for the prediction calculation for converting the high-quality image data with higher image quality than the output image data, the result of the prediction calculation using the student image data that is the image corresponding to the output image data, and the image quality In a learning method for obtaining by learning to minimize an error with teacher image data that is an image corresponding to image data,
A plurality of pixels used in the prediction calculation for converting a target pixel that is a target pixel of the student image data into a pixel of the teacher image data corresponding to the target pixel is used as a prediction tap from the student image data. A prediction tap extraction step to extract;
A class tap extracting step of extracting a plurality of pixels used for class classification to classify the pixel of interest into any one of a plurality of classes as a class tap from the student image data;
A class classification step for classifying the pixel of interest based on the class tap extracted by the class tap extraction step;
For each class of the target pixel classified by the class classification step, an error between the prediction calculation result using the prediction tap and the pixel of the teacher image data corresponding to the target pixel is minimized. A coefficient calculating step for calculating the coefficient, and a learning step for performing a learning process for obtaining the coefficient for each class;
And a control step of controlling the learning step so that the learning process is performed for each processing block of the student image data having a block size determined based on the predetermined number of times. The encoded data obtained by encoding the converted image data obtained by converting the number of pixels of the original image data by a predetermined number of times for each predetermined block is decoded for each predetermined block,
Output image data having the same number of pixels as the number of pixels of the original image data, obtained by multiplying the number of pixels of the image data obtained by decoding the encoded data by a reciprocal of the predetermined number, The coefficient used for the prediction calculation for converting the high-quality image data with higher image quality than the output image data, the result of the prediction calculation using the student image data that is the image corresponding to the output image data, and the image quality In a program that causes a computer to execute processing to be obtained by learning that minimizes an error from teacher image data that is an image corresponding to image data,
A plurality of pixels used in the prediction calculation for converting a target pixel that is a target pixel of the student image data into a pixel of the teacher image data corresponding to the target pixel is used as a prediction tap from the student image data. A prediction tap extraction step to extract;
A class tap extracting step of extracting a plurality of pixels used for class classification to classify the pixel of interest into any one of a plurality of classes as a class tap from the student image data;
A class classification step for classifying the pixel of interest based on the class tap extracted by the class tap extraction step;
For each class of the target pixel classified by the class classification step, an error between the prediction calculation result using the prediction tap and the pixel of the teacher image data corresponding to the target pixel is minimized. A coefficient calculating step for calculating the coefficient, and a learning step for performing a learning process for obtaining the coefficient for each class;
Causing a computer to execute a process including a control step of controlling the learning step so that the learning process is performed for each processing block of the student image data having a block size determined based on the predetermined number of times program.

Images