JP4730105B2 - Image processing apparatus and method, learning apparatus and method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus and method, a learning apparatus and method, and a program, and more particularly, to an image processing apparatus and method, a learning apparatus and method, and a program that can obtain a higher-definition image.

  In an imaging apparatus that captures an image using an imaging element such as a CCD (Charge Coupled Devices), a single-plate CCD is often used as the imaging element in order to reduce the size of the imaging apparatus. In this single-plate CCD, each pixel outputs only data of any one of the three primary colors R (red), G (green), and B (blue), and each pixel outputs data of which color. Whether to output is determined by the color filter array arranged on the front surface of the single-plate CCD.

  For example, a pixel in which a G color filter is arranged outputs G component data, but does not output an R component and a B component. In addition, the pixel in which the R color filter is arranged outputs the R component data, but does not output the G component and the B component. Similarly, the pixel in which the B color filter is arranged outputs the B component, but does not output the G component and the R component.

  By the way, when various processes are performed on image data obtained by imaging, RGB color components are required for each pixel. Therefore, some conventional imaging devices obtain image data equivalent to 3-plate CCD output from image data output by a single-plate CCD by class classification adaptive processing (see, for example, Patent Document 1).

JP 2002-64836 A

  However, in the above-described technique, as a prediction tap used for predicting a predetermined color (pixel value) of a pixel of interest in an image corresponding to a three-plate CCD output, When pixels of different colors are used, if the dynamic range of pixels of different colors is too large compared to the dynamic range of pixels of the color to be predicted, the image will break down.

  For example, as shown in FIG. 1, when a G (green) pixel is used as a prediction tap to predict an R (red) pixel value at a predetermined position A, the obtained R pixel value is It becomes a prominent value compared with the pixel value of R at the position. In the figure, the horizontal axis indicates the position in the captured image, and the vertical axis indicates the pixel value (level) of each color at each position.

  A curve 11 shows a value (waveform) of G at each position in the real world, and the value of G increases steeply before and after the position A. The curve 12 shows the R value (waveform) at each position in the real world, and the R value increases gradually before and after the position A as compared with the G value shown in the curve 11. .

  Further, in the drawing, circles and rectangles indicate G pixels (pixel values) and R pixels (pixel values) obtained by imaging, respectively. Here, using the three G pixels indicated by the circles on the left side of FIG. 1 and the two R pixels indicated by the rectangles as prediction taps, the R pixel value at position A is determined by the class classification adaptive processing. Consider the case of prediction.

  In this case, the dynamic range of the G pixel used as the prediction tap is very large compared to the dynamic range of the R pixel used as the prediction tap, and the G pixel value changes rapidly in the vicinity of the position A. Yes. Here, the dynamic range of a pixel means the difference between the maximum value and the minimum value of the pixel values of the pixels constituting the prediction tap.

  The abrupt change in the G pixel value in the vicinity of the position A also affects the prediction of the R pixel value in the position A, and the magnitude of the slope of the R waveform in the position A is equal to the G waveform in the position A. The R pixel value at the position A is predicted so as to be the same as the inclination.

  Therefore, as shown in the right triangle of FIG. 1, the R pixel value at the position A obtained by the prediction is influenced excessively by the surrounding G pixels, and is excessively emphasized. Compared with the pixel value, it protrudes upward in the figure. Therefore, as indicated by the curve 13, only the pixel value at the position A of the R pixel value of the image obtained by the classification adaptation process is larger than the surrounding pixel values.

  Conversely, as a prediction tap used to predict a predetermined color (pixel value) of a pixel of interest in an image corresponding to a three-plate CCD output, a color different from the color of a pixel predicted in a single-plate CCD image If the dynamic range of a pixel of a different color is too small relative to the dynamic range of the pixel of the color to be predicted, the pixel of the different color has little influence on the pixel value of the color to be predicted. For this reason, the pixels of the color to be predicted are not sufficiently emphasized, and the image corresponding to the three-plate CCD output obtained by the class classification adaptive process has no fineness.

  The present invention has been made in view of such a situation, and makes it possible to obtain a high-definition image with higher accuracy.

  An image processing apparatus according to a first aspect of the present invention provides first image data including pixels having a value of a first component of a component video signal and pixels having a value of a second component. An image processing apparatus for converting to second image data composed of pixels having a value of a first component and a value of the second component, the pixel being focused on in the second image data A prediction tap extracting unit that extracts a plurality of pixels used for predicting a target pixel as a prediction tap from the first image data, and a class classification that classifies the target pixel into one of a plurality of classes. Class tap extracting means for extracting a plurality of pixels to be used as class taps from the first image data; class classifying means for classifying the target pixel using the class tap; The prediction tap is normalized so that the dynamic range of the value of the first component of the pixel constituting the prediction tap is equal to the dynamic range of the value of the second component of the pixel constituting the prediction tap. The first component value of the pixel of interest, or the second value using the normalizing means for converting, the tap coefficient obtained in advance for the class of the pixel of interest, and the normalized prediction tap Prediction calculating means for predicting and calculating the value of the component.

  In the normalizing means, when predicting the value of the first component of the pixel of interest, the dynamic range value of the first component value is calculated as the dynamic range value of the second component value. The prediction tap can be normalized by multiplying the value obtained by the division by the value of the second component of the pixels constituting the prediction tap.

  The first component or the second component may be a component representing any one of red, blue, and green.

  The first component or the second component may be a component representing luminance or color difference.

  In the image processing method according to the first aspect of the present invention, the first image data including pixels having a value of a first component of a component video signal and pixels having a value of a second component is An image processing method for converting to second image data composed of pixels having a value of a first component and a value of the second component, the pixel being focused on in the second image data A plurality of pixels used for predicting a target pixel are extracted as prediction taps from the first image data, and a plurality of pixels used for class classification that classifies the target pixel into one of a plurality of classes. , Extracting as a class tap from the first image data, classifying the pixel of interest using the class tap, and dynamics of the value of the first component of the pixels constituting the prediction tap Normalizing the prediction tap so that the clean range is equal to the dynamic range of the value of the second component of the pixels constituting the prediction tap, and tap coefficients that are obtained in advance for the class of the pixel of interest; Predicting the value of the first component or the value of the second component of the pixel of interest using the normalized prediction tap.

  According to a first aspect of the present invention, there is provided a program for storing first image data including pixels having a value of a first component of a component video signal and pixels having a value of a second component. A program for causing a computer to execute image processing for conversion to second image data composed of pixels having the value of the second component and the value of the second component, and paying attention to the second image data A plurality of pixels used for predicting a pixel of interest that is a pixel are extracted as prediction taps from the first image data, and are used for class classification that classifies the pixel of interest into one of a plurality of classes. Are extracted as class taps from the first image data, and the class tap is used to classify the pixel of interest, and there are pixels constituting the prediction tap. The prediction tap is normalized so that the dynamic range of the value of the first component is equal to the dynamic range of the value of the second component of the pixels constituting the prediction tap, and the class of the target pixel Predicting the value of the first component or the value of the second component of the pixel of interest using a tap coefficient determined in advance and the normalized prediction tap.

  In the first aspect of the present invention, first image data composed of pixels having a value of a first component of a component video signal and pixels having a value of a second component is the first image data. In order to predict a pixel of interest that is a pixel of interest of the second image data when converted into second image data composed of pixels having a component value and a value of the second component A plurality of pixels to be used are extracted as prediction taps from the first image data, and a plurality of pixels used for class classification that classifies the pixel of interest into one of a plurality of classes are the first image data. As a class tap, the class tap is used to classify the pixel of interest, and the dynamic range of the value of the first component included in the pixels constituting the prediction tap The prediction tap is normalized so that the dynamic range of the value of the second component of the pixels constituting the prediction tap is equal, and the tap coefficient obtained in advance for the class of the target pixel is The predicted prediction tap is used to predict the value of the first component or the value of the second component of the pixel of interest.

  The learning device according to the second aspect of the present invention is a pixel of interest which is a pixel of interest from first image data composed of pixels having values of a first component and a second component of a component video signal. Pixel-of-interest extracting means for extracting the value of the first component, a plurality of pixels used for predicting the pixel of interest, a pixel having the value of the first component, and a value of the second component Prediction tap extracting means for extracting as a prediction tap from second image data composed of pixels having a plurality of pixels, and a plurality of pixels used for class classification to classify the pixel of interest into one of a plurality of classes, A class tap extraction unit that extracts a class tap from the second image data, a class classification unit that classifies the pixel of interest using the class tap, and the prediction tap Normalization that normalizes the prediction tap so that the dynamic range of the value of the first component of the pixel is equal to the dynamic range of the value of the second component of the pixel constituting the prediction tap Using the means, the value of the first component of the pixel of interest, and the normalized prediction tap, to predict the value of the first component of the pixel of interest from the normalized prediction tap And calculating means for obtaining a tap coefficient corresponding to the class of the target pixel.

  When obtaining the tap coefficient used for predicting the value of the first component of the pixel of interest, the normalizing means uses the dynamic range value of the value of the first component as the second component. The prediction tap can be normalized by multiplying the value obtained by dividing the dynamic range value by the value of the second component of the pixels constituting the prediction tap.

  The first component or the second component may be a component representing any one of red, blue, and green.

  The first component or the second component may be a component representing luminance or color difference.

  The learning method or program according to the second aspect of the present invention is a pixel of interest from first image data composed of pixels having values of a first component and a second component of a component video signal. A plurality of pixels used for extracting the value of the first component of the target pixel and predicting the target pixel are a pixel having the value of the first component and a pixel having the value of the second component. A plurality of pixels extracted from the second image data configured as a prediction tap and used for class classification to classify the pixel of interest into one of a plurality of classes. Extracting as a tap, classifying the pixel of interest using the class tap, the dynamic range of the value of the first component of the pixels constituting the prediction tap, and the prediction Normalizing the prediction tap so that the dynamic range of the value of the second component of the pixels constituting the pixel is equal to the value of the first component of the pixel of interest, and the normalized prediction Determining a tap coefficient corresponding to the class of the target pixel, which is used to predict the value of the first component of the target pixel from the normalized prediction tap using the tap.

  In the second aspect of the present invention, from the first image data composed of pixels having the values of the first component and the second component of the component video signal, the pixel of interest that is the pixel of interest is described above. The value of the first component is extracted, and the plurality of pixels used for predicting the target pixel are composed of pixels having the value of the first component and pixels having the value of the second component. A plurality of pixels that are extracted as prediction taps from the second image data and used for class classification to classify the pixel of interest into one of a plurality of classes, and are extracted from the second image data as class taps. The class tap is used to classify the pixel of interest, the dynamic range of the value of the first component of the pixels constituting the prediction tap, and the prediction tap The prediction tap is normalized so that the dynamic range of the value of the second component included in the constituent pixels is equal, and the value of the first component of the target pixel and the normalized prediction tap are Used to determine the tap coefficient corresponding to the class of the pixel of interest used to predict the value of the first component of the pixel of interest from the normalized prediction tap.

  According to the first aspect of the present invention, a high-definition image can be obtained. In particular, according to the first aspect of the present invention, a high-definition image can be obtained with higher accuracy.

  Further, according to the second aspect of the present invention, a high-definition image can be obtained. In particular, according to the second aspect of the present invention, a high-definition image can be obtained with higher accuracy.

  Embodiments of the present invention will be described below. Correspondences between the constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. Not something to do.

  An image processing apparatus according to a first aspect of the present invention provides first image data including pixels having a value of a first component of a component video signal and pixels having a value of a second component. An image processing device (for example, the imaging device 41 in FIG. 3) that converts to second image data composed of pixels having a value of a first component and a value of the second component, Prediction tap extraction means for extracting a plurality of pixels used for predicting a pixel of interest, which is a pixel of interest of image data, as a prediction tap from the first image data (for example, the prediction tap blocking circuit of FIG. 5). 114) and a class tap extracting means for extracting, as class taps, a plurality of pixels used for class classification that classifies the target pixel into any one of a plurality of classes. For example, the ADRC blocking circuit 111 in FIG. 5, class classification means for classifying the pixel of interest using the class tap (for example, the class classification circuit 113 in FIG. 5), and the prediction tap are configured. Normalizing means for normalizing the prediction tap so that the dynamic range of the value of the first component included in the pixel is equal to the dynamic range of the value of the second component included in the pixel constituting the prediction tap. (For example, the prediction tap normalization unit 115 in FIG. 5), the tap coefficient obtained in advance for the class of the target pixel, and the normalized prediction tap, the first of the target pixel. Prediction calculating means (for example, the adaptive processing circuit 116 in FIG. 5) for predicting and calculating the component value or the second component value is provided.

  In the normalizing means, when predicting the value of the first component of the pixel of interest, the dynamic range value of the first component value is calculated as the dynamic range value of the second component value. The prediction tap can be normalized by multiplying the value obtained by the division by the value of the second component of the pixels constituting the prediction tap (for example, the process of step S49 in FIG. 6). .

  In the image processing method according to the first aspect of the present invention, the first image data including pixels having a value of a first component of a component video signal and pixels having a value of a second component is An image processing method for converting to second image data composed of pixels having a value of a first component and a value of the second component, the pixel being focused on in the second image data A plurality of pixels used for predicting the pixel of interest are extracted as prediction taps from the first image data (for example, step S48 in FIG. 6), and the pixel of interest is classified into one of a plurality of classes. A plurality of pixels used for class classification to be extracted as class taps from the first image data (for example, step S45 in FIG. 6), and class classification of the pixel of interest is performed using the class taps ( For example, step S47 in FIG. 6, the dynamic range of the value of the first component included in the pixels constituting the prediction tap, and the dynamic range of the value of the second component included in the pixels constituting the prediction tap, The prediction taps are normalized so as to be equal to each other (for example, step S49 in FIG. 6), and the attention tap is obtained using the tap coefficient obtained in advance for the class of the attention pixel and the normalized prediction tap. A step of predicting and calculating the value of the first component or the value of the second component of the pixel (for example, step S50 in FIG. 6).

  Note that the program according to the first aspect of the present invention is basically the same processing as the image processing method according to the first aspect of the present invention described above, and is therefore repeated, so that the description thereof is omitted.

  The learning device according to the second aspect of the present invention (for example, the learning device 201 in FIG. 11) is based on first image data composed of pixels having values of the first component and the second component of the component video signal. A pixel-of-interest extracting means (for example, the teacher image blocking circuit 217 in FIG. 11) for extracting the value of the first component of the pixel of interest that is the pixel of interest, and a plurality of pixels used for predicting the pixel of interest Predictive tap extracting means (for example, FIG. 5) extracts a pixel as a prediction tap from second image data composed of a pixel having the value of the first component and a pixel having the value of the second component. 11 prediction tap blocking circuits 215), and a plurality of pixels used for class classification for classifying the pixel of interest into one of a plurality of classes, from the second image data. Class tap extracting means (for example, the ADRC blocking circuit 212 in FIG. 11) and class classifying means for classifying the pixel of interest using the class tap (for example, the class classifying circuit 214 in FIG. 11). ) And the dynamic range of the value of the first component included in the pixels constituting the prediction tap and the dynamic range of the value of the second component included in the pixels constituting the prediction tap are equal to each other. Normalization using normalization means for normalizing the prediction tap (for example, the prediction tap normalization unit 216 in FIG. 11), the value of the first component of the pixel of interest, and the normalized prediction tap An operation for obtaining a tap coefficient corresponding to the class of the target pixel, which is used to predict the value of the first component of the target pixel from the predicted tap that has been performed. And means (e.g., the arithmetic circuit 220 in FIG. 11).

  When obtaining the tap coefficient used for predicting the value of the first component of the pixel of interest, the normalizing means uses the dynamic range value of the value of the first component as the second component. The prediction tap is normalized by multiplying the value obtained by dividing the value of the dynamic range by the value of the second component of the pixel constituting the prediction tap (for example, FIG. 12). Step S86).

  The learning method or program according to the second aspect of the present invention is a pixel of interest from first image data composed of pixels having values of a first component and a second component of a component video signal. The value of the first component of the pixel of interest is extracted (for example, step S87 in FIG. 12), and a plurality of pixels used for predicting the pixel of interest are pixels having the value of the first component; Extracted as prediction taps from second image data composed of pixels having the value of the second component (for example, step S85 in FIG. 12), and classifying the pixel of interest into one of a plurality of classes A plurality of pixels used for class classification are extracted as class taps from the second image data (for example, step S82 in FIG. 12), and the class tap is used to select the pixel of interest. Classification (for example, step S84 in FIG. 12), the dynamic range of the value of the first component included in the pixels constituting the prediction tap, and the second component included in the pixel constituting the prediction tap. The prediction tap is normalized so that the dynamic range of the value becomes equal (for example, step S86 in FIG. 12), the value of the first component of the pixel of interest, and the normalized prediction tap are used. Obtaining a tap coefficient corresponding to the class of the target pixel used to predict the value of the first component of the target pixel from the normalized prediction tap (for example, step S90 in FIG. 12); Including.

  The present invention is applied to a digital video camera for capturing moving images, a digital still camera for capturing still images, video equipment such as a camera-integrated VTR (Video Tape Recorder), an image processing apparatus used in broadcasting services, a printer, a scanner, and the like. Can be applied.

  Embodiments to which the present invention is applied will be described below with reference to the drawings.

  FIG. 2 shows the principle of processing in which the imaging apparatus to which the present invention is applied generates higher-definition image data from the output of a single-plate CCD.

  As shown in FIG. 2, in the imaging apparatus, n × m R (red) image data, n × m (n × m (vertical 6 × horizontal 8 in FIG. 2) image data output) m G (green) image data and n × m B (blue) image data are respectively generated by direct calculation. In other words, the imaging apparatus converts the image data output from the single-plate CCD into image data corresponding to a three-plate CCD output composed of R image data, G image data, and B image data.

  A color filter array is arranged on the front surface of the single CCD of the imaging apparatus, and the color arrangement of the color filter array is, for example, a color arrangement called a Bayer arrangement as shown on the left side of FIG. In FIG. 2, the G color filters are arranged in a checkered pattern, and the R color filters and the B color filters are alternately arranged for each line in the remaining portion.

  In this way, the imaging apparatus converts image data composed of pixels having an R pixel value, pixels having a G pixel value, and pixels having a B pixel value output from a single-plate CCD into R pixels. Is converted into image data equivalent to a three-plate CCD output composed of pixels having a value, a pixel value of G, and a pixel value of B.

  FIG. 3 is a block diagram illustrating a configuration example of an imaging apparatus that captures a subject and generates high-definition image data according to the principle described above.

  The lens 51 of the imaging device 41 condenses the light from the subject and makes the collected light enter the single-plate CCD 54 via the iris 52 and the color filter array 53. That is, the lens 51 forms an image of the incident light on the CCD 54. The color filter array 53 is composed of color filters assigned to each pixel, and is arranged on the front surface (the left side in the figure) of the CCD 54.

  The CCD 54 photoelectrically converts incident light, accumulates electric charges obtained by the photoelectric conversion, and uses the accumulated electric charges as an image signal that is an analog signal as an AGC (Automatic Gain Control) / CDS (Correlated Double Sampling) circuit 55. To supply.

  The AGS / CDS circuit 55 adjusts and outputs the gain so that the magnitude (level) of the supplied image signal is constant, and removes 1 / f noise generated in the CCD 54. Further, the AGC / CDS circuit 55 performs electronic shutter processing for electrically changing the charge accumulation time in the CCD 54 based on the control of a main CPU (Central Processing Unit) 59. The image signal output from the AGC / CDS circuit 55 is input to an A / D (Analog / Digital) conversion unit 56, converted from an analog signal to image data that is a digital signal, and then input to an image signal processing circuit 57. Is done. The image signal processing circuit 57 performs predetermined processing such as defect correction processing, digital clamp processing, white balance adjustment processing, gamma correction processing, and interpolation processing using class classification adaptation processing on the input image data.

  The timing generator 58 generates various timing signals based on the control of the main CPU 59, and supplies them to the CCD 54, the AGC / CDS circuit 55, the A / D converter 56, the main CPU 59, and the like.

  The motor 60 drives the iris 52 based on the control of the main CPU 59 and controls the amount of light incident on the CCD 54 from the lens 51. The motor 61 is controlled by the main CPU 59 and controls the lens 51 to control the focus state of the lens 51 with respect to the CCD 54. The light emitting unit 62 is controlled by the main CPU 59 and irradiates a predetermined flash to the subject during imaging.

  The recording medium interface 63 stores the image data output from the image signal processing circuit 57 in the memory 64 as necessary, executes a predetermined interface process, and then supplies the image data to the recording medium 65 for recording. The recording medium 65 includes, for example, a non-volatile flash memory, a magnetic disk, an optical disk, a magneto-optical disk, and the like, and is detachable from the imaging device 41.

  The controller 66 controls the image signal processing circuit 57 and the recording medium interface 63 under the control of the main CPU 59. In addition, various commands are input to the main CPU 59 from the operation unit 70 including shutter buttons, switches, and the like in accordance with the operation of the photographer. The power supply unit 67 includes a battery 68 and a DC / DC converter 69, and the DC / DC converter 69 converts power from the battery 68 into a DC voltage having a predetermined value and supplies it to each unit of the imaging device 41. . The rechargeable battery 68 can be attached to and detached from the imaging device 41.

  Next, with reference to the flowchart of FIG. 4, a process in which the imaging device 41 images a predetermined subject and causes the recording medium 65 to record image data obtained by the imaging will be described.

  In step S <b> 11, the lens 51 condenses the light from the subject, and causes the collected light to enter the CCD 54 via the iris 52 and the color filter array 53. The main CPU 59 controls the motor 60 to drive the iris 52 to adjust the light amount from the lens 51 incident on the CCD 54 to a predetermined value, and also controls the motor 61 to adjust the position of the lens 51. Perform focus control.

  In step S12, the main CPU 59 determines whether or not the shutter button has been operated by the photographer. For example, when the imager confirms the angle of view by viewing an image displayed on a monitor or finder (not shown) provided in the imaging device 41 and operates the operation unit 70 as a shutter button, the operation unit 70 performs main operation. The CPU 59 is supplied with a signal indicating that the shutter button has been operated. The main CPU 59 determines that the shutter button has been operated when a signal indicating that the shutter button has been operated is supplied.

  If it is determined in step S12 that the shutter button has not been operated, the process returns to step S11, and the subsequent processes are repeated.

  On the other hand, if it is determined in step S12 that the shutter button has been operated, the process proceeds to step S13. At this time, the main CPU 59 drives the light emitting unit 62 to generate a flash and irradiate the subject, and controls the timing generator 58 to generate various timing signals. The timing generator 58 supplies the generated timing signal to the CCD 54, the AGC / CDS circuit 55, the A / D converter 56, and the main CPU 59.

  In step S13, the CCD 54 photoelectrically converts (converts into charges) the light incident from the lens 51 in synchronization with the timing signal from the timing generator 58, and uses the analog signal obtained thereby as an image signal as an AGC / CDS. This is supplied to the circuit 55.

  In step S14, the AGC / CDS circuit 55 performs processing for removing the 1 / f noise component of the image signal supplied from the CCD 54, and then adjusts the gain so that the magnitude of the image signal is constant, The signal is supplied to the A / D converter 56.

  In step S15, the A / D converter 56 converts the image signal supplied from the AGC / CDS circuit 55 from an analog signal into image data that is a digital signal, and the image data obtained by the conversion is converted into an image signal processing circuit 57. To supply.

  In step S <b> 16, the image signal processing circuit 57 performs image signal processing on the image data supplied from the A / D conversion unit 56. Although details will be described later with reference to the flowchart of FIG. 6, image signal processing such as defect correction processing, digital clamp processing, white balance adjustment processing, gamma correction processing, class classification adaptation processing, and the like is performed. The image signal processing circuit 57 temporarily stores the image data subjected to the image signal processing in the memory 64 as necessary, and supplies the image data to the recording medium 65 via the recording medium interface 63.

  In step S17, the recording medium 65 records the image data supplied from the recording medium interface 63, and the imaging process ends. Note that the image data obtained by imaging may be output in a predetermined signal format to another device connected to the imaging device 41 without being recorded on the recording medium 65.

  In this manner, the imaging device 41 images the subject and records the image data obtained by the imaging on the recording medium 65.

  By the way, the image signal processing circuit 57 that performs image signal processing on image data is configured as shown in FIG. 5, for example.

  The image signal processing circuit 57 includes a defect correction circuit 101, a clamp circuit 102, a white balance circuit 103, a gamma correction circuit 104, an interpolation processing unit 105, a correction circuit 106, and an RGB matrix circuit 107.

  The image signal (image data) output from the single CCD 54 (FIG. 3) is supplied to the defect correction circuit 101 via the A / D converter 56 (FIG. 3). Here, the image data supplied to the defect correction circuit 101 is, for example, a component video signal whose components are R (red), G (green), and B (blue). The defect correction circuit 101 detects a defect component such as a component corresponding to a pixel that does not react to light in the CCD 54 or a component corresponding to a pixel that always holds a charge, and performs A / D based on the detection result. Defect correction processing for correcting the image data supplied from the conversion unit 56 is performed.

  Further, when an image signal is read, if a value smaller than a value included in a signal indicating a pixel value of each pixel of the image is detected, it is determined that reading of the image signal for one line of the image is completed. Since the minimum value of the signal indicating the pixel value is 0, when a negative value is detected, it is determined that reading of the image signal for one line of the image has been completed. Therefore, the A / D conversion unit 56 is a digital signal that is an analog image signal with the signal value slightly shifted in the positive direction in order to prevent the negative value of the image signal from being cut. Convert to image data. The clamp circuit 102 performs clamp processing for restoring the shift amount in the positive direction of the image data supplied from the defect correction circuit 101.

  The white balance circuit 103 corrects the respective gains of the R, G, and B color signals as image data supplied from the clamp circuit 102. The gamma correction circuit 104 corrects the value of each color signal supplied from the white balance circuit 103 according to the gamma curve.

  The interpolation processing unit 105 converts the image data supplied from the gamma correction circuit 104 into image data corresponding to a three-plate CCD output by performing a class classification adaptive process. Further, the interpolation processing unit 105 supplies the image data obtained by the conversion to the correction circuit 106.

  The interpolation processing unit 105 includes an ADRC (Adaptive Dynamic Range Control) blocking circuit 111, an ADRC processing circuit 112, a class classification circuit 113, a prediction tap blocking circuit 114, a prediction tap normalization unit 115, an adaptive processing circuit 116, and a coefficient memory. 117.

  The ADRC blocking circuit 111 converts the image data output from the single-plate CCD 54 and obtains image data equivalent to the 3-plate CCD output (the image data equivalent to the 3-plate CCD output is the image to be obtained from now on). Pixels that are data and are virtually assumed because they do not exist at this stage) are sequentially set as target pixels, and some of the pixels that form the image data supplied from the gamma correction circuit 104 are the target pixels. Is extracted as a class tap for classifying into one of a plurality of classes.

  Further, the ADRC blocking circuit 111 supplies the extracted class tap to the ADRC processing circuit 112.

  The ADRC processing circuit 112 performs ADRC processing on the class tap supplied from the ADRC blocking circuit 111 and supplies the class tap subjected to ADRC processing to the class classification circuit 113. The class classification circuit 113 classifies the target pixel based on the class tap from the ADRC processing circuit 112 (using the class tap), and supplies a class code (class number) indicating the classified class to the adaptive processing circuit 116. To do.

  The prediction tap blocking circuit 114 extracts some of the pixels constituting the image data supplied from the gamma correction circuit 104 as prediction taps used to predict the pixel value of the target pixel, and a prediction tap normalization unit 115.

  The prediction tap normalization unit 115 normalizes the prediction tap supplied from the prediction tap blocking circuit 114 and supplies the normalized prediction tap to the adaptive processing circuit 116.

  The adaptive processing circuit 116 reads the tap coefficient corresponding to the class code supplied from the class classification circuit 113 from the coefficient memory 117, and extracts the tap coefficient from the pixel value of the pixels constituting the prediction tap supplied from the prediction tap normalization unit 115. And a necessary pixel value is predicted and calculated.

  The coefficient memory 117 stores a tap coefficient for each class used for predicting the pixel value of the target pixel (a tap coefficient determined in advance for each class). For example, the coefficient memory 117 predicts the tap coefficient for predicting the R pixel value of the target pixel classified into each class, the tap coefficient for predicting the G pixel value, and the B pixel value. The tap coefficients are stored in advance.

  The correction circuit 106 performs correction processing necessary for visually enhancing the image such as edge enhancement on the image data supplied from the adaptive processing circuit 116 of the interpolation processing unit 105. The RGB matrix circuit 107 outputs the R, G, and B color signals as image data supplied from the correction circuit 106 as they are, or multiplies each color signal by a predetermined conversion matrix, The image data is converted into image data of a predetermined signal format such as YUV method (signal method using luminance Y, color difference U, and color difference V as components) and output.

  Next, image signal processing performed by the image signal processing circuit 57 in the process of step S16 of FIG. 4 will be described with reference to the flowchart of FIG.

  In step S <b> 41, the defect correction circuit 101 performs defect correction processing for correcting any defective pixels on the image data supplied from the A / D conversion unit 56, and the image data subjected to the defect correction processing. Is supplied to the clamp circuit 102. For example, the defect correction circuit 101 replaces the pixel value of a pixel that does not respond to light, or a pixel that always has a charge with the average value of the pixel values of adjacent pixels, thereby replacing the image data. Correct.

  In step S42, the clamp circuit 102 performs a clamp process for correcting the offset value in the A / D conversion unit 56 on the image data supplied from the defect correction circuit 101, and the image data subjected to the clamp process. Is supplied to the white balance circuit 103.

  In step S43, the white balance circuit 103 performs white balance adjustment processing on the image data supplied from the clamp circuit 102, and sets the levels of the R, G, and B color signals as image data to appropriate white levels. Adjust to a level that can be expressed. Then, the white balance circuit 103 supplies the image data subjected to the white balance adjustment process to the gamma correction circuit 104.

  In step S <b> 44, the gamma correction circuit 104 performs gamma correction processing on the image data supplied from the white balance circuit 103, and the image data subjected to the gamma correction processing is converted into the ADRC blocking circuit 111 of the interpolation processing unit 105 and the image data. This is supplied to the prediction tap blocking circuit 114.

  In step S45, the ADRC blocking circuit 111 divides the image data supplied from the gamma correction circuit 104 into p × q blocks (where p and q are positive integers), and classifies each divided block as a class. Extract taps. The ADRC blocking circuit 111 supplies the extracted class tap to the ADRC processing circuit 112.

  For example, as shown in FIG. 7A, when the position of the target pixel is a position corresponding to the G pixel indicated by the arrow K11, the ADRC blocking circuit 111 displays the upper left corner of the G pixel indicated by the arrow K11 in the figure. , Four G pixels adjacent to the lower left, upper right, and lower right, two G pixels adjacent to each other through the R pixel in the vertical direction from the G pixel indicated by the arrow K11, and B in the horizontal direction A total of nine G pixels composed of two G pixels adjacent to each other through the pixel and the G pixel indicated by the arrow K11 are extracted as class taps.

  Also, for example, as shown in FIG. 7B, when the position of the target pixel is a position corresponding to the G pixel indicated by the arrow K12, the ADRC blocking circuit 111 displays the G pixel indicated by the arrow K12 in the drawing. 4 G pixels adjacent to the upper left, lower left, upper right, and lower right, two G pixels adjacent to each other via the B pixel in the vertical direction from the G pixel indicated by the arrow K12, and the horizontal direction A total of nine G pixels consisting of two G pixels adjacent to each other through R pixels and a G pixel indicated by an arrow K12 are extracted as class taps.

  Returning to the description of the flowchart of FIG. 6, when class taps are extracted, in step S46, the ADRC processing circuit 112 performs ADRC processing on the class taps supplied from the ADRC blocking circuit 111, and the resulting ADRC code is obtained. Is supplied to the class classification circuit 113 as a class tap subjected to ADRC processing.

  For example, in the K-bit ADRC processing, the ADRC processing circuit 112 detects the maximum value MAX and the minimum value MIN of the pixel values constituting the class tap, and detects the maximum value MAX and the minimum value MIN of the detected pixel values. The difference DR (DR = MAX−MIN) is a local dynamic range of a set of pixels constituting the class tap. Based on the dynamic range DR, the ADRC processing circuit 112 requantizes the pixels (the pixel values thereof) constituting the class tap into K bits. That is, the ADRC processing circuit 112 subtracts the minimum value MIN from the pixel value of each pixel constituting the class tap, and divides (quantizes) the subtracted value by DR / 2K.

  Then, the ADRC processing circuit 112 outputs, as an ADRC code, a bit string in which the pixel values of the K-bit pixels constituting the class tap obtained as described above are arranged in a predetermined order. Therefore, for example, when a class tap is subjected to 1-bit ADRC processing, the pixel value of each pixel constituting the class tap is the difference between the maximum value MAX and the minimum value MIN after the minimum value MIN is subtracted. Divided by 1/2 (rounded down), the pixel value of each pixel is made 1 bit (binarized). A bit string in which the 1-bit pixel values are arranged in a predetermined order is output as an ADRC code.

  In step S47, the class classification circuit 113 performs class classification based on the ADRC code supplied from the ADRC processing circuit 112. That is, the class classification circuit 113 determines a class corresponding to the ADRC code obtained by ADRC processing the class tap, generates a class code indicating the class, and supplies the generated class code to the adaptive processing circuit 116.

  In step S48, the prediction tap blocking circuit 114 divides the image data supplied from the gamma correction circuit 104 into p × q blocks (where p and q are positive integers), and from each of the divided blocks. Extract prediction taps. The prediction tap blocking circuit 114 supplies the extracted prediction tap to the prediction tap normalization unit 115.

  For example, when the position corresponding to the G pixel indicated by the arrow K11 is the position of the target pixel as shown in FIG. 7A, the prediction tap blocking circuit 114 is indicated by the arrow K11 as shown in FIG. 8A. 5 × 5 pixels around the G pixel including the G pixel to be extracted are extracted as prediction taps. In this case, the pixel of the prediction tap includes an R pixel, a G pixel, and a B pixel.

  As shown in FIG. 7B, when the position corresponding to the G pixel indicated by the arrow K12 is the position of the target pixel, as shown in FIG. 8B, the prediction tap blocking circuit 114 is indicated by the arrow K12. 5 × 5 pixels around the G pixel including the G pixel to be extracted are extracted as prediction taps. Also in this case, as in the case of FIG. 8A, the pixels of the prediction tap include an R pixel, a G pixel, and a B pixel.

  Returning to the description of the flowchart of FIG. 6, when the prediction tap is extracted, the prediction tap normalization unit 115 normalizes the prediction tap supplied from the prediction tap blocking circuit 114 in step S <b> 49. The prediction tap is supplied to the adaptive processing circuit 116.

  For example, as shown in FIG. 9, consider the case where the position of the target pixel is the position of the pixel G33 and the R pixel value of the target pixel is predicted. Note that one square in FIG. 9 represents one pixel, each of the pixels G11 to G55 represents a G pixel, each of the pixels B12 to B54 represents a B pixel, and each of the pixels R21 to R21 Each of the pixels R45 represents an R pixel.

  Here, assuming that the pixels constituting the prediction tap for the target pixel are 5 × 5 pixels centering on the pixel G33 (that is, all the pixels shown in FIG. 9), the prediction taps include pixels G11 to G55. A total of 13 G pixels, a total of 6 B pixels of pixels B12 to B54, and a total of 6 R pixels of pixels R21 to R45 are included.

  In this way, when predicting the R pixel value, when the prediction tap includes G pixels and B pixels of colors different from R, as described with reference to FIG. 1, the R pixels If the dynamic range of the G pixel or the B pixel is too large with respect to the dynamic range, the image may be broken.

  Therefore, the prediction tap normalization unit 115 normalizes the prediction tap so that the dynamic range of the G pixel and the dynamic range of the B pixel that constitute the prediction tap are equal to the dynamic range of the R pixel.

  Specifically, first, the prediction tap normalization unit 115 calculates the dynamic range DR (R) of the R pixel by calculating Expression (1).

  DR (R) = Rmax−Rmin (1)

  Here, Rmax and Rmin in Equation (1) represent the maximum value and the minimum value among the pixel values of the R pixels that constitute the prediction tap, respectively. Therefore, for example, in the example of FIG. 9, the maximum value and the minimum value of the pixel values of the six R pixels of the pixels R21 to R45 constituting the prediction tap are Rmax and Rmin, respectively, and DR ( R) is a difference value between the maximum value and the minimum value.

  Similarly, the prediction tap normalization unit 115 calculates the dynamic range DR (G) of the G pixel and the dynamic range DR of the B pixel that form the prediction tap by calculating Expressions (2) and (3). (B) is obtained.

  DR (G) = Gmax−Gmin (2)

  DR (B) = Bmax−Bmin (3)

  Here, Gmax and Gmin in Expression (2) represent the maximum value and minimum value of the pixel values of the G pixels constituting the prediction tap, respectively, and Bmax and Bmin in Expression (3) are respectively predicted. It represents the maximum value and the minimum value among the pixel values of the B pixels constituting the tap.

  Next, the prediction tap normalization unit 115 calculates Equation (4) using the obtained dynamic range DR (R) and DR (G) of the R pixel and the G pixel, and forms G that constitutes the prediction tap. Normalize the pixel values of the pixels.

  GNij = Gij × (DR (R) / DR (G)) (4)

  Here, Gij represents the pixel value of the G pixel Gij (where 1 ≦ i ≦ 5, 1 ≦ j ≦ 5) in FIG. 9, and GNij represents the normalized pixel value of the pixel Gij. Yes. Therefore, for example, the prediction tap normalization unit 115 normalizes the pixel values of the pixels G11 to G55 by calculating Expression (4).

  Similarly, the prediction tap normalization unit 115 calculates Equation (5) using the obtained dynamic ranges DR (R) and DR (B) of the R pixel and the B pixel, and forms the prediction tap. Normalize the pixel values of the pixels.

  BNij = Bij × (DR (R) / DR (B)) (5)

  Here, Bij represents the pixel value of the B pixel Bij (where 1 ≦ i ≦ 5, 1 ≦ j ≦ 5) in FIG. 9, and BNij represents the normalized pixel value of the pixel Bij. Yes.

  When the prediction tap normalization unit 115 normalizes the pixel value of the G pixel and the pixel value of the B pixel, the pixel value of the R pixel, the pixel value of the normalized G pixel, and the normalized pixel value Let the pixel value of the pixel of B be the pixel value of the pixel which comprises the normalized prediction tap.

  In addition, when predicting the B pixel value of the target pixel, the prediction tap normalization unit 115 sets the dynamic range DR (B) of the B pixel to the pixel value of the G pixel constituting the prediction tap. The pixel value of the G pixel is normalized by multiplying the value divided by the dynamic range DR (G). Further, the prediction tap normalization unit 115 multiplies the pixel value of the R pixel by the value obtained by dividing the dynamic range DR (B) of the B pixel by the dynamic range DR (R) of the R pixel, and Normalize the pixel value of the pixel. Then, the prediction tap normalization unit 115 configures the normalized prediction tap by using the normalized pixel value of the R pixel, the normalized pixel value of the G pixel, and the pixel value of the B pixel. The pixel value of the pixel.

  Similarly, when predicting the G pixel value of the target pixel, the prediction tap normalization unit 115 sets the dynamic range DR (G) of the G pixel to the B pixel value as the pixel value of the B pixel constituting the prediction tap. The pixel value of the B pixel is normalized by multiplying the value divided by the dynamic range DR (B). Further, the prediction tap normalization unit 115 multiplies the pixel value of the R pixel by the value obtained by dividing the dynamic range DR (G) of the G pixel by the dynamic range DR (R) of the R pixel, and Normalize the pixel value of the pixel. Then, the prediction tap normalization unit 115 configures the normalized prediction tap with the normalized pixel value of the R pixel, the pixel value of the G pixel, and the normalized pixel value of the B pixel. The pixel value of the pixel.

  In this way, by normalizing the prediction tap, for example, as shown in FIG. 10, in order to predict the R pixel value at the predetermined position B, the G pixel may be used even if the G pixel is used as the prediction tap. Can be made equal to the dynamic range of the R pixel. In the figure, the horizontal axis indicates the position in the captured image, and the vertical axis indicates the pixel value (level) of each color component at each position.

  A curve 141 indicates the G value (waveform) at each position in the real world, and the G value increases steeply before and after the position B. A curve 142 indicates the R value (waveform) at each position in the real world, and the R value increases gradually before and after the position B as compared to the G value indicated by the curve 141. .

  Further, in the drawing, circles and squares indicate G pixels (pixel values) and R pixels (pixel values) constituting the prediction tap. Here, using the three G pixels indicated by circles in the left side of FIG. 10 and the two R pixels indicated by squares as prediction taps, the pixel value of R at position B is obtained by class classification adaptive processing. Consider the case of prediction.

  In this case, the dynamic range of the G pixel used as the prediction tap is very large compared to the dynamic range of the R pixel used as the prediction tap, and the G pixel value changes rapidly in the vicinity of the position B. Yes.

  Therefore, as described above, when the pixel value of the G pixel is normalized, the normalized G pixel value (waveform) is obtained before and after the position B as shown by the curve 143 in the right diagram of FIG. The curve has almost the same slope as the curve 142 and is a gently increasing curve. As described above, when the G pixel value is normalized, the normalized dynamic range of the G pixel becomes equal to the dynamic range of the R pixel, and the predicted attention is not caused without causing the image to fail. The pixel value of the pixel is a value that is moderately emphasized.

  Returning to the description of the flowchart of FIG. 6, when the prediction tap is normalized, in step S <b> 50, the adaptive processing circuit 116 reads the tap coefficient corresponding to the class code supplied from the class classification circuit 113 from the coefficient memory 117, By multiplying the normalized prediction tap from the prediction tap normalization unit 115 by the tap coefficient, the pixel value of the target pixel (for example, R, G, or B pixel value) is predicted and calculated. The adaptive processing circuit 116 obtains the pixel value of the pixel of interest by prediction calculation, generates image data corresponding to the three-plate CCD output, and supplies the image data to the correction circuit 106.

  In step S51, the interpolation processing unit 105 determines whether or not the class classification adaptation processing has been completed for all blocks.

  If it is determined in step S51 that the class classification adaptation process has not been completed for all blocks, the process returns to step S45, and the subsequent processes are repeated.

  On the other hand, if it is determined in step S51 that the class classification adaptive processing has been completed for all the blocks, the processing proceeds to step S52, and the correction circuit 106 performs image processing on the image data from the adaptive processing circuit 116. Correction processing (so-called image creation) is performed to make the image look good visually, and the image data subjected to the correction processing is supplied to the RGB matrix circuit 107.

  In step S53, the RGB matrix circuit 107 converts the color space such as converting the R, G, and B color signals as image data supplied from the correction circuit 106 into YUV image data as necessary. The image data obtained by the conversion process is supplied to the recording medium interface 63, and the image signal process ends.

  In this way, the image signal processing circuit 57 normalizes the prediction tap extracted from the image data, and performs the class classification adaptation process using the normalized prediction tap.

  In this way, by performing the classification adaptation process by normalizing the prediction tap, the dynamic range of the pixels of each color of the prediction tap can be made equal, and the failure of the image can be suppressed. As a result, higher-definition image data equivalent to a three-plate CCD output can be generated with higher accuracy from the image data output from the single-plate CCD 54.

  In addition, the class classification adaptive processing can obtain R image data, G image data, and B image data as image data equivalent to a three-plate CCD output, thereby increasing the sharpness of edge portions and details. , The S / N evaluation value can be improved, and a clearer image can be obtained. 7 and 8 show an example of the class tap and the prediction tap for the target pixel. However, the positions of the pixels constituting the class tap or the prediction tap are arbitrary and are determined to be most efficient.

  Next, prediction calculation in the adaptive processing circuit 116 in FIG. 5 and learning of tap coefficients stored in the coefficient memory 117 will be described.

  For example, as the class classification adaptive processing, a prediction tap is extracted from image data output from a single-plate CCD 54, and an image pixel corresponding to a three-plate CCD output (hereinafter, appropriately, using the prediction tap and the tap coefficient). Let us consider obtaining (predicting) a pixel value of a high-quality pixel) by a predetermined prediction calculation.

  If, for example, a linear primary prediction calculation is adopted as the predetermined prediction calculation, the pixel value y of the high-quality pixel is obtained by a linear primary expression shown in Expression (6).

・・・（６）

... (6)

However, in Expression (6), x _i is a pixel of the image data output from the i-th single-plate CCD 54 that constitutes a prediction tap for the pixel value y of the high-quality pixel (hereinafter referred to as a low-quality pixel as appropriate). It represents a pixel value of a designated), w _i represents the i th i th tap coefficient multiplied with the low-quality pixel (pixel value of). In equation (6), the prediction tap is composed of N low image quality pixels x ₁ , x ₂ ,..., X _N.

  Here, the pixel value y of the high-quality pixel can be obtained not by the linear primary expression shown in Expression (6) but by an expression of a higher-order linear function of the second or higher order. Further, not only the linear function but also the pixel value of the high image quality pixel may be obtained by a non-linear function.

Now, when the true value of the pixel value of the high-quality pixel of the s-th sample is expressed as y _s and the predicted value of the true value y _s obtained by the equation (6) is expressed as y _s ′, the prediction error _es Is represented by the following equation (7).

e _s = (y _s −y _s ′) (7)

Now, since the predicted value y _s ′ of equation (7) is obtained according to equation (6), the following equation (8) is obtained by replacing y _s ′ of equation (7) according to equation (6). It is done.

・・・（８）

... (8)

In Equation (8), x _{s, i} represents the i-th low-quality pixel that constitutes the prediction tap for the high-quality pixel of the s-th sample.

Prediction error e _s the tap coefficient w _i to 0 in Equation (8) (or formula (7)) is, is the optimal for predicting the high-quality pixel, for all the high-quality pixel, such In general, it is difficult to obtain a large tap coefficient w _i .

Therefore, as a standard indicating that the tap coefficient w _i is optimum, for example, when the least square method is adopted, the optimum tap coefficient w _i is expressed by the square error represented by the following equation (9). It can be obtained by minimizing the sum E.

・・・（９）

... (9)

However, in the equation (9), S constitutes a high image quality pixel y _s, the prediction taps for the high-quality pixel y _s low-quality pixels _{x s, 1, x s,} 2, ···, x s _{, N} represents the number of samples (the number of learning samples).

The minimum value (minimum value) of the sum E of square errors in equation (9) is given by w _i , which is 0 as a result of partial differentiation of the sum E by the tap coefficient w _i as shown in equation (10).

・・・（１０）

... (10)

On the other hand, when the above equation (8) is partially differentiated by the tap coefficient w _i , the following equation (11) is obtained.

・・・（１１）

(11)

  From the equations (10) and (11), the following equation (12) is obtained.

・・・（１２）

(12)

To e _s of formula (12), by substituting equation (8), equation (12) can be represented by normal equations shown in equation (13).

・・・（１３）

... (13)

Further, if X _{i, j} and Y _i are respectively defined by Expression (14) and Expression (15) (where 1 ≦ i ≦ N, 1 ≦ j ≦ N), Expression (13) can be expressed by Expression (16). Can be represented.

・・・（１４）

(14)

・・・（１５）

... (15)

・・・（１６）

... (16)

The normal equation of Expression (16) can be solved for the tap coefficient w _i by using, for example, a sweeping method (Gauss-Jordan elimination method) or the like.

By solving the normal equation of Equation (16) for each class, the optimum tap coefficient w _i (here, the tap coefficient w _i that minimizes the sum E of square errors) can be obtained for each class. it can.

In the interpolation processing unit 105 of FIG. 5, by using the tap coefficient w _i for each class as described above, the calculation of Expression (6) is performed, so that the three-plate CCD is obtained from the image data output from the single-plate CCD 54. Image data corresponding to the output can be obtained.

Next, FIG. 11 shows a configuration example of a learning apparatus that performs learning for obtaining the tap coefficient w _i by solving the normal equation of Expression (16) for each class.

  The learning device 201 includes a thinning circuit 211, an ADRC blocking circuit 212, an ADRC processing circuit 213, a class classification circuit 214, a prediction tap blocking circuit 215, a prediction tap normalizing unit 216, a teacher image blocking circuit 217, an arithmetic circuit 218, A learning data memory 219, an arithmetic circuit 220, and a coefficient memory 221 are included.

The learning device 201 is input with image data equivalent to a 3-plate CCD output (hereinafter also referred to as teacher image data) used for learning the tap coefficient w _i . In the learning device 201, the teacher image data is supplied to the thinning circuit 211 and the teacher image blocking circuit 217.

  The thinning circuit 211 performs thinning processing for thinning out pixels from the teacher image data according to the arrangement of each color in the color filter array. This thinning process is performed by applying a filter assuming a color filter array 53 (optical low-pass filter) disposed in front of the CCD 54 of the imaging device 41 (FIG. 3).

  Further, the thinning circuit 211 converts the image data equivalent to the single-plate CCD output (hereinafter also referred to as student image data) obtained by performing the thinning process on the teacher image data into the ADRC blocking circuit 212 and the prediction tap block. Is supplied to the circuit 215.

  The ADRC blocking circuit 212 extracts a predetermined pixel from the student image data supplied from the thinning circuit 211 as a class tap from the student image data supplied from the thinning circuit 211 while taking correspondence with the target pixel in the teacher image data. To do. Further, the ADRC blocking circuit 212 supplies the extracted class tap to the ADRC processing circuit 213.

  The ADRC processing circuit 213 performs ADRC processing on the class tap supplied from the ADRC blocking circuit 212 and supplies the class tap subjected to ADRC processing to the class classification circuit 214. The class classification circuit 214 classifies the target pixel based on the class tap from the ADRC processing circuit 213 and supplies a class code (class number) indicating the classified class to the arithmetic circuit 218.

  The prediction tap blocking circuit 215 uses a predetermined pixel among the pixels constituting the student image data as a prediction tap from the student image data supplied from the thinning circuit 211 while taking correspondence with the target pixel in the teacher image data. Extract. Further, the prediction tap blocking circuit 215 supplies the extracted prediction tap to the prediction tap normalization unit 216.

  The prediction tap normalization unit 216 normalizes the prediction tap supplied from the prediction tap blocking circuit 215 and supplies the normalized prediction tap to the arithmetic circuit 218.

  The teacher image blocking circuit 217 extracts the target pixel (the pixel value) from the teacher image data by sequentially setting the pixel constituting the teacher image data as the target pixel while taking correspondence with the class tap in the student image data. This is supplied to the arithmetic circuit 218.

  The arithmetic circuit 218 takes the correspondence between the target pixel supplied from the teacher image blocking circuit 217 and the normalized prediction tap from the prediction tap normalization unit 216, and the pixel constituting the target pixel and the prediction tap, Is added according to the class code from the class classification circuit 214.

That is, the pixel value y _{s of} the target pixel of the teacher image data, the prediction tap (the pixel value of the pixel of the student image data constituting the pixel) x _{s, i} , and the class code representing the class of the target pixel are supplied to the arithmetic circuit 218 Is done.

Then, the arithmetic circuit 218 uses the normalized prediction tap (student image data) x _{s, i} for each class corresponding to the class code, and multiplies the student image data between the matrixes on the left side of Expression (13) (x _{s, i} x _{s, i} ) and a calculation corresponding to the calculation of the sum of the multiplied values.

Further, the arithmetic circuit 218 uses the prediction tap (student image data) x _{s, i} and the teacher image data y _s for each class corresponding to the class code, and uses the student image data x _s in the vector on the right side of Expression (13). _{, i} and teacher image data y _s (x _{s, i} y _s ), and a calculation corresponding to the calculation of the sum of the multiplied values.

That is, the arithmetic circuit 218 lastly calculates the left-side matrix component (Σx _{s, i} x _{s, i} ) and the right-side vector component (Σx _{s, i} y _s ), and the newly selected pixel of interest for the matrix component (Σx _{s, i} x _{s, i} ) or vector component (Σx _{s, i} y _s ) the image data is calculated using the teacher image data y _{s + 1} and the student data x _{s + 1, i,} corresponding component _{x s + 1, i x s} + 1, i or x _{s + 1, i} Add y _{s + 1} (addition represented by Σ in equation (13)).

  Then, the arithmetic circuit 218 establishes (generates) the normal equation shown in the equation (13) for each class by performing the above addition using all the pixels of the teacher image data as the target pixel, The normal equation is supplied to the learning data memory 219.

  The learning data memory 219 sequentially reads and stores normal equations (matrix coefficients thereof) obtained by addition by the arithmetic circuit 218. The learning data memory 219 supplies the stored normal equation to the arithmetic circuit 220.

The arithmetic circuit 220 solves the normal equation for each class supplied from the learning data memory 219 to obtain the optimum tap coefficient w _i for each class and supplies it to the coefficient memory 221. The coefficient memory 221 stores the tap coefficient w _i of each class supplied from the arithmetic circuit 220.

  Next, the learning process by the learning device 201 will be described with reference to the flowchart of FIG. This learning process is started when image data corresponding to a 3-plate CCD output as teacher image data is supplied to the thinning circuit 211 and the teacher image blocking circuit 217 of the learning device 201.

  In step S <b> 81, the thinning circuit 211 thins out pixels from the teacher image data, generates image data corresponding to a single-plate CCD output as student image data, and supplies the image data to the ADRC blocking circuit 212 and the prediction tap blocking circuit 215.

  In step S82, the ADRC blocking circuit 212 divides the student image data supplied from the thinning circuit 211 into p × q blocks (where p and q are positive integers), and classifies each divided block as a class. Extract taps. The ADRC blocking circuit 212 supplies the extracted class tap to the ADRC processing circuit 213.

  In step S83, the ADRC processing circuit 213 performs ADRC processing on the class tap supplied from the ADRC blocking circuit 212, and supplies the ADRC code obtained as a result to the class classification circuit 214 as a class tap subjected to ADRC processing. To do.

  In step S 84, the class classification circuit 214 classifies the target pixel based on the class tap from the ADRC processing circuit 213, and supplies a class code indicating the classified class to the arithmetic circuit 218.

  In step S85, the prediction tap blocking circuit 215 divides the student image data supplied from the thinning-out circuit 211 into p × q blocks (where p and q are positive integers), and from each of the divided blocks. Extract prediction taps. The prediction tap blocking circuit 215 supplies the extracted prediction tap to the prediction tap normalization unit 216.

  In step S86, the prediction tap normalization unit 216 normalizes the prediction tap supplied from the prediction tap blocking circuit 215 and supplies the normalized prediction tap to the arithmetic circuit 218. For example, as described with reference to FIG. 9, the prediction tap normalization unit 216 equals the dynamic range of pixels of a predetermined color for predicting the tap coefficient of the target pixel and the dynamic range of pixels of other colors. The prediction tap is normalized so that

  In step S87, the teacher image blocking circuit 217 sequentially sets the pixels constituting the teacher image data as the target pixels while taking correspondence with the class taps in the student image data, and from the teacher image data, the target pixels (R, G of , And B pixel values) are extracted and supplied to the arithmetic circuit 218.

  In step S88, the arithmetic circuit 218 determines the target pixel and the prediction tap while taking the correspondence between the target pixel supplied from the teacher image blocking circuit 217 and the normalized prediction tap from the prediction tap normalization unit 216. For the constituent pixels, addition to the normal equation shown by the equation (13) established for the class corresponding to the class code from the class classification circuit 214 is performed.

  In step S89, the arithmetic circuit 218 determines whether or not all blocks have been added.

  If it is determined in step S89 that addition has not been performed for all blocks, the process returns to step S88, and addition is performed for blocks that have not yet been added.

  On the other hand, if it is determined in step S89 that the addition has been performed for all the blocks, the arithmetic circuit 218 supplies the normal equation established for each class to the learning data memory 219 for storage. Then, the learning data memory 219 supplies the stored normal equation to the arithmetic circuit 220, and the process proceeds to step S90.

In step S90, the arithmetic circuit 220 obtains the tap coefficient w _i of each class by solving the normal equation for each class supplied from the learning data memory 219 by, for example, a sweeping method, and supplies the tap coefficient w _i to the coefficient memory 221. Thereby, each of the tap coefficient w _i used for predicting each of the R pixel value, the G pixel value, and the B pixel value of each class is obtained.

In step S91, the arithmetic circuit 220 determines whether tap coefficients w _i have been obtained for all classes.

If it is determined in step S91 that the tap coefficients w _i have not been obtained for all classes, the process returns to step S90, the normal equation of the class for which tap coefficients w _i have not yet been obtained is solved, and the tap coefficient w is determined. _i is required.

On the other hand, if it is determined in step S91 that the tap coefficients w _i have been obtained for all classes, the process proceeds to step S92, and the coefficient memory 221 receives the tap coefficient w of each class supplied from the arithmetic circuit 220. _i is stored, and the learning process ends.

The coefficient memory 117 of FIG. 5 stores tap coefficients w _i for each of a plurality of sets of classes obtained as described above.

In this way, the learning apparatus 201 normalizes the prediction tap extracted from the image data corresponding to the single-plate CCD output as the student image data, and obtains the tap coefficient w _i of each class using the normalized prediction tap. .

In this way, the prediction tap extracted from the image data corresponding to the single-plate CCD output is normalized so that the dynamic ranges of the pixels of each color are equal, and the tap coefficient w _i of each class is calculated using the normalized prediction tap. By obtaining the tap coefficient w _i for performing the class classification adaptive processing with higher accuracy, the tap coefficient w _i can be obtained. As a result, when the class classification adaptive processing is performed using the obtained tap coefficient w _i , it is possible to suppress the failure of the image, and from the image data output from the single-plate CCD, a higher-definition three-plate CCD Image data corresponding to output can be obtained with higher accuracy.

  In the above description, the Bayer array shown in FIG. 13A is used as an example of the color filter array 53 that transmits the primary color (R, G, B) components of light. In addition, the interline shown in FIG. 13B is used. An arrangement, a G stripe RB checkered arrangement shown in FIG. 13C, a G stripe RB complete checkered arrangement shown in FIG. 13D, a primary color difference arrangement shown in FIG. 13E, or the like.

  In the interline arrangement shown in FIG. 13B, G color filters are arranged in a checkered pattern, and R color filters and B color filters are alternately arranged in the remaining part of each row.

  Further, in the G stripe RB checkered arrangement shown in FIG. 13C, every other column of G color filters is arranged in the vertical direction in the figure, and the R color filter and B color are arranged in the vertical direction of the remaining columns. Are alternately arranged. At this time, the R color filter or the B color filter in each column is arranged so as to be adjacent to the same color filter in the horizontal direction via the G color filter.

  Further, in the G stripe RB complete checkered arrangement shown in FIG. 13D, the G color filters are arranged in the vertical direction in every other row in the figure, and the R color filters and the B color filters are arranged in the vertical direction of the remaining columns. Color filters are arranged alternately. At this time, the R color filter and the B color filter in each row are arranged so as to be adjacent to the B color filter and the R color filter through the G color filter in the horizontal direction. Yes.

  Further, in the primary color difference array shown in FIG. 13E, in the first to third rows, the G color filters are arranged in the vertical direction every other column in the figure, and the vertical direction of the remaining columns. In addition, R color filters and B color filters are alternately arranged. At this time, the R color filter and the B color filter in each column are arranged so as to be adjacent to the same color filter through the G color filter in the horizontal direction. In the fourth row, R color filters and G color filters are alternately arranged.

  Further, the color filter array 53 may be a complementary color filter instead of the primary color filter. In this case, each pixel of the image data obtained by imaging has one value of yellow (Ye), cyan (Cy), magenta (Mg), and green (G).

  Furthermore, in the above description, an example of generating image data equivalent to a three-plate CCD output having the same resolution from image data equivalent to a single-plate CCD output has been described. It is also possible to generate image data equivalent to the three-plate CCD output.

  Further, for example, when obtaining tap coefficients for generating image data equivalent to a three-plate CCD output with a quadruple resolution from image data equivalent to a single-plate CCD output, the learning apparatus 201 has a configuration as shown in FIG. 14A. , R image data, G image data, and B image data corresponding to a three-plate CCD output are input and used as teacher image data.

  Then, the image data corresponding to the three-plate CCD output shown in FIG. 14A is thinned out by the low-pass filter in the thinning circuit 211, and becomes image data having a resolution of 1/4 as shown in FIG. 14B. Each of the R image data, the G image data, and the B image data illustrated in FIG. 14B has a resolution that is ¼ of each of the R image data, the G image data, and the B image data illustrated in FIG. 14A. Image data.

  Further, the image data shown in FIG. 14B is further thinned out by the thinning circuit 211 to form Bayer array image data corresponding to the single-plate CCD output as shown in FIG. 14C. Is student image data.

  Furthermore, as shown in FIG. 15A, each of the pixels corresponding to the respective positions of the four areas having the same area in the pixel of the image data corresponding to the single-plate CCD output is set as the target pixel, and the pixel value of each target pixel is set as the target pixel. A tap coefficient for obtaining is obtained.

  In FIG. 15A, among the pixels of the image data corresponding to the single-plate CCD output, the pixels corresponding to the positions of the four regions having the same area in the G pixel indicated by the arrow K41 are the target pixel 251-1 to the target pixel 251 respectively. -4. Here, the prediction taps for each of the target pixel 251-1 to the target pixel 251-4 are, for example, 3 × 3 pixels centered on the G pixel indicated by the arrow K 41.

  Then, as shown in FIG. 15B, the R pixel value, the G pixel value, and the B pixel value of each of the target pixels 251-1 to 251-4 of the image data corresponding to the three-plate CCD output, and the prediction Based on the taps, tap coefficients for predicting the R pixel values of the target pixels 251-1 to 251-4, respectively, and the G coefficients of the target pixels 251-1 to 251-4, respectively. A total of 12 tap coefficients are obtained for each of the tap coefficients for predicting the pixel value and for each of the tap coefficients for predicting the B pixel value of each of the target pixel 251-1 to target pixel 251-4. .

  In this case, the arithmetic circuit 218 calculates the normal equation represented by the equations (17), (18), and (19) for each color component of R, G, and B corresponding to the equation (16). Stand up and add.

・・・（１７）

... (17)

・・・（１８）

... (18)

・・・（１９）

... (19)

Here, each of _{wr1, i} , _{wr2, i} , _{wr3, i} , and _{wr4, i} in Equation (17) (where 1 ≦ i ≦ N) is the pixel of interest 251-1 to the pixel of interest 251. -4 represents each i-th tap coefficient that is used to predict each R-pixel value of −4 and is multiplied by (i.e., the pixel value of) the i-th pixel constituting the prediction tap.

In addition, each of R _{1, i} , R _{2, i} , R _{3, i} , and R _{4, i} in Formula (17) (where 1 ≦ i ≦ N) is Σx _s, the pixel value y _s in _i y _s, represent those pixel values of the respective R of the pixel of interest 251-1 to the pixel of interest 251-4.

Similarly, each of w _{g1, i} , w _{g2, i} , w _{g3, i} , and w _{g4, i} (where 1 ≦ i ≦ N) in Expression (18) is the target pixel 251-1 to the target pixel 251. -4 represents each i-th tap coefficient that is used to predict each G pixel value of −4 and is multiplied by (the pixel value of) the i-th pixel that forms the prediction tap.

In addition, each of G _{1, i} , G _{2, i} , G _{3, i} , and G _{4, i in} the formula (18) (where 1 ≦ i ≦ N) is Σx _s, the pixel value y _s in _i y _s, represent those the pixel value of each of the G pixel of interest 251-1 to the pixel of interest 251-4.

Furthermore, each of w _{b1, i} , w _{b2, i} , w _{b3, i} , and w _{b4, i in} the equation (19) (where 1 ≦ i ≦ N) is the target pixel 251-1 to the target pixel 251- 4 represents each i-th tap coefficient that is used to predict each of the four B pixel values and is multiplied by (the pixel value of) the i-th pixel constituting the prediction tap.

In addition, each of B _{1, i} , B _{2, i} , B _{3, i} , and B _{4, i} in equation (19) (where 1 ≦ i ≦ N) is represented by Σx _s, the pixel value y _s in _i y _s, represent those the pixel value of each B pixel of interest 251-1 to the pixel of interest 251-4.

  The arithmetic circuit 220 solves the normal equations of the equations (17) to (19) established in this way, thereby obtaining the R pixel value of each of the target pixels 251-1 to 251-4, Twelve tap coefficients for predicting each of the pixel value and the B pixel value are obtained.

  Further, by using these tap coefficients, the adaptive processing circuit 116 generates image data corresponding to a three-plate CCD output (4 times dense image data) with a quadruple resolution from image data corresponding to a single-plate CCD output. can do.

  By the way, in order to evaluate the effect of the above embodiment, the inventors performed a simulation. An image corresponding to a single-plate CCD output is created from an image corresponding to a three-plate CCD output, and tap coefficients are generated by an algorithm that performs the same processing as the processing performed by the learning device 201 described above. Then, a process for converting an image corresponding to a single-plate CCD output into an image corresponding to a three-plate CCD output is generated by interpolation using a class classification adaptive process that normalizes prediction taps using the generated tap coefficients.

  An image corresponding to a 3-plate CCD output generated as a result of such a simulation and an image corresponding to a 3-plate CCD output generated by performing the conventional classification adaptation process were evaluated, and the result of the simulation was generated. In the image corresponding to the three-plate CCD output, it was confirmed that the image breakdown caused by the difference in the dynamic range of each color component was improved in all the generated images. Therefore, the method of generating the image corresponding to the 3-plate CCD output by performing the class classification adaptive processing for normalizing the prediction tap is the method of generating the image corresponding to the 3-plate CCD output by performing the conventional class classification adaptive processing. It can be said that there is an advantage.

  As described above, by normalizing the prediction taps used for the class classification adaptation process, the dynamic ranges of the pixels of each color of the prediction tap can be made equal, and the failure of the image can be suppressed. Thereby, higher-definition image data can be generated more accurately.

  Also, taps for performing class classification adaptation processing more accurately by normalizing prediction taps so that the dynamic ranges of pixels of each color are equal and obtaining tap coefficients for each class using the normalized prediction taps A coefficient can be obtained. Thereby, when the class classification adaptive process is performed using the obtained tap coefficient, it is possible to suppress the failure of the image and to obtain higher-definition image data with higher accuracy.

  In the above, the number of classes into which the target pixel is classified is reduced by performing ADRC processing on the extracted class taps. However, DCT (Discrete Cosine Transform), VQ (Vector Quantization) (vector quantum) ), DPCM (Differential Pulse Code Modulation), BTC (Block Trancation Coding), nonlinear quantization, etc., the number of classes into which the pixel of interest is classified may be reduced.

  In addition, the image data has been described as component video signals having RGB as components, but other component video signals having luminance Y, color difference U, and color difference V as components may be used. Furthermore, the image data may have at least one value of yellow (Ye), cyan (Cy), magenta (Mg), and green (G) as the pixel value of each pixel.

  Furthermore, although it has been described that an image is picked up using a CCD as the image pickup device, for example, a CMOS (Complementary Mental Oxide Semiconductor) or the like can be used as the image pickup device for picking up an image.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 16 is a block diagram showing an example of the configuration of a personal computer that executes the above-described series of processing by a program. The CPU 311 of the personal computer 301 executes various processes according to a program recorded in a ROM (Read Only Memory) 312 or a recording unit 318. A RAM (Random Access Memory) 313 appropriately stores programs executed by the CPU 311 and data. The CPU 311, ROM 312, and RAM 313 are connected to each other via a bus 314.

  An input / output interface 315 is also connected to the CPU 311 via the bus 314. The input / output interface 315 is connected to an input unit 316 including a keyboard, a mouse, and a microphone, and an output unit 317 including a display and a speaker. The CPU 311 executes various processes in response to commands input from the input unit 316. Then, the CPU 311 outputs the processing result to the output unit 317.

  The recording unit 318 connected to the input / output interface 315 includes, for example, a hard disk, and records programs executed by the CPU 311 and various data. The communication unit 319 communicates with an external device via a network such as the Internet or a local area network.

  A program may be acquired via the communication unit 319 and recorded in the recording unit 318.

  The drive 320 connected to the input / output interface 315 drives a removable medium 331 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the program or data recorded therein. Get etc. The acquired program and data are transferred to the recording unit 318 and recorded as necessary.

  As shown in FIG. 16, a program recording medium for storing a program that is installed in a computer and can be executed by the computer is a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory), DVD (Digital Versatile Disc) (including magneto-optical disk), or removable media 331 which is a package medium made of semiconductor memory, or ROM 312 in which a program is temporarily or permanently stored, The recording unit 318 is configured by a hard disk and the like. The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 319 that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program recording medium is not limited to the processing performed in time series in the described order, but is not necessarily performed in time series. Or the process performed separately is also included.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a figure for demonstrating the failure of the image which arises when a pixel of a different color is used as a prediction tap. It is a figure explaining the principle which obtains the image data equivalent to a 3 plate CCD output from the output of a single plate CCD. It is a block diagram which shows the structural example of an imaging device. It is a flowchart explaining an imaging process. It is a block diagram which shows the structural example of an image signal processing circuit. It is a flowchart explaining an image signal process. It is a figure which shows an example of a class tap. It is a figure which shows an example of a prediction tap. It is a figure for demonstrating normalization of a prediction tap. It is a figure for demonstrating normalization of a prediction tap. It is a block diagram which shows the structural example of a learning apparatus. It is a flowchart explaining a learning process. It is a figure which shows the example of the other arrangement | sequence of a color filter array. It is a figure explaining the principle of the learning process which calculates | requires the tap coefficient for obtaining the image data equivalent to the 3 plate CCD output of 4 times the resolution from the output of a single plate CCD. It is a figure for demonstrating the attention pixel in the case of calculating | requiring the tap coefficient which produces | generates the image data of 4 times resolution. And FIG. 16 is a block diagram illustrating a configuration example of a personal computer.

Explanation of symbols

  41 imaging device, 54 CCD, 57 image signal processing circuit, 65 recording medium, 111 ADRC blocking circuit, 113 class classification circuit, 114 prediction tap blocking circuit, 115 prediction tap normalization unit, 116 adaptive processing circuit, 117 coefficient memory , 201 learning device, 211 decimation circuit, 212 ADRC blocking circuit, 214 class classification circuit, 215 prediction tap blocking circuit, 216 prediction tap normalization unit, 218 arithmetic circuit, 220 arithmetic circuit, 301 personal computer, 311 CPU, 312 ROM, 313 RAM, 318 recording unit, 331 removable media

Claims (12)

First image data composed of pixels having the value of the first component of the component video signal and pixels having the value of the second component is obtained by using the values of the first component and the second component. In an image processing apparatus for converting to second image data composed of pixels having values,
A prediction tap extracting means for extracting a plurality of pixels used for predicting a target pixel which is a target pixel of the second image data as a prediction tap from the first image data;
Class tap extracting means for extracting, as class taps, a plurality of pixels used for class classification that classifies the target pixel into any of a plurality of classes;
Class classification means for classifying the pixel of interest using the class tap;
The prediction tap is set so that the dynamic range of the value of the first component included in the pixels constituting the prediction tap is equal to the dynamic range of the value of the second component included in the pixels constituting the prediction tap. Normalization means for normalization;
Predicting the value of the first component or the value of the second component of the pixel of interest using the tap coefficient previously determined for the class of the pixel of interest and the normalized prediction tap An image processing apparatus comprising: The normalizing means, when predicting the value of the first component of the pixel of interest, divides the dynamic range value of the first component value by the dynamic range value of the second component value. The image processing apparatus according to claim 1, wherein the prediction tap is normalized by multiplying a value obtained by multiplying a value of the second component of a pixel constituting the prediction tap. The image processing apparatus according to claim 1, wherein the first component or the second component is a component representing any one of red, blue, and green. The image processing apparatus according to claim 1, wherein the first component or the second component is a component representing luminance or color difference. First image data composed of pixels having the value of the first component of the component video signal and pixels having the value of the second component is obtained by using the values of the first component and the second component. In an image processing method for converting to second image data composed of pixels having values,
Extracting a plurality of pixels used for predicting a target pixel which is a target pixel of the second image data as a prediction tap from the first image data;
Extracting a plurality of pixels used for class classification to classify the target pixel into any one of a plurality of classes as a class tap from the first image data;
Using the class tap, classify the pixel of interest,
The prediction tap is set so that the dynamic range of the value of the first component included in the pixels constituting the prediction tap is equal to the dynamic range of the value of the second component included in the pixels constituting the prediction tap. Normalize,
Predicting the value of the first component or the value of the second component of the pixel of interest using the tap coefficient previously determined for the class of the pixel of interest and the normalized prediction tap An image processing method including a step. First image data composed of pixels having the value of the first component of the component video signal and pixels having the value of the second component is obtained by using the values of the first component and the second component. In a program for causing a computer to execute image processing for conversion into second image data composed of pixels having values,
Extracting a plurality of pixels used for predicting a target pixel which is a target pixel of the second image data as a prediction tap from the first image data;
Extracting a plurality of pixels used for class classification to classify the target pixel into any one of a plurality of classes as a class tap from the first image data;
Using the class tap, classify the pixel of interest,
The prediction tap is set so that the dynamic range of the value of the first component included in the pixels constituting the prediction tap is equal to the dynamic range of the value of the second component included in the pixels constituting the prediction tap. Normalize,
Predicting the value of the first component or the value of the second component of the pixel of interest using the tap coefficient previously determined for the class of the pixel of interest and the normalized prediction tap A program that causes a computer to execute processing including steps. Attention to extract the value of the first component of the pixel of interest, which is the pixel of interest, from the first image data composed of pixels having the values of the first component and the second component of the component video signal Pixel extraction means;
A plurality of pixels used for predicting the pixel of interest are predicted taps from second image data composed of pixels having the value of the first component and pixels having the value of the second component. A prediction tap extracting means for extracting;
Class tap extracting means for extracting, as class taps, a plurality of pixels used for class classification for classifying the target pixel into any of a plurality of classes;
Class classification means for classifying the pixel of interest using the class tap;
The prediction tap is set so that the dynamic range of the value of the first component included in the pixels constituting the prediction tap is equal to the dynamic range of the value of the second component included in the pixels constituting the prediction tap. Normalization means for normalization;
Used to predict the value of the first component of the pixel of interest from the normalized prediction tap using the value of the first component of the pixel of interest and the normalized prediction tap A learning device comprising: a calculation means for obtaining a tap coefficient corresponding to the class of the target pixel. When obtaining the tap coefficient used for predicting the value of the first component of the pixel of interest, the normalizing unit obtains a dynamic range value of the value of the first component of the second component. The learning according to claim 7, wherein the prediction tap is normalized by multiplying a value obtained by dividing the value by a dynamic range value by a value of the second component included in a pixel constituting the prediction tap. apparatus. The learning device according to claim 7, wherein the first component or the second component is a component representing any one of red, blue, and green. The learning device according to claim 7, wherein the first component or the second component is a component representing luminance or color difference. Extracting the value of the first component of the pixel of interest, which is the pixel of interest, from the first image data composed of pixels having the values of the first component and the second component of the component video signal;
A plurality of pixels used for predicting the pixel of interest are predicted taps from second image data composed of pixels having the value of the first component and pixels having the value of the second component. Extract and
Extracting a plurality of pixels used for class classification to classify the pixel of interest into any of a plurality of classes as a class tap from the second image data;
Using the class tap, classify the pixel of interest,
The prediction tap is set so that the dynamic range of the value of the first component included in the pixels constituting the prediction tap is equal to the dynamic range of the value of the second component included in the pixels constituting the prediction tap. Normalize,
Used to predict the value of the first component of the pixel of interest from the normalized prediction tap using the value of the first component of the pixel of interest and the normalized prediction tap A learning method including a step of obtaining a tap coefficient corresponding to the class of the target pixel. Extracting the value of the first component of the pixel of interest, which is the pixel of interest, from the first image data composed of pixels having the values of the first component and the second component of the component video signal;
A plurality of pixels used for predicting the pixel of interest are predicted taps from second image data composed of pixels having the value of the first component and pixels having the value of the second component. Extract and
Extracting a plurality of pixels used for class classification to classify the pixel of interest into any of a plurality of classes as a class tap from the second image data;
Using the class tap, classify the pixel of interest,
The prediction tap is set so that the dynamic range of the value of the first component included in the pixels constituting the prediction tap is equal to the dynamic range of the value of the second component included in the pixels constituting the prediction tap. Normalize,
Used to predict the value of the first component of the pixel of interest from the normalized prediction tap using the value of the first component of the pixel of interest and the normalized prediction tap A program for causing a computer to execute processing including a step of obtaining a tap coefficient corresponding to the class of the target pixel.

Images