JP4613510B2 - Image processing method and apparatus, and program - Google Patents

Description

  The present invention relates to an image processing method, apparatus, and program, and more particularly, to an image processing method, apparatus, and program that enable a clearer image to be obtained, for example, in an imaging apparatus that employs a single plate sensor.

  In recent years, digital cameras have become common as cameras. In a digital camera, an image captured by an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Mental Oxide Semiconductor) (hereinafter referred to as a captured image) is an LCD (Liquid Crystal Display) provided in the digital camera, etc. Displayed on the monitor and confirmed by the user. In addition, the image signal of the captured image is digitized and is transmitted to an image processing apparatus such as a personal computer (PC) via a recording medium such as a flash memory or by wired or wireless communication such as cable connection or infrared communication. Transferred. In the personal computer that is the transfer destination, the captured image of the digital camera is displayed on a monitor such as a CRT (Cathode Ray Tube) or a liquid crystal display (LCD), and the user can confirm or edit the captured image.

  In digital cameras, for example, when shooting a subject that does not have sufficient brightness, such as in a shaded place or in a dimly lit room, the digital camera must have a proper exposure. It is necessary to reduce the shutter speed (shutter time becomes longer).

  In such imaging with a long shutter time, the digital camera may be fixed on a tripod or the like so that the digital camera does not shake (shake). However, for example, when a digital camera is picked up by a hand and the digital camera shakes (so-called camera shake), the digital camera shakes (shakes) while the shutter is open (shutter time). The captured image is a blurred image of the subject. This blurred image is called an “hand shake” image or an “camera shake” image.

  Besides slowing down the shutter speed (increasing the shutter time), as a method for obtaining sufficient exposure, it is possible to obtain exposure equivalent to long-time exposure by simply accumulating a plurality of captured images. A method has been proposed (see, for example, Patent Document 1).

  However, in the method disclosed in Patent Document 1, since a plurality of captured images are simply cumulatively added, if the above-described camera shake occurs, the image becomes blurred as in the case where the shutter time is increased. .

  As a method for preventing blurring of images even when camera shake occurs (to avoid “camera shake” images), for example, “Image Stabilizer (abbreviated as IS)” used in digital cameras manufactured by Canon Inc. There is a method called "."

  In the image stabilizer, a pre-sensor is provided in the optical system lens, and the pre-sensor detects the shake and vibration of the digital camera, and a part of the lens group (correction optical system) according to the digital signal representing the detected shake and vibration of the digital camera. Is moved in a direction perpendicular to the optical axis, and the light beam is refracted in a direction to cancel the image blur.

  The image stabilizer suppresses image blurring caused by hand-held shooting or subtle vibrations of the camera platform caused by wind, which has a large effect when long focus or shutter speed is slow, and sharp (clear) A) image can be provided to the user.

  However, the image stabilizer requires a dedicated sensor for detecting blur and a mechanism for moving a part of the lens group (correction optical system) at high speed, which complicates the structure and increases the manufacturing cost. is there.

  As another method for avoiding “camera-blurred” images, a plurality of images are continuously captured by a high-speed shutter, and the second and subsequent captured images are compared with the first captured image among the captured images. There is a method of detecting a shift amount, correcting the second and subsequent captured images by the shift amount, and sequentially adding to the first image (for example, Patent Documents 2, 3, 4, 5, 6, 7, 8). In these conventional methods, basically, a corrected image having the same data arrangement as that of the first captured image is formed from each of the second and subsequent captured images by predetermined correction or interpolation. A simple addition method or a linear addition method is adopted for the first captured image in units of pixels.

  In this method, each image captured at high speed (continuous) has a short shutter time (exposure time), and thus becomes a dark image with little blur. Since the second and subsequent captured images are added to the first captured image, the final image obtained by the addition can be an image having the same brightness as the appropriate exposure.

  This method detects the amount of deviation of the second and subsequent captured images from the first captured image, and corrects (interpolates) the captured image by the amount of displacement, so that an R (Red) signal (red) of one pixel is detected. Data), G (Green) signal (green data), and B (Blue) signal (blue data) with each interpolation method using various interpolation functions such as linear interpolation and bicubic interpolation. Interpolate (correct).

JP 05-236422 A JP 2000-217032 A JP 2000-224460 A JP 2000-244803 A JP 2000-244797 A JP 2000-066952 A Japanese Patent Laid-Open No. 10-341367 JP 09-261526 A

  However, in the case of an imaging apparatus in which a single-plate sensor is employed as the imaging element, only one color signal from the R signal, G signal, and B signal is output from one pixel. Therefore, with the various interpolation methods as described above, it is not possible to interpolate the shift amounts of a plurality of images, and no other interpolation method applicable to a single plate sensor has been proposed.

  The present invention has been made in view of such a situation. For example, in an imaging apparatus employing a single plate sensor, a clearer image can be obtained even when camera shake occurs. To do.

The image processing method of the present invention uses a predetermined one input image among a plurality of input images captured by an imaging unit that captures an image having a pixel value of one color signal per pixel as a reference image. A detection step for detecting a positional relationship between the input images of the reference image; for each color of R, G, and B, with the pixel center of each pixel of the reference image as the target position; When each input image other than the reference image is corrected to a position on the reference image system, the natural length is zero in relation to the observed pixel value that is the pixel value of each input image other than the reference image observed around the target position. Thus, by generating and solving as a spring relational expression of a spring model with the probability that the observed pixel value is equivalent to the true color light quantity at the target position as a spring constant , R, G, B per pixel the pixel values of the three color signals Characterized in that it comprises an image estimation step of estimating the output image.

In the image estimation step, an output image is estimated by generating and solving a spring relational expression of a spring model using only a predetermined number of observation pixel values set in advance among the observation pixel values around the target position . be able to.

In the image estimation step, an average value of all the observed pixel values around the target position is obtained, and a spring relational expression of the spring model is generated and solved using only a predetermined number of observed pixel values in order from the average value. The output image can be estimated.

In the image estimation step , a spring relational expression of the spring model is generated and solved using only a predetermined number of pixel values in order from the shortest distance to the target position among the observed pixel values around the target position. Can be estimated .

In the image estimation step, it is detected whether there is an edge part around the target position, and a spring relational expression of the spring model excluding the observed pixel value at the position detected as the edge part is generated and solved to obtain an output image. Can be estimated .

In the image estimation step, it is detected whether there is an edge part around the position of interest, and the spring relational expression of the spring model is generated by changing the observed pixel value of the position detected as the edge part to a predetermined pixel value, By solving, the output image can be estimated .

In the image estimation step, the predetermined pixel value can be changed to a pixel value on the target position on a plane having the inclination of the edge portion and passing through the observed pixel value of the edge portion .

The image estimation step includes edges in at least one of the four directions of “vertical edge”, “lateral edge”, “upper left to lower right edge”, and “upper right to lower left edge”. The presence or absence of a part can be detected .

In the image estimation step, in addition to the spring relational expression, at least two of G and R, G and B, and R and B are added on the condition that there is a color correlation and output by solving. An image can be estimated .

In the image estimation step, the position on the reference image is within a predetermined allowable value in the vicinity region centered on the target position as a condition of color correlation between G and R, G and B, or R and B. It is possible to obtain observed pixel values of certain two colors, perform principal component analysis, obtain variances for components orthogonal to the principal components, and add conditions such that the smaller the variance, the greater the spring constant .

The light amount of the true color at the target position is expressed as a weighted addition according to the distance between the target position and the observed pixel value, and the weight is the target position (I ′, J ′) on the reference image system of the observed pixel value. Using the cubic function Cubic, the position (x, y) of Cubic (I′−x) × Cubic (J′−y) can be calculated .

In the image estimation step, it is determined whether ΣCubic (I′−x) × Cubic (J′−y) is equal to or greater than a predetermined threshold value. The pixel value of the target position can also be estimated using the pixel values of peripheral pixels around the pixel .

In the detection step, the positional relationship between the plurality of input images can be detected using the input image captured at an intermediate time or a time in the vicinity of the time when the plurality of input images are captured as a reference image.

The imaging unit may further include an exposure correction step of capturing a plurality of input images with less than proper exposure and increasing the gain of each pixel value of the plurality of input images.

An image processing apparatus according to the present invention includes an imaging unit that captures a plurality of input images having a pixel value of one color signal per pixel, and a predetermined one input image among the plurality of input images as a reference image. Detecting means for detecting a positional relationship among a plurality of input images, and for each of R, G, and B colors, with the pixel center of each pixel of the reference image as a target position, the light amount of the true color at the target position with respect to the target position And the observed pixel value of each input image other than the reference image observed around the target position when each input image other than the reference image is corrected to a position on the reference image system. Is generated as a spring relational expression of a spring model in which the probability that the observed pixel value and the true color light quantity at the target position are equivalent is equivalent to a spring constant, and R, G, Have pixel values of 3 color signals of B Characterized in that it comprises an image estimating means for estimating an output image that.

The program of the present invention uses, as a reference image, a predetermined one input image among a plurality of input images captured by an image capturing unit that captures an image having a pixel value of one color signal per pixel. A detection step for detecting a positional relationship between a plurality of input images, and for each color of R, G, and B, with the pixel center of each pixel of the reference image as the target position, the true color light quantity at the target position with respect to the target position When the input image other than the reference image is corrected to a position on the reference image system, the natural length is related to the observed pixel value that is the pixel value of each input image other than the reference image observed around the target position. It is generated as a spring relational expression of a spring model with the probability that the observed pixel value and the true color light quantity at the target position are equivalent to zero, and is solved by solving, and R, G, B per pixel The three colors of Characterized in that to execute a process including the image estimation step of estimating the output image having the pixel value.

In the present invention, a plurality of input images with a predetermined one input image among a plurality of input images captured by an imaging unit that captures an image having a pixel value of one color signal per pixel as a reference image. The positional relationship between the colors is detected, and for each color of R, G, and B, the pixel center of each pixel of the reference image is set as the target position, and the true color light quantity at the target position and each input other than the reference image for the target position. When the image is corrected to a position on the reference image system, the relationship between the observed pixel value of each input image other than the reference image observed around the position of interest is zero, the natural length is zero, The three color signals R, G, and B per pixel are generated and solved as a spring relational expression of a spring model with the probability that the true color light quantity at the target position is equivalent to the spring constant. An output image having a pixel value of It is constant.

  The image processing apparatus may be an independent apparatus or a block that performs image processing of one apparatus.

  According to the present invention, a clearer image can be obtained.

  Embodiments of the present invention will be described below. Correspondences between constituent elements described in the claims at the beginning of the application and specific examples in the embodiments of the present invention are exemplified as follows. This description is for confirming that a specific example supporting the invention described in the claims at the beginning of the application is described in the embodiment of the invention. Therefore, even if there are specific examples that are described in the embodiment of the invention but are not described here as corresponding to the configuration requirements, the specific examples are not included in the configuration. It does not mean that it does not correspond to a requirement. On the contrary, even if a specific example is described here as corresponding to a configuration requirement, this means that the specific example does not correspond to a configuration requirement other than the configuration requirement. not.

  Further, this description does not mean that the inventions corresponding to the specific examples described in the embodiments of the invention are all described in the claims at the beginning of the application. In other words, this description is an invention corresponding to the specific example described in the embodiment of the invention, and is the existence of an invention that is not described in the claims at the time of filing of this application, that is, in the future, It does not deny the existence of an invention that has been filed for division or added by amendment.

The image processing method according to claim 1 is an image processing method for estimating an output image from a plurality of input images, and a plurality of images captured by an imaging unit that captures an image having a pixel value of one color signal per pixel . A detection step (for example, step S3 in FIG. 2) for detecting a positional relationship between a plurality of input images using a predetermined one of the input images as a reference image, and for each of R, G, and B colors, With the pixel center of each pixel of the reference image as the target position, when the target position is corrected to the true color light quantity of the target position and each input image other than the reference image to a position on the reference image system, With regard to the relationship with the observed pixel value, which is the pixel value of each input image other than the observed reference image, the probability that the natural length is zero and the observed pixel value is equivalent to the true color light quantity at the target position. With spring constant An image estimation step (for example, step S4 in FIG. 2) for estimating an output image having pixel values of three color signals of R, G, and B per pixel by generating and solving as a spring relational expression of the model ; It is characterized by including.

The image processing method according to claim 14 , wherein the imaging unit captures the plurality of input images with less than proper exposure, and performs an exposure correction step (for example, increasing the pixel value of each of the plurality of input images). Further comprising step S2) of FIG.

  The image processing apparatus according to claim 15, wherein the image processing apparatus (for example, the digital camera 1 in FIG. 1) that estimates an output image from a plurality of input images has a pixel value of one color signal per pixel. The positional relationship between the plurality of input images, with an imaging unit (for example, the image sensor 4 in FIG. 1) that captures a plurality of input images and a predetermined one input image of the plurality of input images as a reference image Detection means (for example, the motion detection circuit 23 in FIG. 4), and for each color of R, G, and B, the pixel center of each pixel of the reference image is set as the target position, and the target position of the target position is determined. Observation of the light amount of the true color and the pixel value of each input image other than the reference image observed around the target position when each input image other than the reference image is corrected to a position on the reference image system With pixel value By generating and solving the relationship as a spring relational expression of a spring model in which the natural length is zero and the probability that the observed pixel value and the true color light quantity of the target position are equivalent is a spring constant, Image estimation means (for example, an arithmetic circuit 24 in FIG. 4) for estimating the output image having pixel values of three color signals of R, G, and B per pixel is provided.

  A specific example of each step of the program according to the sixteenth aspect is the same as the specific example in the embodiment of the invention of each step of the image processing method according to the first aspect.

(First embodiment)
FIG. 1 is a block diagram showing a configuration example of an embodiment of a digital (still) camera 1 to which the present invention is applied.

  A digital camera 1 in FIG. 1 includes a lens 2, an aperture 3, an image sensor 4, a correlated double sampling circuit 5, an A / D (Analog / Digital) converter 6, a signal processing circuit 7, a timing generator 8, and a D / A (Digital / Analog) converter 9, video encoder 10, monitor 11, codec (CODEC) 12, memory 13, bus 14, CPU (Central Processing Unit) 15, and input device 16. The A / D converter 6 includes a shift circuit 21 and the signal processing circuit 7 includes a frame memory 22.

  Light from a subject (not shown) passes through an optical system such as the lens 2 and the diaphragm 3 and enters the image sensor 4. The image sensor 4 of a single plate sensor composed of a CCD, a CMOS, or the like has a predetermined number of pixels (image sensors).

  The image sensor 4 receives incident subject light at predetermined intervals for a predetermined time (shutter time) in accordance with the exposure timing signal supplied from the timing generator 8. The image sensor 4 converts the amount of received light reaching each light receiving element on the imaging surface into an electric signal by photoelectric conversion, and supplies the image signal converted to the electric signal to the correlated double sampling circuit 5. Since the image pickup device 4 is a single plate sensor, the electrical signal supplied to the correlated double sampling circuit 5 is one color signal (data) of R signal, G signal, and B signal per pixel. ).

  Here, in order to output a clearer image even if camera shake occurs, the image pickup device 4 has a plurality of imaging elements 4 that are faster (with a shorter shutter time) than the shutter speed (shutter time (exposure time)) at appropriate exposure. Assume that images are captured (hereinafter referred to as N). Accordingly, the N images (input images) captured by the image sensor 4 are darker (those captured below the proper exposure) than the images captured at the appropriate exposure, and are captured at the appropriate exposure. It is assumed that the brightness is 1 / Mk of the image (= 1 / Mk) (k = 1 to N). Note that the value of Mk is determined by, for example, the shutter speed.

  The correlated double sampling circuit 5 removes the noise component of the image signal (electric signal) supplied from the image sensor 4 by correlated double sampling, and supplies it to the A / D converter 6. The A / D converter 6 A / D-converts, ie, samples and quantizes the noise-removed subject image signal supplied from the correlated double sampling circuit 5. Thereafter, the shift circuit 21 multiplies the digital image after A / D conversion, which is a dark image below the appropriate exposure, by Mk, for example, by shifting n ′ bits, thereby obtaining the same brightness (value) as the appropriate exposure. ) (The gain is increased) and supplied to the signal processing circuit 7.

  In the correlated double sampling circuit 5, the noise component of the image signal is removed, but not all of the noise component is completely removed. Therefore, there are also noise components that are not removed by the correlated double sampling circuit 5. In this case, the noise component that is not removed by the correlated double sampling circuit 5 becomes an error with respect to the true value of the image signal, and is multiplied by Mk together with the image signal in the shift circuit 21. Therefore, it can be said that the error depends on the amount of gain increase in the shift circuit 21. Here, E is a noise amount of a noise component that is not removed by the correlated double sampling circuit 5. As the noise amount E, for example, a possible maximum value can be adopted according to the characteristics of the image sensor 4. Here, the image signal supplied from the A / D converter 6 to the signal processing circuit 7 includes noise about Mk times the noise amount E (E × Mk). For example, when Mk = 8, the shift circuit 21 sets n ′ = 3, and the k-th captured image is shifted by 3 bits, so that the captured image has the same brightness as in the case of proper exposure. The

  Image signals of N captured images supplied from the A / D converter 6 that have been converted to a brightness of Mk times and gained up to the same brightness as that of the appropriate exposure are stored in the frame memory 22 of the signal processing circuit 7. Temporarily stored (stored).

  The signal processing circuit 7 performs predetermined processing on the image signals of N captured images gained up to the same brightness as the appropriate exposure stored in the frame memory 22 in accordance with various preset programs.

  That is, the signal processing circuit 7 uses the first captured image as the reference image and the second to Nth captured images among the N captured images as target images, and the target image is compared with the reference image. Whether or not such a positional deviation has occurred is detected. A positional deviation amount (positional relationship) between the reference image and the target image is detected. Then, the signal processing circuit 7 outputs an output image having all of the G signal, the R signal, and the B signal per pixel as one clear image (output image) in which camera shake is corrected based on the deviation amount. And the image signal of the obtained output image is supplied to the D / A converter 9, the codec 12, or both. The signal processing circuit 7 can be configured by a DSP (Digial Signal Processor) or the like. In the following, it is assumed that the image signal processed in the subsequent stage from the A / D converter 6 is gained up to the same brightness as that of the appropriate exposure even when there is no particular notice.

  The timing generator 8 outputs an exposure (exposure) timing signal to the image sensor 4, the correlated double sampling circuit 5, the A / D converter 6, and a signal processing circuit so that N images are captured at a predetermined interval. 7 is supplied. This interval can be changed by the user according to, for example, the brightness of the subject. When the user changes the interval, the interval change value determined by the CPU 15 when the user operates the input device 16 is supplied from the CPU 15 to the timing generator 8 via the bus 14.

  The D / A converter 9 D / A converts the image signal of the output image supplied from the signal processing circuit 7 and supplies it to the video encoder 10. The video encoder 10 converts the image signal (analog signal) supplied from the D / A converter 9 into a video signal that can be displayed on the monitor 11, and supplies the video signal to the monitor 11. The monitor 11 serves as a finder for the digital camera 1 and is composed of an LCD, a CRT, or the like, and displays a video signal supplied from the video encoder 10. As a result, a clear image is displayed on the monitor 11.

  The codec 12 converts the image signal of the output image supplied from the signal processing circuit 7 according to a predetermined method such as a JPEG (Joint Photographic Experts Group) method, an MPEG (Moving Picture Experts Group) method, or a DV (Digital Video) method. Encode and supply to the memory 13.

  The memory 13 is constituted by a semiconductor memory such as a flash memory, and temporarily (permanently) stores (records) the encoded image signal supplied from the codec 12. Instead of the memory 13, a recording medium such as a magnetic disk or an optical (magnetic) disk can be used. The memory 13 or a recording medium used instead of the memory 13 can be detachable from the digital camera 1. It is possible to provide both a recording medium built in the digital camera 1 and a recording medium detachable from the digital camera 1.

  The CPU 15 supplies a control signal to each unit via the bus 14 and controls various processes. For example, a control signal is supplied to each unit so that a subject is imaged in accordance with an imaging start signal supplied from the input device 16 in accordance with a user operation, and the image is stored in the memory 13.

  The input device 16 has operation buttons such as a release button on the digital camera 1 body. Various signals generated when the user operates the operation buttons are supplied to the CPU 15 via the bus 14, and the CPU 15 performs processing according to the various signals supplied from the input device 16 via the bus 14. Control each part. One or more operation buttons of the input device 16 can be displayed on the monitor 11. The operation of the operation button displayed on the monitor 11 can be detected by, for example, providing a transparent tablet on the monitor 11 and using the tablet.

  The imaging process of the digital camera 1 will be described with reference to the flowchart of FIG.

  First, in step S1, the imaging device 4 images a subject. That is, the imaging device 4 captures the incident subject light N times continuously at a predetermined interval in accordance with the exposure timing signal supplied from the timing generator 8 in shooting by pressing the release button (shutter button) once. By receiving light, high-speed imaging is performed N times. Therefore, N captured images are obtained in one shooting, and each captured image is a dark image having an appropriate exposure or less. The received light of the subject is photoelectrically converted, and after the noise component is removed by the correlated double sampling circuit 5, it is supplied to the A / D converter 6. Then, the process proceeds to step S2.

  In step S <b> 2, the A / D converter 6 digitally converts the image signal of the subject from which noise is removed, which is supplied from the correlated double sampling circuit 5. After that, the shift circuit 21 shifts the dark image below the appropriate exposure by n ′ bits, converts it to an image signal having the same brightness (value) as the appropriate exposure (gain increase), and supplies it to the signal processing circuit 7. The process proceeds to step S3.

  In step S3, the signal processing circuit 7 uses the first image as a reference image, sets the second and subsequent images as target images, and sets the target image (second to Nth images) to the reference image. Then, what kind of positional deviation has occurred, that is, the amount of positional deviation (movement amount) of the target image with respect to the reference image is detected, and the process proceeds to step S4.

  In step S4, the signal processing circuit 7 performs image estimation processing on the basis of the N captured images and the amount of positional deviation of the target image with respect to the reference image detected in step S3, and proceeds to step S5. Although details of the image estimation process will be described later, by this process, the signal processing circuit 7 has all the G signal, R signal, and B signal as one pixel as one clear image (output image) in which camera shake is corrected. And an image signal of the obtained output image is supplied to the D / A converter 9 or the codec 12 or both.

  In step S5, the monitor 11 displays the output image, records the output image in the memory 13 such as a flash memory, and the process ends. That is, in step S5, the image signal supplied from the signal processing circuit 7 to the D / A converter 9 in step S4 is converted into an analog signal and supplied to the video encoder 10. In step S 5, the video encoder 10 converts the analog signal of the image signal supplied from the D / A converter 9 into a video signal that can be displayed on the monitor 11, and supplies the video signal to the monitor 11. In step S5, the monitor 11 displays an image based on the video signal supplied from the video encoder 10, and ends the process. In step S5, the image signal supplied from the signal processing circuit 7 to the codec 12 in step S4 is subjected to predetermined encoding such as JPEG or MPEG and recorded in the memory 13 such as a flash memory for processing. finish.

  FIG. 3 shows an array of pixels of the image sensor 4. Note that FIG. 3 shows a part of the upper left part of the image sensor 4 (24 pixels in total, 6 pixels in the horizontal direction and 4 pixels in the vertical direction), but the other pixels are arranged in the same manner. It shall be.

  In FIG. 3, an XY coordinate system is set in which the upper left corner of the image sensor 4 is the origin, the horizontal (right) direction is the X direction, and the vertical (lower) direction is the Y direction. Further, the length (width) of one pixel in the vertical and horizontal directions is set to 1, respectively. In this case, the position (center position) of the i-th pixel from the left and the j-th pixel from the top can be expressed as (i−0.5, j−0.5).

  In FIG. 3, the pixel array of the image sensor 4 is a so-called Bayer array.

  That is, the pixel from which the G signal can be extracted is the first pixel in the X direction from the origin, the first pixel in the Y direction, the third pixel in the X direction, and the first pixel in the Y direction. A pixel G02 that is the fifth pixel in the X direction from the origin and the first pixel G04 in the Y direction, a pixel G11 that is the second pixel in the X direction and the second pixel in the Y direction, and so on. In addition, a pixel G13, a pixel G15, a pixel G20, a pixel G22, a pixel G24, a pixel G31, a pixel G33, and a pixel G35 are arranged.

  Also, the pixel from which the R signal can be extracted is the second pixel in the X direction from the origin, the first pixel R01 in the Y direction, the fourth pixel in the X direction from the origin, and the first pixel in the Y direction. A pixel R03 that is the sixth pixel in the X direction from the origin and a pixel R05 that is the first pixel in the Y direction, a pixel R21 that is the second pixel in the X direction from the origin and the third pixel in the Y direction, and so on. In addition, a pixel R23 and a pixel R25 are arranged.

  Further, the pixel from which the B signal can be extracted is the first pixel in the X direction from the origin, the second pixel in the Y direction, the third pixel in the X direction from the origin, and the second pixel in the Y direction. Pixel B12, the fifth pixel in the X direction from the origin, the second pixel in the Y direction, the pixel B14, the first pixel in the X direction from the origin, and the fourth pixel in the Y direction, and so on. In addition, a pixel B32 and a pixel B34 are arranged.

  Here, with respect to a predetermined position (x, y) (x and y are real numbers) in the XY coordinate system of the image sensor 4, the G signal, R signal, and B signal of one clear image without camera shake are Let Lg (x, y), Lr (x, y), and Lb (x, y), respectively. Further, for the “i-th and j-th pixels” that are i-th from the left and j-th from the top, the G signal, R signal, and B signal of one clear image without camera shake are respectively represented by Lg (i, j), Lr (i, j), and Lb (i, j). That is, each of Lg (x, y), Lr (x, y), Lb (x, y) (Lg (i, j), Lr (i, j), Lb (i, j)) has a predetermined position. In (x, y) (“i-th, j-th pixel”), it represents the true green, red, and blue light amounts (data) without camera shake and noise. Therefore, in the following, Lg (x, y), Lr (x, y), Lb (x, y) (Lg (i, j), Lr (i, j), Lb (i, j)) will be described. True green light quantity Lg (x, y) (Lg (i, j)), true red light quantity Lr (x, y) (Lr (i, j)), true blue light quantity Lb (x, y) ) (Lb (i, j)). When x = i−0.5 and y = j−0.5, Lg (x, y) = Lg (i, j), Lr (x, y) = Lr (i, j), Lb (X, y) = Lb (i, j).

  In the present embodiment, the pixel array of the image sensor 4 is a Bayer array, but the pixel array is not limited to the Bayer array and may be any other array.

  Next, variables ig, jg, ir, jr, ib, and jb used for each pixel of the G signal, R signal, and B signal are defined.

  Each of the variables ig and jg represents a position i in the X direction and a position j in the Y direction for a pixel from which the G signal can be extracted. That is, the combination of variables ig and jg is equal to the combination of variables i and j from which the G signal can be extracted. In the case of the Bayer array, it can be said that the variables ig and jg satisfy the condition that the difference (i−j) between the variables i and j is an even number. Of course, the difference (ig−jg) between the variables ig and jg is an even number. Therefore, “ig-th and jg-th pixels” are pixels from which the G signal can be extracted. In the case of an array other than the Bayer array, the conditions for variables ig and jg according to the nature of the array are used.

  Each of the variables ir and jr represents a position i in the X direction and a position j in the Y direction with respect to a pixel from which the R signal can be extracted. That is, the combination of the variables ir and jr is equal to the combination of the variables i and j from which the R signal can be extracted. In the case of the Bayer array, it can be said that the variables ir and jr satisfy the condition that the variable i is an even number and the difference (ij) between the variables i and j is an odd number. Of course, the difference (ir−jr) between the variables ir and jr is also an odd number. Therefore, the “irth and jrth pixels” are pixels from which the R signal can be extracted. In the case of an array other than the Bayer array, the conditions of the variables ir and jr according to the nature of the array are used.

  Each of the variables ib and jb represents a position i in the X direction and a position j in the Y direction with respect to a pixel from which the B signal can be extracted. That is, the combination of variables ib and jb is equal to the combination of variables i and j from which the B signal can be extracted. In the case of the Bayer array, it can be said that the variable ib and jb satisfy the condition that the variable i is an odd number and the difference (i−j) between the variables i and j is an odd number. Of course, the difference (ib−jb) between the variables ib and jb is also an odd number. Therefore, the “ib-th and jb-th pixels” are pixels from which the B signal can be extracted. In the case of an array other than the Bayer array, the conditions of the variables ib and jb according to the nature of the array are used.

  Next, the values (pixel values) of color signals (G signal, R signal, B signal) obtained by receiving light at each pixel of the image sensor 4 shown in FIG. 3 will be defined.

  As described above, the image sensor 4 captures N images. Accordingly, N pixel values are obtained for one pixel of the image sensor 4. Therefore, the pixel values obtained from the “ig-th and jg-th pixels” of the k-th (k = 1 to N) are set to Gobs (k, ig, jg) and “ir” of the k-th (k = 1 to N). The pixel value obtained at the “th, jr-th pixel” is Robs (k, ir, jr), and the pixel value obtained at the k-th (k = 1 to N) “ib-th, jb-th pixel” is Bobs ( k, ib, jb). For example, the pixel value obtained by the first pixel G00 is represented by Gobs (1, 1, 1), and the pixel value obtained by the second pixel G04 is represented by Gobs (2, 5, 1). The In the following description, k represents an integer of 1 to N unless otherwise specified.

  Conversely, pixels having pixel values Gobs (k, ig, jg), Robs (k, ir, jr), and Bobs (k, ib, jb) are respectively represented by pixel G (jg-1) (ig− 1), R (jr-1) (ir-1), and B (jb-1) (ib-1).

  FIG. 4 shows a detailed configuration example of a part of the signal processing circuit 7 of FIG.

  The signal processing circuit 7 includes a frame memory 22, a motion detection circuit 23, an arithmetic circuit 24, and a controller 25. The frame memory 22 includes frame memories 22-1 to 22-N, and the motion detection circuit 23 includes motion detection circuits 23-1 to 23- (N-1).

  As described above, N captured images are supplied from the A / D converter 6 to the frame memory 22. The frame memory 22-1 stores (stores) the first captured image supplied from the A / D converter 6. The frame memory 22-2 stores the second captured image supplied from the A / D converter 6. Similarly, the frame memory 22-k stores the k-th captured image supplied from the A / D converter 6 in the frame memory 22-k (k = 3 to N).

  The frame memory 22-1 supplies the stored first captured image to the arithmetic circuit 24 and the motion detection circuits 23-1 to 23- (N-1) at a predetermined timing. The frame memory 22-2 supplies the stored second captured image to the arithmetic circuit 24 and the motion detection circuit 23-1 at a predetermined timing. Similarly, the frame memory 22-k supplies the k-th captured image stored therein to the arithmetic circuit 24 and the motion detection circuit 23- (k−1) at a predetermined timing.

  The motion detection circuit 23 detects the positional relationship between the two captured images. That is, the motion detection circuit 23 uses the first captured image as a reference image, sets the second and subsequent captured images as target images, and sets the target image (second to Nth images) to the reference image. Thus, the amount of positional deviation (movement amount) of the target image relative to the reference image is detected. The amount of deviation is caused by, for example, camera shake.

  The motion detection circuit 23-1 is supplied with the first captured image as the reference image from the frame memory 22-1, and the second captured image as the target image from the frame memory 22-2.

  The motion detection circuit 23-1 corresponds to which position of the first captured image each pixel of the second captured image (or each block when the entire screen is divided into a plurality of blocks). , And the positional relationship between the first captured image and the second captured image is expressed by the following equation (1) at a rotation angle θ2, a scale S2, and a parallel movement amount (T2x, T2y). The configured conversion parameters (θ2, T2x, T2y, S2) are obtained and supplied to the arithmetic circuit 24.

・・・（１）

... (1)

Expression (1) is a so-called affine transformation expression. In Expression (1), (X2, Y2) is the pixel position of the second captured image, and (X1 ₍₂₎ , Y1 ₍₂₎ ) Is the position on the first captured image. Subscript (2) indicates that a position of a second captured image has been converted.

  This conversion parameter (θ2, T2x, T2y, S2) indicates that the position (X2, Y2) on the second captured image is, for example, relative to the position (X1, Y1) on the first captured image. Since the digital camera 1 is rotated by an angle θ2 and moved in the direction of the subject for handheld shooting, the image is enlarged by S2 times (reduced when S2 <1), and is parallel to the subject (T2x, T2y) ) Means that it is shifted. In the case of normal camera shake, the influence of lateral blurring (shake) parallel to the surface of the image sensor 4 is large, and there is little blurring (shake) from the digital camera 1 to the direction of the subject. S2 = 1 may be approximated assuming that there is no effect on the direction of the subject.

  The motion detection circuit 23-2 is supplied with the first captured image as the reference image from the frame memory 22-1, and the third captured image as the target image from the frame memory 22-3. The

  In the motion detection circuit 23-2, each pixel of the third captured image (or each block when the entire screen is divided into a plurality of blocks) corresponds to which position of the first captured image. And the positional relationship between the first captured image and the third captured image is expressed by the following equation (2) with the rotation angle θ3, the scale S3, and the parallel movement amount (T3x, T3y). The configured conversion parameters (θ3, T3x, T3y, S3) are obtained and supplied to the arithmetic circuit 24.

・・・（２）

... (2)

Expression (2) is a so-called affine transformation expression. In Expression (2), (X3, Y3) is the pixel position of the third captured image, and (X1 ₍₃₎ , Y1 ₍₃₎ ) Is the position on the first captured image. Subscript (3) indicates that a position of a third captured image has been converted.

  This conversion parameter (θ3, T3x, T3y, S3) indicates that the position (X3, Y3) on the third captured image is hand-held shooting relative to the position (X1, Y1) on the first captured image. Therefore, since the digital camera 1 is rotated by the angle θ3 and moved in the direction of the subject, the image is enlarged by S3 times (reduced when S3 <1) and only in the direction parallel to the subject (T3x, T3y). It means that it has shifted. In the case of normal camera shake, the influence of lateral blurring (shake) parallel to the surface of the image sensor 4 is large, and there is little blurring (shake) from the digital camera 1 to the direction of the subject. S3 = 1 may be approximated assuming that there is no effect on the direction of the subject.

  Similarly, in the motion detection circuit 23- (k-1), the first captured image as the reference image is received from the frame memory 22-1, and the kth captured image as the target image is stored in the frame memory 22-. k, respectively.

  In the motion detection circuit 23- (k-1), each pixel of the k-th captured image (or each block when the entire screen is divided into a plurality of blocks) is located at which position of the first captured image. Rotation angle θk, scale Sk, and parallel movement amount (Tkx) such that the positional relationship between the first captured image and the kth captured image is expressed by the following equation (3). , Tky), conversion parameters (θk, Tkx, Tky, Sk) are obtained and supplied to the arithmetic circuit 24.

・・・（３）

... (3)

Expression (3) is a so-called affine transformation expression. In Expression (3), (Xk, Yk) is the pixel position of the k-th captured image, and (X1 _(k) , Y1 _(k) ) Is the position on the first captured image. The subscript (k) represents that a certain position of the k-th image has been converted.

  The conversion parameters (θk, Tkx, Tky, Sk) are obtained by hand-held shooting with respect to the position (X1, Y1) on the first captured image at the position (Xk, Yk) on the kth captured image. Therefore, since the digital camera 1 is rotated by the angle θk and moved in the direction of the subject, the image is enlarged by Sk times (reduced when Sk <1) and only in the direction parallel to the subject (Tkx, Tky). It means that it has shifted. In the case of normal camera shake, the influence of lateral blurring (shake) parallel to the surface of the image sensor 4 is large, and there is little blurring (shake) from the digital camera 1 to the direction of the subject. Sk = 1 may be approximated assuming that there is no effect on the direction of the subject.

  As described above, the conversion parameter (θk, Tkx, Tky, Sk) is obtained from the positional relationship of the k-th captured image with the first captured image as a reference, and the digital camera 1 is provided with an acceleration sensor. In other words, it can be mechanically obtained from the output of the acceleration sensor.

  N captured images are supplied to the arithmetic circuit 24 from the frame memories 22-1 to 22-N. The arithmetic circuit 24 also includes conversion parameters (θk, Tkx, Tky) representing the positional relationship between the first captured image and the kth captured image from the motion detection circuits 23-1 to 23- (N-1). , Sk).

  The arithmetic circuit 24 performs image estimation processing to be described later based on the positional relationship of the second to N-th captured images supplied from the motion detection circuits 23-1 to 23- (N-1) with respect to the first captured image. As a result, the image signal (G signal, R signal, B signal) of one clear output image corrected for camera shake is estimated and supplied to the D / A converter 9 or the codec 12. Each of the N captured images supplied from the A / D converter 6 to the signal processing circuit 7 is a signal in which one pixel has any one of a G signal, an R signal, and a B signal. The image signal estimated by the arithmetic circuit 24 is a signal (data) having three color signals of G signal, R signal, and B signal per pixel.

  The controller 25 controls the frame memories 22-1 to 22-N, the motion detection circuits 23-1 to 23- (N-1), the arithmetic circuit 24, and the like in the signal processing circuit 7 according to the control of the CPU 15. Note that the CPU 15 (FIG. 1) replaces the controller 25 with the frame memories 22-1 to 22-N, the motion detection circuits 23-1 to 23- (N-1), the arithmetic circuit 24, etc. in the signal processing circuit 7. In this case, the controller 25 can be omitted.

  In a single-plate sensor using a Bayer array or the like, the number of R signal or B signal pixels is smaller than the number of G signal pixels. For this reason, the R signal or B signal in the output image obtained by the signal processing circuit 7 may have a larger error than the G signal. In such a case, it is possible to remove noise by disposing a low-pass filter that limits the band of the high frequency component only for the color difference signal in the subsequent stage of the arithmetic circuit 24 without changing the luminance signal.

  In addition, by making the arithmetic circuit 24 a high-speed arithmetic circuit that performs real-time processing that can be processed while sequentially capturing captured images during continuous imaging, the frame memories 22-1 to 22-N and motion The number of the detection circuits 23-1 to 23- (N-1) can be reduced. Thereby, the scale of the signal processing circuit 7 can be reduced.

  Next, a first embodiment of the image estimation process in the signal processing circuit 7 described above will be described.

  For example, it is assumed that an on-chip lens (not shown) is disposed immediately above each pixel (light receiving portion) of the image sensor 4. The on-chip lens converges all the light of the subject incident on the on-chip lens into one point. Accordingly, the pixel immediately below the on-chip lens can receive all the light of the subject incident on the on-chip lens as an integral value. Thereby, the effect that the detection sensitivity of each pixel becomes good is acquired.

  Therefore, the data received by each pixel of the image sensor 4 (the amount of received light) is not the light value of the subject incident on a certain point of the pixel (that is, not point-sampled data). This is an integrated value of light of a subject incident on a surface (light receiving surface) having a finite area.

  In the first embodiment, a clear image (output image) is obtained as an image estimation process by accurately formulating the characteristics of the on-chip lens. Conventionally, the data of each pixel is generally considered as point-sampled data. However, originally, as described above, the amount of light received by each pixel of the actual image sensor 4 is the value (integrated value) of light incident on a surface having a certain area, so an accurate image is estimated (restored). ) It was hard to say.

  First, for the first image stored in the frame memory 22-1, the relationship between the pixel value obtained at each pixel and the light incident by the on-chip lens is formulated.

  For example, for the pixel G00 that is the “first, first pixel” at the top left in FIG. 3, the pixel value Gobs (1, 1, 1) and the true green light quantity Lg (x, y) This relationship can be expressed as the following equation (4).

・・・（４）

... (4)

  A green filter is attached to the upper left pixel G00 of the light receiving element 4 which is a single plate sensor so as to transmit only the green component of the incident light. The light received by the pixel G00 is a rectangle surrounded by (0, 0), (0, 1), (1, 0), and (1, 1) in the coordinate system of FIG. 3 due to the effect of the on-chip lens. The light is incident on the region. That is, all of the light of the subject incident on the rectangular area surrounded by (0, 0), (0, 1), (1, 0), and (1, 1) is “the first and first pixels. Is received by G00.

  The left side of Expression (4) represents the true green light quantity Lg (x, y) (incident) at the position (x, y) in the coordinate system of the first captured image as (0, 0), (0 , 1), (1, 0), and (1, 1), that is, a rectangular region integrated, that is, integration with 0 ≦ x <1 and 0 ≦ y <1. The right side of the equation (4) is a pixel value Gobs (1, 1, 1) obtained (observed) at the first “first, first pixel” at that time. Equation (4) is a pixel in which the integrated value of the true green light quantity Lg (x, y) incident on the light receiving surface of the pixel G00 at the time of imaging of the first sheet is observed in the pixel G00. It represents that it is equal to the value Gobs (1, 1, 1).

  Depending on the performance of the on-chip lens, there is a case where the light of the subject incident on the peripheral portion of each pixel (so-called “flip” portion) cannot be converged. In that case, for example, the range integrated on the left side of the equation (4) is reduced by a small amount, for example, (0.1, 0.1), (0.1, 0.9), (0.9, 0.1) and (0.9, 0.9).

  In addition, in many digital cameras, an optical low-pass filter is incorporated in front of each pixel in order to avoid false color, which is a drawback of a single-plate sensor. In that case, a slightly wider range than a rectangular area of one pixel is used. Light is input to the pixel. In that case, the range integrated on the left side of the equation (4) may be set to a rectangular area that is slightly larger.

  Next, consider the pixel R01 which is the “second and first pixel” on the right side of the pixel G00 in FIG.

  Regarding the pixel R01 which is the “second and first pixel”, the relationship between the pixel value Robs (1, 2, 1) and the true green light amount Lg (x, y) is expressed by the following equation (5). It can be expressed as

・・・（５）

... (5)

  A red filter is attached to the upper left pixel R01 of the light receiving element 4 which is a single plate sensor so as to transmit only the red component of the incident light. The light received by the pixel R01 is a rectangle surrounded by (1, 0), (1, 1), (2, 0), and (2, 1) in the coordinate system of FIG. 3 due to the effect of the on-chip lens. The light is incident on the region. That is, all of the light of the subject incident on the rectangular area surrounded by (1, 0), (1, 1), (2, 0), and (2, 1) is “the second and first pixels. Is received by R01.

  The left side of Expression (5) represents the true red light amount Lr (x, y) (incident) at the position (x, y) in the coordinate system of the first captured image as (1, 0), (1 , 1), (2, 0), and (2, 1), that is, a rectangular region that is integrated with 1 ≦ x <2 and 0 ≦ y <1. Further, the right side of Expression (5) is a pixel value Robs (1, 2, 1) obtained (observed) at the first “second, first pixel” at that time. Equation (5) is a pixel in which the integral value on the light receiving surface of the true red light quantity Lr (x, y) incident on the light receiving surface of the pixel R01 at the time of the first imaging is observed in the pixel R01. It represents that it is equal to the value Robs (1, 2, 1).

  Depending on the performance of the on-chip lens, there is a case where the light of the subject incident on the peripheral portion of each pixel (so-called “flip” portion) cannot be converged. In that case, for example, the range to be integrated on the left side of the equation (5) is reduced by a little, for example, (1.1, 0.1), (1.1, 0.9), (1.9, 0.1) and (1.9, 0.9).

  In addition, in many digital cameras, an optical low-pass filter is incorporated in front of each pixel in order to avoid false color, which is a drawback of a single-plate sensor. In that case, a slightly wider range than a rectangular area of one pixel is used. Light is input to the pixel. In that case, the range integrated on the left side of the equation (5) may be a rectangular region that is slightly larger.

  A similar expression can be established for other pixels of the first captured image.

  That is, for the pixel G (jg-1) (ig-1) which is the "igth, jgth pixel" in FIG. 3, the pixel value Gobs (1, ig, jg) and the true green light amount Lg The relationship with (x, y) can be expressed as in equation (6).

・・・（６）

... (6)

  The expression (6) is used to calculate the true green light quantity Lg (x, y) in the coordinate system of FIG. 3 (ig-1, jg-1), (ig-1, jg), (ig, jg-1). , And (ig, jg), that is, a pixel area Gobs (, obtained by integrating ig−1 ≦ x <ig and jg−1 ≦ y <jg in the coordinate system of the first captured image. 1, ig, jg).

  In addition, for R (jr−1) (ir−1), which is “irth and jrth pixels” in FIG. 3, the pixel value Robs (1, ir, jr) and the true red light amount Lr ( The relationship with x, y) can be expressed as in equation (7).

・・・（７）

... (7)

  Equation (7) indicates that the true red light amount Lr (x, y) is expressed as (ir-1, jr-1), (ir-1, jr), (ir, jr-1) in the coordinate system of FIG. , And (ir, jr), that is, a pixel region Robs () obtained by integrating ir-1 ≦ x <ir and jr-1 ≦ y <jr in the coordinate system of the first captured image. 1, ir, jr).

  Further, for the pixel B (jb-1) (ib-1) which is the “ibth and jbth pixel” in FIG. 3, the pixel value Bobs (1, ib, jb) and the true blue light quantity Lb The relationship with (x, y) can be expressed as in equation (8).

・・・（８）

... (8)

  The expression (8) is obtained by converting the true blue light quantity Lb (x, y) into (ib-1, jb-1), (ib-1, jb), (ib, jb-1) in the coordinate system of FIG. , And (ib, jb), that is, the integrated value of ib-1 ≦ x <ib and jb-1 ≦ y <jb in the coordinate system of the first captured image is the pixel value Bobs ( 1, ib, jb).

  Actually, as described above, the image signal (observed pixel value) includes the noise amount E, and the noise amount E is gained up to Mk times. Therefore, when the noise component of (E × Mk) is considered in the equations (6), (7), and (8), the light quantity integration conditional expressions represented by the equations (9), (10), and (11) are respectively obtained. can get.

・・・（９）

... (9)

・・・（１０）

... (10)

・・・（１１）

(11)

  Here, | x | represents the absolute value of x.

  In the equation (9), the difference between the integrated value of the light amount Lg (x, y) and the observed pixel value Gobs (1, ig, jg) is equal to or less than the maximum noise amount M1 × E that can be assumed. Represents that. The same applies to equations (10) and (11).

  In the following description, the k-th captured image is also referred to as a k-th image.

  Next, with respect to the second image stored in the frame memory 22-2, the pixel value obtained (observed) in each pixel and the light incident by the on-chip lens, as in the first image. Formulate the relationship.

  The coordinate system of the second and subsequent captured images is the coordinate system of the first captured image using the conversion parameters (θk, Tkx, Tky, Sk) (k = 2 to N) detected by the motion detection circuit 23. Is converted to

  For the pixel G00 which is the “first and first pixels” of the second image, the relationship between the pixel value Gobs (2, 1, 1) and the green light quantity Lg (x, y) is It can be expressed as (12).

・・・（１２）

(12)

  The upper left pixel G00 of the light receiving element 4 is equipped with a green filter so as to absorb only the green component of the incident light. The light received by the pixel G00 is surrounded by (0, 0), (0, 1), (1, 0), and (1, 1) in the coordinate system of the second image due to the effect of the on-chip lens. The light is incident on the rectangular area.

Using the conversion parameters (θ2, T2x, T2y, S2), the positions (0, 0), (0, 1), (1, 0), and (1, 1) in the coordinate system of the second image are used. The position converted to the coordinate system of the first image is (0 ₍₂₎ , 0 ₍₂₎ ), (0 ₍₂₎ , 1 ₍₂₎ ), (1 ₍₂₎ , 0 ₍₂₎ ), and (1 ₍₂₎ , 1 ₍₂₎ ). Therefore, (0 ₍₂₎ , 0 ₍₂₎ ), (0 ₍₂₎ , 1 ₍₂₎ ), (1 ₍₂₎ , 0 ₍₂₎ ), and (1 ₍₂₎ , 1 ₍₂₎ ) All of the light of the subject incident on the rectangular region surrounded by is received by the pixel G00 which is the “first and first pixel” at the time of the second image pickup. The position obtained by converting the position (x, y) of the second image into the coordinates of the first image is also written as P (2, x, y).

The left side of Equation (12) represents the true green light quantity Lg (x, y) as P (2, 0, 0), P (2, 0, 1), P (2, 1, 0), and P In the rectangular area surrounded by (2, 1, 1), that is, in the first frame coordinate system, (0 ₍₂₎ , 0 ₍₂₎ ), (0 ₍₂₎ , 1 ₍₂₎ ), (1 _{(2 )} , 0 ₍₂₎ ), and (1 ₍₂₎ , 1 ₍₂₎ ). In addition, the right side of Expression (12) is a pixel value Gobs (2, 1, 1) obtained at the “first, first pixel” of the second image at that time. Expression (12) indicates that the pixel value Gobs (2, 1, 1) observed at the second pixel G00 is (0 ₍₂₎ , 0 ₍₂₎ ) in the coordinate system at the time of imaging of the first image. , (0 ₍₂₎ , 1 ₍₂₎ ), (1 ₍₂₎ , 0 ₍₂₎ ), and true green that is incident on the rectangular area bounded by (1 ₍₂₎ , 1 ₍₂₎ ) It represents that it is equal to the integral value of the light quantity Lg (x, y). In equation (12), ∫∫dxdy is (0 ₍₂₎ , 0 ₍₂₎ ), (0 ₍₂₎ , 1 ₍₂₎ ), (1 ₍₂₎ , 0 ₍₂₎ ), and (1 ₍₂₎ , 1 ₍₂₎ ) represents the integral of the rectangular region enclosed. The same applies to equation (13) and thereafter.

  Depending on the performance of the on-chip lens, there is a case where the light of the subject incident on the peripheral portion of each pixel (so-called “flip” portion) cannot be converged. In this case, the range that is integrated on the left side of Expression (12) may be a rectangular area that is slightly smaller, as in the case of the first image described above.

  In addition, in many digital cameras, an optical low-pass filter is incorporated in front of each pixel in order to avoid false color, which is a drawback of a single-plate sensor. In that case, a slightly wider range than a rectangular area of one pixel is used. Light is input to the pixel. In this case, the range integrated on the left side of the equation (12) may be a rectangular region that is slightly larger.

  Next, consider the pixel R01 that is the “second and first pixel” that is adjacent to the pixel G00 in the second image.

  For the pixel R01 which is the “second and first pixel” of the second image, the relationship between the pixel value Robs (2, 2, 1) and the true red light quantity Lr (x, y) is It can be expressed as the following formula (13).

・・・（１３）

... (13)

  The upper left pixel R01 of the light receiving element 4 is equipped with a red filter so as to absorb only the red component of the incident light. The portion received by the pixel R01 is surrounded by (1, 0), (1, 1), (2, 0), and (2, 1) in the coordinate system of the second image for the effect of the on-chip lens. The light is incident on the rectangular area.

The positions (1, 0), (1, 1), (2, 0), and (2, 1) in the coordinate system of the second image are converted using the conversion parameters (θ2, T2x, T2y, S2). The position converted to the coordinate system of the first image is (1 ₍₂₎ , 0 ₍₂₎ ), (1 ₍₂₎ , 1 ₍₂₎ ), (2 ₍₂₎ , 0 ₍₂₎ ), and (2 ₍₂₎ , 1 ₍₂₎ ). Therefore, (1 ₍₂₎ , 0 ₍₂₎ ), (1 ₍₂₎ , 1 ₍₂₎ ), (2 ₍₂₎ , 0 ₍₂₎ ), and (2 ₍₂₎ , 1 ₍₂₎ ) All of the light of the subject incident on the rectangular area surrounded by is received by R01 which is the “second and first pixel” at the time of imaging the second image.

The left side of equation (13) represents the true red light quantity Lr (x, y) as P (2,1,0), P (2,1,1), P (2,2,0), and P In the rectangular region surrounded by ₍₂ , ₂ , 1 ₎ , that is, in the first frame coordinate system, (1 ₍₂₎ , 0 ₍₂₎ ), (1 ₍₂₎ , 1 ₍₂₎ ), (2 _{(2 )} , 0 ₍₂₎ ), and (2 ₍₂₎ , 1 ₍₂₎ ). Further, the right side of the equation (13) is a pixel value Robs (2, 2, 1) obtained at the second “second, first pixel” at that time. Expression (13) indicates that the pixel value Robs (2, 2, 1) observed at the second pixel R01 is (1 ₍₂₎ , 0 ₍₂₎ ) in the coordinate system at the time of imaging of the first image. , (1 ₍₂₎ , 1 ₍₂₎ ), (2 ₍₂₎ , 0 ₍₂₎ ), and true red that is incident on the rectangular area bounded by (2 ₍₂₎ , 1 ₍₂₎ ) It represents that it is equal to the integral value of the light quantity Lr (x, y) of the.

  Depending on the performance of the on-chip lens, there is a case where the light of the subject incident on the peripheral portion of each pixel (so-called “flip” portion) cannot be converged. In this case, the range that is integrated on the left side of Equation (13) may be a rectangular region that is slightly smaller than the one.

  In addition, in many digital cameras, an optical low-pass filter is incorporated in front of each pixel in order to avoid false color, which is a drawback of a single-plate sensor. In that case, a slightly wider range than a rectangular area of one pixel is used. Light is input to the pixel. In that case, the range integrated on the left side of the equation (13) may be a rectangular region that is slightly larger.

  The same formula can be established for the other pixels of the second image.

  That is, for G (jg−1) (ig−1) which is the “igth and jgth pixels” of the second image, the pixel value Gobs (2, ig, jg) and the true green light amount The relationship with Lg (x, y) can be expressed as in equation (14).

・・・（１４）

(14)

Expression (14) is for calculating the true green light quantity Lg (x, y) in the coordinate system of the second image (ig-1, jg-1), (ig-1, jg), (ig, jg- 1) and a rectangular region surrounded by (ig, jg), that is, in the coordinate system of the first image, (ig-1 ₍₂₎ , jg-1 ₍₂₎ ), (ig-1 ₍₂₎ , jg ₍₂₎ ), (ig ₍₂₎ , jg-1 ₍₂₎ ), and integrated in a rectangular area surrounded by (ig ₍₂₎ , jg ₍₂₎ ) are pixel values Gobs (2, ig , Jg).

  For R (jr-1) (ir-1), which is the "irth and jrth pixels" of the second image, the pixel value Robs (2, ir, jr) and the true red light quantity The relationship with Lr (x, y) can be expressed as in equation (15).

・・・（１５）

... (15)

The expression (15) is used to calculate the true red light amount Lr (x, y) in the coordinate system of the second image (ir-1, jr-1), (ir-1, jr), (ir, jr- 1) and a rectangular region surrounded by (ir, jr), that is, in the coordinate system of the first image, (ir-1 ₍₂₎ , jr-1 ₍₂₎ ), (ir-1 ₍₂₎ , jr ₍₂₎ ), (ir ₍₂₎ , jr-1 ₍₂₎ ) and (ir ₍₂₎ , jr ₍₂₎ ) are integrated in the rectangular region surrounded by the pixel value Robs (2, ir , Jr).

  Furthermore, for B (jb-1) (ib-1) which is the "ibth and jbth pixels" of the second image, the pixel value Bobs (2, ib, jb) and the true blue light quantity The relationship with Lb (x, y) can be expressed as in Expression (16).

・・・（１６）

... (16)

Expression (16) expresses the true blue light quantity Lb (x, y) in the coordinate system of the second image (ib-1, jb-1), (ib-1, jb), (ib, jb- 1) and (ib, jb) in a rectangular area, that is, in the coordinate system of the first image, (ib-1 ₍₂₎ , jb-1 ₍₂₎ ), (ib-1 ₍₂₎ , jb ₍₂₎ ), (ib ₍₂₎ , jb-1 ₍₂₎ ), and (ib ₍₂₎ , jb ₍₂₎ ) are integrated in a rectangular region surrounded by the pixel value Bobs (2, ib , Jb).

  Actually, as described above, the image signal (observed pixel value) includes the noise amount E, and the noise amount E is gained up to Mk times. When the noise component of (E × Mk) is considered in the equations (14), (15), and (16), the light quantity integration conditional expressions represented by the equations (17), (18), and (19) are obtained, respectively. .

・・・（１７）

... (17)

・・・（１８）

... (18)

・・・（１９）

... (19)

  Here, | x | represents the absolute value of x.

  In Expression (17), the difference between the integrated value of the light amount Lg (x, y) and the observed pixel value Gobs (2, ig, jg) is equal to or less than the maximum noise amount M2 × E that can be assumed. Represents that. The same applies to equations (18) and (19).

  Referring to FIG. 5, the relationship between the position (x, y) of the second image and the position P (2, x, y) obtained by converting the position (x, y) to the coordinates of the first image. explain.

  The left side of FIG. 5 shows predetermined four points (i-1, j-1), (i-1, j), (i, j-1), and (i, j) in the coordinate system of the second image. A pixel 31 having a rectangular region surrounded by is shown.

  The right side of FIG. 5 shows the pixel 31 ′ after the left side pixel 31 is converted into the coordinate system of the first image. Accordingly, the pixel 31 on the left side of FIG. 5 and the pixel 31 ′ on the right side of FIG. 5 show the same subject (a part thereof) (for example, a certain landscape).

  Here, it is assumed that the pixel value of each pixel of the second and subsequent captured images is obtained as an integrated value of light (light from the subject) at the time of capturing the first image that is the reference image. The integration range when obtaining is set with reference to the first coordinate system. As a result, the first to N-th captured images that have been misaligned due to camera shake are so-called aligned (alignment based on the first captured image).

  The position (i-1, j-1) in the coordinate system of the second image is converted by the conversion parameters (θ2, T2x, T2y, S2), so that the position P in the coordinate system of the first image is obtained. (2, i-1, j-1). Further, the position (i−1, j) in the coordinate system of the second image is converted by the conversion parameters (θ2, T2x, T2y, S2), thereby the position P in the coordinate system of the first image. (2, i-1, j). Similarly, the positions (i, j−1) and (i, j) in the coordinate system of the second image are also converted by the conversion parameters (θ2, T2x, T2y, S2), thereby the first image. The positions P (2, i, j−1) and P (2, i, j) in the coordinate system can be respectively set. When the X coordinate axis or the Y coordinate axis is simply expressed as X or Y, it represents the X coordinate axis or the Y coordinate axis in the coordinate system of the first image.

  Similar to the second image described above, the positions (i−1, j−1), (i−1, j), (i, j−1) in the coordinate system of the kth image (k = 3 to N). ), And (i, j) are also converted by the conversion parameters (θk, Tkx, Tky, Sk), so that the position P (k, i−1, j−1) in the coordinate system of the first image is P (k, i-1, j), P (k, i, j-1), and P (k, i, j), respectively.

  Then, for the k-th image (k = 3 to N), when considering the noise amount E gained up to Mk times, Expressions (20) to (22) similar to Expressions (17) to (19) are used. The expressed light quantity integration conditional expression is obtained.

・・・（２０）

... (20)

Expression (20) is for calculating the true green light quantity Lg (x, y) as P (k, ig-1, jg-1), P (k, ig-1, jg), P (k, ig, jg). -1) and P (k, ig, jg), that is, (ig-1 _(k) , jg-1 _(k) ), (ig-1 _{(k) )} , Jg _(k) ), (ig _(k) , jg-1 _(k) ) and (ig _(k) , jg _(k) ) The pixel value Gobs (k, ig, jg) obtained at “ig-th and jg-th pixels” is equal in consideration of the noise amount E (error) gained up to Mk times, that is, true green The difference (absolute value) between the integrated value of the light quantity Lg (x, y) of the pixel and the observed pixel value Gobs (k, ig, jg) is equal to or less than the maximum noise amount that can be assumed. Yes. Here, | x | represents the absolute value of x.

・・・（２１）

... (21)

The expression (21) indicates that the true red light amount Lr (x, y) is expressed as P (k, ir-1, jr-1), P (k, ir-1, jr), P (k, ir, jr). -1) and P (k, ir, jr), ie, in the first coordinate system, (ir-1 _(k) , jr-1 _(k) ), (ir-1 _{(k )} , Jr _(k) ), (ir _(k) , jr-1 _(k) ) and (ir _(k) , jr _(k) ) The pixel value Robs (k, ir, jr) obtained in “ir-th and jr-th pixels” is equal in consideration of the noise amount E (error) gained up by Mk times, that is, true red The difference (absolute value) between the integrated value of the light amount Lr (x, y) and the observed pixel value Robs (k, ir, jr) is equal to or less than the maximum noise amount that can be assumed. Yes. Here, | x | represents the absolute value of x.

・・・（２２）

(22)

Equation (22) is obtained by converting the true blue light quantity Lb (x, y) into P (k, ib-1, jb-1), P (k, ib-1, jb), P (k, ib, jb). -1) and P (k, ib, jb), that is, (ib-1 _(k) , jb-1 _(k) ), (ib-1 _{(k) )} , Jb _(k) ), (ib _(k) , jb-1 _(k) ), and (ib _(k) , jb _(k) ) The pixel value Bobs (k, ib, jb) obtained at “ib-th and jb-th pixels” is equal in consideration of the noise amount E (error) gained up to Mk times, that is, true blue The difference (absolute value) between the integrated value of the light amount Lb (x, y) and the observed pixel value Bobs (k, ib, jb) is equal to or less than the maximum noise amount that can be assumed. Yes. Here, | x | represents the absolute value of x.

  Next, a first embodiment of the image estimation process in step S4 in FIG. 2 will be described with reference to the flowchart in FIG.

  First, in step S11, the arithmetic circuit 24 calculates the light intensity integration represented by the equation (9) for all (ig, jg) from the pixel values Gobs (1, ig, jg) of each pixel of the first image. The conditional expression is changed from Robs (1, ir, jr), and the light quantity integration conditional expression expressed by the expression (10) for all (ir, jr) is changed from Bobs (1, ib, jb) to all (ib , Jb), a light quantity integration conditional expression represented by the expression (11) is generated, and the process proceeds to step S12.

  In step S12, the arithmetic circuit 24 calculates the light quantity integration conditional expression expressed by the expression (17) for all (ig, jg) from the pixel values Gobs (2, ig, jg) of each pixel of the second image. , Robs (2, ir, jr), the light quantity integration conditional expression expressed by the equation (18) for all (ir, jr) from Bobs (2, ib, jb) to all (ib, jb) Is generated, and the process proceeds to step S13. Here, the arithmetic circuit 24 uses the conversion parameters (θ2, T2x, T2y, S2) supplied from the motion detection circuit 23-1 to convert the position of the second image into the position of the first image. .

  In step S13, the arithmetic circuit 24 expresses the expression (20) for all (ig, jg) from the pixel values Gobs (k, ig, jg) of each pixel of the k-th image (k = 3 to N). From Robs (k, ir, jr), the light quantity integration conditional expression expressed by the equation (21) for all (ir, jr) is expressed as Bobs (k, ib, jb). A light quantity integration conditional expression expressed by Expression (22) for all (ib, jb) is generated, and the process proceeds to Step S14. Here, in order to convert the position of the k-th image into the position of the first image, the arithmetic circuit 24 converts the conversion parameters (θk, Tkx, Tky, Sk) supplied from the motion detection circuit 23- (k-1). ).

  In step S14, the arithmetic circuit 24 calculates the equations (9), (10), (11), (17), (18), (19), (20), (21), and the values obtained in steps S11 to S13, and By calculating a solution that satisfies the light quantity integration conditional expression (22), the true green light quantity Lg (x, y), the true red light quantity Lr (x, y), and the true blue light quantity Lb (x, y) y) is estimated and the process returns. The true green light amount Lg (x, y), the true red light amount Lr (x, y), and the true blue light amount Lb (x, y) (estimated values) obtained here are one sheet to be obtained. Are supplied to the D / A converter 9 or the codec 12 as image signals (G signal, R signal, B signal).

  Note that the solution satisfying all the light quantity integration conditional expressions in step S14, that is, the true green light quantity Lg (x, y), the true red light quantity Lr (x, y), and the true blue light quantity Lb (x , Y), a solution for obtaining image data satisfying a plurality of conditions is used. As such a solution, for example, there is a solution of Projection Onto Convex Sets (POCS method). The POCS method is a method for estimating an optimal solution that meets the constraint conditions by repeating convex projection. For example, the paper “DC Youla and H. Webb,“ Image Restoration by the Method of Convex Projections: part 1 theory ”, IEEE Trans Med. Image., Vol. 1 No. 2, pp81-94, Oct. 1982, etc., and the description thereof is omitted. Japanese Patent Laid-Open No. 8-263639 describes a method for creating a high-resolution still image from a plurality of low-resolution moving images using the POCS method. In this prior art method, motion correction with a specific accuracy such as one pixel or half pixel using a motion vector is fundamental, and the number of pixels considered when estimating the value of each pixel is the specific accuracy. It is considered that a fixed one determined by is used. On the other hand, in the present embodiment, motion correction by an analog method can be applied, and the number of pixels considered when estimating the value of each pixel is arbitrarily adaptive depending on the blurring situation. It is configured to change.

  With reference to the flowchart of FIG. 7, the first embodiment of the image estimation process in step S4 of FIG. 2 will be described again.

  First, in step S21, the arithmetic circuit 24 sets a variable k for counting the number of images held therein to 1 and proceeds to step S22.

  In step S22, the arithmetic circuit 24 calculates the light intensity integration condition represented by the equation (20) for all (ig, jg) from the pixel value Gobs (k, ig, jg) of each pixel of the k-th image. An expression is generated, and the process proceeds to step S23.

  In step S23, the arithmetic circuit 24 calculates the light intensity integration condition represented by the equation (21) for all (ir, jr) from the pixel values Robs (k, ir, jr) of each pixel of the k-th image. An expression is generated, and the process proceeds to step S24.

  In step S24, the arithmetic circuit 24 calculates the light intensity integration condition represented by the expression (22) for all (ib, jb) from the pixel values Bobs (k, ib, jb) of each pixel of the k-th image. An expression is generated, and the process proceeds to step S25.

  In steps S22 to S24, when generating the light quantity integration conditional expression, conversion parameters supplied from the motion detection circuits 23-1 to 23- (N-1) are used as necessary.

  In step S25, the arithmetic circuit 24 determines whether or not the internal variable k is equal to the number N of images supplied from the frame memories 22-1 to 22-N. If it is determined that the variable k is not equal to the number N of images, the arithmetic circuit 24 proceeds to step S26, increments the variable k by 1, and then returns to step S22. Then, the processes of steps S22 to S25 are repeated.

  On the other hand, when it is determined that the variable k is equal to the number N of images, the process proceeds to step S27, and the arithmetic circuit 24 calculates the equations (20), (21), and k = 1 to N obtained in steps S22 to S24, and By calculating the light quantity integration conditional expression (22), the true green light quantity Lg (x, y), the true red light quantity Lr (x, y), and the true blue light quantity Lb (x, y) are obtained. Estimate and return. The true green light amount Lg (x, y), the true red light amount Lr (x, y), and the true blue light amount Lb (x, y) obtained here are the one clear image to be obtained. The image signal (G signal, R signal, B signal) is supplied to the D / A converter 9 or the codec 12.

  The true green light amount Lg (x, y), the true red light amount Lr (x, y), and the true blue light amount Lb (x, y) obtained here are analog signals. . That is, Lg (x, y), Lr (x, y), and Lb (x, y) are obtained as a function of (x, y), but the variables x and y are not integers as described above. Real number, including fractional part. Although the number of digits of the decimal part depends on the accuracy of the computing device, it is generally considered to be about 2 to 3 digits in binary representation. Therefore, Lg (x, y), Lr (x, y), and Lb (x, y), which are functions having real numbers (x, y) as arguments, are again (x, y) as necessary. It is possible to perform resampling by re-sampling only integer positions as y), convert it into a digital image signal, and supply it to the D / A converter 9 or the codec 12.

  Further, when the subject is moving during N imaging times by the high-speed shutter, the true green light amount Lg (x, y) and the true red light amount Lr (x, y) of the moving part. Since the true blue light amount Lb (x, y) changes with time, it may be difficult to obtain a correct solution by the above-described method.

  Therefore, a portion where the subject is moving can be processed by simple superposition as exception processing. That is, each of the N images is Bayer array data (only one of the R signal, G signal, and B signal per pixel). From this data, the R signal, G per pixel Demosaicing processing is performed to restore the three signals, the signal and the B signal. Further, the N images after the demosaicing process are aligned by averaging by performing rotation, enlargement, reduction, or parallel movement. As a method of demosaicing processing, any method including a conventional method can be adopted.

  As described above, in the first embodiment, it is possible to estimate a clear image in which camera shake is corrected by performing a process that takes into account the effect of the on-chip lens directly above each pixel.

(Second Embodiment)
Next, a second embodiment of the image estimation process in the signal processing circuit 7 will be described.

  In the second embodiment, an R signal, a G signal, and a B signal in addition to the light quantity integration conditional expressions represented by the equations (20) to (22) in which k = 1 to N in the first embodiment. Color correlation conditions relating to the correlation between each other are added, and the true green light amount Lg (x, y), the true red light amount Lr (x, y), and the true blue light amount Lb satisfying all the conditional expressions. (X, y) is obtained.

  Focusing on the local portion of the image, the true green light amount Lg (x, y) equivalent to the light of the subject incident on the image sensor 4, the true red light amount Lr (x, y), the true light amount. The blue light quantity Lb (x, y) has a correlation between colors (color correlation). Therefore, in addition to equations (20) to (22), which are light intensity integration conditional expressions, by adding a color correlation condition, a more accurate solution, that is, a clearer image that is more faithful to the original light can be obtained. Can be obtained (estimated).

  A specific method for obtaining a color correlation condition will be described with reference to FIGS. In FIG. 8 and FIG. 9, the condition of the color correlation between green and red is considered.

  The ig-th and jg-th green pixels G (jg-1) (ig-1) of the k'th image on the lower left side of FIG. 8 and the irth of the k "th image on the lower-right side of FIG. Note the jrth red pixel R (jr-1) (ir-1).

  The arithmetic circuit 24 determines the positions of the green pixel G (jg-1) (ig-1) of the k'th image and the red pixel R (jr-1) (ir-1) of the k "th image. As described in the first embodiment, conversion is performed using the conversion parameters (θk ′, Tk′x, Tk′y, Sk ′) and the conversion parameters (θk ″, Tk ″ x, Tk ″ y, Sk ″). Thus, the position in the coordinate system of the first image on the upper side of Fig. 8 is obtained.Including the case where k 'and k "are 1, in this case, (θ1, T1x, T1y, S1) = (0 , 0, 0, 1).

  The arithmetic circuit 24 then converts the k′-th green pixel G (jg−1) (ig−1) converted into the coordinate system of the first image and the k converted into the coordinate system of the first image. “Calculate the distance from the first red pixel R (jr−1) (ir−1). Further, the arithmetic circuit 24 allows an allowable value (determination value) delta (for example, the distance) to be regarded as the same position. , 0.25 pixel).

For example, consider the positions of the pixel G (jg-1) (ig-1) and the pixel R (jr-1) (ir-1) at the position (ig, jg) and the position (ir, jr), respectively. , the pixel G (jg-1) in the coordinate system of the k 'th image (ig-1) position of (ig, jg) and the point G _c and the pixel in the coordinate system of the k "th image R (jg-1 ) (Ig-1) at the position (ir, jr) as the point R _c and the position (ig, jg) at the pixel G (jg-1) (ig-1) in the coordinate system of the first image as G _{c ( k ′)} and the position (ir, jr) of the pixel R (jg−1) (ig−1) in the coordinate system of the first image as R _{c (k ″)} , respectively, the point G _c Expression (23) indicating whether the distance between _{(k ′)} and the point R _{c (k ″)} is within the allowable value delta can be written as follows.

・・・（２３）

(23)

Expression (23) is referred to as a distance conditional expression. Here, Dis [G _{c (k ′)} , R _{c (k ″)} ] represents a distance between the point G _{c (k ′)} and the point R _{c (k ″)} . Further, the positions represented by the point G _{c (k ′)} and the point R _{c (k ″)} are the same as the expressions (1) to (3) in the position (ig, jg) and the position (ir, jr). , Affine transformation is performed with the conversion parameters (θk ′, Tk′x, Tk′y, Sk ′) and (θk ″, Tk ″ x, Tk ″ y, Sk ″), respectively.

  The arithmetic circuit 24 has an allowable range in which the k'th green pixel G (jg-1) (ig-1) and the k "th red pixel R (jr-1) (ir-1) are within a certain allowable range. Whether or not there is a pixel that can be considered to be at the same position when considering the margin of delta is centered at a certain position (x, y) in the coordinate system of the first image (x ± dX, y A region near (± dY), that is, a region surrounded by (x−dX, y−dY), (x−dX, y + dY), (x + dX, y−dY), and (x + dX, y + dY) is obtained. , DX and dY are predetermined values for setting the neighborhood region, and can be, for example, the lengths in the X direction and Y direction for two pixels.

  In other words, the arithmetic circuit 24 is a region near (x ± dX, y ± dY) around the position (x, y) in the coordinate system of the first image, that is, (x−dX, y−dY), (x−dX, y + dY), (x + dX, y−dY), and (x + dX, y + dY) and (k ′, ig, jg) satisfying the above equation (23) (K ″, ir, jr) is obtained.

  Then, the arithmetic circuit 24 calculates the pixel value Gobs (k ′, ig, jg) and the pixel value Robs (k ″, kg) corresponding to the obtained (k ′, ig, jg) and (k ″, ir, jr). ir, jr).

  The arithmetic circuit 24 obtains (k ′, ig, jg) and (k ″, ir, jr) satisfying the above equation (23) for all combinations in which k ′ and k ″ are 1 to N, respectively. .

  In general, a combination of a plurality of (k ′, ig, jg) and (k ″, ir, jr) is detected, so that the arithmetic circuit 24 detects the detected (k ′, ig, jg) and The pixel value Gobs (k ′, ig, jg) and the pixel value Robs (k ″, ir, jr) corresponding to (k ″, ir, jr) are shown in FIG. (Gobs (k ′, ig, jg)), and the vertical axis is plotted in the GR space with the R signal (Robs (k ″, ir, jr)).

  FIG. 9 schematically shows pixel values Gobs (k ′, ig, jg) and pixel values Robs (k ″, ir, jr) of pixels that can be regarded as being in the same position plotted in the GR space. FIG.

  The crosses in FIG. 9 indicate pixel values Gobs (k ′, ig, jg) and pixel values Robs corresponding to (k ′, ig, jg) and (k ″, ir, jr) detected by the arithmetic circuit 24. This represents a pair with (k ″, ir, jr).

  Accordingly, in the vicinity region of the position (x, y), the true green light amount Lg (x, y) and the true red light amount Lr (x, y) to be obtained are as shown in FIG. There seems to be a correlation.

  Therefore, in the second embodiment, in addition to the light quantity integration conditional expression expressed by the equations (20) to (22) in the first embodiment, the correlation between green and red shown in FIG. It is added as a condition that there is.

  That is, the arithmetic circuit 24 is represented by a set of pixel values Gobs (k ′, ig, jg) and pixel values Robs (k ″, ir, jr) that satisfy the distance conditional expression of Expression (23) by the arithmetic circuit 24. The principal component analysis is performed on the plurality of points plotted in the GR space of FIG.

  The arithmetic circuit 24 then applies a component (for example, the second principal component) orthogonal to the direction of the principal component (first principal component) (the direction indicated by the thick arrow in FIG. 9), which is the analysis result of the principal component analysis. Find the variance of. Further, for the position (x, y), the arithmetic circuit 24 includes a true green light amount Lg (x, y), a band having a dispersion width of a component orthogonal to the direction of the main component in the GR space, A conditional expression that there is a point represented by the true red light amount Lr (x, y) is set as a condition for color correlation.

  The above color correlation conditions are also considered for green and blue.

  As in the case of the green and red pixels shown in FIG. 8, the arithmetic circuit 24 performs the k′-th green pixel G (jg−1) (ig−1) and the k ″ ′-th blue pixel. As described in the first embodiment, the position of B (jb-1) (ib-1) is converted into the conversion parameter (θk ′, Tk′x, Tk′y, Sk ′) and the conversion parameter (θk ″ ′). , Tk ″ ′ x, Tk ″ ′ y, Sk ″ ′) to obtain the position of the first image in the coordinate system.

  The arithmetic circuit 24 then converts the k′-th green pixel G (jg−1) (ig−1) converted into the coordinate system of the first image and the k converted into the coordinate system of the first image. The distance to the '' th blue pixel B (jb-1) (ib-1) is calculated. Further, the arithmetic circuit 24 is within an allowable value (determination value) delta that the distance is regarded as the same position. Is determined in the same manner as in FIG.

For example, consider the positions of pixel G (jg-1) (ig-1) and pixel B (jb-1) (ib-1) at position (ig, jg) and position (ib, jb), respectively. , 'position of the pixel in the coordinate system of the piece of the image G (jg-1) (ig -1) (ig, jg) and the point G _c a, k "' k pixels in the coordinate system of the piece of the image B (jb- 1) The position (ib, jb) of (ib-1) is the point B _c, and the position (ig, jg) of the pixel G (jg-1) (ig-1) in the coordinate system of the first image is G _{c (k ′)} and the position (ib, jb) of the pixel B (jb−1) (ib−1) in the coordinate system of the first image are respectively represented as B _{c (k ″ ′).} Expression (24) representing whether the distance between G _{c (k ′)} and the point B _{c (k ″ ′)} is within the allowable value delta can be written as follows.

・・・（２４）

... (24)

Expression (24) is referred to as a distance conditional expression. Here, Dis [G _{c (k ′)} , B _{c (k ′ ′)} ] represents a distance between the point G _{c (k ′)} and the point B _{c (k ′ ′)} . Further, the positions represented by the point G _{c (k ′)} and the point B _{c (k ″ ′)} are the positions (ig, jg) and (ib, jb) as in the expressions (1) to (3). , Affine transformation is performed with the conversion parameters (θk ′, Tk′x, Tk′y, Sk ′) and (θk ″ ′, Tk ″ ′ x, Tk ″ ′ y, Sk ″ ′).

  The arithmetic circuit 24 has a k'th green pixel G (jg-1) (ig-1) and a k''th blue pixel B (jb-1) (ib-1). Whether or not there is a pixel that can be regarded as being in the same position when considering the margin of the range delta is centered on a certain position (x, y) in the coordinate system of the first image (x ± dX, A region near y ± dY), that is, a region surrounded by (x−dX, y−dY), (x−dX, y + dY), (x + dX, y−dY), and (x + dX, y + dY) is obtained here. In this case, dX and dY are predetermined values for setting the neighborhood region, and can be, for example, the lengths in the X direction and Y direction for two pixels.

  In other words, the arithmetic circuit 24 is a region near (x ± dX, y ± dY) around the position (x, y) in the coordinate system of the first image, that is, (x−dX, y-dY), (x-dX, y + dY), (x + dX, y-dY), and (x + dX, y + dY) and (k ′, ig, jg) satisfying the above-described expression (24) (K ″ ′, ib, jb) is obtained.

  Then, the arithmetic circuit 24 calculates the pixel value Gobs (k ′, ig, jg) and the pixel value Bobs (k ″ (k ″) corresponding to the obtained (k ′, ig, jg) and (k ″ ′, ib, jb). ', Ib, jb).

  The arithmetic circuit 24 calculates (k ′, ig, jg) and (k ″, ib, jb) satisfying the above equation (24) for all combinations in which k ′ and k ″ ′ are 1 to N, respectively. Ask.

  In general, since a combination of a plurality of (k ′, ig, jg) and (k ″ ′, ib, jb) is detected, the arithmetic circuit 24 detects (k ′, ig, jg). And the pixel value Gobs (k ′, ig, jg) and the pixel value Bobs (k ″ ′, ib, jb) corresponding to (k ″ ′, ib, jb) and the horizontal axis the G signal (Gobs (k ′ (k ′) , Ig, jg)), and the vertical axis is plotted in a GB space such that the B signal (Bobs (k ″ ′, ib, jb)) is used.

  Therefore, in the second embodiment, in addition to the light quantity integration conditional expression expressed by the equations (20) to (22) in the first embodiment, the same as in the case of green and red in FIG. Is added on condition that the color correlation is also in green and blue.

  That is, the arithmetic circuit 24 is a plurality of points plotted in the GB space, and the pixel value Gobs (k ′, ig, jg) and the pixel value Bobs (k ″ ′, ib, jb) detected by the arithmetic circuit 24. The principal component analysis is performed for each pair.

  Then, the arithmetic circuit 24 obtains the variance for the component (for example, the second principal component) orthogonal to the direction of the principal component (first principal component), which is the analysis result of the principal component analysis. Further, for the position (x, y), the arithmetic circuit 24 includes a true green light amount Lg (x, y), a band having a dispersion width of a component orthogonal to the direction of the main component in the GB space, A conditional expression that there is a point represented by the true blue light quantity Lb (x, y) is set as a color correlation condition.

  Therefore, the true green light amount Lg (x, y), the true red light amount Lr (x, y), and the true blue light amount Lb (x, y) that are finally obtained in the arithmetic circuit 24 are: A component in which the point represented by the true green light amount Lg (x, y) and the true red light amount Lr (x, y) at the position (x, y) is orthogonal to the direction of the main component in the GR space. And is represented by a true green light amount Lg (x, y) and a true blue light amount Lb (x, y) at a position (x, y). In the GB space, the points are limited (constrained) to exist in a band shape having a dispersion width of the component orthogonal to the direction of the main component.

  In the present embodiment, only the two color correlation conditions in the above-described GR and GB spaces are added. Similarly, the condition of the color correlation between the R signal and the B signal (in the RB space) is also described. May also be added.

  True green light quantity Lg (x, y), true red light quantity Lr (x, y), and true blue light quantity Lb (x, y) satisfying both the light quantity integration conditional expression and the color correlation conditional expression For example, the POCS method or the like can be adopted as in the first embodiment.

  The position (x, y) to which the color correlation condition is added may be all positions (x, y), or may be only the position of a grid point where x and y are integers, for example.

  A second embodiment of the image estimation process in step S4 of FIG. 2 will be described with reference to the flowchart of FIG.

  In steps S31 to S33, processing similar to that in steps S11 to S13 of the image estimation processing of the first embodiment shown in FIG. 6 is performed.

  That is, in step S31, the arithmetic circuit 24 calculates the light intensity integration condition represented by the equation (9) for all (ig, jg) from the pixel values Gobs (1, ig, jg) of each pixel of the first image. From Robs (1, ir, jr), the light quantity integration conditional expression expressed by Equation (10) for all (ir, jr) is changed from Bobs (1, ib, jb) to all (ib, The light quantity integration conditional expression represented by the expression (11) for jb) is generated, and the process proceeds to step S32.

  In step S <b> 32, the arithmetic circuit 24 calculates a light quantity integration conditional expression expressed by Expression (17) for all (ig, jg) from the pixel values Gobs (2, ig, jg) of each pixel of the second image. , Robs (2, ir, jr), the light quantity integration conditional expression expressed by the equation (18) for all (ir, jr) from Bobs (2, ib, jb) to all (ib, jb) Is generated, and the process proceeds to step S33. Here, the arithmetic circuit 24 uses the conversion parameters (θ2, T2x, T2y, S2) supplied from the motion detection circuit 23-1 to convert the position of the second image into the position of the first image. .

  In step S <b> 33, the arithmetic circuit 24 uses the equation (20) for all (ig, jg) from the pixel values Gobs (k, ig, jg) of each pixel of the k-th (k = 3 to N) image. The light quantity integration conditional expression expressed from Robs (k, ir, jr) and the light quantity integration conditional expression expressed by Expression (21) for all (ir, jr) from Bobs (k, ib, jb). Then, the light quantity integration conditional expression represented by the expression (22) for all (ib, jb) is generated, and the process proceeds to step S34. Here, in order to convert the position of the k-th image into the position of the first image, the arithmetic circuit 24 converts the conversion parameters (θk, Tkx, Tky, Sk) supplied from the motion detection circuit 23- (k-1). ).

  In step S34, the arithmetic circuit 24 satisfies the distance conditional expression (23) for the predetermined position (x, y) in the vicinity region of the position (x, y) (k ′, ig, jg) and (k ″, ir, jr) are obtained, and the process proceeds to step S35.

  In step S35, the arithmetic circuit 24 calculates the pixel values Gobs (k ′, ig, jg) and Robs (k ″, k ”, (k ′, ig, jg) obtained in step S34. The points represented by ir, jr) are plotted in the GR space and the principal component analysis is performed. Then, the arithmetic circuit 24 obtains the variance for the component orthogonal to the direction of the principal component, which is the analysis result of the principal component analysis, and the color correlation that the G signal and the R signal of the same pixel exist within the variance range. The conditional expression is established, and the process proceeds from step S35 to step S36.

  In step S36, the arithmetic circuit 24 satisfies the conditional expression (k ′, ig, jg) for the distance of the expression (24) in the region near the position (x, y) with respect to the predetermined position (x, y). ) And (k ″ ′, ib, jb) are obtained, and the process proceeds to step S37.

  In step S37, the arithmetic circuit 24 obtains the pixel values Gobs (k ′, ig, jg) and Bobs (k ″) of (k ′, ig, jg) and (k ″ ′, ib, jb) obtained in step S36. The points represented by ', ib, jb) are plotted in the GB space, and the principal component analysis is performed. Then, the arithmetic circuit 24 obtains the variance for the component orthogonal to the direction of the principal component, which is the analysis result of the principal component analysis, and the color correlation that the G signal and B signal of the same pixel exist within the variance range. The conditional expression is established, and the process proceeds from step S37 to step S38.

  In step S38, the arithmetic circuit 24 determines whether or not a color correlation conditional expression has been obtained for all the preset positions (x, y). If it is determined in step S38 that the color correlation conditional expression has not yet been obtained for all positions (x, y), the process returns to step S34, and the arithmetic circuit 24 has obtained the color correlation conditional expression. Pay attention to (select) the position (x, y) that does not exist, and repeat the processing of steps S34 to S38.

  On the other hand, if it is determined in step S38 that conditional expressions for color correlation have been obtained for all positions (x, y), the process proceeds to step S39, and the arithmetic circuit 24 is obtained in steps S31 to S33, S35, and S37. The true green light amount Lg (x, y), the true red light amount Lr (x, y), and the true blue light amount Lb (x, y) satisfying all the conditional expressions are calculated, and the process returns. . That is, the arithmetic circuit 24 calculates the true green light quantity Lg (x, y) that satisfies all of the light quantity integration conditional expression obtained in steps S31 to S33 and the color correlation conditional expression obtained in steps S35 and S37. ), The true red light amount Lr (x, y) and the true blue light amount Lb (x, y) are calculated.

  With reference to the flowcharts of FIG. 11 and FIG. 12, the second embodiment of the image estimation process in step S4 of FIG. 2 will be described again.

  Steps S51 to S56 are the same as steps S21 to S26 of the image estimation process of the first embodiment shown in FIG.

  That is, in step S51, the arithmetic circuit 24 sets a variable k for counting the number of images held therein to 1 and proceeds to step S52.

  In step S52, the arithmetic circuit 24 calculates the light intensity integration condition represented by the equation (20) for all (ig, jg) from the pixel value Gobs (k, ig, jg) of each pixel of the k-th image. An expression is generated, and the process proceeds to step S53.

  In step S53, the arithmetic circuit 24 calculates the light intensity integration condition represented by the equation (21) for all (ir, jr) from the pixel values Robs (k, ir, jr) of each pixel of the k-th image. An expression is generated, and the process proceeds to step S54.

  In step S54, the arithmetic circuit 24 calculates the light intensity integration condition represented by the equation (22) for all (ib, jb) from the pixel values Bobs (k, ib, jb) of each pixel of the k-th image. An expression is generated, and the process proceeds to step S55.

  In steps S52 to S54, when generating the light quantity integration conditional expression, conversion parameters supplied from the motion detection circuits 23-1 to 23- (N-1) are used as necessary.

  In step S55, the arithmetic circuit 24 determines whether or not the internal variable k is equal to the number N of images supplied from the frame memories 22-1 to 22-N. If it is determined that the variable k is not equal to the number N of images, the arithmetic circuit 24 proceeds to step S56, increments the variable k by 1, and then returns to step S52. Then, the processes in steps S52 to S56 are repeated.

  On the other hand, if it is determined in step S55 that the variable k is equal to the number N of images, the process proceeds to step S57 in FIG.

  Steps S57 to S62 are the same as steps S34 to S39 in FIG.

  That is, in step S57, the arithmetic circuit 24 satisfies the distance conditional expression (23) for the predetermined position (x, y) in the vicinity region of the position (x, y) (k ′, ig, jg) and (k ″, ir, jr) are obtained, and the process proceeds to step S58.

  In step S58, the arithmetic circuit 24 calculates the pixel values Gobs (k ′, ig, jg) and Robs (k ″, k ”, (k ′, ig, jg) obtained in step S57. The points represented by ir, jr) are plotted in the GR space and the principal component analysis is performed. Then, the arithmetic circuit 24 obtains the variance for the component orthogonal to the direction of the principal component, which is the analysis result of the principal component analysis, and the color correlation that the G signal and the R signal of the same pixel exist within the variance range. After the conditional expression is established, the process proceeds from step S58 to step S59.

  In step S59, the arithmetic circuit 24 satisfies the conditional expression (k ′, ig, jg) for the distance of the expression (24) in the vicinity region of the position (x, y) with respect to the predetermined position (x, y). ) And (k ″ ′, ib, jb) are obtained, and the process proceeds to step S60.

  In step S60, the arithmetic circuit 24 calculates the pixel values Gobs (k ′, ig, jg) and Bobs (k ″ ′) of (k ′, ig, jg) and (k ″ ′, ib, jb) obtained in step S59. , Ib, jb) are plotted in the GB space and the principal component analysis is performed. Then, the arithmetic circuit 24 obtains the variance for the component orthogonal to the direction of the principal component, which is the analysis result of the principal component analysis, and the color correlation that the G signal and B signal of the same pixel exist within the variance range. Is established, and the process proceeds from step S60 to step S61.

  In step S61, the arithmetic circuit 24 determines whether or not a color correlation conditional expression has been obtained for all preset positions (x, y). If it is determined in step S61 that the color correlation conditional expression has not yet been obtained for all positions (x, y), the process returns to step S57, and the arithmetic circuit 24 has obtained the color correlation conditional expression. Pay attention to (select) the position (x, y) that does not exist, and repeat the processing of steps S57 to S61.

  On the other hand, if it is determined in step S61 that the conditional expressions for color correlation have been obtained for all positions (x, y), the process proceeds to step S62, and the arithmetic circuit 24 is obtained in steps S52 to S54, S58, and S60. The true green light amount Lg (x, y), the true red light amount Lr (x, y), and the true blue light amount Lb (x, y) satisfying all the conditional expressions are calculated, and the process returns. . That is, the arithmetic circuit 24 calculates the true green light quantity Lg (x, y) that satisfies all of the light quantity integration conditional expressions obtained in steps S52 to S54 and the color correlation conditional expression obtained in steps S58 and S60. ), The true red light amount Lr (x, y) and the true blue light amount Lb (x, y) are calculated.

  As described above, the R signal, the G signal, and the B signal in addition to the light quantity integration conditional expressions expressed by the equations (20), (21), and (22) in which k = 1 to N in the first embodiment. A color correlation condition relating to the correlation with the signal is added, and the true green light amount Lg (x, y), the true red light amount Lr (x, y), and the true blue color satisfying all the conditional expressions. By obtaining the light amount Lb (x, y) of the light, it is possible to obtain a clear image that is more faithful to the original light.

  In the above-described case, the conditional expression for color correlation in the two-dimensional color space is used. However, it is possible to use the conditional expression in the three-dimensional color space.

(Third embodiment)
Next, a third embodiment of image estimation processing in the signal processing circuit 7 will be described.

  In the first and second embodiments, the characteristics of the on-chip lens immediately above each pixel of the image sensor 4 are accurately formulated (applied), and the true green light amount Lg (x, y), true A red light amount Lr (x, y) and a true blue light amount Lb (x, y) were estimated to obtain a clear image.

  In the third embodiment, the effect of the on-chip lens is ignored, and the data (the amount of received light) received by each pixel of the image sensor 4 is incident on a certain point (for example, the center of gravity of the pixel) of the pixel. The light value of the subject (point-sampled data).

  In the third embodiment, a method using a spring model described later is adopted. Thereby, as compared with the POCS method used in the first and second embodiments, the true green light amount Lg (x, y), the true red light amount Lr (x, y), and the true blue light It is possible to reduce the amount of calculation when obtaining the amount of light Lb (x, y). Also in the third embodiment, the same effect as in the first and second embodiments can be obtained that a clearer image can be estimated as compared with the conventional case.

  In the third embodiment, as described above, since the data received by each pixel is considered as point-sampled data, among the pixels of the image sensor 4 in FIG. The data received by the “jth pixel” is, for example, received at the center position (coordinates) (i−0.5, j−0.5) of the “ith, jth pixel”. .

  Therefore, for example, the pixel value Gobs (1, ig, jg) as data obtained from the “ig-th, jg-th pixel” of the first image is the position (ig-0. 5, jg-0.5) is the amount of green light Lg (ig-0.5, jg-0.5) received. Similarly, the pixel value Robs (1, ir, jr) as data obtained from the “ir-th and jr-th pixels” of the first image is the position (ir−0.5) of the coordinate system of the first image. , Jr-0.5) is the amount of red light Lr (ir-0.5, jr-0.5) received at the first image, and is obtained as “ib-th, jb-th pixel”. Pixel values Bobs (1, ib, jb) of the blue light quantity Lb (ib-0.5, jb-0.5) received at the position (ib-0.5, jb-0.5) of the coordinate system of the first image. jb-0.5).

  FIG. 13 shows the first image stored in the frame memory 22-1.

  In FIG. 13, for the pixel G (jg-1) (ig-1) that receives the green component, the pixel value Gobs (1, ig, jg) is observed at the position indicated by the black circle. For the pixel R (jr-1) (ir-1) that receives the red component, the pixel value Robs (1, ir, jr) is observed at the position indicated by the black square. For the pixel B (jb-1) (ib-1) that receives the blue component, the pixel value Bobs (1, ib, jb) is observed at the position indicated by the black triangle. As described above, the pixel value of the first image is the center (center of gravity) of each pixel on the coordinate system of the first image, that is, (i−0.5, j -0.5) is observed.

  FIG. 14 shows the second image stored in the frame memory 22-2.

  In FIG. 14, for the pixel G (jg-1) (ig-1) that receives the green component, the pixel value Gobs (2, ig, jg) is observed at the position indicated by the black circle. For the pixel R (jr-1) (ir-1) that receives the red component, the pixel value Robs (2, ir, jr) is observed at the position indicated by the black square. For the pixel B (jb-1) (ib-1) that receives the blue component, the pixel value Bobs (2, ib, jb) is observed at the position indicated by the black triangle. As described above, the pixel value of the second image is the central part (center of gravity) of each pixel on the coordinate system of the second image, that is, (i−0.5, j -0.5) is observed.

  A point on the second image is considered with reference to the coordinate system of the first image, as in the first and second embodiments. The points on the second image can be converted into points on the coordinate system of the first image using the conversion parameters (θ2, T2x, T2y, S2) detected by the motion detection circuit 23-1.

  FIG. 14 shows the pixel value Gobs (2, ig, jg), the pixel value Robs (2, ir, jr), the pixel value Bobs of the point on the second image converted into the coordinate system of the first image. (2, ib, jb) is also shown.

That is, the pixel value Gobs (2, ig, jg) of the pixel G (jg-1) (ig-1) in the coordinate system of the second image is set to the position (ig-0. 5, jg-0.5) based on the conversion parameter (θ2, T2x, T2y, S2) detected by the motion detection circuit 23-1, the position ((ig-0.5 ₍₂₎ , (jg−0.5) ₍₂₎ ) is the green light quantity Lg (x, y) observed. In FIG. 14, the position ((ig−0.5) ₍₂₎ , (jg−0.5) ₍₂₎ on the coordinate system of the first image where the pixel value Gobs (2, ig, jg) is observed. ) Are indicated by white circles.

Also, the pixel value Robs (2, ir, jr) of the pixel R (jr-1) (ir-1) in the coordinate system of the second image is the position (ir-0. 5, jr-0.5) based on the conversion parameters (θ2, T2x, T2y, S2) detected by the motion detection circuit 23-1, the position ((ir-0.5) on the coordinate system of the first image. ₍₂₎ , (jr−0.5) ₍₂₎ ) is the red light quantity Lr (x, y) observed. In FIG. 14, the position ((ir−0.5) ₍₂₎ , (jr−0.5) ₍₂₎ on the coordinate system of the first image where the pixel value Robs (2, ir, jr) is observed. ) Is indicated by a white square.

Further, the pixel value Bobs (2, ib, jb) of the pixel B (jb-1) (ib-1) in the coordinate system of the second image is the position (ib-0. 5, jb-0.5) based on the conversion parameter (θ2, T2x, T2y, S2) detected by the motion detection circuit 23-1, the position ((ib-0.5) on the coordinate system of the first image. ₍₂₎ , (jb−0.5) _{(2) The} blue light quantity Lb (x, y) observed. In FIG. 14, the position ((ib−0.5) ₍₂₎ , (jb−0.5) ₍₂₎ on the coordinate system of the first image where the pixel value Bobs (2, ib, jb) is observed. ) Is indicated by a white triangle.

  FIG. 15 shows the third image stored in the frame memory 22-3.

  In FIG. 15, for the pixel G (jg-1) (ig-1) that receives the green component, the pixel value Gobs (3, ig, jg) is observed at the position indicated by the black circle. For the pixel R (jr-1) (ir-1) that receives the red component, the pixel value Robs (3, ir, jr) is observed at the position indicated by the black square. For the pixel B (jb-1) (ib-1) that receives the blue component, the pixel value Bobs (3, ib, jb) is observed at the position indicated by the black triangle. As described above, the pixel value of the third image is the center (center of gravity) of each pixel on the coordinate system of the third image, i.e., (i−0.5, j -0.5) is observed.

  As in the first and second embodiments, points on the third image are considered with reference to the coordinate system of the first image. The point on the third image can be converted into a point on the coordinate system of the first image using the conversion parameters (θ3, T3x, T3y, S3) detected by the motion detection circuit 23-2.

  FIG. 15 shows the pixel value Gobs (3, ig, jg), the pixel value Robs (3, ir, jr), the pixel value Bobs of the point on the third image converted into the coordinate system of the first image. (3, ib, jb) is also shown.

That is, the pixel value Gobs (3, ig, jg) of the pixel G (jg-1) (ig-1) in the coordinate system of the third image is set to the position (ig-0. 5, jg-0.5) by the conversion parameter (θ3, T3x, T3y, S3) detected by the motion detection circuit 23-2, the position ((ig-0.5) in the coordinate system of the first image. ₍₃₎ , (jg−0.5) ₍₃₎ ) is the green light quantity Lg (x, y) observed. In FIG. 15, the position ((ig−0.5) ₍₃₎ , (jg−0.5) ₍₃₎ on the coordinate system of the first image where the pixel value Gobs (3, ig, jg) is observed. ) Are indicated by white circles.

Also, the pixel value Robs (3, ir, jr) of the pixel R (jr-1) (ir-1) in the coordinate system of the third image is the position (ir-0. 5, jr-0.5) based on the conversion parameter (θ3, T3x, T3y, S3) detected by the motion detection circuit 23-2, the position ((ir-0.5) on the coordinate system of the first image. ₍₃₎ , (jr−0.5) ₍₃₎ ) is the red light quantity Lr (x, y) observed. In FIG. 15, the position ((ir−0.5) ₍₃₎ , (jr−0.5) ₍₃₎ on the coordinate system of the first image where the pixel value Robs (3, ir, jr) is observed. ) Is indicated by a white square.

Further, the pixel value Bobs (3, ib, jb) of the pixel B (jb-1) (ib-1) in the coordinate system of the third image is the position (ib-0. 5, jb-0.5) based on the conversion parameter (θ3, T3x, T3y, S3) detected by the motion detection circuit 23-2, the position ((ib-0.5) on the coordinate system of the first image. ₍₃₎ , (jb−0.5) ₍₃₎ ) is the blue light quantity Lb (x, y) observed. In FIG. 15, the position ((ib−0.5) ₍₃₎ , (jb−0.5) ₍₃₎ on the coordinate system of the first image where the pixel value Bobs (3, ib, jb) is observed. ) Is indicated by a white triangle.

  FIG. 16 illustrates pixel values Gobs (1, ig, jg) to Gobs of predetermined pixels G (jg−1) (ig−1) that receive the green component among the pixels of the first to Nth captured images. (N, ig, jg) indicates the position on the coordinate system of the first image where the image is observed.

In FIG. 16, at the center position (pixel center) of the ig-th and jg-th pixels G (jg−1) (ig−1) on the coordinate system of the first image, the pixel value Gobs (1, ig, jg) has been observed. Further, the pixel value Gobs (2, ig, jg) of the second image whose position is converted on the coordinate system of the first image is located on the upper left side of the pixel center of the pixel G (jg-1) (ig-1). ) Has been observed. Further, on the lower left side of the pixel center of the pixel G (jg-1) (ig-1), the pixel value Gobs (3, ig, jg) of the third image whose position is converted on the coordinate system of the first image. ) Has been observed. In addition, on the upper right side of the pixel center of the pixel G (jg-1) (ig-1), the pixel value Gobs (4, ig, jg) of the fourth image whose position is converted on the coordinate system of the first image. ) Has been observed. The pixel values Gobs (k, ig ₎ , jg) (k = 5 to N) of the fifth to Nth images are not shown.

In the third embodiment, the arithmetic circuit 24 determines the position of the first image on the coordinate system based on the pixel values Gobs (k, ig, jg) (k = 1 to N) of the first to Nth images. (i -0.5, j -0.5) true green light intensity Lg (i -0.5, j -0.5) in seeking.

Here, the center position (i−0.5, j−0.5) of the i-th and j-th pixels on the coordinate system of the first image as the reference image is represented as (I ′, J ′). I will decide. That is, I ′ = i−0.5 and J ′ = j−0.5. True green light intensity Lg to be obtained (I ', J'), the center position of the pixel (i -0.5, j -0.5) with Lg (i -0.5, j -0.5 ). Similarly, the true red light amount Lr (I ′, J ′) to be obtained is Lr ( i− 0.5, j− ) using the center position ( i− 0.5, j− 0.5) of the pixel. 0.5), and the true blue light quantity Lb (I ′, J ′) to be obtained is represented by Lb ( i ) using the center position ( i −0.5, j −0.5) of the pixel. -0.5, j -0.5).

  FIG. 17 shows the true red light amount Lr (I ′, J ′), the true red light amount Lr (I ′, J ′), and the true blue light to be obtained by the arithmetic circuit 24 as a clear image signal. The position of the light quantity Lb (I ′, J ′) is shown on the coordinate system of the first image.

  That is, in FIG. 17, the center position (I ′, J ′) of each pixel of the image sensor 4 on the coordinate system of the first image is a true green light amount Lg (I ′, J ′), a true red color. Are indicated by black circles as positions where the true light amount Lr (I ′, J ′) and the true blue light amount Lb (I ′, J ′) should be obtained.

  With reference to FIGS. 18 to 23, the spring model will be described focusing on the pixels that receive the green component. In FIG. 18 and subsequent figures, the true green light amount Lg (I ′, J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′) to be obtained. The position (I ′, J ′) on the coordinate system of the first image is a black circle, and pixel values Gobs (k, ig, jg), Robs (k, ir, jr), Bobs (k, ib, jb) The position on the coordinate system of the first image where (k = 1 to N) is observed is represented by a white circle. In addition, hereinafter, the coordinate system of the first image, which is a reference image, is referred to as a reference coordinate system as appropriate.

The position (x, y) of the point A in FIG. 18 is a position (ig−0.5, jg) where the pixel value Gobs (k, ig, jg) of the ig-th and jg-th pixels of the k-th image is observed. -0.5) is converted by the conversion parameters (θk, Tkx, Tky, Sk) detected by the motion detection circuit 23- (k-1) on the reference coordinate system ((ig-0.5) _{( k)} , (jg−0.5) _(k) ). When k = 1, it can be considered that (θk, Tkx, Tky, Sk) = (0, 0, 0, 1), so (x, y) = (ig−0.5, jg−0. 5). Note that the pixel value of the point A (x, y) is Gobs (k, ig, jg).

  Here, for points A (x, y), integers I and J satisfying I−0.5 ≦ x <I + 0.5 and J−0.5 ≦ y <J + 0.5 are considered. Each position of the four corners of the region surrounded by (I ± 0.5, J ± 0.5) at this time, that is, (I−0.5, J−0.5), (I−0.5, J + 0) .5), (I + 0.5, J−0.5), and (I + 0.5, J + 0.5) are positions (I ′, J ′) for obtaining the true green light quantity Lg (I ′, J ′). J ′).

  Now, the pixel value Gobs (k, ig, jg) at the point A (x, y) is close to the point A (x, y) (I−0.5, J−0.5), (I−0. 5, J + 0.5), (I + 0.5, J-0.5), and (I + 0.5, J + 0.5), the true green light quantity Lg (I-0.5, J-0.5). ), Lg (I−0.5, J + 0.5), Lg (I + 0.5, J−0.5), and Lg (I + 0.5, J + 0.5). That is, the true green light quantity Lg (I−0.5, J−0.5), Lg (I−0.5, J + 0.5), Lg (I + 0.5, J−0.5), Lg ( I + 0.5, J + 0.5) can be approximated by the pixel value Gobs (k, ig, jg) at the point A (x, y).

  Incidentally, the pixel value Gobs (k, ig, jg) observed at the point A (x, y) includes an error (amount of noise) Mk × E as described in the first and second embodiments. It is. Further, here, (I−0.5, J−0.5), (I−0.5, J + 0.5), (I + 0.5, J−0.5), (I + 0.5, J + 0. 5) True green light quantity Lg (I-0.5, J-0.5), Lg (I-0.5, J + 0.5), Lg (I + 0.5, J-0.5) at each of the positions 5) ), Lg (I + 0.5, J + 0.5) is approximated by the pixel value Gobs (k, ig, jg) at the point A (x, y) (hereinafter referred to as an approximate error as appropriate). .

  Here, for example, the true green light quantity Lg (I-0.5, J-0.5) at the position (I-0.5, J-0.5) is the pixel value at the point A (x, y). The approximation by Gobs (k, ig, jg) can be applied to a spring model that is a model using a spring.

  FIG. 19 shows the true green light quantity Lg (I-0.5) at the position (I-0.5, J-0.5) at the pixel value Gobs (k, ig, jg) at the point A (x, y). , J-0.5) is a conceptual diagram of a spring model representing a state approximated.

  In FIG. 19, the pixel value Gobs (k, ig, jg) and the true green light amount Lg (I-0.5, J-0.5) are connected to one end and the other end of the spring BA1, respectively. A state in which the true green light amount Lg (I-0.5, J-0.5) is about to be pulled toward the pixel value Gobs (k, ig, jg) is shown. In the spring model of FIG. 19, it is certain that the true green light quantity Lg (I-0.5, J-0.5) is the pixel value Gobs (k, ig, jg) at the point A (x, y). The true green light amount Lg (I-0.5, J-0.5) is drawn toward the pixel value Gobs (k, ig, jg) side, and conversely, the lower the degree of the probability, The true green light quantity Lg (I-0.5, J-0.5) is moved away from the pixel value Gobs (k, ig, jg).

Here, the approximation error caused by approximating the true green light quantity Lg (I−0.5, J−0.5) with the pixel value Gobs (k, ig, jg) is the point A (x, y). And the position (I-0.5, J-0.5) become larger as the distance between them becomes longer. Therefore, an increasing function F ((x1, y1), (x2, y2)) is introduced in which the function value increases as the distance between the point (x1, y1) and the point (x2, y2) increases. As the function F ((x1, y1), (x2, y2)), for example,
F ((x1, y1), (x2, y2)) = √ {(x1-x2) ² + (y1-y2) ² }
Etc. can be adopted.

  Further, as described above, the pixel value Gobs (k, ig, jg) observed at the point A (x, y) includes a noise amount E × Mk as an error.

The probability that the pixel value Gobs (k, ig, jg) and the true green light quantity Lg (I-0.5, J-0.5) are equivalent (identical), that is, the true green light quantity The approximation accuracy when Lg (I−0.5, J−0.5) is approximated by the pixel value Gobs (k, ig, jg) depends on the increase or decrease in the above-described approximation error and noise amount E × Mk. Decrease or increase respectively. Therefore, the approximation accuracy (probability) is, for example,
Mk × E × F ((x, y), (I−0.5, J−0.5))
Decreases (inversely proportional) to the value of. That is, the certainty that the true green light quantity Lg (I−0.5, J−0.5) is equal to the pixel value Gobs (k, ig, jg) at the point A (x, y) is “ If the value of “Mk × E × F ((x, y), (I−0.5, J−0.5))” is small, the value is large, and “Mk × E × F ((x, y)”. , (I−0.5, J−0.5)) ”is large, it can be said to be small.

Now, the probability that the pixel value Gobs (k, ig, jg) and the true green light quantity Lg (I-0.5, J-0.5) are equivalent is represented by the spring constant of the spring BA1 (spring For example, the spring constant is expressed by the formula {√2-F ((x, y), (I−0.5, J−0.5))} / (Mk × E).
Can be expressed as Here, the denominator (Mk × E) is Mk times the amount of noise E, and as the noise increases, the spring BA1 becomes weaker, and the true green light quantity Lg (I−0.5, J−0. 5), the force pulled toward the pixel value Gobs (k, ig, jg) of the point A by the spring BA1 becomes small. The {√2-F ((x, y), (I-0.5, J-0.5))} of the molecule is surrounded by (I ± 0.5, J ± 0.5) in FIG. From the maximum value of the distance between any two points in the region, √ (1 ² +1 ² ) = √2, the distance F between the point A and (I−0.5, J−0.5) ((x, y), (I−0.5, J−0.5)) minus the value (difference), and the true position A (x, y) where the pixel value Gobs (k, ig, jg) was observed. As the distance from the position (I-0.5, J-0.5) for obtaining the green light quantity Lg (I-0.5, J-0.5) increases, √2-F ((x, y ), (I-0.5, J-0.5)) becomes smaller and the spring BA1 becomes weaker. Accordingly, the force with which the true green light amount Lg (I-0.5, J-0.5) is pulled toward the pixel value Gobs (k, ig, jg) of the point A by the spring BA1 is reduced. In the spring constant, instead of {√2-F ((x, y), (I−0.5, J−0.5))} of the molecule, F ((x, y), (I− The reciprocal of 0.5, J-0.5)) can also be used.

In the spring model, it is assumed that the spring BA1 is a spring whose natural length (the length of the spring BA1 in an unloaded state) is zero. In this case, the length (elongation) of the spring BA1 is the distance in the axial direction of the light quantity (pixel value).
| Gobs (k, ig, jg) -Lg (I-0.5, J-0.5) |
The true green light quantity Lg (I-0.5, J-0.5) is moved to the pixel value Gobs (k, ig, jg) side of the point A (x, y) by the spring BA1. The pulling force is: spring constant × spring length = {{√2−F ((x, y), (I−0.5, J−0.5))} / (Mk × E)} × | It can be expressed as Gobs (k, ig, jg) −Lg (I−0.5, J−0.5) |.

  FIG. 20 shows pixel values Gobs (k, ig, jg) at point A (x, y), (I−0.5, J−0.5), (I−0.5, J + 0.5), True green light amounts Lg (I-0.5, J-0.5), Lg (I-0...) At positions (I + 0.5, J-0.5) and (I + 0.5, J + 0.5). 5, J + 0.5), Lg (I + 0.5, J−0.5), and Lg (I + 0.5, J + 0.5).

  A spring model representing the relationship between the pixel value Gobs (k, ig, jg) at the point A (x, y) described in FIG. 19 and the true green light quantity Lg (I-0.5, J-0.5). In the same way as for the true green light quantity Lg (I-0.5, J + 0.5), Lg (I + 0.5, J-0.5), and Lg (I + 0.5, J + 0.5) The relationship between the pixel values Gobs (k, ig, jg) in A (x, y) can be represented by a spring model.

  That is, in FIG. 20, the pixel value Gobs (k, ig, jg) and the true green light quantity Lg (I−0.5, J + 0.5) are connected to one end and the other end of the spring BA2, respectively. A state in which the true green light amount Lg (I−0.5, J + 0.5) is about to be pulled toward the pixel value Gobs (k, ig, jg) is shown. In the spring model of FIG. 20, the true green light quantity Lg (I−0.5, J + 0.5) is likely to be the pixel value Gobs (k, ig, jg) at the point A (x, y). , The true green light amount Lg (I−0.5, J + 0.5) is drawn toward the pixel value Gobs (k, ig, jg), and conversely, the lower the degree of the probability, the more true green light The light amount Lg (I−0.5, J + 0.5) is moved away from the pixel value Gobs (k, ig, jg).

Here, the approximation error caused by approximating the true green light quantity Lg (I−0.5, J + 0.5) with the pixel value Gobs (k, ig, jg) is the point A (x, y) and the position. The distance between (I−0.5, J + 0.5) increases as the distance increases. Therefore, an increasing function F ((x1, y1), (x2, y2)) is introduced in which the function value increases as the distance between the point (x1, y1) and the point (x2, y2) increases. As the function F ((x1, y1), (x2, y2)), for example,
F ((x1, y1), (x2, y2)) = √ {(x1-x2) ² + (y1-y2) ² }
Etc. can be adopted.

  Further, as described above, the pixel value Gobs (k, ig, jg) observed at the point A (x, y) includes a noise amount E × Mk as an error.

The probability that the pixel value Gobs (k, ig, jg) and the true green light amount Lg (I−0.5, J + 0.5) are equivalent (identical), that is, the true green light amount Lg ( The approximation accuracy when I−0.5, J + 0.5) is approximated by the pixel value Gobs (k, ig, jg) decreases in accordance with the increase or decrease in the approximation error and the noise amount E × Mk. Or increase. Therefore, the approximation accuracy (probability) is, for example,
Mk × E × F ((x, y), (I−0.5, J + 0.5))
Decreases (inversely proportional) to the value of. That is, the probability that the true green light amount Lg (I−0.5, J + 0.5) is equal to the pixel value Gobs (k, ig, jg) at the point A (x, y) is “Mk × If the value of E × F ((x, y), (I−0.5, J + 0.5)) ”is small, the value is large, and“ Mk × E × F ((x, y), (I− If the value of 0.5, J + 0.5)) ”is large, it can be said that the value is small.

Now, the probability that the pixel value Gobs (k, ig, jg) and the true green light quantity Lg (I-0.5, J + 0.5) are equivalent is the spring constant of the spring BA2 (spring strength). ), The spring constant is, for example, the formula {√2-F ((x, y), (I−0.5, J + 0.5))} / (Mk × E)
Can be expressed as Here, the denominator (Mk × E) is Mk times the amount of noise E, and the larger the noise, the weaker the spring BA2, and the true green light quantity Lg (I−0.5, J + 0.5). However, the force pulled by the spring BA2 toward the pixel value Gobs (k, ig, jg) of the point A becomes small. The {√2-F ((x, y), (I−0.5, J + 0.5))} of the molecule is within the region surrounded by (I ± 0.5, J ± 0.5) in FIG. √ the maximum value of the distance between any two points from ^{(1 2 +1 2) = √2} , points a and (I-0.5, J + 0.5 ) distance between F ((x, y), ( I−0.5, J + 0.5)) is a value (difference), and the position A (x, y) where the pixel value Gobs (k, ig, jg) is observed and the true green light quantity Lg ( The larger the distance from the position (I-0.5, J + 0.5) for which I-0.5, J + 0.5) is found, the more the value of √2−F ((x, y), (I−0.5, J + 0) .5)) becomes smaller and the spring BA2 becomes weaker. Therefore, the force with which the true green light amount Lg (I−0.5, J + 0.5) is pulled toward the pixel value Gobs (k, ig, jg) of the point A by the spring BA2 is reduced. In the spring constant, F ((x, y), (I-0...) Instead of {√2-F ((x, y), (I−0.5, J + 0.5))} of the molecule. 5, the reciprocal number of J + 0.5)) can also be used.

In the spring model, the spring BA2 is a spring whose natural length (the length of the spring BA2 in an unloaded state) is zero. In this case, the length (elongation) of the spring BA2 is the distance in the axial direction of the light quantity (pixel value).
| Gobs (k, ig, jg) -Lg (I-0.5, J + 0.5) |
The true green light quantity Lg (I−0.5, J + 0.5) is pulled toward the pixel value Gobs (k, ig, jg) of the point A (x, y) by the spring BA2. The force is: spring constant × spring length = {{√2−F ((x, y), (I−0.5, J + 0.5))} / (Mk × E)} × | Gobs (k, ig, jg) −Lg (I−0.5, J + 0.5) |.

  In FIG. 20, the pixel value Gobs (k, ig, jg) and the true green light amount Lg (I + 0.5, J-0.5) are connected to one end and the other end of the spring BA3, respectively. A state in which the true green light amount Lg (I + 0.5, J−0.5) is about to be pulled toward the pixel value Gobs (k, ig, jg) is shown. In the spring model of FIG. 20, the true green light quantity Lg (I + 0.5, J−0.5) is likely to be the pixel value Gobs (k, ig, jg) at the point A (x, y). , The true green light amount Lg (I + 0.5, J-0.5) is drawn toward the pixel value Gobs (k, ig, jg), and conversely, the lower the degree of certainty, the more true green light The light amount Lg (I + 0.5, J−0.5) is moved away from the pixel value Gobs (k, ig, jg).

Here, the approximation error caused by approximating the true green light amount Lg (I + 0.5, J−0.5) with the pixel value Gobs (k, ig, jg) is the point A (x, y) and the position. The distance between (I + 0.5, J−0.5) increases as the distance increases. Therefore, an increasing function F ((x1, y1), (x2, y2)) is introduced in which the function value increases as the distance between the point (x1, y1) and the point (x2, y2) increases. As the function F ((x1, y1), (x2, y2)), for example,
F ((x1, y1), (x2, y2)) = √ {(x1-x2) ² + (y1-y2) ² }
Etc. can be adopted.

  Further, as described above, the pixel value Gobs (k, ig, jg) observed at the point A (x, y) includes a noise amount E × Mk as an error.

The probability that the pixel value Gobs (k, ig, jg) and the true green light quantity Lg (I + 0.5, J-0.5) are equivalent (identical), that is, the true green light quantity Lg ( The approximation accuracy when I + 0.5, J−0.5) is approximated by the pixel value Gobs (k, ig, jg) decreases as the approximation error and the noise amount E × Mk increase or decrease, respectively. Or increase. Therefore, the approximation accuracy (probability) is, for example,
Mk × E × F ((x, y), (I + 0.5, J−0.5))
Decreases (inversely proportional) to the value of. That is, the probability that the true green light amount Lg (I + 0.5, J−0.5) is equal to the pixel value Gobs (k, ig, jg) at the point A (x, y) is “Mk × If the value of “E × F ((x, y), (I + 0.5, J−0.5))” is small, the value is large, and “Mk × E × F ((x, y), (I + 0. 5, J−0.5)) ”is large, it can be said to be small.

Now, the probability that the pixel value Gobs (k, ig, jg) and the true green light quantity Lg (I + 0.5, J-0.5) are equivalent is the spring constant of the spring BA3 (spring strength). ), The spring constant is, for example, the formula {√2−F ((x, y), (I + 0.5, J−0.5))} / (Mk × E).
Can be expressed as Here, the denominator (Mk × E) is Mk times the noise amount E, and the greater the noise, the weaker the spring BA3 becomes, and the true green light quantity Lg (I + 0.5, J−0.5). However, the force pulled by the spring BA3 toward the pixel value Gobs (k, ig, jg) of the point A becomes small. The {√2-F ((x, y), (I + 0.5, J-0.5))} of the molecule is in the region surrounded by (I ± 0.5, J ± 0.5) in FIG. The distance F between the point A and (I + 0.5, J−0.5) F ((x, y), () from the maximum value of the distance between any two points √ (1 ² +1 ² ) = √2. I + 0.5, J−0.5)) is a value (difference), and the position A (x, y) where the pixel value Gobs (k, ig, jg) is observed and the true green light quantity Lg ( The larger the distance from the position (I + 0.5, J-0.5) for which I + 0.5, J-0.5) is found, the more it becomes √2−F ((x, y), (I + 0.5, J−0). .5)) becomes smaller and the spring BA3 becomes weaker. Therefore, the force by which the true green light amount Lg (I + 0.5, J−0.5) is pulled toward the pixel value Gobs (k, ig, jg) of the point A by the spring BA3 is reduced. In the spring constant, instead of {√2-F ((x, y), (I + 0.5, J-0.5))} of the molecule, F ((x, y), (I + 0.5, The reciprocal of J-0.5)) can also be used.

In the spring model, it is assumed that the spring BA3 is a spring whose natural length (the length of the spring BA3 in an unloaded state) is zero. In this case, the length (elongation) of the spring BA3 is the distance in the axial direction of the light quantity (pixel value).
| Gobs (k, ig, jg) -Lg (I + 0.5, J-0.5) |
The true green light amount Lg (I + 0.5, J−0.5) is pulled toward the pixel value Gobs (k, ig, jg) of the point A (x, y) by the spring BA3. The force is: spring constant × spring length = {{√2−F ((x, y), (I + 0.5, J−0.5))} / (Mk × E)} × | Gobs (k, ig, jg) -Lg (I + 0.5, J-0.5) |.

  Further, in FIG. 20, the pixel value Gobs (k, ig, jg) and the true green light amount Lg (I + 0.5, J + 0.5) are connected to one end and the other end of the spring BA4, respectively, and the pixel value A state where the true green light amount Lg (I + 0.5, J + 0.5) is about to be pulled toward the Gobs (k, ig, jg) side is shown. In the spring model of FIG. 20, the true green light quantity Lg (I + 0.5, J + 0.5) is true so that it is certain that the pixel value Gobs (k, ig, jg) at the point A (x, y). The green light amount Lg (I + 0.5, J + 0.5) is drawn toward the pixel value Gobs (k, ig, jg), and conversely, the lower the probability, the more true green light amount Lg (I + 0) .5, J + 0.5) move away from the pixel value Gobs (k, ig, jg).

Here, the approximation error caused by approximating the true green light quantity Lg (I + 0.5, J + 0.5) with the pixel value Gobs (k, ig, jg) is the point A (x, y) and the position (I + 0). .5, J + 0.5) increases with increasing distance. Therefore, an increasing function F ((x1, y1), (x2, y2)) is introduced in which the function value increases as the distance between the point (x1, y1) and the point (x2, y2) increases. As the function F ((x1, y1), (x2, y2)), for example,
F ((x1, y1), (x2, y2)) = √ {(x1-x2) ² + (y1-y2) ² }
Etc. can be adopted.

  Further, as described above, the pixel value Gobs (k, ig, jg) observed at the point A (x, y) includes a noise amount E × Mk as an error.

The probability that the pixel value Gobs (k, ig, jg) and the true green light quantity Lg (I + 0.5, J + 0.5) are equivalent (identical), that is, the true green light quantity Lg (I + 0. 5, J + 0.5) is approximated by the pixel value Gobs (k, ig, jg), and the approximation accuracy decreases or increases in accordance with the increase or decrease of the approximation error and the noise amount E × Mk, respectively. Therefore, the approximation accuracy (probability) is, for example,
Mk × E × F ((x, y), (I + 0.5, J + 0.5))
Decreases (inversely proportional) to the value of. That is, the probability that the true green light amount Lg (I + 0.5, J + 0.5) is equal to the pixel value Gobs (k, ig, jg) at the point A (x, y) is “Mk × E × If the value of F ((x, y), (I + 0.5, J + 0.5)) ”is small, the value is large, and“ Mk × E × F ((x, y), (I + 0.5, J + 0. If the value of 5)) ”is large, it can be said that the value is small.

Now, the probability that this pixel value Gobs (k, ig, jg) and the true green light quantity Lg (I + 0.5, J + 0.5) are equivalent is the spring constant (spring strength) of the spring BA4. When expressed, the spring constant is, for example, the formula {√2-F ((x, y), (I + 0.5, J + 0.5))} / (Mk × E).
Can be expressed as Here, the denominator (Mk × E) is Mk times the amount of noise E, and the greater the noise, the weaker the spring BA4, and the true green light quantity Lg (I + 0.5, J + 0.5) The force pulled by the spring BA4 toward the pixel value Gobs (k, ig, jg) of the point A becomes small. The {√2-F ((x, y), (I + 0.5, J + 0.5))} of the molecule is an arbitrary value within the region surrounded by (I ± 0.5, J ± 0.5) in FIG. from the maximum value of the distance between two points ^{√ (1 2 +1 2) =} √2, points a and (I + 0.5, J + 0.5 ) distance between F ((x, y), (I + 0.5, J + 0.5)), the position A (x, y) where the pixel value Gobs (k, ig, jg) was observed and the true green light quantity Lg (I + 0.5, J + 0. 5) As the distance from the position (I + 0.5, J + 0.5) to be obtained becomes larger, √2−F ((x, y), (I + 0.5, J + 0.5)) becomes smaller, and the spring BA4 becomes become weak. Therefore, the force by which the true green light amount Lg (I + 0.5, J + 0.5) is pulled toward the pixel value Gobs (k, ig, jg) of the point A by the spring BA4 is reduced. In the spring constant, instead of {√2-F ((x, y), (I + 0.5, J + 0.5))} of the molecule, F ((x, y), (I + 0.5, J + 0. The inverse of 5)) can also be used.

In the spring model, the spring BA4 is a spring whose natural length (the length of the spring BA4 in an unloaded state) is zero. In this case, the length (elongation) of the spring BA4 is the distance in the axial direction of the light amount (pixel value).
| Gobs (k, ig, jg) -Lg (I + 0.5, J + 0.5) |
The force by which the true green light amount Lg (I + 0.5, J + 0.5) is pulled toward the pixel value Gobs (k, ig, jg) of the point A (x, y) by the spring BA4 is , Spring constant × spring length = {{√2−F ((x, y), (I + 0.5, J + 0.5))} / (Mk × E)} × | Gobs (k, ig, jg) -Lg (I + 0.5, J + 0.5) |

  In the above, paying attention to the pixel value Gobs (k, ig, jg) at an arbitrary position A (x, y) on the reference coordinate system, the periphery of the position A (x, y), that is, the expression I-0 0.5 ≦ x <I + 0.5 and an arbitrary true green light quantity Lg (I−0.5, J−0. 5) The relationship between Lg (I-0.5, J + 0.5), Lg (I + 0.5, J-0.5), and Lg (I + 0.5, J + 0.5) is represented by a spring model. Now, paying attention to the position (I ′, J ′) that is the pixel center on the reference coordinate system, the true green light quantity Lg (I ′, J ′) at the position (I ′, J ′), The relationship with the pixel values Gobs (k, ig, jg) observed in the vicinity is represented by a spring model.

  Since the position (I ′, J ′) is the center position of each pixel whose decimal point of I ′, J ′ is 0.5, for example, it is indicated by a black circle in FIG. 20 (I−0.5, J-0.5) is the position (I ′, J ′).

  As shown in FIG. 20, with respect to the point A (x, y), four true green light amounts Lg (I-0.5, J-0.5), Lg (I-0.5) around the point A (x, y). , J + 0.5), Lg (I + 0.5, J−0.5), and Lg (I + 0.5, J + 0.5) can be defined. Furthermore, four spring models are similarly defined for points at which the pixel values Gobs (k, ig, jg) of the first to Nth captured images other than the point A (x, y) are observed. Can do. Therefore, when attention is paid to the position (I ′, J ′), as shown in FIG. 21, the true green light quantity Lg (I ′, J ′) at the position (I ′, J ′) For example, a spring model with pixel values Gobs (k, ig, jg) observed at each of the points A to E is defined.

  That is, the position on the reference coordinate system obtained by converting the position (ig−0.5, jg−0.5) with the conversion parameters (θk, Tkx, Tky, Sk) with respect to a certain position (I ′, J ′). All sets of integers k, ig, and jg satisfying (x, y) satisfying I′−1 ≦ x <I ′ + 1 and J′−1 ≦ y <J ′ + 1 are obtained. For example, as shown in FIG. 21, for the position (I ′, J ′), five pixel values Gobs (k, ig, jg) observed at points A to E are specified (k, ig, jg) is obtained.

  In this case, as shown in FIG. 22, each of the five pixel values Gobs (k, ig, jg) observed at points A to E and the true green light quantity Lg (at position (I ′, J ′)) A spring model representing the relationship with I ′, J ′) can be defined. That is, for the position (I ′, J ′) where the true green light amount Lg (I ′, J ′) is to be obtained, the true green light amount Lg (I ′, J ′) and the observed pixel value Gobs. The relationship with (k, ig, jg) can be expressed by a spring model.

  FIG. 23 is a diagram for explaining an estimation method for estimating the true green light quantity Lg (I ′, J ′) at the position (I ′, J ′) using a spring model focused on the position (I ′, J ′). It is. Note that the axis in FIG. 23 represents the amount of green light (G signal).

  In FIG. 23, an object V having a mass of 0 is connected to one end of each of the five springs, and the other end of each of the five springs has five pixel values Gobs (k, ig, jg) at points A to E, respectively. It is connected to the. That is, the five pixel values Gobs (k, ig, jg) of the points A to E are at positions (ig−0.5, jg−0. The position (x, y) on the reference coordinate system obtained by converting 5) with the conversion parameters (θk, Tkx, Tky, Sk) is I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′. The pixel value Gobs (k, ig, jg) corresponding to the set of (k, ig, jg) that satisfies +1.

The natural length of these five springs is 0, and the spring constant of the spring whose other end is connected to the pixel value Gobs (k, ig, jg) observed at the point (x, y) is as described above. In addition,
{√2-F ((x, y), (I ′, J ′))} / (Mk × E).

  Each of the above five springs pulls the object V toward the five pixel values Gobs (k, ig, jg) with a force proportional to the spring constant, and balances at a certain position. The light amount representing the position is estimated as the true green light amount Lg (I ′, J ′) at the position (I ′, J ′). The balance of the spring can be expressed by the following expression indicating that the sum of the forces applied to the object V is zero.

・・・（２５）

... (25)

  Hereinafter, Expression (25) is referred to as a spring relational expression of the green light quantity. Here, Σ in Expression (25) is a position (ig−0.5, jg−0.5) with respect to a certain position (I ′, J ′), and conversion parameters (θk, Tkx, Tky, Sk). The number of pairs of (k, ig, jg) in which the position (x, y) on the reference coordinate system converted in step 1 satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. Represents the sum of minutes. For example, in the example of FIG. 23, the sum is obtained for a set of five (k, ig, jg) of points A to E.

  Expression (25) is a linear equation with Lg (I ′, J ′) as an unknown, and by solving Expression (25), the true green light quantity Lg (I ′) at position (I ′, J ′). , J ′).

  The true red light amount Lr (I ′, J ′) and the true blue light amount Lb (I ′, J ′) at the position (I ′, J ′) are also described in FIGS. As in the case of the pixel that receives the component of (1), the linear equations (26) and (27) similar to the equation (25) can be established.

・・・（２６）

... (26)

  Hereinafter, the expression (26) is referred to as a spring relational expression for the amount of red light. Here, Σ in equation (26) is a position (ir−0.5, jr−0.5) with respect to a certain position (I ′, J ′), and conversion parameters (θk, Tkx, Tky, Sk). The number of pairs of (k, ir, jr) in which the position (x, y) on the reference coordinate system converted in step 1 satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. Represents the sum of minutes.

・・・（２７）

... (27)

  Hereinafter, the expression (27) is referred to as a blue light quantity spring relational expression. Here, Σ in Expression (27) is a position (ib−0.5, jb−0.5) with respect to a certain position (I ′, J ′), and conversion parameters (θk, Tkx, Tky, Sk). The number of pairs (k, ib, jb) in which the position (x, y) on the reference coordinate system converted in step 1 satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. Represents the sum of minutes.

  Therefore, in the third embodiment, for a certain position (I ′, J ′), the position (i−0.5, j−0.5) is converted with the conversion parameters (θk, Tkx, Tky, Sk). (K, ib, jb) sets in which the position (correction position) (x, y) on the reference coordinate system satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. The true green light amount Lg (I ′, J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′) are obtained based on the pixel values of It is done.

  Next, a third embodiment of the image estimation process in step S4 of FIG. 2 will be described with reference to the flowchart of FIG.

  First, in step S71, the arithmetic circuit 24 pays attention to a certain position (I ′, J ′) on the reference coordinate system (hereinafter referred to as attention position (I ′, J ′)). Here, the target position (I ′, J ′) is the pixel center (i−0.5, j−0.5) of the “i-th and j-th pixels” of the first captured image that is the reference image. Represents.

  Then, the process proceeds from step S71 to S72, and the arithmetic circuit 24 determines the center position (ig−0.5, jg) of the pixel that receives the green component of the k-th image with respect to the target position (I ′, J ′). −0.5) is converted to the position (x, y) on the reference coordinate system obtained by converting the conversion parameters (θk, Tkx, Tky, Sk) by I′−1 ≦ x <I ′ + 1, J′−1 ≦ y. <A set of (k, ig, jg) that satisfies J ′ + 1 is obtained for all the first to Nth images, and the process proceeds to step S73.

  In step S73, the arithmetic circuit 24 generates a spring relational expression of the green light quantity represented by the expression (25) using all the pairs (k, ig, jg) obtained in step S72, Proceed to S74.

  In step S74, the arithmetic circuit 24 receives the center position (ir−0.5, jr−0.5) of the pixel that receives the red component of the kth image with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the data with the conversion parameters (θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. All sets (k, ir, jr) that satisfy (k, ir, jr) are obtained for the first to Nth images, and the process proceeds to step S75.

  In step S75, the arithmetic circuit 24 generates a spring relational expression of the red light quantity represented by the equation (26) using all the pairs (k, ir, jr) obtained in step S74, and the step Proceed to S76.

  In step S76, the arithmetic circuit 24 receives the center position (ib−0.5, jb−0.5) of the pixel that receives the blue component of the k-th image with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the data with the conversion parameters (θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. All the sets of (k, ib, jb) that satisfy (k, ib, jb) are obtained for the first to Nth images, and the process proceeds to step S77.

  In step S77, the arithmetic circuit 24 generates a spring relational expression of the blue light quantity represented by the equation (27) using all the pairs (k, ib, jb) obtained in step S76, and the step Proceed to S78.

  In step S78, the arithmetic circuit 24 calculates the green light amount spring relational expression represented by the expression (25) obtained in step S73 and the red light quantity spring relational expression represented by the expression (26) obtained in step S75. Then, the true green light quantity Lg (I ′, J ′) at the target position (I ′, J ′) is obtained by solving the spring relational expression of the blue light quantity represented by the expression (27) obtained in step S77 as a linear equation. J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′) are obtained, and the process proceeds to step S79.

  In step S79, the arithmetic circuit 24 sets all positions (I ′, J ′) as the target positions, that is, determines all the center positions of the pixels of the first captured image as the target positions (I ′, J ′). ) As to whether the true green light amount Lg (I ′, J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′) are obtained. Determine.

  If it is determined in step S79 that all positions (I ′, J ′) have not yet been obtained as attention positions, the process returns to step S71, and the processes in steps S71 to S79 are repeated. That is, the arithmetic circuit 24 sets the position (I ′, J ′) that has not been noticed as the next attention position (I ′, J ′), and the true green light amount Lg at the attention position (I ′, J ′). (I ′, J ′), true red light quantity Lr (I ′, J ′), and true blue light quantity Lb (I ′, J ′) are obtained.

  On the other hand, if it is determined in step S79 that all positions (I ′, J ′) are the target positions, the process proceeds to step S80, and the arithmetic circuit 24 calculates the true green light amount Lg (I) obtained in step S78. ', J'), a true red light quantity Lr (I ', J'), and a true blue light quantity Lb (I ', J'), an image (signal) is estimated, and the D / A converter 9 or codec 12 is supplied as an output image, and the process returns. For example, in the “i-th and j-th pixels”, the arithmetic circuit 24 calculates the true green light quantity Lg (i−0.5, j−0.5) obtained in step S78 as a green value (G signal). ), The true red light quantity Lr (i-0.5, j-0.5) obtained in step S78 as the red value (R signal), and the blue value (B signal) obtained in step S78. From the true blue light quantity Lb (i−0.5, j−0.5), an image signal of “i th and j th pixels” is estimated (obtained). Then, the arithmetic circuit 24 estimates the output image by performing the estimation for all the pixels having the position (I ′, J ′) as the center position.

  As described above, in the third embodiment, data received by each pixel of the image sensor 4 is regarded as point-sampled data, and the pixel value observed at the center of each pixel and a clear image without camera shake are obtained. By expressing the relationship with the image signal by a spring model, it is possible to obtain a clear image more faithful to the original light.

(Fourth embodiment)
Next, a fourth embodiment of the image estimation process will be described. In the fourth embodiment, a part of the third embodiment described with reference to FIG. 24 is improved.

  That is, in the third embodiment of FIG. 24, in step S72, the arithmetic circuit 24 determines the center position of the pixel that receives the green component of the k-th image with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting (ig−0.5, jg−0.5) with the conversion parameters (θk, Tkx, Tky, Sk) is I′−1 ≦ x <I ′. All sets (k, ig, jg) satisfying +1, J′−1 ≦ y <J ′ + 1 are obtained for the first to Nth images. In step S73, the arithmetic circuit 24 generates a spring relational expression of the green light quantity represented by the expression (25) using all the pairs (k, ig, jg) obtained in step S72. Similarly, for pixels that receive red and blue components, a set of (k, ig, jg) is obtained for the target position (I ′, J ′), and all the obtained (k, A spring relational expression represented by the formula (26) or the formula (27) is generated using a set of (ig, jg).

  By the way, the data of a specific pixel of a specific image among 1 to N images may become unreliable data due to, for example, a shooting mistake. In the digital camera 1 of FIG. 1, N images are taken and stored in the frame memory 22. Accordingly, sufficient data can be secured even if unreliable data is not adopted.

  Therefore, the fourth embodiment adopts only reliable data (discards unreliable data) and estimates a clearer image. Therefore, in the fourth embodiment, the target position (I ′, J ′) is preset as reliable data from among all the obtained (k, ig, jg) pairs. It is assumed that a set of L (k, ig, jg) is adopted. Here, for example, L can be a fixed value such as 8 or can be a variable value set in accordance with a user operation. The arithmetic circuit 24 obtains an average value of the pixel values Gobs (k, ig, jg) of all the obtained (k, ig, jg) pairs, and obtains the set of (k, ig, jg) as the pixel value Gobs. Only L pieces (L = 8) are selected in order from the average value of (k, ig, jg), and the true green light quantity Lg (I ', J') at the position (I ', J') is obtained. adopt.

  Therefore, in the third embodiment, the spring relational expressions for the amounts of green, red, and blue light expressed by the equations (25), (26), and (27) are the same as those in the fourth embodiment. Expressions (28), (29), and (30) can be expressed.

・・・（２８）

... (28)

  Here, Σ in Equation (28) is the average of all the obtained pixel values Gobs (k, ig, jg) for the target position (I ′, J ′). A value is obtained, and the sum of L (k, ig, jg) pairs in the order in which the pixel value Gobs (k, ig, jg) is close to the average value is represented.

・・・（２９）

... (29)

  Here, Σ in Expression (29) is the average of all the obtained pixel values Robs (k, ir, jr) of the set (k, ir, jr) with respect to the target position (I ′, J ′). A value is obtained and the sum of L (k, ir, jr) pairs in the order in which the pixel values Robs (k, ir, jr) are close to the average value is represented.

・・・（３０）

... (30)

  Here, Σ in equation (30) is the average of all the obtained pixel values Bobs (k, ib, jb) of the set (k, ib, jb) with respect to the target position (I ′, J ′). A value is obtained, and the sum of L (k, ib, jb) sets in the order in which the pixel value Bobs (k, ib, jb) is close to the average value is represented.

  A fourth embodiment of the image estimation process in step S4 of FIG. 2 will be described with reference to the flowchart of FIG.

  First, in step S91, the arithmetic circuit 24 pays attention to a certain position (I ′, J ′) on the reference coordinate system (hereinafter referred to as attention position (I ′, J ′)). Here, the target position (I ′, J ′) is the pixel center (i−0.5, j−0.5) of the “i-th and j-th pixels” of the first captured image that is the reference image. Represents.

  Then, the process proceeds from step S91 to S92, and the arithmetic circuit 24 determines the center position (ig−0.5, jg) of the pixel that receives the green component of the k-th image with respect to the target position (I ′, J ′). −0.5) is converted to the position (x, y) on the reference coordinate system obtained by converting the conversion parameters (θk, Tkx, Tky, Sk) by I′−1 ≦ x <I ′ + 1, J′−1 ≦ y. <A set of (k, ig, jg) that satisfies J ′ + 1 is obtained for all the first to Nth images, and the process proceeds to step S93.

  In step S93, the arithmetic circuit 24 obtains an average value of the pixel values Gobs (k, ig, jg) in all the sets (k, ig, jg) obtained in step S92, and obtains the pixel values Gobs (k, ig, Select L (k, ig, jg) pairs in the order in which jg) is close to the average value, and proceed to Step S94. That is, in step S93, data far from the average value of the pixel values Gobs (k, ig, jg) in all (k, ig, jg) pairs obtained in step S92 is discarded as unreliable data. . If all the (k, ig, jg) pairs obtained in step S92 have not reached L originally, all the (k, ig, jg) pairs are selected (not discarded). ).

  In step S94, the arithmetic circuit 24 uses the L (k, ig, jg) pairs selected in step S93 to generate a spring relational expression of the green light quantity expressed by the equation (28). Proceed to step S95.

  In step S95, the arithmetic circuit 24 determines the center position (ir−0.5, jr−0.5) of the pixel that receives the red component of the kth image with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the data with the conversion parameters (θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. All sets (k, ir, jr) that satisfy (k, ir, jr) are obtained for the first to Nth images, and the process proceeds to step S96.

  In step S96, the arithmetic circuit 24 obtains an average value of the pixel values Robs (k, ir, jr) in all the sets (k, ir, jr) obtained in step S96, and obtains the pixel values Robs (k, ir, jr). Select a set of L (k, ir, jr) in the order in which jr) is close to the average value, and proceed to step S97. That is, in step S96, data far from the average value of the pixel values Robs (k, ir, jr) in all the (k, ir, jr) pairs obtained in step S95 is discarded as unreliable data. . If all (k, ir, jr) pairs obtained in step S95 have not reached L, all (k, ir, jr) pairs are selected (not discarded). ).

  In step S97, the arithmetic circuit 24 generates a spring relational expression of the red light quantity represented by the equation (29) using the L (k, ir, jr) sets selected in step S96. Proceed to step S98.

  In step S98, the arithmetic circuit 24 determines the center position (ib−0.5, jb−0.5) of the pixel that receives the blue component of the kth image with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the data with the conversion parameters (θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. All the sets of (k, ib, jb) that satisfy (k, ib, jb) are obtained for the first to Nth images, and the process proceeds to step S99.

  In step S99, the arithmetic circuit 24 obtains an average value of the pixel values Bobs (k, ib, jb) in all the sets (k, ib, jb) obtained in step S98, and obtains the pixel values Bobs (k, ib, Select L (k, ib, jb) pairs in the order in which jb) is close to the average value, and proceed to Step S100. That is, in step S99, data far from the average value of the pixel values Bobs (k, ib, jb) in all the sets (k, ib, jb) obtained in step S98 is discarded as unreliable data. . If all (k, ib, jb) pairs obtained in step S98 have not reached L originally, all (k, ib, jb) pairs are selected (not discarded). ).

  In step S100, the arithmetic circuit 24 generates a spring relational expression of the blue light quantity represented by the equation (30) using the L (k, ib, jb) pairs selected in step S99, and Proceed to step S101.

  In step S101, the arithmetic circuit 24 calculates the green light amount spring relational expression represented by the expression (28) obtained in step S94 and the red light quantity spring relational expression represented by the expression (29) obtained in step S97. By solving the spring relational expression of the blue light quantity represented by the expression (30) obtained in step S100 as a linear equation, the true green light quantity Lg (I ′, J ′) at the target position (I ′, J ′) is obtained. J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′) are obtained, and the process proceeds to step S102.

  In step S102, the arithmetic circuit 24 sets all the positions (I ′, J ′) as the target positions, that is, sets all the center positions of the pixels of the first captured image as the target positions (I ′, J ′). ) As to whether the true green light amount Lg (I ′, J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′) are obtained. Determine.

  If it is determined in step S102 that all positions (I ′, J ′) have not yet been set as the target positions, the process returns to step S91, and the processes in steps S91 to S102 are repeated. That is, the arithmetic circuit 24 sets the position (I ′, J ′) that has not been noticed as the next attention position (I ′, J ′), and the true green light amount Lg at the attention position (I ′, J ′). (I ′, J ′), true red light quantity Lr (I ′, J ′), and true blue light quantity Lb (I ′, J ′) are obtained.

  On the other hand, if it is determined in step S102 that all positions (I ′, J ′) are the target positions, the process proceeds to step S103, and the arithmetic circuit 24 calculates the true green light quantity Lg (I) obtained in step S101. ', J'), a true red light quantity Lr (I ', J'), and a true blue light quantity Lb (I ', J'), an image (signal) is estimated, and the D / A converter 9 or codec 12 is supplied as an output image, and the process returns. For example, in the “i-th and j-th pixels”, the arithmetic circuit 24 calculates the true green light amount Lg (i−0.5, j−0.5) obtained in step S101 as a green value (G signal). ), The true red light quantity Lr (i−0.5, j−0.5) obtained in step S101 as the red value (R signal), and the blue value (B signal) obtained in step S101. From the true blue light quantity Lb (i−0.5, j−0.5), the image signal of the “i th and j th pixels” is estimated. Then, the arithmetic circuit 24 estimates the output image by performing the estimation for all the pixels having the position (I ′, J ′) as the center position.

  As described above, in the fourth embodiment, the pixel value close to the average value is regarded as highly reliable data, and the spring model is applied only to highly reliable data. It becomes possible to obtain a clear image faithful to the original light.

(Fifth embodiment)
Next, a fifth embodiment of the image estimation process will be described. As in the fourth embodiment, the fifth embodiment is reliable from all the (k, ig, jg) pairs obtained for the target position (I ′, J ′). As a data, a set of L (k, ig, jg) set in advance is adopted, and a true green color is obtained by solving a spring relational expression of L, green, red, and blue light amounts. The light amount Lg (I ′, J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′) are obtained.

  However, in the fourth embodiment, in the arithmetic circuit 24, for example, for the pixel that receives the green component, all the obtained pixel values Gobs (k, ig, jg) of (k, ig, jg) are obtained. The average value is obtained, and a set of L (k, ig, jg) in the order in which the pixel values Gobs (k, ig, jg) are close to the average value is selected.

  On the other hand, in the fifth embodiment, the arithmetic circuit 24 determines the center position (ig−0.5, jg) of the pixel that receives the green component of the k-th image with respect to the target position (I ′, J ′). -0.5) with the conversion parameters (θk, Tkx, Tky, Sk) converted in order from the shortest distance between the position (x, y) on the reference coordinate system and the target position (I ′, J ′). Only L sets (k, ig, jg) are selected as highly reliable data. This is because, as described in FIG. 19, the true green light quantity Lg (I ′, J ′) at the target position (I ′, J ′) is changed to the pixel value Gobs (k) (k, k) observed at the position (x, y). , Ig, jg), the approximation error due to the approximation increases as the distance between the position (x, y) and the target position (I ′, J ′) increases. Therefore, the pixel value Gobs (k, ig, jg) observed at the position (x, y) far from the target position (I ′, J ′) is less reliable. For example, the function F ((x1, y1), (x2, y2)) described with reference to FIG. 19 is used as the function for obtaining the distance between the two points (x1, y1) and (x2, y2). be able to.

  In the fifth embodiment, instead of the equations (25), (26), and (27) in the third embodiment, the equations (31), (32), and (33) are used. A spring relational expression of green, red, and blue light amounts is used.

・・・（３１）

... (31)

  Here, Σ in the equation (31) is the center position (ig−0.5, jg−0...) Of the pixel that receives the green component of the kth image with respect to the target position (I ′, J ′). 5) L selected in order from the shortest distance between the position (x, y) on the reference coordinate system obtained by converting the conversion parameters (θk, Tkx, Tky, Sk) on the reference coordinate system and the target position (I ′, J ′). This represents the sum of a set of (k, ig, jg).

・・・（３２）

... (32)

  Here, Σ in the equation (32) is the center position (ir−0.5, jr−0...) Of the pixel that receives the red component of the kth image with respect to the target position (I ′, J ′). 5) L selected in order from the shortest distance between the position (x, y) on the reference coordinate system obtained by converting the conversion parameters (θk, Tkx, Tky, Sk) on the reference coordinate system and the target position (I ′, J ′). This represents the sum of a set of (k, ir, jr).

・・・（３３）

... (33)

  Here, Σ in equation (33) is the center position (ib−0.5, jb−0...) Of the pixel that receives the blue component of the kth image with respect to the target position (I ′, J ′). 5) L selected in order from the shortest distance between the position (x, y) on the reference coordinate system obtained by converting the conversion parameters (θk, Tkx, Tky, Sk) on the reference coordinate system and the target position (I ′, J ′). This represents the sum of a set of (k, ib, jb).

  A fifth embodiment of the image estimation process in step S4 of FIG. 2 will be described with reference to the flowchart of FIG.

  First, in step S121, the arithmetic circuit 24 pays attention to a certain position (I ′, J ′) on the reference coordinate system (hereinafter referred to as attention position (I ′, J ′)). Here, the target position (I ′, J ′) is the pixel center (i−0.5, j−0.5) of the “i-th and j-th pixels” of the first captured image that is the reference image. Represents.

  Proceeding from step S121 to S122, the arithmetic circuit 24 determines the center position (ig−0.5, jg−0) of the pixel that receives the green component of the kth image with respect to the target position (I ′, J ′). .5) with the transformation parameters (θk, Tkx, Tky, Sk), the position (x, y) on the reference coordinate system is I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J. All sets of (k, ig, jg) satisfying '+1 are obtained for the first to Nth images, and the process proceeds to step S123.

  In step S123, the arithmetic circuit 24 selects the green component of the kth image from the set of all (k, ig, jg) obtained in step S122 with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the center position (ig−0.5, jg−0.5) of the pixel that receives light with the conversion parameters (θk, Tkx, Tky, Sk) and the target position ( Only L sets (k, ig, jg) are selected in order of increasing distance from I ′, J ′), and the process proceeds to step S124. In other words, in step S123, among all the (k, ig, jg) pairs obtained in step S122, those having a long distance between the position (x, y) and the target position (I ′, J ′) are reliable. Discarded as irrelevant data. If all (k, ig, jg) pairs obtained in step S122 have not reached L from the beginning, all (k, ig, jg) pairs are selected (not discarded). ).

  In step S124, the arithmetic circuit 24 uses the L (k, ig, jg) pairs selected in step S123 to generate a green light amount spring relational expression represented by the equation (31), and Proceed to step S125.

  In step S125, the arithmetic circuit 24 receives the center position (ir−0.5, jr−0.5) of the pixel that receives the red component of the k-th image with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the data with the conversion parameters (θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. All sets (k, ir, jr) that satisfy (k, ir, jr) are obtained for the first to Nth images, and the process proceeds to step S126.

  In step S126, the arithmetic circuit 24 selects the red component of the kth image from the set of all (k, ir, jr) obtained in step S125 with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the center position (ir−0.5, jr−0.5) of the pixel that receives light with the conversion parameters (θk, Tkx, Tky, Sk) and the target position ( Only L (k, ir, jr) pairs are selected in ascending order of distance from I ′, J ′), and the process proceeds to step S127. In other words, in step S126, among all the (k, ir, jr) pairs obtained in step S125, the longest distance between the position (x, y) and the target position (I ′, J ′) Discarded as irrelevant data. If all (k, ir, jr) pairs obtained in step S125 have not reached L originally, all (k, ir, jr) pairs are selected (not discarded). ).

  In step S127, the arithmetic circuit 24 generates a spring relational expression of the red light quantity represented by the equation (32) using the L (k, ir, jr) pairs selected in step S126. Proceed to step S128.

  In step S128, the arithmetic circuit 24 determines the center position (ib−0.5, jb−0.5) of the pixel that receives the blue component of the kth image with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the data with the conversion parameters (θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. The set of satisfying (k, ib, jb) is obtained for all the 1st to N-th images, and the process proceeds to step S129.

  In step S129, the arithmetic circuit 24 selects the blue component of the kth image from the set of all (k, ib, jb) obtained in step S128 with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the center position (ib−0.5, jb−0.5) of the pixel that receives light with the conversion parameters (θk, Tkx, Tky, Sk) and the target position ( Only L sets (k, ib, jb) are selected in order of increasing distance from I ′, J ′), and the process proceeds to step S130. In other words, in step S129, among all the (k, ib, jb) pairs obtained in step S128, the longest distance between the position (x, y) and the target position (I ′, J ′) Discarded as irrelevant data. If all (k, ib, jb) pairs obtained in step S128 have not reached L originally, all (k, ib, jb) pairs are selected (not discarded). ).

  In step S130, the arithmetic circuit 24 generates a spring relational expression of the blue light quantity represented by the equation (33) using the L (k, ib, jb) pairs selected in step S129. Proceed to step S131.

  In step S131, the arithmetic circuit 24 calculates the green light quantity spring relation represented by the expression (31) obtained in step S124 and the red light quantity spring relation represented by the expression (32) obtained in step S127. By solving the spring relational expression of the blue light quantity represented by the equation (33) obtained in step S130 as a linear equation, the true green light quantity Lg (I ′, J ′) at the target position (I ′, J ′) is obtained. J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′) are obtained, and the process proceeds to step S132.

  In step S132, the arithmetic circuit 24 sets all the positions (I ′, J ′) as the attention positions, that is, determines all the center positions of the pixels of the first captured image as the attention positions (I ′, J ′). ) As to whether the true green light amount Lg (I ′, J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′) are obtained. Determine.

  If it is determined in step S132 that all positions (I ′, J ′) have not yet been set as the target positions, the process returns to step S121, and the processes in steps S121 to S132 are repeated. That is, the arithmetic circuit 24 sets the position (I ′, J ′) that has not been noticed as the next attention position (I ′, J ′), and the true green light amount Lg at the attention position (I ′, J ′). (I ′, J ′), true red light quantity Lr (I ′, J ′), and true blue light quantity Lb (I ′, J ′) are obtained.

  On the other hand, if it is determined in step S132 that all positions (I ′, J ′) are the target positions, the process proceeds to step S133, and the arithmetic circuit 24 calculates the true green light quantity Lg (I) obtained in step S131. ', J'), a true red light amount Lr (I ', J'), and a true blue light amount Lb (I ', J') to generate an image (signal), and the D / A converter 9 or codec 12 is supplied as an output image, and the process returns. For example, in the “i-th and j-th pixels”, the arithmetic circuit 24 calculates the true green light amount Lg (i−0.5, j−0.5) obtained in step S131 as a green value (G signal). ), The true red light quantity Lr (i−0.5, j−0.5) obtained in step S131 as a red value (R signal), and the blue value (B signal) obtained in step S131. From the true blue light quantity Lb (i−0.5, j−0.5), the image signal of the “i th and j th pixels” is estimated. Then, the arithmetic circuit 24 estimates the output image by performing the estimation for all the pixels having the position (I ′, J ′) as the center position.

  As described above, in the fifth embodiment, the pixel value observed at a position close to the target position (I ′, J ′) is used as highly reliable data, and the spring model is applied only to the highly reliable data. As a result, a clearer image that is faithful to the original light than the third embodiment can be obtained.

(Sixth embodiment)
Next, a sixth embodiment of the image estimation process will be described.

  The sixth embodiment is also an improvement of a part of the third embodiment described above. That is, in the sixth embodiment, the edge portion of the image is detected, and the expressions (25), (26), and (27) in the third embodiment are applied to the pixel values in the detected edge portion. The spring relational expression of the green, red, and blue light quantities represented by

  FIG. 27 illustrates a reference coordinate system in which attention is paid to, for example, a green component (G signal) among green, red, and blue.

  In FIG. 27, a pixel value as bright green is observed on the right side with a boundary (edge) 51 as a boundary, and a pixel value as dark green is observed on the left side. Here, assuming that the green pixel value is represented by 8 bits, for example, the bright green pixel value is about 240, and the dark green pixel value is about 16, for example. .

  In FIG. 27, white circles 52-1 to 52-10, white circles 53-1 to 53-9, and white circle 54 are center positions (ig-0.jg) of “ig-th, jg-th” pixels of a certain k-th image. 5, jg−0.5) represents a position on the reference coordinate system obtained by converting the conversion parameters (θk, Tkx, Tky, Sk), and a green pixel value Gobs (k, ig, jg) is observed at the position. ing. Here, the set of (k, ig, jg) is different in each of the white circles 52-1 to 52-10, the white circles 53-1 to 53-9, and the white circle 54.

  Each black circle arranged at a grid-like intersection indicated by a dotted line in FIG. 27 indicates a position (I ′, J ′) at which the arithmetic circuit 24 should obtain the true green light quantity Lg (I ′, J ′). Represents. Here, as described above, the position (I ′, J ′) is the pixel center (i−0.5, j−0...) Of the “i-th and j-th pixels” in the first image as the reference image. 5). Further, the interval between Lg (I ′, J ′) in the X direction and the Y direction is both 1, and therefore the size of the pixel in the X direction and the Y direction is also 1.

  Now, at the positions represented by white circles 52-1 to 52-8 included in the region 61 of FIG. 27, about 240 bright green pixel values Gobs (k, ig, jg) are observed.

  Further, dark green pixel values Gobs (k, ig, jg) of about 16 are observed at positions represented by white circles 53-1 to 53-7 included in the region 62 of FIG.

  Further, the position represented by the white circle 54 in FIG. 27 is a position on the boundary 51. At this position, the bright green light having the pixel value 240 and the dark green light having the pixel value 16 are caused by the effect of the on-chip lens. A pixel value that receives the same amount of light as the light is observed. Accordingly, 128 (= (240 + 16) / 2), which is the average of 240 and 16, is observed as the pixel value Gobs (k, ig, jg) at the position of the white circle 54.

  Since the pixel values Gobs (k, ig, jg) observed at the white circles 52-1 to 52-8, the white circles 53-1 to 53-7, and the white circle 54 described above usually include an error component, To be precise, it should be expressed as about 240, about 16, and about 128, but here, such an error component is ignored and described as 240, 16, and 128.

  Here, for example, the position of the black circle 71 in FIG. 27 is set as the position of interest (I ′, J ′), and the position of interest (I ′, J ′) represented by the black circle 71 by the method of the third embodiment described above. Consider the case of obtaining the true green light quantity Lg (I ′, J ′).

  That is, assuming that the position of the black circle 71 is the target position (I ′, J ′), the pixel values Gobs (k, k) observed at the positions of the surrounding white circles 52-3, 52-6, 52-10, 54, etc. ig, jg) is adopted as Gobs (k, ig, jg) in the equation (25). In this case, the true green light amount Lg (I ′, J ′), which is a solution obtained by solving the equation (25), is pulled (affected) by the pixel value 128 observed at the position of the white circle 54. ), A value smaller than 240. However, since the position of the black circle 71 is originally the position on the right side of the boundary 51 where the bright green pixel value of the pixel value 240 is observed, it is a solution obtained by solving Equation (25). The true green light quantity Lg (I ′, J ′) is preferably 240.

  Also, assuming that the position of the black circle 72 is the target position (I ′, J ′), the pixel values observed at the positions of the white circles 53-2, 53-3, 53-5, 53-9, 54, etc. Gobs (k, ig, jg) is adopted as Gobs (k, ig, jg) in equation (25). In this case, the true green light amount Lg (I ′, J ′), which is a solution obtained by solving the equation (25), is pulled (affected) by the pixel value 128 observed at the position of the white circle 54. ), A value greater than 16. However, since the position of the black circle 72 is originally the position on the left side of the boundary 51 where the dark green pixel value of the pixel value 16 is observed, it is a solution obtained by solving the equation (25). The true green light quantity Lg (I ′, J ′) is preferably 16.

  The same can be said for the positions of the black circle 73 and the black circle 74. That is, the image estimated by the image estimation processing of the third embodiment is blurred at the edge portion where the boundary 51 in FIG. 27 exists (the difference between the pixel values of adjacent pixels becomes small). It can be an image.

  Therefore, in the sixth embodiment, an edge portion where the boundary 51 in FIG. 27 exists is detected, and special processing is performed on the pixel value Gobs (k, ig, jg) at the edge portion, that is, the expression The pixel value Gobs (k, ig, jg) to be substituted into the spring relational expression of the green light quantity represented by (25) is changed.

  A specific method of the sixth embodiment will be described by paying attention to the green component (G signal) as in FIG.

First, the arithmetic circuit 24 detects the edge portion using the pixel values Gobs (k, ig, jg) of all positions converted on the reference coordinate system. That is, the arithmetic circuit 24, "vertical edge", "lateral edge", "edge of the lower right direction from the upper left", the four directional edge respectively of the presence or absence of "edge lower left direction from the right" judge.

  A determination method for determining whether or not there is a vertical edge will be described with reference to FIG.

  FIG. 28 shows a reference coordinate system. In FIG. 28, the positions represented by white circles 81-1 to 81-10, white circles 82-1 to 82-9, and white circle 83 are the positions of the “ig-th and jg-th pixels” of a certain k-th image. The position on the reference coordinate system obtained by converting the center position (ig−0.5, jg−0.5) with the conversion parameters (θk, Tkx, Tky, Sk) is represented, and the pixel value Gobs (k, ig, jg) is observed. Here, the set of (k, ig, jg) is different in each of the white circles 81-1 to 81-10, the white circles 82-1 to 82-9, and the white circle 83.

  Each black circle arranged at a grid-like intersection indicated by a dotted line in FIG. 28 indicates a position (I ′, J ′) at which the arithmetic circuit 24 should obtain the true green light quantity Lg (I ′, J ′). Represents. Here, as described above, the position (I ′, J ′) is the pixel center (i−0.5, j−0...) Of the “i-th and j-th pixels” in the first image as the reference image. 5). Further, the interval between Lg (I ′, J ′) in the X direction and the Y direction is 1, as in FIG. 27, and therefore the size of the pixel in the X direction and the Y direction is both 1.

  The arithmetic circuit 24 pays attention to the position of the white circle 83 in FIG. 28 and determines, for example, whether there is a vertical edge such as the edge 94 shown in FIG. The position (x, y) of the white circle 83 satisfies the formulas I−0.5 ≦ x <I + 0.5 and J−0.5 ≦ y <J + 0.5. In FIG. 28, the position (x, y) of the white circle 83 is a black circle 84 representing the position (I + 0.5, J−0.5), and a black circle 85 representing the position (I−0.5, J−0.5). , A position in an area A11 surrounded by a black circle 86 representing the position (I + 0.5, J + 0.5) and a black circle 87 representing the position (I−0.5, J + 0.5). Here, I and J are integers as before.

  The arithmetic circuit 24 defines nine areas each having the same size as the pixel, with the area A11 including the position of the white circle 83 of interest as the center. In other words, the nine regions include regions A00, I-1.5 ≦ x <I-0.I-1.5 ≦ x <I-0.5, J-1.5 ≦ y <J-0.5. 5, J−0.5 ≦ y <J + 0.5 region A01, I−1.5 ≦ x <I−0.5, J + 0.5 ≦ y <J + 1.5 region A02, I−0.5 ≦ Region A10 where x <I + 0.5, J−1.5 ≦ y <J−0.5, Region A11 where I−0.5 ≦ x <I + 0.5, J−0.5 ≦ y <J + 0.5, Region A12 of I−0.5 ≦ x <I + 0.5, J + 0.5 ≦ y <J + 1.5, Region of I + 0.5 ≦ x <I + 1.5, J−1.5 ≦ y <J−0.5 A20, I + 0.5 ≦ x <I + 1.5, J−0.5 ≦ y <J + 0.5 region A21, I + 0.5 ≦ x <I + 1.5, J + 0.5 ≦ y <J + 1.5 region A22, It is. The position of the white circle 83 of interest is now included in the area A11.

  Here, as described above, the entire nine regions A00 to A22 defined with respect to the position of the white circle 83 of interest are hereinafter referred to as an edge determination region as appropriate.

  In order to determine whether or not there is an edge 94 in the vertical direction, the arithmetic circuit 24 has a region 91 on the left side of the edge determination region (hereinafter, referred to as the left region 91), which is composed of one row of region A00, region A01, and region A02. ), A region A 10, a region A 11, and a region A 12, and a center region 92 of the edge determination region (hereinafter referred to as the center region 92), a region A 20, a region A 21, and a region A 22. The average value and the variance of the pixel values Gobs (k, ig, jg) at the positions of the white circles included in the right region 93 (hereinafter referred to as the right region 93) of the edge determination region are obtained.

  That is, the arithmetic circuit 24 obtains the average value EG0 and the variance SG0 of the pixel values Gobs (k, ig, jg) at the positions of the white circles 82-1 to 82-7 in the left region 91. Further, the arithmetic circuit 24 calculates the average value EG0 ″ of the pixel values Gobs (k, ig, jg) at the positions of the white circles 81-9, 81-10, 82-8 to 82-9, and 83 in the central region 92. Further, the arithmetic circuit 24 obtains the pixel value Gobs (k, ig, jg) average value EG0 ′ and the variance SG0 ′ at the positions of the white circles 81-1 to 81-8 in the right region 93.

  If there is a vertical edge 94 near the position of the white circle 83 of interest, the average value EG0 of the left region 91, the average value EG0 ″ of the central region 92, and the average value EG0 ′ of the right region 93 are Therefore, the relationship of the following equation (a) is established, and the arithmetic circuit 24 determines that there is an edge in the vertical direction when the equation (a) is established.

EG0 <EG0 "<EG0 'or EG0'<EG0"<EG0
... (a)

  Note that in practice, the arithmetic circuit 24 may also use the variances SG0 and SG0 because the above equation (a) may be satisfied even in a portion other than the edge 94 due to, for example, variation in data due to a fine pattern of the subject. By determining whether or not the following expression with 'added holds, it is determined more reliably whether or not there is an edge 94 in the vertical direction. That is, the arithmetic circuit 24 determines that there is an edge in the vertical direction when the formula (b) is established.

EG0 + SG0 <EG0 "<EG0'-SG0 '
Or EG0 ′ + SG0 ′ <EG0 ”<EG0−SG0
... (b)

  According to the formula (b), it is difficult to determine that there is an edge in a portion where the pixel values vary as in the pattern portion and the variances SG0 and SG0 ′ are large, and erroneous determination can be prevented. .

  A determination method for determining whether or not there is a horizontal edge will be described with reference to FIG. Note that portions corresponding to those in FIG. 28 are denoted by the same reference numerals, and description thereof will be omitted below.

  In order to determine whether or not there is a horizontal edge 104, the arithmetic circuit 24 is an area 101 on the upper side of the edge determination area (hereinafter referred to as the upper area 101) composed of one row of the area A00, the area A10, and the area A20. ), An area 102 (hereinafter referred to as the central area 102), an area A02, an area A12, and an area A22. The average value and variance of the pixel values Gobs (k, ig, jg) at the positions of the white circles included in the lower region 103 (hereinafter referred to as the lower region 103) of the edge determination region are obtained.

  That is, the arithmetic circuit 24 calculates the pixel values Gobs (k, ig, jg) at the positions of white circles 81-1 to 81-3, 81-10, 82-1, 82-2, 82-9 in the upper region 101. An average value EG1 and a variance SG1 are obtained. The arithmetic circuit 24 calculates the average value EG1 ″ of the pixel values Gobs (k, ig, jg) at the positions of the white circles 81-4 to 81-6, 82-3 to 82-5, and 83 in the central region 102. Further, the arithmetic circuit 24 distributes the pixel values Gobs (k, ig, jg) average values EG1 ′ at the positions of the white circles 81-7 to 81-9 and 82-6 to 82-8 in the lower region 103 and the variance. SG1 ′ is obtained.

  If there is a lateral edge 104 near the position of the white circle 83 of interest, the average value EG1 of the upper region 101, the average value EG1 ″ of the central region 102, and the average value EG1 ′ of the lower region 103 As a relationship, the following equation (c) is taken into account, taking into account the variation in data due to the fine pattern of the subject, etc. Then, the arithmetic circuit 24 determines that there is a lateral edge when equation (c) holds. To do.

EG1 + SG1 <EG1 "<EG1'-SG1 '
Or EG1 '+ SG1'<EG1"<EG1-SG1
... (c)

  According to the equation (c), it is difficult to determine that there is an edge in a portion where the pixel values vary as in the pattern portion and the variances SG1 and SG1 ′ are large, and erroneous determination can be prevented. . As in the case of the above-described formula (a), it is also possible to determine the presence / absence of an edge in the horizontal direction based on only the average value without considering dispersion.

  A determination method for determining whether there is an edge from the upper left to the lower right will be described with reference to FIG. Note that portions corresponding to those in FIG. 28 are denoted by the same reference numerals, and description thereof will be omitted below.

  In order to determine whether there is an edge 114 in the direction from the upper left to the lower right, the arithmetic circuit 24 has a lower left side region 111 (hereinafter, lower left) of the edge determination region, which is composed of one row of the region A01, the region A02, and the region A12. Side region 111), region A 00, region A 11, region A 22, and center region 112 of the edge determination region (hereinafter referred to as center region 112), region A 10, region A 20, and region A 21. The average value and variance of the pixel values Gobs (k, ig, jg) at the positions of the white circles included in the upper left region 113 (hereinafter referred to as the upper left region 113) of the edge determination region, which is composed of one column. Ask for.

  That is, the arithmetic circuit 24 obtains the average value EG2 and the variance SG2 of the pixel values Gobs (k, ig, jg) at the positions of the white circles 81-9, 82-3 to 82-8 in the lower left region 111. The arithmetic circuit 24 calculates the average value EG2 ″ of the pixel values Gobs (k, ig, jg) at the positions of the white circles 81-7, 81-8, 82-1, 82-2, and 83 in the central region 112. Further, the arithmetic circuit 24 calculates the pixel values Gobs (k, ig, jg) average value EG2 ′ at the positions of the white circles 81-1 to 81-6, 81-10, and 82-9 in the upper right region 113. The variance SG2 ′ is obtained.

  If there is an edge 114 from the upper left to the lower right near the position of the white circle 83 of interest, the average value EG2 of the lower left region 111, the average value EG2 "of the central region 112, and the average of the upper right region 113 As the relationship of the value EG2 ′, the following equation (d) is taken into account, taking into account the data variation due to the fine pattern of the subject, etc. Then, the arithmetic circuit 24 calculates the equation (d) from the upper left to the lower right. It is determined that there is an edge in the direction.

EG2 + SG2 <EG2 "<EG2'-SG2 '
Or EG2 '+ SG2'<EG2"<EG2-SG2
... (d)

  According to the equation (d), it is difficult to determine that there is an edge in a portion where the pixel value varies as in the pattern portion and the variance SG2, SG2 ′ is large, and erroneous determination can be prevented. . As in the case of the above-described formula (a), it is also possible to determine the presence or absence of an edge from the upper left to the lower right by using only the average value without considering the variance.

  A determination method for determining whether there is an edge from the upper right to the lower left will be described with reference to FIG. Note that portions corresponding to those in FIG. 28 are denoted by the same reference numerals, and description thereof will be omitted below.

  In order to determine whether there is an edge 124 in the direction from the upper right to the lower left, the arithmetic circuit 24 includes an upper left area 121 (hereinafter, upper left) of the edge determination area, which is composed of one row of the area A00, the area A01, and the area A10. (Referred to as a region 121), a region A02, a region A11, and a region A20, and a central region 122 of the edge determination region (hereinafter referred to as a central region 122), a region A12, a region A21, and a region A22. An average value of pixel values Gobs (k, ig, jg) at positions of white circles included in each of the lower right region 123 (hereinafter referred to as the lower right region 123) of the edge determination region, which is composed of columns Find the variance.

  That is, the arithmetic circuit 24 calculates the average value EG3 and the variance SG3 of the pixel values Gobs (k, ig, jg) at the positions of the white circles 81-10, 82-1 to 82-5, 82-9 in the upper left region 121. Ask. Further, the arithmetic circuit 24 calculates the average value EG3 ″ of the pixel values Gobs (k, ig, jg) at the positions of the white circles 81-1 to 81-3, 82-6, 82-7, and 83 in the central region 122. Further, the arithmetic circuit 24 calculates pixel values Gobs (k, ig, jg) average values EG3 ′ and variance SG3 ′ at positions of white circles 81-4 to 81-9 and 82-8 in the lower right region 123. Ask.

  If there is an edge 124 from the upper right to the lower left near the position of the white circle 83 of interest, the average value EG3 of the upper left region 121, the average value EG3 ″ of the central region 122, and the average of the lower right region 123 As the relationship of the value EG3 ′, the following equation (e) is taken into consideration, which also considers data variation due to a fine pattern of the subject, etc. Then, the arithmetic circuit 24 moves from the upper right to the lower left when the equation (e) is satisfied. It is determined that there is an edge.

EG3 + SG3 <EG3 "<EG3'-SG3 '
Or EG3 '+ SG3'<EG3"<EG3-SG3
... (e)

  According to the equation (e), it is difficult to determine that there is an edge in a portion where the pixel values vary as in the pattern portion and the variances SG1 and SG1 ′ are large, and erroneous determination can be prevented. . As in the case of the above-described formula (a), it is also possible to determine the presence / absence of an edge from the lower right to the lower left by using only the average value without considering the variance.

  As described with reference to FIGS. 28 to 31, the arithmetic circuit 24 has edges in four directions of “vertical direction”, “horizontal direction”, “upper left to lower right direction”, and “lower right to lower left direction”. It is determined whether or not the above equation holds. Note that the arithmetic circuit 24 determines the presence or absence of an edge not only for green but also for other red and blue colors.

  Here, it is also conceivable that the above-described equation holds for the edges in a plurality of directions among the edges in the four directions. However, since there is only one actual edge, in that case, only the edge in the most prominent direction is adopted and there is no edge in the other direction.

Specifically, for example, the arithmetic circuit 24 has a plurality of directions in which edges are detected.
| (EGm′−EGm) ÷ (SGm ′ + SGm) |
(M = 0 to 3)
And m that maximizes the result of the calculation is determined. If m is 0, “vertical edge” is 1, “horizontal edge” is 1, 2 is “edge from upper left to lower right”, 3 is “ The edge from the lower right to the lower left direction is adopted as the edge in the most prominent direction. Here, | x | represents the absolute value of x.

  Next, according to the direction of the edge detected for the position of interest, the arithmetic circuit 24 performs a special operation on the pixel value Gobs (k, ig, jg) observed at the position of interest at the edge portion. Processing will be described.

  The special processing means that the pixel observed at the position determined to be the edge portion when the pixel value Gobs (k, ig, jg) is substituted into the spring relational expression of the green light quantity expressed by Expression (25). The value Gobs (k, ig, jg) is a process of changing the pixel value Gobs (k, ig, jg) to be substituted.

  As a process for changing the pixel value Gobs (k, ig, jg) to be substituted, for example, the following first process or second process can be employed.

  For example, in the first process, when an edge is detected, for example, at the position of the white circle 83 in FIGS. 28 to 31, the pixel value Gobs (k, ig, jg) at that position is discarded. In other words, the pixel value Gobs (k, ig, jg) at the position where the edge is detected is not included in the spring relational expression of the green light quantity expressed by Expression (25). In this case, in the spring model described in FIG. 22 (FIG. 23), there is no spring that pulls in the wrong direction (light quantity). Therefore, more accurate (clear) Lg (I ′, J ′) can be obtained.

  In the second processing, for example, the pixel value Gobs (k, ig, jg) at the position of the white circle 83 in FIGS. 28 to 31 is defined for the position to which attention is paid in the above-described edge detection. The pixel value Gobs (k, ig, jg) observed in the nine regions A00 to A22 constituting the edge determination region is replaced with another value, and the green light amount spring represented by the equation (25) Substitute into the relational expression.

  The second process will be specifically described below.

  FIG. 32 shows a region A11 including the position of the white circle 83 of interest when the “vertical edge” shown in FIG. 28 is detected. In the figure, one axis represents the X direction in the reference coordinate system, the axis perpendicular to the X direction represents the Y direction in the reference coordinate system, and the axis perpendicular to the X direction and the Y direction represents the pixel value.

  In FIG. 32, the average values EG0, EG0 ′, EG0 ″ obtained for the edge determination region and the variances SG0, SG0 ′ satisfy the expression “EG0 ′ + SG0 ′ <EG0” <EG0−SG0 ”. The edge of "exists.

  In this case, the true green light quantity Lg (I-0.5, J-0.5) at the position (I-0.5, J-0.5) of the black circle 85 and the position (I-0 of the black circle 87). .5, J + 0.5) is the true green light quantity Lg (I−0.5, J + 0.5) is the pixel value Gobs (k) observed at the position (x, y) of the white circle 83 of interest. , Ig, jg). Further, the true green light amount Lg (I-0.5, J-0.5) at the position (I-0.5, J-0.5) of the black circle 85 or the position (I-0. 5, J + 0.5) and the light quantity of the true green light quantity Lg (I−0.5, J + 0.5) and the pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83. The difference in (pixel value) depends on the difference (EG0−EG0 ′) between the average values EG0 and EG0 ′.

  In addition, the true green light amount Lg (I + 0.5, J-0.5) at the position (I + 0.5, J-0.5) of the black circle 84 and the position (I + 0.5, J + 0.5) of the black circle 86. The true green light amount Lg (I + 0.5, J + 0.5) at is smaller than the pixel value Gobs (k, ig, jg) observed at the position (x, y) of the white circle 83 of interest. Conceivable. Further, the true green light amount Lg (I + 0.5, J-0.5) at the position of the black circle 84 (I + 0.5, J-0.5) or the position of the black circle 86 (I + 0.5, J + 0.5). The difference in light amount (pixel value) between each true green light amount Lg (I + 0.5, J + 0.5) and the pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83 is It depends on the difference (EG0−EG0 ′) between the average values EG0 and EG0 ′.

  Therefore, the arithmetic circuit 24 obtains a plane Q1 that passes through the pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83 of interest and has an inclination p in the X direction. Here, p = (EG0−EG0 ′) / 2. That is, the slope p is the difference between the average value EG0 of the pixel values in the left region 91 and the average value EG0 ′ of the pixel values in the right region 93 in FIG. 28, and the center position in the x direction between the left region 91 and the right region 92. It is obtained by dividing by 2 which is the distance between each other. The inclination p represents the degree of change in the amount of light (pixel value) in the direction perpendicular to the edge in the vertical edge portion, that is, the inclination of the edge.

  Then, the arithmetic circuit 24 obtains a value (pixel value) on the plane Q1 at the position (I-0.5, J-0.5) of the black circle 85, and obtains the value as a pixel value Gobs0 (k, ig, jg). And Further, the arithmetic circuit 24 obtains a value (pixel value) on the plane Q1 at the position (I−0.5, J + 0.5) of the black circle 87, and sets the value as the pixel value Gobs1 (k, ig, jg). . Similarly, the arithmetic circuit 24 calculates the value (pixel value) on the plane Q1 at the position (I + 0.5, J−0.5) of the black circle 84 and the position (I + 0.5, J + 0.5) of the black circle 86. The values (pixel values) on the plane Q1 are obtained, and the respective values are set as a pixel value Gobs2 (k, ig, jg) and a pixel value Gobs3 (k, ig, jg).

  Here, since the plane Q1 has an inclination p only in the X direction, the pixel value Gobs0 (k, ig, jg) and the pixel value Gobs1 (k, ig, jg) are equal, and the pixel value Gobs2 (K, ig, jg) and the pixel value Gobs3 (k, ig, jg) are equal.

  In the third embodiment, as shown in FIG. 22, the true green light quantity Lg (I ′, J ′) at a certain position (I ′, J ′) is observed at the peripheral positions. Formula (25) is established by considering a spring model representing a state in which the pixel value Gobs (k, ig, jg) is balanced. The pixel values Gobs (k, ig, jg) observed at positions around the position (I ′, J ′) are the positions (ig−0.5, jg−0.5) in the coordinate system of the k-th image. ) With the conversion parameters (θk, Tkx, Tky, Sk), the position (x, y) on the reference coordinate system is I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. A set of pixel values Gobs (k, ig, jg) satisfying (k, ig, jg).

  In FIG. 32, for example, when the position (I + 0.5, J−0.5) of the black circle 84 is noted as the position (I ′, J ′), the pixel values observed at positions around the position of the black circle 84 The pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83 is included as Gobs (k, ig, jg). Therefore, in the third embodiment, the pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83 is substituted into Expression (25).

  On the other hand, in the sixth embodiment, instead of the pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83, the position (I + 0.5, J) of the black circle 84 of interest. The pixel value Gobs2 (k, ig, jg) which is a value (pixel value) on the plane Q1 in -0.5) is substituted into the equation (25).

  When attention is paid to the position (I−0.5, J−0.5) of the black circle 85 as the position (I ′, J ′), the pixel value Gobs (k, k) of the position (x, y) of the white circle 83 is obtained. In place of (ig, jg), a pixel value Gobs0 (k, ig, jg) which is a value (pixel value) on the plane Q1 at the position (I-0.5, J-0.5) of the black circle 85 of interest. ) Is substituted into equation (25).

  Further, when attention is paid to the position (I + 0.5, J + 0.5) of the black circle 86 as the position (I ′, J ′), the pixel value Gobs (k, ig, jg) of the position (x, y) of the white circle 83 is obtained. Instead of the pixel value Gobs3 (k, ig, jg), which is the value (pixel value) on the plane Q1 at the position (I + 0.5, J + 0.5) of the black circle 86 of interest, in the equation (25) Assigned.

  When attention is paid to the position (I−0.5, J + 0.5) of the black circle 87 as the position (I ′, J ′), the pixel value Gobs (k, ig, In place of jg), a pixel value Gobs1 (k, ig, jg) which is a value (pixel value) on the plane Q1 at the position (I−0.5, J + 0.5) of the black circle 87 of interest Substituted in (25).

  As described above, the pixel value Gobs (k, ig, jg) (pixel value at the position of the white circle 83 in FIG. 32) observed at a position where there is a vertical edge (abrupt change in the amount of green light) Pixel values (Gobs0 (k, ig, jg), Gobs1 (k, ig, jg), Gobs2 (k, ig, jg), or Gobs3 (k, ig, jg) corresponding to the inclination (steepness) p of the edge ) And substituting the changed pixel value into equation (25), Lg (I ′, J ′) as the object V in FIG. 23 is pulled to an appropriate position in the spring model, A more accurate (clear) true green light amount Lg (I ′, J ′) can be obtained.

  FIG. 33 shows a region A11 including the position of the white circle 83 of interest when the “lateral edge” shown in FIG. 29 is detected. In the figure, one axis represents the X direction in the reference coordinate system, the axis perpendicular to the X direction represents the Y direction in the reference coordinate system, and the axis perpendicular to the X direction and the Y direction represents the pixel value.

  In FIG. 33, the average values EG1, EG1 ′, EG1 ″ obtained for the edge determination region and the variances SG1, SG1 ′ satisfy the expression “EG1 ′ + SG1 ′ <EG1” <EG1-SG1 ”. The edge of "exists.

  In this case, the true green light quantity Lg (I + 0.5, J-0.5) at the position (I + 0.5, J-0.5) of the black circle 84 and the position (I-0.5, J of the black circle 85). −0.5) is the true green light quantity Lg (I−0.5, J−0.5), which is the pixel value Gobs (k) observed at the position (x, y) of the white circle 83 of interest. , Ig, jg). Further, the true green light amount Lg (I + 0.5, J-0.5) at the position of the black circle 84 (I + 0.5, J-0.5) or the position of the black circle 85 (I-0.5, J-0.5). 0.5) the true green light quantity Lg (I-0.5, J-0.5) and the light quantity (pixels Gobs (k, ig, jg) at the position (x, y) of the white circle 83 ( The difference in pixel value) depends on the difference (EG1−EG1 ′) between the average values EG1 and EG1 ′.

  Further, the true green light quantity Lg (I + 0.5, J + 0.5) at the position (I + 0.5, J + 0.5) of the black circle 86 and the true light at the position (I−0.5, J + 0.5) of the black circle 87. Is less than the pixel value Gobs (k, ig, jg) observed at the position (x, y) of the white circle 83 of interest. Conceivable. Further, the true green light quantity Lg (I + 0.5, J + 0.5) at the position (I + 0.5, J + 0.5) of the black circle 86 or the true light at the position (I−0.5, J + 0.5) of the black circle 87. The difference in light amount (pixel value) between each of the green light amounts Lg (I−0.5, J + 0.5) and the pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83 is It depends on the difference (EG1−EG1 ′) between the average values EG1 and EG1 ′.

  Therefore, the arithmetic circuit 24 obtains a plane Q2 that passes through the pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83 of interest and has a slope p in the Y direction. Here, p = (EG1-EG1 ′) / 2. That is, the slope p is the difference between the average value EG1 of the pixel values in the upper region 101 and the average value EG1 ′ of the pixel values in the lower region 103 in FIG. It is obtained by dividing by 2 which is the distance between positions. The inclination p represents the degree of change in the amount of light (pixel value) in the direction perpendicular to the edge at the edge portion in the horizontal direction, that is, the inclination of the edge.

  Then, the arithmetic circuit 24 obtains a value (pixel value) on the plane Q2 at the position (I-0.5, J-0.5) of the black circle 85, and obtains the value as a pixel value Gobs0 (k, ig, jg). And Further, the arithmetic circuit 24 obtains a value (pixel value) on the plane Q2 at the position (I−0.5, J + 0.5) of the black circle 87, and sets the value as the pixel value Gobs1 (k, ig, jg). . Similarly, the arithmetic circuit 24 calculates the value (pixel value) on the plane Q2 at the position (I + 0.5, J−0.5) of the black circle 84 and the position (I + 0.5, J + 0.5) of the black circle 86. Values (pixel values) on the plane Q2 are obtained, and the respective values are set as a pixel value Gobs2 (k, ig, jg) and a pixel value Gobs3 (k, ig, jg), respectively.

  Here, since the plane Q2 has an inclination p only in the Y direction, the pixel value Gobs0 (k, ig, jg) is equal to the pixel value Gobs2 (k, ig, jg), and the pixel value Gobs1. (K, ig, jg) and the pixel value Gobs3 (k, ig, jg) are equal.

  Also in the “lateral edge”, as in the case of the “vertical edge” in FIG. 32, the position (I + 0.5, J−0.5) of the black circle 84 is set as the position (I ′, J ′). When attention is paid, instead of the pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83, the plane Q2 at the position (I + 0.5, J-0.5) of the black circle 84 of interest. The pixel value Gobs2 (k, ig, jg), which is the upper value (pixel value), is substituted into Expression (25).

  When attention is paid to the position (I−0.5, J−0.5) of the black circle 85 as the position (I ′, J ′), the pixel value Gobs (k, k) of the position (x, y) of the white circle 83 is obtained. In place of (ig, jg), a pixel value Gobs0 (k, ig, jg) which is a value (pixel value) on the plane Q2 at the position (I-0.5, J-0.5) of the black circle 85 of interest. ) Is substituted into equation (25).

  Further, when attention is paid to the position (I + 0.5, J + 0.5) of the black circle 86 as the position (I ′, J ′), the pixel value Gobs (k, ig, jg) of the position (x, y) of the white circle 83 is obtained. Instead of the pixel value Gobs3 (k, ig, jg), which is the value (pixel value) on the plane Q2 at the position (I + 0.5, J + 0.5) of the black circle 86 of interest, in the equation (25) Assigned.

  Further, when attention is paid to the position (I−0.5, J + 0.5) of the black circle 87 as the pixel value (I ′, J ′), the pixel value Gobs (k, ig) at the position (x, y) of the white circle 83. , Jg), the pixel value Gobs1 (k, ig, jg), which is the value (pixel value) on the plane Q2 at the position (x, y) of the black circle 87 at the position (x, y), is expressed by the equation (25). ).

  As described above, the pixel value Gobs (k, ig, jg) (pixel value at the position of the white circle 83 in FIG. 33) observed at a position where there is a horizontal edge (abrupt change in the amount of green light) Pixel values (Gobs0 (k, ig, jg), Gobs1 (k, ig, jg), Gobs2 (k, ig, jg), or Gobs3 (k, ig, jg) corresponding to the inclination (steepness) p of the edge ) And substituting the changed pixel value into equation (25), Lg (I ′, J ′) as the object V in FIG. 23 is pulled to an appropriate position in the spring model, A more accurate (clear) true green light amount Lg (I ′, J ′) can be obtained.

  FIG. 34 shows a region A11 including the position of the white circle 83 of interest when “the edge from the upper left to the lower right” shown in FIG. 30 is detected. In the figure, one axis represents the X direction in the reference coordinate system, the axis perpendicular to the X direction represents the Y direction in the reference coordinate system, and the axis perpendicular to the X direction and the Y direction represents the pixel value.

  In FIG. 34, the average values EG2, EG2 ′, EG2 ″ obtained for the edge determination region and the variances SG2, SG2 ′ satisfy the expression “EG2 + SG2 <EG2” <EG2′−SG2 ′ ”. There is a “downward edge”.

  In this case, the true green light quantity Lg (I + 0.5, J−0.5) at the position (I + 0.5, J−0.5) of the black circle 84 is the position (x, y) of the white circle 83 of interest. ) Is considered to be larger than the pixel value Gobs (k, ig, jg). Further, the true green light amount Lg (I + 0.5, J−0.5) at the position (I + 0.5, J−0.5) of the black circle 84 and the pixel value Gobs at the position (x, y) of the white circle 83. The difference in the light amount (pixel value) from (k, ig, jg) depends on the difference (EG2′−EG2) between the average values EG2 ′ and EG2.

  The true green light quantity Lg (I-0.5, J + 0.5) at the position (I-0.5, J + 0.5) of the black circle 87 is the position (x, y) of the white circle 83 of interest. Is considered to be smaller than the pixel value Gobs (k, ig, jg) observed in FIG. Further, the true green light amount Lg (I-0.5, J + 0.5) at the position (I−0.5, J + 0.5) of the black circle 87 and the pixel value Gobs at the position (x, y) of the white circle 83. The difference in the light amount (pixel value) from (k, ig, jg) depends on the difference (EG2′−EG2) between the average values EG2 ′ and EG2.

  Therefore, the arithmetic circuit 24 passes through the pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83 of interest, and the position (I + 0.5, J−0.5) of the black circle 84. A plane Q3 having a slope p in the diagonal direction of the area A11 connecting the positions (I−0.5, J + 0.5) of the black circle 87 is obtained. Here, p = (EG2′−EG2) ÷ 2√2. That is, the slope p is the difference between the average value EG2 ′ of the pixel values in the upper right region 113 and the average value EG2 of the pixel values in the lower left region 111 in FIG. 30, and the position of the black circle 84 (I + 0.5, J−0). .5) and the center position of the region A20 of the upper right region 113 and the region A02 of the lower left region 111 in the diagonal direction of the region A11 connecting the position (I−0.5, J + 0.5) of the black circle 87. It is obtained by dividing by the distance 2√2. The inclination p represents the degree of change in the amount of light (pixel value) in the direction perpendicular to the edge at the edge portion from the upper left to the lower right, that is, the inclination of the edge.

  Then, the arithmetic circuit 24 obtains a value (pixel value) on the plane Q3 at the position (I-0.5, J-0.5) of the black circle 85, and obtains the value as a pixel value Gobs0 (k, ig, jg). And Further, the arithmetic circuit 24 obtains a value (pixel value) on the plane Q3 at the position (I−0.5, J + 0.5) of the black circle 87, and sets the value as the pixel value Gobs1 (k, ig, jg). . Similarly, the arithmetic circuit 24 calculates the value (pixel value) on the plane Q3 at the position (I + 0.5, J−0.5) of the black circle 84 and the position (I + 0.5, J + 0.5) of the black circle 86. Values (pixel values) on the plane Q3 are obtained, and these values are set as a pixel value Gobs2 (k, ig, jg) and a pixel value Gobs3 (k, ig, jg), respectively.

  Here, the plane Q3 is inclined only in the diagonal direction of the region A11 connecting the position of the black circle 84 (I + 0.5, J-0.5) and the position of the black circle 87 (I-0.5, J + 0.5). Therefore, the pixel value Gobs0 (k, ig, jg) is equal to the pixel value Gobs3 (k, ig, jg).

  Also in the “edge from the upper left to the lower right”, the position (I + 0.5, J−0.5) of the black circle 84 is changed to the position (I ′, J) as in the case of the “vertical edge” in FIG. When attention is paid to '), instead of the pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83, the position of the black circle 84 of interest (I + 0.5, J-0.5) A pixel value Gobs2 (k, ig, jg), which is a value (pixel value) on the plane Q3, is substituted into Expression (25).

  When attention is paid to the position (I−0.5, J−0.5) of the black circle 85 as the position (I ′, J ′), the pixel value Gobs (k, k) of the position (x, y) of the white circle 83 is obtained. In place of (ig, jg), a pixel value Gobs0 (k, ig, jg) which is a value (pixel value) on the plane Q3 at the position (I-0.5, J-0.5) of the black circle 85 of interest. ) Is substituted into equation (25).

  Further, when attention is paid to the position (I + 0.5, J + 0.5) of the black circle 86 as the position (I ′, J ′), the pixel value Gobs (k, ig, jg) of the position (x, y) of the white circle 83 is obtained. Instead of the pixel value Gobs3 (k, ig, jg) which is the value (pixel value) on the plane Q3 at the position (I + 0.5, J + 0.5) of the black circle 86 of interest. Assigned.

  When attention is paid to the position (I−0.5, J + 0.5) of the black circle 87 as the position (I ′, J ′), the pixel value Gobs (k, ig, In place of jg), a pixel value Gobs1 (k, ig, jg) which is a value (pixel value) on the plane Q3 at the position (I−0.5, J + 0.5) of the black circle 87 of interest Substituted in (25).

  As described above, the pixel value Gobs (k, ig, jg) (pixel value at the position of the white circle 83 in FIG. 34) observed at a position where there is an edge from the upper left to the lower right (a steep change in the amount of green light). Is a pixel value (Gobs0 (k, ig, jg), Gobs1 (k, ig, jg), Gobs2 (k, ig, jg), or Gobs3 (k, ig) corresponding to the inclination (steepness) p of the edge. , Jg)), and by substituting the changed pixel value into equation (25), Lg (I ′, J ′) as the object V in FIG. It is possible to obtain a more accurate (clear) true green light amount Lg (I ′, J ′) by being pulled.

  FIG. 35 illustrates a region A11 including the position of the white circle 83 of interest when the “edge from the upper right to the lower left direction” illustrated in FIG. 31 is detected. In the figure, one axis represents the X direction in the reference coordinate system, the axis perpendicular to the X direction represents the Y direction in the reference coordinate system, and the axis perpendicular to the X direction and the Y direction represents the pixel value.

  In FIG. 35, the average values EG3, EG3 ′, EG3 ″ determined for the edge determination region and the variances SG3, SG3 ′ satisfy the expression “EG3 ′ + SG3 ′ <EG3” <EG3-SG3 ”. There is an “lower left edge”.

  In this case, the true green light quantity Lg (I-0.5, J-0.5) at the position (I-0.5, J-0.5) of the black circle 85 is the position of the white circle 83 of interest. It is considered that the pixel value Gobs (k, ig, jg) observed at (x, y) is larger. Further, the true green light quantity Lg (I-0.5, J-0.5) at the position (I-0.5, J-0.5) of the black circle 85 and the position (x, y) of the white circle 83. The difference in the light amount (pixel value) from the pixel value Gobs (k, ig, jg) depends on the difference between the average values EG3 and EG3 ′ (EG3−EG3 ′).

  Further, the true green light amount Lg (I + 0.5, J + 0.5) at the position (I + 0.5, J + 0.5) of the black circle 86 is observed at the position (x, y) of the white circle 83 of interest. It is considered to be smaller than the pixel value Gobs (k, ig, jg). Further, the true green light amount Lg (I + 0.5, J + 0.5) at the position (I + 0.5, J + 0.5) of the black circle 86 and the pixel value Gobs (k, ig) at the position (x, y) of the white circle 83. , Jg) depends on the difference (EG3−EG3 ′) between the average values EG3 and EG3 ′.

  Therefore, the arithmetic circuit 24 passes through the pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83 of interest and passes through the position (I-0.5, J-0.5) of the black circle 85. ) And the position (I + 0.5, J + 0.5) of the black circle 86, a plane Q4 having a slope p in the diagonal direction of the area A11 is obtained. Here, p = (EG3−EG3 ′) ÷ 2√2. In other words, the slope p is the difference between the average value EG3 of the pixel values in the upper left region 121 and the average value EG3 ′ of the pixel values in the lower right region 123 in FIG. J-0.5) and the center position of the region A00 of the upper left region 121 and the region A22 of the lower right region 123 in the diagonal direction of the region A11 connecting the positions (I + 0.5, J + 0.5) of the black circle 86 It is obtained by dividing by the distance of 2√2. The inclination p represents the degree of change in the amount of light (pixel value) in the direction perpendicular to the edge at the edge portion from the upper right to the lower left, that is, the inclination of the edge.

  Then, the arithmetic circuit 24 obtains a value (pixel value) on the plane Q4 at the position (I-0.5, J-0.5) of the black circle 85, and obtains the value as a pixel value Gobs0 (k, ig, jg). And Further, the arithmetic circuit 24 obtains a value (pixel value) on the plane Q4 at the position (I−0.5, J + 0.5) of the black circle 87, and sets the value as the pixel value Gobs1 (k, ig, jg). . Similarly, the arithmetic circuit 24 calculates the value (pixel value) on the plane Q4 at the position (I + 0.5, J−0.5) of the black circle 84 and the position (I + 0.5, J + 0.5) of the black circle 86. Values (pixel values) on the plane Q4 are obtained, and these values are set as a pixel value Gobs2 (k, ig, jg) and a pixel value Gobs3 (k, ig, jg), respectively.

  Here, the plane Q4 is inclined only in the diagonal direction of the region A11 connecting the position of the black circle 85 (I−0.5, J−0.5) and the position of the black circle 86 (I + 0.5, J + 0.5). Therefore, Gobs1 (k, ig, jg) and Gobs2 (k, ig, jg) are equal.

  Also in “the edge from the upper right to the lower left direction”, the position (I + 0.5, J−0.5) of the black circle 84 is changed to the position (I ′, J ′) as in the case of the “vertical edge” in FIG. ), Instead of the pixel value Gobs (k, ig, jg) at the position (x, y) of the white circle 83, the position at the position (I + 0.5, J-0.5) of the black circle 84 of interest. A pixel value Gobs2 (k, ig, jg), which is a value (pixel value) on the plane Q4, is substituted into Expression (25).

  When attention is paid to the position (I−0.5, J−0.5) of the black circle 85 as the position (I ′, J ′), the pixel value Gobs (k, k) of the position (x, y) of the white circle 83 is obtained. In place of (ig, jg), a pixel value Gobs0 (k, ig, jg) which is a value (pixel value) on the plane Q4 at the position (I-0.5, J-0.5) of the black circle 85 of interest. ) Is substituted into equation (25).

  Further, when attention is paid to the position (I + 0.5, J + 0.5) of the black circle 86 as the position (I ′, J ′), the pixel value Gobs (k, ig, jg) of the position (x, y) of the white circle 83 is obtained. Instead of the pixel value Gobs3 (k, ig, jg), which is the value (pixel value) on the plane Q4 at the position (I + 0.5, J + 0.5) of the black circle 86 of interest, in the equation (25). Assigned.

  When attention is paid to the position (I−0.5, J + 0.5) of the black circle 87 as the position (I ′, J ′), the pixel value Gobs (k, ig, Instead of jg), the pixel value Gobs1 (k, ig, jg), which is the value (pixel value) on the plane Q4 at the position (I−0.5, J + 0.5) of the black circle 87 of interest, is expressed by the equation Substituted in (25).

  As described above, the pixel value Gobs (k, ig, jg) (pixel value at the position of the white circle 83 in FIG. 35) observed at a position where there is an edge from the upper right to the lower left direction (a sharp change in the amount of green light). , Pixel values (Gobs0 (k, ig, jg), Gobs1 (k, ig, jg), Gobs2 (k, ig, jg), or Gobs3 (k, ig, jg)), and by substituting the changed pixel value into equation (25), Lg (I ′, J ′) as the object V in FIG. 23 is pulled to an appropriate position in the spring model. Thus, a more accurate (clear) true green light amount Lg (I ′, J ′) can be obtained.

  From the above, the spring relational expression of the green light quantity in the sixth embodiment corresponding to the expression (25) in the third embodiment can be expressed as follows.

  The true red light quantity Lr (I ′, J ′) and the true blue light quantity Lb (I ′, J ′) other than green are also set in the same manner as the true green light quantity Lg (I ′, J ′). Can be obtained.

・・・（３４）

... (34)

  Here, Σ in equation (34) is the position (ig−0.5, jg−0.5) on the k-th captured image with respect to a certain position (I ′, J ′). The position (x, y) on the reference coordinates converted by θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1 (k, ig). , Jg) represents the sum of the number of sets.

  However, at the position of a certain (k, ig, jg) pixel value Gobs (k, ig, jg) in the set of (k, ig, jg) constituting the expression (34), When any one of “lateral edge”, “upper left to lower right edge”, and “lower right to lower left edge” is detected, Gobs ′ (k, ig, jg) in Expression (34) ), Instead of the pixel value Gobs (k, ig, jg) of the (k, ig, jg), the plane having the edge inclination p and passing through the pixel value Gobs (k, ig, jg) A value (pixel value) on the position (I ′, J ′) is used. In addition, at the position of a certain (k, ig, jg) pixel value Gobs (k, ig, jg) in the set of (k, ig, jg) constituting the expression (34), “vertical edge”, When none of “lateral edge”, “upper left to lower right edge”, and “lower right to lower left edge” is detected, Gobs ′ (k, ig, jg) in Expression (34) The pixel value Gobs (k, ig, jg) of (k, ig, jg) is used as is.

  Similarly, the spring relational expressions of the red and blue light amounts of the sixth embodiment corresponding to the expressions (26) and (27) of the third embodiment are similarly expressed by the expressions (35) and (36) as follows: Each can be represented.

・・・（３５）

... (35)

  Here, Σ in equation (35) is the position (ir−0.5, jr−0.5) on the k-th captured image with respect to a certain position (I ′, J ′). The position (x, y) on the reference coordinates converted by θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1 (k, ir , Jr) represents the sum of the number of sets.

  However, at the position of a certain (k, ir, jr) pixel value Robs (k, ir, jr) in the set of (k, ir, jr) constituting the expression (35), the “longitudinal edge”, When any one of “lateral edge”, “upper left to lower right edge”, and “lower right to lower left edge” is detected, Robs ′ (k, ir, jr) in Expression (35) ), Instead of the pixel value Robs (k, ir, jr) of the (k, ir, jr), the plane having the slope p of the edge and passing through the pixel value Robs (k, ir, jr) A value (pixel value) on the position (I ′, J ′) is used. Further, in the position of a certain (k, ig, jg) pixel value Gobs (k, ig, jg) in the set of (k, ig, jg) constituting the expression (35), the “vertical edge”, When none of “lateral edge”, “upper left to lower right edge”, and “lower right to lower left edge” is detected, Robs ′ (k, ir, jr) in Expression (35) The pixel value Robs (k, ir, jr) of (k, ir, jr) is used as it is.

・・・（３６）

... (36)

  Here, Σ in Expression (36) is a position (ib−0.5, jb−0.5) on the k-th captured image with respect to a certain position (I ′, J ′). The position (x, y) on the reference coordinates converted by θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1 (k, ib , Jb) represents the sum of the number of sets.

  However, at the position of a certain (k, ib, jb) pixel value Bobs (k, ib, jb) in the set of (k, ib, jb) constituting the expression (36), “vertical edge”, When any one of “lateral edge”, “upper left to lower right edge”, and “lower right to lower left edge” is detected, Bobs ′ (k, ib, jb) in Expression (36) ), Instead of the pixel value Bobs (k, ib, jb) of the (k, ib, jb), the plane having the slope p of the edge and passing through the pixel value Bobs (k, ib, jb) A value (pixel value) on the position (I ′, J ′) is used. In addition, at the position of a certain (k, ig, jg) pixel value Gobs (k, ig, jg) in the set of (k, ig, jg) constituting the expression (36), “vertical edge”, When none of the “lateral edge”, “upper left to lower right edge”, and “lower right to lower left edge” is detected, Bobs ′ (k, ib, jb) in Expression (36) The pixel value Bobs (k, ib, jb) of (k, ib, jb) is used as it is.

  A sixth embodiment of the image estimation process in step S4 of FIG. 2 will be described with reference to the flowchart of FIG.

  First, in step S141, the arithmetic circuit 24 pays attention to a certain position (I ′, J ′) on the reference coordinate system (hereinafter referred to as attention position (I ′, J ′)). Here, the target position (I ′, J ′) is the pixel center (i−0.5, j−0.5) of the “i-th and j-th pixels” of the first captured image that is the reference image. Represents.

  Then, the process proceeds from step S141 to S142, and the arithmetic circuit 24 determines the center position (ig−0.5, jg) of the pixel that receives the green component of the k-th image with respect to the target position (I ′, J ′). −0.5) is converted to the position (x, y) on the reference coordinate system obtained by converting the conversion parameters (θk, Tkx, Tky, Sk) by I′−1 ≦ x <I ′ + 1, J′−1 ≦ y. <A set of (k, ig, jg) that satisfies J ′ + 1 is obtained for all the first to Nth images, and the process proceeds to step S143.

  In step S143, the arithmetic circuit 24 “vertical edge”, “lateral edge”, “horizontal edge”, “position in the reference coordinate system corresponding to each (k, ig, jg) pair obtained in step S142”. It is determined whether there is any of “edge from upper left to lower right direction” and “edge from lower right to lower left direction”. If it is determined in step S143 that there is an edge in any one of the four directions described above, the process proceeds to step S144, and the arithmetic circuit 24 determines each pair of (k, ig, jg) corresponding to the position where the edge exists. Then, a plane passing through the pixel value Gobs (k, ig, jg) and having an edge inclination p is created (obtained). The arithmetic circuit 24 calculates a value (pixel value) on the plane at the target position (I ′, J ′), and proceeds from step S144 to S145.

  When it is determined in step S143 that there is no edge in any of the above four directions, or after the processing in step S144, in step S145, the arithmetic circuit 24 determines all the (k, ig, jg) is used to generate a spring relational expression of the green light quantity represented by Expression (34), and the process proceeds to step S146. Here, in step S143, the pixel value Gobs (k, ig, jg) determined that there is an edge in any of the above four directions at a position on the reference coordinate system corresponding to a certain (k, ig, jg). For jg), the arithmetic circuit 24 calculates the value (pixel value) on the plane at the position of interest (I ′, J ′) obtained in step S144 as the pixel value Gobs (k) of (k, ig, jg). , Ig, jg) is substituted into Gobs ′ (k, ig, jg) in equation (34). In step S143, the pixel value Gobs (k, ig, jg) determined to have no edge in any of the above four directions at the position on the reference coordinate system corresponding to a certain (k, ig, jg). , The arithmetic circuit 24 substitutes the pixel value Gobs (k, ig, jg) of the (k, ig, jg) into Gobs ′ (k, ig, jg) in the equation (34) as it is.

  In step S146, the arithmetic circuit 24 receives the center position (ir−0.5, jr−0.5) of the pixel that receives the red component of the kth image with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the data with the conversion parameters (θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. All sets (k, ir, jr) that satisfy (k, ir, jr) are obtained for the first to Nth images, and the process proceeds to step S147.

  In step S147, the arithmetic circuit 24 “vertical edge”, “lateral edge”, “horizontal edge”, “position in the reference coordinate system corresponding to each (k, ir, jr) pair obtained in step S146”. It is determined whether there is any of “edge from upper left to lower right direction” and “edge from lower right to lower left direction”. If it is determined in step S147 that there is an edge in one of the four directions described above, the process proceeds to step S148, and the arithmetic circuit 24 determines each of the (k, ir, jr) pairs corresponding to the position where the edge exists. Then, a plane passing through the pixel value Robs (k, ir, jr) and having an edge inclination p is created (obtained). Further, the arithmetic circuit 24 calculates a value (pixel value) on the plane at the position of interest (I ′, J ′), and proceeds from step S148 to S149.

  When it is determined in step S147 that there is no edge in any of the four directions described above, and after the processing in step S148, in step S149, the arithmetic circuit 24 determines all the (k, Using the set of ir, jr), the spring relational expression of the red light quantity represented by the expression (35) is generated, and the process proceeds to step S150. Here, in step S147, it is determined that the pixel value Robs (k, ir, jr) is determined to have an edge in any of the above four directions at a position on the reference coordinate system corresponding to a certain (k, ir, jr). For jr), the arithmetic circuit 24 calculates the value (pixel value) on the plane at the position of interest (I ′, J ′) obtained in step S148 as the pixel value Robs (k) of (k, ir, jr). , Ir, jr) is substituted into Robs ′ (k, ir, jr) in equation (35). In step S147, the pixel value Robs (k, ir, jr) determined to have no edge in any of the above four directions at the position on the reference coordinate system corresponding to a certain (k, ir, jr). , The arithmetic circuit 24 substitutes the pixel value Robs (k, ir, jr) of the (k, ir, jr) into Robs ′ (k, ir, jr) in the equation (35) as it is.

  In step S150, the arithmetic circuit 24 receives the center position (ib−0.5, jb−0.5) of the pixel that receives the blue component of the kth image with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the data with the conversion parameters (θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. All the sets of (k, ib, jb) that satisfy (k, ib, jb) are obtained for the first to Nth images, and the process proceeds to step S151.

  In step S151, the arithmetic circuit 24 “vertical edge”, “horizontal edge”, “horizontal edge”, “position in the reference coordinate system corresponding to each (k, ib, jb) pair obtained in step S152”. It is determined whether there is any of “edge from upper left to lower right direction” and “edge from lower right to lower left direction”. If it is determined in step S151 that there is an edge in any one of the four directions described above, the operation proceeds to step S152, and the arithmetic circuit 24 determines each pair of (k, ib, jb) corresponding to the position where the edge exists. Then, a plane passing through the pixel value Bobs (k, ib, jb) and having an edge inclination p is created (obtained). In addition, the arithmetic circuit 24 calculates a value (pixel value) on the plane at the position of interest (I ′, J ′), and proceeds from step S152 to S153.

  Further, when it is determined in step S151 that there is no edge in any of the above four directions, and after the processing in step S152, in step S153, the arithmetic circuit 24 determines all the (k, Using the set of ib, jb), a spring relational expression of the blue light quantity represented by Expression (36) is generated, and the process proceeds to Step S154. Here, in step S151, it is determined that the pixel value Bobs (k, ib, jb) is determined to have an edge in any one of the above four directions at a position on the reference coordinate system corresponding to a certain (k, ib, jb). For jb), the arithmetic circuit 24 calculates the value (pixel value) on the plane at the target position (I ′, J ′) obtained in step S152 as the pixel value Bobs (k) of the (k, ib, jb). , Ib, jb) is substituted into Bobs ′ (k, ib, jb) in the equation (36). In step S151, the pixel value Bobs (k, ib, jb) determined that there is no edge in any of the above four directions at the position on the reference coordinate system corresponding to a certain (k, ib, jb). , The arithmetic circuit 24 substitutes the pixel value Bobs (k, ib, jb) of the set of (k, ib, jb) into Bobs ′ (k, ib, jb) in the equation (36) as it is.

  In step S154, the arithmetic circuit 24 calculates the green light amount spring relational expression represented by the expression (34) obtained in step S145 and the red light quantity spring relational expression represented by the expression (35) obtained in step S149. By solving the spring relational expression of the blue light quantity represented by the expression (36) obtained in step S153 as a linear equation, the true green light quantity Lg (I ′, J ′) at the target position (I ′, J ′) is obtained. J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′) are obtained, and the process proceeds to step S155.

  In step S155, the arithmetic circuit 24 sets all the positions (I ′, J ′) as the target positions, that is, sets all the center positions of the pixels of the first captured image as the target positions (I ′, J ′). ) As to whether the true green light amount Lg (I ′, J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′) are obtained. Determine.

  If it is determined in step S155 that all positions (I ′, J ′) have not yet been obtained as attention positions, the process returns to step S141, and the processes in steps S141 to S155 are repeated. That is, the arithmetic circuit 24 sets the position (I ′, J ′) that has not been noticed as the next attention position (I ′, J ′), and the true green light amount Lg at the attention position (I ′, J ′). (I ′, J ′), true red light quantity Lr (I ′, J ′), and true blue light quantity Lb (I ′, J ′) are obtained.

  On the other hand, if it is determined in step S155 that all positions (I ′, J ′) are the target positions, the process proceeds to step S156, and the arithmetic circuit 24 calculates the true green light quantity Lg (I) obtained in step S154. ', J'), a true red light quantity Lr (I ', J'), and a true blue light quantity Lb (I ', J'), an image (signal) is estimated, and the D / A converter 9 or codec 12 is supplied as an output image and the process returns. For example, in the “i-th and j-th pixels”, the arithmetic circuit 24 calculates the true green light amount Lg (i−0.5, j−0.5) obtained in step S154 as a green value (G signal). ), The true red light quantity Lr (i−0.5, j−0.5) obtained in step S154 as the red value (R signal), and the blue value (B signal) obtained in step S154. From the true blue light quantity Lb (i−0.5, j−0.5), the image signal of the “i th and j th pixels” is estimated. Then, the arithmetic circuit 24 estimates the output image by performing the estimation for all the pixels having the position (I ′, J ′) as the center position.

  As described above, in the sixth embodiment, the pixel value observed at the position of the edge portion is changed by changing the pixel value to be substituted into the spring relational expression according to the inclination (steepness degree) of the edge. A more accurate (clear) image can be obtained.

  In the above-described example, the pixel value at the edge portion is changed to the pixel value on the target position (I ′, J ′) according to the slope (steepness) of the edge. The pixel values are used as they are, but weights are changed (changed) according to the distance to the target position (I ′, J ′) so as to reduce the influence of the pixel values at the edge portion in the spring relational expression. You can also

  In the above example, the slopes of the planes Q1 to Q4 to be created are positions in nine regions (3 × 3 pixel width region) centered on a certain position (x, y) on the reference coordinate system. Although it is determined using the observed pixel value, it may be determined based on the pixel value observed at the position of another region such as 16 regions (4 × 4 pixel width region).

(Seventh embodiment)
Next, a seventh embodiment of image estimation processing in the signal processing circuit 7 will be described. The seventh embodiment is an application of the third embodiment described in FIG.

  That is, in the seventh embodiment, in the third embodiment, in addition to the spring relational expressions expressed by the equations (25), (26), and (27), the correlation between the R signal, the G signal, and the B signal. And the true green light quantity Lg (x, y), the true red light quantity Lr (x, y), and the true blue light quantity Lb (x, y) satisfying all the conditional expressions. y).

  Focusing on the local portion of the image, the true green light amount Lg (x, y) equivalent to the light of the subject incident on the image sensor 4, the true red light amount Lr (x, y), the true light amount. The blue light quantity Lb (x, y) has a correlation between colors. Therefore, assuming that the image estimated by the image estimation process also has color correlation, by adding the condition of the color correlation further, an error due to noise or the like is reduced, and a more accurate solution, that is, more original light. A faithful and clear image can be obtained.

  A method for obtaining a specific color correlation condition will be described with reference to FIGS. 37 and 38. FIG. In FIGS. 37 and 38, the condition of the color correlation between green (G signal) and red (R signal) is considered.

  37, the ig-th and jg-th green pixels G (jg−1) (ig−1) of the k′-th image on the lower left side of FIG. 37 and the ir-th number of the k ″ -th image on the lower-right side of FIG. Note the jrth red pixel R (jr-1) (ir-1).

  The arithmetic circuit 24 determines the positions of the green pixel G (jg-1) (ig-1) of the k'th image and the red pixel R (jr-1) (ir-1) of the k "th image. As described in the third embodiment, conversion is performed using the conversion parameters (θk ′, Tk′x, Tk′y, Sk ′) and the conversion parameters (θk ″, Tk ″ x, Tk ″ y, Sk ″). Thus, the position of the first image on the upper side of FIG.

  The arithmetic circuit 24 then converts the k′-th green pixel G (jg−1) (ig−1) converted into the coordinate system of the first image and the k converted into the coordinate system of the first image. “Calculate the distance from the first red pixel R (jr−1) (ir−1). Further, the arithmetic circuit 24 allows an allowable value (determination value) delta (for example, the distance) to be regarded as the same position. , 0.25 pixel).

For example, consider the positions of the pixel G (jg-1) (ig-1) and the pixel R (jr-1) (ir-1) at the position (ig, jg) and the position (ir, jr), respectively. , the pixel G (jg-1) in the coordinate system of the k 'th image (ig-1) position of (ig, jg) and the point G _c and the pixel in the coordinate system of the k "th image R (jg-1 ) (Ig-1) at the position (ir, jr) as the point R _c and the position (ig, jg) at the pixel G (jg-1) (ig-1) in the coordinate system of the first image as G _{c ( k ′)} and the position (ir, jr) of the pixel R (jg−1) (ig−1) in the coordinate system of the first image as R _{c (k ″)} , respectively, the point G _c Expression (37) representing whether the distance between _{(k ′)} and the point R _{c (k ″)} is within the allowable value delta can be written as follows.

・・・（３７）

... (37)

Expression (37) is referred to as a distance conditional expression. Here, Dis [G _{c (k ′)} , R _{c (k ″)} ] represents a distance between the point G _{c (k ′)} and the point R _{c (k ″)} . The position represented by the point G _{c (k ′)} and the point R _{c (k ″)} is obtained by converting the position (ig, jg) and the position (ir, jr) into the conversion parameters (θk ′, Tk′x, Tk). 'y, Sk') and (θk ", Tk" x, Tk "y, Sk"), respectively.

  The arithmetic circuit 24 has an allowable range in which the k'th green pixel G (jg-1) (ig-1) and the k "th red pixel R (jr-1) (ir-1) are within a certain allowable range. Whether or not there is a pixel that can be regarded as being at the same position when considering the margin of delta is centered on a certain position (I ′, J ′) in the coordinate system of the first image (I ′ ± dX, J ′ ± dY), ie, (I′−dX, J′−dY), (I′−dX, J ′ + dY), (I ′ + dX, J′−dY), and (I The area surrounded by '+ dX, J' + dY) where dX and dY are predetermined values for setting neighboring areas, for example, the lengths in the X direction and Y direction for two pixels, etc. can do.

  In other words, the arithmetic circuit 24 is a region near (I ′ ± dX, J ′ ± dY) around the target position (I ′, J ′) in the coordinate system of the first image, that is, ( I′−dX, J′−dY), (I′−dX, J ′ + dY), (I ′ + dX, J′−dY), and (I ′ + dX, J ′ + dY) (K ′, ig, jg) and (k ″, ir, jr) satisfying the expression (37) are obtained.

  Then, the arithmetic circuit 24 calculates the pixel value Gobs (k ′, ig, jg) and the pixel value Robs (k ″, kg) corresponding to the obtained (k ′, ig, jg) and (k ″, ir, jr). ir, jr).

  The arithmetic circuit 24 obtains (k ′, ig, jg) and (k ″, ir, jr) satisfying the above equation (37) for all combinations in which k ′ and k ″ are 1 to N, respectively.

  In general, a combination of a plurality of (k ′, ig, jg) and (k ″, ir, jr) is detected, so that the arithmetic circuit 24 detects the detected (k ′, ig, jg) and A set of pixel values Gobs (k ′, ig, jg) and pixel values Robs (k ″, ir, jr) corresponding to (k ″, ir, jr) is shown in FIG. The G signal (Gobs (k ′, ig, jg)) and the vertical axis are plotted in the GR space with the R signal (Robs (k ″, ir, jr)).

  FIG. 38 shows a GR space in which points represented by sets of pixel values Gobs (k ′, ig, jg) and pixel values Robs (k ″, ir, jr) satisfying Expression (37) are plotted. .

  The cross marks in FIG. 38 indicate pixel values Gobs (k ′, ig, jg) and pixel values Robs corresponding to (k ′, ig, jg) and (k ″, ir, jr) detected by the arithmetic circuit 24. Represents a set of (k ″, ir, jr), that is, a set of pixel value Gobs (k ′, ig, jg) and pixel value Robs (k ″, ir, jr) satisfying Expression (37). .

  In the region near the target position (I ′, J ′), the true green light amount Lg (I ′, J ′) to be obtained and the true red light amount Lr (I ′, J ′) are shown in FIG. It seems that there is a correlation as shown in.

  Therefore, in the seventh embodiment, in addition to the spring relational expressions represented by the expressions (25) to (27) in the third embodiment, there is a correlation between green and red shown in FIG. It is added as a condition.

  That is, the arithmetic circuit 24 is represented by a set of pixel values Gobs (k ′, ig, jg) and pixel values Robs (k ″, ir, jr) that satisfy the distance conditional expression of Expression (37) by the arithmetic circuit 24. The principal component analysis is performed on the plurality of points plotted in the GR space of FIG.

  Then, the arithmetic circuit 24 obtains the direction of the principal component, which is the analysis result of the principal component analysis, and obtains a straight line Qg × G + Qr × R = Qc representing the direction (axis). Here, Qg, Qr, and Qc are constants representing straight lines in the GR space, and G and R are variables representing the G signal and the R signal in the GR space. Further, the arithmetic circuit 24 obtains a variance for a component orthogonal to the direction of the main component.

  Here, a spring having a natural length of 0 and a spring constant (spring strength) of H is introduced. The spring constant H can be an arbitrary function that monotonously decreases with respect to the variance of the component orthogonal to the direction of the principal component obtained in the GR space. For example, H = (1 ÷ dispersion) or the like. Can be adopted. That is, the smaller the variance, the stronger the spring (the larger the spring constant).

  One end of both ends of the spring having the spring constant H is connected to a point (Lg (I ′, J ′), Lr (I ′, J ′)) in the GR space, and the other end of the spring is the main component. It is movably connected to an arbitrary point on the direction straight line Qg × G + Qr × R = Qc. Thereby, the point (Lg (I ′, J ′), Lr (I ′, J ′)) is a straight line Qg × from the point (Lg (I ′, J ′), Lr (I ′, J ′)). It is pulled in the direction of a perpendicular line hanging down to G + Qr × R = Qc. That is, the points (Lg (I ′, J ′), Lr (I ′, J ′)) are pulled so that the color correlation coincidence (the relationship between the color correlation between the G signal and the R signal) is satisfied.

  Here, when the distance between the point (Lg (I ′, J ′), Lr (I ′, J ′)) and the straight line Qg × G + Qr × R = Qc in the direction of the principal component is represented by u, the spring constant H The force with which the spring pulls the point (Lg (I ′, J ′), Lr (I ′, J ′)) is represented by H × u (hereinafter referred to as GR spring force as appropriate). The GR spring force H × u in the GR space is decomposed into a G signal component and an R signal component, and added to the left side of each of the equations (25) and (26) as an addition target, so that color correlation is considered. It will be.

  The above color correlation conditions are also considered for green (G signal) and blue (B signal).

  As in the case of the green and red pixels shown in FIG. 37, the arithmetic circuit 24 calculates the k′th green pixel G (jg−1) (ig−1) and the k′′th blue pixel. The position of B (jb-1) (ib-1) is converted into a conversion parameter (θk ′, Tk′x, Tk′y, Sk ′) and a conversion parameter (θk ″ ′, Tk ″ ′ x, Tk ″ ′ y, The position in the coordinate system of the first image is obtained by performing conversion by Sk ″ ′).

  The arithmetic circuit 24 then converts the k′-th green pixel G (jg−1) (ig−1) converted into the coordinate system of the first image and the k converted into the coordinate system of the first image. The distance from the '' th blue pixel B (jb-1) (ib-1) is calculated. Further, the arithmetic circuit 24 considers that the distance is in the same position (determination value). Determine if it is within delta.

For example, consider the positions of pixel G (jg-1) (ig-1) and pixel B (jb-1) (ib-1) at position (ig, jg) and position (ib, jb), respectively. , 'position of the pixel in the coordinate system of the piece of the image G (jg-1) (ig -1) (ig, jg) and the point G _c a, k "' k pixels in the coordinate system of the piece of the image B (jb- 1) The position (ib, jb) of (ib-1) is the point B _c, and the position (ig, jg) of the pixel G (jg-1) (ig-1) in the coordinate system of the first image is G _{c (k ′)} and the position (ib, jb) of the pixel B (jb−1) (ib−1) in the coordinate system of the first image are respectively represented as B _{c (k ″ ′).} Expression (38) indicating whether the distance between G _{c (k ′)} and the point B _{c (k ″ ′)} is within the allowable value delta can be written as follows.

・・・（３８）

... (38)

Expression (38) is referred to as a distance conditional expression. Here, Dis [G _{c (k ′)} , B _{c (k ′ ′)} ] represents a distance between the point G _{c (k ′)} and the point B _{c (k ′ ′)} . The positions represented by the point G _{c (k ′)} and the point B _{c (k ″ ′)} are obtained by converting the position (ig, jg) and the position (ib, jb) into the conversion parameters (θk ′, Tk′x, Tk′y, Sk ′) and (θk ″ ′, Tk ″ ′ x, Tk ″ ′ y, Sk ″ ′) are affine transformed.

  The arithmetic circuit 24 has a k'th green pixel G (jg-1) (ig-1) and a k''th blue pixel B (jb-1) (ib-1). Whether or not there is a pixel that can be considered to be at the same position when considering the margin of the range delta is centered on a certain position (I ′, J ′) in the coordinate system of the first image (I ′ ± dX, J ′ ± dY), ie, (I′−dX, J′−dY), (I′−dX, J ′ + dY), (I ′ + dX, J′−dY), and ( I ′ + dX, J ′ + dY) where dX, dY are predetermined values for setting neighboring regions, for example, the lengths in the X and Y directions for two pixels, etc. It can be.

  In other words, the arithmetic circuit 24 is a region near (I ′ ± dX, J ′ ± dY) around the target position (I ′, J ′) in the coordinate system of the first image, that is, ( I′−dX, J′−dY), (I′−dX, J ′ + dY), (I ′ + dX, J′−dY), and (I ′ + dX, J ′ + dY) (K ′, ig, jg) and (k ″ ′, ib, jb) satisfying equation (38) are obtained.

  Then, the arithmetic circuit 24 calculates the pixel value Gobs (k ′, ig, jg) and the pixel value Bobs (k ″ (k ″) corresponding to the obtained (k ′, ig, jg) and (k ″ ′, ib, jb). ', Ib, jb).

  The arithmetic circuit 24 calculates (k ′, ig, jg) and (k ″ ′, ib, jb) satisfying the above equation (38) for all combinations in which k ′ and k ″ ′ are 1 to N, respectively. Ask.

  In general, since a combination of a plurality of (k ′, ig, jg) and (k ″ ′, ib, jb) is detected, the arithmetic circuit 24 detects (k ′, ig, jg). And a set of pixel values Gobs (k ′, ig, jg) and pixel values Bobs (k ″ ′, ib, jb) corresponding to (k ″ ′, ib, jb), and the horizontal axis represents a G signal (Gobs). (k ′, ig, jg)), and the vertical axis is plotted in the GB space with the B signal (Bobs (k ″ ′, ib, jb)).

  Therefore, in the seventh embodiment, in addition to the spring relational expressions expressed by the equations (25) to (27) in the third embodiment, it is further provided that there is a correlation between green and blue. Append.

  That is, the arithmetic circuit 24 is a set of a pixel value Gobs (k ′, ig, jg) and a pixel value Bobs (k ″ ′, ib, jb) that satisfy the distance conditional expression of the expression (38) by the arithmetic circuit 24. A principal component analysis is performed on a plurality of points plotted in the GB space.

  Then, the arithmetic circuit 24 obtains the direction of the principal component, which is the analysis result of the principal component analysis, and obtains a straight line Qg ′ × G + Qb ′ × B = Qc ′ representing the direction (axis). Here, Qg ′, Qb ′, and Qc ′ are constants representing straight lines in the GB space, and G and B are variables representing the G signal and the B signal in the GB space. Further, the arithmetic circuit 24 obtains a variance for a component orthogonal to the direction of the main component.

  Here, a spring having a natural length of 0 and a spring constant (spring strength) of H is introduced. The spring constant H can be an arbitrary function that monotonously decreases with respect to the variance of the component orthogonal to the direction of the principal component obtained in the GB space. For example, H = (1 ÷ dispersion) or the like. Can be adopted. That is, the smaller the variance, the stronger the spring (the larger the spring constant).

  One end of both ends of the spring having the spring constant H is connected to a point (Lg (I ′, J ′), Lb (I ′, J ′)) on the GB space, and the other end of the spring is the main component. It is connected so as to be movable to an arbitrary point on the direction straight line Qg ′ × G + Qb ′ × B = Qc ′. Thus, the point (Lg (I ′, J ′), Lb (I ′, J ′)) is changed from the point (Lg (I ′, J ′), Lb (I ′, J ′)) to the straight line Qg ′. It is pulled in the direction of the perpendicular hanging down to × G + Qb ′ × B = Qc ′. That is, the points (Lg (I ′, J ′), Lb (I ′, J ′)) are pulled so that the color correlation coincidence (the relationship between the color correlation between the G signal and the B signal) is satisfied.

  Here, when the distance between the point (Lg (I ′, J ′), Lb (I ′, J ′)) and the straight line Qg ′ × G + Qb ′ × B = Qc ′ in the direction of the principal component is represented by u, the spring The force with which the spring with a constant H pulls the point (Lg (I ′, J ′), Lb (I ′, J ′)) is represented by H × u (hereinafter referred to as GB spring force as appropriate). The GB spring force H × u in the GB space is decomposed into a G signal component and a B signal component, and added to the left side of each of the equations (25) and (27) as an addition target, so that the color correlation is considered. It will be.

  In the present embodiment, only the two color correlation conditions in the above-described GR and GB spaces are added. Similarly, the condition of the color correlation between the R signal and the B signal (in the RB space) is also described. May also be added.

  A seventh embodiment of the image estimation process in step S4 of FIG. 2 will be described with reference to the flowchart of FIG.

  Steps S171 to S177 are the same as steps S71 to S77 of the image estimation processing of the third embodiment shown in FIG.

  That is, in step S171, the arithmetic circuit 24 pays attention to a certain position (I ′, J ′) on the reference coordinate system (hereinafter referred to as attention position (I ′, J ′)). Here, the target position (I ′, J ′) is the pixel center (i−0.5, j−0.5) of the “i-th and j-th pixels” of the first captured image that is the reference image. Represents.

  Then, the process proceeds from step S171 to S172, and the arithmetic circuit 24 determines the center position (ig−0.5, jg) of the pixel that receives the green component of the k-th image with respect to the target position (I ′, J ′). −0.5) is converted to the position (x, y) on the reference coordinate system obtained by converting the conversion parameters (θk, Tkx, Tky, Sk) by I′−1 ≦ x <I ′ + 1, J′−1 ≦ y. <A set of (k, ig, jg) that satisfies J ′ + 1 is obtained for all of the first to Nth images, and the process proceeds to step S173.

  In step S173, the arithmetic circuit 24 generates a spring relational expression of the green light quantity represented by the equation (25) using all the (k, ig, jg) pairs obtained in step S172, The process proceeds to S174.

  In step S174, the arithmetic circuit 24 receives the center position (ir−0.5, jr−0.5) of the pixel that receives the red component of the k-th image with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the data with the conversion parameters (θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. The set of satisfying (k, ir, jr) is obtained for all the first to Nth images, and the process proceeds to step S175.

  In step S175, the arithmetic circuit 24 generates a spring relational expression of the red light quantity represented by the equation (26) using all the (k, ir, jr) pairs obtained in step S174, The process proceeds to S176.

  In step S176, the arithmetic circuit 24 determines the center position (ib−0.5, jb−0.5) of the pixel that receives the blue component of the kth image with respect to the target position (I ′, J ′). The position (x, y) on the reference coordinate system obtained by converting the data with the conversion parameters (θk, Tkx, Tky, Sk) satisfies I′−1 ≦ x <I ′ + 1, J′−1 ≦ y <J ′ + 1. All the sets of (k, ib, jb) that satisfy (k, ib, jb) are obtained for the first to Nth images, and the process proceeds to step S177.

  In step S177, the arithmetic circuit 24 generates a spring relational expression of the blue light quantity represented by the equation (27) using all the pairs (k, ib, jb) obtained in step S176, The process proceeds to S178.

  In step S178, the arithmetic circuit 24 satisfies Expression (37) in a region defined by (I ′ ± dX, J ′ ± dY) centered on the target position (I ′, J ′) (k ′, ig, jg) and (k ″, ir, jr) are all obtained. Further, the arithmetic circuit 24 sets the obtained (k ′, ig, jg) and (k ″, ir, jr). The points on the GR space (Gobs (k ′, ig, jg), Robs (k ″, ir, jr)) specified by the above are plotted and the principal component analysis is performed. The variance of the component orthogonal to the direction is obtained, and the reciprocal of the variance is obtained as the spring constant H. Then, the arithmetic circuit 24 obtains a straight line Qg × G + Qr × R = Qc representing the principal component direction, and points on the GR space. The distance u between (Lg (I ′, J ′), Lr (I ′, J ′)) and the straight line Qg × G + Qr × R = Qc Furthermore, the arithmetic circuit 24 adds the G signal component of the GR spring force H × u to the addition target on the left side of the equation (25), and obtains the GR spring force as an unknown. The H × u R signal component is added as an addition target to the left side of Expression (26), and the process proceeds from step S178 to S179.

  In step S179, the arithmetic circuit 24 satisfies Expression (38) in a region defined by (I ′ ± dX, J ′ ± dY) centered on the target position (I ′, J ′) (k ′, ig, jg) and (k ″ ′, ib, jb) are all obtained. Further, the arithmetic circuit 24 obtains the obtained (k ′, ig, jg) and (k ″ ′, ib, jb) and The points in the GB space (Gobs (k ′, ig, jg), Bobs (k ″ ′, ib, jb)) specified by the set of are plotted and the principal component analysis is performed. The variance of the component orthogonal to the principal component direction is obtained, and the reciprocal of the variance is obtained as the spring constant H. Then, the arithmetic circuit 24 obtains a straight line Qg ′ × G + Qb ′ × B = Qc ′ representing the principal component direction, The distance between the point (Lg (I ′, J ′), Lb (I ′, J ′)) on the GB space and the straight line Qg ′ × G + Qb ′ × B = Qc ′ The GB spring force H × u as an unknown represented by u is obtained, and the arithmetic circuit 24 adds the G signal component of the GB spring force H × u to the addition target on the left side of Expression (25), and The B signal component of the GB spring force H × u is added as an addition target to the left side of Expression (27), and the process proceeds from step S179 to S180.

  In step S180, the arithmetic circuit 24 calculates the spring relational expression of the green light quantity to which the color correlation condition in the GR and GB spaces is added, the spring relational expression of red light quantity to which the condition of the color correlation in the GR space is added, GB The true green light quantity Lg (I ′, J ′) at the target position (I ′, J ′) is obtained by solving the spring relational expression of the blue light quantity to which the color correlation condition in the space is added, for example, as a linear equation. ), The true red light amount Lr (I ′, J ′) and the true blue light amount Lb (I ′, J ′) are obtained, respectively, and the process proceeds to step S181.

  In step S181, the arithmetic circuit 24 sets all the positions (I ′, J ′) as the target positions, that is, sets all the center positions of the pixels of the first captured image as the target positions (I ′, J ′). ) As to whether the true green light amount Lg (I ′, J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′) are obtained. Determine.

  If it is determined in step S181 that all positions (I ′, J ′) have not yet been obtained as attention positions, the process returns to step S171, and the processes in steps S171 to S181 are repeated. That is, the arithmetic circuit 24 sets the position (I ′, J ′) that has not been noticed as the next attention position (I ′, J ′), and the true green light amount Lg at the attention position (I ′, J ′). (I ′, J ′), true red light quantity Lr (I ′, J ′), and true blue light quantity Lb (I ′, J ′) are obtained.

  On the other hand, if it is determined in step S181 that all positions (I ′, J ′) are the target positions, the process proceeds to step S182, and the arithmetic circuit 24 calculates the true green light amount Lg (I) obtained in step S180. ', J'), a true red light quantity Lr (I ', J'), and a true blue light quantity Lb (I ', J'), an image (signal) is estimated, and the D / A converter 9 or codec 12 is supplied as an output image, and the process returns. For example, in the “i-th and j-th pixels”, the arithmetic circuit 24 calculates the true green light amount Lg (i−0.5, j−0.5) obtained in step S180 as a green value (G signal). ), The true red light amount Lr (i−0.5, j−0.5) obtained in step S180 as a red value (R signal), and the blue value (B signal) obtained in step S180. From the true blue light quantity Lb (i−0.5, j−0.5), the image signal of the “i th and j th pixels” is estimated. Then, the arithmetic circuit 24 estimates the output image by performing the estimation on all the pixels having the position (I ′, J ′) as the center position.

  As described above, in the seventh embodiment, a color correlation condition is further added to obtain a more accurate solution that reduces errors due to noise and the like, that is, a clearer image that is more faithful to the original light. be able to.

  Now, in order to clarify the features of the present invention, the difference between the present invention and the prior art will be described again.

  FIG. 40 shows a processing flow of the method of the present invention and the conventional method.

  As shown in FIG. 40, conventionally, “(a) in the figure: an image output from a single-plate sensor (for example, an image of a Bayer array: one of R signal, G signal, or B signal per pixel) There are a number of “conversion methods from only one image) to a normal color image (an image having three data of R signal, G signal, and B signal for each pixel)”. In fact, some digital still cameras using a single-plate sensor sold as a product implement this conversion method. This conversion method is a process called demosaicing.

  Conventionally, “(a) in the figure: from a plurality of dark (or poor S / N ratio) color images (images having three data of R signal, G signal, and B signal per pixel) A method of creating a clear (or a good S / N ratio) color image (an image having three data of R signal, G signal, and B signal for each pixel) is also known. Specifically, there are methods such as Patent Document 2 and Patent Document 3 described in the related art.

  Therefore, by combining these two methods, “an image output from a plurality of dark (or poor S / N) single-plate sensors (for example, an image of a Bayer array: R signal, G signal, or B per pixel) Create a clear (or good signal-to-noise ratio) color image (an image with three data of R, G, and B signals per pixel) from an image that has only one signal) I can do it.

  That is, an image (for example, Bayer array image: one of R signal, G signal, or B signal per pixel) output from a plurality (m) of dark (or poor SN ratio) single-plate sensors. A demosaicing process 202-m is performed on each of the (only images) 201-m, as shown in (a) in the figure, and a dark (or poor S / N ratio) color image (R signal per pixel) The image 203-m is created once. Then, for these color images 203-m, a clear (or good SN ratio) color image (R per pixel) is obtained by using the methods of Patent Document 2 and Patent Document 3 shown in FIG. An image 204 in which three data of the signal, the G signal, and the B signal are gathered may be created. Here, m is an integer representing a predetermined number.

  In the demosaicing process, naturally, an interpolation process or a process similar thereto is performed, so that the sharpness of the image is reduced. In the processes in Patent Document 2 and Patent Document 3 and the like, the interpolation of the digitally sampled data is performed, so that the sharpness of the image is reduced. Thus, in (a) and (b) in the figure, the interpolation processing is performed twice in total, so that the sharpness of the image is considerably diminished. Incidentally, the interpolation process is a process used when restoring data at a specific point from the surrounding data, and it is clear that the sharper the image becomes, the more the interpolation is performed.

  On the other hand, in the present invention, “an image output from a plurality of dark (or poor signal-to-noise ratio) single-plate sensors (for example, an image in a Bayer array: one of R signal, G signal, or B signal per pixel) "It creates a clear (or good S / N ratio) color image (an image with three data of R signal, G signal, and B signal per pixel) directly from" there is only one image ". There is only one process or similar process. Therefore, a clearer image than the conventional method can be restored. Thus, the difference between the present invention and the prior art and the advantages of the present invention are clear.

  Further, in the conventional method, demosaicing processing is performed on each image. For example, when there are eight images, demosaicing processing ((A) in the figure) must be performed eight times. That is, there is a drawback that the calculation amount becomes enormous.

  The difference between the present invention and the conventional method will be described from another viewpoint.

  In the method disclosed in Patent Document 2 described in the background of the prior art, it is described that the positions may be shifted and overlapped by the amount of camera shake. This means that an image whose position is shifted by the amount of camera shake is created by interpolation. Superimposing N images means that N pixels of data are superimposed on all pixels of the output image. Therefore, after overlapping, all pixels are divided by a fixed value (that is, N or “(exposure time at proper exposure) ÷ (exposure time at actual photographing) ÷ N”). . In addition, there has also been a method of obtaining a high-definition image by causing a certain fixed amount of pixel shift, shooting a plurality of images, and combining the images while aligning the images. Even in this case, since the deviation amount is determined, each pixel of the output image is always divided (normalized) by a predetermined value.

  On the other hand, in the first embodiment and the second embodiment of the present invention, the number of conditions at each position (x, y) of the output image, that is, an observed value (observation) that can be used to estimate the true value. The number of pixels varies from case to case. When a certain person captures a scene, there is a possibility that there are many input pixels Gobs (k, ig, jg) near the position (x, y). In that case, the equation (20) involving the position (x, y) increases. Also, if there are only a few input pixels Gobs (k, ig, jg) in the vicinity, the equation (20) involving the position (x, y) is reduced. As described above, the number of conditional expressions related to each position (x, y) of the output image is variable, and depends on the state (shift amount due to camera shake) when the image is actually taken. For this reason, in the estimation of the true value of each pixel position, the observation points that are candidates to be considered and the number thereof can be adaptively changed according to the imaging state (movement between images). In this case, for example, a configuration in which the number of observation points is fixed and that number of observation points is selected from the candidate observation points, a configuration in which both the number of observation points and the selected observation points are changed, or an observation point It is possible to employ a configuration in which both are used in accordance with predetermined conditions regarding the position, number, distribution, and the like.

  In the third to seventh embodiments of the present invention, the position is shifted by the amount of camera shake, but because of the Bayer array, some data is not necessarily added to all output positions. If there is no position-corrected data in the 2 × 2 pixels shown in FIG. 21 for the k-th image, none of the data of the k-th image is added to the output pixel. On the other hand, if there are two pieces of position-corrected data in the 2 × 2 pixels shown in FIG. 21, two data are added (weighted) to the output pixels. More specifically, a certain pixel will eventually become an output pixel by dividing by 10 if the total weight is 10. Another pixel finally becomes an output pixel by dividing by 3.5 if the total weight is 3.5. In other words, in the present invention, in the expressions shown in Expression (25), Expression (26), Expression (27), and the like, the total number added by Σ depends on the actual shooting state (shift amount due to camera shake). . For this reason, in the estimation of the true value of each pixel position, the observation points that are candidates to be considered and the number thereof can be adaptively changed according to the imaging state (movement between images).

  As described above, in the present invention, when restoring a clear image from a plurality of images output from a single-plate sensor (Bayer array sensor), the number of conditional expressions is variable or the weight is variable. In view of the fact that there is a case where it is necessary to do so, an appropriate restoration method has been provided.

  The series of processes described above can be executed by dedicated hardware or can be executed by software. In this case, for example, the digital camera 1 can be realized by causing a computer as shown in FIG. 41 to execute a program.

  41, a CPU (Central Processing Unit) 301 executes various processes according to a program stored in a ROM (Read Only Memory) 302 or a program loaded from a storage unit 308 to a RAM (Random Access Memory) 303. To do. The RAM 303 also appropriately stores data necessary for the CPU 301 to execute various processes. The CPU 301 executes processing performed by the motion detection circuit 23 and the arithmetic circuit 24 of the signal processing circuit 7.

  The CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output interface 305 is also connected to the bus 304.

  The input / output interface 305 includes an input unit 306 including a keyboard and a mouse, a display including a CRT (Cathode Ray Tube) and an LCD (Liquid Crystal display), an output unit 307 including a speaker, and a hard disk. A communication unit 309 including a storage unit 308, a modem, a terminal adapter, and the like is connected. The communication unit 309 performs communication processing via a network such as the Internet. The imaging unit 311 includes the imaging device 4 illustrated in FIG. 1 and the like, images a subject, and supplies image data of the captured subject to the CPU 301 and the like via the input / output interface 305.

  A drive 310 is connected to the input / output interface 305 as necessary, and a magnetic disk 321, an optical disk 322, a magneto-optical disk 323, or a semiconductor memory 324 is appropriately mounted, and a computer program read from these is loaded. It is installed in the storage unit 308 as necessary.

(Eighth embodiment)
Next, an eighth embodiment of image estimation processing in the signal processing circuit 7 will be described. In the eighth embodiment, a part of the third embodiment described with reference to FIG. 24 is improved.

  In the third embodiment, data received by each pixel of the image sensor 4 (FIG. 1) (amount of received light) is regarded as point-sampled data, and the pixel value observed at the center of each pixel is not clear and there is no camera shake. By expressing the relationship with the image signal as an image by a spring model, a clear image more faithful to the original light was obtained.

  For example, regarding the G signal among the G signal, the R signal, and the B signal that form an image signal as a clear image without camera shake, in the third embodiment, the position (I ′, J As described above, the true green light quantity Lg (I ′, J ′) to be obtained by “) is represented by the spring relational expression of the green light quantity in the formula (25).

  {√2-F ((x, y), () of the spring constant {√2−F ((x, y), (I ′, J ′))} / (Mk × E) in the equation (25) I ′, J ′))} is monotonously decreasing with respect to the distance between the position (I ′, J ′) and the position (x, y) in the vicinity of the position (I ′, J ′). Represents the weight according to. Therefore, the pixel value Gobs (k, ig, jg) at the position (x, y) and the true green light amount Lg (I ′, J ′) at the position (I ′, J ′) are equal to the position (I ′ , J ′) and the position (x, y) are increased, the pulling force of the spring is weakened, and the distance between the position (I ′, J ′) and the position (x, y) is decreased. There is a relationship in which the pulling force becomes stronger.

  On the other hand, (Mk × E) in the spring constant {√2−F ((x, y), (I ′, J ′))} / (Mk × E) in the equation (25) is correlated double sampling. The noise amount E of the noise component not removed by the circuit 5 (FIG. 1) is multiplied by Mk (k = 1 to N) due to gain increase. 1 / (Mk × E) represents a weight due to noise that monotonously decreases with respect to the noise amount E. Accordingly, the pixel value Gobs (k, ig, jg) at the position (x, y) and the true green light amount Lg (I ′, J ′) at the position (I ′, J ′) have an amount of noise E. The larger the value, the weaker the pulling force of the spring, and the smaller the noise amount E, the stronger the pulling force of the spring.

  In the eighth embodiment, the position (I ′, J ′) and the pixel value Gobs, which are {√2−F ((x, y), (I ′, J ′))} in Expression (25), are expressed. For example, Cubic (I′−x) using a cubic (Cubic) function as a weight (hereinafter simply referred to as a distance weight) according to the distance from (k, ig, jg) to a certain position (x, y). ) × Cubic (J'-y) is adopted. That is, in the eighth embodiment, instead of {√2-F ((x, y), (I ′, J ′))} in Expression (25), Cubic (I′−x) × Cubic (J '-Y) is adopted.

  Here, the cubic function Cubic (z) is expressed by the following equation (39).

・・・（３９）

... (39)

  In the equation (39), a is a predetermined constant, for example, -1.

  As shown in FIG. 42, the cubic function Cubic (z) becomes 0 when the variable z is 2 ≦ | z | and | z | = 1, and when 1 <| z | <2, , Negative value. When | z | <1, Cubic (z) becomes a positive value and decreases as the value of | z | increases. In other words, the cubic function Cubic (z) is a function having a low-pass filter characteristic when considered on the frequency axis (when Fourier transformed).

In Expression (25), the p-th pixel value Gobs (k, ig, jg) to be subjected to the summation (Σ) is represented by v _p and the spring constant { _{p p} pixel value v _p { √2-F ((x, y), (I ′, J ′))} / (Mk × E) is represented as a spring constant K _p, and further, the expression (25) is converted into a true green light quantity Lg ( Solving for I ′, J ′) yields equation (40).

・・・（４０）

... (40)

  Here, in formula (40), Σ represents a summation about p.

From equation (40), according to the spring model, the true green light quantity Lg (I ′, J ′) is obtained by weighting the spring constant K _p and using the pixel value v _p. It can be said that. The same applies to the true red light amount Lr (I ′, J ′) and the true blue light amount Lb (I ′, J ′).

  FIG. 43 shows the positions (ig−0.5, jg−0.5) of N captured images with respect to the position (I ′, J ′) on the reference coordinate system (first coordinate system). All pairs of integers k, ig, and jg satisfying I′−2 ≦ x <I ′ + 2 and J′−2 ≦ y <J ′ + 2 at the converted position (x, y) are obtained, and the pixel value Gobs ( k, ig, jg) are points G11 to G19 where the observed points are shown.

  In FIG. 43, nine pixel values Gobs (k, ig, jg) observed at points G11 to G19 indicated by white circles are specified for the position (I ′, J ′) indicated by black circles (k , Ig, jg) is obtained. The sets of integers k, ig, and jg at points G11 to G19 are different from each other.

  Here, in the third embodiment, as shown in FIG. 21, the positions (ig−0.5, jg−0.5) of the N captured images with respect to the position (I ′, J ′). ) On the reference coordinate system (first coordinate system) is an integer satisfying I′−1 ≦ x <I ′ + 1 and J′−1 ≦ y <J ′ + 1. Whereas a set of k, ig, and jg is obtained, in the eighth embodiment, as shown in FIG. 43, I′−2 ≦ x <I ′ + 2, J′−2 ≦ y <J ′ + 2 A set of integers k, ig, and jg that satisfy the above is obtained. As described above, this is because the cubic function Cubic (z) represented by the equation (39) takes a value corresponding to the argument z in the range of −2 ≦ z ≦ 2 (| z |> 2). The range is 0 regardless of the argument z), so that the range corresponding to that range, ie, I′−2 ≦ x <I ′ + 2 and J′−2 ≦ y <J ′ + 2 is observed. This is because the true green light amount Lg (I ′, J ′) at the position (I ′, J ′) is estimated using the pixel value Gobs (k, ig, jg).

In the third embodiment, the position of the k-th captured image (ig-0.5, jg-0.5) is set to the first captured image that is the reference coordinate system as follows. position which was converted to _{((ig-0.5) (k} ), (jg-0.5) (k)). That is, the motion detection circuit 23- (k-1) is configured with a rotation angle θk, a scale Sk, and a parallel movement amount (Tkx, Tky) based on the positional relationship between the first captured image and the kth captured image. Conversion parameters (θk, Tkx, Tky, Sk) of the affine transformation to be performed are obtained and supplied to the arithmetic circuit 24. Then, the arithmetic circuit 24 uses the transformation parameters (θk, Tkx, Tky, Sk) supplied from the motion detection circuit 23- (k-1) in the affine transformation equation (3) to obtain the k-th captured image. (Ig−0.5, jg−0.5) is converted into a position ((ig−0.5) _(k) , (jg−0.5) _(k) ) in the reference coordinate system.

  On the other hand, in the eighth embodiment, the motion detection circuit 23- (k-1) uses an affine transformation in which the positional relationship between the first captured image and the kth captured image is expressed by the following equation (41). The conversion parameter of is obtained.

・・・（４１）

... (41)

  That is, the motion detection circuit 23- (k-1) is a matrix (ak ′, bk ′, ck ′, dk) of Expression (41) that represents the positional relationship between the first captured image and the kth captured image. ') And a two-dimensional vector (Tkx', Tky ') are obtained and supplied to the arithmetic circuit 24. Here, the transformation parameters of the affine transformation represented by the matrix (ak ′, bk ′, ck ′, dk ′) and the two-dimensional vector (Tkx ′, Tky ′) in the equation (41) are simply converted into the transformation parameter (ak ', Bk', dk ', Tkx', Tky ').

  In addition, by setting ak ′ = dk ′ = Sk × cos (θk) and −bk ′ = ck ′ = Sk × sin (θk), Expression (41) matches Expression (3).

Here, (Xk, Yk) in Expression (41) is the position of the pixel of the k-th captured image, and (X1 _(k) , Y1 _(k) ) is the position as in Expression (3). This represents the position (position on the reference coordinate system) on the first captured image obtained by affine transformation of (Xk, Yk) with Expression (41). That is, the subscript (k) indicates that a certain position of the k-th image has been converted on the reference coordinate system. Note that the first captured image is regarded as conversion parameters (a1 ′, b1 ′, c1 ′, d1 ′, T1x ′, T1y ′) = (1, 0, 0, 1, 0, 0). , K = 1 to N, the equation (41) is established.

  For example, in the motion detection circuit 23-1, conversion parameters (a2 ′, b2 ′, c2) in which the positional relationship between the first captured image and the second captured image is expressed by the following equation (42). ', D2', T2x ', T2y') are obtained and supplied to the arithmetic circuit 24.

・・・（４２）

... (42)

  Further, for example, in the motion detection circuit 23-2, conversion parameters (a3 ′, b3 ′) in which the positional relationship between the first captured image and the third captured image is expressed by the following equation (43). , C3 ′, d3 ′, T3x ′, T3y ′) are obtained and supplied to the arithmetic circuit 24.

・・・（４３）

... (43)

  Expression (43) indicates that the position (X3, Y3) on the third captured image is compared with the position (X1, Y1) on the first captured image, for example, for the handheld shooting, the conversion parameter ( a3 ′, b3 ′, c3 ′, d3 ′, T3x ′, T3y ′).

  As described above, the conversion parameters (ak ′, bk ′, ck ′, dk ′, Tkx ′, Tky ′) are obtained from the positional relationship of the k-th captured image with reference to the first captured image. In addition to obtaining, an acceleration sensor may be provided in the digital camera 1, and so-called mechanically obtained from the output of the acceleration sensor. In addition, in hand-held camera shake, it can be considered that the positional relationship between the first captured image and the k-th captured image contains almost no rotation component. The rotation components of ak ′, bk ′, ck ′, dk ′, Tkx ′, and Tky ′) may be negligible (fixed values indicating that they are not rotating).

  On the other hand, regarding the weighting due to noise in Expression (25), in the eighth embodiment, it is assumed that the same gain-up (n′-bit shift) is performed for all N captured images. In other words, in the eighth embodiment, the N images (input images) captured by the image sensor 4 are all 1 / M = 1 / Mk brightness of the image captured with appropriate exposure. To do. For example, when the captured image is 1/8 the brightness of an image captured with appropriate exposure, M = 8, and the shift circuit 21 (FIG. 1) can increase the gain by 3-bit shift.

  In the shift circuit 21, the same gain-up is performed on all the N captured images, so that the weight 1 / (Mk × E) due to the noise in Equation (25) is 1 / (Mk × E) in the eighth embodiment. (M × E).

  From the above, the expression (25) in the third embodiment can be expressed as the following expression (44) in the eighth embodiment.

・・・（４４）

... (44)

  Here, Σ in the equation (44) is obtained by converting the position (ig−0.5, jg−0.5) into the conversion parameters (ak ′, bk ′, ck ′) with respect to a certain position (I ′, J ′). , Dk ′, Tkx ′, Tky ′), the position (x, y) on the reference coordinate system satisfies I′−2 ≦ x <I ′ + 2, J′−2 ≦ y <J ′ + 2. k, ig, jg) represents the sum of the number of sets. For example, in the example shown in FIG. 43, it is the sum of nine (k, ig, jg) pairs of points G11 to G19.

  Equation (44) is the green light amount spring relational expression of the eighth embodiment using the cubic function Cubic (z) corresponding to the green light amount spring relational expression of Equation (25). it can.

  Since 1 / (M × E), which is a weight due to noise in Expression (44), is constant regardless of the set of (k, ig, jg) that is the object of Σ, it can be excluded from the object of Σ. Therefore, the equation (44) can be transformed into the following equation (45).

・・・（４５）

... (45)

  Further, when the equation (45) is solved for the true green light quantity Lg (I ′, J ′) to be obtained, the following equation (46) is obtained.

・・・（４６）

... (46)

  In the eighth embodiment, the true green light amount Lg (I ′, J ′) is obtained by Expression (46).

  According to the equation (46), the true green light quantity Lg (I ′, J ′) has a pixel value Gobs (k, ig,) with Cubic (I′−x) × Cubic (J′−y) as a weight. It can be said that it is obtained by performing weighted addition using jg). Here, the weight Cubic (I′−x) × Cubic (J′−y) is the position (I ′, J ′) and the position (x, y) where the pixel value Gobs (k, ig, jg) is located. It has the characteristics of a low-pass filter with respect to distance.

  In the following, Expression (46) is referred to as a green light quantity weight addition expression. In addition, the numerator and denominator of the weight addition formula for the green light quantity in Expression (46) are expressed as Expression (47) and Expression (48) as follows.

・・・（４７）

... (47)

・・・（４８）

... (48)

If the data (light quantity) before the pixel value Gobs (k, ig, jg) is increased by M times in the shift circuit 21 is D _{Gobs (k, ig, jg)} , the equation (46) is It can be expressed as equation (49).

・・・（４９）

... (49)

  Similarly to the equation (46), the true red light amount Lr (I ′, J ′) and the true blue light amount Lb (I ′, J ′) at the position (I ′, J ′) 50) and equation (51), respectively.

・・・（５０）

... (50)

  Hereinafter, the equation (50) is referred to as a red light amount weight addition equation. Here, Σ in the equation (50) is a position (ir−0.5, jr−0.5) is converted into a conversion parameter (ak ′, bk ′, ck ′) with respect to a certain position (I ′, J ′). , Dk ′, Tkx ′, Tky ′), the position (x, y) on the reference coordinate system satisfies I′−2 ≦ x <I ′ + 2, J′−2 ≦ y <J ′ + 2. k, ir, jr) represents the sum of the number of sets.

・・・（５１）

... (51)

  Hereinafter, Expression (51) is referred to as a blue light amount weight addition expression. Here, Σ in the equation (51) is obtained by converting the position (ib−0.5, jb−0.5) into the conversion parameters (ak ′, bk ′, ck ′) with respect to a certain position (I ′, J ′). , Dk ′, Tkx ′, Tky ′), the position (x, y) on the reference coordinate system satisfies I′−2 ≦ x <I ′ + 2, J′−2 ≦ y <J ′ + 2. k, ib, jb) represents the sum of the number of sets.

  Also, the numerator and denominator of the red light amount weight addition formula of Formula (50) are represented by Formula (52) and Formula (53), respectively.

・・・（５２）

... (52)

・・・（５３）

... (53)

  Similarly, the numerator and the denominator of the weight addition formula for the blue light quantity in Expression (51) are represented by Expression (54) and Expression (55), respectively.

・・・（５４）

... (54)

・・・（５５）

... (55)

  The green light quantity weight addition formula represented by Expression (46), the red light quantity weight addition formula represented by Expression (50), and the blue light quantity weight addition represented by Formula (51). Using the equation, the arithmetic circuit 24 calculates the true green light amount Lg (I ′, J ′), the true red light amount Lr (I ′, J ′), and the true blue light amount Lb (I ′, J ′). The processing for obtaining ') is referred to as normal processing in contrast to exception processing described later.

  Next, the weight addition formula for the green light quantity in Expression (46), the weight addition expression for the red light quantity in Expression (50), and the weight addition expression for the blue light quantity in Expression (51) will be considered.

  First, the weight addition formula of the green light quantity in Expression (46) is obtained by calculating the pixel value Gobs (k, ig, jg) and the distance weight Cubic (I′−x) × Cubic (J′−y) at the position (x, y). ) And the numerator of equation (47), which is a weighted addition value of pixel values Gobs (k, ig, jg) by distance weight Cubic (I′−x) × Cubic (J′−y)) Divided by the denominator of the equation (48), which is the sum of the weights Cubic (I′−x) × Cubic (J′−y).

  However, when the expression (48), which is the denominator of the expression (46), is 0 (including almost 0), the true green light amount Lg (I ′, J ′) obtained by the expression (46) is unstable. (Undetermined) Unreliable value. In other words, the position (I ′, J ′) where the expression (48), which is the denominator of the expression (46), is 0 is included in the pixel value Gobs (k, ig, jg) in the numerator of the expression (46). A small noise component (error) is divided by a denominator of 0 to be amplified to a large value. As a result, the true green light amount Lg (I ′, J ′) obtained by the equation (46) is The value is unreliable, including large noise.

  When the denominator formula (48) is 0, for example, at least one of the cubic functions Cubic (I′-x) or Cubic (J′-y) is the entire summation range of the formula (48) In this case, the value becomes 0 over the entire period. When the cubic function Cubic (I′−x) or Cubic (J′−y) is 0, I′−x = ± 1 or J′−y = from the characteristics of the cubic function shown in FIG. This is a case where ± 1, that is, x = I ′ ± 1 or y = J ′ ± 1.

  Therefore, for a certain position (I ′, J ′), the position (ig−0.5, jg−0.5) is converted into the conversion parameters (ak ′, bk ′, ck ′, dk ′, Tkx ′, Tky ′). ) All (k, ig, jg) pixels appearing in the range of I′−2 ≦ x <I ′ + 2 and J′−2 ≦ y <J ′ + 2 when converted into the reference coordinate system. When the position (x, y) of the value Gobs (k, ig, jg) has a relationship of x = I ′ ± 1 or y = J ′ ± 1 with the position (I ′, J ′), the expression (46) The true green light quantity Lg (I ′, J ′) obtained in (1) is an unstable (undefined) and unreliable value because its denominator is 0 (or almost 0). However, all the positions (x, y) of the pixel values Gobs (k, ig, jg) appearing in the range of I′−2 ≦ x <I ′ + 2, J′−2 ≦ y <J ′ + 2 I ′, J ′) and x = I ′ ± 1 or y = J ′ ± 1 is a very special and exceptional state. Hereinafter, this state is referred to as an exceptional state. Call

  Here, the pixel position (k, i, j) of the k-th captured image is converted on the reference coordinate system with the conversion parameters (ak ′, bk ′, ck ′, dk ′, Tkx ′, Tky ′). The position (x, y) thus obtained is hereinafter also referred to as a conversion position (x, y) (correction position) as appropriate.

  FIG. 44 is a diagram illustrating a case where a certain position (I ′, J ′) is in an exceptional state.

  In FIG. 44, a point (conversion position) G11 obtained by converting the position (ig-0.5, jg-0.5) with the conversion parameters (ak ′, bk ′, ck ′, dk ′, Tkx ′, Tky ′). 'And point G15' appear at positions where x = I'-1 and points G12 'and G16' appear at positions where x = I '+ 1.

  Further, points G13 ′ and G14 ′ obtained by converting the position (ig−0.5, jg−0.5) with the conversion parameters (ak ′, bk ′, ck ′, dk ′, Tkx ′, Tky ′) are obtained. , Y = J′−1, the point G17 ′, the point G18 ′, and the point G19 ′ respectively appear at positions that satisfy the relationship y = J ′ + 1.

  In the state shown in FIG. 44, a set of (k, ig, jg) converted into a range of I′−2 ≦ x <I ′ + 2, J′−2 ≦ y <J ′ + 2 on the reference coordinate system. All conversion positions (x, y) have a relationship of position (I ′, J ′) and x = I ′ ± 1 or y = J ′ ± 1. In this state, the region (position X (I ′, J ′) centered on the position (I ′, J ′) and the area (position X (I ′, J ′) centered on the position (I ′, J ′)) is 2 × There is no G signal data (pixel value Gobs (k, ig, jg)) in the × 2 square area.

  In such a state (exceptional state), when the true green light quantity Lg (I ′, J ′) at the position (I ′, J ′) is obtained by the equation (46), as described above, the reliability A low (unstable) true green light quantity Lg (I ′, J ′) is obtained.

  Therefore, when the pixel at the position (I ′, J ′) is in such an exceptional state, the arithmetic circuit 24 performs exception processing.

  That is, for example, assuming that the position (I ′, J ′) is the position of interest, the true green light amount Lg at the pixel of the output image at the position of interest (I ′, J ′) (hereinafter referred to as the pixel of interest as appropriate). Assuming that (I ′, J ′) is estimated, when the target pixel (the target position (I ′, J ′)) is in an exceptional state, the arithmetic circuit 24 selects the target position (I of the target pixel in the exceptional state. ', J') is the true green light quantity Lg (I ', J'), and the pixel of the kth picked-up image whose conversion position (x, y) is in the vicinity of the target position (I ', J') In addition to the pixel value Gobs (k, ig, jg), the pixel value Gobs of the pixel of the k-th captured image whose conversion position (x, y) is in the vicinity of the pixel position of the output image around the target pixel It is also determined using (k, ig, jg). Here, as a pixel around the target pixel at the target position (I ′, J ′) (hereinafter, appropriately referred to as a peripheral pixel), for example, a position (I′-1, J ′) as shown in FIG. , (I ′ + 1, J ′), (I ′, J′−1), (I ′, J ′ + 1) pixels can be employed.

  As described with reference to FIG. 3, the image sensor 4 of the digital camera 1 of the present embodiment has a Bayer array. In the Bayer array, the pixels that receive the green component are arranged every other pixel in both the X direction and the Y direction (see FIG. 3). In the eighth embodiment, it is assumed that the image sensor 4 has a W × H number of pixels of W pixels in the horizontal direction (X direction) and H pixels in the vertical direction (Y direction).

  If the observed value (or pixel value) Gobs (k, ig, jg) of the G signal does not exist in the vicinity of the target position (I ′, J ′), the target position (I ′, J ′) It can be said that the target pixel is not a pixel that receives the green component among the pixels in the Bayer array.

  For example, as shown in FIG. 46, the target pixel at the target position (I ′, J ′) is a pixel that receives a blue component among the pixels in the Bayer array, and is a pixel B12 surrounded by a circle. In some cases, a pixel that receives the green component is present on any of the top, bottom, left, and right of the pixel B12. Similarly, for each pixel that receives either the red or blue component of the image sensor 4, there is a pixel that receives the green component on either the top, bottom, left, or right of the pixel.

  Therefore, when the observed value Gobs (k, ig, jg) of the G signal does not exist in the vicinity of the target position (I ′, J ′) (the conversion position (x, y) is the target position (I ′, J ′). ) Of the captured image in the vicinity of the captured image)), and the neighboring pixels (peripheral pixels) adjacent to the top, bottom, left, or right of the target pixel of the target position (I ', J') In the vicinity of any of the positions (I ′, J′−1), (I ′, J ′ + 1), (I′−1, J ′), or (I ′ + 1, J ′), G There are signal observations Gobs (k, ig, jg). Then, in any peripheral pixel at the position (I′-1, J ′), (I ′ + 1, J ′), (I ′, J′−1), or (I ′, J ′ + 1) Since the observed value Gobs (k, ig, jg) of the G signal exists, no exceptional state has occurred. That is, for any peripheral pixel at position (I'-1, J '), (I' + 1, J '), (I', J'-1), or (I ', J' + 1). The reliable true green light quantity Lg (I ′, J ′) can be obtained from Equation (46).

  From the above, when the target pixel is in an exceptional state, the arithmetic circuit 24 converts the true green light amount Lg (I ′, J ′) at the target position (I ′, J ′) into the conversion position (x, y). In addition to the pixel value Gobs (k, ig, jg) of the pixel in the vicinity of the target position (I ′, J ′), the positions (I′−1, J ′), ( I '+ 1, J'), (I ', J'-1), or exception processing to be obtained using pixel values Gobs (k, ig, jg) of pixels in the vicinity of (I', J '+ 1) I do.

  Specifically, the arithmetic circuit 24 obtains the true green light quantity Lg (I ′, J ′) at the target position (I ′, J ′) as an exception process by the following equation (56).

・・・（５６）

... (56)

  According to Expression (56), the target position (I ′, J ′) of the target pixel and the positions (I′−1, J ′), (I ′ + 1, J ′) of the peripheral pixels around the target pixel. , (I ′, J′−1), and (I ′, J ′ + 1), and the sum of the numerators of Equation (46) calculated during normal processing at each of the five points, ie, Equation (47) The sum total is calculated as the target position (I ', J') of the target pixel and the positions (I'-1, J '), (I' + 1, J '), (I', Dividing by the sum of the denominator of Expression (46) calculated during normal processing at each of the five points J′−1) and (I ′, J ′ + 1), that is, the sum of Expression (48) Thus, the true green light amount Lg (I ′, J ′) is obtained.

  Note that, when the target pixel is in an exceptional state, the true green light amount Lg (I ′, J ′) at the target position (I ′, J ′) is limited to the above equation (56). For example, the following equation (57) may be adopted.

・・・（５７）

... (57)

  In Expression (57), the positions (I′−1, J ′), (I ′ + 1, J ′), (I ′, J′−1), and (I ′, J) of the peripheral pixels around the target pixel. '+1) for each of the four points, the true green light quantity Lg (I'-1, J'), Lg (I '+ 1, J'), Lg (I ', J'-1) obtained by the equation (46) ) And Lg (I ′, J ′ + 1) is an equation that estimates the true green light amount Lg (I ′, J ′) at the target position (I ′, J ′). .

  Next, the weight addition formula for the red light amount in the equation (50) will be considered.

  The true red light quantity Lr (I ′, J ′) obtained by the red light quantity weight addition formula of Expression (50) is also true green light quantity obtained by the green light quantity weight addition expression of Expression (46). Similar to the light amount Lg (I ′, J ′), the value may become unstable, that is, the target pixel may be in an exceptional state. Specifically, the position (ir−0.5, jr−0.5) is converted into the conversion parameters (ak ′, bk ′, ck ′, dk) with respect to the target position (I ′, J ′) of a certain target pixel. (', Tkx', Tky ') all (k) appearing in the range of I'-2≤x <I' + 2, J'-2≤y <J '+ 2 when converted to the reference coordinate system. , Ir, jr), the conversion position (x, y) of the pixel value Robs (k, ir, jr) is the position (I ′, J ′) and x = I ′ ± 1 or In some cases, y = J ′ ± 1. In such a state (exceptional state), a region (position (I ', J') centered around (I '± 1, J' ± 1) centered on the position (I ', J'). No square data (pixel value Robs (k, ir, jr)) exists in a square area of horizontal x vertical 2 x 2.

  In this case, the arithmetic circuit 24 performs the following exception processing.

  That is, the arithmetic circuit 24 calculates the true red light amount Lr (I ′, J ′) at the target position (I ′, J ′) of the target pixel in the exceptional state, and the conversion position (x, y) as the target position (I In addition to the pixel value Robs (k, ir, jr) of the pixel of the k-th captured image in the vicinity of ', J'), the conversion position (x, y) is the pixel of the output image around the pixel of interest. The pixel value Robs (k, ir, jr) of the pixel of the k-th captured image in the vicinity of the position is also obtained. Here, as peripheral pixels around the target pixel at the target position (I ′, J ′), for example, positions (I′−1, J′−1), (I ′, J) as shown in FIG. '-1), (I' + 1, J'-1), (I'-1, J '), (I' + 1, J '), (I'-1, J' + 1), (I ', J '+ 1) and (I' + 1, J '+ 1) pixels can be employed.

  When there is no observed value (or pixel value) Robs (k, ir, jr) of the R signal in the vicinity of the target position (I ′, J ′), the target position (I ′, J ′) is focused. It can be said that the pixel is not a pixel that receives the red component among the pixels in the Bayer array.

  For example, as shown in FIG. 48, the target pixel at the target position (I ′, J ′) is a pixel that receives the green component among the pixels in the Bayer array, and is a circled pixel G11. In some cases, a pixel that receives a red component exists above or below the pixel G11.

  Also, for example, as shown in FIG. 48, the target pixel at the target position (I ′, J ′) is a pixel that receives a green component among the pixels in the Bayer array, and is surrounded by a circle. In the case of G22, there is a pixel that receives the red component on either the left or right side of the pixel G22.

  Further, for example, as shown in FIG. 48, the target pixel at the target position (I ′, J ′) is a pixel that receives a blue component among the pixels in the Bayer array, and is surrounded by a circle. In the case of B14, a pixel that receives the red component exists at any of the upper right, lower right, upper left, and lower left of the pixel B14.

  Similarly, for each pixel of the image sensor 4 that is not a pixel that receives the red component, any one of the top, bottom, left, right, upper right, lower right, upper left, and lower left of the pixel. Includes a pixel that receives a red component.

  Therefore, when the observed value Robs (k, ir, jr) of the R signal does not exist in the vicinity of the target position (I ′, J ′) (the conversion position (x, y) is the target position (I ′, J ′). ), When there is no red pixel in the captured image), diagonally upper left, upper, diagonally upper right, left, right, diagonally lower left of the pixel of interest at the target position (I ', J') , Lower or diagonally lower right adjacent pixels (peripheral pixels) positions (I′−1, J′−1), (I ′, J′−1), (I ′ + 1, J′−1), (I'-1, J '), (I' + 1, J '), (I'-1, J' + 1), (I ', J' + 1), or (I '+ 1, J' + 1) There is an observed value Robs (k, ir, jr) of the R signal in the vicinity of any of them. And position (I'-1, J'-1), (I ', J'-1), (I' + 1, J'-1), (I'-1, J '), (I' + 1) , J ′), (I′−1, J ′ + 1), (I ′, J ′ + 1), or (I ′ + 1, J ′ + 1), the observation of the R signal in any peripheral pixel Since the value Robs (k, ir, jr) exists, no exceptional condition has occurred. That is, positions (I'-1, J'-1), (I ', J'-1), (I' + 1, J'-1), (I'-1, J '), (I' + 1) , J ′), (I′−1, J ′ + 1), (I ′, J ′ + 1), or (I ′ + 1, J ′ + 1) for any peripheral pixel, Expression (50) Thus, a reliable true red light amount Lr (I ′, J ′) can be obtained.

  From the above, the arithmetic circuit 24 converts the true red light amount Lr (I ′, J ′) at the target position (I ′, J ′) to the conversion position (x, y) when the target pixel is in an exceptional state. In addition to the pixel value Robs (k, ir, jr) of the pixel in the vicinity of the target position (I ′, J ′), the positions (I′−1, J′−1) of the peripheral pixels around the target pixel , (I ', J'-1), (I' + 1, J'-1), (I'-1, J '), (I' + 1, J '), (I'-1, J' + 1) ), (I ′, J ′ + 1), or pixel values Robs (k, ir, jr) of pixels in the vicinity of (I ′ + 1, J ′ + 1).

  Specifically, the arithmetic circuit 24 obtains the true red light amount Lr (I ′, J ′) at the target position (I ′, J ′) as an exception process by the following equation (58).

・・・（５８）

... (58)

  According to Expression (58), the target position (I ′, J ′) of the target pixel and the positions (I′−1, J′−1), (I ′, J ′) of peripheral pixels around the target pixel. -1), (I '+ 1, J'-1), (I'-1, J'), (I '+ 1, J'), (I'-1, J '+ 1), (I', J '+1) and (I' + 1, J '+ 1) at the nine points, respectively, the sum of the numerators of the formula (50) calculated during normal processing, that is, the sum of the formula (52) Attention position (I ', J') and positions (I'-1, J'-1), (I ', J'-1), (I' + 1, J ') of peripheral pixels around the attention pixel -1), (I'-1, J '), (I' + 1, J '), (I'-1, J' + 1), (I ', J' + 1), and (I '+ 1, J By dividing by the sum of the denominator of the equation (50) calculated at the time of normal processing at each of the nine points “+1”, that is, the sum of the equation (53), the true red light amount Lr (I ′) J ') is required.

  Similar to the equation (57) in the case of the G signal, instead of the equation (58), the positions (I′−1, J′−1), (I ′, J ′) of the peripheral pixels around the pixel of interest. -1), (I '+ 1, J'-1), (I'-1, J'), (I '+ 1, J'), (I'-1, J '+ 1), (I', J '+1) and 8 points (I' + 1, J '+ 1), the true red light amounts Lr (I'-1, J'-1), Lr (I', J'-1), Lr ( I '+ 1, J'-1), Lr (I'-1, J'), Lr (I '+ 1, J'), Lr (I'-1, J '+ 1), Lr (I', J ' +1) and the average value of Lr (I ′ + 1, J ′ + 1), the true red light amount Lr (I ′, J ′) at the target position (I ′, J ′) may be obtained.

  Further, the true blue light quantity Lb (I ′, J ′) obtained by the blue light quantity weight addition formula of Expression (51) is also true by the green light quantity weight addition formula of Expression (46). Similar to the green light quantity Lg (I ′, J ′) and the true red light quantity Lr (I ′, J ′) obtained by the weighted addition formula of the red light quantity in Expression (50), the value is unstable. In other words, the target pixel at the target position (I ′, J ′) may be in an exceptional state.

  In this case, the arithmetic circuit 24 performs the following exception processing.

  That is, in the Bayer array, the pixels that receive the blue component are arranged in the same positional relationship as the pixels that receive the red component. Therefore, the arithmetic circuit 24 calculates the true blue light quantity Lb (I ′, J ′) at the target position (I ′, J ′) of the target pixel in the exceptional state by the same formula (59) as the formula (58). Perform the exception handling required in.

・・・（５９）

... (59)

  According to Expression (59), the target position (I ′, J ′) of the target pixel and the positions (I′−1, J′−1) and (I ′, J′−1) of the peripheral pixels of the target pixel. ), (I '+ 1, J'-1), (I'-1, J'), (I '+ 1, J'), (I'-1, J '+ 1), (I', J '+ 1) ) And (I ′ + 1, J ′ + 1), the sum of the numerators of Equation (51) calculated during normal processing, that is, the sum of Equation (54), is used as the target position of the target pixel. (I ', J') and positions (I'-1, J'-1), (I ', J'-1), (I' + 1, J'-1) of peripheral pixels of the target pixel, (I'-1, J '), (I' + 1, J '), (I'-1, J' + 1), (I ', J' + 1), and (I '+ 1, J' + 1) The true blue light quantity Lb (I ′, J ′) is obtained by dividing by the sum of the denominator of the equation (51) calculated in the normal processing at each of the nine points, that is, the sum of the equation (55). Demand It is.

  Similar to the equation (57) in the case of the G signal, instead of the equation (59), the positions (I′−1, J′−1), (I ′, J ′) of the peripheral pixels around the pixel of interest. -1), (I '+ 1, J'-1), (I'-1, J'), (I '+ 1, J'), (I'-1, J '+ 1), (I', J '+1) and 8 points (I' + 1, J '+ 1), the true blue light amounts Lb (I'-1, J'-1), Lb (I', J'-1), Lb ( I '+ 1, J'-1), Lb (I'-1, J'), Lb (I '+ 1, J'), Lb (I'-1, J '+ 1), Lb (I', J ' +1) and the average value of Lb (I ′ + 1, J ′ + 1), the true blue light quantity Lb (I ′, J ′) at the target position (I ′, J ′) may be obtained.

  Next, an eighth embodiment of the image estimation process in step S4 of FIG. 2 will be described with reference to the flowcharts of FIGS.

  First, in step S201, the arithmetic circuit 24 sets 1 to a variable J ′ representing a position in the Y direction among positions (I ′, J ′) where a pixel on the reference coordinate system is located, and then proceeds to step S202. move on.

  In step S202, the arithmetic circuit 24 sets 1 to the variable I ′ representing the position in the X direction among the positions (I ′, J ′) at which the pixels are on the reference coordinate system, and proceeds to step S203. Here, the variables I ′ and J ′ are variables representing the positions of the pixels of the output image in the X direction and the Y direction, respectively.

  In step S203, the arithmetic circuit 24 sets the position (I ′, J ′) as the position of interest, and the center of the pixel that receives the green component of the k-th image with respect to the position of interest (I ′, J ′). A conversion position (x, y on the reference coordinate system obtained by converting the position (ig−0.5, jg−0.5) with the conversion parameters (ak ′, bk ′, ck ′, dk ′, Tkx ′, Tky ′). ) Obtain all sets (k, ig, jg) satisfying I′−2 ≦ x <I ′ + 2 and J′−2 ≦ y <J ′ + 2 for the first to Nth images, and the process proceeds to step S204. move on.

  In step S204, the arithmetic circuit 24 uses the set of all (k, ig, jg) obtained in step S203, the denominator of the weight addition formula for the green light quantity expressed by the equation (48), and the equation Each numerator of the weight addition formula of the green light quantity represented by (47) is calculated. Further, the arithmetic circuit 24 stores each result of the arithmetic operation in a memory (not shown), and proceeds to step S205.

  In step S205, the arithmetic circuit 24 determines the center position (ir−0.5, jr−0.5) of the pixel that receives the red component of the k-th image with respect to the target position (I ′, J ′). Is converted with the conversion parameters (ak ′, bk ′, ck ′, dk ′, Tkx ′, Tky ′), the conversion position (x, y) on the reference coordinate system is I′−2 ≦ x <I ′ + 2, All sets of (k, ir, jr) that satisfy J′−2 ≦ y <J ′ + 2 are obtained for the first to Nth images, and the process proceeds to step S206.

  In step S206, the arithmetic circuit 24 uses the set of all (k, ir, jr) obtained in step S205, the denominator of the red light quantity weight addition formula represented by formula (53), and the formula Each numerator of the red light quantity weight addition formula represented by (52) is calculated. Further, the arithmetic circuit 24 stores each result of the arithmetic operation in the memory, and proceeds to step S207.

  In step S207, the arithmetic circuit 24 determines the center position (ib−0.5, jb−0.5) of the pixel that receives the blue component of the k-th image with respect to the target position (I ′, J ′). Is converted with the conversion parameters (ak ′, bk ′, ck ′, dk ′, Tkx ′, Tky ′), the conversion position (x, y) on the reference coordinate system is I′−2 ≦ x <I ′ + 2, All sets (k, ib, jb) satisfying J′−2 ≦ y <J ′ + 2 are obtained for the first to Nth images, and the process proceeds to step S208.

  In step S208, the arithmetic circuit 24 uses the set of all (k, ib, jb) obtained in step S207, the denominator of the blue light quantity weight addition formula represented by the formula (55), and the formula Each numerator of the blue light quantity weight addition formula represented by (54) is calculated. Further, the arithmetic circuit 24 stores each result of the arithmetic operation in the memory, and proceeds to step S209.

  In step S209, the arithmetic circuit 24 determines whether or not the variable I ′ is equal to the number of pixels W in the X direction. If it is determined in step S209 that the variable I ′ is not equal to the number of pixels W, that is, the processing in steps S203 to S208 is performed for all the pixels in the X direction in the current Y = J ′ pixel column. If not, the process proceeds to step S210, the variable I ′ is incremented by 1, and the process returns to step S203.

  On the other hand, if it is determined in step S209 that the variable I ′ is equal to the number of pixels W, that is, the processes in steps S203 to S208 are performed for all the pixels in the X direction in the current Y = J ′ pixel column. If yes, the process proceeds to step S211.

  In step S211, the arithmetic circuit 24 determines whether or not the variable J ′ is equal to the number of pixels H in the Y direction. If it is determined in step S211 that the variable J ′ is not equal to the number of pixels H, that is, if the processing in steps S203 to S208 has not been performed for all columns in the Y direction of the image sensor 4, the process proceeds to step S212. Then, the variable J ′ is incremented by 1, and the process returns to step S202.

  On the other hand, if it is determined in step S211 that the variable J ′ is equal to the number of pixels H, that is, if the processing in steps S203 to S208 has been performed for all columns in the Y direction of the image sensor 4, FIG. Proceed to step S213. Here, the number of pixels W and H are also the numbers of pixels in the X direction and the Y direction of the output image, respectively.

  In step S213, the arithmetic circuit 24 sets 1 to the variable J ′ as in step S201, and proceeds to step S214.

  In step S214, the arithmetic circuit 24 sets 1 to the variable I ′ in the same manner as in step S202, and proceeds to step S215.

  In step S215, the arithmetic circuit 24 sets the position (I ′, J ′) as the position of interest (I ′, J ′), and the true green light amount Lg (with respect to the position of interest (I ′, J ′)). I ', J') is calculated, and the process proceeds to step S216. In other words, in step S215, as will be described later, normal processing (first calculation processing) using the weight addition formula for the green light quantity in Expression (46), or exception processing (second processing) using Expression (56). Thus, the true green light amount Lg (I ′, J ′) for the target position (I ′, J ′) is obtained.

  In step S216, the arithmetic circuit 24 sets the position (I ′, J ′) as the position of interest (I ′, J ′), and the true red light amount Lr (with respect to the position of interest (I ′, J ′)). I ', J') is calculated, and the process proceeds to step S217. That is, in step S216, as will be described later, normal processing (first calculation processing) using the red light amount weight addition formula of Formula (50) or exception processing (second processing) using Formula (58) is performed. Thus, the true red light quantity Lr (I ′, J ′) for the target position (I ′, J ′) is obtained.

  In step S217, the arithmetic circuit 24 sets the position (I ′, J ′) as the position of interest (I ′, J ′), and the true blue light quantity Lb (with respect to the position of interest (I ′, J ′)). I ′, J ′) is calculated, and the process proceeds to step S218. That is, in step S217, as will be described later, normal processing (first calculation processing) using the blue light amount weight addition formula of Formula (51) or exception processing (second processing) using Formula (59) is used. Thus, the true blue light quantity Lg (I ′, J ′) for the target position (I ′, J ′) is obtained.

  In step S218, the arithmetic circuit 24 determines whether or not the variable I ′ is equal to the number of pixels W in the X direction. If it is determined in step S218 that the variable I ′ is not equal to the number of pixels W, that is, the processing in steps S215 to S217 is performed for all the pixels in the X direction in the current Y = J ′ pixel column. If not, the process proceeds to step S219, increments the variable I ′ by 1, and returns to step S215.

  On the other hand, if it is determined in step S218 that the variable I ′ is equal to the number of pixels W, that is, the processes in steps S215 to S217 are performed for all the pixels in the X direction in the current Y = J ′ pixel column. If yes, the process proceeds to step S220.

  In step S220, the arithmetic circuit 24 determines whether or not the variable J ′ is equal to the number of pixels H in the Y direction. If it is determined in step S220 that the variable J ′ is not equal to the number of pixels H, that is, if the processing in steps S215 to S217 has not been performed for all columns in the Y direction of the image sensor 4, the process proceeds to step S221. Then, the variable J ′ is incremented by 1, and the process returns to step S214.

  On the other hand, if it is determined in step S220 that the variable J ′ is equal to the number of pixels H, that is, if the processes in steps S215 to S217 have been performed for all columns in the Y direction of the image sensor 4, the process proceeds to step S222. move on.

  In step S222, the arithmetic circuit 24 calculates the true green light amount Lg (I ′, J ′), the true red light amount Lr (I ′, J ′), and the true light obtained in steps S215, S216, and S217, respectively. An output image (signal) is estimated from the blue light quantity Lb (I ′, J ′), and supplied to the D / A converter 9 or the codec 12, and the process returns. For example, in the “i-th and j-th pixels”, the arithmetic circuit 24 calculates the true green light amount Lg (= Lg (I ′, J ′)) obtained in step S215 as the green value (G signal). , The true red light quantity Lr (i−0.5, j−0.5) (= Lr (I ′, J ′)) obtained in step S216 as the red value (R signal), and the blue value ( From the true blue light quantity Lb (i−0.5, j−0.5) (= Lb (I ′, J ′)) obtained in step S217 as the B signal), the “i th and j th pixels ”Is estimated. Then, the arithmetic circuit 24 performs the estimation on all the pixels of the output image having the position (I ′, J ′) (= (i−0.5, j−0.5)) as the center position, and outputs the result. Estimate the image.

  Next, with reference to the flowchart of FIG. 51, the calculation process for obtaining the true green light amount Lg (I ′, J ′) at the target position (I ′, J ′) in step S215 of FIG. 50 will be described.

  First, in step S251, the arithmetic circuit 24 calculates the denominator of the weight addition formula of the green light quantity of the equation (46) calculated in step S204 of FIG. 49 for the target position (I ′, J ′). It is determined whether or not the absolute value, that is, the absolute value of Expression (48) is equal to or greater than a predetermined threshold value. This predetermined threshold value is a value for determining whether or not to apply exception processing by regarding the absolute value of the equation (48) as 0, and is set in advance in the arithmetic circuit 24, for example. However, the predetermined threshold can also be set according to the user's operation.

  If it is determined in step S251 that the absolute value of the equation (48) for the target position (I ′, J ′) is equal to or greater than a predetermined threshold, that is, the equation (48) for the target position (I ′, J ′). If the absolute value of () is not small enough to be regarded as 0, the process proceeds to step S252, and the arithmetic circuit 24 selects and performs a normal process for calculating the weight addition formula for the green light quantity in the equation (46). That is, the arithmetic circuit 24 calculates the value of the numerator of the weight addition formula of the green light quantity of the equation (46) calculated in step S204, that is, the value of the equation (47), in the equation ( A calculation is performed to divide by the denominator value of the weight addition formula of the green light quantity of 46), that is, the value of formula (48). Thus, in step S252, the true green light amount Lg (I ′, J ′) at the target position (I ′, J ′) is obtained.

  On the other hand, when it is determined in step S251 that the absolute value of Expression (48) with respect to the target position (I ′, J ′) is less than the predetermined threshold, that is, the absolute value of Expression (48) is 0, or When it is close to 0, the process proceeds to step S253, and the arithmetic circuit 24 selects and performs exception processing. That is, the arithmetic circuit 24 calculates the true green light amount Lg (I ′, J ′) at the target position (I ′, J ′) by calculating the equation (56).

  Next, with reference to the flowchart of FIG. 52, the calculation processing for obtaining the true red light amount Lr (I ′, J ′) at the target position (I ′, J ′) in step S216 of FIG. 50 will be described.

  First, in step S271, the arithmetic circuit 24 calculates the denominator of the red light quantity weight addition formula of equation (50) calculated in step S206 of FIG. 49 for the target position (I ′, J ′). It is determined whether the absolute value, that is, the absolute value of Expression (53) is equal to or greater than a predetermined threshold value. This predetermined threshold value is a value for determining whether or not to apply exception processing by regarding the absolute value of the equation (53) as 0, and is set in advance in the arithmetic circuit 24, for example. However, the predetermined threshold can also be set according to the user's operation. Further, this threshold value may be the same as or different from the threshold value in step S251 of FIG.

  If it is determined in step S271 that the absolute value of the equation (53) for the target position (I ′, J ′) is equal to or greater than a predetermined threshold, that is, the equation (53) for the target position (I ′, J ′). If the absolute value of () is not small enough to be regarded as 0, the process proceeds to step S272, and the arithmetic circuit 24 selects and performs a normal process for calculating the red light quantity weight addition formula of formula (50). In other words, the arithmetic circuit 24 calculates the value of the numerator of the red light quantity weight addition formula of Formula (50) calculated in Step S206, that is, the value of Formula (52), in Formula ( 50) The operation of dividing by the denominator value of the red light quantity weight addition formula, that is, the value of formula (53) is performed. Thus, in step S272, the true red light amount Lr (I ′, J ′) at the target position (I ′, J ′) is obtained.

  On the other hand, when it is determined in step S271 that the absolute value of the equation (53) with respect to the target position (I ′, J ′) is less than the predetermined threshold, that is, the absolute value of the equation (53) is 0, or When it is close to 0, the process proceeds to step S273, and the arithmetic circuit 24 selects and performs exception processing. That is, the arithmetic circuit 24 calculates the true red light amount Lr (I ′, J ′) at the target position (I ′, J ′) by calculating the equation (58).

  Next, with reference to the flowchart of FIG. 53, the calculation process for obtaining the true blue light quantity Lb (I ′, J ′) at the target position (I ′, J ′) in step S217 of FIG. 50 will be described.

  First, in step S291, the arithmetic circuit 24 calculates the denominator of the blue light quantity weight addition formula of equation (51) calculated in step S208 of FIG. 49 for the position of interest (I ′, J ′). It is determined whether the absolute value, that is, the absolute value of Expression (55) is equal to or greater than a predetermined threshold value. This predetermined threshold value is a value for determining whether or not to apply exception processing, assuming that the absolute value of Expression (55) is 0, and is set in advance in the arithmetic circuit 24, for example. However, the predetermined threshold can also be set according to the user's operation. Further, this threshold value may be the same as or different from the threshold values in step S251 in FIG. 51 and step S271 in FIG.

  If it is determined in step S291 that the absolute value of the equation (55) for the target position (I ′, J ′) is equal to or greater than a predetermined threshold, that is, the equation (55) for the target position (I ′, J ′). If the absolute value of () is not small enough to be regarded as 0, the process proceeds to step S292, and the arithmetic circuit 24 selects and performs a normal process for calculating the weight addition formula for the blue light quantity in Expression (51). That is, the arithmetic circuit 24 calculates the value of the numerator of the blue light amount weight addition formula of Formula (51) calculated in Step S208, that is, the value of Formula (54), in Formula ( 51) The operation of dividing by the denominator value of the blue light quantity weight addition formula, that is, the value of formula (55) is performed. Thereby, in step S292, the true blue light quantity Lb (I ′, J ′) at the target position (I ′, J ′) is obtained.

  On the other hand, when it is determined in step S291 that the absolute value of Expression (55) for the target position (I ′, J ′) is less than the predetermined threshold, that is, the absolute value of Expression (55) is 0, or When it is close to 0, the process proceeds to step S293, and the arithmetic circuit 24 selects and performs exception processing. That is, the arithmetic circuit 24 calculates the true blue light quantity Lb (I ′, J ′) at the target position (I ′, J ′) by calculating Expression (59).

  As described above, in the eighth embodiment, the weight according to the distance between the target position (I ′, J ′) and the converted position (x, y) near the target position (I ′, J ′). By performing weighted addition using a cubic function having the characteristics of a low-pass filter, the true green light amount Lg (I ′, J ′), the true red light amount Lr (I ′, J ′), and the true Since the blue light quantity Lb (I ′, J ′) is obtained, a clear image faithful to the original light can be obtained.

  In the eighth embodiment, the position (ig−0.5, jg−0.5) is represented on the reference coordinate system by the conversion parameters (ak ′, bk ′, ck ′, dk ′, Tkx ′, Tky ′). , All (k) appearing in the range of I′−2 ≦ x <I ′ + 2 and J′−2 ≦ y <J ′ + 2 with respect to the target position (I ′, J ′). , Ig, jg) pixel value Gobs (k, ig, jg), that is, the pixel value Gobs (k, ig, jg) in which the conversion position (x, y) is in the vicinity of the target position (I ′, J ′). The true green light quantity Lg (I ′, J ′) is obtained by the weighted addition expression of the green light quantity in Expression (46) (normal processing).

  However, when the absolute value of the denominator equation (48) of the weight addition equation of the green light quantity in the equation (46) of the target position (I ′, J ′) is considered to be 0 when it is less than the predetermined threshold, that is, the equation When the value obtained by the weight addition formula of the green light quantity in (46) is unstable, the pixel value Gobs (k) where the conversion position (x, y) is in the vicinity of the attention position (I ′, J ′). , Ig, jg) and weighted addition using pixel values Gobs (k, ig, jg) in the vicinity of the positions of the peripheral pixels around the target pixel at the target position (I ′, J ′). From (56), the true green light quantity Lg (I ′, J ′) is obtained (exception processing).

  The true red light amount Lr (I ′, J ′) and the true blue light amount Lb (I ′, J ′) are obtained in the same manner.

  Therefore, it is possible to obtain a good output image with no noticeable noise.

  The normal process and the exception process can be viewed as follows.

  That is, for example, when attention is paid to green, in the case described above, in the normal processing, the pixel of the captured image in which the conversion position (x, y) is in the vicinity of the position of the target pixel (target position) (I ′, J ′). The weighted addition using the pixel values Gobs (k, ig, jg) is performed. On the other hand, in the exception processing, the pixel value Gobs (k, ig, jg) of the pixel of the captured image whose conversion position (x, y) is in the vicinity of the position (I ′, J ′) of the target pixel and the conversion position (x , Y) performs weighted addition using pixel values Gobs (k, ig, jg) of pixels of the captured image in the vicinity of the positions of the peripheral pixels around the target pixel.

  Accordingly, in the exception processing, the pixel of the captured image whose conversion position (x, y) is in the vicinity of the position (I ′, J ′) of the target pixel and the pixel of the captured image in the vicinity of the position of the peripheral pixel Weighted addition is also performed using the value Gobs (k, ig, jg).

  From the above, the captured image observed in the region of I′−2 ≦ x <I ′ + 2, J′−2 ≦ y <J ′ + 2 as the vicinity of the target position (I ′, J ′) in the normal processing. The true green light quantity Lg (I ′, J ′) at the target position (I ′, J ′) is obtained by weighted addition using the pixel values Gobs (k, ig, jg) of As the vicinity of the target position (I ′, J ′), the region is observed in a region of I′−3 ≦ x <I ′ + 3, J′−3 ≦ y <J ′ + 3, which is wider than the region near the normal process. It can be said that the true green light quantity Lg (I ′, J ′) at the target position (I ′, J ′) is obtained by weighted addition using the pixel value Gobs (k, ig, jg).

  Furthermore, in other words, when the true green light amount Lg (I ′, J ′) at the target position (I ′, J ′) is obtained, I is preliminarily set as the vicinity of the target position (I ′, J ′). The region of “−3 ≦ x <I ′ + 3, J′−3 ≦ y <J ′ + 3 is set, and in the normal processing, I′−2 ≦ x <I ′ + 2, among the neighboring regions. According to the weight addition formula of the green light quantity of the formula (46) in which the weight for the pixel value Gobs (k, ig, jg) observed in the region other than J′−2 ≦ y <J ′ + 2 is 0, the target position ( It can be said that the true green light quantity Lg (I ′, J ′) of I ′, J ′) is required. On the other hand, in the exception processing, pixel values Gobs (k, ig, jg) observed in regions other than I′−2 ≦ x <I ′ + 2, J′−2 ≦ y <J ′ + 2, that is, peripheral pixels As a weight for a pixel value Gobs (k, ig, jg) observed in the vicinity, a value given by a cubic function Cubic (z) having the origin of the position of the surrounding pixel instead of a constant zero weight is used. It can be said that the true green light quantity Lg (I ′, J ′) at the target position (I ′, J ′) is obtained from the equation (56).

  That is, it can be said that the true green light amount Lg (I ′, J ′) is obtained by weighted addition using different weights (first and second weights) in the normal process and the exception process.

  In the eighth embodiment, it corresponds to the distance z between the position of interest (I ′, J ′) and the position (x, y) where the pixel value Gobs (k, ig, jg) in the vicinity is observed. The cubic function Cubic (z) of the equation (39) is adopted as a function representing the weight, but any other function having the characteristics of a low-pass filter such as sin (z) / z is adopted as the weight. can do.

  In the above-described embodiment, the first captured image is used as the reference image and each of the second to Nth captured images is used as the target image. Any of the 2nd to Nth captured images may be used as the reference image.

  That is, FIG. 54 shows N captured images captured by N consecutive imaging (high-speed imaging). FIG. 54 shows an example where N = 8.

The first to eighth captured images 401 _{1 to} 401 ₈ (kth image 401 _k ) are images captured in time series, and are images shifted due to camera shake in the upper right direction in the figure as time passes. It has become.

FIG. 55 shows _eight captured images 401 _{1 to} 4018 similar to FIG.

In the signal processing circuit 7, when the output image is estimated using the _first image of the captured images 401 _{1 to} 40 ₈ as the reference image and the second to eighth images as target images, the output image is obtained as an image on the first image 401 ₁ of the coordinate system is the reference image. Here, in FIG. 55, the first image 401 ₁ , that is, the output image is indicated by a thick line.

In the area of the output image indicated by the bold line in FIG. 55, the light amount (pixel value) is estimated using all the data (pixel values) of the _eight captured images 401 _{1 to} 4018 of the _{first to eighth} images. The area is an area 411 in the upper right part of the output image indicated by a dotted line. Pixel value of the region 411, since it is estimated using data of 1 to 8 th eight all captured images 401 ₁ to 401 _8, a clearer image quality.

However, when estimating the output image, out of the region of the output image shown by the bold line in FIG. 55, in the region other than the region 411, of 1 to 8 th captured images 401 ₁ to 401 _8, the part of the number-th Since only the data of the captured image can be used, that is, all the data of the first to _eighth captured images 401 _{1 to} 4018 cannot be used, the image quality is compared with the area 411 accordingly. The sharpness of the image will deteriorate. As described above, when camera shake occurs in the upper right direction, the data in the lower left direction, which is the opposite, decreases in data that can be used to estimate the output image, and the sharpness of the image quality Degraded compared to 411.

Furthermore, among the 2-8 th captured images 401 ₁ to 401 ₈ is the target image, data with no portion 412 over the area of the output image shown by the bold line in FIG. 55, be used to estimate the output image It cannot be said that it will be thrown away in vain.

  As described above, when the output image is estimated using the first captured image as a reference, if a camera shake in a certain direction occurs, an area in the same direction as the camera shake (for example, from the center of the output image (for example, In the area 411) of FIG. 55, a clear image quality is obtained, but in the area in the opposite direction, the image quality is deteriorated.

  By the way, in general, when a user views an image, the user often pays attention to the central portion thereof, so that it is particularly desirable that the image quality of the central portion of the image is clear.

  Therefore, the signal processing circuit 7 uses a captured image (hereinafter referred to as an intermediate image) captured at an intermediate time or a time in the vicinity of the time when N images are continuously captured as a reference image, The output image can be estimated using the captured image as a target image.

For example, as illustrated in FIG. 56, the signal processing circuit 7 estimates an output image using a fourth captured image indicated by a thick line among the _eight captured images 401 _{1 to} 4018 as a reference image. In this case, in the estimation of the output image, all the data of the _{first to eighth} captured images 401 _{1 to} 4018 are used in the central area 421.

  That is, an intermediate image of a plurality of time-series captured images is used as a reference image, and an output image is estimated using another captured image as a target image. It can be image quality.

  In normal shooting (imaging), the photographer performs shooting so that the target subject is positioned at the center of the image (image frame). As described above, when a person views an image, the image is generally viewed from the center of the image as a viewpoint. In that sense, it can be said that a better image is obtained when the central portion of the image has a clearer image quality than the peripheral portion of the image.

Therefore, as shown in FIG. 56, by using the intermediate image as a reference image and another captured image as a target image, eight captured images 401 _{1 to} 401 whose central portions of the output image are the first to eighth images. 401 since ₈ is estimated using all data, rather than the reference image captured image of the first sheet, it is possible better to the intermediate image with the reference image to obtain a good output image.

Note that the frequency of camera shake is, for example, about 10 to 15 Hz. Therefore, for example, at the shutter speed at which the _eight captured images 401 _{1 to} 40 8 shown in FIG. 56 are captured within 1/50 second, the amount of camera shake (the amount of camera shake) is linearly approximated. Can do. That is, camera shake can be regarded as movement at a constant speed in a certain direction. Therefore, when the eight captured images 401 _{1 to} 401 ₈ are captured in time series, the amount of camera shake during the imaging time can be linearly approximated, and the intermediate image, that is, the fourth captured image 401 _{4 is} captured. and, by the 5 th captured image 401 ₅ the reference image, as described in FIG. 56, the image quality of the center portion of the output image can be clear.

Further, the signal processing circuit 7 shown in FIG. 4, of the eight captured images 401 ₁ to 401 _8, for example, the 4 th captured image 401 ₄ when the reference image is a frame memory 22- 1 is 4 th captured image 401 ₄ of the supply to the memory as a reference image, the first to third sheet of 401 ₁ to 401 _3, and 5 to 8 th captured images 401 ₅ to 401 _8, respectively The frame memories 22-2 to 22-8 may be supplied and stored.

  Here, by using the intermediate image as a reference image, an output image with a clear central portion can be obtained, and the design of the signal processing circuit 7 can be facilitated.

That is, assuming that the amount of camera shake can be linearly approximated as described above, when eight captured images 401 _{1 to} 40 ₁₈ are captured in time series, for example, camera shake between adjacent captured images is detected. Assume that the amount is 10 pixels. In this case, if the first captured image is used as a reference image, a maximum of 70 pixel camera shake occurs even if the capturing of one image can be performed in an infinitely small time. Therefore, it is necessary to design the signal processing circuit 7 so as to be able to cope with camera shake of 70 pixels at the maximum.

In contrast, when the intermediate image and the reference image, i.e., among the captured images 401 ₁ to 401 _8, for example, the case of the 4 th captured image 401 ₄ with the reference image, the maximum amount of motion blur is 40 pixels. Therefore, the signal processing circuit 7 may be designed so as to be able to cope with camera shake of 40 pixels at the maximum, so that the hardware design of the signal processing circuit 7 can be facilitated.

  The series of processes described in the eighth embodiment can be executed by dedicated hardware as well as other embodiments, or can be executed by software. When the series of processes described in the eighth embodiment is executed by software, as in the other embodiments, for example, the digital camera 1 is realized by causing the computer shown in FIG. 41 to execute the program. be able to.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. It also includes processes that are executed individually.

  In the above-described example, the image sensor 4 of the digital camera 1 is configured by a single plate sensor, and an image having three color signals per pixel is estimated from one color signal per pixel output from the image sensor 4. However, the image sensor 4 may not be a single plate sensor. In other words, the image pickup device 4 employs a device that outputs predetermined n color signals per pixel, and in the image estimation process, from the n color signals per pixel, (n + 1) or more per pixel. An image having a color signal can be estimated.

  In the above-described example, the positional relationship of the target image with respect to the reference image is detected using the first captured image or the intermediate image as a reference image. However, any captured image other than the first captured image and the intermediate image is detected. It is also possible to detect the positional relationship of the target image with respect to the reference image using as a reference image.

  Furthermore, as described above, the present invention can be used for a digital still camera as well as a digital video camera.

It is a block diagram which shows the structural example of one Embodiment of the digital camera 1 to which this invention is applied. 3 is a flowchart for describing an imaging process of the digital camera 1 in FIG. 1. It is a figure which shows the arrangement | sequence of the pixel of the image pick-up element 4 of FIG. It is a block diagram which shows the detailed structural example of the signal processing circuit 7 of FIG. It is a figure shown about the relationship of the coordinate of a 1st sheet image and a 2nd sheet image. It is a flowchart explaining 1st Embodiment of an image estimation process. It is a flowchart explaining 1st Embodiment of an image estimation process. It is a figure explaining the conditions of color correlation in a 2nd embodiment. It is a figure explaining the conditions of color correlation in a 2nd embodiment. It is a flowchart explaining 2nd Embodiment of an image estimation process. It is a flowchart explaining 2nd Embodiment of an image estimation process. It is a flowchart explaining 2nd Embodiment of an image estimation process. It is a figure which shows the 1st sheet image memorize | stored in the frame memory 22-1. It is a figure which shows the 2nd sheet image memorize | stored in the frame memory 22-2. It is a figure which shows the 3rd sheet image memorize | stored in frame memory 22-3. 2 is a diagram illustrating pixels of an image sensor 4. It is a figure which shows the arrangement | sequence of the pixel of the image pick-up element 4 of FIG. It is a figure explaining a spring model. It is a figure explaining a spring model. It is a figure explaining a spring model. It is a figure explaining a spring model. It is a figure explaining a spring model. It is a figure explaining a spring model. It is a flowchart explaining 3rd Embodiment of an image estimation process. It is a flowchart explaining 4th Embodiment of an image estimation process. It is a flowchart explaining 5th Embodiment of an image estimation process. It is a figure which shows the state in which each pixel of the image pick-up element 4 is light-receiving the green component (G signal). It is a figure explaining the edge of a vertical direction. It is a figure explaining the edge of a horizontal direction. It is a figure explaining the edge of a lower right direction from upper left. It is a figure explaining the edge of a lower left direction from an upper right. It is a figure explaining the plane Q1 produced when the edge of a vertical direction is detected. It is a figure explaining the plane Q2 created when the edge of a horizontal direction is detected. It is a figure explaining plane Q3 created when the edge of the lower right direction is detected from the upper left. It is a figure explaining plane Q4 created when the edge of a lower left direction is detected from an upper right. It is a flowchart explaining 6th Embodiment of an image estimation process. It is a figure explaining the conditions of color correlation in a 7th embodiment. It is a figure explaining the conditions of color correlation in a 7th embodiment. It is a flowchart explaining 7th Embodiment of an image estimation process. It is a figure explaining the difference between this invention and the conventional method. It is a block diagram which shows the structural example of one Embodiment of the computer to which this invention is applied. It is a figure explaining a cubic function. It is a figure explaining the pixel value on a reference coordinate system in attention position (I ', J'). It is a figure explaining the pixel value on a reference coordinate system in attention position (I ', J') of an exceptional state. It is a figure explaining the exception process of G signal. It is a figure explaining the exception process of G signal. It is a figure explaining the exception process of R signal. It is a figure explaining the exception process of R signal. It is a flowchart explaining 8th Embodiment of an image estimation process. It is a flowchart explaining 8th Embodiment of an image estimation process. It is a flowchart explaining 8th Embodiment of an image estimation process. It is a flowchart explaining 8th Embodiment of an image estimation process. It is a flowchart explaining 8th Embodiment of an image estimation process. Is a diagram showing a captured image 401 ₁ to 401 _8. It is a figure explaining the output image obtained when the 1st captured image is used as a reference image. It is a figure explaining the output image obtained when an intermediate image is used as a standard image.

Explanation of symbols

  1 Digital Camera, 2 Lens, 3 Aperture, 4 Image Sensor, 5 Correlated Double Sampling Circuit, 6 A / D Converter, 7 Signal Processing Circuit, 8 Timing Generator, 9 D / A Converter, 10 Video Encoder, 11 Monitor, 12 Codec, 13 memory, 14 bus, 15 CPU, 16 input device, 21 shift circuit, 22-1 to N frame memory, 23-1 to (N-1) motion detection circuit, 24 arithmetic circuit, 25 controller

Claims (16)

In an image processing method for estimating an output image from a plurality of input images,
Positions of the plurality of input images with a predetermined one input image among the plurality of input images captured by the imaging unit that captures an image having a pixel value of one color signal per pixel as a reference image A detection step for detecting a relationship;
For each color of R, G, and B, with the pixel center of each pixel of the reference image as the target position, for the target position, the light amount of the true color at the target position and each input image other than the reference image are the reference When corrected to a position on the image system, the relationship between the observed pixel value that is the pixel value of each input image other than the reference image observed around the target position, the natural length is zero, and the observed pixel value Three color signals R, G, and B per pixel are generated by solving and solving as a spring relational expression of a spring model with the probability that the true color light quantity at the target position is equivalent to a spring constant. An image estimation step of estimating the output image having a pixel value of. The image estimation step generates a spring relational expression of the spring model using only a predetermined number of observation pixel values set in advance among the observation pixel values around the position of interest, and solves the output. The image processing method according to claim 1, wherein an image is estimated. The image estimation step calculates an average value of all the observed pixel values around the position of interest , and generates a spring relational expression of the spring model using only the predetermined number of observed pixel values in order from the average value. The image processing method according to claim 2 , wherein the output image is estimated by solving . The image estimation step generates and solves a spring relational expression of the spring model using only the predetermined number of pixel values in order of increasing distance from the target position among the observed pixel values around the target position. The image processing method according to claim 2 , wherein the output image is estimated . The image estimation step detects whether there is an edge portion around the target position, and generates and solves a spring relational expression of the spring model excluding the observed pixel value at the position detected as the edge portion. The image processing method according to claim 1 , wherein the output image is estimated by : The image estimation step detects whether there is an edge part around the position of interest, and the observed pixel value at the position detected as the edge part is changed to a predetermined pixel value. The image processing method according to claim 1 , wherein the output image is estimated by generating and solving . The image estimation step is characterized in that the predetermined pixel value is changed to a pixel value on the target position of a plane having the inclination of the edge portion and passing through the observed pixel value of the edge portion. The image processing method according to claim 6 . In the image estimation step, the edge in at least one of the four directions of “vertical edge”, “lateral edge”, “upper left to lower right edge”, and “upper right to lower left edge” is selected. The image processing method according to claim 5 , wherein presence or absence of a part is detected . In the image estimation step, in addition to the spring relational expression, at least two of G and R, G and B, and R and B are added on the condition that there is a color correlation, and by solving, The image processing method according to claim 1 , wherein the output image is estimated . In the image estimation step, a position on the reference image is set to a predetermined allowable value in a vicinity region centered on the target position as a condition of color correlation between G and R, G and B, or R and B. Find the observed pixel values of the two colors that are within, perform principal component analysis, find the variance for the component orthogonal to the principal component, and add the condition that the spring constant increases as the variance decreases The image processing method according to claim 9. The amount of light of the true color at the target position is expressed as a weighted addition according to the distance between the target position and the observed pixel value, and the weight includes the target position (I ′, J ′) and the observed pixel. position on the reference image based value (x, y), a reference to cubic function cubic, to claim 1, characterized in that calculated by cubic (I'-x) × cubic (J'-y) The image processing method as described. The image estimation step determines whether ΣCubic (I′−x) × Cubic (J′−y) is equal to or greater than a predetermined threshold value. The image processing method according to claim 11 , wherein the pixel value of the target position is estimated also using pixel values of peripheral pixels around the pixel . The detecting step includes
The positional relationship between the plurality of input images is detected by using, as the reference image , an input image captured at an intermediate time or a time close to the time when the plurality of input images are captured. Item 8. The image processing method according to Item 1. The imaging means captures the plurality of input images with less than appropriate exposure,
The image processing method according to claim 1, further comprising an exposure correction step of increasing a gain of a pixel value of each of the plurality of input images. In an image processing apparatus that estimates an output image from a plurality of input images,
Imaging means for imaging the plurality of input images having a pixel value of one color signal per pixel;
Detecting means for detecting a positional relationship between the plurality of input images , with a predetermined one input image of the plurality of input images as a reference image ;
For each color of R, G, and B, with the pixel center of each pixel of the reference image as the target position, for the target position, the light amount of the true color at the target position and each input image other than the reference image are the reference When corrected to a position on the image system, the relationship between the observed pixel value that is the pixel value of each input image other than the reference image observed around the target position, the natural length is zero, and the observed pixel value Three color signals R, G, and B per pixel are generated by solving and solving as a spring relational expression of a spring model with the probability that the true color light quantity at the target position is equivalent to a spring constant. And an image estimation means for estimating the output image having a pixel value of. To an image processing computer that estimates an output image from a plurality of input images,
Positions of the plurality of input images with a predetermined one input image among the plurality of input images captured by the imaging unit that captures an image having a pixel value of one color signal per pixel as a reference image A detection step for detecting a relationship;
For each color of R, G, and B, with the pixel center of each pixel of the reference image as the target position, for the target position, the light amount of the true color at the target position and each input image other than the reference image are the reference When corrected to a position on the image system, the relationship between the observed pixel value that is the pixel value of each input image other than the reference image observed around the target position, the natural length is zero, and the observed pixel value Three color signals R, G, and B per pixel are generated by solving and solving as a spring relational expression of a spring model with the probability that the true color light quantity at the target position is equivalent to a spring constant. a program characterized by executing a process including the image estimation step of estimating the output image having the pixel value.

Images