JP2008245236A - Imaging apparatus and defective pixel correcting method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of appropriately correcting a defective pixel. <P>SOLUTION: The imaging apparatus 1A includes: an image pickup element 15 for capturing a photographed image, including a defective pixel, of a subject image; a position change means for changing a relative position of the image pickup element 15 with respect to a photographing optical system from a main photographed image capturing position to an image-for-correction capturing position; a photographing control means for capturing a main photographed image by means of the image pickup element 15 at the main photographed image capturing position and for capturing an image for correction by means of the image pickup element 15 at the image-for-correction capturing position; a matching degree acquiring section 114a for acquiring a degree of matching between the main photographed image and the image for correction; a correction section 114c for the defective pixel for correcting a pixel signal acquired at the defective pixel in the main photographed image; and a correction control section 114b which causes the correction section 114c to carry out defective pixel correction corresponding to the degree of matching. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for correcting defective pixels in an image sensor.

  In recent years, digital single-lens reflex cameras (DSLR) have been rapidly spreading. In a digital single-lens reflex camera, an image (photographed image) related to a subject image is acquired using an image sensor such as a CCD. However, a defective pixel may exist in the image sensor. For example, the defective pixel does not depend on the exposure amount, or does not depend on the high output pixel (white flaw) and the exposure amount that are clearly different from other pixels, or the dependency method clearly differs. There are low output pixels (black flaws) different from these pixels. A high-capacity point due to a high-output pixel or a low-brightness point due to a low-output pixel occurs in the actual captured image acquired by such an image sensor having defective pixels, and a good captured image is obtained. Can not do it.

  As a technique for correcting defective pixels, for example, after a recording image (an image before displacement) relating to a subject image is captured using an imaging device, the relative position between the imaging optical system and the imaging device is changed (displaced). Then, an image related to the same subject image (image after displacement) is taken in a different area of the image sensor, and a pixel signal (pixel value) obtained by a defective pixel in the image before displacement is displaced in the image after displacement. A technique has been proposed in which correction is performed using a pixel signal acquired by a normal pixel having the same absolute coordinate as the defective pixel of the previous image (Patent Document 1).

JP 2001-86411 A

  By the way, when the subject changes due to movement or the like between the two shootings, the pixels having the same absolute coordinates in the two images acquire pixel signals relating to different subject parts.

  Therefore, without considering the subject change between the two shootings, using the pixel signal of the pixel with the same absolute coordinate as the defective pixel of the image before displacement in the image after displacement, the image of the image before displacement is uniformly used. When defective pixel correction is performed, the correction accuracy of defective pixels may be significantly reduced.

  Accordingly, an object of the present invention is to provide an imaging apparatus capable of executing appropriate defective pixel correction according to changes in the subject between two shootings.

  According to a first aspect of the present invention, there is provided an imaging device having a photographic optical system, the imaging device having a defective pixel and acquiring a photographic image related to a subject image, and a relative position of the imaging device with respect to the photographic optical system. Using the position changing means for changing from the first relative position to the second relative position and the position changing means, the first picked-up image is acquired by the image pickup device at the first relative position, and the second Photographing control means for obtaining a second photographed image by the image pickup device at the relative position, a matching degree obtaining means for obtaining a coincidence degree between the first photographed image and the second photographed image, and the first In the captured image, a defective pixel correction unit that corrects a pixel signal acquired from the defective pixel, and a correction control unit that causes the defective pixel correction unit to perform defective pixel correction according to the degree of coincidence. The features.

  According to a second aspect of the present invention, there is provided a method for correcting a defective pixel in a captured image acquired by an image sensor having a defective pixel, wherein a) the relative position of the image sensor with respect to the photographic optical system is a first relative position. And b) acquiring a first photographed image by the image sensor at the first relative position and obtaining a second photographed image by the image sensor at the second relative position. A step of obtaining c) a step of obtaining a degree of coincidence between the first photographed image and the second photographed image; and d) a pixel signal obtained from the defective pixel in the first photographed image. And correcting the defective pixel in accordance with the degree of coincidence in the step d).

  According to a third aspect of the present invention, there is provided an imaging device having a photographing optical system, the imaging device having a defective pixel and acquiring a photographed image relating to a subject image, and a relative position of the imaging device with respect to the photographing optical system. Using the position changing means for changing the position from the first relative position to the second relative position, and using the position changing means, the first picked-up image is obtained by the image pickup device at the first relative position, and A photographing control unit that obtains a second photographed image by the image sensor at the second relative position, and a change between the subject at the time of obtaining the first photographed image and the subject at the time of obtaining the second photographed image. Change acquisition means for acquiring a change vector; and an image corresponding to a subject part imaged on a defective pixel of the image sensor at the first relative position among the pixels in the second photographed image. Identifying means for identifying a corresponding pixel having a signal based on a movement vector representing a displacement from the first relative position to the second relative position and the change vector, and using a pixel signal of the corresponding pixel And defect correction means for correcting defective pixels in the first photographed image.

  According to the present invention, the relative position of the image sensor with respect to the photographic optical system is changed from the actual captured image acquisition position to the correction image acquisition position, the captured image is acquired at each position, and the degree of coincidence between the two captured images To get. Since defective pixel correction is performed in accordance with the degree of coincidence between the two photographed images, appropriate defective pixel correction can be performed.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<1. First Embodiment>
<1-1. Configuration>
1 and 2 are diagrams showing an external configuration of an imaging apparatus 1A according to the first embodiment of the present invention. Here, FIG. 1 is a front external view of the image pickup apparatus 1A, and FIG. 2 is a rear external view of the image pickup apparatus 1A. This imaging device 1A is configured as a lens interchangeable single-lens reflex digital camera.

  As shown in FIG. 1, the imaging device 1 </ b> A includes a camera body (camera body) 2. An interchangeable photographic lens unit (interchangeable lens) 3 can be attached to and detached from the camera body 2.

  The photographic lens unit 3 is mainly composed of a lens barrel 101, a lens group 37 (see FIG. 3) and an aperture (not shown) provided inside the lens barrel 101, and the like. The lens group 37 includes a focus lens that changes the focal position by moving in the optical axis direction.

  The camera body 2 includes an annular mount Mt to which the photographing lens unit 3 is attached at the front center, and an attach / detach button 89 for attaching / detaching the photographing lens unit 3 near the annular mount Mt. Yes.

  Further, the camera body 2 is provided with a mode setting dial 82 in the upper left part of the front surface and a control value setting dial 86 in the upper right part of the front surface. By operating the mode setting dial 82, various camera modes (such as various shooting modes (such as portrait shooting mode, landscape shooting mode, and continuous shooting mode), a playback mode for playing back captured images, and external devices can be used. It is possible to perform a setting operation (switching operation) including a communication mode for performing data communication. Further, by operating the control value setting dial 86, it is possible to set control values in various shooting modes.

  Further, the camera body 2 includes a grip portion 14 for a photographer to hold at the left end of the front. A release button 11 for instructing the start of exposure is provided on the upper surface of the grip portion 14. A battery storage chamber and a card storage chamber are provided inside the grip portion 14. For example, four AA batteries are housed in the battery compartment as a power source for the camera, and a memory card 119 (see FIG. 4) for recording image data of a photographed image is removable in the card compartment. It is designed to be stored in.

  The release button 11 is a two-stage detection button that can detect two states, a half-pressed state (S1 state) and a fully-pressed state (S2 state). When the release button 11 is half-pressed to enter the S1 state, a preparation operation (for example, an AF control operation and an AE control operation) for acquiring a recording still image (main captured image) related to the subject is performed. When the release button 11 is further pushed into the S2 state, a photographing operation for the actual photographed image (an exposure operation for the subject image is performed using the imaging element 15 (described later), and the image signal obtained by the exposure operation is converted into an image signal. A series of operations for performing predetermined image processing) is performed.

  In FIG. 2, a rear monitor 12 is provided in the approximate center of the rear surface of the camera body 2. The rear monitor 12 is configured as a color liquid crystal display (LCD), for example. The rear monitor 12 can display a menu screen for setting shooting conditions and the like, and can reproduce and display a captured image recorded on the memory card 119 in the reproduction mode.

  A finder window 10 is provided at the upper center of the back surface of the camera body 2. The subject image from the photographic lens unit 3 is guided to the finder window 10, and the photographer can view an image equivalent to the subject image acquired by the image sensor 15 by looking into the finder window 10. Specifically, the subject image incident on the photographing optical system is reflected upward by the mirror mechanism 103 (see FIG. 3) and is viewed through the eyepiece lens 106. Thus, the photographer can determine the composition by looking through the finder window 10. At the time of photographing the actual photographed image, the mirror mechanism 103 is retracted from the light path of the light that forms the subject image, and the light from the photographing lens unit 3 (light that forms the subject image) reaches the image sensor 15. A captured image (image data) relating to the subject is obtained.

  A main switch 81 is provided at the upper left of the rear monitor 12. The main switch 81 is a two-point slide switch. When the contact is set to the left “OFF” position, the power is turned off. When the contact is set to the right “ON” position, the power is turned on.

  A direction selection key 84 is provided on the right side of the rear monitor 12. This direction selection key 84 has a circular operation button, and it is possible to detect a pressing operation in four directions of up, down, left and right, and a pressing operation in four directions of upper right, upper left, lower right and lower left on this operation button. It has become. In addition, the direction selection key 84 is configured to detect a pressing operation of a push button at the center, in addition to the pressing operations in the eight directions.

  A setting button group 83 including a plurality of buttons for setting a menu screen, deleting an image, and the like is provided on the left side of the rear monitor 12.

  Next, the internal configuration of the imaging apparatus 1A will be described. FIG. 3 is a longitudinal sectional view of the imaging apparatus 1A. As shown in FIG. 3, the imaging device 1A includes an imaging device 15, a finder unit 102 (finder optical system), a mirror mechanism 103, a focus detection unit 107, an imaging device driving mechanism 112, a shutter unit 40, and the like. ing.

  The imaging element (imaging sensor) 15 is perpendicular to the optical axis L on the optical axis L of the lens group 37 provided in the imaging lens unit 3 when the imaging lens unit 3 is attached to the imaging apparatus 1A. An image signal related to the subject image is generated in the plane. Details will be described later.

  On the optical axis L, a mirror mechanism 103 (reflecting plate) is disposed at a position where the subject light is reflected toward the finder unit 102. The subject light that has passed through the photographing lens unit 3 is reflected upward by a mirror mechanism 103 (a main mirror 1031 described later), and forms an image on a focusing screen 104 (focus glass). Part of the subject light that has passed through the photographing lens unit 3 passes through the mirror mechanism 103.

  The finder unit 102 includes a pentaprism 105, an eyepiece lens 106, and a finder window 10. The pentaprism 105 has a pentagonal cross section, and is a prism for changing the top and right sides of the optical image into an upright image by reflection inside the subject image incident from the lower surface thereof. The eyepiece 106 guides the subject image that has been made upright by the pentaprism 105 to the outside of the finder window 10. With such a configuration, the finder unit 102 functions as an optical finder for confirming the object field during shooting standby.

  The mirror mechanism 103 includes a main mirror 1031 and a sub mirror 1032, and is provided on the back side of the main mirror 1031 so that the sub mirror 1032 can be turned down toward the back of the main mirror 1031. Part of the subject light transmitted through the main mirror 1031 is reflected by the sub mirror 1032, and the reflected subject light enters the focus detection unit 107.

  The mirror mechanism 103 is configured as a so-called quick return mirror. At the time of exposure, the mirror mechanism 103 jumps upwards as indicated by an arrow A with the rotation shaft 1033 as a rotation fulcrum, and stops at a position below the focusing screen 104. At this time, the sub mirror 1032 rotates about the rotation shaft 1034 in the direction indicated by the arrow B with respect to the back surface of the main mirror 1031, and when the mirror mechanism 103 stops at a position below the focusing screen 104, It will be in the state folded so that it may become substantially parallel to 1031. Thereby, the subject light from the photographing lens unit 3 reaches the image sensor 15 without being blocked by the mirror mechanism 103, and the image sensor 15 is exposed. When the exposure is completed, the mirror mechanism 103 returns to the original position (position shown in FIG. 3).

  The focus detection unit 107 is configured as a so-called AF sensor including a distance measuring element that detects focus information of a subject. The focus detection unit 107 is disposed at the bottom of the mirror mechanism 103, and detects the in-focus position by, for example, a known phase difference detection method.

  The image sensor 15 is held so as to be movable two-dimensionally on a plane orthogonal to the optical axis L by an image sensor driving mechanism (also simply referred to as “driving mechanism”) 112 (see FIG. 4). Further, immediately before the image sensor 15 in the optical axis direction, a low-pass filter 108 for preventing the incidence of infrared rays (for IR cut) and for preventing the generation of pseudo color and color moire is disposed. A shutter unit 40 is disposed immediately before the low-pass filter 108. The shutter unit 40 is a mechanical focal plane shutter that includes a curtain that moves in the vertical direction, and that performs an optical path opening operation and an optical path blocking operation of subject light guided to the image sensor 15 along the optical axis L.

  The above components of the image pickup apparatus 1A are connected (fixed) to each other by a chassis made of a metal material such as iron. In the present embodiment, an example is shown in which the above-described chassis includes a front chassis (not shown), a rear chassis 183, and a bottom chassis 184. These chassis serve as a support material that supports the components in the imaging apparatus 1A described above. The bottom chassis 184 is provided with a tripod mounting portion 185.

  Next, an outline of functions of the imaging apparatus 1A will be described. FIG. 4 is a block diagram illustrating a functional configuration of the imaging apparatus 1A.

  As illustrated in FIG. 4, the imaging apparatus 1A includes an overall control unit 110, an imaging processing unit 113, a defect correction unit 114, an image processing unit 115, an operation unit 118, and the like.

  The overall control unit 110 is configured as a microcomputer and mainly includes a CPU, a RAM 4a, a ROM 4b, and the like. The overall control unit 110 implements various functions by reading a program stored in the ROM 4b and executing the program by the CPU.

  For example, based on the output signal (blur signal) of the gyro sensor 117, the overall control unit 110 derives the blur amount of the subject image due to vibration and the movement amount and direction of the image sensor 15 corresponding to the blur direction. Then, by driving the drive mechanism 112 via the drive mechanism control unit 111, the image pickup device 15 is moved in a plane perpendicular to the optical axis L to correct the shake of the image pickup apparatus 1 </ b> A caused by camera shake or the like. It has a function to do. The drive mechanism 112 includes a pitch direction actuator and a yaw direction actuator, and has a function of moving the imaging element 15 in two directions, horizontal and vertical, in a plane perpendicular to the optical axis L.

  The operation unit 118 includes various buttons and switches including the release button 11 (see FIG. 1). In response to a user input operation on the operation unit 118, the overall control unit 110 and the like implement various operations. For example, when the fully-pressed state of the release button 11 is detected, a photographing operation for the main photographed image is executed, and the image pickup device 15 is moved by the driving mechanism 112 to perform a photographing operation for the correction image. The acquired correction image is used for correcting the pixel signal (pixel value) acquired by the defective pixel of the image sensor 15. Details will be described later.

  The imaging device 15 performs exposure (charge accumulation by photoelectric conversion) of a subject image formed on the light receiving surface, and generates an image signal related to the subject image. Specifically, the image sensor 15 has a plurality of pixels configured with photodiodes arranged two-dimensionally in a matrix, and R (red), G (green), and B (blue) are arranged on the light receiving surface of each pixel. ) Primary color transmission filters are arranged as a checkered color sensor (for example, a CCD). The image sensor 15 converts the optical image of the subject formed by the lens group 37 into analog electrical signals of R (red), G (green), and B (blue) color components, and generates an image signal. .

  The image signal acquired by the imaging element 15 is output to the imaging processing unit 113. The image signal is subjected to noise removal and gain adjustment in the imaging processing unit 113, and then converted into digital image data (image data) by A / D conversion. The image data is written into the image memory 116. Thereafter, the defect correction unit 114 and the image processing unit 115 perform various processes on the image data stored in the image memory 116. The image memory 116 is a high-speed accessible image memory for temporarily storing generated image data, and has a capacity capable of storing image data for a plurality of frames.

  The defect correction unit 114 includes a coincidence degree acquisition unit 114a, a correction control unit 114b, and a correction unit 114c. The coincidence degree acquisition unit 114a obtains the degree of coincidence between the subject in the main captured image and the subject in the correction image. The correction control unit 114b selectively controls a defective pixel correction method based on the degree of coincidence. The correction unit 114c corrects the pixel signal acquired by the defective pixel of the image sensor 15 in the actual captured image, using the correction method selected and instructed by the correction control unit 114b. Details of the defective pixel correction executed in the defect correction unit 114 will be described later.

  The image processing unit 115 performs digital signal processing on the image data stored in the image memory 116 to generate image data related to the captured image. Specifically, the image processing unit 115 executes various processes such as a pixel interpolation process, a white balance adjustment process, and an edge enhancement process.

  In the pixel interpolation process, the pixel value of the color signal that is insufficient in each pixel of the Bayer array is calculated by interpolation. For example, in a pixel (G pixel) from which a G component pixel value can be acquired, the R component pixel value is determined as the average value of the R pixel values above and below the G pixel, and the B component Is determined as an average value of the pixel values of the B pixels existing on the left and right of the G pixel. Further, in the R pixel, the pixel value of the insufficient G component or B component is determined based on the pixel values of the four G pixels or the four B pixels existing around the R pixel. Similarly, in the B pixel, the pixel value of the insufficient G component or R component is determined based on the pixel values of the four G pixels or the four R pixels existing around the B pixel.

  In the white balance adjustment process, white balance control of an image is performed by independently performing gain correction on each of the RGB pixels.

  In the contour emphasis process, an edge emphasis process is performed to make the contour stand out with a high-pass filter corresponding to the image data.

  The image data stored in the image memory 116 is appropriately subjected to image processing in the defect correction unit 114 and the image processing unit 115, and then recorded on a recording medium (here, a memory card 119).

<1-2. Operation>
<1-2-1. Overall operation>
Next, defective pixel correction in the present embodiment will be described. FIG. 5 is a diagram illustrating defective pixels in the image sensor 15. Note that each unit area partitioned by a dotted line in the image sensor 15 of FIG. 5 represents one pixel of the image sensor 15, and each pixel of the image sensor 15 is schematically enlarged for simplicity. . FIG. 6 is an overall flowchart regarding defective pixel correction. FIG. 7 is a diagram illustrating two images acquired by changing the relative positions of the imaging optical system and the image sensor 15. In FIG. 7, the image sensor at the position before the movement is represented by reference numeral 15a, and the image sensor at the position after the movement is represented by reference numeral 15b. FIG. 8 is a diagram showing an evaluation area for acquiring the degree of coincidence. FIG. 9 is a diagram illustrating a region corresponding to the evaluation region in the correction image GR2. Note that each unit area defined by the dotted lines in the two images GR1 and GR2 shown in FIGS. 7, 8, and 9 represents one pixel of the image, and each pixel of the image is for simplicity. Is shown in a magnified view.

  In the present embodiment, a plurality of defective pixels included in an area (also referred to as “defective pixel area”) KA in which defective pixels in units of one pixel as illustrated in FIG. 5 are adjacent to each other in the two-dimensional direction. Assuming that the image pickup apparatus 1A is not shaken due to camera shake or the like.

  As shown in FIG. 6, in step SP1, the relative position between the imaging optical system and the image sensor 15 is changed, and two images (specifically, the actual captured image GR1 and the correction image GR2) are acquired. . Specifically, when the S2 state of the release button 11 is detected, the subject image is exposed, and the main captured image GR1 relating to the subject image is acquired. When the exposure of the actual captured image is completed, the image sensor 15 is moved using the above-described camera shake correction function, and the relative position between the imaging optical system and the image sensor 15 is changed. Then, exposure is performed at a new relative position, and an image (correction image) GR <b> 2 relating to the same subject image is captured in a different area of the image sensor 15. More specifically, as shown in FIG. 7, the actual captured image GR1 (FIG. 7 (b)) is acquired by the imaging element 15a (FIG. 7 (a)) at the position before the movement, and the imaging at the position after the movement is performed. The correction image GR2 (FIG. 7B) is acquired by the element 15b (FIG. 7A). Note that an arrow MV in FIG. 7A represents a displacement (relative position change direction and change amount) between the photographing optical system and the image sensor 15, and is hereinafter also referred to as a movement vector MV. In addition, the region surrounded by the alternate long and short dash line on the main captured image GR1 in FIG. 7B is a region having pixel signals acquired by each pixel in the defective pixel region KA of the image sensor 15 in the main captured image GR1 ( KS) (also referred to as “defect signal region”). Further, displacement information (a value expressing the displacement of the relative position between the imaging optical system and the image sensor 15 as a two-dimensional component value) HI relating to the change in the relative position between the imaging optical system and the image sensor 15 is stored in the ROM 4b in advance. Based on the displacement information HI, the movement destination of the image sensor 15 is determined.

  In step SP2, an evaluation value ES indicating the degree of coincidence of the subject in the two acquired images is calculated. In the present embodiment, the evaluation value ES representing the degree of coincidence is acquired by comparing regions assumed to have information on the same subject part in two images acquired at different relative positions.

  Specifically, as shown in FIG. 8, first, an area in the vicinity of the defect signal area KS in the actual captured image GR1 is specified as an evaluation area (also referred to as “matching degree acquisition area”) HR. Here, a case where four regions HR1, HR2, HR3, and HR4 around the defect signal region KS are used as the evaluation region HR is illustrated.

  As a method for setting the four evaluation regions HR1 to HR4, first, each pixel included in the defect signal region KS is set with the position where the pixel (also referred to as “origin pixel”) OP in the lower left corner of the captured image GR1 is present as the origin. The X coordinate minimum value Xs, the X coordinate maximum value Xb, the Y coordinate minimum value Ys, and the Y coordinate maximum value Yb are respectively acquired from the coordinate values indicated by the pixels. Then, in the defect signal region KS, a pixel (in this case, three pixels) located at a position moved from the one pixel selected from the pixels having the minimum value Xs as the coordinate value of the X coordinate in the −X direction (here, three pixels). An area of 3 pixels × 3 pixels centering on CP1 (also referred to as “reference pixel”) is set (specified) as the evaluation area HR1. Similarly, in the defect signal area KS, 3 pixels × centered on a pixel CP3 existing at a position shifted by 3 pixels in the + X direction from one pixel selected from pixels having the maximum value Xb as the coordinate value of the X coordinate × An area of 3 pixels is set as the evaluation area HR3. Similarly, in the defect signal area KS, three pixels centering on a pixel CP2 present at a position shifted by three pixels in the + Y direction from one pixel selected from pixels having the maximum value Yb as the coordinate value of the Y coordinate. A region of × 3 pixels is set as the evaluation region HR2. Similarly, 3 in the defect signal area KS is centered on a pixel CP4 existing at a position shifted by three pixels in the −Y direction from one pixel selected from pixels having the minimum value Ys as the coordinate value of the Y coordinate. An area of pixels × 3 pixels is set as the evaluation area HR4.

  Next, as shown in FIG. 9, in the correction image GR2, corresponding regions TR1, TR2, TR3, and TR4 corresponding to the four evaluation regions HR1 to HR4 are specified. Specifically, when the position where the origin pixel OP in the lower left corner of the actual captured image GR1 is used as the origin, in the correction image GR2, the same as the pixels in the four evaluation regions HR1 to HR4 of the actual captured image GR1. Regions constituted by pixels having absolute coordinates are determined as corresponding regions TR1 to TR4, respectively. Each of the corresponding regions TR1 to TR4 determined in this way is also expressed as a region that is assumed to have information on the same subject part as the corresponding evaluation regions HR1 to HR4. If the absolute coordinates of the evaluation area are given by the absolute coordinates of the central pixel in the evaluation area, the corresponding areas TR1 to TR4 are areas having the same absolute coordinates as the corresponding evaluation areas HR1 to HR4. It is also expressed.

  The evaluation value ES indicating the degree of coincidence is a pixel corresponding to the evaluation pixel in the corresponding region TR of the correction image GR2 and a pixel in the evaluation region HR (also referred to as “evaluation pixel”) in the captured image GR1. And also referred to as “evaluation pixel”). As the evaluation value ES representing the degree of coincidence, for example, the square value of the difference between the G component pixel value of the evaluation pixel and the G component pixel value of the corresponding evaluation pixel corresponding to the evaluation pixel is added for each evaluation pixel. The value obtained in this way is used. More specifically, the evaluation value ES is the pixel value PG1 (x, y) of the G component of the evaluation pixel (coordinate (x, y)) and the pixel value of the G component of the corresponding evaluation pixel (coordinate (x, y)). Using PG2 (x, y), it is expressed as in equation (1).

  The evaluation value ES calculated in this way represents the degree of coincidence between the subject parts imaged in the region (or pixel) having the same absolute coordinates in the two images. That is, the evaluation value ES represented by the expression (1) becomes a small value when the identity between the subject part in the evaluation region HR and the subject part in the corresponding region TR becomes high, and becomes a large value when the identity becomes low.

As shown in FIG. 8, when four regions HR1, HR2, HR3, and HR4 around the defect signal region KS are used as the evaluation region HR, there are nine pixels per evaluation region. The evaluation value ES is acquired based on the difference in pixel value between two corresponding 9 × 4) pixels.

Next, in step SP3 (FIG. 6), the evaluation value ES calculated in step SP2 is compared with a preset first threshold value TH1. When the evaluation value ES is smaller than the first threshold value TH1, the degree of coincidence between the main captured image GR1 and the correction image GR2 is high, that is, between image acquisition between the main captured image GR1 and the correction image GR2. It is determined that there has been no movement (change) of the subject, and the process proceeds to step SP4.

  In step SP4, a corresponding pixel having a pixel signal corresponding to a subject part imaged on a defective pixel of the image sensor 15a at the acquisition position of the main captured image GR1 is specified among the pixels in the correction image GR2. Then, the pixel signal acquired by the defective pixel in the main captured image GR1 is corrected using the pixel signal of the corresponding pixel in the correction image GR2. Details will be described later.

  On the other hand, when the evaluation value ES is equal to or greater than the first threshold value TH1, it is determined that the degree of coincidence between the main captured image GR1 and the correction image GR2 is low, and the process proceeds to step SP5.

  In step SP5, the evaluation value ES is further compared with a preset second threshold value TH2. Note that the second threshold value TH2 is set to a value larger than the first threshold value TH1.

  When the evaluation value ES is smaller than the second threshold value TH2, it is determined that there has been a minute subject change between image acquisition of the main captured image GR1 and the correction image GR2, and the process proceeds to step SP6.

  In step SP6, in the correction image GR2, a region having the same subject part as the subject part included in the evaluation region HR of the captured image GR1 (also referred to as “same region”) is specified, and the absolute coordinates of the same region are determined. A motion vector (details will be described later) is calculated from the absolute coordinates of the evaluation region HR. The absolute coordinates of a certain area are represented by the absolute coordinates of the central pixel in the area.

  Based on the motion vector and the movement vector MV, a corresponding pixel for each defective pixel of the image sensor 15a at the actual captured image GR1 acquisition position is selected from the pixels of the image sensor 15b at the correction image GR2 acquisition position. The pixel signal acquired by the defective pixel in the actual captured image GR1 is corrected using the pixel signal of the corresponding pixel. Details will be described later.

  On the other hand, if the evaluation value ES is equal to or greater than the second threshold value TH2, it is determined that there has been a change in subject between image acquisition of the main captured image GR1 and the correction image GR2, and the process proceeds to step SP7.

  In step SP7, the pixel signal of each pixel in the defect signal region KS is corrected using information (peripheral information) obtained from the non-defective pixels (pixels having no defect) around the defect signal region KS in the captured image GR1. The Details will be described later.

  As described above, the imaging apparatus 1A according to the present embodiment changes the relative position of the imaging element with respect to the imaging optical system from the actual captured image GR1 acquisition position to the correction image GR2 acquisition position, and two images are obtained at each position. An image is taken, and the degree of coincidence between the two images is acquired. Then, the most effective defective pixel correction is executed according to the degree of coincidence of the two images. According to this, it becomes possible to perform appropriate defective pixel correction according to the degree of coincidence.

<1-2-2. Defective pixel correction>
Hereinafter, the defective pixel correction executed in step SP4, step SP6, and step SP7 will be described in detail.

  First, the first defective pixel correction executed in step SP4 will be described. FIG. 10 is a diagram illustrating how the image sensor 15 moves. FIG. 11 is a diagram illustrating a specific state of the corresponding pixel using the movement vector MV.

  In step SP4, first, a pixel (corresponding pixel) corresponding to a defective pixel of the image sensor 15 at the actual captured image GR1 acquisition position from among the pixels of the image sensor 15 at the correction image GR2 acquisition position after the change of the relative position. Is identified. Specifically, as shown in FIG. 10, in the image pickup devices 15a and 15b at the two positions shifted based on the displacement information HI, among the pixels of the image pickup device 15b at the position after the movement, before the movement. Each pixel in the area HA (area surrounded by a two-dot chain line) corresponding to the defective pixel area KA (FIG. 10A) at the position becomes a corresponding pixel of each defective pixel.

  Such a corresponding pixel can be specified based on the displacement information HI. Specifically, each pixel present at a position moved by a displacement obtained by inverting the sign of the amount of movement of each direction component of the displacement information HI from each defective pixel of the image sensor 15 is a corresponding pixel of each defective pixel. It becomes. For example, a case where the corresponding pixel of the defective pixel PK1 in the defective pixel area KAm shown in FIG. In this case, the displacement obtained by inverting the sign of the amount of movement of each direction component (here, +3 pixels in the X direction and −4 pixels in the Y direction) from the defective pixel PK1, that is, from the defective pixel PK1 to the X direction. The pixel PXz existing at a position shifted by −3 pixels and by +4 pixels in the Y direction becomes the corresponding pixel of the defective pixel PK1. Such identification of the corresponding pixel is also executed for each other defective pixel in the defective pixel area KA, and the corresponding pixel for each other defective pixel is specified.

  Such identification of the corresponding pixel can also be described by using a movement vector MV representing the displacement of the relative position (the change direction and the change amount of the relative position). Specifically, when the defective pixel of the image sensor 15 (15b) at the correction image GR2 acquisition position is set as the end point of the movement vector MV, the pixel present at the start point of the movement vector is at the main captured image GR1 acquisition position. It becomes a corresponding pixel of the defective pixel of the image sensor 15 (15a). For example, as shown in FIG. 11, the corresponding pixel of the defective pixel PK1 is a pixel PXz existing at the start point of the movement vector MV when the defective pixel PK1 is the end point of the movement vector MV. As described above, the corresponding pixel in the first defective pixel correction can be specified based on the movement vector MV related to the change of the relative position. Note that the movement vector MV representing the displacement of the relative position is acquired using the displacement information HI stored in the ROM 4b.

  Then, by replacing the pixel signal acquired by the defective pixel of the image sensor 15a at the acquisition position of the main captured image GR1 with the pixel signal acquired by the corresponding pixel of the image sensor 15b at the acquisition position of the correction image GR2, the main shooting is performed. The pixel value of the pixel in the defect signal area KS in the image GR1 is corrected.

  The pixel signal replacement is performed using a pixel signal having the same color as that of the defective pixel of the image sensor 15a. Specifically, pixel interpolation processing is performed on the image data acquired by the image sensor 15b, and pixel signals of color signals that are insufficient at each pixel of the image sensor 15b are acquired. In step SP4, the pixel signal of the defective pixel of the image sensor 15a is replaced with a pixel signal having the same color as the defective pixel of the image sensor 15a among the three color signals in the corresponding pixel of the image sensor 15b. For example, when the defective pixel of the image sensor 15a is an R component pixel and the corresponding pixel in the image sensor 15b is a G component pixel, the defective pixel is detected by the R component pixel signal in the corresponding pixel calculated by pixel interpolation. The pixel signal of the pixel is replaced.

  As described above, the imaging apparatus 1A according to the present embodiment is used for correction when the evaluation value ES representing the degree of coincidence between two images is smaller than the first threshold value TH1, that is, when the degree of coincidence is high. Among the pixels in the image GR2, the corresponding pixel having the pixel signal corresponding to the subject part imaged on the defective pixel of the imaging element 15a at the acquisition position of the main captured image GR1 is acquired from the acquisition position of the main captured image GR1. It is specified based on the movement vector MV representing the displacement to the position, and defective pixel correction in the actual captured image GR1 is performed using the pixel signal of the corresponding pixel.

  According to this, when the degree of coincidence between the two images is high, the pixel signal of the defective pixel is corrected using the accurate information of the subject that actually exists as the information regarding the defective pixel. Correction can be executed. Furthermore, since the pixel signal of the defective pixel is corrected using accurate information, it is possible to perform highly accurate correction not only on a defective pixel on a pixel basis but also on a relatively large defective pixel region. .

  Next, the second defective pixel correction executed in step SP6 will be described. FIG. 12 is a diagram illustrating how the image GF2 in the evaluation region HR2 is shifted in the correction image GR2. Note that only the image GF2 in the evaluation region HR2 in FIG. 12 is represented by vertical line hatching. FIG. 12B shows a state in which the subject has changed compared to the subject (FIG. 12A) in the main captured image GR1. FIG. 13 is a diagram illustrating how the change vector HV is acquired. FIG. 14 is a diagram illustrating a specific state of the corresponding pixel using the movement vector MV and the change vector HV.

  In step SP6, first, in the correction image GR2, a region (same region) having information on the same subject part as the subject part included in the evaluation region HR of the main captured image GR1 is specified. Specifically, matching is performed by sequentially calculating correlation values by shifting an image (also referred to as a “shift image”) GF in the evaluation region HR by a minute width (for example, one pixel) within the reference region FR on the correction image GR2. Processing is performed, and a position (matching position) MP where the shift image GF matches in the correction image GR2 is detected. As the correlation value, for example, when the shift image GF is superimposed on the correction image GR2, a pixel signal between each pixel of the shift image GF and each pixel that overlaps each pixel of the shift image GF in the correction image GR2 is used. An absolute value of the difference is calculated, and an average value (or integrated value) of the absolute values of the difference corresponding to the number of pixels in the shift image GF obtained thereby can be employed. Such a correlation value is a small value when the corresponding pixels include the same part of the subject, and a large value when the corresponding pixels are different parts. Therefore, the position where the correlation value is the smallest in the correction image GR2 is detected as the coincidence position MP, and the same region is specified. Note that the coincidence position MP is represented by absolute coordinates indicated by the center pixel of the shift image GF when the shift image GF is arranged at a position where the correlation value is smallest on the correction image GR2.

  Then, a vector (also referred to as “motion vector” or “change vector”) HV starting from the center pixel (here, the reference pixel) of the evaluation region HR in the captured image GR1 and having the coincidence position MP as the end point is acquired. . The change vector HV acquired in this way is also expressed as a vector representing the motion (change) of the subject itself on the captured image between the two imaging operations for the subject.

  For example, if the image (vertical hatched area) in the evaluation area HR2 shown in FIG. 12A is a shift image GF2, the shift image GF2 is the reference area FR of the correction image GR2 as shown in FIG. 12B. (A region surrounded by a broken line) is shifted as indicated by an arrow SY, for example, and a correlation value is calculated at each shift position in the reference region FR. Then, based on the calculated correlation value, the coincidence position MP is detected from each shift position (see FIG. 13), and the absolute coordinates at the coincidence position MP are acquired. When the coincidence position MP is detected, the origin position is the position where the origin pixel OP in the lower left corner of the actual captured image GR1 is the origin, and the coincidence position MP2 starts from the reference pixel CP2 of the evaluation region HR in the actual captured image GR1. Is obtained as a change vector HV. The reference area FR includes, for example, the evaluation area HR and is slightly wider than the evaluation area HR (for example, if the evaluation area HR is an area of 3 pixels × 3 pixels, 9 pixels × 9 pixels). Area) is set. The range (size) of the reference area FR may be changed according to the acquisition interval of two images. Specifically, the reference area FR may be enlarged as the acquisition interval between two images increases.

  Next, a corresponding pixel having a pixel signal corresponding to a subject part imaged on a defective pixel of the imaging element 15a at the acquisition position of the main captured image GR1 is specified from the pixels in the correction image GR2. In the second defective pixel correction, the corresponding pixel is specified based on the movement vector MV and the change vector HV.

  Specifically, first, when the defective pixel of the image sensor 15 (15b) is set as the end point of the movement vector MV, the pixel existing at the start point of the movement vector MV is specified. Then, when the pixel existing at the start point of the movement vector MV is used as the start point of the change vector HV, the pixel existing at the end point of the change vector HV becomes a corresponding pixel of the defective pixel of the image sensor 15 (15a).

  For example, a case where the corresponding pixel of the defective pixel PK1 is specified will be described with reference to FIG. In this case, the defective pixel PK1 of the image sensor 15b is set as the end point of the movement vector MV, and the pixel Pe existing at the start point of the movement vector MV is specified. When the pixel Pe existing at the start point of the movement vector MV is set as the start point of the change vector HV, the pixel PXa existing at the end point of the change vector HV becomes a corresponding pixel of the defective pixel of the image sensor 15a. As described above, the corresponding pixel in the second defective pixel correction is specified based on the movement vector MV related to the change of the relative position and the change vector HV representing the subject change between image acquisitions.

  Then, by replacing the pixel signal acquired by the defective pixel of the imaging element 15a at the acquisition position of the main captured image GR1 with the pixel signal acquired by the corresponding pixel of the imaging element 15b at the acquisition position of the correction image GR2, the actual captured image The pixel value of the pixel in the defect signal area KS in GR1 is corrected.

  As described above, in the imaging apparatus 1A according to the present embodiment, the evaluation value ES indicating the degree of coincidence between two images is equal to or greater than the first threshold value TH1 and smaller than the second threshold value TH2. If the degree of coincidence is within a predetermined range (an intermediate range in which the threshold value TH1 is the upper limit value and the threshold value TH2 is the lower limit value), the subject and the correction at the time of acquisition of the actual captured image GR1 are corrected. A change vector HV representing a change from the subject at the time of acquisition of the image GR2 is acquired, and a corresponding pixel is specified based on the movement vector MV and the change vector HV. Then, the pixel signal acquired by the defective pixel in the actual captured image is corrected using the pixel signal of the corresponding pixel in the correction image.

  According to this, even when a minute subject change occurs between two image acquisitions, a pixel having information on the same subject part as the subject part imaged on the defective pixel is specified, and the defective pixel Since the pixel signal is corrected, it is possible to execute defective pixel correction with high accuracy. In addition, since the pixel signal of the defective pixel is corrected using accurate information on the subject that actually exists as information regarding the defective pixel, not only the defective pixel in units of one pixel but also a relatively large defective pixel region is accurate. High correction can be performed.

  Next, the third defective pixel correction executed in step SP7 will be described. FIG. 15 is a diagram illustrating the peripheral pixels of the defective pixel area KA used for interpolation.

  If it is determined in step SP5 that the evaluation value ES is greater than or equal to the second threshold value TH2, there has been a change in the subject such as movement of the subject during image acquisition between the main captured image GR1 and the correction image GR2. It is judged that

  If there is a subject change between the two image acquisitions, each pixel in the correction image GR2 has a pixel having information on the same subject part as each pixel in the defect signal area KS in the main captured image GR1. It is thought not to. In such a case, rather than correcting the pixel signal of the defective pixel using the pixel signal of the corresponding pixel in the correction image GR2, the pixel signal of the defective pixel is corrected by interpolation using the peripheral pixels of the defective pixel area KA. The correction accuracy is higher.

  Therefore, in step SP7, the defective pixel correction using the correction image GR2 is not performed, and the defect signal region is obtained using information (peripheral information) obtained from the non-defective pixels around the defect signal region KS in the main captured image GR1. The pixel signal of each pixel in KS is corrected.

  In the present embodiment, pixel signals of defective pixels are interpolated using pixel signals acquired by four non-defective pixels existing closest to the defective pixel in four horizontal and vertical directions of the defective pixel. . Specifically, as shown in FIG. 15, a case where the pixel signal of the defective pixel PK1 in the image sensor 15 is corrected will be described in detail.

  First, in the two horizontal directions (+ X direction and −X direction) of the defective pixel PK1, two non-defective pixels Pa1 and Pa2 existing at positions closest to the defective pixel PK1 are specified. Then, linear interpolation is performed using the X coordinate and the pixel signal in the pixel Pa1 and the X coordinate and the pixel signal in the pixel Pa2, and the pixel signal of the defective pixel PK1 obtained from the pixel in the horizontal direction (also referred to as “horizontal pixel signal”). Is obtained). Similarly, in the two vertical directions (+ Y direction and −Y direction) of the defective pixel PK1, two non-defective pixels Pb1 and Pb2 that are present at positions closest to the defective pixel PK1 are specified. Then, linear interpolation using the Y coordinate and pixel signal in the pixel Pb1 and the Y coordinate and pixel signal in the pixel Pb2 is performed, and the pixel signal of the defective pixel PK1 obtained from the pixel in the vertical direction (also referred to as “vertical pixel signal”). Is obtained).

  Next, an average value of the horizontal pixel signal and the vertical pixel signal is calculated, and the average value is determined as a pixel signal of the defective pixel PK1.

  Such defective pixel correction processing by interpolation is executed for each defective pixel in the defective pixel area KA, and the pixel signal acquired by each defective pixel is corrected.

  As described above, in the imaging device 1A according to the present embodiment, when the evaluation value ES indicating the degree of coincidence between two images is equal to or greater than the second threshold value TH2, the defect signal region KS in the captured image GR1. The pixel signal acquired by the defective pixel in the actual captured image is corrected by interpolation using the pixel signal of the adjacent pixel.

  According to this, when the degree of coincidence between the two images is low, the pixel signal of the defective pixel is corrected by interpolation using the pixel signal of the non-defective pixel around the defective pixel. It is possible to avoid defective pixel correction with low accuracy.

<2. Second Embodiment>
Next, a second embodiment of the present invention will be described. The imaging apparatus 1A according to the first embodiment acquires a degree of coincidence by comparing regions that are assumed to have information on the same subject part in two images acquired at different relative positions. However, the imaging apparatus 1B according to the second embodiment acquires the degree of coincidence based on the change vector HV and performs defective pixel correction. FIG. 16 is an overall flowchart of defective pixel correction according to the second embodiment. 17, FIG. 20 and FIG. 25 are diagrams showing change vectors HV1 to HV4 in the respective evaluation regions HR1 to HR4. 18, FIG. 21, and FIG. 26 are diagrams showing the X component values of the change vectors HV1 to HV4 when the reference pixels CP1 to CP4 are used as a reference. 19, 22 and 27 are diagrams showing the Y component values of the change vectors HV1 to HV4 when the reference pixels CP1 to CP4 are used as a reference. 23 and 24 are diagrams illustrating the subject SJ1 in the main captured image GR1 and the subject SJ2 in the correction image GR2.

  The imaging device 1B according to the second embodiment has the same configuration and function as the imaging device 1A according to the first embodiment (see FIGS. 1, 2, 3, and 4). The description will focus on the differences from the embodiment.

  As shown in FIG. 16, in step SP11, as in step SP1 described above, the relative position between the photographic optical system and the image sensor 15 is changed, and two images (the main photographic image GR1 and the correction image GR2) are changed. ) Is acquired.

  In step SP12, the change vector HV is acquired based on the two images. The method of acquiring the change vector HV is the same as that in the first embodiment. Specifically, the image GF in the evaluation region HR (in FIG. 12, the image GF2 in the evaluation region HR2) is shifted by a minute width within the reference region FR on the correction image GR2, and the matching process is performed. A position MP (coincidence position) MP where the shift images GF coincide in GR2 is detected. Then, a change vector HV is obtained that starts from the central pixel (here, the reference pixel) of the evaluation region HR in the captured image GR1 and ends at the central pixel of the shift image GF at the coincidence position MP.

  In step SP12, as shown in FIG. 17, four change vectors HV1 to HV4 are calculated using the shift images GF1 to GF4 for the four evaluation regions HR1 to HR4 (see FIG. 8).

  In the next step SP13, the magnitudes (absolute values) AB1 to AB4 of the four change vectors HV1 to HV4 are calculated.

  In step SP14, the absolute values AB1 to AB4 of the change vectors HV1 to HV4 are respectively compared with a preset threshold value TH3. When the absolute values AB1 to AB4 are smaller than the threshold value TH3, the degree of coincidence between the main captured image GR1 and the correction image GR2 is high, that is, between image acquisition between the main captured image GR1 and the correction image GR2. It is determined that there has been no movement (change) of the subject, and the process proceeds to step SP15.

  In step SP15, the first defective pixel correction is executed in the same manner as in step SP4 (FIG. 6). Specifically, among the pixels in the correction image GR2, a corresponding pixel having a pixel signal corresponding to the subject part imaged on the defective pixel of the image sensor 15a at the acquisition position of the main captured image GR1 is specified. Then, using the pixel signal of the corresponding pixel in the correction image GR2, the pixel signal acquired by the defective pixel of the imaging element 15a in the main captured image GR1 is corrected.

  On the other hand, if at least one of the absolute values AB1 to AB4 is greater than the threshold value TH3, it is determined that the degree of coincidence between the main captured image GR1 and the correction image GR2 is low, and the process proceeds to step SP16. Transition.

  In step SP16, the change vectors HV1 to HV4 are decomposed into two-dimensional components in the X-axis direction and the Y-axis direction, respectively, and index values (“variation degree”, “ BKx and an index value (variation degree) BKy representing a variation regarding the Y direction component of each of the change vectors HV1 to HV4 are calculated.

  For example, if the distance between two adjacent pixels in the horizontal and vertical direction is “1”, component values (X component values) IX1 to IX4 in the X-axis direction of the change vectors HV1 to HV4 shown in FIG. Is also “2” (see FIG. 18). Then, when the absolute value of the difference between the minimum X component value and the maximum X component value among the X component values IX1 to IX4 of the change vectors HV1 to HV4 is defined as the variation degree BKx of the X components of the change vectors HV1 to HV4. The variation degree BKx of the X component of each of the change vectors HV1 to HV4 is “0”. Further, since the component values (Y component values) IY1 to IY4 in the Y-axis direction of the change vectors HV1 to HV4 are all “0” (see FIG. 19), the Y component values of the change vectors HV1 to HV4 are If the absolute value of the difference between the minimum Y component value and the maximum Y component value is the variation degree BKy of the Y component of each change vector HV1 to HV4, the variation degree BKy of the Y component of each change vector HV1 to HV4 is “0”.

  When the change vectors HV1 to HV4 as shown in FIG. 20 are acquired, the X component values IX1, IX2, and IX4 of the change vectors HV1, HV2, and HV4 are “2”, respectively, and the change vector HV3 The X component value IX3 is “3” (see FIG. 21). The variation degree BKx of the X component of each of the change vectors HV1 to HV4 is “1”. Further, the Y component values IY1, IY2, and IY4 of the change vectors HV1, HV2, and HV4 are “0”, and the Y component value IY3 of the change vector HV3 is “1” (see FIG. 22). The variation degree BKy of the Y component of each of the change vectors HV1 to HV4 is “1”.

  The variation degrees BKx and BKy of the two-dimensional components of the respective change vectors HV1 to HV4 obtained in this way are also expressed as representing a change mode of the subject.

  Specifically, when the subject change between image acquisitions does not affect the shape of the subject, the variation degree of each change vector HV in a plurality of subject parts is small. For example, as shown in FIG. 23, when the subject moves in parallel between the image acquisitions, the shape change between the subject SJ1 in the main captured image GR1 and the subject SJ2 in the correction image GR2 becomes minute, and the degree of variation becomes small. .

  On the other hand, when the subject change between image acquisitions includes a change in the shape of the subject, the variation degree of each change vector HV in a plurality of subject parts increases. For example, as shown in FIG. 24, when the shape change between the subject SJ1 in the main captured image GR1 and the subject SJ2 in the correction image GR2 is large, the degree of variation increases.

  Note that the shape change of the subject includes, for example, the change of the subject itself due to the movement of the person's hand or foot, the change of the face direction, and further, the change of the subject in the acquired image due to the subject approaching or moving away. The change of a magnitude | size etc. are mentioned.

  As described above, the variation degrees BKx and BKy represent the manner in which the subject changes during image acquisition. In the following steps, defective pixel correction is performed (selected) according to the variation degrees BKx and BKy.

  Specifically, in step SP17, whether or not the variations of the X component and the Y component are both within a preset range (predetermined range) LM, more specifically, the variation degree of the X component and the Y component. It is determined whether both BKx and BKy are smaller than threshold value TH4. When the variation degrees BKx and BKy of the X component and the Y component are both smaller than the threshold value TH4, the subject change that occurs during image acquisition between the main captured image GR1 and the correction image GR2 changes the shape of the subject. It is determined that the change is a minute change (for example, a change due to the parallel movement of the subject as shown in FIG. 23), and the process proceeds to step SP18.

  In step SP18, as in step SP6 (FIG. 6), the second defective pixel correction is performed. Specifically, based on the movement vector MV and the change vector HV, from among each pixel of the image sensor 15b at the correction image GR2 acquisition position, each defective pixel of the image sensor 15a at the main captured image GR1 acquisition position. Corresponding pixels are specified, and pixel signals acquired from defective pixels of the image sensor 15a in the actual captured image GR1 are corrected using pixel signals of the corresponding pixels.

  On the other hand, when at least one of the variations BKx and BKy of the X component and the Y component is greater than or equal to the threshold value TH4, this occurred between image acquisition of the main captured image GR1 and the correction image GR2. It is determined that the subject change is a large change that changes the shape of the subject, and the process proceeds to step SP19.

  For example, when change vectors HV1 to HV4 as shown in FIG. 25 are acquired in step SP16, the minimum X component value IX2 and the maximum of the X component values IX1 to IX4 of the change vectors HV1 to HV4 are obtained. Since the absolute value of the difference from the X component value IX1 is “4” (see FIG. 26), the variation degree BKx of the X component of each change vector HV1 to HV4 is “4”. Also, the absolute value of the difference between the minimum Y component value IY2 and the maximum Y component value IY1 among the X component values IX1 to IX4 of the change vectors HV1 to HV4 is “3” (see FIG. 27). The variation degree BKy of the X component of each of the change vectors HV1 to HV4 is “3”. Here, if the threshold value TH4 is set to “4”, the variation degree BKx of the X component is equal to or greater than the threshold value TH4. That is, in step SP17, it is determined that the subject change that occurred between the two image acquisitions changes the shape of the subject, and the process proceeds to step SP19.

  In step SP19, the third defective pixel correction is executed as in step SP7 (FIG. 6). Specifically, the pixel signal of each pixel in the defect signal region KS is corrected using information (peripheral information) obtained from non-defective pixels (pixels having no defect) around the defect signal region KS in the actual captured image GR1. Is done.

  As described above, the imaging device 1B according to the second embodiment specifies the corresponding regions for a plurality of regions in the main captured image GR1 in the correction image GR2, thereby allowing each subject part (a plurality of regions in the plurality of regions to be a plurality of regions). A change vector representing a change state (degree of change) of the (subject part) is acquired. Then, as an index of the degree of coincidence between the subject in the main captured image GR1 and the subject in the correction image GR2, information on the degree of coincidence (specifically, the absolute value of each change vector HV and the variation in the two-dimensional component of each change vector HV) Degree) (also referred to as “matching degree information”), and the most effective defective pixel correction is executed in accordance with the matching degree information. According to this, it is possible to execute appropriate defective pixel correction according to the degree of coincidence between the subject in the main captured image GR1 and the subject in the correction image GR2.

  Further, as in the present embodiment, by using the degree of variation, it is possible to specify not only whether or not there is a change in the subject but also how the subject changes, in other words, how the subject has changed. Therefore, it is possible to perform more appropriate defective pixel correction according to the type (mode) of subject change. For example, when the imaging device 1A and the imaging device 1B are compared with respect to the defective pixel correction operation performed when the subject moves relatively large during image acquisition (see FIG. 23), the imaging device 1A evaluates the evaluation value ES. However, in the imaging apparatus 1B, the absolute value of the change vector HV is large, but the degree of variation is small. Therefore, the second defective pixel correction is executed. Is likely to be. That is, in the imaging apparatus 1B, it is possible to perform the defective pixel correction using the second defective pixel correction having higher correction accuracy than the third defective pixel correction in consideration of the change of the subject.

<3. Modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the contents described above.

  For example, in the first embodiment, the evaluation value ES indicating the degree of coincidence is calculated using the pixel value of the G component of the evaluation pixel and the pixel value of the G component of the corresponding evaluation pixel. However, the present invention is not limited to this. Instead, it may be calculated using the luminance value of the evaluation pixel and the luminance value of the corresponding evaluation pixel.

  In the first embodiment, the change vector HV is calculated based on the image (shifted image) GF2 of the evaluation region HR2 among the four evaluation regions HR1 to HR4. However, the present invention is not limited to this. Specifically, four vectors may be calculated using the shift images GF1 to GF4 for the four evaluation regions HR1 to HR4, and an average vector of these four vectors may be used as the change vector. More specifically, the average coordinates obtained by averaging the absolute coordinates of the start points of each vector calculated for each shift image and the average coordinates obtained by averaging the absolute coordinates of the end points of each vector are respectively the start points. The vector that is the end point may be adopted as the change vector.

  In the first embodiment, the evaluation value ES is compared with the two threshold values TH1 and TH2, and three defect correction methods are selected based on the comparison result. The value ES may be compared with one threshold value, and two defect correction methods may be selected based on the comparison result.

  In the second embodiment, in step SP17 (FIG. 16), the degree of variation BKx related to the X direction component of each change vector HV1 to HV4 and the degree of variation BKy related to the Y direction component of each change vector HV1 to HV4 are respectively determined. Although it is compared with the threshold value TH4, the present invention is not limited to this, and any one degree of variation is compared with the threshold value TH4 (that is, a defect correction method is selected using either degree of variation). May be.

  In each of the above embodiments, the first defective pixel correction is distinguished from the second defective pixel correction. However, the first defective pixel correction is a correction method included in the second defective pixel correction. It can be expressed as Specifically, the process executed when the magnitude of the change vector is “zero” in the second defective pixel correction corresponds to the process executed in the first defective pixel correction.

1 is a diagram illustrating an external configuration of an imaging apparatus according to a first embodiment of the present invention. 1 is a diagram illustrating an external configuration of an imaging apparatus according to a first embodiment of the present invention. It is a longitudinal cross-sectional view of an imaging device. It is a block diagram which shows the function structure of an imaging device. It is a figure which shows the defective pixel in an image sensor. It is a whole flowchart regarding defective pixel correction. It is a figure which shows two images acquired by changing the relative position of an imaging optical system and an image pick-up element. It is a figure which shows the evaluation area | region for acquiring a coincidence degree. It is a figure which shows the area | region corresponding to an evaluation area | region in the image for correction | amendment. It is a figure which shows the mode of a movement of an image pick-up element. It is a figure which shows the specific mode of the corresponding pixel using a movement vector. It is a figure which shows a mode that the image of an evaluation area | region is shifted in the image for correction | amendment. It is a figure which shows a mode that a change vector is acquired. It is a figure which shows the specific mode of the corresponding pixel using a movement vector and a change vector. It is a figure which shows the peripheral pixel of the defective pixel area | region used for interpolation. It is a whole flowchart of defective pixel correction concerning a 2nd embodiment of the present invention. It is a figure which shows each change vector in each evaluation area | region. It is a figure which shows the X component value of each change vector at the time of making each reference | standard pixel into the reference | standard. It is a figure which shows the Y component value of each change vector at the time of making each reference | standard pixel into the reference | standard. It is a figure which shows each change vector in each evaluation area | region. It is a figure which shows the X component value of each change vector at the time of making each reference | standard pixel into the reference | standard. It is a figure which shows the Y component value of each change vector at the time of making each reference | standard pixel into the reference | standard. It is a figure which shows the to-be-photographed object and the to-be-photographed object in a correction image. It is a figure which shows the to-be-photographed object and the to-be-photographed object in a correction image. It is a figure which shows each change vector in each evaluation area | region. It is a figure which shows the X component value of each change vector at the time of making each reference | standard pixel into the reference | standard. It is a figure which shows the Y component value of each change vector at the time of making each reference | standard pixel into the reference | standard.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image pick-up device 11 Release button 15 Image pick-up element GR1 Main image | photographed image GR2 Image for correction KA Defect pixel area KS Defect signal area MV Movement vector HV, HV1-HV4 Change vector BKx, BKy Variability

Claims (11)

An imaging apparatus having an imaging optical system,
An image sensor having defective pixels and acquiring a captured image related to the subject image;
Position changing means for changing the relative position of the image sensor with respect to the photographing optical system from a first relative position to a second relative position;
An imaging control unit that acquires the first captured image by the image sensor at the first relative position and acquires the second captured image by the image sensor at the second relative position using the position changing unit; ,
A degree of coincidence acquisition means for obtaining a degree of coincidence between the first captured image and the second captured image;
In the first captured image, defective pixel correction means for correcting a pixel signal acquired by the defective pixel;
Correction control means for executing defective pixel correction according to the degree of coincidence in the defective pixel correction means;
An imaging apparatus comprising: The imaging apparatus according to claim 1,
The defective pixel correction means includes
Among the pixels in the second photographed image, the corresponding pixel having a pixel signal corresponding to the subject part imaged on the defective pixel of the image sensor at the first relative position is selected from the first relative position. A first defect correction unit that is identified based on a movement vector representing a displacement to a second relative position, and performs the defective pixel correction in the first captured image using a pixel signal of the corresponding pixel;
Have
The image pickup apparatus, wherein the correction control unit causes the defective pixel correction to be performed using the first defect correction unit when the degree of coincidence is higher than a predetermined threshold value. The imaging apparatus according to claim 1,
The defective pixel correction means includes
Change acquisition means for acquiring a change vector representing a change between the subject at the time of acquiring the first captured image and the subject at the time of acquiring the second captured image;
Among the pixels in the second photographed image, the corresponding pixel having a pixel signal corresponding to the subject part imaged on the defective pixel of the image sensor at the first relative position is selected from the first relative position. A second defect correction that is specified based on the movement vector representing the displacement to the second relative position and the change vector, and performs the defective pixel correction in the first captured image using the pixel signal of the corresponding pixel. Means,
Have
The image pickup apparatus, wherein the correction control unit causes the defective pixel correction to be performed using the second defect correction unit when the degree of coincidence is within a predetermined range. The imaging device according to claim 3.
The defective pixel correction means includes
Third defect correction means for correcting the defective pixel in the first photographed image by interpolation using pixel signals of non-defective pixels around the defective pixel in the image sensor;
Further comprising
The image pickup apparatus, wherein the correction control unit causes the defective pixel correction to be performed using the third defect correction unit when the degree of coincidence is lower than a lower limit value in the predetermined range. The imaging apparatus according to claim 4,
The defective pixel correction means includes
A first defect that identifies the corresponding pixel among the pixels in the second captured image based on the movement vector, and performs the defective pixel correction in the first captured image using a pixel signal of the corresponding pixel. Correction means,
Further comprising
The image pickup apparatus, wherein the correction control unit causes the defective pixel correction to be performed using the first defect correction unit when the degree of coincidence is higher than an upper limit value in the predetermined range. The imaging apparatus according to claim 1,
Change acquisition means for acquiring a change vector representing a change between the subject at the time of acquiring the first captured image and the subject at the time of acquiring the second captured image;
Further comprising
The matching degree acquisition means
Information acquisition means for acquiring information on the degree of coincidence based on the change vector;
An imaging device comprising: The imaging device according to claim 6,
The information acquisition means calculates a magnitude of the change vector;
The defective pixel correction means includes
Among the pixels in the second photographed image, the corresponding pixel having a pixel signal corresponding to the subject part imaged on the defective pixel of the image sensor at the first relative position is selected from the first relative position. A first defect correction unit that is identified based on a movement vector representing a displacement to a second relative position, and performs the defective pixel correction in the first captured image using a pixel signal of the corresponding pixel;
Have
The imaging apparatus according to claim 1, wherein the correction control unit causes the defective pixel correction to be performed using the first defect correction unit when the magnitude of the change vector is smaller than a predetermined threshold value. The imaging device according to claim 6,
The change acquisition means acquires the change vectors for a plurality of parts related to the subject,
The information acquisition means includes
A variation degree obtaining means for obtaining a variation degree with respect to each change vector relating to the plurality of portions;
Have
The defective pixel correction means includes
Among the pixels in the second photographed image, the corresponding pixel having a pixel signal corresponding to the subject part imaged on the defective pixel of the image sensor at the first relative position is selected from the first relative position. A second defect correction that is specified based on the movement vector representing the displacement to the second relative position and the change vector, and performs the defective pixel correction in the first captured image using the pixel signal of the corresponding pixel. means,
Have
The image pickup apparatus, wherein the correction control unit causes the defective pixel correction to be performed using the second defect correction unit when the degree of variation is smaller than a predetermined threshold value. The imaging device according to claim 8,
The defective pixel correction means includes
Third defect correction means for correcting the defective pixel in the first photographed image by interpolation using pixel signals of non-defective pixels around the defective pixel in the image sensor;
Further comprising
The image pickup apparatus, wherein the correction control unit causes the defective pixel correction to be performed using the third defect correction unit when the degree of variation is larger than the predetermined threshold value. A method for correcting a defective pixel of a captured image acquired by an image sensor having a defective pixel,
a) changing the relative position of the image sensor relative to the imaging optical system from a first relative position to a second relative position;
b) obtaining a first photographed image by the image sensor at the first relative position and obtaining a second photographed image by the image sensor at the second relative position;
c) obtaining a degree of coincidence between the first captured image and the second captured image;
d) correcting the pixel signal acquired by the defective pixel in the first captured image;
With
In the step d), defective pixel correction according to the degree of coincidence is performed. An imaging apparatus having an imaging optical system,
An image sensor having defective pixels and acquiring a captured image related to the subject image;
Position changing means for changing the relative position of the image sensor with respect to the photographing optical system from a first relative position to a second relative position;
An imaging control unit that acquires the first captured image by the image sensor at the first relative position and acquires the second captured image by the image sensor at the second relative position using the position changing unit; ,
Change acquisition means for acquiring a change vector representing a change between the subject at the time of acquiring the first captured image and the subject at the time of acquiring the second captured image;
Among the pixels in the second photographed image, the corresponding pixel having a pixel signal corresponding to the subject part imaged on the defective pixel of the image sensor at the first relative position is selected from the first relative position. A specifying means for specifying based on a movement vector representing a displacement to a second relative position and the change vector;
Defect correcting means for performing defective pixel correction in the first captured image using a pixel signal of the corresponding pixel;
An imaging apparatus comprising:

Images