WO2016121928A1 - Image detection device, image detection method and image capture device - Google Patents

Abstract

An image detection device comprises: a movement means that physically moves, on the basis of a pixel arrangement of an image pickup element having a transmission wavelength selection element, at least one of an optical element, which is a part of an image capture optical system, and the image pickup element, thereby moving the position of a subject image on the light reception surface of the image pickup element; a signal generation means that picks up, each time the position of the subject image is moved, the subject image taken in the image pickup element and that performs predetermined signal processings of the subject image as picked up, which include a color interpolation processing, thereby generating color difference signals; and a detection means that detects, on the basis of at least one pair of color difference signals for which the positions of the subject image on the light reception surface of the image pickup element are different from each other, an image degradation occurring in a captured image.

Description

Image detecting apparatus, image detecting method, and photographing apparatus

The present invention relates to an image detection device, an image detection method, and a photographing device that detect image deterioration in an image.

It is known that when a subject including a high-frequency component equal to or higher than the pixel pitch of the image sensor is picked up, a moiré such as a false color (color moire) is generated and a captured image is deteriorated. Therefore, various techniques for removing this type of false color have been proposed. For example, Japanese Patent Application Laid-Open No. 2011-109494 (hereinafter referred to as “Patent Document 1”) describes a specific configuration of a photographing apparatus capable of detecting a false color generated in a photographed image.

The imaging apparatus described in Patent Document 1 images a focused subject, and then images a non-focused subject. In the out-of-focus photographed image, the contrast of the subject is reduced and high-frequency components are reduced, so that the false color generated in the in-focus photographed image is also reduced. Therefore, the imaging apparatus described in Patent Document 1 calculates a difference value between the color difference signal of the captured image in the focused state and the color difference signal of the captured image in the out-of-focus state for each block, and falsely calculates a block having a large difference value. Detect as a block with color.

As described above, the imaging apparatus described in Patent Document 1 detects a subject with high contrast and a false color by detecting a subject with low contrast and a reduced false color by comparing processing. The occurrence of color is detected. However, since the occurrence of false color is only reduced in the in-focus shot image compared to the in-focus shot image, even in a block where the difference value of the color difference signal is false. Don't get big. Therefore, it is difficult for the photographing apparatus described in Patent Document 1 to detect false colors with high accuracy.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image detection apparatus, an image detection method, and a photographing apparatus capable of accurately detecting image degradation.

An image detection apparatus according to an embodiment of the present invention relates a subject image incident on an image sensor having a predetermined pixel arrangement via a photographing optical system and the image sensor relative to a light receiving surface direction of the image sensor. A moving means for moving the position, a signal generating means for picking up a subject image in a state where the relative positions are different to generate a color difference signal, and a detection for detecting image deterioration occurring in the captured image based on the color difference signal Means.

In addition, the image detection apparatus according to an embodiment of the present invention physically connects at least one of the optical elements in the imaging optical system and at least one of the imaging elements based on the pixel arrangement of the imaging element with the transmission wavelength selection element. Moving means for moving the position of the subject image on the light receiving surface of the image sensor by moving, and capturing the subject image captured by the image sensor every time the position of the subject image is moved, and color interpolation processing Image generation that occurs in a captured image based on at least a pair of color difference signals that are different from each other in position of the subject image on the light receiving surface of the image sensor, and a signal generation unit that generates a color difference signal by performing predetermined signal processing including And detecting means for detecting.

According to one embodiment of the present invention, by moving the position of the subject image on the light receiving surface of the image sensor, image degradation of different colors occurs in the region including high frequency components in the captured images before and after the movement, Changes between captured images tend to be large. Therefore, it is possible to detect image degradation with high accuracy.

Further, in one embodiment of the present invention, the moving means is arranged on the light receiving surface of the image sensor so that the color information of at least the pair of color difference signals has a complementary color relationship in the generation region where the image degradation occurs. The position of the subject image may be moved by an amount corresponding to the pixel interval.

In one embodiment of the present invention, the moving means moves the position of the subject image on the light receiving surface of the image sensor for n pixels or (m + 0.5) pixels in a direction corresponding to the pixel arrangement (where n = The distance may be moved by an odd natural number, m = 0 or an odd natural number).

In one embodiment of the present invention, the detection means includes the same subject image incident on the pixel having the same address with respect to at least a pair of captured images having different positions of the subject image on the light receiving surface of the image sensor. The address of each pixel constituting one captured image is converted in accordance with the amount of movement of the position of the subject image, and the color difference signal of one captured image and the other captured image are converted. A configuration may be adopted in which image degradation is detected for each pixel based on the color difference signal.

In one embodiment of the present invention, the detection unit may calculate at least one of a difference value and an addition value of at least a pair of color difference signals and detect image deterioration based on the calculation result.

In one embodiment of the present invention, the detection unit calculates a difference value of at least a pair of color difference signals for each pixel, and image degradation occurs in a pixel in which the calculated difference value is equal to or greater than a first threshold value. It is good also as a structure detected as a pixel of an area | region.

In one embodiment of the present invention, the detection unit calculates an addition value of at least a pair of color difference signals for each pixel, and image degradation occurs in a pixel in which the calculated addition value is equal to or less than a second threshold value. It is good also as a structure detected as a pixel of an area | region.

In one embodiment of the present invention, the signal generation unit generates a luminance signal that forms a pair with the color difference signal by performing predetermined signal processing, and the detection unit performs image processing based on at least a pair of luminance signals. It is good also as a structure which detects deterioration.

An imaging device according to an embodiment of the present invention is an imaging device having the above-described image detection device, and includes a stationary state determination unit that determines whether or not the imaging device is in a stationary state. When it is determined by the state determination means that the photographing apparatus is in a stationary state, image deterioration is detected by the image detection apparatus.

Further, a photographing apparatus according to an embodiment of the present invention may be a photographing apparatus having the above-described image detection device, and may include a shake detection unit that detects a shake of the photographing apparatus. In this case, the moving means physically moves at least one of some of the optical elements and the imaging element in the imaging optical system based on the shake of the imaging apparatus detected by the shake detection means, thereby causing the shake of the imaging apparatus. The image blur caused by the image is corrected.

In addition, the image detection method according to an embodiment of the present invention physically connects a part of the optical elements in the photographing optical system and at least one of the image sensors based on the pixel arrangement of the image sensor with the transmission wavelength selection element. The step of moving the position of the subject image on the light receiving surface of the image sensor by moving the image, and the subject image captured by the image sensor is captured every time the position of the subject image is moved, and color interpolation processing is performed. Generating a color difference signal by performing predetermined signal processing, and detecting image deterioration occurring in the captured image based on at least a pair of color difference signals having different positions of the subject image on the light receiving surface of the image sensor. Performing steps.

According to an embodiment of the present invention, there are provided an image detection device, an image detection method, and an imaging device that can detect image degradation with high accuracy.

It is a block diagram which shows the structure of the imaging device of embodiment of this invention. 1 is a diagram schematically illustrating a configuration of an image shake correction apparatus provided in an imaging apparatus according to an embodiment of the present invention. 1 is a diagram schematically illustrating a configuration of an image shake correction apparatus provided in an imaging apparatus according to an embodiment of the present invention. It is a figure which assists description of the LPF drive in embodiment of this invention. It is a figure which shows the false color detection flow by the system controller in embodiment of this invention. It is a figure regarding the to-be-photographed object image | photographed in embodiment of this invention. It is a figure regarding the to-be-photographed object image | photographed in embodiment of this invention. It is a figure which shows each color difference signal plotted in the color space of embodiment of this invention. It is a figure which shows the false color detection flow by the system controller in another embodiment.

Hereinafter, a photographing apparatus according to an embodiment of the present invention will be described with reference to the drawings. In the following, a digital single lens reflex camera will be described as an embodiment of the present invention. Note that the photographing apparatus is not limited to a digital single lens reflex camera, but, for example, a mirrorless single lens camera, a compact digital camera, a video camera, a camcorder, a tablet terminal, a PHS (Personal Handy Phone system), a smartphone, a feature phone, a portable game machine, The apparatus may be replaced with another type of apparatus having a photographing function.

[Configuration of the entire photographing apparatus 1]
FIG. 1 is a block diagram showing a configuration of the photographing apparatus 1 of the present embodiment. As shown in FIG. 1, the photographing apparatus 1 includes a system controller 100, an operation unit 102, a drive circuit 104, a photographing lens 106, a diaphragm 108, a shutter 110, an image shake correction device 112, a signal processing circuit 114, and an image processing engine 116. , Buffer memory 118, card interface 120, LCD (Liquid Crystal Display) control circuit 122, LCD 124, ROM (Read Only Memory) 126, gyro sensor 128, acceleration sensor 130, geomagnetic sensor 132, and GPS (Global Positioning System) sensor 134 It has. Although the photographing lens 106 has a plurality of lenses, it is shown as a single lens for convenience in FIG. Further, the same direction as the optical axis AX of the photographing lens 106 is defined as the Z-axis direction, and the two axial directions orthogonal to the Z-axis direction and orthogonal to each other are the X-axis direction (horizontal direction) and the Y-axis direction (vertical direction), respectively. It is defined as

The operation unit 102 includes various switches necessary for the user to operate the photographing apparatus 1, such as a power switch, a release switch, and a photographing mode switch. When the user operates the power switch, power is supplied from the battery (not shown) to the various circuits of the photographing apparatus 1 through the power line.

The system controller 100 includes a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). After supplying power, the system controller 100 accesses the ROM 126, reads out a control program, loads it into a work area (not shown), and executes the loaded control program to control the entire photographing apparatus 1.

When the release switch is operated, the system controller 100 detects, for example, a photometric value calculated based on an image captured by a solid-state imaging device 112a (see FIG. 2 described later) or an exposure meter built in the photographing apparatus 1. The diaphragm 108 and the shutter 110 are driven and controlled via the drive circuit 104 so that proper exposure is obtained based on the photometric value measured in (not shown). More specifically, drive control of the aperture 108 and the shutter 110 is performed based on an AE function specified by the shooting mode switch, such as a program AE (Automatic Exposure), shutter priority AE, aperture priority AE, or the like. Further, the system controller 100 performs AF (Autofocus) control together with AE control. For AF control, an active method, a phase difference detection method, an image plane phase difference detection method, a contrast detection method, or the like is applied. In addition, the AF mode includes a central single-point ranging mode using a single central ranging area, a multi-point ranging mode using a plurality of ranging areas, a full-screen ranging mode based on full-screen distance information, and the like. is there. The system controller 100 controls driving of the photographing lens 106 via the driving circuit 104 based on the AF result, and adjusts the focus of the photographing lens 106. Since the configuration and control of this type of AE and AF are well known, detailed description thereof is omitted here.

2 and 3 are diagrams schematically showing a configuration of the image blur correction device 112. FIG. As shown in FIGS. 2 and 3, the image blur correction device 112 includes a solid-state image sensor 112a. The light beam from the subject passes through the photographing lens 106, the diaphragm 108, and the shutter 110, and is received by the light receiving surface 112aa of the solid-state imaging device 112a. Note that the light receiving surface 112aa of the solid-state imaging device 112a is an XY plane including the X axis and the Y axis. The solid-state imaging device 112a is a single-plate color CCD (Charge-Coupled Device) image sensor having a color filter with a Bayer-type pixel arrangement as a transmission wavelength selection device. The solid-state imaging device 112a accumulates an optical image formed by each pixel on the light receiving surface 112aa as a charge corresponding to the amount of light, and generates R (Red), G (Green), and B (Blue) image signals. Output. Note that the solid-state imaging device 112a is not limited to a CCD image sensor, and may be replaced with a CMOS (Complementary Metal Oxide や Semiconductor) image sensor or other types of imaging devices. The solid-state imaging device 112a may be a filter having a complementary color filter, or a filter having a periodic color arrangement such as a Bayer arrangement if any color filter is arranged for each pixel. There is no need.

The signal processing circuit 114 performs predetermined signal processing such as clamping and demosaicing (color interpolation) on the image signal input from the solid-state imaging device 112a, and outputs the processed signal to the image processing engine 116. The image processing engine 116 performs predetermined signal processing such as matrix operation, Y / C separation, and white balance on the image signal input from the signal processing circuit 114 to generate a luminance signal Y and color difference signals Cb and Cr. , Compressed in a predetermined format such as JPEG (Joint Photographic 等 Experts Group). The buffer memory 118 is used as a temporary storage location for processing data when the image processing engine 116 executes processing. The captured image storage format is not limited to the JPEG format, and may be a RAW format in which minimal image processing (for example, clamping) is performed.

The memory card 200 is detachably inserted into the card slot of the card interface 120.

The image processing engine 116 can communicate with the memory card 200 via the card interface 120. The image processing engine 116 stores the generated compressed image signal (captured image data) in the memory card 200 (or a built-in memory (not shown) provided in the image capturing apparatus 1).

Also, the image processing engine 116 buffers the generated luminance signal Y and color difference signals Cb and Cr in a frame memory (not shown) in units of frames. The image processing engine 116 sweeps the buffered signal from each frame memory at a predetermined timing, converts it into a video signal of a predetermined format, and outputs it to the LCD control circuit 122. The LCD control circuit 122 modulates and controls the liquid crystal based on the image signal input from the image processing engine 116. Thereby, the photographed image of the subject is displayed on the display screen of the LCD 124. The user can view through a display screen of the LCD 124 a real-time through image (live view) captured with appropriate brightness and focus based on AE control and AF control.

When the user performs a reproduction operation of the photographed image, the image processing engine 116 reads the photographed image data designated by the operation from the memory card 200 or the built-in memory, converts it into an image signal of a predetermined format, and the LCD control circuit 122. Output to. The LCD control circuit 122 performs modulation control on the liquid crystal based on the image signal input from the image processing engine 116, so that a captured image of the subject is displayed on the display screen of the LCD 124.

[Explanation regarding the drive of the shake correction member]
The image shake correction device 112 drives a shake correction member. In the present embodiment, the shake correction member is the solid-state image sensor 112a. Note that the shake correction member is not limited to the solid-state image sensor 112a, but is partially moved with respect to the optical axis AX, such as a part of lenses included in the photographing lens 106, so that the light-receiving surface 112aa of the solid-state image sensor 112a. Another configuration capable of shifting the incident position of the subject image above may be used, or a configuration in which two or more members of these and the solid-state imaging device 112a are combined may be used.

The image blur correction device 112 not only slightly drives (vibrates) the blur correction member in a plane orthogonal to the optical axis AX (that is, in the XY plane) in order to correct the image blur. The shake correction member is driven minutely in a plane perpendicular to the optical axis AX so that an optical low-pass filter (LPF) effect (reduction of moiré such as false color) can be obtained. Rotate). Hereinafter, for convenience of description, driving the shake correction member with image shake correction is referred to as “image shake correction drive”, and driving the shake correction member so as to obtain the same effect as an optical LPF is referred to as “LPF”. "Drive".

(Explanation on image blur correction drive)
The gyro sensor 128 is a sensor that detects information for controlling image blur correction. Specifically, the gyro sensor 128 detects angular velocities around the two axes (around the X axis and around the Y axis) applied to the imaging apparatus 1, and the detected angular velocities around the two axes are within the XY plane (in other words, a solid state This is output to the system controller 100 as a shake detection signal indicating the shake of the light receiving surface 112aa of the image sensor 112a.

As shown in FIGS. 2 and 3, the image blur correction device 112 includes a fixed support substrate 112b fixed to a structure such as a chassis included in the photographing device 1. The fixed support substrate 112b slidably supports the movable stage 112c on which the solid-state imaging device 112a is mounted.

Magnets M _YR , M _YL , M _XD , and M _XU are attached on the surface of the fixed support substrate 112 b facing the movable stage 112 c. In addition, yokes Y _YR , Y _YL , Y _XD , and Y _{XU that} are magnetic bodies are attached to the fixed support substrate 112b. The yokes Y _YR , Y _YL , Y _XD , and Y _XU each have a shape that extends from the fixed support substrate 112 b to the position facing the magnets M _YR , M _YL , M _XD , and M _XU around the movable stage 112 c, A magnetic circuit is formed between the magnets M _YR , M _YL , M _XD , and M _XU . In addition, driving coils C _YR , C _YL , C _XD , and C _XU are attached to the movable stage 112c. When the driving coils C _YR , C _YL , C _XD , and C _XU receive a current in the magnetic field of the magnetic circuit, a driving force is generated. The movable stage 112c (solid-state imaging device 112a) is minutely driven in the XY plane with respect to the fixed support substrate 112b by the generated driving force.

Corresponding magnets, yokes and driving coils constitute a voice coil motor. Hereinafter, for the sake of convenience, a reference numeral VCM _YR is given to a voice coil motor made up of a magnet M _YR , a yoke Y _YR and a driving coil C _YR , and a voice coil motor made up of a magnet M _YL , a yoke Y _YL and a driving coil C _YL is called up. given the VCM _YL, a voice coil motor reference numeral _{VCM XD} magnet _{M XD,} the voice coil motor consisting of a yoke _{Y XD} and the driving coil _{C XD,} consisting of the magnet _{M XU,} yoke _{Y XU} and the driving coil _{C XU} The symbol VCM _XU is appended.

Each voice coil motor VCM _YR , VCM _YL , VCM _XD , VCM _XU (drive coils C _YR , C _YL , C _XD , C _XU ) is PWM (Pulse Width Modulation) driven under the control of the system controller 100. The voice coil motors VCM _YR and VCM _YL are arranged below the solid-state image sensor 112a and arranged side by side with a predetermined interval in the horizontal direction (X-axis direction). The voice coil motors VCM _XD and VCM _XU are On the side of the solid-state image sensor 112a, they are arranged side by side with a predetermined interval in the vertical direction (Y-axis direction).

Hall elements H _YR , H _YL , H _XD , and H _XU are attached to the positions near the driving coils C _YR , C _YL , C _XD , and C _XU on the fixed support substrate 112b. The Hall elements H _YR , H _YL , H _XD , and H _XU detect the magnetic _forces of the magnets M _YR , M _YL , M _XD , and M _XU , respectively, and the position of the movable stage 112 c (solid-state imaging element 112 a) in the XY plane. Is output to the system controller 100. Specifically, the Y-axis direction position and tilt (rotation) of the movable stage 112c (solid-state imaging device 112a) are detected by the Hall elements H _YR and H _YL , and the movable stage 112c (solid-state imaging) is detected by the Hall elements H _XD and H _XU. The X-axis direction position and inclination (rotation) of the element 112a) are detected.

The system controller 100 includes a driver IC for a voice coil motor. The system controller 100 determines the rated power (allowable power) of the driver IC based on the shake detection signal output from the gyro sensor 128 and the position detection signal output from the Hall elements H _YR , H _YL , H _XD , and H _XU. Calculate the duty ratio so as not to disturb the balance of the current flowing through each voice coil motor VCM _YR , VCM _YL , VCM _XD , VCM _XU (drive coil C _YR , C _YL , C _XD , C _XU ) within the range not exceeding To do. The system controller 100 sends a drive current to each of the voice coil motors VCM _YR , VCM _YL , VCM _XD , and VCM _XU with the calculated duty ratio to drive the solid-state imaging device 112 a with image blur correction. As a result, the image blur on the light receiving surface 112aa of the solid-state image sensor 112a is corrected while the solid-state image sensor 112a is held at a predetermined position against gravity, disturbance, or the like (in other words, on the light receiving surface 112aa). The position of the solid-state imaging device 112a is adjusted so that the incident position of the subject image does not fluctuate.

(Explanation regarding LPF drive)
Next, the LPF driving will be described. In the present embodiment, the image blur correction device 112 causes a predetermined drive current to flow through the voice coil motors VCM _YR , VCM _YL , VCM _XD , and VCM _XU , so that the image shake correction device 112 is predetermined in the XY plane for one exposure period. The movable stage 112c (solid-state image sensor 112a) is driven so as to draw a trajectory, and the subject image is incident on a plurality of pixels having different detection colors (R, G, or B) of the solid-state image sensor 112a. Thereby, the same effect as an optical LPF can be obtained.

4 (a) and 4 (b) are diagrams for assisting the description of the LPF drive. As shown in the figure, on the light receiving surface 112aa of the solid-state imaging device 112a, a plurality of pixels PIX are arranged in a matrix at a predetermined pixel pitch P. For convenience of explanation, each pixel PIX in the figure is given a reference (any one of R, G, and B) corresponding to the filter color arranged on the front surface.

FIG. 4A shows an example in which the solid-state imaging device 112a is driven so as to draw a square locus centered on the optical axis AX. For example, the square locus can be a square closed path with the pixel pitch P of the solid-state imaging element 112a as one side. In the example of FIG. 4A, the solid-state imaging device 112a is driven so as to form a square path alternately in units of one pixel pitch P in the X-axis direction and the Y-axis direction.

FIG. 4B shows an example in which the solid-state imaging device 112a is driven to draw a rotationally symmetric circular locus centered on the optical axis AX. This circular locus can be a closed circular path having a radius r of √2 / 2 times the pixel pitch P of the solid-state image sensor 112a, for example.

Note that information on the driving locus including the pixel pitch P is held in advance in the internal memory of the system controller 100 or the ROM 126.

As illustrated in FIG. 4A (or FIG. 4B), during the exposure period, the solid-state imaging device 112a is driven to draw a predetermined square locus (or circular locus) based on the information of the drive locus. Then, the subject image is uniformly incident on the four color filters R, G, B, and G (four (two rows and two columns) pixels PIX). Thereby, an effect equivalent to that of an optical LPF can be obtained. In other words, since the subject image incident on any color filter (pixel PIX) is necessarily incident on the surrounding color filter (pixel PIX), the same effect as when the subject image passes through the optical LPF. (Reduction of moiré such as false color) is obtained.

Note that the user can switch on / off of image blur correction driving and LPF driving by operating the operation unit 102.

[Explanation about false color detection]
Next, a method for detecting a false color, which is a kind of moire, that occurs in a captured image in the present embodiment will be described. FIG. 5 shows a false color detection flow executed by the system controller 100. The false color detection flow shown in FIG. 5 is started when the release switch is pressed, for example.

[S11 in FIG. 5 (state determination)]
In this processing step S11, it is determined whether or not the photographing apparatus 1 is in a stationary state. Illustratively, when the amplitude of a signal component having a frequency equal to or higher than a certain frequency among the vibration detection signals input from the gyro sensor 128 is within a certain threshold continuously for a certain period, it is determined to be in a stationary state. A typical example of the stationary state of the photographing apparatus 1 is a state in which the photographing apparatus 1 is fixed to a tripod. Note that the posture of the photographing apparatus 1 including the stationary state may be detected using information output from other sensors such as the acceleration sensor 130, the geomagnetic sensor 132, and the GPS sensor 134 instead of the gyro sensor 128. In order to improve the detection accuracy, for example, sensor fusion technology may be applied so that information output from these sensors may be used in combination.

When the photographing apparatus 1 is not in a stationary state such as a hand-held photographing state with a low shutter speed setting, a false color is unlikely to occur in the first place due to blurring of the subject due to camera shake (or subject blurring). Therefore, in this embodiment, only when it is determined that the photographing apparatus 1 is in a stationary state (that is, in a state in which false color is likely to occur) (S11: YES), the false color in the photographed image is detected. Therefore, processing step S12 (photographing of the first image) and subsequent steps are executed. When the photographing apparatus 1 is not in a stationary state, the processing load of the system controller 100 is reduced because the processing step S12 (first image photographing) and subsequent steps are not executed. The photographer may be notified, for example, via the display screen of the LCD 124 that the processing step S12 (photographing the first image) and subsequent steps are not executed. Note that, when it is important to detect the false color in the photographed image, the processing step S12 (photographing the first image) and subsequent steps may be executed regardless of whether or not the photographing apparatus 1 is in a stationary state. In this case, the photographer may be notified through the display screen of the LCD 124 that the processing step S12 (photographing the first image) and subsequent steps are executed. Further, the determination threshold value for the still state may be changed according to the shutter speed set in the photographing apparatus 1.

[S12 in FIG. 5 (capturing the first image)]
6 (a) and 7 (a) are enlarged views showing different parts of a subject to be photographed. The subject shown in FIG. 6A is an oblique stripe pattern in which light and dark appear alternately at the same pitch as the pixel pitch P of the pixel PIX of the solid-state image sensor 112a. The subject shown in FIG. It is a vertical stripe pattern in which light and dark appear alternately at the same pitch as P. For convenience of explanation, a 6 × 6 square subject surrounded by a thick solid line among the subjects shown in FIG. 6A is referred to as “obliquely striped subject 6a”, and a thick solid line among the subjects shown in FIG. 7A. The subject of 6 × 6 square surrounded by is denoted as “vertical stripe subject 7a”.

In this processing step S12, a subject image (first image) is taken with appropriate brightness and focus based on AE control and AF control. Here, it is assumed that both the diagonally striped subject 6a and the vertically striped subject 7a are in focus.

FIGS. 6B and 7B are diagrams schematically showing the oblique stripe subject 6a and the vertical stripe subject 7a captured by each pixel PIX of the solid-state image sensor 112a, and the light receiving surface 112aa of the solid-state image sensor 112a. Is a front view from the subject side. In FIG. 6B and FIG. 7B, as in FIG. 4, each pixel PIX is assigned a code (any one of R, G, and B) corresponding to the filter color arranged on the front surface. . In each of FIGS. 6B and 7B, the black pixel PIX indicates that the dark portion of the striped pattern has been captured, and the white pixel PIX has captured the bright portion of the striped pattern. It shows that. For convenience of explanation, pixel addresses (numerals 1 to 8 and symbols 1 to 1) are attached to the diagrams of FIGS. 6B and 7B. Strictly speaking, the subject is imaged on the solid-state imaging device 112a by the imaging action of the photographic lens 106 in a state where the subject is turned upside down. As an example, the upper left corner portion of the “obliquely striped subject 6a” forms an image as the lower right corner portion on the solid-state imaging device 112a. However, in this embodiment, in order to avoid complication of explanation, the upper left corner portion of the “obliquely striped subject 6a” is described as corresponding to the upper left corner portion of FIG.

[S13 in FIG. 5 (Shift of Solid-State Image Sensor 112a)]
In this processing step S13, the movable stage 112c is driven, and the solid-state imaging device 112a is shifted rightward (the direction of the arrow on the X axis in FIG. 6) by a distance of one pixel.

[S14 in FIG. 5 (Taking second image)]
Also in this processing step S14, the subject image (second image) is captured based on the AE control and AF control at the time of capturing the first image. After the second image is captured, the photographer may be notified that the false color detection process using the first image and the second image is to be started.

6 (c) and 7 (c) are similar to FIGS. 6 (b) and 7 (b), respectively, and the diagonally striped subject 6a and the vertically striped subject captured by each pixel PIX of the solid-state imaging device 112a. 7a is shown schematically. As shown in FIGS. 6B and 7B, the incident position of the subject image on the light receiving surface 112aa of the solid-state image sensor 112a is determined by processing step S13 (shift of the solid-state image sensor 112a). Is shifted by a distance corresponding to one pixel in the left direction on the light receiving surface 112aa.

In addition, the shooting conditions in this processing step S14 are other than the point that the solid-state imaging device 112a is shifted rightward by a distance of one pixel with respect to the shooting in the processing step S12 (first image shooting). Are the same. For this reason, the second image is substantially a photograph of a range shifted to the right by one pixel with respect to the first image. As can be seen from FIGS. 6B and 7B, in the second image, the subject is shifted to the left by a distance of one pixel as a whole with respect to the first image. It is reflected.

The signal processing (clamping) of the first image taken at the processing step S12 (photographing the first image) and the second image taken at the processing step S14 (photographing the second image) are both described above. , Demosaic, matrix calculation, Y / C separation, white balance, etc.), and converted into a luminance signal Y and color difference signals Cb, Cr. Hereinafter, for convenience of explanation, the color difference signals (Cb, Cr) of the first image captured in the processing step S12 (capturing the first image) will be referred to as “first color difference signals”, and the processing step S14 (the second image of the second image). The color difference signals (Cb, Cr) of the second image taken in (shooting) are referred to as “second color difference signals”. The “target pixel” refers to a pixel of each image after at least demosaic processing.

[S15 in FIG. 5 (electrical LPF processing)]
In the present embodiment, although described in detail later, the occurrence of false colors is detected based on the signal difference value or signal addition value between the first color difference signal and the second color difference signal. However, at the edge portion with high contrast, the signal difference value between the first color difference signal and the second color difference signal may be large even if a false color is not generated. In this case, there is a risk of false detection that a false color has occurred in the edge portion. As will be described in detail later, the first color difference signal and the second color difference signal used for the calculation of the signal difference value and the signal addition value are the color difference signals of the pixels that capture the same subject image. Due to the fact that the solid-state image sensor 112a is shifted by the shift of the image sensor 112a), demosaic processing is performed using pixels having different addresses. For this reason, the first color difference signal and the second color difference signal may be slightly different in color information even though they are color difference signals of pixels that capture the same subject image. Therefore, in this processing step S15, LPF processing is performed on the image signal (luminance signal Y, color difference signals Cb, Cr). By blurring the image by the LPF processing, false detection of false colors at the edge portion is suppressed, and errors in color information between the first color difference signal and the second color difference signal are suppressed.

[S16 in FIG. 5 (address conversion)]
In the present embodiment, a false color generation area in which a false color is generated in a captured image is detected on a pixel basis. In this case, in order to improve the false color detection accuracy, it is desirable to calculate the signal difference value and the signal addition value between the target pixels on which the same subject image is incident. Therefore, in this processing step S16, the second image captured in the processing step S14 (second image capturing) is configured so that the same subject image is processed as being incident on the pixel having the same address. The address of each pixel corresponds to the shift amount of the solid-state image sensor 112a in processing step S13 (shift of the solid-state image sensor 112a) (in other words, the shift amount of the incident position of the subject image on the light receiving surface 112aa of the solid-state image sensor 112a). Converted accordingly.

As an example, the address (c, 2) of the pixel PIXb in FIG. 6C is the same as the pixel that is located when the pixel PIXb is shifted by one pixel in the right direction according to the shift amount of the incident position of the subject image, that is, the pixel PIXb. Is converted to the same address (d, 2) as the pixel PIXa (see FIG. 6B) for capturing the subject image.

[S17 in FIG. 5 (Calculation of Difference Value of Color Difference Signal)]
The oblique stripe subject 6a and the vertical stripe subject 7a include a high-frequency component having the same pitch as the pixel pitch P of the pixel PIX of the solid-state image sensor 112a. Therefore, if the image signals of the diagonally striped subject 6a and the vertically striped subject 7a are demosaiced in the processing step S12 (first image capturing) and the processing step S14 (second image capturing), a false color is generated.

Specifically, in the example of FIG. 6B, the luminance of the G component pixel PIX is high and the luminance of the R and B component pixels PIX is low. Therefore, the G component is dominant in the color information of each pixel PIX after the demosaic process, and a green false color is generated in the diagonally striped subject 6a. On the other hand, in the example of FIG. 6C, the luminance of the R and B component pixels PIX is high and the luminance of the G component pixel PIX is low, contrary to the example of FIG. 6B. Therefore, the R and B components are dominant in the color information of each pixel PIX after the demosaic process, and a purple false color is generated in the diagonally striped subject 6a.

In the example of FIG. 7B, the luminance of the B component pixel PIX is high and the luminance of the R component pixel PIX is low. Therefore, the color information of each pixel PIX after the demosaic processing becomes a mixed color component of B and G, and a false color of an intermediate color between blue and green (for example, a color near the cyan color center) is generated. On the other hand, in the example of FIG. 7C, the luminance of the R component pixel PIX is high and the luminance of the B component pixel PIX is low, contrary to the example of FIG. 7B. Therefore, the color information of each pixel PIX after the demosaic processing becomes a mixed color component of R and G, and a false color of an intermediate color between red and green (a color near the orange center) is generated.

FIG. 8 shows a color space defined by two axes of Cb and Cr. In FIG. 8, reference numeral 6b is a plot corresponding to the green false color generated at the target pixel in the example of FIG. 6B, and reference numeral 6c is a purple color generated at the target pixel in the example of FIG. 6C. 7b is a plot corresponding to a false color of an intermediate color between blue and green generated in the target pixel in the example of FIG. 7B, and reference numeral 7c is a plot corresponding to FIG. It is a plot corresponding to the false color of the intermediate color of red and green generated in the target pixel in the example of c). The following shows the coordinate information of each plot. The origin O is the coordinates (0, 0).
Plot 6b: (Cb, Cr) = (− M, −N)
Plot 6c: (Cb, Cr) = (M, N)
Plot 7b: (Cb, Cr) = (M ′, −N ′)
Plot 7c: (Cb, Cr) = (− M ′ + α, N ′ + β)
However, all of M, N, M ′, N ′, α, and β are positive numbers.

As shown in FIG. 8, in the present embodiment, the false color itself that is generated when a subject image having a high frequency component equivalent to the pixel pitch P of the pixel PIX of the solid-state image sensor 112a is captured is the light receiving surface 112aa. A portion (a pixel of interest) where a false color is generated is detected by utilizing the change by shifting the incident position of the subject image above. More specifically, in a pixel of interest in which a high-frequency component subject image is captured, a portion in which the color information of the first color difference signal and the second color difference signal has a complementary color relationship is determined to be a portion where a false color is generated. It is detected.

In the present embodiment, in order to obtain the above complementary color relationship, the incident position of the subject image on the light receiving surface 112aa is shifted by one pixel in the left direction (horizontal pixel arrangement direction) (the solid-state imaging device 112a is shifted from the subject image). However, the present invention is not limited to this. For example, the shift direction may be the right direction (horizontal pixel arrangement direction), or the upper direction (vertical pixel arrangement direction), the lower direction (vertical pixel arrangement direction), or the upper right direction. , Lower right, upper left, lower left diagonal directions (directions forming 45 degrees with respect to horizontal and vertical arrangement directions), or other directions according to the pixel arrangement may be used together. . In addition, the shift distance may be, for example, a distance corresponding to another odd pixel such as 3 pixels or 5 pixels, or a half pixel or a half pixel plus an odd pixel (for example, 1.. 5 pixels, 2.5 pixels, etc.) (ie, n pixels or (m + 0.5) pixels (where n = odd natural number, m = 0 or odd natural number) However, it can be selected according to the accuracy of the mechanism for shift driving. Further, the shift distance may be a distance other than the even number of pixels and the vicinity thereof (for example, 1.9 to 2.1 pixels) depending on the subject to be photographed and the photographing conditions. Even when the incident position of the subject image on the light receiving surface 112aa is shifted by these amounts (direction and distance), the first color difference signal is obtained at the target pixel in which the subject image of the high frequency component is captured (false color is generated). Since the color information of the second color difference signal and the second color difference signal are complementary to each other, the occurrence of false color can be detected.

In this processing step S17, after the address conversion in the processing step S16 (address conversion) (hereinafter the same), for each target pixel having the same address, the difference value (Cb _sub) between the first color difference signal and the second color difference signal. , Cr _sub ) is calculated. Specifically, in this processing step S17, Cb and Cr of the first color difference signal are defined as Cb1 and Cr1, respectively, and Cb and Cr of the second color difference signal of the same address are defined as Cb2 and Cr2, respectively. In addition, the difference value (Cb _sub , Cr _sub ) is calculated by the following equation.
Cb _sub = Cb1-Cb2
Cr _sub = Cr1-Cr2

[S18 in FIG. 5 (calculation of first distance information)]
In the process step S18, address for each identical target pixel, a distance in the color space of the first color difference signal and a second color difference signal (the first distance information Saturation_ _sub) is calculated by the following equation.
^{_{Saturation_ sub = √ (Cb sub 2}} + Cr sub 2)

The first distance information Saturation_ _sub are each an example of each plot pair of FIG. 8 (plot 6b and plot 6c, plots 7b and plot ^{7c), 2√ (M 2 +} N 2), √ {(2M'-α) ² + (2N ′ + β) ^{2 −} }.

As understood from the positional relationship of each plot pair of FIG. 8, as the color information with the first color difference signal and a second color difference signal is strong complementary relationship first distance information Saturation_ _sub increases, first color difference signal and the color information having the second color difference signal is not (for example similar hue) in complementary relationship as the first distance information Saturation_ _sub decreases. That is, the first distance information Saturation_ _sub ideally if false color occurrence region is zero, the larger the false color occurrence region in which false color is generated strongly.

[S19 in FIG. 5 (Calculation of Addition Value of Color Difference Signal)]
In this processing step S19, for each target pixel having the same address, an addition value (Cb ′ _add , Cr ′ _add ) of the first color difference signal and the second color difference signal is calculated.

Specifically, in this processing step S19, provisional addition values (Cb _add , Cr _add ) are calculated according to the following equations.
Cb _add = Cb1 + Cb2
Cr _add = Cr1 + Cr2

Next, the average value (Cb _mean , Cr _mean ) of the provisional addition values is calculated by the following equation.
Cb _mean = Cb _add / 2
Cr _mean = Cr _add / 2

Next, the added value (Cb ′ _add , Cr ′ _add ) is calculated by the following equation.
Cb ′ _add = Cb _add −Cb _mean
Cr ' _add = Cr _add -Cr _mean

[S20 in FIG. 5 (calculation of second distance information)]
In the process step S20, address for each identical target pixel, the addition value _{(Cb 'add, Cr' add} ) the second distance information Saturation_ _{the add} in a color space based on the calculation by the following equation.
_{Saturation_ add = √ (Cb 'add} 2 + Cr' add 2)

Second distance information Saturation_ _{the add} each example of each plot pair of FIG. 8 (plot 6b and plot 6c, plots 7b and plot 7c), zero, and ^{√ (α 2 + β 2)} .

Here, the first and second color difference signals change under the influence of the light source, the exposure condition, the white balance and the like at the time of image capturing. However, first, since the second respective color difference signals varies in the same way, mutual relative distance in the color space (i.e., the first distance information Saturation_ _sub) changes little.

On the other hand, the second distance information Saturation_ _{the add,} the first, second light source for the respective color difference signal imaging, exposure conditions, the changes relative to the position of the origin O of the color space under the influence of such as white balance , Change a lot. Therefore, in the process step S19 (calculation of the sum of the color difference signal), the second distance information Saturation_ _{the add} provisional sum value _{(Cb add, Cr} add) without operation immediately using, relative to the origin O by the impact In order to cancel or reduce the change in position (to make the midpoint between the first color difference signal and the second color difference signal far from the origin O closer to the origin O due to the above influence), the added values (Cb ′ _add , Cr ′ _add) ) Is calculated.

As can be understood from the positional relationship between each pair of plots in FIG. 8, when the color information of the first color difference signal and the second color difference signal has a complementary color relationship, the signs are opposite to each other, so that the added value (Cb _'add, Cr' _add) is decreased, the second distance information Saturation_ _{the add} decreases. The second distance information Saturation_ to _add becomes smaller as is strong complementary relationship, the zero ideally. On the other hand, when the color information of the first color difference signal and the second color difference signal is not in a complementary color relationship (for example, the same hue), the signs are the same, so the added values (Cb ′ _add , Cr ′) _add) and increases, the second distance information Saturation_ _{the add} increases. That is, the second distance information Saturation_ _{the add} is larger if false color occurrence region decreases if false color occurrence region.

[S21 in FIG. 5 (Calculation of Difference Value of Luminance Signal)]
For convenience of explanation, the luminance signal Y of the first image taken in the processing step S12 (photographing the first image) will be referred to as “first luminance signal” and taken in the processing step S14 (photographing the second image). The luminance signal Y of the second image is referred to as “second luminance signal”. In this processing step S21, a difference value Y _diff between the first luminance signal and the second luminance signal is calculated for each target pixel having the same address. Specifically, in this processing step S21, when the first luminance signal is defined as Y1 and the second luminance signal is defined as Y2, the difference value Y _diff is calculated by the following equation.
Y _diff = | Y1-Y2 |

[S22 in FIG. 5 (Determination of False Color Generation Area)]
In this processing step S22, it is determined for each pixel of interest having the same address whether or not it is a false color generation region. Specifically, when all of the following conditions (1) to (3) are satisfied, it is determined that the pixel is a false color generation region.

・ Condition (1)
As described above, since the color information having the first color difference signal the larger the first distance information Saturation_ _sub and the second color difference signal is strong complementary relationship, likely the target pixel is false color occurrence region is high. Therefore, condition (1) is defined as follows.
Condition (1): Saturation_ _sub ≧ threshold T1

・ Condition (2)
As described above, since the color information having the second distance information Saturation_ _{the add} is small enough first color difference signal and a second color difference signal is strong complementary relationship, likely the target pixel is false color occurrence region is high. Therefore, condition (2) is defined as follows.
Condition (2): Saturation_ _add ≦ threshold T2

Consider a case where a large image blur occurs in only one of the first image taken in processing step S12 (shooting the first image) and the second image taken in processing step S14 (shooting the second image). . In this case, the difference between the first color difference signal and the second color difference signal becomes large, and there is a possibility that a false color is erroneously detected. Therefore, the condition (3) is defined as follows.
Condition (3): Y _diff ≦ threshold value T3

That is, when the difference value Y _diff is larger than the threshold value T3, there is a risk of the erroneous detection described above, so that false color detection for the target pixel is not performed (assuming that the target pixel is not a false color generation region). It is processed.).

The conditions (1) and (2) are conditions for directly determining whether or not the pixel of interest is a false color generation area. Therefore, in another embodiment, when at least one of the condition (1) and the condition (2) is satisfied, the target pixel may be determined to be a false color generation region.

Also, the user can change the false color detection sensitivity by operating the operation unit 102 to change the settings of the threshold values T1 to T3.

[S23 in FIG. 5 (detection of false color)]
In this processing step S23, the presence or absence of a false color is detected. Illustratively, the number of target pixels determined to be a false color generation region in processing step S22 (determination of the false color generation region) (or the ratio of the target pixels determined to be the false color generation region out of the total number of effective pixels). If it is equal to or greater than the predetermined threshold, a detection result that there is a false color is obtained (S23: YES), and if the number of pixels of interest (or a ratio) is less than the predetermined threshold, a detection result that there is no false color is obtained (S23). : NO).

[S24 in FIG. 5 (Storing Captured Images)]
This processing step S24 is executed when a detection result indicating no false color is obtained in the processing step S23 (detection of false color) (S23: NO). In this processing step S24, false colors are not detected for the first image captured in processing step S12 (capturing the first image) and the second image captured in processing step S14 (capturing the second image). For example, at least one of them is stored in the memory card 200 (or a built-in memory (not shown) provided in the photographing apparatus 1). At this point, the photographer may be notified that the shooting operation has been completed. In particular, when the photographer is informed that the false color detection process using the first image and the second image is started in the processing step S14 (second image capturing), the photographing operation is completed. Communicate to the photographer. As a result, the photographer can proceed to the next work, for example, change of the state (setting) of the photographing apparatus 1.

[S25 in FIG. 5 (Image pickup under LPF drive)]
This processing step S25 is executed when a detection result indicating that there is a false color is obtained in processing step S23 (detection of false color) (S23: YES). In this processing step S25, false colors were detected for the first image captured in processing step S12 (capturing the first image) and the second image captured in processing step S14 (capturing the second image). Therefore, the LPF drive is executed. When imaging under LPF driving has already been performed, the driving cycle (rotation cycle) and driving of the solid-state imaging device 112a are obtained so that a stronger optical LPF effect (reduction of moire such as false colors) can be obtained. The amplitude (rotation radius) is adjusted. That is, it is changed to a more advantageous shooting condition for reducing false colors. Then, the subject is imaged (third image is captured). That is, when a false color is detected, the third image is captured. Therefore, when the photographer is notified that the false color detection process is started in the processing step S14 (photographing of the second image), the photographer is in a state of the photographing device 1 until the photographing of the third image is completed. (Setting) can be maintained.

According to the present embodiment, the difference occurs when a subject image having a high frequency component equal to or greater than the pixel pitch P is captured by shifting the incident position of the subject image on the light receiving surface 112aa of the solid-state imaging device 112a. False colors (false colors that are complementary to each other) are generated, and an image with a large difference is generated. According to this embodiment, since an image with a large difference is used for detecting a false color, the false color is detected with high accuracy.

This completes the description of the exemplary embodiment of the present invention. Embodiments of the present invention are not limited to those described above, and various modifications are possible within the scope of the technical idea of the present invention. For example, the embodiment of the present application also includes an embodiment that is exemplarily specified in the specification or a combination of obvious embodiments and the like as appropriate.

In the above embodiment, the false color is optically removed from the entire image by executing the LPF drive, but the present invention is not limited to this. False colors may be removed using image processing. In the case of image processing, the false color can be removed not only in the entire image but also locally (for example, for each pixel of interest determined as a false color generation region). In addition, if information indicating the false color generation area is stored in association with the first image (second image), even if the false color correction is manually performed by another terminal (such as a computer), A false color generation region can be easily found based on the information. Therefore, the trouble of searching for the false color generation area from the entire image is reduced.

In the above embodiment, the incident position of the subject image on the light receiving surface 112aa is shifted by shifting the solid-state imaging device 112a itself. However, the present invention is not limited to this. For example, the incident position of the subject image on the light receiving surface 112aa may be shifted by driving another shake correction member (such as a part of lenses included in the photographing lens 106) eccentrically, or the photographing lens A parallel plate arranged in the optical path 106 at a slight inclination with respect to the optical axis AX is rotated around the optical axis AX, or the apex variable prism, the cover glass of the solid-state image sensor 112a (the dust adhering to the cover glass). Or the like) may be driven to shift the incident position of the subject image on the light receiving surface 112aa.

In the above embodiment, the threshold values T1 to T3 are unchanged in all the determinations for each pixel in the processing step S22 (determination of false color generation region), but the present invention is not limited to this.

For example, by performing AF control by the system controller 100, a range that can be regarded as being in focus in the image (a focus area, illustratively a range that falls within the depth of field) is obtained. Since the subject in the focus area (in-focus state) has high contrast and tends to include high-frequency components, false colors are likely to occur. On the other hand, a subject outside the in-focus area (out-of-focus state) has a low contrast and is unlikely to contain a high-frequency component, so that false colors are unlikely to occur. Therefore, in the processing step S22 (determination of the false color generation area), the threshold values T1 to T3 are set depending on whether the determination is made for the pixel in the in-focus area obtained by the calculation or the determination for the pixel outside the in-focus area. It may be changed to a different value. Illustratively, since there is a high possibility that a false color is generated (conspicuous) in a pixel in the in-focus area, a threshold setting that increases detection sensitivity (for example, the threshold T1 is set to a low value) is performed. Since the possibility of false color is low (not conspicuous) in other pixels, threshold setting (for example, setting the threshold value T1 to a high value) that lowers the detection sensitivity or false color detection is performed. The process itself is omitted. As a result, the false color detection accuracy is further improved, and the detection speed is improved.

In the above-described embodiment, a pair of images (a first image taken at processing step S12 (photographing the first image) and a second image taken at processing step S14 (photographing the second image)). However, in another embodiment, false color may be detected using three or more images.

For example, for the vertical stripe subject 7a, by shifting the solid-state imaging device 112a in the light / dark direction (left direction) of the stripe in processing step S13 (shift of the solid-state imaging device 112a) as in the above embodiment, It is easy to obtain a false color image having a complementary color relationship. However, when the solid-state image sensor 112a is shifted in the up, down, or diagonal directions with respect to the vertical stripe subject 7a, the color component with high luminance is less likely to change before and after the shift of the solid-state image sensor 112a, resulting in a complementary color relationship. It becomes difficult to obtain a false color image. Therefore, in order to further improve the accuracy of false color detection (in another aspect, to prevent detection omission), a false color detection flow shown in FIG. 9 can be considered. In the description of the false color detection flow in FIG. 9, processing similar to the false color detection flow in FIG. 5 is simplified or omitted as appropriate.

[S111 in FIG. 9 (state determination)]
In this processing step S111, it is determined whether or not the photographing apparatus 1 is in a stationary state.

[S112 in FIG. 9 (capturing the first image)]
This processing step S112 is executed when it is determined in processing step S111 (state determination) that the photographing apparatus 1 is in a stationary state (S111: YES). In this processing step S112, a subject image (first image) is taken with appropriate brightness and focus based on AE control and AF control.

[S113 in FIG. 9 (Shift of Solid-State Image Sensor 112a)]
In this processing step S113, the movable stage 112c is driven, and the solid-state imaging device 112a is shifted rightward by a distance of one pixel.

[S114 in FIG. 9 (Taking Second Image)]
In this processing step S114, a subject image (second image) is captured based on the AE control and AF control at the time of capturing the first image.

[S115 in FIG. 9 (Shift of Solid-State Image Sensor 112a)]
In this processing step S115, the solid-state imaging device 112a is shifted upward by a distance of one pixel with respect to the position at the time of first image shooting in the processing step S112 (first image shooting).

[S116 in FIG. 9 (Third Image Shooting)]
In this processing step S116, a subject image (third image) is photographed based on AE control and AF control at the time of photographing the first image.

[S117 in FIG. 9 (Shift of Solid-State Image Sensor 112a)]
In this processing step S117, the solid-state imaging device 112a is shifted diagonally to the lower left by a distance of one pixel with respect to the position at the time of first image shooting in the processing step S112 (first image shooting).

[S118 in FIG. 9 (Fourth Image Shooting)]
In this processing step S118, an appropriate subject image (fourth image) is captured based on the AE control and AF control at the time of capturing the first image.

[S119 in FIG. 9 (electrical LPF processing)]
In this processing step S119, an LPF process is performed on the image signals (luminance signal Y, color difference signals Cb, Cr) in order to suppress false detection of false colors.

[S120 in FIG. 9 (address conversion)]
In this processing step S120, the same subject image is obtained for the first image captured in processing step S112 (capturing the first image) and the second image captured in processing step S114 (capturing the second image). The address of each pixel constituting the second image is processed as if it were incident on the pixel of the same address, and the shift amount (in other words, the shift amount of the solid-state image sensor 112a in the processing step S113 (shift of the solid-state image sensor 112a)). , The amount of shift of the incident position of the subject image on the light receiving surface 112aa of the solid-state imaging device 112a).

In this processing step S120, the same subject is used for the first image captured in processing step S112 (capturing the first image) and the third image captured in processing step S116 (capturing the third image). The address of each pixel constituting the third image is determined by the amount of shift of the solid-state image sensor 112a in the processing step S115 (shift of the solid-state image sensor 112a) so that the image is processed as being incident on the pixel having the same address. In other words, it is converted according to the shift amount of the incident position of the subject image on the light receiving surface 112aa of the solid-state image sensor 112a.

In this processing step S120, the same subject is used for the first image captured in processing step S112 (capturing the first image) and the fourth image captured in processing step S118 (capturing the fourth image). The address of each pixel making up the fourth image is the shift amount of the solid-state image sensor 112a in the processing step S117 (shift of the solid-state image sensor 112a) so that the image is processed as being incident on the pixel having the same address. In other words, it is converted according to the shift amount of the incident position of the subject image on the light receiving surface 112aa of the solid-state image sensor 112a.

[S121 in FIG. 9 (Calculation of Difference Value of Color Difference Signal)]
For convenience of explanation, the color difference signals (Cb, Cr) of the first image photographed in the processing step S112 (first image photographing) are referred to as “first color difference signal”, and the processing step S114 (second image photographing). The color difference signals (Cb, Cr) of the second image photographed in Step 2 are referred to as “second color difference signals”, and the color difference signals (Cb, Cr) of the third image photographed in processing step S116 (third image photographing). Cr) is referred to as “third color difference signal”, and the color difference signals (Cb, Cr) of the fourth image captured in processing step S118 (fourth image capturing) are referred to as “fourth color difference signal”. In this processing step S121, for each pixel having the same address, for each of the first color difference signal and the second color difference signal, the first color difference signal and the third color difference signal, the first color difference signal and the fourth color difference signal, Cb _sub , Cr _sub ) is calculated.

[S122 in FIG. 9 (Calculation of First Distance Information)]
In this processing step S122, the first color difference signal and the second color difference signal, the first color difference signal and the third color difference signal, the first color difference signal and the fourth color difference signal are respectively determined for each target pixel having the same address. distance information Saturation_ _sub is calculated.

[S123 in FIG. 9 (Calculation of Addition Value of Color Difference Signal)]
In this processing step S123, for each pixel of interest having the same address, the added value for each of the first color difference signal and the second color difference signal, the first color difference signal and the third color difference signal, and the first color difference signal and the fourth color difference signal. (Cb ′ _add , Cr ′ _add ) is calculated.

[S124 in FIG. 9 (Calculation of Second Distance Information)]
In this processing step S124, the second color difference signal and the second color difference signal, the first color difference signal and the third color difference signal, the first color difference signal and the fourth color difference signal are respectively determined for each target pixel having the same address. distance information Saturation_ _{the add} is calculated.

[S125 in FIG. 9 (Calculation of Difference Value of Luminance Signal)]
For convenience of explanation, the luminance signal Y of the first image taken in the processing step S112 (photographing the first image) will be referred to as “first luminance signal” and taken in the processing step S114 (photographing the second image). The luminance signal Y of the second image is denoted as “second luminance signal”, and the luminance signal Y of the third image photographed in the processing step S116 (third image photographing) is denoted as “third luminance signal”. The luminance signal Y of the fourth image shot in the processing step S118 (fourth image shooting) is referred to as “fourth luminance signal”. In this processing step S124, for each target pixel having the same address, a difference value is obtained for each of the first luminance signal and the second luminance signal, the first luminance signal and the third luminance signal, and the first luminance signal and the fourth luminance signal. Y _diff is calculated.

[S126 in FIG. 9 (Determination of False Color Generation Area)]
In this processing step S126, it is determined whether or not each pixel of interest having the same address is a false color generation region. Specifically, when at least one of the following conditions (1 ′) to (3 ′) is satisfied, it is determined that the pixel is a false color generation region.

・ Condition (1 ')
And a first in first color difference signal and a second color difference signal (in the case of the solid-state imaging device 112a at the time of each shot is a positional relationship shifted one pixel in the lateral direction) the first distance information Saturation_ _sub threshold T1 'above, and it second distance information Saturation_ _{the add} is "or less, and the difference value Y _diff threshold T3 'threshold T2 or less.

・ Condition (2 ')
And at the first color difference signal and the third color difference signal (solid-state imaging device 112a at the time of each shooting when a positional relationship shifted one pixel in the vertical direction) the first distance information Saturation_ _sub threshold T1 'above, and it second distance information Saturation_ _{the add} is "or less, and the difference value Y _diff threshold T3 'threshold T2 or less.

・ Condition (3 ')
And at the first color difference signal and the fourth color difference signal (solid-state imaging device 112a at the time of each shooting when a positional relationship shifted one pixel in the diagonal direction) the first distance information Saturation_ _sub threshold T1 'above, and it second distance information Saturation_ _{the add} is "or less, and the difference value Y _diff threshold T3 'threshold T2 or less.

[S127 in FIG. 9 (Detection of False Color)]
In this processing step S127, the presence or absence of a false color is detected.

[S128 in FIG. 9 (Storing Captured Images)]
This processing step S128 is executed when a detection result indicating that there is no false color is obtained in processing step S127 (detection of false color) (S127: NO). In this processing step S128, the first image captured in processing step S112 (capturing the first image), the second image captured in processing step S114 (capturing the second image), and processing step S116 (third capturing) Assuming that no false color is detected for the third image captured in (Image Capture) and the fourth image captured in Processing Step S118 (Fourth Image Capture), at least one of them is the memory card 200 (or (Stored in an unillustrated built-in memory provided in the photographing apparatus 1).

[S129 in FIG. 9 (image pickup under LPF drive)]
This processing step S129 is executed when a detection result indicating that there is a false color is obtained in processing step S127 (detection of false color) (S127: YES). In this processing step S129, the first image photographed in the processing step S112 (first image photographing), the second image photographed in the processing step S114 (second image photographing), and the processing step S116 (third image photographing). LPF driving is executed because a false color is detected from at least one of the third image captured in (image capture) and the fourth image captured in processing step S118 (fourth image capture). Thus, the subject is imaged.

According to the false color detection flow of FIG. 9, since a change in image (occurrence of false color) is determined when the solid-state imaging device 112a is shifted in different directions, in a region where a high frequency component appears in the subject, The change between at least a pair of images becomes large. For this reason, the false color is detected with high accuracy regardless of the appearance direction of the high-frequency component in the subject (in the case of the vertical-striped subject 7a, the left-right direction in which the light and shade alternate).

Claims (12)

1. A subject image incident on an image sensor having a predetermined pixel arrangement via a photographing optical system, and a moving means for relatively moving the position of the image sensor in the direction of the light receiving surface of the image sensor;
   Signal generating means for capturing the subject image in a state where the relative positions are different to generate a color difference signal;
   Detection means for detecting image degradation occurring in the captured image based on the color difference signal;
   Comprising
   Image detection device.
2. An object image on the light receiving surface of the image sensor by physically moving at least one of the optical elements in the imaging optical system and at least one of the image sensors based on the pixel arrangement of the image sensor with a transmission wavelength selection element Moving means for moving the position of
   A signal generating unit that captures a subject image captured by the imaging device each time the position of the subject image is moved, and performs predetermined signal processing including color interpolation processing to generate a color difference signal;
   Detecting means for detecting image degradation occurring in a captured image based on at least a pair of color difference signals in which the positions of the subject images on the light receiving surface of the image sensor are different from each other;
   Comprising
   Image detection device.
3. The moving means is
   The position of the subject image on the light receiving surface of the image sensor is set by an amount corresponding to the pixel interval so that the color information of the at least one pair of color difference signals has a complementary color relationship with each other in an occurrence region where image degradation occurs. Move,
   The image detection apparatus according to claim 2.
4. The moving means is
   The position of the subject image on the light receiving surface of the image sensor is equivalent to n pixels or (m + 0.5) pixels in the direction corresponding to the pixel arrangement (where n = odd natural number, m = 0 or odd natural number). Move the distance of
   The image detection apparatus according to claim 3.
5. The detection means includes
   One photographed image is configured so that the same subject image is processed as being incident on a pixel having the same address with respect to at least a pair of photographed images having different positions of the subject image on the light receiving surface of the image sensor. The address of each pixel to be converted according to the amount of movement of the position of the subject image,
   Detection of image degradation is performed for each pixel based on the color difference signal of the one captured image and the color difference signal of the other captured image in which the address is converted.
   The image detection apparatus according to any one of claims 2 to 4.
6. The detection means includes
   Calculating at least one of a difference value and an addition value of the at least one pair of color difference signals;
   Image degradation is detected based on the calculation result.
   The image detection apparatus according to any one of claims 2 to 5.
7. The detection means includes
   Calculating a difference value of the at least one pair of color difference signals for each pixel;
   Detecting a pixel whose calculated difference value is equal to or greater than a first threshold as a pixel in an occurrence region where image degradation occurs,
   The image detection apparatus according to claim 5.
8. The detection means includes
   An added value of the at least one pair of color difference signals is calculated for each pixel,
   A pixel whose calculated addition value is equal to or smaller than a second threshold is detected as a pixel in a generation region where image degradation occurs;
   The image detection apparatus according to claim 5.
9. The signal generating means includes
   By performing the predetermined signal processing, a luminance signal paired with the color difference signal is generated,
   The detection means includes
   Detecting image degradation based on the at least one pair of luminance signals;
   The image detection apparatus according to any one of claims 6 to 8.
10. A photographing apparatus comprising the image detection device according to any one of claims 1 to 9,
    Comprising a stationary state determining means for determining whether or not the photographing apparatus is in a stationary state;
    When it is determined by the stationary state determining means that the photographing apparatus is in a stationary state, image degradation is detected by the image detecting device.
    Shooting device.
11. An imaging device comprising the image detection device according to any one of claims 1 to 9, or the imaging device according to claim 10,
    Comprising shake detection means for detecting shake of the imaging device;
    The moving means is
    By physically moving at least one of some of the optical elements and the image sensor in the imaging optical system based on the shake of the imaging apparatus detected by the shake detection unit, an image caused by the shake of the imaging apparatus To correct shake,
    Shooting device.
12. An object image on the light receiving surface of the image sensor by physically moving at least one of the optical elements in the imaging optical system and at least one of the image sensors based on the pixel arrangement of the image sensor with a transmission wavelength selection element Moving the position of
    Each time the position of the subject image is moved, capturing the subject image captured by the image sensor, performing predetermined signal processing including color interpolation processing, and generating a color difference signal;
    Detecting image degradation occurring in a captured image based on at least a pair of color difference signals having different positions of the subject image on the light receiving surface of the image sensor; and
    including,
    Image detection method.

Images