WO2012165087A1

WO2012165087A1 - Image capture device and ghosting correction method

Info

Publication number: WO2012165087A1
Application number: PCT/JP2012/060987
Authority: WO
Inventors: 博文渡邊; 善工古田
Original assignee: 富士フイルム株式会社
Priority date: 2011-06-01
Filing date: 2012-04-24
Publication date: 2012-12-06
Also published as: JP2014158061A

Abstract

When the difference between G pixels (52) in a detection area of a first group of pixels is greater than or equal to a threshold (Th1), a system control unit (11) determines that ghosting has occurred in the detection area and an area of a second group of pixels which configures a pair with the detection area, and further, when the difference between the detection signal of each G pixel (52) in the detection area and the detection signal of a G pixel (53) which configures a pair with each G pixel (52) is greater than or equal to a threshold (Th2), carries out a correction which reduces the level difference of the ghosting which occurs in each of the detection area and the area of the second group of pixels which configures the pair with the detection area.

Description

Imaging apparatus and ghost correction method

The present invention relates to an imaging apparatus and a ghost correction method.

Imaging devices such as digital still cameras and digital video cameras generally have an imaging optical system composed of a plurality of lenses and prisms. When the subject includes a bright light source such as the sun or a light bulb, the light from the light source is reflected on the lens surface, the imaging optical system structure (screws, lens barrel, diaphragm, etc.) or the surface of the image sensor, and the reflection An image of light may be formed on the image sensor. An image generated by forming an image of the reflected light is generally called a ghost. Usually, a ghost in the shape of an aperture is generated around a light source such as the sun, or reflection of a structure often becomes a ghost.

Japanese Patent Application Laid-Open No. 2004-133867 determines whether or not there is a ghost using a difference image between a focus image and a defocus image as a determination image, and subtracts the ghost from the focus image when it is determined that there is a ghost, thereby correcting the ghost. Is disclosed.

According to the imaging apparatus described in Patent Document 1, it is possible to accurately correct a ghost in any shooting environment.

Japanese Unexamined Patent Publication No. 2008-54206

However, the imaging apparatus described in Patent Document 1 needs to execute imaging for acquiring a defocused image in order to detect a ghost. For this reason, the photographing time becomes longer and the power consumption becomes larger.

The present invention has been made in view of the above circumstances, and in any shooting environment, an imaging device capable of detecting the presence or absence of a ghost by one shooting and correcting the ghost when there is a ghost. It is another object of the present invention to provide a ghost correction method.

The imaging device of the present invention is an imaging device including a solid-state imaging device having two pixel groups each including a plurality of pixels, and the plurality of pixels of each pixel group are arranged in two dimensions, green The pixel group includes three types of pixels: a first color detection pixel that detects light, a blue detection pixel that detects blue light, and a red detection pixel that detects red light. Each pixel group includes the three types of pixels. The array of the plurality of pixels of each pixel group is arranged in a Bayer shape, the first pixel row in which the green detection pixels and the blue detection pixels are alternately arranged in a row direction, and the red detection pixels And a second pixel row in which the green detection pixels are alternately arranged in a row direction, are alternately arranged in a column direction orthogonal to the row direction, and one pixel group of the two pixel groups Each of the pixels includes the two images in a predetermined direction with respect to the pixels. Pixels of the same type as the respective pixels of the other pixel group of the group are arranged adjacent to each other, and an optical aperture of each pixel of the one pixel group and an optical aperture of each pixel of the other pixel group Are decentered in opposite directions, and each pixel of the one pixel group and a pixel of the other pixel group arranged adjacent to each pixel in the predetermined direction constitute a pixel pair. At least one of the first color detection pixels included in the first pixel row and the first color detection pixels included in the second pixel row obtained by dividing the one pixel group. The other pixel group in which pixels forming a pair with the green area detection pixel and the green area detection pixel in the division area are arranged using the detection signal of each green detection pixel included in the division area. Whether or not there is a ghost in this area A ghost presence / absence determination unit that determines whether a ghost has occurred by the ghost presence / absence determination unit, a detection signal of each green detection pixel in the divided area, and detection of a pixel that forms a pair with each green detection pixel A ghost image quality difference calculating unit that calculates data indicating a difference in image quality of ghosts generated in each of the divided area and the area of the other pixel group determined that the ghost is generated using a signal; When the data indicating the image quality difference exceeds a first threshold value, the detection signal of each green detection pixel in the divided area and each green color detection so that the data indicating the image quality difference is equal to or less than the first threshold value. And a signal correction unit that corrects at least one of detection signals of pixels that form a pair with the pixel.

The ghost correction method of the present invention is a ghost correction method for correcting a ghost of a captured image obtained by imaging with a solid-state imaging device having two pixel groups each including a plurality of pixels, wherein These pixels are arranged in two dimensions, and include three types of pixels: a green detection pixel that detects green light, a blue detection pixel that detects blue light, and a red detection pixel that detects red light, Each pixel group includes the three types of pixels arranged in a Bayer shape, and the plurality of pixels in each pixel group are arranged by alternately arranging the green detection pixels and the blue detection pixels in a row direction. A first pixel row and a second pixel row in which the red detection pixels and the green detection pixels are alternately arranged in a row direction are alternately arranged in a column direction orthogonal to the row direction. , One pixel of the two pixel groups In each of the pixels, pixels of the same type as the respective pixels of the other pixel group of the two pixel groups are arranged adjacent to each other in a predetermined direction with respect to each of the pixels. The optical aperture of the pixel and the optical aperture of each pixel of the other pixel group are decentered in opposite directions, and each pixel of the one pixel group is adjacent to the pixel in the predetermined direction. A pair of pixels is formed by the pixels of the other pixel group arranged in a row, and the green pixel and the second pixel included in the first pixel row obtained by dividing the one pixel group For each divided area including at least one of the green detection pixels included in a pixel row, a detection signal of each green detection pixel included in the divided area is used to detect each green detection pixel in the divided area and the divided area. Before the pixels that make up the pair are placed A ghost presence / absence determination step for determining whether or not a ghost has occurred in the area of the other pixel group, and detection of each green color detection pixel in the divided area determined to have a ghost generated by the ghost presence / absence determination step Ghosts generated in the divided area and the other pixel group area where the ghost is determined to be generated using a signal and a detection signal of a pixel forming a pair with each green detection pixel A ghost image quality difference calculating step for calculating data indicating a difference in image quality, and when the data indicating the image quality difference exceeds a first threshold, the data indicating the image quality difference is equal to or less than the first threshold. A signal correction step for correcting at least one of a detection signal of each green detection pixel in the divided area and a detection signal of a pixel forming a pair with each of the green detection pixels; Is provided.

According to the present invention, it is possible to provide an imaging apparatus and a ghost correction method capable of detecting the presence or absence of a ghost by one shooting and correcting the ghost when there is a ghost in any shooting environment. it can.

The figure which shows schematic structure of the imaging device for describing one Embodiment of this invention FIG. 1 is a schematic plan view showing a schematic configuration of the solid-state imaging device 5 shown in FIG. The figure explaining the level difference of the detection signal of the pixel in the area where the ghost has occurred The flowchart for demonstrating operation | movement of the system control part 11 in the digital camera shown by FIG. Diagram for explaining an example of setting a detection area The flowchart for demonstrating the modification of operation | movement of the system control part 11 in the digital camera shown by FIG. The figure which shows the modification of the solid-state image sensor 5 mounted in the digital camera shown in FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a diagram showing a schematic configuration of an imaging apparatus for explaining an embodiment of the present invention. Examples of the imaging device include an imaging device such as a digital camera and a digital video camera, an imaging module mounted on an electronic endoscope, a camera-equipped mobile phone, and the like. Hereinafter, a digital camera will be described as an example.

The imaging system of the digital camera shown in FIG. 1 includes a photographing lens 1 including a focus lens and a zoom lens, a CCD type or MOS type solid-state imaging device 5, a diaphragm 2 provided therebetween, and an infrared ray cut-off. A filter 3 and an optical low-pass filter 4 are provided.

The system control unit 11 that controls the entire electric control system of the digital camera controls the flash light emitting unit 12 and the light receiving unit 13. In addition, the system control unit 11 controls the lens driving unit 8 to adjust the position of the photographing lens 1 to the focus position or perform zoom adjustment. In addition, the system control unit 11 adjusts the exposure amount by controlling the opening amount of the aperture 2 via the aperture driving unit 9.

Further, the system control unit 11 drives the solid-state imaging device 5 by controlling the imaging device driving unit 10 to output a subject image captured through the photographing lens 1 as a captured image signal. An instruction signal from the user is input to the system control unit 11 through the operation unit 14.

The electric control system of the digital camera further includes an analog signal processing unit 6 that performs analog signal processing such as correlated double sampling processing connected to the output of the solid-state imaging device 5, and RGB output from the analog signal processing unit 6. And an A / D conversion circuit 7 for converting the color signal into a digital signal. The analog signal processing unit 6 and the A / D conversion circuit 7 are controlled by the system control unit 11.

Furthermore, the electric control system of this digital camera generates image data by performing main memory 16, memory control unit 15 connected to main memory 16, interpolation calculation, gamma correction calculation, RGB / YC conversion processing, and the like. A digital signal processing unit 17, a compression / decompression processing unit 18 that compresses image data generated by the digital signal processing unit 17 into a JPEG format or expands compressed image data, and a detachable recording medium 21 are connected. An external memory control unit 20 and a display control unit 22 to which a liquid crystal display unit 23 mounted on the back of the camera or the like is connected.

The memory control unit 15, the digital signal processing unit 17, the compression / decompression processing unit 18, the external memory control unit 20, and the display control unit 22 are connected to each other by a control bus 24 and a data bus 25, and commands from the system control unit 11. Controlled by.

FIG. 2 is a schematic plan view showing a schematic configuration of the solid-state imaging device 5 shown in FIG.

The solid-state imaging device 5 includes a first pixel group including a plurality of pixels 52 arranged in a two-dimensional manner (a square lattice shape in the example of FIG. 2) in a row direction X and a column direction Y orthogonal thereto. And a second pixel group made up of a plurality of pixels 53 having the same arrangement.

Each pixel 52 includes a photoelectric conversion unit such as a photodiode and a color filter provided thereabove. The color filter array included in all the pixels 52 is a Bayer array.

Each pixel 53 includes a photoelectric conversion unit such as a photodiode and a color filter provided thereabove. The array of color filters included in all the pixels 53 is a Bayer array.

In FIG. 2, pixels with “R” are pixels that detect red light. Also, the pixels with “G” are pixels that detect green light. Also, the pixels with “B” are pixels that detect blue light. Hereinafter, the

pixels

52 and 53 with “R” are also referred to as an R pixel 52 and an R pixel 53. The

pixels

52 and 53 with “G” are also referred to as G pixel 52 and G pixel 53. The

pixels

52 and 53 with “B” are also referred to as B pixel 52 and B pixel 53.

The first pixel group of the solid-state imaging device 5 includes a GR pixel row, which is a pixel row in which G pixels 52 and R pixels 52 are alternately arranged in the row direction X, and B pixels 52 and G pixels 52 in the row direction X. Two types of pixel rows, including BG pixel rows, which are alternately arranged pixel rows, are included. These two types of pixel rows are alternately arranged in the column direction Y.

Next to each pixel 52 of the first pixel group, a second pixel group that detects light of the same color as each pixel 52 in a predetermined direction with respect to each pixel 52 (in the diagonally downward direction in the example of FIG. 2). Pixels 53 are arranged. Each pixel 52 and a pixel 53 adjacent thereto in the predetermined direction constitute a pixel pair.

The optical aperture 52k of each pixel 52 (opening of the photoelectric conversion unit included in the pixel 52) and the optical aperture 53k of each pixel 53 (opening of the photoelectric conversion unit included in the pixel 53) are as shown in FIG. In the row direction X, they are eccentric in opposite directions. Therefore, parallax occurs between the image data captured by the first pixel group and the image data captured by the second pixel group. This digital camera generates stereoscopic image data that can be stereoscopically viewed using the two pieces of image data having parallax. The optical aperture 52k of each pixel 52 and the optical aperture 53k of each pixel 53 may be decentered in opposite directions in the column direction Y.

When the solid-state imaging device 5 is used for shooting under conditions where ghost is generated (for example, a state where strong light is incident on the light receiving surface obliquely), the captured image signal output from the first pixel group and the second image signal are output. A ghost occurs in each captured image signal output from the pixel group. The first pixel group and the second pixel group have different optical aperture positions. For this reason, the ghost included in the captured image signal output from the first pixel group and the ghost included in the captured image signal output from the second pixel group are not exactly the same. As a result, when stereoscopic image data is reproduced, the appearance of the ghost may become unnatural. Hereinafter, this phenomenon will be described in detail.

Note that a ghost is generated in the captured image signal, but each signal (ghost component) constituting the ghost in the captured image signal and the output source pixel of each signal have a one-to-one correspondence. For this reason, in the present specification, the fact that a ghost component is included in a pixel signal obtained from an arbitrary pixel of the solid-state imaging device 5 will be described as synonymous with the occurrence of a ghost in the arbitrary pixel.

When a strong oblique light that can generate a ghost (for example, an oblique light that goes from left to right in the X direction in FIG. 3) is incident on the solid-state imaging device 5, a broken line in FIG. Focus on the four pixels 52 of the first pixel group in the enclosed area.

Among the four pixels 52 in the first pixel group surrounded by the broken line in FIG. 3, an R pixel 52 exists on the left side of the upper left G pixel 52 (hereinafter also referred to as the upper left G pixel 52), and the upper left G pixel. An R pixel 53 exists on the left side of a G pixel 53 (hereinafter also referred to as an upper left G pixel 53) that forms a pair with 52. In the R pixel 52 and the R pixel 53, the optical aperture is decentered in the opposite direction in the X direction. For this reason, the amount of light leaking from the

R pixels

52 and 53 and leaking to the adjacent upper left G pixel 52 and upper left G pixel 53 is different. As a result, a level difference is generated between the detection signal of the upper left G pixel 52 and the detection signal of the upper left G pixel 53. This level difference includes a component caused by a difference in optical aperture between the G pixel 52 and the G pixel 53 and a component caused by a difference in the amount of light leaking from adjacent pixels.

Also, the upper left G pixel 52 and the left adjacent to each of the lower right G pixel 52 (hereinafter also referred to as the lower right G pixel 52) of the four pixels 52 of the first pixel group surrounded by the broken line in FIG. Are arranged with different types of pixels (R pixel 52 and B pixel 52) having different detection colors. For this reason, a level difference caused by the difference in the amount of light leaking from the adjacent pixel occurs between the detection signal of the upper left G pixel 52 and the detection signal of the lower right G pixel 52.

Also, adjacent pixels between each pixel 52 except for the lower right G pixel 52 among the four pixels 52 of the first pixel group surrounded by the broken line in FIG. A difference in the level of the detection signal due to the difference in the amount of light leaking from the light occurs.

As described above, when the ghost component is included in the detection signals of the four pixels 52 in the region surrounded by the broken line in FIG. 3 (in other words, when the ghost is generated in the region surrounded by the broken line in FIG. 3). In the area surrounded by the broken line in FIG. 3, a level difference equal to or higher than the first threshold value is generated between the detection signal of the upper left G pixel 52 and the detection signal of the lower right G pixel 52.

Further, when a ghost component is included in the detection signals of the four pixels 52 in the area surrounded by the broken line in FIG. 3, the ghost component is also included in the detection signals of the four pixels 53 that form a pair with the four pixels 52. Will be. However, these two ghost components do not match due to the difference in aperture. Therefore, a level difference equal to or greater than the second threshold value is also generated between the detection signal of each pixel 52 in the region surrounded by the broken line in FIG. 3 and the detection signal of the pixel 53 that forms a pair with each pixel 52. .

In the digital camera shown in FIG. 1, the system control unit 11 includes a total of four pixels 52 in two rows and two columns adjacent to the first pixel group of the solid-state imaging device 5 in the row direction X and the column direction Y. For each detection area (area surrounded by a broken line shown in FIG. 3), a difference between detection signals of two G pixels 52 included in the detection area is obtained, and a ghost is detected in this detection area based on the difference between the detection signals. Whether or not has occurred is determined.

When it is determined that a ghost has occurred in the detection area, the system control unit 11 detects the detection signal of each pixel 52 in the detection area, and the detection signal of the pixel 53 that forms a pair with each pixel 52. If the difference is large enough to affect the appearance of the ghost in the stereoscopic image data, reduce this difference to the extent that the ghost does not appear unnatural. Make corrections.

In this way, the system control unit 11 determines the region where the ghost is generated for each of the first pixel group and the second pixel group, and the first control unit 11 in the region where the ghost is generated. The level difference between the detection signals of the first pixel group and the second pixel group is corrected.

Hereinafter, the operation of the system control unit 11 will be described with reference to flowcharts.

FIG. 4 is a flowchart for explaining the operation of the system control unit 11 in the digital camera shown in FIG.

When a release button included in the operation unit 14 is pressed, the system control unit 11 controls the image sensor driving unit 10 to cause the solid-state image sensor 5 to perform imaging. By this imaging, a captured image signal including the first captured image signal output from the first pixel group and the second captured image signal output from the second pixel group is output from the solid-state image sensor 5. The

The output captured image signal is temporarily stored in the main memory 16 after being subjected to analog signal processing and digital conversion processing. The system control unit 11 performs the processing described below on the captured image signal stored in the main memory 16.

First, as shown in FIG. 5, the system control unit 11 uses a plurality of pixels 52 in a total of two rows and two columns as one detection area (area surrounded by a broken line in FIG. 5). Divide into detection areas.

Then, the system control unit 11 calculates a difference (absolute value) between the detection signals of the two G pixels 52 in the arbitrary detection area for the arbitrary detection area (step S1).

Next, the system control unit 11 determines whether or not the difference calculated in step S1 is greater than or equal to a threshold Th1 (step S2).

As a result of the determination in step S2, when the difference calculated in step S1 is less than the threshold Th1, the system control unit 11 determines that no ghost has occurred in the detection area that is the difference calculation target in step S1. . In step S7, the system control unit 11 determines whether or not the processing after step S1 has been completed for all the detection areas. When the determination in step S7 is NO, the system control unit 11 changes the detection area to an unprocessed one, and performs the processing after step S1 again on the changed detection area.

As a result of the determination in step S2, if the difference calculated in step S1 is greater than or equal to the threshold Th1, the system control unit 11 pairs the detection area that is the difference calculation target in step S1 with each pixel 52 in this detection area. It is determined that a ghost is generated in each of the areas of the second pixel group in which the pixels 53 constituting the pixel are arranged (hereinafter referred to as the area of the second pixel group constituting a pair with the detection area) ( Step S3).

Next, the system control unit 11 pairs a detection signal (hereinafter also referred to as a first detection signal) of each G pixel 52 included in the detection area that is a difference calculation target in step S <b> 1 and the pixel 52. A difference (absolute value) from a detection signal (hereinafter also referred to as a second detection signal) of the constituting G pixel 53 is calculated (step S4).

Note that this difference includes the difference between the detection signal of the upper left G pixel 52 included in the detection area that is the difference calculation target in step S1 and the detection signal of the G pixel 53 that forms a pair with the upper left G pixel 52. The difference between the detection signal of the lower right G pixel 52 included in the detection area subjected to the difference calculation in step S1 and the difference between the detection signal of the G pixel 53 that forms a pair with the lower right G pixel 52 is provided. included. However, since the upper left G pixel 52 and the lower right G pixel 52 have the same optical aperture and are close to each other, the two types of difference values are almost the same. The difference calculated in step S4 is the difference in image quality between the ghost that occurs in the detection area that is the difference calculation target in step S1 and the ghost that occurs in the area of the second pixel group that forms a pair with this detection area. It becomes the data to show.

When the difference calculated in step S4 is equal to or greater than the threshold Th2 (step S5: YES), the system control unit 11 sets a ghost generated in the detection area that is a difference calculation target in step S1 and a pair with this detection area. It is determined that there is an image quality difference in the ghost that occurs in the area of the second pixel group that constitutes the image so as to give a sense of discomfort when each ghost is viewed stereoscopically. Then, the system control unit 11 corrects the level of at least one of the first detection signal and the second detection signal so that the difference calculated in step S4 is equal to or less than the threshold Th2 (step S6).

For example, the system control unit 11 detects the detection signal of the upper left G pixel 52, the detection signal of the G pixel 53 that forms a pair with the upper left G pixel 52, the detection signal of the lower right G pixel 52, and the lower right G pixel 52. Of the detection signals of the G pixel 53 constituting the pair, the detection signal of the upper left G pixel 52 and the detection signal of the lower right G pixel 52 are multiplied by a uniform gain to perform correction.

When the difference calculated in step S4 is less than the threshold Th2 (step S5: NO), the system control unit 11 sets a ghost generated in the detection area that is the difference calculation target in step S1 and a pair with this detection area. It is determined that the ghost image generated in the area of the second pixel group to be configured does not have a difference in image quality that gives a sense of incongruity when each ghost image is stereoscopically viewed. And the system control part 11 performs the process of step S7.

If the determination in step S7 is YES, that is, if the processing from step S1 is performed on all detection areas, the system control unit 11 ends the ghost detection and correction operation.

G pixel 52 and G pixel 53 are pixels whose optical apertures are decentered in opposite directions. For this reason, even if there is a level difference of the threshold Th2 or more between the detection signal of the G pixel 52 and the detection signal of the G pixel 53 that forms a pair with the G pixel 52 in each detection area, the G pixel 52 And it cannot be said that a ghost is generated in the area including the G pixel 53.

Therefore, as described above, the system control unit 11 generates a ghost for each detection area of the first pixel group based on the difference between the detection signals of the two G pixels 52 included in each detection area. The presence or absence of is determined. When the difference between the detection signals is equal to or greater than the threshold Th2 for the G pixel 52 in the detection area where it is determined that a ghost has occurred and the G pixel 53 that forms a pair with the G pixel 52. Only, the system control unit 11 corrects the detection signal so that the difference is less than the threshold Th2. By such processing, the area where the ghost is generated can be accurately determined. Even in an area where a ghost is generated, correction can be performed only for a ghost that causes a sense of incongruity when viewed stereoscopically.

Therefore, according to the digital camera of the present embodiment, it is possible to detect ghosts with high accuracy and to make ghosts without any sense of incongruity even when a stereoscopic image is displayed. Further, in this digital camera, since the ghost level is corrected only in a necessary area, it is possible to visually recognize a ghost without a sense of incongruity without increasing the time required for the ghost correction.

In the flowchart of FIG. 4, after step S <b> 6, the system control unit 11 detects the detection signal (third detection signal) of the R pixel 52 in the detection area that is the difference calculation target in step S <b> 1 and the R pixel. The detection signal difference (absolute value) from the detection signal (fourth detection signal) of the R pixel 53 that forms a pair with 52 is calculated. When the detection signal difference is equal to or greater than the threshold Th3, the system control unit 11 corrects at least one of the third detection signal and the fourth detection signal so that the detection signal difference is less than the threshold Th3. Further, the system control unit 11 may perform the process of step S7 when the detection signal difference is less than the threshold Th3.

Further, after the process of correcting at least one of the third detection signal and the fourth detection signal, the system control unit 11 detects the detection signal (the first detection signal of the B pixel 52 in the detection area that is the difference calculation target in step S1. The detection signal difference (absolute value) between the fifth detection signal) and the detection signal (sixth detection signal) of the B pixel 53 that forms a pair with the B pixel 52 is calculated. When the detection signal difference is equal to or greater than the threshold Th4, the system control unit 11 corrects at least one of the fifth detection signal and the sixth detection signal so that the detection signal difference is less than the threshold Th4. Further, when the detection signal difference is less than the threshold Th4, the system control unit 11 may perform the process of step S7.

A difference calculation process between the third detection signal and the fourth detection signal; a process for correcting at least one of the third detection signal and the fourth detection signal; and a fifth detection signal and a sixth detection signal. The process of calculating the difference and the process of correcting at least one of the fifth detection signal and the sixth detection signal may be reversed in order.

Since the ghost is greatly affected by the luminance component included in the detection signal, as shown in the flowchart of FIG. 4, even if only the detection signal of the G pixel is corrected in step S6, the image quality improvement effect of the ghost is sufficiently obtained. Obtainable.

In the above description, the first pixel group is divided into a plurality of detection areas, and the processing after step S1 in FIG. 4 is performed on each detection area. However, the second pixel group may be divided into a plurality of detection areas, and the processing after step S1 in FIG. 4 may be performed on each detection area. In the flowchart in this case, the notation of the first pixel group and the second pixel group in FIG. 4 is reversed.

Further, the system control unit 11 performs the processing of steps S1 to S3 in FIG. 4 for all the detection areas, and then performs step S4 in FIG. 4 for each detection area determined to have a ghost. Step S6 may be performed.

According to such a process, for example, when only the detection area in the upper right of the six detection areas shown in FIG. 5 is a detection area where it is determined that a ghost has occurred, this detection area The detection signal of each pixel 52 included in each pixel 52 and a pixel 53 that forms a pair with each pixel 52 is received from each pixel 52 included in the detection area adjacent to the detection area and determined to have no ghost. The process of step S6 can be performed by a simple process of replacing the detection signal of the pixel 53 that forms a pair with the pixel 52.

Further, the system control unit 11 divides the first pixel group into a plurality of blocks each including a plurality of detection areas shown in FIG. 5, and processing for determining whether or not a ghost has occurred for each divided block. Correction processing may be performed.

FIG. 6 is a flowchart for explaining a modification of the operation of the system control unit 11 in the digital camera shown in FIG.

When the imaging is finished and the captured image signal output from the solid-state imaging device 5 is stored in the main memory 16, the system control unit 11 selects the first pixel group, for example, the detection area shown in FIG. It is divided into a plurality of blocks including four each.

Then, the system control unit 11 has, for an arbitrary divided block, an average value of the detection signals of the G pixels 52 in the GR pixel row in the arbitrary divided block and the BG pixel row in the arbitrary divided block. A difference (absolute value) from the average value of the detection signals of the G pixel 52 is calculated (step S11).

Next, the system control unit 11 determines whether or not the difference calculated in step S11 is greater than or equal to the threshold Th1 (step S12).

As a result of the determination in step S12, when the difference calculated in step S11 is less than the threshold value Th1, the system control unit 11 determines that no ghost has occurred in the divided block subjected to the difference calculation in step S11. In step S17, it is determined whether or not the processes in and after step S11 have been completed for all the divided blocks. When the determination in step S17 is NO, the system control unit 11 changes the divided block to an unprocessed block, and performs the processing from step S11 on the changed divided block again.

If the difference calculated in step S11 is greater than or equal to the threshold Th1 as a result of the determination in step S12, the system control unit 11 determines that a ghost has occurred in the divided block that is the difference calculation target in step S11 (step S11). S3).

Next, the system control unit 11 calculates the average value of the detection signals (hereinafter also referred to as seventh detection signals) of the G pixels 52 included in the divided block that is the difference calculation target in step S11, and the pixels 52. The difference (absolute value) from the average value of the detection signals (hereinafter also referred to as the eighth detection signal) of the G pixel 53 constituting the pair is calculated (step S14). The difference calculated here is a ghost that occurs in the divided block that is the difference calculation target in step S11 and a ghost that occurs in the area of the second pixel group that forms a pair with each detection area included in this divided block. The data indicates the difference in image quality.

When the difference calculated in step S14 is greater than or equal to the threshold Th2 (step S15: YES), the system control unit 11 includes a ghost that occurs in the divided block that is the difference calculation target in step S11 and the divided block. It is determined that an image quality difference is generated in a ghost that occurs in the area of the second pixel group that forms a pair with each detection area, so as to give a sense of incongruity when each ghost is stereoscopically viewed. Then, the system control unit 11 includes at least the seventh detection signal and the eighth detection signal so that the difference between the average value of the seventh detection signal and the average value of the eighth detection signal is equal to or less than the threshold Th2. One level is corrected (step S16).

For example, the system control unit 11 multiplies the detection signal of each G pixel 52 in the divided block by a uniform gain so that the difference between the average value of the seventh detection signal and the average value of the eighth detection signal is a threshold value. It is made to become below Th2.

When the difference calculated in step S14 is less than the threshold Th2 (step S15: NO), the system control unit 11 includes a ghost that occurs in the divided block that is the difference calculation target in step S11 and the divided block. It is determined that a ghost image generated in the area of the second pixel group that forms a pair with each detection area does not have a difference in image quality that gives a sense of discomfort when each ghost image is stereoscopically viewed. And the system control part 11 performs the process of step S17.

If the determination in step S17 is YES, that is, if the processing from step S11 is performed on all the divided blocks, the system control unit 11 ends the ghost detection and correction operation.

As described above, the time required for detecting and correcting the ghost can be shortened by detecting the ghost in the divided block unit instead of the detection area unit and correcting the ghost.

Note that the optimum values of the threshold values Th1 and Th2 described so far are determined by the aperture value of the aperture 2 mounted on the digital camera, the zoom magnification of the zoom lens included in the photographing lens 1, and the like. For this reason, threshold values Th1 and Th2 are stored in the internal memory of the system control unit 11 for each shooting condition such as an aperture value and a zoom magnification, and the system control unit 11 sets a threshold value Th1 corresponding to the set shooting condition. , Th2 are preferably read from the internal memory and used in the processing shown in FIGS. By doing so, it is possible to maintain ghost detection and correction accuracy even when the shooting conditions are changed.

FIG. 7 is a view showing a modification of the solid-state imaging device 5 mounted on the digital camera shown in FIG.

The solid-state imaging device 5a shown in FIG. 7 has a pixel 53 that detects light of the same color as each of the pixels 52 arranged next to the lower direction of each of the pixels 52 in the first pixel group. The configuration is the same as that shown in FIG. In FIG. 7, the shape of each pixel is obtained by rotating each pixel shown in FIG. 2 by 45 degrees, and the shapes of the

optical apertures

52k and 53k are rectangular.

Even in the solid-state imaging device 5a, the area surrounded by the broken line in FIG. 7 is used as a detection area, the first pixel group is divided into a plurality of detection areas, and the processing shown in FIG. 4 is performed on each detection area. By doing so, the same effect as when the solid-state imaging device 5 is used can be obtained.

As described above, the following items are disclosed in this specification.

The disclosed imaging device is an imaging device including a solid-state imaging device having two pixel groups each including a plurality of pixels, and the plurality of pixels of each pixel group are arranged in a two-dimensional manner, and are green The pixel includes three types of pixels: a green detection pixel that detects light, a blue detection pixel that detects blue light, and a red detection pixel that detects red light, and the array of the plurality of pixels in each pixel group includes: A first pixel row in which green detection pixels and blue detection pixels are alternately arranged in a row direction; and a second pixel row in which red detection pixels and green detection pixels are alternately arranged in a row direction. Each pixel of one pixel group of the two pixel groups is alternately arranged in a column direction orthogonal to the direction, and the other pixel of the two pixel groups in a predetermined direction with respect to each pixel. Pixels of the same type as each pixel in the pixel group are placed adjacent to each other. The optical aperture of each pixel of the one pixel group and the optical aperture of each pixel of the other pixel group are decentered in opposite directions, and each pixel of the one pixel group, A pixel pair is formed by pixels of the other pixel group arranged adjacent to the pixel in the predetermined direction, and is included in the first pixel row obtained by dividing the one pixel group For each of the divided areas each including at least one of the green color detection pixels and the green color detection pixel included in the second pixel row, using the detection signal of each green color detection pixel included in the division area. A ghost presence / absence determination unit that determines whether or not a ghost has occurred in the area and the area of the other pixel group in which pixels forming a pair with each green detection pixel in the divided area are disposed; and the ghost presence / absence determination Go by part It is determined that the ghost is generated using the detection signal of each green detection pixel in the divided area where it is determined that the image is generated and the detection signal of the pixel that forms a pair with each green detection pixel. A ghost image quality difference calculating unit for calculating data indicating a difference in image quality of a ghost generated in each of the divided area and the area of the other pixel group, and when the data indicating the image quality difference exceeds a first threshold, At least one of a detection signal of each green detection pixel in the divided area and a detection signal of a pixel forming a pair with each green detection pixel so that the data indicating the image quality difference is equal to or less than the first threshold value. And a signal correction unit for correction.

In the disclosed imaging device, the divided area includes a plurality of green detection pixels included in the first pixel row and a plurality of green detection pixels included in the second pixel row, and the ghost presence / absence determination unit Are an average value of detection signals of the green detection pixels included in the first pixel row included in the divided area, and the green detection pixels included in the second pixel row included in the divided area. When the difference between the average value of the detection signals is equal to or greater than a second threshold value, it is determined that the ghost has occurred in the divided area and the area of the other pixel group, and the ghost image quality difference calculating unit, The average value of the detection signals of the green detection pixels in the divided areas determined by the ghost presence / absence determination unit and the average value of the detection signals of the pixels forming a pair with each green detection pixel difference A and calculates the data indicating the quality difference.

In the disclosed imaging device, the divided area is included in the green detection pixel included in the first pixel row, the blue detection pixel included in the first pixel row, and the second pixel row. The ghost presence / absence determination unit is included in the first pixel row, which is included in the divided area, and includes the green detection pixel and the red detection pixel included in the second pixel row one by one. When the difference between the detection signal of the green detection pixel and the detection signal of the green detection pixel included in the second pixel row included in the division area is equal to or greater than a second threshold, the division area and the It is determined that the ghost is generated in the area of the other pixel group, and the ghost image quality difference calculation unit determines the green detection pixel of each divided area that is determined that the ghost is generated by the ghost presence / absence determination unit. Detection signal The difference between the detection signals of the pixels constituting the respective green detection pixel pair, and calculates the data indicating the quality difference.

In the disclosed imaging apparatus, the first threshold value and the second threshold value are changed according to shooting conditions.

The disclosed ghost correction method is a ghost correction method for correcting a ghost of a captured image obtained by imaging with a solid-state imaging device having two pixel groups each including a plurality of pixels, and the plurality of the ghost correction methods in each pixel group. The two pixels include three types of pixels arranged in a two-dimensional manner: a green color detection pixel that detects green light, a second color detection pixel that detects blue light, and a third color detection pixel that detects red light. The arrangement of the plurality of pixels of each pixel group includes a first pixel row in which the green detection pixels and the blue detection pixels are alternately arranged in a row direction, the red detection pixels, and the green detection pixels. The second pixel rows alternately arranged in the row direction are alternately arranged in the column direction orthogonal to the row direction, and each pixel of one pixel group of the two pixel groups includes: The two images in a predetermined direction with respect to each pixel. Pixels of the same type as the respective pixels of the other pixel group of the group are arranged adjacent to each other, and an optical aperture of each pixel of the one pixel group and an optical aperture of each pixel of the other pixel group Are decentered in opposite directions, and each pixel of the one pixel group and a pixel of the other pixel group arranged adjacent to each pixel in the predetermined direction constitute a pixel pair. Each divided area obtained by dividing the one pixel group and including at least one of the green color detection pixels included in the first pixel row and the green color detection pixels included in the second pixel row. In addition, using the detection signal of each green detection pixel included in the division area, the division area and the area of the other pixel group in which pixels forming a pair with each green detection pixel of the division area are arranged. Whether a ghost has occurred A ghost presence / absence determination step, a detection signal of each green detection pixel in the divided area determined to have a ghost generated by the ghost presence / absence determination step, and a detection signal of a pixel forming a pair with each of the green detection pixels And a ghost image quality difference calculating step for calculating data indicating a difference in image quality between ghosts occurring in each of the divided area and the other pixel group area where it is determined that the ghost has occurred, and When the data indicating the image quality difference exceeds the first threshold value, the detection signal of each green detection pixel in the divided area and each of the green color detection pixels so that the data indicating the image quality difference is equal to or less than the first threshold value. And a signal correction step of correcting at least one of the detection signals of the pixels constituting the pair.

In the disclosed ghost correction method, the divided area includes a plurality of green detection pixels included in the first pixel row and a plurality of green detection pixels included in the second pixel row, and the ghost presence / absence determination is performed. In the step, an average value of detection signals of the green detection pixels included in the first pixel row included in the division area and the green detection included in the second pixel row included in the division area. When the first difference representing the difference between the average value of the detection signals of the pixels is equal to or greater than a second threshold, it is determined that the ghost has occurred in the area of the divided area and the other pixel group, In the ghost image quality difference calculating step, the average value of the detection signals of the green detection pixels in the divided areas determined by the ghost presence / absence determination unit to generate ghost, The difference between the average value of the detection signal of the pixels constituting the, and calculates the data indicating the quality difference.

In the disclosed ghost correction method, the divided area is divided into the green detection pixel included in the first pixel row, the blue detection pixel included in the first pixel row, and the second pixel row. The green detection pixel included and the red detection pixel included in the second pixel row are included one by one, and the ghost presence determination step includes the first pixel row included in the divided area. When the difference between the detection signal of the green detection pixel and the detection signal of the green detection pixel included in the second pixel row included in the divided area is equal to or greater than a second threshold value, It is determined that the ghost is generated in the area of the other pixel group, and in the ghost image quality difference calculating step, the divided area determined that the ghost is generated by the ghost presence / absence determination unit. A detection signal of the green detection pixel, the difference between the detection signals of the pixels constituting the respective green detection pixel pair, and calculates the data indicating the quality difference.

In the disclosed ghost correction method, the first threshold value and the second threshold value are changed according to shooting conditions.

The imaging apparatus and ghost correction method of the present invention can detect the presence or absence of a ghost by one shooting in any shooting environment, and can correct this ghost if there is a ghost.

Although the present invention has been described in detail and with reference to specific embodiments, it will be apparent to those skilled in the art that various changes and modifications can be made without departing from the spirit and scope of the invention.
This application is based on a Japanese patent application filed on June 1, 2011 (Japanese Patent Application No. 2011-123434), the contents of which are incorporated herein by reference.

5 Solid-State Image Sensor 52 Pixel 53 in First

Pixel Group Pixels

52k and 53k in Second Pixel Group Optical Aperture 11 System Control Unit

Claims

An imaging apparatus including a solid-state imaging device having two pixel groups each including a plurality of pixels,
The plurality of pixels in each pixel group are three-dimensionally arranged in two dimensions: a green detection pixel that detects green light, a blue detection pixel that detects blue light, and a red detection pixel that detects red light. Including
Each pixel group includes the three types of pixels arranged in a Bayer shape,
The arrangement of the plurality of pixels in each pixel group includes a first pixel row in which the green detection pixels and the blue detection pixels are alternately arranged in the row direction, and the red detection pixels and the green detection pixels in the row direction. And the second pixel rows arranged in a row in a column direction orthogonal to the row direction,
In each pixel of one of the two pixel groups, a pixel of the same type as that of each pixel of the other pixel group of the two pixel groups is arranged adjacent to each pixel in a predetermined direction. And
The optical aperture of each pixel of the one pixel group and the optical aperture of each pixel of the other pixel group are decentered in opposite directions,
Each pixel of the one pixel group and a pixel of the other pixel group arranged adjacent to each pixel in the predetermined direction constitute a pixel pair,
A divided area obtained by dividing the one pixel group and including at least one of the first color detection pixel included in the first pixel row and the green detection pixel included in the second pixel row. And using the detection signal of each green detection pixel included in the divided area for each area of the other pixel group in which pixels forming a pair with the green detection pixel in the divided area and the divided area are arranged. A ghost presence / absence determination unit that determines whether or not a ghost has occurred,
The ghost is detected using a detection signal of each green detection pixel in the divided area determined that the ghost is generated by the ghost presence / absence determination unit and a detection signal of a pixel forming a pair with each green detection pixel. A ghost image quality difference calculating unit for calculating data indicating a difference in image quality of a ghost generated in each of the divided area determined to be generated and the area of the other pixel group;
When the data indicating the image quality difference exceeds a first threshold value, the detection signal of each green detection pixel in the divided area and each green color detection so that the data indicating the image quality difference is equal to or less than the first threshold value. An imaging apparatus comprising: a signal correction unit that corrects at least one of detection signals of pixels that form a pair with a pixel.
The imaging apparatus according to claim 1,
The divided area includes a plurality of green detection pixels included in the first pixel row and a plurality of green detection pixels included in the second pixel row, respectively.
The ghost presence / absence determination unit is included in the second pixel row included in the divided area and an average value of detection signals of the green detection pixels included in the first pixel row included in the divided area. When the difference between the average value of the detection signals of the green detection pixels is equal to or greater than a second threshold, it is determined that the ghost has occurred in the divided area and the area of the other pixel group,
The ghost image quality difference calculation unit includes an average value of detection signals of each green detection pixel in the divided area determined by the ghost presence / absence determination unit and a pixel that forms a pair with each green detection pixel An image pickup apparatus that calculates a difference from the average value of the detection signals as data indicating the image quality difference.
The imaging apparatus according to claim 1,
The divided area includes the green color detection pixel included in the first pixel row, the blue color detection pixel included in the first pixel row, and the green color detection pixel included in the second pixel row; Each including the red detection pixels included in the second pixel row;
The ghost presence / absence determination unit includes a detection signal of the green detection pixel included in the first pixel row included in the division area, and the green included in the second pixel row included in the division area. When the difference between the detection signal of the detection pixel and the second threshold value or more, it is determined that the ghost has occurred in the area of the divided area and the other pixel group,
The ghost image quality difference calculation unit is configured to detect a detection signal of each green detection pixel in the divided area determined by the ghost presence / absence determination unit and a detection signal of a pixel forming a pair with each green detection pixel An imaging device that calculates the difference between the two as data indicating the image quality difference.
The imaging device according to claim 2 or 3,
The imaging apparatus in which the first threshold value and the second threshold value are changed according to imaging conditions.
A ghost correction method for correcting a ghost of a captured image obtained by imaging with a solid-state imaging device having two pixel groups each including a plurality of pixels,
The plurality of pixels in each pixel group are two-dimensionally arranged in three dimensions: a green detection pixel that detects green light, a blue detection pixel that detects blue light, and a red detection pixel that detects red light. Including
Each pixel group includes the three types of pixels arranged in a Bayer shape,
The arrangement of the plurality of pixels in each pixel group includes a first pixel row in which the green detection pixels and the blue detection pixels are alternately arranged in the row direction, and the red detection pixels and the green detection pixels in the row direction. And the second pixel rows arranged in a row in a column direction orthogonal to the row direction,
In each pixel of one of the two pixel groups, a pixel of the same type as that of each pixel of the other pixel group of the two pixel groups is arranged adjacent to each pixel in a predetermined direction. And
The optical aperture of each pixel of the one pixel group and the optical aperture of each pixel of the other pixel group are decentered in opposite directions,
Each pixel of the one pixel group and a pixel of the other pixel group arranged adjacent to each pixel in the predetermined direction constitute a pixel pair,
For each divided area obtained by dividing the one pixel group and including at least one of the green color detection pixel included in the first pixel row and the green color detection pixel included in the second pixel row. Using the detection signal of each green detection pixel included in the divided area, a ghost is generated in the divided area and the area of the other pixel group in which pixels forming a pair with each green detection pixel in the divided area are arranged. A ghost presence / absence determination step for determining whether or not an occurrence has occurred;
The ghost is detected using a detection signal of each green detection pixel in the divided area determined to have a ghost in the ghost presence / absence determination step and a detection signal of a pixel forming a pair with each green detection pixel. A ghost image quality difference calculating step for calculating data indicating the image quality difference of the ghost generated in each of the divided area determined to be generated and the area of the other pixel group;
When the data indicating the image quality difference exceeds a first threshold value, the detection signal of each green detection pixel in the divided area and each green color detection so that the data indicating the image quality difference is equal to or less than the first threshold value. A ghost correction method comprising: a signal correction step of correcting at least one of detection signals of pixels forming a pair with a pixel.
The ghost correction method according to claim 5,
The divided area includes a plurality of green detection pixels included in the first pixel row and a plurality of green detection pixels included in the second pixel row, respectively.
In the ghost presence / absence determination step, an average value of detection signals of the green detection pixels included in the first pixel row included in the divided area and included in the second pixel row included in the divided area. When the first difference representing the difference from the average value of the detection signals of the green detection pixels is equal to or greater than a second threshold, the ghost is generated in the area of the divided area and the other pixel group. Judgment,
In the ghost image quality difference calculating step, the average value of the detection signals of the green detection pixels in the divided area determined by the ghost presence / absence determination unit to generate a ghost, and the pixels forming a pair with the green detection pixels A ghost correction method for calculating a difference from the average value of the detected signals as data indicating the image quality difference.
The ghost correction method according to claim 5,
The divided area includes the green color detection pixel included in the first pixel row, the blue color detection pixel included in the first pixel row, and the green color detection pixel included in the second pixel row; Each including the red detection pixels included in the second pixel row;
In the ghost presence / absence determination step, a detection signal of the green detection pixel included in the first pixel row included in the divided area and the green color included in the second pixel row included in the divided area. When the difference between the detection signal of the detection pixel and the second threshold value or more, it is determined that the ghost has occurred in the area of the divided area and the other pixel group,
In the ghost image quality difference calculating step, a detection signal of each green detection pixel in the divided area where it is determined by the ghost presence / absence determination unit that a ghost has occurred, and a detection signal of a pixel forming a pair with each green detection pixel The ghost correction method of calculating the difference between the two as data indicating the image quality difference.
The ghost correction method according to claim 6 or 7,
A ghost correction method in which the first threshold value and the second threshold value are changed according to a shooting condition.