JP2013098920A - Defect correction device, defect correction method, and imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a defect correction device, a defect correction method, and an imaging device with which defect correction is performed on two images composing a 3D image without impairing a stereoscopic effect.SOLUTION: A defect correction device detects each defective pixel in a first image, detects each defective pixel in a second image, detects a defective area composed of the defective pixel detected in the first image, detects a defective area composed of the defective pixel detected in the second image, sets a range, in which defect correction is to be performed in the first image, based on the defective pixel detected in the first image and the detected defective area, sets a range, in which defect correction is to be performed in the second image, based on the defective pixel detected in the second image and the detected defective area, performs defect correction processing on the first image, based on the set correction range, and performs defect correction processing on the second image, based on the set correction range.

Description

  The present technology relates to a defect correction apparatus, a defect correction method, and an imaging apparatus, and more particularly, to a defect correction apparatus, a defect correction method, and an imaging apparatus that perform defect correction on two images constituting a 3D image.

  Conventionally, defect detection and defect correction in an image sensor and the like have been widely performed in 2D cameras (Patent Document 1). In the 3D camera, the same method as the 2D camera is applied to the L image and the R image, respectively.

JP 2009-290653 A

  However, in the case of a 3D camera, if defect correction is performed independently for each of the L image and the R image, the image differs from the original L image and R image, and the natural stereoscopic effect is impaired.

  The present technology has been made in view of such a problem of view resolution, and provides a defect correction device, a defect correction method, and an imaging device that can perform defect correction without impairing the 3D stereoscopic effect. Objective.

  In order to solve the above-described problem, the first technique includes a first defect detection unit that detects a defective pixel in the first image, and a second defect detection unit that detects a defective pixel in the second image. In the first image, a defect area composed of defective pixels detected by the first defect detector is detected, and in the second image, the defect pixels detected by the second defect detector are configured. A defect area detection unit for detecting a defect area to be performed, and a first correction for setting a range for performing defect correction in the first image based on a detection result by the first defect detection unit and a detection result by the defect area detection unit A range setting unit; a second correction range setting unit that sets a range for performing defect correction in the second image based on a detection result by the second defect detection unit and a detection result by the defect region detection unit; A first defect correction unit that performs defect correction processing on the first image based on the correction range set by the positive range setting unit, and a second correction based on the correction range set by the second correction range setting unit. It is a defect correction apparatus provided with the 2nd defect correction part which performs the defect correction process of an image.

  The second technique detects a defective pixel in the first image, detects a defective pixel in the second image, and detects a defective area constituted by the defective pixels detected in the first image. A range in which a defective area composed of defective pixels detected in the second image is detected and defect correction is performed in the first image based on the defective pixels detected in the first image and the detected defective areas Is set, a range for performing defect correction in the second image based on the defective pixel detected in the second image and the detected defective area, and the first range based on the set correction range is set. This is a defect correction method in which defect correction processing is performed on an image and defect correction processing is performed on a second image based on a set correction range.

  Furthermore, the third technique receives a light through the optical system and generates a first image, and a first technique receives the light through the optical system and generates a second image. Two imaging units, a first defect detection unit that detects defective pixels in the first image, a second defect detection unit that detects defective pixels in the second image, and the first image, Detecting a defect area composed of defective pixels detected by the defect detection section of the defect and detecting a defect area composed of defect pixels detected by the second defect detection section in the second image A detection unit; a first correction range setting unit that sets a range for performing defect correction in the first image based on a detection result by the first defect detection unit and a detection result by the defect region detection unit; and a second defect Detection results and defect area detection by the detector A second correction range setting unit that sets a range in which defect correction is performed in the second image based on the detection result of the first image, and a defect in the first image based on the correction range set by the first correction range setting unit An imaging apparatus comprising: a first defect correction unit that performs correction processing; and a second defect correction unit that performs defect correction processing of a second image based on the correction range set by the second correction range setting unit. is there.

  According to the present technology, it is possible to correct an image defect without impairing the 3D stereoscopic effect.

FIG. 1 is a diagram for explaining a problem when the conventional technique is applied to defect correction. FIG. 2 is an explanatory diagram illustrating an outline of defect correction performed by the defect correction apparatus according to the present technology. FIG. 3 is a block diagram showing the configuration of the defect correction apparatus. FIG. 4 is a block diagram illustrating a configuration of an imaging apparatus including the defect correction apparatus according to the present technology. FIG. 5 is a flowchart showing a first method of defect correction processing performed by the defect correction apparatus. FIG. 6 is a flowchart showing a first method of defect correction processing. FIG. 7 is a flowchart showing a second method of defect correction processing performed by the defect correction apparatus. FIG. 8 is a flowchart showing a second method of defect correction processing. FIG. 9 is a diagram for explaining the effect of the second method of the defect correction process. FIG. 10 is a flowchart showing a third method of defect correction processing performed by the defect correction apparatus.

Hereinafter, embodiments of the present technology will be described with reference to the drawings. However, the present technology is not limited to the following embodiments. The description will be given in the following order.
<1. Embodiment>
[1-1. Overview of defect correction processing]
[1-2. Configuration of defect correction apparatus]
[1-3. Configuration of imaging apparatus including defect correction apparatus]
[1-4. First Method of Defect Correction Processing]
[1-5. Second Method of Defect Correction Processing]
[1-6. Third Method of Defect Correction Processing]
<2. Modification>

<1. Embodiment>
[1-1. Overview of defect correction processing]
First, an outline of defect correction performed by the defect correction apparatus according to the present technology will be described with reference to FIGS. 1 and 2. The defect correction apparatus is mounted on, for example, a two-lens imaging apparatus that has an optical imaging system such as an imager, two image engines for the left eye and one for the right eye, and has a three-dimensional image (3D image) imaging function. Used.

  FIG. 1 illustrates a correction process for a defective pixel independently for each of a left-eye image (hereinafter referred to as an L image 10) and a right-eye image (hereinafter referred to as an R image 20) without using the present technology. It is a figure which shows the image at the time of giving.

  Here, the defective pixel means a pixel having a defect that outputs an abnormal image signal instead of an original image signal for some reason and causes an image defect such as intermittent or unclear image in the output image. Defective pixels may occur at the time of manufacture, or may occur due to deterioration over time associated with the use of the imaging device.

  In addition, the defect correction process is performed, for example, by replacing a pixel to be corrected, which is a defective pixel, with an average value of normal pixels in the eight directions around the pixel to be corrected.

  The two images in the upper part of FIG. 1 are the L image 10 and the R image 20 before the defect correction process is performed. 1 shows the L image 10 and the R image 20 after defect correction. The squares in each image represent pixels. In the upper part of FIG. 1, black pixels represent defective pixels. Furthermore, in the lower part of FIG. 1, pixels with diagonal lines represent pixels that have been subjected to correction processing.

  As shown in FIG. 1, the L image 10 includes a plurality of small defects 11, 11, 11 and a large defect composed of a plurality of adjacent defective pixels (hereinafter, composed of a plurality of adjacent defective pixels and having a size equal to or larger than a predetermined size). The defect area 12 is called a defect area.) The R image 20 has a plurality of small defects 21, 21, 21.

  When applying a normal two-dimensional image (2D) defect correction process to the L image 10 and the R image 20 photographed by a two-lens imaging device for 3D photographing, the L image 10 and the R image 20 are independent of each other. The defect correction process is performed. When defect correction processing is performed independently on each of the L image 10 and the R image 20 shown in the upper part of FIG. 1, the image is shown in the lower part of FIG.

  As can be seen from the lower part of FIG. 1, the small defect 11 in the L image 10 is corrected, and the small defect 21 in the R image 20 is also corrected. Further, since the L image 10 has a defect area 12 which is a large defect, the defect area 12 is also corrected. Since there is no defect area in the R image 20, defect correction is not performed in a range 22 corresponding to a large defect in the L image 10 in the R image 20.

  As a result, the L image 10 and the R image 20 have a large difference between the defect area 12 of the L image 10 and the range 22 corresponding to the defect area of the L image in the R image. In the 3D image, a slight difference between the L image and the R image greatly affects the 3D stereoscopic effect. Therefore, as shown in the lower part of FIG. 1, if there is a large difference between the L image 10 and the R image 20, The three-dimensional effect will be impaired. Note that even if small defects are individually corrected for the L image and the R image, there is little risk of impairing the 3D stereoscopic effect due to the size.

  Therefore, the defect correction apparatus according to the present technology performs defect correction as shown in FIG. The upper part of FIG. 2 shows the L image 10 and the R image 20 before the defect correction process is performed. In the lower part of FIG. 2, the L image 10 and the R image 20 after defect correction are shown in the lower part.

  Also in the defect correction apparatus according to the present technology, defect correction is performed independently for the L image 10 and the R image 20 for small defects that are considered to be less likely to impair the 3D stereoscopic effect. On the other hand, the defect area 12 composed of a plurality of adjacent defective pixels existing in the L image 10 has a large area. Therefore, when defect correction is performed only on the L image, there is a large difference between the L image 10 and the R image 20. May occur and the 3D stereoscopic effect may be impaired.

  Accordingly, defect correction is performed on the defect area 12 of the L image 10 and, as shown in the lower part of FIG. 2, the area 23 corresponding to the defect area of the L image in the R image 20 is also regarded as a defect area and is similarly corrected. I do. This is performed even when there is no defective pixel at a position corresponding to the defective area 12 of the L image 10 in the R image 20. As a result, since the defect correction is performed at the same position in the L image and the R image, it is possible to prevent the 3D stereoscopic effect from being impaired due to a large difference between the L image 10 and the R image 20. It becomes possible. In addition, although there is no defect in this way, a range that is regarded as a defect region and is subject to defect correction is referred to as a “deemed defect region”.

  In the above description, the range corresponding to the defect area 12 of the L image 10 is defined as a defect area only in the R image, but the reverse is also true, and if the R image 20 has a defect area, it corresponds to that. The range becomes a deemed defect area in the L image.

[1-2. Configuration of defect correction apparatus]
FIG. 3 is a block diagram illustrating a configuration of the defect correction apparatus 100 according to the present technology. The defect correction apparatus 100 is mounted and used, for example, in an optical imaging system such as an imager, a two-lens imaging apparatus having a 3D image shooting function that includes two image engines for the L side and for the R side. It is done.

  The defect correction apparatus 100 includes an L-side defect detection unit 111, an L-side defect correction unit 112, an L-side correction range setting unit 113, an R-side defect detection unit 121, an R-side defect correction unit 122, an R-side correction range setting unit 123, The defect area detection unit 130 is configured. These are realized by, for example, a circuit, an IC (Integrated Circuit), a microcomputer or the like having the function. Further, it comprises a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and the CPU executes various processes according to programs stored in the ROM, thereby realizing the functions of the respective units. You may do it.

  An L image that is an image for the left eye is input to the L-side defect detection unit 111 and the L-side defect correction unit 112. An R image, which is a right-eye image, is input to the R-side defect detection unit 121 and the R-side defect correction unit 122. For example, the L image is an image generated by an optical imaging system and an image sensor for taking an L image, and the R image is an image generated by an optical imaging system and an image sensor for taking an R image, for example.

  The optical imaging system includes a lens for condensing light from a subject on an imaging device, a driving mechanism for moving a lens to perform focusing and zooming, a shutter mechanism, an iris mechanism, and the like. The optical image of the subject obtained through the optical imaging system is formed on the imaging element.

  The image sensor photoelectrically converts incident light from the subject, converts it into a charge amount, and outputs it as an image signal. The imaging signal output from the imaging element 12 is output to the L-side defect detection unit 111 and the L-side defect correction unit 112. As the image sensor, a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like is used.

  The L-side defect detection unit 111 detects a defect that occurs in an image sensor that generates an L image. Various known detection methods can be employed as the defect detection method. For example, the pixel to be detected is compared with the average value of the pixels in the eight directions around the pixel, and the pixel to be detected is determined as a defective pixel when it is separated from the average value by a predetermined amount. There is a way.

  Further, a test is performed on the image sensor before shipment from a factory or the like, and addresses of all defective pixels obtained as a result of the test are stored in a non-volatile memory (not shown), and defects are determined based on the addresses. There is also a method of detecting pixels. Similarly, the R-side defect detection unit 121 detects a defect that occurs in an image sensor that generates an R image.

  L-side defect pixel information indicating a defective pixel detected by the L-side defect detection unit 111 is supplied to the L-side correction range setting unit 113 and the defect area detection unit 130. The R-side defect pixel information detected by the R-side defect detection unit 121 is supplied to the R-side correction range setting unit 123 and the defect area detection unit 130.

  The defective area detection unit 130 detects an area constituted by a plurality of adjacent defective pixels in the L image based on the defective pixel information supplied from the L-side defect detection unit 111. A region constituted by a plurality of adjacent defective pixels in the L image is referred to as an L-side defective region. Further, based on the defective pixel information supplied from the R-side defect detection unit 121, a region constituted by a plurality of adjacent defective pixels in the R image is detected. A region constituted by a plurality of adjacent defective pixels in the R image is referred to as an R-side defective region. Details of the defect area detection will be described later.

  The defect area detection unit 130 supplies defect area information indicating the position of the defect area in the L image and the R image to the L side correction range setting unit 113 and the R side correction range setting unit 123.

  The L-side correction range setting unit 113 sets a range for performing defect correction in the L image based on the defective pixel information from the L-side defect detection unit 111 and the defect region information from the defect region detection unit 130. L-side correction range information indicating a correction range in the L image is supplied to the L-side defect correction unit 112.

  The R-side correction range setting unit 123 sets a range for performing defect correction in the R image based on the defective pixel information from the R-side defect detection unit 121 and the defect region information from the defect region detection unit 130. R-side correction range information indicating the correction range in the R image is supplied to the R-side defect correction unit 122. The details of determining the range for defect correction will be described later.

  The L-side defect correction unit 112 performs correction processing on the L image based on the correction range information supplied from the L-side correction range setting unit 113. As the correction process, various known correction methods can be used. For example, the correction is performed by replacing a correction target pixel that is a defective pixel with an average value of normal pixels in eight directions around the correction target pixel. An image signal that has been subjected to defect correction processing by the L-side defect correction unit 112, for example, a camera processing unit of the imaging apparatus 200 is supplied.

  The R-side defect correction unit 122 performs correction processing on the R image based on the correction range information supplied from the R-side correction range setting unit 123. The L image and the R image that have been subjected to defect correction are supplied to a camera processing unit or the like of an imaging apparatus to which the defect correction apparatus is applied. The defect correcting apparatus is configured as described above.

[1-3. Configuration of imaging apparatus including defect correction apparatus]
The defect correction apparatus 100 configured as described above is mounted on a two-lens imaging apparatus 200 having a 3D image capturing function including two optical imaging systems, such as an imager, and two image engines for the left eye and the right eye. To be used. FIG. 4 is a block diagram illustrating a configuration of an imaging apparatus 200 having a defect correction apparatus function and a 3D imaging function.

  The imaging apparatus 200 includes an L-side optical imaging system 211, an L-side imaging element 212, an L-side timing generator 213, an L-side preprocessing circuit 214, and an L-side camera processing circuit 215 for taking an L image. In addition, an R-side optical imaging system 221 for capturing an R image, an R-side imaging element 222, an R-side timing generator 223, an R-side preprocessing circuit 224, and an R-side camera processing circuit 225 are provided. Furthermore, a control unit 231, a 3D image compression unit 232, a storage unit 233, a display unit 234, and an input unit 235 are provided.

  Further, the L-side defect detection unit 111, the L-side defect correction unit 112, the L-side correction range setting unit 113, the R-side defect detection unit 121, the R-side defect correction unit 122, and the R-side correction range setting that constitute the defect correction apparatus. Part 123 and defective area detection part 130 are also provided.

  The L-side optical imaging system 211 includes a lens for condensing light from a subject on an imaging device, a driving mechanism for moving a lens to perform focusing and zooming, a shutter mechanism, an iris mechanism, and the like. . These are driven based on a control signal from the control unit 231. The optical image of the subject obtained through the L-side optical imaging system 211 is formed on the L-side imaging element 212 as an imaging device.

  The L-side image sensor 212 is driven based on the timing signal output from the L-side timing generator 213, photoelectrically converts incident light from the subject, converts it into a charge amount, and outputs it as an image signal. The image signal output from the L-side image sensor 212 is supplied to the L-side defect correction unit 112 and the L-side defect detection unit 111. As the L-side image sensor 212, a CCD, a CMOS, or the like is used. The L-side timing generator 213 outputs a timing signal to the L-side image sensor 212 under the control of the control unit 231.

  The image signal that has been subjected to the defect correction processing by the L-side defect correction unit 112 is supplied to the L-side preprocessing circuit 214. The L-side preprocessing circuit 214 performs sample holding on the image signal so as to maintain a good S / N (Signal / Noise) ratio by CDS (Correlated Double Sampling) processing. Furthermore, the gain is controlled by AGC (Auto Gain Control) processing, A / D (Analog / Digital) conversion is performed, and digital image data is output.

  The L-side camera processing circuit 215 performs signal processing such as white balance adjustment processing, color correction processing, gamma correction processing, Y / C conversion processing, and AE (Auto Exposure) processing on the image data from the preprocessing circuit. . Image data that has undergone predetermined processing by the L-side camera processing circuit 215 is supplied to the 3D image compression unit 232.

  An R-side optical imaging system 221, an R-side imaging element 222, an R-side timing generator 223, an R-side preprocessing circuit 224, and an R-side camera processing circuit 225 for capturing an R image are configured to capture the above-described L image. The explanation is omitted because it is the same as.

  The control unit 231 includes, for example, a CPU, a RAM, and a ROM. The ROM stores a program that is read and operated by the CPU. The RAM is used as a work memory for the CPU. The CPU controls the entire imaging apparatus 200 by executing various processes in accordance with programs stored in the ROM and issuing commands. The control unit 231 also functions as an L-side correction range setting unit 113, an R-side correction range setting unit 123, and a defective area detection unit 130 by executing a predetermined program.

  The L image data and the R image data supplied to the 3D image compression unit 232 are respectively compressed by the 3D image compression unit 232, stored in one 3D image file, and finally stored in the storage unit 233. The 3D image file may include object distance information, parallax information, and information related to the convergence angle, etc., together with the 3D compressed image data. The storage unit 233 is a mass storage medium such as a hard disk or a nonvolatile memory.

  The display unit 234 is a display unit configured by, for example, an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an organic EL (Electro Luminescence) panel, or the like. The display unit 234 displays a through image being captured, an image recorded in the storage unit 233, and the like.

  The input unit 235 includes, for example, a power button for switching power on / off, a release button for instructing start of recording of a captured image, an operator for zoom adjustment, and a touch screen configured integrally with the display unit 234. It is an input means consisting of. When an input is made to the input unit 235, a control signal corresponding to the input is generated and output to the control unit 231. Then, the control unit 231 performs arithmetic processing and control corresponding to the control signal.

  In FIG. 4, the L image is configured such that a predetermined process is performed by the L side pre-processing circuit 214 and the L side camera processing circuit 215 after the defect correction process is performed by the L side defect correction unit 112. Yes. However, the configuration is not limited thereto, and defect correction by the L-side defect correction unit 112 may be performed on the L image after processing by the L-side preprocessing circuit 214 and the L-side camera processing circuit 215. . The same applies to the R-side defect correction unit 122.

[1-4. First Method of Defect Correction Processing]
Next, the defect correction process performed by the defect correction apparatus will be described. 5 and 6 are flowcharts showing a first method of defect correction processing.

  First, in step S101, the shutter of the imaging apparatus to which the defect correction apparatus is applied is closed so that both the L image and the R image become black images. As a result, a pixel level value of 0 is obtained for a pixel without a defect, and a pixel level value greater than 0 level is obtained for a defective pixel. Thereby, a defective pixel can be detected.

  Next, in step S102, the L-side defect detection unit 111 scans the entire L image with a first threshold that is a threshold for the pixel level, and has a level that exceeds the first threshold among all the pixels constituting the L image. Detect a pixel. And the 1st L side map is created from the result. The first L-side map corresponds to L-side defective pixel information indicating the position of the defective pixel exceeding the first threshold in the L image.

  In step S103, the R-side defect detection unit 121 scans the entire R image with a first threshold that is a threshold with respect to the pixel level, and has a level exceeding the first threshold among all the pixels constituting the R image. Detect a pixel. And the 1st R side map is created from the result. The first R-side map corresponds to R-side defective pixel information indicating the position of the defective pixel exceeding the first threshold in the R image. The first threshold used when scanning the L image and the first threshold used when scanning the R image are the same value.

  Next, in step S104, the L-side defect detection unit 111 scans the entire L image with a second threshold that is a threshold with respect to the pixel level, and has a level exceeding the second threshold among all the pixels constituting the L image. Detect a pixel. And the 2nd L side map is created from the result. The second L-side map also corresponds to L-side defective pixel information indicating the position of the defective pixel exceeding the second threshold in the L image. Note that the second threshold is set to a value smaller than the first threshold described above.

  In step S105, the R-side defect detection unit 121 scans the entire R image with a second threshold that is a threshold with respect to the pixel level, and has a level exceeding the second threshold among all the pixels constituting the R image. Detect a pixel. Then, a second R side map is created from the result. The second R-side map also corresponds to R-side defective pixel information indicating the position of the defective pixel exceeding the second threshold in the R image. Note that the second threshold value used when scanning the L image and the second threshold value used when scanning the R image are the same value.

  The first L-side map and the second L-side map, which are L-side defective pixel information, are supplied to the defective area detection unit 130 and the L-side correction range setting unit 113. The first R-side map and the second R-side map, which are R-side defective pixel information, are supplied to the defective area detection unit 130 and the R-side correction range setting unit 123.

In addition, the process by the L side defect detection part 111 and the R side defect detection part 121 is performed in parallel.
In addition, the order in which Steps S102 to S105 are performed is not limited to the order described above, and the order is not limited. The scan based on the second threshold may be performed before the scan based on the first threshold.

  Next, in step S106, the defective area detection unit 130 extracts one pixel from a pixel (referred to as a first defective pixel) having a level exceeding the first threshold value represented in the first L-side map. Next, in step S107, it is determined whether or not there is a pixel adjacent to the first defective pixel among the pixels exceeding the second threshold (referred to as a second defective pixel) with reference to the second L-side map. As a result, a “defect region” composed of a plurality of adjacent defective pixels is detected. If there is an adjacent defective pixel, the process proceeds to step S108 (Yes in step S107). The L-side defect area information indicating the detection result is supplied to the L-side correction range setting unit 113.

  In step S108, the L-side correction range setting unit 113 adds, to the first L-side map, the position of the pixel exceeding the second threshold determined to be adjacent to the pixel exceeding the first threshold based on the L-side defect area information. This step S108 is a process in which pixels adjacent to the first threshold-exceeded pixel among the pixels exceeding the second threshold and the first threshold-exceeded pixel are combined into one map. As a result, a defect area constituted by the first pixel and the second pixel adjacent to the first pixel is set as the correction range.

  Then, the process proceeds to step S109. If it is determined in step S107 that there is no adjacent pixel, the process also proceeds to step S109 (No in step S107).

  Next, in step S109, the defect correction apparatus determines whether or not the processing in steps S106 to S108 has been performed for all the first defective pixels in the first L-side map. If all the first defective pixels have not been processed, the process proceeds to step S106 (No in step S109). Then, Steps S106 to S109 are repeated until processing is performed for all the first defective pixels in the first L-side map. When the processes in steps S106 to S108 have been performed for all the first defective pixels in the first L-side map, the process proceeds to step S110 (Yes in step S109).

  Next, in step S110, the defective area detection unit 130 extracts one pixel from a pixel (referred to as a first defective pixel) having a level exceeding the first threshold value represented in the first R-side map. Next, in step S111, it is determined whether or not a pixel adjacent to the first defective pixel exists among pixels exceeding the second threshold (referred to as a second defective pixel) with reference to the second R-side map. As a result, a “defect region” composed of a plurality of adjacent defective pixels is detected. If there is an adjacent defective pixel, the process proceeds to step S112 (Yes in step S110). The defective area information indicating the detection result is supplied to the R-side correction range setting unit 123.

  In step S110, the R-side correction range setting unit 123 adds, to the first R-side map, the position of the pixel exceeding the second threshold determined to be adjacent to the pixel exceeding the first threshold based on the R-side defect area information. . This step S110 is processing that attempts to combine the pixels that are adjacent to the pixel exceeding the first threshold among the pixels exceeding the second threshold and the pixels exceeding the first threshold into one map. As a result, a defect area constituted by the first pixel and the second pixel adjacent to the first pixel is set as the correction range.

  Then, the process proceeds to step S113. In addition, also when it determines with the adjacent pixel not existing by step S111, a process also progresses to step S113 (No of step S111).

  Next, in step S113, the defect correction apparatus 100 determines whether or not the processing in steps S110 to S112 has been performed on all the first defective pixels in the first R-side map. If all the first defective pixels have not been processed, the process proceeds to step S110 (No in step S113). Then, Steps S110 to S113 are repeated until processing is performed for all the first defective pixels in the first R-side map.

  If it is determined in step S113 that the processes in steps S110 to S112 have been performed for all the first defective pixels in the first R-side map, the process proceeds to step S114.

  In step S114, the L-side correction range setting unit 113 adds the defect area of the R image indicated in the defect area information supplied from the defect area detection unit 130 to the first L-side map. In step S115, the R-side correction range setting unit 123 adds the defect area of the L image indicated by the defect area information supplied from the defect area detection unit 130 to the first R-side map. Thereby, a deemed defect area is added to each of the first L-side map and the first R-side map. Note that this process is not performed when there is no defective area.

  The first L-side map created by the above processing becomes L-side correction range information indicating a range in which defect correction is performed on the L image. The first L-side map is supplied from the L-side correction range setting unit 113 to the L-side defect correction unit 112. Further, the first R-side map created by the above processing becomes R-side correction range information indicating a range in which R is subjected to defect correction in the image. The first R-side map is supplied from the R-side correction range setting unit 123 to the R-side defect correction unit 122.

  In step S116, the L-side defect correction unit 112 corrects the defect of the L image based on the first L-side map. In step S117, the R-side defect correction unit 122 corrects the defect of the R image based on the first R-side map.

  In the first L-side map, a defective pixel and a defective area in the L image, and further, a deemed defective area corresponding to the defective area in the R image are represented. Further, the first R-side map shows a defective pixel and a defective area of the R image, and an assumed defective area corresponding to the defective area of the L image. Therefore, when the L-side defect correction unit 112 performs defect correction based on the first L-side map, in the L image, the defect pixel of the L image, the defect region of the L image, and the deemed defect corresponding to the defect region of the R image. Correction is performed for the region. Similarly, when the R-side defect correction unit 122 performs correction based on the first R-side map, in the R image, regarding the defective pixel of the R image, the defective region of the R image, and the assumed defective region corresponding to the defective region of the L image. Correction is performed. Thereby, the defect correction as shown in FIG. 2 can be performed.

  Thus, in the first method of defect correction, the first threshold value and the second threshold value that is smaller than the first threshold value are used. Thereby, even if the defective pixel is isolated only by the first threshold, it is possible to detect a defective area having a large area composed of a plurality of defective pixels by combining defective pixels having a pixel level equal to or higher than the second threshold. It becomes.

[1-5. Second Method of Defect Correction Processing]
Next, a second method of defect correction processing will be described. 7 and 8 are flowcharts showing a second method of defect correction processing. Steps S101 to S117 are the same as those in the first method, and thus description thereof is omitted. The second method is different from the first method in that step S201 and step S202 are performed.

  In step S201, the L-side correction range setting unit 113 calculates the amount of parallax between the L image and the R image, adds a horizontal value to the non-defect area only in the first L-side map according to the amount of parallax, and regards the assumed defect area. Is shifted by the amount of parallax. Note that the parallax is defined as the difference in the horizontal position of the foreground subject included in each of the L image and the R image. As the parallax increases, the depth of the subject displayed in 3D increases, that is, the degree of unevenness increases. The parallax can also be defined as a difference obtained by subtracting the distance from the left end of the image on the R image in the foreground from the distance from the left end of the image on the L image in the foreground.

  Similarly, in step S202, the R-side correction range setting unit 123 calculates the amount of parallax between the L image and the R image, and adds a horizontal value to the non-existent defective area only in the first R-side map according to the amount of parallax, The position of the deemed defect area is shifted by the amount of parallax.

  Here, the effect of the above-described processing in the above-described second method will be described with reference to FIG. The upper part of FIG. 9 shows the L image 10 and the R image 20 to which the second method is not applied. The lower part of FIG. 9 shows the L image 10 and the R image 20 to which the second method is applied.

  There is a parallax between the L image 10 and the R image 20 constituting the 3D video. The parallax increases when the subject in focus is close, such as foreground photography. When the parallax increases, as shown in FIG. 9, the position of the subject (star in FIG. 9) in the image is shifted by the amount of parallax between the L image 10 and the R image 20.

  In this state, as shown in the L image 10 of FIG. 9, when the defective region 40 exists so as to overlap the subject (star) 30, the position of the defective region 40 is directly associated with the R image and the R image 20. In the R image 20, the deemed defect area 50 does not overlap the star.

  When defect correction processing is performed on the L image 10 and the R image 20 in this state, the defect area 40 that overlaps the subject (star) 30 of the L image 10 is corrected. However, in the R image 20, since the deemed defect area 50 does not overlap the subject (star) 30, the area on the subject (star) 30 is not corrected for defects. Then, a difference occurs between the L image 10 and the R image 20 on the subject (star) 30, and the 3D stereoscopic effect is impaired.

  Therefore, as shown in the lower part of FIG. 9, the disparity shift is eliminated by shifting the deemed defect area 50 by the amount of parallax, and both the L image 10 and the R image 20 are defective on the subject (star) 30. Correction will be performed. Thereby, it is possible to prevent a difference between the L image 10 and the R image 20. Note that this second method need not be used when a defective pixel overlaps an unfocused background or the like.

[1-6. Third Method of Defect Correction Processing]
Next, a third method of defect correction processing will be described. First, in step S301, the shutter of the image pickup apparatus provided with the defect correction apparatus is closed to make both L and R black screens. As a result, both the L image and the R image are black images, and a pixel level value of 0 is obtained for a pixel having no defect, and a pixel level value greater than 0 level is obtained for a defective pixel. Thereby, a defective pixel can be detected.

  Next, in step S302, the L-side defect detection unit 111 scans the entire L image and detects a pixel having a pixel level of 0 or more among all the pixels constituting the L image. Then, a third L side map is created from the result. A pixel having a pixel level of 0 or more is defined as a defective pixel. The third L-side map corresponds to L-side defective pixel information indicating the position of the defective pixel in the L image.

  Next, in step S303, the R-side defect detection unit 121 scans the entire R image and detects pixels having a pixel level of 0 or more among all the pixels constituting the R image. And the 1st R side map is created from the result. A pixel having a pixel level of 0 or more is a defective pixel. The third R-side map corresponds to R-side defective pixel information indicating the position of the defective pixel in the R image.

  Next, in step S304, the defective area detection unit 130 detects an area (hereinafter referred to as a defective area) in which a predetermined number or more of defective pixels are adjacent in the L image based on the first L-side map. The predetermined number is, for example, “2 × 2 (4)”, “3 × 3 (9)”, “4 × 4 (16)”, and the like. The L-side defect area information indicating the detected L-side defect area is supplied from the defect area detection unit 130 to the L-side correction range setting unit 113 and the R-side correction range setting unit 123.

  For example, the predetermined number may be set according to the parallax. For example, it may be set according to the zoom magnification, the focus position, and the like of the imaging apparatus 200. For example, based on the focus position, zoom magnification, etc., it is comprehensively determined whether the near-field shooting or the far-field shooting is performed, and in the case of a foreground, the parallax is large and is set to “4 × 4 (16)” or more. On the other hand, in the case of a distant view, the parallax is small and “2 × 2 (four)” is set.

  This is because the near view has a larger parallax than the distant view, and therefore there is a slight difference between the L image and the R image, and the difference may impair the 3D stereoscopic effect. Therefore, in the case of a foreground, the number of pixels detected as a defect area (in other words, the area of the defect area) is reduced compared to the case of a distant view, and an area adjacent to a small number of defective pixels is also regarded as an L image. It is necessary to correct with the R image.

  Next, in step S305, the defective area detection unit 130 similarly detects a defective area in the R image based on the first R-side map. The R-side defect area information indicating the detected R-side defect area is supplied from the defect area detection unit 130 to the R-side correction range setting unit 123 and the L-side correction range setting unit 113.

  In step S306, the L-side correction range setting unit 113 regards the defective area of the R image as a defective area and adds it to the third L-side map. The third L-side map that is correction range information is supplied from the L-side correction range setting unit 113 to the L-side defect correction unit 112.

  In step S307, the R-side correction range setting unit 123 adds the defect area of the L image to the third R-side map. The third R-side map that is correction range information is supplied from the R-side correction range setting unit 123 to the R-side defect correction unit 122.

  In step S308, the L-side defect correction unit 112 corrects the defect of the L image based on the third L-side map. In step S309, the R-side defect correction unit 122 corrects the defect of the R image based on the third R-side map.

  In the third L-side map, a defective pixel in the L image, a defective region, and an assumed defective region corresponding to the defective region in the R image are represented. In addition, the third R-side map shows a defective pixel in the R image, a defective region, and an assumed defective region corresponding to the defective region in the L image. Therefore, when the L-side defect correction unit 112 performs correction based on the third L-side map, in the L image, correction is performed for the defective pixel and the defect region of the L image, and further, the assumed defect region corresponding to the defect region of the R image. Is called. Similarly, when the R-side defect correction unit 122 performs correction based on the third R-side map, in the R image, the defect pixel and the defect region of the R image, and further, the assumed defect region corresponding to the defect region of the L image are corrected. Done. Thereby, the defect correction as shown in FIG. 2 is performed.

<2. Modification>
Although one embodiment of the present technology has been specifically described above, the present technology is not limited to the above-described embodiment, and various modifications based on the technical idea of the present technology are possible.

  The present technology can also be applied to a two-lens digital still camera having a 3D image shooting function and a two-lens digital video camera capable of shooting a 3D moving image.

  In addition, this technique can also take the following structures.

(1) a first defect detection unit that detects defective pixels in the first image;
A second defect detector for detecting defective pixels in the second image;
In the first image, a defect area constituted by the defective pixels detected by the first defect detection unit is detected, and in the second image, detected by the second defect detection unit. A defect area detector for detecting a defect area constituted by the defective pixels;
A first correction range setting unit that sets a range for performing defect correction in the first image based on a detection result by the first defect detection unit and a detection result by the defect region detection unit;
A second correction range setting unit that sets a range for performing defect correction in the second image based on a detection result by the second defect detection unit and a detection result by the defect region detection unit;
A first defect correction unit that performs a defect correction process on the first image based on the correction range set by the first correction range setting unit;
A defect correction apparatus comprising: a second defect correction unit that performs defect correction processing on the second image based on the correction range set by the second correction range setting unit.

(2) The defect correction apparatus according to (1), wherein the defect area detection unit detects an area including a plurality of adjacent defective pixels as the defect area.

(3) The first correction range setting unit further sets a range in which defect correction is performed based on the defect region in the second image detected by the defect region detection unit. The defect correction apparatus as described in 2).

(4) The first correction range setting unit is detected by the defective pixel in the first image, the defective region in the first image detected by the defective region detection unit, and the defective region detection unit. The defect correction apparatus according to any one of (1) to (3), wherein a range corresponding to the defect area of the second image is set as a range for performing defect correction in the first image.

(5) The first defective pixel detection unit compares a first threshold value with a pixel level of the first image, and further sets a second threshold value set to a value smaller than the first threshold value. Comparing the pixel level of the first image, detecting a pixel having a pixel level equal to or higher than the first threshold or the second threshold as a defective pixel;
The defective area detection unit includes a defective pixel having a pixel level equal to or higher than the first threshold and a defect having a pixel level equal to or higher than the second threshold adjacent to the defective pixel having a pixel level equal to or higher than the first threshold. The defect correction apparatus according to any one of (1) to (4), wherein an area including pixels is detected as the defect area.

(6) The first correction range setting unit defines a range corresponding to the defect area of the second image detected by the defect area detection unit in the first image as the first image. The defect correction apparatus according to any one of (1) to (5), wherein a range in which defect correction is performed by shifting the parallax of the second image is set.

(7) The second correction range setting unit further sets a range for performing defect correction based on the defect region in the first image detected by the defect region detection unit. 6) The defect correction apparatus in any one of.

(8) The second correction range setting unit is detected by the defective pixel in the second image, the defective region in the second image detected by the defective region detection unit, and the defective region detection unit. The defect correction apparatus according to any one of (1) to (7), wherein a range corresponding to the defect area of the first image is set as a range for performing defect correction in the second image.

(9) The second defective pixel detection unit compares a first threshold value with a pixel level of the second image, and further, a second threshold value set to a value smaller than the first threshold value, A pixel level of the second image is compared, and a pixel having a pixel level equal to or higher than the first threshold value or the second threshold value is detected as a defective pixel;
The defective area detection unit includes a defective pixel having a pixel level equal to or higher than the first threshold and a defect having a pixel level equal to or higher than the second threshold adjacent to the defective pixel having a pixel level equal to or higher than the first threshold. The defect correction apparatus according to any one of (1) to (8), wherein an area including pixels is detected as the defect area.

(10) The second correction range setting unit defines a range corresponding to the defect area of the first image detected by the defect area detection unit in the second image as the first image. The defect correction apparatus according to any one of (1) to (9), wherein a range in which defect correction is performed by shifting the parallax of the second image is set.

(11) detecting defective pixels in the first image;
Detecting defective pixels in the second image;
Detecting a defective area constituted by the defective pixels detected in the first image, and detecting a defective area constituted by the defective pixels detected in the second image;
Setting a range for performing defect correction in the first image based on the defective pixel detected in the first image and the detected defect area;
Setting a range for performing defect correction in the second image based on the defective pixel detected in the second image and the detected defect area;
Perform defect correction processing on the first image based on the set correction range,
A defect correction method for performing defect correction processing on the second image based on a set correction range.

(12) a first imaging unit that receives light through the optical system and generates a first image;
A second imaging unit that receives light via the optical system and generates a second image;
A first defect detection unit for detecting defective pixels in the first image;
A second defect detector for detecting defective pixels in the second image;
In the first image, a defect area constituted by the defective pixels detected by the first defect detection unit is detected, and in the second image, detected by the second defect detection unit. A defect area detector for detecting a defect area constituted by the defective pixels;
A first correction range setting unit that sets a range for performing defect correction in the first image based on a detection result by the first defect detection unit and a detection result by the defect region detection unit;
A second correction range setting unit that sets a range for performing defect correction in the second image based on a detection result by the second defect detection unit and a detection result by the defect region detection unit;
A first defect correction unit that performs a defect correction process on the first image based on the correction range set by the first correction range setting unit;
An imaging apparatus comprising: a second defect correction unit that performs defect correction processing of the second image based on the correction range set by the second correction range setting unit.

100... Defect correction device 111... L-side defect detection unit 112... L-side defect correction unit 113. ... R-side defect correction unit 123... R-side correction range setting unit 130... Defect region detection unit 200. R side image sensor

Claims (12)

A first defect detector for detecting defective pixels in the first image;
A second defect detector for detecting defective pixels in the second image;
In the first image, a defect area constituted by the defective pixels detected by the first defect detection unit is detected, and in the second image, detected by the second defect detection unit. A defect area detector for detecting a defect area constituted by the defective pixels;
A first correction range setting unit that sets a range for performing defect correction in the first image based on a detection result by the first defect detection unit and a detection result by the defect region detection unit;
A second correction range setting unit that sets a range for performing defect correction in the second image based on a detection result by the second defect detection unit and a detection result by the defect region detection unit;
A first defect correction unit that performs a defect correction process on the first image based on the correction range set by the first correction range setting unit;
A defect correction apparatus comprising: a second defect correction unit that performs defect correction processing on the second image based on the correction range set by the second correction range setting unit. The defect correction apparatus according to claim 1, wherein the defect area detection unit detects an area including a plurality of adjacent defective pixels as the defect area. The defect correction apparatus according to claim 1, wherein the first correction range setting unit further sets a range in which defect correction is performed based on the defect region in the second image detected by the defect region detection unit. . The first correction range setting unit includes a defective pixel in the first image, the defective region in the first image detected by the defective region detection unit, and the defect region detected by the defective region detection unit. The defect correction apparatus according to claim 3, wherein a range corresponding to the defect area of the second image is set as a range in which defect correction is performed in the first image. The first defective pixel detection unit compares a first threshold value with a pixel level of the first image, and further, a second threshold value set to a value smaller than the first threshold value and the first threshold value. A pixel level of the first image or the second threshold value is detected as a defective pixel.
The defective area detection unit includes a defective pixel having a pixel level equal to or higher than the first threshold and a defect having a pixel level equal to or higher than the second threshold adjacent to the defective pixel having a pixel level equal to or higher than the first threshold. The defect correction apparatus according to claim 2, wherein an area composed of pixels is detected as the defect area. The first correction range setting unit sets a range corresponding to the defect area of the second image detected by the defect area detection unit in the first image, as the first image and the second image. The defect correction apparatus according to claim 1, wherein a range in which defect correction is performed by shifting the amount of parallax of the image is set. The defect correction apparatus according to claim 1, wherein the second correction range setting unit further sets a range in which defect correction is performed based on the defect region in the first image detected by the defect region detection unit. . The second correction range setting unit includes a defective pixel in the second image, the defective region in the second image detected by the defective region detection unit, and the defect region detected by the defective region detection unit. The defect correction apparatus according to claim 7, wherein a range corresponding to the defect area of the first image is set as a range in which defect correction is performed in the second image. The second defective pixel detection unit compares the first threshold value with the pixel level of the second image, and further compares the second threshold value set to a value smaller than the first threshold value and the second threshold value. A pixel level of the first image or the second threshold value is detected as a defective pixel.
The defective area detection unit includes a defective pixel having a pixel level equal to or higher than the first threshold and a defect having a pixel level equal to or higher than the second threshold adjacent to the defective pixel having a pixel level equal to or higher than the first threshold. The defect correction apparatus according to claim 2, wherein an area composed of pixels is detected as the defect area. The second correction range setting unit sets a range corresponding to the defect area of the first image detected by the defect area detection unit in the second image, as the first image and the second image. The defect correction apparatus according to claim 1, wherein a range in which defect correction is performed by shifting the amount of parallax of the image is set. Detecting defective pixels in the first image;
Detecting defective pixels in the second image;
Detecting a defective area constituted by the defective pixels detected in the first image, and detecting a defective area constituted by the defective pixels detected in the second image;
Setting a range for performing defect correction in the first image based on the defective pixel detected in the first image and the detected defect area;
Setting a range for performing defect correction in the second image based on the defective pixel detected in the second image and the detected defect area;
Perform defect correction processing on the first image based on the set correction range,
A defect correction method for performing defect correction processing on the second image based on a set correction range. A first imaging unit that receives light via the optical system and generates a first image;
A second imaging unit that receives light via the optical system and generates a second image;
A first defect detection unit for detecting defective pixels in the first image;
A second defect detector for detecting defective pixels in the second image;
In the first image, a defect area constituted by the defective pixels detected by the first defect detection unit is detected, and in the second image, detected by the second defect detection unit. A defect area detector for detecting a defect area constituted by the defective pixels;
A first correction range setting unit that sets a range for performing defect correction in the first image based on a detection result by the first defect detection unit and a detection result by the defect region detection unit;
A second correction range setting unit that sets a range for performing defect correction in the second image based on a detection result by the second defect detection unit and a detection result by the defect region detection unit;
A first defect correction unit that performs a defect correction process on the first image based on the correction range set by the first correction range setting unit;
An imaging apparatus comprising: a second defect correction unit that performs defect correction processing of the second image based on the correction range set by the second correction range setting unit.

Images