JP2020036199A - Imaging apparatus - Google Patents

Abstract

To provide an imaging apparatus that can prevent an SN ratio difference (SN ratio difference resulting from a difference in the number of defective pixels) occurring at a boundary portion at which photographed images are connected, while reducing the time required for defective pixel correction processing.SOLUTION: An imaging apparatus connects photographed images photographed by a plurality of image pick-up devices to create a single composite image, and comprises: storage means that stores information on a defective pixel for at least one image pick-up device of the plurality of image pick-up devices; and correction means that performs defective pixel correction processing on image data output from the image pick-up device on the basis of the information on the defective pixel stored in the storage means to reduce a defective image area appearing in the photographed image due to the defective pixel. The correction means performs the correction processing on partial defective pixels in all defective pixels in the plurality of image pick-up devices so that a difference in defective image area per unit area is reduced between the plurality of photographed images.SELECTED DRAWING: None

Description

  The present invention relates to an imaging device.

  2. Description of the Related Art There is known an imaging apparatus that generates a single composite image by joining captured images captured by a plurality of image sensors arranged so that a part of a captured visual field overlaps with each other. For example, Patent Literature 1 describes a specific configuration of this type of imaging apparatus.

  The imaging device described in Patent Literature 1 has a configuration in which an exposure difference between image sensors is corrected so that a brightness difference does not occur at a boundary portion where images captured by image sensors having different sensitivity differences are joined. Has become.

JP-A-2005-50498

  Patent Literature 1 describes correcting a defective pixel with respect to image data output from each image sensor. However, with the configuration described in Patent Document 1, it is pointed out that it takes time to generate a composite image because the correction processing takes a long time when the number of defective pixels is large.

  If the correction process of the defective pixel is omitted to shorten the time required to generate the composite image, the SN ratio of the entire composite image is reduced. Further, the number of defective pixels differs for each image sensor. Therefore, if the correction process of the defective pixel is omitted, the difference in the number of defective pixels possessed by each image sensor at the boundary portion where the captured images are joined (for example, the area of the defective area such as a white flaw or a black flaw occupying the captured image) (Difference), a composite image in which the SN ratio difference is conspicuous is generated.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to reduce an SN ratio difference (defective pixel) generated at a boundary portion where captured images are joined together while suppressing the time required for correction processing of a defective pixel. It is an object of the present invention to provide an imaging apparatus capable of suppressing an S / N ratio difference caused by a difference between the numbers.

  An imaging apparatus according to an embodiment of the present invention is an apparatus that generates a single composite image by joining captured images captured by a plurality of imaging elements, and at least one of the plurality of imaging elements A storage unit for storing information on the defective pixel, and a correction process for the defective pixel on the image data output from the image sensor based on the information on the defective pixel stored in the storage unit. Correction means for reducing a defective image area appearing in the photographed image. The correction unit performs a correction process on some of the defective pixels of the plurality of image sensors so that the difference in the area of the defective image area per unit area between the plurality of captured images is reduced. Do.

  According to an embodiment of the present invention, an S / N ratio difference (an S / N ratio difference caused by a difference in the number of defective pixels) occurring at a boundary portion where captured images are joined together is suppressed while suppressing a time required for correction processing of a defective pixel. An imaging device capable of performing the above is provided.

FIG. 1 is a block diagram illustrating a configuration of an imaging device according to an embodiment of the present invention. FIG. 2A shows a composite image in the case where the defective pixel correction processing is not performed in the defective pixel correction processing example 1, and FIG. 2B shows the composite image in which the defective pixel correction processing is performed in the defective pixel correction processing example 1. 5 shows a composite image in the case. FIG. 3A shows a composite image in a case where the defective pixel correction processing is not performed in the defective pixel correction processing example 2, and FIG. 3B shows a composite image in which the defective pixel correction processing is performed in the defective pixel correction processing example 2. 5 shows a composite image in the case. FIG. 4A shows a composite image in a case where the defective pixel correction processing is not performed in the defective pixel correction processing example 3, and FIG. 4B shows a composite image in which the defective pixel correction processing is performed in the defective pixel correction processing example 3. 5 shows a composite image in the case.

  Hereinafter, an imaging device according to an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration of an imaging device 1 according to an embodiment of the present invention. The imaging device 1 is a device capable of capturing a spherical image, for example, as described in Patent Document 1. In addition, the imaging device 1 is a digital single-lens reflex camera, a mirrorless single-lens camera, a video camera, a camcorder, a tablet terminal, a PHS (Personal Handy phone System), a smartphone, a feature phone, a portable game machine, or another form having a shooting function. May be used.

  As illustrated in FIG. 1, the imaging apparatus 1 includes a plurality of (two in this embodiment) imaging lenses 10A and 10B and a plurality of (two in this embodiment) imaging corresponding to each of the imaging lenses 10A and 10B. Elements 12A and 12B are provided. In the present embodiment, as described below, a configuration is used in which a single composite image (omnidirectional image) is generated by joining two captured images captured by two pairs of capturing lenses and an image sensor. However, in another embodiment, a single composite image may be generated by joining three or more photographed images photographed by three or more pairs of photographing lenses and image sensors.

  As shown in FIG. 1, the imaging apparatus 1 includes an image processing unit 14, a CPU (Central Processing Unit) 16, a ROM (Read Only Memory) 18, a RAM, in addition to the imaging lenses 10A and 10B and the imaging elements 12A and 12B. (Random Access Memory) 20, RAM interface 22, storage 24, storage interface 26, operation switch group 28, LCD (Liquid Crystal Display) 30, LCD driver 32, 3-axis acceleration sensor 34, and power supply circuit 36.

  The operation switch group 28 includes various operation units, such as a power switch, a release switch, an operation dial, and operation keys, necessary for the user to operate the imaging apparatus 1. When the power switch is pressed by the user, the power circuit 36 supplies power to each unit of the imaging device 1 according to a command from the CPU 16. The CPU 16 accesses the ROM 18 to load the control program into the work area, and executes the loaded control program to control the entire imaging apparatus 1.

  The photographing lens 10A and the photographing lens 10B are fisheye lenses (or wide-angle lenses) having the same optical performance, and have an angle of view exceeding 180 degrees. The photographing lens 10A and the photographing lens 10B are installed so that their optical axes are located coaxially and face each other in opposite directions.

  The imaging element 12A is installed at a position where the center of the imaging surface (the center of the effective pixel area) is orthogonal to the optical axis of the imaging lens 10A and the imaging surface is the imaging plane of the imaging lens 10A. The imaging element 12B is installed at a position where the center of the imaging surface (the center of the effective pixel area) is orthogonal to the optical axis of the imaging lens 10B and the imaging surface is the imaging plane of the imaging lens 10B.

  The image sensor 12A and the image sensor 12B are image sensors having the same specifications (the same number of effective pixels and the same pixel size). Illustratively, the imaging elements 12A and 12B are CMOS (Complementary Metal Oxide Semiconductor) image sensors equipped with a Bayer array filter, and are connected to respective pixels on the imaging surface via corresponding photographing lenses 10A and 12B, respectively. The imaged optical image is accumulated as electric charge corresponding to the light amount. The imaging elements 12A and 12B convert the accumulated charge into a voltage (hereinafter, referred to as an “image signal”) using a floating diffusion amplifier. The image signals output from the imaging elements 12A and 12B are input to the image processing unit 14. The imaging devices 12A and 12B may be CCD (Charge Coupled Device) image sensors, or may be image sensors equipped with complementary color filters.

  The image processing unit 14 performs predetermined signal processing on image data input from the imaging devices 12A and 12B. For convenience of description, data of one captured image including an image signal of each pixel is referred to as “image data”. For convenience of explanation, image data input from the image sensor 12A and processed in the image processing unit 14 is referred to as “image data DA”, and image data input from the image sensor 12B and processed in the image processing unit 14 is described. The data is referred to as “image data DB”.

  The image processing unit 14 performs OB (Optical Black) correction processing, defective pixel correction processing, linear (Linear) correction processing, shading (Shading) correction processing, and area division averaging processing on the image data DA and DB, The image data DA and DB after the division averaging process are stored in the RAM 20 via the RAM interface 22.

  The ROM 18 stores, for example, information (address) of defective pixels of the imaging elements 12A and 12B detected in an inspection process at the time of manufacturing, for example. In the defective pixel correction processing, the defective pixel at the address stored in the ROM 18 is corrected for a pixel value based on a composite signal from a plurality of adjacent pixels. Note that the defective pixel correction processing may be performed after the synthesis processing (described later).

  Next, the image processing unit 14 performs white balance processing, gamma (γ) correction processing, Bayer interpolation processing, YUV conversion processing, edge enhancement processing, and color correction processing on the image data DA and DB stored in the RAM 20. The image data DA and DB after the color correction processing are stored in the RAM 20 via the RAM interface 22.

  The image processing unit 14 performs a cropping process on the color-corrected image data DA and DB stored in the RAM 20, and then performs a distortion correction process and a combining process. In the distortion correction processing, the two image data DA and DB are subjected to distortion correction and top-bottom correction (tilt correction) based on information output from the three-axis acceleration sensor 34. In the combining process, the two image data DA and DB after the distortion correction process are combined (in other words, the two captured images captured by the imaging elements 12A and 12B are joined) to form a celestial sphere image (stereoscopic image). Image data of an angle of 4π radians is generated.

  The image processing unit 14 compresses the spherical image data in a predetermined format such as JPEG (Joint Photographic Experts Group). The compressed spherical image data is stored in the storage 24 via the storage interface 26. The storage 24 is, for example, an SD card.

  The spherical image data may be transferred to an external device such as a smartphone via wireless communication such as Wi-Fi or Bluetooth (registered trademark).

  The LCD driver 32 is a drive circuit that drives the LCD 30. Various information about the imaging device 1 is displayed on the LCD 30. Exemplarily, an icon is displayed to notify that the spherical image data has been stored in the storage 24.

  The image processing unit 14 combines not only a spherical image but also two captured images captured by the image sensors 12A and 12B and combines them to generate a so-called panoramic image that extends 360 degrees only in a horizontal plane. Can also.

  In the present embodiment, the area of a defective image area (a defective area such as a white flaw or a black flaw appearing in a captured image due to defective pixels) per unit area between two captured images captured by the imaging elements 12A and 12B. Is corrected so that only some of the defective pixels of the imaging elements 12A and 12B are corrected so as to reduce the difference.

  In the present embodiment, by correcting only a part of the defective pixels of all the defective pixels of the imaging elements 12A and 12B, the time required for the defective pixel correction process is reduced as compared with the case of correcting all the defective pixels. The two captured images are joined by performing a defective pixel correction process so that the difference in the area of the defective image area per unit area between the two captured images captured by the imaging elements 12A and 12B is reduced. It is possible to suppress the S / N ratio difference (S / N ratio difference caused by the difference in the number of defective pixels) occurring at the boundary portion. By suppressing the SN ratio difference occurring at the boundary portion, the image quality of the entire synthesized image is improved as compared with the case where the SN ratio difference is large.

  The ROM 18 stores only the information of the defective pixel to be subjected to the defective pixel correction processing among all the defective pixels of the imaging elements 12A and 12B. That is, the ROM 18 operates as storage means for storing information on defective pixels for at least one of the plurality of image sensors. The memory capacity of the ROM 18 required for executing the defective pixel correction processing is reduced as compared with the case where the information on all the defective pixels of the imaging elements 12A and 12B is stored in the ROM 18.

  In the present embodiment, the image sensor 12A and the image sensor 12B have the same specifications (the number of effective pixels, the pixel size, and the like are the same). Therefore, the above-mentioned "so that the difference in the area of the defect image area per unit area is small" is, for example, "so that the difference in the number of defect image areas is small", "the defect image area with respect to the total number of effective pixels". To reduce the difference in the ratio of the number of defective image areas "and" to reduce the difference in the number of defective image areas per unit area ".

  Hereinafter, three examples of the defective pixel correction processing executed in the present embodiment will be described. In each defective pixel correction processing example, a panoramic image is shown as a composite image.

<< Defect pixel correction processing example 1 >>
FIG. 2A shows a composite image in the case where the defective pixel correction processing is not performed in the first example of the defective pixel correction processing. FIG. 2B shows a composite image when the defective pixel correction processing is performed in the first example of the defective pixel correction processing. In FIGS. 2A and 2B, for convenience of description, an image captured by the image sensor 12A (hereinafter, referred to as a “photographed image IA”) and an image captured by the image sensor 12B (hereinafter, referred to as “the captured image IA”) are located at the center of the composite image. (Depicted as a photographed image IB)). The boundary line BL is a virtual line that does not appear in an actual composite image. The image on the left side of the boundary line BL is the captured image IA, and the image on the right side of the boundary line BL is the captured image IB. In each of the drawings (FIGS. 3 and 4) after the defective pixel correction processing example 2, the boundary line BL is described at the center of the composite image, the photographed image IA is described on the left side of the boundary line BL, and the photographed image is photographed on the right side of the boundary line BL. Image IB is described.

  In the defective pixel correction processing example 1, the image sensor 12A has n defective pixels, and the image sensor 12B has 3n defective pixels. In the first example of the defective pixel correction processing, the number of the defective image areas of the captured images IA and IB (in other words, the number of defective pixels that are not subjected to the defective pixel correction processing) is n, so that the number of the image pickup elements 12A and 12B is n. The defective pixel correction process is not performed on the image data DA of the imaging element 12A having the smallest number of defective pixels, and the defective pixel correction process is performed on 2n defective pixels on the image data DB.

  In the defective pixel correction processing example 1, only the image signals of some defective pixels of the image data DB of the image data DA and DB (specifically, 2n images exceeding the defective pixel number n of the captured image IA) Since only the defective pixel correction processing is performed on the signal, the time required for the defective pixel correction processing can be reduced as compared with the case where the image signals of all the defective pixels in the image data DA and DB are corrected. Further, the number of defective image areas appearing in each of the captured images IA and IB becomes the same (in other words, the difference in the area of the defective image area per unit area between the captured images IA and IB disappears). The resulting SN ratio difference is substantially eliminated. The image quality of the entire synthesized image is improved as compared with the case where the SN ratio difference is large.

  That is, the image processing unit 14 sets one of all the defective pixels of the plurality of imaging devices 12A and 12B so that the difference in the area of the defective image area per unit area between the plurality of captured images IA and IB becomes small. It operates as a correction unit that performs a correction process on a defective pixel of a part.

  The number of defective image areas appearing in each of the captured images IA and IB may not be the same. If the difference in the number of defective image areas of the captured images IA and IB is smaller than that in the case where the defective pixel correction processing is not performed, the SN ratio difference generated at the boundary line BL is reduced, and the image quality of the entire composite image is improved. The effect is obtained.

<< Defect pixel correction processing example 2 >>
FIG. 3A shows a composite image in the case where the defective pixel correction processing is not performed in the defective pixel correction processing example 2. FIG. 3B shows a composite image when the defective pixel correction processing is performed in the second example of the defective pixel correction processing.

  In the defective pixel correction process example 2, the image sensor 12A has n defective pixels, and the image sensor 12B has 3n defective pixels. Also in the defective pixel correction processing example 2, the defective pixel correction processing is performed only on the image signals of some defective pixels of the image data DB among the image data DA and DB.

  In the defective pixel correction processing example 2, in order to further reduce the time required for the defective pixel correction processing as compared with the defective pixel correction processing example 1, the image data DA of the imaging element 12A having the smallest number of defective pixels among the imaging elements 12A and 12B is used. On the other hand, the defective pixel correction processing is not performed, and the defective pixel correction processing is performed on the image data DB only for the image signals of n defective pixels. Therefore, in the defective pixel correction processing example 2, n defective image areas remain in the captured image IA and 2n defective image areas remain in the captured image IB.

  In the second example of the defective pixel correction process, it is possible to reduce the SN ratio difference generated at the boundary line BL as compared with the case where the defective pixel correction process is not performed. However, since the difference in the number of defective image areas is large, the defective pixel correction process is not performed. It is difficult to say that the S / N ratio difference occurring at the boundary line BL is sufficiently suppressed by simply performing the processing (in other words, simply performing the defective pixel correction processing uniformly over the entire area in the captured image IB).

  Therefore, in the defective pixel correction processing example 2, the difference in the area of the defective image area per unit area from the captured image IA (the difference in the number of the defective image areas) in the image area closer to the boundary line BL in the captured image IB. A defective pixel correction process is performed so as to reduce the size.

  Specifically, in the defective pixel correction processing example 2, the captured image IB is divided into three image areas A1, A2, and A3. In the image area A1, a defective pixel correction process is performed on the image signals of half (50%) of all the defective pixels in the image area A1. In the image area A2, a defective pixel correction process is performed on image signals of 30% of the defective pixels among all the defective pixels in the image area A2. In the image area A3, a defective pixel correction process is performed on image signals of 20% of the defective pixels in all the defective pixels in the image area A3. Note that the number of divided image areas and the correction rate of defective pixels in each image area are not limited to the above, and may be changed as appropriate.

  In the defective pixel correction processing example 2, in the image area A1 closest to the boundary line BL, the difference in the area of the defective image area per unit area from the captured image IA (the difference in the number of the defective image areas) is mainly reduced. be able to. By focusing the defective pixel correction processing on the image area near the boundary line BL where the deterioration of the image quality due to the SN ratio difference is conspicuous, the number of defective pixels to be subjected to the defective pixel correction processing is reduced (in other words, To improve the image quality of the entire synthesized image as compared with a case where the defective pixel correction processing is simply performed on the image data DB with respect to the image signals of n defective pixels while suppressing the time required for the defective pixel correction processing. Can be.

  As described above, in the defective pixel correction processing example 2, the time required for the defective pixel correction processing is longer than that in the defective pixel correction processing example 1 when suppressing the S / N ratio difference generated at the boundary line BL and improving the image quality of the entire composite image. It can be further suppressed.

<< Defect pixel correction processing example 3 >>
FIG. 4A shows a composite image in the case where the defective pixel correction process is not performed in the third example of the defective pixel correction process. FIG. 4B shows a composite image when the defective pixel correction processing is performed in the third example of the defective pixel correction processing.

  In the third example of the defective pixel correction process, the image sensor 12A has n defective pixels, and the image sensor 12B has 3n defective pixels. In the third example of the defective pixel correction processing, the memory capacity for storing the information of the defective pixel in the memory capacity of the ROM 18 is equal to the difference in the number of defective pixels between the image sensor 12A and the image sensor 12B (that is, 2n defective pixels). And smaller than the memory capacity required to store the information of the total number of defective pixels (that is, 4n defective pixels) of the image sensor 12A and the image sensor 12B. . Specifically, the memory capacity is such that information on a maximum of 3n defective pixels can be stored.

  In the defective pixel correction processing example 3, the time required for the defective pixel correction processing is reduced as compared with the case where the image signals of all the defective pixels of the image data DA and DB are corrected, and the memory capacity of the ROM 18 is utilized to the maximum. The image quality of the entire image can be improved as compared with the defective pixel correction processing example 1 and the defective pixel correction processing example 2.

  More specifically, in the third example of the defective pixel correction processing, the image data DA includes 0.5n defective pixel images so that the number of the defective image areas of both the captured images IA and IB is 0.5n. A defective pixel correction process is performed on the signal, and a defective pixel correction process is performed on the image data DB with respect to the image signals of 2.5n defective pixels.

  In the third example of the defective pixel correction process, the number of defective image regions appearing in each of the captured images IA and IB is the same and small (0.5n), so that the SN ratio difference generated at the boundary line BL is substantially eliminated. The image quality of the synthesized image is improved by the small number of defective image areas.

  The above is the description of the exemplary embodiment of the present invention. The embodiments of the present invention are not limited to those described above, and various modifications are possible within the scope of the technical idea of the present invention. For example, the embodiments of the present application include, for example, embodiments explicitly illustrated in the specification or contents obtained by appropriately combining obvious embodiments and the like.

  Some defective pixels occur after shipping of the imaging device 1. A technique for detecting such a defective pixel at a predetermined timing (for example, when the power is turned on or during an imaging operation) is known as a well-known technique. In another embodiment, the image processing unit 14 captures image data (for example, RAW data) in a state where the imaging devices 12A and 12B are shielded from light, and determines whether the signal level is higher than a predetermined threshold for each image signal. Is determined. When an image signal of a pixel whose signal level is higher than a predetermined threshold value and is not stored as a defective pixel in the ROM 18 is detected, this is determined as a newly detected image signal of the defective pixel, and the address thereof is additionally stored in the RAM 20. .

  Each time the number of defective pixels increases, the ratio between the number of defective pixels of the image sensor 12A and the number of defective pixels of the image sensor 12B changes. In another embodiment, even if the number of defective pixels increases, the number of defective image areas appearing in each of the captured images IA and IB is maintained at the same number. The number of defective pixels to be processed may be changed to a number corresponding to the increased number of defective pixels.

  In the above embodiment, the defective pixel correction processing is performed so that the number of defective image areas appearing in each of the captured images IA and IB is the same without distinguishing between white flaws and black flaws. In the embodiment, the defective pixel correction processing is performed so that the number of white defect image areas appearing in each of the captured images IA and IB is the same and the number of black defect image areas appearing in each of the captured images IA and IB is the same. May be performed.

  2. Description of the Related Art There is an imaging unit in which an imaging surface is enlarged by tiling a plurality of rectangular semiconductor substrates (a plurality of imaging elements) cut out in a rectangular shape from a silicon semiconductor wafer on a flat base in a matrix. An image pickup apparatus including an image pickup unit of this type receives an optical image via a single image pickup lens by each image pickup element (in other words, an image pickup area) tiled on a flat base, and receives each image pickup element (image pickup area). ) Is combined to generate a single composite image. One of all the defective pixels of the plurality of imaging elements (each imaging area) is set so that the difference in the area of the defective image area per unit area between the plurality of tiled imaging elements (each imaging area) becomes small. A configuration in which defective pixel correction processing is performed on defective pixels in a part is also within the scope of the present invention.

  The user may be able to set whether or not to execute the defective pixel correction process in order to suppress the S / N ratio difference generated at the boundary portion where the captured images are joined. Exemplarily, the user is required to set a case where the difference in the area of the defect image area per unit area between the plurality of image sensors is equal to or smaller than a predetermined threshold (for example, a value arbitrarily set by the user) ), It can be set not to execute the defective pixel correction processing. By omitting the execution of the defective pixel correction process when generating the composite image, it is possible to reduce the time required to generate the composite image. Conversely, the user can set to execute the defective pixel correction process even if the difference in the area of the defective image area per unit area between the plurality of image sensors is small. In this case, it is possible to generate a composite image in which the above-described SN ratio difference is substantially suppressed. The user may arbitrarily set how much the SN ratio difference is to be suppressed (in other words, the number of defective pixels to be subjected to the defective pixel correction processing).

1 imaging device 10A, 10B imaging lens 12A, 12B imaging device 14 image processing unit 16 CPU
18 ROM
20 RAM
22 RAM interface 24 Storage 26 Storage interface 28 Operation switch group 30 LCD
32 LCD driver 34 3-axis acceleration sensor 36 Power supply circuit

Claims (7)

In an imaging apparatus that generates a single composite image by joining captured images captured by a plurality of imaging elements,
Storage means for storing information on defective pixels for at least one of the plurality of image sensors;
By performing a correction process of the defective pixel on image data output from the image sensor based on information of the defective pixel stored in the storage unit, a defective image appearing in the captured image by the defective pixel Correction means for reducing the area;
With
The correction means,
The correction processing is performed on some of the defective pixels of the plurality of imaging elements so that the difference in the area of the defective image area per unit area between the plurality of captured images is reduced. Do,
Imaging device. The correction means,
Performing the correction process on image data output from some of the plurality of image sensors;
The imaging device according to claim 1. The correction means,
Of the image data output from the plurality of imaging elements, at least, the correction process is not performed on image data of a captured image in which the area of the defective image area per unit area is the smallest,
The imaging device according to claim 2. The correction means,
Performing the correction processing so that the difference in the area of the defect image area per unit area between the plurality of captured images is eliminated;
The imaging device according to claim 1. The correction means,
In the captured image, the correction process is performed such that the difference in the area of the defective image region per unit area between the adjacent captured image and the image region closer to the boundary between the captured images is smaller,
The imaging device according to claim 1. The storage means,
Of all the defective pixels of the plurality of imaging elements, only information of the defective pixel to be subjected to the correction processing is stored.
The imaging device according to claim 1. Further comprising a detecting means for detecting the new defective pixel,
The storage means,
Additionally storing information of the newly detected defective pixel in the detection means,
The correction means,
According to the number of defective pixels additionally stored in the storage unit, change the number of defective pixels to be subjected to the correction process,
The imaging device according to claim 1.