JP2017010114A - Image correction device, image correction method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable an image with a wider dynamic range to be generated by a simpler device.SOLUTION: A light field image having a plurality of element images in which the same subject is imaged is acquired. For each element image, a correction object pixel is detected which satisfies a condition indicating a pixel in a region where the luminance value is deviated in the element image. One or a plurality of alternative pixels which are pixels included in an element pixel different from the correction object pixel and pixels in which the same subject with that in the correction object pixel is imaged are acquired for each correction object pixel. The luminance value of the correction object pixel is then corrected based on the luminance value of the alternative pixel.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for correcting an image.

The image sensor of the digital camera records an image formed by the main lens. The range of the amount of light that can be received by the image sensor at each pixel is determined in advance. Therefore, when the amount of light exceeding the range that can be acquired by the image sensor enters the image sensor, the luminance value of a part of the generated image overflows. Such a phenomenon is called “white jump”. In an 8-bit image with an unsigned color depth, the luminance value of a pixel in which overexposure occurs is the maximum value of 255. Originally, the information of the image that should be represented by a value of 255 or more is all 255, so that it cannot be expressed. On the other hand, when only an extremely small amount of light enters the image sensor, the luminance value of a part of the generated image becomes zero. Such a phenomenon is called “black crushing”.
As described above, there is a predetermined range in the amount of light that can be acquired by the image sensor at one time. This is a hindrance when acquiring an image with a wide dynamic range. As a method for acquiring an image with a wide dynamic range, there are a technique described in Non-Patent Document 1 and a technique described in Non-Patent Document 2.

Fattal, Raanan, Dani Lischinski, and Michael Werman. “Gradient domain high dynamic range compression.” ACM Transactions on Graphics (TOG). Vol. 21. No. 3. ACM, 2002. Saori Uda, Fumihiko Sakagami, Satoshi Sato, "HDR image generation using variable exposure camera" 17th Symposium on Image Recognition and Understanding (MIRU2014), SS1-31

However, in the technique described in Non-Patent Document 1, it is necessary to capture a plurality of images while changing the lens aperture for one scene. Therefore, there is a problem that it cannot be used for a dynamic scene.
In the technique described in Non-Patent Document 2, it is possible to control exposure on a pixel basis by a variable exposure camera using LCoS. Therefore, an image having a wider dynamic range (HDR: High Dynamic Range) can be generated by one imaging. However, since it is necessary to use a special device that is not a camera, it is not convenient.
In view of the above circumstances, an object of the present invention is to provide a technique that enables an image having a wider dynamic range to be generated by a simpler apparatus.

  One aspect of the present invention is an image acquisition unit that acquires a light field image having a plurality of element images in which the same subject is imaged, and a region where the luminance value is biased in each element image A correction target pixel detection unit that detects a correction target pixel that satisfies a condition indicating that the correction target pixel is included, and a pixel that is included in an element image different from the correction target pixel and that is reflected in the correction target pixel; An alternative pixel acquisition unit that acquires one or more alternative pixels that are pixels in which the same subject is captured, and corrects the luminance value of the correction target pixel based on the luminance value of the alternative pixel. And a correction unit.

  One aspect of the present invention is the above-described image correction apparatus, wherein the substitute pixel acquisition unit acquires only substitute pixels that do not satisfy the condition among the substitute pixels.

  One aspect of the present invention is an image acquisition step of acquiring a light field image having a plurality of element images in which the same subject is imaged, and a region where the luminance value is biased in each element image A correction target pixel detecting step for detecting a correction target pixel that satisfies a condition indicating that the correction target pixel is included, and a pixel that is included in an element image different from the correction target pixel, and that is reflected in the correction target pixel; An alternative pixel acquisition step of acquiring one or a plurality of alternative pixels, which are pixels in which the same subject is captured, for each correction target pixel, and correcting the luminance value of the correction target pixel based on the luminance value of the alternative pixel And a correction step.

  One embodiment of the present invention is a computer program for causing a computer to function as the above-described image correction apparatus.

  According to the present invention, an image having a wider dynamic range can be generated by a simpler apparatus.

2 is a schematic block diagram illustrating a functional configuration of the image correction apparatus 10. FIG. A plane subject model is shown. It is a figure which shows the specific example of the microlens image obtained without white-out and black-out. It is a figure which shows the specific example of correction | amendment of overexposure. It is a figure which shows the specific example of correction | amendment of black crushing. 4 is a flowchart showing a processing flow of the image correction apparatus 10.

  FIG. 1 is a schematic block diagram illustrating a functional configuration of the image correction apparatus 10. The image correction apparatus 10 includes a CPU (Central Processing Unit), a memory, an auxiliary storage device, and the like connected by a bus. The CPU of the image correction apparatus 10 executes an image correction program stored in advance in the memory of the image correction apparatus 10. When the CPU executes the image correction program, the image correction apparatus 10 functions as an apparatus including the image acquisition unit 101, the microlens image acquisition unit 102, the correction target pixel detection unit 103, the alternative pixel acquisition unit 104, and the correction unit 105. To do. Note that all or part of the functions of the image correction apparatus 10 may be realized using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). . The image correction program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. The image correction program may be transmitted via a telecommunication line.

  The image acquisition unit 101 inputs a light field image. The light field image is an image formed by arranging a plurality of images (hereinafter referred to as “element images”) obtained by capturing the same subject from slightly different viewpoints according to the position of the viewpoint. The light field image may be formed by, for example, arranging a multi-viewpoint image group photographed using a camera array arranged in parallel. The light field image may be captured by a plenoptic camera, for example. A plenoptic camera is a camera having a microlens array between a main lens and an imaging device. When imaged with a plenoptic camera, an image imaged according to each microlens constituting the microlens array is an element image. In this case, the element image may be called a microlens image. In the following description, it is assumed that the image acquisition unit 101 acquires a light field image captured by a plenoptic camera.

  The microlens image acquisition unit 102 acquires a plurality of microlens images from the light field image input by the image acquisition unit 101. The microlens image acquisition unit 102 sets an independent image coordinate system for each acquired microlens image.

  The correction target pixel detection unit 103 determines whether or not the luminance value satisfies a predetermined condition to be corrected in each pixel of each microlens image. The predetermined condition is a condition indicating a pixel in a region where the luminance value is biased in the microlens image due to the influence of the dynamic range. The predetermined condition is, for example, a condition indicating that a phenomenon called overexposure occurs, or a condition indicating that a phenomenon called blackout occurs.

  More specifically, the predetermined condition is, for example, that the luminance value of each pixel is a value equal to or higher than the first threshold value or a value equal to or lower than the second threshold value. In this case, the first threshold value is larger than the second threshold value. When the luminance value of the pixel can take a value of, for example, 0 to 255, the first threshold value may be “250” and the second threshold value may be “5”. The first threshold and the second threshold may change dynamically. For example, the luminance value of an area where the luminance value is recognized to be biased based on a statistical value (for example, deviation) of the luminance value for each microlens image may be set as the threshold value.

  As another specific example of the predetermined condition, there is a condition regarding a frequency component. For example, the predetermined condition may be a pixel in a region where the frequency component is lower than a predetermined threshold in the microlens image. This is because the region with a low frequency component may be a region where the luminance change is poor due to overexposure or blackout.

  The correction target pixel detection unit 103 detects a pixel that satisfies a predetermined condition as a correction target pixel. The correction target pixel detection unit 103 performs the process of detecting the correction target pixel in this way on all the microlens images.

  The replacement pixel acquisition unit 104 acquires one or a plurality of replacement pixels for each correction target pixel detected by the correction target pixel detection unit 103. The substitute pixel is a pixel included in a microlens image different from the correction target pixel, and is a pixel in which the same subject as the subject shown in the correction target pixel is shown. In the light field image, each microlens image has an image of the subject in an overlapping range with another microlens image arranged in the vicinity. Therefore, by performing stereo matching on a plurality of microlens images, it is possible to acquire information indicating pixels (hereinafter referred to as “alternative pixels”) in which the same subject is captured. Note that any method may be applied to the stereo matching process.

  Strictly speaking, light beams in different directions are collected on each substitute pixel. However, since each microlens image is very densely arranged, it can be assumed that the light rays for the alternative pixels of each microlens image are substantially the same. Therefore, if the lenses having the same transmittance are used, it can be assumed that the alternative pixels have the same luminance value.

  The correction unit 105 corrects the luminance value of the correction target pixel based on the luminance value of the alternative pixel. For example, the correction unit 105 may perform correction by replacing the luminance value of the correction target pixel with the luminance value of the alternative pixel. For example, the correction unit 105 may correct the luminance value of the correction target pixel based on Equation 1 below.

P = P1 * a + P2 * b (Formula 1)
a + b = 1 (Formula 2)
P: luminance value after correction P1: luminance value before correction of the correction target pixel P2: luminance value of the alternative pixel a: weight for the luminance value of the correction target pixel b: weight for the luminance value of the alternative pixel

  By such processing, the luminance value of the pixel that is overexposed or blacked out because the dynamic range is not appropriate is corrected to the luminance value acquired in a more appropriate dynamic range. The correction unit 105 performs HDR processing on the light field image by performing correction processing on all the correction target pixels detected by the correction target pixel detection unit 103.

  The correction process executed by the correction unit 105 will be further described. When imaging is performed with a plenoptic camera, the amount of light observed in the end region of each microlens is less than in the center. In the end region of each microlens, an image is observed with a darker brightness than the region near the center. For this reason, even if whiteout occurs near the center of the microlens image, if the alternative pixel is detected as a pixel in the end region of another microlens image, the luminance value of the pixel that is not overexposed is used. Can be corrected.

  2 to 5 are diagrams illustrating specific examples of the correction process. FIG. 2 shows a planar subject model. FIG. 3 is a diagram illustrating a specific example of a microlens image obtained without whiteout or blackout. FIG. 4 is a diagram showing a specific example of over-exposure correction. FIG. 5 is a diagram illustrating a specific example of blackout correction.

  In FIG. 3, two microlens images 20 (20-1 and 20-2) are shown. In the microlens image 20-1, “something” of the subject model is shown. In the microlens image 20-2, “Aiue” of the subject model is shown. In the pixel 211 of the microlens image 20-1, the upper left portion of the character “A” is shown. In the microlens image 20-2, a pixel in which the same subject as that in the pixel 211 of the microlens image 20-1 is shown is a pixel 212. Therefore, if the pixel 211 of the microlens image 20-1 is a correction target pixel, the pixel 212 of the microlens image 20-2 is one of the alternative pixels. Actually, there may be three or more microlens images 20 having pixels in which the same subject as the subject in the pixel 211 is shown. Therefore, the number of alternative pixels is not limited to one.

  In FIG. 4, two microlens images 20 (20-3 and 20-4) are shown. In the microlens image 20-3, the relative positional relationship between the subject model and the microlens is the same as the microlens image 20-1 in FIG. In the microlens image 20-4, the relative positional relationship between the subject model and the microlens is the same as the microlens image 20-2 in FIG. Therefore, in the microlens image 20-3, “That” out of the subject model should be reflected. Similarly, in the microlens image 20-4, “Aiue” of the subject model should be originally captured. However, in the microlens image 20-3 and the microlens image 20-4, the overexposure has occurred, and thus the characters to be captured are not captured. In the microlens image 20-3, “A” is not shown due to overexposure. In the microlens image 20-4, “up” is not shown due to overexposure. However, “A” that is not shown in the microlens image 20-3 is shown in the microlens image 20-4 due to a decrease in the amount of light due to vignetting. Therefore, the pixel (correction target pixel) in the region 221 where “A” should be represented in the microlens image 20-3 is replaced with the pixel (substitute) in the region 222 where “A” is represented in the microlens image 20-4. The character “A” can be restored in the microlens image 20-3. The same applies to “up” in the microlens image 20-4.

  In FIG. 5, two microlens images 20 (20-5 and 20-6) are shown. In the microlens image 20-5, the relative positional relationship between the subject model and the microlens is the same as the microlens image 20-1 in FIG. In the microlens image 20-6, the relative positional relationship between the subject model and the microlens is the same as the microlens image 20-2 in FIG. For this reason, in the microlens image 20-5, “That” of the subject model should be reflected. Similarly, in the microlens image 20-6, “Aiue” of the subject model should be originally captured. However, in the microlens image 20-5 and the microlens image 20-6, black crushing has occurred, and thus characters to be captured are not captured. In the microlens image 20-5, “U” is not captured due to black crushing. In the microlens image 20-6, “A” is not shown due to black crushing. However, “U” that is not shown in the microlens image 20-5 is shown in the center portion of the microlens where the light amount is relatively high in the microlens image 20-6. Therefore, the pixel (correction target pixel) in the region 231 where “U” should be represented in the microlens image 20-5 is replaced with the pixel (substitute) in the region 232 in which “U” is represented in the microlens image 20-6. The character “U” can be restored in the microlens image 20-5. The same applies to “A” in the microlens image 20-6.

  In the present embodiment, the high dynamic range image is generated by correction such as simple luminance value replacement, but the high dynamic range image may be generated by a method using a histogram such as tone mapping.

  FIG. 6 is a flowchart showing a process flow of the image correction apparatus 10. First, the image acquisition unit 101 acquires a light field image to be processed (step S101). Next, the microlens image acquisition unit 102 extracts a microlens image to be processed from the light field image to be processed acquired by the image acquisition unit 101 (step S102). Next, the correction target pixel detection unit 103 detects a correction target pixel from pixels included in the microlens image to be processed (step S103). Next, the alternative pixel acquisition unit 104 detects one or a plurality of alternative pixels for each correction target pixel detected by the correction target pixel detection unit 103 (step S104). The correcting unit 105 executes the HDR process by correcting the luminance value of the correction target pixel based on the luminance value of the alternative pixel (step S105). The correction unit 105 outputs the light field image that has been subjected to the correction process.

  In the image correction apparatus 10 configured as described above, an image having a wider dynamic range can be generated by a simpler apparatus. Such an effect will be described in detail.

  In the image correction apparatus 10, HDR processing can be executed using a single light field image. Therefore, it is possible to execute the HDR process on the light field image in which a dynamic scene is captured.

  Further, the image correction apparatus 10 can execute the HDR process on a normal light field image. Therefore, a special device such as LCoS is not required.

  Further, by generating a reconstructed image using a light field image that has been subjected to correction processing by the image correction device 10, it is possible to generate an image with a wider dynamic range with a simple device. That is, when a reconstructed image is generated using a light field image that has been subjected to correction processing, a special device is not necessary, and a normal device that can generate a reconstructed image from a light field image may be used. Note that the reconstructed image is an image representing a subject when imaged by one imaging lens (main lens).

(Modification)
The light field image acquired by the image acquisition unit 101 may be an image captured using a plenoptic camera having a plurality of microlenses having different transmittances. In the light field image thus captured, the brightness of the microlens images is different. Therefore, it is possible to acquire a substitute pixel in which the subject is photographed more clearly. Note that the correspondence evaluation index used for stereo matching in this case may be Normalized Cross Correlation (NCC) that is robust to changes in luminance, or other similar methods.

  The correction unit 105 may perform correction processing using the vignetting effect. In this case, in addition to the light field image, the image acquisition unit 101 acquires vignetting parameters specific to the plenoptic camera that captured the light field image. The vignetting parameter is acquired in advance, for example, by performing vignetting calibration. In vignetting calibration, a white paper or a wall with uniform brightness is photographed before the subject is photographed by the plenoptic camera. By such processing, vignetting parameters unique to each plenoptic camera are estimated in advance. The image acquisition unit 101 acquires vignetting parameters at a time before the correction process is performed.

  The correction unit 105 may multiply the luminance value of the alternative pixel by a weight (scale) obtained based on the vignetting parameter when performing the correction process. With this configuration, it is possible to generate a light field image having a more natural luminance value.

  The substitute pixel acquisition unit 104 is a pixel included in an element image different from the correction target pixel, and does not satisfy the condition of the correction target pixel among pixels including the same subject as the subject in the correction target pixel. Only pixels may be acquired. With such a configuration, it is possible to suppress acquisition of a pixel in which overexposure or blackout occurs as an alternative pixel. Therefore, it is possible to acquire a higher quality high dynamic range image by performing correction based only on the alternative pixels in which the subject is clearly shown.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Image correction apparatus, 101 ... Image acquisition part, 102 ... Micro lens image acquisition part, 103 ... Correction object pixel detection part, 104 ... Alternative pixel acquisition part, 105 ... Correction part, 20 ... Micro lens image

Claims (4)

An image acquisition unit for acquiring a light field image having a plurality of element images obtained by imaging the same subject;
A correction target pixel detection unit that detects a correction target pixel that satisfies a condition indicating that the element image is a pixel in a region where the luminance value is biased in the element image;
One or a plurality of alternative pixels, which are pixels included in an element image different from the correction target pixel and are the same subject as the subject shown in the correction target pixel, are acquired for each correction target pixel An alternative pixel acquisition unit,
A correction unit that corrects the luminance value of the correction target pixel based on the luminance value of the substitute pixel; and
An image correction apparatus comprising:   The image correction apparatus according to claim 1, wherein the substitute pixel acquisition unit acquires only substitute pixels that do not satisfy the condition among the substitute pixels. An image acquisition step of acquiring a light field image having a plurality of element images obtained by imaging the same subject;
For each element image, a correction target pixel detection step for detecting a correction target pixel that satisfies a condition indicating that it is a pixel in a region where the luminance value is biased in the element image;
One or a plurality of alternative pixels, which are pixels included in an element image different from the correction target pixel and are the same subject as the subject shown in the correction target pixel, are acquired for each correction target pixel An alternative pixel acquisition step,
A correction step of correcting the luminance value of the correction target pixel based on the luminance value of the substitute pixel;
An image correction method comprising:   The computer program for functioning a computer as an image correction apparatus as described in any one of Claim 1 or 2.