JP6014343B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method.

  Conventionally, various techniques have been disclosed as techniques for correcting a captured image in consideration of the brightness of the captured scene. For example, a technique is disclosed in which a dark portion is picked up by photographing underexposed and performing gamma processing on an image obtained by photographing (see, for example, Patent Document 1). According to this technique, dark part correction can be easily performed.

  In addition, for example, a technique is disclosed in which a plurality of images are shot with different exposures (so-called bracket shooting is performed), and a plurality of images obtained by shooting are combined (see, for example, Patent Document 2). According to this technique, it is possible to enjoy a high dark part correction effect. Also disclosed is a technique for estimating illumination light from one image and correcting a dark part based on the estimated illumination light (see, for example, the same publication).

  In addition, for example, a technique is disclosed in which parallax is detected from a stereo image to darken a near scene with a large parallax and brighten a distant scene with a small parallax (see, for example, Patent Document 3). According to such a technique, it is possible to correct a phenomenon in which the subject in the foreground becomes bright and a phenomenon in which the background becomes dark during flash photography.

JP 2008-160190 A JP 2008-104010 A JP 2010-26018 A

  However, in the technique disclosed in Patent Document 1, the gradation of the appropriate exposure region (for example, the background portion) is lost due to the influence other than the dark portion (for example, the foreground portion), and the image is overexposed. The disadvantage is that this phenomenon can also occur.

  In addition, the technique disclosed in Patent Document 2 has a drawback in that when a fast-moving object is photographed, the position of the object becomes large, and the combined image becomes an unnatural image. . Furthermore, it is also complicated that it is necessary to shoot a plurality of images after setting the mode at the time of shooting. In addition, since it is impossible to distinguish between a dark part and a black object and between a bright part and a white object, a composite image may be unnatural due to erroneous estimation of illumination light.

  In the technique disclosed in Patent Document 3, processing is applied only to an image captured with flash. Therefore, even when an image similar to a flash-captured image is shot, such as when shooting a subject that has received a spotlight or the like without using a flash, the same processing cannot be applied by determining the image content. Further, this process cannot be applied to backlight correction.

  Therefore, the present invention intends to provide a technique capable of correcting a dark part without losing the gradation of a proper exposure region while avoiding synthesis of an unnatural image.

  According to an embodiment of the present invention, a parallax detection unit that detects a parallax based on a left image and a parallax based on a right image between a left image and a right image captured by a stereo camera, and the left A luminance correction unit that generates a left correction image and a right correction image by correcting the luminance of each of the image and the right image, and calculates a mixing ratio for mixing the left image based on the parallax of the left image reference, A mixing ratio calculation unit that calculates a mixing ratio for right image mixing based on the parallax based on the right image, and the left image and the left correction image are mixed based on the mixing ratio for left image mixing, and An image processing apparatus comprising: a mixing unit that mixes the right image and the right correction image based on a mixing ratio for right image mixing.

  According to this configuration, since the brightness of the image is corrected based on the parallax detected from the stereo image, it is possible to perform dark part correction without affecting the appropriate exposure region. Further, according to such a configuration, since only the luminance differs between the corrected image and the original image before correction, the synthesized image becomes unnatural even when a moving object is photographed. The phenomenon can be prevented from occurring. Furthermore, according to such a configuration, it is only necessary to perform correction based on distance information detected from the stereo image, so it is not necessary to perform mode setting for performing dark portion correction at the time of shooting, and correction processing is performed as necessary after shooting. If darkening is performed, dark part correction is possible.

  Furthermore, the technology according to the present embodiment can cope with not only the backlight but also the situation where the background of the subject sunk darkly due to the spotlight or flash. That is, it is possible to perform accurate dark part correction that does not affect the dark part. For example, by correcting brightly only the subject positioned at the front during backlighting, it is possible to correct the dark part without affecting the background area that is appropriately exposed. If there is a bright subject in front, the dark portion can be corrected without affecting the subject that is properly exposed by correcting only the background brightly. According to the technique according to the present embodiment, since it is determined whether to adopt a method for correcting the front side or a method for correcting the back side, the processing is switched, so that both patterns can be handled.

  The mixing ratio calculation unit calculates a mixing ratio for left image mixing for each block based on the left image reference parallax detected for each block, and calculates the right image reference parallax detected for each block. Based on this, the mixing ratio for right image mixing may be calculated for each block. According to such a configuration, the mixing ratio can also be calculated for each block based on the parallax detected for each block.

  The parallax detector calculates the parallax based on the left image for each pixel by linear interpolation between blocks with respect to the parallax based on the left image detected for each block, and the right image standard based on the right image detected for each block. The right image reference parallax is calculated for each pixel by linear interpolation between blocks with respect to the parallax, and the mixing ratio calculation unit calculates a mixing ratio for left image mixing based on the left image reference parallax calculated for each pixel. May be calculated for each pixel, and a mixing ratio for right image mixing may be calculated for each pixel based on the parallax based on the right image calculated for each pixel. According to such a configuration, it is possible to calculate the mixing ratio for each pixel based on the parallax for each pixel by linear interpolation between blocks.

  The brightness correction unit may generate the left correction image and the right correction image by increasing the brightness of each of the left image and the right image. According to such a configuration, it is possible to perform correction for lifting a dark portion on an image.

  When the parallax falls within a predetermined range, the mixing ratio calculation unit may calculate a larger mixing ratio by which the left correction image and the right correction image are multiplied as the parallax is larger. According to such a configuration, for example, when a correction image is generated by performing correction to increase the luminance, correction for increasing the luminance is performed on the subject area in front, so that the backlight correction is appropriately performed. Is possible.

  The image processing apparatus further includes a determination unit that determines a mixture ratio calculation method based on a distribution of combinations of luminance of at least one of the left image and the right image and parallax of the image reference, The mixing ratio calculation unit may calculate the mixing ratio for the left image mixing and the mixing ratio for the right image mixing using the mixing ratio calculation method. According to this configuration, for example, it is possible to correct a dark part without affecting a background area that is appropriately exposed by correcting brightly only a subject positioned in front in backlighting. If there is a bright subject in front, the dark portion can be corrected without affecting the subject that is properly exposed by correcting only the background brightly.

  The determination unit determines either the first mixing ratio calculation method or the second mixing ratio calculation method, and the mixing ratio calculation unit uses the first disparity calculation method when the first mixing ratio calculation method is used. Is within the predetermined first range, the larger the parallax is, the larger the mixture ratio multiplied by the left correction image and the right correction image is calculated, and when the second mixture ratio calculation method is used, When the parallax falls within a predetermined second range, the mixing ratio multiplied by the left correction image and the right correction image may be calculated as the parallax increases. According to such a configuration, for example, when a corrected image is generated by performing correction to increase the brightness, correction is performed to increase the brightness for the subject area in the foreground during backlighting, and a bright subject is in front. In some cases, a correction is made to increase the brightness relative to the background.

  According to another embodiment of the present invention, the step of detecting, for each block, a left image-based parallax and a right image-based parallax between a left image and a right image captured by a stereo camera; Generating a left corrected image and a right corrected image by correcting the luminance of each of the left image and the right image; calculating a mixing ratio for mixing the left image based on the parallax of the left image reference; and Calculating a mixing ratio for right image mixing based on a parallax based on the right image; mixing the left image and the left correction image based on the mixing ratio for left image mixing; and Mixing the right image and the right-corrected image based on the mixing ratio of the image processing method.

  According to this method, since the brightness of the image is corrected based on the parallax detected from the stereo image, it is possible to perform dark part correction without affecting the appropriate exposure region. Further, according to such a configuration, since only the luminance differs between the corrected image and the original image before correction, the synthesized image becomes unnatural even when a moving object is photographed. The phenomenon can be prevented from occurring. Furthermore, according to such a configuration, it is only necessary to perform correction based on distance information detected from the stereo image, so it is not necessary to perform mode setting for performing dark portion correction at the time of shooting, and correction processing is performed as necessary after shooting. If darkening is performed, dark part correction is possible.

  As described above, according to the present invention, it is possible to correct a dark portion without impairing the gradation of the appropriate exposure region while avoiding the synthesis of an unnatural image.

It is a figure which shows the structural example of the image processing apparatus which concerns on embodiment of this invention. It is a figure which shows the structural example of the calculation part of an image processing apparatus. It is a figure which shows the example of the left image and right image which were imaged with the stereo camera. It is a figure for demonstrating the function of a brightness correction part. It is a figure for demonstrating the function of a parallax detection part. It is a figure which shows the example of a classification | category with respect to the combination of a brightness | luminance and parallax. It is a figure which shows the example which estimates the property of an image according to distribution of a combination. It is a figure which shows the example of the mixing ratio calculation method according to the property of an image. It is a flowchart which shows the flow of operation | movement of an image processing apparatus.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

  In the present specification and drawings, a plurality of constituent elements having substantially the same functional configuration may be distinguished by attaching different alphabets after the same reference numeral. However, when it is not necessary to particularly distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are given.

  FIG. 1 is a diagram illustrating a configuration example of an image processing apparatus 10 according to an embodiment of the present invention. As shown in FIG. 1, the image processing apparatus 10 includes a camera 20A, a camera 20B, a development processing unit 110A, a development processing unit 110B, a luminance correction unit 120A, a luminance correction unit 120B, a parallax detection unit 130, a luminance acquisition unit 140, and a calculation. Part 150, mixing part 160A and mixing part 160B. FIG. 2 is a diagram illustrating a configuration example of the calculation unit 150 of the image processing apparatus 10. As illustrated in FIG. 2, the calculation unit 150 includes a determination unit 151 and a mixture ratio calculation unit 152. Hereinafter, the function of each functional block provided in the image processing apparatus 10 will be described in detail sequentially.

  The camera 20A and the camera 20B constitute a stereo camera. That is, each of the camera 20A and the camera 20B acquires a captured image by performing imaging. Hereinafter, an image captured by the camera 20A is referred to as a left image, and an image captured by the camera 20B is referred to as a right image. FIG. 3 is a diagram illustrating an example of a left image and a right image captured by a stereo camera. In the embodiment of the present invention, the image processing apparatus 10 includes a stereo camera. However, the stereo camera may be provided outside the image processing apparatus 10.

  The development processing unit 110A performs development processing on the left image as necessary, and the development processing unit 110B performs development processing on the right image as necessary. For example, when the image sensor is a Bayer sensor, the development process includes a process of demosaicing the captured image to convert it into an RGB signal, a color matrix calculation process, a YUV conversion process, a noise reduction process, and the like.

  The luminance correction unit 120A has a function of generating a left correction image by correcting the luminance of the left image, and the luminance correction unit 120B generates a right correction image by correcting the luminance of the right image. Here, the method of correcting the luminance of each of the left image and the right image is not particularly limited. For example, the luminance correction unit 120A generates a left correction image by increasing the luminance of the left image, and the luminance correction unit 120B generates a right correction image by increasing the luminance of the right image.

  FIG. 4 is a diagram for explaining the function of the luminance correction unit 120. As illustrated in FIG. 4, for example, the luminance correction unit 120 </ b> A can increase the luminance of the left image by performing gamma processing on the Y signal (luminance signal) of the left image. Similarly, for example, the luminance correction unit 120B can increase the luminance of the right image by performing gamma processing on the Y (luminance) signal of the left image. Processing for lifting a dark portion in the captured image is realized by such correction for increasing the luminance.

  Here, as a specific example, luminance correction is performed by performing gamma processing on a Y (luminance) signal. However, luminance correction is performed by performing gamma processing on an RGB signal. Alternatively, the luminance may be corrected by multiplying the RGB signal by a gain.

  FIG. 5 is a diagram for explaining the function of the parallax detection unit 130. As illustrated in FIG. 5, for example, the parallax detection unit 130 uses a left image as a reference between the left image and the right image captured by the stereo camera (hereinafter, referred to as “left image reference parallax”). And the parallax based on the right image (hereinafter referred to as “right image-based parallax”) are detected for each block. The size and shape of each block are not particularly limited. The detection of parallax may be performed by, for example, a block matching method in units of rectangular macroblocks, or may be performed using an optical flow or the like.

  Further, the parallax detection unit 130 may further provide the parallax in units of pixels by interpolating the parallax in units of blocks by linear interpolation between blocks. That is, the parallax detection unit 130 may calculate the parallax based on the left image for each pixel by linear interpolation between blocks with respect to the parallax based on the left image detected for each block. Further, the parallax detection unit 130 may calculate the parallax based on the right image for each pixel by linear interpolation between blocks with respect to the parallax based on the right image detected for each block.

  The luminance acquisition unit 140 acquires the luminance of at least one of the left image and the right image. The luminance acquired by the luminance acquisition unit 140 may be the luminance for each pixel or the luminance for each block. For example, when the luminance acquired by the luminance acquisition unit 140 is the luminance for each block, the average luminance of a plurality of pixels constituting each block may be used as the luminance for each block. The block here is the same as the block used at the time of parallax detection. Whether the luminance is acquired from the left image or the right image may be determined in advance, or may be determined by a user or an application.

  The determination unit 151 determines a mixing ratio calculation method based on the distribution of combinations of the luminance acquired by the luminance acquisition unit 140 and the image-based parallax. For example, the determination unit 151 may classify combinations of luminance and parallax for each block and determine a mixing ratio calculation method based on the classification result.

  FIG. 6 is a diagram illustrating a classification example for a combination of luminance and parallax. As illustrated in FIG. 6, for example, when the threshold value THD1 and the threshold value THD2 are provided for the parallax, the determination unit 151 has a parallax smaller than the threshold THD1 (hereinafter, also referred to as “small parallax”) and larger than the threshold THD1. Thus, each block is classified into either a parallax smaller than the threshold THD2 (hereinafter also referred to as “medium parallax”) or a parallax larger than the threshold THD2 (hereinafter also referred to as “large parallax”).

  Small parallax is a parallax detected when the distance to the subject is long, medium parallax is a parallax detected when the distance to the subject is medium, and large parallax is a short distance to the subject The parallax detected in this case. When the parallax is equal to the threshold value THD1, it may be classified as either a small parallax or a medium parallax. When the parallax is equal to the threshold value THD2, it may be classified as either a medium parallax or a large parallax.

  Further, when the threshold THL1 and the threshold THL2 are provided for the luminance, the determination unit 151 has a luminance smaller than the threshold THL1 (hereinafter also referred to as “low luminance”), a luminance larger than the threshold THL1 and smaller than the threshold THL2. Each block is classified into either one (hereinafter also referred to as “medium luminance”) and luminance greater than the threshold THL2 (hereinafter also referred to as “high luminance”).

  When the luminance is equal to the threshold value THL1, it may be classified as either low luminance or medium luminance, and when the luminance is equal to the threshold value THL2, it may be classified as either medium luminance or high luminance. When such classification is performed, each block is classified into one of the sections C1 to C9 shown in FIG. 6, for example. The determination unit 151 generates the frequency of blocks classified into each category as a frequency distribution. The nature of the image is estimated from this frequency distribution.

  FIG. 7 is a diagram illustrating an example of estimating image properties according to the distribution of combinations. For example, when the subject is irradiated with light from the front (for example, when using a spotlight or when using a flash, this case may be simply referred to as “spotlight” hereinafter). The subject tends to appear bright and the background appears dark. In such a case, the subject has large parallax and high luminance, and therefore belongs to the section C1 shown in FIG. 6, and the background belongs to section C9 because of small parallax and low luminance. As a result, for example, a frequency distribution as shown in “at the time of spotlight” shown in FIG. 7 is taken.

  On the other hand, for example, when the subject is irradiated with light from the back (for example, in the case of backlight), the subject tends to appear dark. In such a case, the subject belongs to the section C7 shown in FIG. 6 because it has a large parallax and low luminance, and the background belongs to the section C3 and the section C6 because it has a small parallax, medium luminance to high luminance. As a result, for example, a luminance distribution as shown in “backlit” shown in FIG. 7 is obtained. As described above, the property of the image can be estimated by looking at the correlation between the parallax and the luminance.

  More specifically, for example, when the determination unit 151 sets the frequency of blocks belonging to the sections C3 and C7 as the first frequency and sets the blocks belonging to the sections C1 and C9 as the second frequency, the first frequency If there is more than the second frequency, it may be estimated that the image was taken “backlit”. In addition, when the second frequency is higher than the first frequency, the determination unit 151 may estimate that the image was taken “during spotlight”. For example, if the determination unit 151 estimates that the image was captured during “backlighting”, the determination unit 151 determines the first mixing ratio calculation method. In addition, for example, if the determination unit 151 estimates that the image was captured during “spotlight”, the determination unit 151 determines the second mixing ratio calculation method.

  The mixing ratio calculation unit 152 calculates a mixing ratio for mixing the left image based on the parallax based on the left image, and calculates a mixing ratio for mixing the right image based on the parallax based on the right image. For example, the mixing ratio calculation unit 152 calculates the mixing ratio for left image mixing for each block based on the left image based parallax detected for each block, and the right image based parallax detected for each block. Based on the above, the mixing ratio for right image mixing is calculated for each block.

  Further, the mixture ratio calculation unit 152 may calculate the mixture ratio based on the parallax calculated for each pixel. In other words, the mixture ratio calculation unit 152 calculates the left image-based mixture ratio for each pixel based on the left-image-based parallax calculated for each pixel, and the right-image-based parallax calculated for each pixel. Based on the above, the mixing ratio for right image mixing may be calculated for each pixel. Therefore, the calculation of the mixing ratio performed for each block may be the calculation of the mixing ratio performed for each pixel.

  The mixing ratio calculation unit 152 may calculate the mixing ratio for left image mixing and the mixing ratio for right image mixing using the mixing ratio calculation method determined by the determination unit 151. For example, when the determination unit 151 estimates that the image was taken “during backlighting”, the first mixing ratio calculation method is determined. Therefore, the mixing ratio may be calculated using the method. In addition, for example, when the determination unit 151 estimates that the image was shot “during spotlight”, the second mixing ratio calculation method is determined. Therefore, if the mixing ratio is calculated using this method, Good.

  FIG. 8 is a diagram illustrating an example of a mixing ratio calculation method according to the properties of an image. As shown in FIG. 8, for example, when it is estimated that the image was captured during “backlighting”, the mixture ratio calculation unit 152 performs left correction as the parallax increases as the parallax falls between the threshold TH3 and the threshold TH4. The mixing ratio multiplied by the image and the right correction image may be calculated to be large. This is because, during “backlight”, it is assumed that the smaller the parallax is, the darker the image is. Therefore, it is preferable to increase the mixing ratio of the corrected images that have been corrected to increase the luminance. The threshold value TH3 and the threshold value TH4 may be changed as appropriate.

  When the parallax is smaller than the threshold TH3, the mixing ratio calculation unit 152 sets the mixing ratio to be multiplied by the left correction image and the right correction image to “0”, and when the parallax is larger than the threshold TH4, the left correction image The mixing ratio multiplied by the right correction image may be “1”. Further, as shown in FIG. 8, the mixing ratio may change linearly in a section where the parallax is larger than the threshold value TH3 and smaller than the threshold value TH4.

  Further, for example, when it is estimated that the image is shot during “spotlight”, the mixing ratio calculation unit 152 increases the left correction image and the right correction as the parallax increases as the parallax falls between the threshold TH1 and the threshold TH2. You may calculate small the mixture ratio multiplied by an image. This is because it is presumed that, when “in spotlight”, the larger the parallax is, the brighter the image is projected, so it is preferable to reduce the mixing ratio of the corrected image that has been corrected to increase the luminance. The threshold value TH1 and the threshold value TH2 may be changed as appropriate.

  A form in which the determination unit 151 is not provided is also assumed. In that case, since there is no function to estimate whether the image was shot “in backlight” or “in spotlight”, it may be determined in advance in which case, or determined by the user or application. May be.

  The mixing unit 160A mixes the left image and the left corrected image based on the mixing ratio for mixing the left image. Further, the mixing unit 160B mixes the right image and the right correction image based on the mixing ratio for right image mixing. More specifically, for example, when the pixel value of the corrected image is P (corrected image), the captured image before correction is P (original image), and the mixing ratio multiplied by the corrected image is K, the following (formula Mixed based on 1).

  P (corrected image) xK + P (original image) x (1-K) (Equation 1)

  Note that the mixing method based on (Equation 1) is merely an example of the mixing method by the mixing unit 160A and the mixing unit 160B, and the mixing method by the mixing unit 160A and the mixing unit 160B is not limited to this example.

  In the above, the function which the image processing apparatus 10 has was demonstrated. Below, the flow of operation of the image processing apparatus 10 will be described. FIG. 9 is a flowchart showing an operation flow of the image processing apparatus 10. In the operation flow shown in FIG. 9, the left image and the right image are not distinguished from each other for ease of explanation. That is, in practice, image processing is performed on both the left image and the right image, but FIG. 9 shows the flow of image processing on one of the left image and the right image.

  First, a captured image (left image and right image) is acquired by the camera 20 (step S1), and development processing is performed on the captured image (left image and right image) as necessary by the development processing unit 110. The brightness correction unit 120 generates a corrected image (left corrected image and right corrected image) by correcting the brightness of the captured image (left image and right image) (step S2). Subsequently, the parallax detection unit 130 detects parallax (left image-based parallax and right-image-based parallax) based on the captured images (left image and right image) (step S3). Subsequently, the determination unit 151 determines the property of the captured image (the left image and the right image) based on the distribution of the combination of the luminance of at least one of the left image and the right image and the image-based parallax. (Step 4). In determining the nature of the captured image (left image and right image), for example, the determination unit 151 captures the captured image (left image and right image) during “backlight” or “when spotlighted”. It is estimated whether it was.

  If the determination unit 151 estimates that the captured image (left image and right image) was captured “backlit”, the determination unit 151 calculates the mixture ratio using the first mixture ratio calculation method (step S6). On the other hand, when it is estimated that the captured image (left image and right image) was captured “at the time of spotlight”, the determination unit 151 calculates the mixing ratio using the second mixing ratio calculation method (step S1). S7). Each of the first mixing ratio calculation method and the second mixing ratio calculation method is as described above.

  Subsequently, the mixing unit 160 mixes the corrected image (left corrected image and right corrected image) and the captured image (left image and right image) based on the mixing ratio calculated by the mixing ratio calculating unit 152 (Step S1). S8). The mixed image is provided by the operation as described above.

  The operation flow of the image processing apparatus 10 has been described above. The technique according to the present embodiment is different from the technique described in the prior art documents that have already been disclosed. The technique disclosed in Japanese Patent Application Laid-Open No. 11-32236 is a technique for raising a dark part by performing gamma processing on an image obtained by photographing, for example. According to this technique, it is possible to easily perform backlight correction. However, according to the technique disclosed in the publication, since a single gamma curve is basically applied to the entire screen, it affects other than the dark part (for example, the foreground part), so the appropriate exposure region (for example, ), The gradation of the background portion is lost, and the phenomenon of whiteout may occur. Further, there are related techniques such as adaptively changing the gamma curve according to the position in the image by looking at the contents of the image, but it is inevitable that the image becomes unnatural due to erroneous determination.

  The technique disclosed in Japanese Patent Laid-Open No. 2000-92378 is a technique for combining two images taken with different exposures, and can enjoy a high backlight correction effect. However, when a fast-moving object is photographed, the displacement of the object becomes large, and the combined image becomes an unnatural image. In addition, it is complicated that it is necessary to take two images after setting the mode at the time of photographing.

  According to the technology according to the present embodiment, it is possible to perform backlight correction processing on an image captured by a stereo camera, and correct only the processing target accurately (correction without affecting the non-correction area). It is possible to obtain a natural processing result. Further, according to the technique according to the present embodiment, it is not necessary to perform mode setting or the like at the time of shooting, and it is possible to enjoy the backlight correction effect by post-processing even after shooting.

  Furthermore, according to the technique according to the present embodiment, it is possible to perform not only the dark part correction at the time of backlighting but also the dark part correction according to the property of the image. For example, it is possible to perform a wide range of dark part correction corresponding to both a situation in which the subject becomes dark during backlighting and a situation in which the background of the subject becomes dark due to being hit with a spotlight or flash.

  The effects produced by the technology according to the present embodiment are as follows, for example. First, as a first effect, it is possible to perform backlight correction without affecting the background. The backlight is a situation where there is a bright object such as a light source in the back and the subject is in front of it, causing the subject to sink darkly. In the technology according to the present embodiment, backlight correction is performed without affecting the background area that is appropriately exposed by brightly correcting only the subject located in front based on the distance information acquired from the stereo image. It can be carried out.

  Furthermore, the technology according to the present embodiment can cope with not only the backlight but also the situation where the background of the subject sunk darkly due to the spotlight or flash. That is, it is possible to perform accurate dark part correction that does not affect the dark part. For example, it is possible to correct a dark part without affecting a background area that is appropriately exposed by correcting brightly only a subject located in front during backlighting. If there is a bright subject in front, the dark portion can be corrected without affecting the subject that is properly exposed by correcting only the background brightly. According to the technique according to the present embodiment, since it is determined whether to adopt a method for correcting the front side or a method for correcting the back side, the processing is switched, so that both patterns can be handled.

  A second effect is that natural dark part correction can be performed even when a moving object is photographed. In terms of synthesizing two images, it is the same as the technique disclosed in Japanese Patent Laid-Open No. 2000-92378. However, in the technology according to the embodiment of the present invention, since only the luminance is different between the corrected image and the original image before correction, the combined image is not correct even when a moving object is photographed. It can prevent the phenomenon of becoming natural. That is, the technology according to the embodiment of the present invention can be applied regardless of the presence or absence of movement of an object to be photographed.

  Further, as a third effect, there is no need to set a mode at the time of shooting, and correction can be made even after shooting. In the technique according to the present embodiment, correction is performed based on distance information detected from a stereo image, so the mode at the time of shooting may be set as usual regardless of whether or not dark portion correction is performed. And dark part correction | amendment is possible if a correction process is performed as needed after imaging | photography.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 20 (20A, 20B) Camera 110 (110A, 110B) Development processing part 120 (120A, 120B) Luminance correction part 130 Parallax detection part 140 Luminance acquisition part 150 Calculation part 151 Determination part 152 Mixing ratio calculation part 160 ( 160A, 160B) Mixing section

Claims (6)

A parallax detection unit that detects, for each block, a left image-based parallax and a right image-based parallax between a left image and a right image captured by a stereo camera;
A luminance correction unit that generates a left correction image and a right correction image by correcting the luminance of each of the left image and the right image;
A mixing ratio calculation unit that calculates a mixing ratio for mixing the left image based on the parallax based on the left image, and calculates a mixing ratio for mixing the right image based on the parallax based on the right image;
Mixing the left image and the left correction image based on the mixing ratio for the left image mixing, and mixing the right image and the right correction image based on the mixing ratio for the right image mixing;
Equipped with a,
The mixing ratio calculation unit calculates a mixing ratio for left image mixing for each block based on the left image reference parallax detected for each block, and calculates the right image reference parallax detected for each block. Based on this, the mixing ratio for right image mixing is calculated for each block,
The image processing apparatus, wherein the brightness correction unit generates the left correction image and the right correction image by increasing the brightness of each of the left image and the right image . The parallax detector calculates the parallax based on the left image for each pixel by linear interpolation between blocks with respect to the parallax based on the left image detected for each block, and the right image standard based on the right image detected for each block. Calculate the parallax based on the right image for each pixel by linear interpolation between blocks with respect to the parallax,
The mixing ratio calculation unit calculates a mixing ratio for left image mixing for each pixel based on the parallax based on the left image calculated for each pixel, and calculates the parallax based on the right image calculated for each pixel. Calculate the mixing ratio for the right image based on each pixel,
The image processing apparatus according to claim 1, wherein: When the parallax falls within a predetermined range, the mixing ratio calculation unit calculates a larger mixing ratio to be multiplied by the left correction image and the right correction image as the parallax increases.
The image processing apparatus according to claim 1, wherein: The image processing apparatus includes:
A determination unit that determines a mixture ratio calculation method based on a distribution of combinations of luminance of at least one of the left image and the right image and parallax of the image reference;
The mixing ratio calculation unit calculates the mixing ratio for the left image mixing and the mixing ratio for the right image mixing using the mixing ratio calculation method.
The image processing apparatus according to claim 1, wherein: The determination unit determines either the first mixing ratio calculation method or the second mixing ratio calculation method,
When the first mixing ratio calculation method is used, the mixing ratio calculation unit multiplies the left correction image and the right correction image as the parallax increases when the parallax falls within a predetermined first range. When the second mixing ratio calculation method is used and the parallax is within a predetermined second range, the larger the parallax is, the larger the parallax is, and the left correction image and the right correction image are calculated. Calculate the mixing ratio to be multiplied
The image processing apparatus according to claim 4 , wherein: Detecting a left image-based parallax and a right image-based parallax between a left image and a right image captured by a stereo camera for each block;
Generating a left corrected image and a right corrected image by correcting the luminance of each of the left image and the right image;
Calculating a left image mixing ratio based on the left image reference parallax, and calculating a right image mixing ratio based on the right image reference parallax;
Mixing the left image and the left correction image based on the left image mixing ratio, and mixing the right image and the right correction image based on the right image mixing ratio;
Only including,
A mixing ratio for left image mixing is calculated for each block based on the parallax based on the left image detected for each block, and for right image mixing based on the parallax based on the right image detected for each block Calculating a mixing ratio for each block;
Generating the left corrected image and the right corrected image by increasing the luminance of each of the left image and the right image .

Images