JP2016072924A - Image processing system and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To speedily and precisely identify a light source color from plural photographic images.SOLUTION: First of all, a criteria image and a reference image are divided into plural blocks and a deviation amount between the images is obtained. Then, alignment is performed by affine transformation on the basis of the obtained deviation amount. Next, at a pixel position of a high-luminance pixel in at least one of the pieces of line data of the aligned images, the difference absolute value of a luminance value is calculated and threshold determination is performed to identify a mirror surface reflection area, and a difference between signal values is calculated to identify the light source color.SELECTED DRAWING: Figure 1

Description

  The present invention particularly relates to an image processing apparatus and an image processing method suitable for use in estimating a light source color.

  Conventionally, a technique for detecting the specular reflection area of an object from an image taken with a digital camera, a digital video camera, etc., estimating the light source color from the detected specular reflection area, and improving the white balance accuracy of the shot image is known. ing. Here, the reflected light from the substance can be broadly divided into two parts, a specular reflection light component and an irregular reflection light component, as shown in FIG.

The specular reflection light component includes a component equal to the light source color, and the intensity is maximum when the incident angle θ _i and the reflection angle θ _{r on} the material surface are equal. On the other hand, the irregularly reflected light component is a reflected light component that is output after incident light is irregularly reflected in the substance, includes the substance color, and is uniform with no dependency on the incident angle and the reflection angle. Assuming that the reflected light at two points A and B on the same material surface is Ia and Ib, the reflected light component is calculated by the following equations (1) and (2).
Ia = fa × S + Da (1)
Ib = fb × S + Db (2)

Here, fa and fb represent the reflectance of the specular reflection light at two points A and B on the object surface, and depend on the incident reflection angle as described above. S represents the intensity of the reflected light when the specular reflection light is reflected 100% on the object surface. Da and Db represent diffusely reflected light components at two points A and B, respectively. When the two points A and B are located in the vicinity of the same object surface, the irregularly reflected light components Da and Db can be regarded as being substantially equal, and therefore the following expression (3) is established.
(Ia−Ib) = (fa−fb) × S (3)

  The above equation (3) indicates that the difference (Ia-Ib) of the reflected light components at the two points A and B is proportional to the specular reflected light component. That is, the specular reflection component (light source color) can be estimated by detecting the luminance difference in the specular reflection region.

  For example, Patent Documents 1 and 2 disclose a method of estimating a light source color by using the property of reflected light. In Patent Document 1, pixel values are classified according to luminance for each block, and the light source color is specified based on at least a partial difference between the average value of the brightest luminance region and the average value of the second brightest luminance region. Techniques to do this are disclosed. Further, Patent Document 2 discloses a specular reflection region by detecting a specific region in which a change amount of a position on a parallax image is not proportional to a movement amount of a viewpoint from a plurality of parallax images captured while moving on a straight line. Techniques for detection are disclosed.

Japanese Patent No. 4447520 Japanese Patent No. 3943288

  However, in the prior art disclosed in Patent Document 1, when the image area for which the difference is taken is not a uniform surface, a difference occurs in the diffuse reflection light component due to a difference in the color or pattern of the object surface. There has been a problem that the reliability of the light component is lowered. In the prior art disclosed in Patent Document 2, in order to extract a curve component, it is necessary to move the image pickup device on a straight line and photograph a plurality of images. Therefore, a device that requires high speed such as a digital video camera is required. Is difficult to apply.

  An object of the present invention is to make it possible to specify a light source color from a plurality of captured images at high speed and with high accuracy.

  An image processing apparatus according to the present invention includes an acquisition unit that acquires a plurality of captured images having parallax, an alignment unit that performs alignment between the plurality of captured images, and an alignment unit that is aligned by the alignment unit. Detecting a region having a luminance value equal to or greater than a predetermined value from the photographed image, and detecting a specular reflection region from a difference in luminance value between the plurality of photographed images in the detected region; and detecting by the detection unit And a specifying means for calculating a difference between signal values of the specular reflection area and specifying a light source color.

  According to the present invention, a light source color can be specified from a plurality of captured images at high speed and with high accuracy.

It is a block diagram which shows the structural example of the digital video camera which concerns on embodiment. It is a flowchart which shows an example of the basic process sequence in embodiment. It is a flowchart which shows an example of the detailed procedure of the alignment process by an image conversion part. It is a flowchart which shows an example of the detailed procedure of the detection process of a specular reflection area | region, and the estimation process of a light source color. It is a figure for demonstrating a specular reflection light component and an irregular reflection light component. It is a figure for demonstrating the parallax image divided | segmented into the some block. It is a figure for demonstrating the procedure which calculates the deviation | shift amount. It is a figure for demonstrating the method to select the deviation | shift amount used for an affine transformation. It is a figure for demonstrating the method to divide | segment a block in the parallax image after alignment. It is a figure which shows an example of a color filter. It is a figure which shows the filter for producing R, G, B plane. It is a figure which shows the direction of line data. It is a figure which shows the position and luminance value which cut out line data. It is a figure for demonstrating the method of specifying a specular reflection area | region. It is a figure which shows the example in case an occlusion area | region generate | occur | produces in two images with parallax. It is a figure which shows an example of the luminance distribution on the line containing a low-intensity area | region.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 is a block diagram illustrating a configuration example of a digital video camera 100 according to the present embodiment.
In FIG. 1, an operation input unit 101 is an input button for transmitting a shooting instruction from a user to the control unit 102, and the control unit 102 controls the entire digital video camera 100 in response to an instruction from the operation input unit 101. I do.

  The imaging unit 103 includes an optical lens and a sensor such as a CCD or CMOS element, and includes a color filter that separates incident light into RGB color components in front of the sensor. As a color filter, a Bayer color filter as shown in FIG. 10A is generally used. The AD conversion unit 104 converts an input signal from the imaging unit 103 into image data and stores the image data in the storage unit 105. The storage unit 105 is an internal memory for temporarily storing image data, and includes a volatile memory such as a DRAM.

  The camera position detection unit 106 is a position detector for detecting the camera position at the time of shooting. Specifically, an acceleration sensor for detecting vertical, horizontal, and front-back position fluctuations of the digital video camera 100 for each image, and a gyro sensor for detecting horizontal and vertical position fluctuations of the digital video camera 100. It consists of. The image conversion unit 107 performs alignment of two pieces of image data captured successively by performing image conversion based on the motion vector in each region of the two pieces of image data having a parallax amount.

  The specular reflection area detection unit 108 detects a high luminance area in the image, calculates a difference in luminance value in the detected high luminance area, and compares the calculated value with a preset threshold value to thereby determine the specular reflection area. Is identified. The light source color detection unit 109 calculates a light source color component from the difference between the R, G, and B signal values in the detected specular reflection region.

  The image processing unit 110 performs white balance processing on the image data based on the calculated light source color component, and performs various development processing such as NR and edge enhancement. The image compression unit 111 compresses the developed image data and converts it into a data format such as JPEG. The external storage device 113 is an external storage memory such as an SD card for storing compressed image data. The external storage control unit 112 controls reading and writing of image data with the external storage device 113.

FIG. 2 is a flowchart showing an example of a basic processing procedure in the present embodiment.
First, when a shooting instruction is received from the operation input unit 101 by a user operation, processing is started. In S201, a plurality of images are continuously shot by the imaging unit 103 and the AD conversion unit 104, and a plurality of images are stored in the storage unit 105. Save the captured images. Moreover, the camera position detection part 106 detects the camera position information at the time of imaging | photography for every image. Based on the detected camera position information, the control unit 102 detects an image (hereinafter referred to as a parallax image) having a desired position variation from continuously captured images.

  Subsequently, in S202, the image conversion unit 107 performs image conversion based on the detected displacement vector (motion vector) of the two parallax images, and aligns the two images. Although details of this processing will be described later, in this embodiment, affine transformation is used as image transformation. For this reason, the parallax image detected in S201 is only an image having a positional variation corresponding to vertical and horizontal translation that can be aligned by affine transformation.

  Also, in S201, an image with horizontal and vertical angle variations that cannot be aligned by affine transformation is not detected as a parallax image. Further, the forward / backward movement that affects the enlargement / reduction of the image may not be used as a parallax image because there is a possibility that the alignment accuracy may be lowered. Instead, the position variation of the horizontal angle and the vertical angle of the camera position can be aligned by using image conversion by projective conversion. However, in this case, the calculation amount for calculating and aligning the deviation amount vector increases as compared with the affine transformation, and therefore it is more preferable to perform the alignment using the affine transformation.

  Subsequently, in S203, the specular reflection area detection unit 108 specifies the specular reflection area by calculating the absolute value of the luminance difference between the two aligned parallax images. Then, the light source color detection unit 109 estimates the light source color from the difference between the R, G, and B signal values in the specified specular reflection region.

  In the next S204, the image processing unit 110 performs WB processing based on the estimated light source color. In step S <b> 205, the image processing unit 110 performs various image processing such as noise reduction and edge enhancement on the image that has been subjected to the WB processing, and converts the image into YUV format data. In step S <b> 206, the image compression unit 111 compresses YUV format data into JPEG format image data. In step S207, the external storage control unit 112 stores the compressed image data in the external storage device 113, and ends the process.

FIG. 3 is a flowchart illustrating an example of a detailed procedure of the alignment processing by the image conversion unit 107 in S202 of FIG.
First, in S301, of the two detected parallax images, the first captured image is defined as a reference image, and the later captured parallax image is defined as a first reference image. Then, the parallax images are simply developed from the Bayer array shown in FIG. 10A to YUV format data by the control unit 102 and the image processing unit 110, and then, as shown in FIG. One reference image 602 is divided into a plurality of blocks.

  Subsequently, in S302, a shift amount between the base image 601 and the first reference image 602 is obtained for each block. As a method of calculating the deviation amount, the correlation value is obtained while moving the reference block 604 within the standard block 603 from the position shown in FIG. Then, as shown in FIG. 7B, the motion vector up to the position where the correlation value is the smallest is taken as the shift amount of the area. Examples of the correlation value include a sum of absolute differences (SAD). As a method for calculating the correlation value based on the sum of absolute differences, for example, a method described in Japanese Patent No. 4781233 is used.

  Subsequently, in S303, a parameter indicating the amount of displacement of the subject is calculated based on the detected amount of displacement of each block. In this embodiment, since affine transformation is used for alignment, affine transformation parameters are calculated. In the parameter calculation procedure, first, as shown in FIG. 8, a shift amount used for affine transformation is selected from the shift amount of each block detected in S302. As shown in FIG. 8, the shift amount is selected by selecting a shift amount that matches the tendency of the overall shift amount vector 801. This is to remove the influence of the misdetected deviation vector due to the influence of the moving body.

  Subsequently, an affine parameter is estimated from the selected shift amount. Assuming that the affine transformation is only parallel movement, rotational movement, and enlargement / reduction, it is expressed as the following equation (4).

  Here, A, B, C, and D represent affine parameters, coordinates (x, y) represent the coordinates of the input image, and coordinates (x ′, y ′) represent the coordinates of the output image. Since the amount of deviation can be considered as a combination of input coordinates and output coordinates, as described above, if at least two deviation amounts are selected, four equations can be made for the four unknowns, so the affine parameters are determined. can do.

  Subsequently, in S304, affine transformation is performed on the first reference image based on the affine parameters calculated in S303. By the affine transformation, the first reference image is deformed (aligned) so as to overlap the base image. The aligned first reference image is stored in the storage unit 105 as a second reference image.

FIG. 4 is a flowchart illustrating an example of a detailed procedure of the specular reflection region detection process and the light source color estimation process in S203 of FIG.
First, in S401, the specular reflection area detection unit 108 divides an image area including the second reference image 901 in the standard image 601 into a plurality of blocks as an effective area 902, as shown in FIG. Hereinafter, a block cut out from each of the standard image 601 and the second reference image 901 will be referred to as a standard block and a reference block. Next, in S402, the specular reflection area detection unit 108 selects one reference block and reference block.

Subsequently, in S403, the specular reflection area detection unit 108 calculates luminance data at each pixel position of the selected standard block and reference block. When calculating the luminance data, first, only the R, G, and B signals are extracted from the image data of the Bayer array as shown in FIGS. The R, G, and B planes are created by applying the filter shown in FIG. Then, luminance data is calculated by obtaining a simple luminance value Y at each pixel position from the same pixel position on each plane using the following equation (5).
Y = 0.3 × R + 0.6 × G + 0.1 × B (5)

  Subsequently, in S404, the specular reflection area detection unit 108 determines whether or not a high-luminance image area having a luminance value equal to or greater than a predetermined value exists in the block, using the calculated luminance data. Since the image area where the specular reflection occurs includes both the irregular reflection light component and the specular reflection light component, the luminance value becomes high. When the specular reflection area is specified using this characteristic, the migration process is performed only on the block including the high-luminance image area. However, the image area where the brightness is saturated is not used because the detection accuracy of the specular reflection light component is lowered. If the result of this determination is that there is no high brightness image area, processing returns to S402 and the next block is selected.

  On the other hand, if the result of determination in S404 is that there is a high-luminance image area, the process proceeds to S405. In step S405, the specular reflection area detection unit 108 determines luminance data for one line (hereinafter, line data) from the luminance data of the base block and the reference block, as shown in FIGS. 13 (a) and 13 (b). ). At this time, the direction of the line data to be cut out is the direction closest to the shift amount direction detected in S303 of FIG. 3 among the four vertical, horizontal, and diagonal directions as shown in FIGS. 12 (a) to 12 (d). To do. Hereinafter, line data extracted from the reference block is referred to as reference line data, and line data extracted from the reference block is referred to as reference line data.

  Subsequently, in S406, as shown in FIG. 14, the specular reflection area detection unit 108 calculates the absolute difference in luminance value at a pixel position where a high luminance pixel exists in at least one of the standard line data and the reference line data. A value is calculated and a threshold value is determined. As described above, in the region where the specular reflection occurs, a difference occurs in the intensity of the reflected light component due to the change in the incident / reflection angle on the object surface due to the parallax. By detecting this difference, a specular reflection region in which the light source color can be estimated is specified.

  However, depending on the shape of the object surface, a blind spot (occlusion area) may occur in two images with parallax, as shown in FIG. If the luminance difference value due to occlusion is detected as the specular reflection region, the accuracy of estimating the light source color is lowered. Since occlusion is likely to occur at a high-contrast edge, it is necessary to detect the high-contrast edge and remove it from the range in which the light source color is estimated. Therefore, for example, as shown in FIG. 16, a distance L between the adjacent high luminance areas A and B is calculated. When the distance L is smaller than the positional shift amount d detected in S303 of FIG. 3 and there is a low luminance area having a luminance value lower than the predetermined threshold Yth2 between the adjacent high luminance areas A and B, Considered as a high contrast edge. Therefore, when this condition is satisfied, the light source color is removed from the estimated range.

  Subsequently, in S407, the specular reflection area detection unit 108 determines whether or not the acquired line data is the last line in the block. If the result of this determination is not the final line, the process returns to S405 and the same processing is repeated. On the other hand, if it is the last line, the process proceeds to the next S408.

  Subsequently, in S408, the light source color detection unit 109 calculates a difference in signal value from each of the R, G, and B planes of the reference block and reference block corresponding to the detected specular reflection area, and extracts the specular reflection light component. .

Here, the specular reflection light component to be extracted differs depending on the luminance difference of each specular reflection area detected in the block. For example, when the signal values in the regions X and Y are X (R, G, B) and Y (R, G, B), and X (100, 200, 400), Y (10, 20, 40) The light intensity is 10 times different. However, when the light source colors are equal, the same white balance gain can be obtained. For example, the white balance gains for R and B calculated from the regions X and Y are GainR (X), GainR (Y), GainB (X), and GainB (Y), respectively. In this case, the calculated R and B white balance gains have the same value as shown in the following equations (6) to (9).
GainR (X) = 200/100 = 2.0 (6)
GainB (X) = 200/400 = 0.5 (7)
GainR (Y) = 20/10 = 2.0 (8)
GainB (Y) = 20/40 = 0.5 (9)

  In S410, which will be described later, it is expected that the accuracy of estimating the light source color is improved by calculating the average value of the white balance gain in the specular reflection area detected in the entire block.

  In step S409, the light source color detection unit 109 determines whether the selected block is the last block in the image. If the result of this determination is that it is not the last block, the process returns to S402 and the same processing is repeated. On the other hand, if the result of determination in S409 is that the selected block is the last block, the process proceeds to S410.

  In S410, the light source color detection unit 109 calculates the average value of the white balance gain calculated in each block, and estimates the light source color. Thus, by calculating the average value of the light source colors (white balance gain) detected from the entire image, the light source color can be estimated with high accuracy.

  As described above, according to the present embodiment, first, the base image and the reference image are aligned, thereby reducing the influence of the irregularly reflected light component when detecting the specular reflection component. Since the specular reflection light component is detected from the luminance difference between the two aligned images, the light source color can be estimated at high speed and with high accuracy. As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 102 Control part 103 Image pick-up part 106 Camera position detection part 107 Image conversion part 108 Specular reflection area detection part 109 Light source color detection part

Claims (6)

Acquisition means for acquiring a plurality of captured images having parallax;
Alignment means for performing alignment between the plurality of captured images;
An area having a luminance value greater than or equal to a predetermined value is detected from each captured image aligned by the alignment means, and a specular reflection area is detected from the difference in luminance value between the plurality of captured images in the detected area. Detecting means for
A specifying means for calculating a difference between the signal values of the specular reflection area detected by the detecting means and specifying a light source color;
An image processing apparatus comprising:   The detecting means is configured such that the distance between adjacent areas having the luminance value equal to or greater than a predetermined value is smaller than the amount of deviation in which the alignment is performed, and the luminance value is lower than a predetermined threshold value between the adjacent areas. The image processing apparatus according to claim 1, wherein when an area exists, the area is not detected as a specular reflection area.   The image processing apparatus according to claim 1, wherein the alignment unit performs alignment between captured images having positional fluctuations in the vertical and horizontal directions.   The image processing apparatus according to claim 1, wherein the alignment unit performs alignment using affine transformation or projective transformation.   The image processing apparatus according to claim 1, wherein the detection unit detects a specular reflection region for each line based on the direction aligned by the alignment unit. An acquisition step of acquiring a plurality of captured images having parallax;
An alignment step of performing alignment between the plurality of captured images;
An area having a luminance value greater than or equal to a predetermined value is detected from each of the captured images aligned in the alignment step, and a specular reflection area is detected from a difference in luminance values between the plurality of captured images in the detected area. A detection step to
A specific step of calculating a difference between the signal values of the specular reflection area detected in the detection step and specifying a light source color;
An image processing method comprising: