JP7054347B2 - Image processing device and its control method - Google Patents

Description

The present invention relates to a technique for adjusting the white balance of an image.

As a technique for performing white balance adjustment processing on images captured under a plurality of light sources having different spectral distributions, Patent Document 1 discloses a technique for dividing an captured image into a plurality of regions and adjusting the white balance for each region. Has been done. More specifically, a method is disclosed in which a user manually divides an captured image into regions and specifies the color temperature of a light source illuminating each region. Further, there is disclosed a method of dividing an captured image into an area where the strobe can reach and an area where the strobe cannot reach and adjusting the white balance for each area based on the distance information of the subject with respect to the image pickup device.

Japanese Unexamined Patent Publication No. 2008-52428

However, in the above-mentioned technique, when manually dividing the area, the user needs to imagine how the subject is illuminated based on knowledge and experience, and the area is divided, which is troublesome. There is. Further, when the region is divided using the distance information, there is a problem that it is difficult to appropriately divide the region when the emission direction and the emission amount of the strobe are different from the assumption.

The present invention has been made in view of such problems, and an object of the present invention is to provide a technique capable of more suitable white balance adjustment.

In order to solve the above-mentioned problems, the image processing apparatus according to the present invention has the following configurations. That is, the image processing device is
The first acquisition means to acquire the input image and
A determination means for determining a representative point in the input image and a white balance coefficient of the representative point.
A second acquisition means for acquiring the first information indicating the inclination of the surface including the representative point and the second information indicating the inclination of the surface including the pixel of interest in the input image.
An adjusting means for adjusting the white balance in the pixel of interest based on the first information, the second information, and the white balance coefficient.
Have,
The first acquisition means further acquires normal information about the subject in each pixel included in the input image.
The adjusting means calculates the degree of similarity between the normal of each pixel included in the input image and the normal at the representative point based on the normal information, and determines the white balance for each pixel included in the input image. , Derived by an interpolation calculation based on the white balance coefficient with the similarity as a weighting coefficient .
Or, the image processing device
The first acquisition means to acquire the input image and
A determination means for determining a representative point in the input image and a white balance coefficient of the representative point, and
An area determining means for determining an adjustment area for adjusting white balance in the input image,
A second acquisition means for acquiring the first information indicating the inclination of the surface including the representative point and the second information indicating the inclination of the surface including the pixel of interest in the input image.
An adjustment means for adjusting the white balance in the pixel of interest in the adjustment region based on the first information, the second information, and the white balance coefficient.
Have,
The first acquisition means further acquires a 3D model of the subject included in the input image, and further acquires the 3D model.
The area determining means determines as the adjustment area a region of pixels determined to have a normal that substantially coincides with the normal of the representative point based on the 3D model.

According to the present invention, it is possible to provide a technique that enables more suitable white balance adjustment.

It is a figure which shows the hardware composition of the image processing apparatus which concerns on 1st Embodiment. It is a functional block diagram of the image processing apparatus which concerns on 1st Embodiment. It is a flowchart of a series of processing in 1st Embodiment. It is a figure which shows the example of the input image and the normal information. It is a figure which shows the example of UI. It is a detailed flowchart of adjustment parameter setting process. It is a figure explaining the coordinate system. It is a figure explaining the degree of similarity of a normal vector. It is a figure which shows the example of the interpolation based on the adjustment parameter. It is a flowchart of a series of processing in 2nd Embodiment. It is a figure which shows the example of the representative point arrangement. It is a figure which shows the example of the representative normal vector. It is a figure which shows the example of the input image, the normal information, and the distance information. It is a flowchart of a series of processing in 3rd Embodiment. It is a figure which shows the example of UI. It is a flowchart of a series of processing in 4th Embodiment. It is a figure which shows the example of a standard face model. It is a flowchart of a series of processing in 5th Embodiment. It is a figure which shows the example of the input image data. It is a figure which shows the example of UI. It is a detailed flowchart of a readjustment area setting process. It is a flowchart of a series of processing in 6th Embodiment. It is a figure which shows the example of UI. It is a flowchart of a series of processing in 7th Embodiment. It is a flowchart of a series of processing in 8th Embodiment.

Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. It should be noted that the following embodiments are merely examples, and are not intended to limit the scope of the present invention.

(First Embodiment)
As a first embodiment of the image processing apparatus according to the present invention, a mode for determining a white balance adjustment parameter will be described below as an example. Here, it is assumed that a plurality of representative points given the coordinates on the captured image and the white balance adjustment parameter are set, and the surface facing the same normal direction as the representative point is illuminated by the same light as the representative point. The white balance adjustment parameter (white balance coefficient) for each pixel is determined. In the following description, the color temperature is used as an adjustment parameter.

<Device configuration>
FIG. 1 is a diagram showing a hardware configuration of the image processing apparatus 100 according to the first embodiment. The image processing device 100 includes a CPU 101, a RAM 102, a ROM 103, an HDD 104, an HDDI / F105, an input I / F106, an output I / F107, and a system bus 108.

The CPU 101 uses the RAM 102 as a work memory to execute a program stored in the ROM 103 and the hard disk drive (HDD) 104, and controls each part described later via the system bus 108. The HDD interface (I / F) 105 is an interface such as Serial ATA (SATA) that connects a secondary storage device such as an HDD 104 or an optical disk drive. The CPU 101 can read data from the HDD 104 and write data to the HDD 104 via the HDDI / F105.

Further, the CPU 101 can expand the data stored in the HDD 104 into the RAM 102, and similarly, can store the data expanded in the RAM 102 in the HDD 104. Then, the CPU 101 can regard the data expanded in the RAM 102 as a program and execute it. The input interface (I / F) 106 is a serial bus interface for connecting an input device such as a mouse / keyboard 109, for example, USB. The CPU 101 can read various signals from the mouse / keyboard 109 via the input I / F 106. The output interface (I / F) 107 is a video output interface such as DVI to which a display device such as a display 110 is connected. The CPU 101 can send data to the display 110 via the output interface 107 to execute the display. If a bidirectional communication interface such as USB or IEEE 1394 is used, the input I / F 106 and the output I / F 107 can be combined into one.

FIG. 2 is a functional block diagram of the image processing device 100. The image data acquisition unit 201 acquires the input image data and the normal information of the subject stored in advance from the HDD 104 via the HDDI / F105. Alternatively, an image pickup device (not shown) such as a digital camera or a 3D scanner may be connected to the image processing device 100 and acquired from the image processing device 100. Here, a normal map is used as normal information.

FIG. 4 is a diagram showing an example of input images and normal information. In the input image data I shown in FIG. 4A, the coordinates (i, j) have the RGB values Ir (i, j), Ig (i, j), and Ib (i, j) of the subject as pixel values. It is stored in I (i, j). Further, in the normal map N shown in FIG. 4B, the values of each element of the unit normal vector relating to the subject are stored as pixel values in the coordinates (i, j). Specifically, the xyz component value of the normal vector corresponding to the coordinates (i, j) of the input image data I is stored as N (i, j). That is, the pixel value N (i, j) = (Nx (i, j), Ny (i, j), Nz (i, j)). Here, the xyz coordinates are set with reference to the coordinates in the input image data. Specifically, the horizontal direction of the input image data is the x-axis, the vertical direction is the y-axis, and the depth direction (that is, the optical axis direction of the image pickup apparatus) is the z-axis. The acquired input image data and the normal map are sent to the parameter setting unit 202.

The parameter setting unit 202 displays the UI for setting the representative point on the display 110 via the output I / F 107. Further, the value instructed by the user on this UI is received via the input I / F 106, and the coordinates on the input image of the representative point and the adjustment parameter for the representative point are set. Then, the adjustment parameter for each pixel in the input image is determined by using the set coordinates of the representative point, the adjustment parameter, and the normal map. Details of these processes will be described later. The determined adjustment parameter is sent to the color processing unit 203.

The color processing unit 203 adjusts the white balance of the input image data using the adjustment parameters and generates the output image data. The generated output image data is sent to the display 111 and displayed.

<Operation of the device>
FIG. 3 is a flowchart of a series of processes according to the first embodiment. This series of processes is realized by reading a computer-executable program describing the procedure shown below from the ROM 103 or the HDD 104 onto the RAM 102, and then executing the program by the CPU 101.

In step S301, the image data acquisition unit 201 acquires the input image data and the normal map which is the normal information corresponding thereto from the HDD 104 or the image pickup apparatus. In step S302, the parameter setting unit 202 outputs and displays the input image data acquired in S301 and the UI for setting the representative point on the display 110.

FIG. 5 is a diagram showing an example of a UI for setting a representative point. The image display area 501 is an area in which input image data is displayed. The user can set the representative point via the UI while referring to the displayed input image data.

In step S303, the parameter setting unit 202 acquires the value instructed by the user on the UI displayed in S302, and sets the coordinates of the representative point and the adjustment parameter. In the example of FIG. 5, when the "add" button 502 is pressed, one representative point to which an ID is newly assigned is generated, and the ID is selected in the ID selection list 503. When the coordinate value is input to the coordinate setting field 504 by the user, the value is set as the coordinates of the representative point selected in the selection list 503. When the user inputs a color temperature value in the adjustment parameter setting field 505, the color temperature is set as an adjustment parameter (representative parameter) for the selected representative point. When changing the coordinates and adjustment parameters of an existing representative point, new values are input to the coordinate setting field 504 and the adjustment parameter setting field 505 with the ID of the representative point selected in the ID selection list 503. In addition to directly inputting a numerical value, by clicking a point in the image display area 501 with the mouse, the coordinate value (i, j) on the image corresponding to that point is acquired and the coordinate value is set. You may enter it in the field 504. In addition to directly inputting a numerical value, the color temperature may be input via the slider 506 or a value corresponding to the selected light source via the preset light source list 507. Hereinafter, the total number of generated representative points will be described as K, and the ID assigned to the representative points will be described as k (k = 1, 2, ..., K).

In step S304, the parameter setting unit 202 uses the normal vector N (i _sk , j) corresponding to the coordinates (i _sk , j _sk ) of each representative point Sk (k = 1, 2, ..., K) set in S303. _sk ) is obtained from the normal map, and this is used as the representative normal vector N _sk . Here, N (i _sk , j _sk ) = (Nx (i _sk , j _sk ), Ny (i _sk , j _sk ), Nz (i _sk , j _sk )). The average value or median value of the normal vector corresponding to the nearby pixels of the coordinates (i _sk , j _sk ) may be used as the representative normal vector N _sk .

In step S305, the parameter setting unit 202 selects one of the pixels constituting the input image for which the adjustment parameter is not set, and sets it as the pixel of interest P (i _p , j _p ). In step S306, the parameter setting unit 202 uses the normal vector N _p = (Nx (i _p , j _p ), Ny (i _p ,) corresponding to the coordinates ( _ip , _jp ) of the pixel of interest P selected in S305. Get j _p ), Nz (i _p , j _p )) from the normal map. In step S307, the parameter setting unit 202 sets the adjustment parameter for the pixel of interest selected in S305.

FIG. 6 is a detailed flowchart of the adjustment parameter setting process (S307). In the following description, k = 2 will be described, but in S602 to 605, the same processing is performed for k = 3 to K.

In step S601, the parameter setting unit 202 stores the color temperature T _s1 of the representative point S1 as the provisional color temperature _T'p of the pixel P of interest. Further, the representative normal vector N _s1 is stored as the intermediate normal vector _N'p of the pixel P of interest.

In step S602, the parameter setting unit 202 calculates the similarity α of the normal vector between the k-th representative point and the pixel of interest. Specifically, the normal vector N _p , the representative normal vector N _sk , and the intermediate normal vector _N'p corresponding to the pixel of interest are converted into polar coordinate display. The conversion of each normal vector from the Cartesian coordinates (x, y, z) to the polar coordinates (r, θ, φ) follows the following formula (1).

FIG. 7 is a diagram illustrating a coordinate system. Specifically, the relationship between the parameters x, y, z of the Cartesian coordinate system and the parameters r, θ, φ of the polar coordinate system is shown. In the example of this figure, the z component of the normal vector of the subject shown in the input image data (that is, facing the direction of the image pickup device) takes a negative value. Further, φ takes a value in the range of π to 2π, and θ takes a value in the range of 0 to π. Since the unit normal vector is stored in the normal map here, the value of r is always "1" from the mathematical formula (1).

Next, the similarity α is calculated using the representative normal vector N _sk represented by polar coordinates, the normal vector N _p of the pixel of interest, and the intermediate normal vector _N'p .

FIG. 8 is a diagram illustrating the similarity α of the normal vector. FIG. 8 is an example in which each normal vector when k = 2 is plotted on a θ−φ plane. The straight line L is a straight line passing through the points _N'p and N _sk , and the point M is a straight line from the point N _p . It is a foot of a perpendicular line lowered to L. Here, the distance d from the point _N'p to the point M in FIG. 8 is normalized by the distance D from the point _N'p to the point N _sk . Then, the value is used as the similarity α between the normal vector N _p of the pixel of interest and the representative normal vector N _sk . At this time, α is expressed by the following mathematical formula (2).

The similarity α is α = 0 when the point M coincides with the intermediate normal vector _N'p , and α = 1 when the point M coincides with the representative normal vector N _sk .

In step S603, the parameter setting unit 202 derives the color temperature of the pixel of interest by interpolation calculation based on the similarity α calculated in S602. Here, the representative color temperature T _sk , the provisional color temperature _T'p , and the similarity α are used. Then, the color temperature T _p (i _p , j _p ) of the coordinates ( _ip , jp) of the pixel of interest _P is derived according to the following mathematical formula (3).

Here, T _th0 and T _th1 are predetermined lower and upper limit values of the color temperature, respectively. The color temperature of sunlight and lighting fixtures, which are typical lighting lights, is about 2000 [K] for low ones such as sunsets and candle flames, and about 12000 [K] for high ones such as clear skies. Therefore, here, T _th0 = 2000 and T _th1 = 12000. In the formula (3), when the interpolation is extrapolated (α <0 or 1 <α), the interpolation method is switched according to the values of the color temperatures _T'p and T _sk and the difference value ΔT thereof.

FIG. 9 is a diagram showing an example of interpolation based on adjustment parameters. 9 (a) to 9 (f) show an example of interpolation when the mathematical formula (3) is used. 9 (a) and 9 (b) are examples in which the provisional color temperature _T'p and the representative point color temperature T _sk are both between [T _th0 , T _th1 ]. In these cases, the color temperature T _p of the pixel of interest is interpolated within the range of [T _th0 , T _th1 ]. 9 (c) to 9 (f) are examples in which the color temperature T _sk of the representative point is not between [T _th0 , T _th1 ]. In these cases, the color temperature T _p of the pixel of interest is interpolated so as not to go outside the range of [T _th0 , T _th1 ].

In step S604, the parameter setting unit 202 obtains a normal vector corresponding to the point M in FIG. 8 according to the following mathematical formula (4) based on the similarity α calculated in S602, and uses this as a new intermediate method for the pixel of interest. Store as a line vector _N'p .

In step S605, when k <K, the parameter setting unit 202 stores the color temperature T _p (i _p , j _p ) of the pixel of interest as a new provisional color temperature _T'p . Then, the value of k is incremented by 1 and the process returns to S602. When k = K, the color temperature of the pixel of interest is determined to be T _p (i _p , j _p ).

In step S308, the parameter setting unit 202 determines whether the adjustment parameters are set for all the pixels in the input image. If the setting for all pixels is completed, the process proceeds to S309, and if not, the process returns to S305.

In step S309, the color processing unit 203 adjusts the white balance of the input image data using the adjustment parameters of each pixel set in S307, and generates the output image data. Specifically, according to the following formula (5), the RGB values Ir (i, j), Ig (i, j), Ib (i, j) of each pixel in the input image data are set to the color temperature T _p (i). Multiply the gain coefficients Gr (T _p (i, j)), Gg (T _p (i, j)), and Gb (T _p (i, j)) corresponding to, j), respectively. Then, the obtained values are the RGB values I'r (i, j), I'g (i, j), and I'b (i, j) of the output image.

The gain coefficients Gr (T _p (i, j)), Gg (T _p (i, j)), and Gb (T _p (i, j)) correspond to the color temperature stored in advance and the gain coefficient. Determine by referring to the table that defines the relationship. If the color temperature that matches T _p (i, j) does not exist on the table, it may be obtained by interpolation from the gain coefficient corresponding to the color temperature in the vicinity thereof.

In step S310, the color processing unit 203 outputs the output image data generated in S309 to the display 110 and displays it.

As described above, according to the first embodiment, the white balance of the captured image can be easily and appropriately adjusted by using the normal information corresponding to the subject in the image.

In the above description of S303, an example of setting all the representative points based on the user's instruction has been described, but some representative points may be automatically set based on the input image data or the normal map. .. For example, the color temperature obtained from the entire input image data I by using a known auto white balance method may be set with k = the color temperature T _s1 of the first representative point S1. Further, the average value or the median value of the normal vector of the normal map N may be set as the representative normal vector N1.

Further, in the above description, an example of interpolating the color temperature of the pixel of interest by repeatedly using the above formula (3) for each representative point of k = 2 to K has been described, but the interpolation is performed by using another method. You may. Interpolation is possible using an arbitrary method in which the corresponding color temperature is largely reflected in the interpolation result as the representative points have similar normal vector directions. For example, by associating the color temperature T with an axis perpendicular to the θ−φ plane and obtaining a curved surface passing through K representative points in this three-dimensional space by using known spline interpolation or the like, the normal vector of the pixel of interest The color temperature of the pixel of interest may be calculated from a point on the curved surface corresponding to.

(Second Embodiment)
In the second embodiment, an example of automatically setting a representative point using normal information will be described. The hardware configuration and the functional block configuration of the image processing device are the same as those of the first embodiment (FIGS. 1 and 2). Hereinafter, the points different from those of the first embodiment in the operation of each functional unit will be mainly described.

The parameter setting unit 202 determines the representative normal vector based on the predetermined representative point arrangement. Further, the input image data is divided into areas using the representative normal vector, and the white balance adjustment parameter corresponding to the representative normal vector is determined based on the color information of each area.

FIG. 10 is a flowchart of a series of processes in the second embodiment. Since steps S301 and S305 to 310 are the same as those in the first embodiment, the description thereof will be omitted.

In step S1002, the parameter setting unit 202 acquires a predetermined representative point arrangement from a storage device such as the HDD 104. Then, the normal vector stored in the corresponding coordinates in the normal map acquired in S301 is acquired and used as the representative normal vector N _sk (k = 1, 2, ..., K).

FIG. 11 is a diagram showing an example of representative point arrangement. In this figure, the black dots represent the positions of the representative points. When obtaining the coordinates on the normal map, the horizontal axis and the vertical axis of the representative point arrangement may be scaled according to the horizontal width and the vertical width of the normal map, respectively. The arrangement of representative points is not limited to the example shown in FIG. 11, and may be different depending on the shooting mode and the arrangement of AF points for focusing.

In step S1003, the parameter setting unit 202 uses the representative normal vector determined in S1002 to divide the input image data acquired in S301 into regions so that pixels having similar normal vectors belong to the same region. Specifically, the inner product of each representative normal vector N _sk (k = 1, 2, ..., K) for the normal vector N (i, j) corresponding to the coordinates (i, j) of the input image data I. Are calculated respectively. Then, the ID (k = kmin) of the representative normal vector when the value becomes maximum is obtained, and the pixels of the coordinates (i, j) are added to the region R _kmin .

In step S1004, the parameter setting unit 202 determines the white balance adjustment parameter T _sk corresponding to the representative normal vector N _sk based on the color information of the pixels belonging to the region R _k . A known auto white balance adjustment method can be used to determine the white balance adjustment parameter based on the color information. For example, the gain coefficients (Gr _k , Gg _k , Gb _k ) such that the average RGB values Ir _k , Ig _k , and Ib _k of the pixels belonging to the region R _k are achromatic are calculated according to the following mathematical formula (6).

Next, the color temperature corresponding to the gain coefficient (Gr _k , Gg _k , Gb _k ) is obtained by referring to the table in which the correspondence relationship between the gain coefficient and the color temperature stored in advance is defined. Then, this is set as the white balance adjustment parameter T _sk corresponding to the representative normal vector N _sk .

As described above, according to the second embodiment, the white balance of the captured image can be adjusted more easily as compared with the first embodiment.

In the above description of S1002, an example of acquiring a representative normal from a normal map based on a predetermined representative point arrangement has been described, but a plurality of predetermined unit normal vectors are represented. It may be acquired as a line. FIG. 12 is a diagram showing an example of a representative normal.

Further, for example, the representative normal vector is acquired from the coordinates of the representative point instructed by the user in the processes of S302 to S304 described in the first embodiment, and the representative method is obtained in the processes of S1003 to S1004 described in the second embodiment. It may be configured to determine the color temperature corresponding to the line vector.

Further, in the first and second embodiments, the example in which the color temperature is used as the white balance adjustment parameter has been described, but the gain coefficient may be used as the white balance adjustment parameter. That is, the gain coefficient for the representative normal vector may be set in S303 or S1004, and the gain coefficient for each pixel may be interpolated in S307.

Further, in the first and second embodiments, an example of obtaining the white balance adjustment parameter using the input image data and the corresponding normal information has been described, but the distance information may also be used in combination. For example, the input image data may be separated into a foreground region and a background region based on the distance information, and the above processing may be applied to each region. In that case, even if the lighting environment of the foreground and the background is significantly different and there is a subject having a normal vector similar to the foreground and the background, it is possible to appropriately adjust the white balance.

(Third Embodiment)
In the third embodiment, an example in which the white balance adjustment parameter is determined by using the input image data and the corresponding normal information in combination with the feature amount information that defines the feature amount other than the normal information will be described. In particular, an example in which distance information is used as feature amount information will be described. The hardware configuration and the functional block configuration of the image processing device are the same as those of the first embodiment (FIGS. 1 and 2). In the following description, the distance information is used as the feature amount information, and the gain coefficient is used as the adjustment parameter.

The image data acquisition unit 201 acquires the distance information of the subject in addition to the input image data and the normal information of the subject. Here, a distance map is used as the distance information.

FIG. 13 is a diagram showing an example of an input image, normal information, and distance information. 13 (a), (b), and (c) show examples of input image data I, normal map N, and distance map Z. The coordinates (i, j) of the distance map Z shown in FIG. 13 (c) are associated with the coordinates (i, j) of the input image data I, respectively. Pixels Z (i, j) of the distance map have pixels (subject distance in the optical axis direction) from the image pickup device that captured the input image data to the subject corresponding to the pixel I (i, j) of the input image data. It is stored as a value.

The parameter setting unit 202 displays a UI for setting the representative point on the display 110 as in the first embodiment, and sets the coordinates on the input image of the representative point and the adjustment parameter for the representative point. Then, the adjustment parameters for each pixel in the input image are calculated by using the coordinates and adjustment parameters of the set representative points, the normal map, and the distance map. At that time, when there are a plurality of representative points whose normal information is substantially the same and whose adjustment parameters are different, the amount (weight coefficient) applied to the adjustment parameters of each representative point is calculated based on the distance information. Details of these processes will be described later. The determined adjustment parameter is sent to the color processing unit 203.

FIG. 14 is a flowchart of a series of processes according to the third embodiment.

In step S1401, the image data acquisition unit 201 acquires the input image data and the corresponding normal map and distance map from the HDD 104 or the image pickup device. In step S1402, the parameter setting unit 202 outputs and displays the input image data acquired in S1401 and the UI for setting the representative point on the display 110.

FIG. 15 is a diagram showing an example of UI. The input image data is displayed in the image display area 1501 of FIG. The user can set the representative point via the UI while referring to the displayed input image data.

In step S1403, the parameter setting unit 202 acquires the value instructed by the user via the UI displayed in S1402, and sets the coordinates of the representative point and the adjustment parameter. In the example of FIG. 15, when one point on the image display area 1501 is selected, one representative point newly assigned an ID is added to the representative point list 1502, and the coordinates on the image corresponding to the selected point are obtained. (I, j) is set as the coordinates of the representative point. Further, when a light source is selected from the preset light source list 1503 in the row corresponding to each representative point on the UI, the gain coefficient corresponding to the light source is set as the adjustment parameter of the representative point.

If the "Delete" button 1505 is pressed while the check box 1504 provided in each row of the adjustment parameter list is checked, the checked row is deleted from the adjustment parameter list and the setting related to the corresponding representative point is discarded. To. When "color temperature" is selected as the light source, the gain coefficient corresponding to the color temperature input in the color temperature setting field 1506 in that row is set as the adjustment parameter. When "click white balance" is selected, a gain coefficient that converts the RGB value of the input image data corresponding to the representative point into an achromatic color is set as an adjustment parameter.

In step S1404, the parameter setting unit 202 has two representative points Sk, Sk'(k, k'= 1, 2, ..., K, k <k') out of the K representative points set in S1403. To make a pair of representative points.

In step S1405, the parameter setting unit 202 determines whether or not the normal information matches between the representative points of the pair selected in S1404. Specifically, the unit normal vectors N _sk and N _{sk'corresponding} to the coordinates of the representative points Sk and Sk'are obtained from the normal map. Then, when these inner products are equal to or more than a predetermined threshold value, it is determined that the normal vectors match. If it is determined that the normal vectors match, the process proceeds to S1406, and if it is determined that the normal vectors do not match, the process proceeds to S1411.

In step S1406, the parameter setting unit 202 determines whether or not the adjustment parameters are different between the representative points of the pair determined in S1405 that the normal vectors match. Specifically, when the gain coefficient corresponding to the representative point Sk and the gain coefficient corresponding to Sk'satisfy the following mathematical formula (7), it is determined that the adjustment parameters are different. Here, the gain coefficients corresponding to the representative points Sk are Gr _k , Gg _k , and Gb _k . The gain coefficients corresponding to Sk'are Gr _k' , Gg _k' , and Gb _k' .

Here, ThG is a predetermined threshold value with respect to the ratio of the gain coefficients. For example, when ThG = 0, it is determined that the adjustment parameters are different except when the ratio of the gain coefficients of R (red) and B (blue) to G (green) completely matches between the representative points Sk and Sk'. Will be done. If it is determined that the adjustment parameters are different, the process proceeds to S1407, and if not, the process proceeds to S1410.

In step S1407, the parameter setting unit 202 determines whether or not the distance information is different between the representative points of the pair for which the normal information is the same and the adjustment parameters are determined to be different in S1405 and S1406. Specifically, the subject distances Z _sk and Z _{sk'corresponding} to the coordinates of the representative points Sk and Sk'are acquired from the distance map. Then, when the absolute value of these differences is equal to or greater than a predetermined threshold value, it is determined that the subject distance is different. If it is determined that the subject distances are different, the process proceeds to S1408, and in other cases, the process proceeds to S1409.

In step S1408, the parameter setting unit 202 classifies the representative points Sk and Sk'of the pair for which the normal information is the same and the adjustment parameters and the subject distance are different in S1405 to S1407 as the distance weight calculation target.

In step S1409, the parameter setting unit 202 invalidates one of the representative points of the pair determined in S1405 to S1407 that the normal information and the distance information match and the adjustment parameters are different. Here, the adjustment parameter instructed later is prioritized over the adjustment parameter instructed to the user earlier, and the representative point Sk with a young ID is invalidated. The time stamp when each representative point is set in S1403 may be stored together with the coordinates and adjustment parameters, and the older one of the time stamps may be invalidated.

In step S1410, the parameter setting unit 202 integrates the representative points of the pair determined in S1405 and S1406 that the normal information and the adjustment parameter match. Specifically, the average value of the unit normal vector and the average value of the gain coefficient are obtained from the unit normal vector and the gain coefficient corresponding to the representative points Sk and Sk'of the pair, respectively. Then, these are stored as new normal information and adjustment parameters for the representative point Sk'set later. Then, the setting related to the representative point Sk previously set is discarded. Alternatively, as in S1409, one representative point may be invalidated.

In step S1411, the parameter setting unit 202 determines whether or not the comparison of S1405 has been completed for all the pairs consisting of the set representative points. If the comparison is completed for all pairs, the process proceeds to S1412, and if not, the process returns to S1404.

In step S1412, the parameter setting unit 202 calculates adjustment parameters for the pixels of the coordinates (i, j) constituting the input image. First, for all the set representative points Sk, the weighting coefficient W _nk (i, j) based on the similarity of the normal vector with the coordinates (i, j) is calculated according to the following mathematical formula (8).

Here, C _valid is a set of IDs indicating valid (that is, not set as invalid) representative points. Further, N _ij and N _sk are unit normal vectors corresponding to the coordinates (i, j) acquired from the normal map and the representative point Sk, respectively.

Β _k (i, j) in the equation (8) corresponds to the similarity between the normal vector of the coordinates (i, j) and the normal vector of the representative point Sk, and the coordinates (i, j) and the representative point Sk. The more similar the normal vectors, the larger the value. The weighting factor W _nk (i, j) in equation (8) is the value obtained by normalizing the similarity of the normal vector using the sum of all the representative points except the invalid representative points for the valid representative points. Become. The larger β _k (i, j) is, the larger the value is (0 ≦ W _nk (i, j) ≦ 1). Further, for an invalid representative point, the weighting coefficient W _nk (i, j) is always "0".

Next, for all the representative points Sk, the weighting coefficient W _zk (i, j) based on the similarity of the distance with the coordinates (i, j) is calculated according to the following mathematical formula (9).

Here, C (Sk) is a set of IDs indicating the representative points classified as the distance weight calculation target whose normal vector coincides with the representative point Sk. Further, Z _ij and Z _sk are subject distances corresponding to the coordinates (i, j) acquired from the distance map Z and the representative point Sk, respectively, and Δz _max is Δz _l (i, j) (l ∈ C (l ∈ C). Sk))) is the maximum value. The γ _k (i, j) in the formula (9) corresponds to the similarity between the subject distance of the coordinates (i, j) and the subject distance of the representative point Sk, and the subject distance between the coordinates (i, j) and the representative point Sk. The more similar, the larger the value.

In the weighting coefficient W _zk (i, j) of the formula (9), the similarity of the subject distances is normalized by using the sum of all the representative points where the normal vectors match for the representative points of the distance weight calculation target. It becomes a value. Therefore, the larger the γ _k (i, j), the larger the value (0 ≦ W _zk (i, j) ≦ 1). Further, the weight coefficient W _zk (i, j) is always "1" for the representative points that are not classified as the distance weight calculation target.

Finally, using the gain coefficients Gr _k , Gg _k , Gb _k and the weighting coefficients W _nk (i, j), W _zk (i, j) of each representative point, the coordinates (i, The gain coefficients Gr (i, j), Gg (i, j), and Gb (i, j) of j) are calculated.

Here, the product W _nk (i, j) * W _zk (i, j) of the weighting coefficients corresponds to the amount of the adjustment parameter coordinates (i, j) of the representative point Sk applied to the pixels. The gain coefficient obtained by the equation (10) reflects the gain coefficient of the representative point as the representative point has a normal vector having a direction similar to that of the normal vector of the coordinates (i, j). Further, when different gain coefficients are set for a plurality of representative points having substantially the same normal information, it is similar to the subject distance of the coordinates (i, j) among the representative points having the same normal vector. The more the representative point has the subject distance, the larger the gain coefficient is reflected. Since W _nk (i, j) = 0 in the equation (8) for the invalid representative point, the applied amount is "0" for the adjustment parameter of the representative point.

In step S1413, the color processing unit 203 generates RGB values of the output image data according to the following mathematical formula (11) using the adjustment parameters of each pixel calculated in S1412. Specifically, the RGB values Ir (i, j), Ig (i, j), and Ib (i, j) of each pixel in the input image data are converted, and the RGB values I'r (i) of the output image data are converted. , j), I'g (i, j), I'b (i, j) are generated.

In step S1414, the color processing unit 203 outputs the output image data generated in S1413 to the display 110 and displays it.

As described above, according to the third embodiment, it is possible to appropriately adjust the white balance of the captured image even when the subject surface having a similar normal vector exists over different regions in the lighting environment. It becomes.

In the above description, an example in which the subject distance is used as a feature amount other than the normal information has been described, but other feature amounts may be used. For example, a known object recognition process is applied to the 3D position coordinates of the subject, the coordinates on the 2D image plane, the color information (brightness, saturation, chromaticity, hue, etc.) of the input image data, and the input image data. It is possible to use the recognition result of the subject obtained by the above. Further, a plurality of these feature quantities may be used in combination. In either case, if it is determined in S1407 whether or not the feature amount other than the normal information is different between the representative points of the pair, and the pair determined to be different is classified into the weight calculation target based on the feature amount in S1408. good. When calculating the weight based on the feature amount in S1412, the weighting coefficient may be calculated based on the degree of similarity of each feature amount so that the representative point whose feature amount is similar to the pixel of interest has a larger weight. ..

Further, when the coordinates match between the representative points of the pair selected in S1404 described above, it may be considered that the user has redone the adjustment instruction for the coordinates, and the previously set representative points may be invalidated. In that case, the processing of S1405 to S1407 may be omitted and the process may proceed to S1409.

(Fourth Embodiment)
In the fourth embodiment, an example of determining a representative normal vector based on the distribution of normal information will be described. Further, an example of determining the white balance adjustment parameter for each pixel in the input image data according to the region to which the pixel belongs will be described. The hardware configuration and functional block configuration of the image processing device are the same as those of the above-described embodiments (FIGS. 1 and 2). Hereinafter, the points different from the above-described embodiments in the operation of each functional unit will be mainly described.

FIG. 16 is a flowchart of a series of processes according to the fourth embodiment. Since S301, S1003, and S309 to 310 are the same as those in the above-described embodiment, the description thereof will be omitted.

In step S1602, the parameter setting unit 202 analyzes the distribution of the normal vector included in the normal map acquired in S301, and calculates the representative normal vector N _sk (k = 1, 2, ..., K). do. Specifically, the normal vector in the normal map is classified into K clusters by using a known clustering method, and the vector corresponding to the center of each cluster is used as the representative normal vector. At this time, the number K of the clusters may be a predetermined number, or may be a larger number as the variance of the normal vector included in the normal map is larger.

In step S1604, the parameter setting unit 202 determines the color temperature as the white balance adjustment parameter for each region divided by S1003. Since each region is a set of pixels having similar normal vectors, the color temperature T _sk corresponding to the region R _k to which the region belongs is determined as the color temperature for the pixel for each pixel in the input image data. Similar to S1004, a known auto white balance adjustment method can be used to determine the color temperature for the region.

As described above, according to the fourth embodiment, it is possible to set a representative normal vector that represents the orientation of the surface of the subject without excess or deficiency according to the distribution of the normal vector of the subject. Further, the white balance adjustment parameter for each pixel can be determined by a simple process.

(Fifth Embodiment)
In the fifth embodiment, an example in which 3D model data representing the three-dimensional shape of the main subject is used as normal information will be described. Further, an example in which the white balance of the entire image is first adjusted and then the white balance is partially readjusted according to the input from the user will be described. The hardware configuration and functional block configuration of the image processing device are the same as those of the above-described embodiments (FIGS. 1 and 2).

The image data acquisition unit 201 acquires the input image data and the 3D model data of the main subject. Here, polygon model data (hereinafter referred to as standard face model data) showing a standard rough shape of a face is used as 3D model data with respect to input image data showing a person's face. FIG. 17 is a diagram showing an example of a standard face model. The acquired input image data and standard face model data are sent to the parameter setting unit 202.

The parameter setting unit 202 determines the adjustment parameter used for the initial adjustment based on the input image data. Further, a UI for setting the representative points is displayed on the display 110, and the adjustment parameters used for readjustment are determined based on the values instructed by the user on the UI and the standard face model data. The details of the process will be described later. The determined adjustment parameter is sent to the color processing unit 203.

The color processing unit 203 adjusts the white balance of various image data using the set adjustment parameters. The adjusted image data is sent to the display 111 and displayed.

FIG. 18 is a flowchart of a series of processes according to the fifth embodiment.

In step S1801, the image data acquisition unit 201 acquires the input image data from the HDD 104 or the image pickup apparatus. Also, the standard face model data is acquired from the HDD 104. FIG. 19A is a diagram showing an example of input image data. In the example of this figure, the white balance of the image is not matched as a whole, and moreover, a part of the person is strongly exposed to light having a color temperature different from that of the other parts.

In step S1802, the parameter setting unit 202 determines one set of gain coefficients (Gra _a , G g _{a, G ba a )} as adjustment parameters used for the initial adjustment. For example, a known auto white balance technique can be used for the entire input image data.

In step S1803, the color processing unit 203 adjusts the white balance of the entire input image data by using the initial adjustment parameter set in S1802. Specifically, the RGB values of all the pixels in the input image data are multiplied by the gain coefficients (Gra _a , G g a, G _{b a} ), respectively. FIG. 19B is a diagram showing an example of an image obtained by performing initial adjustment on the input image of FIG. 19A. That is, FIG. _19B is an example of the result of determining the adjustment parameter according to the color temperature La of the light illuminating most of the subject and uniformly adjusting the white balance of all the pixels using the adjustment parameter. be. In this example, in the part where light with _a color temperature L _p different from La is strongly exposed, the color temperature assumed when determining the adjustment parameter does not match the color temperature of the light actually exposed, so that the color is fogged. Has occurred.

In step S1804, the color processing unit 203 outputs the image data after the initial adjustment performed in S1803 to the display 110 and displays it.

In step S1805, the parameter setting unit 202 outputs the UI for setting the representative point to the display 110 and displays it. Then, when the input regarding the setting of the representative point is performed via the UI, the process proceeds to step S1806, and if there is no input, the series of processes is terminated. FIG. 20 is a diagram showing an example of a UI for setting a representative point in the present embodiment. The image display area 2001 is an area in which image data after white balance adjustment is displayed. The user can set a representative point via the UI if he / she wants to make further adjustments while referring to the displayed image data.

In step S1806, the parameter setting unit 202 acquires the value instructed by the user on the UI displayed in S1805, and sets the coordinates of the representative point S and the adjustment parameter. In the example of FIG. 20, by clicking a point in the image display area 2001 with the mouse, the coordinate values (i, j) on the image corresponding to that point are set as the coordinates of the representative point S. When the slider 2002 is operated by the user, the gain coefficients (Gr _p , Gg _p , Gb _p ) corresponding to the position of the slider are set as adjustment parameters for the representative points. The relationship between the slider position and the gain coefficient is, for example, a gain coefficient corresponding to 2000 [K] when the slider is at the left end and 12000 [K] when the slider is at the right end. The corresponding gain factor is determined by interpolation based on the ratio of distances.

In step S1807, the parameter setting unit 202 determines the readjustment region (region determination) based on the coordinates of the representative point S set in S1806. FIG. 21 is a detailed flowchart of the readjustment area setting process (S1807).

In step S2101, the parameter setting unit 202 extracts a region (face region) in which the face of the subject appears on the input image by using a known face region extraction technique, and the representative point S set in S1806 in that region. Is included or not. If the representative point S is included in the face area, the process proceeds to S2102, and if the representative point S is not included, the process proceeds to S2106.

In step S2102, the parameter setting unit 202 performs a transformation of associating the position on the input image plane with the position on the polygon constituting the standard face model with respect to the pixels included in the face region extracted in S2101. It is calculated based on the position of the face and the size of the face area. A known 3D-2D alignment technique can be used for the mapping.

In step S2103, the parameter setting unit 202 obtains the position Sm on the standard face model corresponding to the coordinates of the representative point S set in _S1806 by using the conversion obtained in S2102.

In step S2104, the parameter setting unit 202 extracts from the polygons constituting the standard face model those having a surface inclination similar to that of the polygon at the position Sm obtained in _S2103 . Here, the inner product of the unit normal vectors of each polygon is used as the similarity of the inclination between the two polygons, and the polygon whose similarity with the polygon at the position _Sm is larger than the predetermined threshold value is extracted.

In step S2105, the parameter setting unit 202 obtains an area occupied by the polygon extracted in S2104 on the input image plane, and uses this as the readjustment area. Specifically, the pixel position corresponding to the apex of the polygon is obtained by using the inverse transformation of the transformation obtained in S2102, and the inside of the figure formed by connecting these is defined as the readjustment region _Rp .

In step S2106, the parameter setting unit 202 sets the region excluding the face region extracted in S2101 from the entire input image as the readjustment region _Rp .

In step S1808, the color processing unit 203 adjusts the white balance of the readjustment region _Rp by using the adjustment parameter of the representative point S set in S1806. Specifically, the RGB values of the pixels included in the readjustment region R _p in the input image data are multiplied by the gain coefficients (Gr _p , Gg _p , Gb _p ), respectively. It should be noted that the similarity calculated when extracting polygons having similar slopes in S2104 is stored, and when the similarity is small, the gain coefficient to be multiplied is brought close to 1 (that is, the change in the pixel value due to readjustment). It may be adjusted so that the amount becomes smaller). FIG. 19 (c) is a diagram showing an example of an image obtained by setting a representative point S at a position marked with a cross with respect to the image after initial adjustment shown in FIG. 19 (b) and performing readjustment. In this example, the representative point S is set at the portion where the color cast occurs in FIG. 19B. Further, based on the position of the representative point S, the area surrounded by the dotted line in FIG. 19C is set as the readjustment area _Rp . By adjusting the white balance in the readjustment region R _p using the adjustment parameter of the representative point S, the color cast that has occurred after the initial adjustment is reduced.

In step S1809, the color processing unit 203 outputs and displays the image data after the readjustment performed in S1808 to the display 110, and returns to S1805.

The face region extracted in S2101 and the conversion obtained in S2102 do not need to be obtained every time the readjustment region setting process (S1807) is repeated, and can be diverted from the second time onward.

As described above, according to the fifth embodiment, it is possible to adjust the white balance based on the inclination of the surface of the main subject.

(Sixth Embodiment)
In the sixth embodiment, an example in which the user sets the representative normal vector via the UI will be described. Further, an example in which normal information cannot be obtained for a part of the input image will be described. Here, a normal map is used as normal information. However, when the normal vector of the subject is unknown in the pixel (i, j), the zero vector (0,0,0) is stored as the pixel value N (i, j) of the normal map. And. The hardware configuration and functional block configuration of the image processing device are the same as those of the above-described embodiments (FIGS. 1 and 2).

The parameter setting unit 202 displays a UI for making settings related to white balance adjustment on the display 110 via the output I / F 107. In addition, the value instructed by the user on this UI is acquired via the input I / F 106, and the adjustment parameter used when the normal vector is unknown, the representative normal vector, and the corresponding adjustment parameter are set. do. Then, based on these settings, the adjustment parameters for each pixel in the input image are determined using the normal map.

FIG. 22 is a flowchart of a series of processes according to the sixth embodiment. Since S301 and S1802 to 1804 are the same as those in the above-described embodiment, the description thereof will be omitted.

In step S2202, the parameter setting unit 202 sets the color temperature corresponding to the gain coefficient used for the initial adjustment as the color temperature (hereinafter referred to as the base color temperature) used when the normal vector is unknown. In addition, the UI for making settings related to white balance adjustment is output to the display 110 and displayed. FIG. 23A is a diagram showing an example of the initial state of the UI in this embodiment. The input image data after the initial adjustment is displayed in the image display area 2301. Further, the area affected when the color temperature setting is changed is highlighted by being surrounded by a dotted line 2302. The color of the base parameter display area 2303 represents the base color temperature. The icon 2304 indicates that the base color temperature is selected as the target whose setting is changed by the operation of the slider 2305. The spherical selector 2306 schematically represents the correspondence between the direction of the normal vector and the color temperature used for adjustment. Each point on the spherical selector 2306 is associated with the normal vector of the sphere at that point, and the color of each point is used to adjust the pixel with the normal vector associated with that point. Represents the color temperature. Here, since the same color temperature (= base color temperature) is used for the entire image in the initial adjustment, the entire spherical selector 2306 is drawn with the hue corresponding to the base color temperature. In the example of the input image in this figure, in addition to the ambient light that illuminates the entire scene, the subject is exposed to light having different color temperatures from three directions. Therefore, different color casts occur in each part.

In step S2203, the parameter setting unit 202 acquires the input from the user via the UI displayed in S2202. Then, if the input related to the setting change is made, the process proceeds to S2204, and if not, the process proceeds to S2207. A specific example will be described below.

<Change of base color temperature>
When the slider 2305 is operated by the user in the state of FIG. 23A, the base color temperature is changed to the color temperature corresponding to the position of the slider, and the process proceeds to S2204. At this time, the base parameter display area 2303 is redrawn with a color corresponding to the changed color temperature.

<New setting of representative normal vector>
When one point 2307 on the spherical selector 2306 is designated by the user in the state of FIG. 23A, the parameter setting unit 202 sets the normal vector corresponding to that point as the representative normal vector N _s1 . Further, the same color temperature as the base color temperature is set as the color temperature T _s1 corresponding to the representative normal vector N _s1 . Further, the similarity between the normal vector and the representative normal vector N _s1 in each pixel is calculated based on the normal map acquired in S301, and the pixel region R _s1 having a high degree of similarity is extracted. Then, the UI is updated to the state shown in FIG. 23 (b).

In FIG. 23B, the endpoint of the representative parameter display area 2308 points to a point 2307 designated by the user. Further, the color of the representative parameter display area 2308 represents the color temperature T _s1 associated with the representative normal vector N _s1 . The icon 2304 indicates that the color temperature T _s1 corresponding to the representative normal vector N _s1 is selected as the target whose setting is changed by the operation of the slider 2305. Icon 2309 indicates that the base color temperature is not selected as the target for which the setting is changed. The dotted line 2302 highlights the region R _s1 having a normal vector similar to the representative normal vector N _s1 . This region is a region affected when the color temperature T _s1 corresponding to the representative normal vector N _s1 is changed. When the slider 2305 is operated by the user in this state, the color temperature T _s1 corresponding to the representative normal vector N _s1 is changed to the color temperature corresponding to the position of the slider, and the process proceeds to S2204. At this time, the UI transitions to the state shown in FIG. 23 (c).

In FIG. 23 (c), the color of the representative parameter display area 2308 is changed to the color representing the changed color temperature T _s1 . Further, on the spherical selector 2306, the hue around the point 2307 (that is, the region where the normal vector is similar to the representative normal vector N _s1 ) changes to the hue corresponding to the changed color temperature T _s1 . There is.

<Completion of setting>
When the setting completion button 2300 is pressed by the user in the state of FIG. 23A, the process proceeds to S2207.

In step S2204, the parameter setting unit 202 sets adjustment parameters for each pixel of the input image data by using the normal map acquired in S2201 and the representative normal vector and the color temperature reflecting the changes in S2203. In the present embodiment, the gain coefficients Gr _p , Gg _p , and Gb _p for the pixels P (i _p , j _p ) in the input image data are calculated according to the following mathematical formula (12).

Here, Gr _B , Gg _B , and Gb _B are gain coefficients corresponding to the base color temperature. Gr _k , Gg _k , and Gb _k are gain coefficients corresponding to the color temperature T _{sk associated} with the representative normal vector N _sk . α _sk is the degree of similarity between the representative normal vector N _sk and the normal vector N _p of the pixel P. Thα is a predetermined threshold value for similarity. The degree of similarity α _sk (= 0 to 1) is maximum when the representative normal vector N _sk and the normal vector N _p of the pixel P match, and the angle formed by N _sk and N _p is a constant value (cos). ^-1 (Thα)) or more, it becomes zero. α _sMAX is the maximum value of similarity α _sk (k = 1 to K). In the equation (12), when the normal vector N _p in the pixel P is unknown, the gain coefficient with respect to the pixel P has the same value as the gain coefficient corresponding to the base color temperature. In other cases, the value is determined by interpolation based on the similarity of the normal vectors.

In step S2205, the color processing unit 203 adjusts the white balance of the input image data. Specifically, the gain coefficients Gr _p , Gg _p , and Gb _p for each pixel set in S2204 are multiplied by the pixel value of the input image data.

In step S2206, the color processing unit 203 outputs the adjusted image data performed in S2205 to the display 110, and updates the display of the image display area 2301. At this time, the UI transitions to the state shown in FIG. 23 (d). In FIG. 23D, the adjusted image data of S2205 is displayed in the image display area 2301, and the color of the area surrounded by the dotted line 2302 is changed by the adjustment. (By adjusting using the color temperature T _s1 corresponding to the representative normal vector N _s1 , the color fog generated in the region R _s1 is reduced.)

When the display of the image data is updated in S2206, the process returns to S2203, and the parameter setting unit 202 again acquires the input from the user. While referring to the updated image data, the user can add a representative normal vector or change the setting via the UI if he / she wants to make further adjustments. FIG. 23 (e) is an example in which the point 2310 and the corresponding color temperature are input by the user from the state of FIG. 23 (d), and the processing of S2204 to 2206 is performed according to the input, and the portion surrounded by the dotted line is shown. Color fog is reduced. Further, FIG. 23 (f) is an example in which a representative normal vector is further added from the state of FIG. 23 (e). By pressing the icon 2309, the base color temperature once set or the color temperature with respect to the representative normal vector may be changed again. Further, by pressing the × button attached to the representative parameter display area 2308, the setting regarding the corresponding representative normal vector may be deleted.

In step S2207, the parameter setting unit 202 determines whether or not it is necessary to confirm the setting regarding the white balance adjustment based on the normal map acquired in S301 and the currently set representative normal vector. In the present embodiment, when the orientation of the subject surface of a pixel does not resemble the orientation of any representative normal vector, the same adjustment parameter as that of a pixel whose normal information is unknown is used for that pixel. However, in such a pixel, a more preferable adjustment result may be obtained by finely adjusting the white balance by adding a representative normal vector. Therefore, the similarity α _sk of the normal vector is calculated according to the equation (13) for each pixel in the input image data in which N (i, j) ≠ (0,0,0), and the maximum similarity is obtained for each pixel. Find the value α _sMAX . Then, when the number of pixels in which α _sMAX = 0 is equal to or greater than a predetermined ratio with respect to the number of pixels in the entire image, it is determined that the setting needs to be confirmed. If it is determined that confirmation is necessary, the process proceeds to S2208, and if not, the process proceeds to S2210.

In step S2208, the parameter setting unit 202 notifies the user of a candidate for a representative normal vector suitable for adjustment. Specifically, the median value of the normal vector is calculated for the pixel having α _sMAX = 0 obtained in S2207, and this is displayed on the UI as a candidate for the representative normal vector. FIG. 23 (g) shows an example of the case where the setting completion button 2300 is pressed in the state of FIG. 23 (d) in S2203. In FIG. 23 (g), the end point of the representative parameter display area 2308 points to a point corresponding to the candidate of the representative normal vector on the spherical selector 2306. Icon 2311 indicates that the normal vector pointed to by the representative parameter display area 2308 is a candidate for the representative normal vector. The dotted line 2302 highlights a region having a normal vector similar to the candidate normal normal vector. This region is a region affected when a candidate of the representative normal vector is set as the representative normal vector and the corresponding color temperature is changed.

In step S2209, the parameter setting unit 202 acquires the input from the user via the UI as in S2203. Then, if the input related to the setting change is made, the process returns to S2204, and if not, the process proceeds to S2210. A specific example will be described below.

<Addition of representative normal vector>
When the slider 2305 is operated by the user in the state of FIG. 23 (g), a candidate for the representative normal vector (that is, the normal vector pointed to by the representative parameter display area 2308 with the icon 2311) is added to the representative normal vector. To. Further, the color temperature corresponding to the position of the slider is set as the color temperature corresponding to the representative normal vector, and the process returns to S2204.

<Rejection of candidates>
When the setting completion button 2300 is pressed in the state of FIG. 23 (g), the candidate of the representative normal vector proceeds to S2210 without being added to the representative normal vector.

In step S2210, the color processing unit 203 stores the image data after the white balance adjustment in a storage device such as the HDD 104. Then, the UI is updated to the state shown in FIG. 23 (h) to notify that the saving is completed.

According to the sixth embodiment described above, the user can set the representative normal vector by an intuitive operation via the UI.

In step S2210, the base color temperature used for the adjustment, the representative normal vector and the corresponding color temperature may be separately stored as the adjustment parameter information. In that case, if the adjustment parameter information saved together with the input image data and the normal information in S301 is acquired, the adjustment once performed can be easily applied to different input image data.

(7th Embodiment)
In the seventh embodiment, an example in which normal information is calculated based on images obtained by capturing images of a subject from a plurality of different viewpoint positions and white balance adjustment is performed using the normal information will be described. The hardware configuration and functional block configuration of the image processing device are the same as those of the above-described embodiments (FIGS. 1 and 2).

The image data acquisition unit 201 acquires the input image data and the sub-input image data (auxiliary input image) obtained by capturing the subject in the input image data from different viewpoint positions. In addition, the camera parameters (information indicating the position and orientation of the camera and the distortion of the lens) when these image data are captured are acquired. Hereinafter, these image data and camera parameters are collectively referred to as multi-viewpoint image data. In the present embodiment, of the two images obtained by capturing with a stereo camera, one of the image data is used as the input image data and the other is used as the sub-input image data.

The parameter setting unit 202 calculates a representative normal vector using the multi-viewpoint image data. The details of the process will be described later.

FIG. 24 is a flowchart of a series of processes according to the seventh embodiment. Since S302 to 303, S305, and S307 to 310 are the same as those in the first embodiment, the description thereof will be omitted.

In step S2401, the image data acquisition unit 201 acquires multi-viewpoint image data from the HDD 104 or the image pickup device.

In step S2404, the parameter setting unit 202 calculates the representative normal vector by obtaining the orientation of the surface including the representative point set in S303 using the multi-viewpoint image data acquired in S2401. First, feature points are extracted from the input image data and the sub-input image data, and the feature points are associated between the images. A known feature-based matching technique can be used for this process. Next, from the associated feature points, three points forming the smallest triangle including the representative points on the input image are selected. Next, for each of the three selected feature points, the three-dimensional coordinates are estimated based on the position on each image and the camera parameters. Then, the unit normal vector of the surface of the triangle formed by the three points is calculated using the obtained three-dimensional coordinates, and this is used as the representative normal vector.

In step S2406, the parameter setting unit 202 calculates the normal vector corresponding to the pixel of interest selected in S305 by using the multi-viewpoint image data acquired in S2401. Specifically, the representative point in S2404 is replaced with the pixel of interest, and the same processing is performed. It is not necessary to associate the feature points again, and the result obtained in S2404 can be diverted. Also, for the estimation of three-dimensional coordinates and the calculation of the unit normal vector, if the feature points used are the same, the result calculated once can be diverted.

According to the seventh embodiment described above, normal information can be calculated from the multi-viewpoint image data, and white balance adjustment can be performed based on the normal information.

The sub-input image data may be an image captured by a camera different from the camera that captured the input image data, or may be two or more images. In addition, a subject is imaged using a Plenoptic camera in which a microlens array is arranged between the main lens and the image pickup element, and one of the obtained plurality of image data can be used as input image data and the other can be used as sub-input image data. good.

(8th Embodiment)
In the eighth embodiment, an example in which normal information is calculated based on the distance information and white balance adjustment is performed using the normal information will be described. The hardware configuration and functional block configuration of the image processing device are the same as those of the above-described embodiments (FIGS. 1 and 2).

The image data acquisition unit 201 acquires the input image data and the angle of view and focal length of the image pickup device when the input image data is captured. In addition, the distance information of the subject is acquired. Here, a distance map is used as the distance information.

The parameter setting unit 202 calculates various normal vectors based on the distance map. The details of the process will be described later.

FIG. 25 is a flowchart of a series of processes according to the eighth embodiment. Since S302 to 303, S305, and S307 to 310 are the same as those in the first embodiment, the description thereof will be omitted.

In step S2501, the image data acquisition unit 201 acquires the input image data, the angle of view and focal length at the time of shooting, and the distance map from the HDD 104 or the image pickup device.

In step S2504, the parameter setting unit 202 calculates the representative normal vector by obtaining the orientation of the surface including the representative point set in S303 based on the distance map acquired in S2501. First, the three-dimensional coordinates of the points on the surface of the subject corresponding to each pixel of the input image are calculated by using the angle of view and the focal length at the time of shooting and the distance map.

Next, the input image is divided into a plurality of regions, and a plane is fitted to each region with respect to the three-dimensional point group data corresponding to the pixels. The input image may be divided into blocks according to a predetermined number of divisions, or may be divided using a general area division method based on pixel values. For the fitting of the plane, for example, a plane having the minimum sum of squared distances with the point cloud corresponding to the pixels included in the region may be obtained. Then, the unit normal vector of the plane fitted to the region including the representative point is defined as the representative normal vector.

In step S2506, the parameter setting unit 202 replaces the representative point in S2504 with the pixel of interest selected in S305, performs the same processing, and calculates the normal vector corresponding to the pixel of interest. It is not necessary to perform the fitting of the plane again, and the result obtained in S2504 can be diverted.

According to the eighth embodiment described above, the normal information can be calculated from the distance map, and the white balance can be adjusted based on the normal information.

In the present embodiment, the surface constituting the subject is obtained by fitting a plane to the three-dimensional point group data calculated based on the distance map, but another shape model that generates a three-dimensional shape from the distance map is obtained. The surface of the subject may be obtained by using a generation technique.

Further, a normal map may be generated from the distance map, and the first to fourth and sixth embodiments may be applied. As for the method of generating a normal map from a distance map, various methods are generally known and are not the main subject of the present invention, and therefore detailed description thereof will be omitted.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 image processing device; 104 HDD; 111 display; 201 image data acquisition unit; 202 parameter setting unit; 203 color processing unit

Claims (12)

The first acquisition means to acquire the input image and
A determination means for determining a representative point in the input image and a white balance coefficient of the representative point, and
A second acquisition means for acquiring the first information indicating the inclination of the surface including the representative point and the second information indicating the inclination of the surface including the pixel of interest in the input image.
An adjusting means for adjusting the white balance in the pixel of interest based on the first information, the second information, and the white balance coefficient.
Have,
The first acquisition means further acquires normal information about the subject in each pixel included in the input image.
The adjusting means calculates the degree of similarity between the normal of each pixel included in the input image and the normal at the representative point based on the normal information, and determines the white balance for each pixel included in the input image. , Derived by an interpolation calculation based on the white balance coefficient with the similarity as a weighting coefficient.
An image processing device characterized by this. The first information is the normal information of the subject at the representative point.
The image processing apparatus according to claim 1, wherein the second information is normal information of a subject in the pixel of interest. The image processing apparatus according to claim 1 or 2, wherein the determination means has a reception means for receiving information for determining the representative point from the user. The reception means receives the designation of the normal vector from the user and receives it.
The image processing apparatus according to claim 3, wherein the determination means determines a pixel corresponding to the normal vector among the pixels included in the input image as the representative point. The determination means determines a plurality of representative points in the input image and a white balance coefficient corresponding to the plurality of representative points.
The first acquisition means further acquires feature amount information other than the normal information regarding the subject in each pixel included in the input image.
The adjusting means multiplies the white balance coefficient by a weighting coefficient based on the normal line information and the feature amount information when there are a plurality of representative points having substantially the same normal lines and different white balance coefficients. The image processing apparatus according to claim 1 , wherein the white balance for each pixel included in the input image is adjusted. The adjusting means calculates a first similarity between the normal line of each pixel included in the input image and the normal line of the representative point based on the normal line information, and the input is based on the feature amount information. A second similarity between the feature amount of each pixel included in the image and the feature amount of the representative point is calculated, and the white balance coefficient is weighted based on the first similarity degree and the second similarity degree. The image processing apparatus according to claim 5 , wherein the white balance for each pixel included in the input image is adjusted by multiplying by a coefficient. The first acquisition means further acquires an auxiliary input image obtained by capturing a subject included in the input image from a viewpoint different from the input image.
The second acquisition means according to any one of claims 1 to 6 , wherein the second acquisition means determines the first information and the second information based on the input image and the auxiliary input image. Image processing equipment. The first acquisition means further acquires the distance information of the subject in each pixel included in the input image.
The image processing apparatus according to any one of claims 1 to 6 , wherein the second acquisition means determines the first information and the second information based on the distance information. The first acquisition means to acquire the input image and
A determination means for determining a representative point in the input image and a white balance coefficient of the representative point, and
An area determining means for determining an adjustment area for adjusting white balance in the input image,
A second acquisition means for acquiring the first information indicating the inclination of the surface including the representative point and the second information indicating the inclination of the surface including the pixel of interest in the input image.
An adjustment means for adjusting the white balance in the pixel of interest in the adjustment region based on the first information, the second information, and the white balance coefficient.
Have,
The first acquisition means further acquires a 3D model of the subject included in the input image, and further acquires the 3D model.
The area determining means determines as the adjustment area a region of pixels determined to have a normal that substantially coincides with the normal of the representative point based on the 3D model.
An image processing device characterized by this. It is a control method for image processing equipment.
The first acquisition process to acquire the input image and
A determination step for determining a representative point in the input image and a white balance coefficient of the representative point.
A second acquisition step of acquiring first information indicating the inclination of the surface including the representative point and second information indicating the inclination of the surface including the pixel of interest in the input image.
An adjustment step for adjusting the white balance in the pixel of interest based on the first information, the second information, and the white balance coefficient.
Including
In the first acquisition step, normal information regarding the subject in each pixel included in the input image is further acquired.
In the adjustment step, the degree of similarity between the normal of each pixel included in the input image and the normal at the representative point is calculated based on the normal information, and the white balance for each pixel included in the input image is determined. , Derived by an interpolation calculation based on the white balance coefficient with the similarity as a weighting coefficient.
A control method characterized by that. It is a control method for image processing equipment.
The first acquisition process to acquire the input image and
A determination step for determining a representative point in the input image and a white balance coefficient of the representative point.
An area determination step of determining an adjustment area for adjusting white balance in the input image, and
A second acquisition step of acquiring first information indicating the inclination of the surface including the representative point and second information indicating the inclination of the surface including the pixel of interest in the input image.
An adjustment step of adjusting the white balance of the pixel of interest in the adjustment region based on the first information, the second information, and the white balance coefficient.
Including
In the first acquisition step, a 3D model of the subject included in the input image is further acquired.
In the area determination step, the area of the pixel determined to have a normal line substantially matching the normal line of the representative point based on the 3D model is determined as the adjustment area.
A control method characterized by that. A program for making a computer function as each means of the image processing apparatus according to any one of claims 1 to 9 .

Images