JP6937603B2 - Image processing equipment and its control methods, programs, and storage media - Google Patents

Description

The present invention relates to an image processing technique for correcting the brightness of a subject in an image.

Conventionally, there is known a technique of performing relighting by irradiating a subject in a captured image with light from a virtual light source to correct shadows. The rewriting process makes it possible to obtain an image in which the shadow of the subject caused by the environmental light source is brightened. In Patent Document 1, a pseudo rewriting process is performed on a captured image, specifically, a brightness region lower than the average brightness of the entire face region is extracted as a shadow region, and the brightness of the extracted shadow region is increased. Is described.

Japanese Unexamined Patent Publication No. 2010-135996

When the shadow of the subject in the image is changed by the rewriting process, it is necessary to appropriately determine the subject area to be rewritten. If the subject area is not appropriate, the relighted image may look unnatural.

In Patent Document 1, the rewriting processing area is determined based on the position and size of the face area in the image, but if the face area is erroneously detected, the rewriting processing is executed on the inappropriate area. there is a possibility. For example, in the face detection process, if an area similar to a face included in the background around the face area is erroneously detected as a face, the rewriting process is executed for a part of the background that is not the face. there is a possibility.

The present invention has been made in view of the above problems, and an object of the present invention is to realize a technique capable of appropriately performing processing based on the reliability of a region for performing processing for correcting shadows and the like of a subject.

In order to solve the above problems and achieve the object, the image processing apparatus of the present invention sets a virtual light source which is a virtual light source, and corrects the brightness of the subject in the image by using the virtual light source. and processing means for processing, acquisition means for acquiring a specific partial area information of the subject in the image, and determining means for determining a subject to the subject area of the correction process in the image obtained by the acquisition unit and and a control means for controlling the correction processing by said processing means in response to the correction processing of the reliability based on a relationship between the determined said object regions by a particular partial area information and said determining means of the subject.

According to the present invention, the processing can be appropriately performed based on the reliability of the region for which the processing for correcting the shadow of the subject or the like is performed.

The block diagram which shows the structure of the image pickup apparatus of this embodiment. The block diagram which shows the structure of the image processing part of this embodiment. The block diagram which shows the structure of the rewriting processing part of this embodiment. The figure explaining the shooting coordinate of this embodiment and the positional relationship between a subject and a virtual light source. The figure explaining the face area information and the subject area information of this embodiment. The figure explaining the image before and after the rewriting process of this embodiment. The flowchart which shows the rewriting process of this embodiment. The figure explaining the rewriting reliability calculation method of this embodiment. The flowchart which shows the rewriting process of a plurality of persons of Embodiment 2.

Hereinafter, embodiments for carrying out the present invention will be described in detail. The embodiments described below are examples for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the apparatus to which the present invention is applied and various conditions. It is not limited to the embodiment of. In addition, a part of each embodiment described later may be appropriately combined and configured.

[Embodiment 1] Hereinafter, an example in which the image processing device of the present embodiment is applied to an image pickup device such as a digital camera capable of capturing a still image or a moving image will be described. In the present embodiment, a digital camera is illustrated as the image pickup device 100, but it is an information processing device such as a camera-equipped mobile phone, a smartphone, a tablet terminal, or a personal computer (PC) with a camera. May be good.

<Device Configuration> First, the configuration and functions of the imaging device of the present embodiment will be described with reference to FIG.

The image pickup apparatus 100 of the present embodiment has a function of performing a relighting process of correcting (changing) shadows by irradiating a subject in an image taken in a certain light source environment with the light of a virtual light source. Has.

In FIG. 1, the photographing lens 101 is a lens group including a zoom lens and a focus lens. The shutter 102 has an aperture function. The image pickup unit 103 is an image pickup device composed of a CCD, a CMOS element, or the like that converts an optical image into an electric signal. The A / D converter 104 converts the analog signal output from the imaging unit 103 into a digital signal. The image processing unit 105 performs various image processing such as white balance (WB) processing, gamma processing, contour enhancement, and color correction processing on the image data output from the A / D converter 104. The memory control unit 107 controls the image memory 106. The image memory 106 stores the image data output from the A / D converter 104 and the image data to be displayed on the display unit 109. The image memory 106 has a storage capacity sufficient to store a predetermined number of still images, moving images for a predetermined time, and audio. Further, the image memory 106 also serves as a memory (video memory) for displaying an image. The D / A converter 108 converts the image display data stored in the image memory 106 into an analog signal and supplies it to the display unit 109. The display unit 109 is a display device such as an LCD or an organic EL.

The codec unit 110 transfers the image data written in the image memory 106 to MPEG or H.A. A video file is generated by compression coding at a predetermined bit rate and format such as 264, and recorded on the recording medium 112. Further, the codec unit 110 decodes the video file recorded on the recording medium 112 at a predetermined bit rate and format, and stores the video file in the image memory 106.

The recording medium I / F 111 is an interface that controls access to the recording medium 112. The recording medium 112 is a built-in and / or detachable memory card, HDD (hard disk drive), or the like for recording captured image data.

The face detection unit 113 detects the face area information of a person as specific subject information in the captured image. The rewriting processing unit 114 performs rewriting processing on the captured image.

The operation unit 115 is an operation member such as various switches, buttons, and a touch panel that receives various operations from the user, and includes a power switch, a shutter button, a recording start / end button, and the like. The operation unit 115 notifies the system control unit 120 of various operation states.

The system control unit 120 realizes each process of the flowchart described later by executing the program stored in the non-volatile memory 116. 117 is a system memory, and RAM is used. In the system memory 117, constants and variables for the operation of the system control unit 120, a program read from the non-volatile memory 116, and the like are expanded. The system control unit 120 also controls the display by controlling the image memory 106, the D / A converter 108, the display unit 109, and the like.

The distance detection unit 118 measures the distance to the subject, and the distance calculation unit 303, which will be described later, calculates the distance information corresponding to each pixel of the photographing pixel as a two-dimensional distance map.

<Shooting operation> Next, a shooting operation by the imaging device 100 of the present embodiment will be described.

The imaging unit 103 photoelectrically converts the light incident through the photographing lens 101 and the shutter 102, and outputs the light as an analog image signal to the A / D converter 104. The A / D converter 104 converts the analog image signal output from the imaging unit 103 into a digital signal and outputs it to the image processing unit 105.

The image processing unit 105 performs color conversion processing, gamma processing, contour enhancement processing, and the like on the image data from the A / D converter 104 or the image data from the memory control unit 107. Further, the image processing unit 105 performs a predetermined evaluation value calculation process (not shown) using the face area information detected by the face detection unit 113 and the captured image data, and the system is based on the obtained evaluation value. The control unit 120 performs exposure control and distance measurement control. As a result, TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, AWB (auto white balance) processing, and the like are performed.

The image data output from the image processing unit 105 is written to the image memory 106 via the memory control unit 107. The image memory 106 stores the image data output from the imaging unit 103 and the image data to be displayed on the display unit 109.

Further, the D / A converter 108 converts the display image data stored in the image memory 106 into an analog signal and supplies it to the display unit 109. The display unit 109 displays on a display such as an LCD according to the analog signal from the D / A converter 108.

The codec unit 110 compresses and encodes the image data stored in the image memory 106 based on a standard such as MPEG. The system control unit 120 associates the encoded image data and stores it in the recording medium 112 via the recording medium I / F 111.

<Image Processing Unit> Next, the configuration and function of the image processing unit 105 of the present embodiment will be described with reference to FIG.

In FIG. 2, the image processing unit 105 includes a simultaneous processing unit 200, a WB amplification unit 201, a brightness / color signal generation unit 202, a contour enhancement processing unit 203, a brightness gamma processing unit 204, a color conversion processing unit 205, and a color gamma processing. A unit 206, a color difference signal generation unit 207, and a shadow information acquisition unit 208 are included.

Next, the processing in the image processing unit 105 will be described.

The image signal input from the A / D converter 104 of FIG. 1 to the image processing unit 105 is input to the simultaneous processing unit 200. The simultaneous processing unit 200 performs simultaneous processing on the input Bayer RGB image data to generate color signals R, G, and B. The WB amplification unit 201 applies a gain to the RGB color signal based on the white balance gain value calculated by the system control unit 120, and adjusts the white balance. The RGB signal output from the WB amplification unit 201 is input to the luminance / color signal generation unit 202. The luminance / color signal generation unit 202 generates a luminance signal Y from the RGB signal, outputs the generated luminance signal Y to the contour enhancement processing unit 203, and outputs the color signal RGB to the color conversion processing unit 205.

The contour enhancement processing unit 203 performs contour enhancement processing on the luminance signal Y and outputs it to the luminance gamma processing unit 204. The luminance gamma processing unit 204 performs gamma correction on the luminance signal Y and outputs it to the image memory 106.

The color conversion processing unit 205 converts the RGB signal into a desired color balance by performing a matrix calculation or the like. The color gamma processing unit 206 performs gamma correction on the RGB color signal. The color difference signal generation unit 207 generates color difference signals RY and BY signals from RGB signals and outputs them to the image memory 106. The luminance / color difference signals (Y, RY, BY) stored in the image memory 106 are compressed and coded by the codec unit 110 and recorded on the recording medium 112.

Further, the RGB signal processed by the color conversion processing unit 205 is output to the shadow information acquisition unit 208. The shadow information acquisition unit 208 acquires information for analyzing the state of the shadow generated on the subject due to the light source environment at the time of shooting. For example, the average brightness information of the subject, the brightness histogram information of the face area, and the like are acquired as shadow information.

<Rewriting Processing Unit> Next, the configuration and function of the rewriting processing unit 114 of the present embodiment will be described with reference to FIG.

When the rewriting process is selected by the user operation, the image data is output from the image processing unit 105 to the rewriting processing unit 114, and the rewriting process is performed by the virtual light source.

In FIG. 3, the RGB signal conversion unit 301 converts the input luminance / color difference signals (Y, BY, RY) into RGB signals. The degamma processing unit 302 performs degamma processing on the RGB signal output from the RGB signal conversion unit 301. The distance calculation unit 303 acquires the distance information between the image pickup device and the subject output from the distance detection unit 118. The normal calculation unit 304 calculates the normal information from the distance information of the subject calculated by the distance calculation unit 303. The virtual light source reflection component calculation unit 305 sets a virtual light source, and calculates a component in which the light emitted from the set virtual light source is reflected on the subject. The virtual light source addition processing unit 306 adds a component obtained by reflecting the light of the virtual light source calculated by the virtual light source addition processing unit 306 to the subject as a rewriting effect to the RGB signal output from the degamma processing unit 302. The gamma processing unit 307 applies a gamma characteristic to the RGB signal output from the virtual light source addition processing unit 306. The luminance / color difference signal conversion unit 308 converts the RGB signal output from the gamma processing unit 307 into a luminance / color difference signal (Y, BY, RY). The subject area determination unit 309 determines the subject area in the captured image based on the distance map acquired from the distance calculation unit 303 and the face area information acquired from the face detection unit 113. The rewriting reliability calculation unit 310 calculates the reliability of the rewriting process based on the relationship between the face area information detected by the face detection unit 113 and the subject area information obtained by the subject area determination unit 309.

Next, the operation of the rewriting processing unit 114 will be described.

The rewriting processing unit 114 reads out the luminance / color difference signals (Y, BY, RY) stored in the image memory 106 and inputs them to the RGB signal conversion unit 301. The RGB signal conversion unit 301 converts the input luminance / color difference signals (Y, BY, RY) into RGB signals and outputs them to the degamma processing unit 302.

The degamma processing unit 302 calculates the gamma characteristic applied by the gamma processing units 204 and 206 of the image processing unit 105 and converts it into linear data. The degamma processing unit 302 outputs the RGB signals (Rt, Gt, Bt) after linear conversion to the virtual light source reflection component calculation unit 305 and the virtual light source addition processing unit 306.

The distance calculation unit 303 calculates a distance map from the subject distance information acquired from the distance detection unit 118. The subject distance information is two-dimensional distance information obtained in units of pixels of the captured image, and the distance map is map data of the distance information for each pixel of the captured image. The normal calculation unit 304 acquires a distance map from the distance calculation unit 303, and calculates a normal map using the normal information corresponding to each pixel of the captured image as map data. A known technique is used as a method for generating a normal map from a distance map, and a specific processing example will be described with reference to FIG. FIG. 4 shows the shooting coordinates and the positional relationship between the subject and the virtual light source. For example, for a subject 401 as shown in FIG. 4, the gradient information can be calculated from the difference ΔD of the distance (depth) D with respect to the horizontal difference ΔH of the captured image, and the normal N can be calculated from the gradient information. It is possible. By performing the above processing on each pixel of the captured image, the normal information N corresponding to each pixel of the captured image can be calculated. The normal calculation unit 304 generates normal information corresponding to each pixel of the captured image as a normal map, and outputs the normal information to the virtual light source reflection component calculation unit 305.

The subject area determination unit 309 determines the subject area in the captured image from the distance map acquired from the distance calculation unit 303 and the face area information acquired from the face detection unit 113. _{Specifically, the average value Def ave} of the distance values in the face area is calculated from the face area information, and an arbitrary range including the average value Def _ave , for example, a region having a distance value such as _{Def ave ± 10 is set as the subject area.} decide.

The rewriting reliability calculation unit 310 calculates the reliability of the rewriting process based on the relationship between the subject area information acquired from the subject area determination unit 309 and the face area information acquired from the face detection unit 113. The rewriting reliability is an index that quantifies whether or not an image whose brightness is corrected as intended by the rewriting process can be obtained. The rewriting reliability is low when there is a possibility that the rewriting process may result in an unnatural image or the pattern may be damaged.

FIG. 5 illustrates the face area information detected by the face detection unit 113 and the subject area information obtained by the subject area determination unit 309. In FIG. 5, (a) is a shooting scene, (b), (c), and (d) are face area information detected by the face detection unit 113, and (e), (f), and (g) are subject area determinations. The subject area information obtained by the unit 309 is illustrated. Further, FIG. 5B shows the face detection result when the face region is appropriately detected by the face detection unit 113. 5 (c) and 5 (d) show the face detection result when the face region is erroneously detected by the face detection unit 113. In particular, FIG. 5C is a case where the background area outside the face area is erroneously detected as a face, and FIG. 5D is an erroneous case where a small area compared to the actual face area is a face. This is the case when it is detected. FIG. 5E shows the subject area information appropriately determined by the subject area determination unit 309. 5 (f) and 5 (g) show subject area information when the subject area is erroneously determined by the subject area determination unit 309. In particular, FIG. 5 (f) shows a case where it is erroneously determined that a portion outside the subject area is also a subject area, and FIG. 5 (g) shows a case where a part of the subject area is erroneously determined to be outside the area. If the rewriting process is performed using the subject area information that is erroneously determined as in FIGS. 5 (c), (d), (f), and (g), the image after the rewriting process may become unnatural. Therefore, in the case of erroneous detection as described above, the rewriting reliability calculated by the rewriting reliability calculation unit 310 becomes a low value.

The virtual light source reflection component calculation unit 305 calculates the component reflected by the virtual light source 402 on the subject 401 based on the distance K between the virtual light source 402 and the subject 401, the normal information N, and the virtual light source parameter L shown in FIG. Specifically, the reflection of the coordinate position corresponding to each pixel of the captured image is inversely proportional to the square of the distance K between the virtual light source 402 and the subject 401, and proportional to the inner product of the normal vector N and the light source direction vector L. Calculate the components. This will be described with reference to FIG. In FIG. 4, 401 indicates the subject and 402 indicates the position of the set virtual light source. The reflection component at the horizontal pixel position H1 (the vertical pixel position is omitted for simplification of explanation) of the image captured by the image pickup apparatus 100 is proportional to the inner product of the normal N1 at the photographing coordinates H1 and the direction vector L1 of the virtual light source. , The value is inversely proportional to the distance K1 between the virtual light source and the subject position.

Expressing the above relationship with a mathematical formula, the subject reflection components (Ra, Ga, Ba) by the virtual light source are given by the following equation 1.
(Equation 1)
Ra = α × (−L ・ N) / K ² × Rt
Ga = α × (−L ・ N) / K ² × Gt
Ba = α × (−L ・ N) / K ² × Bt
Here, α is the strength of the virtual light source, L is the three-dimensional direction vector of the virtual light source, N is the three-dimensional normal vector of the subject, and K is the distance between the virtual light source and the subject. Further, Rt, Gt, and Bt are RGB signals output from the degamma processing unit 302.

When the rewriting reliability calculated by the rewriting reliability calculation unit 310 is higher than a predetermined threshold value, the virtual light source reflection component calculation unit 305 calculates the reflection components (Ra, Ga, Ba) by the virtual light source by the above equation 1. do. When the rewriting reliability is lower than a predetermined threshold value, the gain value α of the rewriting process and the three-dimensional direction vector L and the three-dimensional normal vector N of the virtual light source are controlled to control the reflection components (Ra, Ga, Ba) is calculated. Specific processing will be described later using the flowchart of FIG.

The reflection components (Ra, Ga, Ba) by the virtual light source calculated as described above are output to the virtual light source addition processing unit 306. The virtual light source addition processing unit 306 performs the processing of the following formula 2 for adding reflection components (Ra, Ga, Ba) by the virtual light source.
(Equation 2)
Rout = Rt + Ra
Gout = Gt + Ga
Bout = Bt + Ba
The image signals (Rout, Gout, Bout) output from the virtual light source addition processing unit 306 are input to the gamma processing unit 307. The gamma processing unit 307 performs gamma correction on the input RGB signal. The luminance / color difference signal conversion unit 308 generates a luminance Y, a luminance signal RY, and a BY signal from the RGB signal.

The above is the operation of the rewriting processing unit 114. FIG. 6 shows an example of an image rewritten by the rewriting processing unit 114. FIG. 6A is a photographed image before the rewriting process, and FIG. 6B is a photographed image after the rewriting process. By performing the rewriting process on the dark subject in FIG. 6 (a), the shadow is corrected so as to be bright as shown in FIG. 6 (b).

The system control unit 120 writes the luminance / color difference signals (Y, BY, RY) output from the rewriting processing unit 114 into the image memory 106 by the memory control unit 107. After that, the codec unit 110 performs compression coding and records on the recording medium 112 via the recording medium I / F 111.

<Rewriting Process> Next, the control procedure of the rewriting process of the present embodiment will be described with reference to FIG. 7.

The process of FIG. 7 is realized by reading the program recorded in the non-volatile memory 116 into the system memory 117 and executing the program by the system control unit 120. The same applies to FIG. 9 described later.

In the present embodiment, the system control unit 120 controls the rewriting processing unit 114, calculates the reliability of the rewriting processing, and performs the rewriting processing based on the calculated rewriting reliability.

In S701, the system control unit 120 performs the above-described photographing operation to photograph the subject.

In S702, the system control unit 120 determines whether or not the rewriting processing unit 114 performs the processing by determining whether or not the rewriting processing mode is selected via the operation unit 115 by the user operation. If it is determined that the rewriting process is to be performed, the process proceeds to S703, and if not, the present process is terminated.

In S703, the system control unit 120 acquires the face area information of the subject from the face detection unit 113. The face detection unit 113 acquires information on the coordinate positions of organs such as eyes, nose, and mouth, which are parts of the face of the subject in the captured image, and detects the face region from those coordinate positions. The face detection unit 113 is not limited to the method of detecting the face region from the position of the organ which is a part of the face, and may use another method of learning the past detection result of the face region and detecting the face region. You may use it.

In S704, the system control unit 120 acquires the subject area information from the subject area determination unit 309. As described above, the subject area determination unit 309 acquires face area information from the face detection unit 113 and distance map information from the distance calculation unit 303, and determines an area having a distance value equivalent to the face distance value as the subject area. ..

In S705, the system control unit 120 calculates the rewriting reliability in the rewriting reliability calculation unit 310. The rewriting reliability calculation unit 310 calculates a low value as the rewriting reliability when the detection result by the face detection unit 113 or the determination result by the subject area determination unit 309 is not appropriate as described with reference to FIG. Here, a method of calculating the rewriting reliability will be described with reference to FIG.

FIG. 8A illustrates the relationship between the level of coincidence between the positions of the subject area and the face area and the rewriting reliability. If the face area is included in the subject area, the rewriting reliability is high, and conversely, as shown in FIG. 5C, if the face area is not included in the subject area, the rewriting reliability is low. The farther the position of the subject area and the position of the face area are, the lower the matching degree level and the lower the rewriting reliability.

FIG. 8B illustrates the relationship between the area level of the face region and the rewriting reliability with respect to the subject region. As shown in FIG. 5D, the rewriting reliability is low when the face area is too small (the difference between the two is less than the threshold value) or too large (the difference between the two is greater than or equal to the threshold value) with respect to the subject area. do. Depending on the shooting scene, the size of the subject area differs depending on whether the face is shot up or the whole body is shot. For this reason, it is difficult to determine how small the face area is with respect to the subject area for false detection, but if the face area is too small compared to the subject area, there is a high possibility of false detection. Control to reduce rewriting reliability.

FIG. 8C illustrates the relationship between the protrusion level of the subject area to the peripheral area of the face and the rewriting reliability. As shown in FIG. 5 (f), when the subject area extends to the peripheral area of the face area, the reliability of rewriting is lowered. The subject area includes the face, but also includes the body portion below the neck, so there is no problem even if the subject area extends beyond the face area. However, if the subject area is above the face area, that is, above the head, the rewriting reliability is lowered because there is a high possibility of false detection.

FIG. 8D illustrates the relationship between the missing level of the subject area and the rewriting reliability with respect to the face area. When the subject area lacks a part of the face area as shown in FIG. 5 (g), the reliability of rewriting is lowered. The larger the missing area, the lower the rewriting reliability is controlled to be calculated.

Then, the rewriting reliability calculation unit 310 calculates the final rewriting reliability from the plurality of elements shown in FIGS. 8A to 8D. For example, the rewriting reliability is calculated for each of the four elements, and the average value is used to calculate the final rewriting reliability, or the weight is given according to the importance of each element to give the final rewriting reliability. Calculate the degree.

Returning to FIG. 7, in S706, the system control unit 120 determines whether or not the rewriting reliability calculated in S705 is higher than the predetermined threshold value Th, and if it is determined that it is equal to or higher than the threshold value, proceeds to S707 and is less than the threshold value. If it is determined that, the process proceeds to S708.

In S707, the system control unit 120 executes the rewriting process described above.

In S708, the system control unit 120 changes the parameter of the rewriting process from the normal value and executes the rewriting process.

Specifically, the reflection components (Ra, Ga, Ba) by the virtual light source are calculated using the above-mentioned equation 1, but the gain value α of the rewriting process used in the equation and the three-dimensional effect of the virtual light source. The direction vector L and the three-dimensional normal vector N are controlled. For example, when the rewriting reliability is low, there is a high possibility that the processing result will be unnatural when the rewriting process is performed. Therefore, it is desirable to control the gain value α of the rewriting process to be small. Therefore, the gain value α of the rewriting process is controlled to be set lower than usual according to the low rewriting reliability. For example, if the rewriting reliability is 0, which is the lowest value, the gain value α of the rewriting process is set to zero, and control is performed so that the rewriting process is not performed.

It is also possible to control the reflection component by the virtual light source due to the factor that the rewriting reliability is lowered. For example, when the cause of the low rewriting reliability is an erroneous determination of the subject area information by the subject area determination unit 309, the distance calculation unit 303 that outputs the distance information to the subject area determination unit 309 appropriately calculates. It will not be. In this case, since the three-dimensional normal vector N, which is the output of the normal calculation unit 304 calculated based on the distance information from the distance calculation unit 303, is also considered to be inaccurate, reflection by the virtual light source is performed with N = 1. Find the components (Ra, Ga, Ba).

For example, when the factor of low rewriting reliability is an erroneous determination of the subject area information of the subject area determination unit 309, and the erroneous detection is that the subject area is lacking with respect to the face area as shown in FIG. 5 (g). Will not stand out unless the missing area is rewritten. Therefore, the reflection components (Ra, Ga, Ba) by the virtual light source are obtained by controlling the irradiation range of the three-dimensional direction vector L of the virtual light source so that the missing region is not irradiated with the light of the virtual light source.

The operation of the rewriting processing unit 114 of the system control unit 120 has been described above.

[Embodiment 2] Next, the second embodiment will be described with reference to FIG.

FIG. 9 shows a control procedure of the rewriting process when a plurality of subjects are present in the captured image.

Since S701 to S705 in FIG. 9 are the same as the processing in FIG. 7 when the subject is a single subject, the description thereof will be omitted.

In S901, the system control unit 120 calculates the main subject level. The main subject is, for example, a person who occupies a large area in the captured image, a person whose face is registered in advance, or a person who is closer to the image pickup device. The higher the main subject in the captured image, the higher the level.

In S902, the system control unit 120 determines whether or not the calculation of the rewriting reliability and the main subject level for each subject is completed. Then, when it is determined that the calculation of the rewriting reliability and the main subject level is completed, the process proceeds to S903, and when it is not completed, the process returns to S702 and the processes from S702 to S902 are repeated.

In S903, the system control unit 120 determines whether or not the rewriting reliability of all the subjects is higher than the predetermined threshold value Th, and if it is determined that the rewriting reliability is equal to or higher than the threshold value, the system proceeds to S904 and determines that the rewriting reliability is less than the threshold value. If so, proceed to S905.

In S904, the system control unit 120 executes a normal rewriting process.

In S905, the system control unit 120 determines whether or not the rewriting reliability of a subject having a high main subject level calculated in S901 is higher than a predetermined threshold Th. Then, if it is determined that the value is equal to or higher than the threshold value, the process proceeds to S906, and if it is determined that the value is lower than the threshold value, the process proceeds to S907.

In S906, the system control unit 120 executes normal rewriting for the main subject, and changes the parameters of the rewriting process for the non-main subject that is not the main subject to execute the rewriting process. For example, since the rewriting reliability of a non-main subject is low, the rewriting process is executed using a gain value lower than the gain value of the normal rewriting process.

In S907, the system control unit 120 executes the rewriting process by changing the parameters used in the normal rewriting process for all the subjects. For example, the rewriting process is executed for all the subjects using a gain value lower than the gain value of the normal rewriting process. Further, as described with reference to FIG. 7, the rewriting process is executed by controlling the values of the three-dimensional direction vector L and the three-dimensional normal vector N of the virtual light source of the rewriting parameter due to the factor that the rewriting reliability is low.

The control procedure of the rewriting process when a plurality of subjects are present in the captured image has been described above.

In each of the above-described embodiments, when the rewriting reliability calculation unit 310 calculates the rewriting reliability, the final rewriting is performed from the four elements that cause erroneous detection using FIGS. 8A to 8D. The example of calculating the reliability has been described, but the present invention is not limited to this, and the false detection pattern may have four or more elements.

Further, in each of the above-described embodiments, as shown in FIG. 8, the rewriting reliability calculation unit 310 analyzes the relationship between the face area information of the face detection unit 113 and the subject area information of the subject area determination unit 309. An example of calculating the rewriting reliability has been described, but the present invention is not limited to this. For example, the reliability of the face area information detected by the face detection unit 113 is calculated, the reliability of the subject area information is calculated by the subject area determination unit 309, the information of at least one of the reliabilitys is acquired, and the information is obtained. The rewriting reliability may be calculated based on the configuration.

Further, in each of the above-described embodiments, a case where one final rewriting reliability is calculated for one image or each rewriting reliability is calculated for a plurality of subjects included in one image has been described. It is not limited to this. For example, the rewriting reliability may be calculated for each pixel. In that case, for example, when the positions of the subject area and the face area are misaligned, the larger the amount of misalignment between the subject area and the face area, the larger the pixel. , Reduce rewriting reliability.

Further, in the present embodiment, the case where the shadow (dark part) of the subject is corrected so as to be brightened by the rewriting process has been described, but conversely, the relighting may be performed to darken the subject. In that case, the gain value α of the rewriting process is set to minus.

Further, the method of calculating the position of the virtual light source and the distance D of the pixels of the target area of the rewriting process described with reference to FIG. 4 is not limited to the above-mentioned method, and any calculation method may be used. For example, the positions of the image pickup apparatus and the subject may be acquired as three-dimensional positions and calculated by the distance in three dimensions.

Further, when adding a reflection component by a virtual light source to the RGB signal of each pixel, an equation that is inversely proportional to the square of the distance between the virtual light source and the subject is used, but the present invention is not limited to this. For example, it may be an equation that is inversely proportional to the distance D or an equation in which the irradiation range changes in a Gaussian distribution.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 ... Imaging device, 103 ... Imaging unit, 105 ... Image processing unit, 113 ... Face detection unit, 114 ... Rewriting processing unit, 118 ... Distance detection unit, 120 ... System control unit, 303 ... Distance calculation unit, 309 ... Subject area Decision unit, 310 ... Rewriting reliability calculation unit

Claims (17)

A processing means for setting a virtual light source, which is a virtual light source, and performing correction processing for correcting the brightness of the subject in the image using the virtual light source.
An acquisition unit configured to acquire a certain partial area information of the subject in the image,
Determining means for determining a subject region to be the correction process in the image,
Control for controlling the correction processing by said processing means in accordance with the reliability of the correction process based on a relationship between the determined said object regions by a particular partial area information and said determining means of the subject obtained by the acquisition unit An image processing apparatus comprising: Means. Further comprising a face detecting unit that detects a human face region in the image, and a distance detecting means for detecting the distance information to the subject,
It said determining means, the image processing apparatus according to claim 1, wherein the determining the object region to be the correction process based on the detection result by the face detecting means and the distance detecting means. The image processing apparatus according to claim 2, wherein the specific partial region information of the subject is acquired based on the face region detected by the face detecting means. The claim is characterized in that the control means lowers the reliability of the correction process when the determination means determines the subject region, including the peripheral region of the face region detected by the face detection means. Item 3. The image processing apparatus according to item 3. Said control means, characterized in that the area of the object region determined by the determination unit, when the small area of the face region detected by said face detecting means, to lower the reliability of the correction process The image processing apparatus according to claim 3. Said control means, characterized in that the area of the object region determined by the determination unit, when a larger area of the face region detected by said face detecting means, to lower the reliability of the correction process The image processing apparatus according to claim 3. The face detecting means calculates the reliability of the detected face region and calculates the reliability.
The determining means calculates a reliability of the subject region,
The image according to any one of claims 3 to 6, wherein the control means obtains the reliability of the correction process from at least one of the reliability of the face region and the reliability of the subject region. Processing equipment. Any of claims 1 to 7, wherein the control means controls to change and execute the correction process by the processing means when the reliability of the correction process is less than a predetermined threshold value. The image processing apparatus according to item 1. The control means according to any one of claims 1 to 7, wherein the control means controls so that the correction process by the processing means is not executed when the reliability of the correction process is less than a predetermined threshold value. The image processing apparatus described. Wherein if there are multiple objects in the image, the image processing apparatus according to claim 3, wherein the determination of the reliability of the correction processing for each object. It said control means calculates a level of the object in the image,
The image processing apparatus according to claim 10, wherein the correction process by the processing means is controlled based on the level of the subject and the reliability of the correction process. When there is a subject whose reliability of the correction process is less than a predetermined threshold value, the control means determines the level of the subject.
When the reliability of the correction processing of the first subject whose level of the subject is higher than the threshold value is equal to or higher than the predetermined threshold value, the correction processing by the processing means is executed on the first subject, and the first subject is described. The image processing apparatus according to claim 11, wherein the second subject, which is not the subject, is controlled so that the correction process by the processing means is changed and executed. When the reliability of the correction process of the first subject is less than the predetermined threshold value, the control means controls to change and execute the correction process by the processing means for all the subjects. The image processing apparatus according to claim 12. It photographs the subject, further comprising an imaging means for generating the captured image,
The processing means, the image processing apparatus according to the captured image generated by the imaging means in any one of claims 1 to 13, characterized in that the object of the correction process. A step in which the processing means sets a virtual light source, which is a virtual light source, and performs correction processing for correcting the brightness of the subject in the image using the virtual light source.
Acquiring means, acquiring a specific partial region information of the subject in the image,
Determining means, determining the subject to the subject area of the correction process in the image,
An image characterized in that the control means includes a step of controlling the correction process by the processing means according to the reliability of the correction process based on the relationship between the specific partial area information of the subject and the subject area. Processing method. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 14. A computer-readable storage medium containing a program for causing the computer to function as each means of the image processing apparatus according to any one of claims 1 to 14.

Images