JP2021108023A - Image processing system and image processing method - Google Patents

Abstract

To provide an image processing system and an image processing method that can suppress unnatural effects of virtual light applied to an automatically detected area.SOLUTION: An image processing system determines an irradiation characteristics of virtual light for a feature area detected from image data. Then, an image processing system corrects the determined irradiation characteristics according to the detection reliability of the feature area, and adds an effect of the virtual light to the feature area based on the corrected irradiation characteristics. The image processing system corrects the irradiation characteristics so that the effect of the virtual light is weaker in the case of the second confidence level where the confidence level is lower than the first confidence level than in the case where the detection confidence level is the first confidence level.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus and an image processing method, and more particularly to a technique for imparting an effect of virtual light to an image.

Conventionally, a rewriting technique for imparting an effect of light (virtual light) from a virtual light source to an image has been known (Patent Document 1). Patent Document 1 discloses that the position of a virtual light source is determined so that the contrast value of the subject area to which relighting is applied becomes a target value.

Japanese Patent No. 6442209

When the subject area to which relighting is applied is automatically detected, the quality of rewriting is affected by the detection accuracy of the subject area. For example, if the detection accuracy of the subject area is low, the image after applying relighting may become unnatural.

An object of the present invention is to provide an image processing apparatus and an image processing method capable of suppressing the effect of virtual light applied to an automatically detected region from becoming unnatural.

The above-mentioned objectives are a detection means for detecting a feature region from image data, a determination means for determining the irradiation characteristics of virtual light for the feature region, and an irradiation characteristic determined by the determination means according to the detection reliability of the feature region. When the correction means has a correction means for correcting and an additional means for adding the effect of virtual light to the feature region based on the irradiation characteristics corrected by the correction means, and the correction means has a detection reliability of the first reliability. Achieved by an image processing apparatus characterized in that the irradiation characteristics are corrected so that the effect of virtual light is weaker in the case of the second reliability, which is lower in reliability than the first reliability. NS.

According to the present invention, it is possible to provide an image processing apparatus and an image processing method capable of suppressing the effect of virtual light applied to an automatically detected region from becoming unnatural.

Block diagram showing a functional configuration example of a digital camera according to an embodiment A block diagram showing a functional configuration example of the image processing unit in FIG. A block diagram showing a functional configuration example of the rewriting processing unit in FIG. The figure regarding the rewriting process in an embodiment Flow chart regarding the operation of the light source information correction unit in the embodiment The figure which shows the characteristic example of the reliability used by the light source information correction part in an embodiment. The figure regarding the correction performed by the light source information correction unit in an embodiment. The figure which shows typically the effect of the light source information correction part in an embodiment.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings, based on its exemplary embodiments. The following embodiments do not limit the invention according to the claims. Further, although a plurality of features are described in the embodiment, not all of them are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

In the following, an embodiment in which the present invention is applied to a digital camera will be described. However, the imaging function is not essential to the present invention and can be applied to any electronic device. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game machines, robots, drones, drive recorders and the like. These are examples, and the present invention can be applied to other electronic devices.

FIG. 1 is a block diagram showing a functional configuration example of a digital camera 100 as an example of an image processing device according to an embodiment of the present invention. The optical system 101 includes a focus lens, a diaphragm that also serves as a shutter, and the like, and forms a subject image on the image pickup surface of the image pickup device 102. The control unit 109 controls the operation of the focus lens and the aperture.

The image sensor 102 is, for example, a CCD image sensor or a CMOS image sensor, and has a plurality of photoelectric conversion units (pixels) arranged in two dimensions. The image pickup device 102 generates an analog image signal by converting the subject image formed by the optical system 101 into an electric signal by the photoelectric conversion unit. The A / D conversion unit 103 A / D-converts the analog image signal to generate RAW image data.

The image processing unit 104 applies predetermined image processing to the RAW image data, generates signals and image data, and acquires and / or generates various types of information. The image processing unit 104 may be a dedicated hardware circuit such as an ASIC designed to realize a specific function, or a programmable processor such as a DSP may execute the software to perform the specific function. It may be a configuration to be realized.

Here, the image processing applied by the image processing unit 104 includes preprocessing, color interpolation processing, correction processing, data processing processing, evaluation value calculation processing, special effect processing, and the like. Preprocessing includes signal amplification, reference level adjustment, defective pixel correction, and the like. The color interpolation process is a process of interpolating the values of color components not included in the image data read from the pixels, and is also called a demosaic process or a simultaneous process. The correction processing includes white balance adjustment, gradation correction (gamma processing), processing for correcting aberrations in the optical system of the optical system 101, processing for correcting color, and the like. The data processing process includes synthesis processing, scaling processing, coding and decoding processing, header information generation processing, and the like. The evaluation value calculation process includes generation of signals and evaluation values used for automatic focus detection (AF), calculation process of evaluation values used for automatic exposure control (AE), and the like. Special effects processing includes adding blur, changing color tones, rewriting processing, and the like. Note that these are examples of image processing that can be performed by the image processing unit 104, and do not limit the image processing performed by the image processing unit 104.

The image processing unit 104 applies predetermined image processing to the RAW image data to generate image data for recording and image data for display. The image data for recording is output to the recording unit 108 in the form of a predetermined image data foul. Further, the image data for display is output to the VRAM area of the memory 110.

The recording unit 108 records the image data output from the image processing unit 104 on a recording medium such as a semiconductor memory card under the control of the control unit 109. Further, the recording unit 108 reads data from the recording medium and outputs the data to, for example, the memory 110 under the control of the control unit 109.

The distance map generation unit 105 generates a distance map D_MAP of the shooting range using, for example, image data supplied from the image processing unit 104, and outputs the generated distance map D_MAP to the image processing unit 104. The method for generating the distance map D_MAP is not particularly limited, and a known method can be used. The distance map D_MAP is information representing the subject distance for each pixel, and may be a depth map (sometimes called a distance image, a depth image, etc.) in which the brightness value represents the distance. For example, by obtaining the focus lens position at which the evaluation value for contrast AF generated by the image processing unit 104 is maximized for each pixel, the subject distance can be acquired for each pixel. In addition, the distance information for each pixel is based on the correlation between the amount of blur and the distance from the image data obtained by shooting the same scene multiple times with different focusing distances and the point spread function (PSF) of the optical system. Can also be asked. These techniques are described in, for example, Japanese Patent Application Laid-Open No. 2010-177741 and US Pat. No. 4,965,840. Further, when the parallax image pair can be used, such as when the image sensor 102 supports the shooting of a parallax image, the subject distance can be acquired for each pixel by a technique such as stereo matching.

The face detection unit 106 detects a region (face region) having the characteristics of a person's face from the image data as an example of the feature region (subject region) to which the image processing unit 104 applies the rewriting process. The subject area to which the rewriting process is applied is not limited to the face area, and may be another feature area. The face detection unit 106 can detect the face region by a known method.

Examples of known methods include a method of detecting a face region based on learning using a neural network, a method of detecting facial organs such as eyes, nose, and mouth from an image using template matching, and a skin color. There are methods based on image features such as the shape of the eyes and the shape of the eyes. In addition, a plurality of these methods can be combined to improve the accuracy of detecting the face region.

In the present embodiment, the face detection unit 106 detects the coordinates of the parts (right eye, left eye, mouth, nose, etc., also referred to as facial organs) included in the face region when the face region is detected. Then, the face detection unit 106 calculates the distance between the organs, such as the distance between the right eye and the left eye. The distance between the organs may be, for example, the distance between the representative coordinates (for example, the center or center of gravity coordinates) of the image region (organ region) of the organ. The face detection unit 106 detects an elliptical region centered on the center of gravity of the representative coordinate of the organ region as the face region based on the distance between the organs. At this time, the face detection unit 106 uses, for example, color information and distance information to determine the size of the elliptical region so as not to include the background region.

Further, the face detection unit 106 also calculates the detection reliability indicating the certainty of detection of the face region and the organ region by a known method. For example, when the organ region is detected by pattern matching, the degree of matching (for example, the degree of correlation with the template) can be used as the detection reliability. Further, the detection reliability of the face region can be determined based on, for example, the distance between the organs and the detection reliability of the organ region. Note that these are examples, and the detection reliability may be obtained by other methods. The face detection unit 106 outputs the detection result (for example, the number of face areas, face area information (for example, position, size, detection reliability, etc.)) to the image processing unit 104. The image processing unit 104 may execute the detection process of the subject area.

When a plurality of face regions are detected by the face detection unit 106, the face detection unit 106 selects one face region as the main face region. The selection method is not particularly limited, but the face area having the largest area, the face area closest to the center of the screen, and the face area having the highest detection reliability can be selected as the main face area. Alternatively, the user may be allowed to select the main face area. The image processing unit 104 may select the main face region. In the following description, the main face area and other face areas are not particularly distinguished, but when a plurality of face areas are detected, the main face area is set as the face area to which the rewriting process is applied in the present embodiment.

The shield detection unit 107 detects a region (shield region) of an object that shields (hides) a part of the subject to which the rewriting process is applied from the image data. In this embodiment, since the rewriting process is applied to the face area, an object that hides the face area, such as a mask or sunglasses, is detected as a shield. The method for detecting the shielded area is not particularly limited, and a known method such as template matching can be used. Distance information within the face area may be used. The shield detection unit 107 outputs information on the detected shield region (for example, position, size, detection reliability, etc.) to the image processing unit 104.

The control unit 109 has one or more microprocessors (CPU, MPU), and operates each functional block by, for example, reading a program stored in the ROM in the memory 110 into the RAM in the memory 110 and executing the program. To control. As a result, the control unit 109 realizes various functions of the digital camera 100. For example, the control unit 109 controls the driving of the focus lens based on the AF evaluation value generated by the image processing unit 104, and determines the exposure condition based on the AE evaluation value generated by the image processing unit 104.

The memory 110 has a ROM (nonvolatile area) and a RAM (volatile area). The ROM stores programs, setting values, GUI data, etc. that can be executed by the microprocessor of the control unit 109. The RAM is used to read a program executed by the microprocessor of the control unit 109 and to store values required during the execution of the program. The RAM is also used as a buffer for the image processing unit 104 or as a video memory for the display unit 112.

The operation unit 111 is a general term for input devices such as buttons and switches for a user to input various instructions to the digital camera 100. When the display unit 112 is a touch display, the touch panel is included in the operation unit 111. Further, the operation unit 111 may include an input device of a type such as voice input or line-of-sight input that inputs an instruction in a non-contact manner.

The display unit 112 is, for example, a liquid crystal display installed on the back surface of the digital camera 100. The display unit 112 functions as an electronic viewfinder (EVF) in the shooting standby state or during movie shooting. The display unit 112 also displays the current set value and state of the digital camera 100, the menu screen, the image data read from the recording unit 108, and the like.

FIG. 2 is a block diagram showing a configuration example of the image processing unit 104. The image processing unit 104 includes a simultaneous processing unit 201, a white balance processing unit 202, a rewriting processing unit 203, and a gamma processing unit 204.

The RAW image data output by the A / D conversion unit 103 is input to the simultaneous processing unit 201. The simultaneous processing unit 201 generates values of color components that are lacking in the pixel data constituting the image data. Here, it is assumed that the image sensor 102 is provided with a color filter having a primary color Bayer array. In this case, the pixel data constituting the input RAW image data has only one color component value among the R, G, and B color components. The simultaneous processing unit 201 uses a known color interpolation method to generate the values of the color components that are lacking in the individual pixel data.

The simultaneous processing unit 201 outputs the processed image data to the white balance processing unit 202. The white balance processing unit 202 corrects the white balance of the image data by applying, for example, the white balance gain calculated by the control unit 109. The image data after the white balance correction is output to the rewriting processing unit 203, and is also output to the distance map generation unit 105, the face detection unit 106, and the obstruction detection unit 107.

The image data used by the distance map generation unit 105, the face detection unit 106, and the obstruction detection unit 107 is not limited to the image data after the white balance correction processing, and may be supplied at another processing stage.

The rewriting processing unit 203 receives image data from the white balance processing unit 202, distance map D_MAP from the distance map generation unit 105, and detection results from the face detection unit 106. The rewriting processing unit 203 applies a rewriting process that adds an effect of irradiating virtual light to the image data, and outputs the processed image data (R_out, G_out, B_out) to the gamma processing unit 204. The gamma processing unit 204 applies a gradation conversion process based on a predetermined gamma curve to the image data after the rewriting process, and outputs the processed image data (Rg, Gg, Bg) to the recording unit 108. Although omitted here, various image processing (for example, coding processing) can be applied after the gamma processing until the image is output to the recording unit 108. If the image data after the gamma processing is for display, it is not output to the recording unit 108, but is output to the display unit 112 via the memory 110.

The details of the processing applied to the image data by the rewriting processing unit 203 will be described with reference to FIG. FIG. 3 is a block diagram showing a configuration example of the rewriting processing unit 203. The rewriting processing unit 203 includes a normal acquisition unit 301, a light source information calculation unit 302, a light source information correction unit 303, a diffuse reflection component calculation unit 304, and an additional processing unit 305.

The detection result by the face detection unit 106 is input to the normal acquisition unit 301. The normal acquisition unit 301 acquires the normal information for the face region detected by the face detection unit 106. For example, the normal acquisition unit 301 determines the direction of the face from the position and distance of the organ in the face area detected by the face detection unit 106, and determines the direction of the face with respect to the face of a general person stored in advance in the ROM area of the memory 110. Using the normal information, the appropriate normal information is acquired for the detected face area. The normal acquisition unit 301 outputs the acquired normal information N to the light source information calculation unit 302.

The light source information calculation unit 302 is supplied with image data from the white balance processing unit 202 and normal N from the normal acquisition unit 301. The light source information calculation unit 302 calculates the three-dimensional direction vector L indicating the irradiation direction of the virtual light and the intensity α of the virtual light source by using a known method. For example, as described in Patent Document 1, for setting the contrast value as a target value based on the contrast value obtained based on the brightness of blocks having similar colors and the estimated irradiation direction of ambient light. The irradiation characteristics (irradiation direction and irradiation intensity) of virtual light can be determined. The method for determining the irradiation characteristics (irradiation direction and irradiation intensity) of virtual light is not limited to the method described in Patent Document 1.

In the present embodiment, the direction of the virtual light emitted from the virtual light source to the subject is represented as a virtual light vector L which is a three-dimensional direction vector. Further, the irradiation intensity is converted into the light source intensity α based on the distance between the irradiation target coordinates and the light source coordinates. The vector L can be expressed as the unit vector nL indicating the irradiation direction of the virtual light and the distance | L | to L = | L | * nL between the irradiation target coordinates and the light source coordinates. The light source information calculation unit 302 outputs the obtained vector L and the irradiation intensity α to the light source information correction unit 303.

The light source information correction unit 303 has at least one of a virtual light vector L (irradiation direction) and a light source intensity α (irradiation intensity) representing the irradiation characteristics of virtual light as a characteristic region based on the state of the face region to irradiate the virtual light source. Correct according to the detection reliability. The light source information correction unit 303 outputs the corrected virtual light vector L'and the corrected light source intensity α'to the diffuse reflection component calculation unit 304.

The diffuse reflection component calculation unit 304 calculates the diffuse reflection components Rd, Gd, and Bd of the virtual light by the subject. How to obtain the diffuse reflection component will be described with reference to FIG.
FIG. 4 is a vertical view of the positions of the digital camera 100, the subject 401, and the virtual light source 402 that irradiates the subject with virtual light at the time of shooting. Here, the light rays incident on the pixels arranged in the horizontal direction with the vertical imaging surface will be described. The basic idea is the same for the vertical direction. Further, the shape of the subject 401 is simplified for easy explanation and understanding.

ここで、（式１）におけるＮは被写体の３次元法線ベクトル、Ｋは仮想光源と被写体との距離、ｋｄは被写体の拡散反射率である。α’、Ｌ’はそれぞれ光源情報補正部３０３が出力する補正後の仮想光源の強度と仮想光ベクトルである。ｗθは予め用意された、仮想光源４０２の方向と被写体の法線方向の差に基づく重み関数である。重み関数ｗθは、仮想光源４０２の方向と、仮想光源から注目画素（カメラ座標Ｈ１）に対応する被写体上の位置への方向の角度差に応じて変化する特性をもつ。 The diffuse reflection component of the virtual light incident on the horizontal pixel position H1 of the image taken by the digital camera 100 is proportional to the inner product of the normal N1 and the virtual light vector L1 at the camera coordinates H1, and the virtual light source 402 and the subject position. The value is inversely proportional to the square of the distance K1. Therefore, the intensity Pd of the diffuse reflection component due to the light (virtual light) emitted from the virtual light source 402 being reflected by the subject 401 can be expressed by (Equation 1).

Here, N in (Equation 1) is the three-dimensional normal vector of the subject, K is the distance between the virtual light source and the subject, and kd is the diffuse reflectance of the subject. α'and L'are the intensity and the virtual light vector of the corrected virtual light source output by the light source information correction unit 303, respectively. wθ is a weighting function prepared in advance based on the difference between the direction of the virtual light source 402 and the normal direction of the subject. The weighting function wθ has a characteristic that changes according to the angle difference between the direction of the virtual light source 402 and the direction from the virtual light source to the position on the subject corresponding to the pixel of interest (camera coordinates H1).

Further, using the intensity Pd of the diffuse reflection component, the diffuse reflection components Rd, Gd, and Bd for each color component can be expressed by (Equation 2).
Rd = Pd × Rw × R
Gd = Pd × Gw × G ・ ・ ・ (Equation 2)
Bd = Pd × Bw × B
Here, R, G, and B are parameters indicating the color component value of the pixel of interest, and Rw, Gw, and Bw are parameters indicating the color of the virtual light source 402 (virtual light).

The additional processing unit 305 uses the diffuse reflection components Rd, Gd, and Bd calculated by the diffuse reflection component calculation unit 304 as the color component values R, G, and B of the pixel of interest in the image data input from the white balance processing unit 202. By adding to, the effect of virtual light is added. Therefore, the image data R_out, G_out, and B_out to which the effect of virtual light is added by the additional processing unit 305 can be expressed as in (Equation 3).
R_out = R + Rd
G_out = G + Gd ... (Equation 3)
B_out = B + Bd

The addition processing unit 305 adds a diffuse reflection component to the image data input from the white balance processing unit 202, and outputs the data of the pixels included in the subject area to be rewritten. The additional processing unit 305 outputs the data of the pixels not included in the subject area without changing the color components R, G, and B.

Next, the processing of the light source information correction unit 303 will be described in detail with reference to the flowchart shown in FIG.
In S501, the light source information correction unit 303 acquires face area information, face reliability, and face organ detection reliability from the face detection unit 106. Further, the light source information correction unit 303 obtains the brightness value of the face area from the pixel data included in the face area among the image data input from the white balance processing unit 202, and uses it as the face brightness information. The light source information correction unit 303 can calculate, for example, a value obtained by synthesizing R, G, and B components of pixel data at a predetermined ratio, for example, R: G: B at a ratio of 3: 6: 1, as a luminance value. The light source information correction unit 303 may acquire face brightness information from the image processing unit 104 or the control unit 109. Further, the light source information correction unit 303 compares the arrangement of the organs in the face region obtained from the information of the organ regions included in the face region with the arrangement of the organ regions in the front-facing face held in advance. Obtain the posture of the face area (angle of the subject). The posture of the face region can be obtained as a horizontal angle and a vertical angle when the frontal angle is 0 ° horizontal and 0 ° vertical. Further, the light source information correction unit 303 acquires information on the shield area from the shield detection unit 107.

In S502, the light source information correction unit 303 calculates the light source reliability. The light source reliability is based on the following plurality of reliabilitys.
Reliability based on face angle Conf_FaceAngle
Reliability based on face area Conf_FaceSize
Facial organ detection reliability Conf_FacePartsDetect
Face detection reliability Conf_FaceDetect
Reliability based on face brightness Conf_Luminace
Reliability based on face shield Conf_FaceCover

The reliability Conf_FaceAngle based on the face angle is a reliability that decreases as the angle representing the posture of the face increases, and has the characteristics shown in FIG. 6A, for example. It is assumed that the horizontal and vertical angles are 0 degrees when the face is facing the digital camera 100 (facing the front). Further, when the face is directed to the right and upward when viewed from the digital camera 100, it has a positive angle in the horizontal and vertical directions, and when it is oriented to the left and downward, it has horizontal and vertical directions. It shall have a negative angle. When the weighted addition value FaceAngle of the horizontal angle absolute value and the vertical angle absolute value of the face exceeds the predetermined Th_FaceAngle, the reliability Conf_FaceAngle becomes 0. Basically, the larger the angle from the front, the lower the reliability Conf_FaceAngle.

Here, the weight when adding the horizontal angle and the vertical angle may be arbitrarily set. For example, it can be set based on the characteristics of the face detection unit 106. For example, the face detection unit 106 is configured to detect a face region based on a machine-learned neural network. In this case, when a face image having an angle in the horizontal direction is used as training data more than an image of a face having an angle in the vertical direction, the weight of the angle in the horizontal direction is relatively small so that the Conf_FaceAngle does not become low. Can be.

The reliability Conf_FaceSize based on the face area is a reliability that increases as the ratio of the face area area to the entire image increases, and has the characteristics shown in FIG. 6B, for example. In FIG. 6B, Th_FaceSize_Min is a ratio corresponding to the minimum size of the face region that can be detected by the face detection unit 106, and Th_FaceSize_Max corresponds to the size of the face region that can be regarded as having sufficiently high reliability of face detection. The ratio to do.

Face organ detection reliability Conf_FacePartsDetect is a reliability based on the reliability of the organ region detected by the face detection unit 106. For example, it may be the average value of the detection reliability of individual organ regions.

Face detection reliability Conf_FaceDetect is a reliability based on the detection reliability of the face region detected by the face detection unit 106. Here, the reliability included in the detection result obtained from the face detection unit 106 is used as it is, but the face detection reliability Conf_FaceDetect is set based on the reliability obtained from the face detection unit 106, such as weighting by some factor. You may ask.

The reliability based on face brightness Conf_Luminace is a reliability that decreases when the brightness of the face area is extremely dark or bright, and has a substantially high value in other brightness ranges, for example, FIG. 6 (c). It has the characteristics shown in. The reliability decreases as it is lower than Th_Luminance_Min (first threshold value), and decreases as it is larger than Th_Luminance_Max (second threshold value). Th_Luminance_Min and Th_Luminance_Max are threshold values determined based on the brightness at which the contrast in the face area is maintained in an appropriate range, for example, with respect to the average brightness value or the maximum / minimum brightness value of the pixels included in the face area. For example, when the minimum luminance value is high, most of the face area is overexposed, and it is considered that sufficient contrast is not obtained. Similarly, when the maximum luminance value is low, most of the face area is underexposed, and it is considered that sufficient contrast is not obtained. Th_Luminance_Min and Th_Luminance_Max can be determined experimentally in advance, for example.

The reliability Conf_FaceCover based on the face shield is a reliability that decreases as the ratio of the shield region to the face region increases, and has the characteristics shown in FIG. 6 (d), for example. Th_FaceCover is an upper limit of the allowable occupancy rate of the shield, and can be determined in advance, for example, experimentally.

The six types of light source reliability described above take a value from 0 to 1 representing the lowest reliability to the highest reliability. The light source information correction unit 303 applies a predetermined weight to each reliability, weights and adds them, and calculates the reliability Conf_power of the light source intensity. The reliability of the light source intensity Conf_power takes a value from 0 to 1, and the smaller the value, the lower the intensity of the virtual light source is corrected. There is no particular limitation on the method of setting the weight applied to the light source reliability when calculating the reliability of the light source intensity. Since the light source intensity greatly affects the brightness of the face region to which the effect of virtual light is applied, the weight of the reliability Conf_Luminase based on the face brightness can be relatively increased. Further, when the area of the face area is small, the reliability of the shadow information acquired from the face area may be low, so that the weight of the reliability Conf_FaceSize based on the face area can be relatively increased. These are merely examples, and other weighting may be performed. Further, the average value of the light source reliability may be used without weighting. Basically, there is no limitation on the weight setting method as long as it is possible to suppress the addition of virtual light from an unnatural result by suppressing the intensity of the virtual light source when the reliability of the light source is low.

The light source information correction unit 303 uses the light source intensity reliability Conf_power to correct the light source intensity α according to (Equation 4), and obtains the corrected light source intensity α'.
α'= Conf_power × α ・ ・ ・ (Equation 4)
As described above, the light source information correction unit 303 is more virtual in the case of the second reliability, which is lower in reliability than the first reliability, than in the case where the reliability Conf_power of the light source intensity is the first reliability. The light source intensity α is corrected so that the effect of light is weakened.

In S504, the light source information correction unit 303 corrects the virtual light vector L based on the light source reliability. The processing of S504 will be described with reference to FIG. 7. FIG. 7 is a diagram showing a principle of correcting the virtual light vector L from the relationship between the subject 401 and the virtual light source 402 described in FIG. The virtual light vector L indicating the direction of the virtual light emitted from the virtual light source 402 to the subject 401 has an irradiation target point 701 of the virtual light. The angle θ formed by the straight line 700 passing through the irradiation target point 701 and having the direction of the representative normal line Np of the subject 401 and the virtual light vector L is calculated. Using (Equation 5) with respect to the calculated θ, the angle θ'that the corrected virtual light vector L'is formed with the straight line 700 is calculated.
θ'= Conf_angle × θ ・ ・ ・ (Equation 5)

Here, Conf_angle is the directional reliability of the virtual light source, and takes a value of 0 to 1 like the intensity reliability. As described above, the light source information correction unit 303 is more likely to have a second reliability, which is lower in reliability than the first reliability, than when the directional reliability Conf_angle of the virtual light source is the first reliability. The irradiation direction θ is corrected so that the effect of the virtual light is weakened.

The smaller the reliability Conf_angle, the closer θ'is to 0, and the smaller the angle difference from the representative normal Np of the subject 401. The directional reliability Conf_angle is a reliability obtained by applying a predetermined weight to the six types of light source reliability described in S503 and weighting and adding them. The weight for determining the directional reliability may be different from the weight used for determining the strength reliability. For example, when the facial organ detection reliability Conf_FacePartsDetect is low, the reliability of the normal information acquired from the memory 110 may be low, so that the weight of the facial organ detection reliability Conf_FacePartsDetect is relatively increased. This is just an example and other weighting may be done. Further, the average value of the light source reliability may be used without weighting. Basically, when the light source reliability is low, by correcting the irradiation direction of the virtual light source closer to the direction of the representative normal, the change in shadow (contrast) due to the addition of virtual light will result in an unnatural result. If it can be suppressed, there is no limitation on the weight setting method.

The light source information correction unit 303 obtains the corrected virtual light vector L'so that the straight line 700 extending in the direction of the representative normal line Np of the subject 401 and the angle θ'are formed and | L'| and | L | are equal. ..

In S505, the light source information correction unit 303 outputs the corrected light source intensity α'and the virtual light vector L'to the diffuse reflection component calculation unit 304.

As described above, the diffuse reflection component calculation unit 304 calculates the diffuse reflection component to be added as an effect of virtual light by using the light source intensity α'corrected by the light source information correction unit 303 and the virtual light vector L'. Therefore, the intensity of the effect of the virtual light added by the addition processing unit 305 changes according to the intensity reliability and the direction reliability of the light source. For example, when the detection reliability of the face region or the organ region is low, the intensity of the virtual light is weakened and the irradiation direction of the virtual light becomes closer to the normal direction, so that the effect of the virtual light is suppressed. Therefore, even if the detection of the face region or the organ region fails, it is possible to suppress the image to which the effect of the virtual light is added from being unnaturally conspicuous.

FIG. 8A schematically shows a rewriting process when the effect of virtual light is not suppressed. Here, the illumination picture shown in the lower right from the center of the image schematically shows the direction and intensity of the virtual light source, and is not actually displayed.

For example, when the size of the detected face region is reduced, at least the reliability Conf_FaceSize based on the face area is lower than that in the case of FIG. 8A, out of the above-mentioned six reliabilitys. As a result, the state in which the intensity of the virtual light is corrected to be low is shown in FIG. 8 (b). In FIG. 8B, the irradiation direction of the virtual light is not corrected, but since the irradiation intensity is reduced, the effect of weakening the shadow of the face is weakened. Therefore, even if the brightness of the face region is erroneously detected, it is possible to suppress the occurrence of unnaturalness due to the addition of the effect of virtual light with an erroneous irradiation intensity.

Further, for example, when the contrast is very strong and the eye region cannot be detected in the shaded portion of the face region, at least the reliability Conf_FaceSize based on the face area is lower than that in the case of FIG. 8A. FIG. 8C shows a state in which the angle of the virtual light is corrected so as to approach the normal direction. In FIG. 8C, the irradiation intensity of the virtual light is not corrected, but the irradiation direction is corrected to be closer to the front of the face, so that the effect of the virtual light in the face area is reflected more. There is. Therefore, even when the accuracy of determining the posture of the face is lowered due to the failure to detect the eye region, it is possible to suppress the occurrence of unnaturalness due to the addition of the effect of virtual light in the wrong irradiation direction.

According to the present embodiment, in an image processing apparatus that applies a rewriting process that adds an effect of virtual light to a feature area detected by image processing, the irradiation intensity of virtual light and the irradiation intensity of virtual light are determined according to the detection reliability of the feature area. / Or the irradiation direction was corrected. Specifically, when the detection reliability of the feature region is low, the correction is made so that the effect of the virtual light is suppressed. Therefore, even when it is necessary to apply the rewriting process based on the detection result having low reliability, it is possible to prevent the rewriting process from becoming unnatural.

(Other embodiments)
In the above embodiment, the feature area to which the rewriting process is applied is the face area of a person. However, the feature area is not limited to the face area, and may be any feature area that can be mechanically detected from the image data.

Further, in the above-described embodiment, the light source intensity α is corrected based on the intensity reliability Conf_power based on the light source reliability. However, the intensity of the virtual light applied to the subject may be controlled by controlling the norm | L | indicating the distance between the subject and the light source instead of the intensity reliability Conf_power. That is, when the strength is weakened, | L'| corrected to extend the distance may be used.

Further, in the above-described embodiment, both the strength reliability and the angle reliability were calculated using six types of reliability. However, strength reliability and angular reliability may be determined based on fewer types of reliability. For example, among the six reliabilitys, the reliability of the irradiation intensity is obtained only based on the reliability that has a large influence on the determination of the irradiation intensity, or the irradiation angle is based only on the reliability that has a large influence on the determination of the irradiation angle. You may ask for the reliability of.

For example, the intensity reliability Conf_power that corrects the irradiation intensity can be obtained based on one or more of the reliability Conf_Luminance based on the brightness of the face region and the reliability Conf_FaceSize based on the area of the face region. Further, the directional reliability Conf_angle for correcting the irradiation angle can be obtained based on the facial organ detection reliability Conf_FacePartsDetect. These are examples, and the strength reliability Conf_power and the directional reliability Conf_angle can each be determined based on one or more of the six types of reliability.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The present invention is not limited to the contents of the above-described embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

101 ... Optical system, 102 ... Image sensor, 103 ... A / D conversion unit, 104 ... Image processing unit, 105 ... Distance map generation unit, 106 ... Face detection unit, 107 ... Obstruction detection unit, 108 ... Recording unit, 109 ... control unit, 110 ... memory, 111 ... operation unit, 112 ... display unit

Claims (13)

A detection means that detects a feature area from image data,
Determining means for determining the irradiation characteristics of virtual light for the characteristic region, and
A correction means for correcting the irradiation characteristics determined by the determination means according to the detection reliability of the feature region, and a correction means.
It has an additional means for adding the effect of the virtual light to the characteristic region based on the irradiation characteristic corrected by the correction means.
The effect of the virtual light of the correction means is weaker in the case of the second reliability, which is lower in reliability than the first reliability, than in the case of the detection reliability of the first reliability. To correct the irradiation characteristics,
An image processing device characterized by this. The image processing apparatus according to claim 1, wherein the irradiation characteristics include an irradiation intensity and an irradiation direction, and the correction means corrects at least one of the irradiation intensity and the irradiation direction. The image processing apparatus according to claim 2, wherein the correction means weakens the effect of the virtual light by reducing the irradiation intensity. The correction means according to claim 2 or 3, wherein the correction means weakens the effect of the virtual light by correcting the irradiation direction so that the virtual light irradiates the subject related to the feature region from the front. Image processing equipment. The detection reliability is according to any one of claims 1 to 4, wherein the angle of the subject based on the detection result of the feature region decreases as the angle of the subject increases with respect to the frontal angle of the subject. Image processing equipment. The image processing apparatus according to any one of claims 1 to 4, wherein the detection reliability decreases as the detection reliability of a portion included in the feature region decreases. The image processing apparatus according to any one of claims 1 to 4, wherein the detection reliability decreases as the area of the feature region becomes smaller. The image according to any one of claims 1 to 4, wherein the detection reliability decreases as the brightness of the feature region becomes lower than the first threshold value and becomes larger than the second threshold value. Processing equipment. Further, it has a detecting means for detecting an area of a shield that hides a part of the feature area from the image data.
The image processing apparatus according to any one of claims 1 to 4, wherein the detection reliability decreases as the proportion of the shield region occupied in the feature region increases. The detection reliability is the reliability based on the angle of the feature region, the reliability based on the area of the feature region, the detection reliability of the portion included in the feature region, the detection reliability of the feature region, and the detection reliability of the feature region. The image processing apparatus according to any one of claims 1 to 4, wherein the reliability is based on one or more of the reliability based on the brightness and the reliability based on the ratio in which the feature region is hidden. Image sensor and
The image processing apparatus according to any one of claims 1 to 10, wherein the effect of virtual light is added to the image data obtained by using the image sensor.
An imaging device having. An image processing method executed by an image processing device.
A detection process that detects a feature area from image data,
A determination step for determining the irradiation characteristics of virtual light for the characteristic region, and
A correction step of correcting the irradiation characteristics determined in the determination step according to the detection reliability of the feature region, and a correction step of correcting the irradiation characteristics.
It has an additional step of adding the effect of the virtual light to the feature region based on the irradiation characteristics corrected in the correction step.
In the correction step, the effect of the virtual light is weaker in the case of the second reliability, which is lower in reliability than the first reliability, than in the case where the detection reliability is the first reliability. To correct the irradiation characteristics,
An image processing method characterized by that. A program for causing a computer to function as each means included in the image processing apparatus according to any one of claims 1 to 10.

Images