JP7102192B2 - Image processing device, its control method, program - Google Patents

Description

The present invention relates to an image processing device, and more particularly to an image processing device that adds an effect to an image by using a virtual light source.

Conventionally, in photography, the area of light and shadow generated in a subject is adjusted by adjusting the light by an auxiliary lighting or a reflector. This makes it possible to take photographs with various impressions of the subject. Further, as a technique for adjusting these lights after shooting, there is a method of adding highlights and shadow components due to light reflection to the subject area.

This makes it possible to create an impressive pictorial image or an image that emphasizes the stereoscopic effect.

For example, in Patent Document 1, a shadow pattern is applied to a face region of a subject by applying a shadow pattern prepared in advance. As a result, it is possible to obtain a subject photograph having a desirable shadow while suppressing the processing load.

Japanese Unexamined Patent Publication No. 2012-105016

However, in the method of Patent Document 1 described above, a shadow pattern prepared in advance is applied, and it is difficult to delicately control the shadow feeling according to each subject. In particular, when the facial expression or shape of the subject prepared in advance is significantly different from the facial expression of the photographed subject, there is a problem that the shadow is significantly different from the shadow that should actually be generated, causing a sense of discomfort.

The image processing apparatus according to the present invention extracts feature information from an input means for inputting an image, a setting means for setting the characteristics of a virtual light source with respect to the input image input by the input means, and a subject in the input image. The extraction means, the first shape information indicating the shape information of the predetermined subject, and the specific region in which the accuracy of the shape information in the specific region of the predetermined subject is higher than that of the first shape information. When it is determined that the acquisition means for acquiring the second shape information indicating the shape information, the feature information extracted by the extraction means, and the second shape information acquired by the acquisition means are similar. In addition, a generation means for generating a third shape information as the shape information of the subject in the input image based on the first shape information and the second shape information, the characteristics of the virtual light source, and the generation. It is characterized by having an additional means for adding a virtual light effect to the subject in the input image based on the shape information generated by the means.

According to the present invention, when a virtual light source is applied to a subject, it is possible to add highlights and shadows with less discomfort.

It is a block diagram which shows the structure of the digital camera in this invention. It is a block diagram which shows the structure of the image processing part in this invention. It is a block diagram which shows the structure of the rewriting processing part in this invention. It is a figure which shows the flow of the normal generation in the 1st Example of this invention. It is a figure which shows the example of the normal in this invention. It is a figure which shows the calculation of the reflection component by rewriting in this invention. It is a figure which shows the result of rewriting in this invention. It is a figure which shows the flow of the normal generation in the 2nd Example of this invention. It is a figure explaining the subject feature point in the 2nd Example of this invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this embodiment, an example applied to a digital camera as an image processing device will be described.

<First Example>
Hereinafter, the digital camera according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 7.

FIG. 1 is a block diagram showing a configuration example of a digital camera according to an embodiment of the present invention.

In FIG. 1, 100 is the entire digital camera, 101 is a lens group including a zoom lens and a focus lens, and 102 is a shutter having an aperture function. 103 is an image pickup unit composed of a CCD or CMOS element that converts an optical image into an electric signal, 104 is an A / D converter that converts an analog signal into a digital signal, and 105 is output from the A / D converter 104. This is an image processing unit that performs various image processing such as white balance processing, γ processing, contour enhancement, and color correction processing on image data. 106 is an image memory, 107 is a memory control unit that controls the image memory 106, 108 is a D / A converter that converts an input digital signal into an analog signal, 109 is a display such as an LCD, and 110 is a compression codec for image data. -The codec part to be decrypted. 111 is an interface I / F with the recording medium 112, 112 is a recording medium such as a memory card or a hard disk, 113 is a face detection processing unit that detects an area in which a face is reflected in the captured image, and 114 is a captured image. The rewriting processing unit 50 that performs the rewriting processing is a system control unit that controls the entire system of the digital camera 100. 121 is a non-volatile memory such as EEPROM for storing programs and parameters, and 122 is a system memory for expanding constants and variables for operation of the system control unit 50, programs read from the non-volatile memory 124, and the like. .. Reference numeral 123 denotes a physical light source device such as a strobe. The configuration of the digital camera is not limited to the configuration shown in FIG. For example, one piece of hardware may function as a plurality of processing units or control units depending on the program to be executed. On the contrary, a plurality of hardware may cooperate to function as one processing unit, control unit, or the like. Further, various processes may be executed according to a program, or a circuit for performing various processes may be provided.

Next, the basic operation at the time of shooting a subject in the digital camera 100 configured as described above will be described. The image pickup unit 103 photoelectrically converts the light incident through the lens 101 and the shutter 102, and outputs the light as an input image signal to the A / D converter 104. The A / D converter 104 converts the analog image signal output from the image pickup unit 103 into a digital image signal and outputs the analog image signal to the image processing unit 105.

The image processing unit 105 performs color conversion processing such as white balance, γ processing, contour enhancement processing, and the like on the image data from the A / D converter 104 or the image data from the memory control unit 107. Further, the image processing unit 105 performs a predetermined evaluation value calculation process using the face detection result of the face detection unit 113 and the captured image data, and the system control unit 50 controls the exposure based on the obtained evaluation value result. , Performs distance measurement control. As a result, TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, AWB (auto white balance) processing, and the like are performed.

The image data output from the image processing unit 105 is written to the image memory 106 via the memory control unit 107. The image memory 106 stores the image data output from the imaging unit 103 and the image data to be displayed on the display unit 109.

Further, the D / A converter 108 converts the image display data stored in the image memory 106 into an analog signal and supplies it to the display unit 109. The display unit 109 displays on a display such as an LCD according to the analog signal from the D / A converter 108.

The codec unit 110 compresses and encodes the image data recorded in the image memory 106 based on standards such as JPEG and MPEG. The system control unit 50 associates the encoded image data and stores it in the recording medium 112 via the recording interface 111.

The basic operation when shooting a subject has been described above.

In addition to the above basic operations, the system control unit 50 realizes each process of the present embodiment described later by executing the program recorded in the above-mentioned non-volatile memory 124. The program referred to here is a program for executing various flowcharts described later in this embodiment. At this time, the operating constants and variables of the system control unit 50, the program read from the non-volatile memory 121, and the like are expanded in the system memory 122.

Next, the details of the image processing unit 105 will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the image processing unit 105. In addition, various forms can be considered as a concrete configuration for realizing the configurations shown in FIGS. 2 and 3. For example, one piece of hardware may be prepared for each part shown in FIGS. 2 and 3, or one piece of hardware may function as a plurality of parts. Further, a plurality of hardware including hardware other than the image processing unit 105 may jointly function as any part shown in FIGS. 2 and 3.

In FIG. 2, 200 is a simultaneous processing unit, 201 is a WB amplification unit, 202 is a brightness / color signal generation unit, 203 is a contour enhancement processing unit, 204 is a brightness gamma processing unit, 205 is a color conversion processing unit, and 206 is a color. The γ processing unit and 207 are color difference signal generation units.

Next, the processing in the image processing unit 105 will be described. The image signal input from the A / D conversion unit 104 of FIG. 1 is input to the image processing unit 105.

The image signal input to the image processing unit 105 is input to the simultaneous processing unit 200. The simultaneous processing unit 200 performs simultaneous processing on the input Bayer RGB image data to generate color signals R, G, and B. The WB amplification unit 201 applies a gain to the RGB color signal based on the white balance gain value calculated by the system control unit 50 to adjust the white balance. The RGB signal output by the WB amplification unit 201 is input to the luminance / color signal generation unit 202. Luminance / color signal generation unit 202 A luminance signal Y is generated from the RGB signal, the generated luminance signal Y is output to the contour enhancement processing unit 203, and the color signal RGB is output to the color conversion processing unit 205.

The contour enhancement processing unit 203 performs contour enhancement processing on the luminance signal and outputs it to the luminance gamma processing unit 204. The luminance gamma processing unit 204 performs gamma correction on the luminance signal Y and outputs the luminance signal Y to the image memory 106.

The color conversion processing unit 205 converts the RGB signal into a desired color balance by performing a matrix calculation or the like. The color gamma processing unit 206 performs gamma correction on the RGB color signal. The color difference signal generation unit 207 generates color difference signals RY and BY signals from RGB signals.

The image signals Y, RY, and BY signals output to the image memory 106 are compressed and encoded by the codec unit 110 and recorded on the recording medium 200.

Next, the configuration and operation of the rewriting processing unit 114 will be described with reference to FIG. The rewriting process is a process of assuming a virtual light source and giving an effect illuminated by the virtual light source to the image obtained by shooting. The rewriting process in this example is executed at the time of shooting by setting, for example, whether to perform the rewriting process before shooting based on a user operation or the like. Of course, the configuration may be such that the user is asked whether to perform the rewriting process each time the image is taken, or the user may instruct the photographed / recorded image to perform the rewriting process at an arbitrary timing.

When the rewriting process is selected by the user operation, the data output from the image processing unit 105 is input to the rewriting processing unit 114, and the rewriting process is performed by the virtual light source. It should be noted that a plurality of modes may be selected in the rewriting process. In this embodiment, "shadow mode" and "catch light mode" can be selected. These modes will be described later.

FIG. 3 is a block diagram showing the configuration of the rewriting processing unit 114.

In FIG. 3, 301 is an RGB signal conversion unit that converts input luminance / color difference signals (Y, BY, RY) into RGB signals, and 302 is a degamma processing unit that performs degamma processing. Reference numeral 303 denotes a virtual light source addition processing unit that adds a rewriting signal by the virtual light source. Further, 304 is a gamma processing unit that applies a gamma characteristic to an RGB signal, and 305 is a luminance / color difference signal conversion unit that converts an RGB signal into a luminance / color difference signal (Y, BY, RY). Further, 306 is a normal generation unit that generates a normal line (with respect to the subject surface) of the subject, and 307 is a virtual light source reflection component calculation unit that calculates a component reflected by the virtual light source on the subject.

Hereinafter, the operation of the rewriting processing unit 114 having the above configuration will be described. The rewriting processing unit 114 reads out the luminance / color difference signals (Y, BY, RY) recorded in the image memory 106 and uses them as inputs.

The RGB signal conversion unit 301 converts the input luminance / color difference signals (Y, BY, RY) into RGB signals and outputs them to the degamma processing unit 302.

The degamma processing unit 302 calculates the gamma characteristic multiplied by the gamma processing unit of the image processing unit 105 and converts it into linear data. The degamma processing unit 302 outputs the RGB signals (Rt, Gt, Bt) after linear conversion to the virtual light source reflection component calculation unit 307 and the virtual light source addition processing unit 303.

On the other hand, the normal generation unit 306 generates a normal corresponding to the subject. FIG. 4 shows a flow of normal generation performed by the normal generation unit 306.

In FIG. 4, a base normal model representing general normal characteristics for each subject is read. As the base normal model, for example, when the subject is a human, the human-shaped normal model is recorded in the non-volatile memory 121 in advance, and the model corresponding to the subject is read. The base normal model has a low normal resolution (density) and omits detailed normal information such as eyes, nose, and mouth. An example of the normal model is shown in FIG. A plurality of normal models may be recorded for each face orientation. In that case, a model with an orientation based on the orientation of the subject estimated from the photographed subject is read.

In FIG. 5, FIG. 5 (A) shows a photographed subject. FIG. 5B is a normal model corresponding to a person's face. The normal model is held as two-dimensional image data, and has normal vector data for the subject surface for each coordinate position. 501 shows an example of a normal vector. It is assumed that the digital camera of this embodiment has a normal vector as shown in 501 for each pixel. The normal model also includes the coordinate positions of feature points such as eyes and mouth. For example, 502 indicates the position of the feature point of the eye.

Returning to FIG. 4, in step S402, the position information of the feature points of the main subject is extracted and acquired. In this embodiment, since the main subject is a person, the feature points of the person, specifically the face detected by the face detection unit 113 (FIG. 1), and the feature points of the face such as eyes, mouth, nose, and contour. Read the position.

In step S403, the normal model shown in FIG. 5B is fitted based on the feature points of the subject acquired in S402. In the fitting of this embodiment, the normal model is changed so that the positions of the feature points of the face and the normal model are matched as much as possible. Specifically, optimization is performed so that the difference between the feature points of the face and the feature points of the normal model is minimized, and the conversion characteristics of each pixel are determined. Many techniques are known for fitting the shape information to the captured image, and any of these techniques may be used as long as the fitting can be performed. With the processing up to this point, a base normal model fitted to the subject can be obtained. As described above, since the base normal model is a low-precision model that omits detailed normal information such as eyes, nose, and mouth, it is highly possible that the model is relatively close to the subject shape. However, since the normal information about the details cannot be obtained, the process of acquiring the normal model of these details is performed by the following processing.

In step S404, the current rewriting mode is acquired. The rewriting mode is a mode for determining the effect of the rewriting process, and is assumed to be selected in advance by the user operation. As described above, in this embodiment, it is assumed that there are two rewriting modes, "shadow mode" and "catch light mode".

In step S405, a high resolution normal model is acquired according to the rewriting mode. When the rewriting mode is "shadow mode", the high resolution normal parts of "nose" are read. The high-resolution normal vector part is a model that shows only a specific area (a specific organ in the case of a person, etc.) among the more accurate high-resolution normal models.

An example of a high-resolution normal vector part of the “nose” is shown in FIG. 5 (C).

On the other hand, when the rewriting mode is "catch light mode", the high resolution normal parts of the "eyeball" are read. The high-resolution normal vector parts of the "eyeball" are shown in FIG. 5 (D).

Returning to FIG. 4, in step S406, the high-resolution normal part is fitted to the subject, and the high-resolution normal part is combined with the base normal model. Like the base normal, the high-resolution normal parts of this embodiment each include a plurality of feature points, and fitting the feature points so as to match the feature points of the corresponding subject. Examples of the feature points of the high-resolution normal vector parts are shown in 503 and 504. Also, after fitting, the high resolution normal parts are combined with the fitted base normal model. An example of synthesizing the high-resolution normal parts of the "nose" with the base normal is shown in FIG. 5 (E), and an example of synthesizing the high-resolution normal parts of the "eyeball" is shown in FIG. 5 (F). By doing so, it is possible to partially generate high-resolution normal information according to the rewriting mode of the entire subject. The flow of generating normal information by the normal generator 306 has been described above.

Returning to the description of FIG. In the virtual light source reflection component calculation unit 307, the component that the installed virtual light source reflects on the subject based on the distance K between the light source and the subject, the normal information N, the specular reflection direction S of the virtual light source, the virtual light source parameter, and the reflectance of the subject. Is calculated. The virtual light source parameter and the reflectance of the subject are determined by the flow described later.

A specific example of the method of calculating the reflected light component will be described with reference to FIG. In FIG. 6, 601 indicates the subject and 602 indicates the position of the set virtual light source. In this embodiment, the rewriting process is performed in consideration of diffuse reflection and specular reflection as the reflection component of the virtual light source (point light source).

The diffuse reflection component at the horizontal pixel position H1 (vertical pixel position is omitted for simplification of explanation) of the captured image taken by the camera 100 is proportional to the inner product of the normal N1 at the camera coordinates H1 and the direction vector L1 of the virtual light source. However, the value is inversely proportional to the square of the distance K1 between the virtual light source and the subject position. Further, the specular reflection component is proportional to the inner product of the specular reflection direction S with respect to the subject and the direction V (direction of the line of sight) of the camera from the subject position.

Expressing this relationship with a mathematical formula, the subject reflection components (Ra, Ga, Ba) by the virtual light source are as follows.

Here, α is the strength of the virtual light source, L is the three-dimensional direction vector of the virtual light source, N is the three-dimensional normal vector of the subject, and K is the distance between the virtual light source and the subject. Further, S is the specular reflection vector of the virtual light source, V is the line-of-sight direction vector indicating the direction from the camera to the subject position, kd is the diffuse reflectance of the subject, and ks is the specular reflectance of the subject. Further, β is a brilliance coefficient indicating the degree of spread of the reflected light, and as this value increases, the specular reflection characteristic becomes steep. Rt, Gt, and Bt are captured RGB data output from the degamma processing unit 302. Further, Rw and Bw are parameters for controlling the color of the virtual light source. It is possible to set a plurality of virtual light sources, and it is possible to control the parameters of diffuse reflection and specular reflection for each virtual light source.

At least one of the intensity of the virtual light source, the position of the virtual light source, the diffuse reflectance of the subject, the mirror reflectance of the subject, the brilliance coefficient, the color of the virtual light source, and the irradiation range of the virtual light is set in advance by user operation. It may be set automatically by the digital camera 100. For example, in the catch light mode, it is conceivable to set the position of the virtual light source in front of the subject and set a large mirror reflectance and brilliance coefficient.

The reflection components (Ra, Ga, Ba) by the virtual light source calculated as described above are output to the virtual light source addition processing unit 304.

The virtual light source addition processing unit 303 performs the following processing for adding virtual light source components (Ra, Ga, Ba) to the subject area.
Rout = Rg + Ra
Gout = Gg + Ga
Bout = Bg + Ba

The image signals (Rout, Gout, Bout) output from the virtual light source addition processing unit 303 are input to the gamma processing unit 304. The gamma processing unit 304 performs gamma correction on the RGB input signal. The color difference signal generation unit 305 generates luminance Y, color difference signals RY, and BY signals from RGB signals. The above is the operation of the rewriting processing unit 114.

The system control unit 50 stores the luminance / color difference signal output by the rewriting correction unit 114 in the image memory 106 under the control of the memory control unit 107, and then compresses and encodes it in the codec unit 110. Further, recording is performed on the recording medium 112 via the I / F 111.

The result image after the rewriting process is shown in FIG. FIG. 7A is an example in which the subject is illuminated with light in the “shadow mode”. By relatively increasing the resolution of the normal of the nasal region, it is possible to strengthen the contrast between light and shade caused by the nasal region, which is an important organ in the shadow of the face, and enhance the effect of shading. .. On the contrary, by lowering the resolution of the normal for areas other than the nose area, it is possible to reduce adverse effects such as unnatural shadows and highlights due to the difference between the shape of the subject and the shape of the normal. Will be able to.

On the other hand, FIG. 7B is an example in which the subject is illuminated with light in the “catch light mode”. By brightening the entire subject and relatively increasing the resolution of the normal of the eyeball, it is possible to see a clear catch light as shown in 701. On the contrary, by making the resolution of the normal line relatively low in the area other than the eyeball area, it is possible to reduce the harmful effects such as unnatural shadows and highlights due to the difference between the shape of the subject and the shape of the normal line. Will be able to.

As described above, in this embodiment, the normals having high resolution are partially generated according to the mode of the rewriting process. This makes it possible to obtain an effect suitable for the purpose of rewriting while reducing the harmful effects caused by the difference between the actual subject and the normal.

In this embodiment, the high-resolution normals are partially generated according to the rewriting mode, but the information for determining the generation of the high-resolution normals is not limited to the rewriting mode information. Not limited to the rewriting mode, any information can be used as long as it is a parameter that determines the rewriting characteristics such as the characteristics of reflected light (specular reflection, diffuse reflection), the irradiation range of light, the intensity, and the type of subject to which the light is applied. do not have. For example, in rewriting, if the gain of the specular reflection component is stronger than that of diffuse reflection, add high-resolution normals of "nose" and "cheek" to enhance the highlight effect of the specular reflection component. It is also possible to control it so that it is output. In addition, when the light is applied only to the eyes, it is possible to add a high-resolution normal of the "eyeball" to control the effect of the catch light to be stronger.

In addition, when the gain of the specular reflection component is almost 0 and the gain of the diffuse reflection component is large, the reflected light is only the base normal without adding a partial high-resolution normal so that the light can rotate as a whole. It is also possible to control to produce components. By using the base normal with low resolution, it is possible to produce the effect of shining diffused light with less shadow.

Further, in the above embodiment, the case where there are two rewriting modes, "shadow mode" and "catch light mode", has been described, but the rewriting mode is not limited to these. Any rewriting mode may be used as long as the method of partially adding normal information (or increasing the resolution of the normal) is adopted according to the rewriting mode. For example, by adding a "mode to make the lips glossy" and adding a high-resolution normal of the shape of the lips, it is possible to add a specular reflection component to the lips and make them glossy. It will be possible.

Further, in this embodiment, an example of synthesizing a high-resolution normal with a low-resolution normal model has been described, but a low-resolution normal of a region excluding the nose and eyeball regions is synthesized with a high-resolution normal model. You may. In this case, processing is performed to reduce the resolution of some normals in the normal model of the entire subject.

Further, in this embodiment, an example of fitting the normal model to the subject has been described, but the information held as the model may be any information as long as it indicates the shape characteristics of the subject. For example, it may be three-dimensional wire frame information indicating the three-dimensional shape information of the subject. In this case, the subject is fitted based on the information on the feature points of the wire frame, and the normal is calculated from the angle information of each point on the wire frame.

Further, in this embodiment, the case where the normal vector information is recorded as image data has been described, but the method is not limited to this method as long as the normal vector can be generated. For example, a three-dimensional three-dimensional shape model may be held instead of the normal model, the three-dimensional three-dimensional shape may be fitted to the subject, and the normal information may be generated from the three-dimensional three-dimensional shape model.

<Second Example>
Hereinafter, the digital camera according to the second embodiment of the present invention will be described with reference to FIGS. 8 to 9.

In the first embodiment, an example of selecting the high-resolution normal information to be added based on the rewriting parameters has been described, but in the second embodiment, the subject and the feature points of the high-resolution normal parts match. Determine whether to use high resolution normal information based on the degree. That is, if it is determined that the feature points of the subject and the high-resolution normal parts are similar to some extent, the high-resolution normal information is used.

Since the overall configuration of the digital camera in the second embodiment is the same as that described in the first embodiment (FIGS. 1 to 3), the description thereof will be omitted. In the second embodiment, since the flow in which the normal generation unit (306 in FIG. 3) generates a normal is different from that in the first embodiment, the operation flow of the legal system generation unit 306 will be described in detail with reference to FIG. ..

In FIG. 8, in S801 to S803, the base normal model is read and the base normal is fitted to the subject. Since this process is the same as S401 to 403 in FIG. 4, the description thereof will be omitted.

In step S804, one of a plurality of high-resolution normal vector parts is read. As high-resolution normal vector parts, prepare "eyeball", "nose", "mouth", "cheek", etc. and read them in order. In this embodiment, the case where the normal of the "mouth" is first read will be described as an example.

In step S805, the positional relationship between the feature points of the subject and the feature points of the normal parts is compared. This will be described with reference to FIG. 9 using an example of a subject and a normal. 9 (A) and 9 (B) show photographed subject images, respectively. In FIG. 9, white circles 901 and 902 are the positions of the feature points of the "mouth". Also, FIG. 9B shows the base normal and the high resolution "mouth" normal. 903 indicates the position of the "mouth" feature point on the normal.

At this time, the shapes of the feature points of the normal and the feature points of the photographed subject are compared. For example, in the case of the example of FIG. 9, the feature point 901 of the mouth of the photographed subject and the feature point 903 of the normal line are compared. The comparison method normalizes the size by the distance between the feature points at both ends of the mouth, and then calculates the relative distance between the feature points 901 and 902 of the captured image and the feature points 903 of the normal. When the sum of the distances between all the feature points is equal to or less than a predetermined threshold value, it is determined that the shape of the feature points in the captured image is the same as the normal. If it is equal to or more than the threshold value, it is judged that the shape of the photographed image and the normal line are different. In the case of the examples 901 and 902, the shape of the feature points of the normal is closed at the "mouth". Therefore, it is determined that the feature point 903 of the normal with the "mouth" closed in the same manner has the same shape as the normal. On the other hand, in the case of 902 with the "mouth" open, it is determined that the feature points 902 of the subject and the feature points 903 of the normal have different shapes.

In step S806, if it is determined in step S805 that the shapes of the feature points of the subject and the normal are the same, the process proceeds to step S807, and if it is determined that the shapes of the feature points of the subject and the normal are different, step S807 is skipped. Then, the process proceeds to step S808.

In step S807, a high-resolution normal (for example, a high-resolution normal of the mouth) is fitted to the subject and combined with the base normal. This process is the same as the process of 406 in FIG.

In step S808, it is determined whether all the high resolution normal vector parts have been processed. The above has been described by taking the case where the high-resolution normal vector is the "mouth" as an example, but there are multiple high-resolution normal vector parts such as "eyeball (eye)" and "nose" in addition to the "mouth". The process returns to step S804 and repeats until all these high-resolution normal vector parts are processed in the same manner.

As described above, in this embodiment, when the rewriting process is performed, the high-resolution normal parts and the feature points of the subject are compared, and only the high-resolution normal parts having similar shapes of the feature points are used. As a result, it is possible to reduce adverse effects such as unnatural shadows and highlights due to the difference between the shape of the subject and the shape of the normal.

<Other embodiments>
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiment is supplied to the system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program. This is the process to be executed.

Claims (14)

Input means for inputting images and
A setting means for setting the characteristics of the virtual light source with respect to the input image input by the input means, and
An extraction means for extracting feature information from a subject in the input image, and
The first shape information indicating the shape information of a predetermined subject and the shape information of the specific region in which the accuracy of the shape information in a specific region of the predetermined subject is higher than that of the first shape information are shown. An acquisition means for acquiring the second shape information,
When it is determined that the feature information extracted by the extraction means and the second shape information acquired by the acquisition means are similar, the first shape information and the second shape information are added. Based on this, a generation means for generating a third shape information as the shape information of the subject in the input image, and
An image processing apparatus comprising: an additional means for adding a virtual light effect to a subject in the input image based on the characteristics of the virtual light source and the shape information generated by the generation means. .. The image processing apparatus according to claim 1, wherein the shape information is normal information or three-dimensional shape information with respect to a subject surface. The image processing apparatus according to claim 1 or 2, wherein the generation means determines which of the subject areas is to be a specific area based on the characteristics of the virtual light source. The image processing apparatus according to any one of claims 1 to 3, wherein the specific region is a region showing a person's nose. The image processing apparatus according to claim 1 or 3, wherein the specific region is a region showing a person's eyes. The image processing apparatus according to claim 1 or 3, wherein the specific region is a region showing the lips of a person. The image processing apparatus according to claim 1, wherein the characteristic of the virtual light source is a mode showing an effect by irradiating the virtual light. The mode includes a first mode of adding a shadow to a person in the input image.
The image processing apparatus according to claim 7, wherein in the first mode, the second shape information indicates the shape information of a region indicating a person's nose. The mode includes a second mode that adds a catch light effect to the eyes of a person in the input image.
The image processing apparatus according to claim 8, wherein in the second mode, the second shape information indicates the shape information of a region showing a person's eyes. The mode includes a third mode of adding luster to the lips of a person in the input image.
The image processing apparatus according to claim 9, wherein in the third mode, the second shape information indicates the shape information of a region indicating a person's lips. The item according to any one of claims 1 to 10, wherein the characteristic of the virtual light source is information including any one of a position, an intensity, an irradiation range, a color of the light source, and a reflection characteristic of a subject of the virtual light source. Image processing device. When the generation means determines that the second shape information and the feature information extracted by the extraction means are not similar, the shape of the subject in the input image is formed by using the second shape information. The image processing apparatus according to any one of claims 1 to 11, wherein no information is generated. The input process for inputting images and
A setting process for setting the characteristics of the virtual light source with respect to the input image input in the input process, and a setting process.
An extraction process that extracts feature information from the subject in the input image,
The first shape information indicating the shape information of a predetermined subject and the shape information of the specific region in which the accuracy of the shape information in a specific region of the predetermined subject is higher than that of the first shape information are shown. The acquisition process for acquiring the second shape information and
When it is determined that the feature information extracted in the extraction step and the second shape information are similar, the subject in the input image is based on the first shape information and the second shape information. And a generation process that generates a third shape information as the shape information of
An image processing apparatus comprising: an additional step of adding a virtual light effect to a subject in the input image based on the characteristics of the virtual light source and the shape information generated in the generation step. Control method. An image processing device that can be executed by a computer that causes the computer to function as the image processing device according to any one of claims 1 to 12.

Images