JP2021005418A - Image processing apparatus and control method for the same, and program - Google Patents

Abstract

To provide an image processing apparatus that can create an image having a desired shading state according to the shape of an individual subject.SOLUTION: An image processing apparatus according to the present disclosure has first acquisition means that acquires an image including a subject, second acquisition means that acquires information on a normal line of a predetermined area on a surface of the subject, and processing means that provides an image with the effect of virtual light that is light which is virtual based on the normal line information. The processing means provides the effect of virtual light corresponding to a case where the normal line information acquired by the second acquisition means is changed, according to the direction or the attitude of the predetermined area.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing device, a control method thereof, and a program.

Conventionally, in photography, the impression of a subject can be changed to a desired one by adjusting the areas of light and shadow generated in the subject by using auxiliary lighting or a reflector during photography. On the other hand, there is known a technique of adjusting the light and shadow regions generated in a subject by adding a shadow component to the face region of the subject by image processing after shooting (Patent Document 1). By using this technique, it is possible to obtain an image with an impressive pictorial impression or an image with an emphasized stereoscopic effect.

Japanese Unexamined Patent Publication No. 2012-105016

However, the technique disclosed in Patent Document 1 holds in advance a two-dimensional shadow pattern generated in the face region by light from a predetermined direction, and uses the selected shadow pattern as the outline of a person in a captured image or the eyes and nose. It is applied according to the position of. Therefore, the shadow state is limited to the shadow pattern from a predetermined direction, and it is difficult to obtain a natural shadow state according to the shape of each subject.

The present invention has been made in view of the above-mentioned problems of the prior art, and is an image processing device capable of generating an image having a desired shadow state according to the shape of an individual subject and a control method thereof. , The purpose is to provide the program.

In order to solve this problem, for example, the image processing apparatus of the present invention has the following configuration. That is, based on the first acquisition means for acquiring an image including the subject, the second acquisition means for acquiring the normal information of a predetermined region on the surface of the subject, and the normal information, the image is virtualized. It has a processing means for imparting the effect of virtual light which is a target light, and the processing means has normal information acquired by the second acquisition means according to the direction or orientation of the predetermined region. It is characterized in that the effect of the virtual light corresponding to the case where is changed is imparted.

According to the present invention, it is possible to generate an image having a desired shadow state according to the shape of each subject.

A block diagram showing a functional configuration example of a digital camera as an example of an image processing device according to an embodiment of the present invention. Block diagram showing a functional configuration example of the image processing unit according to the first embodiment Block diagram showing a functional configuration example of the rewriting processing unit according to the first embodiment The figure explaining the reflection component by rewriting which concerns on Embodiment 1. The figure explaining the lighting mode which concerns on Embodiment 1. A flowchart showing a series of operations related to the determination process of the rewriting parameter of the first embodiment. The figure explaining the correction example of the normal which concerns on Embodiment 1. The figure explaining the correction characteristic of the normal which concerns on Embodiment 1. The figure explaining the correction characteristic of the angle formed by the direction of a virtual light source and the direction of a normal, and the correction characteristic of the amount of reflected light which concerns on Embodiment 1. The figure explaining the histogram of the angle formed by the direction of a virtual light source and the direction of a normal line which concerns on Embodiment 1.

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following, as an example of the image processing device, an example in which the present invention is applied to an arbitrary digital camera capable of capturing an image and changing the shadow state of the subject will be described. However, the present invention is not limited to a digital camera capable of taking an image, and can be applied to any device capable of acquiring a captured image and changing the shadow state of a subject. These devices may include, for example, mobile phones, game consoles, tablet terminals, personal computers, clock-type and eyeglass-type information terminals, in-vehicle devices and surveillance systems, medical devices, and the like.

(Configuration of Digital Camera 100)
FIG. 1 is a block diagram showing a functional configuration example of the digital camera 100 as an example of the image processing device of the present embodiment. One or more of the functional blocks shown in FIG. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or MPU executing software. You may. It may also be realized by a combination of software and hardware. Therefore, in the following description, the same hardware can be realized as the main body even if different functional blocks are described as the main body of operation.

The photographing optical system 101 includes a lens group such as a zoom lens and a focus lens. The shutter 102 has an aperture function and exposes the image sensor included in the image pickup unit 103 for a predetermined time. The image pickup unit 103 has an image pickup element in which a plurality of pixels having a photoelectric conversion element are arranged in a two-dimensional manner, converts a subject optical image imaged by the photographing optical system 101 into an electric signal, and outputs an image signal. To do. The image sensor may be a CCD, a CMOS image sensor, or the like. The A / D converter 104 is a circuit that converts an analog signal into a digital signal (image data).

The image processing unit 105 performs various predetermined image processing on the image data output from the A / D conversion unit 104. The memory control unit 107 includes a control circuit and controls input / output to the image memory 106 from the rewriting processing unit 114, the face detection unit 113, and the like, which will be described later.

The D / A conversion unit 108 is a circuit that converts a digital signal read from the memory control unit 107 into an analog signal, and outputs the converted analog signal to the display unit 109. The display unit 109 includes a display device such as an LCD, and displays captured image data and a GUI for operating the digital camera 100. The codec unit 110 includes a processing circuit or a processing module, compresses and encodes the image data, and decodes the compressed image data.

The I / F 111 constitutes an interface with the recording medium 112 and controls writing and reading of image data with and from the recording medium. The recording medium 112 includes a non-volatile recording medium such as a memory card or a hard disk, and records captured image data.

The face detection unit 113 includes a processing circuit or a processing module, inputs captured image data, and detects a face region of a person in the image. The rewriting processing unit 114 inputs the captured image data and performs the rewriting processing described later.

The system control unit 50, for example, includes a CPU (or MPU), expands and executes a program stored in the non-volatile memory 121 in the system memory 122 to control each part of the digital camera 100, or performs a line between each part. Controls the data transfer. Further, the system control unit 50 controls each unit of the digital camera 100 in response to an operation signal from the operation unit 120 that receives an operation from the user.

The operation unit 120 includes switches for inputting various operations related to shooting, such as a power button, a still image recording button, and a button for instructing moving image recording start and stop. Further, the operation unit 120 has a menu display button, an enter button, other cursor keys, a pointing device, a touch panel, and the like, and when the user operates these keys and buttons, an operation signal is transmitted to the system control unit 50.

The non-volatile memory 121 is a non-volatile memory such as an EEPROM that stores programs, parameters, and the like. The system memory 122 includes a volatile memory, and temporarily stores constants, variables, a program read from the non-volatile memory 121, and the like for operation of the system control unit 50. The light source device 123 is, for example, a light source device that generates light including a strobe. The distance measuring sensor 124 includes a sensor that measures the distance between the subject and the digital camera 100, and generates and outputs a two-dimensional distance map image of the subject distance information corresponding to each pixel of the image sensor described above.

Next, a basic operation when shooting a subject with the digital camera 100 will be described. First, the image pickup unit 103 photoelectrically converts the light rays incident through the photographing optical system 101 and the shutter 102, and outputs the light rays as analog image signals to the A / D conversion unit 104. The A / D conversion unit 104 converts the image signal input from the image pickup unit 103 into a digital image signal (image data) and outputs the image signal to the image processing unit 105.

The image processing unit 105 performs predetermined image processing, for example, color conversion processing such as white balance, and gamma for the image data input from the A / D conversion unit 104 or the image data input from the memory control unit 107. Performs processing, contour enhancement processing, etc. In addition, the image processing unit 105 calculates a predetermined evaluation value using the result of the face detection processing by the face detection unit 113 and the captured image data. Then, the system control unit 50 performs exposure control and distance measurement control based on the obtained evaluation value. As a result, TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, AWB (auto white balance) processing, and the like are realized.

The image data output from the image processing unit 105 is stored in the image memory 106 via the memory control unit 107. The image memory 106 stores the image data output from the imaging unit 103 and the image data to be displayed on the display unit 109. Further, the display unit 109 displays on a display device such as an LCD according to the analog signal obtained from the D / A conversion unit 108. The codec unit 110 compresses and encodes the image data stored in the image memory 106 based on standards such as JPEG and MPEG. The system control unit 50 records the encoded image data on the recording medium 112 via the I / F 111.

(Structure of image processing unit 105)
Next, an example of the functional configuration of the image processing unit 105 will be described in detail with reference to FIG.

The image data input to the image processing unit 105 is input to the simultaneous processing unit 200. The simultaneous processing unit 200 performs simultaneous processing on the input Bayer RGB image data to generate color signals R, G, and B, respectively.

The WB amplification unit 201 applies a gain to each of the RGB color signals based on the white balance gain value calculated by the system control unit 50, and adjusts the white balance. The RGB signal output by the WB amplification unit 201 is input to the luminance color signal generation unit 202. The luminance color signal generation unit 202 generates a luminance signal Y from the RGB signal, and outputs the generated luminance signal Y to the contour enhancement processing unit 203 and the color signal RGB to the color conversion processing unit 205.

The contour enhancement processing unit 203 performs contour enhancement processing on the luminance signal Y, and outputs image data with enhanced contours (also referred to as edges) to the luminance gamma processing unit 204. The luminance gamma processing unit 204 performs gamma correction on the luminance signal Y whose contour is emphasized, and converts the luminance signal Y according to, for example, the display characteristics of the display unit 109. The luminance gamma processing unit 204 outputs the converted luminance signal Y as the output luminance signal of the image processing unit 105.

The color conversion processing unit 205 converts the RGB signal into a desired color balance by matrix calculation or the like. The color gamma processing unit 206 performs gamma correction on the RGB color signal to convert the color signal according to, for example, the display characteristics of the display unit 109. The color gamma processing unit 206 outputs the converted RGB color signal to the color difference signal generation unit 207. The color difference signal generation unit 207 generates color difference signals RY and BY signals from RGB color signals and outputs them as output color difference signals of the image processing unit 105. The output image signals Y, RY, and BY signals are input to the codec unit 110 via the image memory 106, compressed and encoded, and recorded on the recording medium 112.

Further, the RGB color signal output from the color conversion processing unit 205 is also input to the evaluation value acquisition unit 208. The evaluation value acquisition unit 208 calculates information for analyzing the state of shadows generated on the subject by the environmental light source. The evaluation value acquisition unit 208 calculates, for example, the average brightness information of the subject and the brightness histogram information of the face region as shadow information, and the calculated evaluation value is output from the image processing unit 105.

(Configuration and operation of rewriting processing unit 114)
Further, with reference to FIG. 3, the configuration of the rewriting processing unit 114 and its specific operation will be described. When the rewriting process is selected by the user operation on the operation unit 120, the rewriting processing unit 114 uses a virtual light source corresponding to the lighting mode described later for the image data output from the image processing unit 105. Perform rewriting processing.

The RGB signal conversion unit 301 converts the luminance / color difference signals (Y, BY, RY) read from the image memory 106 into RGB signals and outputs them to the degamma processing unit 302. The degamma processing unit 302 performs an operation (degamma processing) applying the characteristics opposite to the gamma characteristics applied by the luminance gamma processing unit 204 and the color gamma processing unit 206 of the image processing unit 105 to convert them into a linear signal. The degamma processing unit 302 outputs the RGB signals (Rt, Gt, Bt) converted into linear signals to the reflection component calculation unit 309 and the gain processing unit 303.

The gain processing unit 303 outputs an RGB signal (Rg, Gg, Bg) obtained by multiplying each component of the input RGB signal (Rt, Gt, Bt) by a gain (1 / S) according to Equation 1.

Rg = Rt / S
Gg = Gt / S (expression 1)
Bg = Bt / S
However, S is S> 1, and 1 / S is a gain that reduces the brightness of the input signal.

The normal calculation unit 307 calculates the normal map from the subject distance information (that is, the two-dimensional distance information obtained in pixel units of the captured image) acquired from the distance measuring sensor 124. Since a known method can be used for calculating the normal map according to the present embodiment, a detailed description of the calculation method will be omitted, but FIG. 4 will be referred to for the outline of the normal map in the present embodiment. I will explain.

FIG. 4 shows the relationship between the horizontal coordinates on the captured image and the three-dimensional coordinates in which the subject exists. In the present embodiment, the difference ΔD of the distance (depth) D with respect to the difference ΔH in the horizontal direction on the image can be obtained by using the subject distance information. Therefore, the normal calculation unit 307 can calculate the gradient of the surface shape of the subject 401 based on the difference ΔD of the distance with respect to the difference ΔH. Then, by obtaining the normal with respect to the calculated gradient, the normal N in the surface shape of the subject can be calculated. The normal calculation unit 307 calculates the normal corresponding to each pixel of the captured image by the same processing, and outputs the calculated normal as a normal map to the normal correction unit 308.

The normal correction unit 308 corrects the normal N included in the normal map based on the normal correction characteristic acquired from the system control unit 50 in order to approach the shaded state according to the illumination mode described later (normal). Also called correction processing). Then, the corrected normal map is output to the reflection component calculation unit 309. The normal correction characteristic acquired from the system control unit 50 and the normal correction process will be described later.

The reflection component calculation unit 309 calculates the reflection component of the light beam emitted from the virtual light source on the surface of the subject based on the distance K, normal N, and virtual light source parameter (position and intensity of the virtual light source) between the virtual light source and the subject. .. This reflection component can be calculated for each coordinate position in the captured image. For example, the reflection component at each coordinate position is calculated as being inversely proportional to the square of the distance K between the light source and the subject and proportional to the inner product of the normal vector N of the subject and the direction vector L of the light source.

An example of calculating the reflection component of the above-mentioned light beam will be described with reference to FIG. 4 again. In the example shown in FIG. 4, the virtual light source 402 is set at a position separated from the subject 401 by a distance K1. Further, the light beam emitted from the virtual light source 402 in the direction L1 is reflected on the subject surface at the horizontal pixel position H1 (the vertical pixel position is omitted for simplification of description) of the captured image. The reflection component of the light beam at the horizontal pixel position H1 is proportional to the inner product of the normal N1 at the camera coordinates H1 and the direction vector L1 of the virtual light source, and is inversely proportional to the distance K1 between the virtual light source 402 and the subject position. That is, the reflection components (Ra, Ga, Ba) of the light rays by the virtual light source are as shown in Equation 2.

Ra = α × (−L ・ N) / K ^ 2 × Rw × Rt
Ga = α × (−L ・ N) / K ^ 2 × 1 × Gt (Equation 2)
Ba = α × (−L ・ N) / K ^ 2 × Bw × Bt

Here, α is the intensity of the virtual light source, L is the three-dimensional direction vector of the virtual light source, N is the three-dimensional normal vector of the surface of the subject, and K is the distance between the virtual light source and the subject. Further, Rt, Gt, and Bt are RGB signals output from the degamma processing unit 302, and Rw and Bw are parameters for controlling the color of the virtual light source.

The reflection component calculation unit 309 outputs the reflection components (Ra, Ga, Ba) of the light rays by the virtual light source calculated according to Equation 2 to the virtual light source addition processing unit 304. Then, the virtual light source addition processing unit 304 has a light ray reflection component (Ra, Ga) by the virtual light source with respect to the subject area included in the RGB signals (Rg, Gg, Bg) output by the gain processing unit 303 according to the equation 3. , Ba) is added.

Rout = Rg + Ra
Gout = Gg + Ga (Equation 3)
Bout = Bg + Ba

The virtual light source addition processing unit 304 outputs an image signal (Rout, Gout, Bout) to which a reflection component by the virtual light source is added to the gamma processing unit 305. The gamma processing unit 305 performs gamma correction on the RGB input signal in order to match the output characteristics of the display unit 109. The luminance color difference signal conversion unit 306 converts the RGB signal into a luminance / color difference signal and outputs the converted signal.

(A series of operations related to the determination process of rewriting parameters)
Next, a series of operations related to the rewriting parameter determination process will be described with reference to FIGS. 5 and 6. The rewriting parameters in the present embodiment represent the normal correction characteristics used by the normal correction unit 308 and the virtual light source parameters (position and intensity of the virtual light source) used by the reflection component calculation unit 309. The rewriting parameter determination process is a process for determining the parameters required for the rewriting process by the rewriting processing unit 114 and providing them to the rewriting processing unit 114.

Prior to this processing, the system control unit 50 accepts the user to specify the image data to be processed and to set the lighting mode. The lighting mode can be selected from how to illuminate the subject. For example, the user can select one of a plurality of predetermined lighting modes in the menu display (not shown) included in the operation unit 120. Will be done. The lighting mode includes, for example, four modes: 1) manual lighting mode, 2) small face lighting mode, 3) split lighting mode, and 4) Rembrandt lighting mode.

5 (B) to 5 (D) show an example of the shadow state caused by each lighting mode, and FIG. 5 (A) shows a case where the rewriting process is not performed for comparison with the example of other modes. (That is, the input image data to be processed) is shown as an example. FIG. 5B is an example of an image when the rewriting process is performed by setting the “small face illumination mode”. The small face illumination mode is a mode in which a virtual light source is set on the upper side of the subject and light is applied to the subject from the upper side to add a slightly strong shadow around the contour of the face. In the small face illumination mode, it is possible to give the effect of making the subject's face look like a small face.

FIG. 5C is an example of an image when the rewriting process is performed by setting the “split illumination mode”. The split illumination mode is a mode in which a virtual light source is set right next to the subject and the subject is illuminated from the side. In this mode, you can give a dynamic shading effect.

FIG. 5D is an example of an image when the rewriting process is performed by setting the “Rembrandt illumination mode”. The Rembrandt illumination mode is a mode in which a virtual light source is set at a position diagonally 45 degrees with respect to the subject and the subject is illuminated from an oblique direction. In this mode, it is possible to give an effect that emphasizes the three-dimensional effect of the face.

The manual lighting mode described above is a mode in which the user can set the position of the virtual light source via the operation unit 120 regardless of other modes and can arbitrarily set the lighting.

When the above-mentioned illumination mode is set via the operation unit 120, the system control unit 50 executes the relighting parameter determination process shown in FIG. 6 in order to realize the shadow state according to the illumination mode. This process is realized by the system control unit 50 expanding the program stored in the non-volatile memory 121 into the working area of the system memory 122 and executing the program.

In S601, the system control unit 50 determines whether or not the set lighting mode is the manual lighting mode. If the operation information regarding the lighting mode notified from the operation unit 120 does not indicate the manual lighting mode, the system control unit 50 determines that the set lighting mode is not the manual lighting mode. In this case, the system control unit 50 advances the process to S602. On the other hand, when the operation information regarding the lighting mode indicates the manual lighting mode, it is determined that the set lighting mode is the manual lighting mode, and the process proceeds to S607.

In S602, the system control unit 50 performs face detection processing using the face detection unit 113 to acquire the detected face region. The face detection unit 113 executes face detection processing on the image data to be processed to specify a face region in the image. The face detection process according to the present embodiment may be any as long as the face region in the image can be specified, and a known face detection technique can be used, so detailed description thereof will be omitted.

In S603, the system control unit 50 analyzes the state of ambient light in the image data using the evaluation value output from the image processing unit 105. Specifically, the system control unit 50 first acquires the evaluation value of the image corresponding to the acquired face region (the brightness average value of the minute block unit generated by the evaluation value acquisition unit 208). Then, the relationship between the center of the face region and the bright region of the face region is determined by determining which region of the face region divided by a predetermined number is bright (or which is brighter on the left or right) based on the acquired brightness distribution. Specify the direction of ambient light from. For example, when it is determined that the area on the right side of the center is bright when the face area is divided into 3 × 3 areas, it is specified that the ambient light is shining from the right side.

In S604, the system control unit 50 determines the position of the virtual light source used for rewriting according to the direction of the ambient light and the illumination mode obtained in S603. For example, when the illumination mode is "split illumination mode" or "Rembrandt illumination mode", the system control unit 50 determines the direction so that the virtual light source is applied from the same direction as the ambient light. When the illumination mode is the "small face illumination mode", the direction of the virtual light source is determined above the subject regardless of the direction of the ambient light. The system control unit 50 determines the position of the virtual light source based on, for example, the three-dimensional position of the subject obtained from the subject distance information, the distance of the virtual light source predetermined for each shooting mode, and the direction of the virtual light source. To do.

In S605, the system control unit 50 determines the normal correction characteristic according to the set illumination mode. The normal correction characteristic is information indicating how much what kind of normal the normal correction unit 308 corrects (in the normal correction process) (that is, information indicating the amount of correction of the normal with respect to a predetermined normal component). ). In other words, the normal correction characteristic indicates the amount of correction of the reflection component of the light beam by the virtual light source on the surface of a predetermined subject.

Before explaining a specific example of the normal correction characteristic, a specific example of the normal correction processing will be described with reference to FIG. 7. FIG. 7 shows the positional relationship between the subject and the virtual light source, where 701 is a human subject and 702 is a virtual light source used for rewriting. Further, L703 indicates a vector from the subject to the light source, and N704 indicates a normal vector. The intensity of light reflected by the virtual light source is determined by the angle θ formed by the light source vector L703 and the normal vector N704 as described above. The smaller the angle θ formed by the light source vector L703 and the normal vector N704, the stronger the intensity of the light reflected by the subject, and the larger the θ, the weaker the light reflection.

The normal correction unit 308 corrects the normal vector N704 (that is, the normal included in the normal map) so as to obtain a desired shadow state defined by the illumination mode. For example, when the lighting mode is the small face lighting mode, the normal vector of the contour portion of the subject is corrected to make the contour portion a shaded state in which the contour of the face is strongly shaded as shown in FIG. 5 (B). Gives a stronger shade.

In the example of FIG. 7, the normal vector N704 is corrected to the correction normal vector N705 by the normal correction process. That is, the system control unit 50 assumes a subject 706 smaller (small face) than the subject 701, and makes the normal vector N704 a normal vector (that is, a correction normal vector N705) of the small face subject 706. Correct to. This means that the actual subject 701 is shaded by a subject having a smaller face than the actual subject. As described above, in the normal correction process, the shadow of the contour can be changed by changing the angle θ formed by the light source vector L703 and the correction normal vector N705.

The normal correction unit 308 corrects the above-mentioned normal vector based on the normal correction characteristic. The normal correction characteristics in this embodiment are as shown in FIGS. 8A to 8D, for example. FIG. 8A shows the normal correction characteristics in the small face illumination mode. The horizontal axis shows the normal (input) of the input image, and the vertical axis shows the corrected normal (output). .. In the small face illumination mode, in order to add a shadow to the contour part of the face, the x-axis component of the normal vector (x, y, z) of the input image is corrected at a predetermined position with respect to the x-axis component. Is relatively large (on the negative side of the x-axis, it is large on the negative side). As a result, the correction effect as shown in the correction normal vector N705 of FIG. 7 can be obtained.

FIG. 8B shows the correction characteristics in the split illumination mode in which the subject is illuminated from the right side. The normal is not corrected on the right side of the subject (plus side of the x-axis) where the light source is strongly exposed, but the normal is corrected so that the shadow is strongly generated on the left side of the subject (minus side of the x-axis).

Further, FIGS. 8C and 8D show correction characteristics in the Rembrandt illumination mode in which illumination is applied from an angle of 45 degrees. (C) shows the x-axis component of the normal, and (D) shows the correction characteristic of the y-axis component of the normal. Similar to the split illumination mode, the normal is not corrected on the positive side of the x-axis where the light is strong, but the normal is corrected on the negative side of the x-axis where the shadow is strong. In the Rembrandt illumination mode, since the illumination is applied from an angle, the x and y normals are corrected to make it easier to shade.

On the other hand, in S607, the system control unit 50 determines the position and intensity of the virtual light source by a user operation on the operation unit 120.

In S606, the system control unit 50 sets the determined rewriting parameter in the rewriting processing unit 114. More specifically, the system control unit 50 outputs the normal correction characteristic determined in S605 to the rewriting processing unit 114 and causes the internal normal correction unit 308 to use it. Further, the system control unit 50 outputs the position of the virtual light source determined in S604 and, for example, the intensity of the virtual light source predetermined for each illumination mode to the rewriting processing unit 114 and outputs the position to the internal reflection component calculation unit 309. Let me use it. Then, when the setting of the rewriting parameter in the rewriting processing unit 114 is completed, a series of operations related to this processing is completed.

In the present embodiment, the illumination mode is set and the normal of the subject is corrected so as to obtain the shadow state defined for each illumination mode. However, the shadow state may be set by a mode other than the lighting mode. For example, the shadow state desired by the user may be directly input. Further, when the "shadow intensity" is set by the user operation, the system control unit 50 may determine the normal correction characteristic so that the normal correction amount increases as the shadow intensity increases. ..

Further, when the "subject shape" is specified by the user operation instead of the illumination mode, the shadow state may be defined based on the specified subject shape. The shape of the subject here refers to an appearance shape that makes the shape of the subject appear to have changed by changing the shading method without changing the actual shape of the subject. The above-mentioned "small face" is also one of the shapes of the subject. That is, when the shape of the subject is specified by the user, the system control unit 50 determines the shadow state that looks like the shape, and determines the normal correction characteristic so as to approach the shadow state.

Further, in addition to the above embodiment, a target shadow state may be defined according to the orientation and posture of the subject. For example, when the face or body is facing diagonally, the normal correction amount can be increased in order to give a more shaded feeling.

Further, in the present embodiment, the normal vector is corrected according to the illumination mode, but a configuration other than directly correcting the normal vector may be used. For example, it is possible to take a method of correcting the angle θ formed by the normal vector N and the light source vector L shown in FIG. 7 instead of the normal vector. For example, FIG. 9A shows the correction characteristic of the formed angle θ. In FIG. 9A, the correction characteristic is such that the rate of change increases from the point where the formed angle θ exceeds a certain threshold value TH. In this way, by correcting the angle information instead of directly correcting the normal, it may be easier to control the characteristics.

Further, instead of correcting the angle θ formed by the normal vector N and the light source vector L, the characteristic of the amount of reflected light with respect to the formed angle θ may be changed. FIG. 9B is a diagram showing the reflected light amount characteristics of the virtual light source with respect to the angle θ formed by the normal vector N and the light source vector L. It is also possible to control the feeling of shadow by correcting this reflected light amount characteristic. For example, the reflected light amount characteristic shown in FIG. 9B is corrected to the reflected light amount characteristic shown in FIG. 9C. As a result, the amount of reflected light decreases sharply from the stage where the angle θ formed by the normal vector N and the light source vector L is small, so that it is possible to give a strong sense of shadow. Further, by directly controlling the reflection characteristic with respect to the angle θ formed as shown in FIG. 9C, for example, it becomes easy to control the gradient and the density of the shadow.

Further, in the present embodiment, a case where one shadow state is associated with one lighting mode and the normal correction process always using the normal correction characteristic (FIG. 8) is performed will be described as an example. However, a configuration may be adopted in which the normal correction process is adaptively applied to the illumination mode. For example, the system control unit 50 may acquire a histogram of the angle θ formed by the normal N of the face region of the subject and the light source vector L, and control the normal correction process according to this distribution. FIG. 10 shows an example of a histogram of the angle θ formed by the normal N and the light source vector L. FIG. 10A shows a subject in which the formed angle θ is distributed from a high value to a low value. When the angle θ is distributed from a high value to a low value, rewriting gives a natural gradation of shadows. On the other hand, FIG. 10B shows a subject in which the dioptric power is concentrated on a value having a low angle θ. In the case where the frequency is concentrated on a value with a low angle θ, there is information on a steep angle such that the subject itself has a face with very few irregularities, or the normal estimation accuracy is rough and the angle θ is high. It may be difficult to obtain. Generally, when the formed angle θ is distributed in a predetermined range as shown in FIG. 10 (B), it is difficult to give a shadow feeling of a target lighting mode. Therefore, the system control unit 50 determines whether or not the distribution of the formed angle θ is biased (as shown in FIG. 10B), and only when the distribution of the formed angle θ is biased (B). The configuration may be such that the above-mentioned normal correction is performed. This makes it possible to obtain a crisp and shaded image even when the detection accuracy of the normal line is low or when a person with few irregularities on the face is the subject.

Further, the normal is not limited to the normal correction characteristic according to the illumination mode, and the normal may be corrected based on any information as long as the information affects the determination of the shadow feeling. For example, a desired shading feeling may be specified in advance by the user to add a shading area, a shading density, a shading gradient, or the like, and the normal may be corrected so that an image of the shading feeling can be output.

Further, in the present embodiment, the case where the main subject is a person has been described as an example, but the main subject is not limited to a person, and can be applied to other subjects such as animals. Further, in the present embodiment, an example has been described in which the original captured image is multiplied by 1 / S to reduce the gain when shading, and then the reflection component due to rewriting is added. Any configuration may be used as long as the image signal and the reflection component can be added. For example, the luminance component may be subtracted from the region (shadow region) where the amount of reflection of the virtual light source is small.

As described above, in the present embodiment, the normal correction characteristic is determined according to the predetermined illumination mode, and the normal in the surface shape of the subject is corrected based on the normal correction characteristic. The reflection component of the virtual light source in the surface shape is calculated. As a result, it is possible to perform rewriting that emphasizes a desired shadow state, as compared with the case where the light beam of the virtual light source is simply applied to the surface shape of the subject. In other words, it is possible to generate an image having a desired shadow state according to the shape of each subject.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It is also possible to realize the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

50 ... System control unit, 114 ... Rewriting processing unit, 304 ... Virtual light source addition processing unit, 307 ... Normal calculation unit, 308 ... Normal correction unit, 309 ... Reflection component calculation unit

Claims (10)

The first acquisition means for acquiring an image including the subject,
A second acquisition means for acquiring normal information of a predetermined region on the surface of the subject, and
It has a processing means for imparting an effect of virtual light, which is virtual light, to the image based on the normal information.
The processing means is characterized in that it imparts the effect of the virtual light corresponding to the case where the normal information acquired by the second acquisition means is changed according to the orientation or orientation of the predetermined region. Image processing device. The processing means has an effect of further strengthening shadows as an effect of the virtual light corresponding to a case where the normal information acquired by the second acquisition means is changed according to the direction or orientation of the predetermined region. The image processing apparatus according to claim 1, wherein the image processing apparatus is provided. The image processing apparatus according to claim 1 or 2, wherein the predetermined area is a face or body of a person. The processing means corrects the normal information according to the orientation or orientation of the predetermined region, so that the virtual light corresponding to the case where the normal information acquired by the second acquisition means is changed. The image processing apparatus according to any one of claims 1 to 3, wherein the effect of the above is imparted. The processing means corresponds to a case where the normal information acquired by the second acquisition means is changed by correcting the angle formed by the direction of the virtual light and the normal indicated by the normal information. The image processing apparatus according to any one of claims 1 to 3, wherein the effect of virtual light is imparted. The processing means imparts the effect of the virtual light corresponding to the case where the normal information acquired by the second acquisition means is changed by changing the characteristic of the amount of reflected light of the subject by the virtual light. The image processing apparatus according to any one of claims 1 to 3, wherein the image processing apparatus is used. The image processing apparatus according to any one of claims 1 to 6, wherein the first acquisition means is an image pickup means. Further having a third acquisition means for acquiring subject distance information of the image,
The image processing apparatus according to any one of claims 1 to 7, wherein the second acquisition means acquires the normal information based on the subject distance information. The first acquisition process of acquiring an image including the subject, and
A second acquisition step of acquiring normal information of a predetermined region on the surface of the subject, and
It has a processing step of imparting an effect of virtual light, which is virtual light, to the image based on the normal information.
The processing step is characterized in that the effect of the virtual light corresponding to the case where the normal information acquired in the second acquisition step is changed according to the orientation or orientation of the predetermined region is imparted. Control method of the image processing device. A program that causes a computer to function as each means of the image processing apparatus according to any one of claims 1 to 8.