JP2017138730A - Image processing device, imaging apparatus, control method and program of image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To control processing to a highlight area by using illumination environmental information at the time of imaging and illumination environmental information at the time of relighting.SOLUTION: An image processing device which sets a virtual light source with respect to an image and generates a relighting image obtained by emitting light to a subject image in the image from the virtual light source includes: detection means which detects a highlight area in the image on the basis of a luminance value of pixels constituting the subject image; estimation means which estimates the intensity of environmental light in the image; and processing means which corrects the highlight area in the image on the basis of comparison between the intensity of the light of the virtual light source and the estimated intensity of the environmental light in the processing of generating the relighting image. The processing means does not correct the pixel value of the highlight area when the intensity of the environmental light is larger than the intensity of the light of the virtual light source, and corrects the pixel value of the highlight area when the intensity of the environmental light is equal to or less than the intensity of the light of the virtual light source.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, an image processing apparatus control method, and a program.

  There is a method called relighting that creates an image that has been shot in a lighting environment different from that at the time of shooting. When performing relighting, if a highlight area caused by the lighting environment at the time of photographing remains, the image after relighting may become unnatural. On the other hand, a technique for removing a highlight area caused by an illumination environment at the time of photographing has been proposed (see Patent Document 1).

JP 2004-21388 A

  By using the above-described technique, it is possible to remove a highlight area caused by an illumination environment at the time of shooting. However, in such a case, the highlight area generated by the lighting environment at the time of shooting is always removed, so that it is not possible to keep the highlight area generated by the lighting environment at the time of shooting left.

  Therefore, the present invention makes it possible to control processing for a highlight region using illumination environment information at the time of photographing (hereinafter, ambient light information) and illumination environment information at the time of relighting (hereinafter, virtual light source information).

The invention for solving the above-described problem relates to an image processing apparatus that sets a virtual light source for an image and generates a relighting image in which light is emitted from the virtual light source to a subject image in the image. The device
Detecting means for detecting a highlight area in the image based on a luminance value of a pixel constituting the subject image;
Estimating means for estimating the intensity of ambient light in the image;
In the process of generating the relighting image, a processing unit that corrects a highlight area in the image based on a comparison between the light intensity of the virtual light source and the estimated intensity of the ambient light,
The processing means does not correct the pixel value of the highlight region when the intensity of the ambient light is larger than the intensity of the light of the virtual light source, and the intensity of the ambient light is equal to or less than the intensity of the light of the virtual light source. In this case, the pixel value in the highlight area is corrected.

  Using the ambient light information and the virtual light source information, it is possible to control processing for a highlight area generated in the subject.

1 is a block diagram showing a configuration example of an image processing apparatus corresponding to an embodiment of the invention. The figure which shows an example of a function structure of the image process part corresponding to embodiment of invention. The flowchart which shows an example of the process corresponding to embodiment of invention. The figure which shows an example of the coefficient calculated | required according to the normal line template corresponding to embodiment of invention, and the incident direction of environmental light, and the incident direction of the light of a virtual light source. The figure for demonstrating the coordinate system corresponding to embodiment of invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The image processing apparatus corresponding to the present embodiment sets a virtual light source for an image, and generates a relighting image in which light is emitted from the virtual light source to a subject image in the image. At that time, a highlight region generated by light (including ambient light and light from a virtual light source) irradiated from the light source to the subject is controlled. Here, the highlight area refers to a part of the subject area that appears white due to the light emitted from the light source being reflected by the surface of the subject.

[Embodiment 1]
Hereinafter, an image processing apparatus corresponding to a first embodiment of the invention will be described with reference to the accompanying drawings. In the present embodiment, a digital camera will be described as an example of the image processing apparatus. However, the embodiment of the image processing apparatus includes any information processing terminal or imaging device such as a personal computer, a mobile phone, a smartphone, a PDA, a tablet terminal, and a digital video camera in addition to the digital camera.

  FIG. 1 is a block diagram showing the configuration of the digital camera 100. In FIG. 1, the digital camera 100 has the following configuration. The lens group 101 includes a zoom lens and a focus lens. The lens group 101 may be configured to be attachable as an interchangeable lens when the digital camera 100 is a single-lens reflex digital camera. The shutter 102 has an aperture function. The imaging unit 103 includes a CCD, a CMOS element, or the like that converts an optical image into an electrical signal. The A / D converter 104 converts an analog signal into a digital signal. The image processing unit 105 performs various image processing such as white balance processing, γ processing, and color correction processing on the image data output from the A / D converter 104. The image memory 106 holds an image processed by the digital camera 100. A memory control unit 107 controls the image memory 106 to control image writing and reading. The D / A converter 108 converts the input digital signal into an analog signal. The display unit 109 is a display composed of an LCD, an organic EL, or the like. The codec unit 110 compresses and decodes image data.

  A medium interface (I / F) 111 is an interface with the storage medium 200. The storage medium 112 is a storage medium such as a memory card or a hard disk, and is detachable from the digital camera 100 via a medium I / F 111. The system control unit 50 controls the entire system of the digital camera 100. The operation unit 120 receives various operation instructions from the user. The power switch 121 is a switch that controls on / off of the power of the digital camera 100. The power control unit 122 detects the presence / absence of a battery, the type of battery, and the remaining battery level. The power supply unit 123 is a battery that supplies power to the digital camera 100. The non-volatile memory 124 is an electrically erasable / storable storage medium such as an EEPROM.

  The system timer 125 measures the time used for various controls and the time of a built-in clock. The system memory 126 is a memory for expanding constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 124, and the like. The distance measuring sensor 127 measures the distance to the subject and outputs distance information corresponding to the pixel unit of the photographic pixel as a two-dimensional distance map image.

  Next, a basic processing flow at the time of shooting an object in the digital camera 100 configured as described above will be described. The imaging unit 103 photoelectrically converts light incident through the lens group 101 and the shutter 102 and outputs an image signal to the A / D converter 104. The A / D converter 104 converts the analog image signal output from the imaging unit 103 into a digital image signal (image data) and outputs the digital image signal to the image processing unit 105.

  The image processing unit 105 performs color conversion processing such as white balance and γ processing on the image data from the A / D converter 104 or the image data read from the image memory 106 via the memory control unit 107. Any image data among the Bayer RGB data and the luminance / color difference signals Y, RY, and BY is output. The image data output from the image processing unit 105 is written into the image memory 106 via the memory control unit 107. The image processing unit 105 performs predetermined calculation processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation result. Thus, TTL (through the lens) AF (autofocus) processing, AE (automatic exposure) processing, and the like are performed. The image processing unit 105 further analyzes the captured image data, estimates the light source, and performs AWB (auto white balance) processing based on the estimated light source.

  The image memory 106 stores image data obtained by imaging by the imaging unit 103 and image data to be displayed on the display unit 109. The D / A converter 108 converts the image display data stored in the image memory 106 into an analog signal and supplies the analog signal to the display unit 109. The display unit 109 performs display according to an analog signal from the D / A converter 108 on a display such as an LCD. The codec unit 110 compresses and encodes the image data recorded in the image memory 106 based on a still image or moving image standard such as JPEG or MPEG.

  In addition to the basic operations described above, the system control unit 50 implements each process of the present embodiment, which will be described later, by executing the program recorded in the nonvolatile memory 124 described above. The program here is a program for executing various flowcharts described later in the present embodiment. At this time, constants, variables for operation of the system control unit 50, programs read from the nonvolatile memory 124, and the like are expanded in the system memory 126. The block configuration and basic operation of the digital camera 100 have been described above.

  In the digital camera 100 of FIG. 1 described above, each block may be configured by hardware using a dedicated logic circuit or memory except for a physical device such as an image sensor or a display element. Alternatively, the processing program stored in the memory may be configured by software by a computer such as a CPU executing the processing program. In the above description, the digital camera 100 is described as including the lens group 101, the shutter 103, and the imaging unit 103. However, as an embodiment of the invention, an image editing apparatus, an image processing apparatus, or an information processing apparatus for processing a highlight area may be realized with a configuration that does not have an imaging function.

  Next, an outline of highlight area processing corresponding to the embodiment of the invention will be described with reference to FIGS. FIG. 2 is a diagram illustrating an example of a functional configuration of the image processing unit 105 corresponding to the embodiment of the invention. FIG. 3 is a flowchart showing an example of processing in the image processing unit 105 corresponding to the embodiment of the invention. FIG. 4A is a diagram illustrating an example of a normal template for explaining processing in the image processing unit 105 corresponding to the embodiment of the invention. Hereinafter, the processing will be described according to the flow of the flowchart of FIG. The processing corresponding to the flowchart can be realized by executing a program (stored in a ROM or the like) corresponding to one or more processors functioning as the image processing unit 105, for example.

  First, when an image signal output from the imaging unit 103 is input to the image processing unit 105 as image data via the A / D converter 104, the subject region detection unit 201 detects a subject image from the input image data in S301. To do. In this embodiment, a case where a person is imaged as a subject will be described, and in particular, a face area is detected as a subject image. A known method can be used for the face area detection process. As the method, for example, there is a method of detecting a face region by detecting organs such as eyes and mouth. In step S302, the normal line information acquisition unit 202 acquires normal line information in the face area detected by the subject area detection unit 201 in step S301 with reference to the normal line template as illustrated in FIG. In step S303, the highlight area detection unit 203 detects an area having a high luminance in the face area as a highlight area. In step S304, the ambient light information estimation unit 204 estimates ambient light source information from the normal information acquired in step S302 and the highlight area detected in step S303. In step S305, the virtual light source information setting unit 205 sets light source information of the virtual light source based on the user input. In step S306, the highlight region processing unit 206 performs highlight region processing based on the ambient light source information estimated in step S304 and the virtual light source information set in step S305.

  The above is the outline of the processing of the highlight area corresponding to the present embodiment. Here, the image signal output from the imaging unit 103 is input to the image processing unit 105 as image data through the A / D converter 104. It was explained as a process in the case of. However, embodiments to which the present invention is applicable are not limited to this. For example, after the compressed image data read from the storage medium 112 via the medium interface 111 is decompressed by the codec unit 110 and stored in the image memory 106, the image processing unit 105 performs processing on the image data stored in the image memory 106. However, highlight processing may be performed.

  Next, details of the processing from S302 to S306 will be described. First, in S302, the normal information of the face area of the photographed image is acquired. In this process, a normal template as shown in FIG. 4A may be used. A normal template is an image that represents the face surface by RGB pixel values corresponding to x, y, and z components of each normal vector of the face surface that is an object, and is also referred to as a normal map. . The normal template in FIG. 4A shows normal information when the face is facing the front, but may include normal information on the side of the face. In an actually captured image, the face is not always facing the front, so it is necessary to calculate normal information with reference to a normal template that matches the orientation of the captured face. In order to perform this calculation, it is only necessary to know angle information indicating how many times the direction of the photographed face is tilted from the front direction. As a method for calculating the angle information, a known technique can be used. For example, as disclosed in Japanese Patent Application Laid-Open No. 2014-115859, there is a method of extracting an organ such as a right eye, a left eye, and a nose from an image, and calculating an angle at which the face is directed from the relative positional relationship thereof. . In the present embodiment, the normal information of the captured face region can be acquired by rotating the normal template according to the angle information calculated by the method.

  Next, in the process of detecting a highlight area in S303, the highlight area detection unit 203 detects a high luminance area in the face area as a highlight area. As a method for detecting the highlight area, there is a method in which the luminance value of each pixel constituting the face area detected in S301 is compared with a predetermined threshold Lth, and a pixel area having a luminance value equal to or higher than the threshold is detected as the highlight area. is there. The threshold Lth may be, for example, the average luminance of the face area, but is not limited to this method. When the pixels extracted as high luminance pixels are adjacent to each other, these pixels are collectively treated as one highlight area. In this way, an area composed of pixels having a luminance of Lth or higher among the pixels of the face area is extracted as the highlight area.

  Next, in the process of estimating the ambient light information in S304, the ambient light information estimation unit 204 estimates the ambient light information at the time of image capturing. In this embodiment, the incident direction (position) and intensity of ambient light are estimated. First, a method for estimating the incident direction of ambient light will be described with reference to FIG. First, as shown in FIG. 5A, the highlight region is generated based on the ambient light, as shown in FIG. 5A, from the light source of the ambient light (for example, the sun or indoor lighting) to the object (in this embodiment, the human being who is the subject). In this case, the incident angle θi when the light is incident on the surface of the face and the reflection angle θr when the light is reflected from the surface are substantially equal. Such a reflection form is generally referred to as “specular reflection”, and in the portion of the image data where the reflected light that is specularly reflected is directly incident on the digital camera 100, the light source color is more dominant than the subject itself. In the present embodiment, the incident direction VL1 of the ambient light is estimated based on the angle θr between the normal line of the subject and the digital camera 100 using the fact that θi = θr during specular reflection.

FIGS. 5B and 5C are diagrams illustrating a coordinate system in the present embodiment. As shown in FIG. 5B, when the subject 501 is imaged by the digital camera 100, a three-dimensional space (hereinafter referred to as a camera coordinate system) with the imaging unit 103 of the digital camera 100 as the origin (O _camera ) is set. . The coordinates of an arbitrary point constituting the surface of the subject's face are represented by Xn = (xn, yn, zn) in the camera coordinate system (n is a natural number of 1 or more). At this time, each point to which coordinates are given in the camera coordinate system is the origin of the normal vector of the plane Pn including the point. For example, if the coordinates of the point X1 on the face of the subject in the camera coordinate system are (x1, y1, z1), the direction Xv1 of the point X1 in the camera coordinate system is expressed as a vector by (x1, y1, z1). be able to. The point X1 coincides with the origin (Olocal) in the local coordinate system set for the subject, and the normal vector Nx1 of the plane P1 including the point X1 is Nx1 = (xnl, yn1, zn1) in the local coordinate system. It becomes. It is assumed that the local coordinate system is set in the camera coordinate system, and each axis direction coincides with the camera coordinate system. In addition, the direction of the digital camera 100 viewed from the local coordinate system can be expressed by -Xv1 = (-x1, -y1, -z1).

Therefore, when the plane P1 is included in the highlight region, the incident angle θi of the ambient light coincides with the intersection angle between the normal vector Nx1 and the vector Xv1 of the point X1 in the camera coordinate system. This corresponds to θr of 5 (a). At this time, the vector VL1 indicating the incident direction of the ambient light can be regarded as a vector obtained by rotating Xv1 around the normal vector Nx1 by 180 degrees with respect to the point X1, and is obtained by the following equation (1). it can.
VL1 = Nx1 (Nx1 · (−Xv1)) − [− Xv1−Nx1 (Nx1 · (−Xv1))] (1)

  In the above, n (n · x) represents the projection of the vector x onto the unit vector n. In this way, the incident direction VL1 of the ambient light can be obtained by using the normal template obtained in S302. The above method is described as an example, and the incident direction of the ambient light may be calculated by a method different from the above method. In addition, when a plurality of pixels are included in the highlight region, a value obtained by averaging the calculation results of Expression (1) calculated for each pixel may be used as the incident direction of the ambient light. Alternatively, the average of the normal vectors of each pixel may be calculated first, and VL1 may be obtained by the above calculation using the average normal vector as the normal vector of the plane having the strongest intensity. Alternatively, the information on the incident direction of the ambient light may be given in advance at the time of shooting with the digital camera 100.

Next, a method for estimating the intensity of ambient light will be described. As the intensity of the ambient light, the difference in luminance between the highlight area detected in S303 and the peripheral area of the detected highlight area is used. For example, the intensity of ambient light is obtained as follows. First, assuming that the number of pixels included in the highlight region is i ₁ and the luminance value of each pixel is Yh _i (1 ≦ i ≦ i ₁ ), the average luminance Yh of the highlight region is expressed by Equation (2). I want.
Yh = ΣYh _i / i ₁ (2)

Further, when the region distance is Dth or less from the highlight region and the peripheral region, the average luminance Yl of the peripheral region, the value of luminance of each pixel included in the pixel number j ₁ and the peripheral region included in the peripheral area Yl _{Using j} (1 ≦ j ≦ j ₁ ), it is obtained as shown in Equation (3).
Yl = ΣYl _j / j ₁ (3)
The difference between Yh and Yl obtained by the equations (2) and (3) (see the equation (4)) is used as the ambient light intensity L1.

L1 = Yh−Yl (4)
In step S <b> 305, the virtual light source information setting unit 205 sets light source information of the virtual light source based on a user instruction. The light source information of the virtual light source to be set includes the incident direction (position) of light emitted by the virtual light source and the intensity of light of the virtual light source (hereinafter referred to as the incident direction of the virtual light source and the intensity of the virtual light source for simplicity). Suppose that Here, an operation from the user of the digital camera 100 is accepted via the operation unit 120. Specifically, a relighting process is selected from a menu (not shown) by a user operation on the operation unit 120, and information used in the relighting process is input. In this embodiment, for simplification of description, it is assumed that the user inputs the incident direction (position) VL2 and the intensity L2 of the virtual light source (a method of selecting from presets may be used).

  In step S306, based on the light source information of the ambient light and the light source information of the virtual light source, the highlight area processing unit 206 processes the highlight area in the image data. In the present embodiment, the pixel value of the pixel included in the highlight area generated by the ambient light is corrected as the processing. Hereinafter, a specific processing flow will be described.

  In processing the highlight area, first, the ambient light intensity L1 is compared with the virtual light source intensity L2. When L1 is larger than L2 (L1> L2), the intensity of the ambient light is stronger than the light of the virtual light source, and the pixel value in the highlight area generated by the ambient light is not corrected. On the other hand, when the intensity of the environmental light is equal to or lower than the intensity of the virtual light source (L1 ≦ L2), the pixel value in the highlight area caused by the environmental light can be corrected. At this time, the incident direction VL1 of the ambient light is further compared with the incident direction VL2 of the virtual light source. When the directions of VL1 and VL2 are similar (the positions of the light sources are close to each other), the pixel value in the highlight area caused by the ambient light is not corrected. On the other hand, when the directions of VL1 and VL2 are not similar (the positions of the light sources are far from each other), the pixel value in the highlight area caused by the ambient light is corrected. The correction may be performed, for example, by subtracting a value based on the ambient light intensity L1 from the pixel value in the highlight area. Further, the pixel value of the highlight area may be corrected using the pixel value of the peripheral area of the highlight area. For example, highlighting is performed by using an average value of pixels in the peripheral area or a value obtained by further processing the average value. You may replace the pixel value of an area | region. In addition, a known method for reducing or removing the highlight area can be used.

The determination of the similarity between the directions of VL1 and VL2 (the distance between the light sources) is performed by, for example, obtaining an angle θc at which VL1 and VL2 intersect and determining whether θc is smaller than a predetermined threshold θth. When VL1 and VL2 are used, cos θc can be expressed by the following formula (5).
cos θc = <VL1, VL2> / (|| VL1 || × || VL2 ||) (5)

  Here, <a, b> represents an inner product of a and b, and || a || represents a norm of a. Since cos θc is obtained from Equation (5), θc is obtained using an inverse trigonometric function. When the intersection angle θc is smaller than the threshold value θth, it is determined that the incident direction of the ambient light is close to the incident direction of the virtual light source, and the highlight area caused by the ambient light is not removed. If the calculated θ is equal to or larger than the threshold θth, it is determined that the incident direction of the ambient light is away from the incident direction of the virtual light source, and the highlight area caused by the ambient light is removed.

  The processing in this embodiment has been described above. In the present embodiment described above, the normal template is used as a method for acquiring the normal information. However, the present invention is not limited to this method. For example, distance information corresponding to the pixel unit of the photographing pixel output from the distance measuring sensor 127 is used. Then, normal information may be acquired. A normal template may be prepared for each subject.

In the above-described embodiment, the pixel value in the highlight area caused by the ambient light is corrected when the ambient light intensity is weaker than the virtual light source and the incident direction of the ambient light is not similar to the incident direction of the virtual light source. It was decided to. At that time, the highlight generated by the ambient light may be weakened according to the degree of separation between the incident direction of the ambient light and the incident direction of the virtual light source. For example, as shown in FIG. 4B, the coefficient k is set such that the value increases as the angle θc formed by the incident direction of the ambient light and the incident direction of the virtual light source increases. Then, it is possible to a value obtained by multiplying the k strength L1 of the ambient light is subtracted from the pixel value Yh _i of each pixel of the highlight area caused by ambient light, corrects each pixel value in the highlight region. The corrected pixel value Yh _i ′ can be calculated based on Equation (6). Thereby, the highlight generated by the environmental light can be weakened according to the incident direction of the environmental light and the incident direction of the virtual light source.
Yh _i ′ = Yh _i −kL1 (6)

  Furthermore, in the above-described embodiment, the case where the user directly designates the intensity of the virtual light source has been described. However, the present invention is not limited to this. For example, relative to whether the intensity of the ambient light is stronger or weaker based on the intensity of the ambient light. The user may specify such information.

  In the above-described embodiment, the incident direction (position) and intensity of light from the light source are used as the light source information of the ambient light and the virtual light source. However, the present invention is not limited to this. good. Here, the degree of diffusion refers to the direction dependency of the intensity of light emitted from the light source. It is assumed that the degree of diffusion is low when the intensity of light emitted from the light source is strong only in a specific direction, and the degree of diffusion is high as the intensity of light emitted from the light source is almost uniform regardless of the direction. In addition, when the light source information includes the degree of diffusion, the user can specify the degree of light diffusion of the virtual light source. At this time, the degree of diffusion of ambient light may be used as the degree of diffusion of light from the virtual light source. When the degree of diffusion is included in the light source information, the vectors VL1 and VL2 indicating the direction of the light source indicate the direction in which the intensity of light emitted from the light source is the strongest.

  As described above, according to the present embodiment, the relationship between the incident direction of the ambient light irradiated to the subject and the incident direction of the light irradiated from the newly set virtual light source, and the ambient light and the light of the virtual light source The highlight area based on the ambient light formed in the image data can be processed according to the relationship with the intensity. For example, if the incident direction of the ambient light and the incident direction of the virtual light source are different and the intensity of the light of the virtual light source is stronger than the ambient light, it will be unnatural if the highlight area due to the ambient light remains. So this can be removed. On the other hand, even when the virtual light source is set, if the intensity of the ambient light is stronger than the light of the virtual light source, the highlight area can remain.

[Embodiment 2]
In the first embodiment described above, the case of removing the highlight area caused by the ambient light has been described. On the other hand, in the second embodiment of the invention, a case where a highlight area generated by a virtual light source is added will be described.

  The configurations of the digital camera and the image processing unit corresponding to this embodiment are the same as those shown in FIGS. The flow of highlight processing in the image processing unit 105 is also the same as that described with reference to FIGS. However, the process in S306 of FIG. 3 is different. Hereinafter, a processing method of a highlight area corresponding to the present embodiment will be described. In this embodiment, in step S306, a highlight area generated by the virtual light source is added based on the light source information of the ambient light and the light source information of the virtual light source. Details of the processing will be described below.

  In this process, first, the ambient light intensity L1 is compared with the virtual light source intensity L2. When L1 is less than or equal to L2 (L1 ≦ L2), the highlight region generated by the light from the virtual light source is added on the assumption that the light from the virtual light source is stronger than the ambient light. The highlight area can be set by the following method. First, based on the incident direction VL2 of the virtual light source and the normal information obtained in S302, a vector is calculated for an arbitrary plane constituting the face detected in S301 based on Expression (1). Next, a value corresponding to the coordinates of the camera coordinate system of the origin of the normal vector of the plane on which the calculated vector is calculated is selected. A surface including a plane corresponding to the selected vector can be used as a highlight region based on light from the virtual light source. Further, the generation of the highlight area may be performed, for example, by adding the intensity L2 of the virtual light source to the pixel value of the pixel included in the highlight area.

  On the other hand, when L1 is larger than L2 (L1> L2), the incident direction VL1 of the ambient light is further compared with the incident direction VL2 of the virtual light source. When the directions of VL1 and VL2 are similar, a highlight area generated by the virtual light source is added, and when the directions of VL1 and VL2 are not similar, the highlight area generated by the virtual light source is not added. The method of determining the similarity between the directions of VL1 and VL2 is the same as that described in the first embodiment, and detailed description thereof is omitted here.

  As described above, according to the present embodiment, the relationship between the incident direction of the ambient light irradiated to the subject and the incident direction of the light irradiated from the newly set virtual light source, and the ambient light and the light of the virtual light source The highlight area based on the virtual light source can be added to the image data according to the relationship with the intensity. For example, if the incident direction of the ambient light and the incident direction of the virtual light source are different and the intensity of the light of the virtual light source is stronger than the ambient light, it will be unnatural if the highlight area due to the ambient light remains. So this can be removed. On the other hand, even when the virtual light source is set, if the intensity of the ambient light is stronger than the light of the virtual light source, the highlight area can remain.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100: Digital camera, 101: Lens group, 102: Shutter, 103: Imaging part, 104: A / D conversion part, 105: Image processing part, 106: Image memory, 107: Memory control part, 108: D / A conversion 109: Display unit 110: Codec unit 111: Medium I / F 112: Storage medium 113: External output I / F 50: System control unit 120: Operation unit 121: Power switch 122: Power control unit, 123: power supply unit, 124: nonvolatile memory, 125: system timer, 126: system memory, 127: distance sensor

Claims (12)

An image processing apparatus that sets a virtual light source for an image and generates a relighting image in which light is emitted from the virtual light source to a subject image in the image,
Detecting means for detecting a highlight area in the image based on a luminance value of a pixel constituting the subject image;
Estimating means for estimating the intensity of ambient light in the image;
In the process of generating the relighting image, a processing unit that corrects a highlight area in the image based on a comparison between the light intensity of the virtual light source and the estimated intensity of the ambient light,
The processing means does not correct the pixel value of the highlight region when the intensity of the ambient light is larger than the intensity of the light of the virtual light source, and the intensity of the ambient light is equal to or less than the intensity of the light of the virtual light source. In this case, the image processing apparatus corrects the pixel value of the highlight area. A calculation unit that calculates an incident direction of the ambient light based on information on a normal vector of a region included in the highlight region of the surface of the subject corresponding to the subject image;
The processing means is a case where the intensity of the ambient light is equal to or less than the intensity of the light of the virtual light source, and the incident direction of the ambient light is not similar to the set incident direction of the light of the virtual light source The image processing apparatus according to claim 1, wherein when it is determined, the pixel value of the highlight area is corrected.   The incident direction of the ambient light and the incident direction of the light of the virtual light source are determined to be similar when the angle at which the directions intersect is smaller than a predetermined threshold, and is not similar when the angle is equal to or greater than the threshold. The image processing apparatus according to claim 2, wherein the image processing apparatus is determined as follows.   The processing means corrects the pixel value of the highlight area by subtracting the value of the ambient light intensity corresponding to the magnitude of the angle from the pixel value of the highlight area. Item 4. The image processing apparatus according to Item 3.   The image processing apparatus according to claim 1, wherein the processing unit corrects a pixel value of the highlight area based on a pixel value of a peripheral area of the highlight area.   The processing means further forms a highlight area based on the light of the virtual light source in the subject image of the image when the intensity of the environmental light is equal to or less than the light intensity of the virtual light source. Item 5. The image processing apparatus according to any one of Items 2 to 4. The processing means, when the intensity of the ambient light is greater than the intensity of the light of the virtual light source,
When it is determined that the incident direction of the ambient light and the incident direction of the virtual light source are similar, a highlight region based on the light of the virtual light source is formed in the subject image of the image,
When it is determined that the incident direction of the ambient light is not similar to the incident direction of the virtual light source, a highlight region based on the light of the virtual light source is not formed in the subject image of the image. The image processing apparatus according to claim 6.   The image processing apparatus according to claim 1, wherein the degree of light diffusion of the virtual light source is set so as to correspond to the degree of diffusion of the ambient light.   The said estimation means estimates the intensity | strength of the said ambient light based on the difference of the luminance value of the pixel of the said highlight area | region in the said image, and the peripheral area | region of the said highlight area | region. The image processing apparatus according to any one of the above. Imaging means;
An image processing apparatus comprising: the image processing apparatus according to claim 1 that processes an image including a subject image captured by the image capturing unit. A control method of an image processing apparatus that sets a virtual light source for an image and generates a relighting image in which light is emitted from the virtual light source to a subject image in the image,
A detecting step in which the detecting means of the image processing device detects a highlight area in the image based on a luminance value of a pixel constituting the subject image;
An estimation step in which the estimation means of the image processing apparatus estimates the intensity of ambient light in the image;
The processing unit of the image processing device corrects a highlight area in the image based on a comparison between the light intensity of the virtual light source and the estimated intensity of the ambient light in the process of generating the relighting image. Processing steps,
In the processing step, when the intensity of the ambient light is greater than the intensity of the light of the virtual light source, the pixel value of the highlight region is not corrected, and the intensity of the environment light is equal to or less than the intensity of the light of the virtual light source. And correcting the pixel value of the highlight area.   A program that causes a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 9.

Images