JP2020010167A - Image correction method and image correction device - Google Patents

Abstract

To provide a method and a device capable of capturing an image in which a change in a shadow of a face due to a motion of a person at the time of capturing a moving image is reduced.SOLUTION: An image correction device includes image acquisition means, detecting means for detecting the state of a subject from the acquired image, specifying means for specifying a virtual light source position, determining means for determining the position of the virtual light source on the basis of the state of the subject, and relighting processing means for performing relighting processing on the subject on the basis of the determined position of the virtual light source, and during continuous shooting, the relighting processing is performed such that the relative position of the virtual light source with respect to the subject position and orientation is constant.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a method and an apparatus for correcting an image captured by an image capturing apparatus, and more particularly to a lighting correction of a subject captured as a moving image.

  2. Description of the Related Art Conventionally, a technique has been known in which a subject is detected from a captured image and illumination (lighting) of the subject is corrected later.

  Patent Literature 1 discloses a technique for changing lighting conditions by retouch processing or the like in order to select an optimal face image after capturing a face image of a person.

Japanese Patent Application Laid-Open No. 2000-525025

  However, the related art disclosed in Patent Document 1 does not describe a case where a moving image is captured.

  In particular, in the case of shooting a moving image, depending on the lighting, the manner of shading the face image may change. For example, as shown in FIG. 10, when a person 1003 is photographed by a video camera 1002 using a single light 1001 in a studio, if the lighting is fixed, the lighting on the face of the person is performed according to the movement of the person. Changes the direction of shading. For example, with respect to the person's face in FIG. 11A, it changes as shown in FIGS. That is, even in the case of the same person in one moving image, the shadow of the face changes due to the movement of the person.

  An object of the present invention is to provide an image correction method capable of reducing the change in the shadow by performing a relighting process.

In order to achieve the above object, an image correction method according to the present invention includes:
Image acquisition means, detection means for detecting the state of the subject from the acquired image, designation means for designating the position of the virtual light source, decision means for determining the position of the virtual light source based on the state of the subject, An image correction method having relighting processing means for performing relighting processing on a subject based on the position of a light source, wherein the rewriting processing is performed so that the relative position of the virtual light source with respect to the position and orientation of the subject is constant during continuous shooting. It is characterized by performing.

  According to the present invention, it is possible to provide an image correction method for photographing an image in which the shading of a face image of a subject is small during continuous photographing.

Overall configuration diagram Diagram Face position and orientation Face position and orientation in moving images Flow chart of the first embodiment For multiple wins Flowchart of the second embodiment Face position and orientation for multiple people Flow chart of the third embodiment Shooting status Difference of face shadow depending on lighting direction

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<Example 1>
Hereinafter, an image correction method according to the first embodiment of the present invention will be described with reference to FIGS.

  FIG. 1 shows a camera for performing an image correction method according to a first embodiment of the present invention.

  In FIG. 1, reference numeral 101 denotes the entire apparatus, 102 denotes a photographing lens, 103 denotes an image sensor, 104 denotes a camera signal processing unit, 105 denotes a recording medium, and 106 denotes an image correction processing unit in the present embodiment. The subject image formed by the photographing lens 102 is converted into an electric signal by the image sensor 103, becomes image data by the camera signal processing 104, and is recorded on the recording medium 105. The recording medium 105 may be an externally removable medium or a built-in medium only.

  The image data is input to the image correction processing unit 106 as necessary, and performs image correction processing. FIG. 2 shows details of the image correction processing unit. In the figure, reference numeral 201 denotes the entire image correction processing unit. The image correction processing unit 201 performs an image correction process on the captured image 202, and outputs an output image 203.

  The image correction processing unit 201 detects a specific subject from a captured image. In this embodiment, since a face image of a person is detected, the face detection unit 204 performs face detection, and detects a face image in a captured image. Next, the face orientation determination unit 205 determines the position and orientation of the face in the face image. After the position and orientation of the face are determined, the light source position determination unit 207 determines the light source position for the face according to the light source designation information 206. After the light source position is determined, an image correction process according to the light source position is performed by the relighting processing unit 208.

  Here, the relationship between the position and orientation of the face image and the light source position will be described with reference to FIGS.

  FIG. 3A shows an example of a face image, for example, an image that is slightly rightward toward the screen. Such a face image is detected by a face detection method, and the orientation of the face can be determined from the arrangement of the face organs such as eyes and nose. FIG. 3B is an example in which the face position 301 is determined. The position of the face can be determined from the result of face detection by defining which point of the face is to be the face position. FIG. 3B is an example in which the face position 301 is set between the eyes and the nose. FIG. 3C is an example in which the coordinates of the face are further determined from the face direction.

  In FIG. 3, reference numeral 302 denotes a coordinate axis in the front direction of the face, reference numeral 303 denotes a coordinate axis in the horizontal direction of the face, and reference numeral 304 denotes a coordinate axis in the vertical direction of the face. Since the face coordinate axes 303, 304 and 305 are determined based on the face direction, when the direction of the face image changes, the direction changes accordingly, and a constant direction is maintained for the face.

  Next, FIG. 3D shows an example in which a virtual light source 305 is set for this face position and orientation.

  The position of the virtual light source 305 is specified as the light source specification information 206, and is specified with respect to the face coordinate axis. For example, based on the coordinates of the face coordinate axis, the front coordinates are designated as 1 m in front, the left and right coordinates are designated as 1.5 m to the right, and the up and down coordinates are designated as 2 m above. Alternatively, it may be an angle with respect to the face coordinate axis and a distance from the face position.

  Next, the relighting process of the present invention in the case of capturing a moving image will be described with reference to FIGS. 4A to 4D (the numbers in the drawing are the same as in FIG. 3).

  In the case of capturing a moving image, the face direction of the subject is different for each frame. FIGS. 4A to 4D show cases where the face direction of the subject is (a) slightly rightward, (b) front, (c) slightly leftward, and (d) leftward toward the screen. On the other hand, since the face coordinate axes (302, 303, 304) are determined according to the direction of the face, they change according to the direction of the face.

  In the relighting process of the present invention, the virtual light source position 305 is designated with respect to the coordinate axis, and the virtual light source position in the screen changes according to the face direction. In other words, even if the face direction changes, the position of the virtual light source is at the same position relative to the position and direction of the face. Therefore, when the relighting process is performed on the face image, the effect of the virtual light source on the face image is the same, so that an image with the same shading added to the face can be obtained. If this is a moving image, a face image that maintains a constant shadow between frames is generated. Therefore, an image in which the shadow of the face of the same person in one moving image can be felt as being in a certain state can be captured.

  Here, the operation of this embodiment will be described again with reference to the flowchart of FIG.

  When the moving image photographing is started, a photographed image is acquired at 502 for each frame. This may be an image obtained by normal camera shooting, or an image once recorded on a storage medium.

  Next, at 503, an image of a specific subject, here a face image, is detected from the acquired images. For this, a known face detection technique may be used, and the position, size, inclination, orientation, and the like of the face are obtained. Next, at 504, the face direction (face position and orientation) is determined from the face detection result. This can also be performed using a known face position and face direction determination technique.

  Next, the light source position is determined at 505 according to the light source designation information at 505. In the present invention, the light source specification information 505 is specified for the face direction, and is specified by the distance and angle of the face coordinates, as described above.

Next, rewriting processing of 507 is performed according to the light source position information of 506.
After the rewriting process, if the moving image shooting ends, the process ends. If not, the acquisition of the shot image in step 502 is repeated.

  As described above, according to the present embodiment, it is possible to shoot an image in which the shadow of the face image of the subject is constant.

  When performing the relighting process, the intensity of the relighting (brightness of illumination, contrast, etc.) may be designated in order to adjust the shading of the face shadow. In the case of the present invention, during the shooting of a moving image, It is effective to keep the intensity of this relighting constant.

  In addition, when performing the relighting process, if the process of reducing the influence of separately obtained environmental light (not the virtual light source, the original illumination light, natural light, etc.) is performed, the shading by the virtual light source of the present invention is more effective. Become

<Example 2>
Hereinafter, a second embodiment of the present invention will be described with reference to FIGS. The second embodiment is a case where a plurality of virtual light sources are specified with respect to the first embodiment.

  FIG. 6 shows a face image, the position and orientation of the face, and a state where the virtual light source is specified when a plurality of virtual light sources are specified.

  6, reference numeral 601 denotes a face position, which corresponds to 301 in FIG. Similarly, 602 corresponds to 302, 603 to 303, and 604 to 304. Reference numerals 605 and 606 denote examples of designated virtual light sources, and this figure shows a case where two virtual light sources are designated.

  FIG. 7A shows a processing flow in this case. In the flowchart of FIG. 7, reference numeral 702 corresponds to 502 in FIG. Similarly, 703 corresponds to 503, 704 to 504, 705 to 505, 706 to 506, 708 to 507, and 710 to 508.

  The second embodiment is a case where a plurality of virtual light sources are specified, and the light source specification of 705 naturally specifies a plurality of virtual light sources. Similarly, in the light source position determination in 706, after determining one light source position, it is determined whether or not the designated number of light sources has been reached (707), and the light source position determination is repeated by the number of light sources. Similarly, in the rewriting process of 708, after the end of one rewriting process, it is determined whether or not the specified number of light sources has been reached (709), and the relighting process is repeated by the number of light sources.

  Further, as shown in FIG. 7B, the repetition process performed by the number of light sources is performed by continuously determining one light source position and one re-rating process, and repeating this series of processes by the number of light sources. Is also good.

<Example 3>
Hereinafter, a third embodiment of the present invention will be described with reference to FIGS. The third embodiment is different from the first embodiment in that there are a plurality of subjects to be detected.

  FIG. 8A shows an example in which there are a plurality of subjects. FIG. 8A shows a state where two persons are photographed in one screen. In this embodiment, in this case, since each face image is detected, the face direction (face position and direction) is obtained independently for each face image. The position of the virtual light source is determined according to the face orientation. The designation of the virtual light source may be performed at different positions for different effects for each subject, or a common position may be designated for all subjects.

  FIG. 9 shows a flow in this case. In the flowchart of FIG. 9, reference numeral 902 corresponds to 502 in FIG. Similarly, 903 corresponds to 503, 904 to 504, 905 to 505, 906 to 506, 908 to 507, and 909 to 508.

  In the third embodiment, a plurality of subjects are detected, and the light source designation 905 is designated according to the plurality of subjects. As described above, this may be each light source position for each face image, or a position common to all subjects may be designated. After determining the light source position in 906 according to the light source designation and performing the rewriting process in 907, all the objects have been processed or the number of objects is ranged (908). repeat.

  As shown in FIG. 7 of the second embodiment, it is easy to correct the flow so that the light source position determination and the relighting processing are repeated by the number of subjects, respectively.

  As described above, the preferred embodiments of the present invention have been described, but the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist.

101 overall device, 102 photographing lens, 103 image sensor,
104 camera signal processing unit, 105 recording medium, 106 is an image correction processing unit

Claims (7)

Image acquisition means;
Detecting means for detecting the state of the subject from the acquired image;
Specifying means for specifying a virtual light source position;
Determining means for determining the position of the virtual light source based on the state of the subject;
An image correction method having relighting processing means for performing relighting processing on a subject based on the determined position of the virtual light source,
An image correction method, wherein a relighting process is performed so that a relative position of a virtual light source with respect to a subject position and a direction is constant during continuous shooting.   2. The method according to claim 1, wherein the state of the subject is a position and an orientation of the face image.   There is a setting means for setting the intensity of the rewriting process, and an image correction method for performing the relighting process at the intensity set by the setting device, wherein the intensity of the rewriting process is kept constant during continuous shooting. The image correction method according to claim 1. The image correction method according to claim 1, wherein the image correction method performs environmental light reduction processing as a part of the relighting processing, wherein the environmental light reduction processing is performed for each frame during continuous shooting. Method.   An image correction method capable of designating a plurality of virtual light sources, wherein during continuous shooting, a relighting process is performed so that the relative positions of the plurality of virtual light sources with respect to the subject position and orientation are constant. The image correction method according to claim 1.   An image correction method for performing a relighting process on each subject when a plurality of subjects to be detected are included in the acquired image, wherein a virtual light source is designated for each subject, and continuous shooting is performed. 2. The image correction method according to claim 1, wherein the rewriting process is performed such that the relative position of the virtual light source is constant with respect to the position and orientation of each subject during the processing.   An image correction apparatus for performing the image correction method according to claim 1.