JP2024001398A - Video processing method to protect privacy while maintaining sense of presence - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide signal processing that protects the privacy of others in a video and does not impair the sense of presence that a photograph has.

SOLUTION: The face of an other person shown in a photograph or video image is replaced with another face, which is a face different from the other person. In this case, the different faces in the video can be the face of the same person, the faces of multiple people, the faces of real people, or the faces synthesized using computer graphics. Furthermore, multiple types of parameters such as facial landmarks, facial expressions, and gender are used to replace the face of the other person with the different face.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

TECHNICAL FIELD The present invention relates to processing of a person's face in an image.

When you take a photo or video in a scene where other people are present, such as at a tourist spot, the faces of other people (hereinafter referred to as other people's faces) are often shown. In such cases, it is necessary to refuse to take pictures or to others about being photographed, as it may cause problems in terms of privacy and may make others uncomfortable. However, it is realistically impossible to refuse everyone when there are many other people, and it is troublesome to refuse each and every one even when there are only a few. It would be desirable to be able to technically solve this problem. In this case, the person who does not have the consent or authority to capture the person's face in the video, to process or change the video, or to publish the face in the video, at least It corresponds to someone else.

To address this issue, post-photography processes such as blurring, blacking out, or adding mosaics to other people's faces are often performed. However, these processes have the problem of spoiling the atmosphere of the scene that the photograph or video has.

Therefore, in Patent Document 1, the outline of the face is extracted and only the face is processed as much as possible so as not to spoil the atmosphere of the scene. However, since the face is painted black, it is difficult to say that the atmosphere has not been spoiled.

In Patent Document 2, the background is stored and the other person is erased by replacing the other person with the background. However, it is difficult to say that the atmosphere is intact because other people disappear.

Patent No. 6504364 Patent No. 6419749

The present invention provides signal processing that protects the privacy of others in a video while not impairing the sense of realism that a photograph has.

The present invention provides a method for capturing images of people in the video after taking a photograph or video (including images and other image data equivalent to these, collectively referred to as "video" if there is no need to distinguish them). It is characterized by replacing a face (referred to as "other person's face") with a different face (referred to as "other face") to protect the privacy of others without spoiling the atmosphere of the scene. processing (image processing method). Aspects of the present invention will be described below.
[1] A process characterized by replacing a face of another person in a photo or video with a different face.
[2] The process described in [1] above, which is characterized by replacing the face with another face while using multiple types of parameters such as facial landmarks, facial expressions, and gender.
[3] The process according to any one of [1] to [2] above, wherein the different faces in the video are the faces of the same person.
[4] The process according to any one of [1] to [2] above, wherein the different faces in the video are the faces of multiple people.
[5] The process according to any one of [1] to [4] above, wherein the other face in the video is the face of a real person.
[6] The process according to any one of [1] to [4] above, wherein the other face in the video is a face synthesized using computer graphics.
[7] The process described in any one of [1] to [6] above, characterized in that a text or a pictogram clearly indicating that the replacement has been performed is included in the video.
[8] The process according to any one of [1] to [7] above, characterized in that other people's faces are limited to faces that are captured with a predetermined number of pixels or less.
[9] The process according to any one of [1] to [8] above, characterized in that a specific other person's face is not replaced.
[10] The process according to any one of [1] to [9] above, characterized in that the detection of the orientation of the other person's face is changed for each other person's face.
[11] The process according to any one of [1] to [9] above, characterized in that the detection of the orientation of the other person's face is not changed depending on the other person's face.
[12] The process according to any one of [1] to [9] above, characterized in that the detection of the direction of the other person's face is changed depending on the position where the other person's face is displayed.
[13] Any one of [1] to [12] above, characterized in that when replacing a different face of the same person in two or more time-series photos or videos, a different face of the same person is used. Processing described in Section.
[14] [1] to [13] above, characterized in that changes in multiple types of parameters, such as landmarks and expressions of other people's faces, gender, etc., are limited to below a predetermined value in two or more time-series photos or videos. ] The processing described in any one of the above.
[15] [1] above, characterized in that when replacing a different face of the same person in two or more time-series photographs for stereoscopic viewing or a video for stereoscopic viewing, a different face of the same person is used. ~ The treatment described in any one of [14].
[16] In two or more time-series photographs for stereoscopic viewing or videos for stereoscopic viewing, changes in multiple types of parameters such as landmarks and expressions of other people's faces, gender, etc. are limited to below a predetermined value. The process described in any one of [1] to [15] above.
[17] When replacing a face with a different face while using multiple types of parameters such as facial landmarks, expressions, and gender, select a different face that does not minimize the difference between the parameters of the other person's face and the parameters of the other face. The process described in any one of [1] to [16] above, characterized by:
[18] A method for creating a video using the video processing method according to any one of [1] to [17] above.

Aspects of the present invention will be explained from another perspective. The processing of the present invention includes a means for combining a group of images in which different faces are shown, a means for extracting a group of different faces from a group of images in which different faces are shown, a means for estimating the parameters of the extracted group of different faces, and means for extracting a stranger's face from a video; means for estimating parameters of the extracted stranger's face; means for comparing parameters of the stranger's face with parameters of a group of different faces to select another face from the group of different faces; and means for replacing the other person's face using the other face that has been created. Furthermore, it may include means for displaying on the video that the face has been replaced with a different face. More specifically, do as follows.

The means to compile a group of images showing different faces is to collect photos of actual people, to collect a group of images created using computer graphics (abbreviated as "CG"), or to create a group of images using CG at any time. It is a means. Further, a means for extracting a group of different faces from a group of images showing different faces is a means for extracting a group of different faces using a technology such as computer vision (abbreviated as "CV"). The extracted result is a rectangle, cube, etc. Further, the means for estimating the parameters of the extracted different face group is a means for estimating using CV technology or the like. Facial parameters include 68 landmarks, facial expressions, and gender determination.

The means for extracting the faces of other people from the photographed images is a means for extracting them using CV technology or the like, and is similar to the means for extracting a group of different faces from a group of images in which different faces are shown, as described above. Further, the means for estimating the parameters of the extracted other faces is a means for extracting using CV technology or the like, and is the same as the means for estimating the parameters of the extracted other faces described above.

The means for selecting another face from the other face group by comparing the parameters of the other person's face and the parameters of the other face group is to compare the parameters of the other person's face and the parameters of another face in the other face group, Decide on a different face for all other people's faces. In the case of a video, if there is another face used in the previous frame, the other face of the same person is used. In stereoscopic images, the different faces of each viewpoint are the same.

The means for replacing the other person's face with the selected alternate face is a means for replacing the other person's face using CV technology or the like. Use affine transformation, color correction, etc.

There are ways to display on the video that the face has been replaced with a different face, such as text such as ``The face in this video has been replaced with the face of a different person than the actual person,'' or a pictogram that can be used to easily convey the message. This is a means of overwriting images etc. on video.

According to the process of the present invention, by replacing the face of another person with another face after shooting a video, it is possible to protect the privacy of the other person without spoiling the atmosphere of the scene.

3 is a flowchart showing processing of the present invention. Before processing the captured video, the video that shows another face is processed in advance. 3 is a flowchart showing processing of the present invention. The difference from FIG. 1 is the position of the process 250 for displaying on the video that the face has been replaced with another face. Although the process 250 is superimposed on the photographed video, it can be performed at any timing since it can be performed independently of the process of replacing another person's face. In other words, regarding the process 250, FIGS. 1 and 2 are examples, and the process 250 is not limited thereto. 3 is a flowchart showing processing of the present invention. The difference from FIG. 1 is that another face is created using CG using the other person's face in the captured video as a reference. These are the results of processing according to the present invention. (a) is the photograph taken, and (b) is the photograph after replacement. For ease of understanding, (c) is a photo in which only one of the faces of another person is enlarged, and (d) is a photo in which another face is enlarged.

Embodiments of the present invention will be described in detail below based on the drawings. In the following description, the same reference numerals will be used for processes having the same purpose unless there is a special reason. Furthermore, a description such as "processing 110" indicates "processing 110 of combining a group of images showing different faces" or the like.

[Specific example shown in Figure 1]
FIG. 1 is a flowchart showing the processing of the present invention. Hereinafter, in the processing of the present invention, first, a process 100 is performed for a video in which another face is captured, and then a process 200 is performed for a photographed video. Process 200 is performed after the photo to be converted is taken, but process 100 may be performed before or after.

The process 110 of assembling a group of images in which different faces are shown is performed by collecting images of actual people, or by collecting a group of images created by computer graphics (hereinafter abbreviated as CG), or by creating a group of images using CG at any time. It is processing. In the case of videos of actual people, (1) videos taken from various directions, (2) videos with various facial expressions, (3) videos with no hair covering the face, (4) many people, ( 5) It is desirable that the images take into account the fact that it is known whether or not they are the same person. Even in the case of images created by CG, it is desirable that the images be rendered in consideration of the above. In the case of processing that is created at any time using CG, it is desirable to be able to render with values similar to or the same as the parameters detected in processing 220. The image group created in process 110 is used in the subsequent process as image group 111 showing another face.

The process 120 of extracting a group of different faces from a group of videos showing different faces is a process of extraction using techniques such as machine learning and deep learning. Although the details of the extraction technique are not limited, the extraction results must be usable in subsequent processing. At present, they are rectangles and cubes. Depending on the extraction technique, there is a high possibility of erroneous extraction when the number of pixels of another face is small, so it is desirable to limit the extraction to a predetermined number of pixels or more.

The process 130 of estimating the parameters of the extracted different face group is a process of estimating the parameters of the extracted different face group from the previous processing results using techniques such as machine learning and deep learning. At present, facial parameters include 68 landmarks, facial expressions, and gender determination, but there is no need to limit them to these. The image group created in process 130 is used as another face group and its parameters 131 in the subsequent process. When estimating parameters, the image may be rotated before detection in order to improve estimation accuracy. Furthermore, when the orientation of the face is approximately determined by the position of the image, such as in an all-sky video, the face may be detected after performing a specific rotation. Furthermore, in the case of an image for which it is known in advance that there is distortion, such as an all-sky image, the distortion may be removed before detection. Further, in the case of a stereoscopic photograph, processing may be performed such that the parameters are not changed significantly between the right eye and the left eye, or in the case of a video, between the previous frame and the current frame.

Processing 210 for extracting another person's face from a captured video is similar to processing 120.

Processing 220 for estimating parameters of the extracted other person's face is similar to processing 130.

The process 230 for selecting another face from the other face group by comparing the parameters of the other person's face and the parameters of the other face group compares the parameters of the other person's face and the parameters of another face in the other face group. Remember the difference. The same calculation is performed on all alternate faces in the alternate face group, and the alternate face with the smallest difference is selected. The above calculation is performed for all stranger faces to determine alternate faces for all stranger faces. Also, if you want to intentionally change the photo to have a fun atmosphere, you do not need to select a photo with a small difference.

To compare the parameters of another person's face and the parameters of another face in the group of other faces, for example, the differences in each parameter are integrated. If the importance level differs depending on the parameter, the weight is multiplied before integration. This method is an example and is not limited thereto.

The above is for a photo, and for a video, if there is another face used in the previous frame, the other face of the same person is used. In stereoscopic images, the different faces of each viewpoint are the same. In order to execute these processes, as explained in the process 110, it is desirable to create different face groups and their parameters 131 based on images in which it is known whether they are the same person or not.

In the process 240 of replacing the other person's face with the selected other face, CV technology or the like is used as a means of replacing the other person's face. For example, by finding an affine transformation matrix that makes the corresponding points as close as possible to the 68 landmarks of another person's face and the 68 landmarks of another person's face, perform affine transformation on the other person's face. Get closer to the mark. After that, a color conversion table is obtained so that the average value of the color of the other person's face is the same as the average value of the color of the other person's face, and color conversion is performed on the other person's face to bring it closer to the color of the other person's face. After that, the area surrounded by landmarks of other people's faces is replaced with another face. Then blur the replaced surrounding area. This method is an example and is not limited thereto.

For example, it is desirable to include processing that does not replace specific people, such as not changing only one's face to a different face. The selection of faces not to be replaced may be automatic or may be left to the user.

In the process 250 of displaying on the video that the face has been replaced with a different face, text such as "The face in this video has been replaced with the face of a person different from the person in question" or a pictogram that can be used to easily convey the information is displayed. This is a means of overwriting images etc. on the video. It is desirable to use bordered text to make text and pictograms easier to read. It does not matter whether this process is actually performed or not.

By sequentially performing each of the above processes, it is possible to replace a stranger's face with another face.

[Specific example shown in Figure 2]
FIG. 2 is a flowchart showing the processing of the present invention.

The difference from FIG. 1 is the location of process 250. Although the process 250 is superimposed on the photographed video, it can be performed at any timing since it can be performed independently of the process of replacing another person's face. Therefore, the position shown in FIG. 2 may be used instead of the position shown in FIG. Further, these positions are examples, and the present invention is not limited to these positions.

[Specific example shown in Figure 3]
FIG. 3 is a flowchart showing the processing of the present invention.

The difference from FIG. 1 is that another face is created using CG using the parameters of the other person's face in the captured video. For example, textures prepared in advance to be pasted on the face may be switched depending on facial expression parameters. This is an example and does not have to be limited to this.

In FIG. 3, process 100 cannot be performed before the photo to be converted is taken.

It goes without saying that the present invention is not limited to the embodiments, examples, or specific examples described in this specification and the like.

100 Processing of images showing different faces 110 Processing of combining a group of images showing different faces 120 Processing of extracting a group of different faces from a group of images showing different faces 130 Processing of estimating the parameters of the extracted group of different faces 200 Shooting 210 Processing for extracting another person's face from the captured image 220 Processing for estimating the parameters of the extracted other person's face 230 Selecting another face from the other face group by comparing the parameters of the other person's face and the parameters of the other face group 240 Process of replacing another person's face using the selected alternate face 250 Process of displaying on the video that the face has been replaced with another face

Claims (18)

A video processing method characterized by replacing a face of another person in a photograph or video with a different face. 2. The video processing method according to claim 1, wherein the replacement with the other face is performed using two or more parameters selected from a group consisting of facial landmarks, facial expressions, and gender. 3. The video processing method according to claim 1, wherein the different faces in the video are faces of the same person. 3. The video processing method according to claim 1, wherein the different faces in the video are faces of multiple people. 3. The video processing method according to claim 1, wherein the other face in the video is a face of a real person. 3. The video processing method according to claim 1, wherein the other face in the video is a face synthesized using computer graphics. 3. The video processing method according to claim 1, wherein a text or a pictogram clearly indicating that the replacement has been performed is included in the video. 3. The video processing method according to claim 1, wherein the other person's face is limited to a face captured with a predetermined number of pixels or less. 3. The video processing method according to claim 1, wherein a specific other person's face among the other person's faces is not replaced. 3. The video processing method according to claim 1, wherein the detection of the face direction of the other person's face is changed for each of the other person's faces. 3. The video processing method according to claim 1, wherein the detection of the orientation of the other person's face is not changed depending on the other person's face. 3. The video processing method according to claim 1, wherein the detection of the orientation of the other person's face is changed depending on the position where the other person's face is displayed. 3. The video according to claim 1, wherein when replacing a different face of the same person in two or more time-series photos or videos, a different face of the same person is used. Processing method. In two or more time-series photos or videos, changes in two or more parameters selected from the group consisting of facial landmarks, facial expressions, and gender of the other person's face are limited to below a predetermined value. The video processing method according to claim 2. Any one of claims 1 to 2, characterized in that when replacing a different person's face of the same person in two or more time-series photographs for stereoscopic viewing or a video for stereoscopic viewing, a different face of the same person is used. The video processing method according to item (1). In two or more time-series photographs for stereoscopic viewing or videos for stereoscopic viewing, changes in two or more parameters selected from the group consisting of facial landmarks, facial expressions, and gender of the other person's face are kept below a predetermined value. 3. The video processing method according to claim 2, wherein the video processing method is limited to . When replacing the face with the other face using two or more parameters selected from the group consisting of facial landmarks, expressions, and gender, the difference between the parameters of the other person's face and the parameters of the other face does not become minimum. 3. The video processing method according to claim 2, further comprising selecting another face. A method for creating a video using the video processing method according to any one of claims 1 to 2.

Images