JP7319637B1 - Information processing system, information processing method and information processing program - Google Patents

Abstract

An information processing system, an information processing method, and an information processing program capable of concealing an animal image region of a source image using a character image based on a three-dimensional model of the character are provided. An information processing system for processing a source image including an animal image using a character image based on a three-dimensional model of the character, the system estimating a three-dimensional pose of the animal from the animal image to generate three-dimensional pose data. 3D pose estimating means for generating; pose applying means for changing the pose of the 3D model of the character by applying the 3D pose data; and based on the 3D model of the character whose pose has been changed, the character and an image based on the character image is arranged in the source image to generate a processed image in which the area of the animal image in the source image is substantially hidden. An information processing system comprising character image placement means is provided. [Selection drawing] Fig. 1

Description

The present invention relates to an information processing system, information processing method, and information processing program for processing a source image including an animal image using a character image based on a three-dimensional model of a character. In addition, the "animal image" in this specification includes a person image.

Image processing technology using artificial intelligence technology is known. For example, Non-Patent Document 1 discloses a technique of estimating the position of a key point in a photograph of a person using artificial intelligence. Here, key points are joint points (shoulders, elbows, wrists, hips, knees, ankles, etc.) and characteristic points (eyes, nose, mouth, ears, etc.). Non-Patent Document 2 discloses estimating pre-learned regions such as a person, a bicycle, and an animal from a photograph by semantic segmentation. Non-Patent Document 3 discloses a technique for estimating a three-dimensional pose of a subject from a two-dimensional image. In addition, Non-Patent Document 4 discloses a technique for rotating, moving, and transforming a two-dimensional shape. Furthermore, Non-Patent Document 5 discloses a technique for detecting a bounding box of an object from an image including the object.

By the way, when an image of a person is published, it may be processed such as hiding the face by adding a line of sight or applying a mosaic to the face for privacy protection. However, if the face is only hidden, there is a risk that the person can be identified based on the characteristics of the person's clothing.

On the other hand, a method of painting not only the face but also the whole body with one color is also conceivable. According to this technique, although concealment is possible to some extent, even the existence of a person cannot be recognized.

Therefore, an image processing method has been proposed that allows the presence of a person to be recognized while hiding the whole body of the person (see, for example, Patent Documents 1, 2, and 3).

Photographs are also processed to create animations and comics. However, since the head and body, clothes, and facial features of the person in the photograph do not change, there is a problem that it does not match the style of the animation or manga.

JP 2018-72957 A JP 2019-9752 A JP 2019-91138 A

Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", [online], April 14, 2017, [searched March 25, 2021] , Internet <URL: https://arxiv.org/abs/1611.08050> Evan Shelhamer, Jonathan Long, Trevor Darrell, "Fully Convolutional Networks for Semantic Segmentation", [online], May 20, 2016, [searched on March 25, 2021], Internet < URL: https://arxiv .org/abs/1605.06211> Zdushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, Christian Theobalt，"VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera"，［online］， July 2017, [searched on March 25, 2019], Internet <URL: http://gvv.mpi-inf.mpg.de/projects/VNect/content/VNect_SIGGRAPH2017.pdf> Takeo Igarashi, Tomer Moscovich, John F. Hughes, "As-Rigid-As-Possible Shape Manipulation", [online], [searched March 25, 2021], Internet <URL: https://www-ui .is.s.u-tokyo.ac.jp/~takeo/papers/rigid.pdf＞ Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", [online], [searched May 20, 2021], Internet <URL: https://arxiv.org/pdf/1506.01497.pdf＞

SUMMARY OF THE INVENTION An object of the present invention is to provide an information processing system, an information processing method, and an information processing program capable of hiding an animal image area in a source image using a character image based on a three-dimensional model of the character.

According to one aspect of the present invention, there is provided an information processing system for processing a source image including an animal image using a character image based on a three-dimensional model of a character, wherein the three-dimensional pose of the animal is obtained from the source image or the animal image. 3D pose estimating means for generating 3D pose data by estimating , pose applying means for applying the 3D pose data to change the pose of the 3D model of the character; and Character drawing means for generating a character image, which is a two-dimensional image of the character, based on a three-dimensional model; and character image placement means for generating a substantially hidden processed image.

According to this configuration, the pose of the three-dimensional model of the character is changed by applying the three-dimensional pose data generated from the source image or the animal image, and then the character is based on the three-dimensional model of the character after changing the pose. Since the image is placed in the animal image area of the source image, the animal image area of any source image of the user can be hidden while leaving its existence.

character model transforming means for transforming the three-dimensional model of the character so as to reduce a deviation between the outline of the animal image of the source image and the outline of the image based on the character image; preferably generates said character image based on said posed and deformed three-dimensional model of said character.

According to this configuration, the three-dimensional model of the character is further deformed so as to reduce the deviation between outlines, so that the animal image area of the source image can be hidden more accurately.

The character model transforming means may adjust the length of a predetermined portion of the three-dimensional model of the character.

According to this configuration, the length of the predetermined portion of the character image can be made close to the length of the corresponding predetermined portion of the animal image, so that the deviation between the outline of the animal image and the outline of the image based on the character image is small. , so that the animal image regions in the source image can be more accurately hidden.

The character model transforming means may transform the three-dimensional model of the character based on positions of key points in the animal image and positions of key points of the character.

According to this configuration, the position of the keypoint of the character image can be brought closer to the position of the corresponding keypoint of the animal image. Areas of animal images in the source image can be more accurately obscured.

The apparatus may further include character image transforming means for transforming the character image so as to reduce the deviation between the outline of the animal image and the outline of the character image.

According to this configuration, the character image is deformed so that the deviation between the contours is reduced, so that the animal image area of the source image can be more accurately hidden.

The character image transforming means may transform the character image based on positions of key points in the animal image and positions of key points in the character image.

According to this configuration, the position of the keypoint of the character image can be brought closer to the position of the corresponding keypoint of the animal image. Areas of animal images can be hidden more accurately.

The character image transforming means transforms the character image so that the position of the key point in the animal image and the position of the corresponding key point in the character image substantially coincide with each other, and then transforms the animal image. The character image may be deformed so that a point on the contour of the character image substantially coincides with a corresponding point on the contour of the character image.

According to this configuration, the deviation between the outline of the animal image and the outline of the character image becomes smaller, and the area of the animal image in the source image can be hidden more accurately.

It may further include region estimation means for estimating the region of the animal image and/or the contour of the animal image by semantic segmentation.

According to this configuration, the animal image and the character image can be accurately aligned, and the animal image area of the source image can be more accurately hidden.

The character image arranging means generates a cutout image by removing an area substantially including the animal image from the source image, and the cutout image is formed when an image based on the character image is arranged in the cutout image. a complementary image generating means for generating a complementary image for complementing a gap between the cut-out image and the character image; and synthesizing the cut-out image, an image based on the character image, and the complementary image to produce the processed image. a synthesis means for generating
may be provided.

According to this configuration, since the complementary image is used to fill the gap, even if there is a deviation between the outline of the character image and the outline of the animal image, it is possible to generate a processed image with little unnaturalness.

The cut-out image generating means may generate the cut-out image by removing from the source image an area obtained by enlarging the area of the character image.

With this configuration, even if there is a deviation between the outline of the character image and the outline of the animal image, the area of the animal image can be removed.

The clipped image generating means may generate the clipped image by estimating the animal image region in the source image by semantic segmentation to remove the animal image region from the source image.

According to this configuration, by performing estimation by semantic segmentation, the region of the animal image can be removed more accurately when generating the clipped image.

The complementary image generating means may generate the complementary image by performing blur processing on the cutout image.

According to this configuration, since the complementary image is generated by performing the blur processing, the gap between the clipped image and the character image can be more naturally filled.

The character image arrangement means may adjust the color tone and/or gradation of an image based on the source image and/or the character image.

According to this configuration, it is possible to suppress discomfort in color tone and/or gradation in the processed image.

According to another aspect of the present invention, there is provided an information processing method for processing a source image including an animal image using a character image based on a three-dimensional model of a character, wherein the three-dimensional image of the animal is processed from the source image or the animal image. estimating a pose to generate three-dimensional pose data; applying the three-dimensional pose data to change the pose of the three-dimensional model of the character; generating a character image that is a two-dimensional image of the above, arranging an image based on the character image in the source image, and generating a processed image in which the area of the animal image in the source image is substantially hidden is provided.

According to this configuration, the pose of the three-dimensional model of the character is changed by applying the three-dimensional pose data generated from the source image or the animal image, and then the character is based on the three-dimensional model of the character after changing the pose. Since the image is placed in the animal image area of the source image, the animal image area of any source image of the user can be hidden while leaving its existence.

According to another aspect of the present invention, there is provided an information processing program for processing a source image including an animal image using a character image based on a three-dimensional model of a character, wherein the computer comprises the source image or the animal image. 3D pose estimating means for estimating a 3D pose of an animal to generate 3D pose data; pose applying means for applying the 3D pose data to change the pose of the 3D model of the character; character drawing means for generating a character image, which is a two-dimensional image of the character, based on the three-dimensional model of the character whose character has been changed; An information processing program that operates as character image placement means for generating a processed image in which the area of the animal image is substantially hidden is provided.

According to this configuration, the pose of the three-dimensional model of the character is changed by applying the three-dimensional pose data generated from the source image or the animal image, and then the character is based on the three-dimensional model of the character after changing the pose. Since the image is placed in the animal image area of the source image, the animal image area of any source image of the user can be hidden while leaving its existence.

A character image based on a three-dimensional model of the character can be used to obscure the animal image regions of the source image.

1 is a block diagram showing a schematic configuration of an information processing system according to a first embodiment; FIG. 4 is a flowchart showing an example of processing operations of the information processing system according to the first embodiment; FIG. 4 is a diagram for explaining a source image input to a 3D pose estimation unit; FIG. 4 is a diagram for explaining an initial pose of a three-dimensional model of a character; FIG. 4 is a diagram for explaining a three-dimensional model of a character to which pose data is applied; FIG. 4 is a diagram for explaining a character image generated by a character drawing unit; FIG. 4 is a diagram for explaining generation of a character image using a virtual camera; FIG. 4 is a diagram for explaining generation of key points on a character image using a virtual camera; FIG. 4 is a diagram showing an example of key points on a character image; FIG. 4 is a diagram showing an example of a processed image; FIG. 4 is a diagram showing positions of key points in a person image; FIG. 4 is a diagram showing an example of key points on a three-dimensional model of a character; FIG. 2 is a block diagram showing a schematic configuration of an information processing system according to a second embodiment; FIG. FIG. 4 is a diagram for explaining a method of transforming a three-dimensional model of a character; 9 is a flowchart showing an example of processing operations of an information processing system according to the second embodiment; FIG. 11 is a block diagram showing a schematic configuration of an information processing system according to a third embodiment; FIG. 11 is a flowchart showing an example of processing operations of an information processing system according to a third embodiment; FIG. 4 is a diagram showing an example of transforming a character image; 4 is a flow chart showing an example of a character image deformation method. FIG. 7 is a diagram showing an example of a character image deformation method; FIG. 7 is a diagram showing an example of a character image deformation method; FIG. 11 is a block diagram showing a schematic configuration of an information processing system according to a fourth embodiment; FIG. 14 is a flowchart showing an example of processing operations of an information processing system according to the fourth embodiment; 4A and 4B are diagrams showing examples of cutout images; FIG. The figure which shows the example of a complementary image. FIG. 10 is a diagram showing an example of an image in which the edge of a cutout image is blurred; FIG. 4 is a diagram showing an example of a processed image;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments according to the present invention will be specifically described with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a schematic configuration of an information processing system according to the first embodiment. The information processing system may be composed of one device, or may be composed of a plurality of devices that can communicate with each other. In the latter case, for example, part of the processing may be performed by a user terminal such as a smartphone, and the other part may be performed by the server.

A user inputs an arbitrary source image to this information processing system. This source image contains an image of an animal (animal image) to be hidden. As noted above, the term "animal" shall include "person". In the following, the description will be made assuming that the subject is a human, but the present invention can also be applied to animals other than humans. A three-dimensional model of a computer graphics character is input to the information processing system. Key points may be set in advance in the three-dimensional model of the character. Then, the information processing system uses the character image based on the three-dimensional model of the character to perform processing to hide the person image area of the source image.

Specifically, the information processing system includes a three-dimensional pose estimation section 1 , a pose application section 2 , a character drawing section 3 and a character image placement section 4 . A part or all of the functions of these units may be implemented by hardware, or may be realized by a processor executing a predetermined program.

A three-dimensional pose estimation unit 1 estimates a three-dimensional pose of a person from a source image or an image of a person, and generates three-dimensional pose data. The three-dimensional pose estimation unit 1 also estimates the camera settings when the person image is captured, and outputs the estimated camera settings. The estimated three-dimensional pose is represented by three-dimensional pose data representing the pose of the humanoid object (person or character). The three-dimensional pose data is, for example, information about the angle of each joint and the orientation and position of the entire humanoid object. Alternatively, the 3D pose data may be 3D coordinates of the positions of the joints instead of the angles of the joints, and the format of the 3D pose data is not particularly limited. For example, the method described in Non-Patent Document 3 can be used for estimating the three-dimensional pose.

The three-dimensional model of the character and the three-dimensional pose data output from the three-dimensional pose estimation unit 1 are input to the pose application unit 2 . The 3D pose estimation unit 1 and the pose application unit 2 may be connected, and the 3D pose data output from the 3D pose estimation unit 1 may be automatically input to the pose application unit 2 . Alternatively, the user may manually input the 3D pose data output from the 3D pose estimation unit 1 to the pose application unit 2 . This point also applies to input/output of other units.

The pose application unit 2 applies the three-dimensional pose data to change the pose of the three-dimensional model of the character.

The three-dimensional model of the character whose pose has been changed by the pose applying section 2 and the camera settings output from the three-dimensional pose estimating section 1 are input to the character drawing section 3 . Then, the character drawing unit 3 renders the three-dimensional model of the character whose pose has been changed to generate a character image, which is a two-dimensional image of the character.

A character image generated by the character drawing section 3 and a source image are input to the character image placement section 4 . Then, the character image placement unit 4 places an image based on the character image in the region of the person image of the source image to generate a processed image. The character image placement unit 4 may generate the processed image after adjusting the color tone and/or gradation of the source image and/or the area arbitrarily designated by the user in the character image.

FIG. 2 is a flowchart illustrating an example of processing operations of the information processing system according to the first embodiment. Note that FIG. 2 is an example, and the order of some processes may be changed as appropriate, or some processes may be omitted.

In step S1, the 3D pose estimation unit 1 estimates a 3D pose of a person from a source image or a person image, and generates and outputs 3D pose data. For example, as shown in FIG. 3, assume that the pose of the person 110 included in the source image 100 is the pose of throwing a frisbee. A three-dimensional pose estimation unit 1 generates three-dimensional pose data expressing this pose. Three-dimensional pose data is, for example, information about the angles of each joint such as the neck, arms, and legs, and the orientation and position of the entire humanoid object. Alternatively, the 3D pose data may be 3D coordinates of the position of each joint.

Note that the 3D pose data may implicitly include camera settings. For example, if the three-dimensional pose data contains information on the orientation and position of the entire humanoid object, and the information on the orientation and position is described in the camera coordinate system, this three-dimensional pose data will be the same as the humanoid object. Contains orientation and position information relative to the camera. In this way, when the 3D pose data implicitly includes camera settings, the 3D pose estimation unit 3 does not output the camera settings explicitly, but outputs only the 3D pose data. good.

Subsequently, at step S2 in FIG. 2, the pose application unit 2 applies the three-dimensional pose data to the three-dimensional model of the character to change the pose of the three-dimensional model of the character. For example, as shown in FIG. 4A, assume that the pose (initial pose) of the three-dimensional model 200 of the character is the care pose. The pose applying unit 2 applies the three-dimensional pose data generated from the source image 100 of FIG. 3 to the three-dimensional model 200 of the character. Then, as shown in FIG. 4B, the three-dimensional model 200' of the character assumes the pose of throwing a frisbee.

More specifically, the pose applying unit 2 changes the angles of each joint such as the neck, arms, legs, etc. of the three-dimensional model of the character, as well as the direction and position of the entire character, based on the three-dimensional pose data. Change the pose of the 3D model of In this way, a three-dimensional model of the character with changed poses is obtained.

Subsequently, at step S3 in FIG. 2, the character drawing unit 3 renders the three-dimensional model of the character whose pose has been changed to generate a character image. For example, the character drawing unit 3 renders the three-dimensional model of the character after the pose has been changed shown in FIG. 4B to generate the character image 210 shown in FIG.

The character drawing unit 3 receives a three-dimensional model of a character to which a pose has been applied (in other words, a three-dimensional model of a character whose pose has been changed by the pose applying unit 2) and camera settings. The character drawing unit 3 then applies the input camera settings to the virtual camera. Furthermore, the character drawing unit 3 uses the virtual camera to generate a two-dimensional image of the character (character image) from the three-dimensional model of the character to which the pose has been applied. As a specific example, the character drawing unit 3 virtually draws a three-dimensional model of a pose-applied character from a virtual camera placed in the direction and position indicated by the camera settings, and generates a character image.

For example, FIG. 6A is a diagram showing an example in which camera settings are applied to virtual camera C. As shown in FIG. As shown in FIG. 6A, the character drawing unit 3 can arrange a virtual camera C based on camera settings. FIG. 6 is a diagram showing an example of generating a two-dimensional image of a character (for example, the character image 210 in FIG. 5) from a three-dimensional model 200 of a character to which a pose has been applied using a virtual camera C;

Subsequently, at step S4 in FIG. 2, the character image placement unit 4 places an image based on the character image in the person image area of the source image to generate a processed image. For example, the character image placement unit 4 places the character image 210 shown in FIG. 5 in the region of the person image 110 in the source image 100 shown in FIG. 3 to generate the processed image 300 shown in FIG.

As described above, in the first embodiment, since the character image is arranged in the person image area of the source image, the presence of the person can be made visible while hiding the person image area of the user's arbitrary source image. .

Further, in this embodiment, after changing the pose of the three-dimensional model of the character by applying the three-dimensional pose data generated from the source image or the person image, Place the character image in the person image area of the source image. Therefore, even if the three-dimensional pose of the subject of the person image of the source image is different from the initial pose of the three-dimensional model of the character, the area of the person image of the source image can be hidden.

When arranging a character image in a source image, it is necessary to align the source image and the character image and scale them. One of the images may be further rotated during registration. As a method for this alignment and scale alignment, for example, the following two methods are available.

The first technique is to use the bounding box of the image. In this case, the bounding box of the person image and the bounding box of the character image are used. Then, the source image and the character image are aligned and scaled by enlarging or reducing and moving the character image so that the heights and centers of the bounding boxes match each other. Alternatively, the character image may be enlarged or reduced and moved so that the widths and centers of the bounding boxes match.

In order to obtain the bounding box of the human image, for example, a method of obtaining a bounding box by an object detection method (for example, Non-Patent Document 5) or a method of semantic segmentation (for example, Non-Patent Document 2) is used to obtain a region of the human image. A method of obtaining the bounding box of the area, or the like can be used.

A second technique is to use keypoints. In this case, keypoints on the person image and keypoints on the character image are used.

The position of the keypoint in the human image (see FIG. 8) can be obtained using the method of Non-Patent Document 1, for example. Also, the position of the keypoint in the character image can be obtained using the following two methods (method A or method B).

In method A, keypoint positions (three-dimensional coordinates) are set in advance in a three-dimensional model of a character. In the example of Figure 9, the face keypoints (eyes, nose, mouth, ears) are located on the surface of the 3D model, and the body keypoints (neck, shoulders, elbows, etc.) are located inside the 3D model. do. In step S3 of FIG. 2, when the character drawing unit 3 draws a three-dimensional model of a character using a virtual camera to generate a character image, three-dimensional coordinates (three-dimensional model) are converted to two-dimensional coordinates (virtual screen). A coordinate transformation to is performed. By applying coordinate transformation similar to this to the keypoint coordinates of the three-dimensional model, the positions of the keypoints on the character image can be obtained. Also, in general, bones (skeletal structure) are often set in a three-dimensional model of a character. In such a case, the position (three-dimensional coordinates) of the joint point of the bone may be used as the keypoint position. FIG. 6B shows that the keypoint position of the right wrist on the character image is obtained based on the keypoint of the right wrist set in the three-dimensional model of the character.

As method B, the position of the key point on the character image can be obtained using, for example, the method of Non-Patent Document 1 for the character image.

Alignment and scaling using keypoints are performed as follows.

For example, on each image, the midpoint between the key point of the right waist and the key point of the left waist is obtained, and using this as the position of the waist, the character image is moved so that the positions of the waists of both images match. Then, for each image, a bounding box is created from all keypoints, and the character image is enlarged or reduced so that the height (or width) of the bounding box of both images is the same, resulting in a difference between the source image and the character image. Alignment and scaling can be performed.

Alternatively, a predetermined number (for example, two) of important keypoints are determined, and the character image is moved, scaled, or rotated so that the positions of the keypoints on both images match. For example, the images are moved, scaled, and rotated so that the neck keypoints of both images match and the right ankle keypoints of both images match. Also in this way, the source image and the character image can be aligned and scaled.

(Second embodiment)
A second embodiment described below deforms a three-dimensional model of a character based on a person's image or the positions of points on the person's image in order to more accurately hide the area of the person's image. In the following, explanations common to the first embodiment may be omitted or simplified.

FIG. 10 is a block diagram showing a schematic configuration of an information processing system according to the second embodiment. As a difference from the information processing system of FIG. 1, the information processing system of FIG.

The character model transforming unit 6 also receives a three-dimensional model of a character to which the three-dimensional pose data is applied (changed pose), camera settings, and a source image. Then, the character model transforming unit 6 transforms the three-dimensional model of the character to which the three-dimensional pose data is applied (the pose is changed) so that the deviation between the outline of the person image and the outline of the character image is reduced.

As an example of a method for transforming a three-dimensional model of a character, a method of transforming a three-dimensional model of a character based on the positions of key points on a person's image can be used. In this method, it is necessary to obtain three-dimensional coordinates from the positions of points on the image. Here, for example, as shown in FIG. 11, consider the case of changing the position of the wrist on the character image without changing the position of the elbow. Assume that the position of the wrist on the character image is P0 and the position of the wrist on the person image is P1. In order to reduce the deviation between the outline of the person image and the outline of the character image, the three-dimensional model of the character is deformed so that the position of the wrist on the character image changes from P0 to P1. The position Q of the wrist of the three-dimensional model should be on the straight line L connecting the camera and the point P1, but the position of Q cannot be uniquely determined by this alone. Therefore, as a method of uniquely determining the position of Q, the following method can be used.

In the first method, a plane passing through the original wrist position Q0 and parallel to the virtual screen is created, the intersection of this plane and the straight line L is defined as Q1, and Q1 is defined as the wrist position Q after deformation.
In the second method, we add a constraint condition that "the length of the part (the length from the elbow to the wrist) does not change from the original length", and the post-deformation wrist position is calculated to satisfy this condition. Ask for Q. If there are two solutions for Q (for example, in the example of FIG. 11, there are Q2 and Q3), either can be used, but for example, the one closer to the original pose (Q2 in the example of FIG. 11) ) may be selected.

FIG. 12 is a flowchart illustrating an example of processing operations of the information processing system according to the second embodiment. Since steps S1, S2, and S4 overlap with those of the first embodiment, description thereof is omitted.

In step S11, the character model transforming unit 6 transforms the three-dimensional model of the character to which the three-dimensional pose data is applied (the pose is changed) so that the deviation between the outline of the person image and the outline of the character image becomes small. do. For example, the three-dimensional model of the character is transformed using the method described with reference to FIG. In the example of Fig. 11, the elbow and wrist keypoints are used to transform the region from the elbow to the wrist (forearm). good too.

More specifically, for example, first, the position of the waist of the character image is aligned with the position of the waist of the person image.

Next, the range from the waist to the neck of the three-dimensional model of the character is deformed using the technique shown in FIG. match.

Next, the range from the neck to the right shoulder of the three-dimensional model of the character is deformed using the technique shown in FIG. Align it with your shoulder.

Next, the range from the right shoulder to the right elbow is deformed in the three-dimensional model of the character using the method shown in FIG. align with the right elbow of the

Next, the range from the right elbow to the right wrist of the three-dimensional model of the character is deformed using the method shown in FIG. on the right wrist of the

In the same way, the neck-left shoulder-left elbow-left wrist, hip-right hip-right knee-right ankle, and hip-left hip-left knee-left ankle are processed to determine the position of each part of the character image. may be aligned with the position of the corresponding part of the person image.

There are two methods, methods A and B, for obtaining the positions of key points on the character image, as described above. Method A uses coordinate transformation using a virtual camera. Method B uses character images to estimate keypoint positions.

When method A is used, the character model transformation unit 6 can obtain the positions of key points on the character image without drawing the character image.

When using method B, the character model transforming section 6 needs to draw a character image. To draw the character image, a method similar to that used by the character drawing section may be used.

Subsequently, in step S12, the character drawing section 3 renders the three-dimensional model of the character transformed in step S11 to generate a character image.

Then, in step S4, the character image placement unit 4 places the character image generated in step S12 in the region of the person image in the source image to generate a processed image. By deforming the three-dimensional model of the character so that the position of the key point on the person image and the position of the key point on the character image match, the deviation between the outline of the person image and the outline of the character image can be reduced. can do.

(Third embodiment)
A third embodiment described below transforms a character image based on a person image or the position of a point on the person image in order to more accurately hide the area of the person image. In the following, explanations common to the first embodiment may be omitted or simplified.

FIG. 13 is a block diagram showing a schematic configuration of an information processing system according to the third embodiment. As a difference from the information processing system of FIG. 1, the information processing system of FIG.

The character image output from the character drawing unit 3 and the source image are input to the character image transforming unit 7 . Then, the character image transforming section 7 transforms the character image so that the deviation between the outline of the person image and the outline of the character image is reduced.

As an example of the character image transformation method, the character image can be transformed so that the position of the key point on the person image matches the position of the key point on the character image.

FIG. 14 is a flowchart illustrating an example of processing operations of an information processing system according to the third embodiment. Since steps S1 to S3 overlap with those of the first embodiment, description thereof is omitted.

In step S21, the character image transforming section 7 transforms the character image generated by the character drawing section 3 so as to reduce the deviation between the outline of the person image and the outline of the character image.

The position of the keypoint on the human image (see FIG. 8) can be obtained using the method of Non-Patent Document 1, for example.

There are two methods, methods A and B, for obtaining the positions of key points on the character image, as described above. Method A uses coordinate transformation using a virtual camera. Method B uses character images to estimate keypoint positions.

When method A is used, the character drawing section 3 may further obtain and output the position of the key point on the character image.

When method B is used, the character image transformation unit 7 may use the method of Non-Patent Document 1, for example, to obtain the position of the key point on the character image.

In order to deform the character image so that the position of the key point on the person image and the position of the key point on the character image match, for example, the technique of Non-Patent Document 4 may be used.

In step S4 of FIG. 14, the character image placement unit 4 places the character image generated in step S21 in the person image region in the source image to generate a processed image. By deforming the character image so that the positions of the key points on the person image and the positions of the key points on the character image match each other, it is possible to further reduce the deviation between the outline of the person image and the outline of the character image. can.

In this embodiment, the character image is transformed so that the key points of the person and the key points of the character match. As a result, the positions of the keypoints on the character image after deformation approximately match the positions of the keypoints on the person image. However, if the thickness and width of the corresponding parts of the person image and the character image are different, the contours of the two do not always match.

Therefore, the character image may be further transformed so that the outline of the character image matches the outline of the person image while preventing the key points that are combined from being shifted. That is, the keypoints and contours of the person image and the character image are matched.

FIG. 15 is a diagram showing an example of deforming the outline of a character image. FIG. 15 schematically shows part of the outline (solid line) of the character image and the outline (broken line) of the person image after the deformation processing of the character image using the key points described above. In this example, both keypoints (indicated by "two black circles" in FIG. 15) are aligned by deformation of the character image, but the contours do not match. In this case, the outline of the character image is made as close as possible to the outline of the person image by expanding or contracting the outline of the character image in the direction orthogonal to the line connecting the two key points (indicated by the dashed line in FIG. 15). As a result, the contour of the character image after deformation substantially matches the contour of the person image.

The character image transformation unit 7 may further transform the character image using the following method.

A flowchart of FIG. 16 shows an example of a character image deformation method. First, in step S31, the character image transforming section 7 sets a plurality of points on the outline of the character image based on key points on the character image.

As an example, FIG. 17A shows the outline of a character image and its two key points (black circles) k1 and k2. For example, keypoint k1 is the right shoulder and keypoint k2 is the right elbow. The character image transforming unit 7 arbitrarily adds a point on the straight line connecting the keypoints k1 and k2 (indicated by the dashed dotted line in FIG. 17A, hereinafter referred to as the "reference line") (in the figure, two keypoints k1, Two points (indicated by black squares) m1 and m2 are added at positions dividing the line segment connecting k2 into three equal parts). Then, the character image transforming unit 7 sets points p1 and p2 at positions where a straight line passing through the key point k1 and orthogonal to the reference line intersects the outline of the character image. Similarly, the character image transformation unit 7 sets a plurality of points on the outline of the character image for the other key point k2 and the added points m1 and m2.

At step S32 in FIG. 16, the character image transformation unit 7 sets a plurality of points on the outline of the person image based on the key points on the person image.

As an example, FIG. 17B shows the outline of a person image and its two key points (white circles) k11 and k12. It is assumed that keypoints k1 and k2 in FIG. 17A correspond to keypoints k11 and k12 in FIG. 17B, respectively. For example, if the keypoint k1 in FIG. 17A is the right shoulder and the keypoint k2 is the right elbow, the keypoint k11 in FIG. 17B is also the right shoulder and the keypoint k12 is the right elbow. As described with reference to FIG. 17A, the character image transformation unit 7 sets a plurality of points on the contour of the person image. The character image transformation unit 7 grasps the correspondence relationship between each point set on the contour of the character image in FIG. 17A and each point set on the contour of the person image in FIG. 17B.

In step S33 of FIG. 16, the character image transforming unit 7 transforms the character image so that each point set on the outline of the character image and the corresponding point set on the outline of the person image overlap each other. As described above, the keypoints and contours of the transformed character image substantially match the keypoints and contours of the person image.

(Fourth embodiment)
In the fourth embodiment described below, in order to cope with the case where the character image does not sufficiently hide the character image in the processed image, an area substantially including the character image is cut out from the source image, and then the character image is removed. It complements the space between the placed and cut out image and the character image.

FIG. 18 is a block diagram showing a schematic configuration of an information processing system according to the fourth embodiment. The character image placement section 4 in this information processing system has a clipped image generation section 41 , a complementary image generation section 42 and a synthesis section 43 .

A cut-out image generation unit 41 generates a cut-out image by removing a region substantially including a person image from the source image. Specifically, the cut-out image generation unit 41 identifies the region of the person image in the source image by semantic segmentation. Then, the cut-out image generation unit 41 sets the specified area of the person image as the cover area. Then, the cutout image generation unit generates a cutout image by removing the cover area from the source image. If the accuracy of region identification by semantic segmentation is low, a region larger than the identified region may be set as the cover region.

Alternatively, the clipped image generator 41 may not use semantic segmentation, but instead use character images. In this case, since the character image area and the person image area usually have some deviation, in order to remove the person image area, it is desirable to use the expanded area of the character image as the cover area. Then, the cutout image generation unit generates a cutout image by removing the cover area from the source image.

The complementary image generation unit 42 generates a complementary image that complements the gap between the cutout image and the character image that is formed when the character image is arranged in the cutout image.

The synthesizing unit 43 synthesizes the clipped image, the character image, and the complementary image to generate a processed image. Specifically, the synthesizing unit 43 generates the processed image by superimposing the complementary image, the clipped image, and the character image in this order from the bottom.

FIG. 19 is a block diagram illustrating an example of processing operations of an information processing system according to the fourth embodiment. Since steps S1 to S3 overlap with those of the first embodiment, description thereof is omitted. After step S3, the character image placement unit 4 generates a processed image through the processing of steps S31 to S33 below.

In step S31, the cut-out image generation unit 41 generates a cut-out image by removing a region substantially including the person image from the source image. For example, the cut-out image generation unit 41 removes the cover area obtained by enlarging the area of the character image 210 shown in FIG. 5 from the source image 100 shown in FIG. 3 to generate the cut-out image 700 shown in FIG.

Subsequently, in step S32 of FIG. 19, the complementary image generation unit 42 generates a complementary image that complements the gap between the cut-out image and the character image formed when the character image is arranged in the cut-out image. For example, blur processing is performed on the clipped image 700 shown in FIG. 20 to generate a complementary image as shown in FIG.

Then, in step S33 of FIG. 19, the synthesizing unit 43 synthesizes the clipped image, the character image, and the complementary image to generate a processed image. For example, by synthesizing an image with blurred edges as shown in FIG. 22 and a character image 210 as shown in FIG. 5 on a complemented image as shown in FIG.
A processed image shown in FIG. 23 is obtained.

As described above, according to the fourth embodiment, the gap between the clipped image and the character image in the processed image can be more naturally filled. Note that the second and third embodiments may be combined with the fourth embodiment.

While the second and third embodiments are effective when prioritizing the body shape and head to body of a person image, the fourth embodiment is effective when prioritizing the body shape and head and body of a character. It is valid. This is because the character is not distorted in the fourth embodiment, whereas the character is distorted in the second and third embodiments. That is, which of the methods of the second to fourth embodiments is effective depends on the situation of use. In addition, the techniques of the second to fourth embodiments may be combined with a plurality of these techniques. Alternatively, different techniques may be used depending on the human image or the part of the character.

In each of the above-described embodiments, an area of an image of a person (person image) included in a source image is hidden. Areas may be obscured. In this respect, the "human image" can be generalized to the "animal image". That is, "animal images" include "person images".

The above-described embodiments are described for the purpose of enabling a person having ordinary knowledge in the technical field to which the present invention belongs to implement the present invention. Various modifications of the above-described embodiments can be naturally made by those skilled in the art, and the technical idea of the present invention can also be applied to other embodiments. Therefore, the present invention should not be limited to the described embodiments, but should have the broadest scope in accordance with the spirit defined by the claims.

1 Three-dimensional pose estimation unit 2 Pose application unit 3 Character drawing unit 4 Character image placement unit 41 Cutout image generation unit 42 Complementary image generation unit 43 Synthesis unit 6 Character model transformation unit 7 Character image transformation unit

Claims (14)

An information processing system for processing a source image including an animal image using a character image based on a three-dimensional model of the character,
The information processing system is
3D pose estimation means for generating 3D coordinates or angles of each part of an animal by 3D pose estimation from the source image or the animal image, and obtaining 3D pose data representing the 3D pose of the animal;
Pose applying means for applying the three-dimensional pose data to change the pose of the three-dimensional model of the character;
character drawing means for generating a character image, which is a two-dimensional image of the character, based on the three-dimensional model of the character whose pose has been changed;
character image arrangement means for arranging an image based on the character image in the source image to generate a processed image in which the area of the animal image in the source image is substantially hidden;
character model transforming means for transforming the three-dimensional model of the character so as to reduce the deviation between the outline of the animal image of the source image and the outline of the image based on the character image;
with
The information processing system, wherein the character drawing means generates the character image based on the three-dimensional model of the character whose pose has been changed and transformed. 2. The information processing system according to claim 1 , wherein said character model transforming means adjusts the length of a predetermined part of said three-dimensional model of said character. The character model transforming means includes:
2. The information processing system according to claim 1 , wherein the three-dimensional model of said character is deformed based on the positions of keypoints in said animal image and the positions of keypoints of said character. An information processing system for processing a source image including an animal image using a character image based on a three-dimensional model of the character,
3D pose estimation means for generating 3D coordinates or angles of each part of an animal by 3D pose estimation from the source image or the animal image, and obtaining 3D pose data representing the 3D pose of the animal;
Pose applying means for applying the three-dimensional pose data to change the pose of the three-dimensional model of the character;
character drawing means for generating a character image, which is a two-dimensional image of the character, based on the three-dimensional model of the character whose pose has been changed;
character image arrangement means for arranging an image based on the character image in the source image to generate a processed image in which the area of the animal image in the source image is substantially hidden;
character image transforming means for transforming the character image so as to reduce a deviation between the outline of the animal image of the source image and the outline of the character image ;
An information processing system comprising : The character image transforming means includes:
5. The information processing system according to claim 4 , wherein said character image is transformed based on the positions of key points in said animal image and the positions of key points in said character image. The character image transforming means includes:
deforming the character image so that the position of the key point in the animal image and the position of the corresponding key point in the character image substantially match each other;
6. The information processing system according to claim 5 , wherein the character image is deformed so that a point on the contour of the animal image and a corresponding point on the contour of the character image substantially coincide with each other. 5. The information processing system according to claim 1, further comprising area estimating means for estimating the area of the animal image and/or the contour of the animal image by semantic segmentation. The character image placement means is
a cut-out image generating means for generating a cut-out image by removing an area substantially including the animal image from the source image;
Complementary image generation means for generating a complementary image for complementing a gap between the cut-out image and the character image-based image formed when the character image-based image is arranged in the cut-out image;
synthesizing means for synthesizing the cut-out image, the image based on the character image, and the complementary image to generate the processed image;
The information processing system according to claim 1 or 4 , comprising: The cut-out image generating means includes:
9. The information processing system according to claim 8 , wherein the cutout image is generated by removing an area obtained by enlarging the area of the character image from the source image. The cut-out image generating means includes:
9. The information processing system according to claim 8 , wherein the region of the animal image is removed from the source image to generate the cropped image by estimating the region of the animal image in the source image by semantic segmentation. 11. The information processing system according to any one of claims 8 to 10 , wherein said complementary image generating means generates said complementary image by performing blur processing on said cut-out image. The character image placement means is
5. The information processing system according to claim 1 , wherein the color tone and/or gradation of an image based on said source image and/or said character image is adjusted. An information processing method for processing a source image containing an animal image using a character image based on a three-dimensional model of the character,
The information processing method includes:
generating 3D coordinates or angles of each part of an animal by 3D pose estimation from the source image or the animal image to obtain 3D pose data representing the 3D pose of the animal;
applying the three-dimensional pose data to alter the pose of the three-dimensional model of the character;
generating a character image, which is a two-dimensional image of the character, based on the three-dimensional model of the character whose pose has been changed;
arranging an image based on the character image in the source image to generate a processed image in which the area of the animal image in the source image is substantially hidden;
including
The information processing method further comprises:
deforming the three-dimensional model of the character so as to reduce the deviation between the contour of the animal image of the source image and the contour of the image based on the character image;
The information processing method, wherein the step of generating the character image includes the step of generating the character image based on the three-dimensional model of the character whose pose has been changed and the deformed. An information processing program for processing a source image containing an animal image using a character image based on a three-dimensional model of the character,
The information processing program causes a computer to
3D pose estimation means for generating 3D coordinates or angles of each part of an animal by 3D pose estimation from the source image or the animal image, and obtaining 3D pose data representing the 3D pose of the animal;
Pose applying means for applying the three-dimensional pose data to change the pose of the three-dimensional model of the character;
character drawing means for generating a character image, which is a two-dimensional image of the character, based on the three-dimensional model of the character whose pose has been changed;
character image arrangement means for arranging an image based on the character image in the source image to generate a processed image in which the area of the animal image in the source image is substantially hidden;
to operate as
The information processing program further causes the computer to
An information processing program for operating as character image transforming means for transforming the character image so as to reduce a deviation between the outline of the animal image of the source image and the outline of the character image.