WO2024053235A1

WO2024053235A1 - Three-dimensional avatar generation device, three-dimensional avatar generation method, and three-dimensional avatar generation program

Info

Publication number: WO2024053235A1
Application number: PCT/JP2023/025313
Authority: WO
Inventors: 内田茂樹; 川岸孝輔; 籾倉宏哉
Original assignee: 株式会社PocketRD
Priority date: 2022-09-09
Filing date: 2023-07-07
Publication date: 2024-03-14
Also published as: JP7202045B1; JP2024039293A

Abstract

[Problem] To provide a device, a method, and a program with which it is possible to easily generate a high-quality three-dimensional avatar that reflects the features of an actual person or the like, while protecting the privacy of the person or the like. [Solution] Embodiment 1 relates to a three-dimensional avatar generation device that comprises: a face image input unit 1 for inputting a face image of an object; a head element database 2 for storage concerning head element avatars; a partial surface image database 5 that stores partial surface images that are surface images related to partial regions of the surface of three-dimensional shapes of heads; a partial surface image selection unit 6 that selects a partial surface image matching the three-dimensional shape of the head of the object; a synthesized surface image generation unit 7 that generates, on the basis of the selected partial surface image, a synthesized surface image that is a surface image on the surface of the three-dimensional shape of the head of the object; and a head avatar generation unit 9 that generates a head avatar on the basis of the three-dimensional shape information of the head and the synthesized surface image.

Description

3D avatar generation device, 3D avatar generation method, and 3D avatar generation program

The present invention relates to a three-dimensional avatar generation device, etc. that creates three-dimensional computer graphics data based on imaging data of a person or the like.

In recent years, with improvements in the processing power of computers and other electronic computers, the use of games and video content that utilizes three-dimensional computer graphics is progressing. In other words, famous computer game series that were once expressed in 2D animation have been remade using 3DCG, and movies based on classic fairy tales have been made using 3DCG and become huge hits worldwide. Three-dimensional computer graphics is now widely used as a de facto standard in terms of expression in video content.

As an advanced form of three-dimensional computer graphics, it has also been proposed to use realistic three-dimensional computer graphics modeled after real people. This uses an avatar (character as an alter ego) that imitates the person playing the game as a character in a computer game, or the person of the partner with whom the game is played. This improves the sense of immersion and allows you to enjoy the game as a more realistic experience.

Patent Documents 1 and 2 both disclose examples in which an avatar imitating the player himself or a co-player is used in a computer game that uses a head-mounted display to represent a virtual space.

JP 2019-012509 Publication JP 2019-139673 Publication

However, neither of Patent Documents 1 and 2 discloses how to create an avatar imitating a person such as a player. In particular, computer games and the like are not only those with a realistic world view that imitates the real world, but also those that are set in a specific creative world (for example, those that express the world in paintings or animations), and the latter It is preferable that the avatar used in the game be designed to match the worldview of the work while maintaining the characteristics of the person such as the player, but Patent Documents 1 and 2 do not disclose any technology for creating such an avatar. No disclosure.

In theory, it is possible to create a user's avatar using a method similar to general three-dimensional computer graphics. However, it is not practical to create an avatar using such a method. This is because character creation using 3D computer graphics involves first modeling a mesh structure that corresponds to the skin of a human body, then constructing a skeleton structure that corresponds to the skeleton of a human body, and then building a skeleton structure that corresponds to the skeleton of a human body. This is because it is necessary to perform various tasks such as skinning, which is a function corresponding to a tendon, that is, an association between the skeleton and the skin.

Among these tasks, for example, modeling tasks involve synthesizing images of the user's figure from multiple directions to generate a three-dimensional shape, and automatically modeling the surface shape to some extent based on this. is not impossible. However, it is not possible to automate the skeleton construction and skinning operations, which are tasks related to the internal structure, from images of the user's figure (surface) alone, and these tasks require long hours of manual labor by skilled engineers. I have no choice but to do it. In view of the fact that such a time-consuming process is required, it is not realistic to create a character using general three-dimensional computer graphics for each user's avatar.

In addition, if a 3D avatar is generated by directly compositing videos of the user's figure, the user's figure will of course be faithfully reproduced, but such 3D avatars will be created by the services used. does not necessarily fit into the worldview of Furthermore, using a three-dimensional avatar that faithfully reproduces the user's figure in computer games, SNS, etc. in which an unspecified number of users participate may pose a problem from the viewpoint of privacy protection.

The present invention has been made in view of the above-mentioned problems, and is designed to reflect the characteristics of real people, protect the privacy of those people, and adapt to the world view set by the services used. It is an object of the present invention to provide a device, method, and program for easily generating a suitable, high-quality three-dimensional avatar.

In order to achieve the above object, the three-dimensional avatar generation device according to claim 1 provides a three-dimensional surface shape of an object expressed using a group of vertices that define a positional relationship in a three-dimensional space, and a three-dimensional surface shape that is A three-dimensional avatar generation device that generates a three-dimensional avatar including a surface image displayed on the object, the positional relationship of feature points corresponding to each element constituting the head shape among the vertices based on the facial image of the object. a three-dimensional shape generating means for generating a three-dimensional surface shape of the head of a specified object; and a surface image regarding all or a part of the surface shape of the head; The positional relationship between two or more feature points among the plurality of partial surface images in which the positional relationship of the feature points corresponding to each element constituting the head shape has been specified is the three-dimensional image of the head that has a corresponding relationship. A partial surface image selection means for selecting a partial surface image that matches the positional relationship between two or more feature points of the surface shape, and a partial surface image selected by the partial surface image selection means are synthesized. and a head avatar generator that generates a head avatar based on the three-dimensional surface shape of the head and the composite surface image. It is characterized by having a means.

Further, in order to achieve the above object, the three-dimensional avatar generation device according to claim 2 is provided, in the above invention, wherein the composite surface image generation means selects the partial surface image selected by the partial surface image selection means; The method is characterized in that the combined surface image is generated by combining the facial image of the object with an image of a portion corresponding to the area of the partial surface image.

Further, in order to achieve the above object, the three-dimensional avatar generation device according to claim 3, in the above invention, derives a synthesis ratio of the partial surface image on the three-dimensional surface shape of the head in the synthetic surface image. a composition ratio deriving means for deriving a composition ratio, variation information of the ownership name of the three-dimensional avatar generated with respect to a non-fungible token corresponding to the generated three-dimensional avatar, identification information of a partial surface image constituting the composite surface image, and the information output means for outputting at least the synthesis ratio to a first distributed network in which a first distributed ledger in which the synthesis ratio is recorded; and the three-dimensional information recorded on the first distributed ledger. a second distributed ledger that is generated regarding the ownership name of the avatar, the identification information, the synthesis ratio, and a non-fungible token in correspondence with the partial surface image, and records fluctuation information of the ownership name of the partial surface image; a distribution revenue calculating means for calculating a distribution ratio between the ownership name of the three-dimensional avatar and the partial surface image regarding the revenue generated for the generated three-dimensional avatar based on the ownership name of the partial surface image recorded in the It is characterized by having the following.

In addition, in order to achieve the above object, the three-dimensional avatar generation method according to claim 4 includes a three-dimensional surface shape of an object expressed using a group of vertices that define a positional relationship in a three-dimensional space, and a three-dimensional surface A three-dimensional avatar generation method for generating a three-dimensional avatar including a surface image displayed on a shape, the method comprising: generating a three-dimensional avatar including a surface image displayed on a shape, the method comprising: generating feature points corresponding to each element constituting the head shape among the vertices based on a facial image of the object; a three-dimensional shape generation step of generating a three-dimensional surface shape of the head of the object whose positional relationship has been specified; and a surface image regarding all or a part of the surface shape of the head; Among the plurality of partial surface images in which the positional relationships of feature points corresponding to each element constituting the head shape in the region have been specified, the positional relationships between two or more feature points of the head that have a corresponding relationship are determined. a partial surface image selection step of selecting a partial surface image that matches the positional relationship between two or more feature points of the three-dimensional surface shape; and combining the partial surface images selected in the partial surface image selection step. a synthetic surface image generation step of generating a synthetic surface image that is a surface image that conforms to a three-dimensional surface shape of the head; and a head generating a head avatar based on the three-dimensional surface shape of the head and the synthetic surface image. The method is characterized by including an avatar generation step.

Further, in order to achieve the above object, the three-dimensional avatar generation method according to claim 5, in the above invention, derives a synthesis ratio of the partial surface image on the three-dimensional surface shape of the head in the synthetic surface image. a synthesis ratio deriving step of deriving a synthesis ratio, variation information of the ownership name of the three-dimensional avatar generated regarding the non-fungible token corresponding to the generated three-dimensional avatar, identification information of the partial surface image constituting the synthetic surface image, and the an information output step of outputting at least the composite ratio to a first distributed network in which a first distributed ledger in which the composite ratio is recorded; and the three-dimensional information recorded in the first distributed ledger. a second distributed ledger that is generated regarding the ownership name of the avatar, the identification information, the synthesis ratio, and a non-fungible token in correspondence with the partial surface image, and records fluctuation information of the ownership name of the partial surface image; a distribution revenue calculation step of calculating a distribution ratio between the ownership name of the three-dimensional avatar and the partial surface image regarding the revenue generated for the generated three-dimensional avatar, based on the ownership name of the partial surface image recorded in the It is characterized by including the following.

Further, in order to achieve the above object, the three-dimensional avatar generation program according to claim 6 is configured to display a three-dimensional surface shape and a three-dimensional surface shape of the target object expressed by a group of vertices based on the facial image of the target object. A three-dimensional avatar generation program that causes a computer to generate a three-dimensional avatar including a surface image, the computer having the computer generate a three-dimensional surface of an object expressed using a group of vertices that define positional relationships in three-dimensional space. A three-dimensional avatar generation device that generates a three-dimensional avatar including a shape and a surface image displayed on the three-dimensional surface shape, the elements constituting the head shape among the vertices based on the facial image of the object. a three-dimensional shape generation function that generates a three-dimensional surface shape of the head of an object in which the positional relationship of feature points corresponding to The positional relationship between two or more feature points is determined from among the plurality of partial surface images in which the positional relationship of the feature points corresponding to each element constituting the head shape in the whole or part of the region is specified. a partial surface image selection function that selects a partial surface image that matches the positional relationship between two or more feature points of the three-dimensional surface shape of the head, and a partial surface image selected by the partial surface image selection function. A composite surface image generation function that generates a composite surface image that is a surface image that conforms to the three-dimensional surface shape of the head by combining; A head avatar generation function for generating an avatar.

According to the present invention, it is easy to create a high-quality three-dimensional avatar that reflects the characteristics of a real person, protects the privacy of the person, and conforms to the world view set by the service used. This has the effect of generating

1 is a schematic diagram showing the configuration of a three-dimensional avatar generation device according to a first embodiment; FIG. It is a flowchart for explaining the operation of the head shape generation unit 4 in the embodiment. It is a flowchart for explaining the operation of the partial surface image selection section 6 and the composite surface image generation section 7 in the embodiment. It is a flowchart for explaining the operation of the avatar composition unit 13 in the embodiment. FIG. 2 is a schematic diagram showing the configuration of a three-dimensional avatar generation device according to a second embodiment. It is a schematic diagram showing the composition of a three-dimensional avatar generation device concerning a modification.

Hereinafter, embodiments of the present invention will be described in detail based on the drawings. The following embodiments describe examples that are considered to be the most appropriate embodiments of the present invention, and it goes without saying that the content of the present invention is not limited to the specific examples shown in this embodiment mode. It should not be interpreted in a limited manner. It goes without saying that even configurations other than the specific configurations shown in the embodiments are included in the technical scope of the present invention as long as they provide similar functions and effects.

(Embodiment 1)
First, a three-dimensional avatar generation device according to the first embodiment will be described. As shown in FIG. 1, the three-dimensional avatar generation device according to the first embodiment includes a facial image input unit 1 for inputting a facial image of an object, and a head body database that stores the head body avatar. 2, a head body selection unit 3 that selects a predetermined one from a plurality of head body avatars stored in the head body database 2, and a head body selection unit 3 that selects a predetermined one from a plurality of head body avatars stored in the head body database 2; A head shape generation unit 4 generates three-dimensional shape information that is information about the three-dimensional surface shape of the head of the object based on the information, and a partial surface image that is a surface image about a partial region of the three-dimensional surface shape of the head. A partial surface image database 5 to be stored, a partial surface image selection section 6 that selects a partial surface image that matches the three-dimensional surface shape of the object's head, and a partial surface image selection section 6 that selects a partial surface image that matches the three-dimensional surface shape of the object's head based on the selected partial surface image. A composite surface image generation unit 7 that generates a composite surface image that is a surface image in a dimensional surface shape; a head avatar generation unit 9 that generates a head avatar based on the three-dimensional shape information of the head and the composite surface image; A torso database 10 that stores information on a torso avatar to be combined with a head avatar, a hair database 11 that stores information about a hair avatar to be combined with a head avatar, and torso avatars stored in the torso database 10 and hair database 11. The device includes a component avatar selection section 12 that selects a hair avatar, and an avatar synthesis section 13 that synthesizes the head avatar, torso avatar, and hair avatar to generate an integrated whole body avatar.

The facial image input unit 1 is for inputting a facial image of an object for which a three-dimensional avatar is to be generated. The "facial image" may be a three-dimensional stereoscopic image taken by a 3D scanner or the like, but in the first embodiment, it is a two-dimensional image. Specifically, the facial image input unit 1 may be a data input mechanism for simply inputting data from the outside, or may be configured to include an imaging camera or the like and directly acquire facial images. The facial image inputted by the facial image input unit 1 is an image related to the face of a living thing including a person or a character imitating the living thing, and more specifically, a facial image inputted by the facial image input unit 1 is an image related to the face of a living thing including a person or a character imitating the same. For example, the image includes parts corresponding to the eyes, nose, mouth, eyebrows, ears, chin, etc., and detailed parts thereof. Multiple feature points are set, and ideally it is desirable that the image includes parts corresponding to all the feature points, but if it contains a certain percentage (for example, about 70%) of the parts, this implementation will be effective. There is no problem with avatar generation in this embodiment, and even if the ratio is less than this, as long as three or more feature points can be extracted, avatar generation itself in this embodiment is possible. Regarding parts, it is desirable that the facial image includes the eyes, nose, and mouth.

The head body database 2 is for storing information regarding head body avatars. "Head body avatar" is three-dimensional computer graphics information consisting of the head of a person of average build, for example, and specifically, the head (the part above the neck of a person). .), preferably in addition to this, skeletal information regarding a skeletal structure for controlling head movements, etc., and relationships regarding the relationship between the surface shape and the skeletal structure. information. The head body database 2 stores head body avatars with such information in different formats (data format, usage, feature points (hereinafter referred to as head body avatars) in order to distinguish them from feature points derived from facial images. It has a function of storing a plurality of base avatars corresponding to feature points in a base avatar (referred to as "basic feature points") with different definitions.

"Surface shape information" is information regarding the surface shape of three-dimensional computer graphics, which corresponds to the three-dimensional shape of organs such as eyes and nose, and skin on the human body surface. The information may be in a format in which the entire surface is defined as a collection of minute units such as voxels and the position information of each minute unit is recorded, but from the perspective of reducing the burden on information processing, so-called modeling processing is performed. It is desirable that the three-dimensional surface shape be expressed by defining a predetermined number of vertices and the manner of connection between the vertices. When modeling is performed, edges connecting vertices are formed based on information about vertices and the connection mode between vertices, an area surrounded by three or more edges is defined as a surface (polygon), and a set of surfaces ( The surface shape is specified by the mesh). However, the application target of the present invention is not limited to these, and the surface shape information includes position information of a plurality of vertices and/or a plurality of faces arranged corresponding to the surface shape. If it is, it can be used as surface shape information in the present invention.

Further, for some of the plurality of vertices included in the surface shape information, in addition to the positional information and information regarding the connection mode between the vertices, information regarding the meaning and content of the vertices is also recorded. For example, for vertices that correspond to feature points corresponding to specific points in parts such as the eyes, nose, mouth, etc. and detailed positional relationships among the parts (outer corners of the eyes, pupils, tip of the nose, corners of the mouth, etc.), vertices corresponding to the outer corners of the right eye, The following information shall be attached. Furthermore, regarding the position information of the vertices, it is desirable that the format includes either information regarding the absolute position or information regarding the relative position with respect to the skeletal structure consisting of joints and bones. In this embodiment, not only the former position information but also the latter position information is included, and the position of each vertex changes while maintaining the relative positional relationship according to changes in joint position, bone position, length, etc. It shall be.

"Skeleton information" is information regarding the internal structure that corresponds to the skeleton of a human body and serves as a reference when creating motion in three-dimensional computer graphics. The information format may be a skeletal structure consisting of bones and joints with a predetermined thickness and size, similar to the skeletal structure of the human body, but it may also be a skeletal structure consisting of bones and joints of a predetermined thickness and size. (represented as points) and bones (represented as lines) located between joints and corresponding to bones in the human body. However, the application of the present invention is not limited to these information formats, and includes parts such as joints that can be translated and rotated and also function as a fulcrum in relation to adjacent parts (in the present invention, These are collectively referred to as "joints." Other formats are also acceptable.

"Relationship information" is information that defines the relationship between skeletal information and surface shape information, and more specifically, it is information that defines the relationship between skeletal information and surface shape information. This is information that specifies how much each vertex should follow and operate. If the surface shape were configured to 100% follow the movements of the joints and bones, the character would behave like a tin robot and lack a sense of reality, even though it is a human character. Therefore, when generating three-dimensional computer graphics of a person or the like, it is desirable to set in advance information regarding how much the movement of adjacent bones and joints is followed for each part of the surface shape. Also in this embodiment, for each vertex constituting the surface shape information, numerical information indicating followability with respect to bones and/or joints adjacent to the vertex is set as relevance information. Note that the work of generating relevance information is called skinning processing, weight editing, etc., and weight values are generally used for relevance information as well, but the relevance information in the present invention is not limited to these, and the above-mentioned All information that satisfies the conditions shall be included.

The head element selection unit 3 selects a head element avatar suitable for generating a head avatar of an object from among a plurality of head element avatars stored in the head element database 2. It is something. The head body database 2 stores head body avatars in different data formats depending on the format and usage, and the head body selection unit 3 selects the usage etc. from among the plurality of head body avatars. It has a function of selecting an appropriate head body avatar according to the situation and outputting it to the head shape generation section 4. More specifically, the head body selection unit 3 acquires information regarding the torso avatar selected by the parts avatar selection unit 12 (described later), and selects a head body avatar in a data format consistent with the selected torso avatar. It has the function to select. Furthermore, after satisfying such conditions, it has a function of selecting a head body avatar from a plurality of head body avatars mechanically or according to the user's selection.

The head shape generation section 4 generates a head shape of the object based on the facial image of the object input through the facial image input section 1 and the head element avatar selected by the head element selection section 3. This is for generating three-dimensional shape information that is information about the three-dimensional surface shape of the part. Specifically, the head shape generation unit 4 works with a feature point extraction unit 14 that extracts the positional relationship between feature points in a two-dimensional facial image, and a A positional relationship derivation unit 15 that derives a three-dimensional positional relationship, the coordinates of the feature points from which the three-dimensional positional relationship has been derived, and each vertex that constitutes the three-dimensional surface shape of the head body avatar. a coordinate transformation unit 16 that performs coordinate transformation on the coordinates of the face image; a position adjustment unit 17 that adjusts the positional relationship of the feature points on the facial image extracted by the feature point extraction unit 14; and a region formed by the feature points of the facial image. and a body deformation unit 19 that moves the apex of the head body avatar to match the position of the feature point of the face image.

The feature point extraction unit 14 is for extracting feature points in the facial image of the target object by performing image analysis on the facial image of the target object. Specifically, the feature point extraction unit 14 has a function of extracting feature points from a target facial image by using two-dimensional image analysis technology such as face recognition technology. Note that the feature points extracted by the feature point extraction unit 14 have the same definition as the basic feature points of the head element avatar selected by the head element selection unit 3.

The positional relationship deriving unit 15 is for deriving a three-dimensional positional relationship between the feature points of the facial image extracted by the feature point extracting unit 14. The specific derivation mechanism uses a machine learning model that is trained by giving the two-dimensional feature points in the facial image as the input layer and the three-dimensional positional relationships of the feature points as the output layer as training data. It is also possible to derive the value by using the above method, or other mechanisms may be used. In addition, after deriving the two-dimensional position coordinates (x, y) of the feature points using the plane indicated by the facial image as the xy plane, the depth (z coordinate) of each feature point is calculated using the x and y coordinates of each feature point. It may also be derived using a machine learning model trained by giving coordinates as input layer and z coordinate as output layer as teacher data.

The coordinate conversion unit 16 is for converting the coordinates indicating the position of the feature points extracted from the facial image, and also for converting the coordinates of each vertex including the basic feature points in the head element avatar. Since the feature points of the facial image and the coordinates of each vertex of the head body avatar belong to separate coordinate systems, it is not possible to generate three-dimensional shape information of the object's head based on the two as they are. Therefore, the coordinate transformation unit 16 performs predetermined transformation processing on each coordinate system to develop both on the same coordinate space.

Specifically, the coordinate conversion unit 16 sets a relative coordinate system with a specific feature point (for example, the vertex corresponding to the tip of the nose) as the origin, and converts each of the feature points extracted from the facial image and the head body avatar. By performing affine transformation such as parallel translation, rotation, enlargement/reduction, and skew on the vertices, coordinate transformation processing is performed so that each specific feature point is located at the origin.

The coordinate transformation unit 16 also has a function of performing a coordinate transformation process of returning the coordinate system related to the information to the coordinate system of the head body avatar after generating the three-dimensional shape information of the head.

The position adjustment unit 17 is for adjusting the position of the feature points of the face image after coordinate transformation with the basic feature points of the head body avatar. Specifically, the feature points extracted from the facial image and each vertex of the head body avatar that have been moved to the common coordinates by the processing of the coordinate conversion unit 16 have a specific feature point in a corresponding relationship as the origin. However, the positional relationships of other feature points and vertices are different.For example, the feature points of a face image and the basic feature points and vertices of the head body avatar are misaligned in the direction the face is facing. The situation is as follows.

The position adjustment unit 17 is for adjusting this deviation. For example, assuming a line segment connecting two feature points corresponding to both cheeks, the position adjustment unit 17 adjusts the line segment at the feature point of the facial image and the head body avatar. The position adjustment unit 17 performs a process such as rotating one side to adjust the position so that the line segments are parallel to each other.

The position adjustment unit 17 also has a function of performing shear correction on the coordinates of feature points extracted from the facial image. Depending on the processing method for extracting feature points of a facial image, shearing may occur in the position coordinates of each feature point. In this case, the position adjustment unit 17 has a function of calculating shear amounts for each of the three axes of the three-dimensional coordinate system regarding the feature points of the facial image, and canceling the shear amounts.

The enlarging/reducing unit 18 is for changing the position coordinates of the feature points of the facial image so that the space formed by the feature points of the facial image is expanded or reduced. Specifically, the enlarging/reducing unit 18 expands the space formed by the feature points of the facial image and the space formed by the basic feature points corresponding to the feature points of the facial image among the basic feature points of the head body avatar. It has a function of changing the positional coordinates of the feature points of the facial image while maintaining the mutual positional relationship so that the width of the former space is comparable to the width of the latter space.

Note that the "space formed by feature points" may be a space formed based on the same definition for each of the feature points of the facial image and the basic feature points of the head body avatar. For example, a rectangular parallelepiped (so-called three-dimensional bounding box) whose sides are made of straight lines parallel to the x-axis, y-axis, or z-axis, which is formed so as to contain each feature point and have a minimum volume, or The feature points may be connected by line segments, and a spatial region formed by a surface formed by three or more line segments may be defined as a "space formed by the feature points."

Furthermore, "the widths are about the same" is not limited to the case where the volumes completely match. Preferably, the volume of the space formed by the feature points of the facial image is 0.9 times the volume of the space formed by the basic feature points corresponding to the feature points of the facial image among the basic feature points of the head body avatar. This refers to a range of ~1.1 times, but it is also possible to set a range other than this.

The element body deforming unit 19 transforms the three-dimensional surface shape of a general-purpose head material so that it approaches the three-dimensional surface shape of the object by moving the position of the basic feature points of the head element avatar. It is for the purpose of Specifically, the element transformation unit 19 converts the position coordinates of the basic feature points of the head element avatar that correspond to the feature points of the face image into the position coordinates of the feature points of the face image. Has a function. In addition, the elemental body deformation unit 19 also performs basic feature points such that continuity between the basic feature points and other vertices is maintained for the vertices of the head elemental body avatar that are located in the vicinity of the corresponding basic feature points. It has a function of changing the position at a predetermined rate depending on the distance from the object. Through the processing of the elemental body deformation unit 19, a head shape is generated that has the characteristics of the object in the face region of the three-dimensional surface shape, while remaining in the shape of the head elemental body avatar in other regions.

The elemental body deforming unit 19 not only moves the positions of the basic feature points of the head elemental body avatar to approach the three-dimensional surface shape of the object, but also changes the movement direction and movement of the position of each basic feature point. Record information about distance. The information regarding the movement direction and movement distance corresponds to relative position information indicating how much the feature points in the target facial image have changed with respect to the position of the basic feature points in the element body in the basic state before deformation. Therefore, in comparing the position of the feature point with a partial surface image in the feature point comparison unit 22, which will be described later, it is used as information indicating the position of the feature point in the three-dimensional shape information of the head of the object.

The partial surface image database 5 is for storing partial surface images that are surface images regarding a partial region of the three-dimensional surface shape of the head. Specifically, the partial surface image database 5 includes information on a part of the surface of three-dimensional shape information, such as organs formed on the face such as eyes, nose, mouth, ears, and eyebrows, as well as cheeks, forehead, and chin. It has a function of storing information regarding a partial surface image, which is a surface image regarding a skin area such as a skin area.

"Surface image" is an image related to at least one of color/pattern and texture in the three-dimensional surface shape of the head. "Texture" is also expressed as texture, and refers to external features consisting of minute irregularities on the surface. Generally, in 3D CG, through modeling processing, the outer surface forms edges that connect vertices based on information about the vertices and the connection mode between the vertices, and an area surrounded by three or more edges is defined as a surface (polygon). The surface shape is specified by a set of surfaces (mesh). Since such a method approximates a realistic surface shape, it does not include information regarding texture such as minute irregularities on the surface. Therefore, information regarding texture is created separately and added to the outer surface formed by modeling processing to create a realistic appearance. Note that the information regarding the texture is specifically information regarding the secondary image added on the outer surface, which forms a pattern that reflects Yin and Yang on a two-dimensional plane, such as a height map or a normal map. By doing so, irregularities, etc. are expressed in a pseudo manner. Although the surface image in the first embodiment includes both color/pattern and texture on the surface of the head shape, it may be composed of only one of them.

The information on each partial surface image stored in the partial surface image database 5 includes image information on the partial surface image, that is, information on the color, pattern, and texture in a certain area, and information on the three-dimensional positions of feature points in the area. and information regarding classification. "Information regarding the three-dimensional position of a feature point" refers to the deviation of a predetermined feature point from the basic feature point of a general-purpose head element that corresponds to this, that is, the head element. This is information regarding the movement distance and movement direction from the basic feature point in , and is relative position information with the position of the basic feature point of the head body as a reference.
"Information regarding classification" is information indicating to which classification the partial surface image belongs among a plurality of classifications determined in advance according to predetermined conditions. The "predetermined conditions" can be arbitrary; for example, the generated three-dimensional avatar may be classified differently depending on the design that is suitable for each service such as a game or SNS. Alternatively, the classification may be different depending on the designer who created the partial surface image. Alternatively, different classifications may be made for partial surface images that match the style of a famous painter, illustrator, animator, manga artist, etc., or the world view of a particular movie, painting, animation, or manga. When classifying partial surface images that are consistent with the style and worldview, a machine learning model trained by giving partial surface images as input layer and classification as training data is used to classify individual partial surface images. It may be possible to classify the information, or it may be classified using other mechanisms. Furthermore, the surface image in the partial surface image can be in any form, for example, it may be realistic, cartoon-like, or illustrative.

The partial surface image selection unit 6 selects a surface that matches the three-dimensional surface shape of the object's head generated by the head shape generation unit 4 from among the plurality of partial surface images stored in the partial surface image database 5. This is for selecting a partial surface image consisting of a shape. Specifically, the partial surface image selection unit 6 includes a classification selection unit 21 for selecting a specific classification from among a plurality of classifications regarding partial surface images, and a classification selection unit 21 for selecting a specific classification from among a plurality of classifications regarding partial surface images, and a classification selection unit 21 for selecting a specific classification from among a plurality of classifications regarding partial surface images, and a classification selection unit 21 for selecting a specific classification from among a plurality of classifications regarding partial surface images. A feature point comparison unit 22 that compares the position information with the position information of the feature points of the three-dimensional surface shape of the object's head, and calculates position information about each part of the three-dimensional surface shape of the object's head based on the comparison result. The image determination unit 23 includes an image determination unit 23 that determines which partial surface images match, and an image output unit 24 that outputs the partial surface images determined to match to the composite surface image generation unit 7.

The classification selection unit 21 is for selecting classification items regarding a plurality of partial surface images stored in the partial surface image database 5. In the first embodiment, a configuration is adopted in which a classification item is automatically selected that is suitable for the service in which the generated three-dimensional avatar is used. The configuration may be such that the classification items are selected, or the classification items may be selected according to the user's preferences analyzed based on the browsing history of web pages and the like.

The feature point comparison unit 22 is for comparing the feature points of each partial surface image belonging to the selected classification item with the positions of the feature points in corresponding parts of the three-dimensional surface shape of the object's head. It is. Specifically, the feature point comparison unit 22 compares the positional relationship of feature points of both sides that correspond to each other, and provides information as to whether the positions match or not, and if they do not match, how far apart they are. Outputs information regarding whether the In the comparison of the positional relationships by the feature point comparison unit 22, the position information of the feature points on the surface of the three-dimensional shape information of the object's head is based on the position information of the head element body avatar recorded by the element body transformation unit 19. Relative position information containing the direction and amount of movement from the position of the feature point. Further, the feature point comparison unit 22 may compare the position coordinates of all feature points in a correspondence relationship when comparing feature points, but may also compare only some predetermined feature points. good.

Based on the comparison result of the feature point comparison unit 22, the image determination unit 23 selects features of the three-dimensional shape information of the object's head for each region of the three-dimensional surface shape of the head from among the plurality of partial surface images. This is for determining a partial surface image that includes feature points that are close to the position of the point. Specifically, the image determination unit 23 determines the partial surface image with the smallest total value of the difference in distance between each feature point to be compared derived by the feature point comparison unit 22. However, it may be possible to determine the partial surface image with the largest number of feature points with matching positions, or it may be possible to determine the partial surface image having a surface shape that matches the surface shape of the relevant portion according to another algorithm. Good too.

The image output unit 24 is for outputting the partial surface image determined by the image determination unit 23 to the composite surface image generation unit 7. For the partial surface image determined by the image determination section 23, the image output unit 24 outputs information regarding at least one of color/pattern and texture, the position coordinates of feature points, and the partial surface image regarding which part of the three-dimensional surface shape of the head. It has a function to output information indicating whether the

The composite surface image generation unit 7 synthesizes partial surface images related to each part selected by the partial surface image selection unit 6 by arranging them on each part in a manner that matches the three-dimensional surface shape of the head of the object. This is for generating surface images. Specifically, the composite surface image generation unit 7 includes a surface shape adjustment unit 26 that adjusts the surface shape so that the three-dimensional surface shape of the partial surface image matches the three-dimensional surface shape of the head of the object; An image synthesis unit 27 arranges the partial surface image whose shape has been adjusted on the surface shape of the object's head, and performs interpolation processing on the gap region between the partial surface images that occurs in the three-dimensional surface shape of the object's head. and a complement processing unit 28 that performs the processing.

The surface shape adjustment unit 26 changes the surface shape of the partial surface image selected by the partial surface image selection unit 6 so that the feature points match the positions of the feature points in the three-dimensional surface shape of the object's head. It is for. The surface shape adjustment unit 26 also changes the color, pattern, and texture of the surface image in accordance with the movement of the feature points in the partial surface image. In addition to moving the position of the color, pattern, and texture on the feature point, the position of the color, pattern, and texture near the feature point is also changed according to the distance from the feature point to prevent an unnatural surface image. It is desirable to move the

The image synthesis unit 27 is for arranging the partial surface image that has been adjusted by the surface shape adjustment unit 26 on a corresponding region of the surface shape of the object's head. Specifically, the image synthesis unit 27 determines that the position of the feature point of each partial surface image that has been subjected to shape adjustment by the surface shape adjustment unit 26 is a feature in the corresponding portion of the three-dimensional surface shape of the object's head. It has the function of arranging points so that their positions match.

When the partial surface image is placed on a part of the three-dimensional surface shape of the object's head by the image synthesis section 27, the complementation processing section 28 detects the case where the partial surface image is not placed on the entire surface and a gap area occurs. Second, it is used to perform complementary processing on the void area. Depending on the shape of each partial surface image to be placed, a gap may occur between adjacent partial surface images, and the color, pattern, and texture of the gap cannot be determined. The complementation processing unit 28 performs complementation processing for setting the color, pattern, and texture of the gap portion to match the color, pattern, and texture of the neighboring partial surface image, so that the gap portion does not look unnatural.

The specific content of the interpolation process is to divide the area formed by the main color (for example, skin color) in the partial surface image that exists around the gap into a large number of minute areas, and to extract the color information in each area. A representative color (S _model ) is determined by combining some of the frequently appearing colors (for example, the colors ranked 1st to 5th in frequency). In addition, in deriving the skin color (S), it is derived by the following formula using the maximum value (S _max ), median value (S _median ), and average value (S _mean ) of the brightness in each microregion described above. Ru.

S={(S _model +S _max )/2}*S _model /{(S _median +S _mean )/2}

However, other mathematical formulas may be used to derive the color in the void area, or representative colors, patterns, and textures may be extracted and used as they are as the colors, patterns, and textures in the void area.

Furthermore, the complementation processing unit 28 performs a blurring process to make the boundary unclear so that the boundary between the gap and the adjacent partial surface image does not stand out unnaturally. Specifically, the display area of the color/pattern of the void part and the partial surface image are each enlarged near the boundary to form an area where both overlap, and in the overlap area, the respective mixing ratios are set to 10:0 and 9. :1, ...2:8, 1:9, 0:10 to make the boundary unclear.

The head avatar generation section 9 synthesizes the three-dimensional surface shape of the head of the object formed by the head shape generation section 4 and the composite surface image generated by the composite surface image generation section 7 to generate a head avatar. This is for generating a partial avatar. Through the processing of the head avatar generation section 9, a three-dimensional surface image that has the three-dimensional surface shape of the object's head and reflects the contents of the partial surface image selected by the partial surface image selection section 6 is created as a surface image. The head avatar is completed.

The torso database 10 is for storing torso avatars. The torso avatars that are stored have various shapes and designs, and there are many different data formats (different formats, uses, definitions of basic feature points, etc.). The various data formats of the torso avatar are the same as the various data formats of the head body avatar, and more specifically, when a specific one is selected as the torso avatar, A partial body avatar is selected. Regarding the relationship between a torso avatar and a head body avatar that have the same data format, information regarding the position where the head body avatar is connected in the torso avatar, information regarding the direction of the reference axis of the head body avatar at the time of connection, and Information regarding the size of the head element body avatar to be connected is defined. Using this information, the avatar compositing section 13 performs a whole-body avatar compositing process. The torso avatar may be an imitation of a realistic human body, an animal, an imaginary creature, or even an animated character. The shape of the torso avatar may be 8-headed, 2-headed, or any other shape, and may be nude or wearing clothes, shoes, accessories, etc.

The hair database 11 is for storing hair avatars. The hair avatar is formed to match the outer surface shape near the top of each head element avatar stored in the head element database 2.

The part avatar selection section 12 is for selecting a torso avatar and a hair avatar. The selection algorithm of the torso avatar by the part avatar selection unit 12 may be one that is automatically selected depending on the purpose, or one that is selected based on instructions from the user (for example, the person who is the subject of the facial image). Regarding the selection of the hair avatar in the first embodiment, it is assumed that a hair avatar whose shape matches that of the object with respect to the two-dimensional shape of the hair region projected in one direction (for example, the front direction) is selected. More specifically, the component avatar selection unit 12 clusters the two-dimensional silhouette of the hair avatar and the two-dimensional silhouette of the user's hair using the k-means method or principal component analysis to determine their mutual similarity. A hair avatar having a silhouette similar to that of the user's hair is selected.

The avatar synthesis section 13 is for synthesizing the torso avatar and hair avatar selected by the parts avatar selection section 12 with the head avatar generated by the head avatar generation section 9 to generate an integrated whole body avatar. It is.

The avatar synthesis unit 13 includes a position adjustment unit 29 that aligns the torso avatar and the head avatar when composing them, and a position adjustment unit 29 that aligns the torso avatar with the head avatar, and a reference axis of the head avatar and the head avatar at the time of connection set in the torso avatar. A reference axis adjustment section 30 that adjusts the reference axes so that they are in the same direction, a size adjustment section 31 that adjusts the size of the head avatar, and a synthesis processing section 32 that synthesizes the head avatar with the torso avatar and the hair avatar. Be prepared.

The position adjustment unit 29 is for moving the relative position of the head avatar with respect to the torso avatar to the position of the connection point in the torso avatar. The positional information of the connecting point of the head avatar is set in the torso avatar in advance, and the position adjustment unit 29 adjusts the relative position so that the connecting point of the torso avatar and the connecting point of the head avatar match, based on this positional information. Adjust the position. Note that the connection point in the torso avatar is a place corresponding to the neck between both shoulders in the case of an avatar imitating a normal person, but is not limited to such a place. If it imitates an imaginary creature such as a synthetic beast, it is possible to provide a connection point at a location that matches the design; for example, a connection point may be provided in the palm of the right hand.

The position adjustment unit 29 also adjusts the positions of the head avatar and hair avatar. The hair avatar is formed to match the outer surface shape near the top of the head body avatar, which is the source of the head avatar, and the positional relationship is determined at the vertices of both. With reference to this correspondence, the position adjustment unit 29 performs position adjustment so that the hair avatar is placed at an appropriate position on the head avatar.

The reference axis adjustment unit 30 is for adjusting the directional relationship between the reference axis for the head avatar, which is set in advance for the torso avatar, and the reference axis for the head avatar, so that they match. The direction adjustment by the reference axis adjustment unit 30 is performed, for example, by rotation processing on the torso avatar or head avatar.

The size adjustment unit 31 is for adjusting the size of the head avatar so that it becomes the size preset for the torso avatar. In the torso avatar, information regarding the volume of the head avatar part when an integrated whole avatar is synthesized is determined, and if the volume of the head avatar differs from this, the size adjustment unit 31 adjusts the volume of the torso avatar. The head avatar is enlarged or reduced so that it matches the setting information.

In the synthesis processing section 32, the position adjustment section 29 performs alignment, the reference axis adjustment section 30 matches the directions of the reference axes of the head avatar and the body avatar, and the size adjustment section 31 adjusts the size of the head avatar. This is for compositing the completed head avatar, torso avatar, and hair avatar. Specifically, the synthesis processing unit 32 combines mesh loops (referring to closed curves formed by vertices located at the ends of the connection points) formed at the respective connection points of the head avatar and the body avatar. By doing so, a bond is formed in the surface shape portion. To make it easier, a new line segment is formed between the vertices of the head avatar and the vertices of the body avatar for which the correspondence relationship is defined in advance, and the surface formed by the newly formed line segment is made into a partial polygon. It is also possible to construct a new mesh structure. Further, the head avatar and the torso avatar may be combined by integrating vertices with defined correspondences into one vertex. The same applies to the combination of the head avatar and the hair avatar.

Furthermore, if the head avatar and torso avatar have not only three-dimensional surface shape information but also skeletal information, the compositing processing unit 32 also performs a merging process on their skeletal structures (bones). Regarding the bone joining process, it is desirable to connect the head bone or a similar bone that is generally provided in a head avatar and the neck bone or a similar bone that is generally provided in a torso avatar in a format that sets a parent-child relationship. .

Next, the operation of the head shape generation section 4 will be explained with reference to FIG. 2. First, the feature point extraction section 14 extracts feature points from the input facial image (step S101), and the positional relationship derivation section 15 derives the three-dimensional positional relationship of the feature points (step S102). Thereafter, the coordinate conversion unit 16 converts the position information of the feature points extracted from the facial image and the position information of the basic feature points of the head body avatar to a specific feature point (in this embodiment, a point corresponding to the tip of the nose). ) is converted into position coordinates of a coordinate system with the origin as the origin (step S103), and each feature point extracted from the facial image matches the position of the basic feature point of the corresponding head body avatar. The position is adjusted (step S104), and if shearing of position coordinates occurs during feature point extraction, correction is performed (step S105).

Thereafter, the scaling unit 18 matches the size of the space formed by the feature points extracted from the facial image with the space formed by the basic feature points of the corresponding head body avatar. Then, the position coordinates are changed (step S106). Then, the element body deforming unit 19 converts the position coordinates of the basic feature points of the head element avatar into the positions of the corresponding feature points of the face image in a state where the width of the space formed by the feature points is approximately the same. A process is performed to move the position coordinates to the coordinates (step S107), and finally, a process is performed to return the position coordinates of all vertices of the head element body avatar to the original coordinate system by the coordinate conversion unit 16 (step S108). The operation of the head shape generation unit 4 is thus completed, and the three-dimensional shape information of the head of the object, which has the characteristics of the object derived from the facial image of the object, is completed.

Next, the operations of the partial surface image selection section 6 and the composite surface image generation section 7 will be explained with reference to FIG. First, a classification item is selected by the classification selection unit 21 (step S201), and feature points of a partial surface image belonging to the selected classification item are compared with feature points that have a corresponding relationship in the three-dimensional surface shape of the object's head. A comparison process is performed regarding the positional relationship (step S202).

Thereafter, the image determination unit 23 selects a partial surface image that has the feature point closest to the feature point in the three-dimensional shape information of the object's head (step S203). The selected partial surface image is processed by the surface shape adjustment unit 26 to move its feature points so as to match the three-dimensional surface shape of the object's head (step S204), and the partial surface image is , is placed on the corresponding area in the three-dimensional surface shape of the head of the target part by the image synthesis unit 27 (step S205), and after performing complementation processing as necessary (step S206), the head of the target part is A composite surface image is generated in which the selected partial surface images are arranged in each region on the three-dimensional surface shape of the surface.

Next, the operation of the avatar synthesis section 13 will be explained with reference to FIG. 4. First, the position adjustment unit 29 adjusts the positions of the connection points set for the head avatar and the connection points set for the torso avatar to match (step S301), and adjusts the positions of the head avatar and hair avatar. (Step S302). Thereafter, the reference axis adjustment unit 30 performs adjustment so that the direction of the reference axis for the head avatar set in advance for the torso avatar matches the direction of the reference axis set for the head avatar (step S303). . Further, the size adjustment unit 31 performs an enlargement/reduction process so that the size of the head avatar becomes the size set in advance for the body avatar (step S304). Finally, the head avatar, torso avatar, and hair avatar are combined by the synthesis processing unit 32 (step S305), thereby completing an integrated whole-body avatar.

Next, the advantages of the three-dimensional avatar generation device according to the first embodiment will be explained. First, the three-dimensional avatar generation device according to the first embodiment generates a three-dimensional surface shape of the head of a three-dimensional avatar based on a facial image of a target object, and a surface image to be placed on the three-dimensional surface shape. In this case, instead of using the facial image of the object itself, a composite surface image is used, which is obtained by combining separately prepared partial surface images having shapes close to the three-dimensional surface shape. That is, the three-dimensional avatar generation device according to the first embodiment reflects the physical characteristics of the object (separated eyes, large mouth, high nose, etc.) while also generating specific colors and A partial surface image that differs from the real thing in the pattern is used, thereby making it possible to generate a three-dimensional avatar that has the characteristics of the person but also has a surface image that differs from the appearance of the person. Such three-dimensional avatars give acquaintances who know the person's appearance a friendly impression with the person's characteristics, but third parties cannot guess the person's specific appearance, so they are used by an unspecified number of people. It has the advantage that the user's privacy can be appropriately protected even when used in participating computer games, SNS, etc.

Further, the three-dimensional avatar generation device according to the first embodiment selects partial surface images of a predetermined classification from among a large number of partial surface images according to a predetermined classification to generate a composite surface image. By employing such a configuration, there is an advantage that a three-dimensional avatar with visual uniformity can be generated in the surface image of the head. Furthermore, when the classification items are set according to the type of service that uses the three-dimensional avatar, there is an advantage that the user can easily generate a three-dimensional avatar that corresponds to the service that he/she uses.

(Embodiment 2)
Next, a three-dimensional avatar generation device according to a second embodiment will be described. In Embodiment 2 and its modifications, components with the same names and the same symbols as in Embodiment 1 shall exhibit the same functions as the components in Embodiment 1, unless otherwise specified. .

As shown in FIG. 5, the three-dimensional avatar generation device according to the second embodiment has, in addition to the configuration shown in the first embodiment, a non-fungible token and a partial surface image that correspond to the generated three-dimensional avatar. The token generation unit 33 generates non-fungible tokens that correspond to individual partial surface images recorded in the database, and the composition ratio of the partial surface image to the original target person's facial image in the generated three-dimensional avatar. a combination ratio derivation unit 34 that derives the combination ratio, an information output unit 35 that outputs information regarding the partial surface images constituting the three-dimensional avatar including the derived combination ratio, and distribution of profits obtained by the three-dimensional avatar based on the combination ratio. It also includes a distribution revenue calculation unit 36 that calculates the ratio.

The token generation unit 33 is for generating non-fungible tokens corresponding to three-dimensional avatars and individual partial surface images. A “non-fungible token” is a so-called NFT (Non-Fungible-Token), that is, a token that has unique data and cannot be replaced with other tokens, such as Ethereum (registered trademark). It is published based on the standard ERC721 or other predetermined standards. The transaction history of issued non-fungible tokens is recorded in a distributed ledger stored in a distributed network corresponding to each token.

In the second embodiment, so-called "block chain technology" is used as a distributed ledger management technology using a distributed network. "Blockchain technology" is a technology that uses cryptographic technology to synchronize data between multiple computers that make up a distributed network. Specifically, a "distributed ledger" is one in which each block consists of a collection of records agreed upon among multiple computers and information for connecting with other blocks (information about the previous block). A distributed ledger is constructed by connecting multiple blocks. In the management of distributed ledgers using blockchain technology, even if data is tampered with on some of multiple computers, the correct data will be selected by majority vote among the other computers, so the data in the ledger will be destroyed.・It has the characteristic that it is extremely difficult to tamper with it. However, the distributed ledger management technology in Embodiment 2 is not necessarily limited to blockchain technology, and other technologies may be used as long as they perform similar functions.

The distributed ledger for each token records the transaction history regarding the ownership name of the three-dimensional avatar or partial surface image that corresponds to the token. By adopting such a configuration, in the second embodiment, by referring to the information recorded in the distributed ledger regarding each token, the current holding name of the three-dimensional avatar and partial surface image corresponding to each token can be determined. It is possible to understand the Note that in the claims, a distributed ledger corresponding to a three-dimensional avatar is referred to as a first distributed ledger, and a distributed network in which the first distributed ledger is stored is referred to as a first distributed network; The distributed ledger corresponding to the partial surface image is referred to as a second distributed ledger, and the distributed network in which the second distributed ledger is stored is referred to as a second distributed network. type networks can be composed of the same network. Further, when the composite surface image of the three-dimensional avatar is formed by a plurality of partial surface images, a plurality of second distributed ledgers exist according to the number of partial surface images.

Furthermore, for each token that has a corresponding relationship with a 3D avatar, in addition to the transaction history regarding the ownership name mentioned above, information regarding the partial surface image used for the 3D avatar is recorded in the distributed ledger. Specifically, in addition to the information on which partial surface images are used and to what extent, information on the overall three-dimensional surface shape of the object's head in the three-dimensional avatar, which is derived by the synthesis ratio deriving unit 34 described later, is provided. The composite ratio of the individual partial surface images is recorded.

Regarding the correspondence between each token and the three-dimensional avatar and partial surface image, the identifier of each token is linked one-to-one with the identification information of the corresponding three-dimensional avatar and partial surface image. More directly, the identifier of the token and the identification information of the three-dimensional avatar or the like may have the same content, but there is no problem in an embodiment in which a correspondence relationship is set between the identifiers of the token and the identification information of the three-dimensional avatar or the like.

The synthesis ratio derivation unit 34 is for deriving the synthesis ratio of partial surface images in the three-dimensional surface shape of the head for the three-dimensional avatar synthesized by the avatar synthesis unit 13. In the second embodiment, the composition ratio derivation unit 34 derives the composition ratio based on the ratio of the area of the area replaced by a specific partial surface image to the total surface area of the head surface excluding the hair part of the three-dimensional avatar. I am planning to do so. Furthermore, although the synthesis ratio may be derived individually for all partial surface images, in the second embodiment, it is derived collectively for each partial surface image belonging to the same ownership name.

The information output unit 35 sends information regarding the partial surface image used for the three-dimensional avatar, including the combination ratio derived by the combination ratio derivation unit 34 and identification information of the partial surface image used for combination, to the distributed network. It is for output. Specifically, the information output unit 35 is directly or indirectly connected to a decentralized network regarding tokens that correspond to three-dimensional avatars, and outputs information such as the synthesis ratio to the decentralized network. It has the function of Although the specific configuration of the information output unit 35 can be arbitrary as long as it can realize such a function, in the second embodiment, the information output unit 35 is configured to handle transactions and secrets whose contents include information regarding the composition ratio, etc. The mode is such that a digital signature generated using the key is output. By outputting these, the electronic signature is decrypted in the distributed network and when it is confirmed that the transaction is genuine, the contents of the transaction are recorded in the distributed ledger. Note that although the second embodiment adopts a configuration in which information regarding the change in the holding name of each token is recorded in a distributed ledger using an existing external system, the information output unit 35 can record the change in the holding name in the information output unit 35. It is also possible to output ownership change information consisting of a transaction and an electronic signature using a private key to a distributed network that stores a distributed ledger regarding each token.

Regarding the revenue generated when a 3D avatar is used as a character in a commercial game software, the distribution revenue calculation unit 36 calculates the amount of revenue generated when a 3D avatar is used as a character in commercial game software, and is calculated by the person who owns the 3D avatar and the partial surface image that makes up the 3D avatar. This is to calculate the distribution ratio to the person who holds the name. The distribution revenue calculation unit 36 is directly or indirectly connected to the distributed network, and refers to the transaction history recorded in the distributed ledger regarding non-fungible tokens corresponding to the three-dimensional avatar and each partial surface image. Then, the person who currently owns the three-dimensional avatar, etc. is determined.

Then, the distributed revenue calculation unit 36 calculates the 3D avatar based on the combination ratio derived by the combination ratio derivation unit 34, the identification information of the used partial surface image, the ownership name of the 3D avatar and the partial surface image, The distribution ratio and recipients of the profits generated will be calculated. As for the specific method of calculating the distribution ratio, the simplest structure is to derive it by multiplying the total revenue by the composite ratio, but other methods are possible as long as it is calculated using the composite ratio. But that's fine. For example, the distributed revenue calculation unit 36 weights each region on the head surface according to its importance, and for example, for a partial surface image including the eyes, a predetermined value that is greater than 1 is multiplied by the synthesis ratio. The distribution ratio may be the result of multiplying the coefficients.

Next, the advantages of the three-dimensional avatar generation device according to the second embodiment will be explained. In the second embodiment, by adopting a configuration in which the composition ratio derivation unit 34 derives the composition ratio, and the distribution revenue calculation unit 36 calculates the revenue distribution ratio based on the derived composition ratio, the three-dimensional avatar It has the advantage of being able to easily and fairly distribute profits when profits are earned. In particular, the synthesis ratio is derived using a certain algorithm (for example, in the second embodiment, the synthesis ratio is calculated by calculating the ratio of the area occupied by the partial surface image to the total surface area of the head surface excluding the hair part of the three-dimensional avatar). ), and the calculation of the revenue distribution ratio is also the same, so it is possible to easily realize fair revenue distribution without making arbitrary judgments in the series of operations.

In addition, in the second embodiment, non-fungible tokens are generated corresponding to each of the three-dimensional avatar and the partial surface image, and distribution revenue is calculated based on the recorded contents of the distributed ledger provided corresponding to each token. The structure is such that the division 36 adopts the revenue distribution ratio. When information is managed on a distributed ledger using a decentralized network related to non-fungible tokens, the possibility of information being destroyed or tampered with is extremely low. This has the advantage that errors in the revenue distribution ratio and revenue distribution destination can be significantly suppressed.

(Modified example)
Next, a modification of the second embodiment will be described. In addition to the configuration of Embodiment 2, the three-dimensional avatar generation device according to this modification does not generate a composite surface image by simply arranging the selected partial surface images; It has a configuration that generates a composite surface image by combining the facial image of the target object. Specifically, as shown in FIG. 6, the 3D avatar generation device according to this modification includes a 3D image generation unit 37 that converts a 2D facial image input through the facial image input unit 1 into 3D; A synthetic surface image generation unit 38 superimposes the facial image converted into a dimensional image and the partial surface image selected by the partial surface image selection unit 6, and a synthesis ratio is derived in consideration of the superposition mode etc. by the synthetic surface image generation unit 38. A combination ratio deriving unit 39 is provided.

The three-dimensional image generation unit 37 is for converting the facial image of the two-dimensional object input through the facial image input unit 1 into a three-dimensional image. Specifically, the three-dimensional image generation unit 37 moves the two-dimensional feature points of the facial image so as to correspond to the three-dimensional positional relationship derived by the positional relationship derivation unit 15, and also moves the two-dimensional feature points of the facial image to correspond to the three-dimensional positional relationship derived by the positional relationship derivation unit 15. It has a function of making the facial image three-dimensional by moving the position of the color, pattern, etc. in the surrounding area according to the movement of the feature point. Note that parts that are not represented in the two-dimensional facial image (for example, skin on the sides and back) may be left blank, or may be supplemented in the same manner as the complementation processing section 28. good.

The composite surface image generation unit 38 generates a composite surface image by combining the partial surface image selected by the partial surface image selection unit 6 and the facial image made three-dimensional by the three-dimensional image generation unit 37. It is for. Specifically, in addition to the complementary processing unit 28 in the second embodiment, the composite surface image generation unit 38 includes a surface shape adjustment unit 40 that changes the shape of not only the partial surface image but also the three-dimensional facial image. The apparatus includes an image synthesis section 41 that synthesizes the partial surface image and the facial image that have undergone surface shape adjustment processing by superimposing them at a predetermined mixing ratio.

The surface shape adjustment section 40 has a function of adjusting not only the surface shape of the partial surface image like the surface shape adjustment section 26 in the first and second embodiments, but also the surface shape of the facial image. Specifically, when the positions of a predetermined feature point on the partial surface image and the corresponding feature point on the face image are different, the surface shape adjustment unit 40 adjusts the feature point so that it matches the position of one of them. In addition to adjusting the position of the other, it has a function of moving the positions of both feature points to match their positions. The amount of movement of both feature points in the surface shape adjustment unit 40 may be determined by the user's selection, or the amount of movement of both feature points may be determined based on the results of deep learning etc. so that the shape becomes more natural or more characteristic. The amount of movement of the feature point may also be determined. Furthermore, the amount of movement of both feature points may be set to a uniform value for all partial surface images and facial images, or may be set to a different value for each partial surface image, or may be set to a different value for each feature point in the partial surface image. It may be a different value. Note that the content of determining the amount of movement of both feature points includes a case where only the feature points of the surface shape of the partial surface image are moved, and in this case, the surface shape adjustment unit 40 The same processing as that of the shape adjustment section 26 will be performed.

When the surface shape adjustment unit 40 moves the position of the feature point on the facial image, the surface shape adjustment unit 40 provides the head shape generation unit 4 and the synthesis ratio derivation unit 39 with respect to the position change information (for example, movement direction and movement distance). Output against. The head shape generation unit 4 adjusts the position of the feature point in the three-dimensional shape information of the object's head so that it matches the position of the adjusted feature point based on the variation information. The combination ratio derivation unit 39 derives the combination ratio while also referring to the fluctuation information.

The image synthesis unit 41 does not simply arrange the partial surface image on the three-dimensional surface shape of the object like the image synthesis unit 27 in the first and second embodiments, but instead arranges the partial surface image and the partial surface image. It has a function of arranging facial images (three-dimensionalized by the three-dimensional image generation unit 37) corresponding to the area in a superimposed state at a predetermined mixing ratio. Specifically, the image synthesis unit 41 mixes the partial surface image and the facial image, which have the same shape by adjusting the position of feature points by the surface shape adjustment unit 40, at a predetermined mixing ratio (for example, 8 for the partial surface image, 8 for the facial image, etc.). It has a function of superimposing the images in step 2) and placing them on the corresponding areas on the three-dimensional surface shape of the object's head. Note that the image synthesis unit 41 can also perform image synthesis with a mixing ratio of the partial surface image and the facial image of 10:0; however, in this case, the image synthesis unit 41 may The same processing as the combining section 27 will be performed.

The image synthesis section 41 outputs information regarding the mixing ratio to the synthesis ratio derivation section 39. The combination ratio derivation unit 39 derives the combination ratio while considering the information regarding the mixture ratio.

Similar to the synthesis ratio derivation unit 34 in the first and second embodiments, the synthesis ratio derivation unit 39 is for deriving the synthesis ratio of partial surface images on the head surface of the generated three-dimensional avatar. In this modification, the facial image and three-dimensional surface shape of the subject are transformed not only by simply arranging the partial surface image, but also by moving the feature points according to the shape of the partial surface image. Since a configuration is adopted in which a surface image and a facial image are superimposed and displayed at a predetermined mixing ratio, the synthesis ratio deriving unit 39 calculates the degree of deformation of the facial image and three-dimensional surface shape and the mixture of the partial surface image and the facial image. The composite ratio is derived by also referring to the ratio. Specifically, the synthesis ratio deriving unit 39 derives, for example, the ratio of the area of the area replaced by the specific partial surface image to the total surface area of the head surface excluding the hair part of the three-dimensional avatar. After that, the mixture ratio for the partial surface image is multiplied. For example, if the mixing ratio is partial surface image:facial image=9:1, it is multiplied by 0.9, and when partial surface image:facial image=5:5, it is multiplied by 0.5. After that, the combination ratio deriving unit 38 determines the degree of deformation of the three-dimensional surface shape in the region of the target partial surface image, for example, by multiplying the average moving distance of the feature points in the region by a predetermined coefficient. , derive the synthesis ratio in this modification. The derived combination ratio is output to the distributed network as in the second embodiment, and the distribution revenue calculation unit 36 calculates the revenue distribution ratio based on the combination ratio.

By adopting a configuration like this modified example, it is possible to not only simply arrange partial surface images, but also adjust the shape of the actual facial image of the object as necessary, or mix facial images and partial surface images. This has the advantage that a wide variety of composite surface images can be generated. For example, by deforming the shape of the facial image to match that of the partial surface image and then setting the mixing ratio of the partial surface image to 0, the shape of the facial image can be made to match the shape of the partial surface image, but the color and It is possible to generate a composite surface image in which the pattern of the facial image is maintained. In addition, the synthesis ratio deriving unit 39 derives the synthesis ratio by referring not only to the area ratio but also to the degree of movement of feature points in the surface shape of the object's head and the mixing ratio with the partial surface image. The modified example also has the advantage that it is possible to derive a synthesis ratio that appropriately evaluates the degree of contribution of a partial surface image in a wide variety of synthetic surface images, and that it is possible to calculate an appropriate and fair revenue distribution ratio.

Although the content of the present invention has been described above in Embodiments 1 and 2, the technical scope of the present invention should not be construed as being limited to the specific configuration described in the embodiments, and the functions of the present invention It goes without saying that various modifications and applications of the above-described embodiments fall within the technical scope of the present invention, as long as they can be realized.

First, in the first and second embodiments and the modified examples, the facial image of the object inputted through the facial image input section 1 is a two-dimensional image, but there is no need to limit it to such an aspect. A three-dimensional facial image may also be used. When a three-dimensional facial image is used, it becomes possible to omit the processing by the positional relationship derivation unit 15 and the processing by the three-dimensional image generation unit 37, and a three-dimensional avatar generation device is realized with a simpler configuration. There is also the advantage of being able to do so.

In addition, when setting feature points used for position adjustment, etc., it is possible to set a large number of feature points from the perspective of generating a higher-definition three-dimensional avatar. It may be only a limited part of the vertices of . For the same reason, the number of vertices included in the surface shape information may be reduced in the first place. Furthermore, for the position coordinates of feature points, instead of using relative position information from the corresponding basic feature points in the head element, an absolute coordinate system separate from the head element is used. It may also be configured to perform coordinate transformation as necessary.

Furthermore, in the first embodiment, an avatar consisting only of the head is used as the head body avatar, but an avatar including not only the head but also a torso portion may be used as the head body avatar. In this case, it is possible to generate a full-body avatar having the characteristics of the object based on the facial image without performing avatar synthesis by the avatar synthesis unit 13. Furthermore, when a head body avatar consisting only of a head is used, it may be configured such that the head avatar becomes an independent avatar without being combined with a body avatar or the like. Furthermore, instead of forming the head avatar in the shape of a skin head with the hair removed, a head avatar formed integrally with the hair may be generated.

Further, the moving means in the claims does not need to be interpreted as being limited to aspects such as the coordinate transformation unit 16. The three-dimensional positional relationship of the feature points of the face image is derived on the same coordinate system as the vertices of the head element avatar, without coordinate transformation. It is also possible to move the vertices of the partial element body avatar. Furthermore, regarding the movement/rotation of the feature points of the face image, the basic feature points of the head body avatar, or the group of vertices, it is possible to move not only one of them, but also both of them. .　

The present invention easily generates a high-quality three-dimensional avatar that reflects the characteristics of a real person, protects the privacy of the person, and conforms to the worldview set by the service used. It can be used as a technology to

1 Facial image input unit 2 Head body database 3 Head body selection unit 4 Head shape generation unit 5 Partial surface image database 6 Partial surface image selection units 7, 38 Composite surface image generation unit 9 Head avatar generation unit 10 Torso database 11 Hair database 12 Parts avatar selection section 13 Avatar composition section 14 Feature point extraction section 15 Positional relationship derivation section 16 Coordinate conversion section 17 Position adjustment section 18 Scaling section 19 Base transformation section 21 Classification selection section 22 Feature point comparison section 23 Image judgment section 24 Image output section 26, 40 Surface shape adjustment section 27, 41 Image composition section 28 Complement processing section 29 Position adjustment section 30 Reference axis adjustment section 31 Size adjustment section 32 Composition processing section 33 Token generation section 34, 39 Composition Ratio derivation unit 35 Information output unit 36 Distribution revenue calculation unit 37 Three-dimensional image generation unit

Claims

A 3D avatar generation device that generates a 3D avatar including a 3D surface shape of an object expressed using a group of vertices with defined positional relationships in a 3D space and a surface image displayed on the 3D surface shape. And,
a three-dimensional shape generating means for generating a three-dimensional surface shape of the head of the object in which the positional relationship of feature points corresponding to each element constituting the head shape among the vertices is specified based on the facial image of the object; ,
A surface image relating to all or a part of the surface shape of the head, and a plurality of partial surfaces in which the positional relationships of feature points corresponding to each element constituting the head shape in the whole or part of the region are specified. A partial surface image in which a positional relationship between two or more feature points matches a positional relationship between two or more feature points of the corresponding three-dimensional surface shape of the head from among the images. selection means,
synthetic surface image generating means for generating a synthetic surface image that is a surface image that matches the three-dimensional surface shape of the head by synthesizing the partial surface images selected by the partial surface image selecting means;
head avatar generation means for generating a head avatar based on the three-dimensional surface shape of the head and the composite surface image;
A three-dimensional avatar generation device comprising:
The composite surface image generation means combines the partial surface image selected by the partial surface image selection means with an image of a portion of the facial image of the object corresponding to the area of the partial surface image. The three-dimensional avatar generation device according to claim 1, wherein the three-dimensional avatar generation device generates the composite surface image.
composition ratio deriving means for deriving a composition ratio of the partial surface image on the three-dimensional surface shape of the head in the composite surface image;
A first step in which information on fluctuations in ownership of the three-dimensional avatar, identification information of partial surface images constituting the composite surface image, and the composition ratio are recorded with respect to non-fungible tokens that correspond to the generated three-dimensional avatar. information output means for outputting at least the composite ratio to a first distributed network in which one distributed ledger is stored;
Possession of the partial surface image generated with respect to the holding name of the three-dimensional avatar, the identification information, and the synthesis ratio recorded in the first distributed ledger, and a non-fungible token in correspondence with the partial surface image. Based on the ownership name of the partial surface image recorded in the second distributed ledger in which the change information of the name is recorded, the 3D avatar and the partial surface image regarding the revenue generated for the generated 3D avatar. a distribution revenue calculation means for calculating the distribution ratio among the holding names;
The three-dimensional avatar generation device according to claim 1 or 2, further comprising:
A 3D avatar generation method for generating a 3D avatar including a 3D surface shape of an object expressed using a group of vertices with defined positional relationships in a 3D space and a surface image displayed on the 3D surface shape. And,
a three-dimensional shape generation step of generating a three-dimensional surface shape of the head of the object in which the positional relationship of feature points corresponding to each element constituting the head shape among the vertices is specified based on the facial image of the object; ,
A surface image relating to all or a part of the surface shape of the head, and a plurality of partial surfaces in which the positional relationships of feature points corresponding to each element constituting the head shape in the whole or part of the region are specified. A partial surface image in which a positional relationship between two or more feature points matches a positional relationship between two or more feature points of the corresponding three-dimensional surface shape of the head from among the images. a selection step;
a synthetic surface image generation step of generating a synthetic surface image that is a surface image that conforms to the three-dimensional surface shape of the head by synthesizing the partial surface images selected in the partial surface image selection step;
a head avatar generation step of generating a head avatar based on the three-dimensional surface shape of the head and the composite surface image;
A three-dimensional avatar generation method comprising:
a composition ratio deriving step of deriving a composition ratio of the partial surface image on the three-dimensional surface shape of the head in the composite surface image;
A first step in which information on fluctuations in ownership of the three-dimensional avatar, identification information of partial surface images constituting the composite surface image, and the composition ratio are recorded with respect to non-fungible tokens that correspond to the generated three-dimensional avatar. an information output step of outputting at least the composite ratio to a first distributed network in which one distributed ledger is stored;
Possession of the partial surface image generated with respect to the holding name of the three-dimensional avatar, the identification information, and the synthesis ratio recorded in the first distributed ledger, and a non-fungible token that has a correspondence relationship with the partial surface image. The 3D avatar and the partial surface image regarding the revenue generated for the generated 3D avatar based on the ownership name of the partial surface image recorded in the second distributed ledger in which name change information is recorded. a distribution revenue calculation step of calculating a distribution ratio between holding names;
5. The three-dimensional avatar generation method according to claim 4, further comprising:
A three-dimensional avatar generation program that causes a computer to generate a three-dimensional avatar including a three-dimensional surface shape of the object expressed by a group of vertices and a surface image displayed on the three-dimensional surface shape, based on a facial image of the object. ,
to the computer,
A 3D avatar generation device that generates a 3D avatar including a 3D surface shape of an object expressed using a group of vertices with defined positional relationships in a 3D space and a surface image displayed on the 3D surface shape. And,
a three-dimensional shape generation function that generates a three-dimensional surface shape of the object's head in which the positional relationship of feature points corresponding to each element constituting the head shape among the vertices is specified based on the object's facial image; ,
A surface image relating to all or a part of the surface shape of the head, and a plurality of partial surfaces in which the positional relationships of feature points corresponding to each element constituting the head shape in the whole or part of the region are specified. A partial surface image in which a positional relationship between two or more feature points matches a positional relationship between two or more feature points of the corresponding three-dimensional surface shape of the head from among the images. selection function,
a synthetic surface image generation function that generates a synthetic surface image that is a surface image that matches the three-dimensional surface shape of the head by synthesizing the partial surface images selected by the partial surface image selection function;
a head avatar generation function that generates a head avatar based on the three-dimensional surface shape of the head and the composite surface image;
A three-dimensional avatar generation program that is characterized by realizing the following.