KR20230105110A - Method of generating a target object model imitating the characteristics of the source object and device for the same method - Google Patents

Abstract

The present invention relates to a method of generating a model that mimics the characteristics of a source object and relates to a rendering method. A method of generating a target object model that mimics the characteristics of a source object according to the present invention includes acquiring a plurality of source images of a source object according to lighting conditions; generating a texture map from the source image; extracting a rendered image by applying the generated texture map to a pre-generated target base model; calculating an error between pixels of the extracted rendered image and the source image; and updating a parameter of the texture map according to the calculated error, wherein the step of extracting the rendered image is repeatedly performed to minimize the calculated error between pixels. According to the present invention, it is possible to create a realistic digital human that mimics the facial features of a person using only a photographed camera image.

Description

Method of generating a target object model imitating the characteristics of the source object and device for the same method}

The present invention relates to a method of generating a model that mimics the characteristics of a source object and relates to a rendering method.

Recently, with the development of content production technology using graphic engines such as computer graphics (CG) and visual special effects (VFX), various environments can be implemented in a virtual studio with minimal sets and movies, advertisements, music videos, live performances, etc. was able to film.

In addition, after the spread of Corona 19, the need for VFX technology to create virtual space has greatly expanded as the performance industry, exhibition, and event industries have shifted their activities from offline to online. Through the virtual space called metaverse, it is possible to provide unlimited experience beyond the temporal and spatial limitations of the real world.

Demand for technology that creates and renders more realistic humans in a virtual space where unlimited services are provided extends beyond visual effects for simple entertainment to education and medical fields.

However, since digital humans created through computer graphics must virtually implement various components including human body attributes, in order to create a virtual character that reflects the unique characteristics of humans, it has a greater complexity than general objects. Modeling takes a lot of time and money.

In general, human modeling reproduces human characteristics by rendering the way light interacts with the skin by reflecting the characteristics of light and camera. It is very difficult in itself to perform sampling of incoming and outgoing light at all points of the .

Accordingly, more efficient human rendering methods are continuously being devised.

An object of the present invention is to propose a method for producing a digital human imitating human characteristics.

More specifically, an object of the present invention is to propose an efficient rendering method by extracting human facial features through various cameras and lighting and applying them to a basic model.

In addition, an object of the present invention is to use a deep learning-based feature extraction method to reflect a person's personalized expression or characteristics.

A method for generating a target object model that mimics the characteristics of a source object according to the present invention for solving the above technical problem includes obtaining a plurality of source images of a source object according to lighting conditions; generating a texture map from the source image; extracting a rendered image by applying the generated texture map to a pre-generated target base model; calculating an error between pixels of the extracted rendered image and the source image; and updating a parameter of the texture map according to the calculated error, wherein the step of extracting the rendered image is repeatedly performed to minimize the calculated error between pixels.

Preferably, in the obtaining of the intermediate rendered image, an updated rendered image is extracted by applying the texture map having the updated parameter to the target base model.

extracting a plurality of predetermined 2-dimensional first landmarks from the plurality of source images; matching the extracted first landmark with a second landmark in a 3D space in consideration of an anatomical feature; and generating a target base model by applying the matched second landmark to scan data of a target object.

and aligning (editing) the phase of the interest point of the target object according to the phase of the interest point of the source object.

The updated parameter includes at least one of albedo, specular, and shape values.

The shape value includes a geometry vector defining the global shape of the target object and a normal vector defining the local shape.

Preferably, the step of extracting the first landmark is extracted using a pre-learned neural network.

In the step of updating the parameter, it is preferable to use the extracted second landmark as an update constraint condition of the parameter.

A method for generating a target object model that mimics the characteristics of a source object according to the present invention for solving the above technical problem includes obtaining a plurality of source images of a source object according to lighting conditions; generating a texture map from the source image; extracting a rendered image by applying the generated texture map to a target base model generated by applying second landmarks extracted from the plurality of source images to scan data of a target object; calculating an error between pixels of the extracted rendered image and the source image; and updating a parameter of the texture map according to the calculated error, wherein the step of extracting the rendered image is repeatedly performed to minimize the calculated error between pixels.

Preferably, in the obtaining of the intermediate rendered image, an updated rendered image is extracted by applying the texture map having the updated parameter to the target base model.

In the step of updating the parameter, it is preferable to use the extracted second landmark as an update constraint condition of the parameter.

A method for generating a target object model that mimics the characteristics of a source object according to the present invention for solving the above technical problem includes acquiring a plurality of source images of a face of a source object according to lighting conditions; generating a texture map from the source image; extracting a rendered image by applying the generated texture map to a target base model generated by applying second landmarks extracted from the plurality of source images to scan data of a target object; and calculating an error between pixels of the extracted rendered image and the source image. updating a parameter of the texture map according to the calculated error; and applying the updated rendered image to the face of the virtual character by applying the texture map having the updated parameters to the target base model.

According to the present invention, it is possible to create a realistic digital human that mimics the facial features of a person using only a photographed camera image.

In addition, according to the present invention, dynamic depiction is possible by creating a three-dimensional model reflecting the characteristics of facial muscles that embody a person's unique expression and rendering facial features to the created model.

In addition, the present invention extracts facial landmarks using deep learning and uses the landmarks for 3D modeling, thereby enabling more efficient production of digital humans.

Furthermore, the present invention can extend the digital human by providing an interface for modifying the topology of various objects.

1 shows the structure of a system for performing a method of generating a target object model that mimics the characteristics of a source object according to an embodiment of the present invention.
2 and 3 show a process of a method for generating a target object model that mimics the characteristics of a source object according to an embodiment of the present invention.
4 shows an example of a source image of a method for generating a target object model that mimics the characteristics of a source object according to an embodiment of the present invention.
5 shows an example of a texture map of a method for generating a target object model that mimics the characteristics of a source object according to an embodiment of the present invention.
6 shows a base model creation process of a method for generating a target object model that mimics the characteristics of a source object according to an embodiment of the present invention.
7 shows an example of a neural network of a method for generating a target object model that mimics the characteristics of a source object according to an embodiment of the present invention.
8 illustrates an example of a line landmark of a method for generating a target object model that mimics the characteristics of a source object according to an embodiment of the present invention.
9 illustrates an example of generating a base model of a method for generating a target object model that mimics characteristics of a source object according to an embodiment of the present invention.
10 shows an iterative process of a method for generating a target object model that mimics the characteristics of a source object according to an embodiment of the present invention.

The following merely illustrates the principles of the invention. Therefore, those skilled in the art can invent various devices that embody the principles of the invention and fall within the concept and scope of the invention, even though not explicitly described or shown herein. In addition, all conditional terms and embodiments listed in this specification are, in principle, clearly intended only for the purpose of understanding the concept of the invention, and it should be understood that it is not limited to the specifically listed embodiments and conditions. .

The above objects, features and advantages will become more apparent through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art to which the invention belongs will be able to easily implement the technical idea of the invention. .

In addition, in describing the invention, if it is determined that a detailed description of a known technology related to the invention may unnecessarily obscure the subject matter of the invention, the detailed description will be omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 shows the structure of a system for performing a method of generating a target object model that mimics the characteristics of a source object 10 according to an embodiment of the present invention.

Referring to FIG. 1, the system according to this embodiment generates a camera 15, a lighting device (not shown), and a model that shoots a source object 10 (eg, an actor or a specific person) according to various camera settings. It may be configured as a model generating device 100.

The model generating apparatus 100 receives a plurality of source images taken from the camera, extracts features of the source object 10 from the source images, and uses them for rendering.

Specifically, the image input unit may receive a plurality of source images from the camera, and the model generator may generate a target base model using point information as a 3D depth map of the source images.

The model creation unit modifies the basic template model of a male or female according to the type of object, for example, using information such as a 3D depth map or a point cloud, thereby reflecting the 3D characteristics unique to the source object 10. model can be created.

The rendering unit may generate a target object model by performing a rendering operation of applying a texture map generated from a source image to the target base model.

At this time, since the target object model is rendered based on the first texture map, there may be a difference (loss) from the characteristics of the actual source object 10, and this difference may be repeatedly performed to minimize.

Specifically, the target object model may be updated by extracting a rendered image from the target object model, calculating a pixel-by-pixel difference from the original source image, and changing weights of attributes of the texture map according to the difference.

Furthermore, in this embodiment, the model generation unit extracts landmarks using a neural network learned from the source image and creates a 3D target base model using the landmarks.

Hereinafter, with reference to FIG. 2, a method for generating a target object model that mimics the characteristics of the source object 10 according to the present embodiment will be described in more detail.

First, a plurality of source images of the source object 10 are acquired according to lighting conditions (S100).

That is, lighting according to various conditions is radiated to the source object 10 and the skin tone or texture of the source object 10 can be accurately sensed.

For example, an image including color information of the source object 10 may be obtained from a color camera, and the face of the subject is illuminated as uniformly as possible to obtain color information in a state in which the shadow of the face is excluded. and use cross-polarization to capture directly specularly reflected light from the skin and de-shadowed images (albedo images).

In addition, it is also possible to obtain a color point cloud in which color values are assigned to 3D points having depth information about the source object 10 using an RGB-D camera and depth information.

Next, a texture map is created from the source image (S200).

Create a texture map to express the texture of the face from multiple images according to camera and various lighting conditions.

In addition to using the image received from the camera as it is, it is also possible to generate a feature map in the form of emphasizing various characteristics through post-processing of the captured image.

For example, a specular image may be generated using outputs of a color camera and a cross-polarization camera. The specular image is a model of light reflected from the surface of the skin, and may include details due to pores or fine wrinkles on the skin.

A rendered image is extracted by applying the texture map generated through the above process to a pre-generated target base model (S300).

A first rendering image may be extracted from an initial target object model generated by applying a texture map obtained from a source image to a target base model.

However, since the characteristics of the source object 10 to be imitated may not be fully reflected in the first rendered image at this time, a pixel error between the extracted rendered image and the source image is calculated.

Next, the process of updating the parameters of the texture map according to the calculated error, applying the updated texture map to the target base model, and extracting the second rendered image is repeatedly performed.

The step of extracting the rendered image (S300) may be repeatedly performed so that the calculated inter-pixel error is minimized or becomes less than or equal to a predetermined threshold value.

A method of generating a target object model according to the present embodiment will be described in more detail with reference to FIGS. 3 to 5 .

Referring to FIG. 3 , a source object 10 as a subject may be irradiated with light under a predetermined condition, and a camera may generate a source image by receiving light reflected from the source object 10 . Lighting conditions used for photography may be input as lighting correction values in a subsequent rendering process, and through this, a target object model in which the influence of lighting is minimized may be generated.

A camera that acquires a source image may use a cross-polarized camera or an RGB-D camera in addition to a general color camera, and a plurality of source images obtained from a plurality of cameras may be used to acquire additional source images through post-processing. .

Referring to FIG. 4 , in this embodiment, the source image may generate an original image captured by a color camera, an albedo image captured using cross polarization, and a specular image generated by differentially dividing the original image and the albedo image.

In addition, in the case of an RGB-D camera, 3D information such as a point cloud may be obtained as a source image as depth map information.

The following source images may be generated as texture maps expressing various characteristics by being blended after going through a projection process of images photographed at various angles. In the blending process, for example, a texture may be generated through synthesis with a Laplacian pyramid using a Gaussian pyramid and a mask for an object.

Referring to FIG. 5 , in this embodiment, the texture map includes a feature map defining albedo values, a specular map, and a normal map.

The albedo map and the specular map may be generated by blending the above-described albedo source image and the specular source image, and the normal map may include normal vector information defining the direction of the surface of the object.

The above generated texture maps may be reflected in a base model that models the geometric shape of an object and generated as an initial target model.

In this case, the base model may be generated with scan data of the above-described RGB-D camera, and in this embodiment, the base model is configured in consideration of the shape of the source object 10 and the shape of the muscles of the object expressing motion. It can be.

Referring to FIG. 6 , in the present embodiment, the base model may be created using point cloud information as scan data and line landmarks defining the shape of muscles as landmarks where movements in the anatomical structure of the face are generated. .

In addition, line landmarks can be reconstructed using 2-dimensional landmarks as 3-dimensional information.

Also, in this embodiment, a 2D landmark may be extracted using a learned neural network.

That is, by inputting an image obtained from a color camera to the learned neural network, coordinate information of anatomical features of the face may be output. For example, facial contours, eyes, nose, and mouth positions can be extracted as feature points.

Referring to FIG. 7 , the neural network 70 can be trained to extract the positions of facial feature information in a source image as feature points, and through this, the positions of key features such as the position of the eyes, the position of the nose, or the outline of the face within the face without human manual intervention. Allow information to be output.

A line landmark can be created by reconstructing the position of the feature point created through the above process in 3D in consideration of the shooting angle. In addition, when generating a line landmark, it is possible to extract feature lines by reflecting the structural features of the entire face by reflecting the depth information of the scan data.

Referring to FIG. 8 , a line landmark, unlike a 2D landmark, may be configured in a curved shape in a 3D space, and an anatomical structure for operating a corresponding part may be defined rather than a location of major features such as eyes or a nose. there is.

For example, shape and location information of muscles around the eyes and muscles that move the cheeks and mouth may be reconstructed into the line landmark 125 .

A base model is created by fitting a template model and a corresponding template landmark based on the line landmarks generated through the above process.

Referring to FIG. 9 , a specific shape of the base model may be fitted based on the template model 94 using the line landmark 125 and the template landmark 92 . In addition, a more precise shape of the base model 50 is implemented by utilizing the depth information 96 obtained in advance in the fitting process.

The generated base model can again be configured as a smooth 3D model through surface optimization, and additionally, in this embodiment, the generated base model is phase aligned so that it can reflect the positional characteristics of individual eyes, noses, and mouths. By doing so, it is possible to generate a phase-aligned base model.

The phase alignment process is to align the phase of the interest point of the base model 50 according to the phase of the interest point of the source object 10, reflecting the relative positional relationship of eyes, nose, and mouth peculiar to the source object. let it

Furthermore, alignment of the phase may be performed through a user's correction input, and an interface for this may be directly provided to the user. Also, the interface may be implemented to utilize the above-described line landmark as a tool for alignment.

The user's correction information can be reflected through an elastic deformation process, and the phase alignment model can be updated through surface optimization again after deformation.

Referring back to FIG. 3 , a final target model is created by optimizing an initial target model generated by reflecting a texture map on a generated target base model through a rendering process.

Specifically, referring to FIG. 10 , an image 104 under the same conditions as the source image 102 is obtained by inputting lighting correction and camera correction values according to lighting conditions and camera conditions photographed from the target model through an inverse rendering process, , the difference between the acquired image and the source image can be calculated.

The obtained image and the source image can be compared in units of pixels as a result of the above-described phase alignment process, and through this, an error between pixels can be calculated.

The iterative process may change a parameter or weights of parameters in the texture map to reduce the error by comparing the calculated error with a predetermined threshold. The above process may be repeatedly performed so that the error is less than the threshold value or the error is maximized through numerical prediction.

Specifically, the parameter 106 updated through an iterative process includes at least one of albedo (β), specular (γ), and shape values. Also, the shape value includes a geometry vector α defining the global shape of the target object and a normal vector δ defining the local shape.

The target object model 55 may be generated by applying the texture map optimized through the above process to the base model.

In addition, by applying a texture map with updated parameters to the face of a virtual character in a 3D space, objects in the metaverse environment can be implemented to directly operate as avatars of users or other objects.

According to the present invention, it is possible to create a realistic digital human that mimics human facial features using only the captured camera image.

In addition, according to the present invention, dynamic depiction is possible by creating a three-dimensional model reflecting the characteristics of facial muscles that embody a person's unique expression and rendering facial features to the created model.

In addition, the present invention extracts facial landmarks using deep learning and uses the landmarks for 3D modeling, thereby enabling more efficient production of digital humans.

Furthermore, the present invention can extend the digital human by providing an interface for modifying the phases of various objects.

Furthermore, various embodiments described herein may be implemented in a recording medium readable by a computer or a device similar thereto using, for example, software, hardware, or a combination thereof.

According to hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions. The described embodiments may be implemented as a control module itself.

According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. The software code may be implemented as a software application written in any suitable programming language. The software code may be stored in a memory module and executed by a control module.

The above description is merely an example of the technical idea of the present invention, and those skilled in the art can make various modifications, changes, and substitutions without departing from the essential characteristics of the present invention. will be.

Therefore, the embodiments disclosed in the present invention and the accompanying drawings are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments and the accompanying drawings. . The protection scope of the present invention should be construed according to the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

Claims (12)

acquiring a plurality of source images of a source object according to lighting conditions;
generating a texture map from the source image;
extracting a rendered image by applying the generated texture map to a pre-generated target base model;
calculating an error between pixels of the extracted rendered image and the source image; and
Updating parameters of the texture map according to the calculated error;
Wherein the step of extracting the rendered image is repeatedly performed to minimize the calculated inter-pixel error. According to claim 1,
Wherein the extracting of the rendered image extracts an updated rendered image by applying the texture map having the updated parameters to the target base model. According to claim 2,
extracting a plurality of predetermined 2-dimensional first landmarks from the plurality of source images;
matching the extracted first landmark with a second landmark in a 3D space in consideration of an anatomical feature;
and generating a target base model by applying the matched second landmark to scan data of the target object. According to claim 3,
Editing the phase of the interest point of the target object according to the phase of the interest point of the source object. According to claim 1,
The updated parameter includes at least one of albedo, specular, and shape values. According to claim 5,
The shape value includes a geometry vector defining a global shape of the target object and a normal vector defining a local shape of the target object. According to claim 3,
The method of generating a target object model that mimics the characteristics of a source object, characterized in that the step of extracting the first landmark is extracted using a pre-learned neural network. According to claim 3,
Wherein the updating of the parameter uses the extracted second landmark as an update constraint condition of the parameter. acquiring a plurality of source images of a source object according to lighting conditions;
generating a texture map from the source image;
extracting a rendered image by applying the generated texture map to a target base model generated by applying second landmarks extracted from the plurality of source images to scan data of a target object;
calculating an error between pixels of the extracted rendered image and the source image; and
Updating parameters of the texture map according to the calculated error;
Wherein the step of extracting the rendered image is repeatedly performed to minimize the calculated inter-pixel error. According to claim 9,
Wherein the extracting of the rendered image extracts an updated rendered image by applying the texture map having the updated parameter to the target base model. According to claim 10,
Wherein the updating of the parameter uses the extracted second landmark as an update constraint condition of the parameter. acquiring a plurality of source images of a face of a source object according to lighting conditions;
generating a texture map from the source image;
extracting a rendered image by applying the generated texture map to a target base model generated by applying second landmarks extracted from the plurality of source images to scan data of a target object;
calculating an error between pixels of the extracted rendered image and the source image;
updating a parameter of the texture map according to the calculated error; and
and applying the updated rendered image to the face of the virtual character by applying the texture map having the updated parameters to the target base model.

Images