KR20230075364A - A System and Method for Replacing 3D Character - Google Patents

Abstract

The present invention relates to a 3D character substitution system and method, which includes a modeling data storage unit for storing modeling data of a 3D character, a content image in which 3D content is rendered in 2D, and a character in the received content image. A character information extraction unit recognizing character information including location information of, a character area removal unit recognizing and removing a character area within the content image, controlling modeling data of the 3D character based on the character information, A 3D character replacement system including a character image generation unit that renders and generates a 2D image for the removed character region and an image output unit that synthesizes the generated 2D image with the content image and outputs the same, and an operating method thereof Provided.

Description

3D Character Replacement System and Method {A System and Method for Replacing 3D Character}

The present invention relates to a system and method for replacing a 3D character, and more particularly, to a system and method for enabling a user to enjoy content using his or her own 3D character in content such as a game.

As metaverse technology develops, more and more users express themselves and communicate using 3D characters in virtual space. Accordingly, an increasing number of users continuously manage 3D characters in virtual space and assign identities to them.

As interest in characters in virtual space increases, there is an increasing demand for characters to appear in content while enjoying content such as games.

As in Korean Patent Laid-open Publication No. 10-2000-0037574, "Method for Making Games Using Virtual Characters", a technique for allowing a separately produced virtual character to appear in a game has been developed in the past. However, since this technology is a method of producing a game by registering the image of the character in the production stage of the game program, as in the case of the above prior patent, it is difficult for general users to make their characters appear in the content, There is a problem in that the character cannot be changed after content production is completed, and even if the character is newly decorated, this content cannot be reflected.

Therefore, there is a need for a technology that allows a 3D character managed in a virtual space to appear in content, and when the character changes, the content can be reflected in each content.

Korean Patent Publication No. 10-2000-0037574

An object of the present invention is to enable a user's individual 3D character to appear in content such as a game.

An object of the present invention is to enable a single character created by a user in a metaverse to appear in various contents.

An object of the present invention is to allow a user to modify his/her character, so that the modified content can be reflected in real time within various contents.

In order to achieve this object, the 3D character replacement system according to an embodiment of the present invention receives a modeling data storage unit for storing modeling data of a 3D character, a content image in which 3D content is rendered in 2D, and the A character information extractor for recognizing character information including location information of characters in the received content image, a character region removal unit for recognizing and removing character regions within the content image, It may include a character image generation unit that controls and renders modeling data to generate a 2D image for the removed character region, and an image output unit that synthesizes the generated 2D image with the content image and outputs the result. .

At this time, the character information extractor identifies a pixel representing a character among pixels of the content image, and the character image generator matches the rendered image of the 3D character to the pixel representing the character. Outside pixels can be treated as transparent pixels.

In addition, the character information extraction unit recognizes character information including motions of characters in the received content image, and the character image generation unit controls modeling data of the 3D character to match the motions of characters in the content image. , it is possible to generate a 2D image by rendering the controlled modeling data.

In addition, the character information extractor recognizes character information including the angle of the character in the received content image, and the character image generator controls the modeling data of the 3D character to match the angle of the character in the content image. , it is possible to generate a 2D image by rendering the controlled modeling data.

The present invention enables a user's individual 3D character to appear in content such as a game.

The present invention allows a single character created by a user in a metaverse to appear in various contents.

According to the present invention, when a user modifies his/her character, the modified content can be reflected in real time within various contents.

1 is a configuration diagram showing the internal configuration of a three-dimensional character replacement system according to an embodiment of the present invention.
2 is a diagram showing an example of a screen for replacing a 3D character in a 3D character replacement system according to an embodiment of the present invention.
Figure 3 is a flow chart showing the flow of a three-dimensional character substitution method according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description will be omitted. In addition, in describing the embodiments of the present invention, specific numerical values are only examples, and the scope of the invention is not limited thereby.

The 3D character replacement system according to the present invention may be configured in the form of a server having a central processing unit (CPU) and a memory (Memory) and connectable to other terminals through a communication network such as the Internet. However, the present invention is not limited by the configuration of the central processing unit and memory. In addition, the 3D character replacement system according to the present invention may be physically configured as one device or implemented in a distributed form in a plurality of devices.

1 is a configuration diagram showing the internal configuration of a three-dimensional character replacement system according to an embodiment of the present invention.

The 3D character replacement system 101 according to the present invention includes a modeling data storage unit 110, a character information extraction unit 120, a character area removal unit 130, a character image generation unit 140, and an image output unit 150. ). Each component may be a software module that operates in the same computer system physically, or may be configured so that two or more physically separated computer systems can operate in conjunction with each other. Embodiments belong to the scope of the present invention.

The modeling data storage unit 110 stores modeling data of a 3D character. In general, a 3D object exists in the form of modeling data, and when displayed on a screen, it is converted into a 2D image through a rendering process and then output to the screen. Accordingly, the modeling data may indicate various information such as shape, size, posture, and expression of the 3D character necessary for the 3D character to be displayed in the 3D virtual space.

The modeling data stored in the modeling data storage unit 110 may be controlled according to a user's control command, etc., and through this, it is possible to control the posture, angle, movement, facial expression, etc. of a 3D character in various ways. When the modeling data is controlled in this way, the result appears on the screen where it is rendered.

The character information extractor 120 receives a content image in which 3D content is rendered in 2D, and recognizes character information including location information of a character in the received content image. The content image is an image for expressing a 3D virtual space on the screen, and the content may include various things represented by 3D graphics such as games and metaverses.

The character information extractor 120 may analyze a received image to be displayed on a screen, identify a character to be replaced in the image, and then receive information about the character. An image received to be displayed on a screen may be data generated or streamed in real time, or may be video data generated in advance and stored in a storage device. To identify a character, various artificial intelligence techniques for image processing may be applied, and when a plurality of characters appear, a process of receiving a selection input from a user may be performed to select a character to be replaced among them.

The character information extraction unit 120 may utilize information about the 3D content in order to more accurately identify a character from which character information should be extracted. For example, if an application assumes that the user's character is always located at the bottom center of the screen, the application that generated the received content image is identified as the corresponding application, and the character is always identified in the area where the character is located. more accurately identified.

The character information recognized by the character information extractor 120 is the information on the location of the character, and information on the location of the character and the area occupied by the character in the image can be extracted and replaced. In this way, various artificial intelligence methods can be applied to recognize the character's area, and the present invention is not limited by this method.

The character information extractor 120 may identify a pixel representing a character among pixels of the content image. When the area occupied by the character in the image is identified, among the pixels constituting the image, which pixel represents the character and which pixel represents another object can be distinguished. Through this, character replacement can be made possible by confirming the position of a pixel for expressing a character and expressing a character to be replaced at this position.

The character information extractor 120 may recognize character information including motions of characters in the received content image. In 3D content, the character is not fixed like an image, but moves according to the progress of the content or the user's manipulation. Therefore, even if the character is replaced, the original character's motion must be performed the same as if the substituted character is a real character, rather than simply replacing the character's shape. To this end, it is necessary to recognize the original character's motion.

There may be several methods for recognizing the motion of a character in the character information extractor 120. Recently, a number of technologies for recognizing the motion of each joint using artificial intelligence technology and expressing it as an avatar have emerged, The present invention is not limited by specific recognition methods of such technologies.

The character information extractor 120 analyzes the user's input while using the 3D content in order to accurately recognize the motion of the character, and checks how the user's input is connected to the motion of the existing character in the content, Based on this, it can be implemented to recognize the motion of the character.

The character information extractor 120 recognizes character information including angles of characters in the received content image. A 3D character is expressed differently depending on the angle at which it is located in the 3D coordinate space. The 3D character to be replaced should look at the front at the same angle as the angle at which the front of the character in the original content is viewed so that it is natural without a sense of incongruity. substitution is possible.

To this end, the character information extractor 120 recognizes a standard feature to recognize the direction and angle, such as a character's face, determines the angle of the character based on the position of the recognized feature, and then creates a new character based on this. By replacing , it is possible to provide the same effect that the replaced character operates identically to the existing character.

The character area removal unit 130 recognizes and removes the character area within the content image. As described above, when a character area is recognized in a content image, information on the character area can be checked, and pixels representing a character and pixels that do not constitute a screen can be distinguished from pixels constituting a screen.

Accordingly, the character area removal unit 130 may prevent the existing character from appearing in the content image by removing the color of the pixel portion representing the character in the image.

The character area remover 130 may remove the character area from the image and then fill the character area with an image most similar to the background by referring to an area around the character area. Through this, it is possible to create an image in which only the background exists, as if the character did not exist in the corresponding content. This process can apply various artificial intelligence techniques, and the present invention is not limited by a specific method of filling the area.

The character image generation unit 140 controls and renders modeling data of the 3D character based on the character information to generate a 2D image of the removed character area. As described above, when various information about the character included in the original 3D content is obtained, a character to be replaced is prepared based on this information and synthesized, so that the character in the content is changed to the user's unique character. effect can be provided.

To this end, the character image generation unit 140 can be configured to perform the same motion at the same position, the same angle, and the same motion as the existing character of the actual 3D content by controlling the stored modeling data of the user's unique character. By rendering and generating a 2D image, it is possible for the user to use other 3D content as his own character.

The character image generation unit 140 matches the rendered image of the 3D character to pixels representing the character, and processes other pixels as transparent pixels. Since information on pixels representing characters is removed from the existing content video, this part must be filled with a rendered image of a new 3D character. is filled with the rendered image of the new 3D character, leaving the rest as transparent pixels. When this is combined with the content image from which the character region has been removed, the remaining parts are reproduced as they are through the transparent part, and a final image in which only the character region is converted is created.

The character image generating unit 140 controls modeling data of the 3D character to match the motion of the character in the content image, and renders the controlled modeling data to generate a 2D image. As described above, when the motion of the character in the original content video is recognized, the corresponding motion can be applied to the modeling data of the user's 3D character. Rendered images of dimensional characters can be synthesized. Through this, it is possible to obtain the same effect as if the actual user's unique character is active within the content image.

The character image generation unit 140 controls modeling data of the 3D character to match the angle of the character in the content image, and renders the controlled modeling data to generate a 2D image. In order to replace a character, there must be a 3D character to be replaced at the same angle as the existing character in the content video. For this purpose, the modeling data of the 3D character is controlled based on the angle information of the recognized existing character By rendering, it is possible to replace characters without discomfort.

The image output unit 150 synthesizes the generated 2D image with the content image and outputs the synthesized image. As described above, since the image generated by the character image generator 140 fills the character area with a rendered image of a new 3D character and leaves the remaining area as transparent pixels, it is overlapped with the original content image without complicated calculations. Final video can be created.

2 is a diagram showing an example of a screen for replacing a 3D character in a 3D character replacement system according to an embodiment of the present invention.

As shown in the figure, when there is an application that provides 3D content, the application performs rendering on the terminal to provide an image of the 3D content, or performs rendering on the server and then streams it. It is possible to provide images for 3D content.

Since the 3D character replacement system 201 according to the present invention recognizes and replaces characters in the image by taking the image to be output on the screen as an input, the application does not separately support or insert a module. Even if it is not necessary, it is possible to convert the character of the application into the user's unique character and obtain the same effect as using the user's character in the application.

When a character area is recognized in the content image of an application as indicated by reference numeral 201 and the character area is removed, an image indicated by reference numeral 202 may be obtained. At this time, if the user's unique character is prepared to fit the removed character area and synthesized with the image of reference 202, the user's unique 3D character, as shown by reference 203, can be used in the corresponding 3D content. can be made

Figure 3 is a flow chart showing the flow of a three-dimensional character substitution method according to an embodiment of the present invention.

The 3D character substitution method according to an embodiment of the present invention is a method operating in a 3D character substitution system 101 having a central processing unit (CPU) and a memory, and the above-described 3D character substitution system ( 101) can be applied as it is. Therefore, even if there is no separate description below, it is obvious that all the contents described to describe the 3D character replacement system can be applied as they are to implement the 3D character replacement method.

The modeling data storage step (S301) stores modeling data of a 3D character. In general, a 3D object exists in the form of modeling data, and when displayed on a screen, it is converted into a 2D image through a rendering process and then output to the screen. Accordingly, the modeling data may indicate various information such as shape, size, posture, expression, and the like of the 3D character necessary for the 3D character to be displayed in the 3D virtual space.

The modeling data stored in the modeling data storage step (S301) can be controlled according to a user's control command, etc., and through this, it is possible to control the posture, angle, movement, facial expression, etc. of the 3D character in various ways. When the modeling data is controlled in this way, the result appears on the screen where it is rendered.

In the character information extraction step (S302), a content image in which 3D content is rendered in 2D is received, and character information including location information of a character in the received content image is recognized. The content image is an image for expressing a 3D virtual space on the screen, and the content may include various things represented by 3D graphics such as games and metaverses.

In the character information extraction step (S302), a received image may be analyzed to be displayed on a screen, a character to be replaced in the image may be identified, and then information about the character may be received. An image received to be displayed on a screen may be data generated or streamed in real time, or may be video data generated in advance and stored in a storage device. To identify a character, various artificial intelligence techniques for image processing may be applied, and when a plurality of characters appear, a process of receiving a selection input from a user may be performed to select a character to be replaced among them.

In the character information extraction step (S302), information about the 3D content may be utilized in order to more accurately identify a character from which character information should be extracted. For example, if an application assumes that the user's character is always located at the bottom center of the screen, the application that generated the received content image is identified as the corresponding application, and the character is always identified in the area where the character is located. more accurately identified.

The character information recognized in the character information extraction step (S302) is the information on the character's position, and information on the position of the character and the area occupied by the character in the image can be extracted and replaced. In this way, various artificial intelligence methods can be applied to recognize the character's area, and the present invention is not limited by this method.

In the character information extraction step (S302), a pixel representing a character among pixels of the content image may be identified. When the area occupied by the character in the image is identified, among the pixels constituting the image, which pixel represents the character and which pixel represents another object can be distinguished. Through this, character replacement can be made possible by confirming the position of a pixel for expressing a character and expressing a character to be replaced at this position.

In the character information extraction step (S302), character information including motions of characters in the received content image may be recognized. In 3D content, the character is not fixed like an image, but moves according to the progress of the content or the user's manipulation. Therefore, even if the character is replaced, the original character's motion must be performed the same as if the substituted character is a real character, rather than simply replacing the character's shape. To this end, it is necessary to recognize the original character's motion.

In the character information extraction step (S302), there may be several methods for recognizing the motion of the character. Recently, many technologies have emerged that recognize the motion of each joint using artificial intelligence technology and express it as an avatar. The present invention is not limited by specific recognition methods of such technologies.

The character information extraction step (S302) analyzes the user's input while using the 3D content in order to accurately recognize the motion of the character, and checks how the user's input is connected to the motion of the existing character in the content, Based on this, it can be implemented to recognize the motion of the character.

In the character information extraction step (S302), character information including angles of characters in the received content image is recognized. A 3D character is expressed differently depending on the angle at which it is located in the 3D coordinate space. The 3D character to be replaced should look at the front at the same angle as the angle at which the front of the character in the original content is viewed so that it is natural without a sense of incongruity. substitution is possible.

To this end, in the character information extraction step (S302), a standard feature for recognizing a direction and an angle, such as a character's face, is recognized, and a character's angle is determined based on the position of the recognized feature, and a new character based on this is determined. By replacing , it is possible to provide the same effect that the replaced character operates identically to the existing character.

In the character area removing step (S303), the character area is recognized and removed from the content image. As described above, when a character area is recognized in a content image, information on the character area can be checked, and pixels representing a character and pixels that do not constitute a screen can be distinguished from pixels constituting a screen.

Therefore, in the character area removing step (S303), the existing character may not appear in the content image by removing the color of the pixel portion representing the character in the image.

In the character area removing step (S303), after removing the character area from the image, the character area may be filled with an image that is as similar as possible to the background by referring to an area around the character area. Through this, it is possible to create an image in which only the background exists, as if the character did not exist in the corresponding content. This process can apply various artificial intelligence techniques, and the present invention is not limited by a specific method of filling the area.

In the character image generation step (S304), modeling data of the 3D character is controlled and rendered based on the character information to generate a 2D image of the removed character area. As described above, when various information about the character included in the original 3D content is obtained, a character to be replaced is prepared based on this information and synthesized, so that the character in the content is changed to the user's unique character. effect can be provided.

To this end, the character image generation step (S304) can be configured to perform the same motion at the same angle and the same position as the existing character of the actual 3D content by controlling the stored modeling data for the user's unique character. By rendering and generating a 2D image, it is possible for the user to use other 3D content as his own character.

In the character image generation step (S304), the rendered image of the 3D character is matched to pixels representing the character, and other pixels are treated as transparent pixels. Since information on pixels representing characters is removed from the existing content video, this part must be filled with a rendered image of a new 3D character. is filled with the rendered image of the new 3D character, leaving the rest as transparent pixels. When this is combined with the content image from which the character region has been removed, the remaining parts are reproduced as they are through the transparent part, and a final image in which only the character region is converted is created.

In the character image generation step (S304), modeling data of the 3D character is controlled to match the motion of the character in the content image, and a 2D image is generated by rendering the controlled modeling data. As described above, when the motion of the character in the original content video is recognized, the corresponding motion can be applied to the modeling data of the user's 3D character. Rendered images of dimensional characters can be synthesized. Through this, it is possible to obtain the same effect as if the actual user's unique character is active within the content image.

In the character image generation step (S304), modeling data of the 3D character is controlled to match the angle of the character in the content image, and a 2D image is generated by rendering the controlled modeling data. In order to replace a character, there must be a 3D character to be replaced at the same angle as the existing character in the content video. For this purpose, the modeling data of the 3D character is controlled based on the angle information of the recognized existing character By rendering, it is possible to replace characters without discomfort.

In the video output step (S305), the generated 2D video is synthesized with the content video and output. As described above, since the image generated in the character image generation step (S304) fills the character area with a new 3D character rendering image and leaves the remaining area as transparent pixels, it is overlapped with the original content image without complicated calculations. Final video can be created.

The 3D character substitution method according to the present invention can be produced as a program for a computer to execute and recorded on a computer-readable recording medium.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CDROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those produced by a compiler. The hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa.

Although the above has been described with reference to the embodiments, those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention described in the claims below.

101: 3D character replacement system
110: modeling data storage unit 120: character information extraction unit
130: character area removal unit 140: character image generation unit
150: video output unit

Claims (9)

a modeling data storage unit that stores modeling data of a 3D character;
a character information extractor for receiving a content image in which 3D content is rendered in 2D and recognizing character information including location information of a character in the received content image;
a character area removing unit recognizing and removing a character area within the content image;
a character image generation unit that controls and renders modeling data of the 3D character based on the character information to generate a 2D image of the removed character area; and
An image output unit for synthesizing the generated 2D image with the content image and outputting the synthesized image.
A three-dimensional character replacement system comprising a. According to claim 1,
The character information extraction unit
Identifying a pixel representing a character among pixels of the content image;
The character image generator
Matching the rendered image of the 3D character to a pixel representing the character, and processing other pixels as transparent pixels
Characterized by a three-dimensional character replacement system. According to claim 2,
The character information extraction unit
recognizing character information including motions of characters in the received content image;
The character image generator
Controlling the modeling data of the 3D character to match the motion of the character in the content image, and rendering the controlled modeling data to generate a 2D image.
Characterized by a three-dimensional character replacement system. According to claim 3,
The character information extraction unit
recognizing character information including an angle of a character in the received content image;
The character image generator
Controlling the modeling data of the 3D character to match the angle of the character in the content image, and rendering the controlled modeling data to generate a 2D image.
Characterized by a three-dimensional character replacement system. In the 3D character substitution method operating in a 3D character substitution system having a central processing unit and memory,
A modeling data storage step of storing modeling data of a 3D character;
A character information extraction step of receiving a content image in which 3D content is rendered in 2D and recognizing character information including location information of a character in the received content image;
a character area removal step of recognizing and removing a character area within the content image;
a character image generation step of controlling and rendering modeling data of the 3D character based on the character information to generate a 2D image of the removed character area; and
An image output step of synthesizing the generated 2D image with the content image and outputting the synthesized image.
3-dimensional character replacement method comprising a. According to claim 5,
The character information extraction step is
identifying a pixel representing a character among pixels of the content image;
The character image creation step is
Matching the rendered image of the 3D character to a pixel representing the character, and processing other pixels as transparent pixels
Characterized by a three-dimensional character substitution method. According to claim 6,
The character information extraction step is
recognizing character information including motions of characters in the received content image;
The character image creation step is
Controlling the modeling data of the 3D character to match the motion of the character in the content image, and rendering the controlled modeling data to generate a 2D image.
Characterized by a three-dimensional character substitution method. According to claim 7,
The character information extraction step is
recognizing character information including an angle of a character in the received content image;
The character image creation step is
Controlling the modeling data of the 3D character to match the angle of the character in the content image, and rendering the controlled modeling data to generate a 2D image.
Characterized by a three-dimensional character substitution method. A computer-readable recording medium on which a program for causing a computer to function to execute the method of any one of claims 5 to 8 is recorded.

Images