KR102621814B1 - Video generating device and method therfor - Google Patents

Abstract

An image generating device is provided. The image generating device includes a background generating module that generates a first image with the user removed using background data generated by photographing the appearance of an object interacting with the user, and user motion data related to the user's movement. A character implementation module that generates a second image containing the movement of the metaverse character using a character implementation module, object motion data related to the movement of the object, and the background data, using a first marker attached to the surface of the object and It includes a marker removal module for generating a third image including a related first marker removal object, and an image synthesis module for generating a final image based on the first image and using the second image and the third image. do.

Description

Video generating device and method therfor}

The present invention relates to an image generating device and method. Specifically, the present invention is a device that tracks the user's motion and the motion of the objects with which the user interacts, converts the user into a metaverse character, and generates an image in which the converted character and the object interact without a sense of heterogeneity, and the same. It's about method.

Metaverse character video production is accomplished through a very complex process. In the case of conventional metaverse character production, 2D or 3D computer graphics images require a lot of time and economic costs because the creator makes each movement manually, and also have the disadvantage of lacking realism between movements.

To overcome these shortcomings, recent metaverse character image creation devices are using various methods to preserve the reality of characters, and recent attempts have been made to maximize character reality by implementing the movements of metaverse characters using the movements of actual actors. There is.

Public Patent Publication No. 10-2020-0142391

The object of the present invention is to provide an image generation device and method that maximizes the reality of images in which metaverse characters appear.

Additionally, the object of the present invention is to provide an image generating device and method that maximizes the reality of an image including the process of interaction between metaverse characters and real objects.

In addition, the object of the present invention is to provide an image generating device and method that generates a natural image without any sense of heterogeneity in the overlap between the converted character and the real object in the process of converting the motion of the user performing the act into a character. will be.

The objects of the present invention are not limited to the objects mentioned above, and other objects and advantages of the present invention that are not mentioned can be understood by the following description and will be more clearly understood by the examples of the present invention. Additionally, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof indicated in the patent claims.

An image generating device according to some embodiments of the present invention to solve the above problem generates a first image with the user removed by using background data generated by photographing the appearance of a user and an object interacting with the user. a background generation module, a character implementation module that generates a second image containing the movement of a metaverse character using user motion data related to the user's movement, object motion data related to the movement of the object, and the background data A marker removal module that generates a third image including a first marker removal object related to the first marker attached to the surface of the object, based on the first image, the second image and the third image. It includes an image synthesis module that generates a final image using the image.

According to some embodiments, the marker removal module is a marker removal object creation module that generates a first marker removal object based on surface information of the object included in the background data, and uses the object motion data to create the first marker removal object. and a marker removal object placement module that places the first marker removal object at a position corresponding to the position of the marker, and the first marker may be at least one of a reflective marker, an AR marker, and a mixed marker.

According to some embodiments, the marker removal object creation module extracts appearance information of the object from background data received from an image capture device, and based on the object appearance information, generates an object corresponding to the location where the first marker is attached. A first marker removal object related to the location is created, and the first marker removal object may be at least one of a patch image or a modeling image.

According to some embodiments, the marker removal object placement module receives the object motion data including information about the moving direction and moving speed of the object from a motion capture device, and uses the object motion data to select the first marker. Removal objects can be placed in three-dimensional space.

According to some embodiments, the background generation module includes a user tracking module that generates user tracking data by tracking the user based on the background data, and a user removal module that removes the user from the background data using the user tracking data. module, and may include a background correction module that corrects the background of the location from which the user was removed.

According to some embodiments, the character implementation module includes a skeleton generation module for generating skeleton data using the user motion data, a retargeting module for generating rigging data using the skeleton data, and a rendering operation based on the rigging data. It can include a rendering module that implements the character by performing .

An image generation method according to some embodiments of the present invention for solving the above problem is an image generation method of an image generation device including a background generation module, a character implementation module, a marker removal module, and an image synthesis module, wherein the background generation module A first image with the user removed is generated using background data generated by photographing a user and an object interacting with the user, and the character implementation module is implemented using user motion data received from a motion capture device. A second image containing movement of a character is generated, and the marker removal module uses the background data and object motion data to create a second image containing a first marker removal object associated with a first marker attached to the surface of the object. It includes generating three images, and the image synthesis module generating a final image using the first to third images.

According to some embodiments, the marker removal module includes a marker removal object creation module and a marker removal object placement module, and the marker removal object creation module extracts appearance information of the object from the background data, and extracts appearance information of the object from the background data. Based on , a first marker removal object related to a location corresponding to the location where the first marker is attached may be created.

According to some embodiments, the marker removal object placement module may arrange the marker removal object to move in response to the movement of the object that changes over time.

According to some embodiments, the character implementation module implements a character corresponding to the user's body and implements movement of the character corresponding to the user's movement, wherein the character is the user's body obscured by the object. It can be implemented excluding areas.

The image generating device of the present invention reflects the motion of the user performing the act and applies it to the movement and production effects of the metaverse character, thereby maximizing the reality of the character.

In addition, the image generating device of the present invention can express the actual object that the user is in contact with without any sense of heterogeneity with the character even after the user switches to the character.

Additionally, the image generating device of the present invention can produce images with maximized quality by preventing markers attached to real objects from being visible in the image through image augmentation.

In addition to the above-described content, specific effects of the present invention are described below while explaining specific details for carrying out the invention.

1 is a diagram schematically illustrating a process for generating a final image using an image generating device according to some embodiments of the present invention.
2 and 3 are diagrams for explaining a process of generating a first image through a background generation module according to some embodiments of the present invention.
FIG. 4 is a diagram illustrating a process in which a motion capture device transmits user motion data and object motion data to each module of an image generating device according to some embodiments of the present invention.
Figures 5 and 6 are diagrams for explaining the process of generating a second image through a character implementation module according to some embodiments of the present invention.
Figures 7 and 8 are diagrams for explaining the process of generating a third image through a marker removal module according to some embodiments of the present invention.
FIG. 9 is an exemplary diagram illustrating types of markers used in an image generating device according to some embodiments of the present invention.
FIG. 10 is an exemplary diagram for explaining a first image generated through the process of FIG. 2 according to some embodiments of the present invention.
FIG. 11 is an exemplary diagram for explaining a second image generated through the process of FIG. 4 according to some embodiments of the present invention.
FIG. 12 is an exemplary diagram illustrating a third image generated through the process of FIG. 5 according to some embodiments of the present invention.
Figure 13 is a diagram for explaining the process of generating a final image through an image synthesis module according to some embodiments of the present invention.
Figure 14 is a diagram for explaining the procedure for generating a final image according to some embodiments of the present invention.

Terms or words used in this specification and patent claims should not be construed as limited to their general or dictionary meaning. According to the principle that the inventor can define terms or word concepts in order to explain his or her invention in the best way, it should be interpreted with a meaning and concept consistent with the technical idea of the present invention. In addition, the embodiments described in this specification and the configurations shown in the drawings are only one embodiment of the present invention and do not completely represent the technical idea of the present invention, so they cannot be replaced at the time of filing the present application. It should be understood that there may be various equivalents, variations, and applicable examples.

Terms such as first, second, A, and B used in the present specification and claims may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. The term 'and/or' includes any of a plurality of related stated items or a combination of a plurality of related stated items.

The terms used in the specification and claims are merely used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "include" or "have" should be understood as not precluding the existence or addition possibility of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. .

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains.

Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless clearly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

Additionally, each configuration, process, process, or method included in each embodiment of the present invention may be shared within the scope of not being technically contradictory to each other.

The present invention relates to a metaverse character image generation device, and 'metaverse character image generation' refers to the production of images including the movement and production effects of metaverse characters used in metaverse, virtual reality, augmented reality, or extended reality. It can mean. In other words, in this specification, 'metaverse character' may include a virtual digital idol character, game character, etc. implemented in 2D or 3D. In addition, 'metaverse video' refers to a video that includes metaverse characters, and may mean a video that includes not only metaverse characters, but also metaverse backgrounds and structures.

Hereinafter, an image generating device and method according to some embodiments of the present invention will be described with reference to FIGS. 1 to 14.

1 is a diagram schematically illustrating a process for generating a final image using an image generating device according to some embodiments of the present invention.

Referring to FIG. 1, the image generation device 10 according to some embodiments of the present invention includes a background creation module 100, a character implementation module 200, a marker removal module 300, and an image synthesis module 400. can do.

The background generation module 100 may receive background data (BG_D) from the image capturing device 20. The image capture device 20 may capture an image and generate background data (BG_D). For example, an imaging device may record the entire background, including the user performing the performance with a marker attached (hereinafter referred to as user), the object (OB) interacting with the user (hereinafter referred to as object), and the location where the user is being filmed. You can create background data (BG_D) by shooting. At this time, the background data BG_D may include an image in which the user and the object OB are in direct or indirect contact, that is, the user and the object OB interact with each other. In other words, the background data (BG_D) may include images in which the user and the object (OB) are captured. Background data (BG_D) generated by the image capturing device 20 may be provided to the background generation module 100 and/or the marker removal module 300. The background generation module 100 may generate the first image (V_1) using the background data (BG_D). The first image V_1 may be an image in which the user has been removed from the background data (BG_D) and the area from which the user has been removed has been corrected. A detailed description of the first image (V_1) will be described later. The background generation module 100 may provide the generated first image V_1 to the image synthesis module 400.

The character implementation module 200 may receive user motion data (MD_U) from the motion capture device 30. The motion capture device 30 may generate user motion data (MD_U) by sensing the user's movement. User motion data (MD_U) may include data related to the user's movement direction and movement speed. For example, the motion capture device 30 may generate user motion data (MD_U) by sensing a marker attached to the user. User motion data (MD_U) may include at least one of body motion data (BM_D), facial motion data (FM_D), and hand motion data (HM_D). The character implementation module 200 may generate a metaverse character image including the movement of the metaverse character using user motion data (MD_U). In this specification, the image photographing device 20 and the motion capture device 30 are expressed as different components, but the embodiments are not limited thereto. When the motion capture device 30 uses an optical marker (MK), the image capture device 20 and the motion capture device 30 may be the same component. However, in this specification, for convenience of explanation, the image photographing device 20 and the motion capture device 30 are expressed as distinct from each other.

Additionally, the character implementation module 200 may receive object motion data (MD_O) from the motion capture device 30. The motion capture device 30 may generate object motion data (MD_O) by sensing the movement of the object (OB). For example, the motion capture device 30 may generate object motion data MD_O by sensing a marker MK attached to an object OB. Object motion data MD_O may include data related to the movement direction and movement speed of the object OB. The character implementation module 200 may use the object motion data (MD_O) to generate a second image (V_2) by removing the area of the object (OB) from the above-described metaverse character image. In other words, the second image V_2 may be a metaverse character image from which the object OB area has been removed. A detailed description of the second image (V_2) will be described later. The character implementation module 200 may transmit the second image V_2 generated using user motion data (MD_U) and object motion data (MD_O) to the image synthesis module 400.

The marker removal module 300 may receive object motion data (MD_O) from the motion capture device 30. Additionally, the marker removal module 300 may receive background data (BG_D) from the image capturing device 20. As described above, the background data BG_D may include images of a user and an object OB. That is, the background data (BG_D) may include appearance information about the object (OB). The marker removal module 300 creates a marker removal object using the background data (BG_D) received from the imaging device 20, and uses the object motion data (MD_O) received from the motion capture device 30, The third image (V_3) can be generated by mapping the movement of the object (OB) and the movement of the marker-removed object. In other words, the third image V_3 may be an image in which the marker removal object is placed at a location corresponding to the marker MK of the object OB. A detailed description of the third video (V_3) will be described later. The marker removal module 300 may transmit the generated third image V_3 to the image synthesis module 400.

The image synthesis module uses the first image (V_1) received from the background creation module 100, the second image (V_2) received from the character implementation module 200, and the third image (V_3) received from the marker removal module. Thus, the final image (V_F) can be generated.

According to some embodiments, the image capturing device 20, the motion capture device 30, and the image generating device 10 may exchange data with each other through a network. Data can be transmitted through the network. Networks may include networks based on wired Internet technology, wireless Internet technology, and short-distance communication technology. Wired Internet technology may include, for example, at least one of a local area network (LAN) and a wide area network (WAN).

Wireless Internet technologies include, for example, Wireless LAN (WLAN), DLNA (Digital Living Network Alliance), Wibro (Wireless Broadband), Wimax (World Interoperability for Microwave Access: Wimax), and HSDPA (High Speed Downlink Packet). Access), HSUPA (High Speed Uplink Packet Access), IEEE 802.16, Long Term Evolution (LTE), LTE-A (Long Term Evolution-Advanced), Wireless Mobile Broadband Service (WMBS) and 5G NR (New Radio) technology. However, this embodiment is not limited to this.

Short-range communication technologies include, for example, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, and Near Field Communication. At least one of NFC), Ultrasound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio) may include. However, this embodiment is not limited to this.

The image capture device 20, motion capture device 30, and image generation device 10 that communicate through a network may comply with technical standards and standard communication methods for mobile communication. For example, standard communication methods include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), and EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only). , at least one of Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTEA), and 5G New Radio (NR) may include. However, this embodiment is not limited to this.

2 and 3 are diagrams for explaining a process of generating a first image through a background generation module according to some embodiments of the present invention.

Referring to FIGS. 2 and 3 , the background generation module 100 may include a user tracking module 100_1, a user removal module 100_2, and a background correction module 100_3. The user tracking module 100_1 may receive background data (BG_D) from the image capture device 20. The user tracking module 100_1 may track the user in the background data (BG_D) and generate user tracking data (UT_D) (S100). User tracking data (UT_D) may be data that specifies the user's outline. In other words, the user tracking data (UT_D) may be data that tracks the outline area of a moving user. According to some embodiments, the user tracking module 100_1 may generate user tracking data UT_D using a deep learning-based object detection algorithm. For example, the deep learning-based object detection algorithm may use at least one of YOLO (You Only Look Once), CNN, and SSD (Single Shot Multibox Detector), but the present embodiment is not limited thereto. The user tracking module 100_1 may transmit user tracking data (UT_D) to the user removal module 100_2.

The user removal module 100_2 may receive user tracking data UT_D from the user tracking module 100_1. The user removal module 100_2 may generate a user removal image (UR_V) using the user tracking data (UT_D) (S110). The user removal image (UR_V) may be an image in which the user has been removed from the background data (BG_D). In other words, the user removal module 100_2 can generate the user removal image UR_V by removing the entire user shape, including the user's outline, from the background data BG_D, based on the user tracking data UT_D. The user removal module 100_2 may transmit the user removal image UR_V to the background correction module 100_3.

The background correction module 100_3 may receive the user removal image UR_V from the user removal module 100_2. The background correction module 100_3 may generate the first image V_1 by correcting the background in the user-removed image UR_V (S120). At this time, the user removal image (UR_V) may be an image in which the user's area (i.e., the area inside the user's outline) has been removed from the background data (BG_D), and the area from which the user's outline has been removed is empty without data or in a single color. It may be filled with . The background correction module 100_3 may generate the first image V_1 by correcting the portion from which the user area has been removed in the user removal image UR_V to correspond to the surrounding background area. In other words, the first image V_1 may be an image in which the area from which the user was removed and the surrounding area blend together, as if the user had not existed from the beginning.

According to some embodiments, the background correction module 100_3 may generate the user-removed image UR_V as the first image V_1 using image synthesis or deep learning technology. For example, the background correction module 100_3 may generate the first image (V_1) by correcting the user removal image (UR_V) using Generative Adversarial Networks (GANs) or Convolutional Neural Networks (CNNs). At this time, deep learning technology can be learned using surrounding background information of the removed user's area.

As another example, the background correction module 100_3 may generate image data in an environment where the user does not exist before the background data (BG_D) is generated, and use this image data to generate the first image (V_1). there is. That is, the background correction module 100_3 may generate the first image V_1 by combining the removed user area with pre-photographed image data in the user removal image UR_V. However, the fact that the background correction module 100_3 uses deep learning technology or image synthesis technology is merely illustrative, and the present invention is not limited thereto. The background generation module 100 may transmit the first image V_1 generated using the background data BG_D to the image synthesis module 400.

FIG. 4 is a diagram illustrating a process in which a motion capture device transmits user motion data and object motion data to each module of an image generating device.

According to some embodiments, the motion capture device 30 may include a body capture module 30_1, a facial capture module 30_2, a hand capture module 30_3, and an object capture module 30_4. The motion capture device 30 may record the user's movements as data through sensors and/or markers attached to the user's body. In other words, the motion capture device 30 can sense the user's movement and generate user motion data (MD_U). User motion data (MD_U) may be data expressing the user's skeletal movements and muscle movements. Additionally, the motion capture device 30 may record the movement of the object OB as data through sensors and/or markers attached to the user and the object OB. In other words, the motion capture device 30 may sense the movement of the object OB and generate object motion data MD_O including the movement direction and speed of the object OB.

For example, motion capture device 30 may be implemented as an optical, magnetic, and/or inertial motion capture device. An optical motion capture device may include a marker and a camera. An optical motion capture device may be a device that captures the movements of a user with a marker attached using one or more cameras and generates motion data by inverting the three-dimensional coordinates of the marker attached to the user through triangulation. A magnetic motion capture device may be a device that generates motion data by attaching a sensor capable of measuring a magnetic field to a user's joints, etc., and then measuring movement by calculating the amount of change in the magnetic field of each sensor near the magnetic field generator. An inertial motion capture device may be a device that attaches an inertial sensor including an acceleration sensor, a gyro sensor, and a geomagnetic sensor to a user's joints, reads the user's movement, rotation, and/or direction, and generates motion data. However, the above-described optical, magnetic, and inertial motion capture devices are merely an example of how the motion capture device 30 according to some embodiments of the present invention can be implemented, and the embodiments are not limited thereto. For example, the motion capture device 30 may generate motion data using image processing technology using artificial intelligence or in parallel with the optical, magnetic, and/or inertial motion capture device 30. Hereinafter, for convenience of explanation, it is assumed that the motion capture device 30 is an optical motion capture device including a marker and a camera.

According to some embodiments, the body capture module 30_1 may generate body motion data BM_D. Body motion data (BM_D) may be data about the movement of the user's body, that is, the user's activity motion. In other words, the body capture module 30_1 can generate body motion data BM_D by sensing movements of the user's body, such as the head, neck, arms, body, and legs. For example, the body capture module 30_1 captures the head, neck, arms, etc. when the user performs a specific action (walking, running, crawling, sitting, standing, lying down, dancing, fighting, kicking, swinging arms, etc.) Body motion data (BM_D) can be generated by sensing the movement of the body, legs, etc. The body capture module 30_1 may attach a marker to the user's body and sense the marker to generate body motion data BM_D, but the present embodiment is not limited thereto.

According to some embodiments, the facial capture module 30_2 may generate facial motion data (FM_D). Facial motion data (FM_D) may be data that senses the movement of the user's face, that is, data about the user's facial expression. In other words, the facial capture module 30_2 can generate facial motion data (FM_D) by sensing the movement of the user's face. For example, the facial capture module 30_2 captures the movement of the user's face in a specific expression (a crying expression, a smiling expression, a surprised expression, an angry expression, a disdainful expression, a regretful expression, a loving expression, a disgusting expression, etc.). Facial motion data (FM_D) can be generated by sensing. The facial capture module 30_2 may attach a marker to the user's face and generate facial motion data (FM_D) by additionally using data sensed and image processing technology for the user's facial expression, but this embodiment does not do this. It is not limited.

According to some embodiments, the hand capture module 30_3 may generate hand motion data (HM_D) by sensing the user's hand movements. Hand motion data (HM_D) may be data that senses the user's hand movement, that is, data about the user's hand movement. In other words, the hand capture module 30_3 can generate hand motion data (HM_D) by sensing the user's hand movements. For example, the hand capture module 30_3 may generate hand motion data by sensing the movement of finger joints in the user's hand movements (closed fingers, extended fingers, etc.). The hand capture module 30_3 may be implemented as a special glove or wearable device capable of sensing movement of finger joints, but the present embodiment is not limited thereto.

That is, the motion capture device 30 uses at least one of body motion data (BM_D), facial motion data (FM_D), and hand motion data (HM_D) to perform user motion related to at least one of the user's gesture, facial expression, and hand movement. Data (MD_U) can be created.

The object capture module 30_4 may generate object motion data (MD_O). Object motion data MD_O may include the movement direction and movement speed of the marker MK attached to the object OB. Additionally, the object motion data MD_O may include information about the absolute position of the marker MK attached to the object OB and the relative positional relationship between the plurality of markers. The object capture module 30_4 may be implemented with a reflective marker (MK_1) or an AR marker (MK_2), but the present embodiment is not limited thereto. The motion capture device 30 may transmit object motion data (MD_O) to the marker removal module 300.

Figures 5 and 6 are diagrams for explaining the process of generating a second image through a character implementation module according to some embodiments of the present invention.

According to some embodiments, the character implementation module 200 may include a skeleton creation module 200_1, a retargeting module 200_2, and a rendering module 200_3. The skeleton creation module 200_1 may receive user motion data (MD_U) from the motion capture device 30 (S200). The skeleton generation module 200_1 may generate first skeleton data using user motion data (MD_U) obtained from markers (MK) attached to the user's joints, etc. (S210). At this time, the first skeleton data may include data regarding the user's skeleton size, skeleton location, and skeleton direction. Additionally, the skeleton creation module 200_1 may receive object motion data (MD_O) from the motion capture device 30. The skeleton generation module 200_1 may generate second skeleton data SK_D based on the first skeleton data and object motion data MD_O (S220). That is, the second skeletal data SK_D may be created by modifying the first skeletal data. Specifically, the skeleton creation module 200_1 may use the object motion data MD_O to extract the part where the user's skeleton is obscured by the object OB from the first skeleton data. Thereafter, the part extracted from the first skeleton data can be deleted to generate the second skeleton data (SK_D). The skeleton generation module 200_1 may transmit the second skeleton data SK_D generated using the user motion data MD_U and the object motion data MD_O to the retarget module 200_2. As a result, the second skeleton data SK_D transmitted from the skeleton generation module 200_1 to the retarget module 200_2 may be data in which data about the portion of the user's skeleton obscured by the object OB has been deleted. .

The retarget module 200_2 may receive second skeleton data SK_D from the skeleton generation module 200_1. The retarget module 200_2 may generate rigging data RG_D through a process of rigging the second skeleton data SK_D (S230). At this time, the rigging process may refer to the process of applying movement to the character by applying a structure called a joint when implementing a 3D character. Specifically, it may refer to the process of connecting the bones using the second skeleton data (SK_D) and assigning weights to the direction and angle of movement for each skeleton so that the character can move naturally like a human. The retarget module 200_2 may transmit rigging data RG_D generated using the second skeleton data SK_D to the rendering module 200_3.

The rendering module 200_3 may implement a character by performing a rendering operation on the character's appearance based on the rigging data (RG_D) (S240). Specifically, the rendering process includes steps to determine the character's appearance and texture such as the character's skin, clothes, and hair, steps to provide contrast by setting the lighting surrounding the character, and steps to calculate and reproduce various visual elements such as shadows. You can. The rendering module 200_3 may generate a second image (V_2) including the character's movement (S250). That is, the second image (V_2) relates to a metaverse character image generated based on user motion data (MD_U), and the part where the user's skeleton is obscured by an object (OB) in the metaverse character image has been removed. It may mean a video. The character implementation module 200 may transmit the second image V_2 to the image synthesis module 400.

In this specification, in order to not implement the part of the user's skeleton that is obscured by the object (OB) as a character, it is explained that the first skeleton data is modified in the skeleton creation module 200_1 to generate the second skeleton data (SK_D). However, this embodiment is not limited to this. A person skilled in the art of the present invention will be able to generate the second image (V_2) by removing the object area from the metaverse character image through various methods.

Figures 7 and 8 are diagrams for explaining the process of generating a third image through a marker removal module according to some embodiments of the present invention.

According to some embodiments, the marker removal module 300 may include a marker removal object creation module 300_1 and a marker removal object placement module 300_2. The marker removal object creation module 300_1 may receive background data BG_D from the image capturing device 20 and analyze the appearance of the object OB included in the background data. At this time, the appearance of the object OB may include information such as the shape, shape, color, material, brightness, pattern, and size of the object OB. The marker removal object creation module 300_1 may generate a marker removal object MR_0 that can cover the marker MK attached to the surface of the object OB based on information about the appearance of the object OB ( S300). At this time, the marker removal object MR_0 may be created using surface information of the object OB at a location corresponding to the location where the marker MK is attached, among information about the appearance of the object OB. At this time, the marker removal object creation module 300_1 can use artificial intelligence to learn the surface information of the object (OB) at the location corresponding to the location where the marker (MK) is attached, and create a marker removal object based on the learned information. (MR_0) can be created. Additionally, the marker removal object creation module 300_1 may generate the marker removal object MR_0 using information about the appearance of the object OB before attaching the marker MK. That is, surface information of an object (OB) to which a marker (MK) is not attached can be stored, and a marker removal object (MR_O) can be created using the stored information. Accordingly, the marker removal object MR_0 may be an object having a shape, shape, and color corresponding to the surface of the object OB at the location where the marker MK is attached. However, this embodiment is not limited to this. The marker removal object creation module 300_1 may transmit the marker removal object MR_0 to the marker removal object placement module 300_2.

The marker removal object placement module 300_2 may receive the marker removal object MR_0 from the marker removal object creation module 300_1. The marker removal object placement module 300_2 may receive object motion data (MD_O) from the motion capture device 30. The object motion data MD_O may include the moving direction and moving speed of the object OB. Based on the object motion data (MD_O), the marker removal object placement module 300_2 arranges the marker removal object (MR_0) to correspond to the movement of the marker (MK) attached to the object (OB) to create the third image (V_3). can be created (S310, S320). In other words, the third image V_3 may be an image in which the marker removal object MR_O is arranged to correspond to the movement of the marker MK attached to the object OB. That is, the marker removal object placement module 300_2 may change the arrangement of the marker removal object MR_0 in response to the movement of the marker MK attached to the object OB moving with time. The marker removal object placement module 300_2 may transmit the generated third image V_3 to the image synthesis module 400.

FIG. 9 is an exemplary diagram illustrating types of markers used in an image generating device according to some embodiments of the present invention.

The marker (MK) attached to the user or object (OB) may include a reflective marker (MK_1), an AR marker (MK_2), and a mixed marker (MK_3). At this time, the mixed marker (MK_3) may refer to a marker that combines a reflective marker (MK_1) and an AR marker (MK_2). However, this embodiment is not limited to this.

The reflective marker MK_1 according to some embodiments may be a marker that reflects light in a specific way. The reflective marker (MK_1) can be detected through optical tracking, which is a method of recognizing and tracking specific light. The position and movement of the object (OB) to which the reflective marker (MK_1) is attached can be tracked through optical tracking. Specifically, the optical sensing device detects the light reflected from the reflective marker (MK_1), calculates the position of the reflective marker (MK_1), tracks the movement of the reflective marker (MK_1) based on this position information, and detects the object (OB). ) can measure the relative position and direction of the reflective marker (MK_1) attached.

The AR marker (MK_2) according to some embodiments may be a marker with a unique pattern. The AR marker (MK_2) may be composed of a square or circular one-dimensional flat pattern composed of a black and white pattern. However, this embodiment is not limited to this, and the AR marker (MK_2) may include a color pattern and may be configured in a three-dimensional shape. Each AR marker (MK_2) has a unique pattern, and the pattern is recognized as a unique identifier. The identifier sensing device can detect the identifier and track it. The identifier sensing device may include a camera or optical scanner. However, this embodiment is not limited to this.

The mixed marker (MK_3) according to some embodiments may be a marker in which a reflective marker is combined on the AR marker (MK_2). The mixing marker (MK_3) may be detected by an optical sensing device and/or an identifier sensing device. Additionally, the mixed marker may be detected by a unique sensing device. The mixed marker (MK_3) can be detected by performing at least one of optical tracking or identifier tracking in a situation where it is difficult to recognize the reflective marker (MK_1) or the AR marker (MK_2) alone. For example, in situations where light reflection is difficult due to cloudy weather or a dark location, it may be difficult to detect the reflective marker (MK_1). Additionally, in situations where the object to which the AR marker (MK_2) is attached moves quickly, identifier sensing may not be connected smoothly. At this time, the mixed marker (MK_3) recognizes at least one of the two markers, and when one of the markers is difficult to recognize, it can recognize and track the other marker. In this case, by selectively recognizing two markers alternately depending on the situation, the movement of the object (OB) can be tracked more accurately.

FIG. 10 is an exemplary diagram for explaining a first image generated through the process of FIG. 2 according to some embodiments of the present invention.

The image capturing device 20 according to some embodiments may capture images including movements of a user and an object (OB) interacting with the user. The image capturing device 20 may generate background data (BG_D) based on the captured image. The background generation module 100 may track the user based on the background data (BG_D), remove the user, and then generate the first image (V_1) with the background corrected.

For example, in a studio, which is the background for filming, a user can sit on a chair with a marker (MK) attached and holding an object (OB) to which the marker (MK) is attached. In addition to users, there may also be general performers without markers (MK) in the studio, but embodiments are not limited thereto. The video recording device 20 can capture a user and a general actor performing an act. The image capturing device 20 may transmit the captured background data (BG_D) to the background generation module 100. The background generation module 100 may generate user tracking data (UT_D) by tracking the outline of the user, and remove the user from the background data using the user tracking data (UT_D). At this time, while removing the entire user shape, including the user's outline, from the background data (BG_D), the background portion behind the user shape may also be removed. The background correction module 100_3 may generate the first image V_1 by correcting the removed background portion to correspond to the surrounding background area adjacent to the corresponding portion. Accordingly, only the user is removed from the first image V_1, and the object OB held by the user and the general performer may remain the same. That is, the background generation module 100 may generate the first image V_1 as if the user had not existed from the beginning.

FIG. 11 is an exemplary diagram for explaining a second image generated through the process of FIG. 5 according to some embodiments of the present invention.

The motion capture device 30 according to some embodiments may generate user motion data (MD_U) containing the user's movement information using markers (MK) attached to the user. Additionally, the motion capture device 30 may generate object motion data MD_O including movement information of the object OB using markers MK attached to the object OB. The character implementation module 200 can implement a metaverse character that moves to correspond to the user's movements, based on user motion data (MD_U). At this time, using the object motion data (MD_O), the part obscured by the user's object (OB) may not be implemented as a character. The character implementation module 200 may generate a second image (V_2) including movement of the character. The second image (V_2) may not include the background or object (OB).

For example, the motion capture device 30 may detect a marker (MK) attached to the user's clothes and generate user motion data (MD_U) based on this. Additionally, the motion capture device 30 may detect a marker (MK) attached to an object (OB) and generate object motion data (MD_O) based on this. User motion data (MD_U) and object motion data (MD_O) generated by the motion capture device 30 may be provided to the character implementation module 200.

The character implementation module 200 may generate the second image V_2 using the user motion data (MD_U) and object motion data (MD_O) received from the motion capture device 30. As described above, the second image V_2 may be a character image in which the portion corresponding to the user's external appearance obscured by the object OB has been removed. For example, as shown in FIG. 11 , the second image V_2 may include a metaverse character image in which a portion of the user's left arm that was obscured by the object OB has been removed.

FIG. 12 is an exemplary diagram for explaining a third image generated through the process of FIG. 7 according to some embodiments of the present invention.

The marker removal module 300 according to some embodiments may receive background data (BG_D) and object motion data (MD_O). The marker removal module 300 may generate a third image (V_3) including the marker removal object (MR_0) using the background data (BG_D) and the object motion data (MD_O).

For example, the marker removal module 300 includes background data (BG_D) containing information about the appearance of the object (OB) held by the user and information about the moving direction and moving speed of the object (OB). Motion data (MD_O) can be received. The marker removal object creation module 300_1 uses the surface information of the object (OB) corresponding to the location where the marker (MK) is attached, based on the background data (BG_D), to create a marker (MK) attached to the object (OB). A marker removal object (MR_0) can be created to create an image with the removed image. That is, the marker removal object MR_O can be used to cover the marker MK attached to the object OB. In other words, when the marker removal object MR_O is placed on the corresponding marker MK, the marker MK attached to the object OB may be obscured.

Based on the object motion data (MD_O), the marker removal object placement module 300_2 arranges each marker removal object (MR_0) to correspond to the position and direction of the corresponding marker (MK) to create the third image (V_3). can be created. Since the third image (V_3) is a video whose image changes over time, the marker removal object (MR_O) included in the third image (V_3) is the movement (direction and speed of movement) corresponding to the object motion data (MD_O). etc.). In other words, the third image V_3 may be an image in which the arrangement of the marker removal object MR_0 changes over time.

13 and 14 are diagrams to explain the process of generating a final image through an image synthesis module according to some embodiments of the present invention.

According to some embodiments, the image synthesis module 400 includes the first image (V_1) generated in the background generation module 100, the second image (V_2) generated in the character implementation module 200, and the marker removal module 300. ), the final image (V_F) can be generated using at least a portion of the third image (V_3) generated in .

For example, the image synthesis module 400 may receive the first image (V_1), the second image (V_2), and the third image (V_3) (S400). The image synthesis module 400 may generate the final image (V_F) by combining the second image (V_2) and the third image (V_3) based on the first image (V_1), which is the background with the user removed ( S410, S420). For example, the image synthesis module 400 may set the first image (V_1) as the bottom layer and sequentially arrange the second image (V_2) and the third image (V_3) on the first image (V_1). there is. For another example, the image synthesis module 400 uses the first image (V_1) as the bottom layer and sequentially arranges the third image (V_3) and the second image (V_2) on the first image (V_1). You can.

According to some embodiments, the image synthesis module 400 may generate an image including a metaverse character on behalf of the user by synthesizing the second image (V_2) based on the first image (V_1). As described above, the second image V_2 may be a metaverse character image with the portion obscured by the object OB removed. Therefore, when the second image (V_2) is synthesized based on the first image (V_1), an image in which the metaverse character and the real object (OB) naturally interact with each other can be created. Additionally, as described above, the third image V_3 may be an image of the marker removal object MR_O corresponding to the marker MK attached to the object OB. Therefore, when the third image (V_3) is synthesized based on the first image (V_1), the marker (MK) attached to the object (OB) is obscured by the marker removal object (MR_O), as if the marker (MK) Images of objects (OBs) such as those that are not attached can be generated. In other words, the final video (V_F) may be the same video where a metaverse character and a general actor interact, rather than a user from the beginning, and there is no marker (MK) on the surface of the object (OB) in contact with the character. It may be the same video as before.

Therefore, the image synthesis module 400 synthesizes the second image (V_2) and the third image (V_3) based on the first image (V_1) to generate the final image (V_F), thereby generating the metaverse characters and objects (OB). ) can interact naturally and create a final image (V_F) that covers the marker (MK) attached to the object (OB), thereby creating a more natural image with less heterogeneity.

The above description is merely an illustrative explanation of the technical idea of the present embodiment, and those skilled in the art will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, this embodiment is not intended to limit the technical idea of this embodiment, but to explain it, and the scope of the technical idea of this embodiment is not limited by this example. The scope of protection of this embodiment should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this embodiment.

Claims (10)

a background generation module that generates a first image with the user removed using background data generated by photographing the appearance of a user and an object interacting with the user;
a character implementation module that generates a second image including movement of the metaverse character using user motion data related to the user's movement;
A third image including object motion data related to the movement of the object and a first marker removal object that is related to a first marker attached to the surface of the object and moves based on the movement of the object using the background data. A marker removal module that generates; and
An image synthesis module that synthesizes the second image and the third image based on the first image to generate a final image,
The final image includes an image of the metaverse character and an image of the object in which the first marker is obscured by the first marker removal object.
Image generating device.
According to claim 1,
The marker removal module,
a marker removal object creation module that generates a first marker removal object based on surface information of the object included in the background data; and
A marker removal object placement module that places the first marker removal object at a position corresponding to the position of the first marker using the object motion data,
The first marker is at least one of a reflective marker, an AR marker, and a mixed marker,
Image generating device.
According to clause 2,
The marker removal object creation module,
Extracting appearance information of the object from background data received from the video capture device,
Based on the appearance information of the object, create a first marker removal object related to a location corresponding to the location where the first marker is attached,
The first marker removal object is at least one of a patch image or a modeling image,
Image generating device.
According to clause 3,
The marker removal object placement module,
Receiving the object motion data containing information about the moving direction and moving speed of the object from a motion capture device,
Placing the first marker removal object in a three-dimensional space using the object motion data,
Image generating device.
According to claim 1,
The background creation module is,
a user tracking module that tracks the user based on the background data and generates user tracking data; and
a user removal module that removes the user from the background data using the user tracking data; and
Comprising a background correction module that corrects the background of the location from which the user was removed,
Image generating device.
According to claim 1,
The character implementation module is,
a skeleton generation module that generates skeleton data using the user motion data;
A retargeting module that generates rigging data using the skeleton data; and
Comprising a rendering module that implements a character by performing a rendering operation based on the rigging data,
Image generating device.
An image generation method of an image generation device including a background generation module, a character implementation module, a marker removal module, and an image synthesis module,
The background generation module generates a first image from which the user is removed using background data generated by photographing a user and objects interacting with the user,
The character implementation module generates a second image containing the movement of the metaverse character implemented using user motion data received from a motion capture device,
The marker removal module uses the background data and object motion data to generate a third image including a first marker removal object that is related to a first marker attached to the surface of the object and moves based on the movement of the object. do,
wherein the image synthesis module synthesizes the first to third images to generate a final image,
The final image includes an image of the metaverse character and an image of the object in which the first marker is obscured by the first marker removal object.
How to create video.
According to clause 7,
The marker removal module includes a marker removal object creation module and a marker removal object placement module,
The marker removal object creation module,
Extracting appearance information of the object from the background data,
Based on the appearance information of the object, generating a first marker removal object related to a location corresponding to the location where the first marker is attached,
How to create video.
According to clause 8,
The marker removal object placement module,
Arranging the marker removal object to move in response to the movement of the object that changes with time,
How to create video.
According to clause 7,
The character implementation module is,
Implementing a character corresponding to the user's body,
Implement the movement of the character corresponding to the user's movement,
The character is implemented excluding areas of the user's body that are obscured by the object,
How to create video.

Images