KR20210081082A - Server, method and user device for providing avatar contents based on motion data of object - Google Patents

Abstract

The present invention provides a server for providing avatar content based on motion data of an object, which is capable of extracting the motion data of the object from each frame of an image. The server for providing avatar content based on motion data of an object includes a receiver configured to receive an image of an object from a user terminal and metadata necessary for generating the avatar content; an image analysis unit configured to analyze the image based on the received metadata; a motion data extraction unit configured to extract motion data of the object from each frame of the image based on the analyzed image; a motion data analysis unit configured to analyze whether at least a portion of the motion data is omitted for each frame based on the extracted motion data; a compensator configured to compensate for at least a portion of the missing motion data based on motion data of another frame or a replacement animation for a frame in which at least a portion of the motion data is missing; and a content generation unit configured to generate the avatar content by retargeting the extracted motion data to the avatar.

Description

Server, method, and user terminal providing avatar contents based on motion data of an object {SERVER, METHOD AND USER DEVICE FOR PROVIDING AVATAR CONTENTS BASED ON MOTION DATA OF OBJECT}

The present invention relates to a server, a method, and a user terminal for providing avatar content based on motion data of an object.

An avatar refers to a visual image that means an alter ego in a virtual space, and is in the spotlight as a virtual body replacing a user in Internet chatting, shopping malls, and online games.

Conventionally, a two-dimensional avatar generated by a user selecting an item according to his or her taste has been used, but recently, a three-dimensional avatar based on the user's face created based on the feature point extracted from the user's face has been used. have.

In relation to such an avatar generation, Korean Patent Laid-Open Publication No. 2019-0101834, a prior art, discloses an electronic device for displaying an avatar in which motion is performed according to the movement of a feature point of a face, and an operating method thereof.

Recently, an avatar content service in which an avatar operates based on the user's motion as well as the user's face has been provided. However, in the case of avatar content, there is a problem in that avatar content that does not sufficiently reflect the user's motion is generated by inaccurately recognizing the user's motion due to misrecognition of the user's motion or recognizing an object other than the user. .

A content providing server, method, and user terminal for analyzing an image based on an image of an object photographed from a user terminal and metadata required for avatar content, and extracting motion data of an object from each frame of the image based on the analyzed image would like to provide

Content that analyzes whether at least a part of the motion data is missing for each frame based on the motion data of the object, and supplements at least a part of the missing motion data based on the motion data of another frame or an alternative animation for the missing frame An object of the present invention is to provide a providing server, a method, and a user terminal.

An object of the present invention is to provide a content providing server, a method, and a user terminal that supplement missing motion data and retarget the extracted motion data to an avatar to generate avatar content.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention provides a receiver that receives an image of an object from a user terminal and metadata necessary for generating the avatar content, based on the received metadata. An image analysis unit for analyzing the image, a motion data extraction unit for extracting motion data of the object from each frame of the image based on the analyzed image, and the motion for each frame based on the extracted motion data A motion data analysis unit that analyzes whether at least a portion of the data is missing, a motion data for a frame in which at least a portion of the motion data is missing, based on the motion data of another frame or an alternative animation, the information for supplementing at least a portion of the missing motion data It is possible to provide a content providing server including a content generator for generating the avatar content by retargeting the government and the extracted motion data to the avatar.

Another embodiment of the present invention includes a photographing unit that acquires an image of an object, a transmission unit that transmits an image of the object and metadata necessary for generating the avatar content to a content providing server, and the content providing server A receiving unit for receiving the avatar content generated based on the image and the metadata, and a display unit for displaying the received avatar content, wherein the image is analyzed based on the metadata, and based on the analyzed image to extract the motion data of the object from each frame of the image, and analyze whether at least a part of the motion data is missing for each frame based on the extracted motion data, and at least a part of the motion data is missing Provided is a user terminal in which at least a portion of the missing motion data is supplemented based on the motion data of another frame or an alternative animation for a given frame, and the avatar content is generated by retargeting the extracted motion data to an avatar. can do.

Another embodiment of the present invention provides the steps of: receiving, from a user terminal, an image of an object and metadata necessary for generating the avatar content; analyzing the image based on the received metadata; extracting motion data of the object from each frame of the image based on an image, analyzing whether at least a portion of the motion data is missing for each frame based on the extracted motion data, the motion data Compensating for at least a part of the missing motion data based on motion data or replacement animation of another frame with respect to a frame in which at least a part of which is missing, and retargeting the extracted motion data to an avatar to generate the avatar content It is possible to provide a content providing method including the steps.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, the user's motion data is extracted by analyzing an image of the user's various motions (eg, dance, gesture, etc.) based on the metadata, and the extracted motion A content providing server, a method, and a user terminal for generating avatar content by retargeting an avatar based on data may be provided.

If motion data is missing due to misrecognition of the user's joint, the user's body is out of the shooting area, or recognition of an object, etc., depending on the degree of omission of the missing motion data, motion data of another frame or alternative animation It is possible to provide a content providing server, method, and user terminal to be supplemented using

A content providing server, method, and user terminal for generating avatar content by extracting motion data from an image captured using a 3D depth camera as well as an image captured using a 2D general camera and retargeting it to an avatar can provide

It is possible to provide a content providing server, a method, and a user terminal that provide an authoring tool to a user terminal so that the user can directly edit the customized avatar content.

When the user selects a sound source to be included in an image, the content providing server, method, and user terminal that analyze the genre, tempo, etc. of the sound source and recommend an avatar suitable for the sound source may be provided.

1 is a block diagram of a content providing system according to an embodiment of the present invention.
2 is a block diagram of a content providing server according to an embodiment of the present invention.
3 is an exemplary diagram illustrating an avatar according to an embodiment of the present invention.
4 is an exemplary diagram for explaining a process of extracting motion data and converting a format according to an embodiment of the present invention.
5A and 5B are exemplary views for explaining a process of generating new motion data for a correction section according to an embodiment of the present invention.
6A and 6B are exemplary views for explaining a process of applying an alternative animation to a correction section according to an embodiment of the present invention.
7 is a flowchart of a method for providing avatar content based on motion data of an object in a content providing server according to an embodiment of the present invention.
8 is a block diagram of a user terminal according to an embodiment of the present invention.
9 is an exemplary diagram illustrating edited avatar content according to an embodiment of the present invention.
10 is a flowchart of a method of providing avatar content based on motion data of an object in a user terminal according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in the present specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a content providing system according to an embodiment of the present invention. Referring to FIG. 1 , the content providing system 1 may include a content providing server 110 and a user terminal 120 . The content providing server 110 and the user terminal 120 exemplarily illustrate components that can be controlled by the content providing system 1 .

Each component of the content providing system 1 of FIG. 1 is generally connected through a network. For example, as shown in FIG. 1 , the content providing server 110 may be connected to the user terminal 120 at the same time or at a time interval.

A network refers to a connection structure that enables information exchange between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW: World). Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, etc. are included, but are not limited thereto.

The content providing server 110 may receive from the user terminal 120 an image obtained by photographing an object and metadata necessary for generating the avatar content. Here, the metadata may include information on the avatar to be retargeted and additional information necessary for generating the avatar content. The avatar may include either a human-type avatar or an animal-type avatar, and the additional information may include sound source information included in an image, camera information for capturing an object, background information, and the like.

When the content providing server 110 receives sound source information from the user terminal 120, it analyzes the genre of the sound source (eg, dance song, ballad song, R&B, hip-hop, etc.) based on the received sound source information, Any one of the plurality of avatars may be selected and provided based on the analyzed genre of the sound source.

The content providing server 110 may analyze the image based on the received metadata.

The content providing server 110 analyzes the type information of the image based on the camera information (eg, 2D camera, 3D camera, etc.), and extracts motion data of the object from each frame of the image based on the analyzed image. can In this case, the content providing server 110 may convert the format of the extracted motion data based on the analyzed image type information.

The content providing server 110 primarily analyzes whether motion data is missing for each frame of the image based on the format-converted motion data, and then converts the format-converted motion data to correspond to the size of the avatar to be retargeted. can be scaled.

The content providing server 110 may analyze whether at least a portion of the motion data is missing for each frame based on the extracted motion data. To this end, the content providing server 110 may analyze motion data for the entire frame of the image, and classify the analyzed motion data into any one of body regions.

The content providing server 110 selects a mapping target region to be mapped with any one classified body region based on the avatar information, and selects a mapping target region corresponding to the mapping target region based on coordinate information for each section corresponding to the selected mapping target region. An average value for the coordinates of the entire section can be derived.

If the difference between the coordinate values for each section corresponding to the mapping target area is equal to or greater than the first threshold based on the derived average value, the content providing server 110 determines that motion data is missing from at least one section corresponding to the mapping target area can do.

The content providing server 110 may supplement at least a portion of the missing motion data based on motion data of another frame or an alternative animation for a frame in which at least a portion of the motion data is missing.

The content providing server 110 calculates a correction section to which an alternative animation of the avatar is to be applied based on the motion data classified as a preset first region among the body regions, and the omission rate of the first region in the calculated correction section is the second It can be determined whether or not it is above a threshold value.

For example, when the omission rate of the first region is less than the second threshold, the content providing server 110 interpolates motion data corresponding to at least one of a frame before and after the correction period to obtain new motion data for the correction period. can create

As another example, when the omission rate of the first region is equal to or greater than the second threshold, the content providing server 110 may apply the replacement animation to the correction section. Specifically, the content providing server 110 calculates the length of the correction section to which the replacement animation is to be applied, and detects the motion size of the object based on motion data corresponding to at least one of the frames before and after the correction section, One final replacement animation may be selected from among a plurality of candidate replacement animations based on the length of the correction section and the motion size of the detected object.

The content providing server 110 may perform additional correction by applying camera work and stage effects to the correction section.

The content providing server 110 sets the starting point of the correction section as a key frame, and based on the set key frame, the genre of the sound source included in the image and the motion size of the detected object, short or long, the camera work is applied. can do.

The content providing server 110 may generate avatar content by retargeting the extracted motion data to the avatar.

The user terminal 120 may execute an avatar content service providing app for generating avatar content. Here, the avatar content service providing app may provide an avatar content creation mode, editing mode, and playback mode.

For example, the user terminal 120 may generate the avatar content by retargeting the avatar based on an action taken by the user terminal 120 through the creation mode of the avatar content providing service app. The user terminal 120 may also upload the avatar content previously generated through the creation mode of the avatar content providing service app to the content providing server 110 through the avatar content providing service app. In this case, when uploading the pre-generated avatar content, the user terminal 120 may additionally receive additional information (eg, sound source name, etc.) for analyzing the pre-generated avatar content.

For another example, the user terminal 120 may directly modify the avatar content generated by the content providing server 110 using an authoring tool through the editing mode of the avatar content providing service app.

As another example, the user terminal 120 may enjoy the avatar content generated by retargeting the avatar based on its own motion through the playback mode of the avatar content service app. Alternatively, the user terminal 120 may enjoy the avatar content generated by another user terminal through the play mode of the avatar content service providing app.

When the user terminal 120 receives sound source information to be included in an image, the user terminal 120 may transmit the input sound source information to the content providing server 110 . The user terminal 120 may receive a recommended avatar selected based on sound source information from the content providing server 110 .

The user terminal 120 may acquire an image obtained by photographing the object. Here, the user terminal 120 may photograph an object taking various actions such as a dance or a gesture.

The user terminal 120 may transmit an image of the object and metadata necessary for generating the avatar content to the content providing server 110 .

The user terminal 120 may receive the avatar content generated based on the image and metadata from the content providing server 110 .

The user terminal 120 may display the received avatar content. The user terminal 120 may consume the avatar content in various ways depending on the device. For example, when the user terminal 120 is a head mounted display (HMD), the user terminal 120 may display 2D avatar content as 360° avatar content. In this case, when the user terminal 120 consumes the avatar content in the form of 360˚, the user can enjoy the avatar content in real time at a desired location without using additional corrected information such as camera work. . For another example, when the user terminal 120 is a mobile or general TV, the user terminal 120 may display and enjoy 2D avatar content to which all effects are applied.

The user terminal 120 may edit the received avatar content using an authoring tool.

The user terminal 120 adjusts the viewpoint of the camera work for the avatar content, applies additional animation effects to the avatar, adjusts the sync of the sound source to the avatar content, applies a stage effect, applies a special effect to the avatar, and applies a screen filter You can edit etc.

The user terminal 120 may use various additional functions in addition to generating avatar content through the avatar content providing service app. For example, the user terminal 120 may generate additional content based on the generated avatar content. As another example, when the user terminal 120 captures an image while the avatar content is being reproduced, the captured image may be stored in a photo album of the user terminal 120 . As another example, when a predetermined section is designated by the user in the timeline of the completed avatar content, the user terminal 120 may generate and store a highlight image for the designated predetermined section.

When the avatar content is edited, the user terminal 120 may re-upload the edited avatar content to the content providing server 110 in the form of a project. Also, the user terminal 120 may re-upload a captured image, a highlight image, etc. generated through an additional function used in the user terminal 120 to the content providing server 110 .

The user terminal 120 may include a smart phone, a tablet PC, etc. in which the avatar content is consumed in a 2D form, and a head mounted display (HMD) in which the avatar content is consumed in a 360° form.

2 is a block diagram of a content providing server according to an embodiment of the present invention. Referring to FIG. 2 , the content providing server 110 includes a receiving unit 210 , a recommended avatar providing unit 220 , an image analyzing unit 230 , a motion data extracting unit 240 , a scaling unit 250 , and a motion data analysis unit. It may include a unit 260 , a correction unit 270 , and a content generation unit 280 .

The receiver 210 may receive from the user terminal 120 an image obtained by photographing an object and metadata necessary for generating avatar content. Here, the image may be an image obtained by photographing various motions of an object, such as a dance or gesture, and the metadata may include information on an avatar to be retargeted and additional information necessary for generating avatar content. The additional information may include, for example, sound source information included in an image, camera information (eg, a 2D camera, a 3D camera, etc.) photographing an object, and background information (stage information). An avatar to be retargeted will be described in detail with reference to FIG. 3 .

3 is an exemplary diagram illustrating an avatar according to an embodiment of the present invention. Referring to FIG. 3 , the avatar to be retargeted may include either a human avatar 300 or an animal avatar 310 .

When the recommended avatar providing unit 220 first receives sound source information to be included in the image from the user terminal 120, it analyzes the genre of the sound source based on the received sound source information, and analyzes the genre of the sound source based on the analyzed genre of the sound source. Any one of the avatars may be selected and provided to the user terminal 120 .

For example, if the genre of the sound source is a dance song and the sound source requires a lot of movement because of the fast tempo, the recommended avatar providing unit 220 selects the human avatar 300 that allows the movement of the avatar to be expressed in various ways. Thus, it can be provided to the user terminal 120 .

As another example, when the genre of the sound source is a children's song, the recommended avatar providing unit 220 may select the animal-shaped avatar 310 and provide it to the user terminal 120 .

As another example, the recommended avatar providing unit 220 provides an avatar selected from the avatar contents previously registered in the content providing server 110 in the case of a sound source that does not require much movement because the genre of the sound source is ballad and the tempo is slow. After analyzing, an avatar mainly selected by a plurality of users may be provided to the user terminal 120 as a recommended avatar.

Returning to FIG. 2 , the image analyzer 230 may analyze the image based on the received metadata. Here, the image analyzer 230 may analyze the type information of the image based on the camera information. For example, when the camera information is a 2D camera, the image analyzer 230 may analyze the type information of the image as a 2D image. For another example, the image analysis unit 230 analyzes the type information of the image as a 2D image when the camera information is a 3D camera and is photographed with a simple camera, and converts the type information of the image into a 3D image when photographed with a depth camera. The video can be analyzed.

The motion data extractor 240 may extract motion data of an object from each frame of an image based on the analyzed image. For example, when the analyzed image is a 2D image, the motion data extractor 240 may extract motion data of an object from each frame of the image as x and y coordinates. As another example, when the analyzed image is a 3D image, the motion data extractor 240 may extract motion data of an object from each frame of the image as x, y, and z coordinates.

The motion data extractor 240 may convert the format of the extracted motion data based on the analyzed image type information. A process of converting the format of the motion data will be described in detail with reference to FIG. 4 .

4 is an exemplary diagram for explaining a process of extracting motion data and converting a format according to an embodiment of the present invention. Referring to FIG. 4 , for example, the motion data extractor 240 may extract motion data related to the body region 400 of the object divided based on each joint of the object for each frame of the image.

Here, the motion data has a different format according to the characteristics of the camera (for example, a 2D camera, a 3D camera, a simple camera, a depth camera, etc.) that captured the image, so it is converted into a common format to be retargeted to the avatar ( 410) can be done.

For example, in the case of kinect data, the motion data extraction unit 240 may include eyes 15, 16, ears 17, 18, and soles 19, 22, 21, 24 of the head region. and so on to convert motion data.

For another example, when the image is a 2D image captured by a 2D camera, the motion data extraction unit 240 uses machine learning such as deep learning for motion data composed of x and y coordinates extracted from the 2D image. By generating the z value, we can transform the x and y coordinates to consist of x, y, and z coordinates.

The motion data analyzer 260 may primarily analyze whether motion data is omitted for each frame of the image based on the format-converted motion data 420 .

The scaling unit 250 may scale the format-converted motion data 420 to correspond to the size of the avatar to be retargeted.

Returning to FIG. 2 , the motion data analyzer 260 may secondaryly analyze whether at least a portion of the motion data is missing for each frame based on the extracted motion data. The motion data analyzer 260 may analyze motion data for an entire frame of an image and classify the analyzed motion data into any one of body regions. For example, the motion data analyzer 260 may analyze the extracted motion data in units of all frames of the image (in the case of 30 fps for 1 second, if it is 30, all 900 frames are extracted). Thereafter, the motion data analysis unit 260 classifies the motion data analyzed for the entire frame of the image into a face region, a hand region, a foot region, and a body region among body regions. can

The motion data analyzer 260 may select a mapping target region to be mapped with any one classified body region based on the avatar information. Here, to briefly explain with reference to FIG. 3 , when the avatar to be retargeted is the human avatar 300 , the motion data analyzer 260 maps the entire body of the human avatar 300 to the body region to be mapped. can be selected as For another example, when the avatar to be retargeted is the animal avatar 310 , the motion data analyzer 260 may determine a mapping target region to be mapped with a body region based on the body region 311 of the animal avatar 310 . can be selected as In this case, motion data of the region other than the body region 311 may not be used to generate the avatar content.

The motion data analyzer 260 may derive an average value of the coordinates of the entire section corresponding to the mapping target area based on coordinate information for each section (eg, 1 second) corresponding to the selected mapping target area.

The motion data analysis unit 260 is configured to, based on the derived average value, when the difference between coordinate values for each section corresponding to the mapping target area is equal to or greater than a first threshold (eg, 40% or greater), at least corresponding to the mapping target area. It may be determined that motion data is missing in one section.

The motion data analyzer 260 may analyze the missing motion data for the entire frame and manage it as a timeline. The motion data analyzer 260 may analyze the missing motion data for the entire section to determine whether avatar content can be generated. For example, the motion data analyzer 260 may determine that generation of avatar content using the corresponding image is impossible when the missing motion data in the entire section is about 20% or more. In this case, it is possible to notify the user terminal 120 that generation of avatar content is impossible through a notification unit (not shown). As another example, when the missing motion data in the entire section is less than about 20%, the motion data analyzer 260 may determine that it is possible to generate avatar content using the corresponding image. In this case, a motion size for motion data is detected for each at least one frame, and may be managed for each section.

The compensator 270 may compensate at least a portion of the missing motion data for a frame in which at least a portion of the motion data is missing based on motion data of another frame or an alternative animation.

The compensator 270 calculates a correction section to which an alternative animation of the avatar is to be applied based on the motion data classified as a preset first region among the body regions, and in the calculated correction section, the first region (eg, body region) ), it may be determined whether the omission rate is equal to or greater than a second threshold (eg, 50%).

For example, when the omission ratio of the first region (eg, torso region) is less than a second threshold (eg, 50%), the compensator 270 may be configured to configure at least one of a frame before and after the correction section. By linearly interpolating the motion data corresponding to , new motion data for the correction section may be generated.

As another example, when the omission rate of the first region (eg, body region) is greater than or equal to a second threshold (eg, 50%), the compensator 270 may apply an alternative animation to the correction section. . Here, the alternative animation may include a jumping motion, a wing flapping motion, a shaking motion, and the like, which are pre-generated in relation to the avatar. For example, if the omission rate of the first region (eg, torso region) is equal to or greater than a second threshold (eg, 50%) and the correction period exceeds 1 second, the correction unit 270 may correct Alternative animations can be applied to sections. In this case, the correction section to which the alternative animation is applied may be managed based on the timeline. On the other hand, when the omission rate of the first region (eg, torso region) is greater than or equal to the second threshold (eg, 50%) and the correction period is 1 second or less, the correction unit 270 determines the correction period for the correction period. New motion data for the correction period may be generated by linearly interpolating motion data corresponding to at least one of a frame before and after a frame before and after the correction period by determining the period not to the extent to which the replacement animation is applied.

The compensator 270 may calculate a length of a correction section to which an alternative animation is to be applied, and detect a motion size of the object based on motion data corresponding to at least one of frames before and after the correction section. The compensator 270 may select one final replacement animation from among a plurality of candidate replacement animations based on the length of the compensation section and the detected motion size of the object. Here, as for the plurality of candidate replacement animations, a plurality of candidate replacement animations related to the avatar may be extracted in the order of increasing or decreasing the motion magnitude of the object.

For example, when the correction section corresponds to about 3 seconds, the corrector 270 may select a final replacement animation suitable for 3 seconds from among a plurality of candidate replacement animations. At this time, if there is no candidate replacement animation suitable for 3 seconds, a replacement animation longer than 3 seconds may be selected as the final replacement animation. Alternatively, when the length of the alternative animation, which has the longest playing time, is shorter than 3 seconds, the corrector 270 may adjust the length of the correction section by combining a plurality of candidate alternative animations.

The compensator 270 may perform a blending process so that the selected final replacement animation is naturally connected to the previous frame and the subsequent frame of the correction section. In this case, when the length of the final replacement animation is long, the section may be processed at the same rate as the previous frame and the subsequent frame.

A process of generating new motion data or applying an alternative animation to the correction section will be described in detail with reference to FIGS. 5A to 6B .

5A and 5B are exemplary views for explaining a process of generating new motion data for a correction section according to an embodiment of the present invention.

Referring to FIG. 5A , in the second frame 510 of the first to third frames 500 , 510 , and 520 of the image, a part of the body of the object is out of the screen area, and the motion data of the object is in the second frame 510 . Assume that (511) is missing.

Referring to FIG. 5B , the compensator 270 interpolates the motion data 501 of the first frame 500 and the motion data 521 of the third frame 520 to obtain new motion data for the correction section 530 . (531) can be created.

6A and 6B are exemplary views for explaining a process of applying an alternative animation to a correction section according to an embodiment of the present invention.

Referring to FIG. 6A , in the second frame 610 of the first frame 600 and the second frame 610 of the image, a body part of the object is out of the screen area, and the motion data of the object in the second frame 610 is Assume that (611) is missing.

The compensator 270 may apply an alternative animation 630 in which the avatar shakes the body left and right to the correction section 620 .

Returning to FIG. 2 again, the correction unit 270 may perform additional correction by applying camera work and stage effects to the correction section. Here, the camera work and the stage effect may preferentially apply additional effects to the corresponding corrected section in order to further conceal the corrected section through generation of new motion data or application of an alternative animation.

Here, the camera work may be variously applied, but an action related to the camera work which is mainly used may be defined in advance. For the camera work, functions such as zoom-in/zoom-out, camera rotation around an avatar, and camera viewpoint conversion may be applied, for example. Here, in the case of rotating the camera around the avatar, the camera may be rotated from the left to the front of the avatar or from the right to the front of the avatar. In the case of the camera viewpoint transformation, in the case of the front, viewpoint transformation is applied mainly to the full shot or avatar face, and in the case of the left/right side, viewpoint transformation is applied mainly to the full shot or the avatar's face, and viewpoint transformation to the stage screen can be applied .

The correction unit 270 sets the time point at which the correction section starts as a key frame for applying the camera walk, and based on the set key frame, the genre of the sound source included in the image and the motion size of the detected object. can be applied. For example, when the genre of the sound source corresponds to a dance song and the motion size of the detected object is large, the compensator 270 designates a short application interval of the camera work (for example, within 5 seconds) and applies it. can do. For another example, if the genre of the sound source corresponds to a ballad song and the motion size of the detected object is small, the compensator 270 designates a long application interval of the camera work (for example, set to 10 seconds or more) can be applied.

The compensator 270 may apply a camera zoom-in/zoom-out and a camera rotation speed faster or slower based on the genre of the sound source included in the image and the motion size of the object.

When an equal key frame is set, the compensator 270 may not additionally set an equal key frame if there is a key frame setting due to the correction of the animation of the avatar within 3 seconds from the previous frame and the subsequent frame. can In addition, the compensator 270 may set the camera work randomly from the designation of the key frame, but may not set the previously set effect continuously.

The content generator 280 may generate avatar content by retargeting the extracted motion data to the avatar. For example, the content generator 280 may generate a screen on which a retargeted avatar and a sound source are reproduced based on the motion data of the object as avatar content of the 2D image. In this case, the generated avatar content can be managed in the form of a project that can be reproduced in a 360° environment and edited the avatar content.

7 is a flowchart of a method for providing avatar content based on motion data of an object in a content providing server according to an embodiment of the present invention. The method for providing avatar content based on the motion data of an object in the content providing server 110 shown in FIG. 7 is processed in time series by the content providing system 1 according to the embodiment shown in FIGS. 1 to 6B. includes steps. Accordingly, even if the description is omitted below, it is also applied to the method of providing avatar content based on the motion data of the object in the content providing server 110 according to the embodiment shown in FIGS. 1 to 6B .

In step S710 , the content providing server 110 may receive from the user terminal 120 an image of an object and metadata necessary for generating the avatar content.

In step S720, the content providing server 110 may analyze the image based on the received metadata.

In step S730, the content providing server 110 may extract motion data of the object from each frame of the image based on the analyzed image.

In step S740, the content providing server 110 may analyze whether at least a portion of the motion data is missing for each frame based on the extracted motion data.

In step S750 , the content providing server 110 may supplement at least a portion of the missing motion data based on motion data of another frame or an alternative animation for a frame in which at least a portion of the motion data is missing.

In step S760, the content providing server 110 may generate avatar content by retargeting the extracted motion data to the avatar.

In the above description, steps S710 to S760 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between the steps may be switched.

8 is a block diagram of a user terminal according to an embodiment of the present invention. Referring to FIG. 8 , the user terminal 120 may include an input unit 810 , a photographing unit 820 , a transmitting unit 830 , a receiving unit 840 , a display unit 850 , and an editing unit 860 .

The input unit 810 may receive sound source information to be included in an image. For example, when the input unit 810 receives sound source information to be reproduced while capturing an image, the transmitter 830 may transmit the input sound source information to the content providing server 110 .

The receiver 840 may receive a recommended avatar selected based on sound source information from the content providing server 110 . Here, the recommended avatar may be one selected from among a plurality of characters based on the genre, tempo, and the like of the sound source.

The input unit 810 may directly receive an avatar and additional information required for generating avatar content from a user. The input unit 810 may receive either a human-type avatar or an animal-type avatar the same as a human body type as a character to be retargeted based on the user's motion from the user. Also, the input unit 810 may receive sound source information included in an image, camera information (eg, a 2D camera or a 3D camera) photographing an object, background information (stage information), and the like.

The photographing unit 820 may acquire an image obtained by photographing the object. For example, the photographing unit 820 may photograph an object according to a predetermined guide using a camera (eg, a 2D camera or a 3D camera) provided in the user terminal 120 .

The transmitter 830 may transmit an image of the object and metadata necessary for generating the avatar content to the content providing server 110 .

The receiver 840 may receive the avatar content generated based on the image and metadata from the content providing server 110 . For example, when the receiving unit 840 receives a request from the user to download the avatar content through the avatar content service providing app, the receiver 840 may receive the avatar content from the content providing server 110 . In this case, the receiver 840 may receive the avatar content for simple reproduction or editing of the avatar content according to the user's selection, and the avatar content for the project that can be played in a 360° environment.

The display unit 850 may display the received avatar content.

The editing unit 860 may allow the user to manually edit the received avatar content using an authoring tool. Here, the avatar content may be in the form of a project in which the avatar content can be edited. For example, the editor 860 may adjust the viewpoint of the camera work for the avatar content, apply additional animation effects to the avatar, adjust the sync of the sound source to the avatar content, apply and change stage effects, and apply special effects to the avatar , application and change of screen filters can be edited. The process of editing the avatar content will be described in detail with reference to FIG. 9 .

9 is an exemplary diagram illustrating edited avatar content according to an embodiment of the present invention. Referring to FIG. 9 , the editing unit 860 may edit the avatar content so that the position, rotation range, etc. of the camera are adjusted or changed through the viewpoint adjustment 900 of the camera walk.

The editor 860 may receive a selection of an accessory to be applied to the avatar through an additional animation effect 910 on the avatar, and edit the avatar content so that the selected accessory is applied to the avatar.

The editor 860 may edit the avatar content so that the pollen 920 effect is applied to the avatar by applying a special effect to the avatar, or edit the avatar content so that the wings 930 effect is applied to the avatar.

10 is a flowchart of a method of providing avatar content based on motion data of an object in a user terminal according to the present invention. The method for providing avatar content based on the motion data of an object in the user terminal 120 shown in FIG. 10 includes the steps of being processed time-series by the content providing system 1 according to the embodiment shown in FIGS. 1 to 9 . include those Accordingly, even if omitted below, the method of providing avatar content based on motion data of an object in the user terminal 120 according to the embodiment shown in FIGS. 1 to 9 is also applied.

In step S1010, the user terminal 120 may acquire an image of the object.

In step S1020 , the user terminal 120 may transmit an image of the object and metadata necessary for generating the avatar content to the content providing server 110 .

In step S1030 , the user terminal 120 may receive the avatar content generated based on the image and metadata from the content providing server 110 .

In step S1040, the user terminal 120 may display the received avatar content.

In the above description, steps S1010 to S1040 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between the steps may be switched.

The method for providing avatar content based on motion data of an object in the content providing server and the user terminal described through FIGS. 1 to 10 is a record including a computer program stored in a medium executed by a computer or instructions executable by the computer It can also be implemented in the form of a medium. Also, the method for providing avatar content based on motion data of an object in the content providing server and the user terminal described with reference to FIGS. 1 to 10 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

110: content providing server
120: user terminal
210: receiver
220: recommended avatar provider
230: image analysis unit
240: motion data extraction unit
250: scaling unit
260: motion data analysis unit
270: correction unit
280: content generation unit
810: input unit
820: filming unit
830: transmission unit
840: receiver
850: display unit
860: editorial department

Claims (20)

A server that provides avatar content based on motion data of an object, the server comprising:
a receiving unit for receiving an image of an object and metadata necessary for generating the avatar content from a user terminal;
an image analysis unit that analyzes the image based on the received metadata;
a motion data extraction unit for extracting motion data of the object from each frame of the image based on the analyzed image;
a motion data analysis unit for analyzing whether at least a portion of the motion data is missing for each frame based on the extracted motion data;
a compensator for compensating for at least a portion of the missing motion data based on motion data of another frame or a replacement animation for a frame in which at least a portion of the motion data is missing; and
A content generation unit generating the avatar content by retargeting the extracted motion data to the avatar.
Containing, content providing server.
The method of claim 1,
The metadata includes information on the avatar to be retargeted and additional information necessary for generating the avatar content.
3. The method of claim 2,
The avatar includes either a human avatar or an animal avatar,
The additional information includes at least one of sound source information included in the image, camera information for photographing the object, and background information, the content providing server.
4. The method of claim 3,
The image analysis unit analyzes the type information of the image based on the camera information,
The motion data extraction unit converts the format of the extracted motion data based on the type information of the analyzed image,
The content providing server, wherein the motion data analyzer analyzes whether the motion data is missing for each frame of the image based on the format-converted motion data.
5. The method of claim 4,
The content providing server, further comprising a scaling unit for scaling the format-converted motion data to correspond to the size of the retargeted avatar.
3. The method of claim 2,
The motion data analyzer analyzes the motion data for the entire frame of the image, and classifies the analyzed motion data into any one of body regions.
7. The method of claim 6,
The motion data analyzer selects a mapping target region to be mapped with any one of the classified body regions based on the information of the avatar, and applies the mapping target region to the mapping target region based on coordinate information for each section corresponding to the selected mapping target region. Which is to derive the average value for the entire section, the content providing server.
8. The method of claim 7,
When the difference between coordinate values for each section corresponding to the mapping target area is equal to or greater than a first threshold based on the derived average value, the motion data analysis unit detects that the motion data is missing from at least one section corresponding to the mapping target area. It is determined that, the content providing server.
9. The method of claim 8,
The compensator calculates a correction section to which an alternative animation of the avatar is to be applied based on motion data classified as a preset first region among the body regions, and the omission rate of the first region in the calculated correction section is a second The content providing server to determine whether or not the threshold value or more.
10. The method of claim 9,
When the omission rate of the first region is less than a second threshold, the compensator interpolates motion data corresponding to at least one of a frame before and after the correction section to generate new motion data for the correction section. , content delivery server.
10. The method of claim 9,
The compensator applies the replacement animation to the correction section when the omission ratio of the first region is equal to or greater than a second threshold.
12. The method of claim 11,
The compensator calculates a length of a correction section to which the replacement animation is applied, detects a motion size of the object based on motion data corresponding to at least one of a frame before and after a frame of the correction section, and The content providing server, which selects a final replacement animation of any one of a plurality of candidate replacement animations based on the length and the motion magnitude of the detected object.
13. The method of claim 12,
The content providing server that the correction unit performs additional correction by applying camera work and stage effects to the correction section.
14. The method of claim 13,
The compensator sets a time point at which the correction section starts as a key frame, and applies the camera work based on the genre of the sound source included in the image and the motion size of the detected object based on the set key frame In, content providing server.
3. The method of claim 2,
When the sound source information is received from the user terminal, the genre of the sound source is analyzed based on the received sound source information, and a recommended avatar from among a plurality of avatars is selected and provided based on the analyzed genre of the sound source. A content providing server further comprising a recommended avatar providing unit.
A user terminal providing avatar content based on motion data of an object, the user terminal comprising:
a photographing unit for obtaining an image of photographing an object;
a transmission unit for transmitting the image of the object and metadata necessary for generating the avatar content to a content providing server;
a receiver configured to receive the avatar content generated based on the image and the metadata from the content providing server; and
Comprising a display unit for displaying the received avatar content,
the image is analyzed based on the metadata,
Motion data of the object is extracted from each frame of the image based on the analyzed image,
Whether or not at least a portion of the motion data is missing is analyzed for each frame based on the extracted motion data,
At least a portion of the missing motion data is supplemented for a frame in which at least a portion of the motion data is missing based on motion data of another frame or a replacement animation;
The avatar content is generated by retargeting the extracted motion data to the avatar.
17. The method of claim 16,
The user terminal further comprising an editing unit for editing the received avatar content using an authoring tool.
18. The method of claim 17,
The editing unit adjusts the viewpoint of the camera work with respect to the avatar content, applies additional animation effects to the avatar, adjusts the sync of the sound source to the avatar content, applies and changes stage effects, applies special effects to the avatar, screen Editing at least one of applying and changing the filter, the user terminal.
17. The method of claim 16,
Further comprising an input unit for receiving sound source information to be included in the image,
The transmitter transmits the input sound source information to the content providing server,
The receiving unit receives the recommended avatar selected based on the sound source information from the content providing server, the user terminal.
A server that provides avatar content based on motion data of an object, the server comprising:
receiving an image obtained by photographing an object and metadata necessary for generating the avatar content from a user terminal;
analyzing the image based on the received metadata;
extracting motion data of the object from each frame of the image based on the analyzed image;
analyzing whether at least a portion of the motion data is omitted for each frame based on the extracted motion data;
compensating for a frame in which at least a portion of the motion data is missing based on motion data of another frame or replacement animation; and
generating the avatar content by retargeting the extracted motion data to an avatar
Including, a content providing method.

Images