KR102371072B1 - Method, appararus and system for providing real-time broadcasting platform using motion and facial capture - Google Patents

Abstract

A method of providing a real-time broadcasting platform using motion and face capture is disclosed. The method of providing a real-time broadcasting platform of the present invention includes the steps of extracting the user's motion capture data and face capture data in real time based on the motion detection sensor mounted on the user's body and the image captured by the user, respectively, and the extracted motion capture data Creating an animation of a preset character representing a motion corresponding to , and representing a facial expression corresponding to rigging data to which face capture data is applied, and providing a broadcasting platform for transmitting the generated animation in real time. includes

Description

Method, device and system for providing real-time broadcasting platform using motion and face capture { METHOD, APPARARUS AND SYSTEM FOR PROVIDING REAL-TIME BROADCASTING PLATFORM USING MOTION AND FACIAL CAPTURE }

The present invention relates to a method, apparatus and system for providing a real-time broadcasting platform using motion and face capture, and more particularly, to a method for providing a real-time broadcasting platform using motion and face capture to control a virtual character in real time; It relates to a device and its system.

Recently, as one-person media has become a trend, 'creating your own content' is changing into an era where money goes beyond a simple hobby. The popularity of one-person broadcast video content centered on YouTube, a representative content distribution channel and platform, is reorganizing the existing media market and content industry along with the growth of one-person creators.

In addition to this, various one-person media support projects have been added to create an environment where new creators can more easily enter the one-person media market and produce high-quality content. In addition, the contents produced in this way are loved by viewers around the world by going beyond geographical, cultural, and linguistic limitations.

As such, the growth of one-man media centering on the Internet broadcasting platform is the birth of the Multi Channel Network (MCN), which serves as an agency for one-person creators who create content, a platform that distributes content, viewers who consume content, and one-person creators. It has led to changes and growth throughout the market.

Since the number of viewers and revenue are the criteria for the influence and popularity of creators, creators are also using various methods to attract viewers' attention and induce influx.

Among them, the most representative method is to use a camera to film the creator, and the creator to respond immediately to the chatting of the viewer to induce a response from the viewer. This is a method made possible by the Internet broadcasting platform.

However, this method is difficult to use for creators who are burdened with revealing their faces.

In addition, there are many creators who want to induce viewers to flow by playing popular cartoon or animal characters using motion capture, but these motion capture and animation equipment are expensive and burdensome for creators who want to start broadcasting for the first time.

Therefore, in order to solve the above problems, the need for a motion capture solution that can communicate (react) with a viewer without revealing the face and is inexpensive and has a low price is growing.

An object of the present invention is to provide a method, an apparatus, and a system for providing a real-time broadcasting platform using motion and face capture capable of realizing a virtual character showing the shape and movement of a real person in order to solve the above problems. do it with

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

A method for providing a real-time broadcasting platform using motion and face capture according to an embodiment of the present invention is based on a motion detection sensor mounted on a user's body and an image captured by the user, respectively, the user's motion capture data and face capture Extracting data in real time, generating an animation of a preset character representing a motion corresponding to the extracted motion capture data and representing a facial expression corresponding to rigging data to which the face capture data is applied; and providing a broadcasting platform for transmitting the generated animation in real time.

In this case, the extracting may extract the motion capture data based on a value measured through a gyro sensor attached to the user's body.

In addition, the extracting includes extracting coordinate data corresponding to the user's movement through the gyro sensor, converting the extracted coordinate data into body motion data, and transferring the converted body motion data to a renderer engine. It may include performing correction by comparing it with the simulation data of the present invention, and extracting the corrected body motion data as the motion capture data.

In addition, the extracting may include recognizing a preset object from the user's face in the image captured by the user, and extracting marking data representing the shape of the recognized object as the face capture data.

In addition, in the extracting step, when the preset object is recognized, a plurality of markers disposed at an interval on at least one of a center line and an outline of the recognized object are generated, and then the plurality of generated markers are connected. It is possible to generate the marking data representing the shape of the recognized object.

In addition, the extracting may correct the position of the marker of the marking data according to a user input.

In addition, the generating includes applying the extracted face capture data to the rigging data of the preset character, correcting an error of the rigging data with the face capturing data, and rigging data in which the error is corrected Based on the , it may include the step of generating the change in the expression of the character as an animation.

In addition, the method further includes the step of selecting any one character for generating the animation from among a plurality of characters and providing a UI for customizing any one of the appearance and operation of the selected character through the broadcasting platform can do.

In addition, the UI may be produced as an add-on type UI script.

On the other hand, the apparatus for providing a real-time broadcasting platform using motion and face capture according to an embodiment of the present invention receives the measured value measured from the motion sensor mounted on the user's body and the image captured by the user in real time, respectively. An input unit, a storage unit for storing motion data and rigging data corresponding to the motion capture data, and extracting the user's motion capture data and face capture data from the measured value and the captured image, respectively, and the stored Based on the motion data and the rigging data, an animation of a preset character representing a motion corresponding to the extracted motion capture data and representing a facial expression corresponding to the rigging data to which the face capture data is applied is generated, and the generated animation It includes a processor that provides a broadcasting platform for transmitting in real time.

In this case, the motion detection sensor may be a gyro sensor.

In addition, the processor extracts coordinate data corresponding to the user's movement through the gyro sensor, converts the extracted coordinate data into body motion data, and compares the converted body motion data with simulation data by a renderer engine to perform the correction, and the corrected body motion data may be extracted as the motion capture data.

In addition, the processor may recognize a preset object from the user's face in the image captured by the user, and extract marking data representing the shape of the recognized object as the face capture data.

In addition, when the preset object is recognized, the processor generates a plurality of markers disposed at an interval on at least one of a center line and an outline of the recognized object, and then connects the generated markers to recognize the plurality of markers. It is possible to generate the marking data indicating the shape of the object.

In addition, the device may further include a user interface, and the processor may correct the marker position of the marking data according to a user input input through the user interface.

In addition, the processor applies the extracted face capture data to the rigging data of the preset character, corrects an error of the rigging data with the face capture data, and based on the rigging data for which the error is corrected, the You can create an animation of a character's facial expression change.

In addition, the processor may select any one character for generating the animation from among a plurality of characters and provide a UI for customizing any one of an appearance and operation of the selected character through the broadcasting platform.

In addition, the UI may be produced as an add-on type UI script.

On the other hand, the system for providing a real-time broadcasting platform using motion and face capture according to an embodiment of the present invention includes a camera for photographing a user, a motion detection sensor mounted on the user's body, and a measurement value measured from the motion detection sensor and receiving the images photographed from the camera in real time, extracting motion capture data and face capture data of the user from the received measurement values and the photographed images, respectively, and corresponding to the extracted motion capture data and an electronic device that provides a broadcasting platform for generating an animation of a preset character representing a facial expression corresponding to rigging data to which the face capture data is applied, and transmitting the generated animation in real time. do.

According to various embodiments of the present invention as described above, by performing motion capture of a creator in real time and controlling a character created in advance with such motion capture data, the creator uses a desired character or a created character to communicate in real time with a viewer It can be used as a low-cost broadcasting solution.

In order to more fully understand the drawings cited in the Detailed Description, a brief description of each drawing is provided.
1 is a flowchart for briefly explaining a method of providing a real-time broadcasting platform using motion and face capture according to an embodiment of the present invention;
2 is a flowchart for briefly explaining a data processing method for providing a real-time broadcasting platform according to an embodiment of the present invention;
3 is a flowchart for generating face capture data and animation data of a character according to an embodiment of the present invention;
4 and 5 are screens for explaining a process of extracting face capture data according to an embodiment of the present invention;
6 is a flowchart for explaining a method of generating animation data by applying face capture data to rigging data according to an embodiment of the present invention;
7 is a flowchart for explaining a process of extracting motion capture data by a neuron sensor according to an embodiment of the present invention;
8 is a view showing various examples of a body range in which a neuron sensor is worn according to an embodiment of the present invention;
9 is a view showing a state in which motion capture data is generated by a neuron sensor according to an embodiment of the present invention;
10 is a flowchart for explaining a process of correcting motion captured data by a neuron sensor according to an embodiment of the present invention;
11 is a diagram for explaining a motion capture data extraction optimization algorithm according to an embodiment of the present invention;
12 is a view showing a motion capture result screen by a gyro sensor according to an embodiment of the present invention;
13 is a flowchart for explaining a method of extracting motion capture data by an optical sensor according to an embodiment of the present invention;
14 and 15 are views showing a state in which motion capture data is generated by an optical sensor according to an embodiment of the present invention;
16 and 17 are diagrams showing a character customization UI and a broadcast transmission UI according to an embodiment of the present invention;
18 is a view showing a screen in which a real-time 3D engine is interlocked with a broadcasting platform in real time according to an embodiment of the present invention;
19 is a block diagram schematically illustrating the configuration of an apparatus according to an embodiment of the present invention;
20 is a block diagram schematically illustrating the configuration of a system according to an embodiment of the present invention.

First, terms used in the present specification and claims have been selected in consideration of functions in various embodiments of the present invention. However, these terms may vary depending on the intention of a person skilled in the art, legal or technical interpretation, and emergence of new technology. Also, some terms may be arbitrarily selected by the applicant. These terms may be interpreted in the meanings defined herein, and in the absence of specific definitions, they may be interpreted based on the general content of the present specification and common technical common sense in the art.

Also, the same reference numerals or reference numerals in each drawing appended hereto indicate parts or components that perform substantially the same functions. For convenience of description and understanding, the same reference numbers or reference numerals are used in different embodiments to be described. That is, even though all the components having the same reference number are shown in the plurality of drawings, the plurality of drawings do not mean one embodiment.

Also, in this specification and claims, terms including ordinal numbers such as 'first' and 'second' may be used to distinguish between elements. This ordinal number is used to distinguish the same or similar components from each other, and the meaning of the term should not be limitedly interpreted due to the use of the ordinal number. As an example, for components combined with such an ordinal number, the order of use or arrangement of the elements should not be construed as being limited by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as 'comprise' or 'comprise' are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other It should be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

Also, in an embodiment of the present invention, when it is said that a part is connected to another part, this includes not only direct connection but also indirect connection through another medium. In addition, the meaning that a certain part includes a certain component means that other components may be further included, rather than excluding other components, unless specifically stated to the contrary.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

1 is a flowchart for briefly explaining a method for providing a real-time broadcasting platform using motion and face capture according to an embodiment of the present invention.

First, based on the motion detection sensor mounted on the user's body and the image captured by the user, respectively, the user's motion capture data and face capture data can be extracted in real time (S110).

Specifically, the motion capture data is extracted based on a value measured through a motion sensor mounted on the user's body. In this case, the motion detection sensor may be implemented as a gyro sensor.

The gyroscope may be implemented as an IMU (Inertial Measurement Unit) including a 3-axis gyroscope, a 3-axis accelerometer, and a 3-axis magnetometer, and is attached to the user's body by a strap. and can be tightened. The IMU is also commonly referred to as a neuron sensor.

After receiving a value measured in real time from the gyro sensor, the hub may transmit this data to a device providing a broadcasting platform through wired/wireless communication such as Wi-Fi or USB.

Coordinate data corresponding to the user's movement may be extracted through the gyro sensor mounted on the user's body, and the extracted coordinate data may be converted into body motion data. In this case, correction may be performed by comparing the converted body motion data with simulation data by the renderer engine, and the body motion data on which the correction has been performed may be extracted as motion capture data.

Meanwhile, the user may recognize a preset object from the user's face in the captured image, and may extract marking data representing the shape of the recognized object as the face capture data. Here, the marking data may be data representing the shape of the recognized object by connecting a plurality of markers disposed at an interval to at least one of a center line and an outline of the recognized object.

At this time, it is possible to correct the position of the marker of the marking data according to the user input.

Thereafter, an animation of a preset character representing a motion corresponding to the extracted motion capture data and representing a facial expression corresponding to the rigging data to which the face capture data is applied is generated (S120).

In this case, the extracted face capture data may be applied to rigging data of a preset character, and an error between the rigging data and the face capture data may be corrected. At this time, based on the rigging data for which the error is corrected, the change in the expression of the character may be generated as an animation.

Thereafter, a broadcasting platform for transmitting the generated animation in real time is provided (S130). In this case, any one of the plurality of characters for generating animation may be selected, and a UI for customizing any one of the appearance and operation of the selected character may be provided through the broadcasting platform.

Hereinafter, specific examples of each step will be described later along with the drawings.

2 is a flowchart for briefly explaining a data processing method for providing a real-time broadcasting platform according to an embodiment of the present invention.

When the content is planned (S210), based on 3D character modeling, through a real-time renderer engine, it is possible to implement modeled data close to the real thing (S220). In this case, the real-time renderer engine may be implemented as, for example, an unreal engine.

Thereafter, through character rigging, a range for human movement is designated (S230), and motion capture data and face capture data are extracted through motion/face capture (S240). In this case, the motion capture data can be extracted by converting actual human motion into data through gyro-type motion capture, and the face capture data can be extracted by capturing the motion of each part of the face through an image taken without a marker.

Thereafter, a 3D animation representing a motion corresponding to the extracted motion capture data and representing a facial expression corresponding to the rigging data to which the face capture data is applied can be generated (S250), and a lively animation character is applied by applying it to a preset character (S260) can be implemented.

On the other hand, simultaneously or separately from steps S210 to S260, a software for providing a UI script capable of customizing the character implemented in steps S210 to S260 to a real-time image transmission platform (real-time broadcasting platform) is planned (S270), and the planned It is possible to design a UI for specifying selection, customization, and effects of a character provided by the software (S280). When programming for implementing the designed UI is performed (S290), animation data of the character implemented in step S260 and programming data implemented in step S290 may be transmitted to the real-time image transmission platform 10 .

3 is a flowchart for generating face capture data and animation data of a character according to an embodiment of the present invention.

According to the markerless algorithm shown in Fig. 3, in generating the face capture data for the facial expression performance of the customized character, unlike the existing face capturing method in which a marker is directly captured on the face and recognized, the face capture data without a marker , and based on this, animation data representing the movement of the rigging data can be generated.

First, face data is photographed through the camera (S310), and a position for each face part is set according to a markerless algorithm (S320).

Thereafter, the position data for each face part is tracked and analyzed (S330), and the rigging data to which the movement is applied is loaded (S340).

The tracked position data is added to the rigging data (S350), and the error due to the error and noise of each position for each face part is corrected (S360). At this time, it is determined whether the difference between the value of each position for each actual face part and the corrected value is equal to or greater than a preset threshold value (S370), and if it is equal to or greater than the threshold value, it is determined that the output is abnormal (S370:N) and step S360 redo If it is less than the threshold, it is determined that the output is normal (S370:Y), and a data optimization process is performed (S380), and finally optimized animation data can be generated (S390).

4 and 5 are screens for explaining a process of extracting face capture data according to an embodiment of the present invention.

As shown in FIG. 4 , objects such as eyes, nose, and mouth, which are main points for representing facial expressions in a user's captured image, may be automatically recognized. At this time, after creating markers by spacing them on the center line or outline of each recognized object, the created markers are connected to generate the marking data 41 representing the shape of each object.

At this time, as shown in FIG. 5 , since the automatically recognized marker of each object may have an error with the center line or outline of the actual part, it is possible for the user to fine-tune the position of the marker directly according to a user input.

6 is a flowchart illustrating a method of generating animation data by applying face capture data to rigging data according to an embodiment of the present invention.

As shown in FIG. 6 , when detailed adjustment of the marker is finished, the markers 61-1, 61-2, and 61-3 that are the standards for performing the customized character's facial expression are selected, and the tracked marker data is converted into the character. It can be applied to rigging data. Accordingly, the points 62-1, 62-2, 62-3 corresponding to the selected markers 61-1, 61-2, and 61-3 in the rigging data are the markers 61-1, 61-2, 61 It can be synchronized with the movement of -3).

7 is a flowchart illustrating a process of extracting motion capture data by a gyro sensor according to an embodiment of the present invention.

According to the motion capture algorithm of FIG. 7 , motion can be captured by using a gyro sensor without a marker, instead of the conventional most widely used method of capturing motion by attaching a marker to the body and photographing it.

First, the gyroscope and the hub may be set up (S710), and calibration may be performed on the detected motion capture data (S720). At this time, it is determined whether the output of the motion capture data is within the normal range (S730), and if it is within the normal range (S730:Y), the coordinates of the output motion capture data are stored (S740). If it is not within the normal range (S730:N), calibration may be performed again. Whether the output of the motion capture data is within a normal range may be determined according to whether the difference between the output value and a preset threshold value is greater than or equal to a preset value by comparing the output value.

Thereafter, the coordinate values of the stored data are converted into body motion data (S750), and errors due to noise and errors of the converted body motion data with actual motions are corrected (S760). It is determined whether the output of the corrected body motion data is within the normal range (S770), and if it is within the normal range (S770:Y), data optimization is performed (S780), and final body motion data is output (S790).

When the output of the corrected body motion data is not within the normal range (S770:N), the correction in step S760 may be re-performed.

Whether the output of the body motion data is within a normal range may be determined according to whether the difference between the output value and a preset threshold value is greater than or equal to a preset value by comparing the output value.

8 is a diagram illustrating various examples of a body range in which a neuron sensor is worn according to an embodiment of the present invention.

The neuron sensor checks the movement in real time through an IMU (Inertial Measurement Unit) with a built-in 3-axis gyroscope, 3-axis accelerometer, and 3-axis magnetometer. After being calculated, the 9 axis sensor units are transmitted at a speed of 60 fs to a hub that is mounted together on the body of the user (actor). The data transmitted to the hub may be finally delivered to the electronic device 100 that outputs the motion capture data through Wi-Fi or a USB cable.

The number of sensors mounted on the body may be set as (a) to (d) according to the range for capturing detailed movements.

In the case of 11 neurons as shown in (a), they are attached only to the arms, legs, and torso, excluding the hands and feet, so that motion capture data can be tracked.

In the case of 18 neurons as shown in (b), in the embodiment of (a), they are additionally attached to the shoulder, hand, and foot to track the motion capture data.

In the case of 21 neurons as shown in (c), in the embodiment of (b), they are additionally attached to the finger of the left hand so that motion capture data can be tracked.

In the case of 32 neurons as shown in (d), in the embodiment of (c), they are additionally attached to the finger of the right hand to track the motion capture data.

9 is a diagram illustrating a state in which motion capture data is generated by a gyro sensor according to an embodiment of the present invention.

When a user wears a neuron, modeling as shown in FIG. 9A may be performed. In this case, the body size of the user wearing the neurons may be selected and applied, and calibration and posture correction may be performed.

At this time, since the motion capture method using the neuron sensor has a large margin of error compared to the existing motion capture method for markers, an algorithm for calibrating it

Calibration may be performed to prevent cross-talk of data from external noise and to accurately extract the user's movement. In the case of a motion with a large movement, the calibration may not be accurate. In this case, the accuracy of the motion capture can be maintained through rework of the calibration.

As shown in (b) of FIG. 9 , after taking four postures, it is maintained until these postures are recognized to perform calibration, and then capture can be performed. Figure 9 (b) shows four poses for correction.

12 , according to the detected motion of the user 1210 , motion data 122 corresponding to the motion capture data is generated, and a preset character 1230 is generated according to the movement of the motion data 122 . The movement can be performed to correspond.

10 is a flowchart illustrating a process of correcting motion captured data by a gyro sensor according to an embodiment of the present invention.

First, while performing motion capture through the gyro sensor (S1010-1), the actual coordinate values of the body motion data are extracted (S1020-1). At the same time, simulation through a real-time renderer engine such as Unity or Unreal engine is performed (S1010-2), and an operation of extracting coordinate values of the simulated data (S1020-2) is simultaneously performed. can

Accordingly, when body motion data and simulation data are respectively output (S1130-1 and S1130-2), the output body motion data and simulation data are compared with each other, and correction is performed so that the body motion data is closest to the simulation data. do.

In this case, the correction process may be performed by correcting an error of a position value, an error caused by external noise, an error caused by a violent movement, and the like.

The body motion data corrected in this way is output as final body motion data for expressing the movement of the character (S1040). And, the above comparison operation and correction operation may be scripted.

11 is a diagram for explaining a motion capture data extraction optimization algorithm according to an embodiment of the present invention.

The extracted data of the gyro-type motion capture method to be used in the present invention is based on BVH. Since BVH data is bone data that contains position information on the X, Y and Z axes including motion position and rotation values, the position and rotation values of the rigging data are placed on the character bone to be used. When converges to (0,0,0), the exact position value can be extracted.

However, in the current method 1110, since there is no function to automatically correct the extracted data to (0,0,0), the location data of all data does not come out in a consistent form when outputting BVH data, and the data itself is an error. This occurs, and an additional correction process is performed after output, and as a result, external correction is performed in the progress of outputting the final data.

In order to compensate for this, a conversion process 1120 for correcting X, Y and Z values to (0,0,0) is added in the middle of the process of outputting neuron data and outputting bone data (1130) in real time. This allows the position value of the data to be output normally, thereby reducing the delay until the final data output.

After creating software that runs on the real-time renderer engine, you can create scripts that can be applied internally through C++ and JAVA, and implement data extraction optimization in the form of checking and changing the extracted data when outputting the final data in the program. .

13 is a flowchart for explaining a method of extracting motion capture data by an optical sensor according to an embodiment of the present invention.

The motion capture data of the present invention may be extracted by an optical sensor. The optical motion capture method may be performed with an optical camera, a switching hub, a router, a motion suit with a marker attached thereto, and equipment including a marker.

The optical camera is a configuration that transmits the retro-reflected position data value to the switching hub by projecting the actor (user) infrared rays through a high-resolution camera with an infrared lens.

The switching hub is connected to the optical camera with an Ethernet cable, supplies power, receives captured data, and delivers it to the router.

The router is configured to transmit data received from each switching hub to the electronic device 100 for outputting the final motion capture data.

The electronic device 100 is configured to process the camera position value transmitted from the router in real time through a dedicated program, and output motion capture data.

A motion suit is a suit to which a marker can be attached to express the movement of an actor, and the marker is attached to each joint part of the actor wearing the motion suit and expresses the position value of the body by infrared reflection.

When a high-resolution optical camera with a built-in infrared emitting and detection lens emits infrared light toward an actor, the marker coated with a reflective material reflects the infrared and detects the returned data, and through this, motion capture based on the actor's position and angle data can be saved.

In the motion capture data, when a marker is simultaneously observed through infrared rays emitted from two or more optical cameras to which inertial and optical algorithms are applied, the three-dimensional coordinates of the marker are calculated by a trigonometric function to extract the position value, and the position value is set at 120 fps. It is transmitted to the switching hub at a higher speed. The transmitted data is finally delivered to the electronic device 100 via Ethernet through the router.

The first form of adding foot positions (4 places) to the basic body joint parts (37 places) when capturing whole body movements, or markers for thighs, arms, chest, and head (13 places) in the basic body joint parts (37 places) ) can be added to capture the actor's motion.

Specifically, as shown in FIG. 13 , sensor correction is performed to organize the surrounding environment by setting the camera position and angle ( S1310 ). Remove any shiny objects or unnecessary obstructions that may block the camera view, and use a non-reflective mat on the floor to minimize reflection distortion. In order to efficiently use the captured data, it is necessary to properly arrange the cameras, and after making a plan for the camera layout, the cameras can be arranged.

Since accurate motion values can be obtained only when markers are tracked simultaneously through two or more synchronized cameras, the distance between the cameras is separated by a certain distance, and the camera place the

Using a wand for camera calibration, shake a wand with a marker over the entire capture volume to collect a sample that the camera shakes, and calculate a position value using the collected samples. After all calculations are done, you can define the ground plane for the capture volume and set the position value of the calculated value.

Then, after attaching a marker to an appropriate position of the actor wearing the motion suit, moving to the center of the capture volume, instructing the T-correction pose as shown in Fig. 14 (a), ), an initially measured skeleton that is used for actual capture in the program is generated (S1320).

Thereafter, as shown in (a) of FIG. 15, the movement data of the skeleton is recorded (S1330), and the captured movement of the skeleton is transmitted to the optical camera through infrared reflection and can be checked in real time in the electronic device 100 . In addition, through the Motion Builder plug-in, modeling data expressed in Motion Builder and the skeleton in which motion capture is performed can be linked to express the modeling movement of Motion Builder as the movement of the skeleton.

At this time, a data processing process of confirming the marker position and reconstructing the lost data may be performed (S1340), and as shown in FIG. 15 (b), the position value of the marker is set as closely as possible to the movement of the actor. The movement of the actor may be expressed as the movement of the skeleton in which the body joints of the modeling data are converted (S1350).

In the area where the marker is mounted, both the 2D object captured by each camera through movement and the 3D data reconstructed in real time based on the 2D object are saved. ) and made available for use. The 3D data stored in this way is applied to the 3D character (S1360).

16 and 17 are diagrams illustrating a character customization UI and a broadcast transmission UI according to an embodiment of the present invention.

As shown in FIG. 16 , the broadcasting platform may provide a UI including an authoring tool capable of customizing desired characters, motions, subtitles, environments, special effects, etc. before or during broadcasting.

As shown in (a) of FIG. 16, the UI provided before broadcasting may include a character screen area 1610-1, a character customizing area 162-1, and a character modeling area 1630-1, Each region may be implemented as each region 1610-2, 1620-2, and 1630-2 shown in FIG. 16B. In the character modeling area 1630-1, any one character for generating animation among a plurality of characters may be selected, and in the character customization area 162-1, any one of the appearance and motion of the selected character may be customized. .

In addition, as shown in (a) of FIG. 17 , the UI provided during real-time broadcasting includes a background screen selection area 1710-1, a real-time broadcast screen area 172-1, an effect selection area 1730-1 and It may include an authoring tool area 1740-1, and each area may be implemented like each area 1710-2, 1720-2, 1730-2, 1740-2 shown in FIG. 17B. there is.

18 is a diagram illustrating a screen in which a real-time 3D engine is interlocked with a broadcasting platform in real time according to an embodiment of the present invention.

As shown in (a) and (b) of FIG. 18 , a particle light source real-time processing algorithm using an object's material, material, and pixel unit operation can be implemented through a script. By linking a renderer engine such as Unreal in real time, the character you want to use can be directly imported, and real-time customization is possible.

In addition, the UI may be produced as an add-on type UI script. Specifically, a UI script can be produced to be added-on to the transmission platform by using a blueprint in the renderer engine so that the existing broadcast transmission platform and the renderer engine can be linked.

19 is a block diagram schematically illustrating the configuration of an apparatus according to an embodiment of the present invention.

The electronic device 100 according to an embodiment of the present invention includes an input unit 110 , a storage unit 120 , and a processor 130 .

The input unit 110 is configured to receive, in real time, a measured value measured from a motion sensor mounted on the user's body and an image captured by the user. Accordingly, the input unit 110 may include a motion sensor and a communication unit (not shown) for performing communication through a wired/wireless communication method with a camera for photographing a user.

The storage unit 120 is configured to store motion data and rigging data corresponding to the motion capture data. The storage unit 120 is

The processor 130 is a configuration for overall controlling the electronic device 100 .

Specifically, the processor 130 extracts the user's motion capture data and face capture data from the measured values and the captured image, respectively, and based on the stored motion data and rigging data, performs an operation corresponding to the extracted motion capture data. It is possible to provide a broadcasting platform for generating an animation of a preset character representing a facial expression corresponding to the rigging data to which the face capture data has been applied, and transmitting the generated animation in real time.

At this time, the processor 130 extracts coordinate data corresponding to the user's movement through a motion detection sensor implemented as a gyro sensor, converts the extracted coordinate data into body motion data, and sends the converted body motion data to the renderer engine. It is possible to perform correction by comparing it with the simulation data of the present invention, and extract the corrected body motion data as motion capture data.

In addition, the processor 130 may recognize a preset object from the user's face in the image captured by the user, and extract marking data representing the shape of the recognized object as the face capture data.

In this case, when a preset object is recognized, the processor 130 generates a plurality of markers disposed at an interval on at least one of a center line and an outline of the recognized object, and then connects the plurality of generated markers to recognize the recognized object. Marking data representing the shape of an object can be created.

Also, the electronic device 100 according to an embodiment of the present invention may further include a user interface (not shown). In this case, the processor 130 may correct the marker position of the marking data according to a user command input through the user interface.

In addition, the processor 130 applies the extracted face capture data to the rigging data of the preset character, corrects the error of the rigging data with the face capture data, and changes the expression of the character based on the error corrected rigging data You can create animations.

In this case, the processor 130 may select any one character for generating animation among a plurality of characters and provide a UI for customizing any one of the appearance and operation of the selected character through the broadcasting platform.

20 is a block diagram schematically illustrating the configuration of a system according to an embodiment of the present invention.

The broadcast platform providing system 1000 according to an embodiment of the present invention includes an electronic device 100 , a camera 200 , and a gyro sensor 300 . In this case, the electronic device 100 , the camera 200 , and the gyro sensor 300 may be connected through the network 2000 .

The electronic device 100 is configured to provide a real-time broadcasting platform, and may receive face capture data and motion capture data from the camera 200 and the gyro sensor 300, respectively.

In this case, the electronic device 100 may receive the face capture data and the motion capture data through an external device (not shown) connected to the camera 200 and the gyro sensor 300 .

The electronic device 100 extracts motion capture data and face capture data of the user from the received measurement value and the captured image, respectively. Thereafter, the electronic device 100 may generate an animation of a preset character representing a motion corresponding to the extracted motion capture data and representing a facial expression corresponding to the rigging data to which the face capture data is applied.

Thereafter, the electronic device 100 may transmit the animation generated through the provided broadcasting platform in real time.

According to various embodiments of the present invention as described above, it is possible to communicate in real time with a viewer using a character desired by a creator or a character created by the creator, and can be used as a low-cost broadcasting solution.

The broadcast platform providing method according to the above-described various embodiments may be implemented as a program and stored in various recording media. That is, a computer program that is processed by various processors to execute the above-described method for providing a broadcast platform may be used in a state stored in a recording medium. As an example, i) extracting the user's motion capture data and face capture data in real time based on the motion detection sensor mounted on the user's body and the image captured by the user, ii) corresponding to the extracted motion capture data Creating an animation of a preset character representing a motion to be performed and representing a facial expression corresponding to rigging data to which face capture data is applied; and iii) providing a broadcasting platform for transmitting the generated animation in real time. A non-transitory computer readable medium in which a program for performing the above is stored may be provided.

The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, and the like, and can be read by a device. Specifically, the above-described various applications or programs may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

On the other hand, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims In addition, various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

S110: A step of extracting the user's motion capture data and face capture data in real time based on the motion detection sensor mounted on the user's body and the image captured by the user, respectively
S120: Creating an animation of a preset character representing a motion corresponding to the extracted motion capture data and representing a facial expression corresponding to rigging data to which the face capture data is applied
S130: Step of providing a broadcasting platform for transmitting the generated animation in real time
100: electronic device 110: input unit
120: storage unit 130: processor
200: camera 300: gyro sensor
1000: broadcasting platform providing system 2000: network

Claims (20)

In the method of providing a real-time broadcasting platform using motion and face capture,
extracting motion capture data and face capture data of the user in real time based on a motion sensor mounted on the user's body and an image captured by the user, respectively;
generating an animation of a preset character representing a motion corresponding to the extracted motion capture data and representing a facial expression corresponding to rigging data to which the face capture data is applied; and
Including the step of providing a broadcasting platform for transmitting the generated animation in real time,
The extracting step is
extracting coordinate data corresponding to the user's movement from the measured value of the motion sensor;
converting the coordinate data into body motion data if the extracted coordinate data is within a normal range, and performing calibration of the coordinate data if it is not within the normal range;
Comparing the converted body motion data with simulation data by a renderer engine and performing correction so that the converted body motion data is closest to the simulation data; and
and extracting the corrected body motion data as the motion capture data. According to claim 1,
The motion sensor is a gyro sensor,
The extracting step is
Method of extracting the coordinate data based on the value measured through the gyro (gyro) sensor. delete According to claim 1,
The extracting step is
A method characterized in that the user recognizes a preset object from the user's face in the captured image, and extracts marking data representing the shape of the recognized object as the face capture data. 5. The method of claim 4,
The extracting step is
When the preset object is recognized, a plurality of markers are generated at an interval on at least one of a center line and an outline of the recognized object, and then the marking data is generated by connecting the plurality of markers. method. 6. The method of claim 5,
The extracting step is
Method, characterized in that for correcting the marker position of the marking data according to a user input. 5. The method of claim 4,
The generating step is
applying the extracted face capture data to rigging data of a preset character;
correcting an error between the rigging data and the face capture data; and
Based on the error-corrected rigging data, the method comprising the step of generating a change in the expression of the character as the animation. According to claim 1,
Selecting any one character for generating the animation from among a plurality of characters, and providing a UI for customizing any one of the appearance and operation of the selected character through the broadcasting platform; Method further comprising a. 9. The method of claim 8,
The method, characterized in that the UI is produced as an add-on (add-on) type UI script. A real-time broadcasting platform providing device for a method of providing a real-time broadcasting platform using motion and face capture,
an input unit for receiving, in real time, a measured value measured from a motion sensor mounted on the user's body and an image captured by the user; and
Each of the user's motion capture data and face capture data is extracted from the measured value and the captured image, and based on the motion data and rigging data stored in the storage unit, an operation corresponding to the extracted motion capture data is displayed, and the face capture A processor that provides a broadcasting platform for generating an animation of a preset character representing a facial expression corresponding to the rigging data to which the data is applied, and transmitting the generated animation in real time,
The processor is
Coordinate data corresponding to the user's movement is extracted from the measured value, and if the extracted coordinate data is within a normal range, the coordinate data is converted into body motion data. If it is not within the normal range, the coordinate data is calibrated, and the converted Comparing body motion data with simulation data by a renderer engine, performing correction such that the body motion data is closest to the simulation data at maximum, and extracting the corrected body motion data as the motion capture data. 11. The method of claim 10,
The motion sensor is
Device characterized in that it is a gyro (gyro) sensor. delete 11. The method of claim 10,
The processor is
A device characterized in that the user recognizes a preset object from the user's face in the captured image, and extracts marking data representing the shape of the recognized object as the face capture data. 14. The method of claim 13,
The processor is
When the preset object is recognized, a plurality of markers are generated at an interval on at least one of a center line and an outline of the recognized object, and then the marking data is generated by connecting the plurality of markers. Device. 15. The method of claim 14,
User interface; further comprising,
The processor is
Device characterized in that for correcting the marker position of the marking data according to a user command input through the user interface. 14. The method of claim 13,
The processor is
Applying the extracted face capture data to the rigging data of a preset character, correcting the error between the rigging data and the face capturing data, and generating the change in the expression of the character as the animation based on the error corrected rigging data device characterized. 11. The method of claim 10,
The processor is
Device, characterized in that selecting any one character for generating the animation from among a plurality of characters, and providing a UI for customizing any one of the appearance and operation of the selected character through the broadcasting platform. 18. The method of claim 17,
The UI is
A device, characterized in that it is produced as an add-on type UI script. In the real-time broadcasting platform providing system using motion and face capture,
a camera to photograph the user;
a motion sensor mounted on the user's body; and
Each of the measured values measured from the motion sensor and the captured images from the camera are received in real time, and the user's motion capture data and face capture data are extracted from the measured values and the captured images, respectively, and the extracted motion capture Electronic that provides a broadcasting platform for generating an animation of a preset character representing a motion corresponding to data, representing a facial expression corresponding to rigging data to which the face capture data is applied, and transmitting the animation in real time device, including
The electronic device is
Coordinate data corresponding to the user's movement is extracted from the measured value, and if the extracted coordinate data is within a normal range, the coordinate data is converted into body motion data. If it is not within the normal range, the coordinate data is calibrated, and the converted Comparing body motion data with simulation data by a renderer engine, performing correction such that the body motion data is closest to the simulation data at maximum, and extracting the corrected body motion data as the motion capture data. A computer program stored in a recording medium for executing the method of providing a real-time broadcasting platform using the motion and face capture of any one of claims 1, 2, and 4 to 9.

Images