WO2021251534A1 - Procédé, appareil et système de fourniture de plate-forme de diffusion en temps réel à l'aide d'une capture de mouvement et de visage - Google Patents

Procédé, appareil et système de fourniture de plate-forme de diffusion en temps réel à l'aide d'une capture de mouvement et de visage Download PDF

Info

Publication number
WO2021251534A1
WO2021251534A1 PCT/KR2020/007728 KR2020007728W WO2021251534A1 WO 2021251534 A1 WO2021251534 A1 WO 2021251534A1 KR 2020007728 W KR2020007728 W KR 2020007728W WO 2021251534 A1 WO2021251534 A1 WO 2021251534A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
motion
user
face
capture data
Prior art date
Application number
PCT/KR2020/007728
Other languages
English (en)
Korean (ko)
Inventor
김태원
김진식
Original Assignee
주식회사 이엠피이모션캡쳐
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 주식회사 이엠피이모션캡쳐 filed Critical 주식회사 이엠피이모션캡쳐
Publication of WO2021251534A1 publication Critical patent/WO2021251534A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/251—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving models
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present invention relates to a method, apparatus and system for providing a real-time broadcasting platform using motion and face capture, and more particularly, to a method for providing a real-time broadcasting platform using motion and face capture to control a virtual character in real time; It relates to a device and its system.
  • MCN Multi Channel Network
  • creators are also using various methods to attract viewers' attention and induce influx.
  • the most representative method is to use a camera to film the creator, and the creator to respond immediately to the chatting of the viewer to induce a response from the viewer. This is a method made possible by the Internet broadcasting platform.
  • An object of the present invention is to provide a method, an apparatus, and a system for providing a real-time broadcasting platform using motion and face capture capable of realizing a virtual character showing the shape and movement of a real person in order to solve the above problems. do it with
  • a method for providing a real-time broadcasting platform using motion and face capture is based on a motion detection sensor mounted on a user's body and an image captured by the user, respectively, the user's motion capture data and face capture Extracting data in real time, generating an animation of a preset character representing a motion corresponding to the extracted motion capture data and representing a facial expression corresponding to rigging data to which the face capture data is applied; and providing a broadcasting platform for transmitting the generated animation in real time.
  • the extracting may extract the motion capture data based on a value measured through a gyro sensor attached to the user's body.
  • the extracting includes extracting coordinate data corresponding to the user's movement through the gyro sensor, converting the extracted coordinate data into body motion data, and transferring the converted body motion data to a renderer engine. It may include performing correction by comparing it with the simulation data of the present invention, and extracting the corrected body motion data as the motion capture data.
  • the extracting may include recognizing a preset object from the user's face in the image captured by the user, and extracting marking data representing the shape of the recognized object as the face capture data.
  • the extracting step when the preset object is recognized, a plurality of markers disposed at an interval on at least one of a center line and an outline of the recognized object are generated, and then the plurality of generated markers are connected. It is possible to generate the marking data representing the shape of the recognized object.
  • the extracting may correct the position of the marker of the marking data according to a user input.
  • the generating includes applying the extracted face capture data to the rigging data of the preset character, correcting an error of the rigging data with the face capturing data, and rigging data in which the error is corrected Based on the , it may include the step of generating the change in the expression of the character as an animation.
  • the method further includes the step of selecting any one character for generating the animation from among a plurality of characters and providing a UI for customizing any one of the appearance and operation of the selected character through the broadcasting platform can do.
  • the UI may be produced as an add-on type UI script.
  • the apparatus for providing a real-time broadcasting platform using motion and face capture receives the measured value measured from the motion sensor mounted on the user's body and the image captured by the user in real time, respectively.
  • An input unit a storage unit for storing motion data and rigging data corresponding to the motion capture data, and extracting the user's motion capture data and face capture data from the measured value and the captured image, respectively, and the stored
  • an animation of a preset character representing a motion corresponding to the extracted motion capture data and representing a facial expression corresponding to the rigging data to which the face capture data is applied is generated, and the generated animation
  • It includes a processor that provides a broadcasting platform for transmitting in real time.
  • the motion detection sensor may be a gyro sensor.
  • the processor extracts coordinate data corresponding to the user's movement through the gyro sensor, converts the extracted coordinate data into body motion data, and compares the converted body motion data with simulation data by a renderer engine to perform the correction, and the corrected body motion data may be extracted as the motion capture data.
  • the processor may recognize a preset object from the user's face in the image captured by the user, and extract marking data representing the shape of the recognized object as the face capture data.
  • the processor when the preset object is recognized, the processor generates a plurality of markers disposed at an interval on at least one of a center line and an outline of the recognized object, and then connects the generated markers to recognize the plurality of markers. It is possible to generate the marking data indicating the shape of the object.
  • the device may further include a user interface, and the processor may correct the marker position of the marking data according to a user input input through the user interface.
  • the processor applies the extracted face capture data to the rigging data of the preset character, corrects an error of the rigging data with the face capture data, and based on the rigging data for which the error is corrected, the You can create an animation of a character's facial expression change.
  • the processor may select any one character for generating the animation from among a plurality of characters and provide a UI for customizing any one of an appearance and operation of the selected character through the broadcasting platform.
  • the UI may be produced as an add-on type UI script.
  • the system for providing a real-time broadcasting platform using motion and face capture includes a camera for photographing a user, a motion detection sensor mounted on the user's body, and a measurement value measured from the motion detection sensor and receiving the images photographed from the camera in real time, extracting motion capture data and face capture data of the user from the received measurement values and the photographed images, respectively, and corresponding to the extracted motion capture data and an electronic device that provides a broadcasting platform for generating an animation of a preset character representing a facial expression corresponding to rigging data to which the face capture data is applied, and transmitting the generated animation in real time. do.
  • the creator by performing motion capture of a creator in real time and controlling a character created in advance with such motion capture data, the creator uses the desired character or the created character to communicate in real time with the viewer It can be used as a low-cost broadcasting solution.
  • FIG. 1 is a flowchart for briefly explaining a method of providing a real-time broadcasting platform using motion and face capture according to an embodiment of the present invention
  • FIG. 2 is a flowchart for briefly explaining a data processing method for providing a real-time broadcasting platform according to an embodiment of the present invention
  • FIG. 3 is a flowchart for generating face capture data and animation data of a character according to an embodiment of the present invention
  • 4 and 5 are screens for explaining a process of extracting face capture data according to an embodiment of the present invention.
  • FIG. 6 is a flowchart illustrating a method of generating animation data by applying face capture data to rigging data according to an embodiment of the present invention
  • FIG. 7 is a flowchart for explaining a process of extracting motion capture data by a neuron sensor according to an embodiment of the present invention
  • FIGS. 8 to 11 are views showing various examples of a body range in which a neuron sensor is worn according to an embodiment of the present invention.
  • FIGS. 12 to 13 are diagrams illustrating a state in which motion capture data is generated by a neuron sensor according to an embodiment of the present invention
  • FIG. 14 is a flowchart for explaining a process of correcting motion captured data by a neuron sensor according to an embodiment of the present invention
  • FIG. 15 is a diagram for explaining a motion capture data extraction optimization algorithm according to an embodiment of the present invention.
  • FIG. 16 is a view showing a motion capture result screen by a gyro sensor according to an embodiment of the present invention.
  • 17 is a flowchart for explaining a method of extracting motion capture data by an optical sensor according to an embodiment of the present invention.
  • 18 to 21 are views showing a state in which motion capture data is generated by an optical sensor according to an embodiment of the present invention.
  • 22 to 25 are views showing a character customization UI and a broadcast transmission UI according to an embodiment of the present invention.
  • 26 to 27 are diagrams showing a screen in which a real-time 3D engine is interlocked with a broadcasting platform in real time according to an embodiment of the present invention
  • FIG. 28 is a block diagram schematically illustrating the configuration of an apparatus according to an embodiment of the present invention.
  • 29 is a block diagram schematically illustrating the configuration of a system according to an embodiment of the present invention.
  • ordinal numbers such as 'first' and 'second' may be used to distinguish between elements. This ordinal number is used to distinguish the same or similar components from each other, and the meaning of the term should not be limitedly interpreted due to the use of the ordinal number. As an example, for components combined with such an ordinal number, the order of use or arrangement of the elements should not be construed as being limited by the number. If necessary, each ordinal number may be used interchangeably.
  • a part when it is said that a part is connected to another part, this includes not only direct connection but also indirect connection through another medium.
  • the meaning that a certain part includes a certain component means that other components may be further included, rather than excluding other components, unless otherwise stated.
  • FIG. 1 is a flowchart for briefly explaining a method for providing a real-time broadcasting platform using motion and face capture according to an embodiment of the present invention.
  • the user's motion capture data and face capture data can be extracted in real time (S110).
  • the motion capture data is extracted based on a value measured through a motion sensor mounted on the user's body.
  • the motion detection sensor may be implemented as a gyro sensor.
  • the gyro sensor may be implemented as an IMU (Inertial Measurement Unit) including a 3-axis gyroscope, a 3-axis accelerometer, and a 3-axis magnetometer, and is attached to the user's body by a strap. and can be tightened.
  • IMU Inertial Measurement Unit
  • the IMU is also commonly referred to as a neuron sensor.
  • the hub may transmit this data to a device providing a broadcasting platform through wired/wireless communication such as Wi-Fi or USB.
  • Coordinate data corresponding to the user's movement may be extracted through the gyro sensor mounted on the user's body, and the extracted coordinate data may be converted into body motion data.
  • correction may be performed by comparing the converted body motion data with simulation data by the renderer engine, and the body motion data on which the correction has been performed may be extracted as motion capture data.
  • the user may recognize a preset object from the user's face in the captured image, and may extract marking data representing the shape of the recognized object as the face capture data.
  • the marking data may be data representing the shape of the recognized object by connecting a plurality of markers disposed at an interval to at least one of a center line and an outline of the recognized object.
  • the extracted face capture data may be applied to rigging data of a preset character, and an error between the rigging data and the face capture data may be corrected.
  • the change in the expression of the character may be generated as an animation.
  • a broadcasting platform for transmitting the generated animation in real time is provided (S130).
  • any one of the plurality of characters for generating animation may be selected, and a UI for customizing any one of the appearance and operation of the selected character may be provided through the broadcasting platform.
  • FIG. 2 is a flowchart for briefly explaining a data processing method for providing a real-time broadcasting platform according to an embodiment of the present invention.
  • the real-time renderer engine When the content is planned (S210), based on 3D character modeling, through a real-time renderer engine, it is possible to implement modeled data close to the real thing (S220).
  • the real-time renderer engine may be implemented as, for example, an unreal engine.
  • a range for human movement is designated (S230), and motion capture data and face capture data are extracted through motion/face capture (S240).
  • the motion capture data can be extracted by converting actual human motion into data through gyro-type motion capture
  • the face capture data can be extracted by capturing the motion of each part of the face through an image taken without a marker.
  • a 3D animation representing a motion corresponding to the extracted motion capture data and representing a facial expression corresponding to the rigging data to which the face capture data is applied can be generated (S250), and a lively animation character is applied by applying it to a preset character (S260) can be implemented.
  • a software for providing a UI script capable of customizing the character implemented in steps S210 to S260 to a real-time image transmission platform (real-time broadcasting platform) is planned (S270), and the planned It is possible to design a UI for specifying selection, customization, and effects of a character provided by the software (S280).
  • animation data of the character implemented in step S260 and programming data implemented in step S290 may be transmitted to the real-time image transmission platform 10 .
  • FIG. 3 is a flowchart for generating face capture data and animation data of a character according to an embodiment of the present invention.
  • the markerless algorithm shown in Fig. 3 in generating the face capture data for the facial expression performance of the customized character, unlike the existing face capturing method in which a marker is directly captured on the face and recognized, the face capture data without a marker , and based on this, animation data representing the movement of the rigging data can be generated.
  • face data is photographed through the camera (S310), and a position for each face part is set according to a markerless algorithm (S320).
  • the tracked position data is added to the rigging data (S350), and the error due to the error and noise of each position for each face part is corrected (S360). At this time, it is determined whether the difference between the value of each position for each actual face part and the corrected value is equal to or greater than a preset threshold value (S370), and if it is equal to or greater than the threshold value, it is determined that the output is abnormal (S370:N) and step S360 redo If it is less than the threshold, it is determined that the output is normal (S370:Y), and a data optimization process is performed (S380), and finally optimized animation data can be generated (S390).
  • S370 preset threshold value
  • S380 data optimization process
  • 4 and 5 are screens for explaining a process of extracting face capture data according to an embodiment of the present invention.
  • objects such as eyes, nose, and mouth, which are main points for representing facial expressions in a user's captured image, may be automatically recognized.
  • the created markers are connected to generate the marking data 41 representing the shape of each object.
  • the automatically recognized marker of each object may have an error with the center line or outline of the actual part, it is possible for the user to fine-tune the position of the marker directly according to a user input.
  • FIG. 6 is a flowchart illustrating a method of generating animation data by applying face capture data to rigging data according to an embodiment of the present invention.
  • the markers 61-1, 61-2, and 61-3 that are the standards for performing the customized character's facial expression are selected, and the tracked marker data is converted into the character. It can be applied to rigging data. Accordingly, the points 62-1, 62-2, 62-3 corresponding to the selected markers 61-1, 61-2, and 61-3 in the rigging data are the markers 61-1, 61-2, 61 It can be synchronized with the movement of -3).
  • FIG. 7 is a flowchart illustrating a process of extracting motion capture data by a gyro sensor according to an embodiment of the present invention.
  • motion can be captured by using a gyro sensor without a marker, instead of the conventional most widely used method of capturing motion by attaching a marker to the body and photographing it.
  • the gyroscope and the hub may be set up (S710), and calibration may be performed on the detected motion capture data (S720). At this time, it is determined whether the output of the motion capture data is within the normal range (S730), and if it is within the normal range (S730:Y), the coordinates of the output motion capture data are stored (S740). If it is not within the normal range (S730:N), calibration may be performed again. Whether the output of the motion capture data is within a normal range may be determined according to whether the difference between the output value and a preset threshold value is greater than or equal to a preset value by comparing the output value.
  • the coordinate values of the stored data are converted into body motion data (S750), and errors due to noise and errors of the converted body motion data with actual motions are corrected (S760). It is determined whether the output of the corrected body motion data is within the normal range (S770), and if it is within the normal range (S770:Y), data optimization is performed (S780), and final body motion data is output (S790).
  • step S760 When the output of the corrected body motion data is not within the normal range (S770:N), the correction in step S760 may be re-performed.
  • Whether the output of the body motion data is within a normal range may be determined according to whether the difference between the output value and a preset threshold value is greater than or equal to a preset value by comparing the output value.
  • FIGS. 8 to 11 are diagrams illustrating various examples of a body range in which a neuron sensor is worn according to an embodiment of the present invention.
  • the neuron sensor checks the movement in real time through an IMU (Inertial Measurement Unit) with a built-in 3-axis gyroscope, 3-axis accelerometer, and 3-axis magnetometer. After being calculated, the 9 axis sensor units are transmitted at a speed of 60 fs to a hub that is mounted together on the body of the user (actor). The data transmitted to the hub may be finally delivered to the electronic device 100 that outputs the motion capture data through Wi-Fi or a USB cable.
  • IMU Inertial Measurement Unit
  • the number of sensors mounted on the body may be set as (a) to (d) according to the range for capturing detailed movements.
  • FIGS. 12 to 13 are diagrams illustrating a state in which motion capture data is generated by a gyro sensor according to an embodiment of the present invention.
  • modeling as shown in FIG. 9A may be performed.
  • the body size of the user wearing the neurons may be selected and applied, and calibration and posture correction may be performed.
  • Calibration may be performed to prevent cross-talk of data from external noise and to accurately extract the user's movement.
  • the calibration may not be accurate. In this case, the accuracy of the motion capture can be maintained through rework of the calibration.
  • motion data 122 corresponding to the motion capture data is generated, and a preset character 1230 is generated according to the movement of the motion data 122 .
  • the movement can be performed to correspond.
  • FIG. 14 is a flowchart illustrating a process of correcting motion captured data by a gyro sensor according to an embodiment of the present invention.
  • body motion data and simulation data are respectively output (S1130-1 and S1130-2), the output body motion data and simulation data are compared with each other, and correction is performed so that the body motion data is closest to the simulation data. do.
  • the correction process may be performed by correcting an error of a position value, an error caused by external noise, an error caused by a violent movement, and the like.
  • the body motion data corrected in this way is output as final body motion data for expressing the movement of the character (S1040). And, the above comparison operation and correction operation may be scripted.
  • 15 is a diagram for explaining a motion capture data extraction optimization algorithm according to an embodiment of the present invention.
  • the extracted data of the gyro-type motion capture method to be used in the present invention is based on BVH. Since BVH data is bone data that contains position information on the X, Y and Z axes including motion position and rotation values, the position and rotation values of the rigging data are placed on the character bone to be used. When converges to (0,0,0), the exact position value can be extracted.
  • a conversion process 1120 for correcting X, Y and Z values to (0,0,0) is added in the middle of the process of outputting neuron data and outputting bone data (1130) in real time. This allows the position value of the data to be output normally, thereby reducing the delay until the final data output.
  • 17 is a flowchart for explaining a method of extracting motion capture data by an optical sensor according to an embodiment of the present invention.
  • the motion capture data of the present invention may be extracted by an optical sensor.
  • the optical motion capture method may be performed with an optical camera, a switching hub, a router, a motion suit with a marker attached thereto, and equipment including a marker.
  • the optical camera is a configuration that transmits the retro-reflected position data value to the switching hub by projecting the actor (user) infrared rays through a high-resolution camera with an infrared lens.
  • the switching hub is connected to the optical camera with an Ethernet cable, supplies power, receives captured data, and delivers it to the router.
  • the router is configured to transmit data received from each switching hub to the electronic device 100 for outputting the final motion capture data.
  • the electronic device 100 is configured to process the camera position value transmitted from the router in real time through a dedicated program, and output motion capture data.
  • a motion suit is a suit to which a marker can be attached to express the movement of an actor, and the marker is attached to each joint part of the actor wearing the motion suit and expresses the position value of the body by infrared reflection.
  • the marker coated with a reflective material reflects the infrared and detects the returned data, and through this, motion capture based on the actor's position and angle data can be saved.
  • the three-dimensional coordinates of the marker are calculated by a trigonometric function to extract the position value, and the position value is set at 120 fps. It is transmitted to the switching hub at a higher speed. The transmitted data is finally delivered to the electronic device 100 via Ethernet through the router.
  • the first form of adding foot positions (4 places) to the basic body joint parts (37 places) when capturing whole body movements, or markers for thighs, arms, chest, and head (13 places) in the basic body joint parts (37 places) ) can be added to capture the actor's motion.
  • the sensor correction is performed to organize the surrounding environment by setting the camera position and angle ( S1310 ). Remove any shiny objects or unnecessary obstructions that may block the camera view, and use a non-reflective mat on the floor to minimize reflection distortion.
  • the cameras can be arranged.
  • a wand for camera calibration shake a wand with a marker over the entire capture volume to collect a sample that the camera shakes, and calculate a position value using the collected samples. After all calculations are done, you can define the ground plane for the capture volume and set the position value of the calculated value.
  • the movement data of the skeleton is recorded ( S1330 ), and the captured movement of the skeleton is transmitted to the optical camera through infrared reflection so that the electronic device 100 can check it in real time.
  • modeling data expressed in Motion Builder and the skeleton in which motion capture is performed can be linked to express the modeling movement of Motion Builder as the movement of the skeleton.
  • the data processing process of confirming the marker position and reconstructing the lost data may be performed (S1340), and as shown in FIG. 21, the position value of the marker is set as closely as possible to the movement of the actor, so that the movement of the actor is
  • the body joints of the modeling data may be expressed as the movement of the converted skeleton (S1350).
  • both the 2D object captured by each camera through movement and the 3D data reconstructed in real time based on the 2D object are saved. ) and made available for use.
  • the 3D data stored in this way is applied to the 3D character (S1360).
  • 22 to 25 are diagrams illustrating a character customization UI and a broadcast transmission UI according to an embodiment of the present invention.
  • the broadcasting platform may provide a UI including an authoring tool capable of customizing desired characters, motions, subtitles, environments, special effects, etc. before or during broadcasting.
  • the UI provided before broadcasting may include a character screen area 1610-1, a character customizing area 162-1, and a character modeling area 1630-1, and each area is shown in FIG.
  • Each region 1610-2, 1620-2, and 1630-2 shown in 16b may be implemented.
  • the character modeling area 1630-1 any one character for generating animation among a plurality of characters may be selected, and in the character customization area 162-1, any one of the appearance and motion of the selected character may be customized. .
  • the UI provided during real-time broadcasting includes a background screen selection area 1710-1, a real-time broadcast screen area 172-1, an effect selection area 1730-1, and an authoring tool area ( 1740-1), and each region may be implemented like each of the regions 1710-2, 1720-2, 1730-2, and 1740-2 shown in FIG. 24 .
  • 26 to 27 are diagrams illustrating screens in which a real-time 3D engine is interlocked with a broadcasting platform in real time according to an embodiment of the present invention.
  • a particle light source real-time processing algorithm using an object's material, material, and pixel unit operation may be implemented through a script.
  • a renderer engine such as Unreal in real time, the character you want to use can be directly imported, and real-time customization is possible.
  • the UI may be produced as an add-on type UI script.
  • a UI script can be produced to be added-on to the transmission platform by using a blueprint in the renderer engine so that the existing broadcast transmission platform and the renderer engine can be linked.
  • FIG. 28 is a block diagram schematically illustrating the configuration of an apparatus according to an embodiment of the present invention.
  • the electronic device 100 includes an input unit 110 , a storage unit 120 , and a processor 130 .
  • the input unit 110 is configured to receive, in real time, a measured value measured from a motion sensor mounted on the user's body and an image captured by the user. Accordingly, the input unit 110 may include a motion sensor and a communication unit (not shown) for performing communication through a wired/wireless communication method with a camera for photographing a user.
  • the storage unit 120 is configured to store motion data and rigging data corresponding to the motion capture data.
  • the storage unit 120 is
  • the processor 130 is a configuration for overall controlling the electronic device 100 .
  • the processor 130 extracts the user's motion capture data and face capture data from the measured values and the captured image, respectively, and based on the stored motion data and rigging data, performs an operation corresponding to the extracted motion capture data. It is possible to provide a broadcasting platform for generating an animation of a preset character representing a facial expression corresponding to the rigging data to which the face capture data has been applied, and transmitting the generated animation in real time.
  • the processor 130 extracts coordinate data corresponding to the user's movement through a motion detection sensor implemented as a gyro sensor, converts the extracted coordinate data into body motion data, and sends the converted body motion data to the renderer engine. It is possible to perform correction by comparing it with the simulation data of the present invention, and extract the corrected body motion data as motion capture data.
  • the processor 130 may recognize a preset object from the user's face in the image captured by the user, and extract marking data representing the shape of the recognized object as the face capture data.
  • the processor 130 when a preset object is recognized, the processor 130 generates a plurality of markers disposed at an interval on at least one of a center line and an outline of the recognized object, and then connects the plurality of generated markers to recognize the recognized object. Marking data representing the shape of an object can be created.
  • the electronic device 100 may further include a user interface (not shown).
  • the processor 130 may correct the marker position of the marking data according to a user command input through the user interface.
  • the processor 130 applies the extracted face capture data to the rigging data of the preset character, corrects the error with the face capture data of the rigging data, and changes the expression of the character based on the error corrected rigging data You can create animations.
  • the processor 130 may select any one character for generating animation among a plurality of characters and provide a UI for customizing any one of the appearance and operation of the selected character through the broadcasting platform.
  • 29 is a block diagram schematically illustrating the configuration of a system according to an embodiment of the present invention.
  • the broadcast platform providing system 1000 includes an electronic device 100 , a camera 200 , and a gyro sensor 300 .
  • the electronic device 100 , the camera 200 , and the gyro sensor 300 may be connected through the network 2000 .
  • the electronic device 100 is configured to provide a real-time broadcasting platform, and may receive face capture data and motion capture data from the camera 200 and the gyro sensor 300, respectively.
  • the electronic device 100 may receive the face capture data and the motion capture data through an external device (not shown) connected to the camera 200 and the gyro sensor 300 .
  • the electronic device 100 extracts motion capture data and face capture data of the user from the received measurement value and the captured image, respectively. Thereafter, the electronic device 100 may generate an animation of a preset character representing a motion corresponding to the extracted motion capture data and representing a facial expression corresponding to the rigging data to which the face capture data is applied.
  • the electronic device 100 may transmit the animation generated through the provided broadcasting platform in real time.
  • the broadcast platform providing method may be implemented as a program and stored in various recording media. That is, a computer program that is processed by various processors to execute the above-described method for providing a broadcasting platform may be used in a state stored in a recording medium.
  • a non-transitory computer readable medium in which a program for performing the above is stored may be provided.
  • the non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, and the like, and can be read by a device.
  • a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Security & Cryptography (AREA)
  • General Engineering & Computer Science (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un procédé permettant de fournir une plate-forme de diffusion en temps réel en utilisant la capture de mouvement et de visage. Selon la présente invention, le procédé de fourniture de la plate-forme de diffusion en temps réel comprend les étapes consistant à : extraire, en temps réel, des données de capture de mouvement de l'utilisateur et des données de capture de visage sur la base d'un capteur de détection de mouvement monté sur le corps de l'utilisateur et d'une image capturée par l'utilisateur, respectivement ; générer des animations d'un caractère prédéfini qui représentent un mouvement correspondant aux données de capture de mouvement extraites et représentent une expression faciale correspondant à des données de création d'armature auxquelles les données de capture de visage sont appliquées ; et fournir une plate-forme de diffusion pour transmettre les animations générées en temps réel.
PCT/KR2020/007728 2020-06-10 2020-06-15 Procédé, appareil et système de fourniture de plate-forme de diffusion en temps réel à l'aide d'une capture de mouvement et de visage WO2021251534A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2020-0070557 2020-06-10
KR1020200070557A KR102371072B1 (ko) 2020-06-10 2020-06-10 모션 및 얼굴 캡쳐를 이용한 실시간 방송플랫폼 제공 방법, 장치 및 그 시스템

Publications (1)

Publication Number Publication Date
WO2021251534A1 true WO2021251534A1 (fr) 2021-12-16

Family

ID=78846171

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2020/007728 WO2021251534A1 (fr) 2020-06-10 2020-06-15 Procédé, appareil et système de fourniture de plate-forme de diffusion en temps réel à l'aide d'une capture de mouvement et de visage

Country Status (2)

Country Link
KR (1) KR102371072B1 (fr)
WO (1) WO2021251534A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114638918A (zh) * 2022-01-26 2022-06-17 武汉艺画开天文化传播有限公司 一种实时表演捕捉虚拟直播与录制系统

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102446351B1 (ko) 2021-01-19 2022-09-22 주식회사 엘지에너지솔루션 전극 조립체, 배터리 및 이를 포함하는 배터리 팩 및 자동차

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20060109044A (ko) * 2005-04-15 2006-10-19 엘지전자 주식회사 특징점 알고리즘을 이용한 3차원 얼굴 표정 애니메이션생성방법
KR20120059994A (ko) * 2010-12-01 2012-06-11 삼성전자주식회사 표정 제어점을 이용한 아바타 제어 장치 및 방법
KR20130080442A (ko) * 2010-06-09 2013-07-12 마이크로소프트 코포레이션 표정의 실시간 애니메이션화
KR101681096B1 (ko) * 2010-07-13 2016-12-01 삼성전자주식회사 얼굴 애니메이션 시스템 및 방법
KR20200028830A (ko) * 2018-09-07 2020-03-17 (주)위지윅스튜디오 실시간 cg 영상 방송 서비스 시스템

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20060109044A (ko) * 2005-04-15 2006-10-19 엘지전자 주식회사 특징점 알고리즘을 이용한 3차원 얼굴 표정 애니메이션생성방법
KR20130080442A (ko) * 2010-06-09 2013-07-12 마이크로소프트 코포레이션 표정의 실시간 애니메이션화
KR101681096B1 (ko) * 2010-07-13 2016-12-01 삼성전자주식회사 얼굴 애니메이션 시스템 및 방법
KR20120059994A (ko) * 2010-12-01 2012-06-11 삼성전자주식회사 표정 제어점을 이용한 아바타 제어 장치 및 방법
KR20200028830A (ko) * 2018-09-07 2020-03-17 (주)위지윅스튜디오 실시간 cg 영상 방송 서비스 시스템

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114638918A (zh) * 2022-01-26 2022-06-17 武汉艺画开天文化传播有限公司 一种实时表演捕捉虚拟直播与录制系统
CN114638918B (zh) * 2022-01-26 2023-03-28 武汉艺画开天文化传播有限公司 一种实时表演捕捉虚拟直播与录制系统

Also Published As

Publication number Publication date
KR20210153472A (ko) 2021-12-17
KR102371072B1 (ko) 2022-03-10

Similar Documents

Publication Publication Date Title
US11380068B2 (en) Technique for recording augmented reality data
WO2020171540A1 (fr) Dispositif électronique permettant de fournir un mode de prise de vue sur la base d'un personnage virtuel et son procédé de fonctionnement
WO2019164266A1 (fr) Dispositif électronique permettant de produire une image contenant un avatar 3d reflétant le mouvement du visage grâce à un avatar 3d correspondant au visage et procédé de fonctionnement de celui-ci
WO2017010695A1 (fr) Appareil de génération de contenu tridimensionnel et procédé de génération de contenu tridimensionnel associé
WO2020162673A1 (fr) Dispositif électronique pour générer une animation d'avatar et procédé associé
EP2629267A2 (fr) Composition en temps réel de flux multimédia en direct et en mode graphique informatique
WO2021251534A1 (fr) Procédé, appareil et système de fourniture de plate-forme de diffusion en temps réel à l'aide d'une capture de mouvement et de visage
CN103578135A (zh) 虚拟影像与真实场景相结合的舞台交互集成系统及实现方法
WO2020171541A1 (fr) Dispositif électronique et procédé de mise en place d'une interface d'utilisateur pour l'édition de frimousses pendant l'interfonctionnement avec une fonction de caméra en utilisant ledit dispositif électronique
WO2020054978A1 (fr) Dispositif et procédé de génération d'image
CN203630822U (zh) 虚拟影像与真实场景相结合的舞台交互集成系统
US20220053179A1 (en) Information processing apparatus, information processing method, and program
WO2021221341A1 (fr) Dispositif de réalité augmentée et son procédé de commande
US11231745B1 (en) Wearable article with conduits for a performance capture system
WO2021086018A1 (fr) Procédé d'affichage de réalité augmentée tridimensionnelle
WO2020153785A1 (fr) Dispositif électronique et procédé pour fournir un objet graphique correspondant à des informations d'émotion en utilisant celui-ci
WO2023075508A1 (fr) Dispositif électronique et procédé de commande associé
WO2019054790A1 (fr) Procédé de traitement de contenu, et dispositif électronique pour sa prise en charge
JP7065708B2 (ja) 録画再生装置及びプログラム
WO2020122513A1 (fr) Procédé de traitement d'image bidimensionnelle et dispositif d'exécution dudit procédé
WO2024043527A1 (fr) Procédé de commande de dispositif de réalité virtuelle, et dispositif de réalité virtuelle l'utilisant
WO2024053895A1 (fr) Dispositif de réalité augmentée pour fournir un service de réalité augmentée en vue de commander un objet dans un espace réel, et son procédé de fonctionnement
WO2023224250A1 (fr) Appareil électronique et son procédé de commande
WO2024063343A1 (fr) Procédé et système de capture de mouvement 3d
WO2023113471A1 (fr) Dispositif électronique et procédé d'acquisition de données de squelette tridimensionnel d'un sujet photographié à l'aide d'une pluralité de dispositifs de prise de vues

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20939512

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 12.05.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 20939512

Country of ref document: EP

Kind code of ref document: A1