CN113822970A - Live broadcast control method and device, storage medium and electronic equipment - Google Patents

Live broadcast control method and device, storage medium and electronic equipment Download PDF

Info

Publication number
CN113822970A
CN113822970A CN202111114506.1A CN202111114506A CN113822970A CN 113822970 A CN113822970 A CN 113822970A CN 202111114506 A CN202111114506 A CN 202111114506A CN 113822970 A CN113822970 A CN 113822970A
Authority
CN
China
Prior art keywords
scene
avatar
data
live
actor
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111114506.1A
Other languages
Chinese (zh)
Inventor
王毅
钱骏
尹浩然
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Boguan Information Technology Co Ltd
Original Assignee
Guangzhou Boguan Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Boguan Information Technology Co Ltd filed Critical Guangzhou Boguan Information Technology Co Ltd
Priority to CN202111114506.1A priority Critical patent/CN113822970A/en
Publication of CN113822970A publication Critical patent/CN113822970A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)

Abstract

The disclosure provides a live broadcast control method and device, a storage medium and electronic equipment, and relates to the technical field of live broadcast. The method comprises the following steps: creating an avatar according to the acquired model data of the avatar; associating collected performance data of a moving actor in a real scene with the avatar to drive the avatar based on the performance data of the moving actor; according to the scene mapping rules of the real scene and the live scene, determining a target position matched with the position of the moving capture actor in the real scene in the live scene, and synthesizing the virtual image to the live scene according to the target position to generate a live video stream; and sending the live video stream containing the virtual image to a live client for playing. According to the method and the device, the virtual image is driven by adopting performance data of the moving capture actor, and the virtual image is synthesized to the live broadcast scene, so that the interactivity of a live broadcast mode can be improved.

Description

Live broadcast control method and device, storage medium and electronic equipment
Technical Field
The present disclosure relates to the field of live broadcast technologies, and in particular, to a live broadcast control method and apparatus, a storage medium, and an electronic device.
Background
The development of internet technology makes the network live broadcast become one of important interactive modes, and the live broadcast is widely popular with the public due to the advantages of strong real-time performance, strong interactivity, no region limitation and the like. For example, a user may view a video program broadcasted by a main broadcast in real time through a network, or the user may interact with a live broadcast room in real time by sending information and the like.
However, the anchor image displayed in the existing live interface is mostly the real image of the anchor, and the user can only interact with the live broadcasting room where the anchor is located by actions of sending information, gift and the like, and cannot directly interact with the anchor, so that the interactivity of the live broadcasting is not strong.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The present disclosure provides a live broadcast control method, a live broadcast control device, a computer-readable storage medium, and an electronic device, which improve the problem of poor live broadcast interactivity in the prior art.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.
According to a first aspect of the present disclosure, there is provided a live broadcast control method, the method including: creating an avatar according to the acquired model data of the avatar; associating collected performance data of a moving actor in a real scene with the avatar to drive the avatar based on the performance data of the moving actor; according to the scene mapping rules of the real scene and the live scene, determining a target position matched with the position of the moving capture actor in the real scene in the live scene, and synthesizing the virtual image to the live scene according to the target position to generate a live video stream; sending the live video stream containing the virtual image to a live client for playing; wherein a picture of a live scene containing the avatar and a picture of a real scene containing the action actor are kept frame-synchronized.
In an exemplary embodiment of the present disclosure, the model data of the avatar is acquired by: acquiring a three-dimensional role model of the virtual image by using a role model creation tool; importing the three-dimensional role model into a model engine, and analyzing the three-dimensional role model through the model engine to obtain model data of the virtual image; wherein the model data includes model map data and animation control data, the model map data being generated by baking a model frame of the avatar.
In an exemplary embodiment of the present disclosure, the animation control data includes bone binding data and skinning data of the avatar, the performance data includes motion data of the action actor, and the associating the collected performance data of the action actor in a real scene with the avatar to drive the avatar based on the performance data of the action actor includes: acquiring motion data of the moving capture actor through an inertial sensor, wherein the inertial sensor is configured at a motion part of the moving capture actor; and transmitting the action data to the model engine in real time through a first virtual engine plug-in of the model engine, associating the action data with the limb action of the virtual image, and driving the virtual image to execute the action data through the skeleton binding data and the skin data.
In an exemplary embodiment of the present disclosure, the animation control data includes expression control data of the avatar, the performance data includes expression data of the action actor, and the collected performance data of the action actor in a real scene is associated with the avatar to drive the avatar based on the performance data of the action actor, including: connecting the virtual reality plug-in loaded in the model engine with the expression control data by adopting an animation blueprint; acquiring expression data of the moving capture actor through image acquisition equipment comprising a depth camera; and associating the expression data of the moving actor with the facial expression of the virtual image by using a second virtual engine plug-in of the model engine, and driving the virtual image to execute the expression data through the expression control data.
In an exemplary embodiment of the present disclosure, the determining, according to a scene mapping rule of the real scene and a live scene, a target position in the live scene that matches a position of the action actor in the real scene includes: determining the moving distance and/or moving direction of the moving actor in the real scene through the motion data of the moving actor collected by the inertial sensor; determining a distance value of the virtual image in the live broadcast scene corresponding to the moving distance of the moving actor in the real scene according to a distance mapping proportion in the scene mapping rule; and/or determining a direction value of the virtual image in the live broadcast scene corresponding to the moving direction of the moving actor in the real scene according to a direction mapping proportion in the scene mapping rule; and determining the target position according to the distance value and/or the direction value of the virtual image in the live broadcast scene.
In an exemplary embodiment of the present disclosure, when determining a target position matching a position of the action actor in the real scene in the live scene according to a scene mapping rule of the real scene and the live scene, the method further includes: and in the scene mapping rule, searching a position condition met by the position of the moving capture actor in the real scene, and determining the position in the live scene corresponding to the position condition as the target position.
In an exemplary embodiment of the present disclosure, the image of the real scene including the action actor is captured by a camera, and after determining the target position of the avatar in the real scene, the method further includes: and adjusting the shooting frame rate of the camera and the frame rate of the live broadcast scene of the virtual image so as to keep the frame synchronization between the live broadcast scene picture containing the virtual image and the real scene picture shot by the camera.
In an exemplary embodiment of the present disclosure, the synthesizing the avatar to the live scene according to the target position includes: extracting a keying picture of a dynamic capture actor in a real scene through green screen keying; and synthesizing the virtual image based on the image matting picture and the virtual live broadcast engine, and displaying the target position in the live broadcast scene.
In an exemplary embodiment of the present disclosure, after creating the avatar according to the acquired model data of the avatar, the method further includes: receiving modification parameters about the avatar sent by an anchor client, and modifying the avatar according to the modification parameters, wherein the modification parameters comprise parameters for modifying the display style of the avatar and/or event parameters for controlling the avatar to execute action events.
In an exemplary embodiment of the present disclosure, before creating the avatar from the acquired model data of the avatar, the method further includes: receiving screening information sent by a main broadcasting client, and screening the virtual image in a virtual image set according to the screening information; wherein, screening information includes any one or more of the identification data of the avatar, the scene category of the live broadcast scene and the category of the live broadcast room, the identification data of the avatar includes any one or more of the name of the avatar, the category of the avatar and the size information.
According to a second aspect of the present disclosure, there is provided a live broadcast control apparatus including: the creating module is used for creating the virtual image according to the acquired model data of the virtual image; the association module is used for associating the collected performance data of the moving capture actor in the real scene with the virtual image so as to drive the virtual image based on the performance data of the moving capture actor; the synthesis module is used for determining a target position matched with the position of the moving capture actor in the real scene in the live scene according to the scene mapping rules of the real scene and the live scene, synthesizing the virtual image to the live scene according to the target position and generating a live video stream; the sending module is used for sending the live video stream containing the virtual image to a live client for playing; wherein a picture of a live scene containing the avatar and a picture of a real scene containing the action actor are kept frame-synchronized.
In an exemplary embodiment of the present disclosure, the creation module acquires model data of the avatar by performing the following method: the method comprises the steps of obtaining a three-dimensional role model of the virtual image by using a role model creation tool, importing the three-dimensional role model into a model engine, analyzing the three-dimensional role model by the model engine, and obtaining model data of the virtual image, wherein the model data comprises model map data and animation control data, and the model map data is generated by baking a model frame of the virtual image.
In an exemplary embodiment of the disclosure, the animation control data includes bone binding data and skin data of the avatar, the performance data includes motion data of the action actor, the association module is configured to collect the motion data of the action actor through an inertial sensor, the inertial sensor is configured at a motion part of the action actor, transmit the motion data to the model engine through a first virtual engine plug of the model engine in real time, associate the motion data with limb motions of the avatar, and drive the avatar to execute the motion data through the bone binding data and the skin data.
In an exemplary embodiment of the present disclosure, the animation control data includes expression control data of the avatar, the performance data includes expression data of the action actor, and the association module is configured to connect a virtual reality plug-in loaded in the model engine with the expression control data by using an animation blueprint; acquiring expression data of the moving capture actor through image acquisition equipment comprising a depth camera; and associating the expression data of the moving actor with the facial expression of the virtual image by using a second virtual engine plug-in of the model engine, and driving the virtual image to execute the expression data through the expression control data.
In an exemplary embodiment of the disclosure, the synthesis module is configured to determine a moving distance and/or a moving direction of the actor in the real scene through motion data of the actor captured by the inertial sensor, determine a distance value of the avatar in the live scene corresponding to the moving distance of the actor in the real scene according to a distance mapping ratio in the scene mapping rule, and/or determine a direction value of the avatar in the live scene corresponding to the moving direction of the actor in the real scene according to a direction mapping ratio in the scene mapping rule, and determine the target position according to the distance value and/or the direction value of the avatar in the live scene.
In an exemplary embodiment of the disclosure, when a target position matching the position of the actor in the real scene is determined in the live scene according to a scene mapping rule between the real scene and the live scene, the synthesis module is configured to search a position condition that is satisfied by the position of the actor in the real scene in the scene mapping rule, so as to determine the position in the live scene corresponding to the position condition as the target position.
In an exemplary embodiment of the disclosure, the picture of the real scene containing the live actor is obtained by shooting through a camera, and after the target position of the avatar in the real scene is determined, the synthesis module is further configured to adjust a shooting frame rate of the camera and a picture frame rate of the avatar in the live scene, so that the picture of the live scene containing the avatar and the picture of the real scene obtained by shooting through the camera keep frame synchronization.
In an exemplary embodiment of the present disclosure, the synthesis module is further configured to extract a keyed image of an actor in a real scene through green screen keying; and synthesizing the virtual image based on the image matting picture and the virtual live broadcast engine, and displaying the target position in the live broadcast scene.
In an exemplary embodiment of the present disclosure, after creating the avatar according to the acquired model data of the avatar, the composition module is further configured to receive modification parameters sent by the anchor client and related to the avatar, modify the avatar according to the modification parameters, where the modification parameters include parameters for modifying a display style of the avatar and/or event parameters for controlling the avatar to perform an action event.
In an exemplary embodiment of the present disclosure, before creating an avatar according to the obtained model data of the avatar, the creation module is further configured to receive screening information sent by the anchor client, and screen the avatar in the avatar set according to the screening information, wherein the screening information includes any one or more of identification data of the avatar, a scene category of a live broadcast scene, and a category of a live broadcast room, and the identification data of the avatar includes any one or more of a name of the avatar, an avatar category, and size information.
According to a third aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements any of the above-described live control methods.
According to a fourth aspect of the present disclosure, there is provided an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the above-described live control methods via execution of the executable instructions.
The present disclosure has the following beneficial effects:
according to the live broadcast control method, the live broadcast control device, the computer-readable storage medium, and the electronic device in the present exemplary embodiment, an avatar may be created according to acquired model data of the avatar, collected performance data of a live actor in a real scene may be associated with the avatar to drive the avatar based on the performance data of the live actor, then a target position matching a position of the live actor in the real scene may be determined in the live broadcast scene according to a scene mapping rule of the real scene and the live broadcast scene, the avatar may be synthesized to the live broadcast scene according to the target position to generate a live broadcast video stream, and the live broadcast video stream including the avatar may be transmitted to a live broadcast client to be played. On one hand, the method creates the virtual image, associates the performance data of the moving actor with the virtual image, synthesizes the virtual image into a live broadcast scene according to the target position, generates a live broadcast video stream, can combine the virtual image with the live broadcast scene, realizes the transfer of the virtual image from a virtual world to a real world, and improves the interactivity of live broadcast; meanwhile, cross-dimension live broadcast impression can be brought to a user, the live broadcast scene limitation of the virtual image can be broken, the resource cost is reduced, and the virtual image can comprise various images and styles, so that the content richness of a live broadcast room can be improved, and more diversified live broadcast content is brought to audiences. On the other hand, the frame synchronization of the picture of the live scene containing the virtual image and the picture of the real scene containing the moving actor is kept, so that the moving actor can view the picture of the live scene in real time, and convenience is provided for the moving actor to perform and the anchor to adjust the picture content in real time according to the live picture and the moving actor's performance.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is apparent that the drawings in the following description are only some embodiments of the present disclosure, and that other drawings can be obtained from those drawings without inventive effort for a person skilled in the art.
Fig. 1 shows a flowchart of a live control method in the present exemplary embodiment;
FIG. 2 is a flow chart illustrating the processing of action data in the exemplary embodiment;
fig. 3 shows a flowchart of processing of expression data in the present exemplary embodiment;
FIG. 4 illustrates a flow chart for determining a target location in the present exemplary embodiment;
FIG. 5 illustrates a process flow diagram for matting a picture in the present exemplary embodiment;
fig. 6 is a block diagram showing a configuration of a live control apparatus in the present exemplary embodiment;
FIG. 7 illustrates a computer-readable storage medium for implementing the above-described method in the present exemplary embodiment;
fig. 8 shows an electronic device for implementing the above method in the present exemplary embodiment.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
In view of the foregoing various problems, exemplary embodiments of the present disclosure first provide a live control method. The method can be applied to a server side of a live broadcast platform, so that the server side can generate a live broadcast video stream containing the virtual image and send the live broadcast video stream to a live broadcast client side for playing. For example, as an application scenario example of the live broadcast control method in the present exemplary embodiment, the anchor broadcaster may select a desired avatar through an anchor client, and the anchor broadcaster itself serves as a player of the avatar, that is, a live actor performs performance, and transmits performance data of the live actor captured by a camera to a virtual live broadcast engine, so that the virtual live broadcast engine may synthesize a live broadcast picture according to the performance data of the live actor, generate a live broadcast video stream including the avatar, and transmit the live broadcast video stream to the live broadcast client, and viewers may view the live broadcast video through the live broadcast client. In the live broadcast picture, the audience can see the virtual image to execute the same action, expression and the like as the action-capturing actor, but can not see the performance behavior of the action-capturing actor.
Fig. 1 shows a flow of the present exemplary embodiment, which may include the following steps S110 to S140:
and S110, creating the virtual image according to the acquired model data of the virtual image.
The virtual image can comprise a human-shaped virtual image or a plurality of virtual anthropomorphic images, and the virtual image can be a three-dimensional role model which is created in a role model creation tool by a maker in advance; the model data refers to a three-dimensional role model containing an avatar and related data enabling the three-dimensional role model to present animation characteristics, and may include original drawing data, map data, material data, texture data of the three-dimensional role model, expression control data, skeleton binding data, skin data and the like for controlling the motion of the three-dimensional role model.
In the present exemplary embodiment, a three-dimensional character model of an avatar that is uniformly and reasonably wired can be created in advance by a creator using a character model creation tool. The anchor or the broadcasting personnel can log in the live broadcast platform at the anchor client and select the needed virtual image, and during live broadcast, the server of the live broadcast platform can obtain the model data according to the virtual image selected by the anchor or the broadcasting personnel and establish the virtual image during live broadcast according to the model data.
In an alternative embodiment, the model data of the avatar may be obtained by:
acquiring a three-dimensional role model of the virtual image by using a role model creation tool;
and importing the three-dimensional role model into a model engine, and analyzing the three-dimensional role model through the model engine to obtain model data of the virtual image.
In this exemplary embodiment, the three-dimensional character model may include three-dimensional model data of the avatar, animation control data of the avatar, and the like, and accordingly, after the three-dimensional character model is imported into the model engine, the model data obtained through analysis may include model map data and animation control data, where the model map data is a two-dimensional image representing final information of the surface of the avatar, and may be generated by baking a model frame of the avatar. For example, for a three-dimensional character model, a clothing model, an accessory model and the like of the virtual image, UV expansion can be performed on each model respectively, and then each UV is baked into a map, so that model map data of each model is obtained; the animation control data refers to data for controlling the virtual image to execute actions, such as skeleton binding information and skin information for controlling the virtual image to execute limb actions, and the like, and also can comprise expression control data for controlling the virtual image to present facial expressions, sound control data for controlling the virtual image to make sound, and the like.
The maker can use the character model creating tool to create the three-dimensional character model, the clothing model, the accessory model and the like of the virtual image with uniform and reasonable wiring in advance, then the models are respectively unfolded with UV and baked to form a chartlet, animation control data is created for the three-dimensional character model, such as skeleton binding data and skin data for controlling the virtual image to execute limb actions, the animation control data is exported to be an FBX file (a model file), the FBX file and the chartlet obtained by baking are used as the three-dimensional character model and are imported into the model engine, so that the model engine can analyze the imported three-dimensional character model, integrate the analyzed data into a virtual broadcasting project, and create the virtual image.
By the method, the virtual image can be created according to the model data of the virtual image, and model support is provided for realizing live broadcast of the virtual image.
In addition, in order to facilitate the anchor client to select the avatar, in an alternative embodiment, before creating the avatar according to the obtained model data of the avatar, the server may further receive screening information sent by the anchor client, and collectively screen the avatar in the avatar according to the screening information.
The screening information may include any one or more of identification data of an avatar, a scene category of a live broadcast scene, and a category of a live broadcast room, the identification data of the avatar may be any one or more of a name of the avatar, an image category, and size information, for example, the name of the avatar may be "penguin No. 1", the image category may be an animation image, a human-like image, an animal-like image, and the like, and the size information may be a volume of the avatar and the like.
When the screening information sent by the anchor client is received, the server side can search the virtual images corresponding to the key words in the virtual image set according to the key words and the like in the screening information and use the virtual images as the selected virtual images, or can send the screening result to the anchor client, so that the anchor client can select the required virtual images from the screened virtual images.
And S120, associating the collected performance data of the moving capture actor in the real scene with the virtual image so as to drive the virtual image based on the performance data of the moving capture actor.
The performance data may include any one or more of motion data, expression data, and audio data of the actor. The motion data can be the motion of the whole body of the moving actor, or the motion of the hands or other limb structures of the moving actor, the expression data can be the facial expression data of the moving actor, and the audio data refers to the sound data of the moving actor. In this exemplary embodiment, the number of the action actors may be one or more, for example, the performance data of the action actors may include the motion data of the action actor a and the expression data and the audio data of the action actor B, or the performance data of the action actors may be the motion data, the expression data and the audio data of one action actor.
When the action-capturing actor performs, the action-capturing technology, the audio synchronization technology and the like can be adopted to collect performance data of the action-capturing actor, and then the performance data is associated with the virtual image in the model engine, so that the virtual image can synchronously execute the same action as the action-capturing actor or play the same audio in the live broadcasting process, or the virtual image can execute specific action according to the performance action of the action-capturing actor, for example, when the action-capturing actor jumps in place, the virtual image can show the action of continuously flying to high altitude. In particular, for the audio data, after the audio data of the actor is collected, the audio data may be processed accordingly, such as adjusting the volume, removing noise, changing the sound, and the like, and then the processed audio data may be associated with the avatar in the model engine.
Further, in an alternative embodiment, the animation control data may include skeletal binding data and skinning data for the avatar, and the performance data may include motion data for moving the actor. The skin is a manufacturing technology of three-dimensional animation, and refers to a technology of adding bones to an avatar and binding a model to the bones, skin data can be used for expressing an association relationship between the bones and parts of the model, and bone binding data can be used for expressing a connection relationship between the bones.
Based on this, referring to fig. 2, step S120 can be realized by the following steps S210 to S220:
and step S210, acquiring motion data of the moving actor through an inertial sensor.
The inertial sensor may be configured at a motion location of the moving actor and may be used to measure motion data of the moving actor. For example, when the hand motion data of the actor needs to be collected, the inertial sensor may be configured on the hand of the actor, such as using a Manus glove to access an Xsens software system to collect the hand motion data of the actor, and when the motion data of the whole body of the actor needs to be collected, the inertial sensor may also be configured on a specific part of the body of the actor, such as an ankle, a knee, a thigh, an arm, a hand, a chest, a waist and a shoulder, a head, and the like, and specifically, the inertial sensor may include an accelerometer, a gyroscope, a magnetometer, and the like.
Step S220, the action data is transmitted to the model engine in real time through a first virtual engine plug-in of the model engine, the action data is associated with the limb action of the virtual image, and the virtual image is driven to execute the action data through the skeleton binding data and the skin data.
The first virtual engine plug-in can be various virtual engine plug-ins including Maven Live Link, and can be used for transmitting the motion data collected by the inertial sensor to the model engine in real time, so that the model engine can determine the posture, the action and the like of the virtual image according to the motion data. For example, inertial sensor devices such as an accelerometer, a gyroscope, and a magnetometer may be worn on the body of the actor, and measurement of the attitude angle of the actor may be performed through the inertial navigation principle, so as to obtain motion data. After obtaining the motion data of the action actor, the motion data can be transmitted to the model engine through the first virtual engine plug-in, so that the model engine can associate the motion data with the limb motion of the virtual image, and drive the virtual image to execute the motion data through the skeleton binding data and the skin data, for example, the same limb motion as the action actor is presented.
In an alternative embodiment, the animation control data may include avatar expression control data and the performance data may include actor expression data. The expression control data refers to control data for controlling the avatar to execute expression motions, and may be 52 BlendShapes files produced in advance by the producer.
Based on this, referring to fig. 3, step S120 can also be realized by the following steps S310 to S330:
and S310, connecting the virtual reality plug-in loaded in the model engine with the expression control data by adopting an animation blueprint.
The animated blueprint may perform animation blending, directly control the skeleton of the animation, and output a final pose to the skeletal mesh object at each frame of the live view. The virtual reality plugin may be a technology capable of calculating the position and angle of the image acquisition device in real time and adding a corresponding image, such as an ARKit plugin.
In order to enable the virtual image to present expression effects closer to real characters, the virtual reality plug-in the model engine can be connected with expression control data by adopting an animation blueprint. For example, an ARKit plug-in can be loaded in the model engine, and then the ARKit plug-in is connected with the existing expression control file, namely the character model of 52 Blendshapes, by using an animation blueprint.
Step S320, obtaining expression data of the action actor through an image obtaining device including a depth camera.
The image acquisition device including the depth camera may be a camera device or an intelligent terminal device having depth detection and shooting functions.
Step S330, the second virtual engine plug-in of the model engine is used for associating the expression data of the action actor with the facial expression of the virtual image, and the virtual image is driven to execute the expression data through the expression control data.
The second virtual engine plug-in may be a virtual engine plug-in including livelinekface (an application program) and may be used to associate the emoticon data of the action actor with the avatar so that the avatar may present an expression consistent with the action actor.
Specifically, the depth camera and the server may be placed in the same network through a software livelinekface program provided by the model engine, and an IP (Internet Protocol Address) Address of the server device is input in the software, so as to implement automatic loading of the facial expression in the model engine.
By the method, the virtual image can execute corresponding actions according to the performance data of the moving actor, namely the action data and the expression data, so that the action mapping from the moving actor to the virtual image is realized.
S130, according to scene mapping rules of the real scene and the live scene, determining a target position matched with the position of the moving capture actor in the real scene in the live scene, synthesizing the virtual image to the live scene according to the target position, and generating a live video stream.
The scene mapping rule may include a position association rule between a real scene where an actor is located and a live scene, that is, a position point in the live scene corresponding to a position point in the real scene, for example, a position on the left side in the real scene may correspond to a position on the right side in the live scene; the scene mapping rule may also include a distance mapping ratio and/or a direction mapping ratio of the real scene and the live scene, that is, a distance value or an angle value corresponding to a unit distance or an angle in the real scene in the live scene, for example, 1m in the real scene may correspond to 1cm in the live scene. The target position refers to a position of the avatar in a live scene of the live broadcast room, and can be represented by a coordinate position of the avatar in the live scene.
In the real scene, the position of the moving capture actor may change along with the action, so that when the action of the moving capture actor is mapped onto the virtual image, the target position of the virtual image associated with the moving capture actor in the live broadcast scene when the moving capture actor performs in the real scene can be determined according to the scene mapping rules of the real scene and the live broadcast scene, and then the virtual image can be synthesized to the target position in the virtual scene, and a live broadcast video stream is generated.
Specifically, in an alternative embodiment, referring to fig. 4, the target position of the avatar in the live scene may be determined through the following steps S410 to S430:
step S410, determining a moving distance and/or a moving direction of the moving actor in the real scene according to the motion data of the moving actor collected by the inertial sensor.
For example, whether the position of the moving actor changes or not can be determined through motion data collected by the inertial sensor, such as the posture and the orientation of the body part of the moving actor, and when the position of the moving actor is determined to change, the moving distance and the moving direction of the moving actor in a real scene can be calculated according to the kinematics principle.
Step S420, determining a distance value of the avatar corresponding to the moving distance of the actor in the real scene in the live broadcast scene according to the distance mapping ratio in the scene mapping rule, and/or determining a direction value of the avatar corresponding to the moving direction of the actor in the real scene in the live broadcast scene according to the direction mapping ratio in the scene mapping rule.
Multiplying the moving distance of the moving actor in the real scene by the distance mapping proportion to obtain the distance value of the virtual image in the live scene; meanwhile, the direction value of the moving direction of the moving actor in the real scene is multiplied by the direction mapping proportion, so that the direction value of the virtual image in the live scene can be obtained. In this manner, the avatar associated with the action actor may be moved accordingly in the live scene in the manner in which the action actor is moved.
And step S430, determining the target position according to the distance value and/or the direction value of the virtual image in the live scene.
Specifically, the position of the avatar at the current moment may be used as an initial position, and the target position may be determined according to a distance value and a direction value of the avatar in the live broadcast scene.
Through the steps S410-S430, the position of the avatar in the live broadcast scene can be synchronously determined according to the action data of the moving actor in the real scene, so that the avatar can be accurately positioned in the live broadcast scene, and particularly, when the moving actor and the avatar act synchronously, the action consistency of the moving actor and the avatar can be ensured.
Further, in an alternative embodiment, other positioning sensors may be used to obtain the moving distance and moving direction of the actor in the real scene. For example, a mechanical positioning sensor may be installed at a positioning portion of the moving actor, such as a position of a center of gravity of a body, the moving and rotating positions are recorded through a mechanical gear and a coil in the mechanical positioning sensor, the recorded information is transmitted to the server to be calculated, the position of the moving actor at the current moment is obtained, then the position is compared with the position before the moving of the moving actor, the position of the moving actor at the current moment is obtained, and the moving distance and the moving direction of the moving actor in the real scene are determined. For another example, the moving distance and the moving direction of the moving actor in the real scene may also be determined by using an optical positioning technology, an image positioning technology, or the like, that is, a plurality of infrared emission cameras are installed in the space where the moving actor is located, the whole space is covered and shot, an infrared reflection point may be installed on the body of the moving actor, the position of the moving actor at the current time may be obtained by capturing the light rays of the infrared light emitted by the cameras after being reflected by the reflection point, and performing program calculation, and the moving distance and the moving direction of the moving actor in the real scene may be obtained by comparing the position with the position before moving.
In an optional implementation manner, one or more position conditions may also be set in the scene mapping rule, and when the target position is determined, the position condition that the position of the action actor in the real scene meets may be searched in the scene mapping rule, so as to determine the position in the live broadcast scene corresponding to the position condition as the target position.
For example, when the position of the moving actor in the real scene is determined, whether a position condition that is satisfied exists in the position may be searched in the scene mapping rule, for example, whether the position is at a preset mark position, when the position of the moving actor in the real scene is determined to be at the mark position, the position in the live scene corresponding to the position condition may be set as a target position, for example, the target position may be set as a suspended position in the live scene. By the method, the virtual image can get rid of the influence of the real environment on the action capture actors, and richer interaction effects are realized.
When a live actor performs in a real scene, performance data can be transmitted to the live engine through a collection device, such as an inertial sensor or an image acquisition device, via a network or a physical transmission line. Based on this, when the capturing device is an image capturing device, in order to keep the frame synchronization between the picture of the live broadcast scene and the picture captured by the image capturing device, in an optional implementation, the following method may also be performed:
and adjusting the shooting frame rate of the camera and the frame rate of the live broadcast scene of the virtual image so as to keep the frame synchronization between the picture of the live broadcast scene containing the virtual image and the picture of the real scene shot by the camera.
Because the frame rate of the virtual pictures rendered in the model engine is often variable, and the scene pictures shot by the camera are generally output by adopting a stable frame rate, the shooting frame rate of the camera and the frame rate of the virtual images in the live broadcast scene can be adjusted to obtain the pictures synchronously locked by the virtual pictures and the scene pictures, so that the pictures of the live broadcast scene containing the virtual images and the pictures of the real scene shot by the camera keep frame synchronization. Specifically, the frame rate of the picture of the model engine may be set in a locked manner based on the frame rate of the camera, so that the picture in the live broadcast scene including the avatar and the picture in the real scene keep frame synchronization, or the frame rate of the picture of the camera may be set based on the frame rate of the picture of the avatar in the live broadcast scene so that the picture in the live broadcast scene including the avatar and the picture in the real scene keep frame synchronization, or the frame rates of the picture of the avatar in the live broadcast scene and the picture in the scene may be adjusted simultaneously so that the picture in the live broadcast scene and the picture in the scene keep frame synchronization. By the method, the virtual image and the picture of the real scene where the virtual image is positioned can be kept frame-synchronized with the shot picture of the camera, the synchronous playing of the display picture of the virtual image and the picture of the real scene is realized, and the influence on the performance of the action capturer caused by the asynchronous pictures is avoided.
In an alternative embodiment, the avatar may be composited to the live scene in accordance with the matting of the moving actor. Specifically, referring to fig. 5, the following steps S510 to S520 may be performed:
and step S510, extracting the image matting picture of the moving capture actor in the real scene through green screen image matting.
The moving actor can perform in a green curtain environment, and after the picture of the moving actor in a real scene is obtained, the keying technology can be adopted to extract the keying picture of the moving actor in the picture. Therefore, after the motion data of the moving capture actor shot by the camera is acquired, the shot pictures can be converted into a series of keying pictures arranged according to time sequence.
And step S520, synthesizing an avatar based on the image matting picture and the virtual live broadcast engine, and displaying the target position in the live broadcast scene.
The scene of the moving actor in the real scene may include a series of image frames arranged in time sequence, and the motion state of the moving actor may be determined by using a time sequence analysis method of the image frames. In order to improve the accuracy of determining the action state of the moving actor, the matting picture of the moving actor in each image frame can be extracted from the image frame sequence of the moving actor in the real scene, the action state of the moving actor is determined according to the matting picture, and then the virtual image synthesized in the virtual live broadcast engine is controlled to execute corresponding action according to the action state. For example, algorithms such as a neural network can be adopted to analyze the extracted time sequence characteristics of a series of matting pictures of a moving actor to determine the action state of the moving actor within corresponding time, determine action data of an avatar associated with the moving actor according to the action state, synthesize the avatar through a virtual live broadcast engine, and superimpose the avatar into a live broadcast scene to generate a live broadcast video stream.
Meanwhile, when the virtual image is synthesized to the target position in the live scene, the live scene can be a real scene or a virtual scene. For example, when the live broadcast scene is a real scene, two anchor casts, namely an avatar and a real anchor, can be simultaneously displayed in the live broadcast scene, and the live broadcast scene can be a scene where the real anchor is located, at this time, the avatar can be superimposed on the live broadcast scene where the real anchor is located, that is, the real scene, and a live broadcast video stream is generated. Or when the live broadcast scene is not the real scene of the real anchor but the synthesized virtual scene, the matting pictures of the real anchor in the real scene can be simultaneously extracted, the virtual image and the matting pictures of the real anchor are simultaneously synthesized to the virtual scene, and the live broadcast video stream is generated.
By the method, the virtual image can be synthesized to the live broadcast scene to generate the live broadcast video stream, so that the live broadcast picture can show the cross-dimension picture effect, and the richness of the live broadcast picture can be improved. Meanwhile, when the live broadcast scene comprises the virtual image and the real main broadcast, the action capture actor can control the associated virtual image to interact with the real main broadcast according to the broadcasting requirement, and the diversity and the content richness of the live broadcast mode can be further improved.
And S140, sending the live video stream containing the virtual image to a live client for playing.
After the live video stream is generated, the server side can send the generated live video stream to the live client side, audiences can play the live video stream through the live client side, and in the live video stream, the audiences can see that the virtual image executes the action data which is the same as or different from the action data of the action actor which is captured movably. In addition, when the performance data includes audio data, the live client may also play the audio data of the action actor or the processed audio data synchronously. Through this kind of mode, spectator can watch the performance of virtual image at live client, compares current live performance of real person, can bring more abundant video for spectator and watch the experience.
In order to improve the flexibility and richness of controlling live broadcast of the avatar, in an optional implementation manner, during live broadcast, the server may receive modification parameters about the avatar sent by the anchor client, and modify the avatar according to the modification parameters.
The modification parameter may be a parameter for modifying a display style of the avatar, such as modifying a color, an accessory, a volume, a display position, and the like of the avatar, or an event parameter for controlling the avatar to perform an action event, such as an event parameter for controlling the avatar to perform a pivot rotation action event.
For example, the anchor may trigger a modification operation on the avatar at the anchor client by long pressing the avatar, etc., the anchor may input a modification parameter of the avatar in a modification window displayed at the anchor client, such as inputting a color parameter of "clothes" of the avatar, etc., and the anchor client may send the modification parameter as a modification request to the server after confirming the modification, so that the server may modify the avatar synchronously according to the modification parameter; for another example, the anchor may also trigger an event modification operation of the avatar through operations such as double-click, and then select a corresponding action event from the displayed selectable events, and the server may receive the modification parameters and control the avatar to execute the action event selected by the anchor client. By the method, the anchor can modify the virtual image according to the self requirement or the preference of audiences, the personalized requirement of the anchor can be better met, and the diversity of the live broadcast interaction mode is improved.
In addition, when the anchor in the live broadcast room comprises a real anchor and an avatar, in order to enable the real anchor and the avatar to interact better, live broadcast pictures can be respectively pushed to a screen which can be seen by a moving actor and a screen which can be seen by the real anchor, so that the moving actor and the real anchor can cooperate with each other to perform according to the virtual avatar and the station position, the action, the language and the like of the real anchor, and the interactivity of the avatar and the real anchor is improved.
In summary, according to the live broadcast control method in the exemplary embodiment, an avatar may be created according to acquired model data of the avatar, collected performance data of a moving actor in a real scene is associated with the avatar to drive the avatar based on the performance data of the moving actor, then a target position matching the position of the moving actor in the real scene is determined in a live broadcast scene according to a scene mapping rule of the real scene and the live broadcast scene, the avatar is synthesized to the live broadcast scene according to the target position to generate a live broadcast video stream, and the live broadcast video stream including the avatar is sent to a live broadcast client to be played. On one hand, the method creates the virtual image, associates the performance data of the moving actor with the virtual image, synthesizes the virtual image into a live broadcast scene according to the target position, generates a live broadcast video stream, can combine the virtual image with the live broadcast scene, realizes the transfer of the virtual image from a virtual world to a real world, and improves the interactivity of live broadcast; meanwhile, cross-dimension live broadcast impression can be brought to a user, the live broadcast scene limitation of the virtual image can be broken, the resource cost is reduced, and the virtual image can comprise various images and styles, so that the content richness of a live broadcast room can be improved, and more diversified live broadcast content is brought to audiences. On the other hand, the frame synchronization of the picture of the live scene containing the virtual image and the picture of the real scene containing the moving actor is kept, so that the moving actor can view the picture of the live scene in real time, and convenience is provided for the moving actor to perform and the anchor to adjust the picture content in real time according to the live picture and the moving actor's performance.
Further, the present exemplary embodiment also provides a live broadcast control apparatus, and as shown in fig. 6, the live broadcast control apparatus 600 may include: a creating module 610, which may be configured to create an avatar according to the obtained model data of the avatar; an association module 620, operable to associate the collected performance data of the moving capture actor in the real scene with the avatar, to drive the avatar based on the performance data of the moving capture actor; the synthesizing module 630 may be configured to determine, in the live broadcast scene, a target position that matches the position of the action actor in the real scene according to a scene mapping rule between the real scene and the live broadcast scene, and synthesize the avatar into the live broadcast scene according to the target position to generate a live broadcast video stream; a sending module 640, configured to send a live video stream containing an avatar to a live client for playing; wherein the picture of the live scene containing the avatar and the picture of the real scene containing the action actor are kept frame-synchronized.
In an exemplary embodiment of the present disclosure, the creation module 610 acquires model data of the avatar by performing the following method: the method comprises the steps of obtaining a three-dimensional role model of the virtual image by using a role model creation tool, importing the three-dimensional role model into a model engine, analyzing the three-dimensional role model by the model engine, and obtaining model data of the virtual image, wherein the model data comprise model map data and animation control data, and the model map data are generated by baking a model frame of the virtual image.
In an exemplary embodiment of the disclosure, the animation control data includes bone binding data and skinning data of the avatar, the performance data includes motion data of the moving actor, the association module 620 may be configured to collect the motion data of the moving actor through an inertial sensor, the inertial sensor is configured at a motion part of the moving actor, transmit the motion data to the model engine through a first virtual engine plug-in of the model engine in real time, associate the motion data with limb motions of the avatar, and drive the avatar to execute the motion data through the bone binding data and the skinning data.
In an exemplary embodiment of the present disclosure, the animation control data includes expression control data of an avatar, the performance data includes expression data of a moving actor, and the association module 620 may be configured to connect a virtual reality plug-in loaded in the model engine with the expression control data by using an animation blueprint; acquiring expression data of a moving actor through image acquisition equipment comprising a depth camera; and associating the expression data of the moving actor with the facial expression of the virtual image by using a second virtual engine plug-in of the model engine, and driving the virtual image to execute the expression data through expression control data.
In an exemplary embodiment of the disclosure, the synthesis module 630 may be configured to determine a moving distance and/or a moving direction of the moving actor in the real scene through motion data of the moving actor collected by the inertial sensor, determine a distance value of an avatar corresponding to the moving distance of the moving actor in the real scene in the live scene according to a distance mapping ratio in the scene mapping rule, and/or determine a direction value of the avatar corresponding to the moving direction of the moving actor in the real scene in the live scene according to a direction mapping ratio in the scene mapping rule, and determine the target position according to the distance value and/or the direction value of the avatar in the live scene.
In an exemplary embodiment of the disclosure, when a target position matching a position of a moving actor in a real scene is determined in a live scene according to a scene mapping rule between the real scene and the live scene, the synthesis module 630 may be configured to search for a position condition that is satisfied by the position of the moving actor in the real scene in the scene mapping rule, so as to determine the position in the live scene corresponding to the position condition as the target position.
In an exemplary embodiment of the disclosure, the picture of the real scene containing the moving actor is obtained by shooting through the camera, and after determining the target position of the avatar in the real scene, the synthesizing module 630 may be further configured to adjust a shooting frame rate of the camera and a picture frame rate of the avatar in the live scene, so that the picture of the live scene containing the avatar and the picture of the real scene obtained by shooting through the camera keep frame synchronization.
In an exemplary embodiment of the present disclosure, the composition module 630 may be further configured to extract a keyed image of an actor in a real scene through green screen keying; and synthesizing an avatar based on the image matting picture and the virtual live broadcast engine, and displaying the target position in the live broadcast scene.
In an exemplary embodiment of the present disclosure, after creating the avatar according to the acquired model data of the avatar, the composition module 630 may be further configured to receive modification parameters about the avatar sent by the anchor client, modify the avatar according to the modification parameters, and the modification parameters include parameters for modifying a display style of the avatar and/or event parameters for controlling the avatar to perform an action event.
In an exemplary embodiment of the present disclosure, before creating the avatar according to the obtained model data of the avatar, the creating module 610 may be further configured to receive screening information sent by the anchor client, and collectively screen the avatar in the avatar according to the screening information, where the screening information includes any one or more of identification data of the avatar, a scene category of a live broadcast scene, and a category of a live broadcast room, and the identification data of the avatar includes any one or more of a name of the avatar, an avatar category, and size information.
The specific details of each module in the above apparatus have been described in detail in the method section, and details of an undisclosed scheme may refer to the method section, and thus are not described again.
As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or program product. Accordingly, various aspects of the present disclosure may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
Exemplary embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, various aspects of the disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the disclosure described in the above-mentioned "exemplary methods" section of this specification, when the program product is run on the terminal device.
Referring to fig. 7, a program product 700 for implementing the above method according to an exemplary embodiment of the present disclosure is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
Program product 700 may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
The exemplary embodiment of the present disclosure also provides an electronic device capable of implementing the above method. An electronic device 800 according to such an exemplary embodiment of the present disclosure is described below with reference to fig. 8. The electronic device 800 shown in fig. 8 is only an example and should not bring any limitations to the functionality and scope of use of the embodiments of the present disclosure.
As shown in fig. 8, electronic device 800 may take the form of a general purpose computing device. The components of the electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one memory unit 820, a bus 830 connecting the various system components (including the memory unit 820 and the processing unit 810), and a display unit 840.
Wherein the storage unit 820 stores program code that may be executed by the processing unit 810 to cause the processing unit 810 to perform the steps according to various exemplary embodiments of the present disclosure described in the above section "exemplary method" of this specification. For example, processing unit 810 may perform the method steps shown in fig. 1-5, and so on.
The storage unit 820 may include readable media in the form of volatile storage units, such as a random access storage unit (RAM)821 and/or a cache storage unit 822, and may further include a read only storage unit (ROM) 823.
Storage unit 820 may also include a program/utility 824 having a set (at least one) of program modules 825, such program modules 825 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
Bus 830 may be any of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 800, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 800 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 850. Also, the electronic device 800 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 860. As shown, the network adapter 860 communicates with the other modules of the electronic device 800 via the bus 830. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functions of two or more modules or units described above may be embodied in one module or unit, according to exemplary embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Furthermore, the above-described figures are merely schematic illustrations of processes included in methods according to exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the exemplary embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to make a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the exemplary embodiments of the present disclosure.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims (13)

1. A live broadcast control method, characterized in that the method comprises:
creating an avatar according to the acquired model data of the avatar;
associating collected performance data of a moving actor in a real scene with the avatar to drive the avatar based on the performance data of the moving actor;
according to the scene mapping rules of the real scene and the live scene, determining a target position matched with the position of the moving capture actor in the real scene in the live scene, and synthesizing the virtual image to the live scene according to the target position to generate a live video stream;
sending the live video stream containing the virtual image to a live client for playing;
wherein a picture of a live scene containing the avatar and a picture of a real scene containing the action actor are kept frame-synchronized.
2. A live broadcast control method according to claim 1, wherein model data of the avatar is acquired by:
acquiring a three-dimensional role model of the virtual image by using a role model creation tool;
importing the three-dimensional role model into a model engine, and analyzing the three-dimensional role model through the model engine to obtain model data of the virtual image;
wherein the model data includes model map data and animation control data, the model map data being generated by baking a model frame of the avatar.
3. A live control method according to claim 2, wherein the animation control data comprises skeletal binding data and skinning data of the avatar, the performance data comprises motion data of the action actor, and the associating the captured performance data of the action actor in a real scene with the avatar to drive the avatar based on the performance data of the action actor comprises:
acquiring motion data of the moving capture actor through an inertial sensor, wherein the inertial sensor is configured at a motion part of the moving capture actor;
and transmitting the action data to the model engine in real time through a first virtual engine plug-in of the model engine, associating the action data with the limb action of the virtual image, and driving the virtual image to execute the action data through the skeleton binding data and the skin data.
4. A live broadcast control method according to claim 2, wherein the animation control data comprises expression control data of the avatar, the performance data comprises expression data of the action actor, and the associating the captured performance data of the action actor in a real scene with the avatar to drive the avatar based on the performance data of the action actor comprises:
connecting the virtual reality plug-in loaded in the model engine with the expression control data by adopting an animation blueprint;
acquiring expression data of the moving capture actor through image acquisition equipment comprising a depth camera;
and associating the expression data of the moving actor with the facial expression of the virtual image by using a second virtual engine plug-in of the model engine, and driving the virtual image to execute the expression data through the expression control data.
5. The live broadcast control method according to claim 3, wherein the determining a target position in the live broadcast scene that matches the position of the action actor in the real scene according to a scene mapping rule of the real scene and the live broadcast scene comprises:
determining the moving distance and/or moving direction of the moving actor in the real scene through the motion data of the moving actor collected by the inertial sensor;
determining a distance value of the virtual image in the live broadcast scene corresponding to the moving distance of the moving actor in the real scene according to a distance mapping proportion in the scene mapping rule; and/or
Determining a direction value of the virtual image in the live broadcast scene corresponding to the moving direction of the moving actor in the real scene according to a direction mapping proportion in the scene mapping rule;
and determining the target position according to the distance value and/or the direction value of the virtual image in the live broadcast scene.
6. The live broadcast control method according to claim 1, wherein when a target position matching a position of the action actor in the real scene is determined in the live scene according to a scene mapping rule of the real scene and the live scene, the method further comprises:
and in the scene mapping rule, searching a position condition met by the position of the moving capture actor in the real scene, and determining the position in the live scene corresponding to the position condition as the target position.
7. A live broadcast control method according to claim 1, wherein a picture of a real scene containing the action actor is taken by a camera, and after determining a target position of the avatar in the real scene, the method further comprises:
and adjusting the shooting frame rate of the camera and the frame rate of the live broadcast scene of the virtual image so as to keep the frame synchronization between the live broadcast scene picture containing the virtual image and the real scene picture shot by the camera.
8. The live control method according to claim 1, wherein the synthesizing the avatar to the live scene according to the target position includes:
extracting a keying picture of a dynamic capture actor in a real scene through green screen keying;
and synthesizing the virtual image based on the image matting picture and the virtual live broadcast engine, and displaying the target position in the live broadcast scene.
9. The live broadcast control method according to claim 1, wherein after creating the avatar based on the acquired model data of the avatar, the method further comprises:
receiving modification parameters about the avatar sent by an anchor client, and modifying the avatar according to the modification parameters, wherein the modification parameters comprise parameters for modifying the display style of the avatar and/or event parameters for controlling the avatar to execute action events.
10. A live broadcast control method according to claim 1, wherein before creating an avatar from the acquired model data of the avatar, the method further comprises:
receiving screening information sent by a main broadcasting client, and screening the virtual image in a virtual image set according to the screening information;
wherein, screening information includes any one or more of the identification data of the avatar, the scene category of the live broadcast scene and the category of the live broadcast room, the identification data of the avatar includes any one or more of the name of the avatar, the category of the avatar and the size information.
11. A live control apparatus, the apparatus comprising:
the creating module is used for creating the virtual image according to the acquired model data of the virtual image;
the association module is used for associating the collected performance data of the moving capture actor in the real scene with the virtual image so as to drive the virtual image based on the performance data of the moving capture actor;
the synthesis module is used for determining a target position matched with the position of the moving capture actor in the real scene in the live scene according to the scene mapping rules of the real scene and the live scene, synthesizing the virtual image to the live scene according to the target position and generating a live video stream;
the sending module is used for sending the live video stream containing the virtual image to a live client for playing;
wherein a picture of a live scene containing the avatar and a picture of a real scene containing the action actor are kept frame-synchronized.
12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the live control method of any one of claims 1-10.
13. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the live control method of any of claims 1-10 via execution of the executable instructions.
CN202111114506.1A 2021-09-23 2021-09-23 Live broadcast control method and device, storage medium and electronic equipment Pending CN113822970A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111114506.1A CN113822970A (en) 2021-09-23 2021-09-23 Live broadcast control method and device, storage medium and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111114506.1A CN113822970A (en) 2021-09-23 2021-09-23 Live broadcast control method and device, storage medium and electronic equipment

Publications (1)

Publication Number Publication Date
CN113822970A true CN113822970A (en) 2021-12-21

Family

ID=78915259

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111114506.1A Pending CN113822970A (en) 2021-09-23 2021-09-23 Live broadcast control method and device, storage medium and electronic equipment

Country Status (1)

Country Link
CN (1) CN113822970A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022223029A1 (en) * 2021-04-22 2022-10-27 北京字节跳动网络技术有限公司 Avatar interaction method, apparatus, and device
CN115619912A (en) * 2022-10-27 2023-01-17 深圳市诸葛瓜科技有限公司 Cartoon character display system and method based on virtual reality technology
WO2023174385A1 (en) * 2022-03-18 2023-09-21 郑州泽正技术服务有限公司 Method and system for performing real social contact by using virtual scene, and ar glasses
CN117319628A (en) * 2023-09-18 2023-12-29 四开花园网络科技(广州)有限公司 Real-time interactive naked eye 3D virtual scene system supporting outdoor LED screen
WO2024012459A1 (en) * 2022-07-15 2024-01-18 北京蔚领时代科技有限公司 Method and system for terminal-cloud combined virtual concert rendering for vr terminal
CN117596420A (en) * 2024-01-18 2024-02-23 江西拓世智能科技股份有限公司 Fusion live broadcast method, system, medium and electronic equipment based on artificial intelligence

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106412558A (en) * 2016-09-08 2017-02-15 深圳超多维科技有限公司 Method, equipment and device for stereo virtual reality live broadcasting
CN107248195A (en) * 2017-05-31 2017-10-13 珠海金山网络游戏科技有限公司 A kind of main broadcaster methods, devices and systems of augmented reality
CN110650354A (en) * 2019-10-12 2020-01-03 苏州大禹网络科技有限公司 Live broadcast method, system, equipment and storage medium for virtual cartoon character
GB202003167D0 (en) * 2019-03-07 2020-04-22 Lucasfilm Entertainment Co Ltd On-set facial performance capture and transfer to a three-dimensional computer-generated model
CN111698390A (en) * 2020-06-23 2020-09-22 网易（杭州）网络有限公司 Virtual camera control method and device, and virtual studio implementation method and system
WO2021047430A1 (en) * 2019-09-11 2021-03-18 广州华多网络科技有限公司 Virtual gift special effect synthesis method and apparatus, and live streaming system
CN113395540A (en) * 2021-06-09 2021-09-14 广州博冠信息科技有限公司 Virtual broadcasting system, virtual broadcasting implementation method, device and equipment, and medium

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106412558A (en) * 2016-09-08 2017-02-15 深圳超多维科技有限公司 Method, equipment and device for stereo virtual reality live broadcasting
CN107248195A (en) * 2017-05-31 2017-10-13 珠海金山网络游戏科技有限公司 A kind of main broadcaster methods, devices and systems of augmented reality
GB202003167D0 (en) * 2019-03-07 2020-04-22 Lucasfilm Entertainment Co Ltd On-set facial performance capture and transfer to a three-dimensional computer-generated model
WO2021047430A1 (en) * 2019-09-11 2021-03-18 广州华多网络科技有限公司 Virtual gift special effect synthesis method and apparatus, and live streaming system
CN110650354A (en) * 2019-10-12 2020-01-03 苏州大禹网络科技有限公司 Live broadcast method, system, equipment and storage medium for virtual cartoon character
CN111698390A (en) * 2020-06-23 2020-09-22 网易（杭州）网络有限公司 Virtual camera control method and device, and virtual studio implementation method and system
CN113395540A (en) * 2021-06-09 2021-09-14 广州博冠信息科技有限公司 Virtual broadcasting system, virtual broadcasting implementation method, device and equipment, and medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
宋柏峰;: "虚拟演播室在广电直播中的应用研究", 艺术工作, no. 05 *
高义栋;李曼曼;雍炎炎;贺亚杰;: "基于UDK引擎的图书馆虚拟漫游系统的设计与实现", 现代教育技术, no. 10 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022223029A1 (en) * 2021-04-22 2022-10-27 北京字节跳动网络技术有限公司 Avatar interaction method, apparatus, and device
WO2023174385A1 (en) * 2022-03-18 2023-09-21 郑州泽正技术服务有限公司 Method and system for performing real social contact by using virtual scene, and ar glasses
WO2024012459A1 (en) * 2022-07-15 2024-01-18 北京蔚领时代科技有限公司 Method and system for terminal-cloud combined virtual concert rendering for vr terminal
CN115619912A (en) * 2022-10-27 2023-01-17 深圳市诸葛瓜科技有限公司 Cartoon character display system and method based on virtual reality technology
CN117319628A (en) * 2023-09-18 2023-12-29 四开花园网络科技(广州)有限公司 Real-time interactive naked eye 3D virtual scene system supporting outdoor LED screen
CN117596420A (en) * 2024-01-18 2024-02-23 江西拓世智能科技股份有限公司 Fusion live broadcast method, system, medium and electronic equipment based on artificial intelligence
CN117596420B (en) * 2024-01-18 2024-05-31 江西拓世智能科技股份有限公司 Fusion live broadcast method, system, medium and electronic equipment based on artificial intelligence

Similar Documents

Publication Publication Date Title
CN113822970A (en) Live broadcast control method and device, storage medium and electronic equipment
CN111970535B (en) Virtual live broadcast method, device, system and storage medium
CN109889914B (en) Video picture pushing method and device, computer equipment and storage medium
CN108986189B (en) Method and system for capturing and live broadcasting of real-time multi-person actions based on three-dimensional animation
CN110557625A (en) live virtual image broadcasting method, terminal, computer equipment and storage medium
US10692288B1 (en) Compositing images for augmented reality
KR101328759B1 (en) Augmented reality method and devices using a real time automatic tracking of marker-free textured planar geometrical objects in a video stream
CN109145788B (en) Video-based attitude data capturing method and system
US6522332B1 (en) Generating action data for the animation of characters
KR101713772B1 (en) Apparatus and method for pre-visualization image
US20030227453A1 (en) Method, system and computer program product for automatically creating an animated 3-D scenario from human position and path data
EP2267659A2 (en) System and method for integrating multiple virtual rendering systems to provide an augmented reality
KR100490726B1 (en) Apparatus and method for video based shooting game
CN111968207A (en) Animation generation method, device, system and storage medium
KR101989089B1 (en) Method and system for authoring ar content by collecting ar content templates based on crowdsourcing
CN103258338A (en) Method and system for driving simulated virtual environments with real data
CN114900678B (en) VR end-cloud combined virtual concert rendering method and system
CN111862348B (en) Video display method, video generation method, device, equipment and storage medium
JP2022500795A (en) Avatar animation
CN113792646B (en) Dance motion auxiliary generation method and device and dance equipment
KR20050082559A (en) Dance learning system, internet community service system and internet community service method using the same, dance learning method, and computer executable recording media on which programs implement said methods are recorded
CN116503522A (en) Interactive picture rendering method, device, equipment, storage medium and program product
CN115631287A (en) Digital virtual stage figure display system
CN115937371B (en) Character model generation method and system
EP4354400A1 (en) Information processing device, information processing method, and program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination