CN114979698B - Live broadcast processing method and system - Google Patents
Live broadcast processing method and system Download PDFInfo
- Publication number
- CN114979698B CN114979698B CN202210903928.5A CN202210903928A CN114979698B CN 114979698 B CN114979698 B CN 114979698B CN 202210903928 A CN202210903928 A CN 202210903928A CN 114979698 B CN114979698 B CN 114979698B
- Authority
- CN
- China
- Prior art keywords
- model
- anchor
- image frame
- frame sequence
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
Abstract
The invention discloses a live broadcast processing method and a system. The anchor terminal sends the anchor image frame sequence and the model position information to a mixed reality server; the mixed reality server determines whether action control information aiming at the target simulation model exists or not based on the anchor image frame sequence and the model position information, and if so, generates a corresponding model control instruction and returns the model control instruction to the anchor terminal; the anchor terminal performs display control on the target simulation model based on the model control instruction to obtain an image frame sequence of the controlled simulation model, performs mixed flow processing on the image frame sequence of the controlled simulation model and the anchor image frame sequence to obtain a mixed flow image frame sequence and sends the mixed flow image frame sequence to the user terminal. The method and the device can avoid the situations that target objects such as a real model cannot be effectively built or operation demonstration cannot be effectively carried out and the like caused by the problem of real property limitation, ensure the service quality, bring beyond-reality sense experience for the user and enhance the viscosity of the user.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a live broadcast processing method and system.
Background
With the development of scientific technology, data processing technology is continuously improved.
Currently, in video playing scenes such as on-line classroom teaching and live broadcast of tape goods, teachers, anchor broadcasters and the like are often required to perform corresponding operations and demonstrations in objects such as models, structures or scenes built by related props, so that corresponding services are provided for video viewers. For example, in a physical class online classroom, a physical teacher controls the state of a bulb and a switch in a circuit model built by physical teaching aids such as the bulb and the switch, and teaches students about the related circuit knowledge.
However, when the target object to be built is complex, the number and quality of the props and other limiting factors may cause that an effective target object cannot be built or effective operation demonstration cannot be performed in the target object, thereby affecting the service quality.
Disclosure of Invention
In view of the above problems, the present invention provides a live broadcast processing method and system for overcoming the above problems or at least partially solving the above problems, and the technical solution is as follows:
a live broadcast processing method is applied to a live broadcast processing system, and the live broadcast processing system comprises: the system comprises a main broadcasting end and a mixed reality server; the live broadcast processing method comprises the following steps:
the anchor terminal sends the anchor image frame sequence and the model position information to a mixed reality server; the anchor image frame sequence is an image frame sequence which is acquired by an anchor terminal aiming at an anchor, and the model position information comprises position information of a target simulation model on a display interface of the anchor terminal;
the mixed reality server determines whether action control information aiming at the target simulation model exists or not based on the anchor image frame sequence and the model position information, if so, generates a corresponding model control instruction, and returns the model control instruction to the anchor terminal;
the anchor terminal responds to the model control instruction and performs display control on the target simulation model to obtain an image frame sequence of the controlled simulation model; performing mixed flow processing on the image frame sequence of the controlled simulation model and the anchor image frame sequence to obtain a mixed flow image frame sequence; and sending the mixed flow image frame sequence to a user side.
Optionally, the mixed reality server includes: the motion recognition module, the model processing module and the target simulation model; the mixed reality server determining whether motion control information for the target simulation model exists based on the anchor image frame sequence and the model location information, including:
the action recognition module recognizes anchor action information existing in the anchor image frame sequence by utilizing an action recognition model, and sends the anchor action information to the model processing module;
the model processing module determines whether action control information exists based on the anchor action information, the model location information, and the target simulation model;
the generating of the corresponding model control instruction comprises:
the model processing module generates a model control instruction corresponding to the motion control information.
Optionally, the motion recognition model is a gesture recognition model; the action recognition module recognizes the anchor action information existing in the anchor image frame sequence by utilizing an action recognition model, and the action recognition module comprises the following steps:
the motion recognition module determines hand position information and hand motion types of the anchor in the anchor image frame sequence by utilizing the gesture recognition model, and determines the hand position information and the hand motion types as the anchor motion information.
Optionally, the determining, by the model processing module, whether action control information exists based on the anchor action information, the model location information, and the target simulation model includes:
the model processing module inputs the anchor action information, the model position information and the target simulation model into a simulation model control program to determine whether the action control information exists;
the model processing module generates a model control instruction corresponding to the motion control information, including:
the model processing model obtains the model control instruction output by the simulation model control program.
Optionally, the anchor end includes: the model control module and the model display module; the anchor terminal responds to the model control instruction and performs display control on the target simulation model, and the method comprises the following steps:
the model control module responds to the model control instruction and sends a corresponding display control instruction to the model display module;
and the model display module responds to the display control instruction and displays the target simulation model.
Optionally, the anchor end includes: a plug flow module; the sending the mixed flow image frame sequence to a user side comprises:
and the plug-flow module sends the mixed-flow image frame sequence to the user side through a data distribution network.
Optionally, the target simulation model in the anchor terminal is sent by the mixed reality server.
A live processing system, comprising: the system comprises a main broadcasting terminal and a mixed reality server; wherein:
the anchor terminal sends the anchor image frame sequence and the model position information to a mixed reality server; the anchor image frame sequence is an image frame sequence which is acquired by an anchor terminal aiming at an anchor, and the model position information comprises position information of a target simulation model on a display interface of the anchor terminal;
the mixed reality server determines whether action control information aiming at the target simulation model exists or not based on the anchor image frame sequence and the model position information, if so, a corresponding model control instruction is generated, and the model control instruction is returned to the anchor terminal;
the anchor terminal responds to the model control instruction and performs display control on the target simulation model to obtain an image frame sequence of the controlled simulation model; performing mixed flow processing on the image frame sequence of the controlled simulation model and the anchor image frame sequence to obtain a mixed flow image frame sequence; and sending the mixed flow image frame sequence to a user side.
Optionally, the mixed reality server includes: the system comprises an action recognition module, a model processing module and the target simulation model; the mixed reality server determining, based on the anchor image frame sequence and the model location information, whether motion control information for the target simulation model exists, set to:
the action recognition module recognizes anchor action information existing in the anchor image frame sequence by utilizing an action recognition model, and sends the anchor action information to the model processing module;
the model processing module determines whether action control information exists based on the anchor action information, the model location information, and the target simulation model;
the generating of the corresponding model control command is set as:
the model processing module generates a model control instruction corresponding to the motion control information.
Optionally, the motion recognition model is a gesture recognition model; the action recognition module recognizes the anchor action information existing in the anchor image frame sequence by utilizing an action recognition model, and is set as follows:
the motion recognition module determines hand position information and hand motion types of the anchor in the anchor image frame sequence by using the gesture recognition model, and determines the hand position information and the hand motion types as the anchor motion information.
Optionally, the model processing module determines whether there is motion control information based on the anchor motion information, the model location information, and the target simulation model, and is configured to:
the model processing module inputs the anchor action information, the model position information and the target simulation model into a simulation model control program to determine whether the action control information exists;
the model processing module generates a model control instruction corresponding to the motion control information, and is configured to:
the model processing model obtains the model control instruction output by the simulation model control program.
Optionally, the anchor end includes: the model control module and the model display module; the anchor terminal responds to the model control instruction, performs display control on the target simulation model, and is set as:
the model control module responds to the model control instruction and sends a corresponding display control instruction to the model display module;
and the model display module responds to the display control instruction and displays the target simulation model.
Optionally, the anchor end includes: a plug flow module; the mixed flow image frame sequence is sent to a user side and is set as follows:
and the plug-flow module sends the mixed-flow image frame sequence to the user side through a data distribution network.
Optionally, the target simulation model in the anchor terminal is sent by the mixed reality server.
The live broadcast processing method and system provided by the invention have the advantages that the live broadcast processing system comprises the anchor terminal and the mixed reality server. The anchor terminal sends the anchor image frame sequence and the model position information to a mixed reality server; the anchor image frame sequence is an image frame sequence which is acquired by an anchor terminal aiming at the anchor, and the model position information comprises the position information of a target simulation model on a display interface of the anchor terminal; the mixed reality server determines whether action control information aiming at the target simulation model exists or not based on the anchor image frame sequence and the model position information, if so, a corresponding model control instruction is generated, and the model control instruction is returned to the anchor terminal; the anchor terminal responds to the model control instruction, display control is carried out on the target simulation model, an image frame sequence of the controlled simulation model is obtained, mixed flow processing is carried out on the image frame sequence of the controlled simulation model and the anchor image frame sequence, a mixed flow image frame sequence is obtained, and the mixed flow image frame sequence is sent to the user side. The invention can combine the real live broadcast environment of the anchor and the digital environment of the target simulation model into a mixed environment, provide specific service for the user in the mixed environment, avoid the conditions that the target objects such as the real model cannot be effectively built or the effective operation demonstration cannot be carried out in the target objects due to the limitation problem (such as the problem of the quantity, the quality and the like of the props) when the real props are used in the real environment, ensure the service quality, bring the sensory experience beyond the reality for the user, improve the viewing experience of the user and enhance the viscosity of the user.
The foregoing description is only an overview of the technical solutions of the present invention, and the following detailed description of the present invention is provided to enable the technical means of the present invention to be more clearly understood, and to enable the above and other objects, features, and advantages of the present invention to be more clearly understood.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 shows a signaling diagram of a first live broadcast processing method according to an embodiment of the present invention;
fig. 2 shows a signaling diagram of a second live broadcast processing method according to an embodiment of the present invention;
fig. 3 is a signaling diagram illustrating a fourth live broadcast processing method according to an embodiment of the present invention;
fig. 4 shows a schematic structural diagram of a live broadcast processing system according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
As shown in fig. 1, this embodiment proposes a first live broadcast processing method, where the live broadcast processing method may be applied to a live broadcast processing system, and the live broadcast processing system may include: the system comprises a main broadcasting end and a mixed reality server; the live broadcast processing method can comprise the following steps:
s101, the anchor terminal sends an anchor image frame sequence and model position information to a mixed reality server; the anchor image frame sequence is an image frame sequence acquired by an anchor terminal aiming at an anchor, and the model position information comprises the position information of a target simulation model on a display interface of the anchor terminal;
the anchor terminal is an electronic device terminal used by the anchor for network live broadcast.
The anchor image frame sequence may be an image frame sequence acquired by an anchor terminal through a camera device to acquire an image of a real live scene of the anchor. The anchor image frame sequence may be a sequence of image frames consisting of at least one real scene image. It will be appreciated that each frame image in the anchor image frame sequence may comprise an anchor human body image.
The target simulation model may be a simulation model set up by using a virtual item to implement a certain service. For example, the target simulation model may be a circuit simulation model built using virtual bulbs, virtual switches, and virtual wires to impart physical knowledge to students.
Optionally, the target simulation model in the anchor end may be a pre-established simulation model;
optionally, the target simulation model in the anchor side may be sent to the anchor side by other sides or the electronic device.
Alternatively, the target simulation model in the anchor side may be sent by the mixed reality server.
Specifically, the model position information may include position information of each virtual item in the target simulation model on a display interface of the anchor terminal.
The mixed reality server can be a server for processing images of the anchor image frame sequence, and generating a model control instruction for the target simulation model based on the image processing result and the model position information so as to realize mixed reality and provide specific services for users.
Specifically, when the anchor needs to provide a specific service using a target simulation model on the anchor side, the anchor side may send a model obtaining instruction to the mixed reality server to obtain the target simulation model returned by the mixed reality server.
Specifically, when the anchor uses the target simulation model on the anchor side in the live broadcasting process, the anchor side can determine model position information of the target simulation model, for example, if the anchor is a teacher and the target simulation model is a circuit model, when the teacher uses the circuit model on the anchor side in the live broadcasting process of a church, the anchor side can determine model position information of the circuit model on the display interface; the anchor end may then send the collected anchor image frame sequence and the determined model location information to the mixed reality server.
S102, the mixed reality server determines whether action control information aiming at a target simulation model exists or not based on the anchor image frame sequence and the model position information, and if so, the step S103 is executed; otherwise, the step S103 is prohibited to avoid unnecessary resource consumption.
The action control information may include limb action information used by the anchor to control the target simulation model in the live broadcasting process.
It is to be understood that the motion control information may be present in the positional change information of the anchor human key point in each frame image of the anchor image frame sequence.
Specifically, the mixed reality server may identify, by using the human key point identification model, human key points of the anchor in each frame image of the anchor image frame sequence after receiving the anchor image frame sequence and the model position information, determine position information of the human key points in each frame image, extract position change information of the human key points from the position information of the human key points in each frame image, and then determine whether there is motion control information by analyzing the position change information of the human key points and the model position information, that is, determine whether the anchor performs effective motion control on the target simulation model. For example, if the anchor is a teacher, the target simulation model is a circuit model including a virtual bulb, a virtual wire, a virtual switch, and a virtual power supply, and each frame image of the anchor image frame sequence includes a manual closing motion of the teacher with respect to the virtual switch, the mixed reality server may extract position change information of a hand key point of the teacher from the anchor image frame sequence, and determine that motion control information, which is manual closing control information with respect to the virtual switch, exists based on the position change information of the hand key point and the model position information.
It should be noted that, if there is motion control information, the mixed reality server may execute a subsequent processing procedure for the motion control information to perform corresponding display control on the target simulation model; if the action control information exists, the mixed reality server can avoid executing subsequent steps, and consumption of related resources is avoided.
S103, generating a corresponding model control instruction by the mixed reality server;
the model control instruction is an instruction for performing corresponding display control on the target simulation model.
Specifically, if it is determined that motion control information exists, the mixed reality server may generate a corresponding model control instruction based on the motion control information.
S104, the mixed reality server returns the model control instruction to the anchor terminal;
specifically, the mixed reality server may return the generated model control instruction to the anchor terminal after generating the model control instruction.
S105, the anchor terminal responds to the model control instruction and performs display control on the target simulation model;
specifically, the anchor terminal may perform corresponding display control on the target simulation model based on the model control instruction after receiving the model control instruction. For example, if the anchor is a teacher, the target simulation model is a circuit model composed of a virtual bulb, a virtual wire, a virtual switch, and a virtual power supply, the model control command is a target command corresponding to manual closing control information for the virtual switch, and the target command includes a closing command for the virtual switch and a lighting command for the virtual bulb, the anchor may control the virtual switch in the circuit model to be in a closed state and control the virtual bulb to light after receiving the model control command.
S106, the anchor end obtains an image frame sequence of the controlled simulation model;
the controlled simulation model is the simulation model obtained after the corresponding display control is carried out on the target simulation model.
Specifically, the anchor terminal may acquire a sequence of image frames of the controlled simulation model after performing display control on the target simulation model.
S107, the anchor end performs mixed flow processing on the image frame sequence of the controlled simulation model and the anchor image frame sequence;
specifically, the anchor terminal may perform mixed flow processing on the image frame sequence of the controlled simulation model and the anchor image frame sequence after obtaining the image frame sequence of the controlled simulation model, so as to obtain an image frame sequence of the entire process including that the anchor operates the target simulation model in the live broadcast process, and the target simulation model is correspondingly changed in display.
Specifically, in the mixed flow processing process of the anchor terminal, the anchor image frame sequence may be used as a media source and placed on the bottom layer of the canvas, the whole canvas is paved, then the image frame sequence of the controlled simulation model is used as another media source and placed at a specified position on the bottom layer of the canvas, and then an image frame sequence with well mixed flow processing may be obtained.
Optionally, a mixed flow module may be integrated in the anchor end, and mixed flow processing is performed on the anchor image frame sequence and the image frame sequence of the controlled post-simulation model through the mixed flow module.
S108, acquiring a mixed flow image frame sequence by the anchor terminal;
wherein, the mixed flow image frame sequence is an image frame sequence which is processed by mixed flow.
And S109, the anchor end sends the mixed flow image frame sequence to the user end.
Specifically, the anchor terminal may send the mixed-flow image frame sequence to the user after obtaining the mixed-flow image frame sequence. It can be understood that the user terminal can obtain the corresponding service by playing the mixed flow image frame sequence.
Optionally, the anchor side may include: a plug flow module; at this time, step S109 may include:
and the plug-flow module sends the mixed-flow image frame sequence to the user side through a data distribution network.
The data distribution Network may be a Content Delivery Network (CDN) for live data.
Specifically, the stream pushing module may send the mixed-flow image frame sequence to a user end device for playing through the live CDN, so as to provide a specific service for the user.
It should be noted that the plug-flow module may also send the mixed-flow image frame sequence to the live CDN for storage. And then, the user end equipment can request the live broadcast CDN to acquire the mixed-flow image frame sequence and play the mixed-flow image frame sequence when needed.
It should be noted that, through steps S101 to S109 in fig. 1, the present invention may merge a real live broadcast environment in which a host broadcast is located and a digital environment in which a target simulation model is located into a mixed environment, provide a specific service to a user in the mixed environment, and avoid situations that a target object such as a model cannot be effectively built or an operation demonstration cannot be effectively performed in the target object due to a limitation problem (such as a problem of the number and quality of a property) when the real property is used in a real environment, thereby ensuring the quality of the service, bringing a sensory experience beyond reality to the user, improving the viewing experience of the user, and enhancing the user stickiness.
It should be noted that when the method is applied to a teaching live broadcast scene, the reducibility of a complex teaching scene, such as a complex physical problem teaching scene, can be enhanced, the teaching requirement is met, and the learning effect of students is guaranteed.
It can be understood that, when the anchor end collects the anchor image frame sequence, it can also collect the audio stream data of the anchor at the same time, that is, collect the audio and video data of the anchor in the real environment, including the anchor image frame sequence; of course, the anchor side may then only send the anchor image frame sequence in the audio-video data to the mixed reality server without sending the audio stream data therein. After the anchor terminal obtains the image frame sequence of the controlled simulation model, the anchor terminal can perform mixed flow processing on the image frame sequence of the controlled simulation model and the audio and video data to obtain mixed flow processed data and send the mixed flow processed data to the user terminal. At the moment, the user side can also listen to live broadcast voice of the main broadcast through the mixed flow processed data. At the moment, the invention can further improve the user experience and enhance the user viscosity.
The live broadcast processing method provided by the embodiment can be applied to a live broadcast processing system, and the live broadcast processing system comprises an anchor terminal and a mixed reality server. The anchor terminal sends the anchor image frame sequence and the model position information to a mixed reality server; the anchor image frame sequence is an image frame sequence which is acquired by an anchor terminal aiming at the anchor, and the model position information comprises the position information of a target simulation model on a display interface of the anchor terminal; the mixed reality server determines whether action control information aiming at the target simulation model exists or not based on the anchor image frame sequence and the model position information, if so, a corresponding model control instruction is generated, and the model control instruction is returned to the anchor terminal; the anchor terminal responds to the model control instruction, display control is carried out on the target simulation model, an image frame sequence of the controlled simulation model is obtained, mixed flow processing is carried out on the image frame sequence of the controlled simulation model and the anchor image frame sequence, a mixed flow image frame sequence is obtained, and the mixed flow image frame sequence is sent to the user side. The invention can combine the real live broadcast environment of the anchor and the digital environment of the target simulation model into a mixed environment, provide specific service for the user in the mixed environment, avoid the conditions that the target objects such as the real model cannot be effectively built or the effective operation demonstration cannot be carried out in the target objects due to the limitation problem (such as the problem of the quantity, the quality and the like of the props) when the real props are used in the real environment, ensure the service quality, bring the sensory experience beyond the reality for the user, improve the viewing experience of the user and enhance the viscosity of the user.
Corresponding to the method shown in fig. 1, as shown in fig. 2, the present embodiment proposes a second live broadcast processing method. In the method, the mixed reality server includes: the system comprises an action recognition module, a model processing module and a target simulation model; at this time, step S102 may include steps S201, S202, and S203, in which:
s201, the action recognition module recognizes anchor action information existing in an anchor image frame sequence by utilizing an action recognition model;
wherein the anchor motion information may comprise anchor limb motion information in an anchor image frame sequence.
Specifically, the motion recognition model may perform image processing on each frame image of the anchor image frame sequence, recognize an anchor human key point in each frame image, determine position information of the human key point in each frame image, and determine the anchor limb motion information based on the position information of the human key point in each frame image, that is, determine the anchor motion information.
Optionally, in other live broadcast processing methods provided in this embodiment, the motion recognition model is a gesture recognition model; step S201 may include:
the motion recognition module determines hand position information and hand motion types of a anchor in an anchor image frame sequence by utilizing a gesture recognition model, and determines the hand position information and the hand motion types as anchor motion information.
Specifically, the gesture recognition model may recognize a hand key point in each frame image of the anchor image frame sequence, determine hand key point position information, that is, the hand position information, in each frame image, extract hand key point position variation information from the hand key point position information in each frame image, and determine a corresponding hand motion category based on the hand key point position variation information.
Optionally, the hand action categories may include click and drag. For example, in a physical teaching application scenario, when the anchor is a teacher and the target simulation model is a circuit model composed of a virtual bulb, a virtual wire, a virtual switch, and a virtual power supply, the click may be a click action performed on a certain physical prop, and the drag may be an action of dragging the certain physical prop from a certain position to another position.
Specifically, the motion recognition module may determine the hand position information and the hand motion category determined by the gesture recognition model as anchor motion information.
S202, the action recognition module sends the anchor action information to the model processing module;
specifically, the action recognition module may send the anchor action information to the model processing module for information processing after obtaining the anchor action information.
S203, the model processing module determines whether action control information exists or not based on the anchor action information, the model position information and the target simulation model;
specifically, the model processing module may determine whether action control information exists based on the anchor action information, the model location information, and the target simulation model after receiving the anchor action information.
At this time, step S103 may include step S204, in which:
and S204, generating a model control command corresponding to the motion control information by the model processing module.
Specifically, the model processing module may generate a corresponding model control instruction based on the motion control information after determining that the motion control information exists.
It should be noted that, by the arrangement of the action recognition module and the model processing module, the invention can effectively ensure the recognition of the mixed reality server to the anchor action information and the action control information, and the generation of the model control instruction, and ensure the function realization of the mixed reality server, thereby ensuring the mixed processing effect and the subsequent service quality.
According to the live broadcast processing method provided by the embodiment, the recognition of the mixed reality server to the anchor action information and the action control information and the generation of the model control instruction are effectively guaranteed through the arrangement of the action recognition module and the model processing module, and the function realization of the mixed reality server is guaranteed, so that the mixed processing effect and the subsequent service quality are guaranteed.
Based on fig. 2, the present embodiment proposes a third live broadcast processing method. In the method, step S203 may include:
the model processing module inputs the anchor action information, the model position information and the target simulation model into a simulation model control program to determine whether action control information exists or not;
at this time, step S204 may include:
the model processing model obtains model control instructions output by the simulation model control program.
The simulation model control program may be a control program of the target simulation model.
Specifically, the model processing module may be integrated with a simulation model control program.
Specifically, the model processing module may input the anchor action information, the model position information, and the target simulation model into the simulation model control program, and determine whether the action control information exists through the simulation model control program.
Specifically, if the simulation model control program determines that motion control information exists, the simulation model control program may generate a model control command corresponding to the motion control information.
It should be noted that, the present invention can effectively ensure that the model processing module realizes the recognition of the action control information and the generation of the model control instruction through the setting of the simulation model control program, and further ensure the subsequent mixed flow processing effect and the user service quality.
The live broadcast processing method provided by the embodiment can effectively ensure that the model processing module realizes the recognition of the action control information and the generation of the model control instruction through the setting of the simulation model control program, and further ensure the subsequent mixed flow processing effect and the user service quality.
Based on fig. 1, as shown in fig. 3, the present embodiment proposes a fourth live broadcast processing method. In the method, the anchor may include: the model control module and the model display module; at this time, step S105 may include:
s301, the model control module responds to the model control instruction and sends a corresponding display control instruction to the model display module;
the model control module can be a control module integrated by the anchor terminal and aiming at the target simulation model.
The model display module can be a module which is integrated by the anchor terminal and is used for displaying the target simulation model.
Specifically, the model control module may generate a display control instruction corresponding to the model control instruction and send the display control instruction to the model display instruction when receiving the model control instruction.
And S302, the model display module responds to the display control instruction and displays the target simulation model.
Specifically, the model display module may perform corresponding display changes on the target simulation model when receiving the display control instruction.
It should be noted that, by setting the model control module and the model display module, the invention can realize the control and display of the anchor terminal on the target simulation model, and further ensure the subsequent mixed flow processing effect and the user service quality.
According to the live broadcast processing method provided by the embodiment, the control and display of the anchor terminal on the target simulation model can be realized through the arrangement of the model control module and the model display module, and the subsequent mixed flow processing effect and the user service quality are further ensured.
Corresponding to the method shown in fig. 1, as shown in fig. 4, the present embodiment provides a live broadcast control system. The system may include: a anchor 101 and a mixed reality server 102; wherein:
the anchor terminal 101 sends the anchor image frame sequence and the model position information to the mixed reality server 102; the anchor image frame sequence is an image frame sequence acquired by an anchor terminal 101 aiming at an anchor, and the model position information comprises position information of a target simulation model on a display interface of the anchor terminal 101;
the mixed reality server 102 determines whether action control information aiming at the target simulation model exists or not based on the anchor image frame sequence and the model position information, if so, generates a corresponding model control instruction, and returns the model control instruction to the anchor terminal 101;
the anchor terminal 101 responds to the model control instruction, performs display control on the target simulation model, and obtains an image frame sequence of the controlled simulation model; carrying out mixed flow treatment on the image frame sequence of the controlled simulation model and the anchor image frame sequence to obtain a mixed flow image frame sequence; and sending the mixed flow image frame sequence to a user side.
It should be noted that, for specific processing procedures of the anchor terminal 101 and the mixed reality server 102 and technical effects brought by the processing procedures, reference may be made to the related description of steps S101 to S109 in fig. 1 in this embodiment, and details are not described here again.
Optionally, the mixed reality server 102 includes: the system comprises an action recognition module, a model processing module and a target simulation model; the mixed reality server 102 determines whether motion control information for the target simulation model exists based on the anchor image frame sequence and the model location information, and is set to:
the action recognition module recognizes anchor action information existing in the anchor image frame sequence by utilizing an action recognition model and sends the anchor action information to the model processing module;
the model processing module determines whether action control information exists or not based on the anchor action information, the model position information and the target simulation model;
generating a corresponding model control command, which is set as:
the model processing module generates a model control command corresponding to the motion control information.
Optionally, the motion recognition model is a gesture recognition model; the action recognition module recognizes the anchor action information existing in the anchor image frame sequence by utilizing an action recognition model, and is set as follows:
the motion recognition module determines hand position information and hand motion types of a anchor in an anchor image frame sequence by using a gesture recognition model, and determines the hand position information and the hand motion types as anchor motion information.
Optionally, the model processing module determines whether there is motion control information based on the anchor motion information, the model position information, and the target simulation model, and sets the following settings:
the model processing module inputs the anchor action information, the model position information and the target simulation model into a simulation model control program to determine whether action control information exists or not;
the model processing module generates a model control instruction corresponding to the motion control information, and is configured to:
the model processing model obtains model control instructions output by the simulation model control program.
Optionally, the anchor 101 includes: the model control module and the model display module; the anchor terminal 101 responds to the model control instruction, performs display control on the target simulation model, and sets the display control to:
the model control module responds to the model control instruction and sends a corresponding display control instruction to the model display module;
the model display module responds to the display control instruction and displays the target simulation model.
Optionally, the anchor 101 includes: a plug flow module; sending the mixed flow image frame sequence to a user side, and setting as follows:
and the plug-flow module sends the mixed-flow image frame sequence to the user side through a data distribution network.
Optionally, the target simulation model in the anchor 101 is sent by the mixed reality server 102.
The live broadcast processing system provided by this embodiment may include an anchor 101 and a mixed reality server 102. The anchor terminal 101 sends the anchor image frame sequence and the model position information to the mixed reality server 102; the anchor image frame sequence is an image frame sequence acquired by an anchor terminal 101 aiming at an anchor, and the model position information comprises position information of a target simulation model on a display interface of the anchor terminal 101; the mixed reality server 102 determines whether action control information aiming at the target simulation model exists or not based on the anchor image frame sequence and the model position information, if so, generates a corresponding model control instruction, and returns the model control instruction to the anchor terminal 101; the anchor terminal 101 responds to the model control instruction, performs display control on the target simulation model, obtains an image frame sequence of the controlled simulation model, performs mixed flow processing on the image frame sequence of the controlled simulation model and the anchor image frame sequence, obtains a mixed flow image frame sequence, and sends the mixed flow image frame sequence to the user terminal. The invention can combine the real live broadcast environment of the anchor and the digital environment of the target simulation model into a mixed environment, provide specific service for the user in the mixed environment, avoid the conditions that the target objects such as the real model cannot be effectively built or the effective operation demonstration cannot be carried out in the target objects due to the limitation problem (such as the problem of the quantity, the quality and the like of the props) when the real props are used in the real environment, ensure the service quality, bring the sensory experience beyond the reality for the user, improve the viewing experience of the user and enhance the viscosity of the user.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A live broadcast processing method is applied to a live broadcast processing system, and the live broadcast processing system comprises: the system comprises a main broadcasting end and a mixed reality server; the live broadcast processing method comprises the following steps:
the anchor terminal sends the anchor image frame sequence and the model position information to a mixed reality server; the anchor image frame sequence is an image frame sequence collected by an anchor terminal aiming at an anchor, and the model position information comprises position information of various virtual props in a target simulation model on a display interface of the anchor terminal; the target simulation model is a simulation model built by using the virtual prop;
the mixed reality server determines whether action control information aiming at the target simulation model exists or not based on anchor action information and the model position information extracted from the anchor image frame sequence, if so, generates a corresponding model control instruction, and returns the model control instruction to the anchor end; the action control information comprises limb action information used by the anchor to control the target simulation model in the live broadcasting process;
the anchor terminal responds to the model control instruction and performs display control on the target simulation model to obtain an image frame sequence of the controlled simulation model; performing mixed flow processing on the image frame sequence of the controlled simulation model and the anchor image frame sequence to obtain a mixed flow image frame sequence; and sending the mixed flow image frame sequence to a user side.
2. The live processing method according to claim 1, wherein the mixed reality server includes: the system comprises an action recognition module, a model processing module and the target simulation model; the mixed reality server determining whether motion control information for the target simulation model exists based on anchor motion information and the model position information extracted from the anchor image frame sequence, including:
the action recognition module recognizes anchor action information existing in the anchor image frame sequence by utilizing an action recognition model, and sends the anchor action information to the model processing module;
the model processing module determines whether action control information exists based on the anchor action information, the model location information, and the target simulation model;
the generating of the corresponding model control instruction comprises:
the model processing module generates a model control instruction corresponding to the motion control information.
3. The live processing method according to claim 2, wherein the motion recognition model is a gesture recognition model; the action recognition module recognizes the anchor action information existing in the anchor image frame sequence by utilizing an action recognition model, and the action recognition module comprises the following steps:
the motion recognition module determines hand position information and hand motion types of the anchor in the anchor image frame sequence by utilizing the gesture recognition model, and determines the hand position information and the hand motion types as the anchor motion information.
4. The live processing method of claim 2, wherein the model processing module determines whether action control information is present based on the anchor action information, the model location information, and the target simulation model, and comprises:
the model processing module inputs the anchor action information, the model position information and the target simulation model into a simulation model control program to determine whether the action control information exists;
the model processing module generates a model control instruction corresponding to the motion control information, including:
the model processing module obtains the model control instruction output by the simulation model control program.
5. The live broadcast processing method according to claim 1, wherein the anchor side includes: the model control module and the model display module; the anchor terminal responds to the model control instruction and performs display control on the target simulation model, and the method comprises the following steps:
the model control module responds to the model control instruction and sends a corresponding display control instruction to the model display module;
and the model display module responds to the display control instruction and displays the target simulation model.
6. The live broadcast processing method according to claim 1, wherein the anchor side includes: a plug flow module; the sending the mixed flow image frame sequence to a user side comprises:
and the plug-flow module sends the mixed-flow image frame sequence to the user side through a data distribution network.
7. The live processing method according to claim 1, wherein the target simulation model in the anchor side is sent by the mixed reality server.
8. A live processing system, comprising: the system comprises a main broadcasting end and a mixed reality server; wherein:
the anchor terminal sends the anchor image frame sequence and the model position information to a mixed reality server; the anchor image frame sequence is an image frame sequence collected by an anchor terminal aiming at an anchor, and the model position information comprises position information of various virtual props in a target simulation model on a display interface of the anchor terminal; the target simulation model is a simulation model built by using the virtual prop;
the mixed reality server determines whether action control information aiming at the target simulation model exists or not based on anchor action information and the model position information extracted from the anchor image frame sequence, if so, generates a corresponding model control instruction, and returns the model control instruction to the anchor end; the action control information comprises limb action information used by the anchor to control the target simulation model in the live broadcasting process;
the anchor terminal responds to the model control instruction and performs display control on the target simulation model to obtain an image frame sequence of the controlled simulation model; performing mixed flow processing on the image frame sequence of the controlled simulation model and the anchor image frame sequence to obtain a mixed flow image frame sequence; and sending the mixed flow image frame sequence to a user side.
9. The live processing system of claim 8, wherein the mixed reality server comprises: the system comprises an action recognition module, a model processing module and the target simulation model; the mixed reality server determines whether action control information aiming at the target simulation model exists or not based on anchor action information and the model position information extracted from the anchor image frame sequence, and the action control information is set as follows:
the action recognition module recognizes anchor action information existing in the anchor image frame sequence by utilizing an action recognition model, and sends the anchor action information to the model processing module;
the model processing module determines whether action control information exists based on the anchor action information, the model position information and the target simulation model;
the generating of the corresponding model control command is set as:
the model processing module generates a model control instruction corresponding to the motion control information.
10. The live processing system of claim 9, wherein the motion recognition model is a gesture recognition model; the action recognition module recognizes the anchor action information existing in the anchor image frame sequence by utilizing an action recognition model, and is set as follows:
the motion recognition module determines hand position information and hand motion types of the anchor in the anchor image frame sequence by utilizing the gesture recognition model, and determines the hand position information and the hand motion types as the anchor motion information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210903928.5A CN114979698B (en) | 2022-07-29 | 2022-07-29 | Live broadcast processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210903928.5A CN114979698B (en) | 2022-07-29 | 2022-07-29 | Live broadcast processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114979698A CN114979698A (en) | 2022-08-30 |
| CN114979698B true CN114979698B (en) | 2023-01-06 |
Family
ID=82969499
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210903928.5A Active CN114979698B (en) | 2022-07-29 | 2022-07-29 | Live broadcast processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114979698B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107680157A (en) * | 2017-09-08 | 2018-02-09 | 广州华多网络科技有限公司 | It is a kind of based on live interactive approach and live broadcast system, electronic equipment |
| CN107682729A (en) * | 2017-09-08 | 2018-02-09 | 广州华多网络科技有限公司 | It is a kind of based on live interactive approach and live broadcast system, electronic equipment |
| CN110351514A (en) * | 2019-07-09 | 2019-10-18 | 北京猫眼视觉科技有限公司 | A kind of method that dummy model passes through remote assistance mode and video flowing simultaneous transmission |
| CN111443619A (en) * | 2020-04-17 | 2020-07-24 | 南京工程学院 | Virtual-real fused human-computer cooperation simulation method and system |
| CN113507621A (en) * | 2021-07-07 | 2021-10-15 | 上海商汤智能科技有限公司 | Live broadcast method, device, system, computer equipment and storage medium |
-
2022
- 2022-07-29 CN CN202210903928.5A patent/CN114979698B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107680157A (en) * | 2017-09-08 | 2018-02-09 | 广州华多网络科技有限公司 | It is a kind of based on live interactive approach and live broadcast system, electronic equipment |
| CN107682729A (en) * | 2017-09-08 | 2018-02-09 | 广州华多网络科技有限公司 | It is a kind of based on live interactive approach and live broadcast system, electronic equipment |
| CN110351514A (en) * | 2019-07-09 | 2019-10-18 | 北京猫眼视觉科技有限公司 | A kind of method that dummy model passes through remote assistance mode and video flowing simultaneous transmission |
| CN111443619A (en) * | 2020-04-17 | 2020-07-24 | 南京工程学院 | Virtual-real fused human-computer cooperation simulation method and system |
| CN113507621A (en) * | 2021-07-07 | 2021-10-15 | 上海商汤智能科技有限公司 | Live broadcast method, device, system, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114979698A (en) | 2022-08-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110570698B (en) | Online teaching control method and device, storage medium and terminal | |
| CN111654715B (en) | Live video processing method and device, electronic equipment and storage medium | |
| CN112135160A (en) | Virtual object control method and device in live broadcast, storage medium and electronic equipment | |
| CN104837051B (en) | Video broadcasting method and client | |
| CN112135158B (en) | Live broadcasting method based on mixed reality and related equipment | |
| US11842425B2 (en) | Interaction method and apparatus, and electronic device and computer-readable storage medium | |
| CN109922352A (en) | A kind of data processing method, device, electronic equipment and readable storage medium storing program for executing | |
| CN110930220A (en) | Display method, display device, terminal equipment and medium | |
| CN114339285A (en) | Knowledge point processing method, video processing method and device and electronic equipment | |
| CN112351291A (en) | Teaching interaction method, device and equipment based on AI portrait segmentation | |
| CN113191184A (en) | Real-time video processing method and device, electronic equipment and storage medium | |
| CN114679628B (en) | Bullet screen adding method and device, electronic equipment and storage medium | |
| CN113965813A (en) | Video playing method and system in live broadcast room and computer equipment | |
| CN112702613A (en) | Live video recording method and device, storage medium and electronic equipment | |
| CN114143572A (en) | Live broadcast interaction method and device, storage medium and electronic equipment | |
| CN114979698B (en) | Live broadcast processing method and system | |
| CN108268520B (en) | Courseware control method and device and online course live broadcast system | |
| CN111107283B (en) | Information display method, electronic equipment and storage medium | |
| CN114449355B (en) | Live interaction method, device, equipment and storage medium | |
| CN113641853A (en) | Dynamic cover generation method, device, electronic equipment, medium and program product | |
| CN108076348B (en) | Video playing control method and device | |
| CN114846808A (en) | Content distribution system, content distribution method, and content distribution program | |
| CN105208453A (en) | Method and device for displaying interaction feedback information of television interaction system | |
| CN111866548A (en) | Marking method applied to medical video | |
| CN108939539A (en) | Storage medium, game control method, terminal and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |