CN112995692B - Interactive data processing method, device, equipment and medium - Google Patents
Interactive data processing method, device, equipment and medium Download PDFInfo
- Publication number
- CN112995692B CN112995692B CN202110240735.1A CN202110240735A CN112995692B CN 112995692 B CN112995692 B CN 112995692B CN 202110240735 A CN202110240735 A CN 202110240735A CN 112995692 B CN112995692 B CN 112995692B
- Authority
- CN
- China
- Prior art keywords
- data
- graphic
- interaction
- interactive
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
- H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
- H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the application provides an interactive data processing method, device, equipment and medium. After receiving the interaction request, the server generates graphic interaction data corresponding to the target interaction function through a graphic engine; and pushed to the user terminal of the communication connection. Thus, a live broadcast mode for producing graphic interaction data by the server is provided.
Description
Technical Field
The present invention relates to the field of computers, and in particular, to an interactive data processing method, apparatus, device, and medium.
Background
In the current live broadcast interaction scene, live broadcast video for pushing to audience needs to be produced by a live host. Therefore, in the conventional live broadcast manner, the output of the video content is seriously dependent on the anchor, so that the interaction flow is too single to match the user requirement.
Disclosure of Invention
To overcome at least one of the disadvantages in the prior art, in a first aspect, an embodiment of the present application provides an interactive data processing method, which is applied to a server, where the server is communicatively connected to at least one user terminal, and the method includes:
Receiving an interaction request of the user terminal, wherein the interaction request comprises a target interaction function;
invoking a graphic engine to obtain graphic interaction data matched with the target interaction function;
and sending the graphic interaction data to the user terminal so that the user terminal outputs the graphic interaction data.
In a second aspect, an embodiment of the present application provides an interactive data processing method, applied to a user terminal communicatively connected to a server, where the method includes:
providing an interactive interface, wherein the interactive interface is displayed with a configuration control;
responding to the configuration operation of the configuration control, and sending an interaction request to the server, wherein the interaction configuration request carries a target interaction function;
receiving graphic interaction data matched with the target interaction function and sent by the server;
and displaying the graphic interaction data on the interaction interface.
In a third aspect, an embodiment of the present application provides an interactive data processing apparatus, which is applied to a server, where the server is communicatively connected to at least one user terminal, and the interactive data processing apparatus includes:
the request receiving module is used for receiving an interaction request of the user terminal, wherein the interaction request comprises a target interaction function;
The data generation module is used for calling a graphic engine to obtain graphic interaction data matched with the target interaction function;
and the data sending module is used for sending the graphic interaction data to the user terminal so that the user terminal outputs the graphic interaction data.
In a fourth aspect, an embodiment of the present application provides an electronic device, where the electronic device includes a processor and a memory, where the memory stores a computer program, and the computer program implements the interactive data processing method when executed by the processor.
In a fifth aspect, embodiments of the present application provide a storage medium storing a computer program, where the computer program when executed by a processor implements the interactive data processing method.
Compared with the prior art, the application has the following beneficial effects:
the embodiment of the application provides an interactive data processing method, device, equipment and medium. After receiving the interaction request, the server generates graphic interaction data corresponding to the target interaction function through a graphic engine; and pushed to the user terminal of the communication connection. Thus, a live broadcast mode for producing graphic interaction data by the server is provided.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered limiting the scope, and that other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic view of a scenario provided in an embodiment of the present application;
fig. 2 is a schematic diagram of a server structure provided in an embodiment of the present application;
FIG. 3 is a flowchart illustrating steps of an interactive data processing method according to an embodiment of the present disclosure;
FIG. 4 is a second flowchart illustrating steps of an interactive data processing method according to an embodiment of the present disclosure;
fig. 5 is a schematic diagram of resource scheduling according to an embodiment of the present application;
FIG. 6 is a third flowchart illustrating steps of an interactive data processing method according to an embodiment of the present disclosure;
FIG. 7 is a flowchart illustrating steps of an interactive data processing method according to an embodiment of the present disclosure;
fig. 8 is a schematic diagram of data transmission provided in an embodiment of the present application;
FIG. 9 is a flowchart illustrating steps of an interactive data processing method according to an embodiment of the present disclosure;
Fig. 10 is a schematic diagram of sound and picture synchronization provided in the embodiment of the present application;
FIG. 11 is a flowchart illustrating steps of an interactive data processing method according to an embodiment of the present disclosure;
FIG. 12 is a flowchart of steps of an interactive data processing method according to an embodiment of the present disclosure;
FIG. 13 is a schematic diagram of an interactive interface provided in an embodiment of the present application;
FIG. 14 is a schematic diagram of a display effect according to an embodiment of the present disclosure;
FIG. 15 is a second schematic diagram of a display effect provided in an embodiment of the present application;
fig. 16 is a schematic diagram of an interactive data processing apparatus according to an embodiment of the present application.
Icon: 100-server; 200-audience terminals; 300-anchor terminal; 120-memory; 130-a processor; 140-communication means; 401-a request receiving module; 402-a data generation module; 403-a data transmission module; 501-preview icon; 502-content icon; 601-live pictures; 602-avatar; 701-a first graphical region; 702-second graphic region.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, as provided in the accompanying drawings, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
In the description of the present application, it should be noted that, directions or positional relationships indicated by terms such as "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., are directions or positional relationships based on those shown in the drawings, or are directions or positional relationships that are conventionally put in use of the inventive product, are merely for convenience of description of the present application and simplification of description, and do not indicate or imply that the apparatus or element to be referred to must have a specific direction, be configured and operated in a specific direction, and thus should not be construed as limiting the present application. Furthermore, the terms "first," "second," "third," and the like are used merely to distinguish between descriptions and should not be construed as indicating or implying relative importance.
In the related art, live video for pushing to the audience needs to be produced by a live host. Thus, traditional live approaches rely heavily on the anchor.
As shown in fig. 1, in the conventional live interaction scenario, it is necessary to include a anchor terminal 300, a server 100 and an audience terminal 200, and the anchor terminal 300, the server 100 and the audience terminal 200 are connected through network communication.
Taking live game broadcasting as an example, the host can capture the screen of the host terminal 300 through live broadcast software running on the host terminal 300, and then encode the screen into a live video stream to be pushed to the server 100. The server 100 pushes the live video stream transmitted from the anchor terminal 300 to the viewer terminal 200.
Alternatively, in singing live broadcast, the anchor may collect live video through an image collection device of the anchor terminal 300 and then push the live video to the server 100. The server 100 pushes the live video transmitted from the anchor terminal 300 to the viewer terminal 200.
Therefore, in the conventional live interaction scenario, live videos are all generated by the anchor terminal 300, and the video content is seriously dependent on the anchor, so that the interaction flow is too single and cannot be matched with the user requirement.
In view of this, the embodiment of the application provides an interactive data processing method applied to a server, which is used for generating live video through the server so as to achieve the purpose of providing a new live interaction mode.
The server 100 may be, but is not limited to, a Web server, an FTP (File Transfer Protocol ) server, a data server, and the like. The graphics engine may be, but is not limited to, a fantasy engine, a frost engine, a units engine, and the like.
For convenience and clarity of explanation of the interactive data processing method, the structure of the server is described below, and as shown in fig. 2, the server includes a memory 120, a processor 130, and a communication device 140.
The memory 120, the processor 130, and the communication device 140 are electrically connected directly or indirectly to each other to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines. The memory 120 stores a computer program. The processor 130 implements the interactive data processing method when executing the computer program stored in the memory 120.
The Memory 120 may be, but is not limited to, a random access Memory (Random Access Memory, RAM), a Read Only Memory (ROM), a programmable Read Only Memory (Programmable Read-Only Memory, PROM), an erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), an electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc. The memory 120 is used for storing a program, and the processor 130 executes the program after receiving an execution instruction. The communication device 140 is used for transmitting and receiving data through a network.
The processor 130 may be an integrated circuit chip with signal processing capabilities. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Based on the above-mentioned structure of the server, the steps of the method including the interactive data processing method shown in fig. 3 will be described in detail. The server is in communication connection with at least one user terminal, and the interactive data processing method comprises the following steps:
step S1, an interaction request of a user terminal is received.
Wherein the interactive request includes a target interactive function. As one possible implementation, the target function may be used to instruct the server to render video data as well as audio data that generated the target content through the graphics engine.
The user terminals may include, among other things, anchor terminals and/or audience terminals. When the user terminal is a main broadcasting terminal, the corresponding user can be a live broadcasting platform signed up live broadcasting host, or can be an operator of the live broadcasting platform.
For example, when the user is an operator of the live platform, the operator of the live platform may be understood as a live anchor, access the server through the anchor terminal, send an interaction request to the server, and instruct the server to generate video data and audio data of the target content.
When the user is a host contracted by the live broadcasting platform, the host can access the server through the host terminal, send an interaction request to the server, instruct the server to generate video data and audio data of target content, and push the video data and the audio data to the audience terminal entering the host live broadcasting room.
Of course, in some implementations, the interactive request may also be initiated by the viewer terminal.
In addition, the anchor terminal and the audience terminal may be, but are not limited to, smartphones, personal computers (personal computer, PCs), tablet computers, personal digital assistants (personal digital assistant, PDAs), mobile internet devices (mobile Internet device, MIDs), and the like.
And S2, calling a graphic engine to obtain graphic interaction data matched with the target interaction function.
The graphic engine can be used for rendering and generating various virtual pictures, and the server can develop live broadcasting in various scene forms by utilizing the generated virtual pictures. As shown in fig. 4, an implementation of step S2 includes:
s21, calling a graphic engine to obtain the audio data and the video data matched with the target interaction function.
S22, encoding the audio data and the video data to obtain the graphic interaction data with synchronous audio and video.
Taking the product of AI reading by the avatar as an example, the server may render a live view of a story-telling by the avatar through a graphic engine. Since the graphic engine can generate various virtual pictures, in the embodiment of the application, the server can render the virtual figures corresponding to the main angles for playing according to the story main angles in the target content.
Taking "Christian mountain Coke" as an example, when the role mainly related to the story content is Aide Mengtang Tes, virtual image data of Aide Mengtang Tes and audio data of the story content are rendered through a graphic engine; then, the avatar data and the audio data are encoded into graphic interactive data to be pushed to the user terminal, so that the avatar expressed as Ads Mengtang Tass in the picture content seen by the user of the user terminal is broadcasting "Christian mountain Coke".
Correspondingly, when the role mainly related to the story content is tangglare, generating virtual image data of the tangglare and audio data of the story content through a graphic engine; then, the avatar data and the audio data are encoded to be pushed to the user terminal, so that the avatar expressed as tangglare in the picture content seen by the user of the user terminal is broadcasted in "Christian mountain Comte".
Of course, the aforementioned "Christian mountain Coke" is just one example provided by embodiments of the present application, and matching avatars may also be rendered based on other story content.
And step S3, the graphic interaction data is sent to the user terminal, so that the user terminal outputs the graphic interaction data.
Therefore, in the above steps of the interactive data processing method, after receiving the interactive request, the server generates graphic interactive data corresponding to the target interactive function through the graphic engine; and pushed to the user terminal of the communication connection. Thus, a live broadcast mode for producing graphic interaction data by the server is provided.
In another possible implementation, the server may change different virtual visuals for the same story content, or change different story content based on the same virtual visuals. In order to realize the functions, the server can provide a configuration interface for displaying virtual picture types for a user to configure or automatically change according to preset matching rules.
For example, also taking the example of Aide Mengtang Tes in the aforementioned "Du mountain Coke", the server, upon detecting a holiday, may render an Aide Mengtang Tes avatar carrying the holiday feature through the graphics engine.
Therefore, the server can flexibly adjust the display content of the graphic interaction data in a mode of generating the graphic interaction data by the graphic engine, and the adaptability to various live scenes is improved. And the method is not limited by the work and rest habits of individuals of the live anchor, and can be used at any time.
In order to enable the graphic interaction data to provide good interaction effect when being displayed. The server may also provide corresponding interactions for specific speech segments in the audio data. When the server recognizes a specific voice clip from the audio data, an avatar corresponding to the interactive action is rendered through the image engine.
For example, the speech segment "hot applause" may correspond to a particular applause, and the speech segment "cheering" may correspond to a particular dance.
Also, considering that the graphic engine needs to perform a lot of mathematical operations when rendering the virtual picture, the normal operation of the graphic engine depends on sufficient hardware resources.
To this end, the server may include a plurality of compute nodes, wherein each compute node is configured with GPU (Graphics Processing Unit, graphics processor) resources for running the graphics engine. Of course, the server may also invoke other servers, compute nodes, or GPU resources in the compute cluster to run the graphics engine.
Based on the compute nodes, the server schedules GPU resources for the graphics engine according to the system block diagram shown in FIG. 5. As shown in fig. 5, after receiving an interaction request of a user terminal, the server performs GPU resource scheduling according to the interaction request, and searches whether a computing node meeting a resource requirement exists in a plurality of computing nodes.
If the computing node meeting the resource requirement exists, the server starts a graphic engine on the computing node and outputs video data and audio data of the target content.
And the server performs cloud video processing on the video data and the audio data output by the graphic engine to obtain live video streams encoded by the video data and the audio data, and pushes the live video streams to an audio-video live broadcasting room. For example, the server acquires video data and audio numbers output by the graphic engine through the cloud processing SDK, and encodes them.
Therefore, the graphics engine can normally operate in a GPU resource scheduling mode so as to ensure the flow and definition of the live video.
On this basis, as another implementation manner, the target interaction function may be a cloud interaction function provided by a server for at least one user terminal.
In order to realize interaction with the cloud interaction function provided by the server, the interaction request sent by the user terminal also comprises an interaction instruction. And after the server receives the large interaction request, calling a graphic engine to execute the interaction instruction, and then obtaining graphic interaction data matched with the cloud interaction function.
For example, the server is deployed with an interactive game that relies on a graphics engine so that the interactive game that would otherwise need to run locally runs on the server side. Therefore, the user terminal with general performance can play the interactive game with high performance requirement by sending the interactive instruction provided by the interactive game to the server. The interaction instruction is used for realizing remote control of game roles in the interaction game.
In addition, the interactive game which is needed to be run locally is run on the server side, so that a user can play the interactive game of different platforms through the user terminal of the same platform. For example, when the user terminal is an Android platform, the server may deploy an interactive game of the Windows platform to enable the user to play the interactive game of the Windows platform through the Android platform.
Optionally, the server may also live the graphical interaction data matched with the cloud interaction function. At this time, the user terminals participating in the cloud interaction through the interaction instruction may be regarded as anchor terminals, and the user terminals viewing the graphic interaction data may be regarded as audience terminals.
Also taking the above interactive game as an example, it is assumed that the interactive game supports 4 anchor players simultaneously online. The server generates a corresponding game picture through a graphic engine in the process that the 4 anchor participates in game interaction through the anchor terminal; the game frames are then encoded into live video for pushing to the anchor terminal, and to the audience terminal entering the live room.
Thus, the user of the audience terminal can watch the live broadcast picture when the main broadcasting game interacts. In addition, as the interactive game runs on the server, the anchor can conveniently invite other online anchors to participate in game interaction together.
In addition, in some interactive games, there is a difference in the viewing angle of the picture between different players. Aiming at the interactive games, in the related live broadcast technology, the interactive games are locally operated as a main broadcast of one player, and then recorded screen pictures are encoded into live video streams to be pushed to audience terminals. Since the live video stream originates from only one end of the anchor, the spectator cannot view the game frames from other players' perspectives.
In view of this, the server outputs game pictures under different anchor viewing angles through the graphic engine; and splicing game pictures under different main broadcasting viewing angles into a block for coding, and pushing the live video stream obtained by coding to a viewer terminal entering a live broadcasting room.
Also taking the above interactive game as an example, 4 anchor is anchor a, anchor B, anchor C and anchor D, respectively. Wherein, the game visual angles among the main broadcasters are different, so that the game pictures watched by the main broadcasters are different.
At this time, the server acquires game pictures under each anchor viewing angle through the graphic engine, and then splices together to obtain a spliced picture. The spliced picture comprises 4 areas, namely a 1 main area and 3 auxiliary areas, and the area of the main area is the largest. Before the play, the game pictures under one of the main play angles can occupy the main area, and the game pictures under the other main play angles occupy the auxiliary area.
Thus, the user of the audience terminal can watch the game pictures under 4 main broadcasting viewing angles at the same time.
To obtain video data and audio data output by the graphics engine, the game engine may be provided with at least 2 APIs, a first API and a second API (Application Programming Interface, application program interface), respectively. Referring to fig. 6, a possible implementation manner of obtaining the first API and the second API is provided, and an implementation manner of the step S21 includes:
step S211, the first API and the second API are configured for the graphics engine.
In step S212, audio data generated by the graphics engine based on the target interactive function is obtained through the first API.
In step S213, the video data generated by the graphics engine based on the target interactive function is obtained through the second API.
Note that, for steps S212 and S213, the sequence of the steps is not limited.
In order to facilitate data interaction and signaling interaction with the graphics engine, the first API and the second API need to be configured before the graphics engine outputs the audio data and the video data, and the configuration flow is shown in fig. 7.
The server sends a start instruction to the graphics engine. The starting instruction carries an RPC port for constructing the first API.
The graphic engine establishes a channel corresponding to the first API through the RPC port. The graphic engine can send a heartbeat message to the server or receive a notification event sent by the server through a channel corresponding to the first API. The heartbeat message is used for informing the server of the current running state of the graphic engine, and the notification event comprises events such as coding start and stop, pushing flow start and stop, processing progress, task start and stop and the like, and corresponding processing is carried out on different events respectively. Meanwhile, the graphic engine outputs audio data through a channel corresponding to the first API.
Considering that the size of the video data is much larger than the size of the audio data, for example, DX texture data outputted by the illusion engine 4 is much larger than the size of audio PCM (Pulse Code Modulation ) data, transmission efficiency is low if DX texture data is transmitted by using the channel corresponding to the first API.
In order to improve the transmission efficiency of the video data, the server also establishes a shared area for the graphic engine, for example, a video memory area in the GPU; and constructing a channel corresponding to the second API through the shared area between the server and the graphic engine. Then, the graphic engine outputs the video data through a channel corresponding to the second API.
Based on the shared area, as shown in fig. 8, an implementation manner of the step S213 includes:
S2131A, when the server receives a notification that the graphics engine has stored video data to the shared area through the second API, the video data is acquired from the shared area.
Referring again to fig. 7, in the detailed transmission flow, the graphic engine stores the video data to the shared area and then informs the address of the video data in the shared memory through the first API. The server reads video data from the shared area based on the address. The shared area improves data transmission efficiency because the size of the video data is much larger than the size of the audio data.
In view of that the audio data and the video data are generated by the graphics engine separately, in order to obtain the audio-video synchronized graphics interaction data, referring to fig. 9, an implementation of the step S22 includes:
in step S221A, the audio data and the video data are aligned according to the time stamp of the audio data and the time stamp of the video data.
The audio data and the video data have the same playing time length, and the video data comprise a plurality of video frames, and each video frame corresponds to one audio fragment. Each audio clip has a time stamp, each video frame also has a time stamp, and the server aligns the audio clip and the video frame which satisfy the audio-video synchronization relationship through the respective time stamps of the audio clip and the video frame. Alternatively, the time stamp may be a PTS (Presentation Time Stamp ) that tells the player when to present a video frame or an audio clip of the frame.
Step S222A, the aligned audio data and video data are encoded to obtain the graphic interaction data of the audio and video synchronization.
Therefore, the phenomenon that the sound and the picture are not synchronous when the graphic interaction data are displayed is avoided.
Further, as shown in fig. 10, the server is further provided with a video data queue and an audio data queue. The video data queue is used for caching video data output by the graphic engine, and the audio data queue is used for caching audio data output by the graphic engine.
Based on the two queues, the server extracts video frames and audio fragments from the audio queue and the video queue for alignment.
In addition, the embodiment of the application also considers that the situation of frame loss can occur when the graphic engine outputs video data. Aiming at the problem, the server detects video data, and when a frame loss exists, a predicted video frame at the frame loss position is generated according to adjacent video frames at the frame loss position; and sending the predicted video frames to the user terminal.
In some live scenes, the anchor has a need to superimpose the real live view and the virtual view on each other. For example, in live games, some anchor wants to display cartoon images representing the character of the anchor in addition to the content of the game scene in the scene that the audience sees.
In the related superposition technology, a screen grabbing mode is adopted to grab pictures output by a graphic engine and superpose the pictures with superposition graphic data. However, the inventors have studied and found that since each display unit of the display screen generally includes only three channels of RGB, an image obtained by capturing the screen does not carry information of Alpha channels, which results in that the captured image covers the image content of the number of superimposed graphics when superimposed with the superimposed graphics data.
In view of this, in another possible implementation manner, the target interaction function may also correspond to overlay graphics data. In order to avoid that the graphic data output by the graphic engine influences the display effect of the overlapped graphic data. Referring to fig. 11, an implementation manner of the step S213 includes:
in step S2131B, the graphics engine is instructed to generate video data having transparency properties through the first API.
In step S2132B, video data having transparency properties is obtained through the second API.
In this way, since the video data is rendered by the server through the graphics engine, the rendered video data carries transparency attributes as compared to pictures taken from the screen. When the transparency attribute is overlapped with the overlapped graphic data, the problem that the screen is intercepted to lose transparency in the prior art, so that the picture of the overlapped graphic data is completely blocked can be avoided.
Further, an implementation manner of the step S22 includes:
S221B, superimposing the audio data, the video data having the transparency attribute, and the superimposed graphic data.
For example, assume that image a is an overlay graphic data, and image B is video data having a transparency attribute, wherein the value of the transparency attribute of image B is 125. When the image A and the image B are overlapped, the products are multiplied by 0.5 for two corresponding pixels in the image B and the image A respectively, and the overlapped pixels can be obtained.
S222B, encoding the superimposed data to be encoded to obtain graphic interaction data.
Therefore, in the data to be encoded obtained after the server superimposes the video data carrying the transparency attribute with the superimposed image data, the picture of the video data does not cover the picture corresponding to the superimposed image data.
Illustratively, the anchor sends an interactive request to the server through the anchor terminal before starting the live broadcast. The interactive request includes a target interactive function for instructing the server to initiate the graphics engine to render the target avatar.
Then, after receiving the game video sent by the anchor terminal, the server does not directly push the game video to the live broadcasting room, but pushes the game video to the live broadcasting room after overlapping the game video and the target virtual image, so that the audience can watch the live broadcasting picture, and not only the game picture, but also the target virtual image is displayed.
The embodiment of the application also provides an interactive data processing method applied to the user terminal. The application terminal is in communication connection with the server through a network. The steps of the method are described in detail below in conjunction with fig. 12, wherein the method comprises:
step S101, providing an interactive interface.
The interactive interface is displayed with a configuration control.
Step S102, in response to the configuration operation of the configuration control, an interaction request is sent to the server.
The interactive configuration request carries a target interactive function.
Step S103, receiving graphic interaction data matched with a target interaction function and sent by a server;
step S104, displaying the graphic interaction data on the interaction interface.
The user terminals include anchor terminals and/or audience terminals. Illustratively, in one implementation, the interactive interface is as shown in FIG. 13. The configuration control in the interactive interface comprises preview icons 501 of a plurality of avatars and content icons 502 for playing content, wherein the preview icons 501 of the avatars respectively correspond to different avatar identifications, and the content icons 502 respectively correspond to different content identifications.
And the user terminal responds to the configuration operation of the user in the interactive interface, and sends the image identifier corresponding to the virtual image selected by the user and the content identifier corresponding to the playing content to the server in the form of target interactive function. The server generates graphic interaction data including the avatar data and the play content audio data by the graphic engine according to the avatar identification and the content identification.
In this way, the user terminal sends an interaction request to the server, so that the server generates graphic interaction data corresponding to the target interaction function through the graphic engine and sends the graphic interaction data to the user terminal. The user terminal receives the graphic interaction data for display, and a more flexible interaction form is provided for the user.
In another possible implementation, the anchor has a need to superimpose the real live view and the virtual view on each other. For example, the character image of the anchor is represented by an avatar. The target interactive function corresponds to overlay graphics data, which may be live video data of the anchor, for example.
At this time, the graphics interactive data is data obtained by overlapping the generated audio data, video data and superimposed graphics data with each other by the server.
In order to display the graphic interaction data, the interaction interface comprises a first graphic display area for displaying a picture corresponding to the overlapped graphic data. Correspondingly, the interactive interface also comprises a second graphic area for displaying the picture of the video data generated by the server.
In the embodiment of the application, the anchor terminal or the audience terminal may send an interaction request including the target interaction function to the server before the live broadcast starts or during the live broadcast. The target interaction function comprises a host broadcasting identifier, an image identifier of the virtual image and an effect identifier of the display effect.
The server generates corresponding virtual image data according to the image identification, and then transmits the live video data corresponding to the virtual image data and the anchor identification to the anchor terminal and/or the audience terminal according to the display effect corresponding to the effect identification.
Wherein, the server provides various display effect patterns of the avatar data and the live video data. In one possible display effect, the avatar data is superimposed in the live video data. In order to reduce the influence of the avatar data on the display effect of the live video data, the transparency attribute is carried in the avatar video generated by the server. As shown in fig. 14, when the transparency attribute is superimposed with the live video data, it can be avoided that the avatar 602 corresponding to the avatar data can block the live screen 601 corresponding to the live video data.
As shown in fig. 15, in another possible display effect, avatar data is displayed in a second graphic region 702 and live video data is displayed in a first graphic region 701. Wherein the second graphic region 702 may be independent of the first graphic region 701, partially overlap, or the second graphic region 702 may be a designated region in the first graphic region 701. For example, when the first graphic region 701 is a live video play region, the second graphic region 702 may be a region for displaying a anchor person picture in the region.
The embodiment of the application also provides an interactive data processing device which is applied to the server, and the server is in communication connection with at least one user terminal. As shown in fig. 16, the interactive data processing apparatus includes:
the request receiving module 401 is configured to receive an interaction request of a user terminal, where the interaction request includes a target interaction function.
In this embodiment, when the computer executable instructions corresponding to the request receiving module 401 are executed by the processor, step S1 in fig. 3 is implemented, and the detailed description of the request receiving module 401 may refer to the detailed description of step S1.
The data generating module 402 is configured to invoke a graphics engine to obtain graphics-interactive data matched with the target interactive function.
In this embodiment, when the computer executable instructions corresponding to the data generating module 402 are executed by the processor, the step S2 in fig. 3 is implemented, and the detailed description of the data generating module 402 may refer to the detailed description of the step S2.
The data sending module 403 is configured to send the graphic interaction data to the user terminal, so that the user terminal outputs the graphic interaction data.
In this embodiment of the present application, when the computer executable instructions corresponding to the data sending module 403 are executed by the processor, step S3 in fig. 3 is implemented, and for a detailed description of the data sending module 403, reference may be made to the detailed description of step S3.
The embodiment of the application also provides electronic equipment, which comprises a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to realize the interactive data processing method executed by the server. The electronic device may be the server or the user terminal.
The embodiment of the application also provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the server or the user terminal confidence coefficient interaction data processing method is realized.
In summary, the method, the device, the equipment and the medium for processing interactive data provided in the embodiments of the present application. After receiving the interaction request, the server generates graphic interaction data corresponding to the target interaction function through a graphic engine; and pushed to the user terminal of the communication connection. Thus, a live broadcast mode for producing graphic interaction data by the server is provided.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners as well. The apparatus embodiments described above are merely illustrative, for example, flow diagrams and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, the functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing is merely various embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered in the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (11)
1. An interactive data processing method, applied to a server, said server being communicatively connected to at least one user terminal, said method comprising:
receiving an interaction request of the user terminal, wherein the interaction request comprises a target interaction function;
invoking a graphics engine to obtain graphics-interactive data matching the target-interactive function, comprising:
invoking a graphic engine to obtain audio data and video data matched with the target interaction function, wherein the target interaction function is AI reading, and the video data is video data of a story main angle in target content for playing and speaking the target content by an virtual image of the story main angle;
encoding the audio data and the video data to obtain graphic interaction data with synchronous audio and video;
And sending the graphic interaction data to the user terminal so that the user terminal outputs the graphic interaction data.
2. The interactive data processing method according to claim 1, wherein the calling the graphic engine to obtain the audio data and the video data matched with the target interactive function comprises:
configuring a first API and a second API for the graphic engine;
obtaining, through the first API, the audio data generated by the graphics engine based on a target interactive function;
the video data generated by the graphic engine based on the target interaction function is obtained through the second API.
3. The interactive data processing method according to claim 2, wherein the target interactive function corresponds to superimposed graphics data, the obtaining, through the second API, the video data generated by the graphics engine based on the target interactive function includes:
instructing the graphics engine to generate video data having transparency properties through the first API;
obtaining the video data with transparency attribute through the second API;
the encoding the audio data and the video data to obtain the graphic interaction data with synchronous audio and video comprises the following steps:
Superposing the audio data, the video data with transparency attribute and the superposition graphic data;
and encoding the superimposed data to be encoded to obtain the graphic interaction data.
4. The interactive data processing method according to claim 2, wherein the obtaining, through the second API, the video data generated by the graphics engine based on a target interactive function, comprises:
when receiving a notification that the graphics engine has stored the video data to a shared area through the second API, the video data is acquired from the shared area.
5. The method for processing interactive data according to any one of claims 2 to 4, wherein said encoding said audio data and said video data to obtain audio-visual synchronized graphics interactive data comprises:
aligning the audio data with the video data according to the time stamp of the audio data and the time stamp of the video data;
and coding the aligned audio data and the video data to obtain the graphic interaction data with synchronous audio and video.
6. The interactive data processing method according to claim 1, wherein the at least one user terminal comprises a head office terminal and/or a viewer terminal, and the transmitting the graphic interactive data to the user terminal comprises:
And sending the graphic interaction data to the anchor terminal and/or the audience terminal respectively.
7. An interactive data processing method, applied to a user terminal communicatively connected to a server, the method comprising:
providing an interactive interface, wherein the interactive interface is displayed with a configuration control;
responding to the configuration operation of the configuration control, and sending an interaction request to the server, wherein the interaction request comprises a target interaction function, and the target interaction function is AI reading;
receiving graphic interaction data matched with the target interaction function and sent by the server, wherein the server calls a graphic engine to obtain audio data and video data matched with the target interaction function, and the video data is video data of a story main angle in target content for playing and telling the target content by an virtual image of the story main angle;
encoding the audio data and the video data to obtain the graphic interaction data with synchronous audio and video;
and displaying the graphic interaction data on the interaction interface.
8. The interactive data processing method of claim 7, wherein the interactive interface comprises a first graphical display area, the method further comprising:
Displaying superimposed graphic data in the first graphic display area;
wherein the superimposed graphic data has a matching relationship with the target interaction function; the graphic interaction data is obtained by superposing audio data, video data with transparency attribute and the superposition graphic data by the server, and the target interaction function has a matching relationship with the audio data and the video data.
9. An interactive data processing apparatus, applied to a server, the server being communicatively connected to at least one user terminal, the interactive data processing apparatus comprising:
the request receiving module is used for receiving an interaction request of the user terminal, wherein the interaction request comprises a target interaction function;
the data generating module is used for calling a graphic engine and obtaining a mode of graphic interaction data matched with the target interaction function, and comprises the following steps:
invoking a graphic engine to obtain audio data and video data matched with the target interaction function, wherein the target interaction function is AI reading, and the video data is video data of a story main angle in target content for playing and speaking the target content by an virtual image of the story main angle;
Encoding the audio data and the video data to obtain graphic interaction data with synchronous audio and video;
and the data sending module is used for sending the graphic interaction data to the user terminal so that the user terminal outputs the graphic interaction data.
10. An electronic device comprising a processor and a memory, the memory storing a computer program which, when executed by the processor, implements the interactive data processing method of any one of claims 1-6 or the interactive data processing method of any one of claims 7-8.
11. A storage medium storing a computer program which, when executed by a processor, implements the interactive data processing method of any one of claims 1-6 or the interactive data processing method of any one of claims 7-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110240735.1A CN112995692B (en) | 2021-03-04 | 2021-03-04 | Interactive data processing method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110240735.1A CN112995692B (en) | 2021-03-04 | 2021-03-04 | Interactive data processing method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112995692A CN112995692A (en) | 2021-06-18 |
| CN112995692B true CN112995692B (en) | 2023-05-02 |
Family
ID=76352729
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110240735.1A Active CN112995692B (en) | 2021-03-04 | 2021-03-04 | Interactive data processing method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112995692B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112423111A (en) * | 2020-11-05 | 2021-02-26 | 上海哔哩哔哩科技有限公司 | Graphic engine and graphic processing method suitable for player |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109636884A (en) * | 2018-10-25 | 2019-04-16 | 阿里巴巴集团控股有限公司 | Animation processing method, device and equipment |
| CN111698563A (en) * | 2020-05-06 | 2020-09-22 | 广东康云科技有限公司 | Content sending method and device based on AI virtual anchor and storage medium |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101005609B (en) * | 2006-01-21 | 2010-11-03 | 腾讯科技（深圳）有限公司 | Method and system for forming interaction video frequency image |
| TWI336265B (en) * | 2007-11-23 | 2011-01-21 | Mstar Semiconductor Inc | Command distribution method and multimedia apparatus and system using the same for playing game |
| US10332293B2 (en) * | 2017-06-09 | 2019-06-25 | Facebook, Inc. | Augmenting reality with reactive programming |
| CN109874021B (en) * | 2017-12-04 | 2021-05-11 | 腾讯科技（深圳）有限公司 | Live broadcast interaction method, device and system |
| CN108810561A (en) * | 2018-06-21 | 2018-11-13 | 珠海金山网络游戏科技有限公司 | A kind of three-dimensional idol live broadcasting method and device based on artificial intelligence |
| CN110213601B (en) * | 2019-04-30 | 2021-04-27 | 大鱼互联科技(深圳)有限公司 | Live broadcast system and live broadcast method based on cloud game and live broadcast interaction method |
| CN111314724B (en) * | 2020-02-18 | 2022-03-25 | 华为技术有限公司 | Cloud game live broadcasting method and device |
-
2021
- 2021-03-04 CN CN202110240735.1A patent/CN112995692B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109636884A (en) * | 2018-10-25 | 2019-04-16 | 阿里巴巴集团控股有限公司 | Animation processing method, device and equipment |
| CN111698563A (en) * | 2020-05-06 | 2020-09-22 | 广东康云科技有限公司 | Content sending method and device based on AI virtual anchor and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 朴晶 ; ."云录制"时期，电视节目互动性的优化路径探析――以中央广播电视总台《开讲啦》为例.中国广播影视.2020,(第17期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112995692A (en) | 2021-06-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110798697B (en) | Video display method, device and system and electronic equipment | |
| CN111818359B (en) | Processing method and device for live interactive video, electronic equipment and server | |
| US11303881B2 (en) | Method and client for playing back panoramic video | |
| WO2022088918A1 (en) | Virtual image display method and apparatus, electronic device and storage medium | |
| CN112929684B (en) | Video superimposed information updating method and device, electronic equipment and storage medium | |
| WO2019027202A1 (en) | Synchronizing holographic displays and 3d objects with physical video panels | |
| CN111970524B (en) | Control method, device, system, equipment and medium for interactive live broadcast and microphone connection | |
| WO2022048651A1 (en) | Cooperative photographing method and apparatus, electronic device, and computer-readable storage medium | |
| CN108769824A (en) | A kind of video mixed flow method, apparatus, system, equipment and medium | |
| CN108833809A (en) | A kind of video mixed flow control method, device, system, equipment and medium | |
| CN113630614A (en) | Game live broadcast method, device, system, electronic equipment and readable storage medium | |
| WO2023279793A1 (en) | Video playing method and apparatus | |
| CN112995692B (en) | Interactive data processing method, device, equipment and medium | |
| KR101922968B1 (en) | Live streaming method for virtual reality contents and system thereof | |
| CN109729379B (en) | Method, device, terminal and storage medium for realizing live video microphone connection | |
| CN110730340A (en) | Lens transformation-based virtual auditorium display method, system and storage medium | |
| CN114268823A (en) | Video playing method and device, electronic equipment and storage medium | |
| WO2024027611A1 (en) | Video live streaming method and apparatus, electronic device and storage medium | |
| WO2024082561A1 (en) | Video processing method and apparatus, computer, readable storage medium, and program product | |
| US11812084B2 (en) | Method and device for content recording and streaming | |
| KR101915065B1 (en) | Live streaming system for virtual reality contents and operating method thereof | |
| CN112601110B (en) | Method and apparatus for content recording and sharing | |
| WO2021088973A1 (en) | Live stream display method and apparatus, electronic device, and readable storage medium | |
| CN115225915B (en) | Live recording device, live recording system and live recording method | |
| CN116962744A (en) | Live webcast link interaction method, device and live broadcast system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |