CN112752161B - Interaction method based on real-time communication, social application terminal and related device - Google Patents

Interaction method based on real-time communication, social application terminal and related device Download PDF

Info

Publication number
CN112752161B
CN112752161B CN202010031905.0A CN202010031905A CN112752161B CN 112752161 B CN112752161 B CN 112752161B CN 202010031905 A CN202010031905 A CN 202010031905A CN 112752161 B CN112752161 B CN 112752161B
Authority
CN
China
Prior art keywords
user
real
social application
user interaction
current user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010031905.0A
Other languages
Chinese (zh)
Other versions
CN112752161A (en
Inventor
曹良昭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN202010031905.0A priority Critical patent/CN112752161B/en
Publication of CN112752161A publication Critical patent/CN112752161A/en
Application granted granted Critical
Publication of CN112752161B publication Critical patent/CN112752161B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/44213—Monitoring of end-user related data
    • H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4781—Games
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/141—Systems for two-way working between two video terminals, e.g. videophone

Abstract

The embodiment of the invention discloses an interaction method based on real-time communication, a social application terminal and a related device, which are applied to the technical field of information processing. When the social application terminal of the current user performs real-time video communication with the social application terminals of other users, the acquired video characteristics of the video of the current user are acquired, user operation information is determined according to the video characteristics, further, a user interaction picture operated by a user interaction system is determined, and the user interaction picture is rendered according to the user interaction picture and the image of the current user to form a real-time video communication picture so as to perform real-time video communication with the social application terminals of the other users, so that user interaction in the real-time video communication process is realized, and interestingness is enhanced.

Description

Interaction method based on real-time communication, social application terminal and related device
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to an interaction method based on real-time communication, a social application terminal, and a related device.
Background
Many social applications exist which have the function of real-time or voice of multiple users, so that one user can operate the social application terminal, and the social application terminal initiates real-time or real-time voice communication with the social application terminal of the other user (or other multiple users) according to the operation of the one user, thereby realizing the real-time or voice of the multiple users.
In order to increase the interest in the real-time or voice communication process, some social application terminals can initiate a voice game with the social application terminal of the communication counterpart in the real-time or voice communication process, such as killing wolves and the like, but the game mode is relatively single.
Disclosure of Invention
The embodiment of the invention provides an interaction method based on real-time communication, a social application terminal and a related device, which realize user interaction in the real-time video communication process.
An aspect of an embodiment of the present invention provides an interaction method based on real-time communication, including:
when the social application terminal of the current user performs real-time video communication with the social application terminals of other users, acquiring the video of the current user, wherein the video of the current user comprises the image and voice information of the current user;
acquiring video characteristics of the current user, wherein the video characteristics comprise human body component characteristics in an image of the current user and/or content corresponding to the voice information;
determining user operation information according to the video characteristics of the current user, and determining a user interaction picture of user interaction system operation according to the user operation information;
And rendering according to the user interaction picture and the current user image to form a real-time video communication picture so as to perform real-time video communication with social application terminals of other users.
In a specific implementation of an aspect of an embodiment of the present invention, the method further includes:
receiving another real-time video communication picture sent by the social application terminal of the other user, and displaying the another real-time video communication picture on the interface of the real-time video communication;
and when the user interaction picture is determined aiming at the video characteristics of the video of the current user which is collected later, determining another user interaction picture according to the video characteristics of the video of the current user which is collected later and the another real-time video communication picture.
Another aspect of an embodiment of the present invention provides a social application terminal, including:
the acquisition unit is used for acquiring the video of the current user when the social application terminal of the current user and the social application terminals of other users carry out real-time video communication, wherein the video of the current user comprises the image and voice information of the current user;
the feature unit is used for acquiring the video features of the current user; the video features comprise human body component features and/or content corresponding to voice information in the image of the current user;
The user interaction unit is used for determining user operation information according to the video characteristics of the current user and determining a user interaction picture operated by a user interaction system according to the user operation information;
and the rendering unit is used for rendering the image of the current user according to the user interaction picture to form a real-time video communication picture so as to perform real-time video communication with the social application terminals of the other users.
Another aspect of the embodiments of the present invention provides a storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform the real-time communication based interaction method according to the embodiments of the present invention.
Another aspect of the embodiment of the present invention provides a terminal device, including a processor and a storage medium;
the storage medium is used for storing a plurality of instructions, the instructions are used for being loaded and executed by a processor, and the processor is used for realizing each instruction in the plurality of instructions.
It can be seen that, in the method of the embodiment, when the social application terminal of the current user performs real-time video communication with the social application terminals of other users, the collected video features of the video of the current user are obtained, user operation information is determined according to the video features, further, a user interaction picture operated by the user interaction system is determined, and a real-time video communication picture is formed by rendering according to the user interaction picture and the image of the current user, so that real-time video communication with the social application terminals of other users is performed. In this way, in the process of real-time video communication among social application terminals of a plurality of users, the video characteristics of the video of the current user acquired in the real-time video communication are fully utilized to determine the user operation information, and then the obtained user interaction picture is combined with the acquired user image, so that the user interaction in the process of real-time video communication is realized, and the interestingness is enhanced.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings can be obtained according to these drawings without inventive faculty for a person skilled in the art.
FIG. 1 is a schematic diagram of an interaction method based on real-time communication according to an embodiment of the present invention;
FIG. 2 is a flow chart of an interaction method based on real-time communication according to an embodiment of the present invention;
FIG. 3a is a schematic diagram of an interface for real-time video communication displayed by a social application terminal of user 1 in one embodiment of the invention;
FIG. 3b is a schematic diagram of an interface for real-time video communication displayed by a social application terminal of user 2 in one embodiment of the invention;
FIG. 4 is a schematic diagram of an interaction method based on real-time communication in an application embodiment of the present invention;
FIG. 5 is a flow chart of a small game module acquiring characteristics of a human body component in an embodiment of the invention;
FIG. 6a is a schematic diagram of image processing of a user by a real-time video communication system in an embodiment of the application of the present invention;
FIG. 6b is a schematic diagram of a real-time video communication system forming a real-time video communication screen according to an embodiment of the present invention;
FIG. 7a is a schematic diagram of an interface of a real-time video communication system displaying a game near the end of the game in an embodiment of the application of the present invention;
fig. 7b is a schematic diagram of a real-time video communication screen transmitted between two instant messaging terminals in an application embodiment of the present invention;
FIG. 7c is a schematic diagram of an interface at the end of a game displayed by the real-time video communication system in an embodiment of the application of the present invention;
FIG. 8 is a schematic diagram of a logic structure of a social application terminal according to an embodiment of the present invention;
fig. 9 is a schematic logic structure diagram of a terminal device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented, for example, in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The embodiment of the invention provides an interaction method based on real-time communication, which is mainly applied to the interaction method executed by a social application terminal of any party user in the process of carrying out real-time video communication between the social application terminal of a current user and the social application terminals of other users (at least one user), and specifically comprises the following steps as shown in fig. 1:
when the social application terminal of the current user performs real-time video communication with the social application terminals of other users, acquiring the video of the current user, wherein the video of the current user comprises the image and voice information of the current user; acquiring video characteristics of the current user, wherein the video characteristics comprise human body component characteristics in an image of the current user and/or content corresponding to the voice information; determining user operation information in a user interaction system according to the video characteristics of the current user, and determining a user interaction picture operated by the user interaction system according to the user operation information; and rendering according to the user interaction picture and the current user image to form a real-time video communication picture so as to perform real-time video communication with social application terminals of other users.
In the actual application process, the social application terminal may be an application terminal with a real-time video communication function, such as a WeChat or instant messaging application, and the user interaction system may be a small game system.
In this way, in the process of real-time video communication among social application terminals of a plurality of users, the video characteristics of the video of the current user acquired in the real-time video communication are fully utilized to determine the user operation information, then the user interaction picture operated by the user interaction system is obtained according to the user operation information, and further the user interaction picture is combined with the image of the current user to form a real-time video communication picture, so that the user interaction in the real-time video communication process is realized, and the interestingness is enhanced.
The embodiment of the invention provides an interaction method based on real-time communication, which is mainly a method executed by a social application terminal of a current user, and a flow chart is shown in fig. 2, and comprises the following steps:
step 101, after the social application terminal of the current user performs real-time video communication with the social application terminals of other users, acquiring the video of the current user, wherein the video of the current user comprises the image and voice information of the current user.
It can be understood that a user can operate the social application terminal, so that the social application terminal establishes a channel for real-time video communication with social application terminals of other users according to user operation, and each social application terminal can acquire a user's video in real time and send the user's video to the other social application terminal, wherein the user's video comprises multiple frames of images and multiple frames of voice information of the user; and displaying the multi-frame images of the current user and the multi-frame images of other users on an interface for carrying out real-time video communication with the social application terminals of other users.
In this embodiment, a user interaction interface is displayed on an interface where a social application terminal of a current user performs real-time video communication with social application terminals of other users, in one case, the current user may select a certain user interaction system from the user interaction interface, then the social application terminal may receive information of the selected user interaction system from the user interaction interface, start a corresponding user interaction system, initiate a process of this embodiment, collect a video of the current user according to a certain frame rate, and after a frame of video of the current user is collected, continue to execute the following steps 102 to 104. The social application terminal is used for receiving information of the user interaction system selected by the user.
In the process, after receiving the information of the selected user interaction system, the social application terminal of the current user firstly sends a joining request of the selected user interaction system to the social application terminals of other users, the social application terminals of other users display the user selection information, when the other users choose to agree to join, the social application terminals of other users return the information of agreeing to join, and the social application terminals of the current user receive the information of agreeing to join returned by the social application terminals of other users according to the joining request, and then the step of starting the corresponding user interaction system is executed; when other users select to reject the joining, the social application terminals of the other users return joining rejection information, the social application terminals of the current users receive joining rejection information returned by the social application terminals of the other users according to the joining request, the corresponding user interaction system is not started, and the process is ended.
In another case, in the process of performing real-time video communication with the social application terminals of other users, the social application terminal of the current user may directly receive a request for joining a user interaction system sent by the social application terminal of the other user, and display user selection information, if the current user selects information to agree to join from the user selection information, the social application terminal of the current user receives the information to agree to join selected from the user selection information, then starts the corresponding user interaction system, initiates the process of the embodiment, acquires the video of the current user according to a certain frame rate, and after acquiring a frame of video of the current user, continues to execute the following steps 102 to 104; if the current user can select the information refused to join from the user selection information, the social application terminal of the current user receives the information refused to join selected from the user selection information, and a corresponding user interaction system is not started, so that the process is ended.
For example, fig. 3a shows an interface of real-time video communication displayed by a social application terminal, on which an image of a current user 1 and an image of another user 2 are displayed, and buttons of "beautification", "hang-up" and "play-and-play" are also displayed, where the "play-and-play" button is a user interaction interface; when a user clicks a button for playing one game, the social application terminal displays selection information of a plurality of user interaction systems, specifically icons of 3 small games; when a user selects one of the user interaction systems, i.e. clicks on an icon of the mini-game 2, the social application terminal will send a join request of the mini-game 2 to the social application terminal of the other user 2. Before the social application terminal sends the joining request, a user prompt can be performed on a real-time video communication interface, for example, a word such as "confirm to invite the other party to join in the small game 2" and "cancel" and "confirm" buttons are displayed, and when the user clicks the "confirm" button, the social application terminal sends the joining request to the social application terminal of the user 2.
As shown in fig. 3b, when the social application terminal of the user 2 receives the joining request of the mini-game 2, the word of "whether the invitation of joining the mini-game 2 is received is displayed on the interface of the real-time video communication, and the buttons of" reject "and" start "are displayed; when the user clicks the "start" button, the social application terminal of user 2 will activate the corresponding mini-game system (i.e., the user interaction system described above), and during the time that the mini-game system is activated, the interface for real-time video communication may display a countdown, such as "prepare to start game 3s".
When the social application terminal does not start the small game system, in general, the image of the current user displayed on the interface of the real-time video communication can be displayed in a small frame, the image of the opposite user can be displayed in a large frame, and when the user clicks the small frame, the social application terminal can be switched to the small frame to display the image of the opposite user, and the large frame displays the image of the current user; when the social application terminal starts the small game system, images of the current user and the opposite user displayed on the interface of the real-time video communication can be displayed by using frames with the same size, for example, the images of different users are respectively displayed by using the frames distributed up and down or left and right.
Step 102, obtaining video features of the current user, wherein the video features comprise human body component features and/or content of voice information in an image of the current user.
The social application terminal of the current user may acquire the above-acquired human body component characteristics in the image of the current user, and specifically, may acquire coordinate information of key points of a face in the image of the current user, coordinate information of key points of a mouth (or eyes, etc.) in the face, or coordinate information of key points of other parts (except for the face, such as hands, etc.) of a human body in the image of the current user, and so on, as the human body component characteristics in the image of the current user. When the social application terminal of the current user acquires the content of the voice information, the text information corresponding to the voice information can be identified, namely the content of the voice information, for example, the voice information is right voice, and the content of the voice information is right text.
Step 103, determining user operation information in the user interaction system according to the video characteristics of the current user, and determining a user interaction picture operated by the user interaction system according to the user operation information.
After the social application terminal of the current user obtains the human body component characteristics and/or the voice information, the user interaction system can firstly determine the user operation information of the current user, such as eye closing operation, mouth closing operation or hand swinging operation, or a certain instruction and other operations according to the video characteristics of the current user; and then determining the operation of the user interaction system according to the user operation information of the current user and the operation logic of the user interaction system, and determining the user interaction picture according to the operation of the user interaction system. In this process, the user interaction system may combine video features of successive multiframes of the current user to determine user operation information of the current user.
For example, if the user interaction system is a fruit-cutting small game system, the fruit-cutting small game system determines user operation information of the current user, such as operations of swinging hands, according to the human body component characteristics, and the user's swinging hands in the operation logic of the fruit-cutting small game system can cut fruits in the small game, so that the finally determined user interaction picture is a picture of the cut fruits. For another example, if the user interaction system is the in-flight mini-game system, the in-flight mini-game system determines the content of the voice information of the current user as the user operation information of the current user, such as "fire" and the like, and the "fire" operation in the operation logic of the in-flight mini-game system can fire the in-game, so that the finally determined user interaction picture is the picture of the in-flight or not.
It should be noted that, in this embodiment, when the user interaction system is started, the human body component event and/or the voice event is registered with the application engine, and when the social application terminal executes this step 102, the video feature of the current user is obtained after determining that the user interaction system registers the human body component event and/or the voice event, and the video feature is sent to the user interaction system through the application engine. Similarly, when the user interaction system returns to the user interaction picture, the user interaction system also returns through the application engine.
In addition, it should be noted that if the video feature includes the human body component feature and the content of the voice information, the user interaction system may determine whether the mouth feature in the human body component feature corresponds to the content of the voice information when determining the user operation information of the current user according to the video feature, and if so, may directly determine the user operation information of the user according to the content of the voice information; if the user operation information does not accord with the human body component characteristics, the user operation information of the user can be determined according to the human body component characteristics.
When the mouth characteristics indicate that the motion trail of the mouth accords with the motion trail of the mouth when the content corresponding to the voice information is spoken, the mouth characteristics accord with the content of the voice information. For example, the mouth feature indicates that the trajectory of the mouth is "closed-open", and the content of the voice message is "down" and then corresponds.
And 104, rendering by the social application terminal according to the user interaction picture and the current user image to form a real-time video communication picture so as to perform real-time video communication with social application terminals of other users.
Specifically, the social application terminal can carry out superposition rendering on the image of the current user and the user interaction picture, and then a real-time video communication picture can be obtained; other processes such as beautifying, peeling, or blurring may be performed on the image of the current user, and the order of these processes may be before or after the formation of the real-time video communication screen, which is not limited.
Further, in some practical applications, if the user interaction system is a ranking small game system such as fruit cutting, the user interaction system also counts the interactive points of the current user, and in this case, the social application terminal of the current user sends the interactive points and the real-time video communication picture to the social application terminals of other users for display. And the social application terminal of the current user receives another real-time video communication picture sent by the social application terminal of the other user, and the other real-time video communication picture is directly displayed on the interface of the real-time video communication.
The social application terminal of the current user adds interactive points in the head data of the real-time video communication picture, and sends the real-time video communication picture with the head data to the social application terminals of other users for display.
In other practical applications, if the user interaction system is a combat game system such as a street tyrant, the user interaction system needs to consider not only the operation of the current user but also the operation of other users when determining the user interaction picture, and after the social application terminal of the current user receives another real-time video communication picture sent by the social application terminal of the other users, the user interaction system determines another user interaction picture according to the video feature of the video of the current user collected later and the another real-time video communication picture when determining the other user interaction picture.
For example, the user interaction system is a small fight game system, the virtual image of the other user walks forward by one step, and then the social application terminal of the other user sends the virtual image of the other user to another real-time video communication picture of the social application terminal of the current user, wherein the another real-time video communication picture comprises a game picture (namely a user interaction picture) after the virtual image of the other user walks forward by one step, and the game picture determined by the small fight game system is a game picture which is determined by aiming at the video characteristics of the video of the current user and is added with the movement of the virtual image of the current user, such as backward by one step, on the basis of the game picture after the virtual image of the other user walks forward by one step.
It can be seen that, in the method of this embodiment, when the social application terminal of the current user performs real-time video communication with the social application terminals of other users, the collected video features of the video of the current user are obtained, user number operation information is determined according to the video features of the current user, a user interaction picture operated by the user interaction system is determined according to the user operation information, and a real-time video communication picture is formed by rendering the user interaction picture and the image of the current user, so as to perform real-time video communication with the social application terminals of other users. In this way, in the process of real-time video communication among social application terminals of a plurality of users, the video characteristics of the video of the current user collected in the real-time video communication are fully utilized to determine the user interaction picture operated by the user interaction system, and then the user interaction picture is combined with the collected user image, so that the user interaction in the process of real-time video communication is realized, and the interestingness is enhanced.
In the embodiment, the social application terminal is specifically an instant communication terminal, and the even communication terminal includes a real-time video communication system and a user interaction system, and the user interaction system is specifically a small game system, and the method of the embodiment is mainly applied to user interaction in the real-time video communication process of the instant communication terminals of two users, as shown in fig. 4, and may include the following steps:
in step 201, the user 1 operates the instant communication terminal 1 to enable the instant communication terminal 1 to start the real-time video communication system and establish a channel of real-time video communication with the instant communication terminal 2 of the user 2, so that the real-time video communication system of the instant communication terminal 1 displays a user interaction interface on the interface of the real-time video communication.
In step 202, when the user 1 selects a small game system through the user interaction interface, the real-time video communication system receives information of the small game system from the user interaction interface and sends a joining request of the small game system to the instant communication terminal 2.
In step 203, after the instant communication terminal 2 receives the joining request, user selection information is displayed according to the joining request, and when the user 2 selects joining approval information from the user selection information, the real-time video communication system of the instant communication terminal 2 returns joining approval information to the instant communication terminal 1 and starts a corresponding small game system.
In step 204, the real-time video communication system of the instant communication terminal 1 receives the information of agreeing to join returned by the instant communication terminal 2, and starts the corresponding small game system.
In step 205, during the process of real-time video communication, the real-time video communication system of any instant communication terminal will collect the video of each frame of the current user, including the image and voice information of the current user, and obtain the human body component characteristics of the image of each frame of the current user or the content of the voice information (i.e. the video characteristics), and send the obtained video to the small game system.
Specifically, as shown in fig. 5, communication between the small game system and the real-time video communication system needs to be performed by an application engine, which can be implemented specifically by the following steps:
11. the mini-gaming system registers human component events, such as face events, etc., or voice events with the application engine through a registration event (registerEvent) mechanism each time it is started.
12. The real-time video communication system can comprise a shooting component and an identification sub-module, so that the shooting component can detect that the small game system registers human body component events through a mechanism of detecting registration events (oneeventregister), and after each frame of current user image is acquired, the human body component events are transmitted to the identification sub-module, the identification sub-module identifies human body component characteristics in the current user image, and the human body component events are returned to the shooting component; when the small game system is monitored to register a voice event, after voice information of the current user in each frame is collected, the voice event is transmitted to the recognition submodule, and the recognition submodule recognizes specific content corresponding to the voice information of the current user and returns the specific content to the shooting assembly.
13. The content of the human body component characteristics or the voice information is transmitted to the application engine by the shooting component through a send event (sendEvent) mechanism.
14. The application engine will transmit the content of the human component features or voice information to the mini-gaming system through a callback event (onEvent) mechanism.
It should be noted that, the application engine may be a game engine or the like; the human body component features can be the coordinate information of key points of a human face, the coordinate information of key points of eyes or mouth in the human face or the coordinate information of key points of other parts of the human body, and the like.
In step 206, the small game system determines the user operation information of the current user according to the characteristics of the human body components or the content of the voice information, further determines the operation of the small game system, further determines the game picture (i.e. the user interaction picture), and returns to the real-time video communication system.
In step 207, the real-time video communication system renders according to the game frame and the current user image acquired in step 205 to form a real-time video communication frame, and sends the real-time video communication frame to another instant communication terminal for display.
It can be understood that, as shown in fig. 6a, since a photographing component in a real-time video communication system of a certain instant communication terminal may include: an image capturing device (camera), a texture packaging (surface texture) component, a series of filter (GPUFilter) components, an Encoder (Encoder), a texture display (TextureView) component, an audio recording (AudioRecord) component, a multimedia mixing (MediaMuxer) component, and the like, and in general, when capturing an image of a current user of each frame, the image capturing device generates corresponding texture data, and the corresponding texture data is received by the texture packaging component and is processed by the filter component. On the one hand, the texture display component can directly display the processed texture data on the interface of real-time video communication; and on the other hand, the data are transmitted to an encoder for encoding, and finally, the encoded data obtained by the encoder are encoded into video together with the audio information of each frame recorded by the audio recording component by the multimedia mixing component, and are uplink-transmitted to instant messaging terminals of other users through a network transmission protocol.
Here, the filters refer to special effects processing on texture data of each frame, such as blurring processing, peeling, beautifying, face thinning, and the like, each of which is referred to as one filter, and since a series of filters are pass processing, a series of filters may be referred to as a rendering chain. In practical applications, the filter may use a software development kit (Software Development Kit, SDK) for graphics processing, and coding and downstream in a network transport protocol may use SDKs for Tencent Real-Time Communication (TRTC) for tandem or multiple audio and video calls.
In this embodiment, the texture data of each frame of image is processed by a series of filter components, and the game frame acquired by the real-time video communication system is also the texture data, so that when the real-time video communication system performs rendering according to the game frame and the image of the current user, the rendering and the filter can be mixed together to form a real-time video communication frame, and the mixing can be processed by adopting the texture mixing of the development graphics library (Open Graphics Library, openGL). As shown in fig. 6b, the game rendering is embedded as a filter into any position of the rendering chain, for example, the front, the last or the middle of the rendering chain, the middle position is shown in the figure, when the texture data output by a certain filter is input into the game rendering, the game rendering combines the received texture data with the acquired game picture and outputs the combined texture data to the next filter, and the texture data obtained by the last filter (for example, filter m) is the data of the real-time video communication picture and can be respectively transmitted to the encoder and the structure display component.
In step 208, when the user interaction process (i.e. the mini-game) is finished, the real-time video communication system of any instant communication terminal can acquire data such as game countdown, game progress, game points (i.e. interactive points) and the like from the mini-game system, and send the data to another instant communication terminal for display together with the real-time video communication picture.
An interface for real-time video communication, such as that shown in fig. 7a, displayed by an instant communication terminal, includes: the real-time video communication picture formed by the instant communication terminal of the current user (comprising the combination of the image of the current user and the game picture), and the real-time video communication picture received from the instant communication terminal of another user can also comprise information such as game countdown, game points corresponding to the users and the like.
Specifically, the real-time video communication system adds a game point to the header data of the real-time video communication picture, and transmits the real-time video communication picture to which the header data is added to another instant communication terminal.
For example, as shown in fig. 7b, the instant communication terminal 1 adds a game score or the like corresponding to the user 1 to the header data of the formed real-time video communication screen, and transmits the result to the instant communication terminal 2, and at the same time, the instant communication terminal 2 adds a game score or the like corresponding to the user 2 to the header data of the formed real-time video communication screen, and transmits the result to the instant communication terminal 1.
In step 209, when the user interaction process (i.e. the mini-game) is finished, the real-time video communication screen of any instant communication terminal displays the user interaction result, i.e. the result information of the mini-game.
For example, the instant communication terminal shown in fig. 7c may display the result information of the mini-game on the interface of the real-time video communication, such as displaying the word of "game result game score", and may also display the buttons of "share", "chat" and "get again", when the user clicks "get again", the real-time video communication system may reinitiate the flow of user interaction; when a user clicks a chat button, the real-time video communication system only performs video communication; when the user clicks the "share" button, some game credits are sent to the user interaction space (i.e. circle of friends) of the instant messaging terminal, or to the instant messaging terminal of another user.
It should be noted that, in the above embodiment, the interaction of the small games performed by the two instant messaging terminals corresponding to the two users respectively may also be applied to the interaction of the small games performed by the instant messaging terminals of the multiple users in the process of performing real-time video communication.
Therefore, in the embodiment, the small game of the small game module and the real-time video communication are combined together, so that the interestingness in the real-time video communication process is increased, and meanwhile, the application scene of the small game is enriched.
The embodiment of the invention also provides a social application terminal, the structural schematic diagram of which is shown in fig. 8, and the social application terminal specifically comprises:
the collecting unit 11 is configured to collect a video of a current user when the social application terminal of the current user performs real-time video communication with social application terminals of other users, where the video of the current user includes image and voice information of the current user.
The feature unit 12 is configured to obtain the video feature of the current user acquired by the acquisition unit 11, where the video feature includes a human body component feature in an image of the current user and/or content corresponding to the voice information.
The user interaction unit 13 is configured to determine user operation information according to the video feature of the current user acquired by the feature unit 12, and determine a user interaction picture operated by the user interaction system according to the user operation information.
And the rendering unit 14 is configured to render the image of the current user with the user interaction frame received by the user interaction unit 13 to form a real-time video communication frame, so as to perform real-time video communication with the social application terminals of the other users.
Further, the apparatus of this embodiment may further include:
the starting unit 15 is configured to display a user interaction interface on an interface for performing real-time video communication between the social application terminal of the current user and social application terminals of other users; when receiving the information of the selected user interaction system from the user interaction interface, the corresponding user interaction system is started, and then the user number interaction unit 13 determines a user interaction picture according to the user interaction system started by the starting unit 15. Before starting the corresponding user interaction system, the starting unit 15 is further configured to send a request for adding the selected user interaction system to a social application terminal of another user; and executing the starting of the corresponding user interaction system when receiving the joining approval information returned by the social application terminals of the other users according to the joining request.
The starting unit 15 is further configured to display user selection information when receiving a request for joining the user interaction system sent by the social application terminal of the other user; and receiving the information which is selected from the user selection information and agrees to be added, and starting a corresponding user interaction system.
It should be noted that, the social application terminal may further include an application engine 16, when the startup remote 15 starts the user interaction system, registers a human body component event and/or a voice event with the application engine 16, and the feature unit 12 is specifically configured to, when determining that the user interaction system registers the human body component event and/or the voice event, acquire a video feature in the image of the current user, and send the video feature to the user interaction unit 13 in the user interaction system through the application engine 16. Also, the screen receiving unit 13 receives the user interaction screen returned by the user interaction system through the application engine 16.
Further, the apparatus of this embodiment may further include: a communication unit 17, configured to obtain an interaction score corresponding to the current user; and sending the interactive integral and the real-time video communication picture formed by the rendering unit 14 to the social application terminals of the other users for display.
Specifically, when the interactive score and the real-time video communication picture are sent to the social application terminals of the other users to be displayed, the communication unit 17 is specifically configured to add the interactive score to header data of the real-time video communication picture, and send the real-time video communication picture with the header data added to the social application terminals of the other users to be displayed.
Further, the communication unit 17 is further configured to receive another real-time video communication frame sent by the social application terminal of the other user, and display the other real-time video communication frame on the interface of the real-time video communication.
Further, the apparatus of this embodiment may further include: a transmission unit 18, configured to send the another real-time video communication frame to the user interaction unit 13 in the user interaction system; the user number interaction unit 13 determines another user interaction picture according to the video characteristics of the current user collected later and another real-time video communication picture transmitted by the transmission unit 18 when determining the user interaction picture for the video characteristics of the video of the current user collected later. Typically, the transmission unit 18 also transmits information to the user interaction system via the application engine 16.
It can be seen that, in the social application terminal of the present embodiment, when the social application terminal of the current user performs real-time video communication with the social application terminals of other users, the feature unit 12 may acquire the video feature of the collected video of the current user, the user interaction unit 13 determines the user operation information according to the video feature, and further determines the user interaction picture operated by the user interaction system, and the rendering unit 14 renders the real-time video communication picture according to the user interaction picture and the image of the current user, so as to perform real-time video communication with the social application terminals of other users. In this way, in the process of real-time video communication among social application terminals of a plurality of users, the video characteristics of the video of the current user acquired in the real-time video communication are fully utilized to determine the user operation information, then the user interaction picture is determined, and finally the user interaction picture is combined with the user image, so that the user interaction in the process of real-time video communication is realized, and the interestingness is enhanced.
The embodiment of the present invention further provides a terminal device, whose structure schematic diagram is shown in fig. 9, where the terminal device may generate relatively large differences due to different configurations or performances, and may include one or more central processing units (central processing units, CPU) 20 (e.g., one or more processors) and a memory 21, and one or more storage media 22 (e.g., one or more mass storage devices) storing application programs 221 or data 222. Wherein the memory 21 and the storage medium 22 may be transitory or persistent. The program stored in the storage medium 22 may include one or more modules (not shown), each of which may include a series of instruction operations in the terminal device. Still further, the central processor 20 may be arranged to communicate with the storage medium 22 and execute a series of instruction operations in the storage medium 22 on the terminal device.
Specifically, the application program 221 stored in the storage medium 22 includes an application program for interaction based on real-time communication, and the program may include the acquisition unit 11, the feature unit 12, the user interaction unit 13, the rendering unit 14, the starting unit 15, the application engine 16, the communication unit 17, and the transmission unit 18 in the social application terminal, which are not described herein. Still further, the central processor 20 may be configured to communicate with the storage medium 22 and execute a series of operations corresponding to the interactive application program based on real-time communication stored in the storage medium 22 on the terminal device.
The terminal device may also include one or more power supplies 23, one or more wired or wireless network interfaces 24, one or more input/output interfaces 25, and/or one or more operating systems 223, such as Windows ServerTM, mac OS XTM, unixTM, linuxTM, freeBSDTM, or the like.
The steps performed by the social application terminal described in the above method embodiment may be based on the structure of the terminal device shown in fig. 9.
Embodiments of the present invention also provide a computer storage medium storing a plurality of computer programs adapted to be loaded by a processor and to perform an interaction method based on real-time communication as performed by the social application terminal described above.
The embodiment of the invention also provides a terminal device, which comprises a processor and a computer storage medium; the computer storage medium is used for storing a plurality of computer programs, the computer programs are used for being loaded by a processor and executing the interaction method based on real-time communication and executed by the social application terminal, and the processor is used for realizing each computer program in the plurality of computer programs.
Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program to instruct related hardware, the program may be stored in a computer readable storage medium, and the storage medium may include: read Only Memory (ROM), random Access Memory (RAM), magnetic or optical disks, and the like.
The above describes in detail an interaction method based on real-time communication, a social application terminal and related devices provided by the embodiments of the present invention, and specific examples are applied to describe the principles and embodiments of the present invention, where the description of the above embodiments is only for helping to understand the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.

Claims (10)

1. An interactive method based on real-time communication, comprising:
when the social application terminal of the current user performs real-time video communication with the social application terminals of other users, acquiring the video of the current user, wherein the video of the current user comprises the image and voice information of the current user;
acquiring video characteristics of the current user, wherein the video characteristics comprise human body component characteristics in an image of the current user and contents corresponding to the voice information;
if the mouth feature in the human body component feature is consistent with the content of the voice information, determining user operation information according to the content of the voice information, if the mouth feature in the human body component feature is inconsistent with the content of the voice information, determining user operation information according to the human body component feature, and determining a user interaction picture operated by a user interaction system according to the user operation information and operation logic of the user interaction system, wherein the user interaction system is a game system, and the operation logic comprises: the user operation information corresponds to the operation of the virtual articles in the game system;
And rendering according to the user interaction picture and the current user image to form a real-time video communication picture so as to perform real-time video communication with social application terminals of other users.
2. The method of claim 1, wherein prior to the capturing the video of the current user, the method further comprises:
displaying a user interaction interface on an interface of real-time video communication between the social application terminal of the current user and social application terminals of other users;
and when receiving the information of the selected user interaction system from the user interaction interface, starting the corresponding user interaction system.
3. The method of claim 2, wherein prior to the initiating the corresponding user interaction system, the method further comprises:
sending the joining request of the selected user interaction system to social application terminals of other users;
and when receiving the joining approval information returned by the social application terminals of the other users according to the joining request, executing the step of starting the corresponding user interaction system.
4. The method of claim 1, wherein prior to the capturing the video of the current user, the method further comprises:
When receiving a joining request of the user interaction system sent by the social application terminal of the other users, displaying user selection information;
and receiving the information which is selected from the user selection information and agrees to be added, and starting a corresponding user interaction system.
5. The method of claim 1, wherein when the user interaction system is started, registering human component events and/or voice events with an application engine, the acquiring the video features of the current user further comprises:
and when the user interaction system is determined to register human body component events and/or voice events, acquiring video features of the current user, and sending the video features to the user interaction system through an application engine.
6. The method of any one of claims 1 to 5, further comprising:
acquiring an interactive integral corresponding to the current user;
and sending the interactive integration and the real-time video communication picture to social application terminals of other users for display.
7. The method of claim 6, wherein the sending the interactive score and real-time video communication screen to the social application terminal of the other user for display comprises:
And adding the interactive integration in the head data of the real-time video communication picture, and sending the real-time video communication picture with the head data to the social application terminals of other users for display.
8. A social application terminal, comprising:
the acquisition unit is used for acquiring the video of the current user when the social application terminal of the current user and the social application terminals of other users carry out real-time video communication, wherein the video of the current user comprises the image and voice information of the current user;
the feature unit is used for acquiring the video features of the current user; the video features comprise human body component features in the image of the current user and contents corresponding to the voice information;
the user interaction unit is configured to determine user operation information according to the content of the voice information if the mouth feature in the body part feature matches the content of the voice information, determine user operation information according to the body part feature if the mouth feature in the body part feature does not match the content of the voice information, and determine a user interaction picture operated by the user interaction system according to the user operation information and operation logic of the user interaction system, where the user interaction system is a game system, and the operation logic includes: the user operation information corresponds to the operation of the virtual articles in the game system;
And the rendering unit is used for rendering the image of the current user according to the user interaction picture to form a real-time video communication picture so as to perform real-time video communication with the social application terminals of the other users.
9. A computer storage medium storing a plurality of computer programs adapted to be loaded by a processor and to perform the real-time communication based interaction method according to any of claims 1 to 7.
10. A terminal device comprising a processor and a computer storage medium;
the computer storage medium is configured to store a plurality of computer programs for loading and executing the real-time communication based interaction method according to any one of claims 1 to 7 by a processor configured to implement each of the plurality of computer programs.
CN202010031905.0A 2020-01-13 2020-01-13 Interaction method based on real-time communication, social application terminal and related device Active CN112752161B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010031905.0A CN112752161B (en) 2020-01-13 2020-01-13 Interaction method based on real-time communication, social application terminal and related device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010031905.0A CN112752161B (en) 2020-01-13 2020-01-13 Interaction method based on real-time communication, social application terminal and related device

Publications (2)

Publication Number Publication Date
CN112752161A CN112752161A (en) 2021-05-04
CN112752161B true CN112752161B (en) 2023-10-13

Family

ID=75645078

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010031905.0A Active CN112752161B (en) 2020-01-13 2020-01-13 Interaction method based on real-time communication, social application terminal and related device

Country Status (1)

Country Link
CN (1) CN112752161B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104079863A (en) * 2014-06-27 2014-10-01 联想(北京)有限公司 Information processing method and electronic device
CN108366221A (en) * 2018-05-16 2018-08-03 维沃移动通信有限公司 A kind of video call method and terminal
CN108600680A (en) * 2018-04-11 2018-09-28 南京粤讯电子科技有限公司 Method for processing video frequency, terminal and computer readable storage medium
CN109391792A (en) * 2017-08-03 2019-02-26 腾讯科技（深圳）有限公司 Method, apparatus, terminal and the computer readable storage medium of video communication
CN109831636A (en) * 2019-01-28 2019-05-31 努比亚技术有限公司 Interdynamic video control method, terminal and computer readable storage medium
CN110602516A (en) * 2019-09-16 2019-12-20 腾讯科技（深圳）有限公司 Information interaction method and device based on live video and electronic equipment

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10719713B2 (en) * 2018-05-29 2020-07-21 International Business Machines Corporation Suggested comment determination for a communication session based on image feature extraction

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104079863A (en) * 2014-06-27 2014-10-01 联想(北京)有限公司 Information processing method and electronic device
CN109391792A (en) * 2017-08-03 2019-02-26 腾讯科技（深圳）有限公司 Method, apparatus, terminal and the computer readable storage medium of video communication
CN108600680A (en) * 2018-04-11 2018-09-28 南京粤讯电子科技有限公司 Method for processing video frequency, terminal and computer readable storage medium
CN108366221A (en) * 2018-05-16 2018-08-03 维沃移动通信有限公司 A kind of video call method and terminal
CN109831636A (en) * 2019-01-28 2019-05-31 努比亚技术有限公司 Interdynamic video control method, terminal and computer readable storage medium
CN110602516A (en) * 2019-09-16 2019-12-20 腾讯科技（深圳）有限公司 Information interaction method and device based on live video and electronic equipment

Also Published As

Publication number Publication date
CN112752161A (en) 2021-05-04

Similar Documents

Publication Publication Date Title
CN108900920B (en) Live broadcast processing method, device, equipment and storage medium
US9818225B2 (en) Synchronizing multiple head-mounted displays to a unified space and correlating movement of objects in the unified space
CN109040849B (en) Live broadcast platform interaction method, device, equipment and storage medium
CN108986192B (en) Data processing method and device for live broadcast
CN111698567A (en) Game fighting method and device for live broadcast room
US20230336684A1 (en) Cooperative photographing method and apparatus, electronic device, and computer-readable storage medium
CN111870935B (en) Business data processing method and device, computer equipment and storage medium
JP7431497B2 (en) Game provision method and system based on video calls and object recognition
CN113518264A (en) Interaction method, device, terminal and storage medium
CN113457123A (en) Interaction method and device based on cloud game, electronic equipment and readable storage medium
KR20220065856A (en) Information processing methods, systems, devices, devices and storage media
CN112717423A (en) Live broadcast method, device, equipment and storage medium for game match
CN109963628A (en) Game server and the method for generating context aware game information
CN113996053A (en) Information synchronization method, device, computer equipment, storage medium and program product
CN110336957B (en) Video production method, device, medium and electronic equipment
CN109039851B (en) Interactive data processing method and device, computer equipment and storage medium
CN112752161B (en) Interaction method based on real-time communication, social application terminal and related device
CN112015506B (en) Content display method and device
CN113274727A (en) Live broadcast interaction method and device, storage medium and electronic equipment
CN114430494B (en) Interface display method, device, equipment and storage medium
CN114760520A (en) Live small and medium video shooting interaction method, device, equipment and storage medium
CN114915852A (en) Video call interaction method and device, computer equipment and storage medium
CN114425167A (en) Interaction method and device of virtual object, storage medium and electronic equipment
CN113810253A (en) Service providing method, system, device, equipment and storage medium
WO2023082737A1 (en) Data processing method and apparatus, and device and readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 40044552

Country of ref document: HK

SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant