WO2021135334A1

WO2021135334A1 - Method and apparatus for processing live streaming content, and system

Info

Publication number: WO2021135334A1
Application number: PCT/CN2020/112856
Authority: WO
Inventors: 王云
Original assignee: 广州华多网络科技有限公司
Priority date: 2019-12-31
Filing date: 2020-09-01
Publication date: 2021-07-08
Also published as: CN111083515B; CN111083515A

Abstract

Provided is a method for processing live streaming content. The method comprises: receiving an original video stream sent by an anchor client by means of a video channel and additional live streaming information sent by the anchor client by means of a signaling channel; according to the additional live streaming information, synthesizing the original video stream and the additional live streaming information into a live streaming video stream, and sending the live streaming video stream to an audience client; and storing a specified image frame of the original video stream on the basis of an evaluation index of user behaviors. The audience client can view the live streaming video stream without perception, and the original video stream can be conveniently shared to other video platforms in a timely manner.

Description

Method, device and system for processing live content

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on December 31, 2019, the application number is 201911407509.7, and the invention title is "Methods, Devices, and Systems for Processing Live Content", the entire content of which is incorporated herein by reference. Applying.

Technical field

This application relates to the field of live broadcast, and in particular to a method, device, and system for processing live broadcast content.

Background technique

Nowadays, short videos are appearing and integrated into people’s lives. Using short videos on various short video platforms has become one of the ways for people to entertain and spend their lives. Some short video content is some wonderful moments recorded by the anchor during the live broadcast. Wonderful moments often require the live broadcast platform to filter the live broadcast content of the host to find the brilliant moments during the live broadcast of the high-quality host, and then publish them to a third-party short video platform.

However, the live broadcast pictures of the host on the live broadcast platform often include not only the original picture of the host obtained through the host’s camera, but also some advertisements such as stickers and subtitles added by the host himself or the copyright mark of the live broadcast platform. These advertisements and The logo must appear in the live broadcast screen of the host on the live broadcast platform, but cannot appear in the short video screen of the wonderful moments published on the short video platform of the third party. However, the short video generated on the live screen of the selected host is posted to the third party In the short video platform, the generated short video pictures usually carry these advertisements and logos. As a result, the short video pictures of the wonderful moments posted on the third-party short video platform also include advertisements and logos. Therefore, how to keep the live broadcast While there are advertisements and logos in the live broadcast screens of the anchors on the platform, it is a problem that must be solved to avoid the short video screens of the wonderful moments posted to the short video platform of the third party carrying the advertisements and logos.

Summary of the invention

In view of this, the present application provides a method, device, live broadcast system, equipment, and computer-readable storage medium for processing live broadcast content.

According to a first aspect of the embodiments of the present application, there is provided a method for processing live content, the method including:

Receive the original video stream sent by the host client through the video channel and the additional live broadcast information sent through the signaling channel;

Synthesize the original video stream and the live additional information into a live video stream according to the live additional information, and send the live video stream to the viewer client; and

The specified image frame of the original video stream is saved based on the evaluation index of the user behavior.

According to a second aspect of the embodiments of the present application, there is provided a method for processing live content, the method including:

Obtain the original video stream;

Send the original video stream and the live additional information to the server respectively, so that the server synthesizes the original video stream and the live additional information into the live video stream according to the live additional information, sends the live video stream to the viewer client, and The user behavior evaluation index saves the designated image frames of the original video stream, where the original video stream is sent through a video channel, and the live broadcast additional information is sent through a signaling channel.

According to a third aspect of the embodiments of the present application, a live broadcast system is provided, and the live broadcast system includes an anchor client, a server, and an audience client;

The anchor client is used to obtain the original video stream;

Sending the original video stream and the live broadcast additional information to the server respectively, where the original video stream is sent through a video channel, and the live broadcast additional information is sent through a signaling channel;

The server is configured to synthesize the original video stream and the additional information of the live broadcast into a live video stream, and send the live video stream to the viewer client; and

Saving the designated image frame of the original video stream based on the evaluation index of the user behavior;

The audience client is used to receive and display the live video stream.

According to a fourth aspect of the embodiments of the present application, there is provided an apparatus for processing live content, the apparatus including:

The receiving module is used to receive the original video stream sent by the host client through the video channel and the live broadcast additional information sent through the signaling channel;

A synthesis module for synthesizing the original video stream and the live broadcast additional information into a live video stream according to the live broadcast additional information;

The transmission module is used to send the live video stream to the audience client;

The saving module is configured to save the designated image frame of the original video stream based on the evaluation index of the user behavior.

According to a fifth aspect of the embodiments of the present application, there is provided an apparatus for processing live content, the apparatus including:

The acquisition module is used to acquire the original video stream;

The sending module is configured to send the original video stream and live additional information to the server respectively, so that the server synthesizes the original video stream and the live additional information into the live video stream according to the live additional information signaling to the viewer client The live video stream is sent, and the designated image frame of the original video stream is saved based on the evaluation index of the user behavior, wherein the original video stream is sent through a video channel, and the live broadcast additional information is sent through a signaling channel.

The host client of this application sends the original video stream and live additional information to the server through different transmission channels, and the server completes the action of synthesizing the live additional information and the original video stream into the live video stream instead of pushing the stream by the host client When the time is completed, since the server can obtain the original video stream without additional live broadcast information, this application can not only allow the viewer client to watch the live video stream imperceptibly, but also share the original video stream to other video platforms in a timely and convenient manner.

Description of the drawings

Fig. 1a is a schematic diagram of a picture of an original video stream collected in a live broadcast scenario according to an exemplary embodiment of the present application;

Fig. 1b is a schematic diagram of a synthesized live broadcast picture according to an exemplary embodiment of the present application;

Fig. 2 is a flowchart of a method for processing live content shown in an exemplary embodiment of the present application;

Fig. 3 is a flowchart of another method for processing live content shown in an exemplary embodiment of the present application;

Figure 3a is an application example in an application scenario of this application;

Fig. 4 is a schematic diagram of an apparatus for processing live content shown in an exemplary embodiment of the present application;

Fig. 5 is a schematic diagram of another device for processing live content shown in an exemplary embodiment of the present application;

Fig. 6 is a schematic diagram of an electronic device shown in an exemplary embodiment of the present application.

Detailed ways

The exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the application as detailed in the appended claims.

The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and/or" as used herein refers to and includes any or all possible combinations of one or more associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determination".

With the outbreak of short videos, short videos are now fully integrated into people’s lives. Everyone can be a director anytime, anywhere. They can record their own or the wonderful moments around them through the camera and share them on major short video platforms for viewing on the other side of the screen. Audience watching, which also makes short videos on short video platforms one of the most common ways of entertainment and consumption. The content of some short videos is usually some wonderful moments recorded by the host during the live broadcast. These wonderful moments often require the live broadcast platform to filter the live broadcast content of the host to find the wonderful moments during the live broadcast of the high-quality host. Then publish it to a third-party short video platform.

However, the live broadcast screen of the host on the live broadcast platform often includes not only the original image of the host obtained through the host’s camera, or the original image captured by the host on the terminal display interface, but also includes some stickers and subtitles added by the host himself. Some advertisements or copyright signs of live broadcast platforms, as shown in Figure 1a, this screen is the host’s original image obtained by the host’s camera. No additional live broadcast information is added to the host’s original image, and only the host’s image obtained by the host’s camera is displayed. .

As shown in Figure 1b, the screen includes some text, image advertisements, and copyright identification information added by the host on the original screen obtained by the camera. The host uses this screen as a live broadcast for viewers to watch, Figure 1b These advertisements and logos in the live broadcast screen shown are necessary for the live broadcast platform, so they must appear in the live broadcast screen of the host on the live broadcast platform, but cannot appear on the short video platform published to the third party. In the short video screen at the moment.

This application provides a method for processing live content. The server combines the original video stream sent by the host client and the live additional information into the live video stream, and sends the live video stream to the viewer client, as well as evaluation indicators based on user behavior Save the designated image frame of the original video stream. The live video stream synthesized by the original video stream and the additional information of the live broadcast is used for the live broadcast, and the specified image frames of the original video stream are saved for publishing to a third-party video platform.

Fig. 2 is a flowchart of a method for processing live content shown in an exemplary embodiment of the application. As shown in Fig. 2, the method includes the following steps:

S201: Receive the original video stream sent by the host client through the video channel and the live additional information sent through the signaling channel;

S202: Synthesize the original video stream and the live additional information into a live video stream according to the live additional information, and send the live video stream to the viewer client;

S203: Save the designated image frame of the original video stream based on the evaluation index of the user behavior.

In S201, the original image frame sent by the host client can be the image frame of any screen captured in real time by the camera, and the frame can be the screen that the host wants to use for live broadcast, for example, the host can capture itself through the camera during the live broadcast. , Or some pictures of the surrounding environment of the host. The acquired original image frame can also be the screen captured by the host on the terminal display interface, such as a game or movie being broadcast by the host, the host can capture the game interface or movie playback interface displayed on a mobile device or a fixed device as the original image frame .

The live broadcast additional information signaling carries live broadcast additional information. The live broadcast additional information may indicate some subtitles and copyright identifications added to the original image frame, or texture pictures added manually by the host, and these subtitles and pictures may be some advertisement information.

In S202, the live broadcast additional information includes the attribute information and location information of the live broadcast additional information. The attribute information is some fixed attributes of the live broadcast additional information. For example, when the live broadcast additional information is subtitles, the attribute information can be the style of subtitles, for example, it can be subtitled. Font, size, color, background color and other text styles. When the live broadcast additional information is used to obtain the relevant information of the picture, the attribute information can be the URL of the picture and the zoom factor of the picture according to the attribute information and location information. One or both of these parameters. The location information is the coordinates of the live broadcast additional information added to the image frame of the original video stream. According to the live broadcast additional information signaling, the original video stream and the live broadcast additional information are combined into the live video stream. The original video stream can be combined with the live broadcast according to the attribute information and location information. The additional information is combined into a live video stream.

In S203, the designated image frame is the original image frame determined by the server based on the evaluation index of the user behavior. The evaluation index of the user behavior may include at least one of the number of gifts and the activity of the public screen. Of course, it may be other than these two. Evaluation indicators other than indicators. Which parameter to choose as the evaluation index can be determined according to the characteristics of the specified image frame that the person using this scheme needs to select. For example, the number of gifts or fair activity can also be used as the evaluation index, or other parameters can be used as the evaluation index. index.

For example, the following describes an example in which an image frame of a wonderful moment is used as a designated image frame. In the live broadcast scenario, the server receives the live video stream sent by the host client and sends the live video stream to the audience client for viewing by the audience client. The audience can enter the live broadcast room of the host they like according to their personal preferences. Give a gift to your favorite host or leave a message on the public screen in the live broadcast room, send a bullet screen to communicate with the host or other viewers who watch the live broadcast together. As the number of bullet screens and the number of gifts increases, or the value of the gift is higher, the host’s The heat of the live broadcast room will also rise. When the server sends the live video frame to the audience client, it will use the real-time audience barrage, the number of audience gifts, the current live broadcast room heat, etc. as evaluation indicators, and real-time statistics of the audience barrage in the host's live room corresponding to the current live video frame According to these evaluation indicators, determine whether the current live video frame is a video frame of a wonderful moment. The wonderful moment refers to if the audience barrage in the live broadcast room of the host within a certain period of time If the number, the number of viewers giving gifts, and the current popularity of the live broadcast room reach a certain index, then it is determined that the live video frames within the time period constitute a wonderful moment. The specified image frame can be the original image frame corresponding to the live video frame included in the time period of the wonderful moment, that is, the original image frame of the wonderful moment. Since the server will judge in real time whether the current moment is a wonderful moment, when it is judged that the current moment is a wonderful moment Start recording the received original image frame at the moment, and stop recording and save the recorded file as a video file when the current moment is judged to be a non-exciting moment. The format of the video file can be, for example, MP4 format, AVI format, etc., and the server It will continue to judge whether the current moment is a wonderful moment. Once it is judged that the current moment is a wonderful moment, it will be recorded and saved again. Since the original image frame of the wonderful moment does not carry some additional live broadcast information such as subtitles, copyright signs, or stickers manually added by the anchor, it can be used to publish to other video platforms.

As an example, the process for the server to store the specified file can refer to the following process: the server can store the original video stream in segments according to the time sequence of the received original video stream, and judge whether there is a specified image frame in the saved original video stream. The timing can be performed in real time or asynchronously with the saved action. For example, each time a file is saved, it can be queried whether the time period includes a wonderful moment. If the time period includes a wonderful moment, it is determined that the original image frame of the wonderful moment exists in the file, and the file is compared with other original image frames of the wonderful moment. File integration and archiving; if the time period of the file does not include the wonderful moments, it is determined that the original image frames of the wonderful moments do not exist in the file, and the file can be deleted directly to avoid occupying the memory space of the device. Since the files uniformly sent to the server after integrated archiving are all files that include the original image frames of the wonderful moments, there is no need for the server to filter again, so that users can directly share these files obtained by the server to other video platforms when needed.

In one embodiment, the designated image frame may be distributed to the current live broadcast platform or a third-party video platform. The user can obtain the video files of the wonderful moments saved on the server from the server, and the user can edit the obtained video files, for example, perform appropriate editing, add special effects, etc., and then distribute the processed video files to third-party videos The platform is for viewers of third-party video platforms to watch these wonderful moments, or the processed video files can also be distributed to the current live broadcast platform for viewers of the current live broadcast platform to watch these wonderful moments.

Fig. 3 is a flowchart of a method for processing live content shown in an exemplary embodiment of the application. As shown in Fig. 3, the method includes the following steps:

S301: Obtain the original video stream;

S302: Send the original video stream and the live broadcast additional information signaling to the server respectively, so that the server synthesizes the original video stream and the live broadcast additional information into the live video stream according to the live broadcast additional information signaling, and sends it to the viewer client. The live video stream and the evaluation index based on user behavior save the designated image frames of the original video stream.

This application also provides a live broadcast system, which includes an anchor client, a server, and an audience client;

The anchor client is used to obtain the original video stream;

Separately sending the original video stream and the live broadcast additional information signaling to the server; and,

As an example, the host client can also locally synthesize the original image frame and the additional information of the live broadcast into a local live image frame for the host to watch the live image locally. In addition, the server can directly send the saved file of the specified image frame to the third-party video platform, or the user can obtain the file of the specified image frame from the server and publish it directly or after editing and publishing to the third-party video platform for viewers on the third-party video platform Watch.

In order to better understand this solution, an application example is listed below. In the live broadcast environment as shown in FIG. 3a, there are an anchor client 301a, several viewer clients 302a, a media server 303a, a synthesis server 304a, a live video stream database 305a and an original video stream database 306a in the live broadcast system.

The host can enter the live broadcast room through the host client 301a. When the host performs live broadcast, the camera collects the original video stream, and pushes the collected original video stream to the media server 303a in the form of streaming media data. In addition, it will be presented to the media server 303a. Some additional live broadcast information of the viewer is sent to the composition server 304a in the form of signaling (for example, through a channel for sending control signaling).

After receiving the original video stream, the media server 303a also sends the original video stream to the synthesis server 304a.

The synthesis server 304a needs to synthesize the original video stream and the live broadcast additional information to generate a live video stream for the viewer function client to watch. The live video stream viewed by the viewer is a video picture with subtitles, copyright identification, and other information attached. In addition, the synthesis server 304a stores the synthesized live video stream in the live video stream database 305a, and the original video stream in the original video stream database 306a. When saving the original video stream, the original video stream can be stored in multiple folders according to the time sequence of the received file.

The live video stream is distributed by the synthesis server 304a to each viewer client 302a in real time. As for the video data stored in the original video stream database 306a, it can be judged whether it contains the image frame of the wonderful moment in real time or at an appropriate time according to different needs. The judgment method can be realized by means of AI image recognition. The basis for judgment can be Use certain evaluation indicators that can analyze user behavior. File the retrieved folders containing the image frames of the notable moments; delete the folders that do not contain the image frames of the notable moments.

The original video stream stored in the original video stream database 306a can be distributed to other video platforms at an appropriate time, or the user can use a client to obtain it from the synthesis server 304a and send it to the user, so that the user can send other videos to other video streams. The need for platform sharing.

As can be seen from Figure 3a, when the synthesis server 304a pushes the synthesized live video stream, it may not push it to the anchor client 301a, because the anchor client 301a can obtain the original video stream collected by the anchor locally. , It is also possible to obtain additional live broadcast information such as subtitles and pictures, so the host client 301a also has the ability to synthesize live video streams. The host client 301a can preview the live broadcast effect locally after completing the synthesis of the local live video stream, which makes the preview process more efficient.

Corresponding to the foregoing embodiment of the method for processing live content, this application also provides an embodiment of an apparatus for processing live content.

As shown in FIG. 4, FIG. 4 is an apparatus 400 for processing live content shown in an exemplary embodiment of this application, including:

The receiving module 401 is configured to receive the original video stream sent by the host client through the video channel and the live broadcast additional information sent through the signaling channel;

The synthesis module 402 is configured to synthesize the original video stream and the live broadcast additional information into a live video stream according to the live broadcast additional information;

The transmission module 403 is configured to send the live video stream to the audience client;

The saving module 404 is configured to save the designated image frame of the original video stream based on the evaluation index of the user behavior.

As shown in FIG. 5, FIG. 5 shows an apparatus 500 for processing live content according to an exemplary embodiment of this application, including:

The obtaining module 501 is used to obtain the original video stream;

The sending module 502 is configured to send the original video stream and the live broadcast additional information to the server respectively, so that the server combines the original video stream and the live broadcast additional information into the live video stream according to the live broadcast additional information signaling, and presents it to the audience client The terminal sends the live video stream, and saves the designated image frame of the original video stream based on the evaluation index of the user behavior, wherein the original video stream is sent through a video channel, and the live broadcast additional information is sent through a signaling channel.

The embodiments of the apparatus for processing live content in this application can be applied to devices. The device embodiments can be implemented by software, or can be implemented by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory through the processor of the device where it is located. From a hardware perspective, as shown in FIG. 6, a hardware structure diagram of the device where the device for processing live content of this application is located, except for the processor, memory, network interface, and non-volatile memory shown in FIG. In addition, the device where the device is located in the embodiment usually includes other hardware according to the actual function of the device, which will not be repeated here.

Wherein, the non-volatile memory is used to store executable instructions of the processor, and the processor is configured to execute the instructions to implement the method for processing live content described in any of the foregoing embodiments.

The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method for processing live content described in any of the above embodiments is implemented.

For the implementation process of the functions and roles of each unit in the foregoing device, refer to the implementation process of the corresponding steps in the foregoing method for details, and details are not described herein again.

For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can refer to the part of the description of the method embodiment. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present application. Those of ordinary skill in the art can understand and implement without creative work.

The above descriptions are only preferred embodiments of this application and are not intended to limit this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in this application Within the scope of protection.

Claims

A method for processing live content, characterized in that the method includes:

Receive the original video stream sent by the host client through the video channel and the additional live broadcast information sent through the signaling channel;

Synthesize the original video stream and the live additional information into a live video stream according to the live additional information, and send the live video stream to the viewer client; and

The specified image frame of the original video stream is saved based on the evaluation index of the user behavior.
The method according to claim 1, wherein the additional live broadcast information includes at least:

Subtitles, related information and/or copyright identification used to obtain pictures.
The method according to claim 2, wherein the live broadcast additional information includes attribute information and location information of the live broadcast additional information.
The method according to claim 3, wherein the step of synthesizing a live video stream specifically comprises:

The original video stream and the live broadcast additional information are combined into a live video stream according to the attribute information and location information.
The method according to claim 4, wherein the attribute information includes the style of the subtitle, the URL of the picture, and/or the zoom factor of the picture;

The position information includes the coordinates of the subtitle and/or the picture added on the image frame of the original video stream.
The method according to claim 1, wherein the evaluation index of the user behavior includes the number of gifts and/or public screen activity.
A method for processing live content, characterized in that the method includes:

Obtain the original video stream;

Send the original video stream and the live additional information to the server respectively, so that the server synthesizes the original video stream and the live additional information into the live video stream according to the live additional information, sends the live video stream to the viewer client, and The user behavior evaluation index saves the designated image frames of the original video stream, where the original video stream is sent through a video channel, and the live broadcast additional information is sent through a signaling channel.
The method according to claim 7, wherein the method is executed by the host client, and the method further comprises:

The original video stream and the live broadcast additional information are locally synthesized into a live video stream, and the live video stream is played locally.
A live broadcast system, characterized in that the live broadcast system includes an anchor client, a server, and an audience client;

The anchor client is used to obtain the original video stream;

Sending the original video stream and the live broadcast additional information to the server respectively, where the original video stream is sent through a video channel, and the live broadcast additional information is sent through a signaling channel;

The server is configured to synthesize the original video stream and the additional information of the live broadcast into a live video stream, and send the live video stream to the viewer client; and

Saving the designated image frame of the original video stream based on the evaluation index of the user behavior;

The audience client is used to receive and display the live video stream.
A device for processing live broadcast content, characterized in that the device includes:

The receiving module is used to receive the original video stream sent by the host client through the video channel and the live broadcast additional information sent through the signaling channel;

A synthesis module, configured to synthesize the original video stream and the live broadcast additional information into a live video stream according to the live broadcast additional information;

The transmission module is used to send the live video stream to the audience client;

The saving module is configured to save the designated image frame of the original video stream based on the evaluation index of the user behavior.
A device for processing live broadcast content, characterized in that the device includes:

The acquisition module is used to acquire the original video stream;

The sending module is used to send the original video stream and the live additional information to the server respectively, so that the server combines the original video stream and the live additional information into the live video stream according to the live additional information signaling to the viewer client The live video stream is sent, and the designated image frame of the original video stream is saved based on the evaluation index of the user behavior, wherein the original video stream is sent through a video channel, and the live broadcast additional information is sent through a signaling channel.