WO2024055689A1

WO2024055689A1 - Interaction method and apparatus for streaming media

Info

Publication number: WO2024055689A1
Application number: PCT/CN2023/103469
Authority: WO
Inventors: 林霖; 庞峰
Original assignee: 华为技术有限公司
Priority date: 2022-09-15
Filing date: 2023-06-28
Publication date: 2024-03-21
Also published as: CN117750042A

Abstract

An interaction method and apparatus for streaming media. The method comprises: upon receiving a first interaction indication, which is sent by a first server by means of streaming media and configured to indicate the start of participation in terms of interaction with regard to a program broadcasted by a terminal device, the terminal device may output a first interaction prompt to prompt a user to participate in the interaction with regard to the program broadcasted by the terminal device, and may collect at least one piece of interaction information generated when the user performs interaction according to the first interaction prompt, wherein the at least one piece of interaction information is configured to generate interaction data; and the terminal device may then send the interaction data to a second server. By means of the method, users who view the program currently broadcasted by the terminal device can all participate in the interaction with regard to the program currently broadcasted by the terminal device; moreover, interaction can be implemented without the need to install, in the terminal device, an application program having an interaction function or without the need for the users to use the other terminal devices that are provided with application programs for interaction. Therefore, the interaction is easily implemented.

Description

A streaming media interaction method and device

Cross-references to related applications

This application claims priority to the Chinese patent application filed with the China Patent Office on September 15, 2022, with application number 202211124951.0 and application title "A streaming media interactive method and device", the entire content of which is incorporated into this application by reference. middle.

Technical field

The present application relates to the field of communication technology, and in particular, to a streaming media interaction method and device.

Background technique

Currently, as a form of streaming media, live video or on-demand video services are widely used. For example, during holidays, TV stations broadcast live holiday parties. For another example, users can watch on-demand services provided by TV stations through terminal devices such as televisions. For another example, anchors can use online media communication technology to conduct online live broadcasts.

When users watch live or on-demand videos provided by TV stations, the direction of information transmission is one-way. Users can only watch the video but cannot interact with it.

When watching online live videos, users can interact in the following ways: users enter comment information, including text, pictures, etc., in the comment area below the live broadcast screen; users send barrages and other short comment information, including text, in the live broadcast screen , clap, whistle and other animated sound effects, send various gifts, etc. However, these interactive methods have nothing to do with the control surface and media surface of streaming media. Instead, they superimpose interactive capabilities on streaming media and require corresponding applications to be installed in the user's terminal device.

Contents of the invention

This application provides a streaming media interaction method and device, so that users who watch streaming media programs can participate in interactions with the program.

In the first aspect, embodiments of the present application provide a streaming media interaction method. The method includes: after receiving a first interaction instruction sent by the first server through streaming media to instruct the user to start participating in the interaction of the terminal device playing the program, the terminal device can output a first interaction prompt to prompt the user to participate in the terminal device playing program. interact with the program, and collect at least one interaction information generated when the user interacts according to the first interaction prompt, and the at least one interaction information is used to generate interaction data. Then, the terminal device can send the interaction data to the second server.

Through this method, users who watch the program currently played by the terminal device can participate in the interaction of the program currently played by the terminal device, and there is no need for the terminal device to install an application program with interactive functions or the user needs to use other terminals with applications for interaction Only equipment can achieve interaction and it is easy to implement.

In a possible design, the interaction data includes: at least one interaction information; and the frame sequence number of the streaming media frame of the program being displayed by the terminal device when each interaction information in the at least one interaction information is collected. Through this design, the terminal device not only sends at least one interaction information to the second server, but also sends the frame sequence number of the streaming media frame of the program being displayed by the terminal device when collecting each interaction information in the at least one interaction information, so that it can be based on the The frame sequence number determines the user's response time, which can reduce the unfairness of interaction caused by the different delivery times of streaming media programs to different users.

In a possible design, the interaction data also includes: when each interaction information in at least one interaction information is collected, the displayed duration of the streaming media frame of the program being displayed by the terminal device. Through this design, the terminal device also collects the displayed duration of the streaming media frame of the program being displayed by the terminal device when each interactive information in at least one interactive information is collected from the second server, so that the user's reaction time can be determined more accurately. This further reduces the unfairness of interaction caused by the different delivery times of streaming media programs to different users.

In a possible design, the media plane data of the first video stream of the program played by the terminal device includes the frame sequence number of the streaming media frame in the program. That is to say, the terminal device can obtain the frame sequence number of the streaming media frame in the program through the media plane. Through this design, when the terminal device obtains the program streaming media, it can obtain the frame sequence number of the streaming media frame in the program, thereby efficiently obtaining the frame sequence number. Moreover, the terminal device can obtain the frame sequence number through the media plane, which is easy to implement.

In a possible design, the terminal device may also receive an interactive mode indication sent by the first server through streaming media, where the interactive mode indication is used to instruct the terminal device to use an interactive mode for playing the program. At this time, the first interaction prompt output by the terminal device can prompt the user to participate in the interaction of the program played by the terminal device according to the interaction method, and the terminal device can collect at least one interaction information generated when the user interacts according to the interaction method. Through this design, the terminal device can collect at least one interaction information generated when the user interacts according to the interaction method according to the interaction method instructed by the first server, so that the user's interaction information can be obtained in a targeted manner, and the acquisition and transmission of interaction information can be improved. Efficiency of interactive data.

In a possible design, the interaction method may include at least one of the following: interaction through somatosensory, interaction through voice recognition, interaction through video, interaction through voice, and interaction through remote control. Through this design, the terminal device can support multiple interaction methods.

In one possible design, the terminal device can receive the interaction mode indication through one of the following implementations:

Implementation method 1: The terminal device receives a third video stream of the program from the first server, and the media plane data of the third video stream includes an interaction mode indication. That is to say, the terminal device can receive the interaction mode indication from the first server through the media plane. Through the first implementation method, the terminal device can obtain the interactive mode instructions while obtaining the program streaming media. The acquisition method is fast and efficient. Moreover, the terminal device can obtain interactive mode instructions through the media surface, which is easy to implement.

Implementation method two: the terminal device receives third control plane data from the first server, and the third control plane data includes interaction mode instructions. That is to say, the terminal device can receive the interaction mode instruction from the first server through the control plane. Through the second implementation method, the terminal device can obtain the interactive mode instructions through the control surface, which is easy to implement.

In one possible design, the terminal device may receive an address indication sent by the first server through streaming media, where the address indication is used to indicate a reporting address of the interactive data. Then, the terminal device can send the interaction data to the second server according to the reported address, where the reported address is the address of the second server. Through this design, the terminal device can upload the interactive data to the correct server according to the reporting address indicated by the first server.

In one possible design, the end device may receive the address indication through one of the following implementations:

Implementation Mode 1: The terminal device receives the fourth video stream of the program from the first server, and the media plane data of the fourth video stream includes the address indication. That is to say, the terminal device can receive the address indication from the first server through the media plane. Through implementation method 1, the terminal device can obtain the address indication while obtaining the program streaming media, making it more efficient to obtain the address information. Moreover, the terminal device can obtain the address indication through the media plane, which is easy to implement.

Implementation Mode 2: The terminal device receives fourth control plane data from the first server, where the fourth control plane data includes the address indication. That is to say, the terminal device can receive the address indication from the first server through the control plane. Through implementation method 2, the terminal device can obtain the address indication through the control plane, which is easy to implement.

In a possible design, the terminal device may receive the first interaction indication in one of the following ways:

Method 1: The terminal device receives the first video stream of the program from the first server, and the media plane data of the first video stream includes the first interaction indication. That is to say, the terminal device can receive the first interaction instruction from the first server through the media plane. Through the first method, the terminal device can obtain the first interaction instruction while obtaining the program content, so that it can deliver the corresponding first interaction instruction while delivering the streaming media of the program. Moreover, the terminal device can obtain the first interactive instruction through the media surface, which is easy to implement.

Method 2: The terminal device receives the first control plane data from the first server, where the first control plane data includes the first interaction instruction. That is to say, the terminal device can receive the first interaction instruction from the first server through the control plane. Through the second method, the terminal device can obtain the first interactive instruction through the control surface, which is easy to implement.

In a possible design, the terminal device can send interactive data to the second server in one of the following ways:

Method 1: The terminal device sends the second video stream to the second server, and the media plane data of the second video stream includes interactive data. That is to say, the terminal device can send the second video stream to the second server through the media. Through method 1, the terminal device can send interactive data to the second server through the media stream, achieving higher efficiency.

Method 2: The terminal device sends second control plane data to the second server, and the second control plane data includes interaction data. That is to say, the terminal device can send interactive data to the second server through control. Through method 2, the terminal device can send interactive data to the second server through control, which is easy to implement.

In one possible design, before receiving the first interaction instruction sent by the first server through streaming media, the terminal device may receive the second interaction instruction sent by the first server through streaming media, and the second interaction instruction is used to indicate Prepare to participate in the interaction of programs played on terminal devices. Then, the terminal device can output a second interaction prompt. The second interaction prompt is used to prompt the user to prepare to participate in the interaction of the program played by the terminal device. Through this design, the terminal device can prompt the user to prepare to start participating in the interaction of the current program, giving the user a certain amount of preparation time.

In a possible design, the terminal device may receive the second interaction indication in one of the following ways:

Method A: The terminal device receives the fifth video stream of the program from the first server, and the media plane data of the fifth video stream includes the second interaction indication. That is to say, the terminal device can receive the second interaction instruction from the first server through the media plane. Through method A, the terminal device can obtain the second interaction instruction while obtaining the program content, so that it can deliver the corresponding second interaction instruction while delivering the streaming media of the program. Moreover, the terminal device obtains the second interaction instruction through the media surface, which is easy to implement.

Method B: The terminal device receives the fifth control plane data from the first server, and the fifth control plane data includes the second interaction instruction. also That is to say, the terminal device can receive the second interaction instruction from the first server through the control plane. Through method B, the terminal device can obtain the second interaction instruction through the control surface, which is easy to implement.

In a possible design, the terminal device may also receive a third interaction instruction sent by the first server through streaming media, where the third interaction instruction is used to instruct to stop participating in the interaction of the program played by the terminal device. Then, the terminal device may stop collecting at least one interaction information generated when the user interacts according to the first interaction prompt. Through this design, the terminal device can stop collecting user interaction information in time under the instruction of the first server, thereby saving the energy required by the terminal device to collect user interaction information.

In a possible design, the terminal device can receive the third interaction indication through one of the following implementation methods:

Implementation A: The terminal device receives a sixth video stream containing a program from the first server, and the media plane data of the sixth video stream contains a third interaction indication. That is to say, the terminal device can receive the third interaction instruction from the first server through the media plane. Through implementation method A, the terminal device can obtain the third interaction instruction while obtaining the program content, so that the corresponding third interaction instruction can be issued while delivering the streaming media of the program. Moreover, the terminal device can obtain the third interactive instruction through the media surface, which is easy to implement.

Implementation mode B: The terminal device receives sixth control plane data from the first server, and the sixth control plane data includes the third interaction indication. That is to say, the terminal device may receive the third interaction instruction from the first server through the control plane. Through implementation method B, the terminal device can obtain the third interactive instruction through the control plane, which is easy to implement.

In a second aspect, embodiments of the present application provide a streaming media interactive device, including a unit for executing each of the steps in the above first aspect.

In a third aspect, embodiments of the present application provide a streaming media interactive device, including at least one processing element and at least one storage element, wherein the at least one storage element is used to store programs and data, and the at least one processing element is used to read and execute the programs and data stored in the storage element, so that the method provided in the first aspect of this application is implemented.

In a fourth aspect, embodiments of the present application further provide a computer program, which when the computer program is run on a computer, causes the computer to execute the method provided in the first aspect.

In a fifth aspect, embodiments of the present application further provide a computer-readable storage medium. A computer program is stored in the computer-readable storage medium. When the computer program is executed by a computer, it causes the computer to execute the first aspect. provided method.

In a sixth aspect, embodiments of the present application further provide a chip, which is used to read a computer program stored in a memory and execute the method provided in the first aspect.

In a seventh aspect, embodiments of the present application further provide a chip system. The chip system includes a processor and is configured to support a computer device in implementing the method provided in the first aspect. In a possible design, the chip system further includes a memory, and the memory is used to store necessary programs and data of the computer device. The chip system can be composed of chips or include chips and other discrete devices.

The technical effects that can be achieved by any of the above-mentioned second to seventh aspects can be described with reference to the technical effects that can be achieved by any of the possible designs in the above-mentioned first aspect, and the repeated points will not be discussed.

Description of drawings

Figure 1 is a schematic diagram of a real-time transmission protocol (RTP) fixed header;

Figure 2 is a schematic diagram of an RTP extension header;

Figure 3 is a schematic diagram of a media presentation description (MPD) file;

Figure 4 is an architecture diagram of a communication system;

Figure 5 is a flow chart of the first streaming media interaction method provided by the embodiment of the present application;

Figure 6 is a flow chart of the second streaming media interaction method provided by the embodiment of the present application;

Figure 7 is a flow chart of the third streaming media interaction method provided by the embodiment of the present application;

Figure 8 is a flow chart of the fourth streaming media interaction method provided by the embodiment of the present application;

Figure 9 is a flow chart of the fifth streaming media interaction method provided by the embodiment of the present application;

Figure 10 is a structural diagram of the first streaming media interactive device provided by an embodiment of the present application;

Figure 11 is a structural diagram of a second streaming media interactive device provided by an embodiment of the present application.

Detailed ways

This application provides a streaming media interaction method and device, so that when users watch streaming media programs (similar to satellite TV programs, live broadcast programs, etc.) broadcast by TV stations through terminal devices such as televisions and mobile phones, the user can participate in the program. interaction, and can directly Collect user interaction data directly, thereby eliminating the need to install some interactive applications on terminal devices such as TVs and mobile phones, thereby simplifying the user's interaction process and complexity on the basis of realizing user participation in interaction. Among them, the method and the device are based on the same technical concept. Since the principles of the method and the device to solve the problem are similar, the implementation of the device and the method can be referred to each other, and the repeated points will not be repeated.

In the following, some terms used in this application will be explained first to facilitate understanding by those skilled in the art.

1) Streaming media refers to a media format that uses streaming transmission technology for continuous real-time playback. In this application, streaming media may, but is not limited to, include live video or on-demand video.

2) RTP protocol is a media-side protocol for streaming media.

When transmitting a video stream via the RTP protocol, the RTP header of the video stream may include an RTP fixed header. The format of the RTP fixed header is shown in Figure 1. The RTP protocol allows adding an RTP extension header after the RTP fixed header. Therefore, the RTP header of the video stream can also include an RTP extension header. For example, setting the X field in the RTP fixed header to 1 adds a variable-length RTP extension header after the RTP fixed header. The format of the RTP extension header can be shown in Figure 2.

The payload in the RTP protocol can be used to transmit network abstract layer (NAL) messages. For example, it can be used to transmit NAL messages of the H.264 or H.265 protocol. NAL messages of the H.264 or H.265 protocol can include the following types of messages:

Sequence parameter set (SPS): can save a set of global parameters of a coded video sequence.

Picture parameter set (PPS): corresponds to an image or parameters of an image in a sequence.

Supplemental enhancement information (SEI): Additional information can be added to the video stream.

3) The dynamic adaptive streaming over HTTP (DASH) protocol based on hyper text transfer protocol (HTTP) is a control plane protocol for streaming media.

When playing a video stream using the DASH protocol, the terminal device can obtain a DASH message containing an MPD file corresponding to the video content from the server. Among them, MPD files can be described in extensible markup language (XML) format.

Figure 3 shows the format of an MPD file. As shown in Figure 3, the MPD file includes from the inside out: Segment, Representation, Adaptation Set and Period.

Segment: Represents a small piece of audio or video data. The format of Segment can be fmp4 format.

Representation: Represents an audio stream or video stream, which can include one or more Segments.

Adaptation Set: Can include one or more Representations. When an Adaptation Set contains multiple Representations, the multiple Representations can be code streams generated from the same source stream, and the code rates, resolutions and/or frame rates of the multiple Representations are different.

Period: can represent a time period, including one or more Adaptation Sets. A program can consist of one or more Periods.

4). In this application, the video stream can also be called a media stream.

5). In this application, UC is information with a length of 1 byte, and the value range can be 0~255; US is information with a length of 2 bytes, and the value range can be 0~65025; UL is 4 words The section length information can range from 0 to 4228250625.

In the embodiments of this application, the number of nouns means "singular noun or plural noun", that is, "one or more", unless otherwise specified. "At least one" means one or more, and "plurality" means two or more. "And/or" describes the relationship between associated objects, indicating that there can be three relationships. For example, A and/or B can mean: A alone exists, A and B exist simultaneously, and B alone exists. “At least one of the following” or similar expressions refers to any combination of these items (items), including any combination of a single item (items) or a plurality of items (items).

In addition, it should be understood that in the description of this application, words such as "first" and "second" are only used for the purpose of distinguishing the description, and should not be understood as indicating or implying relative importance, nor should they be understood as To indicate or imply an order.

Figure 4 shows the architecture of a possible communication system to which the streaming media interaction method provided by the embodiment of the present application is applicable. Referring to Figure 4, the communication system includes: a terminal device and a service server. The business server and the terminal device can communicate through networks such as content delivery network (content delivery network, CDN) or media delivery network (media delivery network, MDN).

Among them, the terminal device is a device that can provide streaming media services to users. The terminal device has a display screen and can display streaming media obtained from the service server (for example, live video or on-demand video).

Terminal equipment can also be called user equipment (UE), client, electronic equipment, etc. Currently, some examples of terminal devices are: TVs, TV set-top boxes, mobile phones, tablets, laptops, computers, PDAs, mobile internet devices (mobile internet devices, MIDs), wearable devices, etc.

The service server can provide streaming media services for one or more terminal devices. The service server can obtain the original media stream (for example, original video) from the TV station or host-side terminal device, and transcode the original media stream to generate target media streams of multiple bit rates and formats to provide the target media stream. Play to audience-side terminal devices of different formats. In this application, the business server can be an ordinary server or a server in a cloud platform.

Currently, when using the above communication system to provide media streaming services, there are the following technical problems:

1. How to enable users who watch streaming media programs to participate in interactions with the program.

Currently, when users watch live or on-demand videos provided by TV stations, the direction of information transmission is one-way. Users can only watch the video but cannot interact with it.

Therefore, how to enable all users who watch streaming media programs to participate in the interaction with the program is a problem that needs to be solved in this application.

2. How to overcome the unfairness of interaction caused by the different delivery times of streaming media programs to different users.

Due to different transmission technologies and distances, the time it takes for streaming media programs to be delivered to different users varies greatly. For example, multiple TV stations or companies broadcast a party live. The 8 o'clock program content of the party will appear on the TV screens of the viewers of TV station A at 8:01 seconds, on the TV screens of the viewers of TV station B at 8:02 seconds, and on the TV screens of company C at 8:10 seconds. on the audience's screen. If users are required to interact with the party, how to ensure fairness when users from each TV station or company participate in the interaction is a problem that needs to be solved in this application.

The embodiment of the present application provides a streaming media interaction method, which can solve the above technical problems. This method can be applied to the communication system shown in Figure 4. Referring to the flow chart shown in Figure 5, the flow of this method will be described in detail below.

S501: The first server sends the first interaction instruction to the terminal device through streaming media. Correspondingly, the terminal device receives the first interaction instruction sent by the first server through streaming media. Wherein, the first interaction instruction may be used to instruct to start the interaction of participating in the program played by the terminal device.

The first server may be the business server in Figure 4. The first server may send a first interaction instruction to the terminal device when it is determined to start interacting with the program played by the terminal device. For example, the program being played by the terminal device is a program broadcast live by TV station A. After receiving the instruction input by the TV station staff to start interacting with the program, the first server determines to start interacting with the program played by the terminal device and sends the message to the terminal device. Send the first interaction instruction. For another example, the first server detects that the program content contains interactive information. If it detects that the program host prompts the audience to participate in the interaction, the first server can determine to start interacting with the terminal device playing the program and send the first interaction instruction to the terminal device. .

Optionally, the terminal device may receive the first interaction instruction sent by the first server through streaming media in one of the following ways:

Method 1: The terminal device receives a first video stream from the first server including a program played by the terminal device, and the media plane data of the first video stream includes the first interaction indication. That is to say, the terminal device can receive the first interaction instruction from the first server through the media plane.

Wherein, the first server can send the first interaction instruction to the terminal device through any media plane protocol. The following description takes the media plane protocol as RTP as an example.

In some possible ways, the RTP header of the first video stream contains the first interaction indication. At this time, the media plane data of the first video stream is the RTP header of the first video stream. Optionally, the RTP extension header of the first video stream includes the first interaction indication. For example, as shown in Figure 2, the Define by Profile field in the RTP extension header can identify the interactive instruction (InteractiveInstruction) parameter through a unique number, for example, InteractiveInstruction=0xFFFA. When the value of the Define by Profile field is 0xFFFA, the Header Extension field in the RTP extension header can include the InteractiveStatus field. When the value of the InteractiveStatus field is the first value (for example, 3), the InteractiveStatus field indicates starting the interaction of participating in the terminal device playing the program. Among them, the length of the InteractiveStatus field can be 1 byte.

In other possible ways, the NAL of the first video stream includes the first interaction indication. In this case, the media plane data of the first video stream is the NAL of the first video stream. Optionally, the NAL packet SEI of the first video stream contains the first interaction indication. As an example, the SEI message structure is shown in Table 1 below. Therefore, an SEI payload type (SEI payloadType) can be added to the SEI, and the SEI payloadType can identify the InteractiveInstruction parameter through a unique number, for example, InteractiveInstruction=0xFA. When SEI When the value of the payloadType field is 0xFA, the SEI payload (SEIPayLoad) can contain the InteractiveStatus field. For the specific content of the InteractiveStatus field, please refer to the description of the InteractiveStatus field in "The RTP header of the first video stream contains the first interaction indication", which will not be described again here. The SEI payloadSize field is the actual length of the SEI payload (SEIPayLoad).

Table 1

Optionally, in Table 1, n is an integer greater than 1.

Through the first method, the terminal device can obtain the first interaction instruction while obtaining the program content, so that it can deliver the corresponding first interaction instruction while delivering the streaming media of the program. Moreover, the terminal device can obtain the first interactive instruction through the media surface, which is easy to implement.

Method 2: The terminal device receives the first control plane data from the first server, and the first control plane data includes the first interaction instruction. That is to say, the terminal device can receive the first interaction instruction from the first server through the control plane.

The first control plane data may be control plane data corresponding to the program played by the terminal device.

Wherein, the first server can send the first interaction instruction to the terminal device through any control plane protocol. The following description takes the control plane protocol as DASH as an example.

Optionally, the first control plane data is a DASH message. For example, the DASH message includes an MPD file corresponding to the program played by the terminal device, and the MPD file includes the Interactive instruction parameter. The Interactive instruction parameter may include the first interactive instruction. For example, Interactive instruction parameters may include:

Among them, InteractiveStartSegment represents the segment in the program played by the terminal device that begins to participate in interaction, and InteractiveCloseSegment ends the segment in the program played by the terminal device that ends interaction. In the above example, fragment 15 is the fragment that begins to participate in the interaction of the terminal device playing the program; the fragments in which the user can participate in the interaction of the terminal device playing the program are fragments 15-35.

Additionally, the InteractiveCloseSegment parameter can be optional. For example, if the first server cannot determine when to end the interaction when generating the Interactive instruction parameter, the Interactive CloseSegment parameter may not be included in the Interactive instruction parameter, or the InteractiveCloseSegment parameter in the Interactive instruction parameter may be an invalid value. When it can determine when to end the interaction, the first server fills in the correct value in the InteractiveCloseSegment parameter.

Optionally, the Interactive instruction parameter can be included in the Period, Adaptation Set or Representation in the MPD file.

Optionally, when the program played by the terminal device is a live video, the terminal device can periodically obtain the MPD file of the latest content of the program from the first server. For example, the terminal device can obtain the MPD file of the latest content of the program from the first server every 5 seconds.

Through the second method, the terminal device can obtain the first interactive instruction through the control surface, which is easy to implement.

S502: The terminal device outputs a first interactive prompt. The first interactive prompt can be used to prompt the user to participate in the interaction of the program played by the terminal device.

In some possible ways, the terminal device can display the first interactive prompt through the screen. For example, the terminal device can display text on the screen: "Please participate in the interaction of the current program."

In other possible ways, the terminal device can output the first interaction prompt to the user through voice. For example, after receiving the first interaction instruction, the terminal device may output a voice: "Please participate in the interaction of the current program."

S503: The terminal device collects at least one interaction information generated when the user interacts according to the first interaction prompt, and the at least one interaction information can be used to generate interaction data.

Wherein, the terminal device can collect at least one interaction information generated when the user interacts according to the first interaction prompt through the data collection device. For example, the terminal device can collect the video information generated when the user interacts according to the first interaction prompt through the camera, and based on the video information and the algorithm used to calculate the human body motion skeleton (for example, artificial intelligence (artificial intelligence, AI) algorithm), The user's human body posture description file is determined, and the human body posture description file may include human motion skeleton information. For another example, the terminal device may use a camera to collect images or videos generated when the user interacts according to the first interaction prompt. For another example, the terminal device may collect audio generated when the user interacts according to the first interaction prompt through a microphone. For another example, the terminal device can also collect touch operation information generated when the user interacts according to the first interaction prompt through the touch screen.

S504: The terminal device sends interaction data to the second server.

The second server may be the same server as the first server, or may be a different server. When the second server and the first server are different servers, the address of the second server may be a default address or an address obtained from the first server. For the process of the terminal device obtaining the address from the first server, please refer to step B1 below, which will not be described here.

In addition, the interaction data may include at least one interaction information collected in S503. For the specific content of at least one piece of information, please refer to the description of "at least one piece of information" in S503 below, which will not be elaborated here.

In the method shown in Figure 5, after the terminal device receives the first interaction instruction from the first server for instructing to start participating in the interaction of the program played by the terminal device, it can output the first interaction prompt to prompt the user to participate. The terminal device plays the interaction of the program, and the terminal device collects at least one interaction information generated when the user interacts according to the first interaction prompt. Through this method, users who watch the program currently played by the terminal device can participate in the interaction of the program currently played by the terminal device, and there is no need for the terminal device to install an application program with interactive functions or the user needs to use other terminals with applications for interaction Only equipment can achieve interaction and it is easy to implement.

Optionally, the method shown in Figure 5 also includes step A1:

A1: The first server sends interaction mode instructions to the terminal device through streaming media. Correspondingly, the terminal device receives the interaction mode indication sent by the first server through streaming media. The interactive mode indication is used to instruct the terminal device to use an interactive mode for playing the program.

Among them, the interaction method may include, but is not limited to, at least one of the following: interaction through somatosensory, interaction through voice recognition, interaction through video, interaction through voice, interaction through remote control, etc.

Optionally, the terminal device can receive the interaction mode indication sent by the first server through streaming media through one of the following implementation methods:

Implementation method 1: The terminal device receives a third video stream containing a program from the first server, and the media plane data of the third video stream contains an interaction mode indication. That is to say, the terminal device can receive the interaction mode indication from the first server through the media plane.

The third video stream may be the same as the first video stream, or may be different from the first video stream. When the third video stream is different from the first video stream, for example, the third video stream and the first video stream may be different video streams of the same program.

In this application, the first server can send interaction mode instructions to the terminal device through any media plane protocol. The following description takes the media plane protocol as RTP as an example.

In some possible ways, the RTP header of the third video stream contains an interactive mode indication. In this case, the media plane data of the third video stream is the RTP header of the third video stream. Optionally, the RTP extension header of the third video stream contains an interactive mode indication. For example, as shown in Figure 2, the Define by Profile field in the RTP extension header of the third video stream can identify the InteractiveInstruction parameter through a unique number, for example, InteractiveInstruction=0xFFFA. When the value of the Define by Profile field is 0xFFFA, the Header Extension field of the third video stream may include an interactive mode (InteractiveType) field, and the InteractiveType field may indicate the interactive mode of the terminal device for playing the program. For example, when the value of the InteractiveType field is the second value (for example, 1), the interactive mode of the terminal device playing the program is to interact through somatosensory; when the value of the InteractiveType field is the third value (for example, 2), the terminal device The interactive mode for the device playing the program is to interact through voice recognition; when the value of the InteractiveType field is the fourth value (for example, 3), the interactive mode for the terminal device to play the program is to interact through video; when the value of the InteractiveType field is When the value of the InteractiveType field is the sixth value (for example, 5), the interactive mode for the terminal device to play programs is through remote control. interact with the device. Among them, the length of the InteractiveType field can be 1 byte.

In other possible ways, the NAL of the third video stream includes an interactive mode indication. In this case, the media plane data of the third video stream is the NAL of the third video stream. Optionally, the NAL packet SEI of the third video stream contains an interactive mode indication. For example, the SEI message structure is shown in Table 1 above. Add an SEI payloadType to the SEI. The SEI payloadType can identify the InteractiveInstruction parameter through a unique number, for example, InteractiveInstruction=0xFA. When the value of the SEI payloadType field is 0xFA, the SEIPayLoad may contain the InteractiveType field. For the specific content of the InteractiveType field, please refer to "RTP of the third video stream The description of the InteractiveType field in "Header Contains Interaction Mode Instructions" will not be repeated here.

Through the first implementation method, the terminal device can obtain the interactive mode instructions while obtaining the program streaming media. The acquisition method is fast and efficient. Moreover, the terminal device can obtain interactive mode instructions through the media surface, which is easy to implement.

Implementation method two: the terminal device receives third control plane data from the first server, and the third control plane data includes interaction mode instructions. That is to say, the terminal device can receive the interaction mode instruction from the first server through the control plane.

The third control plane data may be the same as the first control plane data, or may be different from the first control plane data. When the third control plane data is different from the first control plane data, for example, the third control plane data and the first control plane data may be control plane data of the same program.

Wherein, the first server can send the interaction mode instruction to the terminal device through any control plane protocol. The following description takes the control plane protocol as DASH as an example.

Optionally, the third control plane data is a DASH message. For example, the DASH message includes an MPD file corresponding to the program played by the terminal device, and the MPD file includes the Interactive instruction parameter. For the content of the Interactive instruction parameter, please refer to the description of the Interactive instruction parameter in Method 2, and the repeated parts will not be repeated. The Interactive instruction parameter can contain interactive mode instructions. For example, Interactive instruction parameters may include:

Among them, InteractiveType is used to represent the interactive mode of the terminal device playing the program. In the above example, the interaction mode is through somatosensory mode.

Through the second implementation method, the terminal device can obtain the interactive mode instructions through the control surface, which is easy to implement.

In addition, this application does not limit the execution order of step A1 and step S501, that is, step A1 and step S501 can be executed before step S502.

Correspondingly, in S502, the first interaction prompt may be specifically used to prompt the user to participate in the interaction of the program played by the terminal device according to the interaction method.

Specifically, in some possible ways, the terminal device can display the first interactive prompt through the screen. For example, if the interaction method is interaction through body sensing, after the terminal device receives the first interaction instruction and the interaction mode instruction, the terminal device can display text on the screen: "Please participate in the interaction of the current program through body sensing." In other possible ways, the terminal device can output the first interaction prompt to the user through voice. For example, if the interaction method is interaction through voice, after receiving the first interaction instruction and the interaction method instruction, the terminal device can output a voice: "Please participate in the interaction of the current program through voice."

Correspondingly, in S503, the terminal device can collect at least one interactive information generated when the user interacts according to the interactive mode. Specifically, the terminal device can collect at least one interactive information generated when the user interacts according to the interactive mode. Optionally, the terminal device can collect at least one interactive information generated when the user interacts according to the interactive mode through a data collection device corresponding to the interactive mode.

For example, when the interaction method is through somatosensory interaction, the terminal device can collect video information generated when the user interacts according to the interaction method through the camera. Then, the terminal device determines the user's human body motion skeleton information based on the video information and an algorithm (for example, AI algorithm) used to calculate the human body motion skeleton. At this time, the at least one interaction information may include the user's human body movement skeleton information.

For another example, when the interaction method is interaction through voice recognition, the terminal device can collect the audio information generated when the user interacts according to the interaction method through the microphone. Then, the terminal device detects the user's speaking content based on the audio information and an algorithm for recognizing speech (eg, AI algorithm). At this time, the at least one interactive information includes the user's speech content.

For another example, when the interaction method is through video, the terminal device can use a camera to collect video information generated when the user interacts according to the interaction method. At this time, the at least one interactive information includes the video information.

For another example, when the interaction method is interaction through voice recognition, the terminal device can collect the audio information generated when the user interacts according to the interaction method through the microphone. At this time, the at least one interactive information includes the audio information.

For another example, when the interaction method is interaction through a remote control, the terminal device can receive information from the remote control through an infrared receiving device. The information from the remote control is, for example, the result of keystrokes on the remote control. At this time, the at least one interaction information includes the information of the remote control.

Through this method, the terminal device can collect at least one interaction information generated when the user interacts according to the interaction method according to the interaction method indicated by the first server, so that the user's interaction information can be obtained in a targeted manner, and the acquisition and transmission of the interaction information can be improved. Efficiency of interactive data.

Optionally, in S503, the terminal device may also collect at least one interaction information generated when the user interacts according to the default interaction method. At this time, the terminal device does not need to obtain the interaction mode indication from the first server. The default interaction method may include one or more interaction methods in step A1. For example, when the default interaction method is to interact through video, the terminal device can use the camera to collect video information generated when the user interacts according to the interaction method. At this time, the at least one interactive information includes the video information. This can reduce the process of the first server issuing interactive mode instructions, thereby making the solution more efficient and saving transmission resources of the streaming media channel.

Optionally, the method shown in Figure 5 may also include step B1:

B1: The first server sends an address indication to the terminal device through streaming media. Correspondingly, the terminal device receives the address indication sent by the first server through streaming media. Among them, the address indication can be used to indicate the reporting address of the interactive data. The address indication may directly or indirectly indicate the reporting address. The reported address may be the address of the second server. For example, the reported address may be the Internet protocol (IP) address or the uniform resource locator (URL) address of the second server.

Optionally, the terminal device can receive the address indication sent by the first server through streaming media through one of the following implementation methods:

Implementation Mode 1: The terminal device receives a fourth video stream containing a program from the first server, and the media plane data of the fourth video stream contains an address indication. That is to say, the terminal device can receive the address indication from the first server through the media plane.

The fourth video stream may be the same as the first video stream, or may be different from the first video stream. When the fourth video stream is different from the first video stream, for example, the fourth video stream and the first video stream may be different video streams of the same program.

In this application, the first server can send an address indication to the terminal device through any media plane protocol. The following description takes the media plane protocol as RTP as an example.

In some possible ways, the RTP header of the fourth video stream includes an address indication. At this time, the media plane data of the fourth video stream is the RTP header of the fourth video stream. Optionally, the RTP extension header of the fourth video stream contains an address indication. For example, as shown in Figure 2, the Define by Profile field in the RTP extension header can identify the InteractiveInstruction parameter through a unique number, for example, InteractiveInstruction=0xFFFA. When the value of the Define by Profile field is 0xFFFA, the Header Extension field may include a reported address (for example, interactive URL (InteractiveURL)) field, and the InteractiveURL field is an address indication. For example, if the InteractiveURL field contains http://CMCCInteractivevido.com/Interactive, the reported address is http://CMCCInteractivevido.com/Interactive. Among them, the length of the InteractiveURL field can be 128 bytes.

In other possible ways, the NAL of the fourth video stream includes an address indication. In this case, the media plane data of the fourth video stream is the NAL of the fourth video stream. Optionally, the NAL packet SEI of the fourth video stream contains an address indication. For example, the SEI message structure is shown in Table 1 above. Add a SEI payloadType to the SEI. The SEI payloadType can identify the InteractiveInstruction parameter through a unique number, for example, InteractiveInstruction=0xFA. When the value of the SEI payloadType field is 0xFA, SEIPayLoad can include the reporting address (for example, InteractiveURL) field. For the specific content of the InteractiveURL field, please refer to the description of the InteractiveURL field in "The RTP header of the fourth video stream contains address indication" and will not be described again here.

Through implementation method 1, the terminal device can obtain the address indication while obtaining the program streaming media, making it more efficient to obtain the address information. Moreover, the terminal device can obtain the address indication through the media plane, which is easy to implement.

Implementation manner 2: The terminal device receives fourth control plane data from the first server, and the fourth control plane data includes the address indication. That is to say, the terminal device can receive the address indication from the first server through the control plane.

The fourth control plane data may be the same as the first control plane data, or may be different from the first control plane data. When the fourth control plane data is different from the first control plane data, for example, the fourth control plane data and the first control plane data may be control plane data of the same program.

Wherein, the first server can send the address indication to the terminal device through any control plane protocol. The following description takes the control plane protocol as DASH as an example.

Optionally, the fourth control plane data is a DASH message. For example, the DASH message includes an MPD file corresponding to the program played by the terminal device, and the MPD file includes the Interactive instruction parameter. For the content of the Interactive instruction parameter, please refer to the description of the Interactive instruction parameter in Method 2, and the repetitive parts will not be repeated. The Interactive instruction parameter can contain address instructions. For example, the Interactive instruction parameter can include:

Among them, InteractiveURL is used to represent the reporting address of interactive data. In the above example, the reporting address is: http://CMCCInteractivevido.com/Interactive.

Through implementation method 2, the terminal device can obtain the address indication through the control plane, which is easy to implement.

In addition, this application does not limit the execution order of step B1 and step S501 to step S503, that is, step B1 can be executed before step S504. Correspondingly, S504 may include: the terminal device sends the interaction data to the second server according to the reported address. Through this method, the terminal device can upload the interaction data to the correct server according to the reporting address indicated by the first server.

Optionally, in order to solve the above problem 2, the media plane data of the first video stream of the program played by the terminal device includes the frame sequence number of the streaming media frame in the program. That is to say, the terminal device can obtain the frame sequence number of the streaming media frame in the program through the media plane.

The frame serial numbers of the streaming media frames in the program may be accumulated and may be used to uniquely indicate the streaming media frames in the program.

In this application, the first server can send the frame sequence number of the streaming media frame in the program to the terminal device through any media plane protocol. The following description takes the media plane protocol as RTP as an example.

In some possible ways, the RTP header of the first video stream contains the frame sequence number of the streaming media frame in the program. In this case, the media plane data of the first video stream is the RTP header of the first video stream. Optionally, the RTP extension header of the first video stream contains the frame sequence number of the streaming media frame in the program. For example, as shown in Figure 2, the Define by Profile field in the RTP extension header can identify the frame sequence number (FramSequence) parameter through a unique number, for example, FramSequence=0xFFFC. When the value of the Define by Profile field is 0xFFFC, the Header Extension field in the RTP extension header can be as shown in Table 2.

Table 2

In other possible ways, the NAL of the first video stream includes the frame sequence number of the streaming media frame in the program. In this case, the media plane data of the first video stream is the NAL of the first video stream. Optionally, the NAL packet SEI of the first video stream contains the frame sequence number of the streaming media frame in the program. For example, the SEI message structure is shown in Table 1 above. Therefore, a SEI payloadType can be added to the SEI, and the SEI payloadType can identify the FramSequence parameter through a unique number, for example, FramSequence=0xFC. When the value of the SEI payloadType field is 0xFC, SEIPayLoad can contain the FramSequenceNumber field. For the specific contents of the FramSequenceNumber field, please refer to Table 2 above. The length of the SEI payloadSize field can be 4 bytes.

In S504, the interaction data may include:

1. At least one interactive information generated when the user interacts according to the first interactive prompt collected by the terminal device: For specific content, please refer to the description of the at least one interactive information in S503, which will not be described again here.

2. When collecting each interaction information in at least one interaction information, the frame sequence number of the streaming media frame of the program being displayed by the terminal device.

For example, when the interaction mode is interaction through somatosensory, the interaction data may include: when the terminal device collects video information generated when the user interacts according to the interaction mode through the camera, the frame sequence number of the streaming media frame of the program being displayed by the terminal device.

For another example, when the interaction mode is interaction through voice recognition, the interaction data may include: when the terminal device collects the audio information generated when the user interacts according to the interaction mode through the microphone, the frame sequence number of the streaming media frame of the program being displayed by the terminal device .

Optionally, interaction data may also include interaction methods. For the specific content of the interaction method, please refer to step A1 and will not be described again here.

Optionally, in order to obtain the frame sequence number of the streaming media frame of the program being displayed by the terminal device when collecting at least one interaction information, in S503, the terminal device may record when collecting at least one interaction information. When collecting each interaction information in at least one interaction information, the frame sequence number of the streaming media frame of the program being displayed by the terminal device.

Through this method, the terminal device not only sends at least one interaction information to the second server, but also sends the frame sequence number of the streaming media frame of the program being displayed by the terminal device when collecting each interaction information in the at least one interaction information, so that it can be based on the The frame sequence number determines the user's response time, which can reduce the unfairness of interaction caused by the different delivery times of streaming media programs to different users.

For example, multiple TV stations or companies broadcast a party live. The 8 o'clock program content of the party will appear on the TV screen of audience A on TV station A at 8:01 seconds, on the TV screen of audience B on TV station B at 8:02 seconds, and on the company's TV screen at 8:10 seconds. Audience C is on the screen of C. The interactive content of the program is who can complete the actions specified by the target character in the program the fastest. In the program content at 8 o'clock, the specified action of the target character is handstand, and the frame number of the streaming media frame corresponding to this program content is 12358. If the terminal equipment of audience A When A collects that audience A is standing on his head, the frame number of the streaming media frame of the program displayed by terminal equipment A is 12380. When terminal equipment B of audience B collects that audience B is standing on his head, the frame number of the streaming media frame of the program displayed by terminal equipment B is The frame number is 12372. When the terminal device C of the audience C collects the inversion of the audience C, the frame number of the streaming media frame of the program displayed by the terminal device C is 12370. Then through this method, it can be determined that the reaction time of the audience C is the shortest.

Optionally, in S504, the interaction data may also include: when each interaction information in the at least one interaction information is collected, the displayed duration of the streaming media frame of the program being displayed by the terminal device. For example, when the interaction mode is interaction through speech recognition, and the terminal device collects the audio information generated when the user interacts according to the interaction mode through the microphone, the frame sequence number of the streaming media frame of the program being displayed by the terminal device is 12560, and the terminal device The frame has been displayed for 19 milliseconds, so the interaction data also includes: when collecting audio information, the displayed duration of the streaming media frame of the program being displayed by the terminal device is 19 milliseconds.

Optionally, in order to collect the displayed duration of the streaming media frame of the program being displayed by the terminal device when collecting each interactive information in at least one interactive information, in S503, when collecting at least one interactive information, the terminal device may record the collected When each interactive information in at least one interactive information is displayed, the display duration of the streaming media frame of the program being displayed by the terminal device.

Through this method, the terminal device also collects the displayed duration of the streaming media frame of the program being displayed by the terminal device from the second server when each interaction information in the at least one interaction information is collected, so that the user's reaction time can be more accurately determined. This further reduces the unfairness of interaction caused by the different delivery times of streaming media programs to different users.

Optionally, in S504, the terminal device can send interaction data to the second server in one of the following ways:

Method 1: The terminal device sends the second video stream to the second server, and the media plane data of the second video stream includes interactive data. That is to say, the terminal device can send the second video stream to the second server through the media.

The second video stream may be a video of user interaction collected by the terminal device, or may be other videos.

In this application, the terminal device can send interactive data to the second server through any media plane protocol. The following description takes the media plane protocol as RTP as an example.

In some possible ways, the RTP header of the second video stream contains interactive data. In this case, the media plane data of the second video stream is the RTP header of the second video stream. Optionally, the RTP extension header of the second video stream contains interactive data. For example, as shown in Figure 2, the Define by Profile field in the RTP extension header can identify the interactive result (InteractiveResult) parameter through a unique number, for example, InteractiveResult=0xFFFB. When the value of the Define by Profile field is 0xFFFB, the Header Extension field in the RTP extension header can be as shown in Table 3.

table 3

Optionally, in actual applications, the Header Extension field in the RTP extension header may include some of the fields in Table 3. For example, the Header Extension field includes the InteractiveType field and the InteractiveDisplayFrame field. For another example, the Header Extension field includes the InteractiveDisplayFrame field and at least one interactive information.

In other possible ways, the NAL of the second video stream contains interactive data. In this case, the media plane data of the second video stream is the NAL of the second video stream. Optionally, the NAL packet SEI of the second video stream contains interactive data. For example, the SEI message structure is shown in Table 1 above. Therefore, a SEI payloadType can be added to the SEI, and the SEI payloadType can identify the InteractiveResult parameter through a unique number, for example, InteractiveResult=0xFB. When the value of the SEI payloadType field is 0xFB, SEIPayLoad can contain the InteractiveResult field. The specific content of the InteractiveResult field can be found in Table 3 and will not be described again here. The SEI payloadSize field is the actual length of SEIPayLoad.

Through method 1, the terminal device can send interactive data to the second server through the media stream, achieving higher efficiency.

Method 2: The terminal device sends the second control plane data of the streaming media to the second server, and the second control plane data includes interaction data. That is to say, the terminal device can send the second control plane data of the streaming media to the second server through control.

The terminal device may send the second control plane data of the streaming media to the second server through any control plane protocol. The following description takes the control plane protocol as HTTP as an example.

Optionally, the second control plane data is an HTTP message. The following is an example of using the HTTP submission (POST) method to send interaction data whose interaction mode is Human Pose. HTTP messages can include:

Among them, InteractiveType indicates that the interaction method is through somatosensory interaction; Human Pose indicates that the collected interactive information includes: user skeleton key point coordinates, which can be used to determine the user's body movements; InteractiveDisplayFrame indicates that the terminal device when collecting the interactive information The 18353rd frame of the program is being played; the InteractiveDisplayFrameDuration flag indicates that the terminal device has displayed the current frame for 19 milliseconds when the interactive information is collected.

Through method 2, the terminal device can send interactive data to the second server through control, which is easy to implement.

Optionally, the terminal device can report interaction data every time it collects interaction information, where the collection interval is, for example, 33 milliseconds; the terminal device can also report interaction data once in a certain period (for example, 5 seconds); the terminal device can also report After collecting all interaction information, report the interaction data.

Optionally, before S501, the method shown in Figure 5 also includes steps C1-C2:

C1: The first server sends the second interaction instruction to the terminal device through streaming media. Correspondingly, the terminal device receives the second interaction instruction sent by the first server through streaming media. Wherein, the second interaction instruction is used to indicate preparation to participate in the interaction of the terminal device playing the program.

Optionally, the terminal device may receive the second interaction instruction sent by the first server through streaming media in one of the following ways:

Method A: The terminal device receives the fifth video stream from the first server that contains the program played by the terminal device, and the media plane data of the fifth video stream contains the second interaction indication. That is to say, the terminal device can receive the second interaction instruction from the first server through the media plane.

The fifth video stream and the first video stream may be different video streams of the same program. For example, the frame sequence number of the streaming media frame in the fifth video stream is different from the frame sequence number of the streaming media frame in the first video stream.

In this application, the first server can send the second interaction instruction to the terminal device through any media plane protocol. The following description takes the media plane protocol as RTP as an example.

In some possible ways, the RTP header of the fifth video stream contains the second interaction indication. At this time, the media plane data of the fifth video stream is the RTP header of the fifth video stream. Optionally, the RTP extension header of the fifth video stream includes the second interaction indication. For example, as shown in Figure 2, the Define by Profile field in the RTP extension header can identify the InteractiveInstruction parameter through a unique number, for example, InteractiveInstruction=0xFFFA. When the value of the Define by Profile field is 0xFFFA, the Header Extension field in the RTP extension header can contain the InteractiveStatus field. When the value of the InteractiveStatus field is the seventh value (for example, 2), the InteractiveStatus field indicates preparation to participate in the interaction of the terminal device playing the program. Among them, the length of the InteractiveStatus field can be 1 byte.

In other possible ways, the NAL of the fifth video stream includes the second interaction indication. At this time, the media plane data of the fifth video stream is the NAL of the fifth video stream. Optionally, the NAL packet SEI of the fifth video stream contains the second interaction indication. For example, the SEI message structure is shown in Table 1 above. Therefore, a SEI payloadType can be added to the SEI, and the SEI payloadType can identify the InteractiveInstruction parameter through a unique number, for example, InteractiveInstruction=0xFA. When the value of the SEI payloadType field is 0xFA, SEIPayLoad can contain the InteractiveStatus field. For the specific content of the InteractiveStatus field, please refer to the description of the InteractiveStatus field in "The RTP header of the fifth video stream contains the second interaction indication" and will not be described again here. The SEI payloadSize field is the actual length of the SEI payload (SEIPayLoad).

Through method A, the terminal device can obtain the second interaction instruction while obtaining the program content, so that it can deliver the corresponding second interaction instruction while delivering the streaming media of the program. Moreover, the terminal device obtains the second interaction instruction through the media surface, which is easy to implement.

In method B, the terminal device receives the fifth control plane data from the first server, and the fifth control plane data includes the second interaction instruction. That is to say, the terminal device may receive the second interaction instruction from the first server through the control plane.

The fifth control plane data may be control plane data corresponding to the program played by the terminal device.

Wherein, the first server can send the second interaction instruction to the terminal device through any control plane protocol. The following description takes the control plane protocol as DASH as an example.

Optionally, the fifth control plane data is a DASH message. For example, the DASH message includes an MPD file corresponding to the program played by the terminal device, and the MPD file includes the Interactive instruction parameter. The Interactive instruction parameter may include a second interactive instruction. For example, Interactive instruction parameters may include:

Among them, InteractiveReadytSegment is used to indicate the segment that is ready to participate in the interaction of the terminal device playing the program. In the above example, starting from slice 10, preparations are made for participating in the interaction of terminal devices playing programs.

Through method B, the terminal device can obtain the second interaction instruction through the control surface, which is easy to implement.

Optionally, the second interaction instruction may also indicate a first duration, where the first duration is the time interval between the current moment and the start moment of the interaction in which the terminal device plays the program. For example, the first duration is 5 seconds.

C2: The terminal device outputs a second interactive prompt. The second interactive prompt is used to prompt the user to prepare to participate in the interaction of the program played by the terminal device.

In some possible ways, the terminal device can display the second interactive prompt through the screen. For example, the terminal device can display text on the screen: "Please prepare to participate in the interaction of the current program."

In other possible ways, the terminal device can output the second interactive prompt to the user through voice. For example, after receiving the second interaction instruction, the terminal device may output a voice: "Please prepare to participate in the interaction of the current program."

Optionally, the second interaction prompt may also prompt the first duration. For example, the terminal device can display text on the screen: "Please prepare to participate in the interaction of the current program in 5 seconds." For another example, after receiving the second interaction instruction, the terminal device can output a voice: "Please prepare to participate in the interaction of the current program in 5 seconds."

Through this method, the terminal device can prompt the user to prepare to start participating in the interaction of the current program, giving the user a certain amount of preparation time.

Optionally, the method shown in Figure 5 also includes steps D1-D2:

D1: The first server sends the third interaction instruction to the terminal device through streaming media. Correspondingly, the terminal device receives the third interaction instruction sent by the first server through streaming media. Wherein, the third interaction instruction is used to instruct to stop the interaction of participating in the terminal device playing the program.

Optionally, the terminal device may receive the third interaction instruction sent by the first server through streaming media through one of the following implementation methods:

Implementation A: The terminal device receives a sixth video stream from the first server that includes a program played by the terminal device, and the media plane data of the sixth video stream includes a third interaction indication. That is to say, the terminal device can receive the third interaction instruction from the first server through the media plane.

The sixth video stream and the first video stream may be different video streams of the same program. For example, the frame sequence number of the streaming media frame in the sixth video stream is different from the frame sequence number of the streaming media frame in the first video stream.

In this application, the first server can send the third interaction instruction to the terminal device through any media plane protocol. The following description takes the media plane protocol as RTP as an example.

In some possible ways, the RTP header of the sixth video stream contains the third interaction indication. At this time, the media plane data of the sixth video stream is the RTP header of the sixth video stream. Optionally, the RTP extension header of the sixth video stream includes a third interaction indication. For example, as shown in Figure 2, the Define by Profile field in the RTP extension header can identify the InteractiveInstruction parameter through a unique number, for example, InteractiveInstruction=0xFFFA. When the value of the Define by Profile field is 0xFFFA, the Header Extension field in the RTP extension header can contain the InteractiveStatus field. When the value of the InteractiveStatus field is the eighth value (for example, 1), the InteractiveStatus field indicates to stop participating in the interaction of the terminal device playing the program. Among them, the length of the InteractiveStatus field can be 1 byte.

In other possible ways, the NAL of the sixth video stream includes the third interaction indication. In this case, the media plane data of the sixth video stream is the NAL of the sixth video stream. Optionally, the NAL packet SEI of the sixth video stream includes a third interaction indication. For example, the SEI message structure is shown in Table 1 above. Therefore, a SEI payloadType can be added to the SEI, and the SEI payloadType can identify the InteractiveInstruction parameter through a unique number, for example, InteractiveInstruction=0xFA. When the value of the SEI payloadType field is 0xFA, SEIPayLoad can contain the InteractiveStatus field. For the specific content of the InteractiveStatus field, please refer to the description of the InteractiveStatus field in "The RTP header of the sixth video stream contains the third interaction indication" and will not be described again here. The SEI payloadSize field is the actual length of the SEI payload (SEIPayLoad).

Through implementation method A, the terminal device can obtain the third interaction instruction while obtaining the program content, so that the corresponding third interaction instruction can be issued while delivering the streaming media of the program. Moreover, the terminal device can obtain the third interactive instruction through the media surface, which is easy to implement.

In implementation mode B, the terminal device receives sixth control plane data from the first server, and the sixth control plane data includes the third interaction indication. That is to say, the terminal device may receive the third interaction instruction from the first server through the control plane.

The sixth control plane data may be control plane data corresponding to the program played by the terminal device.

Wherein, the first server can send the third interaction instruction to the terminal device through any control plane protocol. The following description takes the control plane protocol as DASH as an example.

Optionally, the sixth control plane data is a DASH message. For example, the DASH message includes an MPD file corresponding to the program played by the terminal device, and the MPD file includes the Interactive instruction parameter. The Interactive instruction parameter may include a third interactive instruction. For example, Interactive instruction parameters may include:

Among them, InteractiveCloseSegment is used to instruct the segment to stop participating in the interaction of the terminal device playing the program. In the above example, the interaction of participating in the terminal device playing the program is stopped from slice 30.

Through implementation method B, the terminal device can obtain the third interactive instruction through the control plane, which is easy to implement.

Optionally, step D1 may be executed after step S501.

D2: The terminal device stops collecting at least one interaction information generated when the user interacts according to the first interaction prompt.

Optionally, the terminal device can also output a third interaction prompt, which is used to prompt the user to stop participating in the interaction of the program played by the terminal device.

In some possible ways, the terminal device can display the third interactive prompt through the screen. For example, the terminal device can display text on the screen: "Please stop participating in the interaction of the current program."

In other possible ways, the terminal device can output the third interactive prompt to the user through voice. For example, after receiving the third interaction instruction, the terminal device can output a voice: "Please stop participating in the interaction of the current program."

Through this method, the terminal device can stop collecting the user's interaction information in time under the instruction of the first server, thereby saving the energy required by the terminal device to collect the user's interaction information.

The embodiment of the present application provides another streaming media interaction method, which is a possible implementation of the method shown in Figure 5. under Referring to the flow chart shown in Figure 6, the flow of this method will be described in detail.

S601: The terminal device joins the multicast group of channel 1.

Optionally, the terminal device can display the program list after it is turned on. The program list may include indication information of at least one channel, and the at least one channel may be at least one television channel. After receiving the user's instruction to select channel 1, the terminal device can join the multicast group of channel 1 through multicast, so that it can receive the media data of channel 1. Wherein, the way in which the terminal device receives the user's instruction to select channel 1 may include one of the following: when the terminal device is a television, the terminal device receives the user's instruction to select channel 1 through the remote control; when the screen of the terminal device is a touch screen, the terminal device The user's touch command in the selected area of channel 1 can be detected.

S602: The first server sends video stream 1 of channel 1 to the terminal device.

Among them, video stream 1 may include: the content of video 1 and interaction instruction 1. Among them, the interaction instruction 1 may indicate not to interact with the program of channel 1. The transmission method of the interaction instruction 1 can refer to the transmission method of the third interaction instruction in implementation method A, and the repeated parts will not be described again. For example, when the value of the InteractiveStatus field is an eighth value (for example, 1), the InteractiveStatus field indicates not participating in the interaction of the terminal device playing the program.

Optionally, video stream 1 also includes: the frame sequence number of the streaming media frame in video 1. For the specific content of the frame serial number of the streaming media frame in video 1, please refer to the description of "the frame serial number of the streaming media frame in the program" in the method shown in Figure 5, and will not be described again here.

Additionally, the first server may send the video stream 1 to the terminal device through multicast.

S603: The terminal device displays video 1 on the screen.

S604: The first server determines that the program on channel 1 is about to enter the interactive session.

For example, after receiving an instruction from a television station staff member to prepare for interaction with a program on Channel 1, the first server determines that the program on Channel 1 is about to enter an interactive session. For another example, the first server detects that the program content contains interaction preparation information. If it detects that the program host prompts the audience to prepare for interaction, the first server can determine that the program on channel 1 is about to enter the interactive session.

S605: The first server sends video stream 2 of channel 1 to the terminal device.

Among them, video stream 2 may include: the content of video 2 and the second interaction instruction.

Video 1 and Video 2 may be continuous videos. For example, both Video 1 and Video 2 are videos of a party, Video 1 is the program content of the party from 8:00 to 8:01, and Video 2 is the program content of the party from 8:01 to 8:02.

The second interaction indication may indicate preparation to participate in the interaction of the program of Channel 1 . For the specific content of the second interaction instruction, please refer to step C1, which will not be described again here.

Optionally, video stream 2 also includes at least one of the following: interaction mode indication, address indication, and frame sequence number of the streaming media frame in video 2. The interactive mode indication is used to indicate the interactive mode of the program on Channel 1. For the specific content of the interactive mode indication, please refer to step A1, which will not be described again here. The address indication can be used to indicate the reporting address of interactive data. For the specific content of the address indication, please refer to step B1, which will not be described again here. For the specific content of the frame serial number of the streaming media frame in video 2, please refer to the description of "the frame serial number of the streaming media frame in the program" in the method shown in Figure 5, and will not be described again here.

The frame serial number of the streaming media frame in video 2 may be accumulated in video 2, or may be continuously accumulated based on the accumulated frame serial number in video 1.

Additionally, the first server may send the video stream 2 to the terminal device via multicast.

S606: The terminal device displays video stream 2 on the screen and prepares to collect at least one interaction information generated when the user interacts.

Among them, the terminal device can prepare to collect the user's interactive information by activating the software and hardware for collecting interactive information. For example, the terminal device can turn on the camera and prepare to collect the user's video information.

Optionally, the terminal device can output a second interaction prompt and/or interaction method. The second interaction prompt is used to prompt the user to prepare to participate in the interaction with the program of Channel 1. For the specific content of the second interaction prompt, please refer to step C2, which will not be described again here. The specific content of the interaction method can be referred to step A1, and will not be described again here.

Optionally, the terminal device can also output a third interaction prompt. The third interaction prompt is used to ask the user whether to participate in the interaction of the program of Channel 1. After the user confirms participation in the interaction, the terminal device is ready to collect the user's interaction information.

In some possible ways, the terminal device can display the third interactive prompt through the screen. For example, the terminal device can display text on the screen: "Whether to participate in the interaction of the current program".

In other possible ways, the terminal device can output the third interactive prompt to the user through voice. For example, after receiving the second interaction instruction, the terminal device may output a voice: "Do you want to participate in the interaction of the current program?"

S607: The first server determines that the program of channel 1 enters the interactive session.

For the specific content in S607, please refer to the description in S501 of the first server determining to start interacting with the terminal device to play the program. No further details will be given.

S608: The first server sends video stream 3 to the terminal device.

Among them, video stream 3 includes: the content of video 3 of channel 1 and the first interaction instruction.

Among them, video 2 and video 3 can be continuous videos. For example, video 2 and video 3 are both videos of a party. Video 2 is the program content of the party from 8:01 to 8:02, and video 3 is the program content of the party from 8:02 to 8:03.

The first interaction instruction may be used to instruct the user to start interacting with the program of Channel 1. For the specific content of the first interaction instruction, reference may be made to method 1 in S501, which will not be described again here.

Optionally, video stream 3 also includes at least one of the following: interaction mode indication, address indication, and frame sequence number of the streaming media frame in video 3. The interactive mode indication is used to indicate the interactive mode of the program on Channel 1. For the specific content of the interactive mode indication, please refer to step A1, which will not be described again here. The address indication can be used to indicate the reporting address of interactive data. For the specific content of the address indication, please refer to step B1, which will not be described again here. For the specific content of the frame serial number of the streaming media frame in video 3, please refer to the description of "the frame serial number of the streaming media frame in the program" in the method shown in Figure 5, and will not be described again here.

The frame serial number of the streaming media frame in video 3 may be accumulated in video 3, or may be continuously accumulated based on the accumulated frame serial number in video 2.

Additionally, the first server may send the video stream 3 to the terminal device via multicast.

S609: The terminal device outputs the first interaction prompt and collects at least one interaction information generated when the user interacts according to the first interaction prompt. The at least one interaction information can be used to generate interaction data.

For the specific content of S609, please refer to S502-S503 and will not be repeated here.

S610: The terminal device sends an HTTP message to the second server. Among them, HTTP messages contain interaction data.

For the specific content of S610, please refer to Method 2 in S504, which will not be described again here.

S611: The second server sends interactive data to the first server.

The second server may send all received user interaction data to the first server.

S612: The first server sends result information to the second server. Optionally, the first server may determine result information based on the received interaction data. The result information may include at least one of the following: a score for the user interaction data, a user's reaction time, etc.

For example (hereinafter referred to as example 2), the interactive content of the program is who can complete the actions specified by the target character in the program the fastest. In the program content at 8 o'clock, the specified action of the target character is handstand, and the frame number of the streaming media frame corresponding to this program content is 12358. If the terminal device A of the audience A collects that the audience A is handstand, the frame number of the streaming media frame of the program displayed by the terminal device A is 12380, and the terminal device B of the audience B collects the handstand of the audience B, the terminal device B displays the The frame number of the streaming media frame of the program is 12372. When the terminal device C of the audience C collects that the audience C is upside down, the frame number of the streaming media frame of the program displayed by the terminal device C is 12370. The first server determines that the result information includes: Audience C has a rating of 100, Audience B has a rating of 90, and Audience A has a rating of 70.

S613: The second server sends an HTTP response message to the terminal device, and the HTTP response message contains the result information.

Optionally, after receiving the result information, the terminal device can output the result information. For example, for the above example 2, after receiving the result information, the terminal device C of the audience C can display on the screen: the interaction score is 100 points.

S614: The first server determines that the live program exits the interactive session.

For example, after receiving an instruction input by a TV station staff member to exit the interaction with the program of Channel 1, the first server determines that the program of Channel 1 exits the interactive session. For another example, the first server detects that the program content contains interaction end information. If it detects that the program host prompts the audience to end the interaction, the first server can determine that the program on channel 1 has exited the interactive session.

S615: The first server sends video stream 4 of channel 1 to the terminal device.

Among them, video stream 4 includes: the content of video 4 and interaction instruction 1. For the specific content of interaction instruction 1, please refer to S602, which will not be described again here.

Optionally, video stream 4 also includes: the frame sequence number of the streaming media frame in video 4. For the specific content of the frame serial number of the streaming media frame in video 4, please refer to the description of "the frame serial number of the streaming media frame in the program" in the method shown in Figure 5, and will not be described again here.

Additionally, the first server may send the video stream 4 to the terminal device via multicast.

S616: The terminal device displays video 4.

Among them, S602-S606 and S614-S616 may be optional steps.

Through the method shown in Figure 6, after the terminal device obtains the first interaction instruction through the media plane for instructing to start participating in the interaction of the program played by the terminal device, it can output the first interaction prompt to prompt the user to participate in the terminal device. The interaction of the program is played, and the terminal device collects at least one interaction information generated when the user interacts according to the first interaction prompt. The terminal device can then use the control panel Transmit interaction data. Through this method, users who watch the program currently played by the terminal device can participate in the interaction of the program currently played by the terminal device, and there is no need for the terminal device to install an application program with interactive functions or the user needs to use other terminals with applications for interaction Only equipment can achieve interaction and it is easy to implement.

The embodiment of the present application provides another media stream interaction method, which is another possible implementation of the method shown in Figure 5. Referring to the flow chart shown in Figure 7, the flow of this method will be described in detail below.

S701: The terminal device joins the multicast group of channel 1.

For the specific content of S701, please refer to S601 and will not be repeated here.

S702: The terminal device sends control plane message 1 to the first server.

The control plane message 1 may be a message using a control protocol such as DASH, and is used to request the acquisition of the MPD file of channel 1.

Optionally, the terminal device may periodically send a control plane message for requesting to obtain the MPD file of channel 1 to the first server. Control plane message 1 may be a control plane message sent by the terminal device in the first cycle.

S703: The first server sends MPD file 1 of channel 1 to the terminal device.

Among them, MPD file 1 is used to indicate not to interact with the program of channel 1. For example, MPD file 1 does not contain the Interactive instruction parameter. For another example, MPD file 1 contains the Interactive instruction parameter, and the value of InteractiveReadytSegment in the Interactive instruction parameter is empty. Among them, the specific content of the Interactive instruction parameter can be referred to the description of the Interactive instruction parameter in the method shown in Figure 5, and will not be described again here.

S704: The terminal device sends a video segment file request to the first server. The video segment file request is used to request to obtain the program content of channel 1.

The terminal device may send a video fragment file request to the first server after parsing the MPD file 1. The specific process of the terminal device parsing the MPD file 1 is not limited by this application.

S705: The first server sends video stream 1 of channel 1 to the terminal device.

Among them, video stream 1 may include: the content of video 1. Video 1 corresponds to MPD file 1.

S706: The terminal device displays video 1 on the screen.

S707: The first server determines that the program on channel 1 is about to enter the interactive session.

For the specific content of S707, please refer to S604 and will not be repeated here.

S708: The terminal device sends control plane message 2 to the first server.

The control plane message 2 may be a control plane message sent by the terminal device to the first server in the second period for requesting to obtain the MPD file of channel 1. The first period and the second period may or may not be adjacent.

S709: The first server sends MPD file 2 of channel 1 to the terminal device.

The MPD file 2 may include a second interaction instruction, and the second interaction instruction is used to indicate preparation for participating in the interaction of the program of channel 1 . For the specific content of the second interaction instruction, please refer to step C1, which will not be described again here.

Optionally, MPD file 2 also includes at least one of the following: interactive mode instructions and address instructions. The interactive mode indication is used to indicate the interactive mode of the program on Channel 1. For the specific content of the interactive mode indication, please refer to step A1, which will not be described again here. The address indication can be used to indicate the reporting address of interactive data. For the specific content of the address indication, please refer to step B1, which will not be described again here.

S710: The first server sends video stream 2 of channel 1 to the terminal device.

Among them, video stream 2 may include: the content of video 2. Video 2 corresponds to MPD file 2.

Optionally, video stream 2 also includes: the frame sequence number of the streaming media frame in video 2. For the specific content of the frame serial number of the streaming media frame in video 2, please refer to the description of "the frame serial number of the streaming media frame in the program" in the method shown in Figure 5, and will not be described again here.

S711: The terminal device displays video 2 on the screen and prepares to collect at least one interaction information generated when the user interacts.

For the specific content of S711, please refer to S606 and will not be repeated here.

Optionally, S708-S711 can be executed repeatedly in different cycles.

S712: The first server determines that the program of channel 1 enters the interactive session.

For the specific content of S712, please refer to S607 and will not be repeated here.

S713: The terminal device sends control plane message 3 to the first server.

The control plane message 3 may be a control plane message sent by the terminal device to the first server in the third period to request to obtain the MPD file of channel 1. The second period and the third period may be adjacent or not.

S714: The first server sends MPD file 3 of channel 1 to the terminal device.

The MPD file 3 may include a first interaction instruction, and the first interaction instruction is used to instruct the user to start participating in the interaction of the program of channel 1 . For the specific content of the first interaction instruction, reference can be made to method two in step S501, which will not be described again here.

Optionally, the MPD file 3 also includes at least one of the following: interactive mode instructions and address instructions. The interactive mode indication is used to indicate the interactive mode of the program on Channel 1. For the specific content of the interactive mode indication, please refer to step A1, which will not be described again here. The address indication can be used to indicate the reporting address of interactive data. For the specific content of the address indication, please refer to step B1, which will not be described again here.

S715: The first server sends video stream 3 of channel 1 to the terminal device.

Among them, video stream 3 includes: the content of video 3. Video 3 corresponds to MPD file 3.

Optionally, video stream 3 also includes: the frame sequence number of the streaming media frame in video 3. For the specific content of the frame serial number of the streaming media frame in video 3, please refer to the description of "the frame serial number of the streaming media frame in the program" in the method shown in Figure 5, and will not be described again here.

S716: The terminal device displays video 3 on the screen, outputs the first interaction prompt, and collects at least one interaction information generated when the user interacts according to the first interaction prompt. The at least one interaction information can be used to generate interaction data.

For the specific content of S716, please refer to S609 and will not be repeated here.

S717: The terminal device sends a second video stream to the second server, where the second video stream includes interactive data.

The second video stream may be an RTP stream.

For the specific content of S717, please refer to method 1 of S504 and will not be described again here.

S718: The second server sends interactive data to the first server.

For the specific content of S718, please refer to S611 and will not be repeated here.

Optionally, after S718, the first server can determine the result information based on the received interaction data. The process of determining the result information by the first server can be referred to S612, which will not be described again here.

S719: The first server determines that the live program exits the interactive session.

Among them, the specific content of S719 can be referred to S614, which will not be described again here.

S720: The terminal device sends control plane message 4 to the first server.

The control plane message 4 may be a control plane message sent by the terminal device to the first server in the fourth cycle for requesting to obtain the MPD file of channel 1. The fourth period and the third period may or may not be adjacent.

S721: The first server sends the MPD file 4 of channel 1 to the terminal device.

Among them, MPD file 4 is used to indicate not to interact with the program of channel 1. For the content of MPD file 4, please refer to the description of MPD file 1 in S703 and the description of the third interaction instruction in implementation mode B, which will not be described again here.

S722: The first server sends video stream 4 of channel 1 to the terminal device.

Among them, video stream 4 contains: the content of video 4. Among them, video 4 corresponds to MPD file 4.

Among them, video 3 and video 4 can be continuous videos. For example, video 3 and video 4 are both videos of a party. Video 3 is the program content from 8:02 to 8:03 of the party, and video 4 is the program content of party 8. Program content from 03:03 to 8:04.

S723: The terminal device displays video 4 on the screen.

Among them, S702-S711 and S719-S723 can be optional steps.

Through the method shown in Figure 7, the terminal device obtains through the control plane the information used to indicate starting to participate in the interaction of the terminal device playing the program. After the first interaction instruction, the first interaction prompt can be output to prompt the user to participate in the interaction of the program played by the terminal device, and the terminal device collects at least one interaction information generated when the user interacts according to the first interaction prompt. Then, the terminal device can upload interactive data through the media plane. Through this method, users who watch the program currently played by the terminal device can participate in the interaction of the program currently played by the terminal device, and there is no need for the terminal device to install an application program with interactive functions or the user needs to use other terminals with applications for interaction Only equipment can achieve interaction and it is easy to implement.

The embodiment of the present application provides another media stream interaction method, which is another possible implementation of the method shown in Figure 5. Referring to the flow chart shown in Figure 8, the flow of this method will be described in detail below.

S801-S809 are the same as S601-S609 and will not be described again here.

S810-S811 are the same as S717-S718 and will not be described again here.

Optionally, the method shown in Figure 8 also includes:

S812-S814: The same as 614-S616 and will not be described again here.

Through the method shown in FIG8 , after the terminal device obtains the first interactive indication for indicating the start of the interaction of the program played by the terminal device through the media interface, the terminal device can output the first interactive prompt to prompt the user to participate in the interaction of the program played by the terminal device, and the terminal device collects at least one interactive information generated when the user interacts according to the first interactive prompt. Then, the terminal device can upload the interactive data through the media interface. Through this method, users who are watching the program currently played by the terminal device can participate in the interaction of the program currently played by the terminal device, and the terminal device does not need to install an application with interactive functions or require the user to use other terminal devices with interactive applications to achieve interaction, which is easy to implement.

The embodiment of the present application provides yet another media stream interaction method, which is yet another possible implementation of the method shown in Figure 5 . Referring to the flow chart shown in Figure 9, the flow of this method will be described in detail below.

S901-S916 are the same as S701-S716 and will not be described again here.

S917-S920 are the same as S610-S613 and will not be described again here.

Optionally, the method shown in Figure 9 also includes:

S921-S925: The same as S719-S723 and will not be described again here.

Through the method shown in Figure 9, after the terminal device obtains the first interaction instruction through the control plane for instructing to start participating in the interaction of the program played by the terminal device, it can output the first interaction prompt to prompt the user to participate in the terminal device. The interaction of the program is played, and the terminal device collects at least one interaction information generated when the user interacts according to the first interaction prompt. Then, the terminal device can upload the interaction data through the media control plane. Through this method, users who watch the program currently played by the terminal device can participate in the interaction of the program currently played by the terminal device, and there is no need for the terminal device to install an application program with interactive functions or the user needs to use other terminals with applications for interaction Only equipment can achieve interaction and it is easy to implement.

Based on the same technical concept as the method embodiments in Figures 5 to 9, the embodiment of the present application provides a streaming media interactive device through Figure 10, which can be used to perform the functions of the relevant steps in the above method embodiments. The functions described can be implemented by hardware, or can be implemented by software or hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The structure of the streaming media interactive device is shown in Figure 10, including a communication unit 1001 and a processing unit 1002. The streaming media interactive device 1000 can be applied to the terminal device in the communication system shown in Figure 4, and can implement the communication method provided by the above embodiments and examples of the present application. The functions of each unit in the streaming media interactive device 1000 are introduced below.

The communication unit 1001 is used to receive and send data.

In some possible ways, the communication unit 1001 can be implemented through a physical interface, a communication module, a communication interface, and an input and output interface. The streaming media interactive device 1000 can connect to a network cable or cable through the communication unit, and then establish a physical connection with other devices.

In other possible ways, the communication unit 1001 can be implemented by a transceiver, for example, a mobile communication module. Among them, the mobile communication module may include at least one antenna, at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA), etc.

The processing unit 1002 may be used to support the streaming media interactive device 1000 in performing the processing actions in the above method embodiments. The processing unit 1002 may be implemented by a processor. For example, the processor can be a central processing unit (CPU), or other general-purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC) , field programmable gate array (FPGA) Or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

In one implementation, the streaming media interaction device 1000 is applied to the terminal device in the embodiment of the present application shown in any one of Figures 5 to 9. The specific functions of the processing unit 1002 in this embodiment will be introduced below.

The processing unit 1002 is configured to: receive, through the communication unit 1001, a first interaction instruction sent by the first server through streaming media, where the first interaction instruction is used to instruct to start participating in the interaction of the terminal device playing the program; Output a first interactive prompt, the first interactive prompt is used to prompt the user to participate in the interaction of the program played by the terminal device; collect at least one interactive information generated when the user interacts according to the first interactive prompt, and the at least An interaction information is used to generate interaction data; the interaction data is sent to the second server through the communication unit 1001.

Optionally, the interactive data includes:

the at least one interactive information;

When collecting each interaction information in the at least one interaction information, the frame sequence number of the streaming media frame of the program being displayed by the terminal device.

Optionally, the interaction data further includes: when each interaction information in the at least one interaction information is collected, the displayed duration of the streaming media frame of the program being displayed by the terminal device.

Optionally, the media plane data of the first video stream of the program played by the terminal device includes the frame sequence number of the streaming media frame in the program.

Optionally, the processing unit 1002 is specifically configured to: before outputting the first interaction prompt, receive an interaction mode indication sent by the first server through streaming media through the communication unit 1001, where the interaction mode indication is used to Instruct the interactive mode of the terminal device to play the program. The first interactive prompt is specifically used to prompt the user to participate in the interaction of the terminal device to play the program according to the interactive mode; collect the interaction of the user according to the interactive mode. At least one interactive message is generated.

Optionally, the interaction method includes at least one of the following: interaction through somatosensory, interaction through voice recognition, interaction through video, interaction through voice, and interaction through remote control.

Optionally, the processing unit 1002 is specifically configured to: receive a third video stream of the program from the first server through the communication unit 1001, where the media side data of the third video stream includes the interaction way instructions.

Optionally, the processing unit 1002 is specifically configured to: receive third control plane data from the first server through the communication unit 1001, where the third control plane data includes the interaction mode indication.

Optionally, the processing unit 1002 is specifically configured to: receive an address indication sent by the first server through streaming media through the communication unit 1001, where the address indication is used to indicate the reporting address of the interactive data; according to The reporting address is used to send the interactive data to the second server through the communication unit 1001, and the reporting address is the address of the second server.

Optionally, the processing unit 1002 is specifically configured to: receive the fourth video stream of the program from the first server through the communication unit 1001, where the media plane data of the fourth video stream includes the address. instruct.

Optionally, the processing unit 1002 is specifically configured to: receive fourth control plane data from the first server through the communication unit 1001, where the fourth control plane data includes the address indication.

Optionally, the processing unit 1002 is specifically configured to: receive the first video stream of the program from the first server through the communication unit 1001, where the media plane data of the first video stream includes the first video stream. An interactive instruction.

Optionally, the processing unit 1002 is specifically configured to: receive first control plane data from the first server through the communication unit 1001, where the first control plane data includes the first interaction indication.

Optionally, the processing unit 1002 is specifically configured to: send a second video stream to the second server through the communication unit 1001, where the media plane data of the second video stream includes the interactive data.

Optionally, the processing unit 1002 is specifically configured to: send second control plane data to the second server through the communication unit 1001, where the second control plane data includes the interaction data.

Optionally, the processing unit 1002 is specifically configured to: before receiving the first interaction instruction sent by the first server through streaming media, receive through the communication unit 1001 the second interaction instruction sent by the first server through streaming media. Interaction indication, the second interaction indication is used to indicate preparation to participate in the interaction of the terminal device playing the program; output the second interaction prompt, the second interaction prompt is used to prompt the user to prepare to participate in the terminal device playback program interaction.

Optionally, the processing unit 1002 is specifically configured to: receive the fifth video stream of the program from the first server through the communication unit 1001, where the media plane data of the fifth video stream includes the third video stream. 2. Interactive instructions.

Optionally, the processing unit 1002 is specifically configured to: receive the fifth control message from the first server through the communication unit 1001. Noodle making data, the fifth control plane data includes the second interaction instruction.

Optionally, the processing unit 1002 is specifically configured to receive, through the communication unit 1001, a third interaction instruction sent by the first server through streaming media, where the third interaction instruction is used to instruct the terminal to stop participating. The device plays a program for interaction; and stops collecting at least one interaction information generated when the user interacts according to the first interaction prompt.

Optionally, the processing unit 1002 is specifically configured to: receive a sixth video stream containing the program from the first server through the communication unit 1001, and the media plane data of the sixth video stream contains the Third interactive instruction.

Optionally, the processing unit 1002 is specifically configured to: receive sixth control plane data from the first server through the communication unit 1001, where the sixth control plane data includes the third interaction indication.

It should be noted that the division of modules in the above embodiments of the present application is schematic and is only a logical function division. In actual implementation, there may be other division methods. In addition, each function in each embodiment of the present application The units can be integrated into one processing unit, or they can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code. .

Based on the same technical concept, the embodiment of the present application provides a streaming media interactive device as shown in Figure 11, which can be used to perform relevant steps in the above method embodiment. The streaming media interactive device can be applied to terminal equipment in the communication system shown in Figure 4, can implement the communication methods provided in the above embodiments and examples of the present application, and has the functions of the streaming media interactive device shown in Figure 10. Referring to FIG. 11 , the streaming media interactive device 1100 includes: a communication module 1101 , a processor 1102 and a memory 1103 . The communication module 1101, the processor 1102 and the memory 1103 are connected to each other.

Optionally, the communication module 1101, the processor 1102 and the memory 1103 are connected to each other through a bus 1104. The bus 1104 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one thick line is used in Figure 11, but it does not mean that there is only one bus or one type of bus.

The communication module 1101 is used to receive and send data to implement communication interaction with other devices. For example, the communication module 1101 can be implemented through a physical interface, a communication module, a communication interface, and an input and output interface.

The processor 1102 may be used to support the streaming media interactive device 1100 to perform the processing actions in the above method embodiment. When the streaming media interactive device 1100 is used to implement the above method embodiments, the processor 1102 may also be used to implement the functions of the above processing unit 1002. The processor 1102 may be a CPU, or other general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, transistor logic device, hardware component or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

In one implementation, the streaming media interaction device 1100 is applied to the terminal device in the embodiment of the present application shown in any one of Figures 5 to 9. The processor 1102 is specifically configured to: receive, through the communication module 1101, a first interaction instruction sent by the first server through streaming media, where the first interaction instruction is used to instruct to start participating in the interaction of the terminal device playing the program; Output a first interactive prompt, the first interactive prompt is used to prompt the user to participate in the interaction of the program played by the terminal device; collect at least one interactive information generated when the user interacts according to the first interactive prompt, and the at least An interaction information is used to generate interaction data; the interaction data is sent to the second server through the communication module 1101.

For the specific functions of the processor 1102, please refer to the description in the above embodiments of the present application and the communication methods provided in the examples, as well as the specific functional description of the streaming media interactive device 1000 in the embodiment of the present application shown in Figure 10, here No longer.

The memory 1103 is used to store program instructions and data. Specifically, the program instructions may include program codes, and the program codes include computer operation instructions. The memory 1103 may include RAM, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory. The processor 1102 executes the program instructions stored in the memory 1103, and uses the memory 1103 to execute the program instructions stored in the memory 1103. The data stored in the memory 1103 realizes the above functions, thereby realizing the communication method provided in the above embodiment of the present application.

It can be understood that the memory 1103 in Figure 11 of this application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be ROM, programmable ROM (PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electrically erasable programmable read-only memory (Electrically EPROM) ,EEPROM) or flash memory. Volatile memory can be RAM, which acts as an external cache. By way of illustration, but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchlink DRAM, SLDRAM) ) and direct memory bus random access memory (Direct Rambus RAM, DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

Based on the above embodiments, embodiments of the present application also provide a computer program, which when the computer program is run on a computer, causes the computer to execute the method provided in the above embodiments.

Based on the above embodiments, embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, it causes the computer to execute the method provided in the above embodiments. .

The storage medium may be any available medium that can be accessed by the computer. Taking this as an example but not limited to: computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or can be used to carry or store instructions or data structures. Any other medium that contains the desired program code and is capable of being accessed by a computer.

Based on the above embodiments, embodiments of the present application also provide a chip, which is used to read the computer program stored in the memory and implement the method provided in the above embodiments.

Based on the above embodiments, embodiments of the present application provide a chip system. The chip system includes a processor and is used to support the computer device to implement the functions involved in each device in the above embodiments. In a possible design, the chip system further includes a memory, and the memory is used to store necessary programs and data of the computer device. The chip system may be composed of chips, or may include chips and other discrete devices.

To sum up, embodiments of the present application provide a streaming media interaction method and device. In this method, after receiving the third message sent by the first server through streaming media to instruct the user to start participating in the interaction of the terminal device playing the program, After an interaction instruction, the terminal device can output a first interaction prompt to prompt the user to participate in the interaction of the program played by the terminal device, and collect at least one interaction information generated when the user interacts according to the first interaction prompt, and the at least one interaction information is used to generate Interactive data. Then, the terminal device can send the interaction data to the second server. Through this method, users who watch the program currently played by the terminal device can participate in the interaction of the program currently played by the terminal device, and there is no need for the terminal device to install an application program with interactive functions or the user needs to use other terminals with applications for interaction Only equipment can achieve interaction and it is easy to implement.

In the various embodiments of this application, if there is no special explanation or logical conflict, the terms and/or descriptions between different embodiments are consistent and can be referenced to each other. The technical features in different embodiments are based on their inherent Logical relationships can be combined to form new embodiments.

Those skilled in the art will understand that embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the present application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device for realizing the functions specified in one process or multiple processes of the flowchart and/or one block or multiple blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that when the computer or other programmable data processing device A series of operational steps are performed on a device to produce a computer-implemented process, whereby instructions executed on a computer or other programmable device provide for implementing a process or processes in a flowchart and/or a block or processes in a block diagram The steps for the function specified in the box.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. In this way, if these modifications and variations of the present application fall within the scope of the claims of the present application and equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

A streaming media interaction method, applied to terminal devices, is characterized by including:

Receive a first interaction instruction sent by the first server through streaming media, where the first interaction instruction is used to instruct to start participating in the interaction of the program played by the terminal device;

Output a first interactive prompt, the first interactive prompt is used to prompt the user to participate in the interaction of the program played by the terminal device;

Collect at least one interaction information generated when the user interacts according to the first interaction prompt, and the at least one interaction information is used to generate interaction data;

Send the interaction data to the second server.
The method of claim 1, wherein the interaction data includes:

the at least one interactive information;

When collecting each interaction information in the at least one interaction information, the frame sequence number of the streaming media frame of the program being displayed by the terminal device.
The method of claim 2, wherein the interaction data further includes:

When each interaction information in the at least one interaction information is collected, the displayed duration of the streaming media frame of the program being displayed by the terminal device.
The method according to any one of claims 1 to 3, characterized in that the media plane data of the first video stream of the program played by the terminal device includes the frame sequence number of the streaming media frame in the program.
The method according to any one of claims 1 to 4, characterized in that, before outputting the first interactive prompt, the method further includes:

Receive an interaction mode indication sent by the first server through streaming media, where the interaction mode indication is used to instruct the terminal device to play a program in an interactive mode;

The first interaction prompt is specifically used to prompt the user to participate in the interaction of the program played by the terminal device according to the interaction method;

Collecting at least one interaction information generated when the user interacts according to the first interaction prompt includes:

Collect at least one interaction information generated when the user interacts according to the interaction method.
The method of claim 5, wherein the interaction method includes at least one of the following: interaction through somatosensory, interaction through voice recognition, interaction through video, interaction through voice, and interaction through remote control.
The method according to claim 5 or 6, characterized in that receiving the interaction mode indication sent by the first server through streaming media includes:

A third video stream of the program is received from the first server, and the media plane data of the third video stream includes the interaction mode indication.
The method according to claim 5 or 6, characterized in that receiving the interaction mode indication sent by the first server through streaming media includes:

Receive third control plane data from the first server, where the third control plane data includes the interaction mode indication.
The method according to any one of claims 1 to 7, characterized in that the method further includes:

Receive an address indication sent by the first server through streaming media, where the address indication is used to indicate the reporting address of the interactive data;

Send interaction data to the second server, including:

Send the interaction data to the second server according to the reporting address, where the reporting address is the address of the second server.
The method of claim 9, wherein receiving the address indication sent by the first server through streaming media includes:

A fourth video stream of the program is received from the first server, and media plane data of the fourth video stream includes the address indication.
The method according to claim 9, wherein receiving the address indication sent by the first server via streaming media comprises:

Fourth control plane data is received from the first server, the fourth control plane data including the address indication.
The method according to any one of claims 1 to 11, characterized in that receiving the first interaction instruction sent by the first server through streaming media includes:

A first video stream of the program is received from the first server, and the media plane data of the first video stream includes the first interaction indication.
The method according to any one of claims 1 to 11, characterized in that receiving the first interaction instruction sent by the first server through streaming media includes:

Receive first control plane data from the first server, where the first control plane data includes the first interaction indication.
The method according to any one of claims 1-13, characterized in that sending interaction data to the second server includes:

Send a second video stream to the second server, where the media plane data of the second video stream includes the interaction data.
The method according to any one of claims 1-13, characterized in that sending interaction data to the second server includes:

Send second control plane data to the second server, where the second control plane data includes the interaction data.
The method according to any one of claims 1 to 15, characterized in that, before receiving the first interaction instruction sent by the first server through streaming media, the method further includes:

Receive a second interaction instruction sent by the first server in a streaming media manner, where the second interaction instruction is used to indicate preparation for participation in interaction with a program played by the terminal device;

The second interaction prompt is output, and the second interaction prompt is used to prompt the user to prepare to participate in the interaction of the program played by the terminal device.
The method of claim 16, wherein receiving the second interaction instruction sent by the first server through streaming media includes:

A fifth video stream of the program is received from the first server, and the media plane data of the fifth video stream includes the second interaction indication.
The method of claim 16, wherein receiving the second interaction instruction sent by the first server through streaming media includes:

Receive fifth control plane data from the first server, where the fifth control plane data includes the second interaction indication.
The method according to any one of claims 1-18, characterized in that the method further includes:

Receive a third interaction instruction sent by the first server through streaming media, where the third interaction instruction is used to instruct to stop participating in the interaction of the terminal device playing the program;

Stop collecting at least one interaction information generated when the user interacts according to the first interaction prompt.
The method of claim 19, wherein receiving the third interaction instruction sent by the first server through streaming media includes:

A sixth video stream containing the program is received from the first server, and the media plane data of the sixth video stream contains the third interaction indication.
The method of claim 19, wherein receiving the third interaction instruction sent by the first server through streaming media includes:

Receive sixth control plane data from the first server, where the sixth control plane data includes the third interaction indication.
A streaming media interaction method, characterized by including:

The first server sends a first interaction instruction to the terminal device through streaming media, where the first interaction instruction is used to instruct to start participating in the interaction of the program played by the terminal device;

The terminal device receives the first interaction instruction sent by the first server through streaming media;

The terminal device outputs a first interactive prompt, and the first interactive prompt is used to prompt the user to participate in the interaction of the program played by the terminal device;

The terminal device collects at least one interaction information generated when the user interacts according to the first interaction prompt, and the at least one interaction information is used to generate interaction data;

The terminal device sends the interaction data to the second server.
The method of claim 22, wherein the interaction data includes:

the at least one interactive information;

When collecting each interaction information in the at least one interaction information, the frame sequence number of the streaming media frame of the program being displayed by the terminal device.
The method of claim 23, wherein the interaction data further includes:

When each interaction information in the at least one interaction information is collected, the displayed duration of the streaming media frame of the program being displayed by the terminal device.
The method according to any one of claims 22 to 24, characterized in that the terminal device plays the media of the first video stream of the program. Decent data includes the frame number of the streaming frame in the program.
The method according to any one of claims 22 to 25, characterized in that, before the terminal device outputs the first interaction prompt, the method further includes:

The first server sends an interaction mode indication to the terminal device through streaming media, and the interaction mode indication is used to instruct the terminal device to play an interactive mode of the program;

The terminal device receives the interaction mode indication sent by the first server through streaming media;

The first interaction prompt is specifically used to prompt the user to participate in the interaction of the program played by the terminal device according to the interaction method;

The terminal device collects at least one interaction information generated when the user interacts according to the first interaction prompt, including:

The terminal device collects at least one interaction information generated when the user interacts according to the interaction method.
The method of claim 26, wherein the interaction method includes at least one of the following: interaction through somatosensory, interaction through voice recognition, interaction through video, interaction through voice, and interaction through remote control.
The method of claim 26 or 27, wherein the first server sends an interactive mode indication to the terminal device through streaming media, including: the first server sends the program to the terminal device. The third video stream, the media plane data of the third video stream includes the interaction mode indication;

The terminal device receiving the interactive mode indication sent by the first server through streaming media includes: the terminal device receiving a third video stream of the program from the first server, and the third video The media side data of the stream contains the interaction mode indication.
The method of claim 26 or 27, wherein the first server sends an interaction mode indication to the terminal device through streaming media, including: the first server sends a third control to the terminal device. Surface data, the third control surface data includes the interaction mode indication;

The terminal device receiving the interaction mode indication sent by the first server through streaming media includes: the terminal device receiving third control plane data from the first server, where the third control plane data includes Indicates the interaction mode.
The method according to any one of claims 22 to 28, characterized in that the method further includes:

The first server sends an address indication to the terminal device through streaming media, where the address indication is used to indicate the reporting address of the interactive data;

The terminal device receives the address indication sent by the first server through streaming media;

The terminal device sends interactive data to the second server, including:

The terminal device sends the interactive data to the second server according to the reporting address, where the reporting address is the address of the second server.
The method of claim 30, wherein the first server sends an address indication to the terminal device through streaming media, including: the first server sends a fourth portion of the program to the terminal device. Video stream, the media plane data of the fourth video stream includes the address indication;

The terminal device receiving the address indication sent by the first server through streaming media includes: the terminal device receiving a fourth video stream of the program from the first server, the fourth video stream The media plane data contains the address indication.
The method of claim 30, wherein the first server sends an address indication to the terminal device through streaming media, including: the first server sends fourth control plane data to the terminal device, The fourth control plane data includes the address indication;

The terminal device receiving the address indication sent by the first server through streaming media includes: the terminal device receiving fourth control plane data from the first server, the fourth control plane data including the address instructions.
The method according to any one of claims 22 to 32, characterized in that the first server sends the first interaction instruction to the terminal device through streaming media, including: the first server sends the program to the terminal device The first video stream, the media plane data of the first video stream includes the first interaction indication;

The terminal device receiving the first interaction instruction sent by the first server through streaming media includes: the terminal device receiving the first video stream of the program from the first server, the first The media plane data of the video stream includes the first interaction indication.
The method according to any one of claims 22 to 32, characterized in that the first server sends the first interaction instruction to the terminal device through streaming media, including: the first server sends the first control to the terminal device. Surface data, the first control surface data includes the first interaction indication;

The terminal device receiving the first interaction instruction sent by the first server through streaming media includes: the terminal device receiving the first control plane data from the first server, the first control plane data Contains the first interaction indication.
The method according to any one of claims 22 to 34, characterized in that the terminal device sends interactive data to the second server, including:

The terminal device sends a second video stream to the second server, and the media plane data of the second video stream includes the interactive data.
The method according to any one of claims 22 to 34, characterized in that the terminal device sends interactive data to the second server, including:

The terminal device sends second control plane data to the second server, where the second control plane data includes the interactive data.
The method according to any one of claims 22 to 36, characterized in that, before the first server sends the first interaction instruction to the terminal device through streaming media, the method further includes:

The first server sends a second interaction instruction to the terminal device through streaming media, where the second interaction instruction is used to indicate preparation to participate in the interaction of the program played by the terminal device;

The terminal device receives the second interaction instruction sent by the first server through streaming media;

The terminal device outputs the second interaction prompt, and the second interaction prompt is used to prompt the user to prepare to participate in the interaction of the program played by the terminal device.
The method of claim 37, wherein the first server sends the second interaction instruction to the terminal device through streaming media, including: the first server sends the program's content to the terminal device. A fifth video stream, the media plane data of the fifth video stream includes the second interaction indication;

The terminal device receiving the second interaction instruction sent by the first server through streaming media includes: the terminal device receiving a fifth video stream of the program from the first server, and the fifth The media plane data of the video stream includes the second interaction indication.
The method of claim 37, wherein the first server sends a second interaction instruction to the terminal device through streaming media, including: the first server sends a fifth control plane to the terminal device. Data, the fifth control plane data includes the second interaction indication;

The terminal device receiving the second interaction instruction sent by the first server through streaming media includes: the terminal device receiving fifth control plane data from the first server, the fifth control plane data Contains the second interaction indication.
The method according to any one of claims 22 to 39, characterized in that the method further includes:

The first server sends a third interaction instruction to the terminal device through streaming media, where the third interaction instruction is used to instruct to stop participating in the interaction of the program played by the terminal device;

The terminal device receives the third interaction instruction sent by the first server through streaming media;

The terminal device stops collecting at least one interaction information generated when the user interacts according to the first interaction prompt.
The method of claim 40, wherein the first server sends a third interaction instruction to the terminal device through streaming media, including: the first server sends a message containing the program to the terminal device. The sixth video stream, the media plane data of the sixth video stream includes the third interaction indication;

The terminal device receiving the third interaction instruction sent by the first server through streaming media includes: the terminal device receiving a sixth video stream containing the program from the first server, and the third The media plane data of the six video streams includes the third interaction indication.
The method of claim 40, wherein the first server sends a third interaction instruction to the terminal device through streaming media, including: the first server sends a sixth control plane to the terminal device. Data, the sixth control plane data includes the third interaction indication;

The terminal device receiving the third interaction instruction sent by the first server through streaming media includes: the terminal device receiving sixth control plane data from the first server, the sixth control plane data Contains the third interaction indication.
A media streaming interactive device, characterized by including:

Communication unit for receiving and sending data;

A processing unit, configured to execute the method according to any one of claims 1-21 through the communication unit.
A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is run on a computer, it causes the computer to execute any one of claims 1-21 the method described.
A chip, characterized in that the chip is coupled to a memory, and the chip reads the computer program stored in the memory and executes the method according to any one of claims 1-21.