WO2018095003A1 - Method and apparatus for real-time transmission of multimedia data - Google Patents

Abstract

During a video live broadcast process, instruction monitoring is carried out on a live broadcast terminal; when the live broadcast terminal is monitored to generate a somatosensory control instruction, a somatosensory control data packet is generated according to the somatosensory control instruction and an image frame that is currently collected by the live broadcast terminal is marked; the somatosensory control data packet and the image frame are associated; and the image frame is sent to a user terminal in real time by means of a video live broadcast platform, and the somatosensory control data packet associated with the image frame is synchronously sent to the user terminal at the same time by means of the video live broadcast platform, so that the user terminal outputs the somatosensory control data packet to a somatosensory sensing device synchronously while outputting the image frame to a video playback device. Therefore, the user terminal may experience somatosensory simulation matching the video live broadcast content while watching the live-broadcast video, so that interactivity of an on-line scenario is substantially improved.

Description

Real-time transmission method and device for multimedia data Technical field

The invention belongs to the technical field of networks, and in particular relates to a real-time transmission method and device for multimedia data.

Background technique

Multimedia refers to a human-computer interactive information communication and communication medium that combines two or more media, including text, pictures, sounds, animations, and movies, as well as interactive functions provided by the program. With the development of science and technology, multimedia technology can not only bring visual and auditory experiences to users through computer, but also simulate other senses of human beings, including touch, power, warmth, wetness and wind. The sense of smell, such as the sense of smell, is simulated by all human beings to give the user an interactive experience of somatosensory.

The realization principle of the somatosensory sensing is to send a somatosensory control signal to the somatosensory sensing device attached to the human body surface to control the somatosensory sensing device to perform various types of somatosensory feedback to the human body. For example, when shaking hands with a virtual character in a virtual reality (VR) game, the force feedback control signal controls the magnitude and duration of the force generated by the force sensor located at the hand, thereby bringing a handshake behavior with the game. Match the force feedback experience. However, during the development process, the technician found that the prior art has at least the following technical defects:

A large part of the existing video files are streamed to the user for online play, and the existing video content providers and various types of somatosensory device suppliers independently provide services, video content and somatosensory sensing control signals. The format is also different, so the prior art cannot match the somatosensory data in real time for the video played online, thereby reducing the interactivity of the online scene.

Summary of the invention

In view of this, the embodiments of the present invention provide a real-time transmission method and device for multimedia data, so as to solve the problem that the prior art cannot match the somatosensory data in real time for the video played online, thereby reducing the interaction of the online scene.

In a first aspect, an embodiment of the present invention provides a real-time sending method for multimedia data, where the method includes:

During the live broadcast of the video, the live broadcast terminal is instructed to listen;

When the live control terminal generates the somatosensory control command, the somatosensory control data packet is generated according to the somatosensory control command, and the image frame currently collected by the live broadcast terminal is marked;

Associating the somatosensory control data packet with the image frame;

Sending the image frame to the user terminal in real time through the video live broadcast platform, and simultaneously transmitting the somatosensory control data packet associated with the image frame to the user terminal through the video live broadcast platform, so that the user terminal While outputting the image frame to the video playback device, the somatosensory control data packet is synchronously outputted to the somatosensory sensing device.

As a first possible implementation manner of the first aspect, the generating the somatosensory control data packet according to the somatosensory control instruction includes:

Obtaining somatosensory control data of M body points, and the somatosensory control data of each body point includes control information of N types of somatosensory sensing devices;

And acquiring the acquired somatosensory control data of the M body points into the somatosensory control data packet;

The M and the N are each an integer greater than or equal to 1.

With reference to the first possible implementation of the first aspect, in a second possible implementation, the generating the somatosensory control data packet according to the somatosensory control command further includes:

If the control information of the jth type of the somatosensory sensing device of the i-th body point is not obtained, each data bit corresponding to the control information is written into the preset character in the somatosensory control data packet;

The i and the j are integers greater than or equal to 1, and the i is less than or equal to the M, and the j is less than or equal to the N.

In conjunction with the second possible implementation of the first aspect, in a third possible implementation, the preset character is 0.

A second aspect is a real-time transmitting device for multimedia data, the device comprising:

The monitoring unit is configured to perform instruction monitoring on the live terminal during the live broadcast of the video;

a generating unit, configured to generate a somatosensory control data packet according to the somatosensory control command when the live broadcast terminal generates the somatosensory control command, and mark an image frame currently collected by the live broadcast terminal;

An association unit, configured to associate the somatosensory control data packet with the image frame;

a sending unit, configured to send the image frame to the user terminal in real time through a video live broadcast platform, and simultaneously send the somatosensory control data packet associated with the image frame to the user terminal by using the video live broadcast platform, And causing the user terminal to synchronously output the somatosensory control data packet to the somatosensory sensing device while outputting the image frame to the video playing device.

As a first possible implementation manner of the second aspect, the generating unit includes:

Obtaining a subunit for acquiring somatosensory control data of M body points, and the somatosensory control data of each body point includes control information of N types of somatosensory sensing devices;

a first writing subunit, configured to write the acquired somatosensory control data of the M body points into the somatosensory control data packet;

The M and the N are each an integer greater than or equal to 1.

With reference to the first possible implementation of the second aspect, in a second possible implementation, the generating unit further includes:

a second writing subunit for controlling information of the jth type of somatosensory sensing device that does not acquire the i-th body point, each data bit corresponding to the control information in the somatosensory control data packet Write preset characters;

The i and the j are integers greater than or equal to 1, and the i is less than or equal to the M, and the j is less than or equal to the N.

In conjunction with the second possible implementation of the second aspect, in a third possible implementation, the preset character is 0.

In the embodiment of the present invention, the live control terminal can generate and transmit the somatosensory control data to the user terminal in real time, so that the user terminal can experience the somatosensory simulation matching the live video content while watching the live video, greatly Improve the interactivity of online scenes.

In a third aspect, an embodiment of the present invention provides a real-time output method for multimedia data, where the method includes:

In the process of watching the live video broadcast, the image broadcast frame is received by the live broadcast platform in real time;

If the image frame is associated with a somatosensory control data packet, the somatosensory control data packet that is synchronously sent by the live broadcast terminal is received by the video live broadcast platform, where the somatosensory control data packet is broadcasted by the live broadcast terminal. Generated according to the somatosensory control command;

The image frame is output to the video playback device, and the somatosensory control data packet is synchronously output to the somatosensory sensing device.

As a first possible implementation manner of the third aspect, the synchronous outputting the somatosensory control data packet to the somatosensory sensing device includes:

Parsing the body sense control data of the M body points from the somatosensory control data packet;

Parsing the control information of the N types of the somatosensory sensing devices from the parsed somatosensory control data of the i-th body point;

Outputting the parsed control information to N types of somatosensory sensing devices on the i-th body point;

The M, the N, and the i are all integers greater than or equal to 1, and the i is less than or equal to M.

In conjunction with the first possible implementation of the third aspect, in the second possible implementation, the synchronizing output of the somatosensory control data packet to the somatosensory sensing device further includes:

If the control information corresponding to the jth type of the somatosensory sensing device of the i-th body point in the somatosensory control data packet is a preset character in each data bit, the somatosensory control data packet is associated with While the image frame is being played, the output of the control information is stopped;

The j is an integer greater than or equal to 1, and the j is less than or equal to N.

In conjunction with the second possible implementation of the third aspect, in a third possible implementation, the preset character is 0.

In a fourth aspect, an embodiment of the present invention provides a real-time output device for multimedia data, where the device includes:

a first receiving unit, configured to receive, by the video live broadcast platform, an image frame sent by the live broadcast terminal in real time during the live broadcast of the video;

a second receiving unit, configured to: when the image frame is associated with a somatosensory control data packet, receive, by the video live broadcast platform, the somatosensory control data packet that is synchronously sent by the live broadcast terminal, where the somatosensory control data packet is sent by the live broadcast The terminal is generated according to the somatosensory control instruction during the live broadcast of the video;

And an output unit, configured to output the image frame to the video playing device, and synchronously output the somatosensory control data packet to the somatosensory sensing device.

As a first possible implementation manner of the fourth aspect, the output unit includes:

a first parsing subunit, configured to parse the body sense control data of the M body points from the somatosensory control data packet;

a second parsing subunit, configured to parse control information of the N types of the somatosensory sensing devices from the parsed somatosensory control data of the i-th body point;

a first output control subunit, configured to output the parsed control information to the N types of somatosensory sensing devices on the i-th body point;

The M, the N, and the i are all integers greater than or equal to 1, and the i is less than or equal to M.

In conjunction with the first possible implementation of the fourth aspect, in a second possible implementation, the output unit further includes:

a second output control subunit, configured to: in the somatosensory control data packet, the control information corresponding to the jth type of the somatosensory sensing device of the i-th body point is a preset character in each data bit, And stopping outputting the control information while playing the image frame related to the somatosensory control data packet;

The j is an integer greater than or equal to 1, and the j is less than or equal to N.

With reference to the second possible implementation manner of the fourth aspect, in a third possible implementation manner, the preset character is 0.

The embodiment of the invention can experience the somatosensory simulation matching the live content of the video while watching the live video, and greatly improves the interactivity of the online scene.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments or the description of the prior art will be briefly described below. It is obvious that the drawings in the following description are only the present invention. For some embodiments, other drawings may be obtained from those of ordinary skill in the art in light of the inventive workability.

1 is a flowchart of implementing a real-time sending method of multimedia data according to an embodiment of the present invention;

2 is a specific implementation flowchart of a real-time sending method S102 of multimedia data according to an embodiment of the present invention;

3 is a structural block diagram of a real-time transmitting apparatus for multimedia data according to an embodiment of the present invention;

4 is a schematic diagram of a computing node according to an embodiment of the present invention;

5 is a flowchart of implementing a real-time output method of multimedia data according to an embodiment of the present invention;

FIG. 6 is a flowchart of a specific implementation of a real-time output method S503 for multimedia data according to an embodiment of the present invention;

7 is a structural block diagram of a real-time output device for multimedia data according to an embodiment of the present invention;

FIG. 8 is a schematic diagram of another computing node according to an embodiment of the present invention.

detailed description

In the following description, for purposes of illustration and description However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, Detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the invention.

The real-time transmission scheme of the multimedia data mentioned in the embodiment of the present invention can insert the somatosensory control data into the live video stream at any time during the live broadcast of the video, and transmit the somatosensory control data along with the live video stream to the live broadcast platform. The user terminal is configured to enable the user terminal to simultaneously experience the somatosensory feedback during the live broadcast of the video. In the embodiment of the present invention, the somatosensory control data is used to control the somatosensory sensing device, and the somatosensory control data is sent from the somatosensory control device in the form of a somatosensory control signal and transmitted to the somatosensory sensor electrically connected to the somatosensory control device. In the device. In an actual product form, the somatosensory control device and one or more somatosensory sensing devices can be combined with the wearable product, for example, attaching the somatosensory control device and the plurality of somatosensory sensing devices to the wearable body that wraps the entire body of the user. The somatosensory control device outputs the somatosensory control data to the plurality of somatosensory sensing devices, so that the plurality of somatosensory sensing devices perform a somatosensory simulation on the user based on the received somatosensory control data.

First, the real-time sending method of the multimedia data provided by the embodiment of the present invention is described in detail, and the implementation process thereof is as shown in FIG. 1:

In S101, during the live broadcast of the video, the live broadcast terminal performs instruction monitoring.

Here, the live video broadcast mainly refers to the live broadcast using the Internet and streaming media technology. The live broadcast terminal can refer to the acquisition terminal for collecting image signals and sound signals, and can be a professional audio and video capture camcorder, and can also be a smart phone. A smart terminal device with a camera function, such as a tablet. The video live broadcast process mainly refers to the image and sound digital signals collected by the audio and video collection terminal being generated by the streaming media encoder, and transmitted to the user terminal through the video live broadcast platform.

In the embodiment of the present invention, when the live broadcast terminal is in the live broadcast process, the live broadcast terminal performs instruction monitoring to detect whether the user has issued a somatosensory control command through the live broadcast terminal.

In S102, when the live broadcast terminal detects that the live control terminal generates the somatosensory control command, the somatosensory control data packet is generated according to the somatosensory control command, and the image frame currently collected by the live broadcast terminal is marked.

During the live broadcast of the video, the live terminal provides an operation menu for issuing the somatosensory control command on the display interface, and the user can issue a somatosensory control command through the operation menu at any time. The somatosensory control data is carried in the order for controlling the body sensing device. The somatosensory control data packet is generated according to the monitored somatosensory control command, and an image frame is marked in the live video stream collected by the live broadcast terminal, where the image frame is an image frame acquired when the somatosensory control command is monitored.

Further, as an embodiment of the present invention, each of the somatosensory control data packets may carry the somatosensory control data of one or more body points, and the somatosensory control data of each body point includes one or more Control information for a type of somatosensory sensing device. Specifically, as shown in FIG. 2, the process of generating a sense control packet is as follows:

S201: Acquire somatic control data of M body points, and the somatosensory control data of each body point includes control information of N types of somatosensory sensing devices.

S202. Write the acquired somatosensory control data of the M body points into the somatosensory control data packet.

Wherein M and the N are integers greater than or equal to 1.

Based on the embodiment shown in FIG. 2, when both M and N are equal to 1, each of the somatosensory control data packets is used to control a somatosensory sensing device on a body point to generate a somatosensory simulation when the image frame is played; When M is greater than 1 and N is equal to 1, each of the somatosensory control data packets is used to control the same somatosensory sensing device on the plurality of body points to simultaneously generate the somatosensory simulation when the image frame is played; when both M and N are greater than At 1 o'clock, each of the somatosensory control data packets is used to control a plurality of somatosensory sensing devices on a plurality of body points to simultaneously generate a somatosensory simulation.

In the embodiment of the present invention, for each body point, at least one or more types of somatosensory sensing devices as listed in Table 1 may be integrated according to product requirements:

Table 1

Serial number	Somatosensory sensor	Serial number	Somatosensory sensor
1	Massage/vibration	8	wind
2	Tactile	9	Wet
3	Stinging	10	Peripheral light
4	Warm/hot	11	Peripheral rheumatism
5	cold	12	Peripheral air cooling
6	odor	13	Mechanical exoskeleton
7	shrink

Preferably, in the embodiment of the present invention, a frame format of a somatosensory control data packet is also proposed, and the frame format comprises a start control frame as a frame header and a somatosensory data frame.

(1) The control data carrying several bytes in the initial control frame:

Exemplarily, as shown in Table 2, the start control frame includes 4 bytes of control data, which are control frame byte 1, control frame byte 2, control frame byte 3, and control frame byte. 4. Each control frame byte can be written into the corresponding data content according to the needs of the somatosensory control. For example, the control frame byte can be used to indicate the number of body points covered by the somatosensory control data packet, or to indicate the somatosensory control data packet. Whether it needs to be shielded by the control device.

Table 2

(2) The somatosensory data frame carries the somatosensory control data of several kinds of somatosensory sensors sequentially written, and also reserves the data bits for writing the somatosensory control data of several other somatosensory sensors according to development needs in the future:

In the somatosensory data frame, the somatosensory control data corresponding to each of the somatosensory sensors may be used to indicate the somatosensory sensor related somatosensory control mode, or to describe the somatosensory sensor related somatosensory function. Now. Except as shown in Table 3, each of the somatosensory sensors corresponds to 4 bytes of somatosensory control data, which are respectively a somatosensory control mode byte 1, a somatosensory function byte 1, a somatosensory function byte 2, and a somatosensory function byte 3 .

table 3

Table 4 shows an example of the frame structure of a complete somatosensory control data packet. As can be seen from Table 4, the frame header of the somatosensory control data packet is a 4-byte initial control frame, and the somatosensory data frame is sequentially written. The somatosensory control data of 13 kinds of somatosensory sensing devices, including 7 kinds of function reservations, can further write the somatosensory control data of the seven kinds of somatosensory sensing devices, and the somatosensory control data of each somatosensory sensing device is 4 byte.

Table 4

Illustratively, Table 5 shows the somatosensory control data frame structures corresponding to the 13 somatosensory sensing preambles mentioned in Table 1:

table 5

In the communication processing process of the somatosensory sensing data, if the somatosensory control data packet is generated by using the frame format described above, on the one hand, the somatosensory control data of a plurality of different types of somatosensory sensing devices can be integrated into a unified data format. Effectively eliminating the fragmentation phenomenon of the sensing device due to different suppliers; on the other hand, writing the somatosensory control data of a plurality of different types of somatosensory sensing devices into one data packet for transmission, To some extent, the occurrence of packet loss during data transmission is reduced, and the reliability of data communication is improved.

Further, on the basis of the above-described somatosensory control packet frame format, as an embodiment of the present invention, in the process of generating a somatosensory control packet:

If the control information of the jth type of the somatosensory sensing device of the i-th body point is not obtained, each data bit corresponding to the control information is written into the preset character in the somatosensory control data packet;

The i and the j are integers greater than or equal to 1, and the i is less than or equal to the M, and the j is less than or equal to the N.

That is, a character is preset, and if a body point is not matched with a certain type of somatosensory simulation, in the somatosensory control data packet, each of the types of somatosensory sensing devices corresponding to the body point is occupied. The data bits are written to the preset character. For example, if the somatosensory control command issued by the user only indicates that force feedback is needed, and no other somatosensory simulation is required, all the body points are all except the data bits occupied by the force feedback device in the somatosensory control data packet. The data bits are written to the preset character.

The frame writing scheme of the above-mentioned somatosensory control data packet can ensure that the data lengths of all the somatosensory control data packets are consistent, so that the data length is the same for each generated somatosensory control data packet, which is helpful in the data verification process. Quickly discover the packet loss situation.

Optionally, the preset character is 0.

In S103, the somatosensory control data packet is associated with the image frame.

Specifically, the somatosensory control data packet generated in S102 is associated with the image frame marked in S102, and the association relationship exists once determined, even after the video live broadcast process ends, from the video live broadcast platform through the user terminal. When the entire live broadcast process is played back, the association relationship always exists to ensure that the somatosensory simulation always corresponds to the same live video stream progress during the live broadcast and playback.

In S104, the image frame is sent to the user terminal in real time through the video live broadcast platform, and the somatosensory control data packet associated with the image frame is synchronously sent to the user terminal through the video live broadcast platform, so that The user terminal synchronously outputs the somatosensory control data packet to the somatosensory sensing device while outputting the image frame to the video playback device.

In the embodiment of the present invention, for the live video stream and the somatosensory control data packet, the data is transmitted in real time. The video broadcast platform (server) is broadcasted by the video live broadcast platform to all user terminals watching the live video broadcast. Therefore, when the image frame is sent to the user terminal in real time through the video live broadcast platform, the image frame is also associated with the image frame. The somatosensory control data packet is synchronously broadcasted to all user terminals watching the live video broadcast through the same video live broadcast platform, so that when the user terminal watches the live video broadcast, the user terminal can simultaneously play the live video stream while simultaneously performing the somatosensory simulation. In order to truly restore the somatosensory control intention of the live terminal user.

Corresponding to the real-time sending method of the multimedia data described in the above embodiments, FIG. 3 is a structural block diagram of a real-time transmitting apparatus for multimedia data provided by an embodiment of the present invention. For the convenience of explanation, only the parts related to the present embodiment are shown.

Referring to Figure 3, the apparatus includes:

The monitoring unit 31 performs instruction monitoring on the live broadcast terminal during the live broadcast of the video;

The generating unit 32, when the live broadcast terminal detects that the live control terminal generates the somatosensory control command, generates a somatosensory control data packet according to the somatosensory control command, and marks the image frame currently collected by the live broadcast terminal;

The associating unit 33 associates the somatosensory control data packet with the image frame;

The sending unit 34 sends the image frame to the user terminal in real time through the video live broadcast platform, and simultaneously sends the somatosensory control data packet associated with the image frame to the user terminal through the video live broadcast platform, so that The user terminal synchronously outputs the somatosensory control data packet to the somatosensory sensing device while outputting the image frame to the video playback device.

Optionally, the generating unit 31 includes:

Obtaining a subunit, acquiring somatosensory control data of M body points, and the somatosensory control data of each body point includes control information of N types of somatosensory sensing devices;

a first writing subunit, wherein the acquired somatosensory control data of the M body points are written into the somatosensory control data packet;

The M and the N are each an integer greater than or equal to 1.

Optionally, the generating unit 31 further includes:

The second writing subunit, if the jth type of somatosensory sensing device of the i-th body point is not obtained Setting control information, in the somatosensory control data packet, each data bit corresponding to the control information is written into a preset character;

The i and the j are integers greater than or equal to 1, and the i is less than or equal to the M, and the j is less than or equal to the N.

Optionally, the preset character is 0.

Corresponding to the real-time sending method of the multimedia data described in the above embodiments, FIG. 4 is a schematic diagram of a computing node 400 according to an embodiment of the present invention. For the convenience of explanation, only the parts related to the present embodiment are shown.

The computing node 400 may be a host server that includes computing power, or a personal computer PC, or a portable computer or terminal that can be carried. The specific embodiment of the present invention does not limit the specific implementation of the computing node. Compute node 400 includes:

A processor 410, a communication interface 420, a memory 430, and a bus 440.

The processor 410, the communication interface 420, and the memory 430 complete communication with each other via the bus 440.

The communication interface 420 is configured to communicate with a network element, such as a virtual machine management center, shared storage, and the like.

The processor 410 is configured to execute a program.

In particular, the program can include program code, the program code including computer operating instructions.

The processor 410 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

The memory 430 is configured to store a program. The memory 430 may include a high speed RAM memory and may also include a non-volatile memory such as at least one disk memory.

The program may be specifically configured to perform a real-time sending method of multimedia data, where the method includes:

During the live broadcast of the video, the live broadcast terminal is instructed to listen;

When the live control terminal generates the somatosensory control command, the somatosensory control data packet is generated according to the somatosensory control command, and the image frame currently collected by the live broadcast terminal is marked;

Associating the somatosensory control data packet with the image frame;

Sending the image frame to the user terminal in real time through the video live broadcast platform, and simultaneously transmitting the somatosensory control data packet associated with the image frame to the user terminal through the video live broadcast platform, so that the user terminal While outputting the image frame to the video playback device, the somatosensory control data packet is synchronously outputted to the somatosensory sensing device.

Optionally, the generating the somatosensory control data packet according to the somatosensory control instruction comprises:

Obtaining somatosensory control data of M body points, and the somatosensory control data of each body point includes control information of N types of somatosensory sensing devices;

And acquiring the acquired somatosensory control data of the M body points into the somatosensory control data packet;

The M and the N are each an integer greater than or equal to 1.

Optionally, the generating the somatosensory control data packet according to the somatosensory control instruction further includes:

If the control information of the jth type of the somatosensory sensing device of the i-th body point is not obtained, each data bit corresponding to the control information is written into the preset character in the somatosensory control data packet;

The i and the j are integers greater than or equal to 1, and the i is less than or equal to the M, and the j is less than or equal to the N.

Optionally, the preset character is 0.

In the embodiment of the present invention, the live control terminal can generate and transmit the somatosensory control data to the user terminal in real time, so that the user terminal can experience the somatosensory simulation matching the live video content while watching the live video, greatly Improve the interactivity of online scenes.

Next, the real-time output method of the multimedia data provided by the embodiment of the present invention is described in detail, and the implementation process thereof is as shown in FIG. 5:

In S501, during the live broadcast of the video, the live broadcast platform receives the image frame sent by the live broadcast terminal.

In S502, if the image frame is associated with a somatosensory control data packet, the somatosensory control data packet that is synchronously sent by the live broadcast terminal is received by the video live broadcast platform, where the somatosensory control data packet is The live broadcast terminal is generated according to the somatosensory control instruction during the live broadcast of the video.

In S503, the image frame is output to the video playback device, and the somatosensory control data packet is synchronously outputted to the somatosensory sensing device.

In the embodiment of the present invention, the user terminal receives the live video broadcast frame from the video live broadcast platform in the manner of watching the live video broadcast in the prior art, and as described in the foregoing embodiment, if the live broadcast terminal user is at a certain moment The somatosensory control command transmits the somatosensory control data packet generated according to the sensing control command and the image frame collected at the moment to the user terminal through the video live broadcast platform, and the user terminal outputs the image frame to the video playing device. At the same time, the somatosensory control data packet is also synchronously outputted to the somatosensory sensing device. The method for generating the somatosensory control data packet, the data format, and the data content have been described in detail in the above embodiments, and are not described herein.

As an embodiment of the present invention, when each of the somatosensory control data packets includes control information of N types of body sensing devices on M body points, as shown in FIG. 6, the synchronous output station The somatosensory control data packet to the somatosensory sensing device includes:

S601. The somatosensory control data of the M body points are parsed from the somatosensory control data packet.

S602. Analyze control information of the N types of the somatosensory sensing devices from the analyzed somatosensory control data of the i-th body point.

S603. The parsed control information is respectively output to N types of somatosensory sensing devices on the i-th body point.

The M, the N, and the i are all integers greater than or equal to 1, and the i is less than or equal to M.

Based on the embodiment corresponding to FIG. 6, in the process of outputting the somatosensory control data packet by the somatosensory control device, the somatosensory control data can be accurately outputted to a specific somatosensory sensing device located at a specific body point, and then for the whole body M A total of M*N individual sensing devices on the body point can simultaneously realize the output of the somatosensory simulation in the process of playing related image frames, bringing a full-scale somatosensory experience.

In addition, as described above, a character is preset on the basis of the above-described somatosensory control packet frame format, and if a body point is not matched to a certain type of somatosensory simulation, the somatosensory control packet is Writing, each data bit occupied by the body sensing device corresponding to the body point to the preset character, for example, writing a preset character 0 to ensure the data length of all the somatosensory control data packets Consistent. Then, under the solution, the synchronously outputting the somatosensory control data packet to the somatosensory sensing device further includes:

If the control information corresponding to the jth type of the somatosensory sensing device of the i-th body point in the somatosensory control data packet is a preset character in each data bit, the somatosensory control data packet is associated with While the image frame is being played, the output of the control information is stopped;

The j is an integer greater than or equal to 1, and the j is less than or equal to N.

That is, if the somatosensory control device detects that the control information corresponding to a specific type of somatosensory sensing device on a specific body point is a preset character in each data bit occupied by the somatosensory data packet, then at the moment Stopping outputting control information to the somatosensory sensing device on the body point, so that although the data length of the somatosensory control data packet associated with each image frame is the same, the somatosensory control device can still be based on each data bit. The upper data content determines whether to output control information or stop outputting control information, thereby flexibly implementing output control of multiple body points and multiple body sensing devices.

In the embodiment of the present invention, the live control terminal can generate and transmit the somatosensory control data to the user terminal in real time, so that the user terminal can experience the somatosensory simulation matching the live broadcast content while watching the live video. , greatly improving the interactivity of online scenes.

Corresponding to the real-time output method of the multimedia data described in the above embodiments, FIG. 7 is a structural block diagram of a real-time output device for multimedia data provided by an embodiment of the present invention. For the convenience of explanation, only the parts related to the present embodiment are shown.

Referring to Figure 7, the apparatus includes:

The first receiving unit 71 receives the image frame sent by the live broadcast terminal in real time through the video live broadcast platform during the live broadcast of the video;

The second receiving unit 72, if the image frame is associated with the somatosensory control data packet, receiving, by the video live broadcast platform, the somatosensory control data packet that is synchronously sent by the live broadcast terminal, where the somatosensory control data packet is used by the live broadcast terminal Generated according to the somatosensory control instruction during the live broadcast of the video;

The output unit 73 outputs the image frame to the video playback device, and synchronously outputs the somatosensory control data packet to the somatosensory sensing device.

Optionally, the output unit 73 includes:

a first parsing subunit, parsing the somatosensory control data of the M body points from the somatosensory control data packet;

The second parsing subunit parses the control information of the N types of the somatosensory sensing devices from the parsed somatosensory control data of the i-th body point;

The first output control subunit outputs the parsed control information to the N types of somatosensory sensing devices on the i-th body point;

The M, the N, and the i are all integers greater than or equal to 1, and the i is less than or equal to M.

Optionally, the output unit 73 further includes:

a second output control subunit, if the control information corresponding to the jth type of the somatosensory sensing device of the i-th body point in the somatosensory control data packet is a preset character in each data bit, then While the somatosensory control data packet related to the image frame is played, the control information is stopped from being output;

The j is an integer greater than or equal to 1, and the j is less than or equal to N.

Optionally, the preset character is 0.

Corresponding to the real-time output method of the multimedia data described in the above embodiments, FIG. 8 is a schematic diagram of a computing node 800 according to an embodiment of the present invention. For the convenience of explanation, only the parts related to the present embodiment are shown.

The computing node 800 may be a host server that includes computing power, or a personal computer PC, or a portable computer or terminal that can be carried. The specific embodiment of the present invention does not limit the specific implementation of the computing node. Computing node 800 includes:

A processor 810, a communications interface 820, a memory 830, and a bus 840.

The processor 810, the communication interface 820, and the memory 830 complete communication with each other via the bus 840.

The communication interface 820 is configured to communicate with a network element, such as a virtual machine management center, shared storage, and the like.

The processor 810 is configured to execute a program.

In particular, the program can include program code, the program code including computer operating instructions.

The processor 810 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

The memory 830 is configured to store a program. The memory 830 may include a high speed RAM memory and may also include a non-volatile memory such as at least one disk memory.

The program may be specifically configured to execute a real-time output method of multimedia data, the method comprising:

In the process of watching the live video broadcast, the image broadcast frame is received by the live broadcast platform in real time;

If the image frame is associated with a somatosensory control data packet, the somatosensory control data packet that is synchronously sent by the live broadcast terminal is received by the video live broadcast platform, where the somatosensory control data packet is broadcasted by the live broadcast terminal. Generated according to the somatosensory control command;

The image frame is output to the video playback device, and the somatosensory control data packet is synchronously output to the somatosensory sensing device.

Optionally, the synchronously outputting the somatosensory control data packet to the somatosensory sensing device comprises:

Parsing the body sense control data of the M body points from the somatosensory control data packet;

Parsing the control information of the N types of the somatosensory sensing devices from the parsed somatosensory control data of the i-th body point;

Outputting the parsed control information to N types of somatosensory sensing devices on the i-th body point;

The M, the N, and the i are all integers greater than or equal to 1, and the i is less than or equal to M.

Optionally, the synchronizing output of the somatosensory control data packet to the somatosensory sensing device further includes:

If the somatosensory control data packet, the jth type of somatosensory sensing device pair of the i-th body point The control information is a preset character in each data bit, and the control information is stopped when the image frame related to the somatosensory control data packet is played;

The j is an integer greater than or equal to 1, and the j is less than or equal to N.

Optionally, the preset character is 0.

It will be clearly understood by those skilled in the art that, for convenience and brevity of description, only the division of each functional unit described above is exemplified. In practical applications, the above function assignment can be completed by different functional units as needed. The internal structure of the device is divided into different functional units or modules to perform all or part of the functions described above. Each functional unit in the embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the integrated unit may be implemented in the form of hardware. It can also be implemented in the form of a software functional unit. In addition, the specific names of the respective functional units are only for the purpose of facilitating mutual differentiation, and are not intended to limit the scope of protection of the present application. For the specific working process of the unit in the foregoing device, reference may be made to the corresponding process in the foregoing device embodiment, and details are not described herein again.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can use different means for implementing the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present invention.

In the embodiments provided by the present invention, it should be understood that the disclosed apparatus and apparatus may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in electrical, mechanical or other form.

The units described as separate components may or may not be physically separated as a unit The displayed components may or may not be physical units, ie may be located in one place, or may be distributed over multiple network elements. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

An integrated unit, if implemented in the form of a software functional unit and sold or used as a standalone product, can be stored in a computer readable storage medium. Based on such understanding, the technical solution of the embodiments of the present invention may contribute to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage. The medium includes a number of instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of various embodiments of the embodiments of the present invention. The foregoing storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read only memory, a random access memory, a magnetic disk, or an optical disk.

The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to be limiting; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that The technical solutions are described in the following, and the technical features of the embodiments of the present invention are not deviated from the spirit and scope of the technical solutions of the embodiments of the present invention.

The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalents, and improvements made within the spirit and scope of the present invention should be included in the scope of the present invention. Inside.

Claims (16)

1. A method for real-time transmission of multimedia data, characterized in that the method comprises:

   During the live broadcast of the video, the live broadcast terminal is instructed to listen;

   When the live control terminal generates the somatosensory control command, the somatosensory control data packet is generated according to the somatosensory control command, and the image frame currently collected by the live broadcast terminal is marked;

   Associating the somatosensory control data packet with the image frame;

   Sending the image frame to the user terminal in real time through the video live broadcast platform, and simultaneously transmitting the somatosensory control data packet associated with the image frame to the user terminal through the video live broadcast platform, so that the user terminal While outputting the image frame to the video playback device, the somatosensory control data packet is synchronously outputted to the somatosensory sensing device.
2. The method according to claim 1, wherein the generating the somatosensory control data packet according to the somatosensory control command comprises:

   Obtaining somatosensory control data of M body points, and the somatosensory control data of each body point includes control information of N types of somatosensory sensing devices;

   And acquiring the acquired somatosensory control data of the M body points into the somatosensory control data packet;

   The M and the N are each an integer greater than or equal to 1.
3. The method of claim 2, wherein the generating the somatosensory control data packet according to the somatosensory control command further comprises:

   If the control information of the jth type of the somatosensory sensing device of the i-th body point is not obtained, each data bit corresponding to the control information is written into the preset character in the somatosensory control data packet;

   The i and the j are integers greater than or equal to 1, and the i is less than or equal to the M, and the j is less than or equal to the N.
4. The method of claim 3 wherein said predetermined character is zero.
5. A real-time output method for multimedia data, characterized in that the method comprises:

   In the process of watching the live video broadcast, the image broadcast frame is received by the live broadcast platform in real time;

   If the image frame is associated with a somatosensory control data packet, the somatosensory control data packet that is synchronously sent by the live broadcast terminal is received by the video live broadcast platform, where the somatosensory control data packet is broadcasted by the live broadcast terminal. Generated according to the somatosensory control command;

   The image frame is output to the video playback device, and the somatosensory control data packet is synchronously output to the somatosensory sensing device.
6. The method of claim 5, wherein the synchronously outputting the somatosensory control data packet to the somatosensory sensing device comprises:

   Parsing the body sense control data of the M body points from the somatosensory control data packet;

   Parsing the control information of the N types of the somatosensory sensing devices from the parsed somatosensory control data of the i-th body point;

   Outputting the parsed control information to N types of somatosensory sensing devices on the i-th body point;

   The M, the N, and the i are all integers greater than or equal to 1, and the i is less than or equal to M.
7. The method of claim 6, wherein the synchronously outputting the somatosensory control data packet to the somatosensory sensing device further comprises:

   If the control information corresponding to the jth type of the somatosensory sensing device of the i-th body point in the somatosensory control data packet is a preset character in each data bit, the somatosensory control data packet is associated with While the image frame is being played, the output of the control information is stopped;

   The j is an integer greater than or equal to 1, and the j is less than or equal to N.
8. The method of claim 7 wherein said predetermined character is zero.
9. A real-time transmitting device for multimedia data, characterized in that the device comprises:

   The monitoring unit is configured to perform instruction monitoring on the live terminal during the live broadcast of the video;

   a generating unit, configured to: when listening to the live broadcast terminal generating a somatosensory control instruction, according to the The somatosensory control command generates a somatosensory control data packet, and marks an image frame currently collected by the live broadcast terminal;

   An association unit, configured to associate the somatosensory control data packet with the image frame;

   a sending unit, configured to send the image frame to the user terminal in real time through a video live broadcast platform, and simultaneously send the somatosensory control data packet associated with the image frame to the user terminal by using the video live broadcast platform, And causing the user terminal to synchronously output the somatosensory control data packet to the somatosensory sensing device while outputting the image frame to the video playing device.
10. The apparatus according to claim 9, wherein said generating unit comprises:

    Obtaining a subunit for acquiring somatosensory control data of M body points, and the somatosensory control data of each body point includes control information of N types of somatosensory sensing devices;

    a first writing subunit, configured to write the acquired somatosensory control data of the M body points into the somatosensory control data packet;

    The M and the N are each an integer greater than or equal to 1.
11. The apparatus according to claim 10, wherein said generating unit further comprises:

    a second writing subunit for controlling information of the jth type of somatosensory sensing device that does not acquire the i-th body point, each data bit corresponding to the control information in the somatosensory control data packet Write preset characters;

    The i and the j are integers greater than or equal to 1, and the i is less than or equal to the M, and the j is less than or equal to the N.
12. The apparatus of claim 11 wherein said predetermined character is zero.
13. A real-time output device for multimedia data, characterized in that the device comprises:

    a first receiving unit, configured to receive, by the video live broadcast platform, an image frame sent by the live broadcast terminal in real time during the live broadcast of the video;

    a second receiving unit, configured to: when the image frame is associated with a somatosensory control data packet, receive, by the video live broadcast platform, the somatosensory control data packet that is synchronously sent by the live broadcast terminal, where the somatosensory control data packet is sent by the live broadcast The terminal is generated according to the somatosensory control instruction during the live broadcast of the video;

    An output unit, configured to output the image frame to a video playback device, and synchronously output the body sense control Data packets to the somatosensory sensing device.
14. The apparatus of claim 13 wherein said output unit comprises:

    a first parsing subunit, configured to parse the body sense control data of the M body points from the somatosensory control data packet;

    a second parsing subunit, configured to parse control information of the N types of the somatosensory sensing devices from the parsed somatosensory control data of the i-th body point;

    a first output control subunit, configured to output the parsed control information to the N types of somatosensory sensing devices on the i-th body point;

    The M, the N, and the i are all integers greater than or equal to 1, and the i is less than or equal to M.
15. The device of claim 14, wherein the output unit further comprises:

    a second output control subunit, configured to: in the somatosensory control data packet, the control information corresponding to the jth type of the somatosensory sensing device of the i-th body point is a preset character in each data bit, And stopping outputting the control information while playing the image frame related to the somatosensory control data packet;

    The j is an integer greater than or equal to 1, and the j is less than or equal to N.
16. The apparatus of claim 14 wherein said predetermined character is zero.

Images