CN112671696B

CN112671696B - Message transmission method, device, computer equipment and computer storage medium

Info

Publication number: CN112671696B
Application number: CN201910985127.6A
Authority: CN
Inventors: 朱启航
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2019-10-16
Filing date: 2019-10-16
Publication date: 2023-05-12
Anticipated expiration: 2039-10-16
Also published as: CN112671696A

Abstract

The application discloses a message transmission method, a message transmission device, computer equipment and a computer storage medium, and belongs to the technical field of video data transmission. According to the method, the obtained time stamp of the message is modified according to the time stamp change rule of the message of the video to be played, so that the use of gateway equipment for message conversion can be avoided, namely, the gateway equipment is not required to be used in the message transmission process, and the resource consumption and the cost in the message transmission are reduced.

Description

Message transmission method, device, computer equipment and computer storage medium

Technical Field

The present invention relates to the field of video data transmission technologies, and in particular, to a method and apparatus for transmitting a message, a computer device, and a computer storage medium.

Background

With the development of multimedia technology, the requirements of users on video quality are increasing, but in the process of transmitting data streams through internet protocol (internet protocol, IP) networks, the video streams, the audio streams and the auxiliary data in the data streams may come from different devices of different producers, and these different devices may send the video streams, the audio streams or the auxiliary data in the form of real-time protocol (real time protocol, RTP) messages.

The current message transmission process may be the following process: the method comprises the steps that a first gateway device receives RTP messages which are sent by a plurality of devices through an IP network and carry time stamps, the received RTP messages are converted into serial digital interface (serial digital interface, SDI) messages which do not carry time stamps, the SDI messages which do not carry time stamps are sent to a second gateway device, the second gateway device converts the received SDI messages into RTP messages which can be sent in the IP network, time stamps are added to the converted RTP messages according to local time, the RTP messages with the added time stamps are sent to an IP matrix, the IP matrix schedules and switches video streams, audio streams and auxiliary data which are formed by the RTP messages and sends the RTP messages to a terminal, and the terminal plays videos based on the time stamps in the received RTP messages.

In the above message transmission process, the first gateway device converts the RTP messages sent by different devices into the SDI messages, then the second gateway device converts the converted SDI messages back into RTP messages, and then modifies the time stamp of the RTP messages, that is, the RTP messages sent by different devices need to be converted twice, in order to complete the two conversions, two additional gateway devices are needed to be added in the path of message transmission, if the gateway device is required to meet the requirement of performing the two conversions, the functional requirement on the gateway device is higher, thereby increasing the resource consumption and the cost in the message transmission process.

Disclosure of Invention

The embodiment of the application provides a message transmission method, a message transmission device, computer equipment and a computer storage medium, which can reduce resource consumption and cost in the message transmission process. The technical scheme is as follows:

in a first aspect, a method for transmitting a message is provided, where the method includes:

acquiring a plurality of messages in a video stream of a video to be played;

modifying the time stamps of the messages according to the time stamp change rule of the message of the video to be played to obtain a modified message, wherein the time stamp of the modified message accords with the time stamp change rule;

and sending the modified message to terminal equipment.

According to the method, the obtained time stamp of the message is modified according to the time stamp change rule of the message of the video to be played, so that the use of gateway equipment for message conversion can be avoided, namely, the gateway equipment is not required to be used in the message transmission process, and the resource consumption and the cost in the message transmission are reduced.

In one possible implementation manner, before modifying the time stamps of the plurality of messages according to the time stamp change rule of the message of the video to be played, the method further includes:

And determining the change rule of the time stamp according to the time stamp of the message for packaging the first target video frame in the video stream.

In one possible implementation, the first target video frame is a first video frame of the video to be played.

In one possible implementation manner, the determining the timestamp change rule according to the timestamp of the packet for encapsulating the first target video frame in the video stream includes:

when the time indicated by the time stamp of the message of the first target video frame is sequentially increased, determining the time stamp change rule as a first time stamp change rule, wherein the first time stamp change rule is used for indicating that the time stamp of the message of each video frame of the video to be played is sequentially increased;

when the time stamps of the messages of the first target video frames are the same, determining the time stamp change rule as a second time stamp change rule, wherein the second time stamp change rule is used for indicating that the time stamps of the messages of each video frame are the same;

when the time stamp of the message of the first target video frame comprises two time stamps, determining the time stamp change rule as a third time stamp change rule, wherein the third time stamp change rule is used for indicating that the time stamp of the message of each video frame is divided according to fields, and the time stamps of the messages belonging to the same field are the same.

In one possible implementation manner, the modifying the time stamps of the plurality of messages according to the time stamp change rule of the message of the video to be played, and obtaining the modified message includes:

modifying the time stamp of a first target message in the plurality of messages to be a target time stamp;

and modifying messages belonging to the same video frame as the first target message in the messages according to the change rule of the time stamp and the target time stamp.

Based on the possible implementation manner, because the time stamps of the modified messages are continuous in time, when the first device sends the messages which are continuous in time to the terminal device, the terminal device does not have short black screen when playing the video based on the time stamps of the messages, so that the experience of the user can be improved.

In one possible implementation manner, the modifying the timestamp of the first target message in the plurality of messages to the target timestamp includes:

acquiring the target time stamp according to a local clock;

and replacing the timestamp of the first target message with the target timestamp.

In one possible implementation manner, before the obtaining the target timestamp according to the local clock, the method further includes:

For any video frame of the video to be played, determining a first received message in the messages belonging to the video frame as a first target message of the video frame; or alternatively, the first and second heat exchangers may be,

and determining the first received message as a first target message of the video frame after the message of the last video frame for packaging the video frame is received.

and modifying the time stamp of a first target message in the plurality of messages into a target time stamp according to a target time length and a time stamp of a message with the minimum message sequence number for packaging a second target video frame, wherein the target time length is the time length of one video frame of the video to be played, and the second target video frame is the last video frame of the video frames to which the first target message belongs.

In one possible implementation manner, before the modifying the timestamp of the first target message in the plurality of messages to the target timestamp according to the target duration and the timestamp of the message with the smallest sequence number in the last video frame used for encapsulating the video frame, the method further includes:

And for any video frame of the video to be played, determining a message with the minimum sequence number in the messages belonging to the video frame as a first target message of the video frame.

In one possible implementation manner, after the determining, as the first target packet of the video frame, the packet with the smallest sequence number in the packets belonging to the video frame, the method further includes:

determining whether the transmission of the messages of the video frames is disordered according to the sequence number of the first target message and the sequence number of a second target message, wherein the second target message is a message with the largest sequence number in the message of the last video frame for packaging the video frames.

In a second aspect, a packet transmission device is provided, configured to perform the above packet transmission method. Specifically, the message transmission device includes a functional module configured to execute the message transmission method provided in the first aspect or any one of the optional manners of the first aspect.

In a third aspect, a computer device is provided, the computer device comprising a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to implement operations performed by a method of transmitting messages as described above.

In a fourth aspect, a computer storage medium having stored therein at least one instruction that is loaded and executed by a processor to perform operations performed by the above-described message transmission is provided.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of a message transmission system according to an embodiment of the present application;

fig. 2 is a schematic diagram of an IP signal transmission process according to an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a computer device according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of a message header according to an embodiment of the present application;

fig. 5 is a flowchart of a message transmission method provided in an embodiment of the present application;

fig. 6 is a flowchart of a message transmission method provided in an embodiment of the present application;

fig. 7 is a flowchart of a message transmission method provided in an embodiment of the present application;

Fig. 8 is a schematic structural diagram of a message transmission device according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

Fig. 1 is a schematic diagram of a message transmission system provided in an embodiment of the present application, referring to fig. 1, the system includes one or more terminal devices 101, one or more first devices 102, and one or more second devices 103, where the terminal device 101 is any device installed with a client, and the client may provide a video playing service, and the any device may be a mobile phone, a notebook computer, or the like.

The first device 102 is configured to modify a timestamp of a message, where the message is used to carry data of a video to be played, the first device 102 is further configured to send the modified message to the terminal device 101, and the terminal device 101 plays the video to be played based on the received message, where the message may be an RTP message, and the message may also be any other message carrying a timestamp, and the first device may be a video matrix, where the video matrix refers to an electronic device that outputs K paths of video signals to H paths of terminal devices at will by an array switching method, where K and H are integers greater than 0, for example, an IP matrix.

The second device 103 is configured to generate a data stream of the video to be played, and send the data stream to the first device 102, where the first device modifies a timestamp of a message that forms the data stream. The data stream may comprise a video stream, an audio stream or auxiliary data and the second device may be any device that generates a video stream, an audio stream or auxiliary data.

For further explanation of the process of transmitting a message in a video transmission system, refer to a schematic diagram of an IP signal transmission process provided in the embodiment of the present application shown in fig. 2, where an IP network device (IP matrix) in fig. 2 is a first device, that is, an IP signal 1 is an RTP message sent by a second device through an IP network, after the IP network device receives the IP signal 1, a local time is determined according to a global positioning system (global positioning system, GPS), and then a timestamp of the IP signal 1 is modified according to the local time to obtain an IP signal 2, and finally, the IP network device outputs the IP signal 2.

The terminal device, the first device and the second device may be computer devices, and in order to further illustrate the hardware structure of the computer devices, referring to the schematic structural diagram of one computer device provided in the embodiment of the present application shown in fig. 3, the computer device 300 includes a relatively large difference that may be generated due to different configurations or performances, and may include one or more processors (central processing units, CPU) 301 and one or more memories 302, where at least one instruction is stored in the memory 302, and the at least one instruction is loaded and executed by the processor 301 to implement the methods provided in the embodiments of the methods described below. Of course, the computer device 300 may also have a wired or wireless network interface, a keyboard, an input/output interface, and other components for implementing the functions of the device, which are not described herein.

In an exemplary embodiment, a computer readable storage medium, such as a memory comprising instructions executable by a processor in a terminal to perform the method of message transmission in the embodiments described below, is also provided. For example, the computer readable storage medium may be a read-only memory (ROM), a random access memory (random access memory, RAM), a compact disc-read only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

In this embodiment of the present application, a transmission process of a message is described by taking an RTP message forming a data stream as an example, and because in the process of transmitting a message, a first device modifies a timestamp of a message sent by a second device, so that a process of modifying the timestamp by the first device is easy to understand, the following description is performed on the RTP message:

an RTP packet consists of two parts, header (header) and payload (payload), where the meaning of the first 12 bytes of the header is fixed and the payload can be audio data or video data. Referring to the schematic structure of a message header provided in the embodiment of the present application shown in fig. 4, an RTP message header may include a version number (version, V), padding (P), extension (X), CSRC Counter (CC), marker bit (M), payload Type (PT), sequence number (payload type, SN), time stamp (time stamp), synchronization source (synchronized source, SSRC) identifier, and special source (CSRC) identifier.

Wherein V: for identifying the version of the RTP protocol used.

P: if the bit is set (p=1), the tail of the RTP packet contains additional stuff bytes.

X: if the bit is set (x=1), an extension header is added after the fixed header of the RTP packet.

CC: for indicating the number of CSRC identifiers in the fixed header.

M: the interpretation of this bit is borne by a profile, which may have different meanings for different loads, and marks the end of a video frame for that frame; for audio, the start of the session is marked.

PT: for identifying the type of load, such as audio or image, etc.

SN: after each time an RTP message is sent, the sender increases the value of the domain by 1, and the receiver can detect the loss condition of the message by the domain, reorder the message and recover the data.

Timestamp: is the sampling instant of the first byte of data in the RTP packet. At the beginning of a session, the timestamp is initialized to an initial value. The value of the timestamp increases continuously over time even when no signal is sent. The receiver can calculate the delay or delay jitter according to the time stamp of the received message and perform synchronous control.

SSRC identifier: indicating the source of the RTP message.

CSRC identifier: for indicating all of the specific sources that contain the payload in the RTP message.

For further explanation of the specific process of modifying the message by the first device, refer to fig. 5, which is a flowchart of a message transmission method provided in the embodiment of the present application, where the flowchart of the method specifically includes the following processes shown in steps 501-507.

501. The first device obtains a plurality of messages in a video stream of a video to be played.

The video to be played is a video to be played by any terminal device, where the video to be played includes a plurality of video frames, each video frame is carried by a plurality of messages, the video frames are each still pictures that form a continuous video, and because data that form each video frame is more, each video frame may be carried by a plurality of messages, the plurality of messages acquired by the first device may be messages for encapsulating one video frame, and the one video frame may be any video frame of the video to be played, for example, an nth video frame, where n is a positive integer greater than 0.

It should be noted that, the first video frame and the second video frame of the video to be played, that is, the first two video frames of the video to be played. When the flag bit m=1 in the message indicates that the message is the last message of the video frame where the message is located, when the first device starts to receive the video stream, the received first message may be regarded as the first message of the first video frame, the first device continues to receive the next message, when the flag bit m=1 of the newly received message is regarded as the last message of the first video frame, and then the first device may take the message received from the time of receiving the first message of the first video frame to the time of receiving the latest message (m=1) as the message of the first video frame, and the process of determining the second other video frames by the first device is similar to the process of determining the first video frame.

When the terminal equipment receives a playing instruction of the video to be played, the terminal equipment sends an instruction of requesting the video stream of the video to be played to the server, when the server receives the instruction of requesting the video stream of the video to be played, the server issues an instruction of transmitting the video stream of the video to be played to one or more second equipment, when any one of the second equipment receives the instruction issued by the server, the video stream of the video to be played is sent to the first equipment, namely, a message for packaging each video frame is sent to the first equipment, and the continuous messages can form the video stream, so that the first equipment acquires a plurality of messages.

It should be noted that, the second device may send the message of each video frame of the video to be played to the same receiving port of the first device, when any receiving port receives any message, if the source address or destination address carried by any message is different from the source address or destination address carried by the message received before, it may be stated that any message and the message received before do not belong to the same video stream, so that the first device may determine when to start receiving a new video stream, so that the first device may modify the timestamp of the message in the new video stream according to the timestamp change rule of the new video to be played, and may avoid using the timestamp change rule of the last video to be played (corresponding to the previous video stream) to modify the time of the message in the new video stream. The source address is the address of the device generating any message, and the target address is the address of the device finally receiving any message.

502. The first device determines the change rule of the time stamp of the message of the video to be played according to the time stamp of the message for packaging the first target video frame in the video stream.

The first target video frame may be a first video frame of the video to be played, or may be a second video frame of the video to be played, or the first target video frame is any one of a preset number of video frames of the video to be played, which is not specifically limited in this embodiment of the present application. The time stamp change rules of the messages belonging to the same video frame are consistent, wherein the time stamp change rules can comprise any one of a first time stamp change rule, a second time stamp change rule and a third time stamp change rule. The first timestamp change rule is used to indicate that the timestamp of the message of each video frame of the video to be played is sequentially increased, and may be a timestamp change rule specified by the us movie and television engineering (society of motion picture and television engineers, SMPTE) 2022-6 standard. The second time stamp change rule is used to indicate that the time stamp of the message of each video frame is the same, and may be a time stamp change rule specified for the case of progressive video under SMPTE ST 2110 standard. The third timestamp change rule is used for indicating that the timestamp of the message in each video frame is divided according to fields, and may be a timestamp change rule specified for the case of interlaced video under SMPTE ST 2110 standard. This step 502 may be implemented by the process shown in steps 21-23 below.

Step 21, when the time indicated by the time stamp of the message of the first target video frame is sequentially increased, the first device determines the time stamp change rule as a first time stamp change rule, where the first time stamp change rule is used to indicate that the time stamp of the message of each video frame of the video to be played is sequentially increased.

The message of the first target video frame, that is, the message for encapsulating the first target video frame, it should be noted that, in this application, which message of the first video frame is the message for encapsulating which video frame, after the first device obtains the message of the first target video frame, the time stamp of each message of the first target video frame may be compared, if the time stamp of each message indicates that the indication time sequentially increases, the time stamp change rule is determined as the first time stamp change rule, that is, the message of each video frame of the video to be played uses the first time stamp change rule, where the time indicated by the time stamp may also be regarded as the value of the time stamp.

Step 22, when the time stamps of the messages of the first target video frame are the same, the first device determines the time stamp change rule as a second time stamp change rule, where the second time stamp change rule is used to indicate that the time stamps of the messages of each video frame are the same.

After the first device obtains the messages of the first target video frame, the time stamp of each message of the first target video frame can be compared, if the indication time indicated by the time stamp of each message is the same, the time stamp change rule is determined to be a second time stamp change rule, that is, the second time stamp change rule is used for the message of each video frame of the video to be played.

In some embodiments, the first device parses the packets of the first target video frame to obtain the payload header of each packet, where each payload header includes a field identification bit (field identification, F), and if values of field identification bits F in the payload header of each packet of the first target video frame are all consistent, for example, values of F are all 0, it indicates that time stamps of packets of the first target video frame are all consistent, and the first device may directly determine a time stamp change rule of the video to be played as a second time stamp change rule.

Step 23, when the time stamp of the message of the first target video frame includes two time stamps, the first device determines the time stamp change rule as a third time stamp change rule, where the third time stamp change rule is used to indicate that the time stamp of the message of each video frame is divided according to the fields, and the time stamps of the messages belonging to the same field are the same.

In some embodiments, the messages of each video frame may be divided into two fields, namely, a 0 th field and a 1 st field, where the time stamps of the messages in the 0 th field of each video frame are identical, the messages in the 0 th field of each video frame include a pre-set number of messages in the front of each video frame, the time stamps of the messages in the 1 st field of each video frame are identical, the messages in the 1 st field of each video frame include messages of each video frame except for the pre-set number of messages, and the time stamps in the 0 th field and the 1 st field are different, that is, each video frame includes two time stamps.

Therefore, after the first device obtains the messages of the first target video frame, the time stamp of each message of the first target video frame can be compared, and if the time stamp of each message includes two time stamps, the time stamp change rule is determined to be a third time stamp change rule, that is, the message of each video frame of the video to be played uses the third time stamp change rule.

In some embodiments, the first device analyzes the packet of the first target video frame to obtain a load header of each packet, if values of field marking bits F in the load header of each packet of the first target video frame are inconsistent, it indicates that time stamps of the packet of the first target video frame are the same according to field division, including two time stamps, and the first device may directly determine a time stamp change rule of the video to be played as a third time stamp change rule, where when the field marking bit f=0 of the load header of the packet, the packet is a packet of a 0 th field of the video frame, and when the field marking bit f=1 of the load header of the packet is a packet of a 1 st field of the video frame, time stamps of the packets in the 0 th field are the same, and time stamps of the packets in the 1 st field are different.

It should be noted that, since the subsequent first device may also modify the timestamp of the packet of each video frame, when the timestamp is modified, the timestamp change rule and the time length of each video frame, that is, the target time length may be used, so the first device may acquire the timestamp change rule in advance according to the first target video frame, or may acquire the target time length in advance according to the first video frame and the second video frame, so as to avoid repeated acquisition, and the first device may acquire the target time length according to the timestamp of the last packet of the first video frame and the timestamp of the last packet of the second video frame, where the target time length is the difference between the timestamp of the last packet of the first video frame and the timestamp of the last packet of the second video frame, and the target time length is the time length of one video frame, that is, and the time lengths of each video frame of the same video frame. Wherein the difference between the time stamps is the difference between the times indicated by the time stamps.

Specifically, the target duration may be denoted by Δts, where Δts=ts_tail (2) -ts_tail (1), where ts_tail (2) is the time indicated by the timestamp of the last packet of the second video frame and ts_tail (1) is the time indicated by the timestamp of the last packet of the first video frame.

In addition, the first device may acquire the number of messages of each video frame in advance, and record the number as the target number of messages. In one possible implementation, the first device obtains a target number of messages according to a sequence number of a last message of the first video frame and a sequence number of a last message of the second video frame, where the target number of messages is a difference between the sequence number of the last message of the first video frame and the sequence number of the last message of the second video frame.

The target number of messages is the number of messages in each video frame. The target number of messages may be represented by formula (1), where P is the target number of messages, sn_tail (2) is the sequence number of the last message of the second video frame, and sn_tail (1) is the sequence number of the last message of the first video frame.

P＝Sn_tail(2)- Sn_tail(1) (1)

Further, after determining the sequence number of the first message and the target duration of each video frame, the first device may determine the range of the sequence number of the message of each video frame according to the target message number and the sequence number of the last message of the second video frame. In one possible implementation, the sequence number range of the message of the nth video frame may be [ sn_tail (2) +1+ (n-3) ×p, sn_tail (2) + (n-2) ×p ], where n is a positive integer greater than or equal to 3. When the sequence number of any message is within the range of [ Sn_Tail (2) +1+ (n-3) P, sn_Tail (2) + (n-2) P ], the any message is the message of the nth video frame, and if the difference value between the sequence number of any message and Sn_tail (2) +1+ (n-3) P is Q, the any message is the Q-th message of the nth video frame.

It should be noted that, in order to obtain the parameters required for modifying the timestamp in the process shown in steps 501 and 502, the parameters may include a timestamp change rule, a target number of packets, a target duration, and a sequence number range of the packets of each video frame. These necessary parameters may be acquired once, and multiple acquisitions are not required, that is, the process shown in steps 501-502 may be performed only when the first device starts to receive the video frame, and the process shown in steps 503-507 may be directly performed when the message of the other video frame after the second video frame is received.

In some embodiments, the parameter may also be preset without being acquired from the first video frame and the second video frame, and when the parameter may also be a preset parameter, the first device may not perform this step 502.

503. For any video frame of the video to be played, the first device determines a first received message in the messages belonging to the video frame as a first target message of the video frame.

Any video frame may be an nth video frame of the video to be played, where n is a positive integer greater than 0. The first device directly uses a first received message, which belongs to the nth video frame and is compared with other messages of the nth video frame, as a first target message, namely a first received message in the plurality of messages, wherein the first target message may be a first message of the nth video frame or any message of the nth video frame except the first message. That is, the first message received by the first device in the message sequence number range of the nth video frame is used as the first target message of the nth video frame.

504. For a first target message of the plurality of messages, the first device obtains a target timestamp according to a local clock.

The local clock is used to indicate a current time local to the first device. The target time stamp is a time stamp after the first target message is modified, and the first device can obtain the target time stamp according to epoch time epoch and a local clock, wherein the epoch time epoch is 1970, 01, 0 minutes and 0 seconds. In one possible implementation, this step 504 may be implemented by the process shown in steps 41-43 below.

Step 41, the first device obtains a first duration from the epoch time epoch to the current time.

The first duration is the time difference between the local time indicated by the local clock and the epoch time epoch.

Step 42, the first device obtains a first time according to the first duration, where the first time is the time of the RTP clock domain corresponding to the first duration.

Specifically, the first time may be used for currenttime_rtp, where currenttime_rtp=currenttime×fm, and FM is the clock frequency.

Step 43, the first device obtains the target timestamp according to the first time.

The first device may modulo the first time and convert to a 32-bit integer to obtain the target timestamp. Specifically, the target timestamp may be represented as Ts0, where ts0=currenttime_rtp mod 2 ζ.

505. The first device replaces the timestamp of the first target message with the target timestamp.

The process shown in step 505 is a process of modifying the first target message. It should be noted that the process shown in steps 504-505 is a process in which the first device modifies the timestamp of the first target message in the plurality of messages to be the target timestamp.

506. The first device modifies messages belonging to the same video frame as the first target message in the plurality of messages according to the time stamp change rule and the target time stamp.

For messages in the video frame other than the first target message, the first device modifies the timestamp in a different manner according to the change rule of the timestamp, and in a possible implementation, this step 506 may be implemented by the procedure shown in steps 61-63.

Step 61, for any message except the first target message in the video, when the time stamp change rule is used for indicating that the time stamp of the message of each video frame of the video to be played is sequentially increased, the first device modifies the time stamp of any message according to the target time stamp, the target duration, the target message number and the sequence number of any message, wherein the target message number is the message number of each video frame.

Because the time stamp change rule is used for indicating that the time stamp of the message of each video frame is sequentially increased, that is, the time stamp change rule is the first time stamp change rule, in order to achieve that the time stamps in the video frames are sequentially increased, the first device can obtain the first target time stamp according to the target time stamp, the target time length, the target message number and the serial number of any message, and replace the time stamp of any message with the first target time stamp, thereby achieving that the time stamp of any message is modified.

When the video frame is the nth video frame of the video to be played, the first target timestamp may be represented by the following formula (2), where Ts is the first target timestamp (i.e. any one of the modified timestamps), ts_1 (n) is the target timestamp, sn is the sequence number of the any one of the messages, and sn_tail is the sequence number of the last message of the last video frame of the video frame.

Ts ＝Ts_1(n)+⊿Ts/P*(Sn-Sn_tail-1) (2)

Step 62, when the time stamp change rule is used to indicate that the time stamps of the messages of each video frame are the same, the first device modifies the time stamp of any message to the target time stamp.

Because the time stamp change rule is used for indicating that the time stamps of the messages of each video frame are the same, that is, the time stamp change rule is the second time stamp change rule, the first device can directly modify the time stamp of any message into the target time stamp.

Step 63, when the time stamp change rule is used to indicate that the time stamps of the messages of each video frame are divided according to the fields and the time stamps of the messages belonging to the same field are the same, if the time stamp of any message is the same as the original time stamp of the first target message, the first device modifies the time stamp of any message into the target time stamp, otherwise, modifies the time stamp of any message according to the target time stamp and the target time length.

At this time, the time stamp change rule is a third time stamp change rule, which indicates that the message of the video frame is divided into 0 field and 1 field, if the time stamp of any message is the same as the original time stamp of the first target message, it indicates that any message is a message in the same field as the first target message, the first device can modify the time stamp of any message into the target time stamp, otherwise, it indicates that any message is not a message in the same field as the first target message, and the first device modifies the time stamp of any message according to the target time stamp and the target time length.

The process of modifying the timestamp of any message according to the target timestamp and the target duration may be that the first device obtains a second target timestamp according to the target timestamp and the target duration; and replacing the time stamp of any message with the second target time stamp. The second target timestamp may be represented by ts_f2, where ts_f2=ts_1 (n) +1/2×Δts.

It should be noted that, when the first device receives the first target message and then receives the third target message, the first device determines whether the third target message is a message of the video frame according to the sequence number of the third target message and the sequence number range of the message of the video frame, and when the sequence number of the third target message is within the sequence number range of the message of the video frame, the first device executes step 506. When the sequence number of the third target message is equal to the maximum value of the sequence number range of the video, it is indicated that the third target message is the last message of the video frame, and if the first device receives a message again, the first device executes step 504 based on the newly received message, which is the first target message of the next video of the video frame.

It should be noted that the process shown in steps 504-506 is a process of modifying the time stamps of the plurality of messages according to the time stamp change rule of the messages of the video to be played. The process shown in steps 503-506 is performed for each video frame of the video to be played, so that the first device can modify the time stamp of each message of each video frame, and since each message modification is based on the time stamps of the first two video frames and the local time, the modified each message in each video frame of the video to be played is continuous in time, and each video frame is continuous in time.

507. The first device sends the modified message to the terminal device.

It should be noted that, each time the first terminal modifies the timestamp of a message, this step 507 may be executed. Because the time stamps of the modified messages are continuous in time, when the first device sends the messages which are continuous in time to the terminal device, the terminal device does not have transient black screen when playing the video based on the time stamps of the messages, and therefore the experience of the user can be improved.

According to the method provided by the embodiment of the application, the obtained time stamp of the message is modified according to the time stamp change rule of the message of the video to be played, so that the use of gateway equipment for message conversion and time stamp modification can be avoided, namely, the gateway equipment is not required in the message transmission process, and the resource consumption and cost in message transmission are reduced.

In the embodiment shown in fig. 5, the first device divides the range of each video frame sequence number directly according to the sequence number of the message, and uses the first message received by the first device in each range as the first target message of the corresponding video frame, so that other messages of each video frame can be modified according to the first target message of each video frame, that is, in the process shown in fig. 5, when the first device modifies the timestamp, the first device bases on the first message received by the first device in each range.

Considering that some messages with large serial numbers may be received by the first device first, and messages with small serial numbers may be received by the first device later, in order to ensure that the time stamp of the message with larger serial numbers is larger, the first device may find a reference message for each video frame first, determine a first target message based on the reference message, and finally modify the time stamp according to the determined first target message. To illustrate this process, reference is made to a flowchart of a message transmission method provided in the embodiment of the present application shown in fig. 6. The method flow includes the process shown in steps 601-605 below.

601. The first device obtains a plurality of messages in a video stream of a video to be played.

The implementation process of this step 601 is the same as the implementation process of step 501, and here, this embodiment of the present application will not be described in detail for this step 601.

602. The first device determines the change rule of the time stamp of the message of the video to be played according to the time stamp of the message for packaging the first target video frame in the video stream.

The implementation process of this step 602 is the same as the implementation process of step 502, and here, the embodiment of the present application will not be described in detail for this step 602.

It should be noted that, in order to determine whether all the messages in the first target video frame are received, before executing step 602, the sequence number of the first message of each video frame may be determined first, where the first message is a frame header message, so that the first device may determine, according to the sequence number of the first message of the first target video frame and the sequence number of the first message of the next video frame of the first target video frame, whether the messages of the first target video frame are received completely, for example, the sequence number of the first message of the first video frame is xxx, the sequence number of the first message of the second video frame is kkk, and after receiving the sequence number of the first message is xxx, the first device continuously receives the messages until when receiving the message with the sequence number of kk, all the messages of the first video frame are received completely.

The process of determining the sequence number of the first message of each video frame by the first device may be: the first device determines the sequence number of the first message of each video frame according to the sequence number of the last message of the first video frame and the sequence number of the last message of the second video frame, wherein the last message is the frame tail message.

In one possible implementation, the process of determining the sequence number of the first packet of each video frame by the first device according to the sequence number of the last packet of the first video frame and the sequence number of the last packet of the second video frame may be the process shown in the following steps 2A-2B.

Step 2A, the first device obtains a target message number according to the sequence number of the last message of the first video frame and the sequence number of the last message of the second video frame, where the target message number is the message number of each video frame.

The process of the first device obtaining the target duration is described in the foregoing and is not described herein.

Step 2B, the first device determines a third video frame and the sequence number of the first message of each video frame after the third video frame according to the sequence number of the last message of the second video frame.

Specifically, this step 2B may be implemented by the following formula (3), where n is the number of a video frame, for indicating the nth video frame of the video to be played, and n is a positive integer greater than or equal to 3; sn_head (n) is the sequence number of the first message of the nth video frame.

Sn_head(n)＝Sn_tail(2)+1+(n-3)*P (3)

In addition, 1 is added to the sequence number of the last message of the first video frame, that is, the sequence number of the first message of the second video frame. After determining the sequence number and the target duration of the first message of each video frame, the first device may determine the range of the sequence number of the message of each video frame according to the sequence number and the target duration of the first message of each video frame. In one possible implementation, the sequence number range of the message of the nth video frame may be [ sn_head (n), sn_head (n) +p ], where when the sequence number of any message is within the range of [ sn_head (n), sn_head (n) +p ], any message is the message of the nth video frame, and if the difference between the sequence number of any message and sn_head (n) is Q, any message is the Q message of the nth video frame.

603. For any video frame of the video to be played, the first device determines a first received message as a first target message of the video frame after the message of the last video frame for encapsulating the video frame is received.

When receiving the second target message with the largest sequence number of the last video frame (namely, the n-1 th video frame) of the video frame, namely, the last video frame of the video frame is received, the second target message is the reference message of the first target message, after the first device receives any message of the last video frame of the video frame, if the mark of any message is m=1 or the any message is the message with the largest sequence number of the last video frame of the video frame, the first target message is the last message of the last video frame of the video frame, namely, the last received message is the first target message of the video frame after the first device receives the second target message.

604. The first device modifies the time stamps of the plurality of messages according to the time stamp change rule and the first target message to obtain a modified message, wherein the time stamp of the modified message accords with the time stamp change rule.

The first device may modify the timestamp of the first target message to a target timestamp, and then modify the messages in the video frame except for the first target message according to the target timestamp. The process of modifying the timestamp of the first target message to the target timestamp is the same as the process shown in steps 504-505, and the process of modifying the messages in the video frame except for the first target message according to the target timestamp is the same as the process shown in step 506.

It should be noted that, when the first device receives the first target packet and then receives the fourth target packet, the first device determines whether the fourth target packet is a packet of the video frame according to the sequence number of the fourth target packet and the sequence number range of the packet of the video frame, when the sequence number of the fourth target packet is within the sequence number range of the packet of the video frame, the first device executes step 604, when the flag bit m=1 of the fourth target packet, it indicates that the fourth target packet is the last packet of the video frame, and if the first device receives a packet again, the first device executes step 603 based on the newly received packet, that is, the newly received packet is the first target packet of the next video of the video frame.

And, the process shown in steps 603-604 is performed for each video frame of the video to be played, so that the first device can modify the time stamp of each message of each video frame, and since each message modification is based on the time stamps of the first two video frames and the local time, each message in each video frame of the modified video to be played is continuous in time and continuous in time.

It should be noted that the process shown in the step 604 is a process of modifying the time stamps of the plurality of messages according to the time stamp change rule of the video to be played.

605. The first device sends the modified message to the terminal device.

Because the time stamps of the modified messages are continuous in time, when the first equipment sends the messages which are continuous in time to the first equipment, the terminal equipment does not have transient black screen when playing the video based on the time stamps of the messages, so that the experience of the user can be improved.

According to the method provided by the embodiment of the application, the obtained time stamp of the message is modified according to the time stamp change rule of the message of the video to be played, so that the gateway equipment is not required to be used for message conversion, namely, the gateway equipment is not required to be used in the message transmission process, and the resource consumption and the cost in the message transmission are reduced. And the last message of each video frame is judged by the mark bit M in the message, so that whether the modification of the video frame is finished can be accurately determined.

When a second device sends a message to a first device, the second device sends the message according to the sequence number of the message, the message is sent first with a large sequence number and then sent with a small sequence number, but when the second device is in a transmission process from the second device to the first device, the message with the small sequence number is influenced by the outside, the message with the small sequence number can reach the first device first, the message received at the first device can generate disorder phenomenon, therefore, the first device can firstly judge whether the received message is disorder, and modify the message timestamp according to the disorder condition to ensure the continuous timestamp of the message in each video frame, and in order to embody the process, the flow of the method can be the process shown in message steps 701-706.

701. The first device obtains a plurality of messages in a video stream of a video to be played.

The process shown in this step 701 is the same as the process shown in step 601, and this step 701 is not specifically limited in this embodiment.

702. The first device determines the change rule of the time stamp of the message of the video to be played according to the time stamp of the message for packaging the first target video frame in the video stream.

The process shown in this step 702 is the same as the process shown in step 602, and here, this step 702 is not specifically limited in this embodiment of the present application. It should be noted that, besides the time stamp change rule, the first device may also obtain other parameters required for modifying the time stamp, where the parameters may also include a sequence number of a frame header message of each video frame.

703. For any video frame of the video to be played, the first device determines a message with the minimum sequence number in the messages belonging to the video frame as a first target message of the video frame.

When the sequence number of any message is any one of the sequence numbers of the frame header messages obtained by the first device in advance, the first device can use any message as the first target message of the video frame where the first device is located, if the sequence number of any message is the first message of the video frame where the first device is located, that is, the message with the minimum sequence number.

704. The first device determines whether the transmission of the message of the video frame is disordered according to the sequence number of the first target message and the sequence number of a second target message, wherein the second target message is a message with the largest sequence number in the message of the last video frame for packaging the video frame.

The video frame is an nth video frame, the first device may use the transmission status to identify whether the transmission of the packet of the video frame is out of order, and in one possible implementation, the first device determines the transmission status of the video frame according to the sequence number of the first target packet and the sequence number of the second target packet, where the second target packet is the last packet of the last video frame of the video frame, and the transmission status is used to indicate whether the transmission of the packet of the video frame is out of order.

The transmission state may include any one of a first state for indicating that transmission of the message of the video frame is not out of order and a second state for indicating that transmission of the message of the video frame is out of order.

When the difference between the sequence number of the first target message and the sequence number of the second target message is a first value, it may indicate that the first target message normally arrives at the first device in the sequence of the sequence numbers, that is, the transmission of the message of the video frame is not disordered, the first device determines the transmission state of the video as a first state, where the first value may be 1. When the difference between the sequence number of the first target message and the sequence number of the second target message is a second value, it may indicate that the first target message normally arrives at the first device not in the sequence of the sequence number, and arrives earlier than the second target message, that is, the transmission of the message of the video frame is not disordered, the first device determines the transmission state of the video as a second state, where the second value may be p+1. It should be noted that, in the embodiment of the present application, the first value and the second value are not specifically limited.

705. The first device modifies the time stamps of the plurality of messages according to whether the transmission of the messages of the video frame is out of order, the time stamp change rule and the first target message, and obtains the modified messages.

For different transmission states and different time stamp change rules, the first device may modify the message time stamp in different manners, and this step 705 may be implemented by a procedure described in the following steps 7051-7052.

Step 7051, when the transmission of the messages of the video frame is not disordered, the first device obtains a target timestamp according to the local clock, replaces the timestamp of the first target message with the target timestamp, and modifies the messages of the plurality of messages and the first target message belonging to the same video frame according to the change rule of the timestamp and the target timestamp.

It should be noted that the modification process shown in step 7051 is the same as the processes shown in steps 504-506, and the embodiment of the present application will not be repeated here for this step 7051.

Step 7052, when the transmission of the packets of the video frame is disordered, the first device modifies the timestamp of the packet of the video frame according to the timestamp change rule of the packet with the smallest sequence number in the last video frame for encapsulating the second target video frame, where the second target video frame is the last video frame of the video frame to which the first target packet belongs.

The first device may calculate the modified timestamp of the first target message, and modify the timestamps of other messages in the plurality of messages according to the timestamp of the first target message. In one possible implementation, this step 7052 may be implemented by the process shown in steps A-B below.

And step A, the first device modifies the time stamp of the first target message in the plurality of messages into a target time stamp according to the target time length and the time stamp of the message with the minimum sequence number in the last video frame of the second target video frame, wherein the target time length is the time length of one video frame of the video to be played.

When the video frame is the nth video frame of the video to be played, the modified target timestamp of the first target message may be expressed as ts_head (n), where ts_head (n) =ts_head (n-1) +Δts, and ts_head (n-1) is the timestamp of the message with the smallest sequence number in the last video frame of the video frame, that is, the timestamp of the first target message of the nth-1 video frame.

And B, the first device modifies messages belonging to the same video frame as the first target message in the plurality of messages according to the change rule of the time stamp and the target time stamp.

For different time stamp change rules, the first device modifies messages belonging to the same video frame as the first target message in the plurality of messages in different ways. In one possible implementation, this step B may be implemented by the process shown in steps B1-B3 below.

And B1, for any message except the first target message in any video frame, when the time stamp change rule is used for indicating that the time stamp of the message of each video frame of the video to be played is sequentially increased, the first device modifies the time stamp of any message according to the target time stamp, the target duration, the target message number and the sequence number of any message, wherein the target message number is the message number of each video frame.

For any message except the first target message in the plurality of messages, that is, any message except the first message in the nth video frame, the first device can calculate a modified timestamp of the any message through a formula (4), wherein Ts is the modified timestamp of the any message, sn is a sequence number of the any message, and sn_head (n) is a sequence number of the first target message of the video frame.

Ts＝Ts_head(n)+ΔTs/P*(Sn-Sn_head(n)) (4)

After calculating Ts according to formula (4), the first device replaces the original timestamp of any message with Ts to modify the timestamp of any message. When the first device completes the replacement of the time stamps of all the messages except the first target message in the video frame, the first device also modifies the time stamps of all the messages in the video frame.

And B2, when the time stamp change rule is used for indicating that the time stamps of the messages of each video frame are the same, the first device modifies the time stamp of any message into the target time stamp.

The process shown in the step B2 is the same as the process shown in the step 62, and the embodiment of the present application will not be described in detail herein.

And B3, when the time stamp change rule is used for indicating that the time stamp of the message of each video frame is divided according to the fields and the time stamps of the messages belonging to the same field are the same, if the time stamp of any message is the same as the original time stamp of the first target message, the first equipment modifies the time stamp of any message into the target time stamp, otherwise, the time stamp of any message is modified according to the target time stamp and the target time length.

The process shown in the step B3 is the same as the process shown in the step 63, and here, the embodiment of the present application will not be described in detail in the step B3.

It should be noted that the process shown in the step a-B is a process of modifying the time stamps of the plurality of messages according to the time stamp change rule of the messages of the video to be played. After the first device modifies the timestamp of the last packet of the video frame, the first device may perform the processes shown in steps 703-705 on the next video frame of the video frame, so that the first device may modify the timestamp of each packet of each video frame, and because each packet is modified based on the timestamps of the first two video frames and the frame header packet, the modified time between each packet in each video frame of the video to be played is more continuous, and the time between each video frame is also more continuous.

It should be noted that the process shown in step 705 is a process of modifying the timestamp of the message of the video frame according to whether the transmission of the message of the video frame is out of order and the change rule of the timestamp.

706. The first device sends the modified message to the terminal device.

Because the time stamps of the modified messages are continuous in time, when the first device sends the messages which are continuous in time to the terminal device, the terminal device does not have transient black screen when playing the video based on the time stamps of the messages, and therefore the experience of the user can be improved.

According to the method provided by the embodiment of the application, the obtained time stamp of the message is modified according to the time stamp change rule of the message of the video to be played, so that the gateway equipment is not required to be used for message conversion, namely, the gateway equipment is not required to be used in the message transmission process, and the resource consumption and the cost in the message transmission are reduced. And the last message of each video frame is judged by the mark bit M in the message, so that whether the modification of the video frame is finished can be accurately determined. The time between each message in each video frame of the video to be played after modification is more continuous, and the time between each video frame is also more continuous, so that the experience of the user can be improved.

Fig. 8 is a schematic structural diagram of a message transmission device provided in an embodiment of the present application, where the device includes:

An obtaining module 801, configured to execute the above step 601;

a modifying module 802, configured to perform the step 603;

a sending module 803, configured to perform step 605.

Optionally, the apparatus further comprises:

a first determining module, configured to perform the step 602;

optionally, the first target video frame is a first video frame of the video to be played.

Optionally, the first module is configured to perform steps 61-63 described above.

Optionally, the modification module includes: :

a first modifying unit, configured to modify a timestamp of a first target message in the plurality of messages to a target timestamp;

and a second modifying unit, configured to perform step 506.

Optionally, the first modification unit is configured to perform the steps 504-505.

Optionally, the apparatus further comprises a second determining module, configured to perform the step 503 or 603.

Optionally, the first modifying unit is configured to perform step a.

Optionally, the apparatus further comprises:

a third determining module, configured to perform step 703.

Optionally, the third determining module is further configured to perform step 704 described above.

Any combination of the above-mentioned optional solutions may be adopted to form an optional embodiment of the present disclosure, which is not described herein in detail.

It should be noted that: in the message transmission device provided in the above embodiment, only the division of the above functional modules is used for illustration, and in practical application, the above functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the embodiments of the method for transmitting a message provided in the foregoing embodiments belong to the same concept, and detailed implementation processes of the method embodiments are described in detail in the foregoing embodiments, which are not repeated herein.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The foregoing description of the preferred embodiments of the present application is not intended to limit the invention to the particular embodiments of the present application, but to limit the scope of the invention to the particular embodiments of the present application.

Claims

1. A method for transmitting a message, the method being performed by a first device, the method comprising:

obtaining a plurality of messages in a video stream of the same video to be played from one or more second devices, wherein the one or more second devices are used for generating a data stream;

and sending the modified message to terminal equipment.

2. The method according to claim 1, wherein before modifying the time stamps of the plurality of messages according to the time stamp change rule of the messages of the video to be played, the method further comprises:

3. The method of claim 2, wherein the first target video frame is a first video frame of the video to be played.

4. The method of claim 2, wherein determining the time stamp change rule based on the time stamp of the message in the video stream that encapsulates the first target video frame comprises:

5. The method according to any one of claims 1-4, wherein modifying the time stamps of the plurality of messages according to the time stamp change rule of the message of the video to be played, and obtaining the modified message includes:

6. The method of claim 5, wherein modifying the timestamp of the first target message of the plurality of messages to the target timestamp comprises:

acquiring the target time stamp according to a local clock;

7. The method of claim 6, wherein prior to obtaining the target timestamp from the local clock, the method further comprises:

8. The method of claim 5, wherein modifying the timestamp of the first target message of the plurality of messages to the target timestamp comprises:

And modifying the time stamp of a first target message in the plurality of messages into a target time stamp according to a target time length and a time stamp of a message with the minimum message sequence number for packaging a second target video frame, wherein the target time length is the time length of one video frame of the video to be played, and the second target video frame is the last video frame of the video frame to which the first target message belongs.

9. The method of claim 8, wherein before modifying the timestamp of the first target message of the plurality of messages to the target timestamp according to the target duration and the timestamp of the message with the smallest sequence number in the last video frame used to encapsulate the video frame, the method further comprises:

10. The method according to claim 9, wherein after determining a message with a smallest sequence number among the messages belonging to the video frame as the first target message of the video frame, the method further comprises:

11. A message transmission apparatus, the apparatus comprising:

the system comprises an acquisition module, a data stream generation module and a data stream generation module, wherein the acquisition module is used for acquiring a plurality of messages in a video stream of the same video to be played from one or a plurality of second devices;

the modifying module is used for modifying the time stamp change rule of the message of the video to be played, so as to obtain a modified message, and the time stamp of the modified message accords with the time stamp change rule;

and the sending module is used for sending the modified message to the terminal equipment.

12. The apparatus of claim 11, wherein the apparatus further comprises:

and the first determining module is used for determining the change rule of the time stamp according to the time stamp of the message for packaging the first target video frame in the video stream.

13. The apparatus of claim 12, wherein the first target video frame is a first video frame of the video to be played.

14. The apparatus of claim 12, wherein the first determining module is configured to:

when the time indicated by the time stamp of the message of the first target video frame is sequentially increased, determining the time stamp change rule as a first time stamp change rule, wherein the first target video frame is at least one video frame of the first video frame and the second video frame, and the first time stamp change rule is used for indicating that the time stamp of the message of each video frame of the video to be played is sequentially increased;

15. The apparatus of any of claims 11-14, wherein the modification module comprises:

and the second modifying unit is used for modifying the messages which belong to the same video frame as the first target message in the plurality of messages according to the change rule of the time stamp and the target time stamp.

16. The apparatus of claim 15, wherein the first modification unit is configured to:

acquiring the target time stamp according to a local clock;

17. The apparatus of claim 16, further comprising a second determination module configured to:

18. The apparatus of claim 15, wherein the first modification unit is configured to:

19. The apparatus of claim 18, wherein the apparatus further comprises:

And the third determining module is used for determining a message with the minimum sequence number in the messages belonging to any video frame of the video to be played as a first target message of the video frame.

20. The apparatus of claim 19, wherein the third determination module is further configured to:

21. A computer device comprising a processor and a memory having stored therein at least one instruction that is loaded and executed by the processor to implement the operations performed by the message transmission method of any one of claims 1 to 10.

22. A computer storage medium having stored therein at least one instruction that is loaded and executed by a processor to implement the operations performed by the message transmission method of any one of claims 1 to 10.