WO2021012180A1

WO2021012180A1 - Transmission control method and device for code streams, and computer readable storage medium

Info

Publication number: WO2021012180A1
Application number: PCT/CN2019/097265
Authority: WO
Inventors: 马宁; 陈颖; 苏文艺
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2019-07-23
Filing date: 2019-07-23
Publication date: 2021-01-28
Also published as: CN111602389B; CN111602389A

Abstract

Disclosed are a transmission control method and device for code streams. The method comprises: receiving at least two code streams, the at least two code streams comprising a first code stream and a second code stream, wherein the first code stream serves as the original main code stream, the code streams other than the first code stream in the at least two code streams serve as original background code streams, and the transmission frame rate of the original main code stream is higher than that of any of the original background code streams; obtaining a main code stream switching signal, the main code stream switching signal indicates switching the main code stream to the second code stream; and continuing receiving the at least two code streams after the switching, wherein the second code stream serves as a new main code stream, the code streams other than the second code stream in the at least two code streams serve as new background code streams, and after the switching, the transmission frame rate of the new main code stream is higher than that of any of the new background code streams. The problem in the prior art of long playback switching response time can be effectively solved. (FIG. 2)

Description

Transmission control method, equipment and computer readable storage medium of code stream

Technical field

The present disclosure relates to the field of data transmission technology, and in particular to a transmission control method, device and computer-readable storage medium of a code stream.

Background technique

With the development of communication technology, more and more scenarios have begun to use multi-channel code stream transmission. For example, in some live broadcasts of matches, multiple cameras are often set up at the match site to achieve live broadcasts of the match site from different angles, and the corresponding live broadcast receiver (such as the host’s screen, TV, network video terminal, etc.) usually has only one Therefore, if you need to watch the live images of different cameras, you need to switch the multiple streams transmitted by multiple cameras.

There is an image encoding and decoding process in the transmission of the code stream. In the image encoding and decoding, a key frame is an image frame that does not refer to any frame for encoding, and its data size is larger than other reference frames. Transmission needs to take up more bandwidth or longer time. Therefore, in the prior art, when switching between multiple streams, to achieve a better display effect, it is necessary to buffer the key frames and buffer them for a period of time before displaying, which leads to a longer switching time; Switching speed, the size of the key frame will be limited, which will cause the switching screen to freeze or blur.

Therefore, when switching between multiple streams, it is a problem to be solved urgently to ensure the display effect of the picture while reducing the switching time and occupying bandwidth.

Summary of the invention

In order to solve the problems of long switching time or poor screen display effect when switching multiple streams in related technologies, the present disclosure provides a stream transmission control method, device and computer-readable storage medium.

In a first aspect, a transmission control method of a code stream, the method includes:

At least two code streams are received, the at least two code streams include a first code stream and a second code stream, wherein the first code stream is used as the original main code stream, and the first code stream is excluded from the at least two code streams Other code streams outside the stream are used as the original background code stream, and the transmission frame rate of the original main code stream is higher than the transmission frame rate of any one of the original background code streams;

Acquiring a main code stream switching signal, where the main code stream switching signal indicates to switch the main code stream to the second code stream;

Continue to receive the at least two code streams after switching, wherein the second code stream is used as the new main code stream, and the other code streams except the second code stream among the at least two code streams are used as the new background code stream, After the switching, the transmission frame rate of the new main code stream is higher than the transmission frame rate of any one of the new background code streams.

In a second aspect, a transmission control method of a code stream, the method includes:

Send at least two code streams, the at least two code streams include a first code stream and a second code stream, wherein the first code stream is used as the original main code stream, and the first code stream is excluded from the at least two code streams Other code streams outside the stream are used as the original background code stream, and the transmission frame rate of the original main code stream is higher than the transmission frame rate of any one of the original background code streams;

Acquiring a frame rate adjustment signal, where the frame rate adjustment signal carries the code stream identifier of the second code stream;

After adjusting the frame rate according to the frame rate adjustment signal, continue to send the at least two code streams according to the adjusted transmission frame rate, wherein the second code stream is used as the new main code stream, and the at least two code streams are Other code streams in the stream except the second code stream are used as the new background code stream. After the frame rate is adjusted, the transmission frame rate of the new main code stream is higher than the transmission frame rate of any one of the new background code streams.

In a third aspect, a code stream receiving device includes a processor, a communication component, a memory, and a communication bus. The memory stores executable instructions, and the processor is configured to perform the following steps:

In a fourth aspect, a code stream sending device includes a processor, a communication component, a memory, and a communication bus. The memory stores executable instructions, and the processor is configured to perform the following steps:

In a fifth aspect, a computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, it implements the above-mentioned code stream transmission control method.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

According to the technical solution of the present disclosure, during the transmission of at least two streams, before the main stream is switched, the transmission frame rate of the original main stream is higher than the frame rate of the original background stream. Transmission; after the main stream switching signal is obtained, the transmission frame rate of the new main stream is higher than the transmission frame rate of the new background stream to continue the transmission of the at least two streams; that is, according to the main stream (original The transmission frame rate of the main code stream or the new main code stream) is greater than the transmission frame rate of the background code stream (the original background code stream or the new background code stream), which fully guarantees the transmission bandwidth of the main code stream and can avoid the background code stream ( The original background code stream or the new background code stream) occupies too much bandwidth and affects the transmission of the main code stream; and realizes the dynamic adjustment of the transmission frame rate of each stream according to the switching of the main code stream.

Furthermore, since the main stream and background stream are transmitted at different transmission frame rates at the same time, when the main stream is used as the stream to be played, it can be ensured that when the playback is switched, seamless switching to the playback location can be ensured. The main stream switched to improves the switching response speed and ensures the playback effect during the playback switching process.

It should be understood that the above general description and the following detailed description are only exemplary and cannot limit the present disclosure.

Description of the drawings

The drawings herein are incorporated into the specification and constitute a part of the specification, show embodiments in accordance with the present invention, and are used together with the specification to explain the principle of the present invention.

Fig. 1 is a schematic diagram showing an implementation environment involved in the present disclosure according to an exemplary embodiment;

Fig. 2 is a flow chart showing a method for transmission control of a code stream according to an exemplary embodiment;

Fig. 3 is a schematic diagram of code stream transmission implemented according to the method of the present disclosure in a specific embodiment;

Fig. 4 is a flowchart showing a method for transmission control of a code stream according to another exemplary embodiment;

Fig. 5 is a flowchart showing a transmission control method of a code stream according to another exemplary embodiment;

Fig. 6 is a flow chart showing a transmission control method of a code stream according to another exemplary embodiment;

Fig. 7 is a hardware block diagram of a UAV according to an exemplary embodiment.

Through the above drawings, the specific embodiments of the present invention have been shown, and there will be more detailed descriptions in the following. These drawings and text descriptions are not intended to limit the scope of the concept of the present invention in any way, but by referring to specific embodiments. The concept of the present invention is explained for those skilled in the art.

Detailed ways

Here, an exemplary embodiment will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present invention. Rather, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

Fig. 1 is a schematic diagram showing an implementation environment involved in the present disclosure according to an exemplary embodiment. The implementation environment includes: a sending end 100 and a receiving end 300.

The sending end 100 and the receiving end 300 establish a network connection in a wired or wireless manner. Therefore, based on the network connection between the two, the sending end 100 and the receiving end 300 interact, for example, the sending end 100 communicates to the receiving end. 300 transmits at least two code streams, and the sending end 100 adjusts the transmission frame rate of the image frames in the code stream according to the playback switching operation triggered by the receiving end 300.

The sending end 100 may be any device that can perform multi-channel code stream transmission, such as a drone, a notebook computer, a desktop computer, etc., that performs real-time image collection and transmission, which is not specifically limited here. The receiving end can be any communication device that can receive the code stream, such as remote control with or without screen, VR/AR display device, TV, tablet computer, smart phone, notebook computer and other electronic devices.

Fig. 2 is a flow chart showing a method for controlling code stream transmission according to an exemplary embodiment. The transmission control method of the code stream can be executed by the receiving end 300 in the implementation environment shown in FIG. 1. As shown in Figure 2, the transmission control method of the code stream may include the following steps:

Step 310: At least two code streams are received. The at least two code streams include a first code stream and a second code stream. The first code stream is used as the original main code stream. Other code streams are used as original background code streams, and the transmission frame rate of the original main code stream is higher than the transmission frame rate of any original background code stream.

In a specific application scenario, the code stream is a video stream formed by sequentially arranging several image frames. In other application scenarios, the code stream can also be an audio code stream.

The transmission frame rate is the number of image frames transmitted per second. The transmission frame rate characterizes the data transmission speed in the code stream. It is worth mentioning that in step 310, the term “receiving at least two streams” refers to sequentially receiving the image frames in the at least two streams, and step 310 does not represent the completion of receiving each of the at least two streams. All image frames of the code stream, and the representative receiving end is in the process of receiving the image frames in each code stream.

For the at least two code streams received by the receiving end, one of the code streams is used as the main code stream, and the other code streams among the at least two code streams are taken as the background code stream relative to the main code stream. In the technical solution of the present disclosure, the transmission frame rate of the main code stream is configured to be higher than the transmission frame rate of any background code stream. In a specific embodiment, the code stream to be played is used as the main code stream. Correspondingly, the other code streams except the played code stream in the received at least two code streams correspond to the background code stream, according to the main code stream. The transmission frame rate is higher than the transmission frame rate of any background code stream for transmission, which can meet the playback requirements of the receiving end for the main code stream and avoid jams.

It is worth mentioning that the background code stream is relative to the main code stream. When the main code stream changes, the corresponding background code stream changes accordingly. For example, at least two bit streams include A bit stream, B bit stream and C bit stream, where the main bit stream is A bit stream initially, then when A bit stream is used as the main bit stream, B bit stream The stream and C stream are the background stream; if the B stream is re-selected as the main stream, the A stream and the C stream will be used as the background stream while the B stream is used as the main stream.

For the main stream, the main stream can be selected based on user needs, or the main stream is set by default on the receiving end, or based on the processing set in the receiving end, for example, it is currently necessary to play the A stream , The A stream is selected as the main stream (original main stream), so that the operation of the image frame in the A stream (such as encoding, transmission, decoding, buffering, etc.) by the sender or receiver All need to meet the playback of the A stream.

In the technical solution of the present disclosure, in order to facilitate description and avoid confusion, the main code stream before the main code stream switch is called the original main code stream, and the background code stream before the switch is called the original background code stream; the main code stream is switched The main stream after that is called the new main stream, and the background stream after switching is called the new background stream.

It is worth mentioning that the first code stream and the second code stream of the at least two code streams do not specifically refer to the specific code stream. The first code stream generally refers to the original master before the main stream is selected. The code stream of the code stream, and the second code stream generally refers to the code stream that is switched as the new main code stream when the main code stream is switched.

Step 330: Obtain the main code stream switching signal. The main code stream switching signal indicates to switch the main code stream to the second code stream.

Based on user requirements, or the processing process of each stream received at the receiving end, the data volume requirements of each stream at the receiving end may change, and the main stream needs to be switched; for example, the A Switch the code stream to play B code stream, then correspondingly, switch the main code stream to B code stream.

The main code stream switching signal is used to indicate the code stream to be switched to the main code stream. In the technical solution of the present disclosure, the second code stream generally refers to the code stream which needs to be switched to the main code stream indicated by the main code stream switching signal. Therefore, the second code stream indicated by the main code stream switching signal is regarded as the new main code stream of the present disclosure.

In a specific embodiment, the main stream switching signal may be generated according to a switching operation triggered in the receiving end, and the user triggers the switching operation in the receiving end to instruct the receiving end to switch the main stream to the second code. flow. For example, when the receiving end is VR glasses and the user wants to switch the displayed code stream, he can trigger the switching operation by operating the corresponding control component of the VR glasses, thereby generating the main code stream switching signal.

In another specific embodiment, the main stream switching signal may also be sent by an external device to the receiving end, or generated according to the main stream switching file configured in the receiving end, and the main stream switching file indicates During the transmission of at least two streams, each time segment is selected as the stream of the main stream. For example, when the receiving end is a monitor screen, the monitor screen can receive a signal sent by an external device, or generate a main stream switching signal according to a configured main stream switching file. The configured main stream switching file can generate the main stream switching signal according to the working status of the receiving end, such as working location, working time and other information. For example, when the monitor screen displays the picture of camera A for a predetermined period of time, the main stream switching signal can be automatically generated according to the pre-configured main stream switching file to automatically switch to the picture of camera B.

Step 330: Continue to receive at least two streams after the switch, where the second code stream is used as the new main code stream, and the other code streams in the at least two code streams except the second code stream are used as the new background code stream, after switching , The transmission frame rate of the new main stream is higher than the transmission frame rate of any new background stream.

As described above, the main code stream switching signal indicates the main code stream after switching. Since the transmission frame rate of the main code stream is configured to be higher than the transmission frame rate of any background code stream, after switching the main code stream, according to The selected stream as the new main stream, correspondingly adjust the transmission frame rate of each of the at least two streams, so that after switching, the transmission frame rate of the new main stream is higher than any new background code The transmission frame rate of the stream.

In a specific embodiment, to ensure that after switching, the transmission frame rate of the new main stream is higher than the transmission frame rate of any new background stream, after the receiving end obtains the main stream switching signal, the receiving end sends The end sends a frame rate adjustment signal, and the frame rate adjustment signal indicates that the main stream is switched to the second stream. It can be understood that the frame rate adjustment signal is generated according to the main stream switching signal. Therefore, after receiving the frame rate adjustment signal, the transmitting end adjusts the transmission frame rate of each of the at least two streams according to the frame rate adjustment signal, and performs each code according to the adjusted transmission frame rate Stream transmission, where the sender adjusts the frame rate according to the transmission frame rate of the new main stream higher than the transmission frame rate of any new background stream.

Before switching the main stream, the second stream is used as one stream in the original background stream, and the first stream is used as the original main stream. The transmission frame rate of the original main stream is higher than that of the original background stream. Rate; after the main code stream is switched, the second code stream is used as the new main code stream, and the first code stream corresponds to the one code stream in the new background code stream; thus, it can be known that after the switch, in order to ensure the new main code stream The transmission frame rate is higher than the transmission frame rate of the new main stream and guarantees the demand of the receiving end for the second code stream. After switching, the transmission frame rate of the second code stream is increased, based on the transmission of the second code stream after the switch Frame rate, correspondingly adjust the frame rate of each stream as the new background stream.

In one embodiment, the transmission frame rate of the background code stream (original background code stream or new background code stream) is preset. When a code stream is used as the background code stream (original background code stream or new background code stream), The transmission of the stream is carried out according to the transmission frame rate set for the main stream. In other words, as the background code stream (original background code stream or new background code stream), the transmission frame rate of each code stream is the same, so the transmission frame rate of the new background code stream after switching is the transmission frame rate of the second code stream before switching Frame rate. In a specific embodiment, it is also possible to set the transmission frame rate as the background code stream (original background code stream or new background code stream) for each bit stream.

In an embodiment, the transmission frame rate of the main code stream (original main code stream or new main code stream) can also be preset. When a code stream is selected as the main code stream (original main code stream or new main code stream) After that, the transmission of the code stream as the main code stream is performed according to the set transmission frame rate. In other words, as the main stream (original background code stream or new background code stream), the transmission frame rate of the main code stream is the same. Therefore, the transmission frame rate of the new main code stream after switching is the transmission frame rate of the original main code stream before switching. Frame rate.

In another embodiment, it is also possible to set the transmission frame rate as the main stream (original main stream or new main stream) and as the background code for each of the at least two streams at the same time. The transmission frame rate of the stream (original background code stream or new background code stream), so that after the main code stream is switched, the transmission frame rate of each code stream is adjusted according to the settings made.

In another embodiment, the transmission frame rate of the new background code stream is a dynamic low frame rate adapted to the remaining bandwidth. The remaining bandwidth is determined according to the transmission frame rate of the new main stream after switching. Specifically, the difference between the current bandwidth of the receiving end and the sending end and the occupied bandwidth of the new main stream when the transmission is performed according to the transmission frame rate of the new main stream after switching is the remaining bandwidth. Then dynamically determine the transmission frame rate for the new background stream according to the remaining bandwidth.

In the technical solution of the present disclosure, during the transmission of at least two streams, before the main stream is switched, the at least two streams are performed according to the transmission frame rate of the original main stream higher than the frame rate of the original background stream After obtaining the main stream switching signal, the transmission frame rate of the new main stream is higher than the transmission frame rate of the new background stream to continue the transmission of the at least two streams; on the one hand, according to the main stream The transmission frame rate of (original main stream or new main stream) is greater than the transmission frame rate of background code stream (original background code stream or new background code stream), which fully guarantees the transmission bandwidth of the main stream to meet the requirements of the receiving end. The data requirement of the main stream is to avoid the situation where the available bandwidth of the main stream is insufficient due to the excessive occupation of bandwidth by the background stream (original background stream or new background stream); on the other hand, according to the switch of the main stream The transmission frame rate of each of the at least two streams is dynamically adjusted, so that the adjustment of the frame rate is adapted to the switching of the main stream.

In an embodiment, after step 310, the method further includes:

Play the original main stream and cache the original background stream.

In other words, the receiving end buffers and plays the original main code stream according to the data of the received original main code stream, and only buffers the original background code stream, so that when the first code stream is used as the original main code stream, the receiving end Only the first stream is played at the end.

In a specific embodiment, since the code stream is transmitted after encoding, correspondingly, the receiving end performs corresponding decoding after receiving the data in the original main code stream and the original background code stream. In other words, the original main stream is decoded and played, and the original background stream is decoded and buffered.

Further, after step 330, the method further includes:

Play the second code stream as the new main code stream, and buffer the new background code stream.

In other words, in the receiving end, the receiving end switches the playback according to the main stream switching signal, that is, stops playing the original first stream and switches to playing the second stream as the new main stream.

As described above, corresponding to the scene of code stream transmission after encoding, the second code stream as the new main code stream is decoded and played, and the new background code stream is decoded and buffered.

In a specific embodiment, the step of playing the second code stream as the new main code stream includes:

Obtain the buffered data buffered for the second code stream before switching.

Switch to play the second stream according to the buffered data.

Because during the playback of the original main stream, data is buffered for the original background stream, including the second stream as the original background stream during this period, so after the main stream is switched, no buffering is needed. Can seamlessly switch to play the second stream.

In the stream playback switching of the prior art, after the receiving end receives the playback switching instruction (equivalent to the main stream switching signal in this embodiment), it requests the sending end correspondingly to the playback switching instruction indicated by the playback switching instruction. The data of the code stream, that is, the buffering of the data in the code stream switched to is performed after the playback switching, which makes the playback switching time long and the playback switching response speed slow.

After the playback switching operation is performed, the receiving end requests the image in the video stream from the sending end. Therefore, the sending end performs encoding and transmission of the image in the requested video stream according to the request of the receiving end. Since the first video frame in the video code stream is an I frame, and the I frame does not refer to any frame, it is obtained by the intra-frame coding method. The compression rate is low, the data volume is high, and the compression is difficult. The streaming playback switching method, in order to ensure that the receiving end responds to the playback switching, the size of the first video frame needs to be limited, that is, the first video frame is obtained by encoding after compression, and when the receiving end obtains the I frame according to the compressed encoding for playback and display, the displayed first video frame The frame image freezes or is blurred. However, if the size of the first video frame is not limited, it will take a long time to transmit the I frame, resulting in a long transmission delay. Regardless of whether the first video frame is stuck or blurred or the transmission delay of the first video frame is large, the user experience is greatly reduced.

With the technical solution of this embodiment, these problems can be effectively avoided. Because before the playback switching, that is, before the receiving end obtains the main stream switching signal, the original main stream is buffered and played, and the data in the original background stream is buffered, so that the main stream is obtained at the receiving end After the signal is switched, it can be seamlessly switched to play the second stream as the new main stream directly according to the buffered data. The playback switching response speed is fast, and there is no blurring of the first frame, ensuring the playback effect.

In the prior art, in order to improve the response time of stream playback switching, for at least two streams to be transmitted, the image frames in each stream are transmitted according to the transmission frame rate that satisfies the receiving end to play each stream. . Although the second cut is achieved during the playback switching, the transmission of the currently selected stream before the switching occupies too much bandwidth in the transmission channel, so that the available bandwidth for transmitting the currently selected stream is lost , It affects the playback of the currently selected stream, resulting in poor playback quality.

In the transmission of at least two streams performed by the present disclosure, combined with the original main stream selected in the receiving end to be played, the transmission frame rate of the original main stream is higher than the transmission of any original background stream The frame rate is transmitted to ensure the playback quality of the original main stream; and after the playback is switched according to the main stream switching signal, the frame rate is adjusted, and the transmission frame rate of the new background stream after adjustment is higher than any new background code The transmission frame rate of the stream, and the transmission of each stream according to the adjusted transmission frame rate, so as to realize the targeted transmission of each stream in combination with the stream that needs to be played in the receiving end. The response speed of switching ensures the playback quality of the code stream.

Fig. 3 is a schematic diagram of code stream transmission according to the method of the present disclosure in an application scenario. In Figure 4, the receiving end receives two streams in parallel, namely, A stream and B stream. Among them, before the code stream playback switch, the receiving end plays the A code stream, and correspondingly, during the playing of the A code stream, the A code stream is used as the original main code stream, and the B code stream is used as the original background code stream, and The transmission frame rate of code stream A is higher than that of code stream B. After the playback switch is performed, the A bit stream is used as the new background bit stream, and the B bit stream is used as the new main bit stream. After the transmission frame rate of the two bit streams is adjusted according to the playback switch, the A bit stream The transmission frame rate is lower than that of the B-channel stream. In Figure 3, the adjustment is to adjust the transmission frame rate of the A stream to the transmission frame rate of the B stream before the playback switch, and adjust the transmission frame rate of the B stream to the A code before the playback switch The transmission frame rate of the stream.

As shown in Figure 3, in the A bit stream and the B bit stream, the first image frame transmitted in each bit stream is an I frame, and the image frame transmitted thereafter is a P frame, thus greatly reducing the The bandwidth occupied during the transmission process. In a specific embodiment, in order to prevent abnormal image frames during transmission from affecting the decoding of subsequently transmitted image frames, when a transmission error occurs in the code stream, encoding is performed again according to the intra-frame encoding method to obtain the I frame of the current image to be encoded And transmit; if there is no transmission error in the code stream, continue encoding according to the inter-frame encoding method to obtain the P frame of the current image to be encoded and transmit it.

In an embodiment, the method further includes:

For the received image frames of each of the at least two code streams, anomaly detection is performed on the received image frames.

In order to ensure the accuracy of bit stream transmission, the receiving end is concerned with whether the received image frame is accurate, that is, whether a data error occurs, and whether the receiving end misses the image frame, that is, whether the image frame is lost.

The abnormality detection performed is to detect whether the image frame received in the receiving end has data errors and whether the image frame loss occurs in the receiving end. Therefore, an abnormal image frame in the code stream is correspondingly determined according to the performed abnormality detection, and the abnormal image frame includes the image frame in which the data error occurs and the lost image frame.

In a specific embodiment, the abnormality detection performed at the receiving end includes:

According to the check code carried by the received image frame, it is checked whether the image frame is an image frame with data errors.

The image frame transmitted by the sending end to the receiving end includes the check code configured for the image frame, such as a CRC check code, so that after the image frame is received, the corresponding check algorithm is used, such as the CRC check The CRC check algorithm of the code is checked to determine whether the check code obtained by the check algorithm is consistent with the check code carried in the received image frame to determine whether the received image frame has a data error. If they are consistent, there is no data error in the received image frame. On the contrary, if they are not consistent, the received image frame has a data error.

In another specific embodiment, the abnormality detection performed on the receiving end further includes:

According to the frame sequence number carried by the received image frame, determine whether the frame sequence numbers of the two adjacent received image frames in the same code stream are continuous, and the image frame corresponding to the frame sequence number between the frame sequence numbers that are judged to be discontinuous is confirmed as lost Image frame.

Regarding whether the image frame is lost, in this embodiment, the sending end configures the frame sequence number for the sent image frame. Therefore, after receiving the image frame, the receiving end passes through the same code stream twice. The continuity of the frame numbers of the received image frames is judged accordingly, that is, if the frame numbers of the image frames received two consecutive times are consecutive, there is no missing image frame in the image frames received twice. ; On the contrary, there are missing image frames, and the missing image frames are the image frames corresponding to the frame numbers between the discontinuous frame numbers.

In an embodiment, after performing abnormality detection on the received image frame, the method further includes:

When it is confirmed that the received image frame is accurate through anomaly detection, feedback information is sent to the sending end for the accurately received image frame, and the feedback information carries the frame number of the image frame.

Through the feedback information, the sending end can know in time which image frames sent are received and accurately received.

In an application scenario, the code stream is transmitted to the receiving end after being encoded. The original image in the code stream is encoded by the sending end according to the inter-frame encoding method or the intra-frame encoding method.

The inter-frame encoding method refers to the method of encoding image data using other frames as reference frames using the temporal correlation of image frames.

The intra-frame coding method refers to a method of encoding image data without referring to time-related lines for spatial prediction.

For the stream transmitted from the sender to the receiver, the first frame of each stream is an intra-encoded frame obtained by intra-encoding, such as H.263, H.264, H.265, MPEG4, etc. The coding standard is an I frame obtained by intra-coding. For intra-frame coded frames, since other frames are not referenced for coding, the receiving end can decode the intra-frame coded frames as long as they correctly receive it.

The image frame transmitted after the first frame in each stream can be an intra-frame coded frame, or an inter-frame coded frame obtained by inter-frame coding, such as H.263, H.264, H.265, P-frames and B-frames obtained by intra-frame coding in video coding standards such as MPEG4. For inter-frame coded frames, since the predictive coding is performed with reference to the time correlation, that is, the coding depends on the reference frame, so even if the receiving end receives the inter-coded frame correctly, if the reference frame of the inter-coded frame is wrong, then The inter-coded frame cannot be decoded correctly.

At the receiving end, it is determined that the received image frame has a data error through anomaly detection. If the image frame with the data error is used as the reference frame of the subsequent image frame, the received image frame will not be correctly translated. Code, which causes the error caused by the image frame to be passed backward.

For example, the code stream transmitted by the sending end to the receiving end includes the following image frames arranged in order: P0, P1, P2, P3, P4, P5, P6, ... Pn, where, in the sending end, the image The frame P5 is obtained by encoding the image frame P4 as the reference frame, and correspondingly, the image frames P6, P6, P6, ... Pn are obtained by encoding the previous image frame as the reference frame. Now the receiving end finds that the image frame P4 is an abnormal image frame (that is, data error occurs or the image frame is lost) through anomaly detection, then, correspondingly, the image frame P4 cannot be correctly decoded at the receiving end.

After confirming that the image frame P4 is an abnormal image frame, if it is not processed in time, it will directly affect the decoding of subsequent image frames. That is, if it is not processed, the receiving end cannot decode correctly after receiving the image frames P5, P6 and subsequent image frames, resulting in the error caused by the image frame P4 being transferred to the subsequent image frames , Resulting in wrong delivery.

If the code stream is played in real time at the receiving end, if no countermeasures are taken for abnormal image frames to avoid the current abnormal image frame from affecting subsequent image frames, it will greatly affect the playback effect of the code stream.

Therefore, in an embodiment, in order to avoid erroneous transmission, the method further includes:

When it is detected that the received image frame is accurately received, feedback information is sent to the sending end for the accurately received image frame, and the feedback information carries the frame number of the corresponding image frame.

The image frame transmitted by the receiving and sending end for encoding the reference frame determined for the initial data to be encoded according to the feedback information. When the sending end detects the discontinuity of the frame sequence numbers in the two adjacent feedback information received, it is the initial data to be transmitted Determine the reference frame, and the determined reference frame is different from the abnormal image frame.

At the sending end, the transmitted image frame is obtained by encoding the initial data.

In the technical solution of this embodiment, the sending end dynamically determines the reference frame corresponding to the initial data to be transmitted in the sending end according to the feedback information sent by the receiving end when it detects that the received image frame is a data error. Ensure that the determined reference frame is the image frame accurately received at the receiving end. So as to avoid wrong backward transmission.

In an embodiment, after the step of sending feedback information to the sending end for the accurately received image frame, the method includes:

Save the accurately received image frame and the corresponding frame sequence number in the reference frame management queue of the corresponding code stream at the receiving end.

In this embodiment, a reference frame management queue is correspondingly configured at the receiving end for each bit stream received, and the reference frame management queue is used to store the accurately received image frame and the corresponding frame sequence number in the receiving end.

In another embodiment, to avoid erroneous delivery, the method further includes:

The error-tolerant data sent again by the sending end for the abnormal image frame is the intra-frame coded frame corresponding to the abnormal image frame or the frame group including the intra-frame coded frame.

By decoding the intra-coded frame in the error-tolerant data, an accurate image frame is obtained.

In this embodiment, regardless of whether the abnormal image frame is an intra-coded frame obtained by inter-coding, or other intra-coded frames are used as reference frames to be obtained by inter-coding, as long as it is received again as An abnormal image frame of an intra-coded frame or an intra-coded frame as a reference frame of the abnormal image frame can be decoded according to the re-received intra-coded frame correspondingly to avoid the error backward transfer.

In a specific embodiment, the sending end periodically sends fault-tolerant data to the receiving end, so that no matter when an abnormal image frame appears in the receiving end, as long as the intra-frame coded frame corresponding to the abnormal image frame is received, it can be Decoding is performed according to the intra-coded frame received again for the abnormal image frame to obtain an accurate image frame.

Fig. 4 is a flow chart showing a method for transmission control of a code stream according to another exemplary embodiment. The transmission control method of the code stream can be executed by the transmitting end 100 of the code stream, as shown in FIG. 4, including the following steps:

Step 410: Send at least two code streams, at least two code streams including a first code stream and a second code stream, where the first code stream is used as the original main code stream, and the at least two code streams except the first code stream Other code streams are used as original background code streams, and the transmission frame rate of the original main code stream is higher than the transmission frame rate of any original background code stream.

In an embodiment, the original main stream is the stream selected for playback in the receiving end, so that after the receiving end receives the image frame of each of the at least two streams, the original main stream is The image frames of the code stream are played, and the image frames in the original background code stream are buffered.

Correspondingly, in the transmission process of each bit stream, the transmission is performed according to the image frame. Therefore, the sending of at least two code streams referred to in step 410 does not mean that the image frames in the at least two code streams have been sent, but it is known that the sending end is in the process of sending image frames.

Step 430: Obtain a frame rate adjustment signal, where the frame rate adjustment signal carries the code stream identifier of the second code stream.

Therefore, the transmitting end can learn that the second code stream is switched to the main code stream according to the code stream identifier of the second code stream carried in the frame rate adjustment signal.

In one embodiment, the frame rate adjustment signal sent by the receiving end according to the acquired main stream switching signal is received to obtain the frame rate adjustment signal. The main stream switching signal is used to instruct to switch the main stream to the second code. flow. In other words, after the receiving end obtains the main code stream switching signal, the frame rate adjustment signal is correspondingly sent to the transmitting end, thereby correspondingly adjusting the transmission frame rate of each of the at least two code streams.

In another embodiment, the frame rate adjustment signal may also be generated by an operation triggered on the transmitting end, that is, switching the main stream at the transmitting end.

Step 450: After adjusting the frame rate according to the frame rate adjustment signal, continue to send at least two streams according to the adjusted transmission frame rate, where the second stream is used as the new main stream, and the second stream is divided by the at least two streams. Other code streams outside the code stream are used as the new background code stream. After the frame rate is adjusted, the transmission frame rate of the new main code stream is higher than the transmission frame rate of any new background code stream.

In one embodiment, after the frame rate is adjusted, the transmission frame rate of the new background code stream is the transmission frame rate of the second code stream before switching, or is a dynamic low frame rate adapted to the remaining bandwidth. The remaining bandwidth is based on the The transmission frame rate of the new main stream is determined.

In one embodiment, after the frame rate is adjusted, the transmission frame rate of the new main stream is the transmission frame rate of the original main stream before switching.

In an embodiment, as shown in FIG. 5, the method further includes:

Step 510: Before sending the image frames of each code stream in the at least two code streams, obtain a reference frame determined for the original image to be encoded in the corresponding code stream.

Step 530: Encode the original image according to the reference frame to obtain an image frame of the original image.

In this embodiment, the inter-frame coding method is used to code the original images in each bit stream.

In one embodiment, step 510 includes:

Select the image frame from the image frame obtained by encoding in the corresponding code stream as the reference frame of the original image.

In a specific embodiment, the transmitting end configures a reference frame management queue corresponding to each code stream. The reference frame management queue is used to store the image frames accurately received by the receiving end in the corresponding code stream, so as to ensure the reference frame used For the image frame accurately received in the receiving end, the steps of selecting the image frame from the image frame obtained by encoding in the corresponding code stream as the reference frame of the original image include:

Select an image frame from the reference frame management queue of the corresponding code stream as the reference frame of the original image.

In an embodiment, the method further includes:

The feedback information sent by the receiving end is received. When the receiving end determines that the received image frame is accurate through anomaly detection, it sends feedback information for the accurately received image frame. The feedback information is used to determine a reference frame for the original image to be encoded.

Therefore, the sending end can determine which image frames are accurately received by the receiving end according to the received feedback information. In a specific embodiment, the feedback information carries the frame number of the corresponding image frame. Therefore, when the sending end receives the feedback information, the corresponding image frame corresponding to the frame sequence number is saved in the reference frame management queue of the corresponding code stream according to the carried frame sequence number.

In a specific embodiment, the receiving end configures a reference frame management queue for each bit stream. When the receiving end determines that the received image frame is accurate through abnormal detection, the image frame and the corresponding frame sequence number are saved to the corresponding code The flow's reference frame management queue.

In an embodiment, after the step of receiving the feedback information sent by the receiving end, the method further includes:

Detect whether the frame sequence numbers carried in the feedback information in the corresponding code streams received twice adjacently are continuous.

If it is continuous, the image frame corresponding to the adjacent data of the original image is determined as the reference frame of the original image.

If it is not continuous, then select the image frame from the reference frame management queue of the corresponding code stream as the reference frame of the original image.

For encoding, the encoding efficiency of using the image frame corresponding to the adjacent data of the original image to be encoded is higher. However, if the image frame corresponding to the adjacent data is not accurately received by the receiving end, it may cause the The image frame corresponding to the adjacent data can not be decoded correctly in the receiving end.

In the technical solution of this embodiment, it is realized that the reference frame for the original image to be encoded is determined according to the feedback information. Specifically, when the frame numbers carried in the detection feedback information are continuous, it means that there is no image frame with data errors or lost in the current receiving end, so that the adjacent data of the original image to be encoded corresponds to the image The frame is determined as the reference frame of the original image, thereby improving the coding efficiency; and when the frame number carried in the feedback information is detected as discontinuous, it means that the image frames between the discontinuous frame numbers are abnormal image frames. The image frame is selected from the reference frame management queue of the corresponding code stream as the reference frame of the original image to ensure that the reference frame used is the image frame accurately received at the receiving end.

In an embodiment, the method further includes:

For at least two streams, the original image of the first frame in each stream is encoded and transmitted according to the intra-frame encoding method.

Further, in an embodiment, after the original image of the first frame in each code stream is encoded and transmitted according to the intra-frame encoding method, as shown in FIG. 6, the method further includes:

Step 610: Receive feedback information sent by the receiving end for the received image frame.

Step 630: Determine whether abnormal transmission occurs in the corresponding code stream through the feedback information.

In one embodiment, as described above, when the receiving end determines the correct image frame of the received image frame through anomaly detection, feedback information is sent to the transmitting end for the accurately received image frame, wherein the feedback information carries the The frame number of the accurately received image frame. Therefore, the sender determines whether there is abnormal transmission in the corresponding code stream by detecting whether the frame sequence number carried in the feedback information received twice in the same code stream is adjacent, that is, if it is continuous, it means that there is no abnormal transmission; if it is not continuous, then Indicates abnormal transmission occurred.

In another embodiment, the receiving end may also send feedback information for the abnormal image frame to the transmitting end after the abnormal image frame is determined by the abnormality detection. Correspondingly, the feedback information carries the frame number of the corresponding abnormal image frame , The abnormal image frame includes the image frame with data error and the image frame with loss. Therefore, after receiving the feedback information, the receiving end can determine that abnormal transmission has occurred in the corresponding code stream, and correspondingly determine the abnormal image frame according to the frame sequence number in the feedback information.

Step 650: If it is determined that abnormal transmission occurs in the corresponding code stream, the original image currently to be transmitted in the code stream is encoded and transmitted according to the intra-frame coding mode.

Step 670: If it is determined that no abnormal transmission occurs in the corresponding code stream, the original image currently to be transmitted is coded and transmitted according to the inter-frame coding mode.

In this embodiment, the first frame of the original image in each stream is encoded and transmitted according to the intra-frame encoding method, such as an encoded I frame, and the subsequent original images in the stream are determined according to whether abnormal transmission occurs during transmission. Coding mode, that is, when abnormal transmission occurs, the intra-frame coding method is used for coding and transmission, such as encoding to obtain I frame and transmitting; when there is no abnormal transmission, the inter-frame coding method is used for encoding and transmitting, for example, encoding to obtain P frame And transfer. Thus, on the one hand, the coding efficiency of the sending end is improved, and the influence of abnormal image frames on subsequent transmission of image frames is reduced.

The following are embodiments of the disclosed device. For details that are not disclosed in the device embodiment of the present disclosure, please refer to the embodiment of the transmission control method of the code stream of the present disclosure.

A code stream receiving device includes a processor, a communication component, a memory, and a communication bus. The memory stores executable instructions, and the processor is configured to perform the following steps:

Receive at least two code streams, at least two code streams including a first code stream and a second code stream, where the first code stream is used as the original main code stream, and at least two code streams other than the first code stream As the original background code stream, the transmission frame rate of the original main code stream is higher than the transmission frame rate of any original background code stream.

Acquire the main code stream switching signal, the main code stream switching signal indicates to switch the main code stream to the second code stream.

Continue to receive at least two streams after switching. The second stream is used as the new main stream, and at least two streams other than the second stream are used as the new background stream. After switching, the new main stream The transmission frame rate of the code stream is higher than the transmission frame rate of any new background code stream.

In an embodiment, after the step of receiving at least two bit streams, the processor further executes the following steps:

Play the original main stream and cache the original background stream.

In an embodiment, after the step of acquiring the main stream switching signal, the processor further executes the following steps:

In an embodiment, in the step of playing the second code stream as the new main code stream, the processor is configured to perform the following steps:

Obtain the buffered data buffered for the second code stream before switching.

Switch to play the second stream according to the buffered data.

In an embodiment, in the step of acquiring the main stream switching signal, the processor is configured to perform the following steps:

Generate the main stream switching signal according to the triggered switching operation.

Send a frame rate adjustment signal to the sending end, and the frame rate adjustment signal indicates to switch the main stream to the second stream;

In the step of continuing to receive at least two code streams after switching, the processor is configured to perform the following steps:

Receive at least two streams that continue to be transmitted after adjusting the frame rate according to the frame rate adjustment signal by the sender, and the sender adjusts the frame rate according to the transmission frame rate of the new main stream higher than the transmission frame rate of any new background stream .

In one embodiment, the transmission frame rate of the new background code stream after the switch is the transmission frame rate of the second code stream before the switch, or to adapt to the dynamic low frame rate of the remaining bandwidth, the remaining bandwidth is based on the new main stream after the switch. The transmission frame rate of the stream is determined.

In one embodiment, the transmission frame rate of the new main stream after switching is the transmission frame rate of the original main stream before switching.

In an embodiment, the processor is further configured to perform the following steps:

In an embodiment, after the step of performing abnormality detection on the received image frame, the processor further performs the following steps:

The specific operations performed by the processor of the receiving device of the code stream in the foregoing embodiments have been described in detail in the embodiment of the transmission control method for the code stream, and will not be elaborated here.

The present disclosure provides a code stream sending device, which includes a processor, a communication component, a memory, and a communication bus. The memory stores executable instructions, and the processor is configured to perform the following steps:

Send at least two code streams, at least two code streams including a first code stream and a second code stream, where the first code stream is used as the original main code stream, and at least two code streams other than the first code stream As the original background code stream, the transmission frame rate of the original main code stream is higher than the transmission frame rate of any original background code stream.

The frame rate adjustment signal is obtained, and the frame rate adjustment signal carries the code stream identifier of the second code stream.

After adjusting the frame rate according to the frame rate adjustment signal, continue to send at least two streams according to the adjusted transmission frame rate, where the second stream is used as the new main stream, and at least two streams except the second stream As the new background code stream, the transmission frame rate of the new main code stream is higher than the transmission frame rate of any new background code stream after the frame rate is adjusted.

In an embodiment, in the step of obtaining the frame rate adjustment signal, the processor is configured to perform the following steps:

The frame rate adjustment signal sent by the receiving end according to the acquired main code stream switching signal is received, and the main code stream switching signal is used to instruct to switch the main code stream to the second code stream.

Before sending the image frames of each code stream in at least two code streams, obtain the reference frame determined for the original image to be encoded in the corresponding code stream;

The original image is encoded according to the reference frame to obtain the image frame of the original image.

In an embodiment, in the step of obtaining the reference frame determined for the original image to be encoded in the corresponding bitstream, the processor is configured to perform the following steps:

In an embodiment, after the step of encoding and transmitting the first frame of the original image in each code stream according to the intra-frame encoding method, the processor is further configured to perform the following steps:

Receiving the feedback information sent by the receiving end for the received image frame;

Determine whether there is abnormal transmission in the corresponding code stream through the feedback information;

If it is determined that abnormal transmission occurs in the corresponding code stream, the original image currently to be transmitted in the code stream is encoded and transmitted according to the intra-frame coding method;

If it is determined that there is no abnormal transmission in the corresponding code stream, the original image currently to be transmitted is coded and transmitted according to the inter-frame coding mode.

The specific operations performed by the processor of the code stream sending device in the foregoing embodiments have been described in detail in the embodiment of the transmission control method for the code stream, which will not be elaborated here.

The following are embodiments of the apparatus of the present disclosure, which can be configured to execute the above-mentioned embodiment of the transmission control method of the code stream executed by the sender 100 or the receiver 300 of the present disclosure. For details not disclosed in the embodiment of the device of the present disclosure, please refer to the embodiment of the transmission control method of the code stream of the present disclosure.

A code stream transmission control device that should be configured as a receiving end, the device includes:

The first receiving module is configured to receive at least two code streams. The at least two code streams include a first code stream and a second code stream. The first code stream is used as the original main code stream. Other code streams outside the code stream are used as the original background code stream, and the transmission frame rate of the original main code stream is higher than the transmission frame rate of any original background code stream.

The main code stream signal acquisition module is configured to obtain the main code stream switching signal, and the main code stream switching signal indicates to switch the main code stream to the second code stream.

The second receiving module is configured to continue to receive at least two code streams after switching, where the second code stream is used as the new main code stream, and the other code streams except the second code stream among the at least two code streams are used as the new background code stream , After switching, the transmission frame rate of the new main stream is higher than the transmission frame rate of any new background stream.

In an embodiment, the device further includes:

The play and storage module is configured to play the original main stream and buffer the original background stream.

In an embodiment, the device further includes:

Switch the playback module, configured to play the second code stream as the new main code stream, and buffer the new background code stream.

In an embodiment, switching the playing module includes:

The buffer data acquisition unit is configured to acquire the buffer data buffered for the second code stream before switching.

The switching playing unit is configured to switch to playing the second code stream according to the buffered data.

In an embodiment, the main stream signal acquisition module includes:

The main stream signal generating unit is configured to generate the main stream switching signal according to the triggered switching operation.

In an embodiment, the device further includes:

The frame rate adjustment signal sending module is configured to send a frame rate adjustment signal to the sending end, and the frame rate adjustment signal indicates that the main stream is switched to the second stream.

In this embodiment, the second receiving module includes:

The receiving unit is configured to receive at least two streams that the transmitter continues to transmit after adjusting the frame rate according to the frame rate adjustment signal, and the transmitter according to the transmission frame rate of the new main stream is higher than any transmission frame of the new background stream Adjust the frame rate.

In an embodiment, the device further includes:

The anomaly detection module is configured to perform anomaly detection on the received image frames for the image frames of each of the received at least two code streams.

In an embodiment, the device further includes:

The feedback information sending module is configured to send feedback information for the accurately received image frame to the sending end when it is confirmed that the received image frame is accurate through anomaly detection, and the feedback information carries the frame number of the image frame.

For the implementation process of the functions and roles of each module/unit in the above-mentioned device, refer to the implementation process of the corresponding steps in the above-mentioned code stream transmission control method for details, which will not be repeated here.

It can be understood that these modules can be implemented by hardware, software, or a combination of both. When implemented in hardware, these modules may be implemented as one or more hardware modules, such as one or more application specific integrated circuits. When implemented in software, these modules can be implemented as one or more computer programs executed on one or more processors.

In another embodiment, a code stream transmission control device should be configured as a code stream transmitter, and the device includes:

The first sending module is configured to send at least two code streams. The at least two code streams include a first code stream and a second code stream. The first code stream is used as the original main code stream. Other code streams outside the code stream are used as the original background code stream, and the transmission frame rate of the original main code stream is higher than the transmission frame rate of any original background code stream.

The frame rate adjustment signal obtaining module is configured to obtain the frame rate adjustment signal, and the frame rate adjustment signal carries the code stream identifier of the second code stream.

The second sending module is configured to, after adjusting the frame rate according to the frame rate adjustment signal, continue to send at least two streams according to the adjusted transmission frame rate, where the second stream is used as the new main stream, and at least two streams The other code streams except the second code stream are used as the new background code stream. After the frame rate is adjusted, the transmission frame rate of the new main code stream is higher than the transmission frame rate of any new background code stream.

In an embodiment, the frame rate adjustment signal acquisition module includes:

The frame rate adjustment signal receiving unit is configured to receive the frame rate adjustment signal sent by the receiving end according to the acquired main stream switching signal, and the main stream switching signal is configured to instruct to switch the main stream to the second stream.

In an embodiment, the device further includes:

The reference frame obtaining module is configured to obtain the reference frame determined for the original image to be encoded in the corresponding code stream before sending the image frame of each code stream in the at least two code streams.

The encoding module is configured to encode the original image according to the reference frame to obtain the image frame of the original image.

In an embodiment, the reference frame acquisition module includes:

The first reference frame acquisition unit is configured to select an image frame from the image frames obtained by encoding in the corresponding code stream as the reference frame of the original image.

In an embodiment, the device further includes:

The feedback information receiving module is configured to receive the feedback information sent by the receiving end. When the receiving end determines that the received image frame is accurate through abnormal detection, it sends feedback information for the accurately received image frame. The feedback information is configured as the original to be encoded The image determines the reference frame.

In an embodiment, the device further includes:

The first encoding transmission module is configured to encode and transmit the original image of the first frame in each code stream according to the intra-frame encoding method for at least two code streams.

In an embodiment, the device further includes:

The information receiving module is configured to receive the feedback information sent by the receiving end for the received image frame;

The abnormal transmission determination module is configured to determine whether abnormal transmission occurs in the corresponding code stream through feedback information;

The second encoding transmission module is configured to, if the abnormal transmission determining module determines that abnormal transmission occurs in the corresponding code stream, the original image currently to be transmitted in the code stream is encoded and transmitted according to the intra-frame encoding method;

The third encoding transmission module is configured to encode and transmit the original image currently to be transmitted according to the inter-frame encoding method if the abnormal transmission determining module determines that no abnormal transmission occurs in the corresponding code stream.

Fig. 7 is a hardware block diagram of a UAV according to an exemplary embodiment, wherein the UAV 700 can be used as a code stream transmitter to correspondingly execute the code stream transmission control method of the present disclosure.

It should be noted that the drone 700 is only an example adapted to the present disclosure, and cannot be considered as providing any limitation of the scope of use of the present disclosure. The unmanned aerial vehicle 700 cannot be considered to be dependent on or must have one or more components of the exemplary unmanned aerial vehicle 700 shown in FIG. 7.

7, the drone 700 may include one or more of the following components: a processing component 702, a memory 704, a power component 706, an image acquisition component 708, a positioning component 710, a sensor component 714, and a communication component 716.

The processing component 702 generally controls the overall operations of the UAV 700, such as operations associated with code stream transmission, communication, data storage, and original image encoding. The processing component 702 may include one or more processors 718 to execute instructions to complete all or part of the steps of the following method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components.

The memory 704 is configured to store various types of data to support the operation of the drone 700. Examples of such data include instructions for any application or method configured to operate on drone 700. The memory 704 can be implemented by any type of volatile or non-volatile storage devices or their combination, such as static random access memory (Static Random Access Memory, SRAM for short), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-only memory ( Read-Only Memory, ROM for short), magnetic memory, flash memory, disk, etc. The memory 704 also stores one or more modules, and the one or more modules are configured to be executed by the one or more processors 718 to complete all or part of any of the foregoing method embodiments executed by the sending end step.

The power supply component 712 provides power for various components of the drone 700. The power supply component 712 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the drone 700.

The power assembly 706 is configured to provide power for the flight of the drone 700. The power assembly 706 may include a power motor that drives the rotor to rotate.

The image acquisition component 708 is configured for the UAV 700 to perform image or video acquisition, so as to encode a video stream. The image acquisition component 708 may be at least one camera mounted on the drone 700.

The positioning component 710 is configured to obtain the position of the UAV 700 in real time. The positioning component 710 may be a GPS positioning module, a low-orbit satellite module, or the like.

The sensor component 714 includes one or more sensors configured to provide the drone 700 with various aspects of status assessment. For example, the sensor component 714 can detect the take-off and stop status of the drone 700, the relative positioning of the components, and the sensor component 714 can also detect the position change of the drone 700 or a component of the drone 700 and the temperature of the drone 700 Variety. In some embodiments, the sensor component 714 may also include a magnetic sensor, a pressure sensor or a temperature sensor, an acceleration sensor, an angular velocity sensor, a distance sensor, etc.

The communication component 716 is configured to facilitate wired or wireless communication between the drone 700 and other devices. The UAV 700 can be connected to a wireless network based on a communication standard, such as WiFi (WIreless-Fidelity, wireless fidelity). In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 further includes a near field communication (Near Field Communication, NFC for short) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth technology and other technologies. .

In an exemplary embodiment, a computer-readable storage medium is also provided. The computer-readable storage medium may be a temporary and non-transitory computer-readable storage medium including instructions. The computer-readable storage medium stores a computer The program, when the computer program is executed by the processor, realizes the transmission control method of the code stream in any method embodiment executed by the receiving end.

In an exemplary embodiment, a computer-readable storage medium is also provided. The computer-readable storage medium may be a temporary and non-transitory computer-readable storage medium including instructions. The computer-readable storage medium stores a computer The program, when the computer program is executed by the processor, realizes the transmission control method of the code stream in any method embodiment executed by the sending end.

It should be understood that the present invention is not limited to the precise structure that has been described above and shown in the drawings, and various modifications and changes can be performed without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

A transmission control method of a code stream, the method comprising:

At least two code streams are received, the at least two code streams include a first code stream and a second code stream, wherein the first code stream is used as the original main code stream, and the first code stream is excluded from the at least two code streams Other code streams outside the stream are used as the original background code stream, and the transmission frame rate of the original main code stream is higher than the transmission frame rate of any one of the original background code streams;

Acquiring a main code stream switching signal, where the main code stream switching signal indicates to switch the main code stream to the second code stream;

Continue to receive the at least two code streams after switching, wherein the second code stream is used as the new main code stream, and the other code streams except the second code stream among the at least two code streams are used as the new background code stream, After the switching, the transmission frame rate of the new main code stream is higher than the transmission frame rate of any one of the new background code streams.
The method according to claim 1, wherein after the receiving at least two code streams, the method further comprises:

Play the original main code stream, and buffer the original background code stream.
The method according to claim 2, wherein after said obtaining the main stream switching signal, the method further comprises:

Playing the second code stream as the new main code stream, and buffering the new background code stream.
The method according to claim 1, wherein the playing the second code stream as the new main code stream comprises:

Acquiring the buffered data buffered for the second code stream before switching;

Switch to playing the second code stream according to the buffered data.
The method according to claim 1, wherein said obtaining the main stream switching signal comprises:

The main stream switching signal is generated according to the triggered switching operation.
The method according to claim 1, wherein after said acquiring the main stream switching signal, the method further comprises:

Sending a frame rate adjustment signal to the sending end, where the frame rate adjustment signal instructs to switch the main code stream to the second code stream;

The continuing to receive the at least two code streams after the handover includes:

Receiving the at least two streams that are continued to be transmitted after the transmitting end performs frame rate adjustment according to the frame rate adjustment signal, and the transmitting end according to the new main stream has a higher transmission frame rate than any one of the The frame rate adjustment is performed on the transmission frame rate of the new background code stream.
The method according to claim 1, wherein the transmission frame rate of the new background code stream after switching is the transmission frame rate of the second code stream before switching, or is a dynamic low frame rate adapted to the remaining bandwidth The remaining bandwidth is determined according to the transmission frame rate of the new main stream after switching.
The method according to claim 1, wherein the transmission frame rate of the new main stream after the switching is performed is the transmission frame rate of the original main stream before the switching.
The method according to claim 1, wherein the method further comprises:

For the received image frames of each of the at least two code streams, abnormality detection is performed on the received image frames.
The method according to claim 9, wherein, after the abnormality detection is performed on the received image frame, the method further comprises:

When it is confirmed through the abnormality detection that the received image frame is accurate, feedback information is sent to the sending end for the accurately received image frame, and the feedback information carries the frame sequence number of the image frame.
A transmission control method of a code stream, the method comprising:

Send at least two code streams, the at least two code streams include a first code stream and a second code stream, wherein the first code stream is used as the original main code stream, and the first code stream is excluded from the at least two code streams Other code streams outside the stream are used as the original background code stream, and the transmission frame rate of the original main code stream is higher than the transmission frame rate of any one of the original background code streams;

Acquiring a frame rate adjustment signal, where the frame rate adjustment signal carries the code stream identifier of the second code stream;

After adjusting the frame rate according to the frame rate adjustment signal, continue to send the at least two code streams according to the adjusted transmission frame rate, wherein the second code stream is used as the new main code stream, and the at least two code streams are Other code streams in the stream except the second code stream are used as the new background code stream. After the frame rate is adjusted, the transmission frame rate of the new main code stream is higher than the transmission frame rate of any one of the new background code streams.
The method according to claim 11, wherein said obtaining the frame rate adjustment signal comprises:

The frame rate adjustment signal sent by the receiving end according to the acquired main stream switching signal is received, where the main stream switching signal is used to instruct to switch the main stream to the second stream.
The method according to claim 11, wherein after the frame rate is adjusted, the transmission frame rate of the new background code stream is the transmission frame rate of the second code stream before switching, or is adapted to the dynamic low of the remaining bandwidth Frame rate, the remaining bandwidth is determined according to the transmission frame rate of the new main stream after the switching.
The method according to claim 11, wherein after the frame rate is adjusted, the transmission frame rate of the new main stream is the transmission frame rate of the original main stream before the switching.
The method according to claim 11, wherein the method further comprises:

Before sending the image frame of each code stream in the at least two code streams, acquiring the reference frame determined for the original image to be encoded in the corresponding code stream;

The original image is encoded according to the reference frame to obtain an image frame of the original image.
The method according to claim 11, wherein said obtaining the reference frame determined for the original image to be encoded in the corresponding bitstream comprises:

Selecting an image frame from the image frames obtained by encoding in the corresponding code stream as the reference frame of the original image.
The method according to claim 15, wherein the method further comprises:

Receiving the feedback information sent by the receiving end, when the receiving end determines that the received image frame is accurate through anomaly detection, it sends the feedback information for the accurately received image frame, and the feedback information is used for The original image to be encoded determines the reference frame.
The method according to claim 15, wherein the method further comprises:

For the at least two code streams, the original image of the first frame in each code stream is coded and transmitted in an intra-frame coding mode.
The method according to claim 18, wherein, after the original image of the first frame in each code stream is encoded and transmitted in an intra-frame coding manner, the method further comprises:

Receiving feedback information sent by the receiving end for the received image frame;

Determine whether abnormal transmission occurs in the corresponding code stream according to the feedback information;

If it is determined that abnormal transmission occurs in the corresponding code stream, the original image currently to be transmitted in the code stream is encoded and transmitted according to the intra-frame coding mode;

If it is determined that there is no abnormal transmission in the corresponding code stream, the original image currently to be transmitted is coded and transmitted according to the inter-frame coding mode.
A code stream receiving device includes a processor, a communication component, a memory, and a communication bus. The memory stores executable instructions, and the processor is configured to perform the following steps:

At least two code streams are received, the at least two code streams include a first code stream and a second code stream, wherein the first code stream is used as the original main code stream, and the first code stream is excluded from the at least two code streams Other code streams outside the stream are used as the original background code stream, and the transmission frame rate of the original main code stream is higher than the transmission frame rate of any one of the original background code streams;

Acquiring a main code stream switching signal, where the main code stream switching signal indicates to switch the main code stream to the second code stream;

Continue to receive the at least two code streams after switching, wherein the second code stream is used as the new main code stream, and the other code streams except the second code stream among the at least two code streams are used as the new background code stream, After the switching, the transmission frame rate of the new main code stream is higher than the transmission frame rate of any one of the new background code streams.
The receiving device according to claim 20, wherein after the step of receiving at least two code streams, the processor further executes the following steps:

Play the original main code stream, and buffer the original background code stream.
The receiving device according to claim 21, wherein, after the step of acquiring the main stream switching signal, the processor further executes the following steps:

Playing the second code stream as the new main code stream, and buffering the new background code stream.
The receiving device according to claim 22, wherein, in the step of playing the second code stream as the new main code stream, the processor is configured to perform the following steps:

Acquiring the buffered data buffered for the second code stream before switching;

Switch to playing the second code stream according to the buffered data.
22. The receiving device according to claim 20, wherein, in the step of acquiring the main stream switching signal, the processor is configured to perform the following steps:

The main stream switching signal is generated according to the triggered switching operation.
The receiving device according to claim 20, wherein, after the step of obtaining the main stream switching signal, the processor further executes the following steps:

Sending a frame rate adjustment signal to the sending end, where the frame rate adjustment signal instructs to switch the main code stream to the second code stream;

In the step of continuing to receive the at least two code streams after switching, the processor is configured to perform the following steps:

Receiving the at least two streams that are continued to be transmitted after the transmitting end performs frame rate adjustment according to the frame rate adjustment signal, and the transmitting end according to the new main stream has a higher transmission frame rate than any one of the The frame rate adjustment is performed on the transmission frame rate of the new background code stream.
The receiving device according to claim 20, wherein the transmission frame rate of the new background code stream after the switching is the transmission frame rate of the second code stream before the switching, or is a dynamic low frame rate adapted to the remaining bandwidth The remaining bandwidth is determined according to the transmission frame rate of the new main stream after switching.
22. The receiving device according to claim 20, wherein the transmission frame rate of the new main stream after the switching is performed is the transmission frame rate of the original main stream before the switching.
The receiving device of claim 20, wherein the processor is further configured to perform the following steps:

For the received image frames of each of the at least two code streams, abnormality detection is performed on the received image frames.
The receiving device according to claim 28, wherein, after the step of performing abnormality detection on the received image frame, the processor further performs the following steps:

When it is confirmed through the abnormality detection that the received image frame is accurate, feedback information is sent to the sending end for the accurately received image frame, and the feedback information carries the frame sequence number of the image frame.
A code stream sending device includes a processor, a communication component, a memory, and a communication bus. The memory stores executable instructions, and the processor is configured to execute the following steps:

Send at least two code streams, the at least two code streams include a first code stream and a second code stream, wherein the first code stream is used as the original main code stream, and the first code stream is excluded from the at least two code streams Other code streams outside the stream are used as the original background code stream, and the transmission frame rate of the original main code stream is higher than the transmission frame rate of any one of the original background code streams;

Acquiring a frame rate adjustment signal, where the frame rate adjustment signal carries the code stream identifier of the second code stream;

After adjusting the frame rate according to the frame rate adjustment signal, continue to send the at least two code streams according to the adjusted transmission frame rate, wherein the second code stream is used as the new main code stream, and the at least two code streams are Other code streams in the stream except the second code stream are used as the new background code stream. After the frame rate is adjusted, the transmission frame rate of the new main code stream is higher than the transmission frame rate of any one of the new background code streams.
The transmitting device according to claim 30, wherein, in the step of acquiring the frame rate adjustment signal, the processor is configured to perform the following steps:

The frame rate adjustment signal sent by the receiving end according to the acquired main stream switching signal is received, where the main stream switching signal is used to instruct to switch the main stream to the second stream.
The sending device according to claim 30, wherein after the frame rate is adjusted, the transmission frame rate of the new background code stream is the transmission frame rate of the second code stream before switching, or is a dynamic response to the remaining bandwidth Low frame rate, the remaining bandwidth is determined according to the transmission frame rate of the new main stream after switching.
The sending device according to claim 30, wherein after the frame rate is adjusted, the transmission frame rate of the new main stream is the transmission frame rate of the original main stream before the switching.
The sending device of claim 30, wherein the processor is further configured to perform the following steps:

Before sending the image frame of each code stream in the at least two code streams, acquiring the reference frame determined for the original image to be encoded in the corresponding code stream;

The original image is encoded according to the reference frame to obtain an image frame of the original image.
The sending device according to claim 30, wherein, in the step of obtaining the reference frame determined for the original image to be encoded in the corresponding code stream, the processor is configured to perform the following steps:

Selecting an image frame from the image frames obtained by encoding in the corresponding code stream as the reference frame of the original image.
The sending device of claim 34, wherein the processor is further configured to perform the following steps:

Receiving the feedback information sent by the receiving end, when the receiving end determines that the received image frame is accurate through anomaly detection, it sends the feedback information for the accurately received image frame, and the feedback information is used for The original image to be encoded determines the reference frame.
The sending device of claim 34, wherein the processor is further configured to perform the following steps:

For the at least two code streams, the original image of the first frame in each code stream is coded and transmitted in an intra-frame coding mode.
The sending device according to claim 37, wherein, after the step of encoding and transmitting the original image of the first frame in each code stream according to an intra-frame encoding method, the processor is further configured to perform the following steps:

Receiving feedback information sent by the receiving end for the received image frame;

Determine whether abnormal transmission occurs in the corresponding code stream according to the feedback information;

If it is determined that abnormal transmission occurs in the corresponding code stream, the original image currently to be transmitted in the code stream is encoded and transmitted according to the intra-frame coding mode;

If it is determined that there is no abnormal transmission in the corresponding code stream, the original image currently to be transmitted is coded and transmitted according to the inter-frame coding mode.
A computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.
A computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 11 to 19 is realized.