WO2020258296A1

WO2020258296A1 - Image processing method, device, unmanned aerial vehicle, and receiving end

Info

Publication number: WO2020258296A1
Application number: PCT/CN2019/093862
Authority: WO
Inventors: 马宁; 朱磊; 陈颖; 赵亮
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2019-06-28
Filing date: 2019-06-28
Publication date: 2020-12-30
Also published as: CN111713107A

Abstract

An image processing method, a device, an unmanned aerial vehicle, and a receiving end. The method comprises: obtaining the current image frame; detecting the communication quality of a wireless uplink communication link between a transmitting end and a receiving end; and determining, according to the communication quality, whether to determine, according to received feedback information, a reference frame used for encoding the current image frame, wherein the feedback information is sent by the receiving end to the transmitting end by means of the wireless uplink communication link, and is used for indicating the transmission status of an image frame sent by the transmitting end to the receiving end before the feedback information. According to the method, it is determined, according to the communication quality of the wireless uplink communication link, whether to determine, according to received feedback information, a reference frame used for encoding the current image frame. In this way, the encoding mode of an image matches the communication quality of the wireless uplink communication link, so that an image transmission error can be recovered in time to avoid affecting the user experience.

Description

Image processing method, equipment, unmanned aerial vehicle and receiving end

Technical field

The embodiments of the present application relate to image processing technology, in particular to an image processing method, equipment, unmanned aerial vehicle, and receiving end.

Background technique

Low-latency video transmission systems on wireless and unreliable channels are currently a hot research and application direction. For video transmission on unreliable channels, data errors may occur during data transmission, resulting in video decoding errors. However, the sender cannot predict when a data error occurs, so a corresponding error recovery mechanism is needed to correct the video data error that has occurred.

Commonly used error recovery mechanisms include feedback coding mode and feedback coding mode. Among them, the feedback-free encoding method can realize encoding without the feedback information sent by the receiving end, and the feedback encoding method adopts encoding based on the feedback information sent by the receiving end.

In the existing encoding mechanisms, the image is often encoded based on one of the feedback-free encoding mode and the feedback encoding mode. However, neither the feedback coding mode nor the feedback coding mode is suitable for real-time changing application scenarios. In some application scenarios, when the image is transmitted incorrectly, it may increase the image recovery delay and affect the user experience. .

Summary of the invention

The embodiments of the present application provide an image processing method, equipment, unmanned aerial vehicle, and receiving end to improve the flexibility of image encoding and image error recovery.

In the first aspect, an embodiment of the present application provides an image processing method at the sending end, including:

Get the current image frame;

Detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end;

It is determined according to the communication quality whether to determine the reference frame used to encode the current image frame according to the received feedback information, wherein the feedback information is transmitted from the receiving end to the transmitting end via the wireless uplink communication link. If sent, the feedback information is used to indicate the transmission status of the image frame sent by the sending end to the receiving end before the feedback information.

In the second aspect, an embodiment of the present application provides an image processing method in a receiving end, including:

Determine whether to send feedback information through the wireless uplink communication link according to the communication quality, where the feedback information is used to indicate the transmission status of the image frame sent by the sending end to the receiving end before the feedback information And instruct the sending end to determine the reference frame used when encoding the current image frame according to the feedback information.

In a third aspect, an embodiment of the present application provides an image processing device at a sending end, including a memory and a processor, where:

The memory is used to store program instructions;

The processor is configured to execute the program instructions, and when the program instructions are executed, the processor executes the following steps:

Get the current image frame;

In a fourth aspect, an embodiment of the present application provides an unmanned aerial vehicle, which includes the image processing equipment at the transmitting end of the above third aspect and various possible designs of the third aspect.

In a fifth aspect, an embodiment of the present application provides an image processing device in a receiving end, including a memory and a processor, where:

The memory is used to store program instructions;

In a sixth aspect, an embodiment of the present application provides a receiving end, including the image processing device in the receiving end of the above fifth aspect and various possible designs of the fifth aspect.

In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium having program instructions stored in the computer-readable storage medium. When the program instructions are executed by a processor, each of the above first aspect and the first aspect is implemented. A possible design of the image processing method at the sending end.

In an eighth aspect, the embodiments of the present application also provide another computer-readable storage medium. The computer-readable storage medium stores program instructions. When the processor executes the program instructions, the above second aspect and second aspect are implemented. In terms of various possible designs, the image processing method in the receiving end is described.

According to the image processing method, equipment, unmanned aerial vehicle, and receiving end provided by the embodiments of the present application, the transmitting end of the method obtains the current image frame, detects the communication quality of the wireless uplink communication link between the transmitting end and the receiving end, and determines according to the communication quality Whether to determine a reference frame for encoding the current image frame according to the received feedback information. That is, according to the communication quality of the wireless uplink communication link, it is adaptively determined whether to use the feedback coding mode, so that the coding mode of the image matches the communication quality of the wireless uplink communication link, so that the image transmission error can be recovered in time and the user experience is improved.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

FIG. 1 is a schematic diagram of the architecture of an image processing system provided by an embodiment of the application;

Figure 2 is a schematic diagram of a feedback-free coding mode provided by an embodiment of the application;

FIG. 3 is a schematic diagram of another non-feedback coding mode provided by an embodiment of the application;

4 is a schematic flowchart of an image processing method provided by an embodiment of the application;

FIG. 5 is a schematic diagram of reference frame selection provided by an embodiment of the application;

6 is a schematic flowchart of another image processing method provided by an embodiment of the application;

FIG. 7 is a schematic flowchart of still another image processing method provided by an embodiment of the application;

FIG. 8 is a schematic structural diagram of an image processing device at the sending end according to an embodiment of the application;

FIG. 9 is a schematic structural diagram of an image processing device at a receiving end according to an embodiment of the application;

FIG. 10 is a schematic diagram of the hardware structure of the image processing device at the sending end according to an embodiment of the application;

FIG. 11 is a schematic diagram of the hardware structure of the image processing device at the receiving end according to an embodiment of the application;

FIG. 12 is a schematic structural diagram of an unmanned aerial vehicle provided by an embodiment of the application;

FIG. 13 is a schematic structural diagram of a receiving end provided by an embodiment of the application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or a central component may also exist. When a component is considered to be "connected" to another component, it can be directly connected to another component or there may be a centered component at the same time.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of the present invention. The terms used in the description of the present invention herein are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The term "and/or" as used herein includes any and all combinations of one or more related listed items.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

FIG. 1 is a schematic diagram of the architecture of an image processing system provided by an embodiment of the application. As shown in FIG. 1, the system provided in this embodiment includes a sending end 101 and a receiving end 102. Among them, the sending end 101 sends an image to the receiving end 102 via a wireless downlink communication link, and the receiving end 102 sends information to the sending end 101 via a wireless uplink communication link.

Real-time image transmission has high requirements for image transmission delay. Traditional real-time image transmission has the problems of limited transmission bandwidth and large bandwidth fluctuations. In addition, wireless channels also have more uncertainties, especially the unreliable characteristics of wireless channels. In the process of data transmission, errors will inevitably occur, leading to video decoding errors. At this time, an error recovery mechanism is needed to correct video data transmission errors that have occurred. Commonly used error recovery mechanisms include two ways of feedback coding and feedback coding.

Before introducing the above-mentioned error recovery mechanisms without feedback and with feedback, the commonly used coding frame types are introduced first. Video coding standards widely used in the industry include H.263, H.264, H.265, MPEG4, etc., which provide two types of frame types:

I frame:

All image regions are coded according to the intra-frame prediction coding method, and a prediction compression method that does not depend on the reference frame is adopted.

Since there is no reference to time correlation for prediction, the compression rate is low, and the code rate of the frame after the generated encoding is high.

Advantages: Because it does not rely on other frames, the receiver can recover the error as long as the I frame is correctly received.

P/B frame:

The image block is coded according to the inter-frame prediction coding mode, and the prediction compression method that depends on the reference frame is adopted.

Since the reference frame is used for prediction, the compression rate is higher, and the code rate of the frame after the generated encoding is lower.

Disadvantage: Because of the dependence on the reference frame, there will be errors when the reference frame is received. Even if the receiving end receives the frame correctly, the frame image cannot be decoded correctly, that is, it cannot be restored.

Regarding the error recovery coding of real-time image transmission, it can be divided into the following two types:

1. No feedback coding mode

1. As shown in FIG. 2, the transmitting end determines the I frame to be encoded according to intra prediction at a fixed frame interval period, and all image regions in the I frame are coded according to the intra prediction encoding method. The images of the preset number of frames after the I frame can be inter-frame predictive coding, for example, P frames of the preset number of frames are obtained. The sending end can send the encoded image to the receiving end through the wireless downlink communication link.

2. The sending end periodically determines and sends multiple image frames including the I area to the sending end at a fixed frame interval, where the I area is an image area coded in the frame. The images of the preset number of frames behind the multiple image frames including the I area may be inter-frame predictive coding, for example, P frames of the preset number of frames are obtained. As shown in FIG. 3, the multiple image frames may include 3 frames, and each frame contains interlaced 1/3 of the frame using intra-frame coding. The sending end can send the encoded image to the receiving end through the wireless downlink communication link.

It can be seen from Figures 2 and 3 that the receiving end receives the image sent by the sending end. Since the image sent by the sending end includes a fixed frame interval I frame or multiple image frames including I area, when a certain frame is in accordance with the frame When the image received in the preset encoding mode is incorrect, the image received error of the next frame encoded in the preset encoding mode between frames can be decoded with reference to the corresponding I frame or multiple image frames including the I area. Get restored.

Advantages of no feedback coding mode:

The receiving end can recover the image transmission error based on the received I frame or multiple image frames including the I area, without relying on the feedback information from the receiving end to the sending end.

Disadvantages of this solution:

It is necessary to periodically generate I frames or multiple image frames including I regions, which will result in a lower image compression rate.

2. Feedback coding mode

The receiving end determines the transmission status of the image frame sent by the sending end, such as whether the image frame is received correctly, and indicates the feedback information of the transmission status of the image frame to the sending end, and the sending end determines the image frame correctly received by the receiving end according to the feedback information As the reference frame of the image currently to be encoded to encode the image currently to be encoded.

The advantages of this solution are:

There is no need to send I frames or multiple image frames including I areas, reducing the pressure on the wireless downlink communication link.

The disadvantages of this scheme are:

Because the receiving end needs to send feedback information to the sending end through the wireless uplink communication link, when the wireless uplink communication link is unreliable, that is, when the communication quality of the wireless uplink communication link is not high, the error recovery time may be longer.

In the existing encoding mechanisms, the image is often encoded based on one of the feedback-free encoding mode and the feedback encoding mode. However, since neither the feedback coding mode nor the feedback coding mode is suitable for real-time changing application scenarios, coding flexibility is poor, and a matching coding mode cannot be selected according to the current application scenario. When the image is transmitted incorrectly, it may cause the delay of image restoration to increase, which affects the user experience.

In order to solve this technical problem, this embodiment provides an image processing method, which adaptively determines whether the feedback coding mode needs to be used according to the communication quality of the wireless uplink communication link, so that the coding mode of the image matches the communication of the wireless uplink communication link. The quality enables timely recovery in case of image transmission errors and improves user experience.

Fig. 4 is a schematic flow chart of the image processing method at the sending end provided by an embodiment of the application. The execution subject of this embodiment may be the sending end in the embodiment shown in Fig. 1, for example, the image processing device in the sending end To execute, the image processing device may include one or more processors for executing the image processing method, and the transmitting end is used to transmit the encoded data of the image frame to the receiving end. As shown in Figure 4, the method includes:

S401: Acquire a current image frame.

Specifically, the receiving end may include a photographing device, wherein the image processing device may obtain the current image frame to be encoded from the photographing device.

S402: Detect the communication quality of the wireless uplink communication link between the sending end and the receiving end.

The image processing device can detect the communication quality of the wireless uplink communication link, and by detecting the wireless uplink communication quality, it can determine whether the feedback information sent by the receiving end can be received in a timely and accurate manner.

Optionally, the detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end includes: acquiring communication status information of the wireless uplink communication link, and determining the communication quality according to the communication status information. Specifically, the image processing device may collect the communication state information of the wireless uplink communication link or receive the communication state information sent by the receiving end through the wireless uplink communication link, and further, determine the communication state information according to the communication state information Communication quality.

Optionally, the communication status information includes one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength. For example, the communication status information includes bandwidth, and the larger the bandwidth, the higher the communication quality.

In addition, the communication status information may also include received signal strength (RSSI), signal-to-noise ratio (SIGNAL-NOISE RATIO, SNR), etc., channel capacity assessment, channel noise power, communication Judgment of the distance between the two parties and so on.

Optionally, the detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end includes: acquiring information sent by the receiving end for indicating the communication quality, and according to the indication of the communication quality The information determines the communication quality.

Specifically, the receiving end can detect the communication quality of the wireless uplink communication link by acquiring the communication status information of the wireless uplink communication link, and send the information indicating the communication quality to the sending end, and the sending end can pass The information indicating the communication device determines the communication quality.

S403. Determine, according to the communication quality, whether to determine a reference frame used to encode the current image frame according to the received feedback information, where the feedback information is transmitted from the receiving end to the wireless uplink communication link via the wireless uplink communication link. Sent by the sending end, the feedback information is used to indicate the transmission status of the image frame sent by the sending end to the receiving end before the feedback information.

It should be noted that the transmission status refers to whether there is an error in the transmission of the image frame. Specifically, it refers to whether data loss or data error occurs during the transmission of the image frame. Or, it can be understood as whether the image frame is correctly transmitted, and whether it is successfully decoded after the correct transmission.

Optionally, the determining according to the communication quality whether to determine the reference frame used for encoding the current image frame according to the received feedback information includes:

When the communication quality is higher than or equal to the preset communication quality threshold, the reference frame used for encoding the current image frame is determined according to the received feedback information.

When the communication quality is less than the preset communication quality threshold, one or more reference frames for encoding the current image frame are determined according to the preset frame number interval, wherein the reference frames include frame-based Inner coded image area.

Here, the preset communication quality threshold can be set according to actual conditions. Illustratively, the communication quality of the wireless uplink communication link between the sending end and the receiving end is expressed by bandwidth, bit error rate, signal-to-noise ratio, and received signal strength. The quality threshold includes bandwidth threshold, bit error rate threshold, signal-to-noise ratio threshold, and received signal strength threshold.

When the communication quality is higher than or equal to the preset communication quality threshold, it indicates that the communication quality of the wireless uplink communication link between the sending end and the receiving end is better. If the image processing device is working in no feedback coding, switch to the feedback coding mode, according to The received feedback information determines the reference frame used to encode the current image frame, otherwise the existing mode remains unchanged.

When the communication quality is less than the preset communication quality threshold, it indicates that the communication quality of the wireless uplink communication link between the sending end and the receiving end is poor. If the image processing device is working in the feedback coding mode, switch the device to the no feedback coding mode Determine one or more reference frames for encoding the current image frame according to the preset frame number interval; otherwise, keep the existing mode unchanged. Wherein, when one or more reference frames are a reference frame, the one reference frame is an I frame as shown in FIG. 2, that is, all image areas in the reference frame are image areas based on intra-frame coding; When one or more reference frames are multi-frame reference frames, the multi-frame reference frames are a group of image frames as shown in FIG. 3, that is, the partial image area in each reference frame in the multi-frame reference frames includes Intra-encoded image area.

In addition, in order to avoid too frequent mode switching, a certain protection time can be added. For example, after switching to A mode, it is not allowed to switch to B mode within N time. Among them, the A mode is a feedback coding mode and the B mode is a feedback coding mode, or the A mode is a feedback coding mode and the B mode is a feedback coding mode. N time can be set according to the actual situation. Here, you can also set dual thresholds: after switching to A mode, it is not allowed to switch to B mode within N1 time; after switching to B mode, it is not allowed to switch to A mode within N2 time to meet the needs of multiple application scenarios.

Optionally, the determining a reference frame for encoding the current image frame according to the feedback information includes:

Determine the image frame that has been correctly transmitted according to the feedback information;

The reference frame is determined according to the image frame that has been correctly transmitted.

Optionally, the feedback information includes at least one of the following information:

Indication information used to indicate whether the last image frame sent before the feedback information is correctly transmitted;

The frame number of the image frame that has been correctly transmitted; and

The frame number of the last image frame that was correctly transmitted before the feedback information.

Among them, the frame number of the image frame is globally unique, and the frame number of the image frame and the encoded data are sent to the receiving end together.

Exemplarily, as shown in Figure 5, suppose that the sending end has sent 5 image frames to the receiving end before the current image frame (number 6), the corresponding numbers are 1-5, and the

numbers

2, 3, and 4 The image frame is transmitted correctly. The feedback information includes the frame number of the image frame that has been correctly transmitted. At this time, the feedback information may specifically be "234". Alternatively, the feedback information includes the frame number of the last image frame that has been correctly transmitted. At this time, the feedback information can be "4", and the image processing device at the sending end can know that the image frame number 5 has not been Successfully transferred. Therefore, when the image processing device at the sending end encodes the image frame number 6, it can select the image frame number 4 as the reference frame, or the image frames

number

3 and 4 as the reference frame.

Or, suppose that before the current image frame (number 9), the sending end has sent 8 image frames to the receiving end, the corresponding numbers are 1-8, and the receiving end has feedback on the image frames numbered 1-5 . In this feedback information, only the transmission state of the image frame number 8 needs to be fed back. The feedback information includes an indicator bit for indicating whether the image frame number 8 is correctly transmitted. Assuming that the value of the indicator bit is "0", it means that the image frame number 8 is not transmitted correctly, and the value of the indicator bit is "1". The image frame number 8 has been transmitted correctly. If the value of the indicator bit is 1, the image processing device at the sending end selects the image frame numbered 8 as the reference frame, and performs inter-frame coding on the image frame numbered 9. If the value of the indicator bit is 0, the image processing device at the sending end can determine whether the image frame number 5 is transmitted correctly, if the image frame number 5 is transmitted correctly, the image frame number 5 is selected as the reference frame, Perform inter-frame coding on the image frame numbered 9.

Or, further, the image processing device in the sending end saves at least a part of the image frames that have been correctly transmitted in the reference frame management queue, and the image processing device in the sending end refers to the image frames in the frame management queue to determine For the reference frame. Therefore, each time the feedback information sent by the receiving end is received, the image frame that has been correctly transmitted is determined according to the feedback information, and the reference frame management queue is updated. Correspondingly, after determining the image frame that has been correctly transmitted, the image processing device in the receiving end also saves at least a part of the correctly transmitted image frame in the reference frame management queue. The reference frame management queue at the sending end is the same as the information stored in the reference frame management queue at the receiving end.

Alternatively, the image processing device at the sending end includes a physical or logical buffer for storing all image frames corresponding to the encoded data, and the image processing device at the sending end may only store at least a part of the image frames that have been correctly transmitted The corresponding frame number is saved in the reference frame management queue, and the image processing device in the sending end determines the image frame corresponding to the frame number in the reference frame queue as the reference frame. Correspondingly, the image processing device in the receiving end includes a physical or logical buffer for storing all image frames corresponding to the decoded data, and the image processing device in the receiving end can correspond to at least a part of the image frames that are correctly transmitted The frame number is stored in the reference frame management queue.

The transmitter in the embodiments of the present application can move in any suitable environment, for example, in the air (for example, a fixed-wing aircraft, a rotary-wing aircraft, or an aircraft without fixed wings and no rotors), water (for example, a ship or a submarine) ), on land (for example, a car or train), space (for example, a space plane, satellite or probe), and any combination of the above various environments. The sending end may be an unmanned aerial vehicle, such as a UAV (Unmanned Aerial Vehicle, UAV for short). In some embodiments, the sending end may carry living bodies, such as humans or animals.

The receiving end in the embodiment of the present application may be a computer, a handheld electronic device, a communication device, a video monitoring device, etc.

In the image processing method at the transmitting end provided in this embodiment, the transmitting end detects the communication quality of the wireless uplink communication link between the transmitting end and the receiving end by acquiring the current image frame, and determines whether to determine whether to use for encoding according to the received feedback information according to the communication quality. The reference frame of the current image frame. That is, according to the communication quality of the wireless uplink communication link, it is adaptively determined whether to use the feedback coding mode, which can ensure that the feedback coding mode is used when the wireless uplink communication link quality is good, and when the feedback link quality is poor, no feedback is used. The coding mode improves the flexibility of coding, so that the coding mode of the image matches the communication quality of the wireless uplink communication link, so that it can recover in time when the image is transmitted incorrectly and improve the user experience.

The image processing method according to the embodiment of the present application is described in detail above with reference to FIGS. 4 and 5 from the sending end side, and the image processing method according to another embodiment of the present invention will be described in detail below from the receiving end side with reference to FIG. 6. It should be understood that the interaction between the receiving end and the sending end and related characteristics and functions described on the sending end side correspond to the description on the receiving end side, and repeated descriptions are appropriately omitted for brevity.

FIG. 6 shows an image processing method according to another embodiment of the present invention. The method may be executed by an image processing device in the receiving end, and the image processing device may be various types of chips for image processing and image processing.器等。 As shown in Figure 6, the method includes:

S601: Detect the communication quality of the wireless uplink communication link between the sending end and the receiving end.

Optionally, the detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end includes:

Acquiring communication state information of the wireless uplink communication link;

The communication quality is determined according to the communication status information.

Optionally, the communication status information includes one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength.

Obtain the information indicating the communication quality sent by the sending end, and determine the communication quality according to the information indicating the communication quality.

S602. Determine whether to send feedback information through the wireless uplink communication link according to the communication quality, where the feedback information is used to indicate the size of the image frame sent by the sending end to the receiving end before the feedback information. Transmission status and instruct the sending end to determine the reference frame used when encoding the current image frame according to the feedback information.

Optionally, the determining whether to send feedback information through the wireless uplink communication link according to the communication quality includes:

When the communication quality is higher than or equal to a preset communication quality threshold, sending the feedback information through the wireless uplink communication link;

Otherwise, the feedback information is not sent.

Optionally, the feedback information is used to instruct the sending end to determine the reference frame used when encoding the current image frame according to the feedback information, including:

The feedback information is used to instruct the sending end to determine the image frame that has been correctly transmitted according to the feedback information; determine the reference frame according to the image frame that has been correctly transmitted.

The frame number of the image frame that has been correctly transmitted; and

Optionally, the sending end is an unmanned aerial vehicle.

In the image processing method at the receiving end provided in this embodiment, the receiving end detects the communication quality of the wireless uplink communication link between the sending end and the receiving end, and determines whether to send feedback information through the wireless uplink communication link according to the communication quality, wherein: The feedback information is used to indicate the transmission status of the image frame sent by the sending end to the receiving end before the feedback information and to instruct the sending end to use when determining to encode the current image frame according to the feedback information Reference frame. That is, according to the communication quality of the wireless uplink communication link, it is adaptively determined whether to send feedback information to the sending end, so that the sending end can determine whether the feedback coding mode needs to be used, so that the coding mode of the image matches the communication quality of the wireless uplink communication link, so that the When the image transmission error occurs, it can be restored in time to improve the user experience.

Fig. 7 shows an image processing method according to still another embodiment of the present invention. As shown in Fig. 7, the method includes:

S701. The sending end sends an image frame to the receiving end.

S702. The receiving end generates feedback information according to the transmission state of the image frame.

S703. The receiving end sends the foregoing feedback information to the sending end.

S704. The sending end detects the communication quality of the wireless uplink communication link between the sending end and the receiving end.

S705: When the communication quality is higher than or equal to the preset communication quality threshold, the sending end determines a reference frame for encoding the current image frame according to the foregoing feedback information.

Exemplarily, the sending end may further perform inter-frame encoding on the current image frame according to the aforementioned reference frame to generate encoded data, and send the aforementioned encoded data to the receiving end. The receiving end may perform inter-frame decoding on the encoded data according to the reference frame to generate decoded data, and display the current image frame according to the decoded data.

S706. When the communication quality is less than the preset communication quality threshold, the sending end determines one or more reference frames for encoding the current image frame according to the preset frame number interval, where the reference frames include intra-frame encoding Image area.

In the same way, the sending end may further perform intra-frame encoding on the current image frame according to the aforementioned reference frame to generate encoded data, and send the aforementioned encoded data to the receiving end. The receiving end may perform intra-frame decoding on the encoded data according to the reference frame to generate decoded data, and display the current image frame according to the decoded data.

It is understandable that the interaction between the receiving end and the sending end in the foregoing FIG. 7 and related characteristics, functions and other methods correspond to the description in the foregoing FIG. 4, and for the sake of brevity, details are not described herein again.

The image processing method according to the embodiment of the present application is described above, and the image processing device according to the embodiment of the present application will be described below. However, it should be understood that the image processing device described below can implement the above image processing method. In order to avoid repetition, the following will be described briefly.

FIG. 8 is a schematic structural diagram of an image processing device at the sending end provided by an embodiment of the application. For ease of description, only parts related to the embodiments of the present application are shown. As shown in FIG. 8, the image processing device 80 includes: an image frame acquisition module 801, a first quality detection module 802, and a reference frame determination module 803.

Among them, the image frame acquisition module 801 is used to acquire the current image frame.

The first quality detection module 802 is configured to detect the communication quality of the wireless uplink communication link between the sending end and the receiving end.

The reference frame determining module 803 is configured to determine, according to the communication quality, whether to determine a reference frame used to encode the current image frame according to received feedback information, wherein the feedback information is transmitted by the receiving end through the wireless uplink The feedback information is sent by the communication link to the sending end, and the feedback information is used to indicate the transmission status of the image frame sent by the sending end to the receiving end before the feedback information.

In a possible design, the reference frame determining module 803 is specifically configured to:

In a possible design, the reference frame determining module 803 determines the reference frame used to encode the current image frame according to the feedback information, including:

In a possible design, the first quality detection module 802 is specifically configured to:

In a possible design, the communication status information includes one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength.

Acquiring the information indicating the communication quality sent by the receiving end, and determining the communication quality according to the information indicating the communication quality.

In one possible design, the sending end is an unmanned aerial vehicle.

The device provided in the embodiment of the present application can be used to implement the technical solution of the method embodiment in FIG. 4, and its implementation principles and technical effects are similar, and the details of the embodiment of the present application are not repeated here.

FIG. 9 is a schematic structural diagram of an image processing device at a receiving end according to an embodiment of the application. For ease of description, only parts related to the embodiments of the present application are shown. As shown in FIG. 9, the image processing device 90 includes: a second quality detection module 901 and an information determination module 902.

Wherein, the second quality detection module 901 is configured to detect the communication quality of the wireless uplink communication link between the sending end and the receiving end;

The information determining module 902 is configured to determine whether to send feedback information through the wireless uplink communication link according to the communication quality, where the feedback information is used to indicate that the sending end sends the feedback information to the receiving end before the feedback information. The transmission status of the sent image frame and instruct the sending end to determine the reference frame used when encoding the current image frame according to the feedback information.

In a possible design, the information determining module 902 is specifically configured to:

Otherwise, the feedback information is not sent.

In a possible design, the feedback information includes at least one of the following information:

The frame number of the image frame that has been correctly transmitted; and

In a possible design, the second quality detection module 901 is specifically configured to:

In a possible design, the sending end is an unmanned aerial vehicle

The device provided in the embodiment of the present application can be used to implement the technical solution of the method embodiment in FIG. 6 above, and its implementation principles and technical effects are similar, and the details of the embodiment of the present application are not repeated here.

FIG. 10 is a schematic diagram of the hardware structure of the image processing device at the sending end according to an embodiment of the application. As shown in FIG. 10, the image processing device 100 of this embodiment includes: a processor 1001 and a memory 1002;

The memory 1002 is used to store program instructions;

The processor 1001 is configured to execute program instructions stored in the memory. When the program instructions are executed, the processor executes the following steps:

Get the current image frame;

Optionally, when the processor determines whether to determine a reference frame for encoding the current image frame according to the received feedback information according to the communication quality, it is specifically configured to:

Optionally, when the processor determines a reference frame for encoding the current image frame according to the feedback information, it is specifically configured to:

Optionally, when the processor detects the communication quality of the wireless uplink communication link between the sending end and the receiving end, it is specifically configured to:

Optionally, the sending end is an unmanned aerial vehicle.

Optionally, the memory 1002 may be independent or integrated with the processor 1001.

When the memory 1002 is independently provided, the image processing device further includes a bus 1003 for connecting the memory 1002 and the processor 1001.

FIG. 11 is a schematic diagram of the hardware structure of the image processing device at the sending end according to an embodiment of the application. As shown in FIG. 11, an embodiment of the present application further provides an image processing device 110 at the receiving end, including: a processor 1101 and a memory 1102; where

The memory 1102 is used to store program instructions;

The processor 1101 is configured to execute program instructions stored in the memory. When the program instructions are executed, the processor executes the following steps:

Optionally, when the processor determines whether to send feedback information through the wireless uplink communication link according to the communication quality, it is specifically configured to:

Otherwise, the feedback information is not sent.

The frame number of the image frame that has been correctly transmitted; and

Optionally, the sending end is an unmanned aerial vehicle.

Optionally, the memory 1102 may be independent or integrated with the processor 1101.

When the memory 1102 is independently provided, the image processing device further includes a bus 1103 for connecting the memory 1102 and the processor 1101.

Fig. 12 is a schematic structural diagram of an unmanned aerial vehicle according to an embodiment of the present application. As shown in FIG. 12, the UAV 120 includes the image processing device 100 shown in FIG.

In addition, the UAV 120 may also include the image processing device 110 shown in FIG. 11.

Fig. 13 is a schematic block diagram of a receiving end according to an embodiment of the present application. As shown in FIG. 13, the receiving end 130 includes the image processing device 110 shown in FIG.

In addition, the receiving end 130 may also include the image processing device 100 shown in FIG. 10.

An embodiment of the present application provides a computer-readable storage medium that stores program instructions, and when a processor executes the program instructions, the image processing method at the sending end as described above is implemented.

The embodiments of the present application also provide another computer-readable storage medium, the computer-readable storage medium stores program instructions, and when the processor executes the program instructions, the image processing method at the receiving end as described above is realized.

In the several embodiments provided by the present invention, it should be understood that the disclosed device and method may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other divisions in actual implementation, for example, multiple modules can be combined or integrated. To another system, or some features can be ignored, or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or modules, and may be in electrical, mechanical or other forms.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional modules in the various embodiments of the present invention may be integrated into one processing unit, or each module may exist alone physically, or two or more modules may be integrated into one unit. The units formed by the above-mentioned modules can be realized in the form of hardware, or in the form of hardware plus software functional units.

The above-mentioned integrated modules implemented in the form of software function modules may be stored in a computer readable storage medium. The above-mentioned software function module is stored in a storage medium and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to execute the various embodiments of the present application Part of the method.

It should be understood that the above-mentioned processor may be a central processing unit (Central Processing Unit, CPU for short), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), and application specific integrated circuits (Application Specific Integrated Circuits). Referred to as ASIC) and so on. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in combination with the invention can be directly embodied as executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage, and may also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

The bus may be an Industry Standard Architecture (ISA) bus, Peripheral Component (PCI) bus, or Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

The above-mentioned storage medium can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Except for programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disks or optical disks. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in Application Specific Integrated Circuits (ASIC for short). Of course, the processor and the storage medium may also exist as discrete components in the electronic device or the main control device.

A person of ordinary skill in the art can understand that all or part of the steps in the foregoing method embodiments can be implemented by a program instructing relevant hardware. The aforementioned program can be stored in a computer readable storage medium. When the program is executed, it executes the steps including the foregoing method embodiments; and the foregoing storage medium includes: ROM, RAM, magnetic disk, or optical disk and other media that can store program codes.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: It is still possible to modify the technical solutions described in the foregoing embodiments, or equivalently replace some or all of the technical features; these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention range.

Claims

An image processing method at the sending end, characterized in that it comprises:

Get the current image frame;

Detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end;

It is determined according to the communication quality whether to determine the reference frame used to encode the current image frame according to the received feedback information, wherein the feedback information is transmitted from the receiving end to the transmitting end via the wireless uplink communication link. If sent, the feedback information is used to indicate the transmission status of the image frame sent by the sending end to the receiving end before the feedback information.
The method according to claim 1, wherein the determining whether to determine a reference frame for encoding the current image frame according to the received feedback information according to the communication quality comprises:

When the communication quality is higher than or equal to the preset communication quality threshold, the reference frame used for encoding the current image frame is determined according to the received feedback information.
The method according to claim 2, wherein the determining according to the communication quality whether to determine a reference frame for encoding the current image frame according to received feedback information comprises:

When the communication quality is less than the preset communication quality threshold, one or more reference frames for encoding the current image frame are determined according to the preset frame number interval, wherein the reference frames include frame-based Inner coded image area.
The method according to any one of claims 1 to 3, wherein the determining a reference frame for encoding the current image frame according to the feedback information comprises:

Determine the image frame that has been correctly transmitted according to the feedback information;

The reference frame is determined according to the image frame that has been correctly transmitted.
The method according to any one of claims 1 to 4, wherein the detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end comprises:

Acquiring communication state information of the wireless uplink communication link;

The communication quality is determined according to the communication status information.
The method according to claim 5, wherein the communication status information includes one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength.
The method according to any one of claims 1 to 4, wherein the detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end comprises:

Acquiring the information indicating the communication quality sent by the receiving end, and determining the communication quality according to the information indicating the communication quality.
The method according to any one of claims 1-7, wherein the sending end is an unmanned aerial vehicle.
An image processing method in a receiving end, characterized in that it comprises:

Detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end;

Determine whether to send feedback information through the wireless uplink communication link according to the communication quality, where the feedback information is used to indicate the transmission status of the image frame sent by the sending end to the receiving end before the feedback information And instruct the sending end to determine the reference frame used when encoding the current image frame according to the feedback information.
The method according to claim 9, wherein the determining whether to send feedback information through the wireless uplink communication link according to the communication quality comprises:

When the communication quality is higher than or equal to a preset communication quality threshold, sending the feedback information through the wireless uplink communication link;

Otherwise, the feedback information is not sent.
The method according to claim 9 or 10, wherein the feedback information includes at least one of the following information:

Indication information used to indicate whether the last image frame sent before the feedback information is correctly transmitted;

The frame number of the image frame that has been correctly transmitted; and

The frame number of the last image frame that was correctly transmitted before the feedback information.
The method according to any one of claims 9-11, wherein the detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end comprises:

Acquiring communication state information of the wireless uplink communication link;

The communication quality is determined according to the communication status information.
The method according to claim 12, wherein the communication status information includes one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength.
The method according to any one of claims 9-11, wherein the detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end comprises:

Obtain the information indicating the communication quality sent by the sending end, and determine the communication quality according to the information indicating the communication quality.
The method according to any one of claims 9-14, wherein the sending end is an unmanned aerial vehicle.
An image processing device at a sending end, which is characterized by comprising a memory and a processor, wherein:

The memory is used to store program instructions;

The processor is configured to execute the program instructions, and when the program instructions are executed, the processor executes the following steps:

Get the current image frame;

Detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end;

It is determined according to the communication quality whether to determine the reference frame used to encode the current image frame according to the received feedback information, wherein the feedback information is transmitted from the receiving end to the transmitting end via the wireless uplink communication link. If sent, the feedback information is used to indicate the transmission status of the image frame sent by the sending end to the receiving end before the feedback information.
The device according to claim 16, wherein when the processor determines whether to determine a reference frame for encoding the current image frame according to the received feedback information according to the communication quality, it is specifically configured to:

When the communication quality is higher than or equal to the preset communication quality threshold, the reference frame used for encoding the current image frame is determined according to the received feedback information.
The device according to claim 17, wherein when the processor determines whether to determine a reference frame for encoding the current image frame according to the received feedback information according to the communication quality, it is specifically configured to:

When the communication quality is less than the preset communication quality threshold, one or more reference frames for encoding the current image frame are determined according to the preset frame number interval, wherein the reference frames include frame-based Inner coded image area.
The device according to any one of claims 16-18, wherein when the processor determines the reference frame used to encode the current image frame according to the feedback information, it is specifically configured to:

Determine the image frame that has been correctly transmitted according to the feedback information;

The reference frame is determined according to the image frame that has been correctly transmitted.
The device according to any one of claims 16-19, wherein when the processor detects the communication quality of the wireless uplink communication link between the sending end and the receiving end, it is specifically configured to:

Acquiring communication state information of the wireless uplink communication link;

The communication quality is determined according to the communication status information.
The device according to claim 20, wherein the communication status information includes one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength.
The device according to any one of claims 16-19, wherein when the processor detects the communication quality of the wireless uplink communication link between the sending end and the receiving end, it is specifically configured to:

Acquiring the information indicating the communication quality sent by the receiving end, and determining the communication quality according to the information indicating the communication quality.
The device according to any one of claims 16-22, wherein the sending end is an unmanned aerial vehicle.
An unmanned aerial vehicle, characterized by comprising the image processing equipment at the transmitting end according to any one of claims 16-23.
An image processing device in a receiving end, which is characterized by comprising a memory and a processor, wherein:

The memory is used to store program instructions;

The processor is configured to execute the program instructions, and when the program instructions are executed, the processor executes the following steps:

Detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end;

Determine whether to send feedback information through the wireless uplink communication link according to the communication quality, where the feedback information is used to indicate the transmission status of the image frame sent by the sending end to the receiving end before the feedback information And instruct the sending end to determine the reference frame used when encoding the current image frame according to the feedback information.
The device according to claim 25, wherein when the processor determines whether to send feedback information through the wireless uplink communication link according to the communication quality, the processor is specifically configured to:

When the communication quality is higher than or equal to a preset communication quality threshold, sending the feedback information through the wireless uplink communication link;

Otherwise, the feedback information is not sent.
The device according to claim 25 or 26, wherein the feedback information includes at least one of the following information:

Indication information used to indicate whether the last image frame sent before the feedback information is correctly transmitted;

The frame number of the image frame that has been correctly transmitted; and

The frame number of the last image frame that was correctly transmitted before the feedback information.
The device according to any one of claims 25-27, wherein when the processor detects the communication quality of the wireless uplink communication link between the sending end and the receiving end, it is specifically configured to:

Acquiring communication state information of the wireless uplink communication link;

The communication quality is determined according to the communication status information.
The device according to claim 28, wherein the communication status information includes one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength.
The device according to any one of claims 25-27, wherein when the processor detects the communication quality of the wireless uplink communication link between the sending end and the receiving end, it is specifically configured to:

Obtain the information indicating the communication quality sent by the sending end, and determine the communication quality according to the information indicating the communication quality.
The device according to any one of claims 25-30, wherein the sending end is an unmanned aerial vehicle.
A receiving end, characterized by comprising the image processing device in the receiving end according to any one of claims 25-31.
A computer-readable storage medium, wherein program instructions are stored in the computer-readable storage medium, and when the processor executes the program instructions, the sending end of any one of claims 1-8 is implemented. Image processing method.
A computer-readable storage medium, wherein program instructions are stored in the computer-readable storage medium, and when the processor executes the program instructions, the receiving device according to any one of claims 9-15 is realized. Image processing method in the terminal.