CN111713107A

CN111713107A - Image processing method and device, unmanned aerial vehicle and receiving end

Info

Publication number: CN111713107A
Application number: CN201980012207.1A
Authority: CN
Inventors: 马宁; 朱磊; 陈颖; 赵亮
Original assignee: SZ DJI Technology Co Ltd
Current assignee: SZ DJI Technology Co Ltd
Priority date: 2019-06-28
Filing date: 2019-06-28
Publication date: 2020-09-25
Also published as: WO2020258296A1

Abstract

An image processing method, image processing equipment, an unmanned aerial vehicle and a receiving end. The method comprises the following steps: acquiring a current image frame; detecting the communication quality of a wireless uplink communication link of a sending end and a receiving end; and determining whether to determine a reference frame for encoding the current image frame according to the received feedback information according to the communication quality, wherein the feedback information is sent by the receiving end to the sending end through the wireless uplink communication link, and the feedback information is used for indicating the transmission state of the image frame sent by the sending end to the receiving end before the feedback information. The method enables determining whether to determine a reference frame for encoding a current image frame based on received feedback information based on a communication quality of a wireless uplink communication link. The coding mode of the image is matched with the communication quality of the wireless uplink communication link, so that the image is recovered in time when the image is transmitted in error, and the influence on the user experience is avoided.

Description

Image processing method and device, unmanned aerial vehicle and receiving end

Technical Field

The embodiment of the application relates to an image processing technology, in particular to an image processing method, image processing equipment, an unmanned aerial vehicle and a receiving end.

Background

Low latency video transmission systems over wireless and unreliable channels are currently a direction of intense research and application. For video transmission of unreliable channels, data errors can occur during data transmission, resulting in errors in video decoding. However, the transmitting end cannot predict when a data error occurs, and therefore, a corresponding error recovery mechanism is required to correct the video data error that has occurred.

Common error recovery mechanisms are non-feedback coding mode and feedback coding mode. The feedback coding mode can realize coding without feedback information sent by a receiving end, and the feedback coding mode adopts coding based on the feedback information sent by the receiving end.

In the existing coding mechanism, an image is often coded based on one of a non-feedback coding mode and a feedback coding mode. However, neither the feedback-free coding mode nor the feedback coding mode is suitable for application scenarios that change in real time, and in some application scenarios, when image transmission is wrong, image recovery delay may be increased, which may affect user experience.

Disclosure of Invention

The embodiment of the application provides an image processing method, image processing equipment, an unmanned aerial vehicle and a receiving end, so that the flexibility of image coding and image error recovery is improved.

In a first aspect, an embodiment of the present application provides an image processing method at a sending end, including:

acquiring a current image frame;

detecting the communication quality of a wireless uplink communication link of the sending end and the receiving end;

determining whether to determine a reference frame for encoding the current image frame according to the received feedback information according to the communication quality, wherein the feedback information is transmitted from the receiving end to the transmitting end through the wireless uplink communication link, and the feedback information is used for indicating a transmission state of an image frame transmitted from the transmitting end to the receiving end before the feedback information.

In a second aspect, an embodiment of the present application provides an image processing method in a receiving end, including:

detecting the communication quality of a wireless uplink communication link of a sending end and a receiving end;

and determining whether to transmit feedback information through the wireless uplink communication link according to the communication quality, wherein the feedback information is used for indicating the transmission state of an image frame transmitted to the receiving end by the transmitting end before the feedback information and indicating a reference frame adopted when the transmitting end determines to encode the current image frame according to the feedback information.

In a third aspect, an embodiment of the present application provides an image processing apparatus at a transmitting end, including a memory and a processor, wherein,

the memory to store program instructions;

the processor is configured to execute the program instructions, and when the program instructions are executed, the processor performs the following steps:

acquiring a current image frame;

In a fourth aspect, the present application provides an unmanned aerial vehicle including the image processing device at the transmitting end as described above in the third aspect and in various possible designs of the third aspect.

In a fifth aspect, an embodiment of the present application provides an image processing apparatus in a receiving end, including a memory and a processor, wherein,

the memory to store program instructions;

In a sixth aspect, the present application provides a receiving end, including the image processing apparatus in the receiving end according to the fifth aspect and various possible designs of the fifth aspect.

In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, where program instructions are stored in the computer-readable storage medium, and when a processor executes the program instructions, the method for processing an image at a transmitting end according to the first aspect and various possible designs of the first aspect is implemented.

In an eighth aspect, the present application provides another computer-readable storage medium, where program instructions are stored in the computer-readable storage medium, and when a processor executes the program instructions, the image processing method in the receiving end according to the second aspect and various possible designs of the second aspect is implemented.

According to the image processing method, the image processing device, the unmanned aerial vehicle and the receiving end, the sending end detects the communication quality of a wireless uplink communication link of the sending end and the receiving end by obtaining the current image frame, and whether the reference frame used for coding the current image frame is determined according to the received feedback information is determined according to the communication quality. Whether a feedback coding mode needs to be used is determined according to the communication quality of the wireless uplink communication link in a self-adaptive mode, so that the coding mode of the image is matched with the communication quality of the wireless uplink communication link, recovery is timely when image transmission errors occur, and user experience is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.

Fig. 1 is a schematic diagram of an architecture of an image processing system according to an embodiment of the present application;

fig. 2 is a schematic diagram of a non-feedback coding mode according to an embodiment of the present application;

fig. 3 is a schematic diagram of another non-feedback coding mode according to an embodiment of the present application;

fig. 4 is a schematic flowchart of an image processing method according to an embodiment of the present application;

FIG. 5 is a diagram illustrating reference frame selection provided in an embodiment of the present application;

fig. 6 is a schematic flowchart of another image processing method according to an embodiment of the present application;

fig. 7 is a schematic flowchart of another image processing method according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of an image processing device at a transmitting end according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of an image processing apparatus at a receiving end according to an embodiment of the present application;

fig. 10 is a schematic diagram of a hardware structure of an image processing apparatus at a transmitting end according to an embodiment of the present application;

fig. 11 is a schematic hardware structure diagram of an image processing apparatus at a receiving end according to an embodiment of the present application;

FIG. 12 is a schematic structural diagram of an unmanned aerial vehicle provided by an embodiment of the present application;

fig. 13 is a schematic structural diagram of a receiving end according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below clearly with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It will be understood that when an element is referred to as being "secured to" another element, it can be directly on the other element or intervening elements may also be present. When a component is referred to as being "connected" to another component, it can be directly connected to the other component or intervening components may also be present.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

Some embodiments of the invention are described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.

Fig. 1 is a schematic architecture diagram of an image processing system according to an embodiment of the present application. As shown in fig. 1, the system provided by this embodiment includes a transmitting end 101 and a receiving end 102. The sending end 101 sends an image to the receiving end 102 through a wireless downlink communication link, and the receiving end 102 sends information to the sending end 101 through a wireless uplink communication link.

The real-time image transmission has higher requirements on image transmission delay, the traditional real-time image transmission has the problems of limited transmission bandwidth and larger bandwidth fluctuation, in addition, the wireless channel has more uncertainty, particularly the characteristic of unreliable wireless channel, errors can occur in the data transmission process inevitably, so that video decoding errors are caused, and at the moment, an error recovery mechanism is needed to correct the occurring video data transmission errors. The common error recovery mechanism has two modes of non-feedback coding and feedback coding.

Before the above error recovery mechanisms with feedback and no feedback are introduced, the types of commonly used coding frames are introduced. Video coding standards widely used in the industry include h.263, h.264, h.265, MPEG4, and the like, all providing two types of frame types:

i frame:

all image areas are coded in an intra-frame predictive coding mode, and a predictive compression mode independent of a reference frame is adopted.

Since no prediction is made with reference to temporal correlation, the compression rate is low and the resulting code rate of the encoded frame is high.

The advantages are that: because the receiver can recover the error only by correctly receiving the I frame without depending on the reference of other frames.

P/B frame:

and the image block is coded according to an inter-frame prediction coding mode, and a prediction compression mode depending on a reference frame is adopted.

Since the reference frame is referred to for prediction, the compression rate is high, and the code rate of the generated encoded frame is low.

The disadvantages are as follows: because of the dependence on the reference frame, it may happen that when the reference frame is received incorrectly, even if the receiving end receives the frame correctly, the frame picture cannot be decoded correctly, i.e. cannot be restored back.

Regarding error recovery coding for real-time image transmission, the following two methods can be used:

one, no feedback coding mode

1. As shown in fig. 2, the transmitting end determines an I frame encoded by intra prediction at regular frame number intervals, and all image areas in the I frame are encoded by intra prediction encoding. The image of the preset frame number following the I frame may be inter-frame prediction encoded, for example, to obtain a P frame of the preset frame number. The sending end can send the image obtained by coding to the receiving end through a wireless downlink communication link.

2. And the transmitting end determines and transmits a plurality of image frames comprising an I area to the transmitting end at intervals of fixed frame numbers, wherein the I area is an intra-frame coded image area. An image of a preset number of frames following the plurality of image frames including the I region may be inter-frame prediction encoded, for example, to obtain a P frame of the preset number of frames. As shown in fig. 3, the plurality of image frames may comprise 3 frames, each frame containing interleaved 1/3 frames using intra-frame coding. The sending end can send the image obtained by coding to the receiving end through a wireless downlink communication link.

As can be seen from fig. 2 and 3, when the image transmitted by the transmitting end is received by the receiving end, and the image transmitted by the transmitting end includes an I frame with a fixed frame number interval period or a plurality of image frames including an I region, when an image received error of a frame encoded according to the inter-frame preset encoding mode occurs, the image received error of the next frame encoded according to the inter-frame preset encoding mode may refer to the corresponding I frame or the plurality of image frames including the I region for decoding, and still may be successfully recovered.

Advantages of the feedback-free coding mode:

the receiving end can recover the image transmission error based on the received I frame or a plurality of image frames including the I area, and does not depend on the feedback information from the receiving end to the transmitting end.

The disadvantages of the scheme are as follows:

it is necessary to periodically generate an I frame or a plurality of image frames including an I region, which results in a low compression rate of the image.

Second, feedback coding mode

The receiving end determines the transmission state of the image frame sent by the sending end, for example, whether the image frame is correctly received or not, and indicates the feedback information of the transmission state of the image frame to the sending end, and the sending end determines the image frame correctly received by the receiving end as the reference frame of the current image to be coded according to the feedback information so as to code the current image to be coded.

The scheme has the advantages that:

the I frame or a plurality of image frames including the I area are not required to be transmitted, and the pressure on the wireless downlink communication link is reduced.

The disadvantages of this solution are:

because the receiving end needs to send feedback information to the sending end through the wireless uplink communication link, when the wireless uplink communication link is unreliable, that is, when the communication quality of the wireless uplink communication link is not high, the error recovery time may be long.

In the existing coding mechanism, an image is often coded based on one of a non-feedback coding mode and a feedback coding mode. However, neither the feedback-free coding mode nor the feedback coding mode is suitable for the application scenario that changes in real time, the flexibility of coding is poor, and the matching coding mode cannot be selected according to the current application scenario. When the image transmission is wrong, the image recovery delay may be increased, which may affect the user experience.

In order to solve the technical problem, the present embodiment provides an image processing method, which adaptively determines whether a feedback coding mode needs to be used according to the communication quality of a wireless uplink communication link, so that the coding mode of an image matches the communication quality of the wireless uplink communication link, and the image processing method improves user experience in time for recovery in case of an image transmission error.

Fig. 4 is a flowchart of an image processing method at a transmitting end according to an embodiment of the present application, where an execution subject of this embodiment may be the transmitting end in the embodiment shown in fig. 1, and may be exemplarily executed by an image processing device in the transmitting end, where the image processing device may include one or more processors for executing the image processing method, and the transmitting end is configured to transmit encoded data of an image frame to a receiving end. As shown in fig. 4, the method includes:

s401, acquiring a current image frame.

Specifically, the receiving end may include a camera, wherein the image processing apparatus may acquire the current image frame to be encoded from the camera.

S402, detecting the communication quality of the wireless uplink communication link of the sending end and the receiving end.

The image processing device can detect the communication quality of the wireless uplink communication link, and can determine whether the feedback information sent by the receiving end can be timely and accurately received by detecting the wireless uplink communication quality.

Optionally, the detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end includes: and acquiring communication state information of the wireless uplink communication link, and determining the communication quality according to the communication state information. Specifically, the image processing device may collect communication state information of the wireless uplink communication link or receive the communication state information sent by a receiving end through the wireless uplink communication link, and then determine the communication quality according to the communication state information.

Optionally, the communication status information includes one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength. For example, the communication state information includes a bandwidth, and the larger the bandwidth, the higher the communication quality.

The communication status information may also include Received SIGNAL Strength (RSSI), SIGNAL-to-NOISE RATIO (SNR), etc., channel capacity estimation, channel NOISE power, distance determination between two communication parties, etc.

Optionally, the detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end includes: and acquiring the information which is sent by the receiving end and used for indicating the communication quality, and determining the communication quality according to the information indicating the communication quality.

Specifically, the receiving end may detect the communication quality of the wireless uplink communication link by acquiring the communication state information of the wireless uplink communication link, and send information indicating the communication quality to the transmitting end, and the transmitting end determines the communication quality by the information indicating the communication apparatus.

And S403, determining whether to determine a reference frame for encoding the current image frame according to the received feedback information according to the communication quality, where the feedback information is sent by the receiving end to the sending end through the wireless uplink communication link, and the feedback information is used to indicate a transmission state of an image frame sent by the sending end to the receiving end before the feedback information.

It should be noted that the transmission state refers to whether or not an error occurs in the transmission of the image frame. Specifically, it refers to whether data loss or data error occurs during the transmission of the image frame. Or, it may be understood whether the image frame was correctly transmitted and, after correct transmission, was successfully decoded.

Optionally, the determining whether to determine a reference frame for encoding the current image frame according to the received feedback information according to the communication quality includes:

and when the communication quality is higher than or equal to a preset communication quality threshold value, determining a reference frame for coding the current image frame according to the received feedback information.

and when the communication quality is smaller than the preset communication quality threshold value, determining one frame or a plurality of frames of reference frames for coding the current image frame according to a preset frame number interval, wherein the reference frames comprise image areas based on intra-frame coding.

Here, the preset communication quality threshold may be set according to actual conditions, and illustratively, the communication quality of the wireless uplink communication link between the sending end and the receiving end is represented by bandwidth, error rate, signal-to-noise ratio, and received signal strength, where the preset communication quality threshold includes a bandwidth threshold, an error rate threshold, a signal-to-noise ratio threshold, and a received signal strength threshold.

And when the communication quality is higher than or equal to a preset communication quality threshold value, the communication quality of a wireless uplink communication link between the sending end and the receiving end is better, if the image processing equipment works in a feedback-free coding mode, the mode is switched to a feedback coding mode, a reference frame for coding the current image frame is determined according to the received feedback information, and otherwise, the existing mode is kept unchanged.

When the communication quality is smaller than the preset communication quality threshold value, the communication quality of a wireless uplink communication link of the sending end and the receiving end is poor, if the image processing equipment works in a feedback coding mode, the equipment is switched to a non-feedback coding mode, one frame or multiple frames of reference frames used for coding the current image frame are determined according to the preset frame number interval, and if not, the existing mode is kept unchanged. When one or more reference frames are reference frames, the reference frame is an I frame as shown in fig. 2, that is, all image areas in the reference frame are image areas based on intra-frame coding; when the one or more reference frames are multi-frame reference frames, the multi-frame reference frames are a group of image frames as shown in fig. 3, that is, each reference frame in the multi-frame reference frames includes an image area based on intra-frame coding.

In addition, in order to avoid the mode switching from being too frequent, a certain guard time can be added, for example, after the mode is switched to the A mode, the mode is not allowed to be switched to the B mode within N time. The mode A is a non-feedback coding mode, the mode B is a feedback coding mode, or the mode A is a feedback coding mode, and the mode B is a non-feedback coding mode. The N time can be set according to actual conditions. Here, a double threshold may also be set: after switching to the A mode, switching to the B mode is not allowed within the time of N1; after the mode is switched to the B mode, the mode is not allowed to be switched to the A mode within the time of N2, and the requirements of various application scenarios are met.

Optionally, the determining a reference frame for encoding the current image frame according to the feedback information includes:

determining an image frame that has been correctly transmitted according to the feedback information;

determining the reference frame from the image frame that has been correctly transmitted.

Optionally, the feedback information includes at least one of the following information:

indication information indicating whether a last frame image frame transmitted before the feedback information is correctly transmitted;

the frame number of the image frame that has been correctly transmitted; and

a frame number of an image frame that is correctly transmitted the last frame before the feedback information.

Wherein the frame number of the image frame is globally unique, and the frame number of the image frame is transmitted to the receiving end together with the encoded data.

Illustratively, as shown in fig. 5, assuming that the transmitting end has transmitted 5 image frames to the receiving end before the current image frame (number 6), the corresponding numbers are 1-5 in sequence, and the image frames numbered 2, 3 and 4 are correctly transmitted. The feedback information includes the frame number of the image frame that has been correctly transmitted. At this time, the feedback information may be "234". Alternatively, the feedback information includes the frame number of the image frame whose last frame has been correctly transmitted, and at this time, the feedback information may be "4", and the image processing apparatus in the transmitting end may know that the image frame numbered 5 has not been successfully transmitted according to the feedback information. Thus, the image processing apparatus in the transmitting end can select the image frame numbered 4 as the reference frame or select the image frames numbered 3 and 4 as the reference frames when encoding the image frame numbered 6.

Or, assuming that the transmitting end has transmitted 8 image frames to the receiving end before the current image frame (number 9), the corresponding numbers are 1-8 in sequence, and the receiving end has fed back for the image frames numbered 1-5. Only the transmission state of the image frame number 8 needs to be fed back in this feedback information. The feedback information includes an indication bit for indicating whether the image frame numbered 8 is correctly transmitted, and if the value of the indication bit is "0", the image frame numbered 8 is not correctly transmitted, and if the value of the indication bit is "1", the image frame numbered 8 is correctly transmitted. If the value of the indicator bit is 1, the image processing apparatus in the transmitting end selects the image frame numbered 8 as a reference frame and inter-encodes the image frame numbered 9. If the value of the indicator bit is 0, the image processing apparatus in the transmitting end may determine whether the image frame numbered 5 is correctly transmitted, and if the image frame numbered 5 is correctly transmitted, select the image frame numbered 5 as a reference frame and inter-frame encode the image frame numbered 9.

Or, further, the image processing device in the sending end stores at least a part of the image frames which have been correctly transmitted into the reference frame management queue, and the image processing device in the sending end determines the image frames in the reference frame management queue as reference frames. Therefore, when the feedback information sent by the receiving end is received each time, the correctly transmitted image frame is determined according to the feedback information, and the reference frame management queue is updated. Accordingly, the image processing apparatus in the receiving end also saves at least a part of the correctly transmitted image frames in the reference frame management queue after determining the image frames that have been correctly transmitted. The reference frame management queue in the sending end is the same as the information stored in the reference frame management queue in the receiving end.

Or, the image processing device at the sending end includes a physical or logical buffer for storing image frames corresponding to all encoded data, the image processing device at the sending end may only store frame numbers corresponding to at least a part of correctly transmitted image frames in a reference frame management queue, and the image processing device at the sending end determines the image frame corresponding to the frame number in the reference frame queue as a reference frame. Correspondingly, the image processing device in the receiving end includes a physical or logical buffer for storing the image frames corresponding to all the decoded data, and the image processing device in the receiving end can store the frame numbers corresponding to at least a part of the correctly transmitted image frames in the reference frame management queue.

The transmitter in embodiments of the present application may be moved in any suitable environment, such as in air (e.g., a fixed-wing aircraft, a rotorcraft, or an aircraft without a fixed wing and without a rotor), in water (e.g., a ship or submarine), on land (e.g., a car or train), in space (e.g., a space plane, a satellite, or a sonde), and any combination thereof. The transmitting end may be an Unmanned Aerial Vehicle (UAV), for example. In some embodiments, the transmitting end may carry a living entity, such as a human or an animal.

The receiving end in the embodiment of the application can be a computer, a handheld electronic device, a communication device, a video monitoring device and the like.

In the image processing method at the sending end provided by this embodiment, the sending end detects the communication quality of the wireless uplink communication link between the sending end and the receiving end by acquiring the current image frame, and determines whether to determine the reference frame for encoding the current image frame according to the received feedback information according to the communication quality. The method comprises the steps of determining whether a feedback coding mode needs to be used or not according to the communication quality of a wireless uplink communication link in a self-adaptive mode, and ensuring that the feedback coding mode is adopted when the quality of the wireless uplink communication link is good, and the non-feedback coding mode is adopted when the quality of the feedback link is poor, so that the coding flexibility is improved, the coding mode of an image is matched with the communication quality of the wireless uplink communication link, the image is recovered in time when the image is transmitted in error, and the user experience is improved.

The method of image processing according to an embodiment of the present application is described in detail above from the transmitting end side in conjunction with fig. 4 and 5, and the method of image processing according to another embodiment of the present invention will be described in detail below from the receiving end side in conjunction with fig. 6. It should be understood that the interaction between the receiving end and the transmitting end described by the transmitting end and the related characteristics, functions and the like correspond to the description of the receiving end, and the repeated description is appropriately omitted for the sake of brevity.

Fig. 6 illustrates a method of image processing according to another embodiment of the present invention, which may be performed by an image processing apparatus in a receiving end, which may be various types of chips for image processing, image processors, and the like. As shown in fig. 6, the method includes:

s601, detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end.

Optionally, the detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end includes:

acquiring communication state information of the wireless uplink communication link;

and determining the communication quality according to the communication state information.

Optionally, the communication status information includes one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength.

and acquiring the information which is sent by the sending end and used for indicating the communication quality, and determining the communication quality according to the information indicating the communication quality.

S602, determining whether to send feedback information through the wireless uplink communication link according to the communication quality, wherein the feedback information is used for indicating the transmission state of the image frame sent by the sending end to the receiving end before the feedback information and indicating a reference frame adopted by the sending end when the sending end determines to code the current image frame according to the feedback information.

Optionally, the determining whether to transmit feedback information through the wireless uplink communication link according to the communication quality includes:

when the communication quality is higher than or equal to a preset communication quality threshold value, sending the feedback information through the wireless uplink communication link;

otherwise, the feedback information is not sent.

Optionally, the feedback information is used to instruct the sending end to determine, according to the feedback information, a reference frame used when encoding the current image frame, and includes:

the feedback information is used for indicating the sending end to determine the correctly transmitted image frame according to the feedback information; determining the reference frame from the image frame that has been correctly transmitted.

the frame number of the image frame that has been correctly transmitted; and

Optionally, the transmitting end is an unmanned aerial vehicle.

In the image processing method of the receiving end provided in this embodiment, the receiving end determines whether to send feedback information through a wireless uplink communication link according to the communication quality by detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end, where the feedback information is used to indicate a transmission state of an image frame sent by the sending end to the receiving end before the feedback information and to indicate a reference frame adopted by the sending end when determining to encode a current image frame according to the feedback information. The method comprises the steps that whether feedback information is sent to a sending end or not is determined according to the communication quality of a wireless uplink communication link in a self-adaptive mode, so that the sending end determines whether a feedback coding mode needs to be used or not, the coding mode of an image is matched with the communication quality of the wireless uplink communication link, the image is recovered in time when the image is transmitted in error, and user experience is improved.

Fig. 7 illustrates a method of image processing according to still another embodiment of the present invention, as illustrated in fig. 7, the method including:

s701, the sending end sends the image frame to the receiving end.

S702, the receiving end generates feedback information according to the transmission state of the image frame.

And S703, the receiving end sends the feedback information to the sending end.

S704, the sending end detects the communication quality of the wireless uplink communication link of the sending end and the receiving end.

And S705, when the communication quality is higher than or equal to the preset communication quality threshold, the sending end determines a reference frame for coding the current image frame according to the feedback information.

For example, the transmitting end may further perform inter-frame coding on the current image frame according to the reference frame to generate coded data, and transmit the coded data to the receiving end. The receiving end may perform inter-frame decoding on the encoded data according to the reference frame to generate decoded data, and display a current image frame according to the decoded data.

S706, when the communication quality is smaller than a preset communication quality threshold, the sending end determines one frame or multiple frames of reference frames for coding the current image frame according to a preset frame number interval, wherein the reference frames comprise image areas based on intra-frame coding.

Similarly, the sending end may further perform intra-frame coding on the current image frame according to the reference frame to generate coded data, and send the coded data to the receiving end. The receiving end may perform intra-frame decoding on the encoded data according to the reference frame to generate decoded data, and display a current image frame according to the decoded data.

It can be understood that the interaction between the receiving end and the transmitting end in fig. 7 and the related features, functions and other methods correspond to those described in fig. 4, and are not described herein again for brevity.

Having described the image processing method according to the embodiment of the present application above, an image processing apparatus according to the embodiment of the present application will be described below. It should be understood that the image processing apparatus described below can implement the above image processing method, and the following will be briefly described in order to avoid repetition.

Fig. 8 is a schematic structural diagram of an image processing device at a transmitting end according to an embodiment of the present application. For convenience of explanation, only portions related to the embodiments of the present application are shown. As shown in fig. 8, the image processing apparatus 80 includes: an image frame acquisition module 801, a first quality detection module 802 and a reference frame determination module 803.

The image frame acquiring module 801 is configured to acquire a current image frame.

A first quality detection module 802, configured to detect communication quality of a wireless uplink communication link between the sending end and the receiving end.

A reference frame determining module 803, configured to determine whether to determine a reference frame used for encoding the current image frame according to the received feedback information according to the communication quality, where the feedback information is sent by the receiving end to the sending end through the wireless uplink communication link, and the feedback information is used to indicate a transmission status of an image frame sent by the sending end to the receiving end before the feedback information.

In one possible design, the reference frame determining module 803 is specifically configured to:

In one possible design, the reference frame determining module 803 determines the reference frame for encoding the current image frame according to the feedback information, including:

In one possible design, the first quality detection module 802 is specifically configured to:

In one possible design, the communication status information may include one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength.

and acquiring the information which is sent by the receiving end and used for indicating the communication quality, and determining the communication quality according to the information indicating the communication quality.

In one possible design, the transmitting end is an unmanned aerial vehicle.

The device provided in the embodiment of the present application may be configured to execute the technical solution of the method embodiment in fig. 4, which has similar implementation principles and technical effects, and is not described herein again in the embodiment of the present application.

Fig. 9 is a schematic structural diagram of an image processing apparatus at a receiving end according to an embodiment of the present application. For convenience of explanation, only portions related to the embodiments of the present application are shown. As shown in fig. 9, the image processing apparatus 90 includes: a second quality detection module 901 and an information determination module 902.

The second quality detection module 901 is configured to detect communication quality of a wireless uplink communication link between a sending end and the receiving end;

an information determining module 902, configured to determine whether to send feedback information through the wireless uplink communication link according to the communication quality, where the feedback information is used to indicate a transmission state of an image frame sent by the sending end to the receiving end before the feedback information and indicate a reference frame used by the sending end when determining to encode a current image frame according to the feedback information.

In one possible design, the information determining module 902 is specifically configured to:

otherwise, the feedback information is not sent.

In one possible design, the feedback information includes at least one of the following information:

the frame number of the image frame that has been correctly transmitted; and

In one possible design, the second quality detection module 901 is specifically configured to:

In one possible design, the transmitting end is an unmanned aerial vehicle

The device provided in the embodiment of the present application may be configured to implement the technical solution of the method embodiment in fig. 6, which has similar implementation principles and technical effects, and is not described herein again in the embodiment of the present application.

Fig. 10 is a schematic diagram of a hardware structure of an image processing device at a transmitting end according to an embodiment of the present application. As shown in fig. 10, the image processing apparatus 100 of the present embodiment includes: a processor 1001 and a memory 1002; wherein

A memory 1002 for storing program instructions;

a processor 1001 for executing memory-stored program instructions that, when executed, perform the steps of:

acquiring a current image frame;

Optionally, when the processor determines whether to determine, according to the received feedback information, a reference frame used for encoding the current image frame according to the communication quality, specifically, the processor is configured to:

Optionally, when the processor determines, according to the feedback information, a reference frame used for encoding the current image frame, specifically, the processor is configured to:

Optionally, when detecting the communication quality of the wireless uplink communication link between the sending end and the receiving end, the processor is specifically configured to:

Optionally, the transmitting end is an unmanned aerial vehicle.

Alternatively, the memory 1002 may be separate or integrated with the processor 1001.

When the memory 1002 is provided separately, the image processing apparatus further includes a bus 1003 for connecting the memory 1002 and the processor 1001.

Fig. 11 is a schematic diagram of a hardware structure of an image processing device at a sending end according to an embodiment of the present application. As shown in fig. 11, an embodiment of the present application further provides an image processing apparatus 110 at a receiving end, including: a processor 1101 and a memory 1102; wherein

A memory 1102 for storing program instructions;

a processor 1101 for executing memory-stored program instructions that, when executed, perform the steps of:

Optionally, when determining whether to send feedback information through the wireless uplink communication link according to the communication quality, the processor is specifically configured to:

otherwise, the feedback information is not sent.

the frame number of the image frame that has been correctly transmitted; and

Optionally, the transmitting end is an unmanned aerial vehicle.

Alternatively, the memory 1102 may be separate or integrated with the processor 1101.

When the memory 1102 is provided separately, the image processing apparatus further includes a bus 1103 for connecting the memory 1102 and the processor 1101.

Fig. 12 is a schematic structural diagram of an unmanned aerial vehicle according to an embodiment of the present application. As shown in fig. 12, the unmanned aerial vehicle 120 includes the image processing apparatus 100 shown in fig. 10.

In addition, the unmanned aerial vehicle 120 may also include the image processing apparatus 110 shown in fig. 11.

Fig. 13 is a schematic block diagram of a receiving end according to an embodiment of the present application. As shown in fig. 13, the receiving end 130 includes the image processing apparatus 110 shown in fig. 11.

In addition, the receiving end 130 may also include the image processing apparatus 100 shown in fig. 10.

An embodiment of the present application provides a computer-readable storage medium, where program instructions are stored in the computer-readable storage medium, and when a processor executes the program instructions, the method for processing an image at a sending end is implemented as described above.

An embodiment of the present application further provides another computer-readable storage medium, where program instructions are stored in the computer-readable storage medium, and when a processor executes the program instructions, the method for processing an image at a receiving end is implemented as described above.

In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules is only one logical division, and other divisions may be realized in practice, for example, a plurality of modules may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.

The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each module may exist alone physically, or two or more modules are integrated into one unit. The unit formed by the modules can be realized in a hardware form, and can also be realized in a form of hardware and a software functional unit.

The integrated module implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods according to the embodiments of the present application.

It should be understood that the Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor, or in a combination of the hardware and software modules within the processor.

The memory may comprise a high-speed RAM memory, and may further comprise a non-volatile storage NVM, such as at least one disk memory, and may also be a usb disk, a removable hard disk, a read-only memory, a magnetic or optical disk, etc.

The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (enhanced Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the buses in the figures of the present application are not limited to only one bus or one type of bus.

The storage medium may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be integral to the processor. The processor and the storage medium may reside in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the storage medium may reside as discrete components in an electronic device or host device.

Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method at a transmitting end, comprising:

acquiring a current image frame;

2. The method of claim 1, wherein said determining whether to determine a reference frame for encoding the current image frame based on the received feedback information based on the communication quality comprises:

3. The method of claim 2, wherein said determining whether to determine a reference frame for encoding the current image frame based on the received feedback information based on the communication quality comprises:

4. The method according to any of claims 1-3, wherein said determining a reference frame for encoding the current image frame from the feedback information comprises:

5. The method according to any of claims 1-4, wherein the detecting the communication quality of the wireless uplink communication link between the transmitting end and the receiving end comprises:

6. The method of claim 5, wherein the communication status information comprises one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength.

7. The method according to any of claims 1-4, wherein the detecting the communication quality of the wireless uplink communication link between the transmitting end and the receiving end comprises:

8. The method of any of claims 1-7, wherein the transmitting end is an unmanned aerial vehicle.

9. An image processing method in a receiving end, comprising:

10. The method of claim 9, wherein the determining whether to send feedback information over the wireless uplink communication link according to the communication quality comprises:

otherwise, the feedback information is not sent.

11. The method according to claim 9 or 10, wherein the feedback information comprises at least one of the following information:

the frame number of the image frame that has been correctly transmitted; and

12. The method according to any of claims 9-11, wherein the detecting the communication quality of the wireless uplink communication link between the transmitting end and the receiving end comprises:

13. The method of claim 12, wherein the communication status information comprises one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength.

14. The method according to any of claims 9-11, wherein the detecting the communication quality of the wireless uplink communication link between the transmitting end and the receiving end comprises:

15. The method of any of claims 9-14, wherein the transmitting end is an unmanned aerial vehicle.

16. An image processing apparatus of a transmitting end, comprising a memory, a processor, wherein,

the memory to store program instructions;

acquiring a current image frame;

17. The device of claim 16, wherein the processor, when determining whether to determine the reference frame for encoding the current image frame based on the received feedback information based on the communication quality, is specifically configured to:

18. The device of claim 17, wherein the processor, when determining whether to determine the reference frame for encoding the current image frame based on the received feedback information based on the communication quality, is specifically configured to:

19. The device according to any of claims 16-18, wherein the processor, when determining a reference frame for encoding the current image frame based on the feedback information, is configured to:

20. The apparatus according to any of claims 16-19, wherein the processor, when detecting the communication quality of the wireless uplink communication link between the transmitting end and the receiving end, is specifically configured to:

21. The apparatus of claim 20, wherein the communication status information comprises one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength.

22. The apparatus according to any of claims 16-19, wherein the processor, when detecting the communication quality of the wireless uplink communication link between the transmitting end and the receiving end, is specifically configured to:

23. The apparatus of any of claims 16-22, wherein the transmitting end is an unmanned aerial vehicle.

24. An unmanned aerial vehicle, characterized by comprising the image processing apparatus of the transmitting end according to any one of claims 16 to 23.

25. An image processing apparatus in a receiving end, comprising a memory, a processor, wherein,

the memory to store program instructions;

26. The device according to claim 25, wherein the processor, when determining whether to send feedback information via the wireless uplink communication link according to the communication quality, is specifically configured to:

otherwise, the feedback information is not sent.

27. The apparatus according to claim 25 or 26, wherein the feedback information comprises at least one of the following information:

the frame number of the image frame that has been correctly transmitted; and

28. The device according to any of claims 25 to 27, wherein the processor, when detecting the communication quality of the wireless uplink communication link between the transmitting end and the receiving end, is specifically configured to:

29. The apparatus of claim 28, wherein the communication status information comprises one or more of bandwidth, bit error rate, signal-to-noise ratio, and received signal strength.

30. The device according to any of claims 25 to 27, wherein the processor, when detecting the communication quality of the wireless uplink communication link between the transmitting end and the receiving end, is specifically configured to:

31. The apparatus of any of claims 25-30, wherein the transmitting end is an unmanned aerial vehicle.

32. A receiving end, characterized in that it comprises an image processing device in a receiving end according to any of claims 25-31.

33. A computer-readable storage medium, characterized in that a program instruction is stored in the computer-readable storage medium, which when executed by a processor, implements the image processing method of the transmitting end according to any one of claims 1 to 8.

34. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored therein program instructions that, when executed by a processor, implement the image processing method in the receiving end according to any one of claims 9 to 15.