WO2018120198A1 - 图像处理方法、装置、无人飞行器和接收端 - Google Patents

图像处理方法、装置、无人飞行器和接收端 Download PDF

Info

Publication number
WO2018120198A1
WO2018120198A1 PCT/CN2016/113899 CN2016113899W WO2018120198A1 WO 2018120198 A1 WO2018120198 A1 WO 2018120198A1 CN 2016113899 W CN2016113899 W CN 2016113899W WO 2018120198 A1 WO2018120198 A1 WO 2018120198A1
Authority
WO
WIPO (PCT)
Prior art keywords
frame
image processing
feedback information
image frame
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/113899
Other languages
English (en)
French (fr)
Inventor
朱磊
马宁
龚明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SZ DJI Technology Co Ltd
Original Assignee
SZ DJI Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SZ DJI Technology Co Ltd filed Critical SZ DJI Technology Co Ltd
Priority to CN201910586174.3A priority Critical patent/CN110225338B/zh
Priority to PCT/CN2016/113899 priority patent/WO2018120198A1/zh
Priority to CN201680003480.4A priority patent/CN107113441B/zh
Priority to PCT/CN2017/120225 priority patent/WO2018121775A1/en
Priority to CN201780070939.7A priority patent/CN109983775A/zh
Publication of WO2018120198A1 publication Critical patent/WO2018120198A1/zh
Priority to US16/446,929 priority patent/US11070732B2/en
Priority to US16/457,809 priority patent/US10911750B2/en
Anticipated expiration legal-status Critical
Priority to US17/164,429 priority patent/US20210160480A1/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/107—Selection of coding mode or of prediction mode between spatial and temporal predictive coding, e.g. picture refresh
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H04N23/683—Vibration or motion blur correction performed by a processor, e.g. controlling the readout of an image memory
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/12—Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16—Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/1607—Details of the supervisory signal
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/164—Feedback from the receiver or from the transmission channel
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • H04N19/89—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving methods or arrangements for detection of transmission errors at the decoder
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H04N23/684—Vibration or motion blur correction performed by controlling the image sensor readout, e.g. by controlling the integration time
    • H04N23/6845—Vibration or motion blur correction performed by controlling the image sensor readout, e.g. by controlling the integration time by combination of a plurality of images sequentially taken
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2101/00—UAVs specially adapted for particular uses or applications
    • B64U2101/30—UAVs specially adapted for particular uses or applications for imaging, photography or videography

Definitions

  • the present invention relates to the field of image processing, and more particularly to an image processing method, apparatus, unmanned aerial vehicle, and receiving end.
  • embodiments of the present invention provide an image processing method and apparatus, an unmanned aerial vehicle, and a receiving end.
  • an image processing method in a transmitting end including: acquiring a current image frame; and determining, according to the feedback information, a reference frame for encoding the current image frame according to the feedback information, where The feedback information is sent by the receiving end at a variable feedback time interval, and the feedback information is used to indicate a transmission state of the image frame sent by the sending end to the receiving end before the feedback information; according to the reference a frame, performing inter-coding on the current image frame to generate encoded data; and transmitting the encoded data to the receiving end.
  • a second aspect provides an image processing method in a receiving end, including: generating feedback information, where the feedback information is used to indicate that a sending end sends the sending end to the receiving end before the feedback information a transmission state of the image frame; transmitting the feedback information to the transmitting end at a variable feedback time interval; receiving encoded data of a current image frame sent by the transmitting end, wherein the transmitting end performs the current image frame
  • the reference frame used in encoding is determined by the transmitting end according to the feedback information.
  • a third aspect provides an image processing apparatus in a transmitting end, comprising: an obtaining module, configured to acquire a current image frame; and a determining module, configured to determine, according to the feedback information, the current information for encoding according to the feedback information a reference frame of the image frame, wherein the feedback information is sent by the receiving end at a variable feedback time interval, and the feedback information is used to indicate that the sending end sends the sending end to the receiving end before the feedback information
  • a coding module configured to perform inter-coding on the current image frame according to the reference frame to generate encoded data
  • a sending module configured to send the encoded data to the receiving end.
  • a fourth aspect provides an image processing apparatus in a transmitting end, comprising: at least one memory for storing computer executable instructions; and at least one processor, used alone or in common, to: access the at least one memory, and The computer executable instructions are executed to implement the method of the first aspect.
  • an unmanned aerial vehicle comprising the image processing apparatus of the third aspect or the fourth aspect.
  • the sixth aspect provides an image processing apparatus in a receiving end, comprising: a generating module, configured to generate feedback information, where the feedback information is used to indicate an image sent by the sending end to the receiving end before the feedback information a transmitting module, configured to send the feedback information to the sending end at a variable feedback time interval, and a receiving module, configured to receive encoded data of a current image frame sent by the sending end, where The reference frame used by the transmitting end to encode the current image frame is determined by the sending end according to the feedback information.
  • an image processing apparatus in a receiving end, at least one memory for storing computer executable instructions; and at least one processor, used alone or collectively, for: accessing the at least one memory, and executing the Computer executable instructions are implemented to implement the method of the second aspect.
  • a receiving end comprising the image processing apparatus of the sixth aspect or the seventh aspect.
  • a storage medium in a ninth aspect, storing instructions operable to perform the method of the first aspect.
  • a storage medium is provided, the storage medium storing an instruction, the finger The method can be used to perform the method of the second aspect.
  • the transmitting end determines a reference frame for encoding the current image frame based on the received feedback information. Therefore, the intra-coded frame or the fault-tolerant frame group can be sent to the receiving end, that is, the receiving end can recover data errors, improve the utilization of channel resources, and improve the user experience.
  • FIG. 1 is a schematic flowchart of an image processing method in a transmitting end according to an embodiment of the present invention
  • FIG. 2 is a schematic diagram of reference frame selection in accordance with an embodiment of the present invention.
  • FIG. 3 is a schematic flowchart of an image processing method in a receiving end according to an embodiment of the present invention
  • FIG. 4 is another schematic flowchart of an image processing method in a receiving end according to an embodiment of the present invention.
  • FIG. 5 is still another schematic flowchart of an image processing method in a receiving end according to an embodiment of the present invention.
  • FIG. 6 is a schematic flowchart of an image processing method according to an embodiment of the present invention.
  • FIG. 7 is a schematic block diagram of an image processing apparatus according to an embodiment of the present invention.
  • FIG. 8 is a schematic block diagram of an image processing apparatus according to another embodiment of the present invention.
  • FIG. 9 is a schematic block diagram of an image processing apparatus according to still another embodiment of the present invention.
  • FIG. 10 is a schematic block diagram of an image processing apparatus according to still another embodiment of the present invention.
  • Figure 11 is a schematic block diagram of an unmanned aerial vehicle in accordance with an embodiment of the present invention.
  • Figure 12 is a schematic block diagram of a receiving end in accordance with an embodiment of the present invention.
  • the size of the sequence numbers of the various processes does not imply a sequence of executions, and the order of execution of the processes should be determined by their function and intrinsic logic, and should not be directed to embodiments of the present disclosure.
  • the implementation process constitutes any limitation.
  • FIG. 1 is a schematic flowchart of an image processing method in a transmitting end according to an embodiment of the present invention.
  • the method is performed by an image processing device in a transmitting end, the image processing device may be various types of chips for image processing, an image processor, etc., and the transmitting end is configured to transmit encoded data of an image frame to a receiving end.
  • the method 100 includes:
  • S120 Determine, according to the feedback information, a reference frame used to encode the current image frame according to the feedback information, where the feedback information is sent by a receiving end at a variable feedback time interval, where the feedback information is used by And indicating a transmission state of the image frame sent by the sending end to the receiving end before the feedback information;
  • the transmission status refers to whether an error occurs in the transmission of the image frame. Specifically, it refers to whether data loss or data error occurs during the transmission of an image frame. Alternatively, it can be understood whether the image frame is correctly transmitted and whether it is successfully decoded after the correct transmission.
  • the coding mode used is an inter-frame coding mode or an inter-prediction coding mode, and a predicted coded frame is obtained.
  • the predictive coding frame refers to not only compressing spatial redundancy information in the image frame, but also compressing data by using time redundancy between the image frame and the specified reference frame, that is, performing time domain prediction and For encoding, such inter-coded frames are much more efficient than intra-coded frames. Therefore, the coded frames sent by the transmitting end to the receiving end are all inter-frame encoded frames, which can improve the utilization of channel resources and reduce the transmission delay.
  • the code rate when inter-coding the current image frame, can be controlled to a specific range (eg, below the maximum code rate, above the minimum code rate) or close to the target average code rate.
  • the code rate can be controlled to vary depending on the complexity of the current image frame, bandwidth limitations, buffer capacity, or other factors.
  • the image processing apparatus in the transmitting end determines the reference frame encoding the current image based on the received feedback information. Because the sender can root According to the feedback information, the image frame that has been correctly transmitted is determined, and the image frame as the reference frame is selected from the image frames that have been correctly transmitted, so that the receiving end can recover the transmission error as long as the encoded data of the current image frame is successfully received. Therefore, the image processing apparatus in the transmitting end does not need to send the intra-coded frame or the fault-tolerant frame group, that is, the image processing apparatus in the receiving end can recover the transmission error, improve the utilization of the channel resources, and improve the user experience.
  • the transmitting end in embodiments of the present invention may be moved in any suitable environment, for example, in the air (eg, a fixed-wing aircraft, a rotorcraft, or an aircraft having neither a fixed wing nor a rotor), in water (eg, a ship or a dive) Boat), on land (for example, a car or train), space (for example, a space plane, satellite or detector), and any combination of the above.
  • the transmitting end may be an unmanned aerial vehicle, such as an Unmanned Aerial Vehicle (UAV).
  • UAV Unmanned Aerial Vehicle
  • the transmittable can carry a living being, such as a human or an animal.
  • the receiving end in the embodiment of the present invention may be a computer, a handheld electronic device, a communication device, a video monitoring device, or the like.
  • the image processing apparatus in the transmitting end determines an image frame that has been correctly transmitted based on the feedback information; and determines a reference frame based on the image frame that has been correctly transmitted.
  • the image processing device in the transmitting end is correctly transmitted according to The reference frame determined by the image frame performs inter-frame coding on the current image frame, and the receiving end can recover the transmission error as long as the encoded data of the current image frame is successfully received. Therefore, the image processing apparatus in the transmitting end does not need to send the fault-tolerant frame or the fault-tolerant frame group to the image processing apparatus in the receiving end, that is, the image processing apparatus in the receiving end can recover the transmission error, improve the utilization of the channel resources, and improve the user.
  • the image processing apparatus in the transmitting end does not need to send the fault-tolerant frame or the fault-tolerant frame group to the image processing apparatus in the receiving end, that is, the image processing apparatus in the receiving end can recover the transmission error, improve the utilization of the channel resources, and improve the user.
  • the image processing apparatus in the transmitting end does not need to send the fault-tolerant frame or the fault-tolerant frame group to the image processing apparatus in the receiving end, that is, the image processing apparatus in the receiving end can recover the transmission error, improve the utilization
  • the feedback information includes at least one of the following information: indication information indicating whether the last frame image frame transmitted before the feedback information is correctly transmitted; The frame number of the correctly transmitted image frame; and the frame number of the image frame in which the last frame has been correctly transmitted before the feedback information.
  • the frame number of the image frame is globally unique, and the frame number of the image frame is sent together with the encoded data to the receiving end.
  • the image frames of 3 and 4 are correctly transmitted.
  • the feedback information includes the frame of the image frame that has been correctly transmitted. number.
  • the feedback information may specifically be “234”.
  • the feedback information includes the frame number of the image frame in which the last frame has been correctly transmitted.
  • the feedback information may be “4”, and the image processing apparatus in the transmitting end may know that the image frame numbered 5 is not according to the feedback information. Successful transmission.
  • the image processing apparatus in the transmitting end can select the image frame numbered 4 as the reference frame or the image frames numbered 3 and 4 as the reference frame when encoding the image frame numbered 6.
  • the feedback information includes an indication bit for indicating whether the image frame numbered 8 is correctly transmitted. It is assumed that the value of the indication bit is “0” indicating that the image frame numbered 8 is not correctly transmitted, and the value of the indication bit is “1”. The image frame numbered 8 has been transferred correctly. If the value of the indication bit is 1, the image processing apparatus in the transmission side selects the image frame numbered 8 as the reference frame, and interframe-codes the image frame numbered 9.
  • the image processing apparatus in the transmitting end may determine whether the image frame numbered 5 is correctly transmitted, and if the image frame numbered 5 is correctly transmitted, the image frame numbered 5 is selected as the reference frame. Interframe coding is performed on the image frame numbered 9.
  • the image processing apparatus in the transmitting end saves at least a part of the image frames that have been correctly transmitted into the reference frame management queue, and the image processing apparatus in the transmitting end determines the image frame in the reference frame management queue. Is the reference frame.
  • the image frame that has been correctly transmitted is determined according to the feedback information, and the reference frame management queue is updated.
  • the image processing apparatus in the receiving end also saves at least a part of the correctly transmitted image frame in the reference frame management queue.
  • the reference frame management queue in the sender is the same as the information stored in the reference frame management queue in the receiver.
  • the image processing apparatus in the transmitting end includes a physical or logical buffer for storing image frames corresponding to all encoded data, and the image processing apparatus in the transmitting end may only at least a part of the image frames that have been correctly transmitted.
  • the corresponding frame number is saved in the reference frame management queue, and the image processing device in the transmitting end determines the image frame corresponding to the frame number in the reference frame queue as the reference frame.
  • the image processing apparatus in the receiving end includes a physical or logical buffer for storing image frames corresponding to all the decoded data, and the image processing apparatus in the receiving end may correspond to at least a part of the correctly transmitted image frames.
  • the frame number is saved in the reference frame management queue.
  • the image processing apparatus in the receiving end sends the feedback information to the transmitting end at a variable feedback time interval, for example, the image processing apparatus in the receiving end is in the time period 1
  • the feedback time interval 1 is used to send feedback information to the image processing apparatus in the transmitting end, and the image processing apparatus in the receiving end sends feedback to the image processing apparatus in the transmitting end by using the feedback time interval 2 in the time period 2 after the time period 1 Information, feedback interval 1 is different from feedback interval 2.
  • the time period 1 and the time period 2 can be preset by the system or set by the user. Thereby, suitable coding efficiency and channel utilization can be obtained.
  • the variable feedback time interval may be determined by the image processing device in the transmitting end and then sent to the image processing device of the receiving end, or may be determined by the image processing device in the receiving end.
  • the feedback time interval may be determined by various methods according to actual conditions.
  • the feedback time interval is determined according to at least one of the following information: a load condition of the feedback channel; a demand for error recovery timelines in the application scenario of the transmitting end; a transmission error rate within a preset time period; and a historical data transmission state.
  • the feedback time interval is dynamically determined according to the posture information of the transmitting end.
  • the feedback time interval may be determined according to the feedback channel load. For example, when the load of the feedback channel reaches the threshold, the feedback time interval is increased; when the load of the feedback channel is less than the threshold, the feedback time interval is decreased.
  • the feedback time interval may be determined according to an error recovery timeliness requirement of the application scenario.
  • the error recovery time-time requirement of the current application scenario is that at least three encoded data of each of the five image frames are correct. In the actual transmission process, if the transmission correct rate reaches 3/5, the feedback time interval is increased. If the transmission accuracy rate is less than 3/5, the feedback time interval is reduced.
  • the feedback time interval may be determined according to a historical data transmission state. For example, in a preset time period, when the number of consecutive feedback information indicating a transmission error is less than the first preset value, or the total number of feedback information indicating a transmission error occurs within a preset time period is less than the second pre-predition When the value is set, the feedback time interval can be increased. During the preset time period, when the number of consecutive feedback information indicating the transmission error is not less than the first preset value, or the total number of feedback information indicating the transmission error occurs within the preset time period is not less than the second When the preset value is used, the feedback time interval can be reduced.
  • the image processing apparatus in the transmitting end may determine the feedback time interval according to the posture information of the transmitting end, wherein the posture information includes but is not limited to a three-dimensional position, a three-dimensional angle, a three-dimensional speed, and a three-dimensional acceleration. And three-dimensional angular velocity.
  • the feedback time interval can be reduced; for example, when the three-dimensional speed of the transmitting end is small (for example, 0), indicating that the difference in the change of the image picture captured on the transmitting end is small (for example, no difference), and the feedback time interval can be adjusted.
  • the sending end may determine the posture information of the transmitting end according to the posture information measured by the sensing system in the transmitting end.
  • the attitude information of the transmitting end may include at least one of the following information: a three-dimensional position, a three-dimensional angle, a three-dimensional speed, a three-dimensional acceleration, and a three-dimensional angular velocity.
  • the sensing system in the transmitting end may include at least one of the following devices: a gyroscope, an electronic compass, an Inertial Measurement Unit ("IMU”), a visual sensor, and a Global Positioning System (GPS). For "GPS" and barometer.
  • IMU Inertial Measurement Unit
  • GPS Global Positioning System
  • the feedback information is sent to the transmitting end through a variable feedback time interval, which can reduce the channel burden to a certain extent.
  • the feedback time interval may be directly determined by the image processing apparatus at the receiving end according to any one or more of the feedback channel load, the error recovery timeliness requirement of the application scenario, and the historical data transmission state. .
  • the feedback time interval may be determined by the sender and the receiver in negotiation.
  • the receiving end determines the feedback time interval 1 according to at least one of the following information: the feedback channel load, the error recovery timeliness requirement of the application scenario, and the historical data transmission state; the receiving end receives the feedback time interval 2 sent by the sending end; The receiving end determines the final feedback time interval 3 according to the feedback time interval 1 and the feedback time interval 2.
  • the receiving end determines that the feedback time interval is the first interval according to the load condition of the feedback channel, and the feedback time interval determined by the transmitting end received by the receiving end according to the posture information of the transmitting end is the second interval.
  • the pre-agreed is that when the first interval is greater than the second interval, the receiving end uses the second interval as the feedback time interval. Then, when the first interval determined by the receiving end is greater than the second interval determined by the first device, the second interval is used as the final feedback time interval.
  • the receiving end sends the feedback information to the transmitting end by using a variable feedback time interval, so that the channel utilization rate can be improved to some extent.
  • the transmitting end since the feedback information is sent by the image processing apparatus in the receiving end at variable feedback time intervals, the transmitting end is not received in the time period when the image processing apparatus in the transmitting end does not receive the feedback information.
  • the image processing apparatus in the medium may determine a preset number of image frames transmitted before the current image frame as a reference frame. For example, select the image frame closest to the current image frame as Reference frame.
  • the image processing apparatus in the transmitting end selects the image frame that has been successfully transmitted as the reference frame to encode the current image. frame.
  • the image processing apparatus in the transmitting end when the image processing apparatus in the transmitting end transmits the encoded data after the inter-frame encoding of the current image frame to the image processing apparatus in the receiving end, the image processing apparatus in the transmitting end performs the packing processing on the encoded data. Obtaining a data packet, wherein the data packet includes at least one of a packet sequence number and a check value. Then, after receiving the data packet, the image processing apparatus in the receiving end determines the transmission state of the current image frame according to at least one of the packet sequence number and the check value.
  • the packet sequence number is an RTP packet sequence number.
  • the packet sequence number is the IP packet sequence number.
  • the image processing apparatus in the transmitting end packs the encoded data encoded by the current image frame into one data packet, and the packet number of the data packet has a correspondence relationship with the frame number of the current image frame. .
  • the image processing apparatus in the receiving end continuously detects whether or not data loss occurs according to the packet sequence number of the received data packet, and the image processing apparatus in the receiving end determines that the packet number of the received data packet is missing from the current image frame.
  • the image processing apparatus in the receiving end determines that data loss has occurred, that is, the current image frame transmission fails.
  • the image processing apparatus in the receiving end calculates the check value according to the data in the data packet, and when the calculated check value does not coincide with the check value carried in the data packet, the image processing apparatus in the receiving end determines to appear.
  • the data is incorrect, that is, the current image frame transmission failed.
  • the image processing apparatus in the transmitting end packs the encoded data encoded by the current image frame into a plurality of data packets, and the packet numbers of all the data packets correspond to the frame numbers of the current image frame.
  • the image processing apparatus in the receiving end continuously detects whether or not data loss has occurred based on the packet number of the packet.
  • the image processing apparatus in the transmitting end packs the encoded data encoded by the current image frame into four data packets, the packet numbers are 1, 2, 3, and 4, respectively, when the image processing apparatus in the receiving end determines the received data.
  • the packet number of the packet is only 1, 2, and 4, it is determined that data loss occurs, that is, the transmission of the current image frame fails.
  • the image processing apparatus in the receiving end determines whether a data error has occurred according to the manner of calculating the check value described above.
  • the check value in the data packet is a Cyclical Redundancy Check (“CRC”) value.
  • CRC Cyclical Redundancy Check
  • method 300 illustrates a method of image processing according to another embodiment of the present invention, which may be performed by an image processing apparatus in a receiving end, which may be various types of chips, images for image processing. Processor, etc. As shown in FIG. 3, method 300 includes:
  • the image processing apparatus in the receiving end transmits the feedback information to the image processing apparatus in the transmitting end at variable feedback time intervals, so that the image processing apparatus in the transmitting end can receive according to the The feedback information determines the reference frame encoding the current image frame. Therefore, the image processing apparatus in the transmitting end does not need to send the intra-coded frame or the fault-tolerant frame group, that is, the image processing apparatus in the receiving end can recover the transmission error, improve the utilization of the channel resources, and improve the user experience.
  • the method 300 further includes:
  • the feedback information includes at least one of the following information: indication information indicating whether a last frame image frame sent before the feedback information is correctly transmitted; The frame number of the correctly transmitted image frame; and the frame number of the image frame to which the last frame was correctly transmitted before the feedback information.
  • the method 300 further includes:
  • variable feedback time interval may be determined according to at least one of the following information: a load condition of the feedback channel, a requirement of the application scenario of the sending end for error recovery timelines, and a preset time period.
  • the transmission error rate within.
  • variable feedback time interval may be determined by receiving the variable feedback time interval sent by the sending end.
  • variable feedback time interval is dynamically determined by the sending end according to the posture change rate of the sending end.
  • the data packet sent by the sending end may be received, where the data packet is obtained by the sending end to perform packet processing on the encoded data, where the data packet further includes: At least one of a package serial number and a check value;
  • the method 300 further includes determining, according to at least one of the packet sequence number and the check value, a transmission state of the current image frame.
  • the transmitting end is an unmanned aerial vehicle.
  • FIG. 6 illustrates a method of image processing according to still another embodiment of the present invention. As shown in FIG. 6, the method 600 includes:
  • the sending end sends an image frame to the receiving end.
  • the receiving end generates feedback information according to a transmission state of the image frame.
  • the receiving end sends the feedback information to the sending end at a variable feedback time interval.
  • the transmitting end determines, according to the feedback information, a reference frame used to encode the current image frame.
  • the transmitting end performs inter-frame coding on the current image frame according to the reference frame to generate encoded data.
  • the sending end sends the encoded data to the receiving end.
  • the receiving end performs inter-frame decoding on the encoded data according to the reference frame to generate decoded data.
  • the receiving end displays the current image frame according to the decoded data.
  • FIG. 7 is a schematic block diagram of an image processing apparatus in a transmitting end according to an embodiment of the present invention. As shown in FIG. 7, the image processing apparatus 10 includes:
  • the obtaining module 11 is configured to: acquire a current image frame
  • a determining module 12 configured to determine, according to the feedback information, based on the feedback information a reference frame for encoding the current image frame, wherein the feedback information is sent by the receiving end at a variable feedback time interval, the feedback information being used to indicate the sending end before the feedback information a transmission state of an image frame sent by the receiving end;
  • the encoding module 13 is configured to perform inter-coding on the current image frame according to the reference frame to generate encoded data.
  • the sending module 14 is configured to send the encoded data to the receiving end.
  • the image processing apparatus in the transmitting end determines the reference frame encoding the current image based on the received feedback information. Therefore, it is not necessary to transmit an intra-coded frame or a fault-tolerant frame group, that is, the image processing device in the receiving end can recover the transmission error, improve the utilization of channel resources, and improve the user experience.
  • the determining module 12 is configured to:
  • the reference frame is determined based on the image frame that has been correctly transmitted.
  • the determining module 12 is further configured to:
  • An image frame in the reference frame management queue is determined as the reference frame.
  • the determining module 12 is further configured to:
  • a predetermined number of image frames transmitted before the current image frame are determined as the reference frame.
  • the feedback information includes at least one of the following information:
  • the frame number of the image frame to which the last frame was correctly transmitted before the feedback information is the frame number of the image frame to which the last frame was correctly transmitted before the feedback information.
  • the determining module 12 is further configured to:
  • the sending module 14 is further configured to: send the variable feedback time interval to the receiving end.
  • the determining module 12 is configured to:
  • variable feedback time interval is determined based on at least one of the following information:
  • the application scenario of the sender is required for error recovery timeliness
  • the determining module 12 is configured to:
  • the feedback time interval is adjusted according to the attitude change rate.
  • the sending module 14 is configured to:
  • the packet processing on the encoded data to obtain a data packet, where the data packet further includes: at least one of a packet sequence number and a check value;
  • the sending end is an unmanned aerial vehicle.
  • the image processing apparatus 10 can perform the method 100 and various alternative embodiments thereof, and for the sake of brevity, no further details are provided herein.
  • FIG. 8 is a schematic block diagram of an image processing apparatus in a receiving end according to an embodiment of the present invention. As shown in FIG. 8, the image processing apparatus 20 includes:
  • a generating module 21 configured to generate feedback information, where the feedback information is used to indicate a transmission state of an image frame sent by the sending end to the receiving end before the feedback information;
  • the sending module 22 is configured to send the feedback information to the sending end at a variable feedback time interval
  • the receiving module 23 is configured to receive the encoded data of the current image frame sent by the sending end, where the reference frame used by the transmitting end to encode the current image frame is determined by the sending end according to the feedback information.
  • the image processing apparatus in the receiving end transmits the feedback information to the image processing apparatus in the transmitting end at variable feedback time intervals, so that the image processing apparatus in the transmitting end can be based on the received feedback information.
  • a reference frame encoding the current image frame is determined. Therefore, the image processing apparatus in the transmitting end does not need to send the intra-coded frame or the fault-tolerant frame group, that is, the image processing apparatus in the receiving end can recover the transmission error, improve the utilization of the channel resources, and improve the user experience.
  • the sending module 22 is further configured to:
  • At least a portion of the image frames that have been correctly transmitted are saved to the reference frame management queue.
  • the feedback information includes at least one of the following information:
  • the frame number of the image frame to which the last frame was correctly transmitted before the feedback information is the frame number of the image frame to which the last frame was correctly transmitted before the feedback information.
  • the sending module 22 is further configured to:
  • variable feedback time interval is determined.
  • the sending module 22 is configured to:
  • variable feedback time interval is determined based on at least one of the following information:
  • the application scenario of the sender is required for error recovery timeliness
  • the receiving module 23 is further configured to:
  • variable feedback time interval is that the sending end dynamically determines the rate of change of the posture of the sending end.
  • the receiving module 23 is configured to:
  • the data packet sent by the sending end, where the data packet is obtained by the sending end, and the data packet further includes: a packet sequence number and/or a check value;
  • the sending end is an unmanned aerial vehicle.
  • the image processing apparatus 20 can perform the method 300 and various alternative embodiments thereof, and for the sake of brevity, no further details are provided herein.
  • FIG. 9 is a schematic block diagram of an image processing apparatus in a transmitting end according to another embodiment of the present invention.
  • image processing apparatus 1000 includes at least one memory 1010 for storing computer executable instructions; at least one processor 1020.
  • the at least one processor 1020 is used alone or in common to: access the at least one memory 1010 and execute the computer executable instructions to implement the method 100 and various alternative embodiments thereof, for the sake of brevity, Let me repeat.
  • FIG. 10 is a schematic diagram of an image processing apparatus in a receiving end according to another embodiment of the present invention. block diagram.
  • image processing apparatus 2000 includes at least one memory 2010 for storing computer executable instructions; at least one processor 2020.
  • the at least one processor 2020 is used alone or collectively to: access the at least one memory 2010 and execute the computer executable instructions to implement the method 300 and various alternative embodiments thereof, for the sake of brevity, Let me repeat.
  • the processor in the embodiment of the present invention may be a central processing unit (Central Processing Unit, abbreviated as "CPU"), a network processor (Network Processor, abbreviated as "NP”), or a combination of a CPU and an NP.
  • the processor may further include a hardware chip.
  • the hardware chip may be an Application-Specific Integrated Circuit ("ASIC"), a Programmable Logic Device (PLD), or a combination thereof.
  • the PLD may be a Complex Programmable Logic Device (CPLD), a Field-Programmable Gate Array (FPGA), and a Generic Array Logic (Generic Array Logic). Referred to as "GAL” or any combination thereof.
  • the memory in the embodiment of the present invention may include a read only memory and a random access memory, and provides instructions and data to the processor.
  • a portion of the memory may also include a non-volatile random access memory.
  • the memory can also store information of the device type.
  • the unmanned aerial vehicle 30 includes the image processing apparatus 10 shown in FIG. 7, or the unmanned aerial vehicle 30 includes the image processing apparatus 1000 shown in FIG.
  • Figure 12 is a schematic block diagram of a receiving end in accordance with an embodiment of the present invention.
  • the receiving end 40 includes the image processing device 20 shown in FIG. 8 or the image processing device 2000 shown in FIG.
  • the disclosed system and device The method of setting can be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in various embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the functions may be stored in a computer readable storage medium if implemented in the form of a software functional unit and sold or used as a standalone product. Based on such understanding, the portion of the technical solution of the present disclosure that contributes in essence or to the prior art or the portion of the technical solution may be embodied in the form of a software product stored in a storage medium, including The instructions are used to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present disclosure.
  • the foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Aviation & Aerospace Engineering (AREA)

Abstract

一种图像处理方法、装置、无人飞行器和接收端，该方法包括：获取当前图像帧；响应于反馈信息，根据所述反馈信息确定用于编码所述当前图像帧的参考帧，其中，所述反馈信息是由接收端以可变的反馈时间间隔发送的，所述反馈信息用于指示在所述反馈信息之前所述发送端向所述接收端发送的图像帧的传输状态；根据所述参考帧，对所述当前图像帧进行帧间编码，以生成编码数据；向所述接收端发送所述编码数据。能够提高信道资源的利用率，提升用户体验。

Description

图像处理方法、装置、无人飞行器和接收端
版权申明
本专利文件披露的内容包含受版权保护的材料。该版权为版权所有人所有。版权所有人不反对任何人复制专利与商标局的官方记录和档案中所存在的该专利文件或者该专利披露。
技术领域
本发明涉及图像处理领域，并且更具体地，涉及图像处理方法、装置、无人飞行器和接收端。
背景技术
无线和不可靠信道上低时延视频传输系统是目前的热点研究和应用方向。对于不可靠信道的视频传输来说，数据传输过程中会发生数据丢失及数据传输错误的情况，导致视频解码出错。但是发送端无法预测何时发生数据丢失和数据传输错误。因此，需要有相应的错误恢复机制，来纠正已经发生的视频数据错误。
但是，目前常用的纠正数据传输错误的方法会浪费信道资源带宽、带来传输时延抖动，影响用户的体验。
发明内容
为了解决现有技术的上述以及其他潜在问题，本发明实施例提供一种图像处理方法、装置、无人飞行器和接收端。
第一方面，提供了一种发送端中的图像处理方法，包括：获取当前图像帧；响应于反馈信息，根据所述反馈信息确定用于编码所述当前图像帧的参考帧，其中，所述反馈信息是由接收端以可变的反馈时间间隔发送的，所述反馈信息用于指示在所述反馈信息之前所述发送端向所述接收端发送的图像帧的传输状态；根据所述参考帧，对所述当前图像帧进行帧间编码，以生成编码数据；向所述接收端发送所述编码数据。
第二方面，提供了一种接收端中的图像处理方法，包括：生成反馈信息，所述反馈信息用于指示在所述反馈信息之前发送端向所述接收端发送 的图像帧的传输状态；以可变的反馈时间间隔向所述发送端发送所述反馈信息；接收所述发送端发送的当前图像帧的编码数据，其中，所述发送端对当前图像帧进行编码时采用的参考帧是所述发送端根据所述反馈信息确定的。
第三方面，提供了一种发送端中的图像处理装置，包括：获取模块，用于获取当前图像帧；确定模块，用于响应于反馈信息，根据所述反馈信息确定用于编码所述当前图像帧的参考帧，其中，所述反馈信息是由接收端以可变的反馈时间间隔发送的，所述反馈信息用于指示在所述反馈信息之前所述发送端向所述接收端发送的图像帧的传输状态；编码模块，用于根据所述参考帧，对所述当前图像帧进行帧间编码，以生成编码数据；发送模块，用于向所述接收端发送所述编码数据。
第四方面，提供了一种发送端中的图像处理装置，包括：至少一个存储器，用于存储计算机可执行指令；至少一个处理器，单独或共同地用于：访问所述至少一个存储器，并执行所述计算机可执行指令，以实施第一方面所述的方法。
第五方面，提供了一种无人飞行器，包括第三方面或第四方面的图像处理装置。
第六方面，提供了一种接收端中的图像处理装置，包括：生成模块，用于生成反馈信息，所述反馈信息用于指示在所述反馈信息之前发送端向所述接收端发送的图像帧的传输状态；发送模块，用于以可变的反馈时间间隔向所述发送端发送所述反馈信息；接收模块，用于接收所述发送端发送的当前图像帧的编码数据，其中，所述发送端对当前图像帧进行编码时采用的参考帧是所述发送端根据所述反馈信息确定的。
第七方面，提供了一种接收端中的图像处理装置，至少一个存储器，用于存储计算机可执行指令；至少一个处理器，单独或共同地用于：访问所述至少一个存储器，并执行所述计算机可执行指令，以实施第二方面所述的方法。
第八方面，提供了一种接收端，包括第六方面或第七方面的图像处理装置。
第九方面，提供了一种存储介质，所述存储介质存储指令，所述指令可以用于执行第一方面中的方法。
第十方面，提供了一种存储介质，所述存储介质存储指令，所述指 令可以用于执行第二方面中的方法。
根据本发明的实施例，发送端根据接收到的反馈信息确定用于编码当前图像帧的参考帧。由此，可以不用向接收端发送帧内编码帧或容错帧组，即可以使得接收端恢复数据错误，提高信道资源的利用率，提升用户体验。
附图说明
为了更清楚地说明本发明实施例的技术方案，下面将对本发明实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1根据本发明实施例的发送端中的图像处理方法的示意性流程图；
图2是根据本发明实施例的参考帧选择的示意图；
图3是根据本发明实施例的接收端中的图像处理方法的示意性流程图；
图4是根据本发明实施例的接收端中的图像处理方法的另一示意性流程图；
图5是根据本发明实施例的接收端中的图像处理方法的再一示意性流程图；
图6是根据本发明实施例的图像处理方法的示意性流程图；
图7是根据本发明实施例的图像处理装置的示意性框图；
图8是根据本发明另一实施例的图像处理装置的示意性框图；
图9是根据本发明再一实施例的图像处理装置的示意性框图；
图10是根据本发明再一实施例的图像处理装置的示意性框图；
图11是根据本发明实施例的无人飞行器的示意性框图；
图12是根据本发明实施例的接收端的示意性框图。
具体实施方式
下面将结合附图，对本发明实施例中的技术方案进行描述。
应理解，本文中的具体的例子只是为了帮助本领域技术人员更好地理解本公开实施例，而非限制本公开实施例的范围。
还应理解，在本公开的各种实施例中，各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本公开实施例的实施过程构成任何限定。
图1是根据本发明实施例的发送端中的图像处理方法的示意性流程图。该方法由发送端中的图像处理装置执行，所述图像处理装置可以是各种类型的用于图像处理的芯片、图像处理器等，所述发送端用于向接收端发送图像帧的编码数据。如图1所示，方法100包括：
S110，获取当前图像帧；
S120，响应于反馈信息，根据所述反馈信息确定用于编码所述当前图像帧的参考帧，其中，所述反馈信息是由接收端以可变的反馈时间间隔发送的，所述反馈信息用于指示在所述反馈信息之前所述发送端向所述接收端发送的图像帧的传输状态；
需要说明的是，传输状态指的是图像帧的传输是否出现错误。具体地说，指的是图像帧的传输过程中是否出现数据丢失或数据错误。或者，可以理解为图像帧是否被正确传输，以及正确传输后是否被成功解码。
S130，根据所述参考帧，对所述当前帧进行帧间编码，以生成编码数据；
可以看出，每次在编码当前图像帧时，采用的编码方式均为帧间编码方式或称为帧间预测编码方式，得到预测编码帧。其中，预测编码帧是指不但压缩本图像帧内的空间冗余信息，还会利用本图像帧与指定的参考帧之间的时间冗余来压缩数据的编码方式，也就是进行时域预测并进行编码，这种帧间编码帧的压缩效率要比帧内编码帧高很多。由此，发送端向接收端发送的编码后的帧均为帧间编码帧，能够提高信道资源的利用率，降低传输时延。
可选地，在对当前图像帧进行帧间编码时，可以将码率控制到一个特定的范围内(例如，低于最大码率，高于最小码率)或者接近目标平均码率。可选地，可以控制码率依赖于当前图像帧的复杂度，带宽限制，缓存容量或其他因素而变化。
S140，向所述接收端发送所述编码数据。
因此，根据本发明实施例的图像处理方法，发送端中的图像处理装置根据接收到的反馈信息，确定编码当前图像的参考帧。由于发送端可以根 据反馈信息确定出已被正确传输的图像帧，并从已被正确传输的图像帧中选择作为参考帧的图像帧，使得接收端只要成功接收当前图像帧的编码数据，即可以恢复传输错误。由此，发送端中的图像处理装置无需发送帧内编码帧或者容错帧组，即可以使得接收端中的图像处理装置恢复传输错误，提高信道资源的利用率，提升用户体验。
本发明实施例中的发送端可以在任何合适的环境下移动，例如，空气中(例如，定翼飞机、旋翼飞机，或既没有定翼也没有旋翼的飞机)、水中(例如，轮船或潜水艇)、陆地上(例如，汽车或火车)、太空(例如，太空飞机、卫星或探测器)，以及以上各种环境的任何组合。发送端可以是无人飞行器，例如无人机(Unmanned Aerial Vehicle，简称为“UAV”)。在一些实施例中，可发送端可以承载生命体，例如，人或动物。
本发明实施例中的接收端可以是计算机、手持式电子设备、通讯设备、视频监控设备等。
在S120中，发送端中的图像处理装置根据反馈信息确定已被正确传输的图像帧；并根据已被正确传输的图像帧确定参考帧。
与现有通过周期性发送容错帧(帧内编码帧)或容错帧组来纠正数据传输错误的技术方案不同，本发明实施例中的方法，发送端中的图像处理装置通过根据已被正确传输的图像帧确定的参考帧，对当前图像帧进行帧间编码，接收端只要成功接收当前图像帧的编码数据，即可以恢复传输错误。由此，发送端中的图像处理装置无需向接收端中的图像处理装置发送容错帧或容错帧组，即可以使得接收端中的图像处理装置恢复传输错误，提高信道资源的利用率，提升用户体验。
在本发明实施例中，作为示例而非限定，反馈信息包括下列信息中的至少一种：用于指示在所述反馈信息之前发送的最后一帧图像帧是否被正确传输的指示信息；已被正确传输的图像帧的帧号；以及在所述反馈信息之前最后一帧已被正确传输的图像帧的帧号。
可以理解的是，图像帧的帧号是全局唯一的，并且图像帧的帧号和编码数据一起发送给接收端的。
举例来说，如图2中所示出的，假设在当前图像帧(编号6)之前，发送端已经向接收端发送了5个图像帧，对应编号依次为1-5，并且编号为2、3和4的图像帧被正确传输。反馈信息中包括已被正确传输的图像帧的帧 号。此时，反馈信息具体可以为“234”。或者，反馈信息包括最后一帧已被正确传输的图像帧的帧号，此时，反馈信息可以为“4”，发送端中的图像处理装置根据反馈信息可以知道编号为5的图像帧未被成功传输。由此，发送端中的图像处理装置在编码编号为6的图像帧时，可以选择编号为4的图像帧为参考帧，或者选择编号为3和4的图像帧为参考帧。
或者，假设在当前图像帧(编号9)之前，发送端已经向接收端发送了8个图像帧，对应编号依次为1-8，并且接收端已经针对编号为1-5的图像帧进行过反馈。在此次反馈信息中只需要对编号为8的图像帧的传输状态进行反馈。反馈信息中包括用于指示编号为8的图像帧是否正确传输的指示位，假定指示位的值为“0”表示编号为8的图像帧未被正确传输，指示位的值为“1”表示编号为8的图像帧已被正确传输。如果指示位的值为1，发送端中的图像处理装置选择编号为8的图像帧作为参考帧，对编号为9的图像帧进行帧间编码。如果指示位的值为0，发送端中的图像处理装置可以确定编号为5的图像帧是否被正确传输，如果编号为5的图像帧被正确传输，选择编号为5的图像帧作为参考帧，对编号为9的图像帧进行帧间编码。
或者，进一步地，发送端中的图像处理装置将已被正确传输的图像帧中的至少一部分保存到参考帧管理队列中，发送端中的图像处理装置将参考帧管理队列中的图像帧，确定为参考帧。由此，每次接收到接收端发送的反馈信息时，根据反馈信息确定已被正确传输的图像帧，并更新参考帧管理队列。相对应地，接收端中的图像处理装置在确定已被正确传输的图像帧后，也将正确传输的图像帧中的至少一部分保存在参考帧管理队列中。发送端中的参考帧管理队列与接收端中的参考帧管理队列中保存的信息相同。
或者，发送端中的图像处理装置中包括用于存储所有编码数据对应的图像帧的物理或逻辑上的缓存，发送端中的图像处理装置可以只将已被正确传输的图像帧中的至少一部分对应的帧号保存到参考帧管理队列中，发送端中的图像处理装置将参考帧队列中的帧号对应的图像帧确定为参考帧。相对应地，接收端中的图像处理装置中包括用于存储所有解码数据对应的图像帧的物理或逻辑上的缓存，接收端中的图像处理装置可以将正确传输的图像帧中的至少一部分对应的帧号保存在参考帧管理队列中。
在本发明实施例中，接收端中的图像处理装置是以可变的反馈时间间隔向发送端发送反馈信息的，例如，接收端中的图像处理装置在时间段1 内采用反馈时间间隔1向发送端中的图像处理装置发送反馈信息，接收端中的图像处理装置在时间段1之后的时间段2内采用反馈时间间隔2向发送端中的图像处理装置发送反馈信息，反馈时间间隔1与反馈时间间隔2不同。具体地，时间段1与时间段2可以系统预设或用户设置。由此可以获得合适的编码效率和信道利用率。其中，可变的反馈时间间隔可以是发送端中的图像处理装置确定后发送给接收端的图像处理装置的，也可以是接收端中的图像处理装置自己确定的。
具体来说，反馈时间间隔可以是根据实际情况采用多种方法确定的。例如，反馈时间间隔是根据下列信息中的至少一种确定的：反馈信道的负载情况；发送端的应用场景对错误恢复及时性的需求；预设时段内的传输错误率；以及历史数据传输状态。或者，当该发送端为无人机时，该反馈时间间隔是根据发送端的姿态信息动态确定的。
其中，可以根据反馈信道负载确定该反馈时间间隔，例如，当反馈信道的负载达到阈值时，将反馈时间间隔调大；当反馈信道的负载小于阈值时，将反馈时间间隔调小。
其中，可以根据应用场景的错误恢复及时性需求确定该反馈时间间隔。例如，当前应用场景的错误恢复及时性需求为每发5个图像帧的编码数据至少有3个是正确的，实际传输过程中，若传输正确率达到3/5，则调大该反馈时间间隔，若传输正确率不到3/5，则调小该反馈时间间隔。
其中，可以根据历史数据传输状态确定该反馈时间间隔。例如在预设时间段内，当连续出现指示传输错误的反馈信息的个数小于第一预置数值时，或者在预设时间段内出现指示传输错误的反馈信息的总个数小于第二预置数值时，则可以调大该反馈时间间隔。在预设时间段内，当连续出现指示传输错误的反馈信息的个数不小于第一预置数值时，或者在预设时间段内出现指示传输错误的反馈信息的总个数不小于第二预置数值时，则可以调小该反馈时间间隔。
其中，当发送端例如为无人飞行器时，发送端中的图像处理装置可以根据发送端的姿态信息确定该反馈时间间隔，其中该姿态信息包括但不限于三维位置、三维角度、三维速度、三维加速度和三维角速度。例如，当第一设备的三维速度较大时，说明在发送端上捕获的图像画面的变化差异较大，可以调小该反馈时间间隔；例如，当发送端的三维速度较小时(例如为 0)，说明在发送端上捕获的图像画面的变化差异较小(例如无差异)，可以调大该反馈时间间隔。
其中，发送端可以根据发送端中的传感系统测量的姿态信息确定发送端的姿态信息。发送端的姿态信息可以包括下列信息中的至少一种：三维位置、三维角度、三维速度、三维加速度和三维角速度。发送端中的传感系统可以包括下列器件中的至少一种：陀螺仪、电子罗盘、惯性测量单元(Inertial Measurement Unit，简称为“IMU”)、视觉传感器、全球定位系统(Global Positioning System，简称为“GPS”)和气压计。
本发明实施例通过可变的反馈时间间隔，向发送端发送反馈信息，能在一定程度上降低信道负担。
可选地，作为一种实施例，该反馈时间间隔可以直接由接收端的图像处理装置根据反馈信道负载、应用场景的错误恢复及时性需求与历史数据传输状态中的任一种或多种信息确定。
具体方法参见上文描述，这里不再赘述。
可选地，作为一种实施例，该反馈时间间隔可以由发送端与接收端协商确定。
具体地，接收端根据下列信息中的至少一种确定反馈时间间隔1：反馈信道负载、应用场景的错误恢复及时性需求与历史数据传输状态；该接收端接收发送端发送的反馈时间间隔2；该接收端根据反馈时间间隔1与反馈时间间隔2，确定最终的反馈时间间隔3。举例来说，假设接收端根据反馈信道的负载情况确定出反馈时间间隔为第一间隔，而接收端接收到的发送端根据发送端的姿态信息确定的反馈时间间隔为第二间隔。假设预先约定为，当第一间隔大于第二间隔时，接收端采用第二间隔作为反馈时间间隔。则当接收端确定的第一间隔大于第一设备确定的第二间隔时，以第二间隔作为最终的反馈时间间隔。
在本发明实施例中，接收端通过采用可变的反馈时间间隔向发送端发送反馈信息，可以在一定程度上提高信道利用率。
在本发明实施例中，由于反馈信息是接收端中的图像处理装置以可变的反馈时间间隔发送的，因此，在发送端中的图像处理装置未接收到反馈信息的时间段内，发送端中的图像处理装置可以将在当前图像帧之前发送的预设数量的图像帧确定为参考帧。例如，选择当前图像帧最近的图像帧作为 参考帧。当发送端中的图像处理装置接收到反馈信息，并且反馈信息中携带的图像帧的帧号出现跳变时，发送端中的图像处理装置选择已被成功传输的图像帧作为参考帧编码当前图像帧。
在上述所有实施例中，发送端中的图像处理装置向接收端中的图像处理装置发送对当前图像帧进行帧间编码后的编码数据时，发送端中的图像处理装置对编码数据进行打包处理，获得数据包，其中，数据包中包括包序号和校验值中的至少一项。之后接收端中的图像处理装置接收到数据包后根据包序号和校验值中的至少一项确定当前图像帧的传输状态。
可选地，在实时传输协议(Real-time Transport Protocol，简称为“RTP”)中，包序号是RTP包序号。在传输控制协议/因特网互联协议(Transmission Control Protocol/Internet Protocol，简称为“TCP/IP”)中，包序号是IP包序号。
在本发明实施例中，作为示例而非限定，发送端中的图像处理装置将当前图像帧编码后的编码数据打包为一个数据包，数据包的包号与当前图像帧的帧号具有对应关系。接收端中的图像处理装置根据接收到的数据包的包序号是否连续检测是否发生数据丢失，当接收端中的图像处理装置确定接收到的数据包的包号中缺少与所述当前图像帧对应的数据包的包号时，接收端中的图像处理装置确定发生数据丢失，即所述当前图像帧传输失败。或者，接收端中的图像处理装置根据数据包中的数据计算校验值，当计算得出的校验值与该数据包中携带的校验值不一致时，接收端中的图像处理装置确定出现数据错误，即当前图像帧传输失败。
或者，发送端中的图像处理装置将当前图像帧编码后的编码数据打包为多个数据包，所有数据包的包号均对应所述当前图像帧的帧号。此时，同样的，接收端中的图像处理装置根据数据包的包号是否连续检测是否发生数据丢失。例如，发送端中的图像处理装置将当前图像帧编码后的编码数据打包为4个数据包，包号分别为1、2、3和4，当接收端中的图像处理装置确定接收到的数据包的包号只有1、2和4时，确定发生数据丢失，即当前图像帧的传输失败。或者接收端中的图像处理装置根据上述的计算校验值的方式确定是否发生数据错误。
可选地，在上述实施例中，数据包中的校验值为循环冗余校验(Cyclical Redundancy Check，简称为“CRC”)值。
以上结合图1和图2从发送端侧详细描述了根据本发明实施例的图像处理的方法，下面将结合图3从接收端侧详细描述根据本发明另一实施例的图像处理的方法。应理解，发送端侧描述的接收端与发送端的交互及相关特性、功能等与接收端侧的描述相应，为了简洁，适当省略重复的描述。
图3示出了根据本发明另一实施例的图像处理的方法，该方法可以由接收端中的图像处理装置执行，所述图像处理装置可以是各种类型的用于图像处理的芯片、图像处理器等。如图3所示，方法300包括：
S310，生成反馈信息，所述反馈信息用于指示在所述反馈信息之前发送端向所述接收端发送的图像帧的传输状态；
S320，以可变的反馈时间间隔向所述发送端发送所述反馈信息；
S330，接收所述发送端发送的当前图像帧的编码数据，其中，所述发送端对当前图像帧进行编码时采用的参考帧是所述发送端根据所述反馈信息确定的。
因此，根据本发明实施例的图像处理方法，接收端中的图像处理装置向发送端中的图像处理装置以可变的反馈时间间隔发送反馈信息，使得发送端中的图像处理装置能够根据接收到的反馈信息，确定编码当前图像帧的参考帧。由此，发送端中的图像处理装置无需发送帧内编码帧或者容错帧组，即可以使得接收端中的图像处理装置恢复传输错误，提高信道资源的利用率，提升用户体验。
可选地，如图4所示，在S320之前，所述方法300还包括：
S340，将已被正确传输的图像帧中的至少一部分保存到参考帧管理队列中。
在本发明实施例中，可选地，所述反馈信息包括下列信息中的至少一种：用于指示在所述反馈信息之前发送的最后一帧图像帧是否被正确传输的指示信息；已被正确传输的图像帧的帧号；以及在所述反馈信息之前最后一帧被正确传输的图像帧的帧号。
在本本发明实施例中，可选地，如图5所示，所述方法300还包括：
S350，确定所述可变的反馈时间间隔。
在本发明实施例中，可以根据下列信息中的至少一种确定所述可变的反馈时间间隔：反馈信道的负载情况、所述发送端的应用场景对错误恢复及时性的需求、和预设时段内的传输错误率。
在本发明实施例中，可以通过接收所述发送端发送的所述可变的反馈时间间隔，确定所述可变的反馈时间间隔。
在本发明实施例中，可选地，所述可变的反馈时间间隔是所述发送端根据所述发送端的姿态变化速率动态确定的。
可选地，在本本发明实施例中，可以接收所述发送端发送的数据包，其中，所述数据包是所述发送端将对编码数据进行打包处理获得的，所述数据包还包括：包序号和校验值中的至少一项；
其中，所述方法300还包括：根据所述包序号和校验值中的至少一项，确定所述当前图像帧的传输状态。
在本本发明实施例中，可选地，所述发送端为无人飞行器。
图6示出了根据本发明再一实施例的图像处理的方法，如图6所示，方法600包括：
S610，发送端向接收端发送图像帧；
S620，接收端根据图像帧的传输状态，生成反馈信息；
S630，接收端以可变的反馈时间间隔向发送端发送反馈信息；
S640，发送端根据反馈信息确定用于编码当前图像帧的参考帧；
S650，发送端根据参考帧，对当前图像帧进行帧间编码，以生成编码数据；
S660，发送端向接收端发送所述编码数据；
S670，接收端根据所述参考帧，对所述编码数据进行帧间解码，以生成解码数据；
S680，接收端根据解码数据，显示当前图像帧。
可以理解的是，方法600中接收端与发送端之间的交互及相关特性、功能等与方法100中的描述相对应，为了简洁，在此不再赘述。
以上描述了根据本发明实施例的图像处理方法，以下将描述根据本发明实施例的图像处理装置。但应理解，以下描述的图像处理装置可以实现以上的图像处理方法，为了避免重复，以下将简洁描述。
图7是根据本发明实施例的发送端中的图像处理装置的示意性框图。如图7所示，图像处理装置10包括：
获取模块11，用于：获取当前图像帧；
确定模块12，用于响应于发送的反馈信息，根据所述反馈信息确定 用于编码所述当前图像帧的参考帧，其中，所述反馈信息是由接收端以可变的反馈时间间隔发送的，所述反馈信息用于指示在所述反馈信息之前所述发送端向所述接收端发送的图像帧的传输状态；
编码模块13，用于根据所述参考帧，对所述当前图像帧进行帧间编码，以生成编码数据；
发送模块14，用于向所述接收端发送所述编码数据。
因此，根据本发明实施例的发送端中的图像处理装置根据接收到的反馈信息，确定编码当前图像的参考帧。由此，无需发送帧内编码帧或者容错帧组，即可以使得接收端中的图像处理装置恢复传输错误，提高信道资源的利用率，提升用户体验。
可选地，作为一个实施例，所述确定模块12用于：
根据所述反馈信息确定已被正确传输的图像帧；
根据所述已被正确传输的图像帧确定所述参考帧。
可选地，作为一个实施例，所述确定模块12还用于：
将所述已被正确传输的图像帧中的至少一部分保存到参考帧管理队列中；
将所述参考帧管理队列中的图像帧，确定为所述参考帧。
可选地，作为一个实施例，所述确定模块12还用于：
响应于未接收到所述反馈信息，将在所述当前图像帧之前发送的预设数量的图像帧确定为所述参考帧。
可选地，作为一个实施例，所述反馈信息包括下列信息中的至少一种：
用于指示在所述反馈信息之前发送的最后一帧图像帧是否被正确传输的指示信息；
已被正确传输的图像帧的帧号；以及
在所述反馈信息之前最后一帧被正确传输的图像帧的帧号。
可选地，作为一个实施例，所述确定模块12还用于：
确定所述可变的反馈时间间隔；
其中，所述发送模块14还用于：向所述接收端发送所述可变的反馈时间间隔。
可选地，作为一个实施例，所述确定模块12用于：
根据下列信息中的至少一种确定所述可变的反馈时间间隔：
反馈信道的负载情况；
所述发送端的应用场景对错误恢复及时性的需求；以及
预设时段内的传输错误率。
可选地，作为一个实施例，所述确定模块12用于：
确定所述发送端的姿态变化速率；
根据所述姿态变化速率调整所述反馈时间间隔。
可选地，作为一个实施例，所述发送模块14用于：
将所述编码数据进行打包处理，获得数据包，其中，所述数据包还包括：包序号和校验值中的至少一项；
向所述接收端发送所述数据包。
可选地，作为一个实施例，所述发送端为无人飞行器。
应理解，图像处理装置10可以执行方法100及其各种可选实施例，为了简洁，在此不再赘述。
图8是根据本发明实施例的接收端中的图像处理装置的示意性框图。如图8所示，图像处理装置20包括：
生成模块21，用于生成反馈信息，所述反馈信息用于指示在所述反馈信息之前发送端向所述接收端发送的图像帧的传输状态；
发送模块22，用于以可变的反馈时间间隔向所述发送端发送所述反馈信息；
接收模块23，用于接收所述发送端发送的当前图像帧的编码数据，其中，所述发送端对当前图像帧进行编码时采用的参考帧是所述发送端根据所述反馈信息确定的。
因此，根据本发明实施例的接收端中的图像处理装置向发送端中的图像处理装置以可变的反馈时间间隔发送反馈信息，使得发送端中的图像处理装置能够根据接收到的反馈信息，确定编码当前图像帧的参考帧。由此，发送端中的图像处理装置无需发送帧内编码帧或者容错帧组，即可以使得接收端中的图像处理装置恢复传输错误，提高信道资源的利用率，提升用户体验。
可选地，作为一个实施例，所述发送模块22还用于：
将已被正确传输的图像帧中的至少一部分保存到参考帧管理队列中。
可选地，作为一个实施例，所述反馈信息包括下列信息中的至少一种：
用于指示在所述反馈信息之前发送的最后一帧图像帧是否被正确传输的指示信息；
已被正确传输的图像帧的帧号；以及
在所述反馈信息之前最后一帧被正确传输的图像帧的帧号。
可选地，作为一个实施例，所述发送模块22还用于：
确定所述可变的反馈时间间隔。
可选地，作为一个实施例，所述发送模块22用于：
根据下列信息中的至少一种确定所述可变的反馈时间间隔：
反馈信道的负载情况；
所述发送端的应用场景对错误恢复及时性的需求；以及
预设时段内的传输错误率。
可选地，作为一个实施例，所述接收模块23还用于：
接收所述发送端发送的所述可变的反馈时间间隔。
可选地，作为一个实施例，所述可变的反馈时间间隔是所述发送端根据所述发送端的姿态变化速率动态确定的。
可选地，作为一个实施例，所述接收模块23用于：
接收所述发送端发送的数据包，其中，所述数据包是所述发送端将所述编码数据进行打包处理获得的，所述数据包还包括：包序号和/或校验值；
根据所述包序号和校验值中的至少一项，确定所述当前图像帧的传输状态。
可选地，作为一个实施例，所述发送端为无人飞行器。
应理解，图像处理装置20可以执行方法300及其各种可选实施例，为了简洁，在此不再赘述。
图9是根据本发明另一实施例的发送端中的图像处理装置的示意性框图。如图9所示，图像处理装置1000包括至少一个存储器1010，用于存储计算机可执行指令；至少一个处理器1020。所述至少一个处理器1020单独或共同地用于：访问所述至少一个存储器1010，并执行所述计算机可执行指令，以实施方法100及其各种可选实施例，为了简洁，在此不再赘述。
图10是根据本发明另一实施例的接收端中的图像处理装置的示意性 框图。如图10所示，图像处理装置2000包括至少一个存储器2010，用于存储计算机可执行指令；至少一个处理器2020。所述至少一个处理器2020单独或共同地用于：访问所述至少一个存储器2010，并执行所述计算机可执行指令，以实施方法300及其各种可选实施例，为了简洁，在此不再赘述。
可选地，本发明实施例中的处理器可以是中央处理器(Central Processing Unit，简称为“CPU”)，网络处理器(Network Processor，简称为“NP”)或者CPU和NP的组合。处理器还可以进一步包括硬件芯片。上述硬件芯片可以是专用集成电路(Application-Specific Integrated Circuit，简称为“ASIC”)，可编程逻辑器件(Programmable Logic Device，简称为“PLD”)或其组合。上述PLD可以是复杂可编程逻辑器件(Complex Programmable Logic Device，简称为“CPLD”)，现场可编程逻辑门阵列(Field-Programmable Gate Array，简称为“FPGA”)，通用阵列逻辑(Generic Array Logic,简称为“GAL”)或其任意组合。
可选地，本发明实施例中的存储器可以包括只读存储器和随机存取存储器，并向处理器提供指令和数据。存储器的一部分还可以包括非易失性随机存取存储器。例如，存储器还可以存储设备类型的信息。
图11是根据本发明实施例的无人飞行器的示意性框图。如图11所示，无人飞行器30包括图7所示的图像处理装置10，或者无人飞行器30包括图9所示的图像处理装置1000。
图12是根据本发明实施例的接收端的示意性框图。如图12所示，接收端40包括图8所示的图像处理装置20或图10所示的图像处理装置2000。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本公开的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本公开所提供的几个实施例中，应该理解到，所揭露的系统、装 置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本公开各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本公开各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本公开的具体实施方式，但本公开的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应所述以权利要求的保护范围为准。

Claims (42)

  1. 一种发送端中的图像处理方法，其特征在于，包括：
    获取当前图像帧；
    响应于反馈信息，根据所述反馈信息确定用于编码所述当前图像帧的参考帧，其中，所述反馈信息是由接收端以可变的反馈时间间隔发送的，所述反馈信息用于指示在所述反馈信息之前所述发送端向所述接收端发送的图像帧的传输状态；
    根据所述参考帧，对所述当前图像帧进行帧间编码，以生成编码数据；
    向所述接收端发送所述编码数据。
  2. 根据权利要求1所述的图像处理方法，其特征在于，所述根据所述反馈信息确定用于编码所述当前图像帧的参考帧，包括：
    根据所述反馈信息确定已被正确传输的图像帧；
    根据所述已被正确传输的图像帧确定所述参考帧。
  3. 根据权利要求2所述的图像处理方法，其特征在于，所述方法还包括：
    将所述已被正确传输的图像帧中的至少一部分保存到参考帧管理队列中；
    其中，所述根据所述已被正确传输的图像帧，确定所述参考帧，包括：
    将所述参考帧管理队列中的图像帧，确定为所述参考帧。
  4. 根据权利要求1至3中任一项所述的图像处理方法，其特征在于，所述方法还包括：
    响应于未接收到所述反馈信息，将在所述当前图像帧之前发送的预设数量的图像帧确定为所述参考帧。
  5. 根据权利要求1至4中任一项所述的图像处理方法，其特征在于，所述反馈信息包括下列信息中的至少一种：
    用于指示在所述反馈信息之前发送的最后一帧图像帧是否被正确传输的指示信息；
    已被正确传输的图像帧的帧号；以及
    在所述反馈信息之前最后一帧被正确传输的图像帧的帧号。
  6. 根据权利要求1至5中任一项所述的图像处理方法，其特征在于，所述方法还包括：
    确定所述可变的反馈时间间隔；
    向所述接收端发送所述可变的反馈时间间隔。
  7. 根据权利要求6所述的图像处理方法，其特征在于，所述确定所述可变的反馈时间间隔，包括：
    根据下列信息中的至少一种确定所述可变的反馈时间间隔：
    反馈信道的负载情况；
    所述发送端的应用场景对错误恢复及时性的需求；以及
    预设时段内的传输错误率。
  8. 根据权利要求6所述的图像处理方法，其特征在于，所述确定所述可变的反馈时间间隔，包括：
    确定所述发送端的姿态变化速率；
    根据所述姿态变化速率调整所述反馈时间间隔。
  9. 根据权利要求1至8中任一项所述的方法，其特征在于，所述向所述接收端发送所述编码数据，包括：
    将所述编码数据进行打包处理，获得数据包，其中，所述数据包还包括：包序号和校验值中的至少一项；
    向所述接收端发送所述数据包。
  10. 根据权利要求1至9中任一项所述的方法，其特征在于，所述发送端为无人飞行器。
  11. 一种接收端中的图像处理方法，其特征在于，包括：
    生成反馈信息，所述反馈信息用于指示在所述反馈信息之前发送端向所述接收端发送的图像帧的传输状态；
    以可变的反馈时间间隔向所述发送端发送所述反馈信息；
    接收所述发送端发送的当前图像帧的编码数据，其中，所述发送端对当前图像帧进行编码时采用的参考帧是所述发送端根据所述反馈信息确定的。
  12. 根据权利要求11所述的图像处理方法，其特征在于，在以可变的反馈时间间隔向所述发送端发送所述反馈信息之前，所述方法还包括：
    将已被正确传输的图像帧中的至少一部分保存到参考帧管理队列中。
  13. 根据权利要求11或12所述的图像处理方法，其特征在于，所述反馈信息包括下列信息中的至少一种：
    用于指示在所述反馈信息之前发送的最后一帧图像帧是否被正确传输 的指示信息；
    已被正确传输的图像帧的帧号；以及
    在所述反馈信息之前最后一帧被正确传输的图像帧的帧号。
  14. 根据权利要求11至13中任一项所述的图像处理方法，其特征在于，所述方法还包括：
    确定所述可变的反馈时间间隔。
  15. 根据权利要求14所述的图像处理方法，其特征在于，所述确定所述可变的反馈时间间隔，包括：
    根据下列信息中的至少一种确定所述可变的反馈时间间隔：
    反馈信道的负载情况；
    所述发送端的应用场景对错误恢复及时性的需求；以及
    预设时段内的传输错误率。
  16. 根据权利要求15所述的图像处理方法，其特征在于，所述确定所述可变的反馈时间间隔，包括：
    接收所述发送端发送的所述可变的反馈时间间隔。
  17. 根据权利要求16所述的图像处理方法，其特征在于，所述可变的反馈时间间隔是所述发送端根据所述发送端的姿态变化速率动态确定的。
  18. 根据权利要求11至17中任一项所述的图像处理方法，其特征在于，所述接收所述发送端发送的所述当前图像帧的编码数据，包括：
    接收所述发送端发送的数据包，其中，所述数据包是所述发送端将所述编码数据进行打包处理获得的，所述数据包还包括：包序号和校验值中的至少一项；
    其中，所述方法还包括：
    根据所述包序号和校验值中的至少一项，确定所述当前图像帧的传输状态。
  19. 根据权利要求11至18中任一项所述的图像处理方法，其特征在于，所述发送端为无人飞行器。
  20. 一种发送端中的图像处理装置，其特征在于，包括：
    获取模块，用于获取当前图像帧；
    确定模块，用于响应于反馈信息，根据所述反馈信息确定用于编码所述当前图像帧的参考帧，其中，所述反馈信息是由接收端以可变的反馈时间间 隔发送的，所述反馈信息用于指示在所述反馈信息之前所述发送端向所述接收端发送的图像帧的传输状态；
    编码模块，用于根据所述参考帧，对所述当前图像帧进行帧间编码，以生成编码数据；
    发送模块，用于向所述接收端发送所述编码数据。
  21. 根据权利要求20所述的图像处理装置，其特征在于，所述确定模块用于：
    根据所述反馈信息确定已被正确传输的图像帧；
    根据所述已被正确传输的图像帧确定所述参考帧。
  22. 根据权利要求21所述的图像处理装置，其特征在于，所述确定模块还用于：
    将所述已被正确传输的图像帧中的至少一部分保存到参考帧管理队列中；
    将所述参考帧管理队列中的图像帧，确定为所述参考帧。
  23. 根据权利要求20至22中任一项所述的图像处理装置，其特征在于，所述确定模块还用于：
    响应于未接收到所述反馈信息，将在所述当前图像帧之前发送的预设数量的图像帧确定为所述参考帧。
  24. 根据权利要求20至23中任一项所述的图像处理装置，其特征在于，所述反馈信息包括下列信息中的至少一种：
    用于指示在所述反馈信息之前发送的最后一帧图像帧是否被正确传输的指示信息；
    已被正确传输的图像帧的帧号；以及
    在所述反馈信息之前最后一帧被正确传输的图像帧的帧号。
  25. 根据权利要求20至24中任一项所述的图像处理装置，其特征在于，所述确定模块还用于：
    确定所述可变的反馈时间间隔；
    其中，所述发送模块还用于：向所述接收端发送所述可变的反馈时间间隔。
  26. 根据权利要求25所述的图像处理装置，其特征在于，所述确定模块用于：
    根据下列信息中的至少一种确定所述可变的反馈时间间隔：
    反馈信道的负载情况；
    所述发送端的应用场景对错误恢复及时性的需求；以及
    预设时段内的传输错误率。
  27. 根据权利要求25所述的图像处理装置，其特征在于，所述确定模块用于：
    确定所述发送端的姿态变化速率；
    根据所述姿态变化速率调整所述反馈时间间隔。
  28. 根据权利要求20至27中任一项所述的图像处理装置，其特征在于，所述发送模块用于：
    将所述编码数据进行打包处理，获得数据包，其中，所述数据包还包括：包序号和校验值中的至少一项；
    向所述接收端发送所述数据包。
  29. 根据权利要求20至28中任一项所述的图像处理装置，其特征在于，所述发送端为无人飞行器。
  30. 一种发送端中的图像处理装置，其特征在于，包括：
    至少一个存储器，用于存储计算机可执行指令；
    至少一个处理器，单独或共同地用于：访问所述至少一个存储器，并执行所述计算机可执行指令，以实施根据权利要求1至10中任一项所述的方法。
  31. 一种无人飞行器，其特征在于，包括权利要求20至30中任一项所述的图像处理装置。
  32. 一种在接收端中的图像处理装置，其特征在于，包括：
    生成模块，用于生成反馈信息，所述反馈信息用于指示在所述反馈信息之前发送端向所述接收端发送的图像帧的传输状态；
    发送模块，用于以可变的反馈时间间隔向所述发送端发送所述反馈信息；
    接收模块，用于接收所述发送端发送的当前图像帧的编码数据，其中，所述发送端对当前图像帧进行编码时采用的参考帧是所述发送端根据所述反馈信息确定的。
  33. 根据权利要求32所述的图像处理装置，其特征在于，所述发送模 块还用于：
    将已被正确传输的图像帧中的至少一部分保存到参考帧管理队列中。
  34. 根据权利要求32或33所述的图像处理装置，其特征在于，所述反馈信息包括下列信息中的至少一种：
    用于指示在所述反馈信息之前发送的最后一帧图像帧是否被正确传输的指示信息；
    已被正确传输的图像帧的帧号；以及
    在所述反馈信息之前最后一帧被正确传输的图像帧的帧号。
  35. 根据权利要求32至34中任一项所述的图像处理装置，其特征在于，所述发送模块还用于：
    确定所述可变的反馈时间间隔。
  36. 根据权利要求35所述的图像处理装置，其特征在于，所述发送模块用于：
    根据下列信息中的至少一种确定所述可变的反馈时间间隔：
    反馈信道的负载情况；
    所述发送端的应用场景对错误恢复及时性的需求；
    以及预设时段内的传输错误率。
  37. 根据权利要求36所述的图像处理装置，其特征在于，所述接收模块还用于：
    接收所述发送端发送的所述可变的反馈时间间隔。
  38. 根据权利要求37所述的图像处理装置，其特征在于，所述可变的反馈时间间隔是所述发送端根据所述发送端的姿态变化速率动态确定的。
  39. 根据权利要求32至38中任一项所述的图像处理装置，其特征在于，所述接收模块用于：
    接收所述发送端发送的数据包，其中，所述数据包是所述发送端将所述编码数据进行打包处理获得的，所述数据包还包括：包序号和/或校验值；
    根据所述包序号和校验值中的至少一项，确定所述当前图像帧的传输状态。
  40. 根据权利要求32至39中任一项所述的图像处理装置，其特征在于，所述发送端为无人飞行器。
  41. 一种接收端中的图像处理装置，其特征在于，包括：
    至少一个存储器，用于存储计算机可执行指令；
    至少一个处理器，单独或共同地用于：访问所述至少一个存储器，并执行所述计算机可执行指令，以实施根据权利要求11至19中任一项所述的方法。
  42. 一种接收端，其特征在于，包括权利要求32至41中任一项所述的图像处理装置。
PCT/CN2016/113899 2016-12-30 2016-12-30 图像处理方法、装置、无人飞行器和接收端 Ceased WO2018120198A1 (zh)

Priority Applications (8)

Application Number Priority Date Filing Date Title
CN201910586174.3A CN110225338B (zh) 2016-12-30 2016-12-30 图像处理方法、装置、无人飞行器和接收端
PCT/CN2016/113899 WO2018120198A1 (zh) 2016-12-30 2016-12-30 图像处理方法、装置、无人飞行器和接收端
CN201680003480.4A CN107113441B (zh) 2016-12-30 2016-12-30 图像处理方法、装置、无人飞行器和接收端
PCT/CN2017/120225 WO2018121775A1 (en) 2016-12-30 2017-12-29 System and methods for feedback-based data transmission
CN201780070939.7A CN109983775A (zh) 2016-12-30 2017-12-29 用于基于反馈的数据发送的系统和方法
US16/446,929 US11070732B2 (en) 2016-12-30 2019-06-20 Method for image processing, device, unmanned aerial vehicle, and receiver
US16/457,809 US10911750B2 (en) 2016-12-30 2019-06-28 System and methods for feedback-based data transmission
US17/164,429 US20210160480A1 (en) 2016-12-30 2021-02-01 System and methods for feedback-based data transmission

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2016/113899 WO2018120198A1 (zh) 2016-12-30 2016-12-30 图像处理方法、装置、无人飞行器和接收端

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/446,929 Continuation US11070732B2 (en) 2016-12-30 2019-06-20 Method for image processing, device, unmanned aerial vehicle, and receiver

Publications (1)

Publication Number Publication Date
WO2018120198A1 true WO2018120198A1 (zh) 2018-07-05

Family

ID=59676408

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/113899 Ceased WO2018120198A1 (zh) 2016-12-30 2016-12-30 图像处理方法、装置、无人飞行器和接收端

Country Status (3)

Country Link
US (1) US11070732B2 (zh)
CN (2) CN107113441B (zh)
WO (1) WO2018120198A1 (zh)

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107005714B (zh) 2016-12-30 2019-05-14 深圳市大疆创新科技有限公司 图像处理方法与设备
CN109983775A (zh) * 2016-12-30 2019-07-05 深圳市大疆创新科技有限公司 用于基于反馈的数据发送的系统和方法
CN107113441B (zh) 2016-12-30 2019-07-26 深圳市大疆创新科技有限公司 图像处理方法、装置、无人飞行器和接收端
WO2019153351A1 (en) * 2018-02-12 2019-08-15 SZ DJI Technology Co., Ltd. Image processing
WO2019169640A1 (en) * 2018-03-09 2019-09-12 SZ DJI Technology Co., Ltd. System and method for supporting video coding based on fast feedback
CN108769450B (zh) * 2018-06-21 2021-01-05 深圳市道通智能航空技术有限公司 数据传输控制方法、信息发送端、接收端及飞行器图传系统
CN110113610B (zh) * 2019-04-23 2021-04-20 西安万像电子科技有限公司 数据传输方法及装置
WO2020248267A1 (zh) * 2019-06-14 2020-12-17 深圳市大疆创新科技有限公司 数据传输系统、方法、发送端和计算机可读介质
WO2020258296A1 (zh) * 2019-06-28 2020-12-30 深圳市大疆创新科技有限公司 图像处理方法、设备、无人飞行器和接收端
CN110996122B (zh) * 2019-12-12 2022-11-15 腾讯云计算（北京）有限责任公司 视频帧传输方法、装置、计算机设备及存储介质
CN111629278B (zh) * 2020-05-22 2022-07-26 新石器慧通（北京）科技有限公司 图像传输方法、装置、可移动平台、服务器及存储介质
CN113316930A (zh) * 2020-09-24 2021-08-27 深圳市大疆创新科技有限公司 无线图像传输方法及设备、摄影装置、可移动平台
CN114679586B (zh) * 2020-12-24 2025-06-10 重庆中星微人工智能芯片技术有限公司 帧内编码帧发送方法、装置、电子设备和计算机可读介质
US12337232B2 (en) * 2021-01-04 2025-06-24 Microsoft Technology Licensing, Llc Systems and methods for streaming interactive applications
CN112911223B (zh) * 2021-01-15 2022-11-22 四川一电航空技术有限公司 基于飞行器的图像处理方法、装置、设备及存储介质
CN113573063B (zh) * 2021-06-16 2024-06-14 百果园技术(新加坡)有限公司 视频编解码方法及装置
CN114925226B (zh) * 2022-06-22 2023-03-10 上海威固信息技术股份有限公司 一种图像存储方法、系统、图像存储设备和存储介质
CN115631131A (zh) * 2022-09-09 2023-01-20 维沃移动通信有限公司 图像处理方法、装置、电子设备和可读存储介质
CN119363980B (zh) * 2024-12-26 2025-04-25 济南博观智能科技有限公司 图像帧校验方法、装置、发送端设备和存储介质
CN119893052B (zh) * 2025-03-27 2025-07-11 云南省第一人民医院 呼吸内镜视频采集处理方法、电子设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101155311A (zh) * 2006-09-27 2008-04-02 中兴通讯股份有限公司 一种视频通信中的视频码流错误检测和处理方法
CN101193312A (zh) * 2006-11-22 2008-06-04 中兴通讯股份有限公司 基于反馈的自适应错误恢复装置、视频通信系统和方法
US20140086446A1 (en) * 2012-09-25 2014-03-27 Samsung Electronics Co., Ltd. Method and apparatus for image data processing, and electronic device including the apparatus
CN105681342A (zh) * 2016-03-08 2016-06-15 随锐科技股份有限公司 一种基于h264的多路视频会议系统的抗误码方法及系统

Family Cites Families (46)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3043552B2 (ja) * 1993-10-04 2000-05-22 シャープ株式会社 電子写真装置の画質安定化装置
US20110122063A1 (en) 2002-12-10 2011-05-26 Onlive, Inc. System and method for remote-hosted video effects
US20110126255A1 (en) 2002-12-10 2011-05-26 Onlive, Inc. System and method for remote-hosted video effects
JP4971594B2 (ja) 2004-03-31 2012-07-11 キヤノン株式会社 プログラム及び表示制御装置
CN100558167C (zh) 2006-10-16 2009-11-04 华为技术有限公司 多媒体视频通信方法及系统
TWI508478B (zh) * 2006-10-30 2015-11-11 Interdigital Tech Corp 無線發射／接收單元及用於處理在無線發射／接收單元中執行的回饋的方法
US8150441B2 (en) * 2006-11-06 2012-04-03 Magnolia Broadband Inc. Modifying a signal by controlling transmit diversity parameters
US8494049B2 (en) * 2007-04-09 2013-07-23 Cisco Technology, Inc. Long term reference frame management with error video feedback for compressed video communication
CN101207823A (zh) * 2007-11-22 2008-06-25 武汉大学 用于视频通信的综合抗误码视频编码方法
US8265171B2 (en) 2008-02-26 2012-09-11 Richwave Technology Corp. Error resilient video transmission using instantaneous receiver feedback and channel quality adaptive packet retransmission
US8406296B2 (en) 2008-04-07 2013-03-26 Qualcomm Incorporated Video refresh adaptation algorithms responsive to error feedback
JP5142828B2 (ja) 2008-05-29 2013-02-13 キヤノン株式会社 送信装置、送信方法及びプログラム
CN101360243A (zh) * 2008-09-24 2009-02-04 腾讯科技（深圳）有限公司 基于反馈参考帧的视频通信系统及方法
US8737475B2 (en) 2009-02-02 2014-05-27 Freescale Semiconductor, Inc. Video scene change detection and encoding complexity reduction in a video encoder system having multiple processing devices
US8537699B2 (en) * 2009-06-16 2013-09-17 Qualcomm Incorporated Managing video adaptation algorithms
EP2285122B1 (en) 2009-07-17 2013-11-13 Canon Kabushiki Kaisha A method and device for reconstructing a sequence of video data after transmission over a network
GB2477515B (en) * 2010-02-03 2012-09-26 Orbital Multi Media Holdings Corp Data flow control method and apparatus
US20110249729A1 (en) 2010-04-07 2011-10-13 Apple Inc. Error resilient hierarchical long term reference frames
US8411743B2 (en) 2010-04-30 2013-04-02 Hewlett-Packard Development Company, L.P. Encoding/decoding system using feedback
ES2870332T3 (es) 2010-10-20 2021-10-26 Guangdong Oppo Mobile Telecommunications Corp Ltd Optimización de la distorsión de la tasa resistente a errores para la codificación de imágenes y video
CA2722993A1 (fr) 2010-12-01 2012-06-01 Ecole De Technologie Superieure Systeme d'ecodage video parallele multitrames et multitranches avec encodage simultane de trames predites
CN102075741B (zh) * 2010-12-14 2012-12-12 北京华环电子股份有限公司 一种图像传输方法、终端及系统
CN102014286B (zh) * 2010-12-21 2012-10-31 广东威创视讯科技股份有限公司 一种视频编解码方法及装置
CN102547266B (zh) * 2010-12-29 2015-03-11 中国移动通信集团公司 一种视频数据传输方法及设备
US9071484B1 (en) 2011-04-05 2015-06-30 Elemental Technologies, Inc. Feedback optimized video coding parameters
CN102780547A (zh) * 2011-05-11 2012-11-14 中兴通讯股份有限公司 一种信道状态信息反馈方法及装置
GB2495467B (en) * 2011-09-02 2017-12-13 Skype Video coding
CN102946534A (zh) * 2011-09-02 2013-02-27 斯凯普公司 视频编码
CN102299784B (zh) 2011-09-13 2015-05-27 华为技术有限公司 数据传输方法和设备
CN102510494B (zh) 2011-10-09 2014-02-05 杭州华三通信技术有限公司 对i帧进行同步的方法及设备
US9584832B2 (en) 2011-12-16 2017-02-28 Apple Inc. High quality seamless playback for video decoder clients
US8687654B1 (en) 2012-01-05 2014-04-01 Google Inc. Method to packetize an encoded video frame
US9106936B2 (en) 2012-01-25 2015-08-11 Altera Corporation Raw format image data processing
CN102684837B (zh) 2012-05-16 2014-09-03 西安交通大学 协作多点联合传输中协方差矩阵的动态更新方法
EP2913919A4 (en) 2012-10-26 2016-08-10 Renesas Electronics Corp MOTOR CONTROL DEVICE AND MOTOR DRIVE DEVICE
US9332398B2 (en) 2012-11-30 2016-05-03 Qualcomm Incorporated Provision of positioning data based on device capability
US10951310B2 (en) 2012-12-27 2021-03-16 Panasonic Intellectual Property Corporation Of America Communication method, communication device, and transmitter
CN104660315A (zh) 2013-11-25 2015-05-27 上海益尚信息科技有限公司 新型在协同多点传输系统中控制协作传输下行信号的方法
US9807407B2 (en) 2013-12-02 2017-10-31 Qualcomm Incorporated Reference picture selection
CN105745587B (zh) * 2014-07-31 2018-09-21 深圳市大疆创新科技有限公司 使用无人飞行器实现的虚拟观光系统及方法
CN113031652A (zh) 2015-03-31 2021-06-25 深圳市大疆创新科技有限公司 针对限飞区域的开放平台
CN105163126B (zh) 2015-09-06 2018-05-25 福州瑞芯微电子股份有限公司 一种基于hevc协议的硬件解码方法和装置
US10313685B2 (en) 2015-09-08 2019-06-04 Microsoft Technology Licensing, Llc Video coding
US10595025B2 (en) 2015-09-08 2020-03-17 Microsoft Technology Licensing, Llc Video coding
CN107113441B (zh) 2016-12-30 2019-07-26 深圳市大疆创新科技有限公司 图像处理方法、装置、无人飞行器和接收端
CN107005714B (zh) * 2016-12-30 2019-05-14 深圳市大疆创新科技有限公司 图像处理方法与设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101155311A (zh) * 2006-09-27 2008-04-02 中兴通讯股份有限公司 一种视频通信中的视频码流错误检测和处理方法
CN101193312A (zh) * 2006-11-22 2008-06-04 中兴通讯股份有限公司 基于反馈的自适应错误恢复装置、视频通信系统和方法
US20140086446A1 (en) * 2012-09-25 2014-03-27 Samsung Electronics Co., Ltd. Method and apparatus for image data processing, and electronic device including the apparatus
CN105681342A (zh) * 2016-03-08 2016-06-15 随锐科技股份有限公司 一种基于h264的多路视频会议系统的抗误码方法及系统

Also Published As

Publication number Publication date
US11070732B2 (en) 2021-07-20
CN110225338A (zh) 2019-09-10
CN107113441B (zh) 2019-07-26
CN107113441A (zh) 2017-08-29
US20190313027A1 (en) 2019-10-10
CN110225338B (zh) 2021-12-10

Similar Documents

Publication Publication Date Title
WO2018120198A1 (zh) 图像处理方法、装置、无人飞行器和接收端
US10924745B2 (en) Image processing method and device
US10911750B2 (en) System and methods for feedback-based data transmission
JP6405416B2 (ja) データ送信システム、及びデータ送信方法
US20140277850A1 (en) Remote motion control using a wireless mobile device
US11877253B2 (en) Aircraft time synchronization system and method
US10020916B2 (en) Method and apparatus for data communication of vehicle
US20240331180A1 (en) Data transmission throttling and data quality updating for a slam device
JP5523130B2 (ja) 通信装置、通信方法、及びプログラム
US20240032118A1 (en) Wireless communication method, movable platform, system, and computer-readable storage media
JP5381109B2 (ja) 通信装置及びその制御プログラム
JP5998923B2 (ja) プログラム、情報処理装置、及び通信方法
WO2022000350A1 (zh) 视频传输方法、可移动平台、终端设备、系统及存储介质
US10735142B2 (en) Multi-system data transfer protocol
KR101183118B1 (ko) 군집 비행 물체를 위한 무선 분산 측정 장치 및 방법
US20200413074A1 (en) System and method for supporting video coding based on fast feedback
WO2022141121A1 (zh) 图像传输方法、可移动平台、遥控装置、系统及存储介质
WO2022140971A1 (zh) 图像传输方法、装置、平台、设备及计算机可读存储介质
KR102955914B1 (ko) 신호를 수신하는 전자 장치의 동작 방법 및 이를 지원하는 전자 장치
US11917020B2 (en) Communication device and communication method
WO2022174431A1 (zh) 图像传输方法、可移动平台、设备及计算机可读存储介质
CN112655189A (zh) 传输数据的方法、装置、发送端和接收端
JP2007129421A (ja) 遅延時間を考慮したリンク層フレームの送信方法、送信装置及びプログラム
WO2015196356A1 (en) Method and apparatus for interactive communication

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16925182

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16925182

Country of ref document: EP

Kind code of ref document: A1