WO2022266974A1

WO2022266974A1 - Image processing method and device, service server, and storage medium

Info

Publication number: WO2022266974A1
Application number: PCT/CN2021/102220
Authority: WO
Inventors: 马宁; 尹小俊; 陈颖
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2021-06-24
Filing date: 2021-06-24
Publication date: 2022-12-29

Abstract

An image processing method and device, a service server, and a storage medium. The method is applicable to a transit end, and comprises: receiving coded data of an image frame from a transmitting end (S101); transmitting first feedback information to the transmitting end, the first feedback information being used for indicating a transmission condition of the image frame currently received by a transit end (S102); transmitting the coded data of the image frame to a receiving end (S103); and obtaining second feedback information from the receiving end, the second feedback information being used for indicating the transmission condition of the image frame currently received by the receiving end (S104).

Description

Image processing method, device, service server and storage medium

technical field

The present application relates to the technical field of image processing, and in particular to an image processing method, an image processing device at a transfer terminal, a service server, and a storage medium.

Background technique

Due to the unreliability of data transmission, especially wireless data transmission, image data generally uses periodic error-tolerant frames and error-tolerant frame groups for error recovery during end-to-end transmission; or, try to ensure that less error-tolerant frames are transmitted or Error-tolerant frame group: When there is no problem in the transmission, no error-tolerant frame or error-tolerant frame group is transmitted; when there is an error in the transmission, the receiving end feeds back the error to the sending end, and the sending end sends an error-tolerant frame or error-tolerant frame group.

However, in fact, the end-to-end process may undergo multiple wireless or wired transmission processes, intermediate processing, etc., which results in longer error recovery time and increased error probability.

Contents of the invention

Based on this, the present application provides an image processing method, an image processing device at a transfer terminal, a service server, and a storage medium.

In the first aspect, the present application provides an image processing method, which is applicable to the transfer terminal, and the method includes:

receiving the encoded data of the image frame from the sender;

Sending first feedback information to the sending end, where the first feedback information is used to indicate the transmission status of the image frame currently received by the transfer end;

Send the encoded data of the image frame to the receiving end;

Acquire second feedback information from the receiving end, where the second feedback information is used to indicate the transmission status of the image frame currently received by the receiving end.

In a second aspect, the present application provides an image processing device at a transfer terminal, the device comprising: a communication module, a memory, and a processor;

The memory is used to store computer programs;

The processor is configured to execute the computer program and when executing the computer program, implement the following steps:

Controlling the communication module to receive encoded data of the image frame from the sending end;

controlling the communication module to send first feedback information to the sending end, where the first feedback information is used to indicate the transmission status of the image frame currently received by the transfer end;

controlling the communication module to send encoded data of the image frame to the receiving end;

In a third aspect, the present application provides a service server, where the service server includes the image processing device at the transfer terminal as described above.

In a fourth aspect, the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the image processing method as described above.

The embodiment of the present application provides an image processing method, an image processing device at the transfer end, a service server, and a storage medium, which receive encoded data of image frames from the sending end; send first feedback information to the sending end, and the first feedback The information is used to indicate the transmission status of the image frame currently received by the transit end; send the encoded data of the image frame to the receiving end; obtain second feedback information from the receiving end, and the second feedback information is used to indicate The transmission status of the image frame currently received by the receiving end. Since the transfer terminal receives the encoded data, it will feed back the transmission status of the received image frame to the sending end, and after the transfer terminal sends the encoded data to the receiving end, it will also receive the transmission status of the received image frame fed back by the receiving end. Compared with related technologies, when the end-to-end transmission has an error, the receiving end directly feeds back the error to the sending end. In the embodiment of this application, the transfer end is used as an intermediate node, and the end-to-end transmission process is divided into at least two groups. Feedback on the reception status, and receive feedback from the receiving end, so that the feedback path can be shortened, so the delay of image transmission and feedback during the entire transmission process can be reduced; when the intermediate end feeds back the receiving error to the sending end, the sending end can quickly Take measures for error correction and recovery. When the relay end receives feedback from the receiver that it has received an error, the relay end can take measures to perform error correction and recovery, that is, the receiving error between the sender and the receiver is divided into the error between the sender and the relay. Receiving errors, receiving errors between the transfer end and the receiving end, on the one hand, when a receiving error occurs, the sending end or the transfer end can quickly receive feedback, and can quickly perform error correction and recovery, so the error recovery time can be reduced; on the other hand , since the error correction path becomes shorter, the probability of transmission errors during error correction is also reduced.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application. Ordinary technicians can also obtain other drawings based on these drawings on the premise of not paying creative work.

Fig. 1 is a schematic flow chart of an embodiment of the image processing method of the present application;

2 is a schematic diagram of the connection between the transfer terminal and the 5G base station in an application scenario in the image processing method of the present application;

Fig. 3 is a schematic flow chart of another embodiment of the image processing method of the present application;

FIG. 4 is a schematic diagram of the principle of an error recovery strategy in an embodiment of the image processing method of the present application;

Fig. 5 is a schematic flow chart of another embodiment of the image processing method of the present application;

FIG. 6 is a schematic flowchart of another embodiment of the image processing method of the present application;

Fig. 7 is a schematic flow chart of another embodiment of the image processing method of the present application;

FIG. 8 is a schematic structural diagram of an embodiment of an image processing device at the relay end of the present application.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

The flow charts shown in the drawings are just illustrations, and do not necessarily include all contents and operations/steps, nor must they be performed in the order described. For example, some operations/steps can be decomposed, combined or partly combined, so the actual order of execution may be changed according to the actual situation.

Due to the unreliability of data transmission, image data generally uses periodic error-tolerant frames and error-tolerant frame groups for error recovery during end-to-end transmission; or try to ensure that less error-tolerant frames or error-tolerant frame groups are transmitted: when there is no When there is a problem, the error-tolerant frame or error-tolerant frame group is not transmitted; when there is an error in the transmission, the receiving end feeds back the error to the sending end, and the sending end sends the error-tolerant frame or error-tolerant frame group at this time. However, in fact, in the end-to-end process, there may be multiple transmission processes, intermediate processing, etc., which results in longer error recovery time and increased error probability.

Some implementations of the present application will be described in detail below in conjunction with the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

Referring to Fig. 1, Fig. 1 is a schematic flow chart of an embodiment of the image processing method of the present application. The method of the embodiment of the present application is applicable to the transfer terminal, and the sending process of sending the coded data of the image frame from the sending end to the receiving end includes sending the sending end to the transfer end. The process of sending the coded data of the image frame and the process of sending the coded data of the image frame from the relay end to the receiving end.

It should be noted that the sending end in the embodiment of the present application can be the source sending end of the encoded data, or the intermediate sending end of the encoded data (that is, the sending end can be another transit end, and the encoded data of the sending end is another sending end. end); the receiving end of the embodiment of the present application can be the destination receiving end of the encoded data, and can also be the intermediate receiving end of the encoded data (that is, the receiving end can be another transit end, and the receiving end also receives the encoded data. need to be forwarded to other receivers).

The method includes: step S101, step S102, step S103 and step S104.

Step S101: Receive the coded data of the image frame from the sending end.

Step S102: Sending first feedback information to the sending end, where the first feedback information is used to indicate the transmission status of the image frame currently received by the transfer end.

Step S103: Send the coded data of the image frame to the receiving end.

Step S104: Obtain second feedback information from the receiving end, where the second feedback information is used to indicate the transmission status of the image frame currently received by the receiving end.

In this embodiment, the transmission status may refer to whether there is an error in the transmission of the image frame, including but not limited to: whether data loss or data error occurs during the transmission of the image frame; whether the image frame is transmitted correctly, and whether it is successfully transmitted after the correct transmission. decode; etc.

The sending end in this embodiment can be a platform that can move in any suitable environment, such as various airplanes, drones, etc. flying in the air, various ships, submarines, etc. moving in water, and moving on land. Various cars, trains, etc. In one embodiment, the sending end includes a common drone. The sending end in this embodiment may also be an intermediate node in the image transmission process, or another transfer end, that is, more than two transmission transfers occur during the image transmission process. Therefore, the embodiment of the present application is also applicable to multi-segment transmission.

The receiving end in this embodiment may be a computer, a handheld electronic device, a communication device, a video monitoring device, and the like. The receiving end in this embodiment may also be an intermediate node in the image transmission process, or another transfer end, that is, more than two transmission transfers occur during the image transmission process. Therefore, the embodiment of the present application is also applicable to multi-segment transmission.

For example, the transmission process needs to go through 4 nodes successively, which are: device 1, device 2, device 3, and device 4 from beginning to end; the transmission process can be decomposed into two transmission and transfer processes, the first one is: device 1, device 2. Device 3, the second one is: device 2, device 3, and device 4; for the first transmission and transfer process, device 1 can be used as the sending end of the embodiment of this application, and device 2 (transfer end) can be used as this For the transfer terminal of the embodiment of the application, device 3 (ie another transfer terminal) can be used as the receiving end of the embodiment of the application; for the second transmission transfer, device 2 (another transfer terminal) can be used as the receiving end of the embodiment of the application The sending end, device 3 (transfer end) can be used as the transfer end of the embodiment of the application, and the device 4 can be used as the receiving end of the embodiment of the application; as a whole transmission process, the device 2 and the device 3 are both the embodiment of the application The relay end, device 2 and device 3 are applicable to the method of the embodiment of the present application.

In an embodiment, the transfer terminal is connected to a 5G base station. For example, as shown in Figure 2, the transfer end can be a service server deployed on a mobile edge computing node, and the service server is connected to a 5G base station, which can further reduce the transmission and transmission costs from both ends (sending end and receiving end) to the service server. Feedback delay, so as to achieve a better image transmission experience.

In this embodiment, wireless communication, wired communication, or optical communication may be used between the transfer end and the sending end and between the transfer end and the receiving end, or a mixed communication of the two or more ,as shown in picture 2. As 5G applications become more and more widespread, in an embodiment, the communication between the transfer terminal and the sending end and between the transfer terminal and the receiving end is through a 5G module.

The transfer terminal in the embodiment of the present application is the same as the transfer terminal in the related art: as an intermediate node between the sending end and the intermediate end, it receives the coded data sent by the sending end and forwards the coded data to the receiving end; The difference of the transfer terminal is that the transfer terminal in this embodiment will send the first feedback information to the sending end, and will also receive the second feedback information sent by the receiving end. The first feedback information is used to indicate the transmission status of the image frame currently received by the transfer terminal. The second feedback information is used to indicate the transmission status of the image frame currently received by the receiving end.

Therefore, as a transfer terminal, on the one hand, it can feed back the first feedback information to the sender, so that the sender can understand the transmission status of the image frame sent by itself, and provide technical support for subsequent error correction and recovery between the sender and the transfer terminal; On the other hand, the second feedback information fed back by the receiving end can be received, which is convenient for the transfer end to understand the transmission status of the image frame sent by itself, and provides technical support for subsequent error correction and recovery between the transfer end and the receiving end. That is, in the subsequent error correction recovery, the original error correction recovery between the sending end and the receiving end is changed to: the error correction recovery between the sending end and the transit end, and the error correction recovery between the transit end and the receiving end. The original error correction path, the error correction path of the embodiment of the present application is also shortened, which can reduce the error recovery time and reduce the probability of transmission errors during error correction.

It should be noted that there is no obvious sequence relationship between the above step S102 and step S103.

In the embodiment of the present application, the coded data of the image frame is received from the sending end; the first feedback information is sent to the sending end, and the first feedback information is used to indicate the transmission status of the image frame currently received by the transfer end; Sending the encoded data of the image frame to the receiving end; obtaining second feedback information from the receiving end, where the second feedback information is used to indicate the transmission status of the image frame currently received by the receiving end. Since the transfer terminal receives the encoded data, it will feed back the transmission status of the received image frame to the sending end, and after the transfer terminal sends the encoded data to the receiving end, it will also receive the transmission status of the received image frame fed back by the receiving end. Compared with related technologies, when the end-to-end transmission has an error, the receiving end directly feeds back the error to the sending end. In the embodiment of this application, the transfer end is used as an intermediate node, and the end-to-end transmission process is divided into at least two groups. Feedback on the reception status, and receive feedback from the receiving end, so that the feedback path can be shortened, so the delay of image transmission and feedback during the entire transmission process can be reduced; when the intermediate end feeds back the receiving error to the sending end, the sending end can quickly Take measures for error correction and recovery. When the relay end receives feedback from the receiver that it has received an error, the relay end can take measures to perform error correction and recovery, that is, the receiving error between the sender and the receiver is divided into the error between the sender and the relay. Receiving errors, receiving errors between the transfer end and the receiving end, on the one hand, when a receiving error occurs, the sending end or the transfer end can quickly receive feedback, and can quickly perform error correction and recovery, so the error recovery time can be reduced; on the other hand , since the error correction path becomes shorter, the probability of transmission errors during error correction is also reduced.

When a receiving error occurs from the sender to the transit end, the coded data sent by the sender can be used for error recovery. Based on the feedback from the transit end, the sender usually has two common error recovery methods. Then the coded data received by the transit end corresponds to There are two different encoding data.

The first one is that the encoded data of the image frame from the sending end includes encoded data obtained by encoding according to a preset error recovery strategy when the first feedback information received by the sending end indicates a reception error.

The second type is that the coded data of the image frame from the sending end includes the coded data obtained by the sending end determining a reference frame according to the first feedback information and performing inter-frame coding according to the reference frame.

The reception errors in this embodiment of the present application generally include data loss and data verification errors.

For example, before sending the encoded data, the encoded data is packaged, and each obtained data packet has a unique packet serial number, and the data in the data packet has a check code; the packet loss is judged by the discontinuous detection of the packet serial number, then The reception of the image frame is considered to be a reception error; the check code carried in the data packet is inconsistent with the check code calculated from the data in the data packet, and the reception of the image frame can be considered to be a reception error. It is judged that there is no packet loss through the discontinuous detection of the packet sequence number, and the check code carried in the data packet is consistent with the check code calculated from the data in the data packet, then it can be considered that the reception of the image frame is correct.

When the first feedback information prompts a reception error, the encoded data subsequently received by the transfer terminal may be the encoded data obtained by encoding according to a preset error recovery strategy at the transmitting end, or it may be that the transmitting end determines a reference frame according to the first feedback information and Coded data obtained by performing inter-frame coding according to the reference frame. The details of these two error recovery methods will be described later.

When the relay terminal obtains the second feedback information from the receiving terminal indicating a reception error, the relay terminal usually adopts two commonly used error recovery methods for recovery.

The first is to encode the current image frame according to a preset error recovery policy and send the encoded data.

That is, the method further includes: step S105A and step S106A, as shown in FIG. 3 .

Step S105A: When the second feedback information from the receiving end indicates a receiving error, encode the current image frame according to a preset error recovery strategy to obtain encoded data of the current image frame.

Step S106A: Send the encoded data of the current image frame to the receiving end.

The preset error recovery strategy can refer to an encoding strategy that enables the transfer end to obtain the correct decoded data of the image frame from the encoded data of the image frame encoded according to the strategy, including but not limited to: error-tolerant frame error recovery strategy, error-tolerant frame group Error recovery strategies, other possible error recovery strategies, etc. The receiving end receives the image frame encoded with the preset error recovery strategy, and can perform error recovery on the subsequent image frame to be decoded, so as to ensure the reliability of the image frame transmission between the transfer end and the receiving end.

In an embodiment, the encoded data of the current image frame includes at least intra-frame encoded data of a partial frame of the current image frame.

In this embodiment, the frame size of the image frame may refer to the frame size of the image frame, which has nothing to do with the data information of the image frame. For example, an image frame is divided into five blocks, and each block may be called a partial frame of the image frame, and these five blocks constitute a complete frame of the image frame.

Intra-frame coding, as the name suggests, does not seek outside but only inside, regardless of the external data, it only encodes through its own existing data, regardless of the previous (or subsequent) frame (or several frames) of the frame. The coded part is used to guess what the current part of the data to be coded is; since the intra-frame coding only refers to the data of this frame, the compression rate is small. In inter-frame coding, it is estimated from the previous (or subsequent) frame (or several frames) of the frame what the part of the data to be compressed is; since the inter-frame coding needs to refer to other frame data, the compression rate is high.

Currently, video coding standards widely used in the industry, including H.263, H.264, H.265, MPEG4, etc., provide two types of frame types: I frame and P/B frame. The characteristics of the I frame are: all images are divided into blocks, and the predictive compression method that does not depend on any other frame information is adopted; since there is no reference to the temporal correlation for prediction, the compression rate is low, and the bit rate of the frame after encoding is relatively high. ;Because it does not rely on and does not refer to other frames, as long as the receiving end receives the I frame correctly, it can recover from the error; the I frame belongs to intra-frame coding. The characteristics of the P/B frame are: the image is divided into blocks, and the predictive compression method that depends on other frame information can be used; because the prediction is made with reference to the time correlation, the compression rate is high, and the code rate of the resulting encoded frame is low; Because of relying on and referring to other frames, even if the receiving end receives the frame correctly, if an error occurs before, the error cannot be recovered; the P/B frame belongs to inter-frame coding.

In this embodiment, when the second feedback information indicates a reception error, intra-frame coded data is sent, which can prevent error diffusion and implement error recovery.

Wherein, the encoded data of the current image frame includes intra-frame encoded data of a partial frame of the current image frame and inter-frame encoded data of the remaining partial frames.

At this time, step S106A, after sending the encoded data of the current image frame to the receiving end, further includes: sending the encoded data of M image frames to the receiving end, each image frame in the M image frames The encoded data includes the intra-frame encoded data of the partial frame and the inter-frame encoded data of the remaining partial frames. Among the M+1 image frames including the M image frames and the current image frame, the intra-frame encoded M The sum of +1 partial frames forms a complete frame, M is a positive integer, i and j are different, and both i and j are greater than or equal to 1 and less than or equal to M+1.

Referring to Figure 4, it is assumed that the length of the error recovery image frame group is 3 frames, which are image frame 1, image frame 2, and image frame 3, and the frames of each image frame are divided into 3 blocks, which are respectively partial frame I and partial frame II , Partial frame III; Partial frame I of image frame 1 carries out intra-frame coding, obtains the intra-frame coding data of image frame 1, other partial frame II of image frame 1, partial frame III carries out inter-frame coding, obtains the frame of image frame 1 Inter-coded data; the partial frame II of image frame 2 carries out intra-frame encoding, obtains the intra-frame encoded data of image frame 2, and other partial frame I and partial frame III of image frame 2 carry out inter-frame encoding, obtains the inter-frame encoding of image frame 2 Coding data; the partial frame III of the image frame 3 is intraframe coded to obtain the intraframe encoded data of the image frame 3, and other partial frame I and partial frame II of the image frame 3 are interframe coded to obtain the interframe coding of the image frame 3 data. The receiving end correctly receives these 3 frames of image frames, and can recover the wrong content of the different frame areas of the previous image.

In this embodiment, when the second feedback information indicates a reception error, the intra-coded data of the partial frame of the current image frame and the inter-coded data of the remaining partial frames are sent. In this way, the channel utilization rate can be improved.

Wherein, the encoded data of the current image frame includes intra-frame encoded data of the entire frame of the current image frame.

In this embodiment, the encoded data of the current image frame includes intra-frame encoded data of the entire frame of the current image frame, the encoded data of the current frame may be referred to as an error-tolerant frame, and the error recovery strategy may be referred to as error-tolerant frame error recovery Strategy.

Wherein, the method further includes: when the second feedback information from the receiving end indicates that the reception is correct, sending the inter-frame encoded data of the image frame to the receiving end.

In this embodiment, when the second feedback information indicates that the reception is correct, the interframe coded data of the image frame is sent to the receiving end, which can effectively improve the channel utilization rate.

The second is to perform inter-frame coding on the current image frame according to the determined reference frame.

That is, the method further includes: step S105B, step S106B, and step S107B, as shown in FIG. 5 .

Step S105B: Determine a reference frame for encoding the current image frame according to the second feedback information from the receiving end.

Step S106B: Perform inter-frame encoding on the current image frame according to the reference frame to obtain encoded data of the current image frame.

Step S107B: Send the encoded data of the current image frame to the receiving end.

In this embodiment, the coded image frame adopts an inter-frame coding method (also called an inter-frame predictive coding method). This coding method not only compresses the spatial redundant information in the image frame, but also utilizes the The temporal redundancy between reference frames is used to compress data, that is, to perform temporal prediction and encoding. The compression efficiency of this inter-frame coding method is much higher than that of the intra-frame coding method. In this way, sending the coded data of the current image frame obtained through inter-frame coding to the receiving end can improve the utilization rate of channel resources and reduce transmission delay.

In practical applications, when performing inter-frame coding on the current image frame, the bit rate can be controlled within a specific range or close to the target average bit rate. The basis for rate control can be the complexity of the current image frame, bandwidth constraints, buffer capacity, or other factors.

In this embodiment, the reference frame used to encode the current image frame is determined according to the second feedback information, and the current image frame is inter-frame encoded according to the reference frame. Therefore, as long as the receiving end successfully receives the encoded data of the current image frame, it can To recover from transmission errors, there is no need for the transit end to send intra-coded frames or error-tolerant frame groups, which can improve the utilization of channel resources and improve user experience.

In an embodiment, step S105B, the determining the reference frame used for encoding the current image frame according to the second feedback information from the receiving end may include: sub-step S105B1 and sub-step S105B2, as shown in FIG. 6 .

Sub-step S105B1: According to the second feedback information, determine the image frames that have been correctly transmitted.

Sub-step S105B2: Determine the reference frame according to the correctly transmitted image frame.

In this embodiment, the second feedback information may include at least one of the following information: indication information used to indicate whether the last image frame sent before the second feedback information has been correctly transmitted, and information that has been correctly transmitted The frame number of the image frame is the frame number of the last image frame that has been correctly transmitted before the second feedback information. It should be noted that the frame number of the image frame is unique, and the frame number of the image frame and the encoded data are sent to the receiving end together.

In an embodiment, the method further includes: saving at least a part of the correctly transmitted image frames into a reference frame management queue.

At this time, the sub-step S105B2 of determining the reference frame according to the correctly transmitted image frame may include: determining an image frame in the reference frame management queue as the reference frame.

In this embodiment, each time the second feedback information is received, the image frame that has been correctly transmitted is determined according to the second feedback information, and the reference frame management queue is updated. Correspondingly, after determining that the image frames have been correctly transmitted, the receiving end also saves at least a part of the correctly transmitted image frames in the reference frame management queue. The information stored in the reference frame management queue at the transfer terminal is the same as that stored in the reference frame management queue at the receiving terminal.

In an embodiment, the method further includes: if the second feedback information is not received, determining a preset number of image frames sent before the image frame as the reference frame.

In an embodiment, the transfer terminal packs the coded data, and sends the data package to the receiving terminal. That is, step S103, the sending the coded data of the image frame to the receiving end may include: sub-step S1031 and sub-step S1032, as shown in FIG. 7 .

Sub-step S1031: pack the encoded data of the image frame to generate a data packet, and the data packet includes a packet sequence number and/or a check code.

The transfer terminal may pack the encoded data of the image frame into a data packet, and the packet sequence number of the data packet has a corresponding relationship with the frame number of the image frame. The receiving end can detect whether data loss occurs continuously according to the packet sequence numbers of the received data packets. When the packet sequence numbers of the data packets corresponding to the image frames are missing in the packet sequence numbers of the received data packets, it can be determined that data loss occurs, that is The image frame transmission failed. Or, the receiving end calculates the check value according to the data in the received data packet, and when the calculated check value is inconsistent with the check value carried in the data packet, it is determined that a data error occurs, that is, the transmission of the image frame fails .

The transfer terminal may also pack the encoded data of the image frame into multiple data packets, and the packet sequence numbers of all the data packets correspond to the frame numbers of the image frame. The receiving end detects whether data loss occurs according to whether the packet sequence numbers of the data packets are continuous. Or the receiving end determines whether a data error occurs according to the way of calculating the check value of the data in the received data packet.

Wherein, the check code carried in the data packet may be a cyclic redundancy check code (CRC, Cyclic Redundancy Check).

Sub-step S1032: Send the data packet to the receiving end.

In an embodiment, the encoded data received by the transfer terminal from the sender is a packaged data packet, that is, step S101, the receiving the encoded data of the image frame from the sender may include: receiving the image frame from the sender A data packet obtained after the encoded data is packaged, and the data packet includes a packet sequence number and/or a check code.

At this time, step S102, before sending the first feedback information to the sending end, may include: detecting the encoded data of the image frame according to the packet sequence number and/or check code of the data packet; As a result, the first feedback information is generated.

Wherein, the first feedback information is used to indicate whether the reception is wrong. When the sending end receives the first feedback information indicating a reception error, the sending end may send encoded data obtained by encoding according to a preset error recovery strategy to the transfer end for error recovery.

Wherein, the first feedback information is used to indicate the frame number of the correctly decoded image frame and whether the reception is wrong. When the sending end receives the first feedback information, the sending end can determine the reference frame according to the frame number of the correctly decoded image frame, and send the encoded data obtained by performing inter-frame encoding according to the reference frame to the transfer end for error recovery.

Referring to FIG. 8, FIG. 8 is a schematic structural diagram of an embodiment of an image processing device at the relay end of the present application.

It should be noted that the device in this embodiment can execute the steps in the above image processing method. For detailed description of relevant content, please refer to the relevant content of the above image processing method, which will not be repeated here.

The device 100 includes: a communication module 3, a memory 1, and a processor 2; the processor 2 is connected to the memory 1 and the communication module 3 through a bus, respectively.

Wherein, the processor 2 may be a micro control unit, a central processing unit or a digital signal processor, and so on.

Wherein, the memory 1 may be a Flash chip, a read-only memory, a magnetic disk, an optical disk, a U disk or a mobile hard disk, and the like.

The memory 1 is used to store a computer program; the processor 2 is used to execute the computer program and when executing the computer program, implement the following steps:

controlling the communication module to receive the coded data of the image frame from the sending end; controlling the communication module to send first feedback information to the sending end, the first feedback information being used to indicate that the transfer end currently receives the The transmission situation of the image frame; controlling the communication module to send the coded data of the image frame to the receiving end; obtaining second feedback information from the receiving end, the second feedback information is used to indicate the current received by the receiving end Describe the transmission of image frames.

Wherein, the encoded data of the image frame from the sending end includes encoded data obtained by encoding according to a preset error recovery strategy when the first feedback information received by the sending end indicates a reception error.

Wherein, the encoded data of the image frame from the sending end includes the encoded data obtained by the sending end determining a reference frame according to the first feedback information and performing inter-frame encoding according to the reference frame.

Wherein, when the processor executes the computer program, the following steps are implemented: when the second feedback information from the receiving end indicates a receiving error, encode the current image frame according to a preset error recovery strategy to obtain the current The encoded data of the image frame; controlling the communication module to send the encoded data of the current image frame to the receiving end.

Wherein, the encoded data of the current image frame includes at least intra-frame encoded data of a partial frame of the current image frame.

Wherein, the encoded data of the current image frame includes intra-frame encoded data of a partial frame of the current image frame and inter-frame encoded data of the remaining partial frames; when the processor executes the computer program, the following steps are implemented: Controlling the communication module to send encoded data of M image frames to the receiving end, the encoded data of each image frame in the M image frames includes intra-frame encoded data of a partial frame and inter-frame encoded data of the remaining partial frames , in the M+1 image frames including the M image frames and the current image frame, the sum of M+1 partial frames for intra-frame coding constitutes a complete frame, M is a positive integer, i and j are different, and both i and j are greater than or equal to 1 and less than or equal to M+1.

Wherein, when the processor executes the computer program, the following steps are implemented: when the second feedback information from the receiving end indicates that the reception is correct, control the communication module to send the interframe coding of the image frame to the receiving end data.

Wherein, when the processor executes the computer program, the following steps are implemented: according to the second feedback information from the receiving end, determining a reference frame for encoding the current image frame; according to the reference frame, the current The image frame is inter-frame encoded to obtain the encoded data of the current image frame; the communication module is controlled to send the encoded data of the current image frame to the receiving end.

Wherein, when the processor executes the computer program, the following steps are implemented: according to the second feedback information, determine the image frame that has been correctly transmitted; determine the reference frame according to the image frame that has been correctly transmitted .

Wherein, when the processor executes the computer program, the following steps are implemented: saving at least a part of the correctly transmitted image frames into the reference frame management queue; Determining the reference frame for an image frame includes: determining an image frame in the reference frame management queue as the reference frame.

Wherein, when the processor executes the computer program, the following steps are implemented: if the second feedback information is not received, determining a preset number of image frames sent before the image frame as the reference frame .

Wherein, when the processor executes the computer program, the following steps are implemented: packaging the encoded data of the image frame to generate a data packet, the data packet includes a packet sequence number and/or a check code; The communication module sends the data packet to the receiving end.

Wherein, when the processor executes the computer program, the following steps are implemented: controlling the communication module to receive a data packet obtained after packaging the encoded data of the image frame from the sending end, the data packet includes a packet Serial number and/or check digit.

Wherein, when the processor executes the computer program, the following steps are implemented: detecting the encoded data of the image frame according to the packet sequence number and/or check code of the data packet; Describe the first feedback information.

Wherein, the first feedback information is used to indicate whether the reception is wrong.

Wherein, the first feedback information is used to indicate the frame number of the correctly decoded image frame and whether the reception is wrong.

Wherein, the communication module is a 5G module.

Wherein, the sending end is another transit end.

Wherein, the receiving end is another transfer end.

Wherein, the transfer terminal is connected to a 5G base station.

Wherein, the sending end includes an unmanned aerial vehicle.

The present application also provides a service server, which includes the image processing device at the transfer terminal as described in any one of the above items. For a detailed description of related content, please refer to the related content of the above-mentioned image processing device at the transfer terminal, and details are not repeated here.

The present application also provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the image processing method described in any one of the above items. For a detailed description of the relevant content, please refer to the relevant content above, and will not repeat it here.

Wherein, the computer-readable storage medium may be an internal storage unit of the above device, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk provided, a smart memory card, a secure digital card, a flash memory card, and the like.

It should be understood that the terminology used in the specification of the present application is only for the purpose of describing specific embodiments and is not intended to limit the present application.

It should also be understood that the term "and/or" used in the description of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

The above is only a specific embodiment of the application, but the scope of protection of the application is not limited thereto. Any person familiar with the technical field can easily think of various equivalents within the scope of the technology disclosed in the application. Modifications or replacements, these modifications or replacements shall be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

An image processing method is characterized in that it is suitable for a transfer terminal, and the method includes:

receiving the encoded data of the image frame from the sender;

Sending first feedback information to the sending end, where the first feedback information is used to indicate the transmission status of the image frame currently received by the transfer end;

Send the encoded data of the image frame to the receiving end;

Acquire second feedback information from the receiving end, where the second feedback information is used to indicate the transmission status of the image frame currently received by the receiving end.
The method according to claim 1, wherein the encoded data of the image frame from the sending end includes that when the first feedback information received by the sending end indicates a reception error, the sending end recovers according to a preset error Encoded data obtained by policy encoding.
The method according to claim 1, wherein the coded data of the image frame from the sending end includes the data obtained by the sending end determining a reference frame according to the first feedback information and performing inter-frame coding according to the reference frame Encode the data.
The method according to claim 1, further comprising:

When the second feedback information from the receiving end indicates a reception error, encoding the current image frame according to a preset error recovery strategy to obtain encoded data of the current image frame;

Send the coded data of the current image frame to the receiving end.
The method according to claim 4, wherein the encoded data of the current image frame includes at least intra-frame encoded data of a partial frame of the current image frame.
The method according to claim 5, wherein the encoded data of the current image frame comprises intra-frame encoded data of a partial frame of the current image frame and inter-frame encoded data of the remaining partial frames;

After sending the encoded data of the current image frame to the receiving end, it also includes:

Send the encoded data of M image frames to the receiving end, the encoded data of each image frame in the M image frames includes intra-frame encoded data of partial frames and inter-frame encoded data of remaining partial frames, including the Among the M image frames and the M+1 image frames of the current image frame, the sum of M+1 partial frames for intra-frame encoding constitutes a complete frame, M is a positive integer, i and j are different, and i and j are greater than or equal to 1 and less than or equal to M+1.
The method according to claim 5, wherein the encoded data of the current image frame includes intra-frame encoded data of the entire frame of the current image frame.
The method according to claim 4, characterized in that the method further comprises:

When the second feedback information from the receiving end indicates that the reception is correct, send the interframe encoded data of the image frame to the receiving end.
The method according to claim 1, further comprising:

determining a reference frame for encoding the current image frame according to the second feedback information from the receiving end;

performing inter-frame encoding on the current image frame according to the reference frame, to obtain encoded data of the current image frame;

Send the coded data of the current image frame to the receiving end.
The method according to claim 9, wherein the determining a reference frame for encoding a current image frame according to the second feedback information from the receiving end comprises:

Determine the image frame that has been correctly transmitted according to the second feedback information;

The reference frame is determined according to the correctly transmitted image frame.
The method according to claim 10, characterized in that the method further comprises:

saving at least a part of the correctly transmitted image frames into a reference frame management queue;

The determining the reference frame according to the correctly transmitted image frame includes:

An image frame in the reference frame management queue is determined as the reference frame.
The method according to claim 9, characterized in that the method further comprises:

If the second feedback information is not received, determining a preset number of image frames sent before the image frame as the reference frame.
The method according to claim 1, wherein the sending the coded data of the image frame to the receiving end comprises:

Packing the coded data of the image frame to generate a data packet, the data packet includes a packet sequence number and/or a check code;

Send the data packet to the receiving end.
The method according to claim 1, wherein said receiving the coded data of the image frame from the sending end comprises:

and receiving a data packet obtained after packaging the encoded data of the image frame from the sending end, the data packet includes a packet sequence number and/or a check code.
The method according to claim 14, wherein, before sending the first feedback information to the sending end, comprising:

Detecting the coded data of the image frame according to the packet sequence number and/or check code of the data packet;

Generate the first feedback information according to the detection result.
The method according to claim 15, wherein the first feedback information is used to indicate whether the reception is wrong.
The method according to claim 15, wherein the first feedback information is used to indicate the frame number of the correctly decoded image frame and whether the reception is wrong.
The method according to claim 1, characterized in that, the communication between the transfer terminal and the sending end and between the transfer terminal and the receiving end is through a 5G module.
The method according to claim 18, characterized in that the sending end is another transit end.
The method according to claim 18, characterized in that the receiving end is another transfer end.
The method according to claim 18, wherein the transfer terminal is connected to a 5G base station.
The method according to claim 1, wherein the sending end comprises a drone.
An image processing device at a transfer terminal, characterized in that the device includes: a communication module, a memory, and a processor;

The memory is used to store computer programs;

The processor is configured to execute the computer program and when executing the computer program, implement the following steps:

Controlling the communication module to receive encoded data of the image frame from the sending end;

controlling the communication module to send first feedback information to the sending end, where the first feedback information is used to indicate the transmission status of the image frame currently received by the transfer end;

controlling the communication module to send encoded data of the image frame to the receiving end;

Acquire second feedback information from the receiving end, where the second feedback information is used to indicate the transmission status of the image frame currently received by the receiving end.
The device according to claim 23, wherein the coded data of the image frame from the sending end includes when the first feedback information received by the sending end indicates a reception error, the sending end recovers according to a preset error Encoded data obtained by policy encoding.
The device according to claim 23, wherein the coded data of the image frame from the sending end includes the data obtained by the sending end determining a reference frame according to the first feedback information and performing inter-frame coding according to the reference frame Encode the data.
The device according to claim 23, wherein the processor implements the following steps when executing the computer program:

When the second feedback information from the receiving end indicates a receiving error, encoding the current image frame according to a preset error recovery strategy to obtain encoded data of the current image frame;

and controlling the communication module to send the encoded data of the current image frame to the receiving end.
The device according to claim 26, wherein the encoded data of the current image frame includes at least intra-frame encoded data of a partial frame of the current image frame.
The device according to claim 27, wherein the encoded data of the current image frame includes intra-frame encoded data of a partial frame of the current image frame and inter-frame encoded data of the remaining partial frames;

When the processor executes the computer program, the following steps are implemented:

Controlling the communication module to send encoded data of M image frames to the receiving end, the encoded data of each image frame in the M image frames includes intra-frame encoded data of a partial frame and inter-frame encoded data of the remaining partial frames , in the M+1 image frames including the M image frames and the current image frame, the sum of M+1 partial frames for intra-frame coding constitutes a complete frame, M is a positive integer, i and j are different, and both i and j are greater than or equal to 1 and less than or equal to M+1.
The device according to claim 27, wherein the encoded data of the current image frame includes intra-frame encoded data of the entire frame of the current image frame.
The device according to claim 26, wherein the processor implements the following steps when executing the computer program:

When the second feedback information from the receiving end indicates that the receiving is correct, the communication module is controlled to send the interframe encoded data of the image frame to the receiving end.
The device according to claim 23, wherein the processor implements the following steps when executing the computer program:

determining a reference frame for encoding the current image frame according to the second feedback information from the receiving end;

performing inter-frame encoding on the current image frame according to the reference frame, to obtain encoded data of the current image frame;

and controlling the communication module to send the encoded data of the current image frame to the receiving end.
The device according to claim 31, wherein the processor implements the following steps when executing the computer program:

Determine the image frame that has been correctly transmitted according to the second feedback information;

The reference frame is determined according to the correctly transmitted image frame.
The device according to claim 32, wherein the processor implements the following steps when executing the computer program:

saving at least a part of the correctly transmitted image frames into a reference frame management queue;

The determining the reference frame according to the correctly transmitted image frame includes:

An image frame in the reference frame management queue is determined as the reference frame.
The device according to claim 31, wherein the processor implements the following steps when executing the computer program:

If the second feedback information is not received, determining a preset number of image frames sent before the image frame as the reference frame.
The device according to claim 23, wherein the processor implements the following steps when executing the computer program:

Packing the coded data of the image frame to generate a data packet, the data packet includes a packet sequence number and/or a check code;

and controlling the communication module to send the data packet to the receiving end.
The device according to claim 23, wherein the processor implements the following steps when executing the computer program:

The communication module is controlled to receive a data packet obtained by packaging the coded data of the image frame from the sending end, and the data packet includes a packet sequence number and/or a check code.
The device according to claim 36, wherein the processor implements the following steps when executing the computer program:

Detecting the coded data of the image frame according to the packet sequence number and/or check code of the data packet;

Generate the first feedback information according to the detection result.
The device according to claim 37, wherein the first feedback information is used to indicate whether the reception is wrong.
The device according to claim 37, wherein the first feedback information is used to indicate the frame number of the correctly decoded image frame and whether the reception is wrong.
The device according to claim 23, wherein the communication module is a 5G module.
The device according to claim 40, wherein the sending end is another transit end.
The device according to claim 40, wherein the receiving end is another transfer end.
The device according to claim 40, wherein the relay terminal is connected to a 5G base station.
The device according to claim 23, wherein the sending end comprises a drone.
A service server, characterized in that the service server includes the image processing device at the transfer terminal according to any one of claims 23-44.
A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor realizes any one of claims 1-22. image processing method.