WO2023246936A1

WO2023246936A1 - Image processing method and apparatus, and device

Info

Publication number: WO2023246936A1
Application number: PCT/CN2023/102139
Authority: WO
Inventors: 马强; 高在伟
Original assignee: 杭州海康威视数字技术股份有限公司
Priority date: 2022-06-24
Filing date: 2023-06-25
Publication date: 2023-12-28

Abstract

The present application provides an image processing method and apparatus, and a device. The method comprises: dividing all P frames in a GOP sequence into at least two types of P frames, the at least two types of P frames comprising P frames that cannot be used as reference frames and P frames that can be used as reference frames; and if it is determined that the P frames in the GOP sequence need to be subjected to step printing, performing step printing on the P frames that cannot be used as reference frames, or if it is determined that the P frames in the GOP sequence need to be subjected to step printing, performing step printing on the P frames that cannot be used as reference frames and the P frames that can be used as reference frames. By means of the technical solution of the present application, frame dropping can be performed according to a certain rule, such that a video frame sequence can be ensured to be played in order, and some P frames can also be dropped in some special scenarios to save bandwidth.

Description

An image processing method, device and equipment

Technical field

The present application relates to the field of video coding and decoding, and in particular, to an image processing method, device and equipment.

Background technique

In order to save space, video images are transmitted after encoding. The complete video coding method can include prediction, transformation, quantization, entropy coding, filtering and other processes, and predictive coding can include intra-frame coding and inter-frame coding. . Inter-frame coding uses the correlation in the video time domain to predict the pixels of the current image using the pixels of adjacent encoded images to achieve the purpose of removing video time domain redundancy. Intra-frame coding takes into account the strong spatial correlation between adjacent blocks, uses the surrounding reconstructed pixels as reference pixels to predict the current uncoded block, and only needs to perform subsequent coding processing on the residual value instead of Encoding the original value effectively removes redundancy in the spatial domain and greatly improves compression efficiency.

In the video encoding process, all images can be divided into I frames and P frames. I frames are images that use intra-frame coding, that is, they can be independently encoded without referring to other images. P frames are images that use inter-frame coding. , that is, you need to refer to other images to complete the encoding, such as relying on the previous image to complete the encoding. For example, continuous video frames include frames 1 to 10 in sequence, frame 1 is an I frame, frame 2 to frame 5 are P frames, frame 6 is an I frame, and frame 7 to frame 10 are P frames. Then, frame 1 adopts frame Intra-coding, frame 2 refers to frame 1 to complete inter-frame coding, frame 3 refers to frame 2 to complete inter-frame coding, frame 4 refers to frame 3 to complete inter-frame coding, frame 5 refers to frame 4 to complete inter-frame coding, frame 6 uses intra-frame coding, Frame 7 completes interframe coding with reference to frame 6, and so on.

Obviously, in the above method, all P frames cannot be discarded. Once a certain P frame is discarded, subsequent P frames will not be processed correctly. For example, if frame 3 is discarded, frame 4 will not be able to complete inter-frame coding with reference to frame 3, and frame 5 will not be able to complete inter-frame coding with reference to frame 4.

Contents of the invention

The present application provides an image processing method. The method includes: dividing all P frames in the picture group GOP sequence into at least two types of P frames, and any of the at least two types of P frames cannot be used as reference frames. P frames, and there are P frames that can be used as reference frames among the at least two types of P frames; if it is determined that the P frames in the GOP sequence need to be framed, then the P frames that cannot be used as reference frames are Perform frame extraction, or if it is determined that P frames within the GOP sequence need to be extracted, then extract frames that cannot be used as reference frames and P frames that can be used as reference frames.

In a possible implementation, the method includes: dividing all P frames in the GOP sequence into a first type of P frame and a second type of P frame; wherein the first type of P frame cannot be used as Reference frame, the second type P frame can be used as a reference frame of the first type P frame, and the second type P frame can be used as a reference frame of the second type P frame; if it is determined that it is necessary to When the P frame is extracted, the first type of P frame is extracted, or the first type of P frame and the second type of P frame are extracted.

In a possible implementation, dividing all P frames in the GOP sequence into first type P frames and second type P frames includes: based on The position sorting relationship divides all P frames in the GOP sequence into first type P frames and second type P frames; wherein, there are M1 first type P frames between two adjacent second type P frames, There are M2 first-type P frames between the I frame of the GOP sequence and the first second-type P frame; where M1 is a positive integer and M2 is a positive integer.

For each P frame of the first type, if there is a P frame of the second type before the P frame of the first type, the most recent P frame of the second type before the P frame of the first type is used as the reference frame of the P frame of the first type. , if there is no P frame of the second type before the P frame of the first type, then the I frame of the GOP sequence is used as the reference frame of the P frame of the first type;

For each second type P frame, if there is a second type P frame before the second type P frame, the nearest second type P frame before the second type P frame is used as the reference frame of the second type P frame. , if there is no second type P frame before the second type P frame, then the I frame of the GOP sequence is used as the reference frame of the second type P frame.

Exemplarily, if it is determined that the P frames in the GOP sequence need to be extracted, then extract the first type of P frames, or extract the first type of P frames and the second type of P frames. Extracting frames of P-type frames includes: if it is determined based on the service information that the first sampling ratio needs to be used to extract P frames in the GOP sequence, then extracting all P-frames of the first type or part of the P-frames of the first type. frames; if it is determined based on the service information that the second sampling ratio needs to be used to extract frames from the P frames in the GOP sequence, then extract frames from all first type P frames, and extract frames from all second type P frames or part of the second type P-like frames are used to extract frames; or,

If it is determined based on the business information that the first sampling ratio needs to be used to extract frames from the P frames in the GOP sequence, then some of the first type P frames will be extracted; if it is determined based on the business information that the second sampling ratio needs to be used to extract the frames, To extract frames for P frames in the GOP sequence, all P frames of the first type are extracted. If it is determined based on the service information that it is necessary to use the third sampling ratio to extract frames for P frames in the GOP sequence, then all P frames of the first type are extracted. The P frames of the first type are extracted, and all the P frames of the second type or part of the P frames of the second type are extracted;

Among them, the second sampling proportion is greater than the first sampling proportion, and the third sampling proportion is greater than the second sampling proportion.

In a possible implementation, the method further includes: if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoding end, based on the resolution, the The frame rate and the decoding capability determine whether frame extraction needs to be performed on the P frames in the GOP sequence; wherein, if the decoding capability does not support decoding of images with the resolution and the frame rate, it is determined that the P frames in the GOP sequence need to be decoded. The P frames in the GOP sequence are extracted; if so, determine the sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate and the decoding capability; or,

If the service information includes the network quality corresponding to the target network between the sending end and the receiving end, it is determined based on the network quality whether it is necessary to extract the P frames in the GOP sequence; wherein, if the network quality If the network quality is not the optimal level, it is determined to extract frames from the P frames in the GOP sequence; if so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality;

Wherein, the sampling ratio is the first sampling ratio or the second sampling ratio; or,

The sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio.

In a possible implementation, the method includes: dividing all P frames in the GOP sequence into a first type of P frame, a second type of P frame and a third type of P frame; wherein, the first type of P frame The P frame of the second type cannot be used as a reference frame, the P frame of the second type can only be used as the reference frame of the P frame of the first type, the P frame of the third type can be used as the reference frame of the P frame of the first type, and can be used as the reference frame of the P frame of the first type. The reference frame of the second type of P frame and can be used as the reference frame of the third type of P frame; if it is determined that the P frame in the GOP sequence needs to be extracted, then the first type of P frame is extracted, or , perform frame extraction on the first type P frame and the second type P frame, or perform frame extraction on the first type P frame, the second type P frame and the third type P frame. .

In a possible implementation, dividing all P frames in the GOP sequence into first type P frames, second type P frames and third type P frames includes: based on The positional sorting relationship between each P frame divides all P frames in the GOP sequence into the first type of P frame, the second type of P frame and the third type of P frame; wherein, two adjacent third type P frames There are two P frames of the first type and one P frame of the second type between frames, and the P frame of the second type is located between the two P frames of the first type; the I frame of the GOP sequence and the P frame of the second type There are two first-type P frames and one second-type P frame between a third-type P frame, and the second-type P frame is located between the two first-type P frames.

For example, for each first type P frame, if the previous frame of the first type P frame is not the first type P frame, then the previous frame of the first type P frame is regarded as the first type P frame. reference frame, if the previous frame of the first type P frame is a first type P frame, then the most recent non-first type P frame will be used as the reference frame of the first type P frame; for each second type P frame P frame, if there is a third type P frame before the second type P frame, then the latest third type P frame before the second type P frame is used as the reference frame of the second type P frame, if the second type P frame If there is no P frame of the third type before the P frame, then the I frame of the GOP sequence is used as the reference frame of the P frame of the second type; for each P frame of the third type, if there is a P frame of the third type before the P frame of the third type. If there are three types of P frames, then the latest type three P frame before the third type P frame will be used as the reference frame of the third type P frame. If there is no third type P frame before the third type P frame, then the third type P frame will be used as the reference frame. The I frame of the GOP sequence serves as the reference frame of the third type of P frame.

Exemplarily, if it is determined that the P frames in the GOP sequence need to be extracted, then extract the first type of P frames, or extract the first type of P frames and the second type of P frames. Performing frame extraction on type P frames, or performing frame extraction on the first type P frame, the second type P frame and the third type P frame, including:

If it is determined based on the service information that the first sampling ratio needs to be used to extract frames from the P frames in the GOP sequence, then extract frames from all first-type P frames or part of the first-type P frames;

If it is determined based on the service information that the second sampling ratio needs to be used to extract the P frames in the GOP sequence, then all the first type P frames will be extracted, and all the second type P frames or part of the second type P frames will be extracted. The frame is extracted; wherein the second sampling ratio is greater than the first sampling ratio;

If it is determined based on the service information that the third sampling ratio needs to be used to extract frames from the P frames in the GOP sequence, then extract frames from all P frames of the first type, extract frames from all P frames of the second type, and extract frames from all P frames of the second type. The third type P frame or part of the third type P frame is extracted; the third sampling ratio is greater than the second sampling ratio.

The method further includes: if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoding end, determining based on the resolution, the frame rate, and the decoding capability. Whether it is necessary to extract frames from P frames in the GOP sequence; wherein, if the decoding capability does not support decoding images of the resolution and frame rate, it is determined to extract frames from the P frames in the GOP sequence. frame; if so, determine the sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate and the decoding capability, and the sampling ratio is the first sampling ratio, or the second sampling ratio, or the third sampling ratio proportion; or, if the service information includes the network quality corresponding to the target network between the sending end and the receiving end, determine whether it is necessary to extract frames from the P frames in the GOP sequence based on the network quality; wherein, if If the network quality is not the optimal level, it is determined to extract frames from the P frames in the GOP sequence; if so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality, and the sampling ratio is The first sampling proportion, or the second sampling proportion, or the third sampling proportion.

Exemplarily, the method further includes: when the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence of the image sequence set, then replace the first GOP sequence with The I frame serves as the reference frame for the first frame of the GOP sequence.

The present application provides an image processing device. The device includes: a dividing module, configured to divide all P frames in a GOP sequence into at least two types of P frames. There are inaccuracies in the at least two types of P frames. A P frame as a reference frame, and there is a P frame that can be used as a reference frame among the at least two types of P frames; a processing module, configured to extract frames from the P frame in the GOP sequence if it is determined that the P frame needs to be extracted. The P frames that cannot be used as reference frames are extracted, or if it is determined that the P frames in the GOP sequence need to be extracted, the P frames that cannot be used as reference frames and the P frames that can be used as reference frames are extracted. frame.

In a possible implementation, the dividing module is specifically configured to divide all P frames in the GOP sequence into first type P frames and second type P frames; wherein the first type P frame cannot As a reference frame, the second type P frame can be used as a reference frame of the first type P frame, and the second type P frame can be used as a reference frame of the second type P frame; the processing module is specifically used if If it is determined that the P frames in the GOP sequence need to be extracted, the first type of P frames may be extracted, or the first type of P frames and the second type of P frames may be extracted.

Wherein, when the dividing module divides all P frames in the GOP sequence into first type P frames and second type P frames, it is specifically used to: based on the position ordering relationship between the P frames in the GOP sequence, All P frames within the GOP sequence are divided into first type P frames and second type P frames; where two adjacent There are M1 P frames of the first type between the P frames of the second type, and M2 P frames of the first type between the I frame of the GOP sequence and the first P frame of the second type; M1 and M2 are both positive integer.

Wherein, for each first type P frame, if there is a second type P frame before the first type P frame, the nearest second type P frame before the first type P frame is used as the first type P frame. Reference frame, if there is no P frame of the second type before the P frame of the first type, then the I frame of the GOP sequence is used as the reference frame of the P frame of the first type; for each P frame of the second type, if the P frame of the second type If there is a P frame of the second type before the P frame of the second type, the most recent P frame of the second type before the P frame of the second type is used as the reference frame of the P frame of the second type. If there is no P frame of the second type before the P frame of the second type. If the frame is a type P frame, the I frame of the GOP sequence is used as the reference frame of the second type P frame.

Wherein, if the processing module determines that it is necessary to extract frames from P frames in the GOP sequence, then extract frames from the first type of P frames, or extract frames from the first type of P frames and the second type of P frames. When extracting P frames, it is specifically used: if it is determined based on the service information that the first sampling ratio needs to be used to extract P frames in the GOP sequence, then extract all P frames of the first type or part of the P frames of the first type. Frame extraction; if it is determined based on the service information that the second sampling ratio needs to be used to extract frames from the P frames in the GOP sequence, then extract frames from all first type P frames, and extract frames from all second type P frames or part of the first type P frames. The second type P frames are extracted; or, if it is determined based on the service information that the first sampling ratio needs to be used to extract the P frames in the GOP sequence, then some first type P frames are extracted; if based on the service information, If it is determined that the second sampling ratio needs to be used to extract the P frames in the GOP sequence, then all the first type P frames need to be extracted; if it is determined based on the service information that the third sampling ratio needs to be used to extract the P frames in the GOP sequence To extract frames from P frames, extract frames from all P frames of the first type, and extract frames from all P frames of the second type or part of P frames of the second type; wherein, the second sampling ratio is greater than the first sampling ratio, and the second sampling ratio is larger than the first sampling ratio. The third sampling proportion is greater than the second sampling proportion.

Wherein, the processing module is also configured to: if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence and the decoding capability corresponding to the decoding end, based on the resolution, the frame rate and the decoding capability The decoding capability determines whether it is necessary to extract frames from the P frames in the GOP sequence; wherein, if the decoding capability does not support decoding images of the resolution and frame rate, it is determined that the P frames in the GOP sequence need to be extracted. The P frame is extracted; if so, determine the sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate and the decoding capability; or, if the service information includes the communication between the sending end and the receiving end. According to the network quality corresponding to the target network, it is determined based on the network quality whether it is necessary to extract frames from the P frames in the GOP sequence; wherein, if the network quality is not the optimal level of network quality, it is determined whether the P frames in the GOP sequence need to be extracted. The P frames in the sequence are extracted; if so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality; the sampling ratio is the first sampling ratio or the second sampling ratio; or, the sampling ratio is The first sampling proportion, or the second sampling proportion, or the third sampling proportion.

Among them, the dividing module is specifically used to divide all P frames in the GOP sequence into first type P frames, second type P frames and third type P frames; the first type P frame cannot be used as Reference frame, the second type of P frame can only be used as a reference frame of the first type of P frame, the third type of P frame can be used as a reference frame of the first type of P frame, can be used as a reference frame of the second type of P frame , and can be used as a reference frame for the third type of P frame; the processing module is specifically configured to extract frames for the first type of P frame if it is determined that the P frame in the GOP sequence needs to be extracted, Alternatively, perform frame extraction on the first type P frame and the second type P frame, or perform extraction on the first type P frame, the second type P frame and the third type P frame. frame.

Wherein, when the dividing module divides all P frames in the GOP sequence into the first type of P frame, the second type of P frame and the third type of P frame, it is specifically used: based on the position between each P frame in the GOP sequence. The sorting relationship divides all P frames in the GOP sequence into the first type of P frame, the second type of P frame and the third type of P frame; wherein, for all P frames in the said GOP sequence, two adjacent There are two P frames of the first type and one P frame of the second type between the P frames of the third type, and the P frame of the second type is located between the two P frames of the first type; the GOP sequence There are two first-type P frames and one second-type P frame between the I frame and the first third-type P frame, and the second-type P frame is located between the two first-type P frames.

Wherein, for each first type P frame, if the previous frame of the first type P frame is not the first type P frame, then the previous frame of the first type P frame is used as the reference of the first type P frame. frame, if the previous frame of the first type P frame is a first type P frame, then the most recent non-first type P frame is used as the reference frame of the first type P frame; for each second type P frame , if there is a P frame of the third type before the P frame of the second type, then the most recent P frame of the third type before the P frame of the second type is used as the reference frame of the P frame of the second type, if the P frame of the second type If there is no P frame of the third type in front, the I frame of the GOP sequence is used as the reference frame of the P frame of the second type; for each P frame of the third type, if there is a P frame of the third type in front of the P frame of the third type, P frame, then the latest P frame of the third type preceding the P frame of the third type is used as the reference frame of the P frame of the third type. If there is no P frame of the third type before the P frame of the third type, the P frame of the third type is used as the reference frame. The I frame of the GOP sequence serves as the reference frame of the third type of P frame.

Wherein, if the processing module determines that it is necessary to extract frames from P frames in the GOP sequence, then extract frames from the first type of P frames, or extract frames from the first type of P frames and the second type of P frames. Performing frame extraction on P frames, or performing frame extraction on the first type P frame, the second type P frame and the third type P frame is specifically used for: if it is determined based on the service information that it is necessary to use the first sampling If the proportion of P frames in the GOP sequence is used to extract frames, all first-type P frames or part of the first-type P frames are extracted; if it is determined based on the service information that a second sampling ratio needs to be used to extract P frames in the GOP sequence. To extract frames, extract frames for all P frames of the first type, and extract frames for all P frames of the second type or part of P frames of the second type; the second sampling ratio is greater than the first sampling ratio; if it is determined based on the business information, Using the third sampling ratio to extract frames from P frames in the GOP sequence, all P frames of the first type are extracted, all P frames of the second type are extracted, and all P frames of the third type or part of the P frames are extracted. Three types of P frames are used for frame extraction; the third sampling ratio is greater than the second sampling ratio.

Wherein, the processing module is also configured to: if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence and the decoding capability corresponding to the decoding end, based on the resolution, the frame rate and the decoding capability The decoding capability determines whether it is necessary to perform frame extraction on the P frames in the GOP sequence; wherein, if the decoding capability does not support decoding images of the resolution and frame rate, it is determined whether the P frames in the GOP sequence need to be decoded. The P frame is extracted; if so, the sampling ratio corresponding to the GOP sequence is determined based on the resolution, the frame rate and the decoding capability, and the sampling ratio is the first sampling ratio, or the second sampling ratio, Or the third sampling ratio; or, if the service information includes the network quality corresponding to the target network between the encoding end and the decoding end, determine whether the P frame in the GOP sequence needs to be extracted based on the network quality. ; Wherein, if the network quality is not the optimal level of network quality, it is determined to extract frames from the P frames in the GOP sequence; if so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality, so The above-mentioned sampling ratio is the first sampling ratio, or the second sampling ratio, or the third sampling ratio.

Wherein, when the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence of the image sequence set, the dividing module is also used to divide the first GOP sequence into The I frame serves as the reference frame for the first frame of the GOP sequence.

The present application provides an image processing device, including: a processor and a machine-readable storage medium. The machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is configured to execute machine-executable instructions. Execute instructions to implement the image processing method disclosed in the above embodiments of this application.

It can be seen from the above technical solution that in the embodiment of the present application, all P frames in the GOP (Group of Pictures) sequence can be divided into P frames that cannot be used as reference frames and P frames that can be used as reference frames. Here Basically, the P frames that cannot be used as reference frames can be extracted, or the P frames that cannot be used as reference frames and the P frames that can be used as reference frames can be extracted. The above method provides a new type of video frame sequence. The video frame sequence includes different types of P frames. In the video frame sequence, frames can be dropped according to certain rules, which can not only ensure that the video frame sequence is played in order, but also has the ability to interleave according to the rules. It has the characteristics of permanently discarding P frames without affecting viewing, and can also discard part of P frames in some special scenarios to achieve the purpose of saving bandwidth. This video frame sequence has better prospects for use in decoding, network transmission, and storage, and realizes decoding Features include doubled channel number, low-bandwidth transmission, and low bit rate storage.

Description of the drawings

Figure 1A and Figure 1B are schematic diagrams of the GOP sequence;

Figure 2 is a schematic flowchart of an image processing method in an embodiment of the present application;

Figure 3A and Figure 3B are schematic diagrams of the GOP sequence;

Figures 4A-4C are schematic diagrams of extracting frames from P frames in a GOP sequence;

Figures 5A-5E are schematic diagrams of images displayed through multiple channels;

Figure 6 is a schematic diagram of an image sequence collection;

Figure 7 is a schematic flowchart of an image processing method in an embodiment of the present application;

Figures 8A and 8B are schematic diagrams of the GOP sequence;

Figures 9A-9C are schematic diagrams of extracting frames from P frames in a GOP sequence;

Figure 10 is a schematic diagram of an image sequence collection;

Figure 11 is a schematic structural diagram of an image processing device in an embodiment of the present application;

Figure 12 is a hardware structure diagram of an image processing device in an embodiment of the present application.

Detailed ways

In the process of video encoding and video decoding, all images can be divided into I frames and P frames. I frames are images that use intra-frame coding, that is, they can be encoded independently without referring to other images. P frames use inter-frame coding. The image needs to refer to other images to complete the encoding, such as relying on the previous image to complete the encoding.

Video coding is a compression technology developed for image storage. It can compress images into smaller code streams and store the code streams on the hard disk. When the video needs to be played, the code stream is re-decoded into images and the images are displayed. When video encoding images, the encoding methods include H.264, H.265, etc.

Video decoding is a technology that re-decodes a code stream into an image. After the code stream (such as a code stream stored on a hard disk or transmitted from a network) is decoded into an image, the image can be displayed for video playback.

In the process of video encoding and video decoding, the I frame and all P frames between the I frame and the next I frame can be composed into a GOP sequence, that is, the first frame of the GOP sequence is an I frame, and the GOP sequence has only one I frame. , the I frame of the GOP sequence may be followed by a P frame. For example, continuous video frames include frames 1 to 10 in sequence, frame 1 is an I frame, frame 2 to frame 5 are P frames, frame 6 is an I frame, and frame 7 to frame 10 are P frames. Then, GOP sequence 1 includes Frame 1 to frame 5, and GOP sequence 2 includes frame 6 to frame 10. Referring to FIG. 1A , which shows GOP sequence 1 and GOP sequence 2, each GOP sequence may include one I frame and multiple P frames.

For each GOP sequence, each P frame in the GOP sequence needs to refer to the previous frame. For example, for GOP sequence 1, frame 2 refers to frame 1 to complete inter-frame coding, and frame 3 refers to frame 2 to complete inter-frame coding. Frame 4 refers to frame 3 to complete interframe coding, and frame 5 refers to frame 4 to complete interframe coding. However, in the above structure, all P frames cannot be discarded. Once a P frame is discarded, subsequent P frames will not be processed correctly. For example, as shown in Figure 1B, if frame 3 is discarded, frame 4 will not be able to complete inter-frame coding with reference to frame 3, and frame 5 will not be able to complete inter-frame coding with reference to frame 4. That is to say, all the frames after frame 3 will not be able to complete inter-frame coding with reference to frame 3. None of the P frames can be processed correctly. Under normal circumstances, there will be between 2 seconds and 10 seconds between two I frames. If a P frame is discarded, the image will not be displayed normally for several seconds.

To address the above problems, embodiments of the present application provide a new type of video frame sequence. The video frame sequence may include different types of P frames, such as P frames that cannot be used as reference frames and P frames that can be used as reference frames. The video frame sequence The frame sequence can be dropped according to certain rules, which not only ensures that the video frame sequence is played in order, but does not affect viewing even if P frames are discarded, it is also possible to save bandwidth by discarding some P frames. The new video frame sequence in the embodiment of the present application can divide all P frames in the GOP sequence into at least two types of P frames. There are inaccuracies in at least two types of P frames. P frames that serve as reference frames, and there are P frames that can serve as reference frames among at least two types of P frames. On this basis, if it is determined that the P frame within the GOP sequence needs to be extracted, then the P frame that cannot be used as a reference frame will be extracted, or if it is determined that the P frame within the GOP sequence needs to be extracted, , then the P frames that cannot be used as reference frames and the P frames that can be used as reference frames are extracted.

The technical solutions of the embodiments of the present application will be described below with reference to specific examples.

The embodiment of the present application proposes an image processing method, which can be applied to any device. See Figure 2, which is a schematic flowchart of the image processing method. The method may include:

Step 201: Divide all P frames in the GOP sequence into first type P frames and second type P frames; for example, the first type P frame cannot be used as a reference frame, and the second type P frame can be used as the second type P frame. A reference frame of a type of P frame, and the second type of P frame can also be used as a reference frame of a second type of P frame.

For example, for each GOP sequence, the GOP sequence may include an I frame and multiple P frames. In this embodiment of the present application, all P frames within the GOP sequence may be distinguished into two types of P frames. These two types of P frames can be called first type P frames and second type P frames respectively.

Among them, the first type of P frame can also be called a normal P frame, and the first type of P frame cannot be used as a reference frame for other frames. The P frame of the first type can use other frames as reference frames of the P frame of the first type. For example, the P frame of the first type can use the P frame of the second type as a reference frame. The P frame of the first type can also use the P frame of the GOP sequence as a reference frame. I frames serve as reference frames. However, a first-type P frame cannot use other first-type P frames as reference frames.

Among them, the second type of P frame can also be called a referenced P frame (that is, the second type of P frame can be used as a reference frame). The second type of P frame can be used as a reference frame of the first type of P frame. The second type of P frame can also be called a reference frame. Can be used as a reference frame for the second type of P frame. The second type of P frame can use other frames as reference frames of the second type of P frame. For example, the second type of P frame can use other second type of P frames as reference frames. The second type of P frame can also use the GOP sequence. The I frame is used as the reference frame. However, the P frame of the second type cannot use the P frame of the first type as a reference frame.

In a possible implementation, when dividing all P frames in the GOP sequence into first type P frames and second type P frames, any division method can be used, and there is no limit to this division method, as long as both The first type of P frame and the second type of P frame are enough. For example, all P frames in the GOP sequence can be divided into first type P frames and second type P frames based on the position ordering relationship between the P frames in the GOP sequence, that is, the division method is based on the position ordering relationship. . The positional sorting relationship refers to which P frame within the GOP sequence the P frame is, such as the first P frame, the second P frame, etc., which is used to indicate which position in the GOP sequence it is in.

For example, the first P frame is a P frame of the first type, the second P frame is a P frame of the second type, the third P frame is a P frame of the first type, and the fourth P frame is a P frame of the second type. And so on. For another example, the first P frame and the second P frame are P frames of the first type, the third P frame and the fourth P frame are P frames of the second type, and so on. For another example, the first P frame and the second P frame are the first type of P frames, the third P frame is the second type of P frame, and so on. For another example, the first P frame, the second P frame and the third P frame are the first type of P frames, the fourth P frame is the second type of P frame, and so on. For another example, the first P frame is a P frame of the first type, the second P frame and the third P frame are P frames of the second type, and so on. Of course, the above division methods are just examples.

For example, when all P frames are divided into first type P frames and second type P frames, the number of first type P frames and the number of second type P frames may be the same, and the number of first type P frames may be the same as the number of second type P frames. The number of P-frames of the second type can also be different. When the number of P frames of the first type is different from the number of P frames of the second type, the number of P frames of the first type may be greater than the number of P frames of the second type, and the number of P frames of the first type may also be smaller than the number of P frames of the second type. The number of frames, for example, in the following example, the number of P frames of the first type is greater than the number of P frames of the second type.

For example, when all P frames are divided into first type P frames and second type P frames, at least two first type P frames may appear continuously, but two second type P frames will not appear continuously. For example, what follows the first type of P frame can be the first type of P frame or the second type of P frame, and what follows the second type of P frame can be the first type of P frame, but what follows the second type of P frame will not It is the second type of P frame.

For example, the I frame of the GOP sequence may be followed by the P frame of the first type, and the I frame of the GOP sequence may be followed by the P frame of the second type. For example, the first P frame after the I frame in the GOP sequence is the first type of P frame.

In a possible implementation, when all P frames in the GOP sequence are divided into first type P frames and second type P frames, there are M1 first type P frames between two adjacent second type P frames. P frame, there are M2 first type P frames between the I frame (i.e. the first frame) of the GOP sequence and the first second type P frame, M1 is a positive integer, M2 is a positive integer, both M1 and M2 can be based on Experience configuration, M1 and M2 can be the same or different. Among them, when there are M1 first-category P frames between two adjacent second-category P frames, M1 can change. For example, between the first second-category P frame and the second second-category P frame There are 3 P frames of the first type between, there are 4 P frames of the first type between the second P frame of the second type and the third P frame of the second type, and there are 4 P frames of the first type between the third P frame of the second type and the fourth P frame of the second type. There are three P frames of the first type between the P frames of the second type, and so on.

For example, two adjacent P frames of the second type need to include P frames of the first type. The number of P frames of the first type can be maintained at a fixed number. For example, the number of P frames of the first type is more than 1, but , in order to prevent poor image quality, the number of first type P frames cannot be too many. For the convenience of description, in the embodiment of the present application, three P frames of the first type are included between two adjacent P frames of the second type as an example, and the I frame of the GOP sequence and the first P frame of the second type are There are 3 type 1 P frames in between. Of course, the above division method is just an example, and there is no limit to this. For example, two adjacent second-category P frames can include four first-category P frames, or two adjacent second-category P frames. For example, two P frames of the first type may be included between P frames.

For the convenience of description, in the embodiment of the present application, taking the GOP sequence shown in Figure 3A as an example, the first frame after the I frame is a first type P frame, and there are 3 frames between the I frame and the first second type P frame. A type 1 P frame. There are three first-type P frames between two adjacent second-type P frames. Referring to Figure 3A, the frames in the GOP sequence are: I frame, first type P frame, first type P frame, first type P frame, second type P frame, first type P frame, first type Class P frames, first class P frames, second class P frames, and so on.

In a possible implementation, based on "the first type of P frame is not used as a reference frame, the second type of P frame can be used as the reference frame of the first type of P frame, and the second type of P frame can be used as the second type of P frame. "reference frame" characteristics, then: for each first-type P frame, if there is a second-type P frame before the first-type P frame frame, then the nearest P frame of the second type preceding the P frame of the first type is used as the reference frame of the P frame of the first type. If there is no P frame of the second type before the P frame of the first type, the P frame of the GOP sequence is The I frame serves as the reference frame of the first type P frame. For each second type P frame, if there is a second type P frame before the second type P frame, the nearest second type P frame before the second type P frame is used as the reference frame of the second type P frame. , if there is no second type P frame before the second type P frame, then the I frame of the GOP sequence is used as the reference frame of the second type P frame.

Referring to FIG. 3B , the relationship between each P frame and the reference frame is shown. For the first type P frame, the reference frame of the first type first P frame is the previous I frame, the reference frame of the second type first P frame is the previous I frame, and the reference frame of the third type first P frame is the previous I frame. The reference frame of the frame is the previous I frame, the reference frame of the 4th type 1 P frame is the previous type 2 P frame, and the reference frame of the 5th type 1 P frame is the previous type 2 P frame. The reference frame of the sixth type-1 P frame is the previous type-2 P frame, and so on. For the second type P frame, the reference frame of the first type II P frame is the previous I frame, the reference frame of the second type II P frame is the previous type II P frame, and the reference frame of the third type II P frame is the previous type II P frame. The reference frame of a Type II P frame is the previous Type II P frame, and so on.

Step 202: If it is determined that the P frames in the GOP sequence need to be extracted, then extract the first type of P frames, or extract the first type of P frames and the second type of P frames. For example, extracting a P frame means not processing the P frame. For example, when the encoding end extracts the P frame, it means that the P frame is not encoded. When the decoding end extracts the P frame, it means that the P frame is not processed. This P frame is not decoded.

In a possible implementation, if it is determined that the P frames in the GOP sequence need to be extracted, all the first type P frames or part of the first type P frames can be extracted; or, all the first type P frames can be extracted. The P-type frames are extracted, and all the P-frames of the second type or part of the P-frames of the second type are extracted. In summary, it can be seen that when extracting frames of the first type of P frames, it is not necessary to consider whether the frames of the second type of P frames are extracted, and the frames of the first type of P frames can be extracted directly. When extracting frames of the second type of P frames, it is necessary to consider whether the first type of P frames have been extracted. Only after all first type of P frames have been extracted, can the second type of P frames be extracted.

In a possible implementation, in order to extract frames from P frames in a GOP sequence, multiple sampling ratios can be divided, and the sampling ratio is used to indicate how many P frames to extract. For example, when the sampling ratio is x, Indicates that x in all P frames is sampled, and different sampling ratios represent different sampling methods.

For example, two sampling proportions can be divided. These two sampling proportions are recorded as the first sampling proportion and the second sampling proportion, and the first sampling proportion is smaller than the second sampling proportion. For the first sampling ratio, frame extraction can be performed on all P frames of the first type or part of the P frames of the first type; for the second sampling ratio, frames can be extracted on all P frames of the first type, and on all P frames of the second type The frame or part of the second type P frame is extracted.

For another example, three sampling proportions can be divided. These three sampling proportions are recorded as the first sampling proportion, the second sampling proportion and the third sampling proportion. The first sampling proportion is smaller than the second sampling proportion, and the second sampling proportion is smaller than the third sampling proportion. Proportion. For the first sampling ratio, some P frames of the first type can be extracted; for the second sampling ratio, all P frames of the first type can be extracted; for the third sampling ratio, all P frames of the first type can be extracted Perform frame extraction, and perform frame extraction on all Type II P frames or part of Type II P frames.

For another example, 4 sampling proportions can be divided, recorded as the first sampling proportion, the second sampling proportion, the third sampling proportion and the fourth sampling proportion. The first sampling proportion is smaller than the second sampling proportion, and the second sampling proportion is smaller than the third sampling proportion. Proportion, the third sampling proportion is smaller than the fourth sampling proportion. For the first sampling ratio, frame extraction is performed on some P frames of the first type; for the second sampling ratio, frames are extracted on all P frames of the first type; for the third sampling ratio, frames are extracted on all P frames of the first type , extract frames for some P-frames of the second type; for the fourth sampling ratio, extract frames for all P-frames of the first type, and extract frames for all P-frames of the second type.

Of course, the above is just an example of the division of sampling proportions, and there is no limit to this. Any number of sampling proportions can be divided. For convenience of description, the following explanation will be based on dividing 2 or 3 sampling proportions as an example.

For example, on the basis of dividing the first sampling ratio, the second sampling ratio and the third sampling ratio, if it is necessary to use the first sampling ratio to extract the P frames in the GOP sequence, then some of the first type P frames Perform frame extraction; if it is necessary to use the second sampling ratio to extract frames from P frames in the GOP sequence, then extract frames from all first-type P frames; if it is necessary to use the third sampling ratio to extract P frames within the GOP sequence frames, then extract frames for all P-frames of the first type, and extract frames for all P-frames of the second type or part of the P-frames of the second type.

Assume that the total number of P frames of the first type accounts for x1 of the total number of P frames, and the total number of P frames of the second type accounts for x2 of the total number of P frames. The sum of x1 and x2 is 1, then the first sampling ratio is less than x1, The second sampling proportion is less than or equal to x1. Taking the second sampling proportion as x1 as an example, the third sampling proportion is less than or equal to x1+x2. Referring to Figure 3A and Figure 3B, x1 is 0.75 and x2 is 0.25. On this basis, if the sampling proportion is less than or equal to 0.5, it can be determined that the sampling proportion is the first sampling proportion. If the sampling proportion is greater than 0.5, and less than or If equal to 0.75, it can be determined that the sampling proportion is the second sampling proportion. If the sampling proportion is greater than 0.75 and less than or equal to 1, it can be determined that the sampling proportion is the third sampling proportion.

In summary, it can be seen that when extracting P frames, priority is given to extracting all or part of the P frames of the first type. After all P frames of the first type are extracted, all or part of the P frames of the second type are extracted. Frame extraction is performed locally. When extracting a part of the second type P frame, the previous second type P frame is retained, and the following second type P frame is extracted, such as after the kth (greater than 1) second type P frame. All second type P frames are extracted.

Referring to Figure 4A, if it is determined that the first sampling ratio needs to be used to extract the P frames in the GOP sequence, then some of the first type P frames will be extracted. In this case, the first sampling ratio is less than or equal to 0.5 , taking the first sampling ratio as 0.5 as an example, two first-category P frames need to be extracted between two second-category P frames, that is, one first-category P frame is retained between two second-category P frames. P frame. For example, three first-category P frames between two second-category P frames are extracted according to the strategy of discarding one, retaining one, and discarding one. Referring to Figure 4B, if it is determined that the second sampling ratio needs to be used to extract the P frames in the GOP sequence, then all first type P frames will be extracted. In this case, the second sampling ratio is less than or equal to 0.75 , and greater than 0.5. Referring to Figure 4C, if it is determined that the third sampling ratio needs to be used to extract frames from the P frames in the GOP sequence, then all first type P frames can be extracted, and all second type P frames can be extracted, In this case, the third sampling ratio is less than or equal to 1 and greater than 0.75.

As shown in Figure 4A, a GOP sequence can achieve 1/2 frame extraction. When 1/2 frame extraction cannot meet business requirements, as shown in Figure 4B, a The GOP sequence can achieve 3/4 frame extraction. When 3/4 frame extraction cannot meet the business requirements, as shown in Figure 4C, a GOP sequence can achieve complete frame extraction.

For example, on the basis of dividing the first sampling ratio and the second sampling ratio, if it is determined that the first sampling ratio needs to be used to extract the P frames in the GOP sequence, then all the first type P frames can be extracted. ; If it is determined that the second sampling ratio needs to be used to extract frames from P frames in the GOP sequence, then extract frames from all first-type P frames, and extract frames from all second-type P frames or part of second-type P frames. .

Assume that the total number of P frames of the first type accounts for x1 of the total number of P frames, the total number of P frames of the second type accounts for x2 of the total number of P frames, and the sum of x1 and x2 is 1, then the first sampling ratio is less than or equal to x1, the second sampling proportion is less than or equal to x1+x2. Referring to Figure 3A and Figure 3B, x1 is 0.75 and x2 is 0.25. On this basis, if the sampling proportion is less than or equal to 0.75, then the sampling proportion is determined to be the first sampling proportion. If the sampling proportion is greater than 0.75 and less than or equal to 1, then the sampling proportion is determined to be the second sampling proportion.

Referring to Figure 4B, if it is determined that the first sampling ratio needs to be used to extract the P frames in the GOP sequence, then all the first type P frames can be extracted. In this case, the first sampling ratio is less than or equal to 0.75. Referring to Figure 4C, if it is determined that the second sampling ratio needs to be used to extract the P frames in the GOP sequence, then all the first type P frames can be extracted, and all the second type P frames can be extracted, In this case, the second sampling ratio is less than or equal to 1 and greater than 0.75.

As shown in Figure 4B, a GOP sequence can achieve 3/4 frame extraction. When 3/4 frame extraction cannot meet business requirements, as shown in Figure 4C, a GOP sequence can achieve complete frame extraction.

In a possible implementation, on the basis of dividing the first sampling proportion, the second sampling proportion and the third sampling proportion, the sampling proportion corresponding to the GOP sequence can be determined based on the business information. For example, the sampling proportion corresponding to the GOP sequence can be determined based on the business information. A sampling ratio is used to extract frames from the P frames in the GOP sequence, or a second sampling ratio is determined based on the service information to extract frames from the P frames in the GOP sequence, or a third sampling ratio is determined based on the service information to extract the frames from the GOP sequence. The P frames within are extracted.

If the first sampling ratio is used to extract P frames in the GOP sequence, the service requirements corresponding to the service information can be met. For example, if 50% of the P frames in the GOP sequence are extracted to meet the service requirements, then the first sampling ratio is determined based on the service information. Sampling proportion. If the first sampling ratio cannot meet the business requirements, but the second sampling ratio is used to extract P frames within the GOP sequence, the business requirements corresponding to the service information can be met. That is, extracting 75% of the P frames from the GOP sequence can meet the requirements. For business needs, the second sampling ratio is determined based on business information. If the first sampling ratio cannot meet the business needs, and the second sampling ratio cannot meet the business needs, the third sampling ratio will be determined based on the business information.

In another possible implementation, on the basis of dividing the first sampling ratio and the second sampling ratio, the sampling ratio corresponding to the GOP sequence can be determined based on the business information. For example, the first sampling ratio is used to determine the GOP sequence based on the business information. The P frames within the GOP sequence are extracted, or the second sampling ratio is determined based on the service information to extract the P frames within the GOP sequence. For example, if the first sampling ratio is used to extract P frames within the GOP sequence, the service requirements corresponding to the service information can be met, that is, extracting 75% of the P frames from the GOP sequence can meet the service requirements, then the selection method is determined based on the service information. First sampling ratio. If the first sampling ratio cannot meet business needs, the second sampling ratio will be determined based on business information.

On the basis of dividing the first sampling ratio, the second sampling ratio, and the third sampling ratio, the above frame extraction process will be explained below in combination with two specific application scenarios of business information. Of course, on the basis of dividing the first sampling ratio and the second sampling ratio, the implementation method can be similar, and this article will not go into details.

Application scenario 1: The service information may include the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoder. Based on the resolution, the frame rate, and the decoding capability, it is determined whether it is necessary to modify the P in the GOP sequence. Frame is extracted. If not, frame extraction will not be performed on the P frames in the GOP sequence. If so, the sampling ratio corresponding to the GOP sequence is determined based on the resolution, the frame rate and the decoding capability. For example, the sampling ratio corresponding to the GOP sequence may be the first sampling ratio, the second sampling ratio, or the third sampling ratio. Alternatively, the sampling ratio corresponding to the GOP sequence is determined based on the resolution, the frame rate and the decoding capability, such as the first sampling ratio or the second sampling ratio.

For example, assuming that the NVR device has a decoding capability of 4*1080P30, that is, the NVR device can decode 4*30 frames of 1920*1080 images in one second. Images exceeding this decoding capability cannot be decoded.

a. If the GOP sequence corresponds to an image of 4*30 frames (the resolution is 1920*1080, the frame rate is 4*30 frames), then based on the resolution, frame rate and decoding capability, it is determined that there is no need to modify the P in the GOP sequence Frames are extracted, that is, the resolution, frame rate and decoding capabilities match. Therefore, the NVR device does not extract frames within the GOP sequence. It can decode 4*30 frames of images in one second and display 1080P30 through 4 channels. For an image, see Figure 5A.

b. If the GOP sequence corresponds to an image of 8*30 frames (the resolution is 1920*1080, the frame rate is 8*30 frames), it is determined based on the resolution, frame rate and decoding capability that the P frames in the GOP sequence need to be To perform frame extraction, that is, the resolution, frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images of this resolution and frame rate). Therefore, the NVR device determines that it is necessary to extract frames from the P frames in the GOP sequence. And each channel of image is extracted according to the first sampling ratio (i.e. 1/2 frame extraction method). When 4*30 frames of images can be decoded in one second, 1080P15 images can be displayed through 8 channels, as shown in Figure 5B , obviously, the 15 frames of 8 channels correspond to 4*30 frames.

Obviously, when the number of decoding channels increases, such as from 4 channels to 8 channels, the decoding capability can be supported by decoding images of the resolution and frame rate by extracting frames for each channel to meet business needs.

c. If the GOP sequence corresponds to an image of 4*60 frames (the resolution is 1920*1080, the frame rate is 4*60 frames), it is determined based on the resolution, frame rate and decoding capability that the P frames in the GOP sequence need to be To perform frame extraction, that is, the resolution, frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images of this resolution and frame rate). Therefore, the NVR device determines that it is necessary to extract frames from the P frames in the GOP sequence. And each channel of image is extracted according to the first sampling ratio (that is, 1/2 frame extraction method). When 4*30 frames of images can be decoded in one second, 1080P30 images can be displayed through 4 channels, as shown in Figure 5C , obviously, after 4*60 frames are extracted by 1/2, 4*30 frames remain.

Obviously, when playing back at double speed, such as changing from a 30-frame image to a 60-frame image, you can extract frames for each image so that the decoding capability can support decoding images of this resolution and frame rate to meet business needs. Obviously, 30 frames at 2x speed is equivalent to 60 frames per second, and 1/2 frame sampling is equal to 30 frames per second.

d. If the GOP sequence corresponds to an image of 12*30 frames (the resolution is 1920*1080 and the frame rate is 12*30 frames), it is determined based on the resolution, frame rate and decoding capability that the P frames in the GOP sequence need to be To perform frame extraction, that is, the resolution, frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images of this resolution and frame rate). Therefore, the NVR device determines that it is necessary to extract frames from the P frames in the GOP sequence. And each channel image is extracted according to the second sampling ratio (ie, 3/4 frame extraction method). When 4*30 frames of images can be decoded in one second, 1080P10 images can be displayed through 12 channels, as shown in Figure 5D , obviously, 10 frames of 12 channels correspond to 4*30 frames.

Obviously, when the number of decoding channels increases, such as from 4 channels to 12 channels, the decoding capability can be supported by decoding images of the resolution and frame rate by extracting frames for each channel of images to meet business needs.

e. If the GOP sequence corresponds to an image of 16*30 frames (the resolution is 1920*1080 and the frame rate is 16*30 frames), it is determined based on the resolution, frame rate and decoding capability that the P frames in the GOP sequence need to be To perform frame extraction, that is, the resolution, frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images of this resolution and frame rate). Therefore, the NVR device determines that it is necessary to extract frames from the P frames in the GOP sequence. And each channel of image is extracted according to the third sampling ratio (i.e. 4/4 frame extraction method). When 4*30 frames of images can be decoded in one second, 1080P7.5 images can be displayed through 16 channels, see Figure 5E As shown in the figure, obviously, the 7.5 frames of 16 channels correspond to 4*30 frames.

Obviously, when the number of decoding channels increases, such as from 4 channels to 16 channels, the decoding capability can be supported by decoding images of the resolution and frame rate by extracting frames for each channel of images to meet business needs.

In summary, it can be seen that when the conditions of multiple speeds and/or the number of multiple channels change arbitrarily, frames can be extracted according to the first sampling ratio, the second sampling ratio, or the third sampling ratio to meet business needs.

Application scenario 2: The service information may include the network quality corresponding to the target network between the sending end and the receiving end. Based on the network quality, it is determined whether the P frames in the GOP sequence need to be extracted. If not, frame extraction will not be performed on the P frames in the GOP sequence. If so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality. For example, the sampling ratio corresponding to the GOP sequence may be the first sampling ratio, the second sampling ratio, or the third sampling ratio. Alternatively, the sampling ratio corresponding to the GOP sequence is determined based on the network quality. The sampling ratio corresponding to the GOP sequence may be the first sampling ratio or the second sampling ratio.

For example, when the sender sends a GOP sequence to the receiver, it needs to detect the network quality (such as jitter, packet loss rate, etc.) corresponding to the target network used to carry the GOP sequence. There is no restriction on this detection method.

In a possible implementation, four levels of network quality can be designed. If the network quality corresponding to the target network is the network quality of the first level (ie, the optimal level), it is determined not to extract the P frames in the GOP sequence. frame. If the network quality corresponding to the target network is level 2 network quality, it is determined to extract frames from the P frames in the GOP sequence, and the sampling ratio is the first sampling ratio, such as 1/2 frame extraction. If the network quality corresponding to the target network is level 3 network quality, it is determined to extract frames from the P frames in the GOP sequence, and the sampling ratio is the second sampling ratio, such as 3/4 frame extraction. If the network quality corresponding to the target network is the network quality of level 4 (i.e., the worst level), it is determined to extract frames from the P frames in the GOP sequence, and the sampling ratio is the third sampling ratio, such as 4/4 frame extraction. Way.

In another possible implementation, two levels of network quality can be designed. If the network quality corresponding to the target network is the network quality of the first level (i.e., the optimal level), it is determined that the P frames in the GOP sequence are not to be processed. Frame extraction. If the network quality corresponding to the target network is the second level network quality, it is determined to extract frames from the P frames in the GOP sequence, and the sampling ratio is the first sampling ratio. During the transmission process of the GOP sequence, if the network quality is still If the network quality is Level 2, the sampling ratio will be adjusted to the second sampling ratio. During the transmission process of the GOP sequence, if the network quality is still Level 2, the sampling ratio will be adjusted to the third sampling ratio. In addition, during the transmission process of the GOP sequence, if the network quality returns to the first level of network quality, the P frames in the GOP sequence will not be extracted.

Of course, the above two application scenarios are just two examples of determining the sampling ratio. Other methods can also be used to determine the sampling ratio corresponding to the GOP sequence, and there are no restrictions on this. For example, the sampling ratio corresponding to the GOP sequence is determined based on the size of each frame. If the frame size is less than the first threshold, the P frames in the GOP sequence will not be sampled. The frame size is greater than the first threshold and less than the second threshold. limit, the sampling ratio is the first sampling ratio. When the frame size is greater than the second threshold and less than the third threshold, the sampling ratio is the second sampling ratio. When the frame size is greater than the third threshold, the sampling ratio is Third sampling ratio.

In a possible implementation, at least two GOP sequences can also be composed into an image sequence set. Based on this, when the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the The first GOP sequence of the image sequence set, then the I frame of the first GOP sequence can be used as the reference frame of the first frame of the GOP sequence. In this embodiment, if the GOP sequence is not the first GOP sequence in the image sequence set, the first frame of the GOP sequence can be called a key P frame (that is, it is no longer called an I frame), and the reference frame of the key P frame is the most recent I frame.

For example, for an image sequence set, the image sequence set may include at least one key P frame, that is, at least one key P frame is allowed to exist. The key P frame is used to replace the position of the I frame. The key P frame can only refer to the I frame. Its frame structure is the closest preceding I frame, and the key P frame can effectively reduce the overall bandwidth.

Referring to FIG. 6 , which is a schematic diagram of an image sequence set, the image sequence set may include key P frames. Between the I frame and the key P frame, there may be a first type of P frame and a second type of P frame. For the structural relationship between the first type of P frame and the second type of P frame, please refer to the above embodiment. Between two adjacent key P frames, there may be a first type of P frame and a second type of P frame. For the structural relationship between the first type of P frame and the second type of P frame, please refer to the above embodiment. On this basis, when extracting P frames, frames of the first type of P frames can be extracted, and frames of the second type of P frames can be extracted, but frames of key P frames cannot be extracted.

It can be seen from the above technical solution that in the embodiment of the present application, all P frames in the GOP sequence can be divided into first type P frames and second type P frames. The first type P frame cannot be used as a reference frame, and the second type P frame cannot be used as a reference frame. The frame can be used as the reference frame of the first type of P frame, and the second type of P frame can also be used as the reference frame of the second type of P frame. On this basis, the first type of P frame can be extracted, or the first type of P frame can be extracted. Class P frames and second class P frames are extracted. The above method provides a new type of video frame sequence. The video frame sequence includes different types of P frames, namely the first type of P frame and the second type of P frame. In the video frame sequence, frames can be dropped according to certain rules, which can It ensures that the video frame sequence is played in order, and has the characteristics of discarding P frames according to the rules without affecting viewing. It can also discard some P frames in some special scenarios to achieve the purpose of saving bandwidth. This video frame sequence is used in decoding and network It has better application prospects in transmission and storage, realizing features such as multiplied decoding channels, low-bandwidth transmission, and low bit rate storage. It can also solve problems such as long-term black screens and blurred screens after frame loss in weak network environments. The frame loss strategy can be dynamically implemented and arbitrarily adjusted according to on-site conditions, flexibly and effectively reducing consumption without affecting decoding integrity. This video frame sequence can ensure the flexibility of frame extraction and the stability of image quality, and can effectively improve customer experience. Certain rules of frame dropping can be performed during the decoding process to achieve decoding of 1/2 frame, 3/4 frame, etc. Or, realize low-bandwidth transmission of 1/2 frame and 3/4 frame in network transmission. This frame loss strategy can be dynamically implemented and the strategy can be adjusted arbitrarily according to the on-site situation. It is flexible and effective without affecting the decoding integrity. reduce consumption.

The embodiment of the present application proposes an image processing method, which can be applied to any device. See Figure 7, which is a schematic flow chart of the image processing method. The method may include:

Step 701: Divide all P frames in the GOP sequence into first type P frames, second type P frames and third type P frames; for example, the first type P frame cannot be used as a reference frame, and the second type P frame cannot be used as a reference frame. The P frame of the first type can only be used as a reference frame of the P frame of the first type. The P frame of the third type can be used as the reference frame of the P frame of the first type, and can be used as the reference frame of the P frame of the second type, and can be used as the reference frame of the P frame of the third type. The reference frame of the P frame.

For example, for each GOP sequence, the GOP sequence may include an I frame and multiple P frames. In this embodiment of the present application, all P frames within the GOP sequence may be distinguished into three types of P frames. These three types of P frames are called first type P frames, second type P frames and third type P frames.

Among them, the first type of P frame can also be called a normal P frame, and the first type of P frame cannot be used as a reference frame for other frames. The first type P frame can use other frames as reference frames of the first type P frame. For example, the first type P frame can use the second type P frame as a reference frame, and the first type P frame can use the third type P frame as a reference frame. frame as a reference frame, the first type of P frame can also use the I frame of the GOP sequence as a reference frame. However, the first type P frame cannot use the first type P frame (that is, other first type P frames except the first type P frame) as a reference frame.

Among them, the second type of P frame can also be called 2X type P frame. The second type of P frame can only be used as a reference frame of the first type of P frame, but cannot be used as a reference frame of the second type of P frame, nor can it be used as a reference frame of the third type of P frame. Reference frame for P-like frames. The second type of P frame can use other frames as reference frames of the second type of P frame. For example, the second type of P frame can use the third type of P frame as a reference frame. The second type of P frame can also use the reference frame of the GOP sequence. I frames serve as reference frames. However, the second type of P frame cannot use the second type of P frame (that is, other second type of P frame except the second type of P frame) as a reference frame, and the second type of P frame cannot use the first type of P frame as a reference frame. frame as a reference frame.

Among them, the third type of P frame can also be called a 4X type P frame. The third type of P frame can be used as a reference frame of the first type of P frame, and can be used as a reference frame of the second type of P frame, and can be used as a reference frame of the third type of P frame. The reference frame of the P frame. The third type P frame can use other frames as reference frames of the third type P frame. For example, the third type P frame can use the third type P frame (that is, other third type P frames except the third type P frame). Class P frame) as a reference frame, the third class P frame can also use the I frame of the GOP sequence as a reference frame. However, the third type P frame cannot use the second type P frame as a reference frame, and the third type P frame cannot use the first type P frame as a reference frame.

In a possible implementation, when dividing all P frames in the GOP sequence into the first type of P frames, the second type of P frames and the third type of P frames, any division method can be used. In this embodiment, There is no restriction on the division method, as long as the first type of P frame, the second type of P frame and the third type of P frame can exist at the same time. For example, all P frames in the GOP sequence can be divided into first type P frames, second type P frames and third type P frames based on the position ordering relationship between each P frame in the GOP sequence, that is, divided The method is based on the position ordering relationship. The position ordering relationship refers to which P frame within the GOP sequence the P frame is, such as the first P frame, the second P frame, the third P frame, etc., which is used to indicate which position in the GOP sequence it is in.

For example, the first P frame is a P frame of the first type, the second P frame is a P frame of the second type, the third P frame is a P frame of the third type, and the fourth P frame is a P frame of the first type. The fifth P frame is the second type of P frame, the sixth P frame is the third type of P frame, and so on. For example, the first P frame and the second P frame are P frames of the first type, the third P frame and the fourth P frame are P frames of the second type, and the fifth P frame and the sixth P frame are The third type of P frame, and so on. For another example, the first P frame is a P frame of the first type, the second P frame is a P frame of the second type, the third P frame is a P frame of the first type, and the fourth P frame is a P frame of the third type. , the fifth P frame is the first type of P frame, the sixth P frame is the second type of P frame, and so on. For example, the first P frame and the second P frame are the first type P frames, the third P frame is the second type P frame, the fourth P frame is the third type P frame, the fifth P frame and The sixth P frame is the first type of P frame, and so on. Of course, the above divisions are just a few examples.

For example, when all P frames are divided into the first type of P frames, the second type of P frames and the third type of P frames, the number of the first type of P frames and the number of the second type of P frames may be the same. The number of P-type frames may be different from the number of P-frames of the second type. The number of P frames of the first type and the number of P frames of the third type may be the same, and the number of P frames of the first type and the number of P frames of the third type may be different. The number of P frames of the second type and the number of P frames of the third type may be the same, and the number of P frames of the second type and the number of P frames of the third type may be different.

For example, the number of P frames of the first type may be greater than the number of P frames of the second type, and the number of P frames of the first type may be less than the number of P frames of the second type. The number of P frames of the first type may be greater than the number of P frames of the third type, and the number of P frames of the first type may be less than the number of P frames of the third type. The number of P frames of the second type may be greater than the number of P frames of the third type, and the number of P frames of the second type may be less than the number of P frames of the third type.

For example, the number of P frames of the first type is greater than the number of P frames of the second type, the number of P frames of the first type is greater than the number of P frames of the third type, and the number of P frames of the second type is equal to the number of P frames of the third type. quantity.

For another example, the number of P frames of the second type is equal to the number of P frames of the third type, and the number of P frames of the first type is greater than or equal to the number of P frames of the second type and the number of P frames of the third type. The sum of the number of P frames of the three types.

For example, when all P frames are divided into the first type of P frames, the second type of P frames and the third type of P frames, the first type of P frames, the second type of P frames and the third type of P frames may appear alternately. , that is, two P frames of the first type will not appear continuously, two P frames of the second type will not appear continuously, and two P frames of the third type will not appear continuously. For example, the P frame of the first type may be followed by the P frame of the second type or the P frame of the third type, but it will not be the P frame of the first type. The P frame after the second type P frame may be the P frame of the first type or the P frame of the third type. The third type of P frame, but not the second type of P frame. The third type of P frame can be followed by the first type of P frame or the second type of P frame, but it will not be the third type of P frame.

For example, a P frame of the first type may be followed by a P frame of the second type or a P frame of a third type, a P frame of the second type may be followed by a P frame of the first type, and a P frame of the third type may be followed by a P frame of the first type. Class P frame.

For example, the I frame of the GOP sequence may be followed by the P frame of the first type, the I frame of the GOP sequence may be followed by the P frame of the second type, and the I frame of the GOP sequence may be followed by the P frame of the third type. This is not done. limit.

For example, the first P frame after the I frame in the GOP sequence is the first type of P frame.

In a possible implementation, when all P frames in the GOP sequence are divided into first type P frames, second type P frames and third type P frames, between two adjacent third type P frames There may be two first-type P frames and one second-type P frame, and a second-type P frame may be located between two first-type P frames. In addition, there may be two first-type P frames and one second-type P frame between the I frame (i.e., the first frame) of the GOP sequence and the first third type P frame, and this second type P frame may be located between these two first-class P frames.

Of course, the above division method is just an example, and there is no limit to this. For example, there can be three first-type P frames and two second-type P frames between two adjacent third-type P frames, and the second type A class P frame is located between two first class P frames. For another example, there may be two first-type P frames and three second-type P frames between two adjacent third-type P frames, and the first-type P frame is located between two second-type P frames. For another example, there may be two P frames of the first type and two P frames of the second type between two adjacent P frames of the third type. The P frame of the second type is located between two P frames of the first type, and the P frame of the second type is located between the P frames of the second type. A P-frame of type 1 is located between two P-frames of type 2. For another example, there may be three P frames of the first type and three P frames of the second type between two adjacent P frames of the third type. The P frame of the second type is located between two P frames of the first type, and the P frame of the second type is located between two P frames of the first type. A P-frame of type 1 is located between two P-frames of type 2.

For the convenience of description, in this embodiment, taking the GOP sequence shown in Figure 8A as an example, the first frame after the I frame is a first type P frame, and there are two types of P frames between the I frame and the first third type P frame. A P frame of the first type and a P frame of a second type, and the P frame of the second type is located between two P frames of the first type. There are two first-type P frames and one second-type P frame between two adjacent third-type P frames, and the second-type P frame is located between two first-type P frames. Referring to Figure 8A, the frames in the GOP sequence are: I frame, first type P frame, second type P frame, first type P frame, third type P frame, first type P frame, second type P frame. P frames of the first type, P frames of the third type, P frames of the first type, P frames of the second type, P frames of the first type, P frames of the third type, and so on.

In a possible implementation, based on "the first type of P frame is not used as a reference frame, the second type of P frame is used as the reference frame of the first type of P frame, and the third type of P frame is used as the reference frame of the first type of P frame. , and as the reference frame of the second type of P frame, and as the reference frame of the third type of P frame", then: for each first type of P frame, if the previous frame of the first type of P frame is not the first type of P frame, For a P frame of the first type, the frame preceding the P frame of the first type (which may be an I frame, a P frame of the second type, or a P frame of the third type) is used as the reference frame of the P frame of the first type. If the previous frame of the first type P frame is a first type P frame, then the nearest non-first type P frame (such as an I frame, a second type P frame, or a third type P frame) before the first type P frame is P frame) as the reference frame of the first type of P frame.

For each P frame of the second type, if there is a P frame of the third type before the P frame of the second type, the nearest P frame of the third type before the P frame of the second type is used as the reference frame of the P frame of the second type. . If there is no P frame of the third type before the P frame of the second type, the I frame of the GOP sequence is used as the reference frame of the P frame of the second type.

For each type 3 P frame, if there is a type 3 P frame before the type 3 P frame, then the latest type 3 P frame before the type 3 P frame is used as the reference frame of the type 3 P frame. . If there is no third type P frame before the third type P frame, the I frame of the GOP sequence is used as the reference frame of the third type P frame.

Referring to FIG. 8B , the relationship between each P frame and the reference frame is shown. For the first type of P frame, the reference frame of the first type of P frame is the previous I frame, the reference frame of the second type of first P frame is the previous type of second P frame, and the reference frame of the third type of P frame is the previous type of P frame. The reference frame of a P frame of the first type is the previous P frame of the third type, the reference frame of the fourth P frame of the first type is the previous P frame of the second type, and the reference frame of the fifth P frame of the first type is the previous P frame of the first type. The third type of P frame, and so on. For type 2 P frames, the reference frame of the first type 2 P frame is the previous I frame, the reference frame of the second type 2 P frame is the previous type 3 P frame, and the reference frame of the third type 2 P frame is the previous type 3 P frame. The reference frame of the second type P frame is the previous third type P frame, and so on. For the third type of P frame, the reference frame of the first type III P frame is the previous I frame, the reference frame of the second type III P frame is the previous type III P frame, and the reference frame of the third type III P frame is the previous type III P frame. The reference frame of the third type P frame is the previous third type P frame, and so on.

Step 702: If it is determined that the P frames in the GOP sequence need to be extracted, the first type of P frames are extracted, or the first type of P frames and the second type of P frames are extracted, or the first type of P frames are extracted, or the first type of P frames are extracted. P-frames of the first type, P-frames of the second type and P-frames of the third type are extracted. For example, extracting the P frame may mean: not processing the P frame. For example, when the encoding end extracts the P frame, it means that the P frame is not encoded. When the decoding end extracts the P frame, Just don't decode this P frame.

In a possible implementation, if it is determined that the P frames in the GOP sequence need to be extracted, all the first type P frames or part of the first type P frames can be extracted; or, all the first type P frames can be extracted. Frame extraction is performed on P-type frames, and all P-frames of the second type or part of the P-frames of the second type are extracted; alternatively, all P-frames of the first type can be extracted, and all P-frames of the second type can be extracted. , and perform frame extraction on all the third type P frames or part of the third type P frames. In summary, it can be seen that when extracting frames of the first type of P frames, there is no need to consider whether the second type of P frames and third type of P frames are extracted, and the first type of P frames can be extracted directly. When extracting frames of the second type of P frames, it is necessary to consider whether the first type of P frames have been extracted. It is not necessary to consider whether the third type of P frames have been extracted. Only after all first type of P frames have been extracted. , only then can the second type of P frame be extracted. When extracting frames of the third type of P frames, it is necessary to consider whether the first type of P frames are extracted, and whether the second type of P frames are extracted. Only when all the first type of P frames are extracted, And only after all the P frames of the second type have been extracted, the P frames of the third type can be extracted.

For example, three sampling proportions are divided, recorded as the first sampling proportion, the second sampling proportion and the third sampling proportion. The first sampling proportion is smaller than the second sampling proportion, and the second sampling proportion is smaller than the third sampling proportion. For the first sampling ratio, frame extraction is performed on all P frames of the first type or part of the P frames of the first type; for the second sampling ratio, frames are extracted on all P frames of the first type, and on all P frames of the second type or Frame extraction is performed on some P-frames of the second type; for the third sampling ratio, all P-frames of the first type are extracted, all P-frames of the second type are extracted, and all P-frames of the third type or part of the third-category P-frame are extracted. P-like frames are used for frame extraction.

For another example, six sampling proportions are divided, recorded as the first sampling proportion, the second sampling proportion, the third sampling proportion, the fourth sampling proportion, the fifth sampling proportion and the sixth sampling proportion. The first sampling proportion is smaller than the second sampling proportion. , the second sampling proportion is smaller than the third sampling proportion, the third sampling proportion is smaller than the fourth sampling proportion, the fourth sampling proportion is smaller than the fifth sampling proportion, and the fifth sampling proportion is smaller than the sixth sampling proportion. For the first sampling ratio, frame extraction is performed on some P frames of the first type; for the second sampling ratio, frames are extracted on all P frames of the first type; for the third sampling ratio, frames are extracted on all P frames of the first type , extract frames for some P-frames of the second type; for the fourth sampling ratio, extract frames for all P-frames of the first type, and for all P-frames of the second type; for the fifth sampling ratio, extract frames for all P-frames of the first type P frames are extracted, and frames are extracted for all P frames of the second type, and some P frames of the third type are extracted; for the sixth sampling ratio, frames are extracted for all P frames of the first type, and all P frames of the second type are extracted. Frame extraction is performed on all P frames of the third type.

Of course, the above is just an example of the division of sampling proportions, and there is no restriction on this. Any number of sampling proportions can be divided. For the convenience of description, the following description will be made by taking the division of three sampling proportions as an example.

On this basis, if it is determined that the first sampling ratio needs to be used to extract frames from P frames in the GOP sequence, all first type P frames or part of the first type P frames can be extracted; if it is determined that the second sampling ratio needs to be used, If the sampling ratio is used to extract the P frames in the GOP sequence, all the first type P frames can be extracted, and all the second type P frames or part of the second type P frames can be extracted; if it is determined that the first type P frame needs to be used, The three-sampling ratio is used to extract frames from P frames in the GOP sequence. Then, all P frames of the first type can be extracted, all P frames of the second type can be extracted, and all P frames of the third type or part of the third type can be extracted. P-like frames are used for frame extraction.

For example, assume that the total number of P frames of the first type accounts for x1 of the total number of P frames, the total number of P frames of the second type accounts for x2 of the total number of P frames, and the total number of P frames of the third type accounts for x2 of the total number of P frames. x3 of Or equal to x1+x2+x3.

Referring to Figure 8A and Figure 8B, x1 is 0.5, x2 is 0.25, and x3 is 0.25. On this basis, if the sampling ratio is less than or equal to 0.5, it means that the sampling ratio is the first sampling ratio. If the sampling ratio is greater than 0.5, And less than or equal to 0.75, it means that the sampling ratio is the second sampling ratio. If the sampling ratio is greater than 0.75 and less than or equal to 1, it means that the sampling ratio is the third sampling ratio.

In summary, it can be seen that when extracting P frames, priority is given to extracting all or part of the P frames of the first type. After all P frames of the first type are extracted, all P frames of the second type can be extracted. Or partial frame extraction is performed. After all second type P frames are extracted, frame extraction can be performed on all or part of the third type P frame. When extracting a part of the third type P frame, the previous third type P frame is retained, and the following third type P frame is extracted, such as after the kth (k is greater than 1) third type P frame. All third-category P frames are extracted.

For example, as shown in Figure 9A, if it is determined that the first sampling ratio needs to be used to extract the P frames in the GOP sequence, then all the first type P frames can be extracted. In this case, the first sampling ratio The ratio is less than or equal to 0.5. Referring to Figure 9B, if it is determined that the second sampling ratio needs to be used to extract frames from the P frames in the GOP sequence, then all first type P frames can be extracted, and all second type P frames can be extracted, In this case, the second sampling ratio is less than or equal to 0.75 and greater than 0.5. Referring to Figure 9C, if it is determined that the third sampling ratio needs to be used to extract frames from P frames in the GOP sequence, frames can be extracted from all P frames of the first type, and frames can be extracted from all P frames of the second type, and Frame sampling is performed on all P frames of the third type. In this case, the third sampling ratio is less than or equal to 1 and greater than 0.75.

As shown in Figure 9A, a GOP sequence can achieve 1/2 frame extraction. When 1/2 frame extraction cannot meet business requirements, as shown in Figure 9B, a GOP sequence can achieve 3/4 frame extraction. When 3/4 frame extraction When frame extraction cannot meet business requirements, as shown in Figure 9C, a GOP sequence can achieve complete frame extraction.

In a possible implementation, the sampling ratio corresponding to the GOP sequence can be determined based on the service information. For example, it is determined based on the service information that the first sampling ratio is used to extract the P frames in the GOP sequence, or the sampling ratio is determined based on the service information. The second sampling ratio is used to extract frames from the P frames in the GOP sequence, or the third sampling ratio is used to extract frames from the P frames in the GOP sequence based on the service information.

If the first sampling ratio is used to extract P frames in the GOP sequence, the service requirements corresponding to the service information can be met, that is, the P frames extracted by x1 (such as 0.5) from the GOP sequence can meet the service requirements, and the selection is determined based on the service information. First sampling ratio. If the first sampling ratio cannot meet the business requirements, but the second sampling ratio is used to extract frames from P frames in the GOP sequence, the business requirements corresponding to the business information can be met, that is, x1+x2 (such as 0.75) is extracted from the GOP sequence. The P frame can meet the business requirements, and the second sampling ratio is determined based on the business information. If the first sampling ratio cannot meet the business needs, and the second sampling ratio cannot meet the business needs, the third sampling ratio will be determined based on the business information.

The above frame extraction process is explained below based on two specific application scenarios of business information.

Application scenario 1: Business information includes the resolution corresponding to the GOP sequence (for example, 4*30 frames represents the frame rate), the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoder. Determine whether it is needed based on the resolution, frame rate, and decoding capability. Extract frames from P frames within the GOP sequence. If not, frame extraction will not be performed on the P frames in the GOP sequence. If so, the sampling ratio corresponding to the GOP sequence is determined based on the resolution, the frame rate and the decoding capability, such as the first sampling ratio, the second sampling ratio, or the third sampling ratio.

a. If the GOP sequence corresponds to an image of 4*30 frames (resolution is 1920*1080), it is determined based on the resolution, frame rate and decoding capability that there is no need to extract the P frames in the GOP sequence, that is, the resolution , frame rate and decoding capability match, therefore, the NVR device does not extract frames from P frames in the GOP sequence, can decode 4*30 frames of images in one second, and display 1080P30 images through 4 channels, see Figure 5A.

b. If the GOP sequence corresponds to an 8*30 frame image (resolution is 1920*1080), it is determined based on the resolution, frame rate and decoding capability that the P frames in the GOP sequence need to be extracted, that is, the resolution, The frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images of this resolution and frame rate). Therefore, it is determined that the P frames in the GOP sequence need to be extracted, and each image must be sampled according to the first sampling ratio (i.e. 1/2 frame extraction method), when 4*30 frames of images can be decoded in one second, 1080P15 images can be displayed through 8 channels, as shown in Figure 5B. Obviously, 15 frames of 8 channels correspond to 4*30 frames.

c. If the GOP sequence corresponds to an image of 4*60 frames (resolution 1920*1080), based on the resolution, frame rate and decoding capability, it is determined that the P frames in the GOP sequence need to be extracted, that is, the resolution, frame The rate and decoding capability do not match (that is, the decoding capability does not support decoding images of this resolution and frame rate). Therefore, it is determined that the P frames in the GOP sequence need to be extracted, and each image must be sampled according to the first sampling ratio (i.e. 1 /2 frame extraction method) to extract frames, when 4*30 frames of images can be decoded in one second, 1080P30 images can be displayed through 4 channels, as shown in Figure 5C. Obviously, 4*60 frames are extracted by 1/2 4*30 frames remain after the frame.

Obviously, when playing back at double speed, such as changing from a 30-frame image to a 60-frame image, you can extract frames for each image so that the decoding capability can support decoding images of this resolution and frame rate to meet business needs, that is, Say, 30 frames at 2x speed is equivalent to 60 frames per second, and 1/2 frame extraction is equal to 30 frames per second.

d. If the GOP sequence corresponds to an image of 12*30 frames (resolution is 1920*1080), it is determined based on the resolution, frame rate and decoding capability that the P frames in the GOP sequence need to be extracted, that is, the resolution, The frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images of this resolution and frame rate). Therefore, it is determined that the P frames in the GOP sequence need to be extracted, and each image must be sampled according to the second sampling ratio (i.e. 3/4 frame extraction method), when 4*30 frames of images can be decoded in one second, 1080P10 images can be displayed through 12 channels, as shown in Figure 5D. Obviously, 10 frames of 12 channels correspond to 4*30 frames.

e. If the GOP sequence corresponds to an image of 16*30 frames (resolution is 1920*1080), it is determined based on the resolution, frame rate and decoding capability that the P frames in the GOP sequence need to be extracted, that is, the resolution, The frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images of this resolution and frame rate). Therefore, it is determined that the P frames in the GOP sequence need to be extracted, and each image must be sampled according to the third sampling ratio (i.e. 4/4 frame extraction method), when 4*30 frames of images can be decoded in one second, 1080P7.5 images can be displayed through 16 channels, as shown in Figure 5E. Obviously, 7.5 frames of 16 channels correspond to It's 4*30 frames.

Application scenario 2: If the service information includes the network quality corresponding to the target network between the sending end and the receiving end, it is determined based on the network quality whether the P frame in the GOP sequence needs to be extracted. If not, frame extraction will not be performed on the P frames in the GOP sequence. If so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality, such as the first sampling ratio, the second sampling ratio, or the third sampling ratio.

In another possible implementation, two levels of network quality can be designed. If the network quality corresponding to the target network is the network quality of the first level (i.e., the optimal level), it is determined that the P frames in the GOP sequence are not to be processed. Frame extraction. If the network quality corresponding to the target network is the second level network quality, it is determined to extract frames from the P frames in the GOP sequence, and the sampling ratio is the first sampling ratio. During the transmission process of the GOP sequence, if the network quality is still If the network quality is Level 2, the sampling ratio will be adjusted to the second sampling ratio. During the transmission process of the GOP sequence, if the network quality is still Level 2 network quality, the sampling ratio will be adjusted to the third sampling ratio. In addition, during the transmission process of the GOP sequence, if the network quality returns to the first level of network quality, the P frames in the GOP sequence will not be extracted.

Referring to FIG. 10 , which is a schematic diagram of an image sequence collection, the image sequence collection may include key P frames. Between the I frame and the key P frame, there may be the first type of P frame, the second type of P frame and the third type of P frame, and the structural relationship of the first type of P frame, the second type of P frame and the third type of P frame. See the above embodiments. Between two adjacent key P frames, there may be a first type of P frame, a second type of P frame and a third type of P frame, and the structure of the first type of P frame, the second type of P frame and the third type of P frame For the relationship, please refer to the above embodiment.

When extracting P frames, you can extract frames of the first type of P frames, you can extract frames of the second type of P frames, you can extract frames of the third type of P frames, but you cannot extract frames of key P frames. frame.

It can be seen from the above technical solution that this embodiment provides a new type of video frame sequence. The video frame sequence includes a first type of P frame, a second type of P frame, and a third type of P frame. Frames can be dropped according to certain rules, which can It ensures that the video frame sequence is played in order, and has the characteristics of discarding P frames according to the rules without affecting viewing. It can also discard some P frames in some special scenarios to achieve the purpose of saving bandwidth. The video frame sequence is decoded and transmitted over the network. , has better application prospects in storage, realizing the characteristics of multiplied decoding channels, low-bandwidth transmission, low bit rate storage, etc., and can also solve problems such as long-term black screen and blurred screen after frame loss in weak network environment. The frame loss strategy can be dynamically implemented and arbitrarily adjusted according to on-site conditions, flexibly and effectively reducing consumption without affecting decoding integrity. This video frame sequence can ensure the flexibility of frame extraction and the stability of image quality, and can effectively improve customer experience.

Based on the same application concept as the above method, an image processing device is proposed in the embodiment of the present application. See Figure 11, which is a schematic structural diagram of the image processing device. The device may include:

The dividing module 111 is configured to divide all P frames in the picture group GOP sequence into at least two types of P frames. Among the at least two types of P frames, there are P frames that cannot be used as reference frames, and the at least Among the two types of P frames, there are P frames that can serve as reference frames;

The processing module 112 is configured to extract frames that cannot be used as reference frames if it is determined that the P frames in the GOP sequence need to be extracted, or if it is determined that the P frames in the GOP sequence need to be extracted. When performing frame extraction, frame extraction is performed on P frames that cannot be used as reference frames and P frames that can be used as reference frames.

Exemplarily, the dividing module 111 is specifically used to divide all P frames in the GOP sequence into first type P frames and second type P frames; wherein the first type P frame cannot be used as a reference frame, and the second type P frame cannot be used as a reference frame. The frame can be used as a reference frame of the first type of P frame, and the second type of P frame can be used as a reference frame of the second type of P frame; the processing module 112 is specifically used to extract the P frame in the GOP sequence if it is determined that it is necessary to extract the frame. Then, frame extraction is performed on the P frame of the first type, or frame extraction is performed on the P frame of the first type and the P frame of the second type.

Exemplarily, when the dividing module 111 divides all P frames in the GOP sequence into the first type of P frames and the second type of P frames, it is specifically used to: based on the position ordering relationship between the P frames in the GOP sequence, divide the GOP into All P frames in the sequence are divided into first type P frames and second type P frames; among them, there are M1 first type P frames between two adjacent second type P frames, and the I frame of the GOP sequence is the same as the first type P frame. There are M2 P frames of the first type between the P frames of the second type; M1 and M2 are both positive integers.

For example, for each P frame of the first type, if there is a P frame of the second type before the P frame of the first type, the most recent P frame of the second type before the P frame of the first type is regarded as the P frame of the first type. The reference frame of the frame, if there is no P frame of the second type before the P frame of the first type, then the I frame of the GOP sequence is used as the reference frame of the P frame of the first type; for each P frame of the second type, if the P frame of the second type If there is a P frame of the second type before the P frame of the second type, the most recent P frame of the second type before the P frame of the second type is used as the reference frame of the P frame of the second type. If there is no P frame of the second type before it, For the second type of P frame, the I frame of the GOP sequence is used as the reference frame of the second type of P frame.

For example, if the processing module 112 determines that it is necessary to extract the P frames within the GOP sequence, then the first type of P frame is extracted, or specifically when the first type of P frame and the second type of P frame are extracted. Used for: If it is determined based on the business information that the first sampling ratio needs to be used to extract the P frames in the GOP sequence, then all first type P frames or part of the first type P frames are extracted; if it is determined based on the business information that it is necessary Using the second sampling ratio to extract frames from P frames in the GOP sequence, extract frames from all P frames of the first type, and extract frames from all P frames of the second type or part of the P frames of the second type; or, if If it is determined based on the business information that the first sampling ratio needs to be used to extract the P frames in the GOP sequence, then some of the first type P frames will be extracted; if it is determined based on the business information that the second sampling ratio needs to be used to extract the P frames in the GOP sequence If the frame is extracted, all first-type P frames will be extracted; if it is determined based on the service information that the third sampling ratio needs to be used to extract P frames in the GOP sequence, then all first-category P frames will be extracted. , and perform frame sampling on all second type P frames or part of the second type P frames; wherein the second sampling ratio is greater than the first sampling ratio, and the third sampling ratio is greater than the second sampling ratio.

Exemplarily, the processing module 112 is also configured to determine whether it is necessary to perform decoding based on the resolution, frame rate, and decoding capability if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoding end. The P frames in the GOP sequence are extracted; if the decoding capability does not support images with decoding resolution and frame rate, it is determined to extract the P frames in the GOP sequence; if so, based on the resolution, frame rate and decoding Ability determined The sampling ratio corresponding to the GOP sequence; or, if the service information includes the network quality corresponding to the target network between the sending end and the receiving end, determine whether the P frame in the GOP sequence needs to be extracted based on the network quality; where, if the network If the quality is not the optimal level of network quality, it is determined to extract frames from the P frames in the GOP sequence; if so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality; the sampling ratio is the first sampling ratio or the second sampling ratio; Alternatively, the sampling ratio is the first sampling ratio, or the second sampling ratio, or the third sampling ratio.

Exemplarily, the dividing module 111 is specifically used to divide all P frames in the GOP sequence into first type P frames, second type P frames and third type P frames; the first type P frame cannot be used as a reference. frame, the second type of P frame can only be used as a reference frame of the first type of P frame, the third type of P frame can be used as a reference frame of the first type of P frame, can be used as a reference frame of the second type of P frame, and can be used as a reference frame of the first type of P frame. The reference frame of the three types of P frames; the processing module 112 is specifically used to extract frames from the P frames in the GOP sequence if it is determined that the P frames in the GOP sequence need to be extracted, or extract the frames from the first type P frames and the first type P frames. The second type of P frame is extracted, or the first type of P frame, the second type of P frame and the third type of P frame are extracted.

Exemplarily, when the dividing module 111 divides all P frames in the GOP sequence into the first type of P frame, the second type of P frame and the third type of P frame, it is specifically used to: based on the difference between the P frames in the GOP sequence. The position sorting relationship divides all P frames in the GOP sequence into the first type of P frames, the second type of P frames and the third type of P frames; among them, for all P frames in the GOP sequence, two adjacent third type P frames There are two P frames of the first type and one P frame of the second type between P frames, and a P frame of the second type is located between two P frames of the first type; the I frame of the GOP sequence is the same as the first P frame of the third type. There are two P frames of the first type and one P frame of the second type between the P frames, and the P frame of the second type is located between the two P frames of the first type.

For example, for each first type P frame, if the previous frame of the first type P frame is not the first type P frame, then the previous frame of the first type P frame is regarded as the first type P frame. reference frame, if the previous frame of the first type P frame is a first type P frame, then the most recent non-first type P frame will be used as the reference frame of the first type P frame; for each second type P frame P frame, if there is a third type P frame before the second type P frame, then the latest third type P frame before the second type P frame is used as the reference frame of the second type P frame, if the second type P frame If there is no P frame of the third type before the P frame, the I frame of the GOP sequence is used as the reference frame of the P frame of the second type; for each P frame of the third type, if there is a P frame of the third type before the P frame of the third type, P frame, then the latest P frame of the third type before the P frame of the third type is used as the reference frame of the P frame of the third type. If there is no P frame of the third type before the P frame of the third type, the GOP sequence The I frame serves as the reference frame of the third type of P frame;

For example, if the processing module 112 determines that the P frames in the GOP sequence need to be extracted, then the first type of P frames need to be extracted, or the first type of P frames and the second type of P frames need to be extracted, or , when extracting frames of the first type of P frame, the second type of P frame and the third type of P frame, it is specifically used: if it is determined based on the service information that the first sampling ratio needs to be used to extract the P frame in the GOP sequence, then Perform frame extraction on all first-type P frames or part of the first-type P frames; if it is determined based on the service information that the second sampling ratio needs to be used to extract frames within the GOP sequence, then all first-type P frames shall be extracted. frame, and perform frame sampling on all second type P frames or part of the second type P frames; the second sampling ratio is greater than the first sampling ratio; if it is determined based on the business information that a third sampling ratio needs to be used to perform frame extraction on the P frames within the GOP sequence To extract frames, all P frames of the first type are extracted, all P frames of the second type are extracted, and all P frames of the third type or part of the P frames of the third type are extracted; the third sampling ratio is greater than Second sampling ratio.

Exemplarily, the processing module 112 is also configured to determine whether it is necessary to perform decoding based on the resolution, frame rate, and decoding capability if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoding end. The P frames in the GOP sequence are extracted; if the decoding capability does not support images with decoding resolution and frame rate, it is determined to extract the P frames in the GOP sequence; if so, based on the resolution, frame rate and decoding The ability to determine the sampling ratio corresponding to the GOP sequence, the sampling ratio is the first sampling ratio, or the second sampling ratio, or the third sampling ratio; or, if the service information includes the network quality corresponding to the target network between the encoding end and the decoding end, Then it is determined based on the network quality whether the P frame in the GOP sequence needs to be extracted; if the network quality is not the optimal level of network quality, it is determined to extract the P frame in the GOP sequence; if so, based on the network quality Determine the sampling ratio corresponding to the GOP sequence. The sampling ratio is the first sampling ratio, the second sampling ratio, or the third sampling ratio.

For example, when the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence of the image sequence set, the dividing module 111 is also used to divide the first GOP sequence into The I frame serves as the reference frame for the first frame of the GOP sequence.

Based on the same application concept as the above method, an image processing device is proposed in the embodiment of the present application. As shown in Figure 12, the image processing device includes a processor 1201 and a machine-readable storage medium 1202. The machine-readable storage medium 1202 stores Machine-executable instructions that can be executed by the processor 1201; the processor 1201 is used to execute the machine-executable instructions to implement the image processing method disclosed in the above examples of this application.

Based on the same application concept as the above method, embodiments of the present application also provide a machine-readable storage medium. Several computer instructions are stored on the machine-readable storage medium. When the computer instructions are executed by the processor, the present invention can be realized. Apply the image processing method disclosed in the above example.

Wherein, the above-mentioned machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device, which can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Radom Access Memory, random access memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard drive), solid state drive, any type of storage disk (such as CDs, DVDs, etc.), or similar storage media, or a combination thereof.

The above descriptions are only examples of the present application and are not intended to limit the present application. To those skilled in the art, various modifications and variations may be made to this application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application shall be included in the scope of the claims of this application.

Claims

An image processing method, characterized in that the method includes:

Divide all P frames in the picture group GOP sequence into at least two types of P frames, there are P frames among the at least two types of P frames that cannot be used as reference frames, and the at least two types of P frames There is a P frame that can be used as a reference frame;

If it is determined that the P frame within the GOP sequence needs to be extracted, then the P frame that cannot be used as a reference frame will be extracted, or if it is determined that the P frame within the GOP sequence needs to be extracted, then the P frame will be extracted. P frames that cannot be used as reference frames and P frames that can be used as reference frames are extracted.
The method according to claim 1, characterized in that the method includes:

All P frames in the GOP sequence are divided into first type P frames and second type P frames; wherein, the first type P frame cannot be used as a reference frame, and the second type P frame can be used as the first type P frame. A reference frame of the P frame type, and the P frame of the second type can be used as a reference frame of the P frame of the second type;

If it is determined that the P frames in the GOP sequence need to be extracted, the first type of P frames are extracted, or the first type of P frames and the second type of P frames are extracted. .
The method according to claim 2, characterized in that:

The method of dividing all P frames in the GOP sequence into first type P frames and second type P frames includes:

Based on the positional sorting relationship between the P frames in the GOP sequence, all P frames in the GOP sequence are divided into first type P frames and second type P frames; wherein, two adjacent second type P frames There are M1 P frames of the first type between P frames, and there are M2 P frames of the first type between the I frame of the GOP sequence and the first P frame of the second type; wherein, the M1 is a positive integer, so M2 is a positive integer.
The method according to claim 2 or 3, characterized in that,

For each P frame of the first type, if there is a P frame of the second type before the P frame of the first type, the most recent P frame of the second type before the P frame of the first type is used as the reference frame of the P frame of the first type. , if there is no P frame of the second type before the P frame of the first type, then the I frame of the GOP sequence is used as the reference frame of the P frame of the first type;

For each second type P frame, if there is a second type P frame before the second type P frame, the nearest second type P frame before the second type P frame is used as the reference frame of the second type P frame. , if there is no second type P frame before the second type P frame, then the I frame of the GOP sequence is used as the reference frame of the second type P frame.
The method according to claim 2, characterized in that:

If it is determined that the P frames in the GOP sequence need to be extracted, then extract the P frames of the first type, or extract the P frames of the first type and the P frames of the second type. Frame extraction, including:

If it is determined based on the business information that the first sampling ratio needs to be used to extract the P frames in the GOP sequence, then all the first type P frames or part of the first type P frames need to be extracted; if it is determined based on the business information that it is necessary to use The second sampling ratio is to extract frames from P frames in the GOP sequence, then extract frames from all P frames of the first type, and extract frames from all P frames of the second type or part of the P frames of the second type; or,

If it is determined based on the business information that the first sampling ratio needs to be used to extract frames from the P frames in the GOP sequence, then some of the first type P frames will be extracted; if it is determined based on the business information that the second sampling ratio needs to be used to extract the frames, To extract frames for P frames in the GOP sequence, all P frames of the first type are extracted. If it is determined based on the service information that it is necessary to use the third sampling ratio to extract frames for P frames in the GOP sequence, then all P frames of the first type are extracted. The P frames of the first type are extracted, and all the P frames of the second type or part of the P frames of the second type are extracted;

Among them, the second sampling proportion is greater than the first sampling proportion, and the third sampling proportion is greater than the second sampling proportion.
The method of claim 5, further comprising:

If the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoding end, then it is determined based on the resolution, the frame rate, and the decoding capability whether it is necessary to modify the GOP sequence. The P frames in the sequence are extracted; if the decoding capability does not support decoding the image with the resolution and the frame rate, it is determined to extract the P frames in the GOP sequence; if so, based on The resolution, the frame rate and the decoding capability determine the sampling ratio corresponding to the GOP sequence; or,

If the service information includes the network quality corresponding to the target network between the sending end and the receiving end, it is determined based on the network quality whether it is necessary to extract the P frames in the GOP sequence; wherein, if the network quality If the network quality is not the optimal level, it is determined to extract frames from the P frames in the GOP sequence; if so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality;

Wherein, the sampling ratio is the first sampling ratio or the second sampling ratio; or,

The sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio.
The method according to claim 1, characterized in that the method includes:

All P frames in the GOP sequence are divided into first type P frames, second type P frames and third type P frames; wherein the first type P frame cannot be used as a reference frame, and the second type P frame cannot be used as a reference frame. A P frame can only be used as a reference frame of the first type of P frame, and the third type of P frame can be used as a reference frame of the first type of P frame, and can be used as a reference frame of the second type of P frame, and can be used as a reference frame of the third type of P frame. Reference frame of P frame;

If it is determined that the P frames in the GOP sequence need to be extracted, the first type of P frames are extracted, or the first type of P frames and the second type of P frames are extracted. , or perform frame extraction on the first type P frame, the second type P frame and the third type P frame.
The method according to claim 7, characterized in that dividing all P frames in the GOP sequence into first type P frames, second type P frames and third type P frames includes:

Based on the positional ordering relationship between the P frames in the GOP sequence, divide all P frames in the GOP sequence into first type P frames, second type P frames and third type P frames;

Wherein, there are two P frames of the first type and one P frame of the second type between two adjacent P frames of the third type, and the P frame of the second type is located between the two P frames of the first type. ;

There are two first-type P frames and one second-type P frame between the I-frame of the GOP sequence and the first third-type P frame, and this second-type P frame is located between the two first-type P frames. between.
The method according to claim 7 or 8, characterized in that,

For each first type P frame, if the previous frame of the first type P frame is not the first type P frame, then the previous frame of the first type P frame is regarded as the first type P frame. The reference frame of the frame, if the previous frame of the first type P frame is the first type P frame, then the most recent non-first type P frame is used as the reference frame of the first type P frame;

For each P frame of the second type, if there is a P frame of the third type before the P frame of the second type, the nearest P frame of the third type before the P frame of the second type is used as the reference frame of the P frame of the second type. , if there is no third type P frame before the second type P frame, then the I frame of the GOP sequence is used as the reference frame of the second type P frame;

For each type 3 P frame, if there is a type 3 P frame before the type 3 P frame, then the latest type 3 P frame before the type 3 P frame is used as the reference frame of the type 3 P frame. , if there is no third type P frame before the third type P frame, then the I frame of the GOP sequence is used as the reference frame of the third type P frame.
The method according to claim 7, characterized in that:

If it is determined that the P frames in the GOP sequence need to be extracted, then extract the P frames of the first type, or extract the P frames of the first type and the P frames of the second type. Frame extraction, or frame extraction for the first type of P frame, the second type of P frame and the third type of P frame, including:

If it is determined based on the service information that the first sampling ratio needs to be used to extract frames from the P frames in the GOP sequence, then extract frames from all first-type P frames or part of the first-type P frames;

If it is determined based on the service information that the second sampling ratio needs to be used to extract the P frames in the GOP sequence, then all the first type P frames will be extracted, and all the second type P frames or part of the second type P frames will be extracted. The frame is extracted; wherein the second sampling ratio is greater than the first sampling ratio;

If it is determined based on the service information that the third sampling ratio needs to be used to extract frames from the P frames in the GOP sequence, then extract frames from all P frames of the first type, extract frames from all P frames of the second type, and extract frames from all P frames of the second type. The third type P frame or part of the third type P frame is extracted; the third sampling ratio is greater than the second sampling ratio.
The method of claim 10, further comprising:

If the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoding end, then it is determined based on the resolution, the frame rate, and the decoding capability whether it is necessary to modify the GOP sequence. The P frames in the sequence are extracted; if the decoding capability does not support decoding the image with the resolution and the frame rate, it is determined to extract the P frames in the GOP sequence; if so, based on The resolution, the frame rate and the decoding capability determine the sampling ratio corresponding to the GOP sequence, and the sampling ratio is the first sampling ratio, or the second sampling ratio, or the third sampling ratio; or,

If the service information includes the network quality corresponding to the target network between the sending end and the receiving end, it is determined based on the network quality whether it is necessary to extract the P frames in the GOP sequence; wherein, if the network quality If the network quality is not the optimal level, it is determined to extract frames from the P frames in the GOP sequence; if so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality, and the sampling ratio is the first sampling ratio. , or the second sampling ratio, or the third sampling ratio.
The method according to claim 2 or 7, characterized in that, the method further includes:

When the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence of the image sequence set, the I frame of the first GOP sequence is regarded as the first of the GOP sequence. The reference frame of the frame.
An image processing device, characterized in that the device includes:

A dividing module, configured to divide all P frames in the picture group GOP sequence into at least two types of P frames. Among the at least two types of P frames, there are P frames that cannot be used as reference frames, and the at least two types of P frames Among the types of P frames, there is a P frame that can be used as a reference frame;

A processing module, configured to extract frames from P frames that cannot be used as reference frames if it is determined that the P frames in the GOP sequence need to be extracted, or if it is determined that the P frames in the GOP sequence need to be extracted. For frame extraction, frame extraction is performed on P frames that cannot be used as reference frames and P frames that can be used as reference frames.
The device according to claim 13, characterized in that:

The dividing module is specifically used to divide all P frames in the GOP sequence into first type P frames and second type P frames; wherein, the first type P frame cannot be used as a reference frame, and the second type P frame cannot be used as a reference frame. The P frame can be used as the reference frame of the first type of P frame, and the second type of P frame can be used as the reference frame of the second type of P frame; the processing module is specifically used to determine that it is necessary to When the P frame is extracted, the first type of P frame is extracted, or the first type of P frame and the second type of P frame are extracted.
The device according to claim 14, characterized in that:

When the dividing module divides all P frames in the GOP sequence into first type P frames and second type P frames, it is specifically used to: based on the position ordering relationship between the P frames in the GOP sequence, divide the All P frames in the GOP sequence are divided into first type P frames and second type P frames; wherein, there are M1 first type P frames between two adjacent second type P frames, and the I frame of the GOP sequence There are M2 type-1 P frames between it and the first type-2 P frame; M1 and M2 are both positive integers.
The device according to claim 14 or 15, characterized in that,

For each P frame of the first type, if there is a P frame of the second type before the P frame of the first type, the most recent P frame of the second type before the P frame of the first type is used as the reference frame of the P frame of the first type. , if there is no P frame of the second type before the P frame of the first type, then the I frame of the GOP sequence is used as the reference frame of the P frame of the first type; for each P frame of the second type, if the P frame of the second type If there is a P frame of the second type before the frame, the most recent P frame of the second type before the P frame of the second type is used as the reference frame of the P frame of the second type. If there is no P frame of the second type before the P frame of the second type. frame, the I frame of the GOP sequence is used as the reference frame of the second type of P frame.
The device according to claim 14, characterized in that:

When the processing module determines that it is necessary to extract frames from P frames in the GOP sequence, extract frames from the first type of P frames, or extract frames from the first type of P frames and the second type of P frames. When performing frame extraction, it is specifically used: if it is determined based on the service information that the first sampling ratio needs to be used to extract the P frames in the GOP sequence, then extract the frames for all the first type P frames or part of the first type P frames. ; If it is determined based on the service information that the second sampling ratio needs to be used to extract the P frames in the GOP sequence, then extract the frames for all the first type P frames, and extract all the second type P frames or part of the second type. P frames are extracted; or, if it is determined based on the service information that it is necessary to use the first sampling ratio to extract the P frames in the GOP sequence, then some of the first type P frames are extracted; if it is determined based on the service information that it is necessary Using the second sampling ratio to extract frames from the P frames in the GOP sequence, all P frames of the first type are extracted; if it is determined based on the service information that it is necessary to use the third sampling ratio to extract the P frames in the GOP sequence To perform frame extraction, all P-frames of the first type are extracted, and all P-frames of the second type or part of the P-frames of the second type are extracted; among them, the second sampling ratio is greater than the first sampling ratio, and the third sampling ratio The proportion is greater than the second sampling proportion.
The device according to claim 17, characterized in that:

The processing module is also configured to: if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence and the decoding capability corresponding to the decoding end, based on the resolution, the frame rate and the decoding The ability determines whether it is necessary to perform frame extraction on the P frames in the GOP sequence; wherein, if the decoding capability does not support decoding images of the resolution and the frame rate, determine whether the P frames in the GOP sequence need to be decoded. Perform frame extraction; if so, determine the sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate and the decoding capability; or, if the service information includes the target network between the sending end and the receiving end According to the corresponding network quality, it is determined based on the network quality whether it is necessary to extract frames from the P frames in the GOP sequence; wherein, if the network quality is not the optimal level of network quality, it is determined that the P frames in the GOP sequence need to be extracted. P frames are extracted; if so, determine the sampling ratio corresponding to the GOP sequence based on the network quality; the sampling ratio is the first sampling ratio, or the second sampling ratio; or, the sampling ratio is the first sampling ratio Sampling proportion, or second sampling proportion, or third sampling proportion.
The device according to claim 12, characterized in that:

The division module is specifically used to divide all P frames in the GOP sequence into first type P frames, second type P frames and third type P frames; the first type P frame cannot be used as a reference frame. , the second type of P frame can only be used as a reference frame of the first type of P frame, the third type of P frame can be used as a reference frame of the first type of P frame, can be used as a reference frame of the second type of P frame, and Can be used as a reference frame of the third type of P frame; the processing module is specifically configured to extract frames of the first type of P frame if it is determined that the P frame in the GOP sequence needs to be extracted, or, The P frames of the first type and the P frames of the second type are extracted, or the P frames of the first type, the P frames of the second type and the P frames of the third type are extracted.
The device according to claim 19, characterized in that:

When the division module divides all P frames in the GOP sequence into first type P frames, second type P frames and third type P frames, it is specifically used to: based on the positional sorting relationship between each P frame in the GOP sequence. , all P frames in the GOP sequence are divided into first type P frames, second type P frames and third type P frames; wherein, for all P frames in the GOP sequence, two adjacent third type P frames There are two first type P frames and one second type P frame between the type P frames, and the one second type P frame is located between the two first type P frames; the I frame of the GOP sequence There are two P frames of the first type and one P frame of the second type between the first P frame of the third type, and the P frame of the second type is located between the two P frames of the first type.
The device according to claim 19 or 20, characterized in that,

For each first type P frame, if the previous frame of the first type P frame is not the first type P frame, then the previous frame of the first type P frame is used as the reference frame of the first type P frame, If the previous frame of the first type P frame is a first type P frame, then the most recent non-first type P frame is used as the reference frame of the first type P frame; for each second type P frame, if There is a P frame of the third type before the P frame of the second type, then the P frame of the third type most recent before the P frame of the second type is used as the reference frame of the P frame of the second type. If there is no P frame of the second type before it, If there is a P frame of the third type, the I frame of the GOP sequence is used as the reference frame of the P frame of the second type; for each P frame of the third type, if there is a P frame of the third type before the P frame of the third type , then the latest P frame of the third type before the P frame of the third type is used as the reference frame of the P frame of the third type. If there is no P frame of the third type before the P frame of the third type, then the GOP sequence The I frame serves as the reference frame of the third type of P frame.
The device according to claim 19, characterized in that:

When the processing module determines that it is necessary to extract frames from P frames in the GOP sequence, extract frames from the first type of P frames, or extract frames from the first type of P frames and the second type of P frames. Performing frame extraction, or performing frame extraction on the first type P frame, the second type P frame and the third type P frame is specifically used for: if it is determined based on the service information that the first sampling ratio needs to be used To extract frames for P frames in the GOP sequence, all first-type P frames or part of the first-type P frames are extracted; if it is determined based on the service information that a second sampling ratio needs to be used to extract frames for P frames in the GOP sequence , then extract frames from all first-type P frames, and extract frames from all second-type P frames or part of second-type P frames; the second sampling ratio is greater than the first sampling ratio; if it is determined based on the service information that it is necessary to use the second The three-sampling ratio is used to extract frames from P frames in the GOP sequence. Then all P frames of the first type are extracted, all P frames of the second type are extracted, and all P frames of the third type or part of the third type are extracted. The P frame is extracted; the third sampling ratio is greater than the second sampling ratio.
The device according to claim 22, characterized in that:

The processing module is also configured to: if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence and the decoding capability corresponding to the decoding end, based on the resolution, the frame rate and the decoding The ability determines whether it is necessary to perform frame extraction on the P frames in the GOP sequence; wherein, if the decoding capability does not support decoding images of the resolution and the frame rate, determine whether the P frames in the GOP sequence need to be decoded. Perform frame extraction; if so, determine the sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate and the decoding capability, and the sampling ratio is the first sampling ratio, or the second sampling ratio, or the third sampling ratio. Three sampling ratios; or, if the service information includes the network quality corresponding to the target network between the encoding end and the decoding end, determine whether the P frames in the GOP sequence need to be extracted based on the network quality; where , if the network quality is not the optimal level of network quality, it is determined to extract frames from the P frames in the GOP sequence; if so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality, and the sampling The proportion is the first sampling proportion, or the second sampling proportion, or the third sampling proportion.
The device according to claim 14 or 19, characterized in that,

Wherein, when the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence of the image sequence set, the dividing module is also used to divide the first GOP sequence into The I frame serves as the reference frame for the first frame of the GOP sequence.
An image processing device, characterized in that it includes: a processor and a machine-readable storage medium, the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine The instructions are executable to implement the method steps described in any one of claims 1-12.