WO2022252567A1 - Method and device for determining priority order of video encoding and decoding on basis of correlation comparison - Google Patents

Abstract

Disclosed are a method and a device for determining the priority order of video encoding and decoding on the basis of correlation comparison, the method comprising: determining a corresponding correlation between each encoding block and each video analysis task; determining, on the basis of the correlation, a preferentially encoded encoding block; and encoding and/or transmitting the encoding block. The present invention determines, according to the requirements of a video analysis task, the correlation between sub-blocks and the video analysis task, determines encoding order on the basis of correlation, the encoding order and the transmission order of encoded data thereof preferentially processing important data according to the requirements of video analysis, performs video encoding, and converts the encoded data into a video code stream, which can ensure that the accuracy of video analysis results is within an acceptable range, and at the same time ensures that the data compression rate is optimal.

Description

Method and device for determining priority order of video encoding and decoding based on correlation degree comparison technical field

The present invention relates to the technical field of multimedia processing, in particular to a method and a device for determining video encoding and decoding priorities based on correlation degree comparison.

Background technique

Video compression based on video analysis tasks is somewhat different from other video compression scenarios. In scenarios such as film and television video compression, the quality of compressed video is mainly judged by people subjectively, and the impact of distortion caused by video compression is mainly the impact of human subjective perception. Video analysis is generally done automatically by a computer, and it is required that the analysis result of the compressed video should be consistent with the analysis result of the uncompressed video, otherwise it will cause erroneous analysis results. In order to take into account greater compression efficiency and accurate video analysis, a public document (CN111901594A) has proposed the concept of "just identifiable distortion". The just identifiable distortion is the distortion threshold of the coded image that causes the visual analysis accuracy to be less than the preset accuracy, and the coding parameters are determined according to the just identifiable distortion of the image, and the image is coded according to the coding parameters to obtain a corresponding coded image, which can Under the same bit rate, the performance of visual analysis tasks on encoded images can be improved, which can save more bit rates. Existing publications disclose the use of just identifiable distortion prediction models to determine encoding parameters, but do not disclose how to transmit the encoding parameters and their encoding blocks to overcome the problem of limited transmission channels.

It is easy to infer that determining the encoding parameters according to the just identifiable distortion of the image can concentrate most of the bit rate on the encoding blocks that affect the visual analysis task, and they have a higher impact on the video analysis results than other encoding blocks. In a transmission environment such as limited bandwidth, it is very important to prioritize the transmission of more important coded block data that affects visual analysis tasks. However, since the determination of the coding order in the standard is only related to video content, it has nothing to do with video analysis. Therefore, The encoding order and the transmission order of the encoded data cannot prioritize important data according to the needs of video analysis.

Contents of the invention

The purpose of the present invention is to provide a method and device for determining the priority order of video encoding and decoding based on the correlation degree comparison, which is used to solve the above-mentioned technology that the encoding order and the transmission order of encoded data cannot prioritize important data according to the needs of video analysis question.

A method for determining video encoding priority based on correlation comparison, comprising the following steps:

Determine the degree of correlation between each encoding block and each video analysis task;

determining a coding block for priority coding based on the correlation;

The encoded block is encoded and/or transmitted.

The above-mentioned method for determining video encoding priority based on correlation comparison, said determining the coding block for priority encoding includes:

Preset a reference threshold, and compare whether the correlation between the coding block and each video analysis task exceeds the reference threshold;

Selecting coding blocks whose correlation degrees exceed the reference threshold are preferentially coded.

In the above-mentioned method for determining video coding priority based on correlation degree comparison, the determination of the coding block for priority coding also includes the coding sequence formed by horizontal scanning, vertical scanning and diagonal scanning.

In the above-mentioned method for determining video coding priority based on correlation comparison, the coding block represents a video data block that is subjected to intra-frame prediction coding.

In the above-mentioned method for determining video coding priority based on correlation comparison, when there are multiple video analysis tasks, the coding block has a corresponding correlation for each video analysis task.

In the above-mentioned method for determining video encoding priority based on correlation degree comparison, the determination of the correlation degree corresponding to each coding block includes: performing numerical modification on each coding block to determine the correlation degree between each coding block and a video analysis task.

A method for determining priority order of video decoding based on correlation comparison, comprising the following steps:

Converting the encoding sequence indication information and video data into a video code stream;

The video code stream is decoded for the decoder to decode to obtain corresponding image frames.

A device for determining video encoding priority based on correlation comparison, comprising:

A determination module is used to modify the numerical value of each coding block to determine the correlation between each coding block and the video analysis task;

A comparison module, configured to preset a reference threshold, and compare the correlation corresponding to each coding block with the threshold;

An encoding module, configured to select encoding blocks whose correlation degrees exceed the reference threshold and encode them preferentially.

The above-mentioned device for determining video coding priority based on correlation comparison further includes a block module for dividing the entire image frame into one or more coding blocks, and each coding block is a rectangular area containing an integer number of CTUs.

A device for determining video decoding priority based on correlation comparison, comprising:

A conversion module, configured to convert the encoding sequence indication information and video data into a video code stream;

The decoding module is used to decode the video code stream for the decoder to decode to obtain corresponding image frames.

An electronic device, comprising: a memory, a processor, and a computer program stored on the memory and operable on the processor, and executed when the processor runs the computer program to achieve any of the above A method for prioritizing video coding based on correlation comparison.

A computer-readable storage medium, on which computer-readable instructions are stored, and the computer-readable instructions can be executed by a processor to implement the method for determining video encoding priority based on correlation comparison described in any one of the above.

The above technical solution has the following advantages or beneficial effects:

In the present invention, the method and device for determining the priority order of video encoding and decoding based on the correlation degree comparison determine the correlation degree between each sub-block and the video analysis task according to the video analysis task, determine the encoding sequence based on the correlation degree, the encoding sequence and the sequence of the encoded data The transmission sequence prioritizes important data according to the needs of video analysis, performs video encoding, and converts the encoded data into video code streams, which can ensure the accuracy of video analysis results within an acceptable range, and at the same time ensure the best data compression rate .

Description of drawings

FIG. 1 is a schematic diagram of a method for determining video coding priority based on correlation comparison in the present invention;

Fig. 2 is a schematic diagram of the horizontal scanning coding sequence in the embodiment of the present invention.

Detailed ways

In order to make the above objects, features and advantages of the present invention more comprehensible, specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "upper", "lower", "front" and "rear" are based on the orientation or positional relationship shown in the drawings, and are only for convenience The present invention is described and simplified descriptions do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and thus should not be construed as limiting the present invention.

The terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features.

As shown in FIG. 1 and FIG. 2, a method for determining video coding priority based on correlation comparison includes the following steps:

Block the entire image frame into several coding blocks of the same shape;

Determine the degree of correlation between each encoding block and each video analysis task;

determining a coding block for priority coding based on the correlation;

The encoded block is encoded and/or transmitted.

The encoding blocks for determining priority encoding include:

Preset a reference threshold, and compare whether the correlation between the coding block and each video analysis task exceeds the reference threshold;

Selecting coding blocks whose correlation degrees exceed the reference threshold are preferentially coded.

The coding blocks for determining priority coding also include coding sequences formed by horizontal scanning, vertical scanning and diagonal scanning.

Preset a reference threshold, compare whether the correlation degree corresponding to the coding block exceeds the reference threshold, preset a threshold, compare the correlation degree of each coding block with the threshold, and the ones higher than the threshold are considered to be closely related to a specific video analysis task relevant, has the highest encoding priority;

In the coding order, the coding blocks whose correlation degrees exceed the reference threshold are selected for priority coding, and in the coding sequence formed by horizontal scanning, vertical scanning, and diagonal scanning, several coding blocks with correlation degrees exceeding a certain threshold are selected for coding priority.

For example, the block scan order of H.266/VVC includes diagonal scan, horizontal scan and vertical scan order.

As an example, assume that the correlation of each coding block is as follows:

coding block	relativity
1	0.5
2	0.85
3	0.5
4	0.3
5	0.1
6	2
7	0.75
8	0.8
9	0.2
10	0.2
11	0.1
12	0.3
13	0.2
14	0.1
15	0
16	0

Table 1

If the correlation threshold is set to 0.6, the coding blocks that are preferentially coded are 2, 7, and 8. The horizontal scanning coding order is still maintained in the priority coding block and other coding blocks. The complete coding sequence is: 2, 7, 8, 1, 3, 4, 5, 6, 9, 10, 11, 12, 1, 14, 15, 16.

In this embodiment, video analysis tasks include image classification, object detection, and pose estimation. For an original image Pic0, distorted images Pic1, Pic2, Pic3, . The increase of the degree of distortion will reduce the accuracy of the visual analysis results. Among them, the encoding parameters that have a greater impact on the distortion degree include quantization parameters (QP), target code rates, and the like. When the degree of distortion increases to a certain threshold, the prediction of the image by the visual analysis model will completely fail. Therefore, in Pic1, Pic2, Pic3, . Therefore, by trying different sets of encoding parameters and then analyzing the corresponding distorted image with a specific visual analysis process, it is judged whether the accurate result is obtained. Theoretically speaking, a set of optimal parameter combinations can be obtained, just enough to ensure that the accuracy of the video analysis results is within an acceptable range, and at the same time ensure the best data compression rate and other beneficial effects.

Determine the degree of correlation between the sub-blocks and the video analysis task according to the needs of the video analysis task, determine the coding order based on at least part of the correlation, perform video coding, and convert the coded data into a video code stream.

Further, in a preferred embodiment of the method for determining video coding priority based on correlation comparison in the present invention, the coding block represents a video data block that implements intra-frame prediction coding.

Wherein, in one embodiment, the block method of VVC and HEVC is followed. A frame of image is divided into one or more coded block rows and coded block columns, and each coded block is a rectangular area containing an integer number of CTUs. According to the HEVC standard, CTUs are divided into CUs according to the quadtree division method, and each CU can be divided into PUs and TUs. In VVC, the concepts of CU, PU and TU will no longer be distinguished. Either way, in the present invention, a coded block or sub-block or sub-block is used to represent a block of video data subjected to intra-frame predictive coding.

Further, in a preferred embodiment of the method for determining video encoding priorities based on correlation comparison in the present invention, when there are multiple video analysis tasks, each of the encoding blocks has one corresponding correlation.

Further, in a preferred embodiment of the method for determining video encoding priority based on correlation degree comparison in the present invention, the determination of the correlation degree corresponding to each coding block includes: performing numerical modification on each coding block to determine each coding block The block is related to the video analysis task.

In one embodiment, one coding block or a combination of multiple adjacent coding blocks (for example, 2, 4, 8) are sequentially selected, and its pixel values (luminance and/or chrominance values) are numerically reversed (binary representation 0 becomes 1, 1 becomes 0), while the values of other coding blocks remain unchanged. Then carry out video analysis based on the modified image frame, observe the change of analysis accuracy rate (0-100%), and calculate correlation degree=(1-analysis accuracy rate). This enables the determination of how relevant each coding block or combination of adjacent coding blocks is to a specific video analysis task. Apparently, a high degree of correlation has a high degree of importance. If multiple video analysis tasks exist at the same time, a coding block has a corresponding correlation degree for each video analysis task.

A method for determining priority order of video decoding based on correlation comparison, comprising the following steps:

Converting the encoding sequence indication information and video data into a video code stream;

The video code stream is decoded for the decoder to decode to obtain corresponding image frames.

Compression coding is performed on all codes in the above sequence, and all possible video compression technologies, existing and future, can be utilized. In an embodiment, the framed data is compressed using an intra-frame coding technique, that is, at least including intra-frame prediction, DCT transformation, quantization, and entropy coding processes.

A device for determining video encoding priority based on correlation comparison, comprising:

The block module is used to divide the entire image frame into one or more coding blocks, and each coding block is a rectangular area containing an integer number of CTUs;

A determining module, configured to modify the values of each coding block to determine the correlation between each coding block and the video analysis task;

A comparison module, configured to preset a reference threshold, and compare the correlation corresponding to each coding block with the threshold;

An encoding module, configured to select encoding blocks whose correlation degrees exceed the reference threshold and encode them preferentially.

A device for determining video decoding priority based on correlation comparison, comprising:

A conversion module, configured to convert the encoding sequence indication information and video data into a video code stream;

The decoding module is used to decode the video code stream for the decoder to decode to obtain corresponding image frames.

An electronic device, comprising: a memory, a processor, and a computer program stored on the memory and operable on the processor, and executed when the processor runs the computer program to achieve any of the above A method for prioritizing video encoding based on correlation comparison.

Wherein, the memory is connected in communication with the one or more processors, the memory stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors , so that the one or more processors execute the methods in the foregoing embodiments of the present application.

Specifically, the processor and the memory may be connected through a bus or in other ways, taking connection through a bus as an example. The processor may be a central processing unit (Central Processing Unit, CPU). The processor can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), field-programmable gate array (Field-Programmable Gate Array, FPGA) or other Chips such as programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above-mentioned types of chips.

As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as the cascaded progressive network in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running non-transitory software programs/instructions and functional modules stored in the memory.

The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created by the processor, and the like. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include memory located remotely from the processor, such remote memory may be connected to the processor via a network (eg, via a communication interface). Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

An embodiment of the present application further provides a computer-readable storage medium, where computer-executable instructions are stored, and the computer-executable instructions are executed to perform the methods in the foregoing embodiments of the present application.

The foregoing computer-readable storage media include physically volatile and non-volatile, removable and non-removable media implemented in any manner or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. medium. Computer-readable storage media specifically include, but are not limited to, U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), erasable programmable read-only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash or other solid-state memory technology, CD-ROM, Digital Versatile Disk (DVD), HD-DVD, Blue-Ray or other optical storage device, tape, disk storage or other magnetic storage device, or any other medium that can be used to store the desired information and that can be accessed by a computer.

Although the subject matter described herein is presented in the general context of being executed in connection with the execution of operating systems and application programs on a computer system, those skilled in the art will recognize that other types of program modules can also be used to execute other programs. accomplish. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Those skilled in the art will appreciate that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. , can also be used in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Those skilled in the art can appreciate that the units and method steps of the examples described in conjunction with the embodiments of the present application herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or the part that contributes to the prior art or the technical solution, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.

In the present invention, the method and device for determining the priority order of video encoding and decoding based on the correlation degree comparison determine the correlation degree between each sub-block and the video analysis task according to the video analysis task, determine the encoding sequence based on the correlation degree, the encoding sequence and the sequence of the encoded data The transmission sequence prioritizes important data according to the needs of video analysis, performs video encoding, and converts the encoded data into video code streams, which can ensure the accuracy of video analysis results within an acceptable range, and at the same time ensure the best data compression rate .

The above are only preferred embodiments of the present invention, and are not intended to limit the implementation and protection scope of the present invention. For those skilled in the art, they should be able to realize the equivalent replacement and The solutions obtained by obvious changes shall all be included in the protection scope of the present invention.

Claims (10)

1. A method for determining video encoding priority based on correlation comparison, characterized in that it comprises the following steps:

   Determine the degree of correlation between each encoding block and each video analysis task;

   determining a coding block for priority coding based on the correlation;

   The encoded block is encoded and/or transmitted.
2. The method for determining video coding priority based on correlation comparison according to claim 1, wherein said determining a coding block for priority coding comprises:

   Preset a reference threshold, and compare whether the correlation between the coding block and each video analysis task exceeds the reference threshold;

   Selecting coding blocks whose correlation degrees exceed the reference threshold are preferentially coded.
3. The method for determining video coding priority based on correlation comparison according to claim 1, wherein said determining the coding block for priority coding further includes the coding sequence formed by horizontal scanning, vertical scanning and diagonal scanning.
4. The method for determining video coding priority based on correlation comparison according to claim 1, wherein the coding block represents a video data block for implementing intra-frame prediction coding.
5. The method for determining video encoding priority based on correlation comparison according to claim 1, wherein when there are multiple video analysis tasks, each of the encoding blocks has a corresponding correlation for each video analysis task. Spend.
6. The method for determining video coding priority based on correlation degree comparison according to claim 1, wherein said determining the correlation degree corresponding to each coding block comprises: modifying the values of each coding block to determine the correlation between each coding block and the video coding block. Analyze task dependencies.
7. A method for determining video decoding priority order based on correlation comparison, is characterized in that, comprises the following steps:

   Converting the encoding sequence indication information and video data into a video code stream;

   The video code stream is decoded for the decoder to decode to obtain corresponding image frames.
8. A device for determining video coding priority based on correlation comparison, characterized in that it includes:

   A determination module is used to modify the numerical value of each coding block to determine the correlation between each coding block and the video analysis task;

   A comparison module, configured to preset a reference threshold, and compare the correlation corresponding to each coding block with the threshold;

   An encoding module, configured to select encoding blocks whose correlation degrees exceed the reference threshold and encode them preferentially.
9. The device for determining video coding priority based on correlation comparison according to claim 8, further comprising a block module, which is used to divide the entire image frame into one or more coding blocks, and each coding block is A rectangular area contains an integer number of CTUs.
10. A device for determining video decoding priority based on correlation comparison, characterized in that it includes:

    A conversion module, configured to convert the encoding sequence indication information and video data into a video code stream;

    The decoding module is used to decode the video code stream for the decoder to decode to obtain corresponding image frames.

Images