CN115278241A

CN115278241A - Image coding method and device

Info

Publication number: CN115278241A
Application number: CN202110475373.4A
Authority: CN
Inventors: 陈志波; 解广琪; 李鑫; 吴仕林; 刘森; 林永兵
Original assignee: University of Science and Technology of China USTC; Huawei Technologies Co Ltd
Current assignee: University of Science and Technology of China USTC; Huawei Technologies Co Ltd
Priority date: 2021-04-29
Filing date: 2021-04-29
Publication date: 2022-11-01

Abstract

The application provides an image coding method and device, which can be applied to intelligent automobiles, internet automobiles and automatic driving automobiles. Wherein, the method comprises the following steps: reversible color transformation and discrete wavelet transformation are carried out on a region to be coded in a frame to be coded to obtain a transformation coefficient; determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the reference region, wherein the code rate control parameter comprises a coarse granularity parameter and a fine granularity parameter; searching the code rate control parameter of the area to be coded according to the initial code rate control parameter; quantizing the transform coefficient according to the searched code rate control parameter; and carrying out Columbus encoding and zero run encoding on the quantized transform coefficients. The technical scheme of the application can improve the speed of video coding and transmission.

Description

Image coding method and device

Technical Field

The present application relates to the field of image encoding and decoding technology, and more particularly, to an image encoding method and apparatus.

Background

With the rapid development of multimedia technologies such as image/video and related industries, various new applications and requirements emerge endlessly. Under the application scenes of unmanned vehicles, virtual reality, unmanned aerial vehicle reconnaissance, live game, broadcast streaming transmission and the like, video data acquired by acquisition equipment needs to be streamed to a receiving end instead of being cached on the acquisition equipment. This application scenario has two core requirements, namely low latency and low time complexity. The low delay means that the buffer space consumed for processing the acquired data is as small as possible; low time complexity means that end-to-end codec time consumption is as small as possible.

Although the existing video coding technologies such as High Efficiency Video Coding (HEVC) and multi-function video coding (VVC) can achieve a doubling of coding performance, their complex intra/inter prediction modes will generate a larger delay and a higher complexity of coding and decoding time, thereby affecting the speed of video coding and transmission.

Therefore, how to increase the speed of video encoding and transmission is an urgent technical problem to be solved.

Disclosure of Invention

The application provides an image coding method and device, which can improve the speed of video coding and transmission.

In a first aspect, an image encoding method is provided, and the method includes: reversible color transformation and discrete wavelet transformation are carried out on a region to be coded in a frame to be coded to obtain a transformation coefficient; determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the reference region, wherein the code rate control parameter comprises a coarse granularity parameter and a fine granularity parameter; searching the code rate control parameter of the area to be coded according to the initial code rate control parameter; quantizing the transform coefficient according to the searched code rate control parameter; and performing Golomb coding and zero run coding on the quantized transform coefficient.

It should be understood that the frame to be encoded is an image to be encoded or an image frame to be encoded, and this application does not distinguish between these.

Optionally, the image coding in the present application may be intra-frame coding of a frame to be coded, where the frame to be coded may be an image frame existing alone or an image frame in a video stream, and at this time, the frame to be coded may be recorded as an intra-frame parameter prediction frame; the image coding may also be inter-frame coding of a frame to be coded, where the frame to be coded is an image frame in a video stream, and at this time, the frame to be coded may be marked as an inter-frame parameter prediction frame, which is not limited in this application. Based on this, the image encoding in the present application may include intra-frame encoding; and/or include inter-frame coding. It should be understood that, when encoding a video stream, the intra-frame parameter prediction frame and the inter-frame parameter prediction frame in the video stream may be intra-frame encoded and inter-frame encoded respectively by using the image encoding method provided in the present application.

It should be understood that the initial rate control parameters include an initial coarse-granularity parameter and an initial fine-granularity parameter.

It should be understood that the coarse-granularity parameter and the fine-granularity parameter are rate control parameters defined in the JPEG-XS standard of the low-latency lightweight image coding system. The region is a part of a spatial region of an image defined by the JPEG-XS standard, specifically, an integer number (e.g., 4) of image lines in a frame of image. The reversible color transform and the discrete wavelet transform are transform coding methods used in the JPEG-XS standard, in which the reversible color transform generates almost no delay and the discrete wavelet transform generates a low delay. It should be understood that the discrete wavelet transform used in the JPEG-XS standard refers specifically to an asymmetric discrete wavelet transform, the delay of which is approximately 32 image lines. It should be understood that golomb encoding and zero-run encoding are entropy encoding methods used in the JPEG-XS standard, which also result in lower delays. Therefore, the image encoding method provided by the embodiment of the application can also be regarded as being realized based on a low-delay lightweight image encoding system JPEG-XS.

In the image coding method provided by the embodiment of the application, a region to be coded in a frame to be coded is transformed through reversible color transformation and discrete wavelet transformation to obtain a transform coefficient, an initial code rate control parameter of the region to be coded is determined according to a code rate control parameter of a reference region, then the code rate control parameter of the region to be coded is searched according to the initial code rate control parameter, finally the transform coefficient is quantized according to the searched code rate control parameter, and then Golomb coding and zero run length coding are carried out on the quantized transform coefficient. On one hand, in the image coding process, a low-delay transformation coding method and an entropy coding method are adopted, so that low-delay image coding can be realized; on the other hand, the initial code rate control parameter of the region to be coded is determined according to the code rate control parameter of the reference region, so that when the code rate control parameter of the region to be coded is searched, the search can be carried out based on the better initial code rate control parameter, and the time complexity of image coding can be reduced. Based on this, the image coding method provided by the embodiment of the application can also realize video coding with low delay and low time complexity; further, the speed of video encoding and transmission can be increased. It should be understood that video is composed of a plurality of image frames including intra-frame parameter prediction frames and inter-frame parameter prediction frames, and video encoding includes intra-frame encoding and inter-frame encoding. .

With reference to the first aspect, in some implementations of the first aspect, if the frame to be encoded is an intra-parameter prediction frame, the reference region includes a first reference region, and the first reference region is a first target region already encoded in the frame to be encoded; the determining the initial rate control parameter of the region to be encoded according to the rate control parameter of the reference region includes: and determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the first reference region.

In the embodiment of the application, when predicting the code rate control parameter of the region to be coded in the intra-parameter prediction frame, the initial code rate control parameter of the region to be coded can be determined according to the code rate control parameter of the first target region coded in the frame to be coded, that is, the better initial code rate control parameter is provided for the region to be coded by combining the content correlation among different regions in the same frame of image, so that the optimal code rate control parameter value of the region to be coded can be searched through fewer search steps, and the time complexity of image coding can be reduced.

With reference to the first aspect, in certain implementations of the first aspect, the first target region is a previous region vertically adjacent to the region to be encoded.

In the application embodiment, the first target region may be a previous region vertically adjacent to the region to be encoded, so that when predicting the code rate control parameter of the region to be encoded in the intra-frame parameter prediction frame, the initial code rate control parameter of the region to be encoded may be determined according to the code rate control parameter of the previous region vertically adjacent to the region to be encoded in the frame to be encoded, that is, in combination with content correlation between adjacent regions in the same frame of image, a better initial code rate control parameter is provided for the region to be encoded, so that the optimal code rate control parameter value of the region to be encoded can be searched through fewer search steps, and thus time complexity of image encoding can be reduced.

With reference to the first aspect, in some implementations of the first aspect, if the frame to be encoded is an inter-frame parameter prediction frame, the reference region includes a second reference region and a third reference region, the second reference region is a second target region already encoded in the frame to be encoded, and the third reference region is a third target region already encoded in the target frame; the determining the initial rate control parameter of the region to be encoded according to the rate control parameter of the reference region includes: and determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the second reference region and the code rate control parameter of the third reference region.

In the embodiment of the application, when the code rate control parameter of the region to be coded in the inter-parameter prediction frame is predicted, the initial code rate control parameter of the region to be coded can be determined according to the code rate control parameter of the coded second target region in the frame to be coded and the code rate control parameter of the third target region in the coded target frame, that is, the better initial code rate control parameter is provided for the region to be coded by combining the content correlation between different regions in the same frame image and the content correlation between different frame images, so that the optimal code rate control parameter value of the region to be coded can be searched by fewer search steps, and the time complexity of image coding can be reduced.

With reference to the first aspect, in some implementations of the first aspect, the determining the initial rate control parameter of the region to be encoded according to the rate control parameter of the second reference region and the rate control parameter of the third reference region includes: determining the content complexity of the second reference area, the third reference area and the area to be coded; determining a first weight coefficient of the second reference area and a second weight coefficient of the third reference area according to the content complexity of the second reference area, the third reference area and the area to be coded; and determining an initial code rate control parameter of the region to be coded according to the first weight coefficient, the second weight coefficient, the code rate control parameter of the second reference region and the code rate control parameter of the third reference region.

It should be understood that the rate control parameters for regions with closer content complexity have higher reference values. Therefore, in the embodiment of the present application, a first weight coefficient of a second reference region and a second weight coefficient of a third reference region are determined according to content complexity of the second reference region, the third reference region and a region to be encoded, and then an initial code rate control parameter of the region to be encoded is determined according to the first weight coefficient, the second weight coefficient, a code rate control parameter of the second reference region and a code rate control parameter of the third reference region. Therefore, a better and more accurate initial code rate control parameter can be provided for the code rate control parameter of the region to be coded, and the time complexity of image coding is further reduced.

With reference to the first aspect, in certain implementations of the first aspect, the second target region is a previous region vertically adjacent to the region to be encoded; and/or, the target frame is a previous frame of the frame to be coded; and/or the third target area is an area corresponding to the area to be coded in the target frame.

In this embodiment of the present application, the second target region may be a previous region vertically adjacent to a region to be encoded, the target frame may be a previous frame of a frame to be encoded, and the third target region may be a region corresponding to the region to be encoded in the target frame, so that when predicting a rate control parameter of the region to be encoded in an inter-parameter prediction frame, an initial rate control parameter of the region to be encoded may be determined according to the rate control parameter of the previous region vertically adjacent to the region to be encoded in the frame to be encoded and the rate control parameter of the region corresponding to the region to be encoded in the previous frame of the frame to be encoded, that is, a better initial rate control parameter is provided for the region to be encoded by combining content correlation between adjacent regions in the same frame of image and content correlation between adjacent frame of images, so that an optimal rate control parameter value of the region to be encoded can be searched through fewer search steps, thereby reducing time complexity of image encoding.

With reference to the first aspect, in certain implementations of the first aspect, the method further includes: determining a first code rate upper limit pre-allocated to the region to be coded; the searching the code rate control parameter of the region to be coded according to the initial code rate control parameter comprises: calculating a second code rate required for coding the region to be coded according to the initial coarse granularity parameter; adjusting the initial coarse-grained parameter according to the size relationship between the first code rate and the second code rate, so that a third code rate corresponding to the adjusted coarse-grained parameter is less than or equal to the first code rate, and the absolute value of the difference between the third code rate and the first code rate is less than or equal to a first target threshold; calculating a fourth code rate required by coding the region to be coded according to the adjusted coarse granularity parameter and the initial fine granularity parameter; and adjusting the initial fine-grained parameter according to the size relationship between the first code rate and the fourth code rate, so that a fifth code rate corresponding to the adjusted fine-grained parameter is smaller than or equal to the first code rate, and the absolute value of the difference between the fifth code rate and the first code rate is smaller than or equal to a second target threshold.

With reference to the first aspect, in some implementations of the first aspect, the frame to be encoded is an original image file, and performing reversible color transform and discrete wavelet transform on a region to be encoded in the frame to be encoded to obtain a transform coefficient includes: 4 image channels are output after the reversible color transformation is carried out on the region to be coded in the frame to be coded; and performing the discrete wavelet transform on the 4 image channels to obtain transform coefficients.

It should be understood that the frame to be encoded is an original image file, which means that the frame to be encoded is a RAW image file, i.e., a bayer image before Image Signal Processing (ISP).

In a second aspect, there is provided an image encoding apparatus, the apparatus comprising a processing unit configured to: reversible color transformation and discrete wavelet transformation are carried out on a region to be coded in a frame to be coded to obtain a transformation coefficient; determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the reference region, wherein the code rate control parameter comprises a coarse granularity parameter and a fine granularity parameter; searching the code rate control parameter of the area to be coded according to the initial code rate control parameter; quantizing the transform coefficient according to the searched code rate control parameter; and carrying out Columbus encoding and zero run encoding on the quantized transform coefficients.

With reference to the second aspect, in some implementations of the second aspect, if the frame to be encoded is an intra-parameter prediction frame, the reference area includes a first reference area, and the first reference area is a first target area already encoded in the frame to be encoded; the processing unit is further configured to: and determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the first reference region.

With reference to the second aspect, in certain implementations of the second aspect, the first target region is a previous region vertically adjacent to the region to be encoded.

With reference to the second aspect, in some implementations of the second aspect, if the frame to be encoded is an inter-frame parameter prediction frame, the reference region includes a second reference region and a third reference region, the second reference region is a second target region already encoded in the frame to be encoded, and the third reference region is a third target region already encoded in the target frame; the processing unit is further configured to: and determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the second reference region and the code rate control parameter of the third reference region.

With reference to the second aspect, in some implementations of the second aspect, the processing unit is further configured to: determining the content complexity of the second reference area, the third reference area and the area to be coded; determining a first weight coefficient of the second reference area and a second weight coefficient of the third reference area according to the content complexity of the second reference area, the third reference area and the area to be coded; and determining an initial code rate control parameter of the region to be coded according to the first weight coefficient, the second weight coefficient, the code rate control parameter of the second reference region and the code rate control parameter of the third reference region.

With reference to the second aspect, in certain implementations of the second aspect, the second target region is a previous region vertically adjacent to the region to be encoded; and/or, the target frame is a previous frame of the frame to be coded; and/or the third target area is an area corresponding to the area to be coded in the target frame.

With reference to the second aspect, in some implementations of the second aspect, the processing unit is further configured to: determining a first code rate upper limit pre-allocated to the region to be coded; calculating a second code rate required by coding the region to be coded according to the initial coarse granularity parameter; adjusting the initial coarse-grained parameter according to the size relationship between the first code rate and the second code rate, so that a third code rate corresponding to the adjusted coarse-grained parameter is less than or equal to the first code rate, and the absolute value of the difference between the third code rate and the first code rate is less than or equal to a first target threshold; calculating a fourth code rate required by coding the region to be coded according to the adjusted coarse granularity parameter and the initial fine granularity parameter; and adjusting the initial fine-grained parameter according to the size relationship between the first code rate and the fourth code rate, so that a fifth code rate corresponding to the adjusted fine-grained parameter is smaller than or equal to the first code rate, and the absolute value of the difference between the fifth code rate and the first code rate is smaller than or equal to a second target threshold.

With reference to the second aspect, in some implementations of the second aspect, the frame to be encoded is an original image file, and the processing unit is further configured to: 4 image channels are output after the reversible color transformation is carried out on the region to be coded in the frame to be coded; and performing the discrete wavelet transform on the 4 image channels to obtain transform coefficients.

In a third aspect, a control apparatus is provided, which includes an input/output interface, a processor, and a memory, where the processor is configured to control the input/output interface to send and receive signals or information, and the memory is configured to store a computer program, and the processor is configured to call and execute the computer program from the memory, so that the control apparatus executes an image encoding method as in the first aspect or any possible implementation manner of the first aspect.

In a fourth aspect, there is provided an image encoding apparatus comprising: a memory for storing a program; a processor configured to execute the program stored in the memory, and when the program stored in the memory is executed, implement the image encoding method as in the first aspect or any possible implementation manner of the first aspect.

In a fifth aspect, a camera is provided, which includes the image encoding apparatus as in the second aspect or any possible implementation manner of the second aspect; alternatively, an image encoding apparatus as the fourth aspect is included.

A sixth aspect provides a vehicle comprising the image encoding apparatus as in the second aspect or any possible implementation manner of the second aspect; alternatively, an image encoding apparatus as the fourth aspect is included.

In a seventh aspect, a computing device is provided, comprising: at least one processor and a memory, the at least one processor being coupled to the memory and configured to read and execute instructions in the memory to perform an image encoding method as in the first aspect or any possible implementation manner of the first aspect.

In an eighth aspect, a computer program product containing instructions is provided, which when run on a computer causes the computer to perform the image encoding method of the first aspect or any of the possible implementations of the first aspect.

In a ninth aspect, there is provided a computer readable storage medium storing program code for execution by a device, the program code comprising instructions for performing the image encoding method of the first aspect or any of its possible implementations.

In a tenth aspect, a chip is provided, where the chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the image encoding method in the first aspect or any possible implementation manner of the first aspect.

Optionally, as an implementation manner, the chip may further include a memory, where instructions are stored in the memory, and the processor is configured to execute the instructions stored in the memory, and when the instructions are executed, the processor is configured to execute the image encoding method in the first aspect or any possible implementation manner of the first aspect.

In an eleventh aspect, a chip system is provided, the chip system comprising at least one processor for supporting the functionality involved in implementing the first aspect or some implementations of the first aspect, e.g. receiving or processing data and/or information involved in the method described above.

In one possible design, the system-on-chip further includes a memory to hold program instructions and data, the memory being located within the processor or external to the processor. The chip system may be formed by a chip, or may include a chip and other discrete devices.

Drawings

FIG. 1 is an illustration of an encoded stream Cheng Shi of a low latency lightweight image coding system according to an embodiment of the present disclosure;

fig. 2 is a diagram illustrating an example of a video encoding process according to an embodiment of the present application;

fig. 3 is a diagram illustrating an example of an image encoding method according to an embodiment of the present application;

FIG. 4 is a diagram of an example of a parameter search process for an intra-frame parameter prediction frame according to an embodiment of the present application;

FIG. 5 is a diagram illustrating an exemplary process of performing a parameter search according to initial parameters according to an embodiment of the present application;

FIG. 6 is a diagram illustrating an exemplary parameter searching process for an inter-frame parameter prediction frame according to an embodiment of the present application;

FIG. 7 is a diagram illustrating an example of an image encoding apparatus according to an embodiment of the present application;

fig. 8 is an exemplary block diagram of a hardware structure of an apparatus according to an embodiment of the present disclosure.

Detailed Description

To facilitate understanding, the following first presents several concepts and terms related to the present application.

Delay (latency): the buffer space consumed by processing the collected data (according to the ISO/IEC 21122 standard).

Time complexity: end-to-end codec time consumption.

Code rate control: and determining the optimal code rate control parameter under the condition of meeting the bandwidth limitation, so that the compression distortion is minimum.

Slice (slice): a set of integer number of regions (precincts) that can be independently entropy decoded. The slice may be a frame of image or a portion of a frame of image. In the embodiment of the present application, the slice refers to a set of 16 regions that can be independently entropy-decoded.

Region (precinct): the JPEG-XS standard defines a part of the spatial area of an image, specifically, an integer number (e.g., 4) of image lines in a frame of image.

JPEG-XS: is a new standard (ISO/IEC 21122 standard) of a low-delay lightweight image coding system (low-latency light image coding system).

Fig. 1 is an illustration of an encoded stream Cheng Shi of a low-latency lightweight image coding system according to an embodiment of the present application. It should be understood that fig. 1 may be an encoding flowchart of a new standard JPEG-XS of a low-latency lightweight image encoding system, and may also be another standard, which is not limited in this application. As shown in fig. 1, the process 100 includes steps S110 to S160, which will be briefly described below.

And S110, inputting an image. First, a frame of image is input.

And S120, image transformation coding.

And then, performing Reversible Color Transformation (RCT) and Discrete Wavelet Transformation (DWT) on each region to be coded in the input image to obtain a corresponding transformation coefficient of each region to be coded. Among them, the reversible color transform RCT produces almost no delay, and the discrete wavelet transform DWT produces a low delay. It should be understood that the discrete wavelet transform DWT in the JPEG-XS standard is specifically an asymmetric discrete wavelet transform DWT, which has a delay of approximately 32 image lines.

Wherein the reversible color transform RCT and the discrete wavelet transform DWT are used to remove inter-component and intra-component correlations, respectively.

And S130, code rate control.

Firstly, allocating code rate to each region to be coded, and then performing unidirectional search (namely, setting the initial code rate control parameter as 0 and then starting the search) on the optimal code rate control parameter according to the allocated code rate.

It should be understood that the rate control parameters in the JPEG-XS standard include coarse-grain parameters and fine-grain parameters.

And S140, quantizing.

And quantizing the transform coefficient obtained in the step S120 according to the optimal code rate control parameter obtained in the step S130 to obtain a quantized transform coefficient. That is, the quantization step is calculated by the optimal code rate control parameter, and then the quantized transform coefficient is obtained by quantizing the transform coefficient by using the calculated quantization step. Wherein, the quantized transform coefficient can also be described as a quantization index.

And S150, entropy coding.

The quantized transform coefficients obtained in step S140 are entropy-encoded with low delay. It should be understood that entropy coding methods in the JPEG-XS standard include Golomb coding and zero run coding. Among them, the delay generated by the golomb encoding and the zero run length encoding is low. After the entropy encoding is finished, the encoding result can be output to the code stream, and then the code stream is output through step S160.

In the process of the transformation coding and the entropy coding, the transformation coding technology with low delay and the entropy coding technology with low delay are adopted, so that the low-delay lightweight image coding system achieves low delay under the condition of visual lossless reconstructed images.

The technical solution in the present application will be described below with reference to the accompanying drawings.

At present, the conventional low-delay lightweight image coding system JPEG-XS achieves low delay under the condition of visual lossless reconstructed picture, but the one-way search algorithm of the code rate control parameter still has more time complexity. Through statistics, the time complexity of the rate control module accounts for more than seventy percent of the overall end-to-end coding and decoding complexity, and therefore further optimization is necessary. In addition, the conventional JPEG-XS supports only image coding and does not support video coding.

Based on the above, the application provides an image coding method, which is mainly based on a low-delay lightweight image coding system JPEG-XS to perform intra-frame coding and inter-frame coding on image frames in a video, and designs a parameter fast search algorithm for intra-frame coding and a parameter fast search algorithm for inter-frame coding, so that a fast video coding technology with low delay and low time complexity is realized, and the speed of video coding and transmission can be increased.

It should be understood that the image coding method of the present application can be applied to scenes pursuing low delay and low time complexity, such as unmanned driving, unmanned aerial vehicle reconnaissance, virtual reality, etc. Specifically, the image encoding method of the present application may be applied to the acquisition device of the above-mentioned scene, and the acquisition device may be a camera.

Fig. 2 is a diagram illustrating an example of a video encoding process according to an embodiment of the present application. It should be understood that the encoding flow 200 shown in FIG. 2 may be implemented based on the Low-latency lightweight image coding System JPEG-XS. It should be understood that the video encoding process 200 shown in fig. 2 is only an example and is not intended to limit the scope of the present disclosure. As shown in fig. 2, the video encoding process 200 includes steps S210 to S250, which are briefly described below.

S210, a video stream and a group size are input.

It should be understood that for a video stream, there is content correlation between different regions within the same frame image and also between different frame images. Thus, with this correlation, the video stream can be grouped, each group comprising one intra parametric prediction frame and a plurality of inter parametric prediction frames. It should be understood that the present application does not limit the grouping manner of the video stream. Optionally, the number of frames corresponding to different groups may be the same or different. For convenience of description, the present application describes that the number of frames per group (i.e., group size) is N, and in this case, the number of groups into which the video stream can be divided may be determined according to the group size N.

As shown in fig. 2, when inputting a video stream, a group size N may be input simultaneously, that is, each N frames is defined as a group, and each group includes 1 intra-frame parameter prediction frame followed by N-1 inter-frame parameter prediction frames. It should be understood that the groupings shown in FIG. 2 are merely examples and are not to be construed as limitations of the present application.

And S220, transform coding.

Both the intra-frame parameter prediction frame and the inter-frame parameter prediction frame in the video stream are transform-coded to obtain transform coefficients, which can be referred to as the related description in the method 300.

And S230, controlling the code rate.

The optimal bitrate control parameter satisfying the bitrate constraint is searched according to the searching method of bitrate control parameters provided below, which can be referred to in the related descriptions of the method 300, the method 400, and the method 600.

And S240, quantizing and entropy coding the transformation coefficient.

The search result (i.e., the searched optimal rate control parameter) is used to calculate a quantization step, the transform coefficient is quantized according to the obtained quantization step, and the quantized transform coefficient is entropy-encoded, which can be referred to as the related description in the method 300, and finally step S250 is performed.

And S250, outputting the code stream.

Fig. 3 is a diagram illustrating an example of an image encoding method according to an embodiment of the present application. The method shown in FIG. 3 is implemented mainly based on the JPEG-XS (low-latency lightweight image coding system). As shown in fig. 3, the method 300 includes steps S310 to S350. It should be understood that, the sequence of the above steps is not limited in the embodiment of the present application, and any sequence of the above steps that can implement the scheme of the present application falls within the scope of the present application. These steps are described in detail below.

S310, reversible color transformation and discrete wavelet transformation are carried out on the region to be coded in the frame to be coded to obtain a transformation coefficient.

Optionally, the image coding in the present application may be intra-frame coding of a frame to be coded, where the frame to be coded may be an image frame existing alone or an image frame in a video stream, and at this time, the frame to be coded may be recorded as an intra-frame parameter prediction frame; the image coding may also be inter-frame coding of a frame to be coded, where the frame to be coded is an image frame in a video stream, and at this time, the frame to be coded may be marked as an inter-frame parameter prediction frame, which is not limited in this application. Based on this, the image encoding in the present application may include intra-frame encoding; and/or, inter-frame coding. It should be understood that, when encoding a video stream, the intra-frame parameter prediction frame and the inter-frame parameter prediction frame in the video stream may be intra-frame encoded and inter-frame encoded respectively by using the image encoding method provided in the present application.

And for the convenience of description, hereinafter, it is considered to perform image coding on a frame to be coded in a video stream. Then before step S310 is performed, the video stream needs to be input first, and the video stream is grouped, where each group includes 1 intra-frame parametric prediction frame and N-1 inter-frame parametric prediction frames, as described above in relation to step S210 of the method 200. After receiving the video stream, reversible color transform and discrete wavelet transform can be performed on each region of each frame of image in the video stream to obtain a transform coefficient corresponding to each region. For convenience of description, the following description will be given taking a region to be encoded in a frame to be encoded as an example. It should be understood that the frame to be encoded may be an intra-frame parameter prediction frame or an inter-frame parameter prediction frame, which is not limited in this application. Wherein a reversible color transform and a discrete wavelet transform are used to remove inter-and intra-component correlations, respectively.

It should be understood that a region is a portion of the spatial region of an image defined by the JPEG-XS standard, specifically referring to an integer number (e.g., 4) of image lines in a frame of an image. It should be understood that the reversible color transform and the discrete wavelet transform, which generate little delay, produce the delay of the transform coding method used in the JPEG-XS standard. It should be understood that the discrete wavelet transform used in the JPEG-XS standard refers specifically to an asymmetric discrete wavelet transform, the delay of which is approximately 32 image lines. Based on this, the image encoding method provided by the embodiment of the present application can also be considered to be implemented based on the low-delay lightweight image encoding system JPEG-XS.

It should be understood that the frame to be encoded is an original image file. It should be understood that the frame to be encoded is an original image file, which means that the frame to be encoded is a RAW image file, i.e., a bayer image before Image Signal Processing (ISP). It should be understood that, the reversible color transform and the discrete wavelet transform performed on the region to be coded in the frame to be coded to obtain the transform coefficients specifically are: 4 image channels are output after the reversible color transformation is carried out on the region to be coded in the frame to be coded; and performing the discrete wavelet transform on the 4 image channels to obtain transform coefficients. It should be understood that the original image file may be a separate image file, or may be an original image file in a video, which is not limited in this application.

S320, determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the reference region.

The code rate control parameters include coarse granularity parameters and fine granularity parameters, and the initial code rate control parameters also include initial coarse granularity parameters and initial fine granularity parameters. It should be understood that the coarse-granularity parameter and the fine-granularity parameter are rate control parameters defined in the JPEG-XS standard.

Optionally, if the frame to be encoded is an intra-parameter prediction frame, the reference area may include a first reference area, where the first reference area is a first target area already encoded in the frame to be encoded. In this case, the initial rate control parameter of the region to be encoded may be determined according to the rate control parameter of the first reference region.

In the embodiment of the application, when predicting the code rate control parameter of the region to be coded in the intra-parameter prediction frame, the initial code rate control parameter of the region to be coded can be determined according to the code rate control parameter of the coded first target region in the frame to be coded, that is, the better initial code rate control parameter is provided for the region to be coded by combining the content correlation among different regions in the same frame of image, so that the optimal code rate control parameter value of the region to be coded can be searched by fewer subsequent searching steps, and the time complexity of image coding can be reduced.

Optionally, the first target region may be a previous region vertically adjacent to the region to be encoded, or may be another region that is already encoded and vertically adjacent to the region to be encoded, which is not limited in this application.

As a preferred example, the following embodiments will be described by taking a previous region vertically adjacent to a region to be encoded as an example, and refer to the related description in the method 400.

In the application embodiment, the first target region may be a previous region vertically adjacent to a region to be encoded, so that when predicting a code rate control parameter of the region to be encoded in an intra-frame parameter prediction frame, an initial code rate control parameter of the region to be encoded may be determined according to the code rate control parameter of the previous region vertically adjacent to the region to be encoded in the frame to be encoded, that is, in combination with content correlation between adjacent regions in the same frame of image, a better initial code rate control parameter is provided for the region to be encoded, so that an optimal code rate control parameter value of the region to be encoded can be searched through fewer search steps, and thus time complexity of image encoding can be reduced.

Alternatively, if the frame to be encoded is an inter-parameter prediction frame, the reference region may include a second reference region and a third reference region. The second reference area is a second target area already coded in the frame to be coded, and the third reference area is a third target area in the already coded target frame. In this case, the initial rate control parameter of the region to be encoded may be determined according to the rate control parameter of the second reference region and the rate control parameter of the third reference region.

In the embodiment of the application, when predicting the code rate control parameter of the region to be coded in the inter-parameter prediction frame, the initial code rate control parameter of the region to be coded can be determined according to the code rate control parameter of the coded second target region in the frame to be coded and the code rate control parameter of the third target region in the coded target frame, that is, the better initial code rate control parameter is provided for the region to be coded by combining the content correlation between different regions in the same frame image and the content correlation between different frame images, so that the optimal code rate control parameter value of the region to be coded can be searched through fewer search steps, and the time complexity of image coding can be reduced.

It should be understood that the present application does not limit an implementation manner of determining the initial rate control parameter of the region to be encoded according to the rate control parameter of the second reference region and the rate control parameter of the third reference region. Optionally, the initial code rate control parameter of the region to be encoded may be directly determined according to an average value of the code rate control parameter of the second reference region and the code rate control parameter of the third reference region; the method may further include determining weighting coefficients of a second reference region and a third reference region, and performing weighted summation on the code rate control parameter of the second reference region and the code rate control parameter of the third reference region to determine an initial code rate control parameter of the region to be encoded, which is not limited in the present application. It should be understood that the present application does not limit the manner in which the second reference region and the third reference region weight coefficients are determined. Alternatively, the respective weighting coefficients may be set directly or may be determined according to the complexity of the content.

For example, a first weight coefficient of the second reference region and a second weight coefficient of the third reference region may be determined according to the content complexity of the second reference region, the third reference region, and the region to be encoded; then, an initial code rate control parameter of the region to be encoded is determined according to the first weight coefficient, the second weight coefficient, the code rate control parameter of the second reference region, and the code rate control parameter of the third reference region, which may be specifically described in the related description of the method 600.

It should be understood that the rate control parameters for regions with more similar content complexity have higher reference values. Therefore, in the embodiment of the present application, a first weight coefficient of a second reference region and a second weight coefficient of a third reference region are determined according to content complexity of the second reference region, the third reference region and a region to be encoded, and then an initial code rate control parameter of the region to be encoded is determined according to the first weight coefficient, the second weight coefficient, a code rate control parameter of the second reference region and a code rate control parameter of the third reference region. Therefore, a better and more accurate initial code rate control parameter can be provided for the code rate control parameter of the region to be coded, and the time complexity of image coding is further reduced.

Optionally, the second target region may be a previous region vertically adjacent to the region to be encoded, or may be another region already encoded in the frame to be encoded, which is not limited in this application. As a preferred example, the following embodiments will be described by taking the previous region vertically adjacent to the region to be encoded as an example, and refer to the related description in the method 600.

Optionally, the target frame may be a previous frame of a frame to be encoded, or may also be another encoded frame, which is not limited in this application. Optionally, the third target region may be a region corresponding to a region to be encoded in the target frame, or may also be another region in the target frame, which is not limited in this application. As a preferred example, in the following embodiments, a region corresponding to a region to be encoded in a frame previous to a frame to be encoded is taken as an example for description, and reference may be made to the related description in the method 600.

In this embodiment of the present application, the second target region may be a previous region vertically adjacent to a region to be encoded, the target frame may be a previous frame of a frame to be encoded, and the third target region may be a region corresponding to the region to be encoded in the target frame, so that when predicting a rate control parameter of the region to be encoded in an inter-parameter prediction frame, an initial rate control parameter of the region to be encoded may be determined according to the rate control parameter of the previous region vertically adjacent to the region to be encoded in the frame to be encoded and the rate control parameter of the region corresponding to the region to be encoded in the previous frame of the frame to be encoded, that is, a better initial rate control parameter is provided for the region to be encoded by combining content correlation between adjacent regions in the same frame of image and content correlation between adjacent frame of images, so that an optimal rate control parameter value of the region to be encoded can be searched through fewer search steps, thereby reducing complexity of image encoding.

S330, searching the code rate control parameter of the area to be coded according to the initial code rate control parameter.

It should be understood that the parameter searching process in the embodiment of the present application may also be understood as an adjustment process of a parameter, which is not distinguished in the present application.

Specifically, the rate control parameter of the region to be coded may be searched as follows: determining a first code rate upper limit pre-allocated to a region to be coded; calculating a second code rate required by coding the region to be coded according to the initial coarse granularity parameter; adjusting the initial coarse-grained parameter according to the size relationship between the first code rate and the second code rate, so that a third code rate corresponding to the adjusted coarse-grained parameter is less than or equal to the first code rate, and the absolute value of the difference between the third code rate and the first code rate is less than or equal to a first target threshold; calculating a fourth code rate required by encoding the region to be encoded according to the adjusted coarse granularity parameter and the initial fine granularity parameter; according to the size relationship between the first code rate and the fourth code rate, the initial fine-grained parameter is adjusted, so that a fifth code rate corresponding to the adjusted fine-grained parameter is smaller than or equal to the first code rate, and an absolute value of a difference between the fifth code rate and the first code rate is smaller than or equal to a second target threshold, which may be specifically referred to in the related descriptions of the method 400 and the method 600.

It should be understood that, the third code rate corresponding to the adjusted coarse-grained parameter is less than or equal to the first code rate, and the absolute value of the difference between the third code rate and the first code rate is less than or equal to the first target threshold, it can also be described that the third code rate corresponding to the adjusted coarse-grained parameter needs to take the maximum possible value less than or equal to the first code rate. It should be understood that, the fifth code rate corresponding to the adjusted fine-grained parameter is less than or equal to the first code rate, and the absolute value of the difference between the fifth code rate and the first code rate is less than or equal to the second target threshold, which can also be described as that the fifth code rate corresponding to the adjusted fine-grained parameter needs to take the maximum possible value less than or equal to the first code rate. It should be understood that the values of the first target threshold and the second target threshold need to be determined according to actual situations, which is not limited in the present application.

It should be understood that the third code rate corresponding to the adjusted coarse-grained parameter is the code rate required for encoding the region to be encoded according to the adjusted coarse-grained parameter; the fifth code rate corresponding to the adjusted fine granularity parameter is the code rate required by the coding of the region to be coded according to the adjusted coarse granularity parameter and the adjusted fine granularity parameter.

And S340, quantizing the transform coefficient according to the searched code rate control parameter.

Specifically, a quantization step is calculated according to the searched rate control parameter, and then the transform coefficient obtained in step S310 is quantized according to the quantization step.

And S350, performing Golomb coding and zero run coding on the quantized transform coefficient.

That is, the quantized transform coefficient obtained in step S340 is entropy-encoded with low delay. It should be understood that Golomb encoding and zero run length encoding are entropy encoding methods used in the JPEG-XS standard. Among them, the delay generated by the golomb encoding and the zero run length encoding is low. It should be understood that the method of entropy coding is not limited in this application, and may use golomb coding and zero run coding, or may use only golomb coding, or may use other methods for coding, as long as the effect of low delay is achieved. After the entropy coding is finished, the coding result can be output to the code stream, and finally the code stream is output to finish the coding.

In the image coding method provided by the embodiment of the application, a region to be coded in a frame to be coded is transformed through reversible color transformation and discrete wavelet transformation to obtain a transform coefficient, an initial code rate control parameter of the region to be coded is determined according to a code rate control parameter of a reference region, then the code rate control parameter of the region to be coded is searched according to the initial code rate control parameter, finally the transform coefficient is quantized according to the searched code rate control parameter, and then Golomb coding and zero run length coding are carried out on the quantized transform coefficient. On one hand, in the image coding process, a low-delay transformation coding method and an entropy coding method are adopted, so that low-delay image coding can be realized; on the other hand, the initial code rate control parameter of the region to be coded is determined according to the code rate control parameter of the reference region, so that when the code rate control parameter of the region to be coded is searched, the search can be carried out based on the better initial code rate control parameter, and the time complexity of image coding can be reduced. Based on this, the image coding method provided by the embodiment of the application can also realize video coding with low delay and low time complexity; further, the speed of video encoding and transmission can be increased. It is to be understood that video is composed of a plurality of image frames, the video including intra-frame parameter prediction frames and inter-frame parameter prediction frames, and the video encoding includes intra-frame encoding and inter-frame encoding.

In the scheme of the application, after the code rate control parameters are obtained by the fast search algorithm of the intra-frame parameter prediction frame and the parameter prediction algorithm of the inter-frame parameter prediction frame, the quantization step size is calculated, then the quantization step size is used for quantizing the transformation coefficient, then the quantized transformation coefficient is subjected to entropy coding and written into a code stream, so that the video information can be reconstructed by decoding the quantization step size, inversely quantizing other coefficients and performing inverse image transformation on the reconstructed video at a decoding end, and the operations can be performed in parallel, thereby ensuring the fast and efficient decoding.

The parameter search process for the intra-frame parameter prediction frame and the parameter search process for the inter-frame parameter prediction frame according to the present application will be described in detail below with reference to fig. 4 to 6.

Fig. 4 is a diagram illustrating an example of a parameter search process for an intra parameter prediction frame according to an embodiment of the present application. It should be understood that fig. 4 can also be regarded as an exemplary diagram of a rate control process for an intra parameter prediction frame, and this is not distinguished in the present application. As shown in fig. 4, the process 400 includes steps S410 to S430. It should be understood that, the sequence of the above steps is not limited in the embodiment of the present application, and all solutions that can implement the present application through any sequence of the above steps fall within the protection scope of the present application. These steps are described in detail below.

S410, allocating a code rate upper limit B to a region to be coded in the intra-frame parameter prediction frame_max。

It should be understood that, in the code rate control process of the intra parameter prediction frame, in order to pre-allocate the upper limit of the code rate B to each region to be coded_maxFirst, the total code rate B of a slice (slice) needs to be divided equally according to the number of the areas contained in the slice_allTo obtain B_avgThen pre-assigning B to the first region of the slice_max:＝B_avgIf some code rates are saved after each region is coded, the part of code rates are transmitted to the next region, which means that the upper limit B of the pre-allocated code rate of the next region is_maxIs B_avgAnd the sum of the code rates saved by the previous encoding. In this way, the upper limit B of the pre-allocated code rate of the current region p to be coded is determined_max。

S420, determining an initial code rate control parameter of a to-be-coded area of the intra-frame parameter prediction frame.

In the present application, an initial code rate is determinedThe control parameters are the initial coarse granularity parameter and the initial fine granularity parameter. In this embodiment, the coarse-grained parameter Q of the previous region p-1 (i.e. the first reference region) vertically adjacent to the region p to be encoded will be used_p-1And a fine particle size parameter R_p-1As an initial coarse-grained parameter and an initial fine-grained parameter of the region p to be encoded.

S430, searching code rate control parameters of the region to be coded of the intra-frame parameter prediction frame.

Aiming at a region p to be coded (namely the current region p to be coded), the upper limit B of the code rate needs to be pre-allocated according to the region p to be coded_max(i.e., the first code rate upper limit) and the initial coarse-grained parameter Q of the region to be coded_p-1And an initial fine-grained parameter R_p-1Coarse grain parameter Q from the region p to be encoded_pAnd a fine granularity parameter R_pAnd searching to determine the optimal coarse-grained parameter and fine-grained parameter of the region p to be coded. It should be understood that the coarse-grained parameter may also be referred to as a coarse-grained control parameter, and the fine-grained parameter may also be referred to as a fine-grained control parameter, which is not distinguished in the present application. The following coarse-grained parameters Q of the regions p to be coded are each_pAnd a fine granularity parameter R_pThe search process of (2) is introduced.

As shown in FIG. 5, the coarse granularity parameter Q is first calibrated_pA search is conducted.

Specifically, the coarse-grained parameter Q of the region p to be encoded is first determined_pAnd a fine granularity parameter R_pThe initialization is as follows:

Q_p:＝Q_p-1；R_p:＝0；

then, the code rate (which can also be expressed as bit number) B required for coding the region p to be coded by using the set of parameters is calculated_total(i.e., the second code rate). According to the code rate B required by coding_total(i.e., second code rate) and a pre-allocated code rate cap B_maxThe magnitude relation of (A) and (B)_totalAnd Q_pMonotonic relationship of (B)_totalAnd Q_pIs monotonically decreasing), a coarse granularity parameter Q is determined_pIn the search direction of B_totalGet less than or equal to B_maxStops searching when the maximum possible value (third code rate) of (c) is obtainedObtaining the optimal coarse grain size parameter Q_pAnd fixed.

Specifically, the coarse-grained parameter Q is searched_pIn the process of (1), if B_total＞B_maxThen gradually increase Q_pUp to B_total＜B_maxWhen the search is over, the search ends and the current Q is fixed_pIs the optimal coarse granularity parameter. If B is_total＜B_maxThen gradually decrease Q_pUp to B_total＞B_maxWhen the search is over, then at the current Q_pAdding 1 on the basis and fixing to satisfy B_total＜B_maxWhile making B_totalThe maximum possible value is taken.

At this time Q_pFixed, then, for the fine granularity parameter R_pA search is conducted.

In particular, the fine-grained parameter R of the region p to be coded_pThe initialization is as follows:

R_p:＝R_p-1；

then according to the fixed Q_pAnd R obtained from the current initialization_pCalculating the code rate B required by coding the region p to be coded_total(i.e., the fourth code rate). According to the code rate B required by coding_total(i.e., fourth code rate) and a pre-allocated upper limit B for the code rate_maxThe magnitude relation of (A) and (B)_totalAnd R_pMonotonic relationship (B)_totalAnd R_pIs monotonically increasing), a fine-grained parameter R is determined_pIn the search direction of (1), also in B_totalGet less than or equal to B_maxStopping searching when the maximum possible value (namely the fifth code rate) is obtained, thereby obtaining the optimal fine-grained parameter B_maxAnd fixed.

In particular, in searching for a fine-grained parameter R_pIn the process of (1), if B_total＞B_maxThen gradually decrease R_pUp to B_total＜B_maxWhen the search is finished, the current R is fixed_pIs the optimal fine-grained parameter. If B is_total＜B_maxThen gradually increase R_pUp to B_total＞B_maxWhen the search is finished, then atCurrent R_pIs reduced by 1 and fixed on the basis to satisfy B_total＜B_maxWhile making B_totalThe maximum possible value is taken. It should be understood that the above search fixed Q_pAnd R_pNamely searching and determining the optimal coarse granularity parameter and the optimal fine granularity parameter for the region to be coded.

It should be understood that, compared with the original one-way search algorithm (that is, when searching for the code rate control parameter of the region to be encoded, 0 is used as an initial value for searching), in the intra-frame parameter prediction of the frame to be encoded, the content correlation of adjacent regions in the same frame image is considered, and the code rate control parameter determined by the previous region vertically adjacent to the region to be encoded is used as the initial value of the code rate control parameter of the region to be encoded, so that the optimal value can be searched through fewer search steps, and the time complexity is greatly reduced.

After the optimal coarse-grained parameter and the optimal fine-grained parameter of the region p to be coded are obtained through searching, the quantization step length can be calculated according to the group of parameters, then the transformation coefficient of the region to be coded is quantized by the quantization step length, and then the output code stream of the region to be coded of the intra-frame parameter prediction frame is obtained through entropy coding of the quantized transformation coefficient.

It should be understood that, through preliminary experiments, by using the fast search algorithm for intra-frame parameters designed by the present application, the end-to-end encoding and decoding process can save more than ten percent of time complexity.

Fig. 6 is a diagram illustrating an example of a parameter search process for an inter-frame parameter prediction frame according to an embodiment of the present application. It should be understood that fig. 6 can also be regarded as an exemplary diagram of a rate control process for an inter-parameter prediction frame, and this is not distinguished in the present application. As shown in fig. 6, the process 600 includes steps S610 to S630. It should be understood that, the sequence of the above steps is not limited in the embodiment of the present application, and all solutions that can implement the present application through any sequence of the above steps fall within the protection scope of the present application. These steps are described in detail below.

S610, allocating a code rate upper limit B to a region to be coded in the inter-frame parameter prediction frame_max。

It should be understood that, in the code rate control process of the inter-frame parameter prediction frame, in order to pre-allocate the upper limit B of the code rate to each region to be coded_maxFirstly, the total code rate B of a slice (slice) of a frame needs to be equally divided according to the number of areas contained in the slice_allTo obtain B_avgThen pre-assigning B to the first area of the slice_max:＝B_avgIf some code rates are saved after each region is coded, the part of code rates are transmitted to the next region, which means the upper limit B of the pre-allocated code rate of the next region_maxIs B_avgAnd the sum of the code rates saved by the previous encoding. In this way, the pre-allocated code rate upper limit B of the region p to be coded of the frame to be coded (noted as the ith frame) in the inter-frame parameter prediction frame is determined_max。

S620, determining the initial code rate control parameter of the inter-frame parameter prediction frame to-be-coded area.

In this application, determining the initial rate control parameter is to determine an initial coarse-grained parameter and an initial fine-grained parameter. In this embodiment, the coarse granularity parameter Q of the encoded region p (i.e., the third reference region) in the (i-1) th frame will be incorporated_i-1,pAnd a fine particle size parameter R_i-1,pAnd the coarse-grained parameter Q of the coded region p-1 (i.e. the second reference region) in the frame to be coded (i.e. the ith frame)_i,p-1And a fine particle size parameter R_i,p-1To determine an initial coarse-grained parameter Q of a current region to be encoded in a frame to be encoded_i,p' and initial Fine granularity parameter R_i,p'. It should be understood that the present application is not limited to determining the initial coarse-grained parameter Q of the region to be encoded based on the second reference region and the third reference region_i,p' and initial Fine granularity parameter R_i,pThe method of' is described. Optionally, the initial coarse-grained parameter of the region to be encoded may be determined according to an average value of a sum of the coarse-grained parameter of the second reference region and the coarse-grained parameter of the third reference region, and the initial fine-grained parameter of the region to be encoded may be determined according to an average value of a sum of the fine-grained parameter of the second reference region and the fine-grained parameter of the third reference region. Optionally, the respective weight systems of the second reference region and the third reference region may be determined firstNumbers (i.e. β and α, and α + β = 1), and then determining initial parameter values of the region to be encoded according to the respective weight coefficients and the respective parameter values, as shown in the following formula:

it should be understood that the values of the weight coefficients α and β are not limited in the present application, for example, both α and β may be 0.5, and for example, α may be 0.4 and β may be 0.6.

It should be understood that, in general, the parameters of the regions with similar content complexity have higher reference values, and therefore, as a preferred scheme, the content complexity may be calculated for the two reference regions and the region to be encoded respectively, and the respective weight coefficients (i.e., the second weight coefficient α and the first weight coefficient β) of the two reference regions may be determined according to the content complexities of the three regions.

In particular, the content complexity w of the third reference region is calculated_i-1,pContent complexity w of the second reference region_i,p-1And the content complexity w of the region to be encoded_i,pThen, α and β are determined according to the following formula:

it should be understood that the content complexity referred to in the present application may be calculated using the scene content average activity, may also be calculated using the human visual system characteristics, and may also be calculated using other existing methods, which are not limited in this application.

It should be understood that the weight coefficients of the reference regions are determined according to the content complexity to further determine the reference values of different reference regions, so that better initial parameter values can be determined, and the time complexity of the search is further reduced. It should be understood that, while determining the weight coefficients of the reference region based on the complexity of the picture content, the method also brings great computational complexity. In order to reduce the complexity of calculation, a simplified method is to directly take a constant for alpha and beta, for example, both the constant and the constant are 0.5.

S630, searching code rate control parameters of the region to be coded of the inter-frame parameter prediction frame.

Aiming at an inter-frame parameter prediction frame (i frame) to-be-coded region p (namely the current to-be-coded region p), the code rate upper limit B needs to be pre-allocated according to the to-be-coded region_max(i.e., the first code rate upper limit) and the initial coarse-grained parameter Q of the region to be coded_i,p' and initial Fine granularity parameter R_i,p' coarse-grained parameter Q for region to be encoded p_i,pAnd a fine particle size parameter R_i,pAnd searching to determine the optimal coarse-grained parameter and fine-grained parameter of the region p to be coded. It should be understood that the coarse-grained parameter may also be referred to as a coarse-grained control parameter, and the fine-grained parameter may also be referred to as a fine-grained control parameter, which is not distinguished in the present application.

It should be understood that the initial coarse granularity parameter Q is based on_i,p' and initial Fine granularity parameter R_i,p' to treat the coarse-grained parameter Q of the coding region p_i,pAnd a fine granularity parameter R_i,pThe process of performing the search may be as shown in fig. 4.

Similarly, the coarse-grained parameter Q is first adjusted_i,pA search is conducted.

Specifically, the coarse-grained parameter Q of the region p to be encoded is first determined_i,pAnd a fine particle size parameter R_i,pThe initialization is as follows:

Q_i,p:＝Q_i,p'；R_i,p:＝0；

then, the code rate (which can also be expressed as bit number) B required for coding the region p to be coded by using the set of parameters is calculated_total(i.e., second code rate). According to the code rate B required by the coding_totalAnd a pre-allocated code rate upper limit B_maxThe magnitude relation of (A) and (B)_totalAnd Q_i,pMonotonic relationship of (B)_totalAnd Q_i,pIs monotonically decreasing), a coarse granularity parameter Q is determined_i,pIn the search direction of B_totalGet less than or equal to B_maxIs stopped to obtain the optimal coarse-grained parameter Q_i,pAnd fixed.

Specifically, the coarse-grained parameter Q is searched_i,pIn the process of (1), if B_total＞B_maxThen gradually increase Q_i,pUp to B_total＜B_maxWhen the search is over, the search ends and the current Q is fixed_i,pIs the optimal coarse granularity parameter. If B is_total＜B_maxThen gradually decrease Q_i,pUp to B_total＞B_maxWhen the search is over, then at the current Q_i,pAdding 1 on the basis and fixing to satisfy B_total＜B_maxWhile making B_totalThe maximum possible value is taken.

At this time Q_i,pFixed, then, for the fine granularity parameter R_i,pA search is conducted.

In particular, the fine-grained parameter R of the region p to be coded_i,pThe initialization is as follows:

R_i,p:＝R_i,p'；

then according to the fixed Q_i,pAnd R obtained by current initialization_i,pCalculating the code rate B required by coding the region p to be coded_total(i.e., the fourth code rate). According to the code rate B required by the coding_total(i.e., fourth code rate) and a pre-allocated code rate cap B_maxSize relationship of (A) and (B)_totalAnd R_i,pMonotonic relationship of (B)_totalAnd R_i,pIs monotonically increasing), a fine-grained parameter R is determined_i,pIn the search direction of (1), also in B_totalGet less than or equal to B_maxStopping searching when the maximum possible value (namely the fifth code rate) is obtained, thereby obtaining the optimal fine-grained parameter B_maxAnd fixed.

In particular, in searching for a fine-grained parameter R_i,pIn the process of (A), if B_total＞B_maxThen gradually decrease R_i,pUp to B_total＜B_maxWhen the search is over, the search is ended and the current R is fixed_i,pIs the optimal fine-grained parameter. If B is_total＜B_maxThen gradually increase R_i,pUp to B_total＞B_maxWhen the search is over, then at the current R_i,pIs reduced by 1 and fixed on the basis to satisfy B_total＜B_maxWhile making B_totalThe maximum possible value is taken. It should be understood that the above search fixed Q_i,pAnd R_i,pNamely, the optimal coarse granularity parameter and the optimal fine granularity parameter which are determined by searching the region to be coded in the inter-frame parameter prediction frame.

It should be understood that when the parameter prediction of the inter-frame parameter prediction frame is performed, the content correlation of adjacent frames and the content correlation of adjacent areas in the frames are combined, and the initial value of the code rate control parameter of the area to be coded is determined according to the code rate control parameters of the corresponding area of the previous frame and the previous area vertically adjacent to the area to be coded, so that a better initial parameter is provided, an optimal value can be searched through fewer searching steps, and the time complexity is greatly reduced. Meanwhile, the method is realized based on a low-delay lightweight image coding system JPEG-XS, so that the inter-frame parameters needing to be stored in the parameter prediction process of the inter-frame parameter prediction frame are few, and the influence on the delay can be ignored. This is because the delay of the low-delay lightweight image coding system JPEG-XS is about 32 image lines, and the extra delay brought by the system in the parameter prediction process of the inter-frame parameter prediction frame in the present application is that every 4 image lines (1 region) correspond to two parameters. For example, when the resolution of a frame of image in a video is 3840 × 2160 pixels (3840 pixels wide and 2160 pixels high), the delay of JPEG-XS is 3840 × 32 pixels, and the extra delay caused by the parameter prediction process of the inter-frame parameter prediction frame using the system of the present application is (2160/4) × 2 pixels, which is less than one percent of the delay of JPEG-XS, and thus can be ignored.

After the optimal coarse-grained parameter and the optimal fine-grained parameter of the region p to be coded in the inter-frame parameter prediction frame are obtained through searching, the quantization step length can be calculated according to the group of parameters, then the transformation coefficient of the region to be coded is quantized by the quantization step length, and then the quantized transformation coefficient is subjected to entropy coding to obtain the output code stream of the region to be coded in the inter-frame parameter prediction frame.

It should be understood that, through preliminary experiments, the end-to-end encoding and decoding process can further save more than ten percent of time complexity by adopting the inter-frame parameter fast search algorithm designed by the present application.

Fig. 7 is a diagram illustrating an example of an image encoding apparatus according to an embodiment of the present application, where the apparatus 700 includes a processing unit 710.

The processing unit 710 is configured to perform reversible color transform and discrete wavelet transform on a region to be coded in a frame to be coded to obtain a transform coefficient; determining initial code rate control parameters of a region to be coded according to the code rate control parameters of the reference region, wherein the code rate control parameters comprise coarse granularity parameters and fine granularity parameters; searching the code rate control parameter of the area to be coded according to the initial code rate control parameter; quantizing the transform coefficient according to the searched code rate control parameter; and performing Golomb coding and zero run coding on the quantized transform coefficient.

Optionally, if the frame to be encoded is an intra-frame parameter prediction frame, the reference region includes a first reference region, and the first reference region is a first target region already encoded in the frame to be encoded; the processing unit 710 may also be configured to: and determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the first reference region.

Alternatively, the first target region may be a previous region vertically adjacent to the region to be encoded.

Optionally, if the frame to be encoded is an inter-frame parameter prediction frame, the reference region includes a second reference region and a third reference region, the second reference region is a second target region already encoded in the frame to be encoded, and the third reference region is a third target region already encoded in the target frame; the processing unit 710 may also be configured to: and determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the second reference region and the code rate control parameter of the third reference region.

Optionally, the processing unit 710 may be further configured to: determining the content complexity of the second reference area, the third reference area and the area to be coded; determining a first weight coefficient of the second reference area and a second weight coefficient of the third reference area according to the content complexity of the second reference area, the third reference area and the area to be coded; and determining an initial code rate control parameter of the region to be coded according to the first weight coefficient, the second weight coefficient, the code rate control parameter of the second reference region and the code rate control parameter of the third reference region.

Alternatively, the second target region may be a previous region vertically adjacent to the region to be encoded; and/or, the target frame may be a frame previous to the frame to be encoded; and/or, the third target region may be a region in the target frame corresponding to the region to be encoded.

Optionally, the processing unit 710 may be further configured to: determining a first code rate upper limit pre-allocated to a region to be coded; calculating a second code rate required by coding the region to be coded according to the initial coarse granularity parameter; adjusting the initial coarse-grained parameter according to the size relationship between the first code rate and the second code rate, so that a third code rate corresponding to the adjusted coarse-grained parameter is less than or equal to the first code rate, and the absolute value of the difference between the third code rate and the first code rate is less than or equal to a first target threshold; calculating a fourth code rate required by encoding the region to be encoded according to the adjusted coarse granularity parameter and the initial fine granularity parameter; and adjusting the initial fine-grained parameter according to the size relationship between the first code rate and the fourth code rate, so that a fifth code rate corresponding to the adjusted fine-grained parameter is smaller than or equal to the first code rate, and the absolute value of the difference between the fifth code rate and the first code rate is smaller than or equal to a second target threshold.

Optionally, the frame to be encoded is an original image file, and the processing unit 710 may be further configured to: performing reversible color transformation on a region to be coded in a frame to be coded and outputting 4 image channels; and performing discrete wavelet transform on the 4 image channels to obtain transform coefficients.

Fig. 8 is an exemplary block diagram of a hardware structure of an apparatus provided in an embodiment of the present application. Alternatively, the apparatus 800 may be a computer device. Alternatively, the apparatus 800 may be a chip for encoding, or the apparatus 800 may be an encoding device. The apparatus 800 includes a memory 810, a processor 820, a communication interface 830, and a bus 840. The memory 810, the processor 820 and the communication interface 830 are connected to each other through a bus 840.

The memory 810 may be a Read Only Memory (ROM), a static memory device, a dynamic memory device, or a Random Access Memory (RAM). The memory 810 may store a program, and the processor 820 is configured to perform the steps of the image encoding method of the embodiment of the present application when the program stored in the memory 810 is executed by the processor 820.

The processor 820 may be a general-purpose Central Processing Unit (CPU), a microprocessor, an Application Specific Integrated Circuit (ASIC), a Graphics Processing Unit (GPU) or one or more integrated circuits, and is configured to execute related programs to implement the image encoding method according to the embodiment of the present application.

Processor 820 may also be an integrated circuit chip having signal processing capabilities. In implementation, the image encoding method of the present application may be implemented by an integrated logic circuit of hardware or an instruction in the form of software in the processor 820.

The processor 820 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, or discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 810, and the processor 820 reads information in the memory 810, and performs, in combination with hardware thereof, functions required to be performed by modules included in the apparatus according to the embodiment of the present application, or performs the image encoding method according to the embodiment of the method of the present application.

Communication interface 830 enables communication between apparatus 800 and other devices or communication networks using transceiver means such as, but not limited to, transceivers.

Bus 840 may include a pathway to transfer information between various components of device 800, such as memory 810, processor 820, and communication interface 830.

An embodiment of the present application further provides an image encoding apparatus, including: a memory for storing a program; and a processor for executing the program stored in the memory, and implementing the image encoding method in the embodiment of the present application when the program stored in the memory is executed.

The embodiment of the application also provides a camera which comprises the image coding device in the embodiment of the application.

The embodiment of the application also provides a vehicle comprising the image coding device in the embodiment of the application.

Optionally, the vehicle related to the present application may be a conventional internal combustion engine vehicle, a hybrid vehicle, a pure electric vehicle, a centralized drive vehicle, a distributed drive vehicle, and the like, which is not limited in this application.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image encoding method, characterized in that the method comprises:

reversible color transformation and discrete wavelet transformation are carried out on a region to be coded in a frame to be coded to obtain a transformation coefficient;

determining an initial code rate control parameter of the region to be coded according to a code rate control parameter of a reference region, wherein the code rate control parameter comprises a coarse granularity parameter and a fine granularity parameter;

searching the code rate control parameter of the area to be coded according to the initial code rate control parameter;

quantizing the transform coefficient according to the searched code rate control parameter;

and carrying out Columbus encoding and zero run encoding on the quantized transform coefficients.

2. The method of claim 1, wherein if the frame to be encoded is an intra-parameter prediction frame, the reference region comprises a first reference region, and the first reference region is a first target region encoded in the frame to be encoded;

the determining the initial code rate control parameter of the region to be encoded according to the code rate control parameter of the reference region includes:

and determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the first reference region.

3. The method of claim 2, wherein the first target region is a previous region vertically adjacent to the region to be encoded.

4. The method according to any one of claims 1 to 3, wherein if the frame to be encoded is an inter-parameter prediction frame, the reference region comprises a second reference region and a third reference region, the second reference region is a second target region encoded in the frame to be encoded, and the third reference region is a third target region in the encoded target frame;

and determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the second reference region and the code rate control parameter of the third reference region.

5. The method of claim 4, wherein the determining the initial rate control parameter of the region to be encoded according to the rate control parameter of the second reference region and the rate control parameter of the third reference region comprises:

determining the content complexity of the second reference region, the third reference region and the region to be encoded;

determining a first weight coefficient of the second reference area and a second weight coefficient of the third reference area according to the content complexity of the second reference area, the third reference area and the area to be coded;

and determining an initial code rate control parameter of the region to be coded according to the first weight coefficient, the second weight coefficient, the code rate control parameter of the second reference region and the code rate control parameter of the third reference region.

6. The method according to claim 4 or 5, wherein the second target region is a previous region vertically adjacent to the region to be encoded; and/or the target frame is a frame before the frame to be coded; and/or the third target area is an area corresponding to the area to be coded in the target frame.

7. The method according to any one of claims 1 to 6, further comprising: determining a first code rate upper limit pre-allocated to the region to be coded;

the searching the code rate control parameter of the region to be coded according to the initial code rate control parameter comprises:

calculating a second code rate required by coding the region to be coded according to the initial coarse granularity parameter;

adjusting the initial coarse-grained parameter according to the size relationship between the first code rate and the second code rate, so that a third code rate corresponding to the adjusted coarse-grained parameter is less than or equal to the first code rate, and the absolute value of the difference between the third code rate and the first code rate is less than or equal to a first target threshold;

calculating a fourth code rate required by encoding the region to be encoded according to the adjusted coarse granularity parameter and the initial fine granularity parameter;

and adjusting the initial fine-grained parameter according to the size relation between the first code rate and the fourth code rate, so that a fifth code rate corresponding to the adjusted fine-grained parameter is smaller than or equal to the first code rate, and the absolute value of the difference between the fifth code rate and the first code rate is smaller than or equal to a second target threshold.

8. The method according to any one of claims 1 to 7, wherein the frame to be encoded is an original image file, and reversible color transform and discrete wavelet transform are performed on a region to be encoded in the frame to be encoded to obtain transform coefficients, including:

the reversible color transformation is carried out on the region to be coded in the frame to be coded, and then 4 image channels are output;

and performing the discrete wavelet transform on the 4 image channels to obtain transform coefficients.

9. An image encoding apparatus, characterized in that the apparatus comprises a processing unit configured to:

reversible color transformation and discrete wavelet transformation are carried out on a region to be coded in a frame to be coded to obtain a transformation coefficient; determining initial code rate control parameters of the region to be coded according to code rate control parameters of a reference region, wherein the code rate control parameters comprise coarse granularity parameters and fine granularity parameters; searching the code rate control parameter of the area to be coded according to the initial code rate control parameter; quantizing the transform coefficient according to the searched code rate control parameter; and carrying out Columbus encoding and zero run encoding on the quantized transform coefficients.

10. The apparatus of claim 9, wherein if the frame to be encoded is an intra-parameter prediction frame, the reference region comprises a first reference region, and the first reference region is a first target region encoded in the frame to be encoded;

the processing unit is further to: and determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the first reference region.

11. The apparatus of claim 10, wherein the first target region is a previous region vertically adjacent to the region to be encoded.

12. The apparatus according to any one of claims 9 to 11, wherein if the frame to be encoded is an inter-parameter prediction frame, the reference region comprises a second reference region and a third reference region, the second reference region is a second target region encoded in the frame to be encoded, and the third reference region is a third target region in the encoded target frame;

the processing unit is further to: and determining an initial code rate control parameter of the region to be coded according to the code rate control parameter of the second reference region and the code rate control parameter of the third reference region.

13. The apparatus of claim 12, wherein the processing unit is further configured to:

determining the content complexity of the second reference region, the third reference region and the region to be encoded; determining a first weight coefficient of the second reference area and a second weight coefficient of the third reference area according to the content complexity of the second reference area, the third reference area and the area to be coded; and determining an initial code rate control parameter of the region to be coded according to the first weight coefficient, the second weight coefficient, the code rate control parameter of the second reference region and the code rate control parameter of the third reference region.

14. The apparatus according to claim 12 or 13, wherein the second target region is a previous region vertically adjacent to the region to be encoded; and/or the target frame is a previous frame of the frame to be coded; and/or the third target area is an area corresponding to the area to be coded in the target frame.

15. The apparatus according to any one of claims 9 to 14, wherein the processing unit is further configured to:

determining a first code rate upper limit pre-allocated to the region to be coded;

calculating a second code rate required by encoding the region to be encoded according to the initial coarse granularity parameter; adjusting the initial coarse-grained parameter according to the size relationship between the first code rate and the second code rate, so that a third code rate corresponding to the adjusted coarse-grained parameter is less than or equal to the first code rate, and the absolute value of the difference between the third code rate and the first code rate is less than or equal to a first target threshold; calculating a fourth code rate required by encoding the region to be encoded according to the adjusted coarse granularity parameter and the initial fine granularity parameter; and adjusting the initial fine-grained parameter according to the size relationship between the first code rate and the fourth code rate, so that a fifth code rate corresponding to the adjusted fine-grained parameter is smaller than or equal to the first code rate, and the absolute value of the difference between the fifth code rate and the first code rate is smaller than or equal to a second target threshold.

16. The apparatus according to any of claims 9 to 15, wherein the frame to be encoded is an original image file, and wherein the processing unit is further configured to:

the reversible color transformation is carried out on the region to be coded in the frame to be coded, and then 4 image channels are output; and performing the discrete wavelet transform on the 4 image channels to obtain transform coefficients.

17. An image encoding device, comprising:

a memory for storing a program;

a processor for executing the memory-stored program, which when executed, implements the image encoding method of any one of claims 1 to 8.

18. A camera comprising the image encoding device according to any one of claims 9 to 16; or comprising an image encoding device as claimed in claim 17.

19. A vehicle characterized by comprising the image encoding device according to any one of claims 9 to 16; or, comprising an image encoding device according to claim 17.

20. A computer-readable storage medium characterized in that the computer-readable medium stores instructions for execution by a computing device, which when executed by the computing device, implement the image encoding method of any of claims 1 to 8.