CN110662048A

CN110662048A - Image coding method and device

Info

Publication number: CN110662048A
Application number: CN201810691606.2A
Authority: CN
Inventors: 何泽宇; 王豪; 刘瑞欢
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2018-06-28
Filing date: 2018-06-28
Publication date: 2020-01-07

Abstract

The implementation mode of the application discloses an image coding method and equipment, which are used for determining a boundary region in an image frame to be coded and coding. The implementation method comprises the following steps: determining a boundary line of a first area and a second area which are adjacent in an image frame to be coded, wherein the first area and the second area have color difference; determining a boundary region according to the boundary, wherein the boundary region comprises at least two non-overlapping boundary region units (AUs), each of the at least two boundary AUs contains a part of the boundary, and the boundary region contains all the boundaries, the size of each of the at least two boundary AUs is smaller than or equal to that of a reference AU, the size of the reference AU is smaller than the maximum coding size which can be supported by a current coding protocol used for coding the image frame to be coded; and encoding the boundary area in units of the boundary AU.

Description

Image coding method and device

Technical Field

The present application relates to the multimedia field, and in particular, to an image encoding method and apparatus.

Background

Video coding refers to data compression of video presented in the form of pixel-grouped frames by a specific compression technique to reduce storage space and facilitate transmission. The video can be compressed because there is a strong correlation between the image data of the video, i.e. at least one of the image data of one portion and the image data of the other portion is redundant information, i.e. there is a large amount of redundant information in the video. The compression technique is a technique of removing redundant information by using correlation between image data. Currently, international and common video coding algorithm standards specify that blocks of different sizes are divided, and for the blocks of different sizes, different intra/inter coding modes and different coding tools can be used.

In the h.265 standard, which is one of international common video Coding algorithm standards, it is specified that the size of available Coding Units (CUs) may be 64x64, 32x32, 16x16, 8x8 and 4x4, wherein the Largest Coding Unit (LCU) is used as a basic Unit for Coding, typically 64x 64. After the LCU is determined to be divided in one frame, the CUs with different sizes in a single LCU are divided, so that different dividing modes can be obtained.

In order to meet the requirements of reducing the calculated quantity and the high-quality visual effect, the distortion and the bit Rate of each division mode can be calculated by traversing all possible division modes, and then Rate Distortion Optimization (RDO) is calculated, and the optimal balance points for reducing the calculated quantity and the high-quality visual effect are found by comparing one by one, and the optimal coding mode is selected.

When a video having a black border region is encoded at a low bit rate, if the division method of the video is determined by the above method, there is a possibility that a region included in a CU having a large size simultaneously has pixels of the black border region and pixels of an effective region, and then, after the region included in the CU is encoded, an abnormal streak having a large area appears in the effective region near the boundary, resulting in a poor visual effect.

Disclosure of Invention

The embodiment of the application provides an image coding method and equipment, which are used for determining a boundary region in an image frame to be coded and coding.

A first aspect of the present application provides an image encoding method, including:

and determining a boundary line of a first area and a second area which are adjacent in the image frame to be coded, wherein the first area and the second area have color difference.

In some possible embodiments, the image frame to be encoded is an image that can be displayed on a display in the form of a pixel combination. It should be noted that the image frame to be encoded may be a photo, a drawing, a video screenshot, or the like, and in some possible embodiments, may also be an image of a frame in a video, which is not limited herein. It should be noted that, when displayed on the display, the image frame to be encoded may be a rectangle, a circle, a polygon or other irregular figure, which is not limited herein. In some possible embodiments, the image frame to be encoded may be displayed on a spherical display, or may be displayed on a dome-shaped display, or other curved-screen display, or other irregular-shaped display, which is not limited herein.

Determining a boundary area according to the boundary, wherein the boundary area includes at least two non-overlapping boundary Area Units (AUs) (area units), each of the at least two boundary AUs includes a part of the boundary, and the boundary area includes all the boundaries, the size of each of the at least two boundary AUs is smaller than or equal to that of a reference AU, the size of the reference AU is smaller than the maximum coding size that can be supported by a current coding protocol used for coding the image frame to be coded.

In the embodiment of the present application, after the boundary of the image frame to be encoded is determined, the boundary region may be determined based on the boundary. It should be noted that the AU is used to divide the image frame to be encoded, and the image frame to be encoded may be divided into at least one AU. Such as a rectangular AU with a size of 64x64, which can be denoted as AU-64, wherein each small region has a size of 64x64, which can be denoted as AU-64. In the embodiments of the present application, there may be a plurality of AU with different sizes, such as 32 × 32, 16 × 16, or 8 × 8, which is not limited herein. In some possible embodiments, the shape of AU may also be rectangular, hexagonal, etc., and is not limited herein. In some possible embodiments, the AU may be a coding unit in the protocol h.264 or the protocol h.265, such as a length 64 × 64, 32 × 32, 16 × 16, or 8 × 8, which is not limited herein.

The boundary area is coded by taking the boundary AU as a unit, and because the size of any AU in the boundary AU is smaller than or equal to the size of a reference AU, and the size of the reference AU is smaller than the maximum coding size which can be supported by the current coding protocol, after the boundary area is coded by taking the boundary AU as a unit, compared with the maximum coding size which can be supported by the current coding protocol, the boundary area is coded, abnormal stripes with smaller area appear on an effective area near the boundary, or the abnormal stripes do not appear, thereby improving the visual effect.

In some possible embodiments, the color difference between the first region and the second region includes: the difference between the color parameter of the first region and the color parameter of the second region is greater than or equal to a predetermined color difference value, and the difference between the color parameter of the first region and the color parameter of the second region is greater than or equal to the predetermined color difference value in a region of a predetermined size centered on any pixel of the boundary. In some possible embodiments, the first region and the second region in the image frame to be encoded may be arbitrarily divided, or may be divided according to some principles. In the embodiment of the application, the first region and the second region of the image frame to be encoded are adjacent, and the first region and the second region have a distinct boundary line, and on the boundary line, the two regions can be seen to have distinct colors. In some possible embodiments, for any one pixel in the image frame to be encoded, a region with a preset size may be determined with the pixel as the center, whether two parts of color difference exist in the region is determined, if yes, the pixel is determined as one pixel of a boundary, and the boundary between the first region and the second region in the image frame to be encoded is determined by traversing all pixels in the image frame to be encoded.

In some possible embodiments, the length of the reference AU is equal to half of the length of the maximum coding size that can be supported by the current coding protocol, and the width of the reference AU is equal to half of the width of the maximum coding size that can be supported by the current coding protocol, so as to avoid the situation that the size of the reference AU is equal to the maximum coding size that can be supported by the current coding protocol.

In some possible embodiments, the image frame to be encoded is any one frame of a video to be encoded, the first region is a black border region, and the black border region is a region covered by a video black border of the video to be encoded in the image frame to be encoded. In some possible embodiments, for example, the image of each frame of the video to be encoded is rectangular when it is played on a display, for example, the length and width of the screen of the display may be in a ratio of 16:9, but the length and width ratio of the old television screen is 4:3, so that the images in the old television play are in a ratio of 4: 3. When a video with the length-width ratio of 4:3 is played on a screen with the length-width ratio of 16:9, if the video is displayed normally, black edge areas appear on the left side and the right side of the screen; on the contrary, if the video with the aspect ratio of 16:9 is played on the screen with the aspect ratio of 4:3, if the video is normally displayed, black areas appear on the upper and lower sides of the screen. Therefore, in some possible embodiments, there may be black border areas on all 4 sides of the rectangular screen, which is not limited herein. The method can be applied to the video to solve the problem of the visual effect of abnormal horizontal stripes caused by the black edge of the video in the effective area at low code rate.

In some possible embodiments, the determining a boundary between adjacent first and second regions in the image frame to be encoded includes: and determining the black edge area in the image frame to be coded, and determining the boundary according to the black edge area.

In some possible embodiments, the determining the black border region in the image frame to be encoded includes:

calculating the average brightness value of the ith row of pixels in the image frame to be coded, and comparing the average brightness value of the ith row of pixels with a preset brightness value, wherein i is a non-negative integer; when the average brightness value of the ith row of pixels is less than or equal to the preset brightness value, determining that the ith row of pixels belongs to the black edge area; and when the average value of the brightness of the ith row of pixels is greater than the preset brightness value, determining that the ith row of pixels does not belong to the black edge area.

Or, calculating the brightness average value of the ith row of pixels in the image frame to be coded, and comparing the brightness average value of the ith row of pixels with a preset brightness value, wherein i is a non-negative integer; when the average brightness value of the ith row of pixels is less than or equal to the preset brightness value, determining that the ith row of pixels belongs to the black edge area; and when the average value of the brightness of the ith row of pixels is greater than the preset brightness value, determining that the ith row of pixels does not belong to the black edge area.

In some possible embodiments, the determining the boundary line according to the black-edge region includes:

when the ith row of pixels belongs to the black edge area and the (i + 1) th row of pixels does not belong to the black edge area, determining the ith row of pixels or the (i + 1) th row of pixels as the boundary line; when the ith row of pixels does not belong to the black edge region and the (i + 1) th row of pixels belongs to the black edge region, determining the ith row of pixels or the (i + 1) th row of pixels as the boundary; when the ith row of pixels belongs to the black edge area and the (i + 1) th row of pixels does not belong to the black edge area, determining the ith row of pixels or the (i + 1) th row of pixels as the boundary; and when the ith row of pixels does not belong to the black edge area and the (i + 1) th row of pixels belongs to the black edge area, determining the ith row of pixels or the (i + 1) th row of pixels as the boundary.

In some possible embodiments, the at least two dividing AUs have equal sizes, and the size of the reference AU is the minimum coding size that can be supported by the current coding protocol, so that when the reference AU is used to code the dividing line region, abnormal horizontal stripes with irregular sizes do not occur. In some possible embodiments, the size of the reference AU is the minimum coding size that can be supported by the current coding protocol, so that the total area of the abnormal horizontal stripes is further reduced to achieve the optimal visual effect.

In some possible embodiments, the determining a boundary region from the boundary includes:

dividing the image frame to be coded by taking the first AU as a unit to obtain at least two first AUs, wherein the size of each first AU is larger than that of the reference AU; dividing each first AU containing the pixels in the boundary in the at least two first AUs by taking the reference AU as a unit to obtain at least two reference AUs; and determining all AUs containing the pixels in the boundary line in the at least two reference AUs as boundary lines AUs, and determining a region formed by all the boundary lines AUs as the boundary line region.

The image frame to be coded is divided by using the first AU with larger size as a unit, and each first AU containing the pixels in the boundary is divided by using the reference AU as a unit instead of directly using the reference AU as a unit, so that the excessive fine division of an effective area or a black edge area is avoided, and the calculation amount of a system is reduced.

dividing the image frame to be coded by taking a first AU as a unit to obtain at least two first AUs, wherein the size of the first AU is larger than that of the reference AU, dividing each first AU, which contains the pixels in the boundary, in the at least two first AUs by taking a second AU as a unit to obtain at least two second AUs, wherein the second AU is an AU, the size of which is larger than that of the reference AU and smaller than that of the first AU, dividing each second AU, which contains the pixels in the boundary, in the at least two second AUs by taking the reference AU as a unit to obtain at least two reference AUs, and determining all the reference AUs, which contain the pixels in the boundary, in the at least two reference AUs as the boundary AU and a region formed by the boundaries of all the AUs as the boundary region. The image frame to be coded is divided by using the first AU with larger size as a unit, and each first AU containing the pixels in the boundary is divided by using the reference AU as a unit instead of directly using the reference AU as a unit, so that the excessive fine division of an effective area or a black edge area is avoided, and the calculation amount of a system is reduced.

In some possible embodiments, after the dividing the image frame to be encoded by taking the first AU as a unit to obtain at least two first AUs, the method further includes: and coding all first AUs which belong to the black border area and do not overlap with the boundary area in the at least two first AUs by taking the first AU as a unit. The black border area does not need to be finely divided, and the black border area is coded by taking AUs with the largest size as possible as a unit, so that the visual effect is not influenced, and the calculation amount during coding is relatively reduced.

In some possible implementations, the size of the first AU is 2^ Q1 times as long as the size of the reference AU, and the width of the first AU is 2^ Q1 times as wide as the size of the reference AU, where Q1 is a positive integer.

In some possible embodiments, the first AU is 64 pixels long and wide; when the image frame to be coded is an I frame in the video to be coded, the size of the reference AU is 4 pixels in length and width; when the image frame to be coded is not an I frame in the video to be coded, the size of the reference AU is 8 pixels in length and width, so that the dividing AU can be continuously subdivided according to the reference AU.

In some possible embodiments, after dividing each first AU including the pixel in the boundary in units of second AUs to obtain at least two second AUs, the method further includes: and coding all second AUs which belong to the black border area and do not overlap with the boundary area in the at least two second AUs by taking the second AU as a unit, wherein the black border area does not need to be finely divided, and the black border area is coded by taking AUs with the largest size as possible as a unit, so that the visual effect is not influenced, and the calculated amount during coding is reduced relatively.

In some possible embodiments, the size of the first AU is 2^ Q2 times as long as the size of the second AU and 2^ Q2 times as wide as the size of the second AU, wherein Q2 is a positive integer; the length of the second AU is 2^ Q3 times the length of the reference AU, and the width of the second AU is 2^ Q3 times the width of the reference AU, wherein Q3 is a positive integer; q1 is the sum of Q2 and Q3.

Since the length of the size of the first AU is 2^ Q2 times the length of the size of the second AU, the width of the size of the first AU is 2^ Q2 times the width of the size of the second AU, and Q2 is a positive integer, it can be determined that the first AU can be divided into an integer of regions of the same size in units of the second AU; the length of the size of the second AU is 2^ Q3 times the length of the size of the reference AU, the width of the size of the second AU is 2^ Q3 times the width of the size of the reference AU, and Q3 is a positive integer, it can be determined that the second AU can be divided into an integer of regions of the same size in units of the reference AU without occurrence of a missing region.

In some possible embodiments, the second AU has a size of 32 pixels long and 32 pixels wide, or 16 pixels long and 16 pixels wide.

In some possible embodiments, the image frame to be encoded may be divided in units of AU _0, the size of AU _0 is 2^ k times as long as the size of the reference AU, and is 2^ k times as wide as the size of the reference AU, where k is a positive integer, and let j have an initial value of 1, and the following determination process is performed to determine the boundary area:

dividing each AU _ j-1 containing the dividing line in the image frame to be coded by taking AU _ j as a unit to obtain at least two AU _ j, wherein the length of the size of the AU _ j is 2 times of the length of the size of the AU _ j-1, the width of the AU _ j is 2 times of the width of the size of the AU _ j-1, and judging the sizes of j and k; when j is smaller than k, j is updated to be j +1, and the determining process is executed again; when j is equal to k, the determination process ends.

In some possible embodiments, after dividing each AU _ j-1 containing the dividing line in the image frame to be encoded by the unit of AU _ j to obtain at least two AU _ j, the method further includes: and coding each AU _ j area belonging to the black edge area in the image frame to be coded by taking the AU _ j as a unit.

In some possible embodiments, the determining a boundary region from the boundary includes: dividing the image frame to be encoded by taking AU-16 as a unit to obtain P1 AU-16, wherein the AU-16 is an AU with the length of 16 pixels and the width of 16 pixels; dividing all AU-16 including the boundary pixel in the P1 AU-16 by taking AU-8 as a unit, wherein each AU-16 including the boundary pixel is divided into 4 corresponding AU-8, and the AU-8 is an AU with the length of 8 pixels and the width of 8 pixels; dividing all AU-8 including the boundary pixel in the AU-8 by taking AU-4 as a unit, wherein each AU-8 including the boundary pixel is divided into 4 corresponding AU-4, and the AU-4 is an AU with the length of 4 pixels and the width of 4 pixels; all AU-4 including the boundary pixel in the AU-4 is determined as a boundary region.

As the characteristic of the black edge area is aimed at, 4 AU-4 in the AU-8 do not need to be calculated one by one for position judgment in each step, 2 AU-4 can be directly determined to be used as a part of the boundary area, and the calculation steps are saved.

A second aspect of the present application provides an image encoding device including:

and the boundary determining module is used for determining a boundary of a first area and a second area which are adjacent in the image frame to be coded, wherein the first area and the second area have color difference. In some possible embodiments, the image frame to be encoded is an image that can be displayed on a display in the form of a pixel combination. It should be noted that the image frame to be encoded may be a photo, a drawing, a video screenshot, or the like, and in some possible embodiments, may also be an image of a frame in a video, which is not limited herein. It should be noted that, when displayed on the display, the image frame to be encoded may be a rectangle, a circle, a polygon or other irregular figure, which is not limited herein. In some possible embodiments, the image frame to be encoded may be displayed on a spherical display, or may be displayed on a dome-shaped display, or other curved-screen display, or other irregular-shaped display, which is not limited herein. In the embodiments of the present application, a description will be given taking an example in which a rectangular image is displayed on a display of a rectangular screen.

A boundary region determining module, configured to determine a boundary region according to the boundary, where the boundary region includes at least two non-overlapping boundary region units AU, each of the at least two boundary AUs includes a part of the boundary, and the boundary region includes all of the boundaries, and the size of each of the at least two boundary AUs is smaller than or equal to the size of a reference AU, and the size of the reference AU is smaller than the maximum coding size that can be supported by a current coding protocol used for coding the image frame to be coded.

In the embodiment of the present application, after the boundary of the image frame to be encoded is determined, the boundary region may be determined based on the boundary. It should be noted that the AU is used to divide the image frame to be encoded, and divide the image frame to be encoded into a plurality of small areas, which are referred to as AUs. Such as a rectangular AU with a size of 64x64, which can be denoted as AU-64, wherein each small region has a size of 64x64, which can be denoted as AU-64. In the embodiments of the present application, there may be a plurality of AU with different sizes, such as 32 × 32, 16 × 16, or 8 × 8, which is not limited herein. In some possible embodiments, the shape of AU may also be rectangular, hexagonal, etc., and is not limited herein. In some possible embodiments, the AU may be a coding unit in the protocol h.264 or the protocol h.265, such as a length 64 × 64, 32 × 32, 16 × 16, or 8 × 8, which is not limited herein.

And the coding module is used for coding the boundary area by taking the boundary AU as a unit.

Because the size of any one of the boundary AUs is smaller than or equal to the size of the reference AU, and the size of the reference AU is smaller than the maximum coding size which can be supported by the current coding protocol, after the boundary region is coded by taking the boundary AU as a unit, compared with the maximum coding size which can be supported by the current coding protocol, the abnormal stripe with a smaller area appears in the effective region near the boundary, or the abnormal stripe does not appear, thereby improving the visual effect.

In some possible embodiments, the image frame to be encoded is any one frame in a video to be encoded, the first region is a black-edge region, and the black-edge region is a region covered by a black edge of the video to be encoded in the image frame to be encoded.

In some possible embodiments, the demarcation line determination module includes:

a black edge area determining submodule for determining the black edge area in the image frame to be coded; and the boundary determining submodule is used for determining the boundary according to the black edge area.

In some possible embodiments, the black-edge region determination submodule is specifically configured to:

calculating the average brightness value of the ith row of pixels in the image frame to be coded, and comparing the average brightness value of the ith row of pixels with a preset brightness value, wherein i is a non-negative integer; when the average brightness value of the ith row of pixels is less than or equal to the preset brightness value, determining that the ith row of pixels belongs to the black edge area; when the average brightness value of the ith row of pixels is greater than the preset brightness value, determining that the ith row of pixels does not belong to the black edge area;

In some possible embodiments, the demarcation line determination submodule is specifically configured to:

In some possible embodiments, the at least two dividing AUs are equal in size, and in some possible embodiments, the reference AU is the smallest coding size that can be supported by the current coding protocol, so that when the reference AU is used to code the boundary region, abnormal horizontal stripes with irregular sizes do not occur. In some possible embodiments, the size of the reference AU is the minimum coding size that can be supported by the current coding protocol, so that the total area of the abnormal horizontal stripes is further reduced to achieve the optimal visual effect.

In some possible embodiments, the boundary region determination module is specifically configured to: dividing the image frame to be coded by taking the first AU as a unit to obtain at least two first AUs, wherein the size of each first AU is larger than that of the reference AU;

dividing each first AU containing the pixels in the boundary in the at least two first AUs by taking the reference AU as a unit to obtain at least two reference AUs; and determining all AUs containing the pixels in the boundary line in the at least two reference AUs as boundary lines AUs, and determining a region formed by all the boundary lines AUs as the boundary line region.

In some possible embodiments, the boundary region determination module is specifically configured to:

dividing the image frame to be coded by taking the first AU as a unit to obtain at least two first AUs, wherein the size of each first AU is larger than that of the reference AU; dividing each first AU including the pixels in the boundary in the at least two first AUs by taking a second AU as a unit to obtain at least two second AUs, wherein the second AU is an AU of which the size is larger than that of the reference AU and smaller than that of the first AU; dividing each second AU containing the pixels in the boundary in the at least two second AUs by taking the reference AU as a unit to obtain at least two reference AUs; and determining all reference AUs containing the pixels in the boundary line from the at least two reference AUs as a boundary line AU, and determining a region formed by all the boundary line AUs as a boundary line region.

In some possible embodiments, the encoding module is further configured to:

and coding all first AUs which belong to the black edge area and do not overlap with the boundary area in the at least two first AUs by taking the first AU as a unit, wherein the black edge area does not need to be finely divided, and the black edge area is coded by taking AUs with the largest size as possible as a unit, so that the visual effect is not influenced, and the calculated amount during coding is reduced relatively.

In some possible embodiments, the encoding module is further configured to:

and coding all second AUs which belong to the black border area and do not overlap with the boundary area in the at least two second AUs by taking the second AU as a unit, wherein the black border area does not need to be finely divided, and the black border area is coded by taking AUs with the largest size as possible as a unit, so that the visual effect is not influenced, and the calculated amount during coding is reduced relatively.

In some possible embodiments, the boundary region determination module is further configured to:

the length of the first AU is 2^ Q2 times the length of the second AU, and the width of the first AU is 2^ Q2 times the width of the second AU, wherein Q2 is a positive integer; the length of the second AU is 2^ Q3 times the length of the reference AU, and the width of the second AU is 2^ Q3 times the width of the reference AU, wherein Q3 is a positive integer; q1 is the sum of Q2 and Q3, since the length of the first AU is 2^ Q2 times the length of the second AU, the width of the first AU is 2^ Q2 times the width of the second AU, and Q2 is a positive integer, it can be determined that the first AU can be divided into an integer of the same size of area in units of the second AU; the length of the size of the second AU is 2^ Q3 times the length of the size of the reference AU, the width of the size of the second AU is 2^ Q3 times the width of the size of the reference AU, and Q3 is a positive integer, it can be determined that the second AU can be divided into an integer of regions of the same size in units of the reference AU without occurrence of a missing region.

dividing the image frame to be encoded by taking AU-16 as a unit to obtain P1 AU-16, wherein the AU-16 is an AU with the length of 16 pixels and the width of 16 pixels; dividing all AU-16 including the boundary pixel in the P1 AU-16 by taking AU-8 as a unit, wherein each AU-16 including the boundary pixel is divided into 4 corresponding AU-8, and the AU-8 is an AU with the length of 8 pixels and the width of 8 pixels; dividing all AU-8 including the boundary pixel in the AU-8 by taking AU-4 as a unit, wherein each AU-8 including the boundary pixel is divided into 4 corresponding AU-4, and the AU-4 is an AU with the length of 4 pixels and the width of 4 pixels; all AU-4 including the boundary pixel in the AU-4 is determined as a boundary region.

The present application also provides an image encoding apparatus including:

a communication port and at least one processor; the communication port is used for acquiring an image frame to be encoded and sending the image frame to the at least one processor; the at least one processor configured to perform the steps of:

determining a boundary line between a first area and a second area which are adjacent in an image frame to be coded, wherein the first area and the second area have color difference; determining a boundary region according to the boundary, wherein the boundary region comprises at least two non-overlapping boundary region units (AUs), each of the at least two boundary AUs comprises a part of the boundary, the boundary region comprises all the boundaries, the size of each of the at least two boundary AUs is smaller than or equal to that of a reference AU, the size of the reference AU is smaller than the maximum coding size which can be supported by a current coding protocol used for coding the image frame to be coded; the boundary area is encoded in units of the boundary AU.

In some possible embodiments, in the step of determining a boundary between adjacent first and second regions in the image frame to be encoded, the at least one processor is further configured to perform the steps of:

determining the black edge area in the image frame to be coded; the boundary is determined based on the black border region.

In some possible embodiments, in the step of determining the black border region in the image frame to be encoded, the at least one processor is further configured to perform the following steps:

calculating the average brightness value of the ith row of pixels in the image frame to be coded, and comparing the average brightness value of the ith row of pixels with a preset brightness value, wherein i is a non-negative integer; when the average brightness value of the ith row of pixels is less than or equal to the preset brightness value, determining that the ith row of pixels belongs to the black edge area; when the average brightness value of the ith row of pixels is greater than the preset brightness value, determining that the ith row of pixels does not belong to the black edge area; or, calculating the brightness average value of the ith row of pixels in the image frame to be coded, and comparing the brightness average value of the ith row of pixels with a preset brightness value, wherein i is a non-negative integer; when the average brightness value of the ith row of pixels is less than or equal to the preset brightness value, determining that the ith row of pixels belongs to the black edge area; and when the average value of the brightness of the ith row of pixels is greater than the preset brightness value, determining that the ith row of pixels does not belong to the black edge area.

In some possible embodiments, in the step of determining the boundary from the black-edge region, the at least one processor is further configured to:

when the ith row of pixels belongs to the black edge area and the (i + 1) th row of pixels does not belong to the black edge area, determining the ith row of pixels or the (i + 1) th row of pixels as the boundary line; when the ith row of pixels does not belong to the black edge region and the (i + 1) th row of pixels belongs to the black edge region, determining the ith row of pixels or the (i + 1) th row of pixels as the boundary; when the ith row of pixels belongs to the black edge area and the (i + 1) th row of pixels does not belong to the black edge area, determining the ith row of pixels or the (i + 1) th row of pixels as the boundary; or, when the ith row of pixels does not belong to the black border region and the (i + 1) th row of pixels belongs to the black border region, determining the ith row of pixels or the (i + 1) th row of pixels as the boundary.

In some possible embodiments, in the step of determining a demarcated area according to the demarcations, the at least one processor is further configured to:

In some possible embodiments, after the step of dividing the image frame to be encoded by the first AU to obtain at least two first AUs, the at least one processor is further configured to:

In some possible embodiments, after the step of dividing, for each first AU including the pixel in the boundary, the second AU into at least two second AUs, the at least one processor is further configured to:

A fourth aspect of the present application provides a computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of the above-described aspects.

According to the technical scheme, the implementation mode of the application has the following advantages:

in the application, when a video to be coded is coded at a low code rate, a boundary between a first area and a second area which are adjacent in a picture frame to be coded is determined, the boundary area comprises at least two non-overlapping boundary area units AU, each boundary AU of the at least two boundary AUs comprises a part of the boundary, the boundary area comprises all the boundaries, the size of each boundary AU of the at least two boundary AUs is smaller than or equal to the size of a reference AU, the size of the reference AU is smaller than the maximum coding size which can be supported by a current coding protocol, the current coding protocol is used for coding the picture frame to be coded, after the boundary area is coded by taking the boundary AU as a unit, compared with the maximum coding size which can be supported by the current coding protocol, abnormal stripes with smaller areas appear on an effective area near the boundary, or abnormal stripes do not appear, thereby improving the visual effect.

Drawings

FIG. 1-1 is a schematic diagram of a system for image encoding and display;

fig. 1-2 are schematic structural views of a mobile phone;

FIGS. 1-3 are schematic diagrams of a server architecture;

FIGS. 1-4 are schematic diagrams of different divisions;

FIGS. 1-5 are schematic views of images having black-edged regions;

fig. 1-6 are schematic diagrams of coding the boundary of a black-edge region by a CU with a larger size;

FIGS. 1-7 are schematic diagrams of encoded images;

FIGS. 1-8 are schematic views of abnormal horizontal lines;

FIG. 2-1 is a diagram illustrating an image encoding method;

fig. 2-2 is a schematic diagram of an image frame to be encoded with a black cat;

2-3 are schematic illustrations of the boundary of a first region and a second region of an image frame to be encoded having a black border;

FIGS. 2-4 are schematic views of a borderline region of an image of a black cat;

FIGS. 2-5 are schematic diagrams of boundaries of image frames to be encoded;

FIGS. 2-6 are schematic diagrams of boundary regions of image frames to be encoded;

FIGS. 2-7 are schematic views of an image frame to be encoded without abnormal horizontal stripes;

FIG. 3-1 is a diagram illustrating an image encoding method;

FIG. 3-2 is a schematic diagram of determining a black border region;

3-3 are schematic views of the black border region;

FIGS. 3-4 are schematic views of active areas;

FIGS. 3-5 are schematic diagrams of defining a demarcation region;

FIGS. 3-6 are schematic diagrams of a screen being partitioned using LCUs;

FIGS. 3-7 are schematic diagrams of dividing an image frame to be encoded using an LCU;

fig. 3 to 8 are schematic diagrams illustrating dividing of the first AU including the boundary in the at least one first AU by the reference AU;

FIGS. 3-9 are schematic diagrams of defining a demarcation region;

FIGS. 3-10 are further schematic diagrams of defining a demarcation region;

FIGS. 3-11 are further schematic diagrams of defining a demarcation region;

FIGS. 3-12 are further schematic diagrams of defining a demarcation region;

FIGS. 3-13 are further schematic diagrams of defining a demarcation region;

FIGS. 3-14 are further schematic diagrams of defining a demarcation region;

fig. 3-15 are schematic diagrams illustrating that the area of each AU with the size of 64 × 64, which includes the boundary, in the image frame to be encoded is divided in units of AUs with the size of 32 × 32;

FIGS. 3-16 are further schematic diagrams illustrating the determination of a demarcation region;

FIGS. 3-17 are further schematic diagrams of defining a demarcation region;

FIGS. 3-18 are schematic diagrams of AUs that determine valid regions;

fig. 3-19 are further schematic diagrams of an AU for determining a valid region;

fig. 3-20 are further schematic diagrams of an AU for determining a valid region;

FIGS. 3-21 are schematic diagrams of implementations of determining a demarcation region;

FIGS. 3-22 are schematic diagrams of the division of the image frame to be encoded in AU-16 units;

FIGS. 3-23 are schematic diagrams of the P1 AU-16 including the AU-16 in the boundary divided by AU-8;

FIGS. 3-24 are schematic diagrams of the relationship of 4 AU-8 to the dividing line;

FIGS. 3-25 are schematic diagrams of the division of the 2 AU-8 in AU-4 units;

FIGS. 3-26 are schematic diagrams of defining a demarcation region;

FIG. 4-1 is a schematic diagram of an implementation of an image encoding apparatus;

FIG. 4-2 is a schematic diagram of another implementation of an image encoding apparatus;

fig. 5 is a schematic diagram of an implementation of an image encoding apparatus.

Detailed Description

The implementation mode of the application provides an image coding method and equipment, which are used for determining a boundary region in an image frame to be coded and coding the boundary region.

In order to make the technical solutions of the embodiments of the present application better understood by those skilled in the art, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, and not all embodiments.

The terms "first," "second," "third," "fourth," and the like in the description and in the claims of the present application and in the drawings described above, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be implemented in other sequences than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Referring to fig. 1-1, an image encoding and displaying system 100 includes an image encoding apparatus 101 and an image display apparatus 102. The image encoding device 101 is configured to encode an image frame to be encoded and send the encoded image frame to the image display device 102, and the image display device 102 is configured to receive the encoded image frame to be encoded and display the encoded image frame to be encoded.

In some possible implementations, the image encoding apparatus 101 and the image display apparatus 102 may be both a terminal and a server, and the image display apparatus 102 is not limited herein as long as the image encoding apparatus 101 has the capability of encoding an image and the image display apparatus 102 has the capability of displaying an image. The image encoding device may obtain an encoded image after encoding, and may display the encoded image by itself without transmitting the encoded image to the image display device, which is not limited herein.

A terminal as referred to in implementations of the present application may be a device that provides voice and/or data connectivity to a user, a handheld device having wireless connection capability, or other processing device connected to a wireless modem. Wireless terminals, which may be mobile terminals such as mobile telephones (or "cellular" telephones) and computers having mobile terminals, such as portable, pocket, hand-held, computer-included, or vehicle-mounted mobile devices, may communicate with one or more core networks via a Radio Access Network (RAN Radio Access Network) to exchange speech and/or data with the Radio Access Network. For example, Personal Communication Service (PCS) phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, Personal Digital Assistants (PDAs), and the like. A wireless Terminal may also be referred to as a system, a Subscriber Unit (Subscriber Unit), a Subscriber Station (Subscriber Station), a Mobile Station (Mobile), a Remote Station (Remote Station), an Access Point (Access Point), a Remote Terminal (Remote Terminal), an Access Terminal (Access Terminal), a User Terminal (User Terminal), a Terminal Device, a User Agent (User Agent), a User Device (User Device), or a User Equipment (User Equipment).

Taking a mobile phone as an example, please refer to fig. 1-2 (which is a schematic structural diagram of the mobile phone), the mobile phone 110 includes: radio Frequency (RF) circuitry 1110, memory 1120, input unit 1130, display unit 1140, sensors 1150, audio circuitry 1160, wireless fidelity (WiFi) module 1170, processor 1180, and power supply 1190. Those skilled in the art will appreciate that the handset configuration shown in fig. 1-2 is not intended to be limiting and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.

The server 120, as referred to in the subject implementations, as shown in fig. 1-3 (which are schematic diagrams of the server architecture), may include one or more Central Processing Units (CPUs) 1222 (e.g., one or more processors) and memory 1232, one or more storage media 1230 (e.g., one or more mass storage devices) that store applications 1242 or data 1244. Memory 1232 and storage media 1230 can be, among other things, transient storage or persistent storage. The program stored in the storage medium 1230 may include one or more modules (not shown), each of which may include a series of instruction operations for the server. Still further, the central processor 1222 may be configured to communicate with the storage medium 1230, to execute a series of instruction operations in the storage medium 1230 on the server 120.

The server 120 may also include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input-output interfaces 1258, and/or one or more operating systems 1241, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

In this implementation, encoding refers to data compression of images combined by pixels by a specific technique to reduce storage space and facilitate transmission. It should be noted that the encoding technique is commonly used in images of a video, where one frame in the video can be regarded as one video frame image. Currently, the International universal Video Coding algorithm standards are commonly established by the International Telecommunication Union Telecommunication standards institute (International Telecommunication Union Telecommunication standardization Sector, ITU-T) and the Moving Picture Experts Group (MPEG) as h.264/Advanced Video Coding (AVC) and h.265/High performance Video Coding (HEVC), in which blocks of different sizes are specified to be divided, and different Coding modes and different Coding tools can be used for the blocks of different sizes.

Taking the protocol h.265 as an example, it is specified that one image can be encoded in units of Coding Units (CU), where the sizes of available Coding Units (CU) may be 64x64, 32x32, 16x16, 8x8, and 4x 4. Here, the Largest Coding Unit (LCU) can be used as a basic Unit of Coding, and is usually 64 × 64. After the partition in units of LCUs in one frame is determined, the partition in units of CUs of different sizes, such as 32x32, 16x16, 8x8 or 4x4, is performed on a single LCU. As shown in fig. 1-4 (schematic diagrams of different division modes), the image is divided by using coding units of different sizes as units according to needs, so as to obtain different division modes.

In the implementation manner of the present application, in order to meet the requirements of reducing the amount of calculation in the encoding process and ensuring the high-quality visual effect, all possible partition manners may be traversed for the image in the video, and the Distortion and the bitrate of each combination are calculated, so as to calculate Rate Distortion Optimization (RDO), where the calculation formula is:

J＝D+λ·R

wherein J is the final measurement data, D is the distortion measurement data, lambda is the Lagrange multiplier, and R is the data volume measurement data. Different division modes are compared one by one, and the division mode which enables the value of J to be minimum, namely the best balance point which reduces the calculation quantity and the high-quality visual effect is found out to be the optimal division mode.

However, as shown in fig. 1 to 5 (which are schematic diagrams of images having black-edge regions), when the image of the video having the black-edge regions is encoded at a low bit rate, the division manner of the image is determined by the above method, and there may be a region included in a CU (e.g., LCU) having a large size and having both pixels of the black-edge regions and pixels of the effective region, as shown in fig. 1 to 6 (which are schematic diagrams of encoding the boundaries of the black-edge regions by CUs having a large size). The region included in the CU is encoded to obtain an encoded image, and as shown in fig. 1 to 7 (which are schematic diagrams of encoded images), an abnormal streak having a large area appears in the effective region near the boundary line, as shown in fig. 1 to 8 (which are schematic diagrams of abnormal horizontal streaks), resulting in a poor visual effect.

In view of the above, an image encoding method according to an embodiment of the present invention is provided with reference to fig. 2-1, and the method includes:

201. and determining a boundary line of a first area and a second area which are adjacent in the image frame to be coded, wherein the first area and the second area have color difference.

In some possible implementations, the image frame to be encoded is an image that can be displayed on a display in the form of a combination of pixels. It should be noted that the image frame to be encoded may be an image of one frame in a video, or may be a video screenshot, which is not limited herein. It should be noted that, when the image frame to be encoded is displayed on the display, the image frame to be encoded may be a rectangle, a circle, a polygon, or other irregular figure, which is not limited herein.

In some possible implementations, the image frame to be encoded may be displayed on a spherical display, a dome-shaped display, or other curved-screen display, or other irregular-shaped display, which is not limited herein. In the implementation of the present application, a description will be given taking an example in which a rectangular image is displayed on a display of a rectangular screen.

In some possible implementations, in a region of a predetermined size centered on any pixel of the boundary, a difference between the color parameter of the first region and the color parameter of the second region is greater than or equal to a predetermined color difference value. As shown in fig. 2-2 (which is a schematic diagram of an image frame to be encoded with a black cat), there is a black cat in the image frame to be encoded, an area of a black hair of the black cat on the image frame to be encoded is a first area, and a background outside a body of the black cat is a second area.

It should be noted that the first region and the second region are not coincident and are adjacent to each other, i.e. there is a boundary line, which has at least one pixel, and any pixel in the boundary line is adjacent to at least one pixel in the first region and at least one pixel in the second region at the same time, as shown in fig. 2-3 (which is a schematic diagram of the boundary line of the first region and the second region of the image frame to be encoded with a black border), and the body of the black cat has a distinct boundary line with the background. Note that the pixels in the boundary may belong to the first region or the second region, and this is not limited here.

In an implementation of the present application, for any pixel in the boundary, such as the pixel Q, a pixel belonging to the first area exists in an area of a predetermined size centered on the pixel Q, and a pixel belonging to the second area also exists in the area. In the area with the pixel Q as the center and the preset size, the difference value between the color parameter of the pixel belonging to the first area and the color parameter of the pixel of the second area is larger than or equal to the preset color difference value. The region of the predetermined size may be a rectangle having a length of 5 pixels and a width of 5 pixels (denoted by 5 × 5) centered on the pixel in the boundary, may be a rectangle having a length of 3 × 3, may be a rectangle having unequal lengths of length and width (for example, 3 × 5), and is not limited herein. In some possible implementations, the area with the predetermined size may also be a circle with any one pixel in the boundary as a dot, and the area with the predetermined size is a circle with a fixed length (e.g., 3 or 5 pixels) as a radius, which is not limited herein. From another perspective, in the vicinity of the boundary, the first region and the second region have a significant color difference, which may be a difference in brightness, a difference in gray scale value, a difference in red (R), green (G), and blue (B) in RGB, or other types of color differences, and is not limited herein.

In some possible implementations, for any one pixel in the image frame to be encoded, it may be determined whether there are two portions having a color difference value greater than or equal to a preset color difference value in an area with a preset size, with the pixel as a center, and one of the two portions is adjacent to the pixel and the other portion includes the pixel. If so, the pixel is determined to be a pixel of the boundary line, and the two parts are respectively a part of the first area and a part of the second area. In some possible implementations, the boundary between the first region and the second region in the image frame to be encoded may be determined by traversing all pixels in the image frame to be encoded in the manner described above.

202. And determining a boundary area according to the boundary, wherein the boundary area comprises at least two boundary Area Units (AUs) which do not overlap with each other, each of the at least two boundary AUs comprises a part of the boundary, the boundary area comprises the whole boundary, the size of each of the at least two boundary AUs is smaller than or equal to that of a reference AU, the size of the reference AU is smaller than the maximum coding size which can be supported by a current coding protocol used for coding the image frame to be coded.

In an implementation of the present application, after a boundary of an image frame to be encoded is determined, a boundary region may be determined according to the boundary. In some possible implementations, the size of the boundary AU may be different, such as 32 × 32, 16 × 16, or 8 × 8, which is not limited herein. In some possible implementations, the shape of the AU may also be rectangular, hexagonal, etc., and is not limited herein. In some possible implementations, the AU may be a CU in the protocol h.264 or the protocol h.265, such as a length 64 × 64, 32 × 32, 16 × 16, or 8 × 8, and is not limited herein as long as it is smaller than the maximum coding size that can be supported by the current coding protocol. It should be noted that, if the protocol h.264 or the protocol h.265 is used as the current encoding protocol, the maximum supported encoding size is 64 × 64. In some possible implementations, the boundary region may be determined by dividing the image frame to be encoded into several regions using a plurality of AUs with different sizes, and then using a region including any pixel in the boundary as one region of the boundary region.

In this implementation, all AUs in the boundary AU may have the same size, that is, the same size as the reference AU, which may be a unit in which the boundary area is encoded. The dividing line region includes N reference AUs in units of the reference AU, where N is an integer greater than 1, and the dividing line is included in the dividing line region.

It should be noted that, in some feasible implementations, the size of the reference AU is half the length of the maximum coding size that can be supported by the current coding protocol, and the width is half the width of the maximum coding size that can be supported by the current coding protocol, so as to avoid the situation that the size of the reference AU is equal to the maximum coding size that can be supported by the current coding protocol. The reference AU may have a size of 32 × 32, as the maximum code size that can be supported by the current coding protocol is 64 × 64.

When all of the division AUs are reference AUs, as shown in fig. 2 to 4 (which are schematic diagrams of boundary regions of images of black cats), an image frame to be encoded can be divided into 18 columns of small regions of 19 lines, which are referred to as reference AUs, in units of reference AUs. After the dividing mode of the image frame to be encoded is determined, all reference AUs are traversed, and a reference AU including the boundary can be determined as the boundary area.

In some possible implementations, the size of the reference AU is the minimum coding size that can be supported by the current coding protocol, for example, in the protocol h.265, for an image of a normal frame in a video, the minimum coding size that can be supported by the current coding protocol is 8 × 8; as in protocol h.264, the minimum coding size that can be supported by the current coding protocol is 4x4 for pictures of a normal frame in a video. In some possible implementations, whether protocol h.264 or protocol h.265, the minimum coding size that can be supported by the current coding protocol is 4x4 for pictures of an I-frame in a video.

It should be noted that, in some feasible implementation manners, the image frame to be encoded may be directly divided by the reference AU to obtain a plurality of reference AUs, or may be divided by a larger AU to obtain a plurality of regions, and then all the regions including the pixels in the boundary in the plurality of regions are divided by a smaller AU until the image frame is divided by the reference AU, which is not limited herein. For example, the image frame to be encoded is divided by using an AU of 64 × 64 as a unit to obtain a plurality of AU-64 (an area with a size of 64 × 64), and then the AU-64 including the pixels in the boundary is divided by using an AU with a size of 32 × 32 or 16 × 16 as a unit, and so on until the image frame is divided by using a reference AU (e.g., 8 × 8 or 4 × 4).

203. The boundary area is encoded in units of the boundary AU.

In this implementation, when the image frames to be encoded in fig. 1 to 5 are encoded at a low bit rate, the boundary shown in fig. 2 to 5 (which is a schematic diagram of the boundary of the image frames to be encoded) may be determined, and then the boundary region may be determined according to the boundary shown in fig. 2 to 6 (which is a schematic diagram of the boundary region of the image frames to be encoded). In the present application, when encoding a video to be encoded at a low code rate, a boundary between a first region and a second region adjacent to each other in an image frame to be encoded is determined, where the first region and the second region have a color difference, a boundary region is determined according to the boundary, the boundary region includes mutually non-overlapping demarcation AUs, each AU in the demarcation AUs includes a part or all of the boundary, and the boundary is included in the boundary region, since the size of any one AU in the demarcation AUs is smaller than or equal to the size of a reference AU, and the size of the reference AU is smaller than the maximum encoding size that can be supported by a current encoding protocol, after the boundary region is encoded by taking the demarcation AU as a unit, the effect shown in fig. 2-7 (schematic diagram without abnormal horizontal stripes for the image frame to be encoded) is obtained, and encoding is performed by using the maximum encoding size that can be supported by the current encoding protocol, abnormal stripes having a small area appear in the effective region near the boundary, or abnormal stripes do not appear, thereby improving the visual effect.

Note that in some videos, there is a video black border. The so-called video black border is that besides the normal picture, the image displayed in the video picture has more black borders around the original photographed picture, and the black borders are caused by the conversion of the original picture and the displayed picture due to the different size specifications. Generally, when a video to be encoded has a video black border, the image of each frame in the video to be encoded has a video black border, and the positions of the images are the same. In an implementation manner of the present application, the image frame to be encoded is any one frame of a video to be encoded, the first area is a black border area, the black border area is an area covered by a video black border of the video to be encoded in the image frame to be encoded, the second area is an effective area, and the effective area is an area of the image frame to be encoded except the black border area. In the effective area, an abnormal horizontal streak occurs near the boundary between the black border area and the effective area, which results in poor viewing effect.

In order to alleviate or eliminate the influence of the black border region on the effective region in the image frame to be encoded, please refer to fig. 3-1, the present application provides an implementation manner of an image encoding method, which includes:

301. determining a black edge area in the image frame to be encoded, wherein the black edge area is an area covered by a video black edge of the video to be encoded in the image frame to be encoded, and the first area is the black edge area.

In this implementation manner, the video to be encoded may be a documentary, an animation, a tv show, or a movie, which is not limited herein. It should be noted that the video to be encoded may be a video combined in units of frames, that is, a video to be encoded includes at least 2 frames, and in the fast frame conversion (for example, 60 frames per 1 second), the video effect is achieved. In some possible implementations, the video to be encoded may be played on a mobile phone, a television, a large screen, a dome screen, or a movie (imax), which is not limited herein.

In some possible implementations, taking the video to be encoded played on the display as an example, the image of each frame of the video to be encoded is rectangular. The display may have a screen with a ratio of 16:9 in length to width, but the old version of the television screen has a ratio of 4:3 in length to width, so the images in the old version of the television show have a ratio of 4:3 in length to width. When a video with the length-width ratio of 4:3 is played on a screen with the length-width ratio of 16:9, if the video is displayed normally, black edge areas appear on the left side and the right side of the screen; on the contrary, if the video with the aspect ratio of 16:9 is played on the screen with the aspect ratio of 4:3, if the video is normally displayed, black areas appear on the upper and lower sides of the screen. Therefore, in some possible implementations, there may be black-edge regions on all 4 sides of the rectangular screen, which is not limited herein.

In the implementation of the present application, since the shape of the black-edge area has a certain rule, i.e., a rectangle on four sides, the boundary (i.e., the boundary) of the black-edge area is parallel to the boundary of the screen. It should be noted that, since the black edge area is pure black, the brightness of the black edge area is generally smaller than a fixed value (a preset brightness value), and the application utilizes this feature to determine the black edge area.

Specifically, referring to fig. 3-2, for the image frame to be encoded shown in fig. 1-5, in order to determine the black border region, the boundary pixels of the image frame to be encoded may be the 0 th row or column pixels, the i +1 th row pixels are adjacent to the i th row pixels, and the i +1 th column pixels are adjacent to the i th column pixels, where i is a non-negative integer, and the initial value of i is 0, the following determination process is performed to determine the black border region:

s1, calculating the brightness average value of the ith row of pixels in the image frame to be coded, and comparing the brightness average value of the ith row of pixels with a preset brightness value; and calculating the brightness average value of the ith row of pixels in the image frame to be coded, and comparing the brightness average value of the ith row of pixels with a preset brightness value, wherein i is a non-negative integer.

In some possible implementations, a user input preset brightness, such as 24 candelas per square meter, may be received, and other values may be set, which are not limited herein. In some possible implementations, the average value of the luminance of the pixels in the ith row or column may be calculated, for example, the luminance of all the pixels in the 0 th row is obtained, and the average value is calculated. After obtaining the average value of the brightness of the pixel of the ith row or column, judging whether the pixel of the ith row or column belongs to the black edge area. In some possible implementations, the determination may also be performed by an RBG value or a gray value, which is not limited herein.

S2, when the average value of the brightness of the ith row of pixels is less than or equal to the preset brightness value, determining that the ith row of pixels belongs to the black edge area, and executing the judging process again; when the average brightness value of the ith row of pixels is less than or equal to the preset brightness value, determining that the ith row of pixels belongs to the black edge area, and executing the judging process again.

In this implementation manner, when the average value of the luminance of the pixels in the ith row or column is less than or equal to a preset luminance value (e.g., 24 candelas per square meter), it is determined that the pixels in the ith row or column belong to the black-edge area, otherwise, it is determined whether the pixels in the ith row or column do not belong to the black-edge area but belong to the active area. When it is determined that the pixel of the ith row or column belongs to the active area, the determination process may be ended. In some possible implementations, the location of the ith row or column may be denoted as Black _ up, Black _ down, Black _ left, Black _ right. Specifically, if the upper edge of the image frame to be encoded has a Black edge region, the ith line of pixels counted from the upper boundary of the image frame to be encoded is marked as Black _ up; if the lower edge of the image frame to be coded has a Black edge area, marking the ith row of pixels counted from the lower boundary of the image frame to be coded as Black _ down; if the left side of the image frame to be coded has a Black edge area, marking the ith row of pixels counted from the left boundary of the image frame to be coded as Black _ left; if there is a Black border area on the right side of the image frame to be encoded, the ith row of pixels counted from the right border of the image frame to be encoded is marked as Black _ right.

In some possible implementations, the average luminance value for each row or column may be determined by scanning row-by-row or column from the upper, lower, left, and right boundaries, respectively, to the horizontal centerline. The average value of luminance is a value obtained by dividing the luminance value of each pixel in the row or column by the number of the row or column. If there are 500 pixels in the row or column and the luminance values are 500 values such as 23, 24, 25, 26, and 21 …, the luminance of the 500 pixels is averaged to obtain the luminance average value. If 22 is obtained, the pixel of the row or the column is determined to belong to the black border area, and if 25 or more is obtained, the pixel of the row or the column is determined to belong to the effective area.

In this implementation, after it is determined that the pixel in the ith row or column belongs to the black border region, i may be updated to be the original i plus 1, and the determination process is executed again. The black border region is determined through the above steps, and the black border region shown in fig. 3-3 (which is a schematic diagram of the black border region) is obtained. In some possible implementations, the calculation of the above steps may be performed once per frame, or may be performed only for an I frame, or may be performed once every first frame of a whole second, or may be performed once every 5 seconds, which is not limited herein. In some possible implementations, since the black border region and the valid region of each frame of the video to be encoded are located at the same position, the calculation may also be performed on the first frame of the video to be encoded, which is not limited herein.

S3, when the average value of the brightness of the ith row of pixels is larger than the preset brightness value, determining that the ith row of pixels does not belong to the black edge area, and ending the judging process; and when the average value of the brightness of the ith row of pixels is greater than the preset brightness value, determining that the ith row of pixels does not belong to the black edge area, and ending the judging process.

In an implementation manner of the present application, if it is determined that the average luminance value of the pixels in the ith row or column is greater than the preset luminance value, it is determined that the ith row or column does not belong to the black-edge region, and because the black-edge region and the effective region are distinct, if the ith row or column does not belong to the black-edge region, it belongs to the effective region, and the determination process may be ended, and the black-edge region is the pixels in the 0 th row or column to the pixels in the ith row or column.

302. The boundary is determined based on the black border region.

In the implementation manner of the present application, the effective region is a region where a viewer views the expression information of the image frame to be encoded or the video to be encoded, and in the image frame to be encoded having the black border region, the effective region is an image frame to be encoded other than the black border region, and after the black border region is determined, as shown in fig. 3-4 (which is a schematic diagram of the effective region), the effective region may be determined, and then a boundary between the black border region and the effective region may be determined.

In this application, since the black edge area is black and has a luminance smaller than the predetermined luminance value, that is, the luminance in the active area is greater than or equal to the predetermined luminance value, and since the boundary (i.e., the boundary) of the black edge area is parallel to the boundary of the screen, 2 adjacent rows or columns of pixels may be determined, and if the average luminance of 1 row or column is smaller than the predetermined luminance value and the average luminance of the other 1 row or column is not smaller than the predetermined luminance value, it is known that 1 row or column of the 2 rows or columns of pixels belongs to the black edge area, and the other 1 row or column of pixels is an active area, one row of the 2 rows or columns of pixels may be determined as the boundary.

In the implementation of S1-S3 in step 301, when the average luminance value of the pixels in the ith row or column is less than or equal to the predetermined luminance value and the average luminance value of the pixels in the (i + 1) th row or column is greater than the predetermined luminance value, the pixels in the ith row or column, or the pixels in the (i + 1) th row or column are determined as the boundary. It should be noted that the boundary may be in the form of a pixel combination, for example, the boundary includes a set of 600 pixels, such as { (X1, Y1), (X2, Y2), … …, (X600, Y600) }. In some possible implementations, the pixels of the boundary may belong to the black border region or the active region, which is not limited herein.

303. A boundary region is determined based on the boundary.

In an implementation of the present application, the boundary area is N reference AUs in units of the reference AU, where N is an integer greater than 1, the boundary is included in the boundary area, and a size of the reference AU is smaller than or equal to a size of the reference AU. In some possible implementations, the image frame to be encoded may be divided in units of a larger AU, and when a region including pixels in the boundary is divided by the reference AU, the AU may be gradually reduced until the reference AU, which is not limited herein.

In this implementation manner, please refer to fig. 3-4, in order to determine a schematic diagram of the boundary region, different size strategies may be adopted to divide the image frame to be encoded so as to obtain the boundary region, and the following four strategies are discussed separately:

first, first AU- > base AU- > boundary region.

1.1, dividing the image frame to be coded by taking the first AU as a unit to obtain at least two first AUs, wherein the size of each first AU is larger than that of the reference AU.

Specifically, please refer to fig. 3-5, which are schematic diagrams illustrating an implementation of determining the boundary region.

In this implementation manner, the size of the first AU may be 64 × 64, 32 × 32, or 16 × 16, and taking the encoding standard of the protocol h.265 as an example, the image frame to be encoded is generally divided by using a Largest Coding Unit (LCU), as shown in fig. 3-6 (which is a schematic diagram of dividing a screen by using an LCU), so as to obtain at least one first AU. As shown in fig. 3 to 6, if the first AU is an LCU, at least one first AU covers a black border area and a valid area. It should be noted that the size of the first AU is larger than that of the reference AU, for example, the size of the first AU is 64 × 64, and the reference AU is 8 × 8, so that at least one first AU can be further subdivided according to the reference AU. After the image frames to be encoded shown in fig. 1-5 are divided by the above method, the dividing manners shown in fig. 3-7 (schematic diagrams for dividing the image frames to be encoded by using the LCU) can be obtained.

And 1.2, coding all first AUs which belong to the black border area and do not overlap with the boundary area in the at least two first AUs by taking the first AU as a unit.

In the implementation of the present application, as shown in fig. 3 to 7, the first AUs in the first row from top to bottom all belong to the black border region, and since the black border region does not need to be finely encoded, the largest AU can be used for encoding. Because of the black border area, the coding is not required to be too fine, and the coding is carried out by selecting the AU with the maximum size which can be used, so that the visual effect of the black border area is not influenced, and the calculation amount is effectively reduced.

And 1.3, dividing each first AU containing the pixels in the boundary in the at least two first AUs by taking a second AU as a unit to obtain at least two second AUs, wherein the second AU is an AU with the size larger than that of the reference AU and smaller than that of the first AU.

In this implementation, in the at least one first AU, as shown in fig. 3 to 7, a boundary is located in a plurality of first AUs in the second line, and thus as shown in fig. 3 to 8 (which is a schematic diagram of dividing all first AUs including the boundary in the at least one first AU in units of the reference AU), the plurality of first AUs in the second line may be divided in units of the reference AU. Wherein each first AU may be divided into a corresponding at least two reference AUs. If the size of the first AU is 64 × 64 and the size of the reference AU is 8 × 8, each first AU can be divided into 64 corresponding reference AUs; if the first AU has a size of 64 × 64 and the reference AU has a size of 4 × 4, each of the first AUs can be divided into 256 reference AUs.

And 1.4, dividing each second AU containing the pixels in the boundary line in the at least two second AUs by taking the reference AU as a unit to obtain at least two reference AUs.

In the implementation of the present application, as shown in fig. 3 to 9 (which are schematic diagrams for determining a boundary area), all of the at least two reference AUs including the pixel in the boundary may be determined as the boundary area. It should be noted that the length of the first AU is 2^ Q1 times the length of the reference AU, and the width is 2^ Q1 times the width of the reference AU, where Q1 is a positive integer, and the size of the first AU is 64 pixels long and 64 pixels wide; when the image frame to be coded is an I frame in the video to be coded, the size of the reference AU is 4 pixels long and 4 pixels wide; when the image frame to be encoded is not an I-frame in the video to be encoded, the size of the reference AU is 8 pixels long and 8 pixels wide. That is, when the size of the first AU is 64 × 64 and the size of the reference AU is 8 × 8, Q1 is equal to 3; when the size of the first AU is 64 × 64 and the size of the reference AU is 4 × 4, Q1 is equal to 4. In some possible implementations, when the image frame to be encoded is an I frame in the video to be encoded, the size of the reference AU is 4 × 4, otherwise, the size of the reference AU is 8 × 8, and the first AU in units of the first AU may be divided into exactly 64 or 128 reference AUs in units of the reference AU. It should be noted that an I frame is also called an intra-frame prediction frame, and the I frame is usually the first frame of each Group of Pictures (GOP) and can be used as a reference point for random access after being compressed moderately. Since the length of the size of the first AU is 2^ Q1 times the length of the size of the reference AU, the width of the size of the first AU is 2^ Q1 times the width of the size of the reference AU, and Q1 is a positive integer, it can be determined that the first AU can be divided into an integer of regions of the same size in units of the reference AU, no missing region occurs, and the shape of the reference AU is the same as that of the first AU.

And 1.5, determining all reference AUs containing the pixels in the boundary line in the at least two reference AUs as the boundary line AU, and determining a region formed by all the boundary line AUs as the boundary line region.

And for the first AU belonging to the black-edge area in the at least one first AU, coding is carried out by taking the first AU as a unit, so that the black-edge area is prevented from being divided finely, and therefore, the black-edge area is coded by taking the AU with the largest size as possible as a unit, namely, the visual effect is not influenced, and the calculation amount during coding is reduced.

Two, a first AU- > a second AU- > a reference AU- > a boundary region.

Referring to fig. 3-10, in order to determine another schematic diagram of the boundary region, in some possible implementations, after the image frame to be encoded is divided in units of first AUs, the image frame is not divided in units of reference AUs, but may be transited by a second AU with an intermediate size, so that the effective region or the black-edge region is further prevented from being divided in a fine manner, and the amount of calculation of the system is reduced.

And 2.1, dividing the image frame to be coded by taking the first AU as a unit to obtain at least two first AUs, wherein the size of each first AU is larger than that of the reference AU.

Step 2.1 is the same as step 1.1, and is not described herein.

And 2.2, coding all first AUs which belong to the black border area and do not overlap with the boundary area in the at least two first AUs by taking the first AU as a unit.

Step 2.2 is the same as step 1.2, and is not described herein.

And 2.3, dividing each first AU containing the pixels in the boundary in the at least two first AUs by taking a second AU as a unit to obtain at least two second AUs, wherein the second AU is an AU with the size larger than that of the reference AU and smaller than that of the first AU.

As shown in fig. 3-11 (another schematic diagram for defining the boundary region), when the size of the first AU is 64 × 64 and the size of the reference AU is 8 × 8, the size of the second AU may be 32 × 32 or 16 × 16, which is not limited herein. When the size of the second AU is 32 × 32, dividing each first AU including the pixel in the boundary line in the at least one first AU by taking the second AU as a unit to obtain 4 second AUs; when the size of the second AU is 16 × 16, 16 second AUs are obtained.

It should be noted that the length of the first AU may be 2^ Q2 times the length of the second AU, and the width may be 2^ Q2 times the width of the second AU, i.e. when the second AU has a size of 32 × 32, Q2 equals 1, and when the second AU has a size of 16 × 16, Q2 equals 2.

It should be noted that, since the width of the size of the first AU is 2^ Q2 times the length of the size of the second AU, the width of the size of the first AU is 2^ Q2 times the width of the size of the second AU, and Q2 is a positive integer, it can be determined that the first AU can be divided into an integer of the same size of the second AU in units of the second AU, the length of the size of the second AU is 2^ Q3 times the length of the size of the reference AU, and the width is 2^ Q3 times the width of the size of the reference AU, where Q3 is a positive integer, and Q1 is the sum of Q2 and Q3, it can be determined that the second AU can be divided into an integer of the same size of the reference AU in units of the reference AU without missing regions. For example, the second AU has a size of 32 pixels long and 32 pixels wide, or a size of 16 pixels long and 16 pixels wide. If the size of the first AU is 64 × 64, the size of the second AU is 32 × 32, and the size of the reference AU is 8 × 8, then Q1 is equal to 3, Q2 is equal to 2, and Q3 is equal to 1, then the region in units of the first AU can be exactly divided into 4 or 8 regions in units of the second AU, and then the region in units of the second AU can be exactly divided into 16 or 64 regions in units of the reference AU.

And 2.4, coding all second AUs which belong to the black border area and do not overlap with the boundary area in the at least two second AUs by taking the second AUs as units.

For the second AU belonging to the effective area in the at least two second AUs, the division may be performed by using the above-mentioned division manner for the effective area, which is not described herein again. The black border area does not need to be finely divided, and the black border area is coded by taking AUs with the largest size as possible as a unit, so that the visual effect is not influenced, and the calculation amount during coding is relatively reduced.

And 2.5, dividing each second AU containing the pixels in the boundary line in the at least two second AUs by taking the reference AU as a unit to obtain at least two reference AUs.

In the present implementation, as shown in fig. 3 to 12 (which are another schematic diagrams for determining the boundary region), for each of the at least two second AUs including the pixel in the boundary, the division may be performed in units of the reference AU. In an implementation manner of the present application, after the division, at least two corresponding reference AUs may be obtained, for example, the size of the second AU is 32 × 32, the size of the reference AU is 8 × 8, and the size of the first AU is 64 × 64.

In this implementation manner, after at least two reference AUs including the pixel in the boundary are determined in the at least two second AUs, as shown in fig. 3 to 11, the reference AU belonging to the black-edge area in the at least two reference AUs may be encoded by using the reference AU as a unit, which is not described herein again. It should be noted that, for the reference AU belonging to the effective area in the at least two reference AUs, the division and encoding are performed in the above-mentioned manner for dividing the effective area, which is not described herein again. The black border area does not need to be finely divided, and the black border area is coded by taking AUs with the largest size as possible as a unit, so that the visual effect is not influenced, and the calculation amount during coding is relatively reduced.

And 2.6, determining all reference AUs containing the pixels in the boundary line in the at least two reference AUs as the boundary line AU, and determining a region formed by all the boundary line AUs as the boundary line region.

In the implementation manner of the present application, after the image frame to be encoded is divided by using the first AU with a larger size as a unit, each first AU including the pixel in the boundary in the at least one first AU may be divided by using the second AU with a middle size as a unit, and then divided by using the reference AU as a unit, thereby further avoiding performing too fine division on the effective area or the black-edge area, and reducing the calculation amount of the system.

As shown in fig. 3 to 13 (another schematic diagram for determining the boundary area), when at least two reference AUs corresponding to the division are determined in units of reference AUs, it can be determined that all reference AUs including the pixel in the boundary belong to the boundary area.

And three, AU _0- > AU _1- > AU _1- > … - > base AU (AU _ k) - > boundary region.

Please refer to fig. 3-14, which are another schematic diagrams illustrating the determination of the boundary region.

In some possible implementations, as shown in fig. 3-7, the image frame to be encoded may be divided in units of AU _0 to obtain N _0 AU _0, where the length of the size of AU _0 is 2^ k times the length of the reference AU, the width of the AU _0 is 2^ k times the width of the reference AU, k is a positive integer, let j have an initial value of 1, and the following determination process is performed to determine the boundary area:

and T1, dividing each AU _ j-1 containing the dividing line in the image frame to be coded by taking AU _ j as a unit to obtain N _ j AU _ j areas corresponding to each AU _ j-1 in each AU _ j-1 containing the dividing line, wherein the length of the size of AU _ j is 2 times of the length of the size of AU _ j-1, and the width of AU _ j is 2 times of the width of the size of AU _ j-1.

In the implementation manner of the present application, the image frame to be encoded is divided by using AU _0 of the maximum encoding size that can be supported by the current encoding protocol as a unit to obtain N _0 AU _0, an area that further includes a boundary is divided by using AU _1 that is one level smaller than AU _0 as a unit to divide the N _1 AU _1, and an area that includes a boundary is divided by using AU _2 that is one level smaller as a unit until division is performed by using a reference AU (AU _ k). In an implementation of the present application, the size of AU _0 is greater than the size of AU _ k, where AU _ k is the reference AU and k is a positive integer.

As shown in fig. 3-15 (which is a schematic diagram of dividing, by an AU of size 32 × 32, each area of the image frame to be encoded, which includes the boundary and is based on an AU of size 64 × 64), each AU _ j-1(AU _0) of the image frame to be encoded, which includes the boundary, is divided by an AU _ j (AU _1) to obtain N _ j (4 AU _1) corresponding to each AU _ j-1(AU _0) including the boundary. From the reference AU of 8 × 8, the size of AU _0 is 64 × 64, the size of AU _1 is 32 × 32, the size of AU _2 is 16 × 16, the size of AU _3 is 8 × 8, and k is equal to 3.

Updating j to be j plus 1, and executing the determining process again because j is equal to 2 after updating and is smaller than k. Therefore, the division of the image frame to be coded for each AU _ j-1(AU _1) containing the boundary is performed in units of AU _ j (AU _2), as shown in fig. 3-16 (which is another schematic diagram for determining the boundary region), so as to obtain N _ j AU _ j (4 AU _2) corresponding to each of the AU _ j-1(AU _1) containing the boundary.

Updating j to the original j plus 1, and re-executing the determining process because the updated j is equal to 3 and is equal to k. Therefore, the image frame to be coded is divided into the respective AU _ j-1(AU _2) containing the dividing line by the unit of AU _ j (AU _3), and N _ j AU _ j areas (4 AU _3) corresponding to each of the respective AU _ j-1(AU _2) containing the dividing line are obtained.

And coding each AU _ j area belonging to the black edge area in the image frame to be coded by taking the AU _ j as a unit.

T2, determining the sizes of j and k.

Ending the determination process when j is greater than or equal to k; when j is smaller than k, j is updated to j +1, and the determination process is re-executed.

Updating j to the original j plus 1, and since j equals 4, is greater than k, the loop is ended, resulting in the demarcation regions shown in fig. 3-17 (which is another schematic diagram of defining the demarcation regions).

It should be noted that, each AU belonging to the black border region in the image frame to be encoded is encoded by taking the AU belonging to the black border region as shown in fig. 3 to 17 as a unit, for example, 4 AUs in the first line are encoded by taking 64 × 64 as AU-64, and for example, 16 AUs in the second line are encoded by taking 16 × 16 as AU-16, which is not described herein again.

After the image frame to be coded is divided by using AU _0 with a larger size as a unit, the AU containing the boundary is divided by AUs with a smaller size by one level step by step until the division is performed by using the reference AU as the unit, so that the excessively detailed division of an effective area or a black edge area is avoided to the maximum extent, and the calculation amount of a system is reduced.

In some possible implementations, for an AU belonging to an effective area in at least one first AU and an AU belonging to the effective area in the at least two reference AUs, traversing all possible partition manners, calculating a manner of calculating distortion and a bit rate of each combination, and further calculating a value of RDO, and comparing different partition manners one by one, the calculation formula is as follows: j ═ D + λ · R, and the optimal division method is found as the division method that minimizes the value of J, i.e., the optimal balance point that reduces the amount of computation and the high-quality visual effect. Wherein J is the final measurement data, D is the distortion measurement data, lambda is the Lagrange multiplier, and R is the data volume measurement data.

Taking an AU of 8 by 8 size as an example, as shown in fig. 3-18 (schematic diagrams of AUs defining the active area), the left figure is a 64 by 64 AU, which contains the boundary line, and an AU of 8 by 8 is located near the boundary line. Here, by calculating the RDO value of the 8 × 8 AU, the 16 × 16 AU (shown in the right diagram of fig. 3 to 18) where the 8 × 8 AU is located can be determined, and it can be seen that the 16 × 16 AU includes a boundary line, and therefore, in the implementation of the present application, it is not necessary to further calculate the RDO value of the 16 × 16 AU, and encoding is performed by directly using 8 × 8 as the AU. Meanwhile, the same operation is performed for other 8 × 8 AUs not including the boundary among the 16 × 16 AUs, that is, 8 × 8 is directly used as an AU for encoding.

As shown in fig. 3-19 (another schematic diagram of the AU defining the active area), the top left figure is a 64x64 AU, which contains the boundary. For 18 × 8 AU from the 64 × 64 AUs shown in fig. 3-19, 16 × 16 AUs (shown in the upper right of fig. 3-19) where the 8 × 8 AU is located can be determined, and since the 16 × 16 AU does not include a boundary, 32 × 32 AUs (shown in the lower of fig. 3-19) where the 16 × 16 AU is located can be determined, it can be seen that the 32 × 32 AU includes a boundary. Thus, in the present implementation, the RDO values of the 4 AU 8x8 and the 16 AU 16 are compared to determine the better partition of the 16 AU without further RDO values of the 32 AU 32x32 AU. After calculation by the formula (J ═ D + λ · R), RDO values (J) of the 4 8 × 8 AUs are determined, the sum of RDO values (J) of the 4 8 × 8 AUs is determined, RDO values (J) of the 16 × 16 AUs are determined, and the sum of RDO values (J) of the 4 8 × 8 AUs is compared with RDO value (J) of the 16 × 16 AU. If the sum of RDO values (J) of the 4 AU 8 × 8 is small, the AU 16 × 16 is divided into the 4 AU 8 × 8, otherwise the AU 16 × 16 is divided into the 16 AU 16 × 16. Meanwhile, the same operation is performed on another 16 × 16 AU not including the boundary among the 32 × 32 AUs, so as to select a better coding division method.

As shown in fig. 3-20 (another schematic diagram of the AU defining the active area), the top left figure is a 64x64 AU, and likewise, the 64x64 AU contains the boundary. For 18 × 8 AU among the 64 × 64 AUs, 16 × 16 AUs (shown in the upper right of fig. 3 to 20) where the 8 × 8 AU is located may be determined, and since the 16 × 16 AU does not include a boundary, 32 × 32 AUs (shown in the lower left of fig. 3 to 20) where the 16 × 16 AU is located may be determined, and since the 32 × 32 AU does not include a boundary, 64 × 64 AU (shown in the lower left of fig. 3 to 20) where the 32 × 32 AU is located may be determined, and it may be seen that the 64 × 64 AU includes a boundary, and therefore, in the present implementation, the RDO values of different division modes of 4 32 × 32 AUs among the 64 AU may not be compared to determine the optimum division mode by calculating the RDO values of the 64 × 64 AU as the coding unit. Specifically, for 8 × 8 AUs shown in fig. 3 to 20, the RDO value of 16 × 16 AUs where the 8 × 8 AUs is located is calculated, and the RDO value of 4 8 × 8 AUs in the 16 × 16 AUs is calculated, and the division method with a smaller RDO value is preferred.

When the merits of encoding with 1 AU 16 × 16 and encoding with 4 AU 8 × 8 are determined, 32 × 32 AUs in which the AU 16 × 16 is located can be determined, and the optimal division manner of the other 3 AU 16 × 16 among the AU 32 × 32 can be further calculated. Further, the comparison between the quality of the 32 × 32 AU encoded by 1 32 × 32 AU and the quality of the encoded by 4 16 × 16 AU requires that the RDO value of any 16 × 16 AU among the 32 × 32 AUs is the value of the RDO of the optimum division method among the 16 × 16 AUs. If the optimal coding mode of the 16 × 16 AU is 1 AU 16 × 16, the RDO value of the 16 × 16 AU is equal to the RDO coded by 1 AU 16 × 16; if the optimal coding of the 16 × 16 AU is performed with 4 8 × 8 AUs, the RDO value of the 16 × 16 AU is equal to the sum of the RDO values encoded with 4 8 × 8 AUs. After the comparison of the above methods, the division modes as in fig. 1-4 can be obtained.

In other AU of 64 × 64 not including the boundary, the advantages and disadvantages of the division methods of 8 × 8, 16 × 16, 32 × 32, and 64 × 64 are directly compared to select the optimum division method without considering the influence of the boundary.

And IV, AU _16- > AU _8- > AU _4 (reference AU) - > boundary region.

Referring to fig. 3-21, an implementation of determining the boundary region is illustrated. In some possible implementations, when encoding using the protocol h.264, the partitioning may be performed by:

and 4.1, dividing the image frame to be coded by taking AU-16 as a unit to obtain P1 AU-16, wherein the AU-16 is an AU with the length of 16 pixels and the width of 16 pixels.

In the implementation manner of the present application, in the coding standard of the protocol h.264, one Macro Block (MB) is taken as the maximum coding unit, so that the image frame to be coded can be divided by taking AU-16 as the unit as shown in fig. 3-22 (which is a schematic diagram for dividing the image frame to be coded by taking AU-16 as the unit), so as to obtain P1 AU-16, where the AU-16 is an AU 16 pixels long by 16 pixels wide by 16 pixels, and the AU-16 is an AU 16 by taking AU-16 as the unit.

4.2, dividing all AU-16 including the boundary pixel in the P1 AU-16 by taking AU-8 as a unit, wherein each AU-16 including the boundary pixel is divided into 4 corresponding AU-8, and the AU-8 is an AU with the length of 8 pixels and the width of 8 pixels.

As can be seen from fig. 3-22, if the boundary in the image frame to be encoded is on the third row, and the plurality of AU-16 on the third row includes the boundary, as shown in fig. 3-23 (which is a schematic diagram of dividing AU-16 in the P1 AU-16 by AU-8), the plurality of AU-16 in the P1 AU-16 including any pixel of the boundary are divided by AU-8, so as to obtain 4 AU-8 corresponding to each AU-16 in the plurality of AU-16, where the AU-8 is an AU with a length of 8 pixels and a width of 8 pixels.

4.3, determining all AU-8 including the boundary pixel in the AU-8.

In an implementation manner of the present application, after obtaining 4 AU-8 corresponding to each AU-16 in the plurality of AU-16, for the 4 AU-8, since the black border area is generally 4-sided, and only 2 AU-8 of the 4 AU-8 will include a boundary, a relationship between the 4 AU-8 and the boundary may be determined, and 2 AU-8 including any pixel in the boundary in the 4 AU-8 may be determined.

4.4, dividing all AU-8 containing the boundary pixel by taking AU-4 as a unit, wherein 4 AU-4 corresponding to each AU-8 in the 2 AU-8 take the AU-4 as the reference AU.

In the implementation manner of the present application, one of the 4 cases shown in fig. 3-24 (which is a schematic diagram of the relationship between the 4 AU-8 and the boundary) can be determined according to the relationship between the 4 AU-8 and the boundary, without the need of determining the relationship one by one AU-8. In the implementation manner of the present application, when it is determined that the second case is shown in fig. 3 to 24, each AU-8 of the 4 AU-8 may be directly divided in the second case as shown in fig. 3 to 24, so as to obtain the division manner as shown in fig. 3 to 25 (which is a schematic diagram of dividing the 2 AU-8 by taking AU-4 as a unit), and obtain the boundary area as shown in fig. 3 to 26 (which is a schematic diagram of determining the boundary area).

It should be noted that, for the 4 AU-8 and the AU-8 belonging to the black border area in the boundary, the coding may be performed by directly using AU-8 as a unit, that is, dividing the 2 AU-8 by using AU-4 as a unit to obtain 4 AU-4 corresponding to each AU-8 in the 2 AU-8, and using the AU-4 as the reference AU.

Note that the 4 AU-8 and AU-8 belonging to the black border area in the boundary can be encoded directly using AU-8 as a unit.

304. The boundary area is encoded in units of the boundary AU.

In the application, when a video to be coded is coded at a low code rate, a boundary line between a first area and a second area which are adjacent in a picture frame to be coded is determined, a boundary line area is determined according to the boundary line, when the boundary AUs are all the reference AUs, the boundary line area is N reference AUs which take the reference AUs as a unit, the boundary line is contained in the boundary line area, the ratio of the size of the reference AUs to the size of the picture frame to be coded is smaller than a preset ratio, wherein N is an integer larger than 1, because the size of any AU in the boundary AUs is smaller than or equal to the size of the reference AU and the size of the reference AU is smaller than the maximum coding size which can be supported by a current coding protocol, after the boundary line area is coded by taking the boundary AU as a unit, the coding is carried out by comparing the maximum coding size which can be supported by the current coding protocol, abnormal stripes having a small area appear in the effective region near the boundary, or abnormal stripes do not appear, thereby improving the visual effect.

Referring to fig. 4-1, a schematic diagram of an implementation of an image encoding apparatus is shown, where the image encoding apparatus 400 includes:

a boundary determining module 401, configured to determine a boundary between a first region and a second region that are adjacent to each other in the image frame to be encoded, where the first region and the second region have a color difference.

In some possible implementations, the image frame to be encoded is an image that can be displayed on a display in the form of a combination of pixels. It should be noted that the image frame to be encoded may be a photo, a drawing, a video screenshot, and the like, and in some feasible implementations, the image frame may also be an image of a frame in a video, which is not limited herein. It should be noted that, when displayed on the display, the image frame to be encoded may be a rectangle, a circle, a polygon or other irregular figure, which is not limited herein. In some possible implementations, the image frame to be encoded may be displayed on a spherical display, or may be displayed on a dome-shaped display, or other curved-screen display, or other irregular-shaped display, which is not limited herein. In the implementation of the present application, a description will be given taking an example in which a rectangular image is displayed on a display of a rectangular screen.

A dividing line region determining module 402 for determining a dividing line region according to the dividing line.

The boundary area comprises at least two boundary Area Units (AUs) which are not overlapped with each other, each boundary AU of the at least two boundary AUs comprises a part of the boundary, the boundary area comprises the whole boundary, the size of each boundary AU of the at least two boundary AUs is smaller than or equal to that of a reference AU, the size of the reference AU is smaller than the maximum coding size which can be supported by a current coding protocol, and the current coding protocol is used for coding the image frame to be coded.

In an implementation of the present application, after a boundary of an image frame to be encoded is determined, a boundary region may be determined according to the boundary. It should be noted that the AU is used to divide the image frame to be encoded, and divide the image frame to be encoded into a plurality of small areas, which are referred to as AUs. Such as a rectangular AU with a size of 64x64, which can be denoted as AU-64, wherein each small region has a size of 64x64, which can be denoted as AU-64. It should be noted that in the implementation of the present application, there may be AU with various sizes, such as 32 × 32, 16 × 16, or 8 × 8, which is not limited herein. In some possible implementations, the shape of the AU may also be rectangular, hexagonal, etc., and is not limited herein. In some possible implementations, the AU may be a coding unit in the protocol h.264 or the protocol h.265, such as a length 64 × 64, 32 × 32, 16 × 16, or 8 × 8, which is not limited herein.

An encoding module 403, configured to encode the boundary area by taking the boundary AU as a unit.

In some feasible implementation manners, the image frame to be encoded is any one frame in the video to be encoded, the first region is a black edge region, and the black edge region is a region covered by a video black edge of the video to be encoded in the image frame to be encoded.

In some possible implementations, referring to fig. 4-2, a schematic diagram of another implementation of an image encoding apparatus is shown, where the boundary determining module 401 includes:

a black edge region determining submodule 4011, configured to determine the black edge region in the image frame to be encoded; a boundary determining submodule 4012, configured to determine the boundary according to the black border area.

In some possible implementations, the black border region determining sub-module 4011 is specifically configured to:

In some possible implementations, the dividing line determining submodule 4012 is specifically configured to:

In some possible implementations, the at least two demarcation AUs are equal in size, and in some possible implementations, the reference AU is the smallest coding size that can be supported by the current coding protocol, so that when the reference AU is used to code the boundary region, abnormal horizontal stripes of irregular sizes do not occur. In some possible implementations, the size of the reference AU is the minimum coding size that can be supported by the current coding protocol, so that the total area of the abnormal horizontal stripes is further reduced to achieve the optimal visual effect.

In some possible implementations, the demarcation region determination module 402 is specifically configured to: dividing the image frame to be coded by taking the first AU as a unit to obtain at least two first AUs, wherein the size of each first AU is larger than that of the reference AU; dividing each first AU containing the pixels in the boundary in the at least two first AUs by taking the reference AU as a unit to obtain at least two reference AUs; and determining all AUs containing the pixels in the boundary line in the at least two reference AUs as boundary lines AUs, and determining a region formed by all the boundary lines AUs as the boundary line region.

In some possible implementations, the demarcation region determination module 402 is specifically configured to:

In some possible implementations, the encoding module 403 is further configured to:

In some possible implementations, the dividing line region determining module 402 is further configured to:

In some possible implementations, the image frame to be encoded may be divided in units of AU _0, the size of AU _0 is 2^ k times as long as the size of the reference AU, and is 2^ k times as wide as the size of the reference AU, where k is a positive integer, and let j have an initial value of 1, and the following determination process is performed to determine the boundary area:

In some possible implementations, after dividing each AU _ j-1 containing the dividing line in the image frame to be encoded by taking the AU _ j as a unit to obtain at least two AU _ j, the method further includes: and coding each AU _ j area belonging to the black edge area in the image frame to be coded by taking the AU _ j as a unit.

Referring to fig. 5, a schematic diagram of an implementation of an image encoding apparatus 500 is shown, including:

a communication port 501 and at least one processor 502; the communication port 501 is configured to acquire an image frame to be encoded, and send the image frame to the at least one processor 502; the at least one processor 502 is configured to perform the steps of:

In some possible implementations, in the step of determining a boundary between adjacent first and second regions in the image frame to be encoded, the at least one processor 502 is further configured to perform the following steps:

In some possible implementations, in the step of determining the black border region in the image frame to be encoded, the at least one processor 502 is further configured to perform the following steps:

In some possible implementations, in the step of determining the boundary from the black border region, the at least one processor 502 is further configured to:

In some possible implementations, in the step of determining a demarcated area according to the demarcator, the at least one processor 502 is further configured to:

In some possible implementations, after the step of dividing the image frame to be encoded by the first AU to obtain at least two first AUs, the at least one processor 502 is further configured to:

In some possible implementations, after the step of dividing in units of second AUs for each first AU including a pixel in the boundary to obtain at least two second AUs, the at least one processor 502 is further configured to perform the following steps:

The above implementations may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the procedures or functions described in the present application are wholly or partially generated. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method implementation manners, and are not described herein again.

In the several implementations provided in this application, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. For example, the above-described device implementations are merely illustrative, and for example, the division of the units is only one logical division, and in actual implementations, there may be other divisions, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the implementation scheme.

In addition, functional units in each implementation manner of the present application may be integrated into one processing unit, or each unit may exist alone physically, or at least two units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the implementation manner of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The above implementation manner is only used for illustrating the technical solution of the present application, and is not limited thereto; although the present application has been described in detail with reference to the foregoing implementations, those of ordinary skill in the art will understand that: the technical solutions described in the foregoing implementation manners may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present application.

Claims

1. An image encoding method, comprising:

determining a boundary line of a first area and a second area which are adjacent in an image frame to be coded, wherein the first area and the second area have color difference;

determining a boundary region according to the boundary, wherein the boundary region comprises at least two non-overlapping boundary region units (AUs), each of the at least two boundary AUs contains a part of the boundary, and the boundary region contains all the boundaries, the size of each of the at least two boundary AUs is smaller than or equal to that of a reference AU, the size of the reference AU is smaller than the maximum coding size which can be supported by a current coding protocol used for coding the image frame to be coded;

and encoding the boundary area in units of the boundary AU.

2. The method of claim 1, wherein the color difference between the first region and the second region comprises: the difference value between the color parameter of the first area and the color parameter of the second area is greater than or equal to a preset color difference value.

3. The method of claim 1, wherein the size of the reference AU is equal to half the length of the maximum coding size that can be supported by the current coding protocol, and wherein the size of the reference AU is equal to half the width of the maximum coding size that can be supported by the current coding protocol.

4. The method according to any one of claims 1 to 3, wherein the image frame to be encoded is any one of video frames to be encoded, the first area is a black border area, and the black border area is an area covered by a video black border of the video to be encoded in the image frame to be encoded.

5. The method of claim 4, wherein determining the boundary between the adjacent first region and the second region in the image frame to be encoded comprises:

determining the black edge area in the image frame to be coded;

and determining the boundary line according to the black edge area.

6. The method of claim 5, wherein the determining the black border region in the image frame to be encoded comprises:

calculating the brightness average value of the ith row of pixels in the image frame to be coded, and comparing the brightness average value of the ith row of pixels with a preset brightness value, wherein i is a non-negative integer; when the average value of the brightness of the ith row of pixels is less than or equal to the preset brightness value, determining that the ith row of pixels belongs to the black edge area; when the average brightness value of the ith row of pixels is larger than the preset brightness value, determining that the ith row of pixels does not belong to a black edge area;

or the like, or, alternatively,

calculating the average brightness value of the ith row of pixels in the image frame to be coded, and comparing the average brightness value of the ith row of pixels with a preset brightness value, wherein i is a non-negative integer; when the average value of the brightness of the ith row of pixels is less than or equal to the preset brightness value, determining that the ith row of pixels belongs to the black edge area; and when the average value of the brightness of the ith row of pixels is greater than the preset brightness value, determining that the ith row of pixels does not belong to the black edge area.

7. The method of claim 6, wherein determining the boundary line according to the black border region comprises:

when the pixel of the ith row belongs to the black edge area and the pixel of the (i + 1) th row does not belong to the black edge area, determining the pixel of the ith row or the pixel of the (i + 1) th row as the boundary line; or the like, or, alternatively,

when the pixel of the ith row does not belong to the black edge area and the pixel of the (i + 1) th row belongs to the black edge area, determining the pixel of the ith row or the pixel of the (i + 1) th row as the boundary line;

when the ith row of pixels belongs to the black edge area and the (i + 1) th row of pixels does not belong to the black edge area, determining the ith row of pixels or the (i + 1) th row of pixels as the boundary; or the like, or, alternatively,

and when the ith row of pixels does not belong to the black edge area and the (i + 1) th row of pixels belongs to the black edge area, determining the ith row of pixels or the (i + 1) th row of pixels as the boundary.

8. The method according to any of claims 4 to 7, wherein the at least two demarcation AUs are equal in size.

9. The method of claim 8, wherein the size of the reference AU is a minimum coding size that can be supported by the current coding protocol.

10. The method of claim 8 or 9, wherein determining the boundary region according to the boundary comprises:

dividing the image frame to be coded by taking a first AU as a unit to obtain at least two first AUs, wherein the size of each first AU is larger than that of the reference AU;

dividing each first AU, which contains the pixels in the boundary, of the at least two first AUs by taking the reference AU as a unit to obtain at least two reference AUs;

and determining all AUs containing the pixels in the boundary line in the at least two reference AUs as boundary lines AUs, and determining a region formed by all the boundary lines AUs as the boundary line region.

11. The method of claim 8 or 9, wherein determining the boundary region according to the boundary comprises:

dividing each first AU including the pixels in the boundary in the at least two first AUs by taking a second AU as a unit to obtain at least two second AUs, wherein the second AU is an AU of which the size is larger than that of the reference AU and smaller than that of the first AU;

dividing each second AU, which contains the pixels in the boundary, of the at least two second AUs by taking the reference AU as a unit to obtain at least two reference AUs;

and determining all reference AUs containing the pixels in the boundary line in the at least two reference AUs as boundary lines AUs, and determining a region formed by all the boundary lines AUs as the boundary line region.

12. The method according to claim 10 or 11, wherein after the dividing the image frame to be encoded by taking the first AU as a unit to obtain at least two first AUs, the method further comprises:

and encoding all first AUs belonging to the black-border region and not overlapping with the boundary region, among the at least two first AUs, in units of the first AU.

13. The method of claim 12, wherein the size of the first AU is 2^ Q1 times as long as the size of the reference AU and the width of the first AU is 2^ Q1 times as wide as the size of the reference AU, wherein Q1 is a positive integer.

14. The method according to any one of claims 10 to 13,

the length and width of the first AU are both 64 pixels; when the image frame to be coded is an I frame in the video to be coded, the size of the reference AU is 4 pixels in length and width; and when the image frame to be coded is not an I frame in the video to be coded, the size of the reference AU is 8 pixels in length and width.

15. The method of claim 11,

after dividing each first AU including the pixel in the boundary line in the at least one first AU by taking a second AU as a unit to obtain at least two second AUs, the method further includes:

and encoding all second AUs which belong to the black-border region and do not overlap with the boundary region, among the at least two second AUs, by using the second AU as a unit.

16. The method of claim 11 or 15,

the length of the first AU is 2^ Q2 times the length of the second AU, and the width of the first AU is 2^ Q2 times the width of the second AU, wherein Q2 is a positive integer;

the size of the second AU is 2^ Q3 times as long as the size of the reference AU and 2^ Q3 times as wide as the size of the reference AU, wherein Q3 is a positive integer;

q1 is the sum of Q2 and Q3.

17. The method of any of claims 11, 15 or 16, wherein the second AU has a size of 32 pixels long and 32 pixels wide, or 16 pixels long and 16 pixels wide.

18. The method of claim 8 or 9, wherein determining the boundary region according to the boundary comprises:

dividing the image frame to be encoded by taking AU-16 as a unit to obtain P1 AU-16, wherein the AU-16 is an AU with the length of 16 pixels and the width of 16 pixels;

dividing all AUs-16 including the boundary pixel in the P1 AUs-16 by taking AU-8 as a unit, wherein each AU-16 including the boundary pixel is divided into 4 corresponding AUs-8, and the AUs-8 are AUs with the length of 8 pixels and the width of 8 pixels;

dividing all AU-8 including the boundary pixel in the AU-8 by taking AU-4 as a unit, wherein each AU-8 including the boundary pixel is divided into 4 corresponding AU-4, and the AU-4 is an AU with a length of 4 pixels and a width of 4 pixels;

and determining all AU-4 comprising the boundary pixels in the AU-4 as boundary regions.

19. The method of any one of claims 1 to 3, wherein the sizes in the at least two demarcation AUs are equal.

20. The method of claim 19, wherein the size of the reference AU is a minimum code size that can be supported by a current coding protocol.

21. An image encoding device characterized by comprising:

the boundary determining module is used for determining a boundary of a first area and a second area which are adjacent in an image frame to be coded, wherein the first area and the second area have color difference;

a boundary region determining module, configured to determine a boundary region according to the boundary, where the boundary region includes at least two non-overlapping boundary region units AU, each of the at least two boundary AUs includes a part of the boundary, and the boundary region includes all of the boundaries, a size of each of the at least two boundary AUs is smaller than or equal to a size of a reference AU, the size of the reference AU is smaller than a maximum coding size that can be supported by a current coding protocol, and the current coding protocol is used to code the image frame to be coded;

22. The apparatus according to claim 21, wherein the image frame to be encoded is any one frame of a video to be encoded, the first area is a black border area, and the black border area is an area covered by a black border of the video to be encoded in the image frame to be encoded.

23. The apparatus of claim 22, wherein the demarcation line determination module comprises:

a black edge region determining submodule, configured to determine the black edge region in the image frame to be encoded;

and the boundary determining submodule is used for determining the boundary according to the black edge area.

24. The device of claim 23, wherein the black-edge region determination submodule is specifically configured to:

or the like, or, alternatively,

25. The device of claim 24, wherein the demarcation line determination submodule is specifically configured to:

26. The apparatus of any of claims 22 to 25, wherein the at least two demarcation AU are equal in size.

27. The apparatus of claim 26, wherein the reference AU has a size that is a minimum coding size that can be supported by the current coding protocol.

28. The device of claim 27, wherein the demarcation region determination module is specifically configured to:

29. The device according to claim 26 or 27, wherein the demarcation region determination module is specifically configured to:

30. The apparatus of claim 28 or 29, wherein the encoding module is further configured to:

31. The apparatus of claim 30, wherein the encoding module is further configured to:

32. The device according to claim 29 or 31, wherein the demarcation region determination module is further configured to:

q1 is the sum of Q2 and Q3.

33. The device according to claim 26 or 27, wherein the demarcation region determination module is specifically configured to:

34. An image encoding device characterized by comprising:

a communication port and at least one processor;

the communication port is used for acquiring an image frame to be encoded and sending the image frame to the at least one processor;

the at least one processor configured to perform the steps of:

and encoding the boundary area in units of the boundary AU.

35. The device according to any one of claims 34, wherein the image frame to be encoded is any one of video frames to be encoded, the first area is a black border area, and the black border area is an area covered by a video black border of the video to be encoded in the image frame to be encoded.

36. The device of claim 35, wherein in the step of determining a boundary between adjacent first and second regions in the image frame to be encoded, the at least one processor is further configured to:

determining the black edge area in the image frame to be coded;

and determining the boundary line according to the black edge area.

37. The device of claim 36, wherein in the step of determining the black border region in the image frame to be encoded, the at least one processor is further configured to:

or the like, or, alternatively,

38. The apparatus of claim 37, wherein in the step of determining the boundary from the black border region, the at least one processor is further configured to:

39. The apparatus of any of claims 35 to 38, wherein the at least two demarcation AUs are equal in size.

40. The apparatus of claim 39, wherein the reference AU has a size that is a minimum code size that can be supported by the current coding protocol.

41. The apparatus according to claim 39 or 40, wherein in the step of determining a demarcated area according to the demarcations, the at least one processor is further configured to perform the steps of:

42. The apparatus according to claim 39 or 40, wherein in the step of determining a demarcated area according to the demarcations, the at least one processor is further configured to perform the steps of:

43. The apparatus of claim 41 or 42, wherein after the step of dividing the image frame to be encoded by the first AU to obtain at least two first AUs, the at least one processor is further configured to:

44. The apparatus of claim 42, wherein after the step of dividing, for each first AU in the boundary first AU that contains a pixel in the boundary, in units of second AUs to obtain at least two second AUs, the at least one processor is further configured to:

45. The apparatus according to claim 39 or 40, wherein in the step of determining a demarcated area according to the demarcations, the at least one processor is further configured to perform the steps of:

46. A computer-readable storage medium having instructions stored therein, which when executed on a processor, cause one or more processors to perform the method of any one of claims 1-20.