CN112823378A

CN112823378A - Image depth information determination method, device, equipment and storage medium

Info

Publication number: CN112823378A
Application number: CN202080005529.6A
Authority: CN
Inventors: 高明明; 李泽祥; 李涛; 杨康; 张�林
Original assignee: SZ DJI Technology Co Ltd
Current assignee: SZ DJI Technology Co Ltd
Priority date: 2020-07-03
Filing date: 2020-07-03
Publication date: 2021-05-18
Also published as: WO2022000458A1

Abstract

An image depth information determination method, apparatus, device and storage medium, the method comprising: respectively obtaining a plurality of images related to an object through a shooting device; dividing a first image in a plurality of images into a plurality of blocks; determining a first matching cost aggregate value between pixels in a corresponding reference row of at least a first block of the plurality of blocks and corresponding pixels in a second image of the plurality of images; determining a second matching cost aggregate value between pixels in a corresponding reference column of the first block and corresponding pixels in a second image of the plurality of images; and determining a third matching cost aggregation value of the pixels of the first block in at least one cost aggregation direction according to the first matching cost aggregation value and the second matching cost aggregation value, so as to determine the depth information of the first block. By the method, the matching cost aggregate values of all pixels in the image matrix can be prevented from being stored, and the occupation of data read-write bandwidth can be greatly reduced.

Description

Image depth information determination method, device, equipment and storage medium

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for determining image depth information.

Background

The depth information of the shooting scene, that is, the distance between each object included in the shooting scene and the plurality of cameras is calculated through images shot by the plurality of cameras, and the method has important significance in many application scenes.

A process of calculating depth information provided by the related art includes: matching cost calculation, cost aggregation, parallax calculation and optimization and parallax refinement. For the step of cost aggregation, it is necessary to store the intermediate result of the cost aggregation process in the memory, and when the intermediate result is used, read the intermediate result from the external memory.

Because the data volume of the intermediate result is large, the data volume needing to be read and stored is large, and the process of cost aggregation can be completed only by occupying more read-write bandwidth.

Disclosure of Invention

The embodiment of the invention provides a method, a device, equipment and a storage medium for determining image depth information, which are used for reducing the read-write bandwidth occupied in the process of cost aggregation.

In a first aspect, an embodiment of the present invention provides an image depth information determining method, where the method includes:

respectively obtaining a plurality of images related to an object through a shooting device;

dividing a first image of the plurality of images into a plurality of blocks;

determining a first matching cost aggregate value between pixels in a corresponding reference row of at least a first block of the plurality of blocks and corresponding pixels in a second image of the plurality of images;

determining a second matching cost aggregate value between pixels in a corresponding reference column of the first block and corresponding pixels in a second image of the plurality of images;

determining a third matching cost aggregation value of the pixels of the first block in at least one cost aggregation direction according to the first matching cost aggregation value and the second matching cost aggregation value, so as to determine depth information of the first block;

wherein the first matching cost aggregation value and the second matching cost aggregation value have different cost aggregation directions.

Optionally, before determining a first matching cost aggregate value between pixels in a corresponding reference row of at least a first block of the plurality of blocks and corresponding pixels in a second image of the plurality of images, the method further comprises:

obtaining a first matching cost aggregate value between pixels in a reference row of at least one first block in the plurality of blocks and corresponding pixels in a second image in the plurality of images;

storing the first matching cost aggregate value in a storage space;

prior to determining a second matching cost aggregate value between a pixel in a reference column of the first block and a corresponding pixel in a second image of the plurality of images, the method further comprises:

acquiring a second matching cost aggregation value between pixels in the reference column of the first block and corresponding pixels in the second image;

storing the second matching cost aggregate value into the storage space.

Optionally, the determining, according to the first matching cost aggregation value and the second matching cost aggregation value, a third matching cost aggregation value of the corresponding pixel of the first block in at least one cost aggregation direction includes:

and determining third matching cost aggregation values of pixels in other rows and columns in the first block in at least one cost aggregation direction according to the first matching cost aggregation value and the second matching cost aggregation value.

Optionally, the determining, according to the first matching cost aggregation value and the second matching cost aggregation value, third matching cost aggregation values of pixels in other rows and columns in the first block in at least one price aggregation direction includes:

for the first block, according to the first matching cost aggregation value and the second matching cost aggregation value, determining third matching cost aggregation values of pixels in other rows and columns in the first block in at least one cost aggregation direction in a row-by-row and column-by-column mode.

and for the first block, determining third matching cost aggregation values of pixels in other rows and columns in the first block in at least one cost aggregation direction in a point-by-point mode according to the first matching cost aggregation value and the second matching cost aggregation value.

In a second aspect, an embodiment of the present invention provides an image depth information determining method, including:

acquiring matching cost aggregation values corresponding to pixels in a reference row and a reference column of an image matrix in a first group of cost aggregation directions, wherein the first group of cost aggregation directions are partial aggregation directions in a plurality of cost aggregation directions, the image matrix corresponds to images shot by a plurality of cameras, and the matching costs are used for measuring the similarity among corresponding pixels in the images shot by the plurality of cameras;

storing matching cost aggregation values corresponding to the pixels in the reference row and the reference column in a first group of cost aggregation directions into a storage space;

determining matching cost aggregation values corresponding to pixels in other rows and columns in the first set of cost aggregation directions according to matching cost aggregation values corresponding to the pixels in the reference row and the reference column in the first set of cost aggregation directions, wherein the other rows and columns are rows and columns in the image matrix except the reference row and the reference column;

determining matching cost aggregation values corresponding to pixels of the image matrix in a second set of cost aggregation directions, wherein the second set of cost aggregation directions are composed of aggregation directions which are not included in the first set of cost aggregation directions in the multiple cost aggregation directions;

determining a total matching cost aggregation value corresponding to each pixel according to the matching cost aggregation values corresponding to each pixel in the first group of cost aggregation directions and the second group of cost aggregation directions;

and determining the depth information of the image according to the total matching cost aggregation value corresponding to each pixel.

Optionally, the storing, into a storage space, matching cost aggregation values corresponding to pixels in the reference row and the reference column in the first set of cost aggregation directions includes:

storing matching cost aggregation values corresponding to first pixels in the reference row and the reference column in the first set of cost aggregation directions into a storage space, wherein the first pixels are located in the reference row and the reference column at the same time;

only storing matching cost aggregation values corresponding to the second pixels in the reference row in a first target cost aggregation direction into the storage space; wherein the second pixel is in the reference row and not in the reference column;

storing only matching cost aggregation values corresponding to third pixels in the reference column in a second target cost aggregation direction into the storage space, wherein the third pixels are in the reference column and not in the reference row;

the first set of cost aggregation directions includes the first target cost aggregation direction and the second target cost aggregation direction, and the first target cost aggregation direction and the second target cost aggregation direction are different.

Optionally, the determining matching cost aggregation values corresponding to pixels of the image matrix in the second set of cost aggregation directions includes:

acquiring matching cost aggregation values corresponding to pixels in a reference row and a reference column of the image matrix in a second group of cost aggregation directions;

and determining the matching cost aggregation values of the pixels in the other rows and columns in the second group of cost aggregation directions according to the matching cost aggregation values of the pixels in the reference row and the reference column in the second group of cost aggregation directions.

Optionally, the determining, according to the matching cost aggregation values corresponding to the pixels in the reference row and the reference column in the second set of cost aggregation directions, the matching cost aggregation values corresponding to the pixels in the other rows and columns in the second set of cost aggregation directions includes:

rotating the image matrix by a preset angle;

and determining the matching cost aggregation values of the pixels in the other rows and columns in the second group of cost aggregation direction according to the matching cost aggregation values of the pixels in the reference row and the reference column in the rotated image matrix in the second group of cost aggregation direction.

Optionally, the determining, according to the matching cost aggregation values corresponding to the pixels in the reference row and the reference column in the first set of cost aggregation directions, the matching cost aggregation values corresponding to the pixels in the other rows and columns in the first set of cost aggregation directions includes:

dividing the image matrix into a plurality of image blocks according to the reference rows and the reference columns, wherein each image block comprises a part of pixels in the reference rows and the reference columns;

determining matching cost aggregation values, corresponding to a part of pixels in other rows and columns in each image block, in a first group of cost aggregation directions according to the matching cost aggregation values, corresponding to a part of pixels in a reference row and a reference column in each image block, in the first group of cost aggregation directions;

the determining matching cost aggregation values corresponding to pixels of the image matrix in a second set of cost aggregation directions includes:

determining matching cost aggregation values corresponding to the pixels in each image block in a second group of cost aggregation directions;

determining a total matching cost aggregation value corresponding to each pixel according to the matching cost aggregation values corresponding to each pixel in the first group of cost aggregation directions and the second group of cost aggregation directions, including:

and determining a total matching cost aggregation value corresponding to each pixel according to the matching cost aggregation values corresponding to each pixel in the first group of cost aggregation directions and the second group of cost aggregation directions in each image block.

Optionally, the determining matching cost aggregation values corresponding to pixels in other rows and columns in each image block in the first set of cost aggregation directions includes:

traversing and calculating the matching cost aggregation values of the pixels in other rows and columns in each image block in the first set of cost aggregation directions according to a first set direction;

the determining the matching cost aggregation values corresponding to the pixels in each image block in the second set of cost aggregation directions includes:

traversing and calculating the matching cost aggregation values of the pixels in each image block in the second set of cost aggregation directions according to a second set direction; the first setting direction is opposite to the second setting direction.

Optionally, the determining, according to matching cost aggregation values of a part of pixels in a reference row and a reference column included in each image block in the first set of cost aggregation directions, matching cost aggregation values of pixels in other rows and columns in each image block in the first set of cost aggregation directions, which respectively correspond to the pixels in the other rows and columns in each image block, includes:

for any image block in each image block, according to matching cost aggregation values, corresponding to a part of pixels in a reference row and a reference column, included in the image block in the first group of cost aggregation directions, matching cost aggregation values, corresponding to pixels in other rows and columns in the image block in the first group of cost aggregation directions, are determined in parallel in a row-by-row and column-by-column manner.

for any image block in each image block, according to matching cost aggregation values, corresponding to a part of pixels in a reference row and a reference column in the first group of cost aggregation directions, of the reference row and the reference column included in the any image block, matching cost aggregation values, corresponding to pixels in other rows and columns in the first group of cost aggregation directions, of the pixels in each image block are determined in a point-by-point mode.

Optionally, the method further comprises:

and determining a reference row and a reference column of the image matrix according to one or more of the size of the image matrix, the computing capacity of equipment for performing matching cost aggregation computing processing and the size of a storage space for storing a matching cost aggregation value.

In a third aspect, an embodiment of the present invention provides an image depth information determining apparatus, including a memory and a processor; wherein the memory has stored thereon executable code that, when executed by the processor, causes the processor to:

dividing a first image of the plurality of images into a plurality of blocks;

Optionally, the processor is further configured to:

storing the first matching cost aggregate value in a storage space;

storing the second matching cost aggregate value into the storage space.

Optionally, the processor is configured to:

In a fourth aspect, an embodiment of the present invention provides an apparatus for determining image depth information, including a memory and a processor; wherein the memory has stored thereon executable code that, when executed by the processor, causes the processor to:

Optionally, the processor is configured to:

rotating the image matrix by a preset angle;

Optionally, the processor is configured to:

Optionally, the processor is further configured to:

In a fifth aspect, an embodiment of the present invention provides an electronic device, which includes a shooting device and the image depth information determining device in the third and fourth aspects.

In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, where the storage medium is a computer-readable storage medium, and program instructions are stored in the computer-readable storage medium, where the program instructions are used to implement the image depth information determining methods in the first aspect and the second aspect.

By the image depth information determining method, the image depth information determining device, the image depth information determining equipment and the storage medium, data reading and writing bandwidth can be reduced.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.

Fig. 1 is a schematic flowchart of an image depth information determining method according to an embodiment of the present invention;

fig. 2 is a schematic diagram of 16 aggregation directions according to an embodiment of the present invention;

fig. 3 is a schematic diagram of 8 aggregation directions of cost according to an embodiment of the present invention;

FIG. 4 is a schematic diagram illustrating the locations of reference rows and reference columns in an image matrix according to an embodiment of the present invention;

fig. 5 is a schematic diagram of an image block segmentation result according to an embodiment of the present invention;

fig. 6 is a schematic diagram of an image block according to an embodiment of the present invention;

fig. 7 is a schematic diagram of a matching cost aggregate value of a point-by-point recovery image block according to an embodiment of the present invention;

fig. 8 is a schematic flow chart illustrating another method for determining image depth information according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an image depth information determining apparatus according to an embodiment of the present invention;

fig. 10 is a schematic structural diagram of another image depth information determining apparatus according to an embodiment of the present invention;

fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the examples of the present invention and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise, and "a plurality" typically includes at least two.

The words "if", as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrases "if determined" or "if detected (a stated condition or event)" may be interpreted as "when determined" or "in response to a determination" or "when detected (a stated condition or event)" or "in response to a detection (a stated condition or event)", depending on the context.

In addition, the sequence of steps in each method embodiment described below is only an example and is not strictly limited.

Fig. 1 is a flowchart of an image depth information determining method according to an embodiment of the present invention, as shown in fig. 1, the method includes the following steps:

101. the method comprises the steps of obtaining matching cost aggregation values corresponding to pixels in a reference row and a reference column of an image matrix in a first set of cost aggregation directions, wherein the first set of cost aggregation directions are partial aggregation directions in a plurality of cost aggregation directions, the image matrix corresponds to images shot by a plurality of cameras (for example, binocular cameras or a plurality of cameras located at different positions), and the matching costs are used for measuring the similarity between corresponding pixels in the images shot by the plurality of cameras. In another embodiment, the image matrix may also correspond to images captured by a movable camera at different positions, and the matching cost is used to measure the similarity between the images captured by the movable camera at different positions.

102. And storing the matching cost aggregation values respectively corresponding to the pixels in the reference row and the reference column in the first group of cost aggregation directions into a storage space.

103. And determining the matching cost aggregation values of the pixels in other rows and columns in the first group of cost aggregation directions according to the matching cost aggregation values of the pixels in the reference row and the reference column in the first group of cost aggregation directions, wherein the other rows and columns are rows and columns in the image matrix except the reference row and the reference column.

104. And determining matching cost aggregation values corresponding to the pixels of the image matrix in a second group of cost aggregation directions, wherein the second group of cost aggregation directions are composed of aggregation directions which are not included in the first group of cost aggregation directions in the multiple cost aggregation directions.

105. And determining a total matching cost aggregation value corresponding to each pixel according to the matching cost aggregation values corresponding to each pixel in the first group of cost aggregation directions and the second group of cost aggregation directions.

106. And determining the depth information of the image according to the total matching cost aggregation value corresponding to each pixel.

In practical applications, in order to determine the distance between an object in the physical world (i.e., the real world) and the imaging device, an image including the object may be captured, depth information of the captured image may be determined, and the distance between the object and the imaging device may be determined based on the depth information of the image. The embodiment of the invention provides a method for determining image depth information.

In addition, a plurality of images of a certain object may be captured by the imaging device, and depth information of a target image among the plurality of images may be determined based on the plurality of images. The shooting device can comprise a plurality of cameras, or only one camera. When the shooting device comprises a plurality of cameras, the plurality of cameras can be arranged at different positions, so that the plurality of cameras can shoot a certain object from different angles respectively to obtain images of the certain object at different angles. When the shooting device comprises one camera, an object can be shot from different positions through the same camera at different times, and images of the object at different angles can be obtained. The following description will be given taking a binocular camera as an example. However, one skilled in the art will appreciate that multiple cameras may be employed to implement various embodiments of the present invention, as well as combinations of different embodiments.

In a possible implementation manner, a binocular camera may be used to shoot an object, so as to obtain a left viewpoint image and a right viewpoint image containing the object. And calculating image depth information by taking the left image as a target image. The main process of calculating the image depth information comprises the following steps:

1. and calculating matching cost (matching cost). The matching cost is also referred to as a matching cost value.

2. Cost aggregation (cost aggregation).

3. Disparity calculation and optimization (disparity calculation/optimization).

4. Disparity refinement (disparity refinement).

Wherein the matching cost represents the similarity between two pixels. Cost aggregation is used to utilize all pixels in the corresponding image of the entire scene to affect each single pixel to obtain a more accurate result than the matching cost. The parallax calculation and optimization and the parallax refinement are used for checking the calculation result, performing sub-pixel processing and the like. Through the above-listed calculation process of the image depth information, the depth information of the target image can be calculated, and then the distance from a certain object to the shooting device can be further obtained according to the triangulation principle. The embodiment of the invention specifically provides a method for cost aggregation, and the mathematical principle of cost aggregation will be described below.

Mathematically, the process of cost aggregation can be viewed as finding a global energy minimum:

wherein, C (p, D)_p) For a matching cost of a certain pixel, p is any pixel in the image. P₁T[|D_p-D_q|＝1]A penalty term T [ | D ] when the parallax variation range in adjacent pixels is small_p-D_q|＝1]Representing adjacent pixels D_pAnd D_qThe disparity difference between them is 1, and the Np range is the neighboring area of the pixel p. P₂T[|D_p-D_q＞1|]The penalty term is when the parallax variation range is large (i.e., the parallax difference is greater than 1) in the adjacent pixels. The cost aggregation process involves a high computational complexity, and is a Non-deterministic problem (NP) of Polynomial complexity.

In order to reduce the computational complexity of the cost aggregation process, cost aggregation may be performed using only pixels in certain directions. As shown in fig. 2 and fig. 3, schematic diagrams of 16 cost aggregation directions and 8 cost aggregation directions are sequentially shown, where p in the diagrams is a certain pixel in an image. In addition to performing cost aggregation from 16 cost aggregation directions, cost aggregation may also be performed from 8 cost aggregation directions, or cost aggregation may also be performed from other number of cost aggregation directions, which is not limited in this embodiment of the present invention.

For a certain pixel p in the image, cost aggregation in a certain cost aggregation direction can be achieved by the following formula:

wherein the first term C (p, d) in the polynomial is the matching cost of the pixel p. The second term is the minimum of 4 data, the 4 data including L_r(p-r,d)、L_r(p-r,d-1)+P₁、L_r(p-r,d+1)+P₁、L_r(p-r,d+1)+P₁And min_iL_r(p-r,i)+P₂。L_r(p-r, d) represents a cost aggregate value of the last pixel of the pixel p in the r direction under the disparity d. L is_r(p-r, d-1) represents the cost aggregate value of the last pixel of the pixel p in the r direction under the disparity d-1. L is_r(p-r, d +1) represents a cost aggregate value of the last pixel of the pixel p in the r direction under the disparity d + 1. min_iL_r(p-r, i) represents the minimum cost aggregation value of the last pixel of the pixel p in the r direction under the parallax i, wherein the value of i ranges from 0 to any value between the maximum parallax values. P₁And P₂For the penalty factor, the farther the distance from the pixel p, the larger the penalty factor. Item three min_kL_r(p-r, k) is to prevent L_r(p, d) saturation, i.e. preventing letting L go with iteration_rThe growth of (p, d) is too great.

Through the above formula, the cost aggregate value of a certain pixel p in a certain cost aggregation direction can be calculated, the formula is repeatedly used, the cost aggregate values respectively corresponding to the certain pixel p in all cost aggregations can be obtained, and then the cost aggregate values respectively corresponding to the certain pixel p in all cost aggregations are added, so that the total matching cost aggregate value of the certain pixel p can be obtained. For a certain pixel p, the disparity d corresponding to the minimum total matching cost aggregation value is the final disparity value.

The above describes calculating the total matching cost aggregate value of a certain pixel p, and by repeating this way, the total matching cost aggregate value of all pixels in the image can be calculated. In practical application, the matching cost aggregate value of all pixels in the image on the upper left part path can be calculated first, and the calculation result can be stored in the memory. The memory may be a double data rate synchronous dynamic random access memory (DDR) or other type of memory. Then the matching cost aggregate value on the lower right path of all pixels in the image is calculated. After the matching cost aggregation values of all pixels in the image on the lower right path are calculated, the matching cost aggregation values of all pixels in the image on the upper left path are read from the memory, the matching cost aggregation values of all pixels in the image on the upper left path are added to the matching cost aggregation values of corresponding pixels in the image on the lower right path, and total matching cost aggregation values corresponding to all pixels in the image are obtained. Of course, the calculation order of the upper left partial path and the lower right partial path may be switched.

For example, the upper left part path may be a path shown by L0_ pd-L3_ pd in fig. 3, and the lower right part path may be a path shown by L4_ pd-L7_ pd in fig. 3, taking 8 cost aggregation directions as examples.

In the above process, since the matching cost aggregation values of all pixels in the image on a certain part of the path need to be temporarily stored in the memory, and the data volume of the matching cost aggregation values of all pixels in the image on a certain part of the path is large, a large data read-write bandwidth needs to be occupied. In order to reduce the occupation of the data read-write bandwidth, in the embodiment of the present invention, only part of the matching cost aggregation values may be temporarily stored in the memory, and when the matching cost aggregation values of all the pixels in the image on a certain part of the path are used, the matching cost aggregation values of all the pixels in the image on a certain part of the path are restored based on the part of the matching cost aggregation values, so that the occupation of the data read-write bandwidth can be greatly reduced.

As can be seen from equation 2, for the cost aggregation value of any pixel in the image in a single aggregation direction, it is determined by two parts. Specifically, the cost aggregation value includes the matching cost of the pixel and the cost aggregation value of the last pixel in the current cost aggregation direction. Based on this, the process of cost aggregation can be expressed by the following formula:

L0_pd_i＝f(C_i,L0_pd_i-1) (formula 3)

Where i is the current pixel in the image, C_iL0_ pd, representing the matching cost of the current pixel i_i-1A cost aggregation value, L0_ pd, representing the last pixel i-1 in the current cost aggregation direction_iRepresenting the cost aggregation value of the current pixel i in the current aggregation direction. Taking 8 cost aggregation directions as an example, when performing cost aggregation summation of the lower right part path, the cost aggregation summation result of the upper left part path needs to be used, and the cost aggregation summation result of the upper left part path can be expressed as:

the following describes an implementation process for reducing the occupation of the data read-write bandwidth based on the above analysis by using the theoretical basis analyzed above.

First, the matching cost aggregation values corresponding to all pixels of the image matrix in the first set of cost aggregation directions can be calculated in the manner described above. The image matrix corresponds to an image shot by the binocular camera.

The first set of cost aggregation directions may be partial aggregation directions of a plurality of cost aggregation directions. The multiple cost aggregation directions may be the aforementioned 16 cost aggregation directions, may also be 8 cost aggregation directions, or may be other numbers of cost aggregation directions. The first set of cost aggregation directions may also be an upper left part path or a lower right part path of the plurality of cost aggregation directions.

The position of the pixel in the reference row and the reference column can be determined in the entire image matrix. As shown in fig. 4, taking a 32 × 32 image as an example, the reference row and the reference column may be the 1 st row, the 17 th row, the 1 st column, and the 17 th column in the image. In this example, row 1, middle row (i.e., row 17), column 1, and middle column (i.e., column 17) of the image are selected as reference row and reference column to store the aggregate costs as reference row and/or reference column in the DDR, while the aggregate costs for the other rows and other columns are discarded (i.e., not stored in the DDR). In other cases, a certain row or a certain column of the image may be selected as the reference row and the reference column, and the reference row and/or the reference column may be stored in the DDR as the aggregation cost of the reference row and/or the reference column, and the aggregation cost of the non-reference row or the reference column is discarded, which is not limited in the embodiment of the present invention.

Alternatively, the above process of determining the position of the pixel in the reference row and the reference column in the whole image matrix may be implemented as: and determining a reference row and a reference column of the image matrix according to one or more of the size of the image matrix, the computing capacity of a device for performing matching cost aggregation computing processing and the size of a storage space for storing matching cost aggregation values.

In practical applications, if the size of the image matrix is large, the number of reference rows and reference columns can be considered to be increased, so that the operation speed of performing the cost aggregation processing can be increased. If the computational power of the device performing the matching cost aggregation computation process is high, it can be considered to reduce the number of reference rows and reference columns. If the storage space for storing the matching cost aggregation value is small, the number of reference rows and reference columns can be considered to be increased more, so that the storage space for storing the matching cost aggregation value can be saved.

After determining the positions of the pixels in the reference row and the reference column in the image matrix, the matching cost aggregation values, corresponding to the pixels in the reference row and the reference column of the image matrix in the first set of cost aggregation directions, may be obtained, for example, the matching cost aggregation values, corresponding to the pixels in the labeled positions except for the white frame in fig. 4 in the first set of cost aggregation directions, are obtained. And storing the obtained matching cost aggregation values into a memory, so that the matching cost aggregation values corresponding to the pixels in the reference row and the reference column in the first group of cost aggregation directions are reserved. In one embodiment, the first set of cost aggregation directions are the cost aggregation directions corresponding to the paths shown by L0_ pd-L3_ pd shown in fig. 3.

Alternatively, the process of storing the matching cost aggregation values corresponding to the pixels in the reference row and the reference column in the first set of cost aggregation directions into the storage space may be implemented as follows: storing matching cost aggregation values corresponding to first pixels in the reference row and the reference column in a first group of cost aggregation directions into a storage space, wherein the first pixels are positioned in the reference row and the reference column at the same time; only storing matching cost aggregation values corresponding to the second pixels in the reference row in the first target cost aggregation direction into a storage space; wherein the second pixel is in the reference row and not in the reference column; and only storing the matching cost aggregation value corresponding to the third pixel in the reference column in the second target cost aggregation direction into the storage space, wherein the third pixel is in the reference column and is not in the reference row.

The first group of cost aggregation directions comprise a first target cost aggregation direction and a second target cost aggregation direction, and the first target cost aggregation direction is different from the second target cost aggregation direction.

For the convenience of understanding, the implementation process of storing the matching cost aggregation values is specifically described by taking 8 cost aggregation directions as an example. Assuming that cost aggregation needs to be performed from 8 cost aggregation directions, cost aggregation values of pixels at the 1 st row, 17 th row, 1 st column and 17 th column positions of the image matrix at L0_ pd-L3_ pd on the upper left partial path have been calculated. At this time, for the first pixel located at the overlapping position in the 1 st row, the 17 th row, the 1 st column and the 17 th column, the matching cost aggregation values corresponding to the first pixel in the L0_ pd-L3_ pd directions are stored in the storage space. The first pixel is, for example, a pixel on the first row at the first column.

For the second pixels except for the first pixel in the 1 st and 17 th rows, only the matching cost aggregation values corresponding to the second pixels in the L0_ pd, L1_ pd, and L2_ pd directions may be stored in the storage space. It should be noted that the matching cost aggregation value of the second pixel in the L3_ pd direction may not be stored.

For the third pixels except for the first pixel in the 1 st column and the 17 th column, only the matching cost aggregate value of the third pixel in the L3_ pd direction may be stored in the storage space. It should be noted that the matching cost aggregation values corresponding to the third pixel in the directions L0_ pd, L1_ pd, and L2_ pd may not be stored.

It should be noted that, if there is no previous pixel in a certain price aggregation direction, the matching price aggregation value corresponding to the previous pixel in the certain price aggregation direction of any pixel may be recorded as a certain default value or as the matching price value of any pixel.

When the cost aggregation summation in the second set of cost aggregation directions is performed, the cost aggregation values of all pixels in the image matrix in the first set of cost aggregation directions need to be used, so that the cost aggregation values of the pixels in the rows and columns in the image matrix except the reference rows and the reference columns in the first set of cost aggregation directions can be recovered based on the matching cost aggregation values of the pixels in the reserved reference rows and reference columns in the first set of cost aggregation directions, which correspond to each other. Wherein the second set of cost aggregation directions consists of aggregation directions of the plurality of cost aggregation directions that are not included in the first set of cost aggregation directions.

Optionally, according to the matching cost aggregation values of the pixels in the reference row and the reference column in the first set of cost aggregation directions, the process of determining the matching cost aggregation values of the pixels in the other rows and columns in the first set of cost aggregation directions may be implemented as: dividing the image matrix into a plurality of image blocks according to the reference rows and the reference columns, wherein each image block comprises a part of pixels in the reference rows and the reference columns; and determining matching cost aggregation values respectively corresponding to the pixels in other rows and columns in each image block in the first group of cost aggregation directions according to the matching cost aggregation values respectively corresponding to a part of the pixels in the reference row and the reference column in each image block in the first group of cost aggregation directions.

Based on this, the process of determining the matching cost aggregation value corresponding to each pixel of the image matrix in the second set of cost aggregation direction may be implemented as follows: and determining the matching cost aggregation value corresponding to each pixel in each image block in the second group of cost aggregation directions.

According to the matching cost aggregation values of the pixels in the first group of cost aggregation directions and the second group of cost aggregation directions, the process of determining the total matching cost aggregation value corresponding to each pixel can be realized as follows: and determining a total matching cost aggregation value corresponding to each pixel according to the matching cost aggregation values corresponding to each pixel in the first group of cost aggregation directions and the second group of cost aggregation directions in each image block.

In practical applications, as shown in fig. 5, still taking the process of performing cost aggregation on 32 × 32 images from 8 cost aggregation directions as an example, the whole image may be divided into 4 image block blocks according to the positions of the reference rows and the reference columns, including image block 1-image block 4. Each image block may be processed in turn in the order of image block 1-image block 2-image block 3-image block 4. That is, the processing order is a zigzag (i.e., zigzag) processing order. And the image block in the lower right corner of the image is taken as the image block processed first. Processing procedures of each image block are similar, the processing procedure of the image block is described below by taking the image block1 as an example, and the processing procedures of other image blocks may refer to the processing procedure of the image block1, which is not described in detail in the embodiment of the present invention.

The image block1 obtained after the slicing is shown in fig. 6. For image block1, matching cost aggregation values corresponding to pixels in other rows and columns in image block1 in the first set of cost aggregation directions may be determined according to matching cost aggregation values corresponding to a part of pixels in reference rows and reference columns included in image block1 in the first set of cost aggregation directions. Wherein a part of the pixels in the reference row and the reference column may be pixels at the marked positions other than the white frame shown in fig. 6, and the pixels in the other rows and columns in the image block1 may be pixels at the white frame positions in fig. 6.

In the embodiment of the present invention, two ways of determining matching cost aggregation values corresponding to pixels in other rows and columns in an image block in a first group of cost aggregation directions are provided according to matching cost aggregation values corresponding to a part of pixels in a reference row and a reference column included in the image block in the first group of cost aggregation directions. And determining the matching cost aggregation values respectively corresponding to the pixels in other rows and columns in the image block in the first set of cost aggregation directions in a row-by-row or point-by-point mode.

For a row-by-row manner, according to matching cost aggregation values, corresponding to a part of pixels in a reference row and a reference column included in any image block, in the first set of cost aggregation directions, matching cost aggregation values, corresponding to pixels in other rows and columns in each image block, in the first set of cost aggregation directions can be determined in parallel in a row-by-row manner and a column-by-column manner.

In practical applications, taking fig. 6 as an example, the matching cost aggregation values of the L0_ pd, L1_ pd, and L2_ pd of the row 2 can be calculated from the matching cost aggregation values of the L0_ pd, the L1_ pd, and the L2_ pd directions of the row 1 and the matching cost value of the row 2 (i.e., the matching cost described above). The matching cost aggregation value of the 2 nd column in the L3_ pd direction can also be calculated by the matching cost aggregation value of the 1 st column in the L3_ pd direction and the matching cost value of the 2 nd column.

The calculation processes of the matching cost aggregation values in different directions of L0_ pd-L3_ pd are independent, so that the matching cost aggregation values in different directions can be calculated in parallel. For example, the recovery of the matching cost aggregation values in the L0_ pd, L1_ pd, and L2_ pd directions of the row 2 and the matching cost aggregation values in the L3_ pd direction of the column 2 can be completed in one cycle, and for the image block shown in fig. 6, the computation of the matching cost aggregation values of the pixels in the other rows and columns in the image block in the first set of cost aggregation directions can be completed in only 16 cycles. The calculation speed is high for the mode of restoring the matching cost aggregation value of the image block line by line.

It is worth noting that 4 equal-sized blocks of storage space can be opened up in the memory to store the matching cost aggregate value in the L0_ pd-L3_ pd direction for each pixel in the image block. Taking the image block shown in fig. 6 as an example, assuming that the data size of each matching cost aggregation value is 16 bits, a storage space of 16 × 16 × 16 bits of 4 blocks can be opened up to store the matching cost aggregation values of each pixel in the image block in the direction L0_ pd-L3_ pd. It should be noted that, since the matching cost aggregation values of the pixels at each position in an image block in the L0_ pd-L3_ pd directions are needed, including the pixels at other rows and columns in the image block, and the pixels at the reference row and the reference column, the matching cost aggregation values of the pixels at each position in the whole image block in the L0_ pd-L3_ pd directions can be stored in the opened storage space, and thus, for one cost aggregation direction, a 16 × 16 × 16 bit storage space is needed for one image block.

For a point-by-point manner, according to matching cost aggregation values, corresponding to a part of pixels in a reference row and a reference column included in any image block, in the first set of cost aggregation directions, matching cost aggregation values, corresponding to pixels in other rows and columns in each image block, in the first set of cost aggregation directions, can be determined in a point-by-point manner.

It can be understood that, since the image matrix is divided into a plurality of image blocks according to the reference rows and the reference columns, each image block includes not a complete reference row of a whole row or a complete reference column, but a part of the reference row or a part of the reference column, that is, each image block includes a part of pixels in the reference row and the reference column. For example, a portion of the pixels in the image block that include the reference row and the reference column may be a portion of the pixels in the reference row and the reference column indicated in fig. 6.

When the matching cost aggregation values of the image blocks are restored point by point, the matching cost aggregation values corresponding to the pixels in other rows and columns in each image block in the first group of cost aggregation directions can be calculated in a traversing manner according to a first set direction; traversing and calculating the matching cost aggregation value of each pixel in each image block in the second set of cost aggregation direction according to the second set direction; the first setting direction is opposite to the second setting direction.

When the first set of cost aggregation directions is the upper left partial path of the image matrix, the first set direction may be a top-to-bottom and left-to-right direction, and the second set direction may be a bottom-to-top and right-to-left direction.

In practical applications, as shown in fig. 7, the matching cost aggregate value in the direction of the (1,1) position can be calculated from the matching cost aggregate value in the direction of the L3_ pd of the (1,0) position, and the matching cost aggregate values in the directions of the L0_ pd, L1_ pd, and L2_ pd of the (0,1) position can be calculated to obtain the matching cost aggregate values in the directions of the L0_ pd, L1_ pd, and L2_ pd of the (1,1) position. After the matching cost aggregation values of the (1,1) position in 4 directions are calculated, they may be added.

It is to be understood that after the addition result of the matching cost aggregation values of the (1,1) position in the 4 directions is calculated, the result may be stored into a storage space in the memory. In this way, the addition result of the matching cost aggregation values of all the pixels in the image block in 4 directions can be finally calculated, and the results are stored in the storage space in the memory. Still taking the image block shown in fig. 6 as an example, assuming that the data amount of the addition result of the matching cost aggregation values of each pixel in 4 directions is 16 bits, a storage space of 16 × 16 × 16 bits of 1 block may be opened up to store the addition result of the matching cost aggregation values of each pixel in 4 directions in the image block. Compared with a mode of restoring the matching cost aggregation values of the image blocks line by line, the method of restoring the matching cost aggregation values of the image blocks point by point can directly restore the matching cost aggregation values of one pixel in 4 directions, so that the matching cost aggregation values in the 4 directions can be directly added, and an addition result is stored in a storage position corresponding to a storage space. However, by means of restoring the matching cost aggregation values of the image block line by line, the matching cost aggregation values of one pixel in 4 directions cannot be restored at the same time, so that the matching cost aggregation values of one pixel in 4 directions need to be stored in the storage space respectively, and after the matching cost aggregation values of all pixels of the image block in 4 directions are obtained, the data are obtained from the storage space and added to obtain a final result. Therefore, for the way of restoring the matching cost aggregation value of the image block point by point, the occupied memory space is small.

In one embodiment, if the matching cost aggregation values a1 to am of the image block1 in m cost aggregation directions need to be calculated in total, the m cost aggregation directions are divided into two groups. I.e. a first set of cost aggregation directions and a second set of cost aggregation directions. Wherein the first set of cost aggregation directions comprises m1 cost aggregation directions, and the second set of cost aggregation directions comprises m2 cost aggregation directions. m1+ m2 ═ m, and m1, m2, and m are integers. After the matching cost aggregation values of all the rows and columns in the image block1 in the first group of cost aggregation directions are restored, cost aggregation calculation of all the rows and columns in the image block1 in the second group of cost aggregation directions can be performed, so that the matching cost aggregation values of all the rows and columns in the image block1 in multiple cost aggregation directions can be finally obtained.

After the matching cost aggregation values of each pixel in image block1 in the multiple cost aggregation directions are calculated, the matching cost aggregation values of each pixel included in image block1 in the first set of cost aggregation directions and the second set of cost aggregation directions may be added to determine a total matching cost aggregation value of each pixel included in image block 1. In order to save the storage space, the parallax with the smallest aggregate value of the total matching costs in all the parallaxes can be directly selected as the final result.

Optionally, in the process of performing cost aggregation on any pixel in the image matrix in the second set of cost aggregation directions, the cost aggregation may be performed with reference to a manner that any pixel performs cost aggregation in the first set of cost aggregation directions. Based on this, the process of performing cost aggregation on any pixel in the image matrix in the second set of cost aggregation directions may be implemented as: acquiring matching cost aggregation values corresponding to pixels in a reference row and a reference column of the image matrix in a second group of cost aggregation directions; and determining the matching cost aggregation values respectively corresponding to the pixels in other rows and columns in the second group of cost aggregation directions according to the matching cost aggregation values respectively corresponding to the pixels in the reference row and the reference column in the second group of cost aggregation directions.

In practical application, matching cost aggregate values corresponding to pixels in all rows and columns of the image matrix in the second group of cost aggregate directions can be calculated, positions of the pixels in the reference rows and the reference columns of the image matrix are determined, and the matching cost aggregate values corresponding to the pixels in the reference rows and the reference columns of the image matrix in the second group of cost aggregate directions are reserved. When the matching cost aggregation values corresponding to the pixels in all the rows and columns of the image matrix in the second set of cost aggregation directions are used, the matching cost aggregation values corresponding to the pixels in the other rows and columns of the image matrix except the reference row and the reference column in the second set of cost aggregation directions can be restored based on the matching cost aggregation values corresponding to the pixels in the reference row and the reference column of the image matrix in the second set of cost aggregation directions.

It should be noted that, the step of recovering the matching cost aggregation values corresponding to the pixels in the other rows and columns of the image matrix except the reference row and the reference column in the second set of cost aggregation directions, based on the matching cost aggregation values corresponding to the pixels in the reference row and the reference column of the image matrix in the second set of cost aggregation directions, includes: the middle row, the last row, the middle column, and the last column may be selected as the reference row and the reference column, or the 1 st row, the middle row, the 1 st column, and the middle column may be selected as the reference row and the reference column. In the process of dividing the image block according to the selected reference row and the selected reference column, it is to be ensured that a matching cost aggregation value corresponding to each of a part of pixels in the reference row and the reference column in the second group of cost aggregation directions exists in each divided image block. In one embodiment, corresponding pixels in a reference row are scribed into an image block adjacent to and to the right of the reference row, and corresponding pixels in a reference column are scribed into an image block adjacent to and below the reference column. In an embodiment, a single image is divided into 4 image blocks by using the 1 st row, the 17 th row, the 1 st column and the 17 th column as reference rows and reference columns, then partial pixels in the 1 st row and the 1 st column may be divided into one image block, and partial pixels in the 1 st row, the 1 st column and the 17 th row may not be divided into one image block, because if partial pixels in the 1 st row, the 1 st column and the 17 th row are divided into one image block, partial pixels in the reference rows may not be included in the image block arranged below the image block, and thus it is difficult to restore matching cost aggregate values of other rows and columns of pixels.

In one embodiment, the tiles used for cost-polymerizing from the direction of L0_ pd-L3_ pd may be divided in the same way as the tiles used for cost-polymerizing from the direction of L4_ pd-L7_ pd.

In another embodiment, the image blocks used for cost-polymerizing from the direction of L0_ pd-L3_ pd may be divided differently from the image blocks used for cost-polymerizing from the direction of L4_ pd-L7_ pd. In one embodiment, when the cost aggregation is performed from the L0_ pd-L3_ pd direction, the image may be divided into reference rows and reference columns by the 1 st row and the 1 st column, and when the cost aggregation is performed from the L4_ pd-L7_ pd direction, the image may be divided into reference rows and reference columns by the 17 th row, the 17 th column, the 32 th row and the 32 th column. Thus, an image block in the upper left corner contains a portion of the pixels in row 17 and column 17. And the other image blocks are analogized.

In another embodiment, when the cost aggregation is performed from the L0_ pd-L3_ pd direction, the image may be divided into reference rows and reference columns with the 1 st row and the 1 st column, and when the cost aggregation is performed from the L4_ pd-L7_ pd direction, the image matrix may be rotated by 180 °, and the image matrix may be divided into reference rows and reference columns with the 1 st row, the 1 st column, the 17 th row and the 17 th column in the obtained rotated image.

In yet another embodiment, when performing cost aggregation from the direction L0_ pd-L3_ pd, the image may be divided into reference rows and reference columns with the 1 st row, the 1 st column, the 17 th row and the 17 th column, and when performing cost aggregation from the direction L4_ pd-L7_ pd, the image matrix may be divided into reference rows and reference columns with the 1 st row, the 1 st column, the 17 th row and the 17 th column, and each image block does not include pixels in the reference rows and reference columns, and only includes pixels in the white frame position as shown in fig. 4. Optionally, in the process of recovering the matching cost aggregation values of other row-column pixels, the calculation is performed after the image block is rotated by 180 °.

Optionally, according to the matching cost aggregation values of the pixels in the reference row and the reference column in the second set of cost aggregation directions, the process of determining the matching cost aggregation values of the pixels in the other rows and columns in the second set of cost aggregation directions may be implemented as: rotating the image matrix by a preset angle; and determining matching cost aggregation values corresponding to the pixels in other rows and columns in the second group of cost aggregation direction according to the matching cost aggregation values corresponding to the pixels in the reference row and the reference column in the rotated image matrix in the second group of cost aggregation direction.

The preset angle may be 180 °, and after the image matrix is rotated by 180 °, the lower right portion of the image matrix before rotation becomes the upper left portion of the image matrix after rotation, so that the matching cost aggregation values corresponding to any pixel in the image matrix in the second set of cost aggregation directions may be calculated by referring to a similar manner for performing cost aggregation on any pixel in the first set of cost aggregation directions.

It will be appreciated that the first set of cost aggregation directions mentioned above may be the upper left part path of the image matrix, while the second set of cost aggregation directions is the lower right part path of the image matrix. When the first set of cost aggregation directions is the lower right part path of the image matrix, the second set of cost aggregation directions is the upper left part path of the image matrix.

When the number of the reference rows and the reference columns is adjusted, the size of the image block bl ock is changed. When the image block is larger, the reduction of the data reading and writing bandwidth is more obvious.

By the method provided by the embodiment of the invention, only the matching cost aggregation values respectively corresponding to the pixels in the reference row and the reference column in the image matrix in the first group of cost aggregation directions are temporarily stored in the memory, and the matching cost aggregation values of other rows or columns of the non-reference row and the non-reference column are not stored, so that only the matching cost aggregation values respectively corresponding to part of the pixels in the image matrix in the first group of cost aggregation directions are stored. When the matching cost aggregation values respectively corresponding to the pixels in the other rows and columns in the image matrix except the reference row and the reference column in the first group of cost aggregation directions are used, the matching cost aggregation values respectively corresponding to the pixels in the other rows and columns in the image matrix in the first group of cost aggregation directions can be restored based on the stored matching cost aggregation values respectively corresponding to part of the pixels in the image matrix in the first group of cost aggregation directions. By avoiding the way of storing the matching cost aggregation values corresponding to all pixels in the image matrix in the first group of cost aggregation directions, the occupation of data read-write bandwidth can be greatly reduced.

Fig. 8 is a schematic flowchart of another image depth information determining method according to an embodiment of the present invention, and as shown in fig. 8, the method includes the following steps:

801. a plurality of images about an object are respectively obtained by a photographing device.

802. A first image of the plurality of images is divided into a plurality of blocks.

803. A first matching cost aggregate value between pixels in a corresponding reference row of at least a first block of the plurality of blocks and corresponding pixels in a second image of the plurality of images is determined.

804. A second matching cost aggregate value between pixels in a corresponding reference column of the first block and corresponding pixels in a second image of the plurality of images is determined.

805. And determining a third matching cost aggregation value of the pixels of the first block in at least one cost aggregation direction according to the first matching cost aggregation value and the second matching cost aggregation value, so as to determine the depth information of the first block.

And the cost aggregation directions of the first matching cost aggregation value and the second matching cost aggregation value are different.

In practical applications, a plurality of images of an object may be captured by a camera, and depth information of a target image in the plurality of images may be determined based on the plurality of images. The shooting device can comprise a plurality of cameras, or only one camera. When the shooting device comprises a plurality of cameras, the plurality of cameras can be arranged at different positions, so that the plurality of cameras can shoot a certain object from different angles respectively to obtain images of the certain object at different angles. When the shooting device comprises one camera, an object can be shot from different positions through the same camera at different times, and images of the object at different angles can be obtained.

After a plurality of images of an object are captured, one of the images can be selected as a first image, and then the first image in the plurality of images can be divided into a plurality of blocks. The plurality of blocks may be a plurality of image blocks in the above embodiments.

For at least a first tile of the plurality of tiles, a first matching cost aggregate value between pixels in a corresponding reference row of the first tile and corresponding pixels in a second image of the plurality of images may be calculated. The cost aggregation direction corresponding to the first matching cost aggregation value may be a first set of cost aggregation directions of the plurality of cost aggregation directions.

A second matching cost aggregate value between pixels in a corresponding reference column of the first tile and corresponding pixels in a second image of the plurality of images may also then be determined. The cost aggregation direction corresponding to the second matching cost aggregation value may be a second set of cost aggregation directions of the plurality of cost aggregation directions other than the first set of cost aggregation directions.

Finally, a third matching cost aggregation value of the pixels of the first block in at least one cost aggregation direction can be determined according to the first matching cost aggregation value and the second matching cost aggregation value, so as to determine the depth information of the first block. The third matching cost aggregate value may be a matching cost aggregate value in at least one cost aggregation direction for pixels of rows and columns of the first block other than the reference row and the reference column. The matching cost aggregation values of the pixels of other rows and columns in the first block in at least one cost aggregation direction can be recovered through the first matching cost aggregation value and the second matching cost aggregation value.

For the specific implementation process in the embodiment shown in fig. 8, reference may be made to the related descriptions in the foregoing embodiments, and details are not repeated here.

Optionally, the step of determining a first matching cost aggregate value between pixels in a corresponding reference row of at least a first block of the plurality of blocks and corresponding pixels in a second image of the plurality of images comprises: acquiring a first matching cost aggregation value between pixels in a reference row of at least one first block in the plurality of blocks and corresponding pixels in a second image in the plurality of images; storing the first matching cost aggregate value in a storage space. The step of determining a second matching cost aggregate value between a pixel in the reference column of the first block and a corresponding pixel in a second image of the plurality of images comprises: acquiring a second matching cost aggregation value between pixels in the reference row of the first block and corresponding pixels in the second image; storing the second matching cost aggregate value in a storage space.

Optionally, according to the first matching cost aggregation value and the second matching cost aggregation value, the process of determining the third matching cost aggregation value of the corresponding pixel of the first tile in the at least one cost aggregation direction may be implemented as: and determining third matching cost aggregation values of the pixels in other rows and columns in the first block in at least one price aggregation direction according to the first matching cost aggregation value and the second matching cost aggregation value.

The cost aggregation direction corresponding to the first matching cost aggregation value may be a first group of cost aggregation directions, and the cost aggregation direction corresponding to the second matching cost aggregation value may be a second group of cost aggregation directions. The first set of cost aggregation directions may be partial aggregation directions of the plurality of cost aggregation directions. The multiple cost aggregation directions may be the aforementioned 16 cost aggregation directions, may also be 8 cost aggregation directions, or may be other numbers of cost aggregation directions. The first set of cost aggregation directions may also be an upper left part path or a lower right part path of the plurality of cost aggregation directions. And the second set of cost aggregation directions may be aggregation directions of the plurality of cost aggregation directions other than the first set of cost aggregation directions.

The third matching cost aggregate value may be a cost aggregate value of pixels in other rows and columns in the first block in at least one cost aggregation direction, for example, a cost aggregate value of pixels at positions marked by white boxes in fig. 4 in at least one cost aggregation direction.

Optionally, the process of determining third matching cost aggregation values of pixels in other rows and columns in the first block in at least one cost aggregation direction according to the first matching cost aggregation value and the second matching cost aggregation value may be implemented as follows: and for the first block, determining third matching cost aggregation values of pixels in other rows and columns in the first block in at least one cost aggregation direction in a row-by-row and column-by-column mode according to the first matching cost aggregation value and the second matching cost aggregation value.

Optionally, the process of determining third matching cost aggregation values of pixels in other rows and columns in the first block in at least one cost aggregation direction according to the first matching cost aggregation value and the second matching cost aggregation value may be implemented as follows: and for the first block, determining third matching cost aggregation values of pixels in other rows and columns in the first block in at least one price aggregation direction in a point-by-point mode according to the first matching cost aggregation value and the second matching cost aggregation value.

The at least one valence aggregation direction may be, for example, any one or more of L0_ pd-L7_ pd. Assuming that at least one of the generation cost aggregation directions is the L0_ pd direction, assuming that it is currently necessary to calculate the matching cost aggregation value of the pixel at the (1,1) position of fig. 7 in the L0_ pd direction, the matching cost aggregation value in the L0_ pd direction of the (1,1) position can be calculated from the matching cost aggregation value in the L0_ pd direction of the (0,1) position. The matching cost aggregation value in the L0_ pd direction of the (1,1) position can be used as the third matching cost aggregation value in at least one cost aggregation direction for the pixels in other rows and columns in the first block.

The above process may refer to the related descriptions in the foregoing other embodiments, which are not described herein.

Still another exemplary embodiment of the present invention provides an image depth information determining apparatus, as shown in fig. 9, including:

a first memory 1910 for storing a computer program;

a first processor 1920 for executing the computer program stored in the memory to implement:

dividing a first image of the plurality of images into a plurality of blocks;

Optionally, the first processor 1920 is further configured to:

storing the first matching cost aggregate value in a storage space;

storing the second matching cost aggregate value into the storage space.

Optionally, the first processor 1920 is configured to:

Still another exemplary embodiment of the present invention provides another image depth information determining apparatus, as shown in fig. 10, including:

a second memory 2010 for storing a computer program;

a second processor 2020 for executing the computer program stored in the memory to implement:

Optionally, the second processor 2020 is configured to:

rotating the image matrix by a preset angle;

Optionally, the second processor 2020 is configured to:

Optionally, the second processor 2020 is further configured to:

The image depth information determining apparatus shown in fig. 10 may perform the method of the embodiment shown in fig. 1-7, the image depth information determining apparatus shown in fig. 9 may perform the method of the embodiment shown in fig. 8, and parts of the embodiment not described in detail may refer to the related description of the embodiment shown in fig. 1-8. The implementation process and technical effect of the technical solution refer to the descriptions in the embodiments shown in fig. 1 to 8, and are not described herein again.

Fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. As shown in fig. 11, the electronic apparatus includes: camera 2110 and image depth information determining device 2120.

The image depth information determining device 2120 may be the device of the embodiment shown in fig. 9 and 10. For parts of this embodiment not described in detail, reference may be made to the description of the embodiment shown in fig. 1 to 8. The implementation process and technical effect of the technical solution refer to the descriptions in the embodiments shown in fig. 1 to 8, and are not described herein again.

In addition, an embodiment of the present invention further provides a computer-readable storage medium, where executable codes are stored in the computer-readable storage medium, and the executable codes are used to implement the image depth information determining method provided in the foregoing embodiments.

By the method provided by the embodiment of the invention, only the matching cost aggregation values respectively corresponding to the pixels in the reference row and the reference column in the image matrix in the first group of cost aggregation directions can be temporarily stored in the memory, so that only the matching cost aggregation values respectively corresponding to part of the pixels in the image matrix in the first group of cost aggregation directions are stored. When the matching cost aggregation values respectively corresponding to the pixels in the other rows and columns in the image matrix except the reference row and the reference column in the first group of cost aggregation directions are used, the matching cost aggregation values respectively corresponding to the pixels in the other rows and columns in the image matrix in the first group of cost aggregation directions can be restored based on the stored matching cost aggregation values respectively corresponding to part of the pixels in the image matrix in the first group of cost aggregation directions. By avoiding the way of storing the matching cost aggregation values corresponding to all pixels in the image matrix in the first group of cost aggregation directions, the occupation of data read-write bandwidth can be greatly reduced.

The technical solutions and the technical features in the above embodiments may be used alone or in combination without conflict, and all embodiments that fall within the scope of the present invention are equivalent embodiments within the scope of the present invention as long as they do not exceed the knowledge of those skilled in the art.

The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes performed by the present specification and drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image depth information determination method, comprising:

dividing a first image of the plurality of images into a plurality of blocks;

determining a second matching cost aggregate value between pixels in a corresponding reference column of the first block and corresponding pixels in the second image of the plurality of images;

2. The method of claim 1, wherein prior to determining a first matching cost aggregate value between pixels in a corresponding reference row of at least a first tile of the plurality of tiles and corresponding pixels in a second image of the plurality of images, the method further comprises:

storing the first matching cost aggregate value in a storage space;

storing the second matching cost aggregate value into the storage space.

3. The method according to claim 1, wherein the determining a third matching cost aggregation value of the corresponding pixel of the first tile in at least one cost aggregation direction according to the first matching cost aggregation value and the second matching cost aggregation value comprises:

4. The method according to claim 3, wherein the determining third matching cost aggregation values of pixels in other rows and columns in the first block in at least one cost aggregation direction according to the first matching cost aggregation value and the second matching cost aggregation value comprises:

and according to the first matching cost aggregation value and the second matching cost aggregation value, determining third matching cost aggregation values of pixels in other rows and columns in the first block in at least one cost aggregation direction in a row-by-row and column-by-column mode.

5. The method according to claim 3, wherein the determining third matching cost aggregation values of pixels in other rows and columns in the first block in at least one cost aggregation direction according to the first matching cost aggregation value and the second matching cost aggregation value comprises:

and determining third matching cost aggregation values of pixels in other rows and columns in the first block in at least one price aggregation direction in a point-by-point mode according to the first matching cost aggregation value and the second matching cost aggregation value.

6. An image depth information determination method, comprising:

7. The method of claim 6, wherein storing the matching cost aggregation values corresponding to the pixels in the reference row and the reference column in the first set of cost aggregation directions into a storage space comprises:

8. The method of claim 6, wherein determining the matching cost aggregation value corresponding to each pixel of the image matrix in the second set of cost aggregation directions comprises:

9. The method of claim 8, wherein determining matching cost aggregation values corresponding to pixels in the other rows and columns in the second set of cost aggregation directions according to matching cost aggregation values corresponding to pixels in the reference row and the reference column in the second set of cost aggregation directions comprises:

rotating the image matrix by a preset angle;

10. The method according to claim 6, wherein determining matching cost aggregation values corresponding to pixels in other rows and columns in the first set of cost aggregation directions according to the matching cost aggregation values corresponding to pixels in the reference row and the reference column in the first set of cost aggregation directions comprises:

11. The method according to claim 10, wherein the determining the matching cost aggregation values corresponding to the pixels in other rows and columns in each image block in the first set of cost aggregation directions comprises:

12. The method according to claim 10, wherein the determining, according to the matching cost aggregation values of a part of pixels in the reference row and the reference column included in each image block in the first set of cost aggregation directions, the matching cost aggregation values of pixels in other rows and columns in each image block in the first set of cost aggregation directions respectively corresponds to each other includes:

13. The method according to claim 10, wherein the determining, according to the matching cost aggregation values of a part of pixels in the reference row and the reference column included in each image block in the first set of cost aggregation directions, the matching cost aggregation values of pixels in other rows and columns in each image block in the first set of cost aggregation directions respectively corresponds to each other includes:

14. The method of claim 6, further comprising:

15. An image depth information determination apparatus, comprising a memory, a processor; wherein the memory has stored thereon executable code that, when executed by the processor, causes the processor to:

dividing a first image of the plurality of images into a plurality of blocks;

16. The apparatus of claim 15, wherein the processor is further configured to:

storing the first matching cost aggregate value in a storage space;

storing the second matching cost aggregate value into the storage space.

17. The apparatus of claim 15, wherein the processor is configured to:

18. The apparatus of claim 17, wherein the processor is configured to:

19. The apparatus of claim 17, wherein the processor is configured to:

20. An image depth information determination apparatus, comprising a memory, a processor; wherein the memory has stored thereon executable code that, when executed by the processor, causes the processor to:

21. The apparatus of claim 20, wherein the processor is configured to:

22. The apparatus of claim 20, wherein the processor is configured to:

23. The apparatus of claim 22, wherein the processor is configured to:

rotating the image matrix by a preset angle;

24. The apparatus of claim 22, wherein the processor is configured to:

25. The apparatus of claim 24, wherein the processor is configured to:

26. The apparatus of claim 24, wherein the processor is configured to:

27. The apparatus of claim 24, wherein the processor is configured to:

28. The apparatus of claim 20, wherein the processor is further configured to:

29. An electronic device, comprising:

a photographing device;

and an image depth information determining apparatus as claimed in any one of claims 15 to 28.

30. A computer-readable storage medium, characterized in that the storage medium is a computer-readable storage medium having stored therein program instructions for implementing the image depth information determination method according to any one of claims 1 to 14.