CN115661223B

CN115661223B - Light field depth estimation method, light field depth estimation device, computer equipment and storage medium

Info

Publication number: CN115661223B
Application number: CN202211577595.8A
Authority: CN
Inventors: 王彦; 韩凯; 崔文达; 来文昌; 梦琪; 雷国忠; 陈俊侣
Original assignee: National University of Defense Technology
Current assignee: National University of Defense Technology
Priority date: 2022-12-09
Filing date: 2022-12-09
Publication date: 2023-03-28
Anticipated expiration: 2042-12-09
Also published as: CN115661223A

Abstract

The application relates to a light field depth estimation method, a light field depth estimation device, a computer device and a storage medium. The method comprises the following steps: traversing the center coordinates of all microlenses in the microlens array according to the center calibration result of the microlens array, establishing index numbers of the microlenses, establishing a kdtree data structure according to the center coordinates of the microlenses in the microlens array, traversing kdtree by adopting a KNN algorithm, establishing a temporary tuple data structure to store microlens information in a microlens proximity domain, then performing registration twice through the microlens information, namely coarse registration and fine registration respectively, thereby establishing a depth estimation model according to the registration result, solving the depth estimation model and obtaining light field depth information. By adopting the method, the speed of the depth calculation of the optical field can be improved, and the random error of the depth estimation result can be effectively reduced.

Description

Light field depth estimation method, light field depth estimation device, computer equipment and storage medium

Technical Field

The present application relates to the field of light field depth estimation technologies, and in particular, to a light field depth estimation method, apparatus, computer device, and storage medium.

Background

Light field camera technology is an emerging field of research to achieve specific functions by jointly optimizing optical systems and signal processing. It is not a simple complement to optical imaging and digital image processing, but an organic combination of optical manipulation of the front-end physical domain and the back-end digital domain. The light field camera obtains multi-dimensional images and information in a calculation reconstruction mode by carrying out optical coding and mathematical modeling on an illumination and imaging system. Among many light field camera models, the focus light field camera model is the most excellent light field camera model at present, and is generally called as a light field camera 2.0. The focusing type light field camera is different from a general camera in that it is mounted with a micro lens array at a specific position in front of a camera target surface. When the microlens array is located at the main lens focal plane, the focusing light field camera is degraded to a conventional light field camera, i.e., light field camera 1.0. In the 2.0 model of the light field camera, light rays are converged from a space object point through a main lens and then intersect with a target surface through the center of a micro lens to obtain a target surface imaging image, and the image is called a light field data image. The light field data image is formed by arranging a plurality of microlens subaperture images on the target surface of the camera according to the arrangement mode of the microlens array, so that the light field data image is not a focusing image which is used by human eyes. From the structure of the light path, the 2.0 model of the light field camera is that the micro lens array carries out secondary imaging on the primary imaging point of the main lens. Therefore, the imaging positions of the secondary imaging points of the object points in the sub-aperture images of the different microlenses are different, namely, the parallax exists between the different microlenses. The image depth of the object point can be estimated by using the disparity, and the image depth is called virtual depth or depth. In the light field camera model, the virtual depth represents a normalized value of the distance between the primary imaging point and the microlens array plane. The light field camera 2.0 needs to first calculate the virtual depth values of the object points and generate a depth image of the full field of view before computing the in-focus imaging, a process called depth estimation. Based on the depth estimation result, the actual coordinates of the secondary imaging point of any object point in the field of view on the light field data image can be obtained through a specific algorithm, and then the focused image is obtained through pixel superposition. Thus, in general, the core of light field data image decoding is depth estimation decoding.

The depth estimation process is a very complicated calculation process, and not only needs to accurately know the center coordinates, the size and the type (classified according to focal length) of the micro lens, but also needs algorithms such as image feature point registration, virtual depth value calculation, depth image coordinate positioning and the like to estimate the depth value of the object point and the image coordinate of the object point. Considering the complexity of the implementation, real-time full depth-of-field estimation is very difficult. At present, the company Raytrix in Germany has introduced Raytrix series light field cameras based on the light field camera 2.0 model of a multi-focal-length microlens array, which is also referred to as light field camera 2.5. The light field camera 2.5 is identical to the light field camera 2.0 in the light path structure, and only the micro lens array adopts 3 kinds of micro lenses with different focal lengths to be arranged according to a certain rule, so that the detection depth of field is further expanded. However, because the time complexity and the space complexity of the depth estimation algorithm are too high, the Raytrix camera needs to be used by a computer equipped with a very high-performance video card, and the frame rate is low, so that the real-time processing is difficult to achieve. In addition, the depth estimation method of the Raytrix light field camera searches the micro lenses from near to far according to a specific path and finds the registration characteristic points under the micro lenses, and the depth estimation result is obtained based on the parallax of the micro lenses to which the two registration points which are farthest away belong. However, in principle, the random error of depth estimation using the parallax between two microlenses is much larger than the result of depth estimation using a plurality of microlenses. This also makes the actual accuracy of Raytrix in making three-dimensional measurements still less than satisfactory.

Disclosure of Invention

In view of the above, it is necessary to provide a light field depth estimation method, apparatus, computer device and storage medium for solving the above technical problems.

A light field depth estimation method, the method comprising:

traversing the center coordinates of all microlenses in the microlens array according to the center calibration result of the microlens array, and establishing index numbers of the microlenses;

establishing a kdtree data structure according to the central coordinates of the microlenses in the microlens array;

traversing kdtree by adopting a KNN algorithm, establishing a temporary tuple data structure to store microlens information in a microlens proximity domain, determining an initial feature point from a sub-aperture image of a first microlens in the temporary tuple, and retrieving the proximity domain in the temporary tuple data structure according to the microlens corresponding to the sub-aperture image;

performing coarse registration on the adjacent domain microlens sub-image by adopting a phase correlation method according to the initial feature point to obtain a coarse registration point in the adjacent domain sub-image;

the neighbor matrix of the initial characteristic point and the neighbor matrix of the rough registration point are subjected to up-sampling, and the two up-sampled neighbor matrices are subjected to phase coherent calculation to obtain an accurate registration point;

and constructing a depth estimation model according to the coordinate information of the precise registration point and the offset from the precise registration point to the center of the corresponding micro lens, and solving the depth estimation model to obtain the light field depth information.

In one embodiment, the temporary tuple data structure is { microlens index, microlens center coordinates, distance of microlens center from initial microlens }.

In one embodiment, the method further comprises the following steps: intercepting a first feature point matrix with a preset size by taking the initial feature point as a center, and intercepting a first registration point matrix with the size of the first feature point matrix by taking the coarse registration point as a center; calculating a correlation coefficient of the first characteristic point matrix and the first registration point matrix, and calculating distance information between the coarse registration point and a characteristic point base line; and when the correlation coefficient and the distance information meet a threshold range, judging that the rough calibration is finished.

In one embodiment, the first feature point matrix and the first registration point matrix are both 5 × 5 matrices.

In one embodiment, the method further comprises the following steps: intercepting a second feature point matrix with a preset size by using the initial feature point, and intercepting a second registration point matrix with the size of the second feature point matrix by using the coarse registration point as a center; performing up-sampling on the second characteristic point matrix and the second registration point matrix by preset times to obtain an expanded characteristic point matrix and an expanded registration point matrix; and carrying out phase coherent calculation on the expanded characteristic point matrix and the expanded registration point matrix to obtain sub-pixel precision registration points until the matching of all precision registration points in the neighborhood is completed.

In one embodiment, the method further comprises the following steps: and the second characteristic point matrix and the second registration point matrix are both 3 x 3 matrixes.

In one embodiment, the method further comprises the following steps: according to the coordinate information of the accurate registration point and the offset information from the accurate registration point to the center of the corresponding micro lens, constructing a depth estimation model as follows:

，

wherein the content of the first and second substances,

and/or>

The offset of the precise registration point, which is the object point, in the X-direction and in the Y-direction relative to the center of the associated microlens, respectively, is based on>

And &>

Is the X coordinate and the Y coordinate of the center of the micro lens to which the precise registration point belongs>

And/or>

Is the image coordinate of the object point in the X direction and the Y direction>

Representing depth information;

the step of solving the depth estimation model comprises:

changing the depth estimation model into a homogeneous equation set as follows:

，

wherein, the first and the second end of the pipe are connected with each other,

，/>

and with

Offset vectors which are formed in each case by the offset of the precise registration point in relation to the respective associated microlens center in the X-direction and in the Y-direction>

And &>

Respectively forming a microlens center coordinate vector by the center X coordinate and the center Y coordinate of the microlens to which the precise registration point belongs; solving the homogeneous equation set by adopting SVD (singular value decomposition) to obtain the depth value of the light fieldvAnd the image coordinates of the object point.

A light field depth estimation device, the device comprising:

the data structure component module is used for traversing the center coordinates of all the micro lenses in the micro lens array according to the center calibration result of the micro lens array and establishing the index numbers of the micro lenses; establishing a kdtree data structure according to the central coordinates of the microlenses in the microlens array;

the registration module is used for traversing kdtree by adopting a KNN algorithm, establishing a temporary tuple data structure to store microlens information in a microlens proximity domain, determining an initial feature point from a sub-aperture image of a first microlens in the temporary tuple, and retrieving the proximity domain in the temporary tuple data structure according to the microlens corresponding to the sub-aperture image; performing coarse registration on the adjacent domain microlens sub-image by adopting a phase correlation method according to the initial feature point to obtain a coarse registration point in the adjacent domain sub-image; the neighbor matrix of the initial characteristic point and the neighbor matrix of the rough registration point are subjected to up-sampling, and the two up-sampled neighbor matrices are subjected to phase coherent calculation to obtain an accurate registration point;

and the solving module is used for constructing a depth estimation model according to the coordinate information of the accurate registration point and the offset information from the accurate registration point to the corresponding object point, and solving the depth estimation model to obtain the light field depth information.

A computer device comprising a memory storing a computer program and a processor implementing the following steps when the computer program is executed:

A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:

traversing kdtree by adopting a KNN algorithm, establishing a temporary tuple data structure to store microlens information in a microlens neighborhood, determining an initial feature point from a sub-aperture image of a first microlens in the temporary tuple, and retrieving the neighborhood in the temporary tuple data structure according to the microlens corresponding to the sub-aperture image;

the neighbor matrix of the initial characteristic point and the neighbor matrix of the rough registration point are up-sampled, and the two up-sampled neighbor matrices are subjected to phase coherent calculation to obtain an accurate registration point;

and constructing a depth estimation model according to the coordinate information of the accurate registration point and the offset from the accurate registration point to the center of the corresponding micro lens, and solving the depth estimation model to obtain the light field depth information.

According to the light field depth estimation method, the light field depth estimation device, the light field depth estimation computer equipment and the storage medium, firstly, according to a center calibration result of a micro lens array, center coordinates of all micro lenses in the micro lens array are traversed, index numbers of the micro lenses are established, a tree data structure is established through the center coordinates of the micro lenses in the micro lens array, all micro lenses in an adjacent domain in the tree data structure are traversed for each micro lens and are stored in a tuple data structure according to a preset sequence, initial feature points are determined through the establishment of the data structure when the registration is carried out, only the adjacent domain needs to be searched from the tuple data structure, the calculation amount during the registration can be greatly saved, then, the sub-pixel level registration can be realized through two times of registration, and on the basis of the sub-pixel level registration, a depth estimation model is established, so that a depth estimation result is obtained.

Drawings

FIG. 1 is a flow diagram of a light field depth estimation method in one embodiment;

FIG. 2 is a schematic diagram of a microlens retrieval path in one embodiment;

FIG. 3 is a schematic view of a 2.0 depth estimation model of a light field camera in another embodiment;

FIG. 4 is a block diagram showing the structure of a light field depth estimating apparatus according to an embodiment;

FIG. 5 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

In one embodiment, as shown in fig. 1, there is provided a light field depth estimation method, comprising the steps of:

and 102, traversing the center coordinates of all the microlenses in the microlens array according to the center calibration result of the microlens array, and establishing the index numbers of the microlenses.

The invention adopts the light field depth estimation of the image shot by the micro lens array. The principle of the micro-lens calibration is as follows: the white light is made to diffuse and reflect into the light field camera to generate special light field data image, and when the main lens of the camera is adjusted to match the number of the micro lenses F, the arrangement, shape and size of the sub-aperture image of the micro lenses in the light field data image are completely consistent with the arrangement, shape and size of the micro lenses in the micro lens array, and the sub-aperture images are closely connected and do not overlap with each other. Each microlens sub-aperture image pixel value exhibits a gradual distribution with the center strongest edge being slightly weaker. Parameters such as white balance parameters, microlens size, microlens center coordinates, pixel vignetting correction parameters and the like of the camera can be obtained through an image processing means.

104, establishing a kdtree data structure according to the central coordinates of the microlenses in the microlens array;

in this step, a kdtree data structure is established according to the microlens index and the center coordinates of the microlens. In the depth estimation process, the index and the center coordinate of the micro lens in the adjacent domain are quickly searched on kdtree by adopting a KNN algorithm, a temporary metadata structure is established according to the sequence from far to near, and the relative position relation among the micro lenses and the center coordinate of the micro lens in the adjacent domain are stored. Therefore, for any microlens, all feature points in the sub-aperture image of the microlens can search for registration points in the microlens in the same metadatum, and therefore the traversing speed of the microlens is greatly improved. KNN (K-nearest neighbor) algorithm, K-nearest neighbor algorithm: if a sample belongs to a certain class in the K most similar samples in the feature space (i.e., the nearest neighbors in the feature space), then the sample also belongs to this class. The method only determines the category of the sample to be classified according to the category of the nearest sample or a plurality of samples in the classification decision.

And 106, traversing kdtree by adopting a KNN algorithm, establishing a temporary tuple data structure to store the microlens information in the microlens proximity domain, determining an initial characteristic point from the sub-aperture image of the first microlens in the temporary tuple, and searching the proximity domain in the temporary tuple data structure according to the microlens corresponding to the sub-aperture image.

And 108, performing coarse registration on the microlens sub-images in the adjacent domain by adopting a phase correlation method according to the initial characteristic points to obtain coarse registration points in the sub-images in the adjacent domain.

According to the epipolar constraint, the registration point of a feature point in a microlens sub-aperture image should be located on a straight line passing through the feature point and parallel to the baseline, and the straight line is called the feature point baseline.

And 110, performing up-sampling on the neighbor matrix of the initial characteristic point and the neighbor matrix of the coarse registration point, and performing phase coherent calculation on the two up-sampled neighbor matrices to obtain the accurate registration point.

And 112, constructing a depth estimation model according to the coordinate information of the precise registration point and the offset from the precise registration point to the center of the corresponding micro lens, and solving the depth estimation model to obtain the light field depth information.

According to the light field depth estimation method, firstly, according to a center calibration result of a micro-lens array, the center coordinates of all micro-lenses in the micro-lens array are traversed, index numbers of the micro-lenses are established, a tree data structure is established through the center coordinates of the micro-lenses in the micro-lens array, all micro-lenses in adjacent domains in the tree data structure are traversed by each micro-lens, the micro-lenses are stored in a tuple data structure according to a preset sequence, initial feature points are determined during registration through the establishment of the data structure, only the adjacent domains need to be searched from the tuple data structure, the calculation amount during registration can be greatly saved, then, sub-pixel level registration can be achieved through twice registration, and on the basis of the sub-pixel level registration, a depth estimation model is established, so that a depth estimation result is obtained.

It is noted that, in the above steps 106-110, the registration algorithm in the conventional technology only traverses pixels and registers on the interval where the characteristic point base line is located in the adjacent microlens sub-aperture image, and the registration algorithm generally adopts a correlation coefficient method. Although this approach can greatly reduce the number of pixel registration times, the amount of calculation is still considerable. Especially, when sub-pixel precision registration is performed, pixels on the characteristic point baseline need to be up-sampled, and the number of times of relevant registration is still large. In addition, the accuracy of the method for traversing the registration along a specific straight line cannot be guaranteed. This is because: firstly, the base line direction depends on the connecting line direction of the central coordinates of the two microlenses, and the calibration of the central coordinates of the microlenses has random errors, so that the theoretical optimal registration point is probably not on the base line of the characteristic point; secondly, the assembly error and the mirror surface phase difference of the micro lens array can also cause the theoretical optimal registration point not to be on the characteristic point baseline; and thirdly, the target surface pixels have certain actual sizes, and a straight line does not necessarily pass through the centers of the pixels when passing through the pixels. However, by the technical means, the phase coherence calculation is insensitive to illumination, has good stability and has certain noise immunity. Meanwhile, compared with the traditional pixel-by-pixel registration method, the method has remarkable speed advantage.

In one embodiment, the temporary tuple data structure is { microlens index, microlens center coordinates, distance of microlens center from initial microlens }. By the aid of the element group data structure, coordinate information, distances and the like of the micro lenses can be quickly searched.

In one embodiment, after the coarse registration, the coarse registration result needs to be verified, and the specific steps are as follows: intercepting a first feature point matrix with a preset size by taking the initial feature point as a center, and intercepting a first registration point matrix with the size of the first feature point matrix by taking the coarse registration point as the center; calculating a first characteristic point matrix and a correlation coefficient of the first registration point matrix, and calculating distance information between the registration point and a characteristic point base line; and when the correlation coefficient and the distance information meet the threshold range, judging that the rough calibration is finished. Specifically, the first feature point matrix and the first registration point matrix are both 5 × 5 matrices.

In one embodiment, the step of accurately registering comprises: intercepting a second feature point matrix with a preset size by using the initial feature points, and intercepting a second registration point matrix with the size of the second feature point matrix by using the coarse registration points as centers; performing up-sampling on the second characteristic point matrix and the second registration point matrix by preset times to obtain an expanded characteristic point matrix and an expanded registration point matrix; and carrying out phase coherent calculation on the expanded characteristic point matrix and the expanded registration point matrix to obtain sub-pixel precision registration points until the matching of all precision registration points in the neighborhood is completed. Specifically, the second feature point matrix and the second registration point matrix are both 3 × 3 matrices. If the upsampling multiple is m, then the registration accuracy is 1/m pixels.

In one embodiment, the step of obtaining light field information comprises: according to the coordinate information of the accurate registration point and the offset information from the accurate registration point to the corresponding object point, a depth estimation model is constructed as follows:

，

wherein the content of the first and second substances,

and/or>

And/or>

And/or>

Representing a depth value;

the step of solving the depth estimation model comprises:

the depth estimation model is expanded into a homogeneous equation set as follows:

，

wherein the content of the first and second substances,

，/>

and

offset vectors which are formed in each case by the offset of the precise register point in relation to the respective associated microlens center in the X-direction and in the Y-direction>

And &>

Respectively forming a microlens center coordinate vector by the center X coordinate and the center Y coordinate of the microlens to which the precise registration point belongs; and solving a homogeneous equation set by adopting SVD (singular value decomposition) to obtain a depth value and a coordinate.

It is worth noting that the conventional light field camera 2.0 estimates the virtual depth value of the object point using the two farthest microlens disparities. However, due to the influence of system installation errors and calibration errors, for the same object point, even if the virtual depth value is obtained according to two pairs of microlens sub-aperture images with the same distance, a certain random error often exists. Especially when the virtual depth is small, this error is more significant. This introduces a large error uncertainty to the three-dimensional measurement of the light field camera. In addition, in the conventional light field camera 2.0, the image coordinates of the object point are determined according to the intersection point of the two registration points and the connecting line of the centers of the respective microlenses. However, due to the calibration accuracy of the center of the microlens, certain random errors may exist in the image coordinates determined based on different registration points, and the problem of inaccurate positioning of the image coordinates corresponding to the object points in the depth map is caused.

In particular, in the light field camera 2.0, for each microlens sub-aperture image that twice images an object point, the object pointpThe virtual depth of (a) is calculated as:

，

wherein, in the above formula

Is the virtual depth of the object point>

Is the distance from the secondary imaging point of the object point to the center of the microlens belonging to the object point, and/or>

The distance between the center of the micro lens to which the secondary imaging point belongs and the coordinate center of the object point image.

In the invention, the formula is expanded to three dimensions, specifically:

，/>

the solution of the three-dimensional calculation formula is not described herein again.

Examples of the present invention are explained below.

In the depth estimation process, the first step of the depth estimation process is to create a feature point image from the light field data image. The feature points are determined from sharp imaging points in the microlens sub-aperture image. Since a sharp imaging point in an image has a significant gradient feature, it can be determined whether it is a feature point from the gradient values of pixels in the light-field data image. In the implementation process, an all-0 matrix image with the size consistent with that of the light field data image may be established first, and once a certain pixel is determined as a feature point in the detection process of the feature point in the light field data image, the corresponding position of the empty matrix is marked as 1 until all the feature points are detected. In the depth estimation process, the feature point image can be traversed, a certain feature point is used as an initial registration point, a microlens is retrieved in the vicinity of the microlens, registration is carried out in the corresponding microlens sub-aperture image, and finally the virtual depth value of the object point is obtained through calculation. Because not all pixel points in the light field data image can become feature points, only a full-field sparse depth map can be obtained after the traversal calculation of the feature points in the feature point image is completed. The full-field dense depth map is obtained by interpolating the sparse depth map by an interpolation method. It is clear that the denser the sparse depth map, the finer the dense depth map. Therefore, when generating the feature point map, a gradient threshold value is reasonably selected, and an appropriate feature point is screened.

As shown in fig. 2, a schematic diagram of a microlens search path is provided, assuming that the initial feature point is located in microlens 0. According to the arrangement rule of the microlenses, taking the microlens 0 as the initial microlens, the arrangement of the microlenses in the neighboring domain can be regarded as an annular layered arrangement around the initial microlens with a specific distance, as shown by the dashed arc in fig. 2. The distance from the center of each layer of microlens to the center of the initial microlens is 1 time of the diameter of the microlens,

Double, 2 times and/or greater>

And (5) doubling, 3 times, and the like. According to the ray tracing principle, when registering the pointWhen the center of the microlens is located at the edge of the projection circular domain, the registration point should be located at the outermost side of the microlens sub-aperture image. At this time, the virtual depth calculation formula may be expressed as:

，

wherein, in the formula, the first and second groups,

in order to project the radius of the circular field,Dthe microlens diameter. Obviously, the larger the radius of the projection circle, the farther the farthest microlens having the registration point is from the initial microlens, and the larger the virtual depth value. Meanwhile, according to the binocular vision principle, the larger the virtual depth is, the longer the base line is, and the higher the theoretical limit precision of depth estimation is. The red route in fig. 2 is a path searched for a microlens when the depth of a conventional light field camera is estimated at 2.0, and a specific strategy of the depth estimation method is to perform feature point registration in an adjacent microlens from an initial microlens in a near-to-far manner until path search is finished when a valid registration point cannot be found. The last microlens for which the registration point exists is the most distant microlens, such as microlens 5-2 in fig. 2. Finally, the virtual depth values are estimated from the microlens pairs (0, 5-2) and their included registration point pairs. In fact, even though the distances from the microlenses 5-2, 5-1, and 5-2 to the microlens 0 are the same, the virtual depth values calculated from (0, 5-2), (0, 5-1), and (0, 5-3) and their corresponding feature points may be different, i.e., random errors exist, due to the influence of factors such as the aberration of the microlenses themselves and the mounting error of the microlens array. According to the distance parameter between the center of the micro lens in the micro lens information temporary data structure and the center of the initial micro lens, the micro lenses are searched layer by layer from near to far according to the annular layering in the figure 2 until no micro lens on a certain layer has a legal registration point. And finally, estimating the virtual depth value of the object point by adopting a multi-view visual depth estimation model. The distance from the center of the microlens to the center of the initial microlens in the microlens information data structure actually corresponds to the radius of the microlens arrangement layer, and is the main basis for performing microlens retrieval.

Fig. 3 is a schematic diagram of a 2.0 depth estimation model of a light field camera. In the context of figure 3, it is shown,

is a primary imaging point of the object point>

For a projection point of an image point on the microlens array surface, i.e.>

The coordinates of the imaging point corresponding to the object point. />

Is the center of the ith microlens>

Is->

To>

The distance of (c). The primary imaging point of the light passing through the main lens of the camera is projected on the micro-lens array surface, and the projection domain has a radius ofRAnd is round, andRsize and virtual depth->

It is related. When the microlens center is located at the projection circle region boundary, there is ^>

. It is clear that the virtual depth is essentially the normalized distance between the primary imaging point of the main lens and the microlens array. From the triangular similarity in fig. 3, it is easy to derive a theoretical model for depth estimation, and from fig. 3, additionally, at an imaginary depth->

When known, it is easy to calculate all the secondary imaging point coordinates of the object point as follows:

，

in the above formula, the first and second carbon atoms are,

and &>

Is the imaging point under the microlens relative to the center of the microlens

The amount of offset of (c). At this time, it is>

The corresponding pixel value of the focus imaging point is

. This is also the reason that the light field camera 2.0 has to generate a full field of view dense depth map before focused imaging.

It should be understood that, although the steps in the flowchart of fig. 1 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 1 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

In one embodiment, as shown in fig. 4, there is provided a light field depth estimation apparatus including: a data structure building block 402, a registration block 404, and a solution block 406, wherein:

a data structure component module 402, configured to traverse center coordinates of all microlenses in a microlens array according to a center calibration result of the microlens array, and establish an index number of the microlens; and establishing a kdtree data structure according to the central coordinates of the microlenses in the microlens array.

A registration module 404, configured to traverse kdtree by using a KNN algorithm, establish a temporary tuple data structure to store microlens information in a microlens proximity domain, determine an initial feature point from a sub-aperture image of a first microlens in the temporary tuple, and retrieve a proximity domain in the temporary tuple data structure according to a microlens corresponding to the sub-aperture image; performing coarse registration on the adjacent domain microlens sub-image by adopting a phase correlation method according to the initial feature point to obtain a coarse registration point in the adjacent domain sub-image; and performing up-sampling on the neighbor matrix of the initial characteristic point and the neighbor matrix of the rough registration point, and performing phase coherent calculation on the two up-sampled neighbor matrices to obtain an accurate registration point.

And the solving module 406 is configured to construct a depth estimation model according to the coordinate information of the precise registration point and the offset from the precise registration point to the center of the corresponding microlens, and solve the depth estimation model to obtain the light field depth information.

In one embodiment, the tuple data structure is { microlens index, microlens center coordinate, distance of microlens center from initial microlens }.

In one embodiment, the registration module 404 is further configured to intercept a first feature point matrix with a preset size with the initial feature point as a center, and intercept a first registration point matrix with the size of the first feature point matrix with the coarse registration point as a center; calculating a correlation coefficient of the first characteristic point matrix and the first registration point matrix, and calculating distance information between the registration point and a characteristic point base line; and when the correlation coefficient and the distance information meet a threshold range, judging that the rough calibration is finished.

In one embodiment, the registration module 404 is further configured to intercept a second feature point matrix of a preset size from the initial feature points, and intercept a second registration point matrix of a size of the second feature point matrix from the registration points; performing up-sampling on the second characteristic point matrix and the second registration point matrix by preset times to obtain an expanded characteristic point matrix and an expanded registration point matrix; and carrying out phase coherent calculation on the expanded characteristic point matrix and the expanded registration point matrix to obtain sub-pixel precision registration points until the matching of all precision registration points in the neighborhood is completed.

In one embodiment, the second feature point matrix and the second registration point matrix are both 3 × 3 matrices.

In one embodiment, the solving module 406 is further configured to construct a depth estimation model according to the coordinate information of the precise registration point and the offset information from the precise registration point to the corresponding object point, as follows:

，

wherein the content of the first and second substances,

and/or>

And &>

And/or>

Representing depth information;

the step of solving the depth estimation model comprises:

changing the depth estimation model into a homogeneous equation set as:

，/>

wherein the content of the first and second substances,

，/>

and with

And &>

Respectively forming a microlens center coordinate vector by the center X coordinate and the center Y coordinate of the microlens to which the precise registration point belongs; and solving the homogeneous equation set by adopting SVD (singular value decomposition) to obtain the depth information and coordinates of the light field.

For specific limitations of the light field depth estimation apparatus, reference may be made to the above limitations of the light field depth estimation method, which are not described herein again. The various modules in the light field depth estimation device described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 5. The computer device comprises a processor, a memory, a network interface, a display screen and an input device which are connected through a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a light field depth estimation method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.

Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In an embodiment, a computer device is provided, comprising a memory storing a computer program and a processor implementing the steps of the method in the above embodiments when the processor executes the computer program.

In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the method in the above-mentioned embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent application shall be subject to the appended claims.

Claims

1. A light field depth estimation method, the method comprising:

2. The method of claim 1, wherein the temporary tuple data structure is { microlens index, microlens center coordinates, distance of microlens center from initial microlens center }.

3. The method of claim 1, further comprising, after performing coarse registration in the neighborhood microlens subimages using a phase correlation method based on the initial feature points:

intercepting a first feature point matrix with a preset size by taking the initial feature point as a center, and intercepting a first registration point matrix with the consistent size of the first feature point matrix by taking the rough registration point as a center;

calculating a correlation coefficient of the first characteristic point matrix and the first registration point matrix, and calculating distance information between the registration point and a characteristic point base line;

and when the correlation coefficient and the distance information meet the threshold range, judging that the rough calibration is finished.

4. A method according to claim 3, wherein the first matrix of feature points and the first matrix of registration points are each a 5 x 5 matrix.

5. The method of claim 1, wherein upsampling the neighbor matrix of the initial feature point and the neighbor matrix of the coarse registration point, and performing phase coherent calculation on the two upsampled neighbor matrices to obtain a precise registration point, comprises:

intercepting a second feature point matrix with a preset size by using the initial feature point, and intercepting a second registration point matrix with the size of the second feature point matrix by using the coarse registration point as a center;

performing up-sampling on the second characteristic point matrix and the second registration point matrix by preset times to obtain an expanded characteristic point matrix and an expanded registration point matrix;

and carrying out phase coherent calculation on the expanded characteristic point matrix and the expanded registration point matrix to obtain sub-pixel precision registration points until the matching of all precision registration points in the neighborhood is completed.

6. The method of claim 5, wherein the second matrix of feature points and the second matrix of registration points are each a 3 x 3 matrix.

7. The method of any one of claims 1 to 6, wherein constructing a depth estimation model based on the coordinate information of the precision registration points and the offset of the precision registration points from the centers of the corresponding microlenses comprises:

according to the coordinate information of the accurate registration point and the offset from the accurate registration point to the center of the corresponding micro-lens, constructing a depth estimation model as follows:

，

wherein the content of the first and second substances,

and/or>

And/or>

And/or>

Representing depth information;

the step of solving the depth estimation model comprises:

expanding the depth estimation model into a matrix form of a homogeneous equation set as follows:

，

wherein the content of the first and second substances,

，/>

and/or>

Are each accurateThe offset vector is composed of offset of the registration point relative to the X direction and the Y direction of the center of each micro lens,

and &>

Respectively forming a micro lens center coordinate vector composed of a micro lens center X coordinate and a micro lens center Y coordinate to which the precise registration point belongs;

and solving the homogeneous equation set by adopting SVD (singular value decomposition) to obtain the depth information and coordinates of the light field.

8. A light field depth estimation apparatus, characterized in that the apparatus comprises:

the registration module is used for traversing kdtree by adopting a KNN algorithm, establishing a temporary tuple data structure to store microlens information in a microlens proximity domain, determining an initial feature point from a sub-aperture image of a first microlens in the temporary tuple, and retrieving the proximity domain in the temporary tuple data structure according to the microlens corresponding to the sub-aperture image; performing coarse registration on the adjacent domain microlens sub-image by adopting a phase correlation method according to the initial feature point to obtain a coarse registration point in the adjacent domain sub-image; the neighbor matrix of the initial characteristic point and the neighbor matrix of the rough registration point are up-sampled, and the two up-sampled neighbor matrices are subjected to phase coherent calculation to obtain an accurate registration point;

and the solving module is used for constructing a depth estimation model according to the coordinate information of the accurate registration point and the offset from the accurate registration point to the center of the corresponding micro lens, and solving the depth estimation model to obtain the light field depth information.

9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.