JP2024055772A - Computer vision method and system - Google Patents

Abstract

A computer vision method for generating a three-dimensional reconstruction of an object includes receiving a first set of photometric stereo images of the object, the first set including at least one image taken from a camera at a first position using illumination from a different direction, and a second set of photometric stereo images of the object, the second set including at least one image taken from a camera at a second position using illumination from a different direction, generating a first normal map of the object using the first set of photometric stereo images and a second normal map of the object using the second set of photometric stereo images, and determining a stereo estimate of the shape of the object by performing stereo matching between normal patches in the first normal map and normal patches in the second normal map. The normal map is used together with the first estimate of the shape of the object to generate a reconstruction of the object.

Description

Embodiments relate to computer vision systems and methods for performing 3D imaging of objects.

Many computer vision tasks require deriving an accurate 3D reconstruction of an object from the way it reflects light. However, reconstructing 3D geometry is a challenge because global lighting effects such as cast shadows, self-reflection, and ambient light come into play, especially for specular surfaces.

Photometric stereo is a long-standing problem in computer vision. Recent methods have achieved impressive normal estimation accuracy on both real and synthetic datasets. However, progress in the quality and practicality of the estimated shape is less compelling, as progress is severely limited due to inaccuracies in retrieving the global geometry, especially when dealing with general light reflections.

When photometric stereo techniques rely only on the acquisition of images from a single camera position (monocular photometric stereo), the accuracy of the reconstruction usually depends on a rough estimate of the distance between the camera and the object of interest. For example, an initial estimate of the object geometry is obtained before or after image acquisition, and a depth map is initialized based on the initial object geometry.

FIG. 1 is a schematic diagram of an example system useful for understanding the present invention. FIG. 2 is a schematic diagram showing an arrangement of cameras and light sources for performing 3D imaging of an object. FIG. 3 is a high-level schematic of a method for recovering surface normals from a set of photometric stereo images of an object or scene. FIG. 4 is a schematic diagram showing an arrangement of cameras and light sources for performing 3D imaging of an object. FIG. 5 is a flow diagram of a method for 3D reconstruction according to one embodiment. FIG. 6 is a schematic diagram of a CNN that can be used to extract a normal map from an observation map. FIG. 7 illustrates the operating principle of a reconstruction method according to an embodiment. FIG. 8 illustrates steps of a method according to one embodiment. FIG. 9 shows the results of our method compared to monocular (single-view) photometric stereo. FIG. 10 is a schematic diagram of a system according to one embodiment.

In a first aspect, there is provided a computer vision method for generating a three-dimensional reconstruction of an object, comprising the steps of:
receiving a first set of photometric stereo images of an object and a second set of photometric stereo images of the object, the first set including at least one image taken from a first camera at a first location using illumination from a different direction and the second set including at least one image taken from a second camera at a second location using illumination from a different direction;
generating a first normal map of the object using a first set of photometric stereo images;
generating a second normal map of the object using a second set of photometric stereo images;
determining a stereo estimate of the shape of the object by performing stereo matching between normal patches in the first normal map and normal patches in the second normal map;
A method is provided that includes using the first normal map and the second normal map together with a stereo estimate of the shape of the object to generate a reconstruction of the object.

The disclosed method is linked to computer technology and addresses a technical problem arising in the field of computing, namely, generating a 3D reconstruction of an object or a scene. The disclosed method solves this technical problem of how to improve the quality of the reconstruction. The improvement is provided by providing a binocular-based photometric stereo that merges the strengths of photometric stereo in estimating dense local shape changes and the strengths of Structure from Motion in providing a sparse but accurate depth estimation.

The object may be placed within the depth range of the capture device's camera, and the distance between the object and the camera (i.e., depth z) is approximately measured using a ruler. The depth map is then initialized by setting the depth of all points to a constant value, as estimated using the ruler, or other method of estimating the average distance from the camera to the object. Other methods of initializing the depth may be used, for example using a CAD model, a kinect-type depth sensor, etc. Adding any further sophistication in the estimation of the initial object geometry may reduce the execution time of the method, for example by a single iteration.

In short, an inherent ambiguity in photometric stereo arises from determining the rough depth of the target object from the camera. This is especially crucial for real-world applications where scale factors must be determined to ensure measurability of the geometry.

To address this issue, at least in part, binocular-based photometric stereo is described herein that merges the strengths of photometric stereo in estimating dense local shape changes with the strengths of structure-from-motion in providing sparse but accurate depth estimation.

The embodiments described herein perform matching on normals instead of the first two sets of images, allowing more reliable stereo matching to be performed that is robust even when dealing with objects that have no texture or gloss. As a result, constraining the reconstruction of the object to pairwise correspondences in the stereo matching step can provide a more accurate representation of the object geometry, which provides a better initial guess for the iterative procedure of updating the object geometry to converge to a single object geometry.

In one embodiment, the first normal map and the second normal map are generated using an estimate of the shape of the object to recalculate the light distribution for near-field effects of the illumination on the object.

Once a first reconstruction of the object is generated, the method includes recalculating the light distribution due to near-field effects using the reconstruction of the object, recalculating the first normal map and the second normal map from the recalculated light distribution, and generating a further reconstruction of the object.

The above may be used in an iterative manner, where recalculating the light distribution due to near field effects includes:
(a) using a reconstruction of the object to recalculate a first normal map and a second normal map from a recalculated light distribution;
(b) determining a further estimate of the shape of the object by performing stereo matching between patches of normals in the recomputed first normal map and patches of normals in the recomputed second normal map;
(c) generating a further reconstruction of the shape from at least one of the recalculated first normal map and the recalculated second normal map; and
and repeating (a) through (c) until a further reconstruction of the object converges.

In one embodiment, using the first normal map and the second normal map together with a stereo estimate of the shape of the object to generate a reconstruction of the object includes:
Integrating the first normal map with a stereo estimation constraint of the shape to generate a first reconstruction;
Integrating the second normal map with constraints of the stereo estimation of shape to generate a second reconstruction;
and combining the first reconstruction and the second reconstruction to generate a fused reconstruction, which is a reconstruction of the object.

The fused reconstruction can be generated using a Poisson solver.

Performing stereo matching on the first normal map and the second normal map includes:
Selecting at least one group of pixels on the first normal map;
and searching for a matched group of pixels in the second normal map by scanning across the second normal map for a match.

To reduce the amount of data processed while searching for a match, scanning over the second normal map for a match may be performed over the epipolar lines. The search for a matched pixel group may also be constrained by the current estimated reconstruction of the object. Thus, when the object shape converges, searching for a match can be done more efficiently.

In a further embodiment, performing stereo matching on the first normal map and the second normal map includes:
performing patch warping on at least one group of pixels of the first normal map;
and determining a corresponding group of pixels in a second normal map using the patch-warped at least one group of pixels in the first normal map.

In a further embodiment, searching for matched pixel groups in the second normal map by scanning across the second normal map for matches is used to generate a first partial stereo estimate;
The method is:
Selecting at least one group of pixels on the second normal map;
searching for matched pixel groups in the first normal map by scanning across the first normal map for matches to generate a second partial stereo estimate;
and further comprising combining the first partial stereo estimate and the second partial stereo estimate to form a stereo estimate of the shape, wherein points from the first partial stereo estimate and the second partial stereo estimate that do not match are discarded.

The above allows two stereo maps to be generated that are combined to provide robustness to errors, since points on the partial stereo map that show different depths for the same point can be discarded.

The first set of photometric stereo images includes a first plurality of images taken from a first camera at a first location using illumination from different directions, and the second set includes a second plurality of images taken from a second camera at a second location using illumination from different directions.

In one embodiment, generating the first normal map includes inputting information representing the first set of photometric stereo images, an estimate of the object's shape, and light source position information into a neural network trained to output the first normal map.

For example, information representing a first set of photometric stereo images, an estimate of the object's shape, and light source location information is provided in the form of an observation map, where an observation map is generated for each pixel of the first camera, where each observation map includes a projection of a lighting direction onto a 2D plane, where the lighting direction for each pixel is obtained from each photometric stereo image.

In a further embodiment, there is provided a system for generating a three-dimensional (3D) reconstruction of an object, the system comprising an interface and a processor:
The interface has an image input and is configured to receive a set of photometric stereo images of the object, the set of photometric stereo images including a plurality of images using illumination from different directions with one or more light sources;
The processor
receiving a first set of photometric stereo images of an object and a second set of photometric stereo images of the object, the first set including at least one image taken from a first camera at a first location using illumination from a different direction and the second set including at least one image taken from a second camera at a second location using illumination from a different direction;
generating a first normal map of the object using a first set of photometric stereo images;
generating a second normal map of the object using a second set of photometric stereo images;
determining a stereo estimate of the shape of the object by performing stereo matching between patches of normals in the first normal map and patches of normals in the second normal map;
The system is configured to use the first normal map and the second normal map together with a stereo estimate of the shape of the object to generate a reconstruction of the object.

In the above system, the first camera is configured to be different from the second camera. In a further embodiment, a single camera is used that is moved between the first and second positions.

A carrier medium may be provided carrying computer readable instructions adapted to cause a computer to carry out the method as described above.

Figure 1 shows a schematic diagram of a system that may be used to capture three-dimensional (3D) image data of an object and reconstruct the object. As used herein, the term "object" is used to denote what is being imaged. However, it should be understood that the term may encompass multiple objects, a background, or a combination of objects and background, etc.

3D image data of object 10 is captured using device 11. Further details of device 11 are provided with reference to FIG. 2.

The 3D image data captured by device 11 is sent to computer 12, where the 3D image data is processed. In FIG. 1, computer 12 is shown as a desktop computer, but it should be understood that it may be any processor, such as a distributed processor, or a processor in a mobile phone. Details of an exemplary processor are described further below in this description with reference to FIG. 10.

The system of FIG. 1 may be installed in an existing hardware setup and used for quality control of the setup's process. Such setups include, but are not limited to, 3D printing setups and industrial pipelines. For example, the system of FIG. 1 may be installed in a 3D printer setup where the system is used to perform quality control of the printing process. More specifically, the system may be used to capture photometric stereo images of intermediate results of the printing process and verify proper execution of the printing process. Furthermore, the system may be implemented as a handheld device used by a user to obtain 3D models of everyday items.

Figure 2 shows an example arrangement of the apparatus 11 of Figure 1 that can be used for photometric stereo. Figure 2 shows a mount that holds a camera 20 with multiple pixels and multiple light sources 21 in a fixed relationship to each other and to the camera 20. This arrangement of the apparatus 11 allows the camera 20 and multiple light sources 21 to be moved together while maintaining a fixed separation from each other.

In a particular exemplary arrangement of the apparatus 11, the light sources 21 are arranged surrounding the cameras 20. However, it is understood that the light sources 21 and the cameras 20 may be arranged differently. The cameras 20 are used together with the light sources 21 to obtain photometric stereo data of the object 10. The individual light sources 21 are activated in sequence to enable the cameras 20 to capture the photometric stereo data.

In a particular arrangement of the device 11, a FLEA 3.2 megapixel camera with an 8 mm lens is used as the camera 20. The camera 20 has an 8 mm lens and is rigidly mounted to a printed circuit board. The printed circuit board further comprises 16 white bright LEDs used as light sources 21 arranged in a flush manner with the image plane and surrounding the camera 20 with a maximum difference of 6.5 centimeters.

The device 11 may be used to capture photometric stereo images of an object 10. The object 10 is placed in front of the camera 20 within the depth range of the camera 20. The depth range, also called depth of field (DOF), is a term used to indicate the range of distances between the camera 20 and the object 10 within which the object 10 is in focus. If the object 10 is too close or too far from the camera 20 outside the depth range of the camera 20, the object will be out of focus and no details can be resolved. For example, a camera 20 equipped with an 8 mm lens may have a depth range between 5 centimeters and 30 centimeters.

The light sources 21 are activated individually and in sequence to enable the camera 20 to capture photometric stereo data of the object under different lighting conditions. This is achieved by switching on only one light source at a time. For example, the device 11 may be equipped with 16 LEDs and therefore a set of 16 photometric stereo images may be captured.

In photometric stereo, each light source 21 is individually activated relative to the object 10 to capture photometric stereo data of the object under different lighting conditions. This is accomplished by switching on only one light source at a time. The amount of light reflected from a known surface or object onto each pixel of the camera 20 is then measured. For example, the device 11 may be equipped with 16 LEDs, and thus a set of 16 photometric stereo images may be captured.

Photometric stereo allows the surface normal to be estimated. In its most basic form, for any point on a Lambertian surface, the reflected light intensity I at the camera is

can be expressed as:

where I is the reflected light intensity at the camera, L is the normal vector corresponding to the light direction of the illuminating light, n is the normal to the surface at that point, and k is the albedo reflectance at that point. By solving the above for three or more different light directions, it is possible to estimate the surface normal n. Once all of the surface normals have been determined for all points on the surface, it is possible to reconstruct the surface by integrating the normals.

In practice, the situation is more complicated, since surfaces are unlikely to have a constant albedo. Near-field optical attenuation can also affect the shape of objects reconstructed using photometric stereo.

To model this, the following may be used, where each set of photometric stereo images contains m images, the number of images corresponding to the number of lighting directions (which in this example corresponds to the number of light sources used): Each image i _j,p , for j=1,...,m, can be viewed as a set of pixels p. For each of the m light sources, the direction L _j and the intensity Φ _j are known.

As mentioned above, near-field optical attenuation affects the reflection of light from an object. Near-field optical attenuation is modeled using the following nonlinear radiation model of dissipation:

where Φ _m is the intrinsic luminance of the light source, S _m is the principal direction of the LED point source, μ _m is the angular dissipation coefficient, and the lighting direction is

is expressed as:

We assume a calibrated point light source at position P _m relative to the camera center at point 0, which results in a variable light vector L _m =P _k -X, where X is the 3D surface point coordinate, denoted as X = [x, y, z] ^T.

Gaze vector

of

The general image irradiance equation is:

where N is the surface normal, B is assumed to be a general bidirectional reflectance distribution function (BRDF), and ρ is the surface albedo, where the albedo ρ and the images are RGB and the reflectance varies per channel, thus allowing for the most general case. Additionally, global lighting effects such as shadows and self-reflection can also be incorporated in B.

To allow the above to be modeled, the photometric stereo images are converted into observation maps. Observation maps are generated to account for pixel-by-pixel illumination information. In one embodiment, an observation map is generated for each pixel. Each observation map contains a projection of the lighting direction onto a 2D plane, where the lighting direction for each pixel is obtained from each photometric stereo image.

The details of how the observation map may be constructed will be explained later. However, in short, the observation map is generated using _Lm , the light vector. As mentioned above, this is defined relative to the 3D location of the point on the surface of the object that corresponds to the pixel. Thus, the shape of the object affects _Lm and therefore the observation map. When the observation map is generated for the first time, a simple estimate of the shape is used. For example, it may be assumed that all pixels are at a constant distance from the camera.

The observation map is intended to take into account all lighting directions at each surface pixel and to allow merging the information of a variable number of images into a single dxd image map.

Given multiple images, for each pixel p of camera 20, its value in image j is denoted as i _j,p . All observations of a pixel in images with modified illumination are combined into a single d x d x 4 map O, where d is the dimension. How this is achieved is described below.

Figure 3 shows a high-level diagram of the basic steps of a method for recovering surface normals from a set of photometric stereo images of an object or scene, which can take into account near-field optical effects.

Images of the object are obtained using device 11 according to the procedure described above. Alternatively, photometric stereo images may be obtained from a remote photometric stereo imaging device and communicated to computer 12, thus providing it as input.

For each pixel in the photometric image, an observation map is rendered by combining all photometric image pixel observations onto the observation map (as described above). The observation map is then processed to generate a normal (orientation) map. In one embodiment, the observation map may be processed by a convolutional neural network (CNN). A possible configuration of such a CNN is described with reference to FIG. 6.

Figure 4 shows an apparatus 40 according to one embodiment. Figure 4 shows a mount that holds a first camera 41 and a second camera 42. The mount further holds a number of light sources 43 in a fixed relationship to each other and to the first camera 41 and the second camera 42. This arrangement of the apparatus 40 allows the first camera 41, the second camera 42, and the multiple light sources 43 to be moved together while maintaining a fixed separation from each other.

In one embodiment, 15 LEDs are used as the multiple light sources, but any number and arrangement of light sources may be used as long as their positions relative to each other and to the first camera 41 and second camera 42 are known.

Although FIG. 4 shows an apparatus with a first camera 41 and a second camera 42, the described method may be performed with a single camera, where the single camera is configured to move between the position of the first camera 41 in FIG. 4 and the position of the second camera 42 in FIG. 4 for a given position of the LED and known distances between the first position and the second position, the first position and the LED, and the second position and the LED.

Figure 5 is a flow chart outlining a method according to one embodiment.

In step S501 of data capture using camera 1, a first set of photometric stereo images of the object is acquired. For example, the first set of photometric stereo images may be acquired from a first camera (or camera 1) shown in FIG. 4 at a first position. At the same time, in step S503 of data capture using camera 2, a second set of photometric stereo images is acquired from a second camera (or camera 2) shown in FIG. 4 at a second position. Both the first and second sets of images are acquired under known light (LED) positions and known camera positions as described above.

In an alternative embodiment, two sets of photometric stereo images of the object are acquired from a memory device, where the two sets of photometric stereo images were acquired once in advance before the execution of the described method. In a further variation, there is only one camera, where a first set of photometric stereo images is acquired at a first position and a second set of photometric stereo images is acquired at a second position by the same camera.

For ease of reference in the following description, the first set of photometric stereo images may be referred to as left stereo images and the second set of photometric stereo images may be referred to as right stereo images. However, the first location may be to the right of the second location, above the second location, below the second location, or at any location away from the second location. The terminology of first set, second set, left and right may be used to distinguish an image or set of images obtained from a camera at the first location from an image or set of images obtained from a camera at the second location.

Each set of photometric stereo images contains m images, the number of images corresponding to the number of light sources used. Each image i _j,p can be seen as a set of pixels p, for j=1,...,m. For each of the m light sources, the direction L _j and the intensity Φ _j are known and are used to calculate the normal N _p .

In step S505, a normal map is generated from the data capture step S501. This is called the left normal map N _L. In step S507, a normal map is generated from the data capture step S503. This is called the right normal map N _R.

In this embodiment, the left and right normal maps are generated using the method described above, with the left observation map generated from data from camera 1 and the right observation map generated from data from camera 2. To generate these initial observation maps, an estimate of the initial object geometry z _est is used. This may be a very rough depth initialization z _est (which may be accurate to within a few centimeters). The estimate of the object geometry comprises a depth map that has a constant value corresponding to z _est .

The two sets of photometric images provided by S501 are assumed to provide several calibrated stereo pairs consisting of left and right views (views from a first camera position and a second camera position).

Assuming a camera focal length f and a stereo baseline b, the well-known relationship between depth (z) and disparity (d) is:

applies.

Given photometric stereo images for both the first and second positions, and a depth initialization z _est , left and right normal maps N _L and N _R can be calculated.

In step S509, stereo matching is performed on the left and right normal maps. In this step, stereo matching (or disparity estimation) is performed on both the left and right normal maps to output a set of sparse-pair, shape-preserving correspondences between the left and right normal maps.

The normal maps N _L and N _R from S505 and S507 are expected to be accurate to within a few degrees relative to the ground truth. However, the depth maps (Z _0L and Z _0R , which are calculated from N _L and N _R , respectively, using numerical integration) suffer from scale ambiguity. This is because numerical integration preserves the overall average depth, i.e.,

This is a necessary consequence of the single-view version of the problem, since . Thus, for an almost perfectly integrable surface with true depth _ZT ,

It is expected that.

In practice, equation (6) only applies to very smooth surfaces and is very sensitive to error propagation for any reasonably sized image. This in turn means that the overall shape of the object, _Z0 , will exhibit some low-frequency deformation or bending. However, if attention is restricted to small sized image patches, the smoothness constraint is much easier to satisfy (for patches that do not contain depth discontinuities), and thus equation (6) can be utilized to facilitate left-right matching.

In one embodiment, sparse stereo matching (or disparity estimation) is used to generate a sparse stereo representation of the object. Formally, the goal of this stereo matching step is to find, for each pixel with position (x,y) on the left normal map N _L , the best matching position (x-d,y) in the right normal map N _R , since this directly allows to recover the absolute surface depth using Equation 1.

Single pixel matching may be too ambiguous and therefore too sensitive to noise. In one embodiment, the goal is to match a patch at w pixels around (x,y), which may be expressed as NP _L =N _L [x-w:x+w,y-w:y+w], where NP _L is a patch in the left normal map N _L .

In one embodiment, for a given pixel patch on the left image NP _L centered on pixel location (x,y), a search in NP _R [x-d,y] is performed on the right normal map _NR starting with a first candidate pixel patch at (x,y) in the right normal map _NR . Further candidate matches are obtained, for example, by shifting the first candidate pixel patch left or right along the x dimension. The search is constrained on epipolar lines (or scan lines). The epipolar constraint states that given a pixel in one of the images, its potential conjugate image in the other image belongs to a straight line called an epipolar line. This constraint shows that stereo matching is essentially a one-dimensional problem.

A new pixel patch is then selected on the left image NP _L and the search for a match in the right normal map is performed again.

The candidate pixel patches may overlap. Each candidate pixel patch is associated with a temporal disparity _dt , which is the shift in the x-direction of the first candidate pixel patch.

For a given pixel patch on the left image NP _L , each candidate pixel patch on the right image NP _R is estimated to determine which candidate pixel patch is the best match. In one embodiment, each candidate pixel patch is estimated using angular distance (cosine similarity of normals). However, other metrics may be used for evaluation, such as least squares difference.

This results in a pairwise, shape-preserving correspondence between the left normal map N _L and the right normal map N _R .

In one embodiment, left-right consistency of disparity is enforced. This strengthens the disparity estimation by referring to information from the opposite view, i.e., from right to left. In one embodiment, this is achieved by determining a given pixel patch on the left image NP _L and identifying a matching pixel patch NP _R on the right normal map. Then, for NP _R , centered at pixel location (x, y), a search for NP _L [x-d, y] is performed on the left normal map _NL starting with a first candidate pixel patch at (x, y) in the left normal map _NL . Further candidate matches are obtained by shifting the first candidate pixel patch to the left or right. The candidate pixel patches may overlap. Each candidate pixel patch is associated with a temporal disparity d _t .

Next, for a given pixel patch on the right image NP _R , each candidate pixel patch on the left image NP _L is evaluated to determine which candidate pixel patch is the best match. In one embodiment, each candidate pixel patch is evaluated using angular distance (cosine similarity of normals), however, other metrics may be used for evaluation, such as least squares difference.

Left-right consistency of disparity is enforced by the condition that matches between patches on the left normal map N _L and the right normal map N _R and matches between right and left have a disparity difference below a given threshold. If the disparity difference exceeds the threshold, the match may be rejected. For example, in one embodiment, all matches with a disparity difference greater than 0.5 are rejected. However, other metrics based on disparity may be used.

In one embodiment, left-right consistency of single pixel normal matches is also enforced by keeping only points with normal differences less than 5°.

Although the above describes a window method for searching for candidate pixel patches, alternative methods of identifying candidate patches may be used.

In one embodiment, patch warping may be used to correct for the curvature associated with the normal map to give a more accurate stereo matching. Note that for any non-front facing plane, pixels with different depths have different disparities, so the corresponding patch in the right normal map NP _R is not a square of size 2w centered at (x-d,y). However, the approximate relative depth equation (6) may be used to compute a more accurate mapping (i.e., patch warping) from the left image to the right image.

More precisely, in one embodiment, for each temporal disparity _dt , the warping procedure is as follows:

1.

Calculate the average temporary depth as follows

2. Calculate the relative depth scale factor

3. For each patch pixel (x _p , y _p ), scale its relative depth to z _p = sZ _0L [x _p , y _p ]

4.

Calculate the patch pixel disparity as

5. Sample an interpolated value from the right normal map at position N _R [x _p - d _p , y _p ]

Thus, using the warping procedure described above, for each patch of pixels on the left normal map N _L , a temporal correspondence in the right normal map N _R is calculated. As described above, matching is estimated using angular distance (cosine similarity of normals). In one embodiment, left-right consistency of disparity is enforced as described above.

This results in a set of sparse-pair, shape-preserving correspondences between the left normal map N _L and the right normal map N _R. From this, two shapes are reconstructed, one from patch matching starting from the left normal map and one from patch matching starting from the right normal map.

The two constructed shapes (partial stereo estimates) are then combined into one. If corresponding points are estimated to be at different depths on the two partial stereo estimates, they are discarded together. The recombined stereo estimate is sparse and can be integrated or interpolated to increase its number of points. However, the sparse points from the stereo estimate can be used directly in the next step.

From the pairwise correspondences, a set of stereo points with absolute depth is computed using a triangulation method that exploits the principle of equation (5) and the camera calibration matrix. The set of sparse pairwise shape-preserving correspondences is then input to the 3D shape extraction steps S511 and S513.

In step S511, a first 3D shape (or reconstruction) is obtained using the normal map obtained from camera 1 in step S505, constrained by the pairwise correspondences obtained from stereo matching as detailed above.

A second 3D shape (or reconstruction) is obtained in step S513 using the normal map obtained from camera 2 in step S507, constrained by the pairwise correspondences obtained from stereo matching as detailed above.

The reconstruction is generated by numerically integrating the normal map to produce a depth map, where the integral is constrained by depth information determined from stereo matching of the two depth maps.

In one embodiment, this may be done by following the variational method of Queau and Durou [Y. Queau and J.-D. Durou. Edge-preserving integration of a normal field: Weighted least squares, TV and L1 approaches. In SSVM, 2015], where the surface derivatives are approximated using first-order finite differences.

The numerical integration of each normal map (N _L ,N _R ) to obtain the depth map (Z _0L ,Z _0R ) can be constrained to follow these stereo points. This essentially adds another term λ||z-z0|| to the integral equation, where z is the depth, z ₀ is the estimated depth, and λ is a constant weight that can also be scaled based on the confidence of the match. This removes scale ambiguity and reduces low frequency bending, thus greatly improving single-view photometric stereo depth accuracy.

The integral term with respect to dxdy is:

where D is the divergence operator, z is the known depth, _zd is the depth estimated from the normals, and where _λst ||z- _zst || is a constraint using the stereo points and _λprev ||z- _zprev || is a constraint using a quadratic prior determined from a previous reconstruction. If the sparse stereo reconstruction reconstructs a point that has no corresponding value from the stereo reconstruction, the term _λst ||z- _zst || is set to 0.

This sparse stereo reconstruction is less affected by global distortions than monocular photometric stereo reconstruction.

Note that at this stage (in the case of multi-view photometric stereo, which can accommodate multiple rectified stereo pairs) the output is still a set of 2.5D depth maps per view, and thus fusion is still required to obtain a complete surface. The single-view reconstructions (Z _OL , Z _OR ) subject to the stereo constraint are metrically consistent and can therefore be merged onto a unified surface in step S515. To minimize the impact of inaccuracies in the different view based fusion, each point is weighted using its view term, i.e., n·v (as detailed in Fotios Logothetis, Ignas Budvytis, Roberto Mecca, and Roberto Cipolla. A differential volumetric approach to multi-view photometric stereo. In ICCV, 2019). This allows points in the center of the image to be given a higher weight since they are more likely to be correct.

In addition, points outside the object's visual hull are completely rejected.

In one embodiment, a Poisson reconstruction is performed. A stereo point cloud of sparse stereo points can be made denser using Poisson reconstruction. The output of the Poisson reconstruction is a dense surface that can be projected onto all views and used as a better initialization for the next stage of the iterative procedure.

In one embodiment, the above process is iterated multiple times to improve the reconstruction. In subsequent iterations, the merged stereo constrained configuration from the previous iteration is used as a rough depth estimate or initial 3D plane. Using this estimate, normals are calculated, stereo matching is performed based on the normals, and an image of the object is reconstructed constrained by the stereo points obtained from the stereo matching.

In addition, having a dense surface estimate also makes it possible to estimate discontinuous boundaries that can be taken into account in the single-view reconstruction stage (essentially removing the usual integrability constraints on pixels across boundaries).

The stereo-constrained single-view reconstruction merged in step S515 is then used as the regenerated shape in step S517 to recalculate the light distribution that takes into account near-field effects, as described above in connection with equations (1)-(4), which describe how the object's shape affects near-field effects.

In step S519, new observation maps are generated for cameras 1 and 2. These observation maps are generated in the same manner as described above, however, now the shape regenerated in S517 is used as z _est . Normal maps are generated in step S521 from the observation maps generated in S519.

The method then loops back to S509 if the new normal map generated in step S521 is stereo matched. The process then continues as above until the reconstructed shape converges in S515.

The above method describes the formation of observation maps and the generation of normal maps from these observation maps.

More specifically, if the photometric stereo images are RGB images, a pre-processing may be performed in which the RGB channels are averaged, thus converting the images to grayscale images. In the pre-processing stage, the photometric stereo images are also compensated with the intrinsic light source luminance, where Φ _m is a constant property of each LED. The values of the resulting images are called RAW gray image values and are included in the observation map as further described herein below.

The general image irradiance equation can therefore be rearranged into a BRDF inverse problem as follows:

where j _m denotes a BRDF sample.

Note that while , is known, the lighting direction _L and the near-field optical attenuation _α are unknown due to their nonlinear dependence on the distance between the object represented by the depth z and the photometric stereo capture device. Thus, the goal of the convolutional neural network is to solve the inverse BRDF problem of a general line of sight direction, input to the network via the third and fourth channels of the map, as further explained below, to recover the surface normal N and subsequently the depth z.

Given an initial estimate of the depth z of each pixel (local surface depth), the near-field attenuation α _m (X) can be calculated according to equation (1), and therefore the equivalent far-field reflectance samples j _m representing the observations in the photometric stereo image after compensation for the near-field attenuation can be obtained using the first part of equation (8). The second part of equation (8) is modeled by a convolutional neural network (CNN) that is used to calculate the surface normal N and is used to approximate the normal N.

In step S407, the set of equivalent far-field reflectance samples _jm is used to generate an observation map, which is then fed to the input of the CNN. Each equivalent far-field reflectance sample _jm comprises a set of pixels x={ _x1 ,..., _xp }, where for each sample _jm , m denotes the number of light sources and thus the number of samples in the set of equivalent far-field reflectance samples, and the light direction _Lm and brightness _Φm are known and are used to estimate the surface normal N(x) for each pixel x.

The observation map is computed by combining the information from all light sources into a single map: in particular, for each pixel x, all observations in the set of equivalent far-field reflectance samples j _m are merged into a single d × d observation map.

First normalized observation for each pixel x

Normalization of the observations is performed to compensate for variations in the light source luminance Φ _m by dividing by the maximum luminance over all light sources m.

The compensation for illumination variations is aimed at compensating for albedo variations at different pixels. As a result, this also reduces the range of data associated with the observation of each pixel.

Then, for each pixel x, the normalized observations

is placed on the normalized observation map O _n . The normalized observation map is a square observation map with dimensions d×d. In some embodiments, d is 32. The size of the observation map is independent of the number or size of the photometric stereo images used.

Normalized observations are based on the light source direction

By projecting onto the dxd map, it is mapped onto the normalized observation map according to the following formula:

In some instances, normalized observation data can be corrupted by a division operation. For example, corruption of normalized observation data can occur when the maximum value of the observation is saturated. Dividing by the saturation value leads to an overestimation of the normalized observation. In other instances, segmentation of very faint points in the observation can become numerically unstable, amplifying the amount of noise or imprecision of any distinction.

Thus, the d×d observation map for each pixel x is expanded to a three-dimensional observation map with dimensions d×d×2 by the addition of a raw channel map O _r . The raw channel map may be a raw grayscale channel map. The raw channel map is defined as follows:

A dxdx2 observation map, hereafter denoted by O, is generated by a concatenation operation on the third axis of the normalized observation map O _n and the raw channel map O _r .

In some embodiments, an observation map O having dimensions d x d x 2 is used to obtain the first two components of the line of sight vector by augmenting the observation map to include two additional channels.

and

can be expanded to a dxdx4 observation map, where each component is constant.

and

is a scalar component, itself the line of sight vector used in the BRDF inverse problem formula

Completely determine.

The observation map records the relative pixel intensities from the BRDF samples on a 2D grid of discretized light directions. The observation map representation is a very convenient representation to use with classical CNN architectures because it gives a 2D input in terms of pixel length, despite the potential variation in the light used, and therefore the number of photometric stereo images.

In steps S505, S507, and S521, the observation map for each pixel x is fed to the input of a CNN, which is used to solve the BRDF inverse problem and calculate the surface normal for each point based on the relative pixel intensities in the observation map. Because the CNN is designed to be robust to real-world effects, the modeled BRDF inverse problem equation is an inexact representation of the BRDF inverse problem equation.

A high-level flow diagram of a convolutional neural network that can be used to generate surface normals from a pixel-wise observation map is presented in Figure 6.

The network comprises seven convolutional layers that are used to learn robust features to deal with real-world data. This is done by using an augmentation strategy during the training of the network, as described in GB2598711. The network further comprises two fully connected layers and a fully connected layer that is used to solve the inverse BRDF problem and is therefore combined with a logarithmic layer at the end that computes the surface normal for each pixel (steps S505 and S507).

The network has a total of about 4.5 million parameters. An overall view of the network's architecture is shown in FIG. 6. Although FIG. 6 depicts a particular network architecture, it is understood that different neural network architectures may be used to estimate surface normals from the observed map.

The proposed network includes a single-branch network. Seven convolutional layers 603, 605, 609, 613, 619, 623, and 627, as well as the first two fully connected layers 631 and 635, each followed by a RELU activation function. The size of the convolutional filters of each convolutional layer is shown in FIG. 6, and thus the size of the output volume for each layer can be inferred. In particular, the convolutional layer 603 comprises 32 convolutional filters with dimensions (3×3) and outputs an output volume with dimensions (32, 32, 32), and the convolutional layer 605 comprises 32 convolutional filters with dimensions (3×3) and outputs an output volume with dimensions (32, 32, 32). The output volume of the first concatenated layer 608 has dimensions of (32, 32, 64). The convolution layer 609 includes 32 convolution filters with dimensions (3x3) and outputs an output volume with dimensions (32, 32, 32). The output volume of the second concatenation layer 612 has dimensions (32, 32, 96). The convolution layer 613 includes 64 convolution filters with dimensions (1x1) and outputs an output volume with dimensions (32, 32, 64). The output volume of the average pooling layer 617 has dimensions (16, 16, 64). The convolution layer 619 includes 64 convolution filters with dimensions (3x3) and outputs an output volume with dimensions (16, 16, 64).

The output volume of the third concatenation layer 622 has dimensions (16, 16, 128). The convolution layer 623 has 64 convolution filters with dimensions (3x3) and outputs an output volume with dimensions (16, 16, 64). The convolution layer 627 has 128 filters with dimensions (3x3) and outputs an output volume with dimensions (16, 16, 128).

In addition, dropout layers 607, 611, 615, 621, and 625 are used after convolutional layers 605, 609, 613, 619, and 623, respectively. Dropout is a training technique that reduces independent learning between neurons in the network. During training, a random set of nodes is dropped from the network such that a reduced version of the network is created. The reduced version of the network learns independently of other sections of the neural network, thus preventing neurons from developing codependency among each other.

Each dropout layer is associated with a dropout parameter that specifies the probability that the output of the layer is dropped out. In layers 607, 611, 615, 621, and 625, the dropout parameter is 0.2, thus 20% of the output is dropped out.

Skip connections are also used in the network architecture to accelerate convergence. Skip connections are used to allow the output of a convolutional layer to skip a proceeding layer, allowing the outputs of subsequent layers to be concatenated together before being presented as input to the next layer of the network. An average pooling layer 617 is also used.

A first part of the convolutional neural network, which includes seven convolutional layers 603, 605, 609, 613, 619, 623, and 627, is effectively separated by a flattening layer 629 from a second part of the network, which includes fully connected layers 631, 635, 637 and a logarithmic layer 633. Fully connected layers are used because they provide good approximations of nonlinear mathematical functions.

The flattening layer 629 rearranges the feature maps output by the convolutional layer 627 into a vector of input data that is fed to the first fully connected layer 631. The output volume of the convolutional layer 627 is flattened into a 32,768 element vector, which is fed to the first fully connected layer 631.

The BRDF model described above in this specification follows the Blinn-Phong reflectance model, which is believed to be a good approximation of many real BRDFs, where the BRDF is modeled as a sum of diffuse and exponential components. Therefore, to solve the inverse BRDF problem, the inverse of these operations is approximated as a combination of linear and logarithmic sums. The combination of linear and logarithmic sums is implemented in the CNN using a combination of a densely coupled layer 635 and a logarithmic layer 633. The densely coupled layer 635 represents the linear component of the sum, and the logarithmic layer represents the logarithmic component of the sum.

Finally, a normalization layer 639 is used to convert what has been extracted by the network features into unit vectors (pixel surface normals) so that the network outputs surface normals from the pixel-wise observation map input.

Figure 7 illustrates the steps of a method according to one embodiment. Figure 7 illustrates a set of images arranged in rows and columns. In the following description, each image is referred to as image (row, col). If an image spans two rows, the image is referred to as (row1:row2, col). Images are acquired of an object for which the ground truth is known for each stage of the method.

Image (1,1) is an image from a first set of photometric stereo images, and image (2,1) is an image from a second set of photometric stereo images. Image (1,1) can be the right image and image (2,1) can be the left image, or vice versa.

Image (1,2) shows the error of the estimated normal map calculated for image (1,1) relative to the ground truth. Image (2,2) shows the error of the estimated normal map calculated for image (2,1) relative to the ground truth.

Image (2,3) shows the structure from normals (SfN) obtained by integrating the normal map estimated for image (2,1). Image (2,4) shows the error of the SfN map against the ground truth.

Image (1,3) shows Structure from Normals (SfN) obtained by integrating the normal map estimated for image (1,1). Image (1,4) shows the error of the SfN map against the ground truth.

Image (1,6) shows a disparity estimate map determined from matching the normal map for image (1,1) with the normal map for image (2,1) as described with reference to S509. Image (2,6) shows a disparity estimate map determined from matching the normal map for image (2,1) with the normal map for image (1,1). Black is low disparity while white is high disparity. Image (1,5) shows the error in the disparity map for image (1,6). Image (2,5) shows the error in the disparity map for image (2,6).

Image (1:2,7) shows the stereo points obtained from stereo matching. Image (1:2,8) shows the error of the stereo points against the ground truth.

Image (1:2,9) shows the reconstructed object resulting from fusing the depth maps obtained for images (1,1) and (2,1), constrained by the stereo points of image (1:2,7). Image (1:2,10) shows the error of the fused shape against the ground truth.

Image (1:2,11) shows the reconstructed object using Poisson reconstruction. Image (1:2,12) shows the error of the fused shape against the ground truth.

Lines 3-4 show the process steps for a second iteration of the method. Lines 5-6 show the process steps for a third iteration of the method.

Figure 8 shows the working principle of the reconstruction method. S901 shows a sample of input images originating from two cameras where the object is illuminated by one LED light source (out of 15). Following the arrow path to S902 is the pipeline where a normal map is computed from monocular photometric stereo. Such a normal map is then used to geometrically match the two views and give a sparse stereo representation of the object. Patch warping can be used in S903. S904 shows a sparse stereo point reconstruction. S905 shows the fusion of two stereo constrained single view reconstructions. In S906, a Poisson reconstruction is performed to further refine the reconstruction.

In S907, the reconstruction from S906 is used as an initial estimate of the shape geometry for subsequent iterations of the method. This is used together with the input image to compensate the input image for near-field optical attenuation.

Figure 8 shows, from left to right, the steps of the fusion process: point cloud from stereo; normal integration using a point cloud prior; fusion from all views and backprojection to initialize the next round; integration using a surface prior and considering depth discontinuities.

Figure 9 shows the results of the proposed system compared to monocular (single view) photometric stereo. In 1001, binocular photometric stereo image pairs of three objects (rabbit, queen, and squirrel) made of different materials are shown. After the ground truth (1002), there is a monocular photometric stereo reconstruction (1003), followed by the reconstruction of the proposed method (1004).

The above method and system are described with respect to binocular photometric stereo, i.e., two views or two sets of images taken from cameras in two different positions. However, the described method may also be applied to more views. For example, in one embodiment, there are three views or three sets of images taken from cameras in three different positions. The described stereo matching may then be performed between the first and second views, between the first and third views, and between the second and third views. The resulting stereo constrained single-view reconstructions (first, second, and third reconstructions) may be merged together.

The above method can be applied to any number of views, or set of photometric images.

Figure 10 is a schematic diagram of hardware that can be used to implement the method according to an embodiment. Note that this is only an example and other configurations may be used.

The hardware comprises a computing section 1100. In this particular example, the components of this section are described together; however, it will be understood that they are not necessarily in the same location.

The components of the computing system 1100 may include, but are not limited to, a processing unit 1113 (such as a central processing unit, CPU), a system memory 1101, and a system bus 1111 that couples various system components including the system memory 1101 to the processing unit 1113. The system bus 1111 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures, etc. The computing system 1100 also includes an external memory 1115 connected to the bus 1111.

The system memory 1101 includes computer storage media in the form of volatile and/or nonvolatile memory, such as read-only memory. A basic input/output system (BIOS) 1103, containing routines that help transfer information between elements within the computer, such as during start-up, is typically stored in the system memory 1101. Additionally, the system memory contains an operating system 1105, application programs 1107, and program data 1109 that are used by the CPU 1113.

Also connected to bus 1111 is interface 1125. The interface may be a network interface through which the computer system receives information from additional devices. The interface may also be a user interface that allows a user to respond to certain commands, etc.

The graphics processing unit (GPU) 1119 is particularly well suited for the above-described method due to its operation of multiple parallel invocations. Thus, in one embodiment, processing may be split between the CPU 1113 and the GPU 1119.

The embodiment described above combines two steps: (i) estimating shape-preserving correspondences for sparse pairs, and (ii) using them to initialize a reconstruction guided by the estimated pixel-wise normals, e.g., a Poisson reconstruction, although other reconstruction methods may be used.

Stereo matching requires matching view-invariant features, which is very challenging for textureless objects, especially for objects that undergo specular reflections that change appearance and pixel intensity in different views. In addition, local surface curvature distorts the local appearance between the left and right (or first and second) views, so patch-matching based methods try to match rectangular patches of pixels in the two views.

Single-view near-field photometric stereo can compute dense surface normals (for every foreground pixel) that can be very accurate even with approximate depth initialization. Surface normals are view-invariant features and also exhibit variation for any non-planar surface, which inherently enables patch matching.

In addition, normal integration gives a shape estimate that is locally accurate (and only suffers from low-frequency deformations or bending), and this local shape can be used to perform patch warping, and thus maximize stereo matching, as described below.

Therefore, when performing matching on normals instead of the initial two sets of images, more reliable stereo matching can be performed that is robust even when dealing with objects that have no texture or gloss. As a result, constraining the object reconstruction to pairwise correspondences in the stereo matching step can give a more accurate representation of the object geometry that gives a better initial guess for the iterative procedure of updating the object geometry to converge to a single object geometry.

The above architecture is also suitable for mobile phones that use GPUs. Although several embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the present invention. Indeed, the novel devices and methods described herein may be embodied in a variety of other forms, and further, various omissions, substitutions, and changes in the form of the devices, methods, and products described herein may be made without departing from the spirit of the present invention. The appended claims and their equivalents are intended to cover such forms or modifications as fall within the scope and spirit of the present invention.

Claims (17)

1. A computer vision method for generating a three-dimensional reconstruction of an object, comprising:
receiving a first set of a plurality of photometric stereo images of the object and a second set of a plurality of photometric stereo images of the object, the first set including at least one image taken from a first camera at a first location using illumination from a plurality of different directions and the second set including at least one image taken from a second camera at a second location using illumination from a plurality of different directions;
generating a first normal map of the object using the first set of photometric stereo images;
generating a second normal map of the object using the second set of photometric stereo images;
determining a stereo estimate of the shape of the object by performing stereo matching between a plurality of patches of normals in the first normal map and a plurality of patches of normals in the second normal map;
using the first normal map and the second normal map together with the stereo estimate of the shape of the object to generate a reconstruction of the object. The method of claim 1, wherein the first normal map and the second normal map are generated using an estimate of the shape of the object to recalculate a light distribution for multiple near-field effects of the illumination on the object. The method of claim 2, further comprising: recalculating the light distribution due to multiple near-field effects using the reconstruction of the object; recalculating the first normal map and the second normal map from the recalculated light distribution; and generating a further reconstruction of the object. Recalculating the light distribution due to a plurality of near field effects includes:
(a) using the reconstruction of the object to recalculate the first normal map and the second normal map from the recalculated light distribution;
(b) determining a further estimate of the shape of the object by performing stereo matching between a plurality of patches of normals in the recomputed first normal map and a plurality of patches of normals in the recomputed second normal map;
(c) generating a further reconstruction of the shape from at least one of the recalculated first normal map and the recalculated second normal map; and
and repeating (a) through (c) until a plurality of further reconstructions of the object converge. Using the first normal map and the second normal map together with the stereo estimate of the shape of the object to generate a reconstruction of the object, comprising:
Integrating the first normal map with constraints of the stereo estimation of the shape to generate a first reconstruction;
Integrating the second normal map with constraints of the stereo estimation of the shape to generate a second reconstruction;
and combining the first reconstruction and the second reconstruction to generate a fused reconstruction, which is the reconstruction of the object. The method of claim 5, wherein the fused reconstruction is generated using a Poisson solver. Performing stereo matching on the first normal map and the second normal map includes:
Selecting at least one group of pixels on the first normal map;
and searching for a matched group of pixels in the second normal map by scanning across the second normal map for multiple matches. 8. The method of claim 7, wherein scanning across the second normal map for matches is performed across epipolar lines. The method of claim 7, wherein the search for a matching pixel group is constrained by a current estimated reconstruction of the object. Performing stereo matching on the first normal map and the second normal map includes:
performing patch warping on the at least one group of pixels of the first normal map;
and determining a corresponding group of pixels of the second normal map using the patch-warped at least one group of pixels of the first normal map. Selecting at least one group of pixels on the first normal map;
Searching for matched pixel groups in the second normal map by scanning across the second normal map for a number of matches is used to generate a first partial stereo estimate;
The method comprises:
Selecting at least one group of pixels on the second normal map;
searching for matched pixel groups in the first normal map by scanning across the first normal map for a number of matches to generate a second partial stereo estimate;
8. The method of claim 7, further comprising: combining the first partial stereo estimate and the second partial stereo estimate to form a stereo estimate of the shape, wherein points from the first partial stereo estimate and the second partial stereo estimate that do not match are discarded. The method of claim 1, wherein the first set includes a first plurality of images taken from the first camera at the first location using illumination from a plurality of different directions, and the second set includes a second plurality of images taken from the second camera at the second location using illumination from a plurality of different directions. The method of claim 1, wherein generating the first normal map includes inputting information representing the first set of photometric stereo images, an estimate of the shape of the object, and position information for a number of light sources into a neural network trained to output the first normal map. The method of claim 13, wherein the information representing the first set of photometric stereo images, the estimate of the shape of the object, and the position information of the light sources is provided in the form of an observation map, where the observation map is generated for each pixel of the first camera, each observation map comprising a projection of multiple lighting directions onto a 2D plane, the multiple lighting directions for each pixel being obtained from each photometric stereo image. 1. A system for generating a three-dimensional (3D) reconstruction of an object, comprising: an interface and a processor;
the interface has an image input and is configured to receive a set of photometric stereo images of an object, the set of photometric stereo images including a plurality of images using illumination from a plurality of different directions with one or more light sources;
The processor,
receiving a first set of a plurality of photometric stereo images of the object and a second set of a plurality of photometric stereo images of the object, the first set including at least one image taken from a first camera at a first location using illumination from a plurality of different directions and the second set including at least one image taken from a second camera at a second location using illumination from a plurality of different directions;
generating a first normal map of the object using the first set of photometric stereo images;
generating a second normal map of the object using the second set of photometric stereo images;
determining a stereo estimate of the shape of the object by performing stereo matching between a plurality of patches of normals in the first normal map and a plurality of patches of normals in the second normal map;
a system configured to use the first normal map and the second normal map together with the stereo estimate of the shape of the object to generate a reconstruction of the object. The system of claim 15, wherein the first camera is configured differently from the second camera. A carrier medium carrying computer readable instructions adapted to cause a computer to perform the method of claim 1.