FR3099279A1 - Method for determining a normal map from images by uncalibrated stereo photometry - Google Patents

Abstract

A method of determining a normal map from a plurality of 2D images by uncalibrated stereo photometry, comprising the steps of: receiving a set of images with different uncalibrated lighting conditions; -extract an intermediate Normals field and an intermediate Enlightenment field from a singular value decomposition of Matrix I containing the image data; -determine six minor coefficients of the inverse of the linear transformation to be applied to the field of Normals in order to make it integrable; -modify the field of Normals in order to make its induced surface integrable; -extract the Lambertian Diffuse Reflectance Maxima (LDRM); -determine the ambiguity of generalized bas relief based on LDRMs using a statistical approach; -apply the GBR to the fields of Intermediate Normals and Intermediate Lights; -normalize the Normals of the intermediate Normals field. Figure 5

Description

Method for determining a map of normals from images by uncalibrated stereo photometry

The present invention relates to a method for determining a map of normals from a plurality of 2D images by uncalibrated stereo photometry.

The field of stereo photometry strives to extract geometric information from a set of images obtained from the same point of view where for each image, the static scene captured is observed under different lighting conditions. stereo photometry is of great interest in the field of artistic creation, because the resulting results allow an artist to focus on the modification of the material (and therefore its originality) rather than on the integral creation of the latter, avoiding thus a long process where tedious steps are often necessary.

There are two main classes of so-called stereo photometry methods: those of the calibrated type where the light information (i.e. direction and intensity) is known for each image, and those called uncalibrated where the light information is unknown and is therefore part of the variables to solve (the other variables being the field of normals).

Calibrated approaches, although simpler to solve from a mathematical point of view, have the strong constraint of the need to determine information about light or illumination data. This step is complex and generally requires specific equipment (for example a reflecting ball to be placed in the scene to be captured) combined with software dedicated to the extraction of this information. The results often lack precision, thus negatively impacting the results obtained.

The document WO2008/066368 relates to a method and an apparatus intended for three-dimensional modeling which uses the principle of stereo vision with an integrated lighting system in order to allow restoration in depth without prior knowledge of the reflection of the surface. The method includes various initial stereo and radiometric calibration steps and rectification methods to achieve accurate results. Furthermore, an active stereo technique is applied in order to restore dense depth maps just from a few stereo images. This technique is a combination of classical approximations and stereo photometry and is based on the property of reciprocity of reflection which considers an arbitrary surface distribution of reflection. This particular property is obtained if the object to be restored was acquired under specific lighting conditions previously planned. The active stereo technique is followed by the application of an in-depth map calibration process. The method also includes an image acquisition process for obtaining sets of images needed for calibration, rectification and three-dimensional restoration processes. The method and the device are relatively complex because of the calibration steps to be implemented.

There is therefore a need for a method for determining image characteristics that overcomes hardware constraints, while allowing reliable results to be obtained simply and quickly.

To overcome these various drawbacks, the invention provides various technical means.

First of all, a first objective of the invention consists in providing a method making it possible to improve the productivity of artists for the creation of objects or scenes in the field of artistic creation, for example for video games or movies.

Another object of the invention consists in providing a method making it possible to determine the characteristics of images without having to use specific equipment to take the shots of these images.

To do this, the invention provides a method for determining a map of normals from a plurality of 2D images by uncalibrated stereo photometry, comprising the steps of:
i) receive a set of at least three images with different uncalibrated (or unknown) lighting conditions;
ii) preparing the data of the received images in order to make them compatible for processing intended for Lambertian data devoid of shadows;
iii) extracting a field of intermediate Normals and a field of intermediate Lights resulting from a singular value decomposition of the Matrix I containing the image data;
iv) determine six minor coefficients of the inverse of the linear transformation to be applied to the field of Normals in order to make it integrable;
v) modify the field of Normals in order to make its induced surface integrable using the six minor coefficients and any generalized low relief (GBR) transformation, i.e. the three values determining the last column of the matrix describing the inverse of the linear transformation to be applied to the field of Normals; (for example, the three values are set equal to 1)
vi) extract Lambertian Diffuse Reflectance Maxima (LDRM);
vii) determine the generalized low relief ambiguity (GBR ambiguity) based on the LDRMs using a statistical approach;
viii) apply GBR to Intermediate Normals and Intermediate Lights fields;
ix) normalize the Normals of the intermediate Normals field.

Using an uncalibrated stereo photometry approach allows an artist to dispense with the light calibration steps and focus solely on capturing the scene with common hardware (e.g. a phone camera). To prevent the equational model from being too complex, this method describes an uncalibrated stereo photometry approach where the material is considered Lambertian. Also, to further simplify the model, we consider the lights to be directional and the scene to be without shadows. Although these prerequisites are rarely met in reality, the tests carried out have made it possible to obtain results of a high quality level. In addition, the low rank approximation applied to the input data makes it possible to satisfy these prerequisites as closely as possible and thus to obtain even more qualitative results.

In addition, being in a context of digital material creation tools (texture, image, material or other), one of the objectives is to minimize and optimize the use of memory and computing resources in order to make the method available and attractive. to the widest possible range of users under advantageous productivity conditions.

At step vii, the statistical approach is preferably of the "online" type in the absence of "offline", i.e. convergence can and does happen in practice much earlier due to data processing ( be the local GBRs resulting from the intersection of the 2D segments) progressively with a heuristic as to convergence (variance of the GBR below a certain threshold).

According to various advantageous embodiments, the step of preparing the data comprises one or more of the following phases: a phase of converting the image data into float, a phase of linearizing the data, a phase of converting the RGB values into values of luminance, a phase of vectorization of the images as columns of a matrix I, a phase of approximation of low rank, preferably close to three, in order to respect the Lambertian prerequisites and of absence of shadows of the images.

Advantageously, the statistical approach of the step of determining the generalized low relief ambiguity (GBR ambiguity) is based on a plurality of local analytical resolutions of the GBR with detection of convergence when the distribution of the set of local GBRs satisfies a normal distribution law.

This approach is relatively simple, particularly fast and provides reliable and accurate results. The use of histograms makes it possible to have a low memory complexity.

According to an advantageous embodiment, the method comprises a preliminary step of transforming the input image data into gray levels.

As the process is functional with images without coloring, this step makes it possible to broaden the field of application to the universe of color images, by providing a simple and quick step to make these images compatible.

Advantageously, the set of input images comprises at least three, and more preferably at least eight images.

This embodiment makes it possible, insofar as the sampling of the light is distributed in a uniform manner relative to the scene, to guarantee a sufficient level of quality for a large number of use cases, and for a large part of the usable images.

According to an exemplary embodiment, the resulting normals map data is used to generate a 3D rendering of the input image.

This embodiment is particularly useful for example in the field of artistic creation. For example, it allows you to draw inspiration from everyday objects to integrate them into a work in progress.

In another example, the resulting Normals map data is used to generate a Heights map of the input image.

This embodiment makes it possible to add micro-detail (high frequencies) to the material without disturbing the underlying geometry. (Normal mapping). The “displacement mapping” makes it possible to add macro-detail (low frequency) to the material by disturbing the underlying geometry.

According to yet another exemplary embodiment, the resulting normal map data is used to supplement the description data of a procedurally defined material for use on a texture or digital image.

All the implementation details are given in the following description, supplemented by figures 1 to 23, presented solely for the purpose of non-limiting examples, and in which:

Fig.1

Figure 1 illustrates an example of a stereo photometry data set (subset of 8 images instead of the original 40 images preprocessed by applying a rank minimization method).

Fig.2

FIG. 2 illustrates an example of a map of normals in vector view obtained by the method according to the invention from the data illustrated in the example of FIG. 1.

Fig.3

Figure 3 shows an example height map extracted from the normals map in Figure 2.

Fig.4

Figure 4 illustrates an example of different 3D renderings taking as input the height map of Figure 3.

Fig.5

FIG. 5 is a functional flowchart showing an example of the progress of the main steps of a method for determining a map of normals according to the invention.

Fig.6

Figure 6 is a functional flowchart showing an example data preparation step.

Fig.7

Figure 7 is a functional flowchart showing an example of a normal and lumen extraction step.

Fig.8

FIG. 8 is a functional flowchart showing an example of a calculation step for the linear transformation of a field of normals by imposing the surface integrability condition.

Fig.9

. FIG. 9 is a functional flowchart showing an example of a step for modifying intermediate fields of normals and lights by imposing the condition of completeness of surface.

Fig.10

FIG. 10 is a functional flowchart showing an example of the calculation step of Lambertian Diffuse Reflectance Maxima (LDRM) for a set of stereo photometry images.

Fig.11

Figure 11 is a functional flowchart showing an example Generalized Low Relief (GBR) ambiguity resolution step based on Lambertian Diffuse Reflectance Maxima (LDRM).

Fig.12

Figure 12 is the continuation of the functional flowchart of Figure 11.

Fig.13

Figure 13 is a functional flowchart showing an example application of a GBR to intermediate normal and light fields.

Fig.14

FIG. 14 is a functional flowchart showing an example of a calculation step for normalizing an intermediate field of normals.

Figures 15 to 23 illustrate examples of preparatory steps for implementing the method.

Fig.15

FIG. 15 is a functional flowchart showing an example of a calculation step for a Lambertian Diffuse Reflectance Maxima (LDRM) mask for an image.

Fig.16

Figure 16 is a functional flowchart showing an example step for calculating a 2D segment (u,v) for GBRs.

Fig.17

Figure 17 is a functional flowchart showing an example step for calculating the intersection of two 2D segments.

Fig.18

Fig. 18 is a functional flowchart showing an example lambda calculation step of a 2D segment point of an LDRM.

Fig.19

Fig. 19 is a functional flowchart showing an example of an initialization step of a histogram class.

Fig.20

Figure 20 is a functional flowchart showing an example step of adding a value to a histogram class.

Fig.21

Fig. 21 is a functional flowchart showing another example of a histogram initialization step.

Fig.22

Figure 22 is a functional flowchart showing another example step of adding a value to a histogram.

Fig.23

Fig. 23 is a functional flowchart showing an example step for calculating a median of a histogram.

This method describes an uncalibrated stereo photometry approach where the material is considered Lambertian, the lights are considered to be directional and the scene is devoid of shadows. Although these prerequisites are rarely met in reality, the method in practice behaves surprisingly satisfactorily from the point of view of the quality of the results.

Figure 1 illustrates an example of a set of images obtained from the same viewpoint (i.e. with a single camera position). For each of the images, the captured static scene is observed under different lighting conditions.

Figures 2, 3 are maps of normals in vector mode and of heights obtained from the images of Figure 1 by applying the method according to the invention.

Figure 4 illustrates an example of different 3D renderings based on the so-called "displacement mapping" technique taking as input the height map of figure 3, each image showing a variation of the point of view as well as the illumination of the scene .

Figure 5 schematically illustrates the main steps for implementing the method for determining a map of normals from a plurality of 2D images by uncalibrated stereo photometry. Stage 1 consists of receiving a set of at least three images with different uncalibrated lighting conditions. Step 2, sometimes optional, consists in preparing the data of the images received in order to make them compatible for processing intended for Lambertian data devoid of shadows. Step 3 consists in extracting a field of intermediate Normals and a field of intermediate Lights resulting from a singular value decomposition of the Matrix I containing the image data.

Then, in step 4, it is a question of determining six minor coefficients of the inverse of the linear transformation to be applied to the field of Normals in order to make it integrable. This allows, in step 5, to modify the field of Normals in order to make its induced surface integrable using the six minor coefficients and any GBR, i.e. the three values determining the last column of the matrix describing the inverse of the linear transformation to be applied to the field of Normals.

Step 6 then consists in extracting Lambertian Diffuse Reflectance Maxima (LDRM), in order to be able, in step 7, to determine the generalized low relief ambiguity (GBR ambiguity) based on the LDRM using a statistical approach. Finally, step 8 consists in applying the GBR to the fields of intermediate Normals and intermediate Lights, and step 9 to normalize the Normals of the intermediate Normals field.

The normal maps obtained make it possible, for example, to complete the description of a material, and/or to calculate a map of heights, and/or to produce an edition based on the image of the geometry, etc.

Examples of calculations to perform the operations required in each of steps 1 to 9 are shown in Figures 6 to 14.

Examples of calculations allowing to carry out the useful operations to prepare the data to be processed are illustrated in figures 15 to 24.

IMPLEMENTATION

The implementation of the various modules of the previously described stereo photometry system is advantageously carried out by means of processor instructions or commands, allowing the modules to perform the operation or operations specifically provided for the module concerned. Processor instructions may be in the form of one or more software or software modules implemented by one or more microprocessors. The module(s) and/or the software(s) are advantageously provided in a computer program product comprising a recording medium or recording medium usable by a computer and comprising a programmed code readable by a computer integrated into said medium or medium, allowing application software to run on a computer or other device comprising one or more microprocessors such as a tablet.

According to various variant embodiments, the microprocessor, just like the working memory with the instructions, can be centralized for all the modules or even be arranged externally, with connection to the various modules, or even be distributed locally so that one or more modules each have a microprocessor and/or a working memory.

Claims (8)

A method of determining a map of normals from a plurality of 2D images by uncalibrated stereo photometry, comprising the steps of:
i) receive a set of at least three images with different uncalibrated lighting conditions;
ii) preparing the data of the received images in order to make them compatible for processing intended for Lambertian data devoid of shadows;
iii) extracting a field of intermediate Normals and a field of intermediate Lights resulting from a singular value decomposition of the Matrix I containing the image data;
iv) determine six minor coefficients of the inverse of the linear transformation to be applied to the field of Normals in order to make it integrable;
v) modify the field of Normals in order to make its induced surface integrable using the six minor coefficients and any generalized low relief (GBR) transformation, i.e. the three values determining the last column of the matrix describing the inverse of the linear transformation to apply to the field of Normals;
vi) extract Lambertian Diffuse Reflectance Maxima (LDRM);
vii) determining the generalized low relief ambiguity (“GBR ambiguity”) based on the LDRMs using a statistical approach;
viii) apply GBR to Intermediate Normals and Intermediate Lights fields;
ix) normalize the Normals of the intermediate Normals field. Method for determining a map of normals according to claim 1, in which the step of preparing the data comprises a phase of converting the image data into floating point, a phase of linearizing the data, a phase of converting the RGB values in luminance values, a vectorization phase of the images as columns of a matrix I, and a low rank approximation phase, preferably close to three, in order to respect the Lambertian prerequisites and the absence of shadows images. Method for determining a map of normals according to any one of claims 1 or 2, in which the statistical approach of the step of determining the generalized low relief ambiguity (GBR ambiguity) is based on a plurality of local analytical resolutions of the GBR with detection of convergence when the distribution of the set of local GBRs satisfies a normal distribution law. Method for determining a map of normals according to any one of Claims 1 to 3, comprising a preliminary step of transforming the input image data into gray levels. A method of determining a normal map according to any of claims 1 to 4, wherein the set of input images comprises at least eight images. A method of determining a normals map according to any of claims 1 to 5, wherein the resulting normals map data is used to generate a 3D rendering of the input image. A method of determining a normals map according to any of claims 1 to 5, wherein the resulting normals map data is used to generate a heights map of the input image. A method of determining a normal map according to any one of claims 1 to 5, wherein the resulting normal map data is used to supplement description data of a procedurally defined material for use on a texture or a digital image.