CN111340929A - Non-vision field imaging method based on ray tracing algorithm - Google Patents

Abstract

The invention discloses a non-visual field imaging method based on a ray tracing algorithm, which comprises the steps of utilizing a common camera to shoot diffuse reflection information generated on a diffuse reflection surface of a non-visual field scene, analyzing a path of rays corresponding to each pixel in a diffuse reflection image from an NLOS scene to the camera through the algorithm by means of the thought of the ray tracing algorithm, carrying out reverse calculation on the propagation path, and combining a Bidirectional Reflectance Distribution Function (BRDF) of the diffuse reflection surface so as to reversely deduce an image of an NLOS scene object. The method does not need other expensive equipment and instruments, and greatly reduces the equipment cost compared with a non-visual field imaging method which needs to emit laser and complex imaging equipment.

Description

A non-horizontal imaging method based on ray tracing algorithm

technical field

The invention discloses a non-view field imaging method based on a ray tracing algorithm, which belongs to the field of computational imaging.

Background technique

With the continuous increase of imaging devices, imaging methods are also constantly enriched. Since the emergence of non-horizontal imaging methods using lasers and gated-based ICCDs as imaging devices in 2009, non-horizontal imaging has begun to appear in military detection. , detection of dangerous scenes, and detection of blind spots in vehicles related to autonomous driving. With the continuous innovation of researchers, non-line-of-sight imaging methods continue to increase, but most of them cannot get rid of lasers as the active light source, which largely limits the application of non-line-of-sight imaging outside the laboratory. Until the passive non-horizontal imaging method began to appear in the past two years, non-horizontal imaging once again burst out the application potential in detection.

SUMMARY OF THE INVENTION

Purpose of the present invention: Aiming at the problems and deficiencies of the above-mentioned prior art, the present invention proposes a non-visual field imaging method based on a ray tracing algorithm, which does not use active light sources such as lasers, has low cost, and can be used outside the laboratory. The environment can still implement the application.

Technical solution: a non-visual field imaging method based on a ray tracing algorithm, comprising the following steps:

Step 1: Use the camera to obtain the diffuse reflection image generated by the NLOS scene on the diffuse reflection surface;

Step 2: using inverse perspective transformation and resampling technology to extract the image in the area corresponding to the rectangle in the actual space in the diffuse reflection image and transform this part of the image into a rectangular image with a specified resolution;

Step 3: Determine the size and spatial position of the rectangular image obtained in Step 2, and obtain the spatial coordinates of the center of each pixel according to the number of pixels existing in the rectangular image;

Step 4: Determine the size and spatial position of the area where the NLOS scene to be imaged is located;

Step 5: A light transmission matrix is established from the size and spatial position of the rectangular image obtained in step 3 and the size and spatial position of the area where the NLOS scene to be imaged obtained in step 4 is located; the dimension of the light transmission matrix depends on the total size of the rectangular image. The number of pixels and the total number of pixels of the NLOS scene to be imaged, the number of rows is the total number of pixels of the diffuse reflection image obtained in step 1, and the number of columns is the total number of pixels of the NLOS scene to be imaged; each element in the light transmission matrix Represents the conversion relationship of one pixel information in the NLOS scene to be imaged to one pixel information of the diffuse reflection image, and the conversion relationship is calculated by the light radiation propagation formula in the ray tracing algorithm;

Step 6: An optimization problem is established according to the rectangular image obtained in step 2 and the light transmission matrix obtained in step 5, and the NLOS scene image is obtained by solving the optimization problem.

Further, before step 3 is performed, the rectangular image may be down-sampled according to actual requirements.

Further, the size and spatial position of the area where the NLOS scene to be imaged is located and the size and spatial position of the rectangular image are determined in the same coordinate system.

Further, the conversion relationship is represented by a conversion matrix T:

In the formula, any pixel p _i on the diffuse reflection image, where i∈[1, m*n], comes from any pixel p′ _j on the NLOS scene image to be imaged, where j∈[1 , x*y], ρ represents the reflectivity of the diffuse surface, θ′ represents the angle between the incident light and the normal of the NLOS scene surface, _wi is the direction vector of the incident light, _wo is the direction vector of the reflected light, L _o (p, w _o ) represents the brightness of the reflected light along the direction w _o at the point p of the diffuse reflection surface, θ _i is the angle between the incident light direction vector and the normal vector of the diffuse reflection material at p, s is the NLOS to be solved the area of the scene surface;

Assuming that the diffuse reflection image is represented as an image matrix d, and the NLOS scene to be imaged is represented as an image matrix f, the relationship is expressed as follows:

d=Tf (9)

Further, the step 6 specifically includes the following sub-steps:

Determine whether the transformation matrix T is invertible, if it is invertible, then calculate the image matrix f corresponding to the NLOS scene to be imaged according to formula (9);

f = T ^-1 d (10)

If it is irreversible, perform the following steps:

为f的最小二乘逼近；get the pseudo-inverse of the transformation matrix

is the least squares approximation of f;

Establish the corresponding optimization problem according to the type of diffuse reflection image, and solve it to obtain the image matrix f corresponding to the NLOS scene to be imaged:

f=arg min(||Tf-d|| ² +λ ₁ ||f|| _TV +λ ₂ B) (12)

Among them, ||f|| _TV is the total variation of the image matrix f, which is calculated as:

B is a barrel function, which is calculated as:

Among them, λ ₁ , λ ₂ are the regularization coefficients, f _{i, j} are the pixel values of the i row and j column of the image matrix f, and x, y respectively represent the total number of rows and columns of the image matrix f.

Further, the diffuse reflection image includes any one of a single-channel grayscale image, an RGB three-channel color image, and a Bayer filter mode RGBG four-channel color image;

If the diffuse reflection image is a single-channel grayscale image, an optimization problem is established, and the matrix f corresponding to the NLOS scene image is solved; if the diffuse reflection image is an RGB three-channel color image, the data of the three channels are separated, and the three The optimization problem is established for each channel, and the three-channel data is obtained by solving it. The three-channel data is combined to obtain the matrix f corresponding to the NLOS scene image; if the diffuse reflection image is an RGBG four-channel color image in Bayer filter mode, the four-channel data is separated. , the data of the two G channels are averaged to obtain the data of the three channels, an optimization problem is established for the three channels, the three-channel data are obtained by solving, and the matrix f corresponding to the NLOS scene image is obtained by combining the three-channel data.

Beneficial effects: Compared with the traditional active non-horizontal imaging method based on the ray tracing algorithm of the present invention, it does not require an expensive active light source of laser and a specific image acquisition device. The passive imaging method of information requires the acquisition of polarization information acquisition devices and the passive imaging method based on coherence information requires an interferometer to acquire coherence information. The intensity information based on diffuse reflection in the present invention only requires an ordinary camera, which greatly improves the possibility of application .

Description of drawings

1 is a side view of a non-visual field scene according to an embodiment of the present invention;

FIG. 2 is a top view of a non-visual field scene according to an embodiment of the present invention;

Fig. 3 is a schematic diagram of luminance propagation calculation;

FIG. 4 is a schematic diagram of a diffuse reflection surface receiving incident light from a hemispherical space;

Figure 5 is a schematic diagram of the diffuse reflection surface receiving light from the NLOS scene;

6 is a schematic diagram of an inverse perspective transformation area in the camera FOV;

FIG. 7 is a schematic diagram of the diffuse reflection image matrix and the NLOS scene image matrix.

Detailed ways

The technical solutions of the present invention will now be further described with reference to the accompanying drawings and embodiments.

The basic idea of the present invention is as follows: the existing passive non-visual field imaging is mainly divided into three principles using light intensity, polarization information and coherence information, and the present invention is a non-visual field scene reconstruction using diffuse reflection intensity information. The method does not require expensive imaging equipment, only a common camera is used to record the intensity information (grayscale or RGB information) of the non-view field scene reflected on the diffuse surface, and then use the light propagation in the ray tracing algorithm. The principle is used to derive the back-propagation process of the light corresponding to each pixel in the diffuse reflection image, so as to reconstruct the image of the non-field of view scene.

Therefore, this embodiment utilizes the projection generated by the non-visual field scene on the diffuse reflection surface in front of it. The projection information has lost the specific information of the original scene due to the anisotropic scattering of the scene light source and the irregular reflection of the diffuse reflection surface. However, it still contains part of the information of the non-horizontal scene. Using this information combined with the path of simulated light propagation in the ray tracing algorithm and the bidirectional reflectance distribution function of the diffuse surface, it is still possible to restore the original image of the non-horizontal scene.

In the ray tracing algorithm, the ray entering the camera from the reflection of the diffuse surface can be described by the computer graphics formula (1):

In the formula, p is a point on the diffuse reflection material, p' is a point on other objects in space, _wi is the direction vector of the incident light, wo is the direction vector of the reflected light, and L _{o (} p, wo ₎ represents The brightness of the reflected light along the direction _wo at the point p of the diffuse reflection surface, _fr (p, _wi , _wo ) is the bidirectional direction of the diffuse reflection material when the direction of the incident light at point p is _wi and the direction of the reflected light is _wo The reflectivity distribution function (BRDF) value, Li (p', _{-wi ) is the brightness of the light emitted (or reflected) from point p' and propagated in the direction of wi, θ i is the} incident light direction vector and the diffuse The included angle of the normal vector of the reflective material at p, _dwi is integrated within the range of 2π solid angle, indicating that the incident light at point p comes from the brightness of all incident rays in the entire hemispherical space, as shown in Figure 4.

According to the definition of solid angle:

The integral of the above formula (1) over the solid angle can be converted to the area integral:

where θ′ represents the angle between the incident ray and the surface normal of the NLOS scene. For the scene shown in Figure 1, it can be assumed that the light incident on the diffuse surface in the environment is only the light from the NLOS scene, ignoring other reflected light. This is because if only the objects in the NLOS scene emit light, the other reflected light is much weaker than the light directly from the objects in the NLOS scene, so other reflected light can be ignored under the condition that the calculation result is guaranteed, which is actually With the help of the importance sampling theorem in the ray tracing algorithm. So if only the incoming rays coming directly from the NLOS scene are considered, as shown in Figure 5. Then the range of the area integral in the above formula (3) is the area of the NLOS scene. In fact, when the NLOS scene image is reconstructed, it is assumed that the NLOS scene is a two-dimensional rectangular image, so the integral area of formula (3) eventually becomes a rectangular area. For ease of calculation and understanding, it can be assumed that the NLOS scene is an image displayed on a display screen, and the task of non-horizontal imaging is to reconstruct this image using diffuse reflectance information.

Integration is inconvenient in actual programming calculations, in addition, because the ray tracing algorithm discretizes each ray. Therefore, with the help of the two-dimensional Monte Carlo integration theorem in calculus, the area integral shown in equation (3) is converted into a summation calculation:

In the ray tracing algorithm, the bidirectional reflectance distribution function of an ideal diffuse surface is a constant independent of incident and reflected light and surface points:

where ρ represents the reflectivity of the diffuse surface, from which Equation (4) can be converted into a more compact form:

For formula (6), point p is the point on the diffuse reflection surface, that is, the position in the actual space corresponding to each pixel in the diffuse reflection image captured by the camera, and point p' is the point on the NLOS scene. The above formula (6) establishes the conversion relationship between the brightness (or RGB information) of the point on the image on the NLOS scene to be restored to the brightness of the point on the diffuse reflection image captured by the camera.

As shown in Figure 1, in one instance, there is a partition between the camera and the NLOS scene, so that the camera cannot directly acquire the image of the NLOS scene, but the camera can capture the projected image generated by the NLOS scene on the diffuse surface. Since the projection information of the NLOS scene is mainly on the diffuse reflection surface close to the side of the NLOS scene, in order for the camera to capture more information from the NLOS scene, the camera should be shifted to the side of the NLOS scene by a certain angle, as shown in Figure 2. . Since the camera is deflected by a certain angle, the image captured by the camera has certain depth of field information, that is, the FOV of the camera is not a rectangular area. In order to facilitate the calculation, a rectangular area in the FOV can be extracted. This area needs to contain as much information from the NLOS scene as possible, so the area needs to be as close to the side of the NLOS scene as possible. Assuming that the area has been marked on the diffuse reflection surface (that is, it has been marked in the early stage), the information of the area can be extracted by inverse perspective transformation and transformed into a rectangular image, as shown in Figure 6.

Assuming that the extracted part is an image with a resolution of (m,n), in the actual calculation process, if the value of (m,n) is large, the downsampling technique can be used to reduce the resolution of the image to a certain extent . Assume that the resolution of the NLOS scene image to be restored is (x, y), as shown in Figure 6, this resolution can be set by itself in the calculation as needed, but if it is too large, it will affect the calculation speed, and in the actual calculation x and The magnitude of y is between 10 and 100, otherwise the calculation speed will be greatly affected.

For any pixel p _i (i∈[1, m*n]) on the diffuse reflection image, the luminance value (or RGB value) comes from all the pixels p′ _j (j∈[1 , x*y]), which can be expressed as:

The two images can be regarded as two matrices mathematically. If there is a transformation relationship between the two matrices, then a transformation matrix can be used to describe the transformation relationship. Therefore, a transformation matrix T is defined to describe the transformation relationship between the two images. It can be seen from the above formula that each element of the conversion matrix T should describe the luminance conversion relationship from p _i to p′ _j , where i∈[1, m*n], j∈[1, x*y], then the The dimension should be [m*n, x*y]. The dimension of this matrix is very large, so the diffuse reflection image needs to be downsampled, and the resolution of the NLOS scene graph to be restored cannot be set too large, otherwise the calculation time will be reduced. very long. In summary, the transformation matrix T should be as follows:

After the conversion matrix is established, the conversion relationship between the diffuse reflection image and the NLOS scene image can be represented by matrix operations, assuming that the diffuse reflection image is a matrix d (the dimension of d needs to be converted to [m*n, 1] during calculation), The NLOS scene image is a matrix f (the dimension of f needs to be converted to [x*y, 1] during calculation), then the relationship is shown in the following formula:

d=Tf (9)

However, in the actual operation process, since the information that can be obtained is the diffuse reflection image d, and the NLOS scene image f needs to be solved, the above formula must be reversed, but this needs to be discussed in two cases:

(1) When the transformation matrix is invertible, the matrix f corresponding to the NLOS scene image is obtained by solving according to Equation 10:

f = T ^-1 d (10)

则

为f的最小二乘逼近。另外还需建立优化问题进行求解f：(2) When the transformation matrix is irreversible, an optimization problem needs to be established to solve the matrix f corresponding to the NLOS scene image. At this point, you can first find the pseudo-inverse of the transformation matrix

but

is the least squares approximation of f. In addition, an optimization problem needs to be established to solve f:

f=argmin(||Tf-d|| ² ) (11)

However, the above optimization problem still has too few restrictions on the convergence conditions of f. Referring to the regularization chapter in the relevant "Convex Optimization", with the help of the minimum total variation ubiquitous in the image, the total variation regularization is introduced to further converge on the optimization problem. In addition, since f is an image, in order to further accelerate the convergence, it is necessary to limit f to a positive range. Here, a barrel function can be added to limit it between 0 and 10000:

f=argmin(||Tf-d|| ² +λ ₁ ||f|| _TV +λ ₂ B) (12)

where ||f|| _TV is the total variation of f, which is calculated as:

B is a barrel function, which is calculated as:

The specific operation steps of this embodiment include:

Step 1: The camera obtains the diffuse image generated by the light from the NLOS scene reflected on the diffuse reflection surface; specifically, when the camera captures the diffuse reflection image, in order to obtain more diffuse reflection information, the viewing angle will have a certain deflection to the NLOS side Angle, the captured image has a certain depth of field information, and the captured rectangular image corresponds to not a rectangular area in the actual space. Contains diffuse information. In practical applications, the diffuse reflection image captured by the camera may be one of single-channel grayscale image, RGB three-channel color image, and RGBG Bayer filter mode four-channel color image. There will be differences in the algorithm processing of images;

Step 2: Use inverse perspective transformation and resampling technology to extract the part of the image that corresponds to the rectangular area of the actual scene; the reference points used in the inverse perspective transformation can come from the four corners of the rectangle in the actual scene that already exists in the image, or can be obtained in advance. The inverse perspective transformation of the rectangular four-corner vertices calibrated on the diffuse surface can be transformed into a rectangular image by using the relevant API in OpenCV, or by extracting the ROI area in the image and then performing interpolation;

Step 3: Determine the size and spatial position of the resampled image corresponding to the rectangular image in the actual scene. After determining the spatial position and size of the rectangular image, it is necessary to calculate the number of pixels existing in the area to determine each pixel. The actual spatial coordinates of the corresponding position of the center. The coordinates here refer to relative coordinates. The specific spatial coordinate system can be established according to the calculation, but it is necessary to ensure that the same coordinate system is used to determine the position and size of the subsequent NLOS scene. In addition, if the pixels of the captured image are too high, in order to avoid taking too long to calculate, and to reduce the calculation amount of the algorithm without losing most of the accuracy, it is necessary to downsample the image first to reduce the resolution of the image, and then determine The actual spatial coordinates of the corresponding position of the center of each pixel;

Step 4: Determine the size and spatial position of the rectangular area where the NLOS scene to be imaged is located; the actual NLOS scene may be a scene composed of three-dimensional objects, but in the calculation process, it can be assumed that the NLOS scene is a flat rectangular image, which can simplify the calculation on the one hand The process, on the other hand, will not affect the actual imaging effect. For the size and spatial position of the rectangular area where the NLOS scene to be imaged is located, certain assumptions can be made in the calculation, or the calibration can be completed before the calculation. It is assumed that the size and spatial position of the rectangular area where the NLOS scene is located will only affect the range of the final imaged scene, but will not affect the content of the imaged scene. When the specific location of the scene is uncertain, the size of the rectangle can be assumed to be larger to contain the NLOS scene that needs to be reconstructed. As mentioned above, the size and spatial position of the rectangular area where the NLOS scene to be imaged is located and the size and spatial position of the rectangular area in the actual scene corresponding to the diffuse image are determined in the same coordinate system.

Step 5: Establish a light transmission matrix from the size and spatial position of the actual rectangular area corresponding to the diffuse image and the NLOS scene respectively; the dimension of the light transmission matrix depends on the total number of pixels of the rectangular image and the total number of pixels of the NLOS scene to be restored, The actual number of rows is the total number of pixels of the diffuse image, and the number of columns is the total number of pixels of the NLOS scene to be restored. Each element of the light transmission matrix represents the conversion relationship of one pixel information in the NLOS scene image to one pixel information in the diffuse reflection image, which is calculated by the light radiation propagation formula in the ray tracing algorithm.

Step 6: Establish and solve an optimization problem from the resampled diffuse image and the light transmission matrix. The establishment of the optimization problem is related to the type of diffuse reflection image captured. If the captured image is a single-channel grayscale image, only one optimization problem needs to be established; if the captured image is an RGB three-channel color image, the three-channel Separate the data, and establish an optimization problem for the three channels; if the captured image is an RGBG four-channel color image in Bayer filter mode, it is necessary to separate the four-channel data, and average the data of the two G channels to obtain three The data of the channel, and finally the optimization problem is established for the three channels. After the optimization problem is established for the color image, it is necessary to combine the solved three-channel data to obtain the solved color image.

Claims (7)

1. A non-visual field imaging method based on ray tracing algorithm is characterized in that: the method comprises the following steps:

step 1: acquiring a diffuse reflection image generated by an NLOS scene on a diffuse reflection surface by using a camera;

step 2: extracting an image in a rectangular area in a corresponding actual space in the diffuse reflection image by adopting an inverse perspective transformation and resampling technology and transforming the partial image into a rectangular image with specified resolution;

and step 3: determining the size and the spatial position of the rectangular image obtained in the step (2), and obtaining the spatial coordinate of the center of each pixel point according to the number of the pixel points in the rectangular image;

and 4, step 4: determining the size and the spatial position of an area where an NLOS scene to be imaged is located;

and 5: establishing a light transmission matrix according to the size and the spatial position of the rectangular image obtained in the step 3 and the size and the spatial position of the region where the NLOS scene to be imaged is obtained in the step 4; the dimensionality of the light transmission matrix depends on the total pixel number of the rectangular image and the total pixel number of the NLOS scene to be imaged, the number of lines is the total pixel number of the diffuse reflection image obtained in the step 1, and the number of columns is the total pixel number of the NLOS scene to be imaged; each element in the light transmission matrix represents a conversion relation from one pixel information in an NLOS scene to be imaged to one pixel information of a diffuse reflection image, and the conversion relation is calculated by a light radiation propagation formula in a ray tracing algorithm;

step 6: and (4) establishing an optimization problem according to the rectangular image obtained in the step (2) and the optical transmission matrix obtained in the step (5), and solving the optimization problem to obtain an NLOS scene image.

2. The ray tracing algorithm-based non-visual field imaging method according to claim 1, wherein: before step 3 is performed, the rectangular image may be down-sampled according to actual requirements.

3. The ray tracing algorithm-based non-visual field imaging method according to claim 1, wherein: and the size and the spatial position of the region where the NLOS scene to be imaged is located and the size and the spatial position of the rectangular image are determined under the same coordinate system.

4. The ray tracing algorithm-based non-visual field imaging method according to claim 1, wherein: the conversion relation is expressed by a conversion matrix T:

in the formula, any pixel point p on the diffuse reflection image_iWherein i ∈ [1, m n]From any pixel point p 'on the NLOS scene image to be imaged'_jWherein j ∈ [1, x y]ρ represents the reflectivity of the diffuse reflecting surface, θ' represents the angle between the incident ray and the normal of the NLOS scene surface, w_iIs the direction vector of the incident ray, w_oIs the direction vector of the reflected light, L_o(p，w_o) Representing the edge w at the point p of the diffuse reflection surface_oBrightness of directionally reflected light, theta_iThe included angle between the incident light direction vector and the normal vector of the diffuse reflection material at the position p is included, and s is the area of the surface of the NLOS scene to be solved;

assuming that the diffuse reflection image is represented as an image matrix d and the NLOS scene to be imaged is represented as an image matrix f, the relationship is expressed as follows:

d＝Tf (9)。

5. the ray tracing algorithm-based non-visual field imaging method according to claim 4, wherein: the step 6 specifically includes the following substeps:

judging whether the conversion matrix T is reversible, if so, calculating according to the formula (9) to obtain an image matrix f corresponding to the NLOS scene to be imaged;

f＝T^-1d (10)

if the current is not reversible, the following steps are carried out:

obtaining a pseudo-inverse of the transformation matrix

A least squares approximation of f;

establishing a corresponding optimization problem according to the type of the diffuse reflection image, and solving to obtain an image matrix f corresponding to the NLOS scene to be imaged:

f＝argmin(||Tf-d||²+λ₁||f||_TV+λ₂B) (12)

wherein | f | purple_TVThe total variation of the image matrix f is calculated as follows:

b is a barrel function, and the calculation method is as follows:

wherein λ is₁,λ₂For regularizing coefficients, f_i，jThe i rows and j columns of pixel values of the image matrix f, x and y respectively represent the total row number and the total column number of the image matrix f.

6. The ray tracing algorithm-based non-visual field imaging method according to claim 5, wherein: the diffuse reflection image comprises any one of a single-channel gray image, an RGB three-channel color image and an RGBG four-channel color image in a Bayer filtering mode.

7. The ray tracing algorithm-based non-visual field imaging method according to claim 6, wherein:

if the diffuse reflection image is a single-channel gray image, establishing an optimization problem, and solving to obtain a matrix f corresponding to the NLOS scene image; if the diffuse reflection image is an RGB three-channel color image, separating three-channel data, respectively establishing an optimization problem for the three channels, solving to obtain three-channel data, and combining the three-channel data to obtain a matrix f corresponding to the NLOS scene image; and if the diffuse reflection image is an RGBG four-channel color image in a Bayer filtering mode, separating data of four channels, averaging data of two G channels to obtain data of three channels, establishing an optimization problem aiming at the three channels, solving to obtain three-channel data, and combining the three-channel data to obtain a matrix f corresponding to the NLOS scene image.

Images