TWI399524B - Method and apparatus for extracting scenery depth imformation - Google Patents

Description

Method and device for obtaining scene depth information

The invention relates to the acquisition of a scene depth information, in particular to a method and a device for acquiring scene depth information.

How to capture the distance or depth information of different objects in the actual scene has always been an important issue in the field of stereoscopic display, ranging system or depth profile display system. Traditional methods and devices can be classified into "active" and "passive". The concept of "parallax" is used in the "passive" system, which is mainly based on a multi-lens, multi-lens and similar concept apertured aperture imaging system. However, such systems require multiple imaging lenses and multiple image sensors to obtain stereoscopic image information. The "active" method uses a signal source (usually a light source) that is additionally actively transmitted to determine the distance of different objects based on the difference in time of flight after the source of the signal is illuminated. A specific form of distribution fringes is projected onto the object to be photographed, and the degree of deformation of the stripe on the surface of the object is utilized to evaluate the relative distance between the photographed objects.

In addition, U.S. Patent No. 5,521,695 discloses that a specially designed optical element causes an object to pass through the optical element to form four image points on the imaging surface thereof, and the relative positional changes of the four image points are obtained. The relative distance of the object being shot.

On the other hand, in the imaging system, the point spread function (PSF) of the lens changes with the object distance. Therefore, the object distance information can be obtained according to the characteristics of the diffusion function at this point. however, The point spread function of the lens changes in addition to the object distance. In the case of the same object distance but different fields of view, the point spread function of the lens will also change. Therefore, in order to obtain the difference in distance between the objects to be photographed and the depth information, the back-end image processing needs to consider the influence of the point spread function with the object distance and the visual field.

The invention provides a method for obtaining depth information and an optical capturing device thereof, which improve the similarity of a point spread function of different field of view conditions by means of a device having a coding element, along the optical axis of the optical capturing device The point spread function is still different, and then the depth information of the scene is obtained by the on-axis point spread function in different object distance changes.

The invention provides an optical information capture device for depth information, comprising an optical component and a coding component. Wherein the coding element is placed in a path through the optical element to be captured by the optical element for modulating a point spread function of the optical element.

The invention provides a method for obtaining depth information, comprising: acquiring an image by an optical capturing device; obtaining at least one diffusion function information of the optical capturing device; and scanning different regions of the image and according to the diffusion function information of the point Perform a restore comparison procedure.

1 is a block diagram of a depth information acquisition apparatus according to an embodiment of the present invention. After the incident light passes through the depth information acquisition device 100, it is received by the sensor. The depth information acquisition device 100 includes an optical component 101 and an encoding component 102. The optical element 101 for imaging may be a single lens, a lens group or a reflective imaging lens set or the like. The encoding component 102 can be a wavefront phase encoding component, a wavefront amplitude encoding component, or a wavefront phase and amplitude hybrid encoding component. Where the encoding component 102 is a wavefront phase encoding component, the encoding mode can be an axis. Symmetric coding. The wavefront coding of the coding element 102 can be represented by mutually orthogonal coordinate systems. In the present embodiment, the wavefront coding of the coding element 102 is expressed by the following equation: Where x and y are the coordinate positions of the coding element in the x-axis direction and the y-axis direction, n is a positive integer, l+m is an even number, and A _nx and A _ny are the nth term in the x-axis direction and the y-axis direction, respectively. The coefficient size, A _lmxy is the coefficient size of the xy coupling term.

The coding element 102 can be placed at the aperture of the depth information acquisition device 100, near the aperture, near the exit pupil plane, near the exit pupil plane, near the entrance pupil plane or near the entrance pupil plane. The coding element 102 can also be combined with the optical element 101, for example, the coding element 102 can be fabricated on the lens surface of the optical element 101. The combined wavefront coding can be expressed by the following equation: W '( x , y ) = W ( x , y ) + W ₀ ( x , y ), where W' ( x , y ) is the code added to the depth information acquisition device 100 The wavefront after the element 102, W ₀ ( x , y ), is the wavefront of the depth information acquisition device 100 that is not added to the coding element 102. Those skilled in the art familiar with the art know that W '( x , y ), W ( x , y ) and W ₀ ( x , y ) can also be represented by a zernike polynomial. The coding element 102 is placed in such a manner that the optical wavefront generated by the depth information acquisition device 100 after the wavefront of the captured object light is added to the coding element is mainly composed of W ( x , y ). Furthermore, the coding element 102 can be a refractive element or a diffractive element or an element having both optical properties described above.

According to one embodiment of the present invention, the optical element 101 has an effective focal length of 10.82 mm, a F# of 5, and a full viewing angle of 10.54 degrees. The sensor 103 has a diagonal length of 2 mm. The phase encoding component 102 encodes using the above-described wavefront program, where n = 2, A _2x = A _2y = 22.8 PI, and A _lmxy =0.

2A illustrates a point spread function of a scene light having a wavelength of 656.3 nm, a wavelength of 587.6 nm, and a wavelength of 486.1 nm in a red, green, and blue light band when the object is imaged at a distance of 1000 mm using the optical element 101. Spot diagram. FIG. 2B is a speckle diagram of a point spread function of different fields of view when the scene depth information acquisition device 100 of the phase encoding element 102 is used in combination with the optical element 101 to image a scene having an object distance of 1000 mm.

FIG. 3A illustrates a point spread function of a scene light having a wavelength of 656.3 nm, a wavelength of 587.6 nm, and a wavelength of 486.1 nm at different fields of view when the object is imaged at an object distance of 790 mm using the optical element 101. Spotted map. FIG. 3B is a speckle diagram showing the point spread function of different fields of view when the depth information acquisition device 100 of the phase encoding element 102 is used in combination with the optical element 101 to image a scene having an object distance of 790 mm.

4A illustrates a point spread function of a scene light having a wavelength of 656.3 nm, a wavelength of 587.6 nm, and a wavelength of 486.1 nm in a red, green, and blue light band when the object is imaged at a distance of 513 mm using the optical element 101. Spotted map. 4B illustrates the use of the optical element 101 in combination with the depth information acquisition device 100 of the phase encoding element 102 to image a scene having an object distance of 513 mm, which is represented by red, green, and blue wavelengths of 656.3 nm, a wavelength of 587.6 nm, and a wavelength of 486.1 nm. A speckle pattern of the spread function of the scene light at different points of view.

According to another embodiment of the present invention, the optical element 101 has an effective focal length of 10.82 mm, a F# of 5, and a full viewing angle of 10.54 degrees. The sensor 103 has a diagonal length of 2 mm. The phase encoding component 102 encodes using the above-described wavefront program, where n = 3, A _3x = A _3y = 12.7 PI, and A _lmxy =0. According to another embodiment of the present invention, FIG. 5 illustrates the use of the optical component 101 in conjunction with the depth information acquisition device 100 of the phase encoding component 102 to image scenes of different object distances (513 mm, 790 mm, and 1000 mm), which are represented as red, green, and A speckle pattern of a point spread signal of a wavelength of 656.3 nm, a wavelength of 587.6 nm, and a wavelength of 486.1 nm in a blue light band at different fields of view (ie, 0 mm, 0.7 mm, and 1 mm in the high position).

According to still another embodiment of the present invention, the optical element 101 has an effective focal length of 10.82 mm, a F# of 5, and a full viewing angle of 10.54 degrees. The sensor 103 has a diagonal length of 2 mm. The phase encoding component 102 encodes using the above-described wavefront program, where n = 2, A _2x = A _2y = 19.1 PI, and A _22xy = 9.55 PI. According to still another embodiment of the present invention, FIG. 6 illustrates the use of the optical component 101 in conjunction with the depth information acquisition device 100 of the phase encoding component 102 to image different objects (513 mm, 790 mm, and 1000 mm), which are represented as red, green, and A speckle pattern of a point spread signal of a wavelength of 656.3 nm, a wavelength of 587.6 nm, and a wavelength of 486.1 nm in a blue light band at different fields of view (ie, 0 mm, 0.7 mm, and 1 mm in the high position).

Compared with the point spread function using only the optical element 101, it can be seen from FIG. 2A to FIG. 6 that the depth information obtaining apparatus 100 of the embodiment of the present invention has the appearance of the point spread function in the case of the same object distance but different fields of view. The degree of variation in shape is quite small. In order to further confirm the improvement of the similarity of the point spread function Degree, the similarity calculation of the point spread function is performed using the Hilbert space angle. Figure 7 shows a comparison of similarity of diffusion function at the same object distance but at different points of view (comparative reference object distance is 790 mm). Figure 8 shows a comparison of the similarity of the spread function along the optical axis at different object distances. As can be seen from FIG. 7, the similarity of the point spread function of the depth information obtaining apparatus 100 according to the embodiment of the present invention in different fields of view is improved (in the Hilbert space angle, the smaller the calculated value represents the similarity. Higher). On the other hand, it can be seen from FIG. 8 that the point spread function on the optical axis of the depth information acquiring apparatus 100 still has a difference at different object distances.

In addition, the phase encoding component 102 combined with the optical component 101 by the aforementioned wavefront coding equation design may be limited to a wavelength of 656.3 nm and a wavelength of 587.6 depending on the wavelength band of the scene light when the optical element 101 or the depth information acquiring apparatus 100 actually operates. Nm and the wavelength of the light of 486.1 nm.

In order to enable the ordinary knowledge in the art to implement the present invention through the teachings of the present embodiment, a method implementation example is further provided below with the device for obtaining the scene depth information.

FIG. 9 is a flow chart showing the steps of the depth information obtaining method according to an embodiment of the present invention. In step S901, the acquisition device 100 is used to image the point spread function information when the object has an object distance of 1000 mm. This point spread function information can be obtained by measuring the device 100 or according to the design parameters of the device 100. This point spread function information is stored in step S902. On the other hand, in step S903, images of object distances of 1000 mm, 980 mm, 900 mm, 790 mm, and 513 mm are respectively obtained. Next, in step S904, the stored point spread function information is used to perform scanning and restoration comparison on images of different object distances, respectively. sequence. This reduction comparison program can restore the image using a Wiener filter or a direct inverse operation. After the image is restored, the mean square error (MSE) of the restored images is separately calculated and compared with the threshold value preset by the user. In step S905, the object distance of the image can be determined according to the comparison result, thereby obtaining the depth information. In addition, in step S904, the restored image quality may be evaluated by using a method of determining whether the boundary of the boundary is clear, and then the object distance of the image is determined and the depth information is obtained in step S905.

In accordance with an embodiment of the present invention, FIG. 10A shows a restored image with an object distance of 1000 mm. Fig. 10B shows a restored image with an object distance of 980 mm. Fig. 10C shows a restored image with an object distance of 900 mm. Figure 10D shows a reduced image with an object distance of 790 mm. Figure 10E shows a reduced image with an object distance of 513 mm. Because the scanning and reduction comparison program is performed using the point spread function information when the object distance is 1000 mm, the reduction effect of the object distance of 1000 mm is the best compared with the image of other object distances, and the MSE is 2.4×10 ^-7 . Therefore, in step S905, the information of the depth of the image is 1000 mm.

In the conventional technology system, the back-end image processing needs to consider both the point spread function and the influence of the object distance and the visual field. The embodiment of the present invention improves the similarity of the point spread function of different field of view conditions by the obtaining device 100 having the encoding element 102, and maintains the difference of the point spread function on the axis, and thus the difference function on the on-axis point is different. The change of the object distance gives the depth information of the scene. The technical and technical features of the present invention have been disclosed as above, and those skilled in the art can still make various substitutions and modifications without departing from the spirit and scope of the invention. Therefore, the present invention The scope of the protection should not be limited by the scope of the invention, but should be construed as being included in the following claims.

100‧‧‧Site device for obtaining depth information

101‧‧‧Optical components

102‧‧‧ coding components

S901-S905‧‧‧Steps

1 is a block diagram of a scene depth information obtaining apparatus according to an embodiment of the present invention; and FIG. 2A is a fragment diagram showing a point spread function of different fields of view when an optical element is used to image a scene having an object distance of 1000 mm; FIG. 2B When using the depth information acquisition device to image a scene with a object distance of 1000 mm, the speckle pattern of the point spread function of different fields of view; FIG. 3A shows the point spread function of different fields of view when using an optical element to image a scene with an object distance of 790 mm. FIG. 3B illustrates a speckle pattern of a point spread function of different fields of view when a depth information acquisition device is used to image a scene with an object distance of 790 mm; FIG. 4A illustrates the use of an optical element to image a scene with a object distance of 513 mm. , the speckle pattern of the point spread function of different fields of view; FIG. 4B shows the speckle pattern of the point spread function of different fields of view when the depth information acquiring device is used to image the object with the object distance of 513 mm; FIG. 5 shows the use of depth information. The device (without the xy coupling term) is used to image the speckle pattern of the point spread function at different image height positions when different objects are from the scene; Figure 6 shows the use of the depth information acquisition device (using the xy coupling term) When plotting different objects from the scene, the speckle pattern of the point spread function of different image height positions; Figure 7 shows the comparison of the similarity of the diffusion function at the same object distance but at different points of view; Figure 8 shows the depth information obtained at different object distances. Point spread on the optical axis of the device FIG. 9 is a flow chart showing the steps of the depth information obtaining method according to the embodiment of the present invention; FIG. 10A shows a restored image with an object distance of 1000 mm; FIG. 10B shows a restored image with an object distance of 980 mm; It is a restored image of 900 mm; FIG. 10D shows a restored image with an object distance of 790 mm; and FIG. 10E shows a restored image with an object distance of 513 mm.

100‧‧‧Site device for obtaining depth information

101‧‧‧Optical components

102‧‧‧ coding components

Claims (16)

A scene depth information obtaining device comprising: an optical component; and an encoding component, wherein the encoding component is placed on a path of the scene light passing through the optical component to modulate a point spread function of the optical component; The coding of the coding element can be expressed by the equation W(x, y) = ΣA _nx x ²ⁿ +A _ny y ²ⁿ +A _lmxy x ^l y ^m , where x and y are the coordinate positions of the coding element in the x-axis direction and the y-axis direction, respectively. n is a positive integer, l+m is an even number, A _nx and A _ny are the coefficient values of the nth term in the x-axis direction and the y-axis direction, respectively, and A _lmxy is the coefficient size of the xy coupling term. The device of claim 1, wherein the optical component is a single lens, a lens group, or a reflective imaging lens set. The apparatus of claim 1, wherein the coding element is a wavefront phase coding element. The apparatus of claim 1, wherein the coding element is a wavefront amplitude coding element. The apparatus of claim 1, wherein the coding element is a wavefront phase and amplitude hybrid coding element. The device of claim 1, wherein the coding element is an axisymmetric coding element. The apparatus of claim 1, wherein the coding of the coding element is represented by a coordinate system that is orthogonal to each other. The device of claim 1, wherein the coding element is positionable near the aperture of the device, near the exit pupil, near the exit pupil plane, into the pupil plane or near the pupil plane. The device of claim 1, wherein the coding element is a refractive element or A diffractive element or an element having both optical properties described above. The device of claim 1, wherein the coding element is integrated with the optical element. A method for obtaining scene depth information includes: acquiring an image by an optical device, wherein the optical device is provided with a wavefront phase encoding component, wherein the wavefront phase encoding is W(x, y)=ΣA _nx x ²ⁿ + ^{_{A ny y 2n + A lmxy x}} l y m , where x and y coordinate position of the coding element for each x-axis direction and the direction of the axis y, n is a positive integer, 1 + m is an even number, A _nx A _ny, respectively, and The coefficient size of the nth term in the x-axis direction and the y-axis direction, A _lmxy is the coefficient size of the xy coupling term; obtaining at least one diffusion function information of the optical device; and scanning the region of the acquired image, and according to the point The diffusion function information performs a restore comparison procedure. According to the method of claim 11, the method further comprises obtaining a depth information according to the result of the reduction ratio program. The method of claim 11, wherein the point spread function information is obtained by measuring the optical device or according to design parameters of the optical device. According to the method of claim 11, wherein the reduction comparison program can use a Wiener filter or a direct inverse operation. The method of claim 11, wherein the reduction comparison program can include a mean square error operation. According to the method of claim 15, wherein the reduction comparison program further compares the mean square error operation result with a user-set threshold value.