JP5174684B2 - 3D detection using speckle patterns - Google Patents

Description

( Cross-reference to related applications )
This application claims the benefit of US Provisional Patent Application 60 / 785,187, filed March 24, 2006. This application is a continuation-in-part of PCT patent application PCT / IL2006 / 000335 filed March 14, 2006, which is a US provisional patent application 60 / 724,903 filed October 11, 2005. I claim the benefit. All of these related applications are assigned to the assignee of the present patent application, the disclosures of which are incorporated herein by reference.

(Technical field)
The present invention relates generally to methods and systems for mapping three-dimensional (3D) objects, and more particularly to three-dimensional optical imaging using speckle patterns.

As the coherent light beam passes through the diffuser and is projected onto one surface, a primary speckle pattern can be observed on that surface. This primary speckle is caused by the interference of different components of the diffused beam. In this patent application and in the claims, “first order speckle” is used in this sense and is distinguished from second order speckle caused by diffuse reflection of coherent light from the rough surface of the subject.

Hart describes the use of speckle patterns in a high-speed three-dimensional imaging system in US Pat. The system includes a single lens camera subsystem with an active imaging device and a CCD device, and a correlation processing subsystem. The active imager can be a rotary stop, which allows for adjustable non-equal spacing between defocused images to increase depth of field and increase displacement accuracy below the pixel. The speckle pattern is projected onto the subject and the resulting pattern image is obtained from multiple angles. These images are locally cross-correlated using image correlation techniques, and the surface uses relative camera position information to calculate the three-dimensional coordinates of each locally correlated region. To be resolved.

Patent document 3 by Hunter et al. (The disclosure of which is incorporated herein by reference) describes another speckle based three-dimensional imaging technique. The random speckle pattern, is projected into one three-dimensional table surface is imaged by a plurality of cameras, a plurality of two-dimensional digital image is obtained. Two-dimensional image is processed, one three-dimensional characteristics of the front surface is obtained.
Taiwan patent TW527528 B US patent application 09 / 616,606 US Pat. No. 6,101,269

(Summary of the Invention)
Embodiments of the present invention perform accurate and real-time mapping of a three-dimensional subject using a primary speckle pattern. In the method and system described in the embodiments above PCT patent application and below, using a single coherent light source, stationary and a single image sensor which is held at a fixed angle relative to the light source Such a three-dimensional mapping can be performed.

One aspect of the present invention, one reference image of the speckle pattern is in that first are acquired on a single lookup table surface of known contour. Next, an image of a speckle pattern projected on the subject is acquired, and the three-dimensional contour of the subject is determined by comparing this image with a reference image.

Another aspect of the present invention is that a continuous image of a speckle pattern on the subject is acquired as the subject moves. Each image is compared to one or more of the preceding images in order to track the movement of the subject in three dimensions. In one embodiment described below, the light source and the image sensor are arranged in one straight line, by calculating a one-dimensional correlation coefficients between successive images, tracks the rapid and accurate motion be able to.

In some embodiments , novel illumination and image processing configurations are used to increase accuracy, depth of field , and computational speed of the 3D mapping system.

Therefore, according to one embodiment of the present invention, an illumination device having a coherent light source arranged a to project the primary speckle pattern onto the object and the diffusion plate, relative to the lighting device A single image acquisition device arranged to acquire a primary speckle pattern image on a subject from a single, fixed position and angle, and a single image to derive a three-dimensional map of the subject An object three-dimensional mapping apparatus is provided having a processor connected to process an image of a primary speckle pattern acquired at a fixed angle.

In some embodiments, the apparatus, an image capturing device in order to spatially fixed relative to the illumination device, having a base attached to the lighting device and an image acquisition apparatus. In one embodiment, the image acquisition device includes a detector array and an objective optical system arranged in a linear pattern defining first and second mutually orthogonal axes, the objective optical system comprising: A beam having an entrance pupil and arranged to focus the image on the array, where the beam produced by the light source coherent with the entrance pupil and parallel to the first axis is coherent The illuminating device and the image acquisition device are arranged by a table so as to define the axis of the device passing through the spot passing through the plate. Thus, on only the first axis, the deviation value between the primary speckle pattern acquired in one or more images and the reference image of the primary speckle pattern is By finding, a processor is arranged to derive a three-dimensional map.

Cubic by finding each deviation value between a primary speckle pattern of multiple regions on a subject acquired in one or more images and a reference image of the primary speckle pattern In some embodiments, a processor is arranged to derive the original map, with each shift indicating each distance between the region and the image acquisition device . Usually, the image acquisition device is positioned at a predetermined interval from the illumination device , and each offset is proportional to each distance at a rate determined by this interval. In the disclosed embodiment, the primary speckle pattern projected by the illuminating device has a speckle characteristic dimension, the dimension of speckle in an image, the image by the tolerance that depends on the spacing The distance varies across the entire area, and the interval is selected such that the tolerance is within a predetermined range.

Additionally or alternatively, a processor is arranged to associate each shift with each coordinate of the three-dimensional map using a distortion coefficient model in the image acquisition device . Additionally or alternatively, the first speckle pattern in the first area of the subject and the area of the reference image corresponding to the first shift value relative to the first area. A range expansion method is used based on the first shift to find each shift by finding an initial fit between and to each shift value of a pixel adjacent to the first region. As such, a processor is arranged.

In certain disclosed embodiments, a processor is arranged to process successive acquired images while the subject is moving in order to map the three-dimensional motion of the subject, A body part, a three-dimensional movement, is an action performed by a part of the human body, and a processor is connected to provide input to a computer application in response to the action .

In some embodiments , the illuminating device comprises a beam forming device, which is arranged to reduce the variation in the contrast of the speckle pattern created by the diffuser throughout the detection volume of the device. In one embodiment, the beam former has a diffractive optical element (DOE) and a lens arranged to define a Fourier plane of the diffuser, the DOE being positioned in the Fourier plane. Beam forming apparatus may be arranged to reduce the divergence of the light emitted from the diffusion plate, or the intensity of the light emitted from the diffusion plate, in order to equally whole surface transverse to the optical axis of the illumination device It may be arranged.

In some embodiments, the processor includes an optical correlator, the optical correlator includes a diffractive optical element (DOE) that includes a reference speckle pattern, and the image acquisition device includes a plurality of sub- subjects of the subject. A small lens array is arranged to project the image onto the DOE and generate each correlation peak indicative of the three-dimensional coordinates of the subject.

In some embodiments, the coherence length of the coherent light source is less than 1 cm. Additionally or alternatively, the primary speckle pattern has speckles with characteristic dimensions, and the speckle features can be changed by changing the distance between the coherent light source and the diffuser. The lighting device is configured so that the general dimensions can be adjusted.

According to one embodiment of the present invention, in order to project the primary speckle pattern onto the object, in a coherent light beam diffused from a light source, comprising the steps of illuminating the object with respect to the light source in comparison, the position and angle of the single, fixed, a step of acquiring the images of the primary speckle pattern on the object, to derive the three-dimensional map of the object is acquired at a single, fixed angle There is also provided a method for three-dimensional mapping of a subject comprising the step of processing an image of a primary speckle pattern.

In addition, according to one embodiment of the present invention, a coherent light source having a coherence length of less than 1 cm and a diffuser plate are arranged to project a primary speckle pattern on the subject. In addition, an illumination device , an image acquisition device arranged to acquire a primary speckle pattern image on the subject, and an image of the primary speckle pattern to derive a three-dimensional map of the subject An object three-dimensional mapping apparatus is provided having a processor connected for processing.

In certain embodiments, the coherence length of the coherent light source is less than 0.5 mm. Additionally or alternatively, greater than 5 ° diverging coherent light source.

  The present invention will be more fully understood by reference to the following detailed description of embodiments of the invention in conjunction with the drawings.

(Brief description of the drawings)
FIG. 1 is a schematic diagram illustrating a three-dimensional mapping system according to an embodiment of the present invention.
FIG. 2 is a schematic top view of a speckle imaging device according to an embodiment of the present invention.
FIG. 3 is a flowchart schematically illustrating one method of three-dimensional mapping according to an embodiment of the present invention.
FIG. 4 is a schematic side view of a lighting device used in a three-dimensional mapping system according to another embodiment of the present invention.
FIG. 5 is a schematic side view of a beam forming apparatus according to an embodiment of the present invention.
FIG. 6 is a schematic side view of a beam forming apparatus according to still another embodiment of the present invention.
FIG. 7 is a schematic side view of an optical correlator used in a three-dimensional mapping system, according to yet another embodiment of the present invention.

FIG. 1 is a schematic diagram illustrating a three-dimensional mapping system 20 according to an embodiment of the present invention. The system 20 includes a speckle imaging device 22 that generates a primary speckle pattern, projects it onto a subject 28, and displays an image of the primary speckle pattern that appears on the subject. To win. The detailed design and operation of the device 22 is shown in the following drawings and will be described in connection therewith.

The image processor 24 processes the image data generated by the device 22 to obtain a three-dimensional map of the subject 28. The term "three-dimensional map" as used in this patent application and in the claims, refers to a set of three-dimensional coordinate group representing the front surface of the object. Deriving such a map based on image data can also be referred to as “three-dimensional reconstruction”. Such an image processor 24 which reconstructs the performed may be made from the general purpose computer processor, which is programmed to software for implementing the functions described below. This software may be downloaded to processor 24 in electronic form, for example via a network, or may be provided on tangible media such as optical, magnetic or electronic memory media. Alternatively or in addition, some or all of the functionality of the image processor may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). . The processor 24 is shown in FIG. 1 as an example separate unit from the imaging device 22, but some or all of the processing functions of the processor 24 may be performed by appropriate dedicated circuitry within the imaging device housing, and it may be implemented included with the imaging device.

The three-dimensional map generated by the processor 24 can be used for a wide variety of purposes. For example, the map can be sent to an output device such as a display 26 showing a pseudo 3D image of the subject. In the example of FIG. 1, the object 28 is composed of all parts or a part of a subject body (e.g., a hand). In this case, the system 20 may be used to provide a user interface based on gestures, movement of the user detected by the device 22 is a mouse, tactile type interface member such as a joystick or other accessories on behalf of, to control the interactive computer applications such as games. Alternatively, the system 20 for virtually any application outline of the three-dimensional coordinates Ru is required, may be used to generate a three-dimensional map of the other type of subject.

FIG. 2 is a schematic top view of one device 22 according to one embodiment of the invention. One of the lighting device 30 has one coherent light source 32. This usually consists of one laser and one of the diffusion plate 33. (In the contexts of the present patent application, the term "light" means also the optical radiation of any kind, for example, infrared, ultraviolet and visible light.) The beam of light emitted from the light source 32 , it passes through the diffuser plate 33 in one spot 34 of radius _{w 0,} that generates a one diverging beam 36. As described in the above-mentioned PCT patent application PCT / IL2006 / 000335, the primary speckle pattern created at a distance of Z _obj1 and Z _obj2 by the diffusion plate 34 is
The on-axis dimension ΔZ of the speckle pattern at the subject distance Z _obj is
ΔZ = (Z _obj / w ₀ ) ² · λ
By the _{Z obj1} and _{Z obj2} the range of that given distance, if there is an image that has been scaled increases linearly to each other to the extent of good approximation.

One image acquisition apparatus 38 acquires one image of the speckle pattern projected onto the object 28. 38 consists of an objective optical system 39 which focuses images on one image sensor 40. Usually, the sensor 40, such as a CCD or CMOS based image sensor array, consisting of linear arrays of detector elements 41. The objective optical system 39 has one entrance pupil 42 that defines one field 44 of the image acquisition device along with the dimensions of the image sensor. Detection area of the device 22 is comprised of one overlapping region 46 between the beam 36 and the field 44.

The characteristic transverse speckle dimension projected by the lighting device 30 at a distance of Z _obj (which is defined by the secondary statistics of the speckle pattern) is
ΔX = (Z _obj / W ₀ ) · λ
It is. We, to optimize performance of the optical image processing, speckle size imaged on the sensor 40 on, depending on the range and resolution requirements, should be 1 to 10 pixels, i.e. It has been found that each speckle imaged on the sensor 40 by the optical system 39 should be expanded by the number of 1 to 10 detection elements 41 in the horizontal direction. In a typical application, it results in good speckle size of 2-3 pixels.

From the above equation for ΔX, it can be seen that the speckle dimension can be adjusted by changing the distance between the light source 32 and the diffuser plate 33, which means that the radius w _{0 of the} spot 34 is the distance from the light source. This is because as the length increases, the length increases. As described above, the speckle coefficient of the illumination device 30 can be controlled by simply moving the light source in the lateral direction without using a lens or other optical system. Illumination device 30 is thus being adjustable to be used together with different dimensions and resolution image sensor and different magnifications of the objective optical system. Given the small speckle size that is forced by a factor of above, have large divergence (5 ° or more) and short coherence length (less than 1 cm, and optionally, less than 0.5 mm), low cost, such as a laser diode A good light source can be used in the system 20.

The illumination device 30 and the image acquisition device 38 are held at positions fixed by the base 43. In the embodiment illustrated in FIG. 2, the platform has a housing that holds these devices . Alternatively, a desired positional relationship between the lighting device and the image acquisition device can be maintained by using another appropriate mechanical base . With the configuration of the apparatus 22 and the processing technique described below, three-dimensional mapping can be performed using a single image acquisition apparatus without relatively moving between the illumination apparatus and the image acquisition apparatus and without moving the member. It can be carried out. The image acquisition device 38 can thus acquire an image at a single and fixed angle relative to the illumination device 30.

In order to simplify the calculation of the map change due to the movement of the three-dimensional map and the subject 28, the axis passing through the center of the entrance pupil 42 and the spot 34 is one of the axes of the sensor 40 as shown below. The platform 43 preferably holds the two devices 30 , 38 so that they are parallel to each other. That is, (puts the origin to the optical axis of the objective optical system 3 9) to define the X and Y axes which are orthogonal to each other, through taking the columns and rows of the array of detector elements 41, the pupil 42 and spot 34 The axis should be parallel to one of the array axes (for convenience, the X axis). The advantages of this arrangement are further explained below.

で示される値をとるように生じる。 The optical axes of the devices 30 and 38 (which respectively pass through the center of the spot 34 and the pupil 42) are separated by a distance S. Therefore, when Z _obj changes, distortion of the speckle pattern occurs in the subject image acquired by the image acquisition device 38. In particular, the triangulation, as can be seen from FIG. 2, Z-axis direction shift δZ of one point on the subject, shift ancillary transverse speckle pattern in observed during image δX The

It takes to take the value indicated by.

Shift of the Z coordinate by Z-coordinate and time points on the object, the speckle in the images that are acquired by the image acquisition apparatus 38 of the X-coordinate, the one reference image obtained by the known distance Z It can be calculated by measuring the deviation . That is, to find the speckle group of best fit in the reference image, speckle group in each region of the acquired image is compared with the reference image. The relative shift between the groups to meet the speckle in the image indicates the Z direction of the shift of the region of the acquired image against the reference image. The speckle pattern shift can be measured using an image correlation method or other image fitting calculation methods known in the art. Some of these methods are described in the above-mentioned PCT patent application. Another particularly useful method applied with apparatus 22 is U.S. Provisional Patent Application 60 / 785,202, filed Mar. 24, 2006 (assigned to the assignee of the present patent application, the disclosure of which is hereby incorporated by reference). It is included as a).

Further, in the arrangement in which the X axis passing through the pupil 42 and the spot 34 is parallel to the X axis of the sensor 40 as shown in FIG. 2, the speckle pattern deviation δZ is (distortion by the optical system 39). Is strictly limited to the X direction and there is no Y component of the shift . Thus, the image matching operation is simplified and only needs to find the closest matching group of speckles due to X shift . That is, in order to calculate the shift δZ that against the reference image areas of the current image (anything good if the previous image of the speckle pattern), the value of shift δX that best fit to the reference image In order to find out, it is only necessary to check a copy of the region of the current image shifted in the X-axis direction with respect to the reference image.

Alternatively, if the geometry of the components of device 22 deviates from the above criteria, or if lens distortion is significant , the processor may use a single coefficient model to compensate for this deviation. it can. That is, the known displacement is measured, or modeled, the processor, in order to find the actual three-dimensional coordinates of the object plane, thus the coefficient model deviation appropriate against the reference image (X, Y) A copy of the area of the current image that is shifted by the direction shift can be checked.

Usually, for convenience of construction and operation, operation factor of the system 20 is selected to be S _{<< Z obj.} (On the other hand, since the resolution in the Z direction of the system 20 depends on the ratio S / Z _obj , S must be relatively large relative to the intended working distance of the system to obtain the desired resolution. as long as it is not) S << Z _obj, from the lighting device and an image acquisition device, the value of each distance to each object point, becomes a value close to, but in general should exactly equal. Accordingly, the size of the speckle in the speckle pattern image acquired by the image acquisition device 38 varies with a certain tolerance γ throughout the area 46. Some of the computation methods known in the art have been described in the above-mentioned PCT patent application, and they are used to change these dimensional changes in the fit region of the current image relative to the corresponding region of the reference image. Can be compensated.

である。したがって、寸法Ｎの枠の局所的なスペックル・パターンの実質的な寸法不変性は、
（Ｓ・ｓｉｎ（θ）・Ｎ／２・Ｚ_ｏｂｊ）＜０．３（λ・Ｚ_ｏｂｊ／ｗ_０・ｐｓｉｚｅ（Ｚ_ｏｂｊ））
の条件が成り立つときであり、ここで、ｐｓｉｚｅ（Ｚ_ｏｂｊ）は、Ｚ_ｏｂｊにおける画素の寸法である。これらの条件の下で、装置３８により獲得された連続する像フレーム中での被写体のＺ軸方向の偏移は、一般に、スペックル寸法の変化を考慮することなく演算されうる。 However, in general, in order not to put an excessive calculation load on the processor 24, γ falls within a predetermined range according to the size of the frame to be adapted and the characteristic speckle size . It is desirable to maintain it. The inventors generally found that γ should be limited to the extent that the variation in characteristic frame dimensions does not exceed 30% of one speckle dimension . Using the diagonal θ of the field of view of the image acquisition device 38,

It is. Thus, the substantial dimensional invariance of the local speckle pattern of the dimension N frame is
(S · sin (θ) · N / 2 · Z _obj ) <0.3 (λ · Z _obj / w ₀ · psize (Z _obj ))
It is when the condition is satisfied, wherein, psize _{(Z obj) is} the dimension of the pixels in the _{Z obj.} Under these conditions, the shift in the Z-axis direction of the subject in successive image frames acquired by the device 38 can generally be calculated without taking into account changes in speckle dimensions .

FIG. 3 is a flowchart schematically illustrating a three-dimensional mapping method using the system 20 according to an embodiment of the present invention. This method is especially speckle pattern projected by the illuminating device 30 is based on the recognition that does not substantially change over time. Therefore, the three-dimensional map of the subject can be accurately calculated using each image of the speckle pattern projected on the subject acquired by the image acquisition device 38 at a fixed position and a fixed angle with respect to the device . Can do.

Before mapping the object, in a calibration step 50, from a and 22 a known spatial shape to a certain object separated by a known distance, by projecting the speckle patterns from the device 30, calibration of the apparatus 22 I do. Usually, for this purpose, a flat subject over the entire region 46 at a known distance Z _obj is used as a calibration target. Image acquisition device 38 acquires a reference image of the object, to be saved in the memory of the processor 24. This calibration step may be performed at the time of manufacture, reference images stored in the memory as long as the relative movement between different parts material of the device 22 is not performed without control, be used in this region Can do. To save memory, simplifying the subsequent operations, the reference image may also be stored in a suitable, such as an image of based rather binary threshold, and reduce the amount of data format for fitting algorithm used.

When the system 20 is ready for use, in a first image acquisition step 52 , the system 20 is activated and uses the device 22 to acquire an image of the target subject (in this case, the subject 28). Processor 24, in the map calculation step 54, the image is compared with speckle patterns in the stored calibration image. Dark regions of an image that have a pixel value less than a predetermined threshold (or do not contain significant speckle information) are usually classified as shadow regions, from which depth (Z) information is not extracted. . Other part of the image, for an efficient adaptation of pairs in the reference image, possibly using a known one adaptive threshold in the art, or binarized Ru, or reduced less of the amount of data.

The processor 24 selects one of the frame in the non-shadow portion of the image, the sub-image in this frame, as compared to the portion of the reference image, which until that part of the reference image to which the sub picture is best fit Continue. See above, as shown in FIG. 2, device 30, 38, if it is arranged along the X-axis, the processor, the sub-image, it is displaced relative X direction with respect to the sub-images (as noted above, it is the magnification dimensional change of the speckle pattern to magnification gamma) portion of the image and comparing lever is sufficient. Processor is to fit part of the reference picture, using a shift in the transverse direction of the sub image, based on the principle of triangulation described above, determines the Z coordinate of the surface region of the object 28 in the sub-image . This region of the object table surface rather than being an X-Y plane direction, when tilted, the speckle patterns within the sub-image shows the distortion. The processor 24 may optionally analyze speckle distortion and estimate the tilt angle, thereby improving the accuracy of the three-dimensional mapping.

The processor 24 may calculate the coordinates of adjacent areas of the image using the map coordinates of the first frame as a starting point. Especially, the processor, and a region of an image, finds a high correlation between the corresponding regions in the reference image, shift against the reference image of this region, shift of the adjacent pixels in the image Can be useful in predicting Processor, a pixel in which these adjacent more deviation in the range close equal shift or to the shift of the region adapted initially to try to conform to the reference image. In this way, the processor expands the range of the fit region and continues this until it reaches the end. In this way, the processor calculates the Z coordinate of the non-shadow region of the image until the three-dimensional contour of the subject 28 is completed. This approach has the advantage that it can be quickly and robustly matched even with small frames and images with poor signal to noise ratios. Details of the computing methods that can be used for this purpose are described in the aforementioned PCT patent application.

As a result of the above steps, the processor 24 has computed a complete three-dimensional map of the portion of the object plane that can be seen in the first image. However, in the next image step 56 , the method can be easily extended to acquire and analyze successive images to track the three-dimensional movement of the subject. The device 22 acquires successive images at a predetermined frame rate, and the processor 24 updates the three-dimensional map based on each successive image. The three-dimensional map may be computed on a stored and calibrated reference image if desired. Alternatively, the subject typically does not move much from one image frame to the next, so it is often more efficient to use each successive image as a reference image for the next frame.

In this manner, processor 24, in shift operation step 58, in order to calculate the speckle of each sub-image, a relative shift in the X direction for the same speckles preceding in the image, each successive image it can be compared with the preceding image a. Usually, this shift does not exceed several pixels, so that the calculation can be performed quickly and efficiently. After each new image has been processed in this way, the processor 24 outputs an updated three-dimensional map in a new map output step 60. This image acquisition and update process can thus be performed indefinitely. Since operation of the continuous three-dimensional map is easy, the system 20 is a simple, while using low-cost image hardware and processing hardware, fast real-time video rates than or about 30 frames / sec, map coordinates Can be operated and output. Furthermore, the efficient image matching computation and range expansion method as described above allows the system 20 to operate at video speeds even when a local shift cannot be computed from the preceding image.

Because of this capability of the system 20, the system 20 can be used properly in a wide range of applications , and can be implemented in machine interfaces based on human movements in particular. In such an interface, a computer (having a processor 24 or capable of receiving a three-dimensional map output by the processor) allows a user's body part (eg, arm, hand and / or finger, and possibly the head, Identify one or more quantities in the three-dimensional map corresponding to the torso and other extremities. The computer is programmed to identify actions corresponding to certain movements of these body parts and to control computer applications in response to these actions . Examples of such operations and applications include the following.
Mouse Interpretation and Click—The computer interprets the movement of the user's hand and fingers as if the user moved the mouse on the table and clicked the mouse button.
-Point, select, and interpret the subject freehand on the computer screen.
A computer game in which an actual or virtual subject used in the game is hit, grabbed, moved, and released by user action .
A computer interface for the disabled user based on detecting limited movements that the user can perform.
Other types of applications on the virtual keyboard will be obvious to those skilled in the art.

Returning to FIG. 2, the beam 36 and extends beyond the Rayleigh distance, intensity of illumination to be dropped on the subject 28 is reduced substantially in proportion to the Z ^2. The contrast of the speckle pattern projected on the subject also decreases accordingly. This particularly decreases when there is strong ambient light having the same wavelength as that of the light source 32. Accordingly, the depth (Z coordinate) range over which the system 20 can produce useful results can be limited due to the large Z and weak illumination. This point Ri known der in the art, can be mitigated by your adaptive control and image system. Some examples of such suitable methods are described in the above-mentioned PCT patent application PCT / IL2006 / 000335. Alternatively, or in addition, as described below, the optical beam-shaped formation, it is possible to improve the contour illumination.

FIG. 4 is a schematic side view of a lighting device 70 used in the system 20 to widen the useful depth range of the system, according to one embodiment of the present invention. The device 70 includes a beam forming device 72 together with the light source 32 and the diffusion plate 33. The beam forming apparatus, reduces the divergence between the intermediate region 76, it has been designed to generate a beam 74 to maintain the linear dimension of the speckle pattern of the axial distance Z in this region. As a result, over the entire area 76 in the image of the object 28, it is maintained contrast high speckle, resulting depth range is widened, which is covered by a three-dimensional mapping system. Several optical designs that can be used to improve performance in region 76 are described below.

FIG. 5 is a schematic side view of a beam forming device 72 according to one embodiment of the present invention. The beam forming apparatus includes a single diffractive optical element (DOE) 80, consists of one conical lens 82. The DOE 80 may be in contact with the diffusion plate 33 or may be incorporated as an etching layer or a deposition layer on the surface of the diffusion plate. Various diffraction designs can be used to reduce beam divergence in region 76. For example, the DOE 80 may have a pattern that is concentric with the center on the optical axis of the light source 32 and that the radius of the circle is randomly distributed. The conical lens 82 has a conical outline centered on the optical axis, that is, a kind of rotationally symmetric prism. Both the DOE 80 and the conical lens 82 have the effect of creating a long focal region along the optical axis, so that any one of these members can be used to create a region with reduced beam divergence. This reduction in divergence is further enhanced by using two members together.

FIG. 6 is a schematic side view of a beam forming apparatus 90 according to another embodiment of the present invention. The beam forming apparatus 90 includes one DOE 92 and lenses 94 and 96 having a focal length F. As shown, these lenses are separated from the diffuser plate 33 and the DOE 92 by a distance equal to the focal length, so that the DOE is located in the Fourier plane of the diffuser plate. Therefore, the Fourier transform of the diffuser is multiplied by the transmission function of the DOE. In the far region, the speckle pattern is multiplied by the Fourier transform of the pattern on the DOE.

As shown in FIG. 4, the DOE pattern can be selected such that the Fourier transform reduces divergence and / or provides more uniform illumination across the illumination beam. The latter problem is that transmission in the central region is lower than that in the peripheral region (as opposed to the intensity distribution due to the angle of the beam from the diffuser plate 33 which is bright in the center and tends to decrease as the angle increases from the optical axis). Can be achieved by designing the diffractive optical element ( DOE) 92 as follows. To light intensity of interest, also other designs of DOE92 or DOE80 (Figure 5) for providing a light intensity contrast more uniform speckle, be apparent to those skilled in the art, it is within the scope of the present invention it is conceivable that.

FIG. 7 is a schematic side view of one optical correlator 110 for determining the Z coordinate of the region of the subject 28 that may be used in the system 20 according to yet another embodiment of the present invention. That is, the correlator 110 uses optical techniques that perform some of the functions of the processor 24 described above. The correlator is very fast and can determine the coordinates of multiple regions of the subject in parallel at approximately the same time. Therefore, it is very useful for an application field in which the subject moves quickly.

One lenslet array 116 forms a plurality of sub-images of the subject 28 under speckle illumination by the device 30. One aperture array 118 limits the field of view of the lenslet array 116 so that each sub-image contains only light from a narrow angular region. The second single lenslet array 120 projects the sub-image onto one DOE 122. The lenslet array 120 is separated from the surface of the sub-image by a distance equal to the focal length of the lenslets in the array and is separated from the DOE 122 surface by an equal distance. One lenslet array 124 of the rear is located between the DOE122 and the sensor 40, by a distance equal to the focal length of the lenslet away from the respective.

DOE122 is a spatial Fourier transform of the reference speckle pattern speckle image of the subject 128 are compared, has one reference diffraction patterns. For example, the reference diffraction pattern can be a Fourier transform of the calibration speckle image formed in step 50 using a plane that is a known distance from the light source. In this case, the reference diffraction pattern can be etched or deposited on the DOE surface. Alternatively, the DOE 122 may consist of a single spatial modulator (SLM) that is driven to dynamically project the reference diffraction pattern .

In either case, the correlator 110 is a sub-image of the (formed by the lenslets during lenslet array 116) subjects, multiplied by the reference speckle pattern in the Fourier space. Therefore, the intensity distribution projected onto the sensor 40 by the lenslet array 124 matches the cross-correlation between the reference speckle pattern and each sub-image . In general, the intensity distribution on the sensor has a plurality of correlation peaks, each peak corresponding to one of the sub-images . The transverse shift value of each peak relative to the axis of the corresponding sub-image (defined by the corresponding aperture in the aperture array 118) is the transverse displacement of the speckle pattern on the corresponding region of the subject 28. It is proportional to This displacement is further, as described above, it is proportional to the Z-direction displacement of the area to be against the surface of the reference speckle pattern. Thus, the output of the sensor 40, in order determine the Z coordinate of the area of each sub-image, and, in order to calculate the three-dimensional map of the object can be processed.

Although the embodiments described above relate to the particular system 20 configuration and device 22 design described above, the principles of the present invention are also applicable to other types of speckle based 3D mapping systems and devices. It can be applied to. For example, the aspect of the above-described embodiment may be applied to a system using a plurality of image acquisition devices , or may be applied to a system in which the image acquisition device and the illumination device are relatively movable. .

Although the above embodiments have been shown by way of example, it is understood that the present invention is not limited to what has been particularly shown and described herein. Rather, the scope of the present invention is the combination of the various features described above in this application, any combination thereof, or any combination thereof, as would be conceived by one of ordinary skill in the art upon reading the above description and not disclosed in the prior art. Includes changes and modifications.

1 is a schematic diagram illustrating a three-dimensional mapping system according to an embodiment of the present invention. 1 is a schematic top view of a speckle imaging device according to an embodiment of the present invention. 4 is a flowchart schematically illustrating one method of three-dimensional mapping according to an embodiment of the present invention. FIG. 6 is a schematic side view of a lighting device used in a three-dimensional mapping system according to another embodiment of the present invention. 1 is a schematic side view of a beam forming apparatus according to an embodiment of the present invention. It is a schematic side view of the beam forming apparatus by further another embodiment of this invention. FIG. 6 is a schematic side view of an optical correlator used in a three-dimensional mapping system, according to yet another embodiment of the present invention.

20 three-dimensional mapping system 22 speckle imaging device 24 image processor 28 subjects 30 illumination device 32 coherent light source 33 diffusion plate 38 image acquisition apparatus 39 objective optical system 40 imaging sensor

Claims (42)

One illumination device having one coherent light source and one diffuser arranged to project one primary speckle pattern onto one subject;
A single image acquisition device arranged to acquire the first speckle pattern image on the subject from a single and fixed position and angle relative to the illumination device;
The subject acquired within one or more of the images of the primary speckle pattern acquired at the single and fixed position and angle to derive a three-dimensional map of the subject. said first primary speckle pattern of the plurality of regions of the upper reference of the primary speckle pattern obtained on the reference surface of known contour from said single image acquisition device a known distance One processor connected to find each shift between the images;
Here, the respective shifts indicate the respective distances between the plurality of regions and the image acquisition device,
A three-dimensional mapping apparatus for a subject characterized by comprising:   2. The apparatus of claim 1, further comprising: a stand attached to the illumination device and the image acquisition device to hold the image acquisition device in a fixed positional relationship with the illumination device. The device described. The image acquisition device includes:
One detector element array disposed in one linear pattern defining first and second mutually orthogonal axes;
An objective optical system having one entrance pupil and arranged to focus the image on the detection element array;
Defines the axis of one device that is parallel to the first axis and passes through the entrance pupil and a spot through which one beam emitted by the coherent light source passes through the diffuser. The apparatus according to claim 2, wherein the lighting device and the image acquisition device are arranged by the platform.   Only on the first axis, a deviation between the first speckle pattern acquired in one or more of the images and the reference image of the first speckle pattern. 4. The apparatus of claim 3, wherein the processor is arranged to derive a three-dimensional map by finding a transition.   The image acquisition device is arranged at a predetermined space from the lighting device, wherein the respective shifts are proportional to the respective distances at a rate determined by the space. The apparatus according to claim 1.   The primary speckle pattern projected by the illumination device consists of speckles having one characteristic dimension, where the dimension of the speckle in the image depends on the space 1 6. An apparatus according to claim 5, characterized in that it varies over the entire image by one tolerance and the space is selected to keep the tolerance within one predetermined range.   The processor is configured to associate the respective shifts with respective coordinates of the three-dimensional map using a single coefficient model of distortion in the image acquisition device. The device described in 1.   The processor, between the first speckle pattern in the first region of the subject and the corresponding region of the reference image, in a first one shift with respect to the first region. To find the respective shifts by finding the first one match and to find the respective shifts of pixels adjacent to the first region based on the first shift The apparatus according to claim 1, wherein the apparatus is configured to apply one range extension procedure.   The processor according to claim 1, wherein the processor is configured to process a series of images acquired while the subject is moving in order to map one three-dimensional movement of the subject. 9. The apparatus according to any one of 8.   The subject is a part of a human body, and the three-dimensional movement is an operation performed by the human body part, wherein the processor is responsive to the operation by a computer. 10. Device according to claim 9, characterized in that it is connected to provide one input to the application.   The illumination device is arranged from one beam forming device arranged so as to reduce the fluctuation of the light intensity contrast of the speckle pattern created by the diffuser plate over the entire detection area of the three-dimensional mapping device. The device according to claim 1, characterized in that   The apparatus according to claim 11, wherein the beam forming device comprises one diffractive optical element (DOE).   The beam forming device includes one lens arranged to form one Fourier plane of the diffusion plate, wherein the diffractive optical element (DOE) is arranged on the Fourier plane. The apparatus of claim 12.   The apparatus according to claim 11, wherein the beam forming apparatus is arranged to reduce divergence of light emitted from the diffusion plate.   The beam forming device is arranged to make the intensity of light emitted from the diffuser plate and crossing one surface orthogonal to one optical axis of the illumination device uniform. 11. The apparatus according to 11.   The apparatus according to claim 1, wherein the processor comprises one optical correlator.   The optical correlator is composed of one diffractive optical element (DOE) including one reference speckle pattern, where the image acquisition device generates respective correlation peaks indicating the three-dimensional coordinates of the subject. The apparatus of claim 16, comprising a lenslet array that projects a plurality of sub-images of the subject onto the DOE.   9. A device according to any preceding claim, wherein the coherent light source has a coherence length of less than 1 cm.   The primary speckle pattern comprises speckles having one characteristic dimension, wherein the illumination device changes the distance between the coherent light source and the diffuser by changing the distance between the speckle patterns. 9. A device according to any one of the preceding claims, characterized in that it is configured to be able to adjust the characteristic dimensions of the cable. Illuminating the subject with a beam of coherent light diffused from a light source to project a primary speckle pattern onto a subject;
Using a single image acquisition device to acquire an image of the first speckle pattern on the subject from a single and fixed position and angle relative to the light source;
Processing the image of the first speckle pattern acquired at the single and fixed position and angle to derive a three-dimensional map of the subject, and within one or more of the images said first primary speckle pattern of the plurality of areas on acquired the object in the primary that has been acquired on the reference surface of known contour from said single image acquisition device a known distance Finding each shift between the reference image of the speckle pattern;
Wherein each of the shifts indicates a respective distance between the plurality of regions and the single image acquisition device;
A method for three-dimensional mapping of a subject characterized by comprising:   21. The method of claim 20, wherein the image acquisition device is supported in one fixed positional relationship with respect to the light source while acquiring the image. The image acquisition device includes:
Having one detector element array arranged in one linear pattern defining first and second mutually orthogonal axes;
Here, the light source is composed of one diffusion plate,
Acquiring the image comprises aligning one entrance pupil and one spot of the image acquisition device, wherein the spot is one device with one beam parallel to the first axis. The method of claim 21, wherein the location is through the diffuser plate along an axis.   The step of processing the image comprises: only on the first axis, the first speckle pattern acquired in one or more images and the reference image of the first speckle pattern. 23. The method of claim 22, comprising the step of finding one shift between   21. The respective shifts according to claim 20, characterized in that the respective shifts are proportional to the respective distances at a rate determined by a space at a fixed position from the light source. Method. The primary speckle pattern comprises speckles having one characteristic dimension, wherein the dimension of the speckle in the image is spread over the entire image by one tolerance that depends on the space. Change across,
25. The method of claim 24, wherein acquiring the image comprises selecting the space such that the tolerance is within a predetermined range.   The step of finding each shift comprises the step of associating each shift with a respective coordinate of the three-dimensional map using a coefficient model of distortion in the image acquisition device; The method of claim 20. The step of finding each deviation is:
In the first one shift relative to the first one region, the first speckle pattern in the first region of the subject and the corresponding one region of the reference image Discovering said respective shifts by finding one match;
Applying one range extension procedure to find each respective shift of a pixel adjacent to the first region based on the first shift. The method described in 1.   The step of processing the image comprises the step of processing a series of images acquired while the subject is moving in order to map one three-dimensional movement of the subject. The method according to any one of 20 to 27.   The subject is a part of the human body, and the three-dimensional movement is an action performed by the part of the human body, wherein the step of processing the image is responsive to the action. 30. The method of claim 28, comprising providing one input to one computer application.   The step of illuminating the subject comprises the step of forming the beam so as to reduce fluctuations in the light intensity contrast of the speckle pattern created by the light source over the entire detection area of the three-dimensional mapping device. 28. A method according to any of claims 20 to 27, characterized in that   The method of claim 30, wherein forming the beam comprises passing the beam through a diffractive optical element (DOE).   The light source includes a single diffusion plate, wherein the step of passing the beam includes the step of disposing the diffractive optical element (DOE) on a single Fourier plane of the diffusion plate. Item 32. The method according to Item 31.   The method of claim 30, wherein forming the beam comprises reducing divergence of the beam.   The method of claim 30, wherein forming the beam comprises uniforming the intensity of the beam across a plane orthogonal to one optical axis of the light source.   28. A method according to any of claims 20 to 27, wherein the step of processing the image comprises applying the image to a single optical correlator.   The optical correlator is composed of a diffractive optical element (DOE) including a reference speckle pattern, wherein the step of acquiring the image generates respective correlation peaks indicating the three-dimensional coordinates of the subject. 36. The method according to claim 35, comprising projecting a plurality of sub-images of the subject onto the diffractive optical element (DOE) to achieve this.   28. A method according to any of claims 20 to 27, wherein the coherent light source has a coherence length of less than 1 cm.   Illuminating the subject comprises passing the light from the coherent light source through the one diffuser plate to generate the primary speckle pattern, wherein the primary The speckle pattern consists of speckles having one characteristic dimension, and the method includes changing the distance between the coherent light source and the diffuser to change the characteristic of the speckle. 28. A method according to any of claims 20 to 27, comprising the step of adjusting the dimensions. An illuminator comprising a coherent light source having a coherence length of less than 1 cm and a diffusing plate arranged to project a first speckle pattern on an object;
One image acquisition device arranged to acquire the first speckle pattern image on the subject;
In order to derive a three-dimensional map of the subject , the primary speckles of a plurality of regions on the subject acquired in one or more of the images of the primary speckle pattern Finding each shift between a pattern and a reference image of the first speckle pattern acquired on a reference surface of known contour at a known distance from the single image acquisition device One processor connected to the
Here, the respective shifts indicate the respective distances between the plurality of regions and the image acquisition device,
A three-dimensional mapping apparatus for a subject characterized by comprising:   40. The apparatus of claim 39, wherein the coherence length of the coherent light source is less than 0.5 mm.   41. An apparatus according to any of claims 39 or 40, wherein the coherent light source has a divergence greater than 5 [deg.].   The primary speckle pattern comprises speckles having one characteristic dimension, wherein the illumination device changes the distance between the coherent light source and the diffuser by changing the distance between the speckle patterns. 41. The device according to claim 39 or 40, characterized in that it is arranged to be able to adjust the characteristic dimensions of the cable.

Images