JP2017528714A - Method for optical measurement of three-dimensional coordinates and control of a three-dimensional measuring device - Google Patents

Abstract

A method is provided for scanning and obtaining three-dimensional (3D) coordinates. The method includes providing a 3D measurement device having a projector, a first camera, and a second camera. The method records an image of a pattern of light projected onto an object by a projector. The 3D measurement device is moved along the second path from the first position and the second position. Operation and corresponding control functions are determined based at least in part on the first position and the second position.

Description

Cross-reference of related applications (if applicable)
The present application is German Patent Application No. 10 2015 106 836.2 filed on May 2, 2015, German Patent Application No. 10 2015 106 837.0 filed on May 2, 2015, And claims the benefit of German Patent Application No. 10 2015 106 838.9 filed May 2, 2015. The contents of all of which are incorporated herein by reference in their entirety.

  This application is also a continuation-in-part of US patent application Ser. No. 14/826859 filed Aug. 14, 2015. US14 / 82859 claims the benefit of German Patent Application No. 10 2014 113 015.4, filed on September 10, 2014, and US Provisional Application No. 62 filed on May 14, 2015. This is also a partially continuation application of US Patent Application No. 14 / 712,993, filed on May 15, 2015, which is a non-provisional application of / 161,461. US patent application No. 14 / 712,993 further claims the benefit of German Patent Application No. 10 2014 013 677.9, filed on September 10, 2014. The contents of all of which are incorporated herein by reference in their entirety.

  This application is further a continuation-in-part of US patent application Ser. No. 14 / 722,219 filed on May 27, 2015, which is a non-provisional application of the aforementioned US provisional application No. 62 / 161,461. is there. US patent application No. 14 / 722,219 claims the benefit of German patent application No. 10 2014 013 678.7, filed on Sep. 10, 2014. The contents of all of which are incorporated herein by reference in their entirety.

  The subject matter disclosed herein relates to portable scanners, and more particularly to portable scanners having a display.

  A portable scanner includes a projector that projects a pattern of light onto the surface of an object to be scanned. The position of the projector is determined by the projected encoded pattern. Two (or more) cameras whose relative position and alignment are known or have been determined can record an image of the surface with an additional uncoded pattern. The three-dimensional coordinates (of the pattern points) can be determined by mathematical methods known per se, such as epipolar geometry.

  In video game applications, the scanner is known as a tracking device, in which the projector projects the encoded light pattern onto the target to be followed, preferably the user who is playing, and then this The encoded light pattern is recorded with a camera and the user's coordinates are determined. This data is represented on a suitable display.

  In its simplest form, a system for scanning a scene including distance measurement may include a camera unit having two cameras, an illumination unit, and a synchronization unit. These cameras may optionally include a filter and are used for three-dimensional superposition of target areas. The illumination unit is used to generate a pattern in the target area, such as by a diffractive optical element. The synchronization unit synchronizes the irradiation unit and the camera unit. The camera unit and the irradiation unit can be installed at selectable relative positions. Optionally, two camera units or two illumination units can be used.

  According to one aspect of the invention, a method is provided for optically scanning and measuring an environment. A method includes a three-dimensional (3D) measurement device having a first camera, a second camera, and a projector, the control and evaluation device having a memory and operably coupled to at least one camera and projector. Furthermore, providing a 3D measurement device and providing a corresponding rule between each of the plurality of commands and each of the plurality of operations, wherein each operation from the plurality of operations is a 3D measurement. Including movement along the path by the device, wherein the corresponding rule is based at least in part on movement along the first path, the corresponding rule is stored in memory, and on the object at the projector Projecting a light pattern onto the first time, recording a first set of light pattern images with a first camera and a second camera at a first time, and a second time Recording a second set of images of a pattern of light with a first camera and a second camera and creating a 3D scan of the object based at least in part on the first image and the second image Moving the 3D measurement device along the second path from the first position to the second position and operating based at least in part on the movement from the first position to the second position. Determining, determining a first control function based at least in part on the operation and corresponding rules, and performing the first control function on the 3D measurement device.

  These and other advantages and features will become more apparent from the following description taken in conjunction with the drawings.

  The subject matter regarded as the invention is particularly pointed out and distinctly claimed in the claims that follow at the end of this specification. The foregoing and other features and advantages of the present invention will be apparent from the following detailed description taken in conjunction with the accompanying drawings.

1 is a perspective view of an object in a handheld scanner and in an environment. FIG. It is a figure of the front side of a handheld scanner. It is a figure of the back side of a handheld scanner. FIG. 2 is a top view of a handheld scanner from above. FIG. 3 is a view of a handheld scanner from the right side. FIG. 2 is a perspective view corresponding to FIG. 1 without a housing. Fig. 2 represents a control and evaluation device having a display. FIG. 8 illustrates the control and evaluation device of FIG. 7 from the right side. It is a figure which shows the visual field of a camera which shaded the overlap. It is a figure which shows the geometrical relationship of an image plane, a projector plane, and an epipolar line. FIG. 3 illustrates an exemplary method for optically scanning and measuring an environment. It is a figure which shows the example which has one epipolar line. It is a figure which shows the example which has two epipolar lines. It is a figure which shows the example which has dispersion | variation. It is a figure which shows the step of the method for controlling by the step and movement of a 3rd process block. FIG. 3 shows a first group of movements for a 3D measurement device. FIG. 3 shows a second group of movements for a 3D measuring device. FIG. 6 shows a third group of movements for a 3D measuring device. FIG. 8 is a top view of the control and evaluation device from FIG. 7 with the scale corrected on the display.

  The detailed description explains embodiments of the invention, together with advantages and features, by way of example with reference to the drawings.

  In one embodiment, the carrying structure is mechanically and thermally stable and defines the relative distance and relative alignment of the camera and projector. Placing the surface of the 3D measurement device in contact with the environment on the front side provides an advantage in that these distances and alignments do not change with changes in the shape of the housing.

  The term “projector” is defined to generally refer to a device that creates a pattern. Pattern generation can be by deflection methods such as by diffractive optics or microlenses (or a single laser) or by shading methods such as shutters, transparency (as used in overhead projectors), and other masks Can be done. The deflection method has the advantage that less light is lost and consequently higher intensity is available.

  Depending on the number of assemblies provided for the distance measurement, a corresponding number of arms of the carrier structure are provided, which protrude from a common center located at the intersection of the arms. The assembly can include a camera and projector combination and is provided in the area of the end of the assigned arm. The assemblies may each be arranged on the back side of the support structure. Each of these optics is guided through an assigned aperture in the support structure and is operatively oriented such that each assembly faces the environment from the front side. A housing covers the back side and forms a handle portion.

  In one embodiment, the support structure comprises a carbon or glass fiber reinforced synthetic or ceramic (or another material) matrix. This material provides stability and low weight, and at the same time can be configured with a viewing area. The concave (spherical) curvature on the front side of the support structure not only has structural advantages, but also protects the optical components located on the front side when the front side surface of the 3D measuring device is placed on the work surface .

  The projector creates a projected pattern that may or may not be in the visible wavelength range. In one embodiment, the projected pattern has a wavelength in the infrared range. The two cameras are configured to acquire an image from light in this wavelength range and simultaneously filter out scattered light and other interference in the visible wavelength range. For example, a color camera or a 2D camera can be provided as a third camera for additional information such as color. Such cameras record images of the environment and the object being scanned. In an embodiment where the camera captures colors, a point cloud (a set of points) generated from a scanning process (referred to herein as a “3D scan”) uses a color value assigned from the color information contained in the color image. Can have.

  In operation, the 3D measurement device generates multiple 3D scans of the same scene from different locations. 3D scans are overlaid in a collaborative coordinate system. There are advantages to linking two overlapping 3D scans that the structure in the 3D scan may be recognizable. Preferably, such recognizable structures are searched and displayed continuously or at least after the recording process. Within a determined area, if the density is not the desired level, a further 3D scan of this area can be generated. By refining the video image and the display used to represent the 3D point cloud (which is partly adjacent to it), it helps to recognize in which region the scan should be further generated it can.

  In one embodiment, the 3D measurement device is designed as a portable scanner, i.e. it operates at high speed and is of a size and weight suitable for transportation and use by one person. However, this 3D measurement device is mounted on a tripod (or on another table), on a cart (or another cart) that can be moved by hand, or on an autonomously moving robot, that is, this 3D measurement. It is possible that the device—optionally further—for example by using a separate housing without a transport handle—is not transported by the user. While the embodiments herein describe the 3D measurement device as being handheld, it should be appreciated that this is for purposes of illustration and the claimed invention should not be limited thereto. In other embodiments, the 3D measurement device can also be configured as a compact unit that is fixed or mobile and, where appropriate, built with other devices.

  With reference to FIGS. 1-6, a 3D measurement device 100 is provided as a portable portion of the device for optically scanning and measuring the environment of the 3D measurement device 100 having an object O. As used herein, the side facing the user of the device 100 is referred to as the back side, and the side facing the environment of the device 100 is referred to as the front side. This definition extends to the components of the 3D measurement device 100. The 3D measurement device 100 is provided with a carrying structure 102 having three arms 102a, 102b, 102c so that it can be seen (on its front side). These arms give the support structure 102 a T-shaped or Y-shaped, i.e. triangular arrangement. The region from which the three arms 102a, 102b, 102c intersect and are connected to each other and from which the three arms 102a, 102b, 102c protrude defines the center of the 3D measurement device 100. From the user's viewpoint, the support structure 102 is provided with a left arm 102a, a right arm 102b, and a lower arm 102c. In one embodiment, the angle between the left arm 102a and the right arm 102b is, for example, about 150 ° + 20 °, and the distance between the left arm 102a and the lower arm 102c is, for example, about 105 ° + 10 °. The lower arm 102c is somewhat longer than the other two arms 102a, 102b in some embodiments.

  The support structure 102 is preferably composed of a fiber reinforced synthetic material, such as a carbon fiber reinforced synthetic material (CFC). In another embodiment, the support structure 102 is made from a carbon fiber reinforced ceramic or from a glass fiber reinforced synthetic material. This material makes the support structure 102 mechanically and thermally stable, while providing low weight. The thickness of the support structure 102 is much smaller (eg 5 to 15 mm) than the length of the arms 102a, 102b, 102c (eg 15 to 25 cm). The support structure 102 therefore has a flat basic shape. In some embodiments, the arms 102a, 102b, 102c may include a reinforced spine near the center of the arm. However, this is preferably configured to be curved rather than planar. Such curvature of the support structure 102 is adapted to be a sphere curvature having a radius of about 1 to 3 m. The front side (facing the object O) of the support structure 102 is thereby configured to be concave, and the back side is configured to be convex. The curved shape of the support structure 102 is advantageous to provide stability. The front side of the carrier structure 102 (and the visible area in one embodiment) is configured to be a viewing area, i.e., a concealment, cover, cladding, or other type of packaging. No material is provided. A preferred construction with fiber reinforced synthetic material or ceramic is particularly suitable for this purpose.

  A housing 104 is arranged on the back side of the carrying structure 102, which in the region of the ends of the three arms 102a, 102b, 102c, by suitable connection means, for example by rubber rings and screws with a slight gap. The floating structure 102 is connected to the supporting structure 102 in a floating manner. As used herein, a floating connection is one that reduces or eliminates the transmission of vibrations from the housing 104 to the carrier structure 102. In one embodiment, the floating connection is formed by a rubber isolation mount disposed between the housing 104 and the carrier structure. In one embodiment, a sealing portion made of an elastomer such as rubber is disposed between the outer periphery of the support structure 102 and the housing 104. The carrier structure 102 and the housing 104 are then clamped using an elastomeric bushing. The seal and bushing cooperate to form a floating connection between the carrier structure 102 and the housing 104. Within the region of the left arm 102a and the right arm 102b, the edge of the housing 104 extends to the immediate vicinity of the carrying structure 102, while the housing 104 is within the region of the lower arm 102c from the center of the 3D measurement device 100, Extending a distance to the support structure 102 to form a handle portion 104g, bent at the end of the handle portion 104g to approach the lower arm 102c, where the housing 104 floats in a manner that floats. Connected. The edge of the handle 104g extends to the immediate vicinity of the carrier structure 102. In some embodiments, a portion of the support structure 102 can include a reinforced spine 102r. The spine 102r protrudes into the housing 104. The housing 104 serves as a hood for covering the back side of the support structure 102 and defining an internal space.

  The protective element 105 can be attached to the housing 104 or to the carrier structure 102. In one embodiment, the protective element 105 is located at the ends of the arms 102a, 102b, 102c and extends outwardly therefrom to protect the 3D measurement device from impact and resulting damage. When not in use, the 3D measurement device 100 can be placed with its front side down. Due to the concave curvature on the front side, the 3D measuring device will contact the surface only at the ends of the arms 102a, 102b, 102c. In embodiments where the protective element 105 is positioned at the ends of the arms 102a, 102b, 102c, an advantage is obtained because the protective element 105 will provide additional clearance with the surface. Furthermore, if the protective element 105 is made from a soft material, such as rubber, this provides the desired tactile feel for the user's hand. This soft material can optionally be attached to the housing 104, in particular to the handle 104g.

  On the back side of the 3D measuring device 100, a control actuator or control knob 106 is arranged on the housing 104, so that at least the optical scanning and measurement, ie the scanning process, can be started and stopped. The control knob 106 is located in the center of the housing 104 adjacent to one end of the handle. The control knob 106 may be multifunctional and may provide various functions based on a series of operations by the user. These operations can be time-based (eg, multiple buttons are pressed within a given time), space-based (eg, buttons are moved in a given set of directions), or a combination of both. It's okay. In one embodiment, the control knob 106 can be tilted in several directions (eg, left, right, up, down). In one embodiment, there is at least one status lamp 107 around the control knob 106. In one embodiment, multiple status lamps 107 may be present. These status lamps 107 can be used to indicate the actual state of the 3D measuring device 100 and thus facilitate its operation. The status lamp 107 is preferably capable of showing different colors (eg green or red) to distinguish several states. The status lamp 107 may be a light emitting diode (LED).

  A first camera 111 is placed on the left arm 102a (at its end region) on the carrier structure 102, spaced apart from each other by a defined distance, and on the right arm 102b (at its end portion). The second camera 112 is arranged in the area). The two cameras 111 and 112 are disposed on and fixed to the back side of the support structure 102, in which case the support structure 102 has openings through which the respective cameras 111, 112 can acquire images. It is provided through the front side. The two cameras 111, 112 are preferably surrounded by connecting means for the housing 104 to be floatingly connected to the carrying structure 102.

  Each of the cameras 111 and 112 has a field of view associated with them. The alignment relationship of the first camera 111 and the second camera 112 with respect to each other can be adjusted or adjusted in such a way that these fields of view overlap to allow a stereoscopic image of the object O. If the alignment relationship is fixed, there is a desired predetermined overlap range depending on the application for which the 3D measurement device 100 is used. Depending on environmental conditions, ranges of a few decimeters or even a few meters may be desired. In another embodiment, the alignment relationship of the cameras 111, 112 can be adjusted by the user, for example, by pivoting the cameras 111, 112 in opposite directions. In one embodiment, the alignment relationship of the cameras 111, 112 is tracked and is thus known to the 3D measurement device 100. In another embodiment, the alignment relationship is initially arbitrary (and unknown) and then determined, for example, by measuring the position of the camera, and thus becomes known to the 3D measurement device 100. In yet another embodiment, the alignment relationship is set and fixed during manufacturing or calibration of the 3D measurement device 100. In embodiments where the first camera 111 and the second camera 112 are adjusted, calibration to determine the camera angle and position in the 3D measurement device 100 may be performed in the field. The types of calibration that can be used are further discussed herein.

  The first camera 111 and the second camera 112 are preferably monochromatic, i.e. they can sense a narrow wavelength range, for example by being provided with corresponding filters, which are then scattered. Filter out other wavelength ranges, including light. This narrow wavelength range may be in the infrared range. In order to obtain color information about the object O, the 3D measuring device 100 is preferably aligned symmetrically with respect to the first camera 111 and with respect to the second camera 112, and the 3D measuring device between the cameras 111, 112. It includes a 2D camera, such as a color camera 113, placed in the center of 100. The 2D camera 113 may include an image sensor that can sense light in the visible wavelength range. The 2D camera 113 captures a 2D image of a scene including the object O contained therein, that is, the environment of the 3D measurement device 100.

  In order to illuminate the scene for a 2D camera in adverse lighting conditions, the illustrated embodiment provides at least one light source, for example, four (powerful) light emitting diodes (LEDs) 114. One radiating element 115 is associated with each of the LEDs 114. The light projected from the light emitting diode 114 is deflected from the corresponding LED 114 corresponding to the alignment relationship of the 3D measurement device 100. Such a radiating element 115 can be, for example, a lens or a suitably configured end of a light guide. The radiating elements 115 (four in the illustrated embodiment) are evenly arranged around the color camera 113. Each LED 114 is connected to a radiating element 115 each assigned by one light guide. The LED 114 can thus be structurally arranged, such as by being fixed on its substrate in the control unit 118 of the 3D measurement device 100.

  A sensor such as an inclinometer 119 is provided for later reference to images recorded by the cameras 111, 112, 113. In one embodiment, the inclinometer 119 is an acceleration sensor (having one or several sensitive axes) manufactured in a manner known per se, such as a MEMS (microelectromechanical system). . Other embodiments and combinations of the inclinometer 119 are possible. The data of the 3D measurement device 100 each has a direction of gravity provided by the inclinometer 119 (as one component).

  During operation, images are recorded by the first camera 111 and by the second camera 112. Three-dimensional data can be determined from these images, ie a 3D scan of the object O can be created, for example by photogrammetry. However, the object O has almost no structure or features and may have many smooth surfaces. As a result, it is difficult to generate a 3D scan from the scattered light of the object O.

  In order to solve this difficulty, the projector 121 arranged in the lower arm 102c (the region at the end thereof) can be used. The projector 121 is disposed in the internal space on the back side of the support structure 102 and is fixed to the support structure 102. The support structure 102 is provided with an opening through the front side of the support structure 102 through which the projector 121 can project a light pattern. In one embodiment, the projector 121 is surrounded by connecting means for providing a floating connection between the housing 104 and the carrying structure 102. The projector 121, the first camera 111, and the second camera 112 are arranged in a triangular arrangement with respect to each other and are aligned to the environment of the 3D measurement device 100. The projector 121 is aligned corresponding to the two cameras 111 and 112. The relative alignment relationship between the cameras 111 and 112 and the projector 121 can be preset or set by the user.

  In one embodiment, the cameras 111 and 112 and the projector 121 form an equilateral triangle and have a common tilt angle. When arranged in this manner, and when the views of cameras 111, 112 and projector 121 are similar, the centers of view will meet at a common point at a specific distance from scanner 100. With this arrangement, a maximum amount of overlap can be obtained. In embodiments where the tilt or angle of the cameras 111, 112 and projector 121 can be adjusted, the distance or range to the intersection of the fields of view can be varied.

  When the user places the 3D measuring device 100 on a surface with the front side down, i.e. the front side is facing this surface, the concave curvature on the front side is between the cameras 111, 112, 113 and the projector 121. Create a gap from the surface, which protects each lens from damage.

  Cameras 111, 112, 113, projector 121, control knob 106, status lamp 107, light-emitting diode 114, and inclinometer 119 are connected to a common control unit 118 disposed in housing 104. This control unit 118 can be part of a control and evaluation device incorporated in the housing. In one embodiment, the control unit 118 is connected to a standardized communication interface at the housing 104, such that this interface is wirelessly connected as a projection and reception unit (eg Bluetooth®, WLAN, DECT). Or, as appropriate, as described in DE 10 2009 010 465 B3, the contents of which are incorporated herein by reference, to be cabled (eg USB, LAN) as also being a defined interface, Composed. The communication interface is connected to an external control and evaluation device 122 (as a further component of the device for optically scanning and measuring the environment of the 3D measurement device 100) by means of the wireless connection or cable connection. In this case, the communication interface is configured to be connected by a cable, in which case the cable 125 is plugged into the housing 104, for example at the lower end of the handle portion 104g, so that the cable 125 is The handle portion 104g extends.

  The control and evaluation device 122 may include one or more processors 122a having memory. The processor 122a is configured to operate and control the 3D measurement device 100 and to perform methods for evaluating and storing the measured data. The control and evaluation device 122 may be a portable computer (notebook) or tablet (or smartphone), such as that shown in FIGS. 7 and 8, or any external or distal (eg, in the web) computer. Good. The control and evaluation device 122 may be configured in software for controlling the 3D measurement device 100 and for evaluating the measured data. However, the control and evaluation device 122 can be implemented in separate hardware or it can be incorporated into the 3D measurement device 100. The control and evaluation device 122 may be a distributed component system in which at least one component is incorporated into the 3D measurement device 100 and one component is external. In this way, the processor 122a for performing the method can be implemented in the 3D measurement device 100 and / or in an external computer.

  The projector 121 projects the pattern X that it creates, for example, with a diffractive optical element onto the object O to be scanned. The pattern X does not need to be coded (ie, unary), for example, it may not be coded periodically (ie, multi-valued). This multi-value is solved by using two cameras 111, 112 in combination with accurate knowledge available of pattern shape and orientation. The uncoded pattern may be, for example, the same pattern elements (eg, light spots or lines) that are periodically spaced. The correspondence between the pattern elements projected from the projector 121 and the pattern elements in the image on the photosensitive array of the cameras 111, 112 is through simultaneous epipolar constraints and as discussed further herein. Determined using calibration parameters. By using the two cameras 111 and 112 in combination with knowledge of the shape and orientation of the pattern, point correspondence uniqueness is achieved, and this combined knowledge is obtained from calibration of the 3D measurement device 100.

  As used herein, the term “pattern element” shall mean the shape of an element of pattern X, while the term “point” refers to 3D (of a pattern element or something else). It shall indicate the position in the coordinates.

  The non-coded pattern X (FIG. 1) is preferably a point pattern with points regularly arranged in the grid. In the present invention, for example, 100 points are projected about 100 times (total of 10,000 points) over a field of view FOV (FIG. 9) at an angle of about 50 ° and a distance of about 0.5 m to 5 m. Is done. The pattern X can also be a line pattern or a combination of points and lines, each of which is formed by closely spaced light spots. The first camera 111 includes a first image plane B111, and the second camera 112 includes a second image plane B112. The two cameras 111 and 112 receive at least a part of the pattern X at their respective image planes B111 and B112, and are sensitive to capture a part of the pattern X reflected from the object O in these image planes. An array (eg, a CMOS or CCD array) is placed.

  There is a correlation between the point density, the distance between the projector 121 and the object O, and the resolution obtained for the created pattern X. In the case of pattern generation by diffraction, light from one light source is dispersed over the pattern. In this case, when the total output of the light source is limited, the brightness of the pattern element depends on the number of elements in the pattern. Depending on the intensity of light scattered from the object and the intensity of the background light, it can be determined whether it is desirable to have fewer but brighter pattern elements. Fewer pattern elements means a decrease in the density of acquired points. Thus, it seems useful to be able to generate at least one other pattern besides pattern X. In order to use the patterns desired for the current situation, dynamic transitions between patterns and / or spatial mixing are possible depending on the generation of the patterns. In some embodiments, the projector 121 can produce two patterns that are offset relative to each other with respect to time, are in different wavelength ranges, or have different light intensities. The other pattern may be a pattern deviating from the pattern X, such as an uncoded pattern. In the illustrated embodiment, the pattern is a point pattern with a regular arrangement of points having different distances (grid length) to each other.

  In one embodiment, the pattern X is a single color pattern. The pattern X can be created by the diffractive optical element 124 in the projector 121. The diffractive optical element 124 converts a single light beam from the light source 121a in FIG. 20 into a set of light beams each having a light intensity lower than the single light beam. Each set of rays travels in a different direction and creates a spot when it strikes the object O. The light source 121a may be, for example, a laser, a super luminescent diode, or an LED. In some embodiments, the wavelength of the light source 121a is in the infrared range. The lateral resolution is in this case limited only by the diameter and spacing of the light spots in the projected pattern X. When the pattern X is in the infrared range, it is possible to capture an image of the object O and the surrounding environment without receiving interference from the pattern X using the 2D camera 113. Similarly, if the pattern X is created within the ultraviolet wavelength range, the image acquired by the 2D camera 113 will not receive interference from the pattern X.

  In one embodiment, the projector 121 creates a pattern X on the object O only during the time period in which the cameras 111, 112 (and 113 if available) are recording an image of the object O. This provides advantages in energy efficiency and eye protection. The two cameras 111, 112 and the projector 121 are synchronized or linked with each other in terms of both time and the pattern X used. Each recording process begins with the projector 121 creating a pattern X, similar to a flash in photography, followed by the cameras 111, 112 (and 113 if available), in other words, image pairs. For example, one image is acquired from each of the two cameras 111 and 112. As used herein, a pair of these images acquired at substantially the same time is called a “frame”. The recording process may include one single frame (shot) or a series of multiple frames (video). Such a shot or such video is triggered by the control knob 106. After processing the data, each frame then constitutes a 3D scan consisting of a point cloud in 3D space. This point cloud is defined within the relative local coordinate system of the 3D measurement device 100.

  Data supplied by the 3D measurement device 100 is processed in the control and evaluation device 122 to generate a 3D scan from the frame. The 3D scans are then connected or overlaid in a joint coordinate system. For superposition, known methods can be used, such as by identifying natural or artificial targets (ie recognizable structures) in the overlapping region of two 3D scans. Through identification of these targets, the assignment of two 3D scans can be determined by corresponding pairs. The entire scene (multiple 3D scans) is thus progressively superimposed by the 3D measurement device 100. The control and evaluation device 122 is provided with a display 130 (display device) that is incorporated or connected externally.

In this exemplary embodiment, the projector 121 is not collinear with the two cameras 111 and 112, but the cameras 111 and 112 are arranged to form a triangle. As shown in FIG. 10, this triangular configuration allows the use of epipolar geometry based on optical mathematical methods. Due to the epipolar geometry constraint, the point on projector plane P 121 of projector _{121 is} on the first epipolar line on first image plane B _{111 and} on the second epi plane on second image plane B _112. The epipolar line for each of the image planes B ₁₁₁ and B ₁₁₂ will be determined by the relative geometry of the projector 121 and the two cameras 111 and 112. Furthermore, the points on the first image plane B ₁₁₁ fall on the epipolar line of the projector plane P _{121 and on} the epipolar line of the second image plane B _{112, and} are epipoles in the projector plane and the second image plane. The line is determined by the relative geometry of the projector 121 and the cameras 111 and 112. Still further, points on the second image plane B ₁₁₂ fall on the epipolar line of the projector plane P _{121 and on} the epipolar line of the first image plane B ₁₁₁ , and the epi planes in the projector plane and the first image plane The polar line is determined by the relative geometry of the projector 121 and the cameras 111 and 112. Through the use of at least two cameras and one projector, the image plane B ₁₁₁ even if the projected pattern elements do not have distinctive features, for example having the same shape (uncoded pattern). It can be seen that, for B and B ₁₁₂ and points on the projector plane P ₁₂₁ , sufficient epipolar constraints are provided to allow the correspondence between the points in the pattern X to be determined.

In one embodiment, at least three units (eg, projector 121 and two cameras 111, 112) are used to generate a 3D scene. This realizes an unambiguous triangular relationship between points and epipolar lines, from which the correspondence of the projection of the pattern (X) in the two image planes B ₁₁₁ and B ₁₁₂ can be determined. . With the additional stereo geometry for the camera pair, a considerable number of otherwise indistinguishable pattern points can be further identified on the epipolar line “e”. The density of features on the object O can therefore be increased at the same time, and the size of the features of the pattern X (eg spots) can be kept very low. This utilizes a coded pattern in which the size of features in the pattern has a lower limit based on the resolution of the projector, and this size limit in the coded pattern limits the lateral resolution of the 3D scan. This is in contrast to this method. When the correspondence between the points X on the projector 121 and the cameras 111 and 112 is determined, the three-dimensional coordinates of the points on the surface of the object O can be determined by triangulation with respect to the 3D scan data.

  A triangulation calculation can be performed between the two cameras 111, 112 based on the baseline distance between the two cameras 111, 112 and the relative angle of the tilt of the two cameras 111, 112. Triangulation calculations can also be performed between the projector 121 and the first camera 111 and between the projector 121 and the second camera 112. In order to perform these triangulation calculations, it is necessary to determine a baseline distance between the projector 121 and the first camera 111, and another baseline between the projector 121 and the second camera 112. It is necessary to determine the distance. In addition, the relative angle of tilt between the projector / first camera and the projector / second camera is used.

  In principle, any one of these three triangulation calculations is sufficient to determine the 3D coordinates of the point X on the object O, so the extra two triangulation relationships are redundant information ( Redundancy), which can be usefully employed to provide self-checking of measurement results and to provide self-calibration functionality, as further described herein. As used herein, the term “redundancy” refers to determining 3D coordinates multiple times for a particular point or set of points on an object.

  Additional 3D data can be obtained by photogrammetry methods, for example from several parts of the image acquired from the 2D camera 113 or by the cameras 111, 112, by using several frames with different camera positions. In order to perform photogrammetry calculations, the object seen by the cameras 111, 112, 113 should be illuminated. Such illumination may be background illumination, such as from the sun or artificial lighting. Background illumination may be provided by the 3D measurement device 100 or by another external light source. In some embodiments, the object is illuminated with light from LED 114. Irradiation allows the two-dimensional cameras 111, 112, 113 to distinguish object characteristics such as color, contrast, and shadow, which facilitates identification of object features.

  The measurement process also has a temporal aspect. Typically, an uncoded pattern is used with a stationary device to allow a whole series of patterns to be projected and an image captured to determine a single 3D scan. It was. In order to determine this 3D scan, both the scanner and the object had to remain stationary relative to each other. In contrast, embodiments of the present invention generate one 3D scan for each set of images acquired by cameras 111, 112. In another embodiment (not shown), a second projector is placed adjacent to the projector 121 or additional diffractive optical elements are provided. This second projector projects on the object O at least one second pattern in addition to the pattern X. In an embodiment with two projectors, it is possible to switch between the pattern X and the second pattern in order to continuously capture one set of images having different patterns. Therefore, the 3D scan has a higher resolution by combining evaluation results obtained from different patterns.

  In order to determine the coordinates of the entire object, the 3D scans created from the images acquired by the cameras 111, 112 need to be superimposed. In other words, the 3D point cloud of each frame must be inserted into a common coordinate system. For example, superposition is possible by videogrammetry such as “structure from motion” (SFM) or “simultaneous localization and mapping” (SLAM) method. The natural texture of the object O is also used as a common reference point. In one embodiment, a stationary pattern is projected and used for overlay.

  Data provided by the 3D measurement device 100 is processed in the control and evaluation device 122 by generating a 3D scan from the frame. The 3D scans are then subsequently connected or overlaid in a joint coordinate system. As discussed above, superposition methods known in the art can be used, such as by using natural or artificial targets (ie, recognizable structures). These targets can be localized and identified to determine the assignment or alignment of two different 3D scans relative to each other by corresponding pairs. A 3D scan (sometimes referred to colloquially as a “scene”) can then be progressively superimposed by the 3D measurement device 100. In this exemplary embodiment, the control and evaluation device 122 is provided with a display 130 (display device) so that the user can inspect the 3D scan.

  FIG. 11 shows an embodiment for an optical scanning and measuring process of the environment of the 3D measuring device 100. In a first process block 201, a pair of images for one frame is captured and the measured values are determined and recorded. In this exemplary embodiment, each image is a 2D array of values representing voltages corresponding to the number of electrons in each pixel well.

  The process then proceeds to a second process block 202 where the measured value is evaluated and 3D scan data is generated from the frame. In this second process block, the number is evaluated in combination with the known positions of the spots projected from the projector, and the 3D coordinates of the points on the object O are determined. Using triangulation calculations (including epipolar geometry calculations), the distance for each of the pixels is obtained, and the two angles for each pixel are the pixel location and camera geometry (lens focal length, pixel size, etc.) To be determined.

  The process then proceeds to a third process block 203 where multiple frames of 3D scan data are overlaid in a common coordinate system. In one embodiment, video grammetry information is obtained using a central color camera 113 to assist in the overlay. In one embodiment, a representation of 3D scan data is displayed and saved. Finally, the process proceeds to a fourth process block 204 where automatic calibration is performed. In one embodiment, the automatic calibration performed in the fourth process block 204 can be performed simultaneously with the second process block 202 or can be performed sequentially. As long as the operator continues to operate so that additional frames are acquired, such as by continuously pressing the control knob 106, the process returns to the first process block 201.

In a second process block 202, an image of a particular point in the frame is selected to automatically determine the correspondence. These selected points correspond to points on the object O, in particular pattern elements of the pattern X (eg spots). For each image of a selected point in the first image plane B ₁₁₁ , the epipolar line “e” is successively located in the second image plane B ₁₁₂ and in the projector plane P ₁₂₁ . This procedure for positioning the epipolar line “e” is repeated for the point image in the second image plane B ₁₁₂ and for the point image in the projector plane P ₁₂₁ . These multiple epipolar constraint conditions allow the determination of a one-to-one correspondence (eg, the same spot) between the projected pattern element and the received pattern element in each of the three planes. As shown in FIG. 10, the point X _O (of the pattern X on the object O) can be seen on all three planes B ₁₁₁ , B ₁₁₂ , P ₁₂₁ , and the image is on each plane On the two epipolar lines “e”. A point X _O is an intersection of three straight lines, the light beam of the projector 121 and one light beam of each of the two cameras 111 and 112. The point X _O can thus be shown unambiguous if the density of the points in the pattern X is sufficiently low.

The automatic calibration performed at block 204 is automatic and does not require the use of an external calibration object. There is no variation from a spatial perspective (geometric variation) and a temporal perspective to correct the calibration where appropriate, using the initial calibration first and then the calibration currently used The frame is examined for variations (parameter changes over time). The variation may have a thermal cause, for example, due to an increase in operating temperature. The variation may also have a mechanical cause, such as a mechanical shock caused by the 3D measurement device 100 hitting the surface or floor. These variations are manifested through shifts in measured positions, angles, and other geometric features of points on the object or in the planes B ₁₁₁ , B ₁₁₂ , P ₁₂₁ , for example.

  Calibration parameters that may be corrected may be extrinsic parameters, intrinsic parameters, or operating parameters. The extrinsic parameters for each unit (cameras 111, 112, 113 and projector 121) include six rigid degrees of freedom, namely three positions and three angles. Of particular relevance are the relative geometries between the cameras 111, 112, 113 and the projector 121, such as the relative distance and angle of their alignment. Internal parameters include, but are not limited to, focal length, principal point location, distortion parameters, centering of the photosensitive or MEMS projector array, scale of the array in each direction, and the array dimensions relative to the local coordinate system of the 3D measurement device 100. It may be related to camera or projector device characteristics such as rotation and aberration correction factors for the camera or projector lens system. The operating parameters may be the wavelength of the light source 121a, air temperature and humidity.

With respect to FIG. 10, the variation may be the actual position shift Δ of point X _O relative to its expected theoretical position in one of the three planes.

  As described hereinabove, epipolar constraints are used to determine the correspondence between the projected and imaged pattern elements (ie, these images) on the two cameras 111, 112 and projector 121. It is solved simultaneously. Some redundant information (redundancy) from these simultaneous equations can be used to account for variations in correspondence.

  In addition, as described hereinabove, three separate triangulation calculations may be performed to obtain three sets of 3D coordinates. These three triangulation calculations are: 1) between the first camera 111 and the second camera 112 (stereoscopic camera), 2) between the projector 121 and the first camera 111, and 3) the projector 121. And the second camera 112. The 3D coordinates obtained from these different triangulation calculations can be compared and the calibration parameters can be changed if variations are seen.

FIG. 12 shows a simplified situation of variation for the _two units U ₁ , U ₂ . The units U ₁ and U ₂ may be located either between the two cameras 111 and 112 or between the projector 121 and one of the cameras 111 and 112, for example. Each unit U ₁ , U ₂ comprises a plane in which a point can be selected. The two epipolar lines “e” are common to both planes. A selected point 236 in the plane of unit U ₁ corresponds to a point 216 in the plane of unit U ₂ . It should be appreciated that both points 216, 236 are images of real points on the object O. For example, when the points 216 and 236 are images of the spot of the pattern X on the object O, the correspondence can be detected. In one embodiment, a “spot” on the object O is illuminated and the adjacent area is dark or darker than the illuminated spot. However, when the distance of the points 216, 236 perpendicular to the epipolar line “e” is not the same in both planes, the deviation Δ, ie the deviation Δ between the position of the point 216 and the expected position 218, Occurs.

In general, the deviation Δ is a vector. In an embodiment with _two units U ₁ , U ₂ , such as projector 121 and camera 111, for example, only the component of the deviation Δ perpendicular to the epipolar line “e” is known. The component parallel to the epipolar line “e” disappears when determining the 3D coordinates. In an embodiment having more than two units, such as the projector 121 and both cameras 111, 112, the component of the deviation Δ in both dimensions of the plane is represented by redundant measurements (and in detecting the deviation). Redundancy in determining 3D coordinates). If we have a deviation Δ for several selected points, all the determined deviations Δ can be drawn in the map shown in FIG. 13, which is sometimes called the error field. In the case of only two units, only one component of each deviation Δ can be drawn in the error field. If the variation is due to a single cause or reason, the error field is typical for a particular type of variation. FIG. 13 illustrates an example of an error field generated by the rotation of the first camera 111 about the direction of the line of sight, such as when the calibration parameter relating to the roll angle of the first camera 111 is not accurate.

Referring now to FIGS. 12-14, the shift Δ is now described with respect to epipolar constraint conditions. In general, there are two possibilities for the units involved (camera and projector). In one possibility, one of the units U ₁ , U ₂ is a camera and the other is a projector. In the other possibility, both units U ₁ , U ₂ are cameras. It should be appreciated that the use of a single projector and two cameras is for illustrative purposes and the claimed invention should not be limited thereto. In other embodiments, additional projectors or cameras may be used. Having more than two units (eg, having two cameras and one projector) provides additional capabilities in the automatic determination of calibration parameters, as discussed in more detail herein.

Each of the units U ₁ , U ₂ has an origin, sometimes referred to as the perspective center point O ₁ , O ₂ or the center of projection. This point represents the point through which all rays pass when moving out of the unit (in the case of a projector) or into the unit (in the case of a camera). In a real device, not all rays pass through the fluoroscopy center point, but it is understood that the software can calibrate the calibration parameters of the camera system and pass these points through the corrected ray. I want. The two perspective center points O ₁ and O ₂ define a baseline distance 208 between the units U ₁ and U ₂ .

Each of the units U ₁ and U ₂ also has planes 210 and 230 on which images are formed. In the projector this plane is known as the projector plane P ₁₂₁ and in the camera this is known as the image planes B ₁₁₁ , B ₁₁₂ . Typically, in an actual projector or camera, the projector plane P ₁₂₁ or the image planes B ₁₁₁ , B ₁₁₂ are not behind the perspective centers O ₁ , O ₂ as illustrated in FIG. 14, but behind them. It is in. In most cases, a hardware device such as a photosensitive array (in a camera) or a pattern generator (in a projector) is placed in a plane position behind the perspective center. However, it should be appreciated that the presence of the plane in front of the perspective center as shown in FIG. 14 is mathematically equivalent to the plane being positioned on the other side of the perspective center.

The perspective centers O ₁ and O ₂ are separated by a baseline distance B. A base line 208 connecting the perspective centers O ₁ and O ₂ intersects the planes 230 and 210 at points E ₁ and E ₂ . This point of intersection is called the epipole and is also known as the epipole. A line drawn on a corresponding plane through one of the epipoles is called an epipolar line. For the plane 210 and the corresponding epipolar E ₂ , the epipolar line is 212. The point P ₁ on the plane 230 is on the epipolar line 212.

As explained above, each ray such as ray 232 passes through the perspective center point O ₁ and reaches a plane such as 230 where an image is formed. If the plane 230 is a projector plane, the point P ₁ is projected onto the object at a point such as P _A , P _B , P _C , or P _D depending on the distance to the object. These points P _A , P _B , P _C , P _D share a common ray 232, and in this example the corresponding points Q _A , Q _B , Q _C , or on unit U ₂ , which is a camera, in Q _D, it fits on epipolar 212. Both beam 232 and epipolar 212 is on a plane including the point _O 1, _{O 2,} and _{P D.} If the plane 230 is a camera plane rather than a projector plane, the point P ₁ received by this camera image plane is from any point on the epipolar line 212, eg, points Q _A , Q _B , Q _C , or may be reached from any of the Q _D.

FIG. 13 shows an epipolar line 234 on the plane 230 of the unit U ₁ in addition to the epipolar line 212 on the plane 210 of the unit U ₂ . Any point V ₁ (and W _A , W _B , W _C ) on the epipolar line 212 corresponds to a corresponding point (image or projection point) U _A , U _B , U _C , U _D on the epipolar line 234. Must have. Similarly, any point _U A on the epipolar _{234, U B, U C,} U D is should have _W A point corresponding on epipolar _212, W _B, W C, the _{V 1} . The point set 240 represents points in the space that can intersect the object O, for example, V _A , V _B , V _C , V _D.

  In the normal execution of measurements by the 3D measurement device 100, the 3D measurement device 100 performs self-calibration or automatic calibration if the measurement results are also used to obtain a corrected calibration parameter (ie corrected calibration). ing. In an embodiment of the invention, the first step to obtain a corrected new calibration parameter is selected on the projector plane or image plane in relation to the expected position based on epipolar constraints. It is to detect the variation in the position of the image of the point.

  An example of such variation is shown in FIG. Point 236 on plane 230 intersects the object at point 238. According to epipolar geometric constraints, point 238 should appear on epipolar line 212, specifically at point 218. In this case, the actual point was observed at position 216. In general, when using only two planes 210, 230, it is only known that for a point 236 that falls on the epipolar line 234, the corresponding point should fall on the epipolar line 212. It is. The fact that point 216 does not fit on epipolar line 212 suggests that there is a problem with the calibration parameter, but whether the defective calibration parameter is an extrinsic parameter or an intrinsic parameter or operation It is difficult to state whether it is a parameter based on the observation of a single point 216.

FIG. 14 shows some errors that may be seen in the extrinsic calibration parameters. One type of error that may be seen is the baseline distance B, as well as the particular location of the perspective centers O ₁ , O ₂ , at the baseline 208. In other words, in the coordinates of the perspective center point along the direction of the nominal baseline 208 (error 252) and in the direction perpendicular to the direction of the nominal baseline 208 (error 254). Another possible extrinsic error is in the angular orientation of unit U ₁ or unit U ₂ . One way to characterize this orientation is to focus on the pitch angle 256 about the axis 255 and the yaw angle 258 about the axis 257. If the calibration parameters for the pitch and yaw angles of the planes 230 and 210 are not correct, the points on the plane will not correspond to the positions expected based on the epipolar geometric constraints. Unit U ₁ and unit U ₂ may have incorrect calibration parameters with respect to the roll angle of the camera or projector. Such errors can be detected and corrected in an automatic calibration technique as discussed further herein. The roll angle calibration parameter may be considered an intrinsic calibration parameter rather than an extrinsic parameter.

If deviation is too large, when solved by simultaneous equations epipolar constraint, it may be difficult to find the correspondence of the point X _O occurs. In some embodiments, pattern X consists of a large number (eg, 10,000) of low light intensity and a smaller number (eg, 1,000) of high light intensity. Due to this difference in light intensity, the 3D measurement device 100 can recognize an object having a surface with high or low reflectivity. If difficulty arises in finding the correspondence, the spots projected in the pattern X can be further spaced apart to reduce ambiguity in determining the correspondence in the fourth process block. In embodiments where the light intensity is different, the lower intensity spots can be filtered out or at least reduced by reducing the exposure time and / or lowering the total output of the projector 121. In this example, only the spots with high light intensity (with larger intervals) are visible with the cameras 111 and 112. This reduces ambiguity when determining the correspondence.

In order to determine the calibration parameters sufficiently and correctly, it is advantageous to use the entire volume around the 3D measuring device 100 for the measurement, in particular using the depth information for the determination of the extrinsic parameters. . As an example, FIG. 19 shows (schematically) how the relative geometry of the two cameras 111 and 112 is verified. For this purpose, two points X ₁ , X ₂ with different distances to the 3D measuring device 100 (ie different depths) are selected. Each of the points X ₁ or X ₂ and the previous calibration can be used to verify whether the cameras 111 and 112 still provide consistent results. If the variation occurs due to the relative distance or relative alignment shift of the two cameras 111 and 112, it is possible to distinguish between the two types of errors for the two different distances. Due to the high mechanical and thermal stability of the carrier structure 102, deformation due to heat or its mechanical shock of the 3D measurement device 100 is considered rare, but for example in cameras 111, 112, 113 and projectors. In the mounting means 121, this may occur. It is sufficient in most cases to verify the calibration after switching the starting operation of the 3D measuring device 100 and after a long time interval, for example after acquiring 20 to 100 frames.

The calibration parameters can be reviewed according to the determined deviation Δ, for example, the deviation detected by the selected point X ₀ , X ₁ , or X ₂ can be compensated. Using the redundancy in measuring the 3D coordinates of the points, new calibration parameters can be determined. For example, the calibration parameters can be changed (corrected) according to an optimization strategy until the deviation Δ is minimized or below a predetermined threshold. The new calibration parameter is compared with the currently used calibration parameter and if appropriate, ie the deviation between the (corrected) new calibration parameter and the currently used calibration parameter is a predetermined threshold. If it exceeds, replace them. Thus, the calibration is corrected. It should be appreciated that many of the deviations due to mechanical problems are single events and can be corrected by permanent correction of calibration parameters, particularly extrinsic or intrinsic parameters.

  Errors associated with deviations in projector 121 operating parameters can be determined much faster than extrinsic parameters or intrinsic parameter deviations. In some embodiments, the operating parameter to be checked is the wavelength of the light source 121a. This wavelength may change due to, for example, a temperature rise of the light source 121a or a change in pump current. The embodiment in FIG. 20 is shown (schematically). The scale of the pattern X generated by the diffraction element 124 changes as the wavelength changes. As shown in the error field of FIG. 21, there is no shift in the position of the central pattern element when the wavelength changes at the center of the pattern X, such as the zero-order diffraction position of the laser beam. The shift Δ appears as a position shift at a higher diffraction order, for example in a pattern element at the periphery. A single camera 111 or 112 can be used to recognize such shifts in the individual diffraction positions of the pattern elements.

If the position and alignment of the 3D measurement device 100 (and consequently the projector 121) with respect to the object O remain unchanged, the selected point on the object O is the theoretical coordinate X ₁ ( That is, the point having the coordinate (determined by the most recently used calibration) is moved to a point having an actual coordinate X ₂ different from this point. In other words, _{X 1} that is expected, the actual point _{X 2.} As described above in connection with FIG. 19, using the depth information, from each single shot frame, it is possible to determine the actual 3D coordinates of a point selected X _2. If a deviation Δ of the actual coordinates of the point X _{2 from} the theoretical coordinates X ₁ of the selected point occurs, a change in wavelength is detected. This new wavelength is determined from the size of the shift Δ. In another embodiment, the actual coordinates of X ₂ selected points, in order to detect the deviation from the coordinate X ₁ of the theoretical two frames captured at different times are compared.

  If a change in wavelength is observed, action can be taken to counter this change, if appropriate, or the calibration can be corrected. Calibration correction can be performed by replacing the calibration wavelength with the detected new wavelength. In other words, the wavelength calibration parameter is changed. Treatment to counter the change may include, for example, cooling the laser (if the light source 121a is a laser) or reducing the pump current to return the wavelength to its unchanged (ie, original) value. . By applying feedback in the control loop to the pump current or to laser cooling, the laser wavelength can be stabilized.

  When the control system is not used, the wavelength may drift depending on the light source 121a. In these embodiments, a broadband optical bandpass filter may be used to cover a range of possible wavelengths. However, broadband optical filters may pass more unwanted ambient light, which can be a problem when working outdoors. Therefore, a wavelength control system is desirable depending on the application. It should be appreciated that the embodiments described herein for directly observing wavelength changes provide advantages in simplifying wavelength control systems that would normally require stabilization of both temperature and current. .

Some types of semiconductor lasers, such as Fabry-Perot (FP) lasers, project a single spatial mode, but can also support multiple longitudinal modes, where each longitudinal mode has a slightly different frequency. Have In some cases, the current can be adjusted to switch between operation in a single longitudinal mode and operations in multiple longitudinal modes. The number of longitudinal modes of the laser can be an operating parameter of the projector 121. Usually, one single wavelength is desired because it creates a single well-defined pattern X. If two wavelengths are created by the laser, the second wavelength creates some kind of shadow in the desired pattern X. This result can be described with reference to FIG. Depending on the difference in wavelength of the plurality of longitudinal modes, the circular point may be an ellipse or barbell shape or a pair of points X ₁ and X ₂ . These shifts can be detected by applying image processing to the received pattern X. If multiple modes are observed, the pump current or cooling can be adjusted to reduce the number of longitudinal modes to one.

  Intrinsic parameters of cameras 111 and 112, such as deviations in focal length, can be automatically recognized and compensated in some cases. In other examples, intrinsic parameter deviations may occur due to intentional actions by the operator or through operation of the 3D measurement device 100 rather than from thermal or mechanical causes. These actions may include, for example, by changing the optical properties of the cameras 111, 112, or by focusing or zooming the cameras 111, 112.

A further possibility to verify the calibration result is to combine 3D scan data based on triangulation using 2D images captured by the 2D camera 113. These 2D images typically comprise the color of the object O and other features. In one embodiment, the projector 121 and the two cameras 111 and 112 are synchronized with the camera 113 to capture the frame of the object O synchronously. In this embodiment, the projector 121 projects the infrared light received by the cameras 111, 112 through an infrared filter, while the camera 113 passes the object O through a filter that blocks the infrared light wavelength and thus the pattern X. The image of is received. Thus, the camera 113 receives a clear image of the feature of the object O. By comparing the object features observed by the cameras 111, 112 with those observed by the camera 113, errors in the calibration parameters of the triangulation system including the elements 111, 112 and 121 can be determined. The method used to determine the error in the calibration parameters is similar to that described here, such as the 2D camera 113 that is considered _one of the units U ₁ , U ₂ . Correspondence is detected in the captured frame and the 3D coordinates of the selected object features are determined. Due to the redundancy in determining the 3D coordinates, new calibration parameters can be determined and compared to the calibration parameters currently used. If the difference between the current calibration parameter and the new calibration parameter exceeds the threshold, the new calibration parameter replaces the one currently used.

  In another embodiment, the light received by camera 113 is not filtered to exclude infrared light. Instead, in this embodiment, the camera 113 can detect and acquire an image of the pattern X when it is on the object O. The camera 113 can then operate in either of two modes. In the first mode, the camera 113 collects images in the time interval of the pulse-like projection of the pattern X by the projector 121. In this mode, camera 113 captures an image of features that can be compared to features calculated by a triangulation system including 111, 112, and 121 to determine errors in calibration parameters. In the second mode, the camera 113 acquires an image when the pattern X is projected on the object O. In this case, an error in the relative alignment of the six degrees of freedom of the camera 113 relative to the triangulation elements 111, 112, and 121 can be determined.

  In the case where the image acquired by the 2D camera 113 is received between the image frames of the cameras 111, 112, 3D and 2D coordinate data can be interpolated to provide a basis for image interleaving. Such interpolation may be mathematical. In one embodiment, the interpolation may be based at least in part on a Kalman filter calculation. In some embodiments, the 3D measurement device 100 is provided with an inertial measurement unit (or another position tracking device). The inertial measurement unit measures the acceleration in three directions of space using three sensors (accelerometer) and the angular velocity about three coordinate axes using three additional sensors (gyroscope). Movement can thus be measured on a short time scale. These movement measurements allow time-related compensation cancellation between the distances measured by the projector and cameras 111 and 112 and the image acquired by the 2D camera 113. This compensatory cancellation allows data combinations.

  One embodiment of the display 130 shown in FIG. 7 illustrates a subdivided image or a subdivided screen. In this embodiment, the display 130 is divided into a first display unit 130a and a second display unit 130b. In the present embodiment, the first display unit 130a is a central part of the (rectangular) display 130, and the second display unit 130b is a peripheral region around the first display unit 130a. In another embodiment, the two display units may be in a vertical column. In the illustrated embodiment, the first display portion 130a is shown as having a rectangular shape, but this is for illustrative purposes and the claimed invention should not be limited to this. In other embodiments, the first display unit 130a may be, but is not limited to, a circle, a square, a trapezoid, an unequal quadrilateral, a parallelogram, an oval, a triangle, or any number of sides having any number of sides. It can have other shapes, including squares. In one embodiment, the shape of the first display unit 130a is user-defined or selectable.

  On the first display unit 130a, a video live image VL, for example, one captured by the 2D camera 113 is displayed. In the second display unit 130b, an image of the latest 3D scan (or a plurality of superimposed 3D scans) is displayed as at least a partial view of the 3D point cloud 3DP. The size of the first display unit 130 a is variable, and the second display unit 130 b is disposed in a region between the first display unit 130 a and the boundary 131 of the display 130. As the video live image VL changes, such as when the user moves the device 100, the image of the 3D point cloud 3DP changes accordingly to reflect changes in the position and orientation of the device 100.

  By placing an image of the 3D point cloud 3DP around the periphery of the video live image VL, it is easy for the user to see where additional scanning may be required without taking his eyes off the display 130 It should be appreciated that an advantage is provided in enabling. In addition, it may be desirable for the user to determine if the computer alignment of the current camera position with respect to already acquired 3D data is within the desired specifications. If this alignment is out of specification, you will notice this as a discontinuity at the boundary between the image and the 3D point cloud 3DP.

  The image captured by the camera 113 is a two-dimensional (2D) image of the scene. A 2D image that can be changed to a three-dimensional view usually includes a pincushion-shaped or barrel-shaped distortion, depending on the type of optical lens used in the camera. In general, when the field of view (FOV) of the camera 113 is narrow (for example, about 40 degrees), the distortion is not easily seen by the user. Similarly, an image of 3D point cloud data may appear distorted depending on how the image is processed with respect to the display. The point cloud data 3DP can be viewed as a plan view. In this case, an image is obtained in the original coordinate system (for example, a spherical coordinate system) of the scanner and mapped on the plane. When viewed in plan, the straight line appears curved. Further, images near the central top edge and the central bottom edge (eg, poles) may be distorted with respect to a line (eg, equator) that extends along the midpoint of the image. There may also be distortion created by trying to represent the sphere surface on a rectangular grid (similar to the Mercator projection problem).

  It is appreciated that in order to provide a continuous and seamless image experience to the user, it is desirable to have the image in the first display unit 130a look similar to the image in the second display unit 130b. I want to be. If the image of the 3D point cloud 3DP is significantly distorted, this can make it difficult for the user to determine which regions can use additional scans. Since the planar image of the point cloud data 3DP may be distorted with respect to the 2D camera image, one or more processing steps may be performed on the image generated from the point cloud data 3DP. In one embodiment, the field of view (FOV) of the second display unit 130b is limited so that only the central portion of the planar image is shown. In other words, the image is truncated or cropped to remove the highly distorted portion of the image. When the FOV is small (for example, less than 120 degrees), the distortion is limited, and the point cloud data 3DP viewed in plan will look as desired by the user. In one embodiment, the planar view is processed to scale and shift the planar image to be provided to match the image of the camera 113 in the first display unit 130a.

  In another embodiment, the 3D point cloud data 3DP is processed to generate a panoramic image. As used herein, the term panoramic refers to a display that is capable of angular movement about a point in space (generally the user's location). The panoramic view does not cause distortion at the pole as in the plan view. The panoramic view may be a spherical panorama including 360 degrees in the azimuth direction and +/− 45 degrees in the zenith direction. In one embodiment, the spherical panoramic view may be only a portion of the sphere.

  In another embodiment, the point cloud data 3DP can be processed to generate a 3D display. 3D display refers to a display that is arranged to allow not only rotation about a fixed point, but also translational movement from point to point in space. This provides an advantage for the user to move around in the environment and provides a continuous and seamless display between the first display part 130a and the second display part 130b.

  In one embodiment, the video live image VL in the first display unit 130a and the image of the 3D point cloud 3DP in the second display unit 130b are continuously one (with respect to the displayed content). Adapted to. A portion of the 3D point cloud 3DP is initially selected (by the control and evaluation device 122) in such a way that it is viewed from the viewpoint of the 2D camera 113 or from a position aligned with at least the 2D camera 113. The selected part of the 3D point cloud 3DP is then selected in such a way that it is continuously joined to the video live image VL. In other words, the displayed image of the 3D point cloud 3DP is a series of video live images VL with respect to the region beyond the view of the 2D camera 113 on the left side, the right side, the top side, and the bottom side with respect to the view of the 2D camera. Become a body. As discussed above, selected portions of the three-dimensional point cloud 3DP can be processed to reduce or eliminate distortion. In other embodiments, this representation may correspond to a fisheye lens representation, but it is preferably undistorted. A part of the three-dimensional point cloud 3DP positioned in the area occupied by the first display unit 130a, in other words, a part under or hidden by the video live image VL is not displayed.

  It should be appreciated that the density of points in the three-dimensional point cloud 3DP in the region where the first display unit 130a is located is not visible to the user. Normally, the video live image VL is displayed using natural coloring. However, the coloring of the video live image VL can be artificially changed, for example by an overlay, in order to show the density of points in the area covered by or behind the video live image VL. In this embodiment, the artificial color (and its intensity, where appropriate) used to represent the artificially colored video live image VL corresponds to the density of the points. For example, a green coloring for a video live image VL can indicate a (sufficiently) high density, while indicating a medium or low point density (eg an area where scan data can still be improved) Yellow coloring can be used. In another embodiment, this color coding can be used to display the accuracy of data points that are distance dependent.

  A flag or mark 133 (FIG. 7) is shown in the first display portion 130a to indicate the structure (ie, possible target) recognized by the control and evaluation device 122 to assist in the overlay of 3D scans. Can be inserted. The mark 133 may be a symbol such as a small “x” or “+”. The recognizable structure can be a point, corner, edge, or texture of the object. A recognizable structure can be found by the most recent 3D scan or video live image VL being applied to the beginning of the overlay process (i.e. for target localization). The use of the most recent video live image VL provides an advantage in that the overlay process does not need to be performed as often. If the mark 133 has a high density, it is considered that the 3D scan overlay was successful. However, if a lower density of the mark 133 is recognized, an additional 3D scan may be performed using a relatively slow movement of the 3D measurement device 100. By slowing the movement of the distance 100 during the scan, additional or higher density points can be acquired. Correspondingly, the density of the mark 133 may be used as a quality measure for the success of the overlay. Similarly, the point density of the 3D point cloud 3DP may be used to indicate the success of the scan. As discussed above, the density of points in the scan may be represented by artificial coloring of the video live image VL.

  The movement of the 3D measurement device 100 and the processing of the captured frames can also be performed by a tracking function, i.e. the 3D measurement device 100 tracks the relative movement of its environment in the manner used during tracking. If the tracking is lost, for example if the 3D measuring device 100 moves too fast, the tracking can simply be performed again. In this embodiment, the raw video live image VL provided by the 2D camera 113 and the latest video still image from the tracking provided thereby are arranged side by side on the display 130 for the user. It can be expressed adjacent to each other. The user can then move the 3D measurement device 100 until the two video images match.

  Referring now to FIGS. 15-19, an embodiment for controlling the 3D measurement device 100 based on the movement or operation of the device 100 will be described. These movements or actions by the user can also be used to control the representation of the video image VL or the 3D point cloud 3DP. In certain embodiments, these steps or operations may be incorporated into a method of measurement, such as the subsequent block 203 described with reference to FIG. In general, three steps are provided: a movement recognition step 311, a movement evaluation step 312 and a control step 314.

  In the movement recognition step 311, the movement of the 3D measurement device 100 is recorded, for example, during the recognition performed in the process block 203. The movement evaluation step 312 determines the amount and direction of speed or acceleration of movement. This determination can incorporate the temporal aspects of movement, as well as the quality and quantity of movement. In the quality assessment, the movement can be compared to a predetermined movement stored in memory. If the predetermined threshold is exceeded (or not exceeded), the defined movement is identified. In the control step 313, the operation of the 3D measurement device 100 is changed based on the evaluated movement. All three steps 311, 312, 313 are either controlled by the control and evaluation device 122 or via an external computer coupled to the control and evaluation device 122 to communicate (eg, via a serial connection or network connection). Please appreciate what you can do either.

  Control of the 3D measurement device 100 includes at least three categories of control: measurement (eg, a measurement process involving acquisition of measured data), visualization (eg, preliminary evaluation of the measured data and its representation), and post processing ( For example the final evaluation of the measured data). It should be appreciated that the 3D measurement device 100 may use operations to control one or a combination of categories. Furthermore, the control function performed according to the operation may be context dependent. For example, during post processing, the control step 313 may only provide a storage of the result of the movement evaluation step 312 otherwise the storage of this result will be performed after the end of the measurement process. The three control categories include, but are not limited to, a 3D measurement device 100, a control and evaluation device 122, a display 130, or a computing device connected locally or remotely (such as via a serial or network connection). Can interact with any of the components, subcomponents, or auxiliary components.

  Motion and movement assessments can be subdivided into three groups as represented in FIGS. Although the embodiments herein describe the movements as being individual movements, it should be appreciated that these movements may be combined or performed sequentially. In one embodiment, the user interface can be selected based on movement or movement, such as selecting a configuration of the control knob 106.

Referring now to FIG. 16, a first group of movements or movements is illustrated. In this mode of operation, 3D measurement device 100 is moved in the direction _{V 1.} Motion V ₁ was be measured both in terms of speed and acceleration. Depending on the direction of movement, speed, or acceleration, various control functions change, for example, the measurement being acquired or the visualization of data on the display 130. When the measurement function is controlled, a series of images of one or more cameras 111, 112, 113 is used with a low dynamic range, but with exposure of the cameras 111, 112, 113 using the stationary movement of the 3D measurement device 100. You can capture at different times. In another embodiment, various illumination intensities can be projected from the light source 121a or the LED 114 by a stationary operation. In yet another embodiment, stationary motion can result in various exposure times (pulse lengths) from the light source 121a during the sequence, and images can be acquired using high dynamic range (HDR) techniques. it can. As used herein, the term “stationary” means to stop moving the 3D measurement device 100, in other words, to have zero velocity in either direction. In certain embodiments, the term “stationary” includes when the user reduces the speed of the 3D measurement device 100 below a threshold.

  In one embodiment, the scale of the representation of the video image VL on the display 130 and / or the 3D point cloud 3DP may depend on the speed and / or acceleration of the movement of the 3D measurement device 100. The term “scale” is defined as the ratio between the size of the first display portion 130a (either linear dimension or area) and the size of the display 130 as a whole and is shown as a percentage.

  The small field of view of the 2D camera 113 is assigned to a small scale. In this embodiment with a subdivided display 130 where the central first display portion 130a shows a video live image VL, this first display portion 130a is therefore of a smaller size than in the standard case. There may also be a second display part 130b (around the periphery of the first display part 130a) showing a larger part of the three-dimensional point cloud 3DP. A larger field of view is assigned to a larger scale. In one embodiment, the video live image VL can fill the entire display 130.

  When high-speed movement of the 3D measurement device 100 is detected, the scale of the expression may be configured smaller than the case of low speed, and the reverse relationship also holds. Similarly, this may be true for the acceleration of movement of the 3D measurement device 100. For example, the scale of the displayed image is reduced for positive acceleration, and the scale is increased for negative acceleration. The scale also depends on the speed and / or acceleration components of the movement of the 3D measurement device 100, for example, components that are arranged perpendicular or parallel to the alignment relationship of the 3D measurement device 100. If the scale is determined based on the component of movement parallel to the alignment relationship (ie in the direction of the alignment relationship), this scale can also depend on the change in the average distance from the 3D measurement device 100 to the object O. .

  In some embodiments, a change in scale due to movement, a stop of movement of the 3D measurement device 100, or a threshold speed value of movement that has not been achieved, using a series of stationary motions of the camera 113 having a low dynamic range. Images can be recorded. These images can be captured within this series of images with a low dynamic range but with different exposure times or illumination intensities to produce a high dynamic range image therefrom.

Referring now to FIG. 17, a second group of mobile includes operation or movement as indicated by the arrow V _2. In this embodiment, the user uses the 3D measurement device 100 to perform certain small movements, such as circular movements. This circular movement may be at least in the region of the space having a radius of the length of the user's arm. FIG. 17 illustrates movement V ₂ as centered on an axis extending orthogonal to the front of the 3D measurement device 100, but this is for illustration purposes and the claimed invention is Please understand that it should not be limited to. In other embodiments, the central axis on which the rotation takes place can be different, such as an axis extending from the side of the device 100 or extending vertically through the center of the device 100. Evaluation of operation V ₂ by comparing the type of movement stored its movement, and by identifying an operation V ₂ Thus, with respect to preferably quality is performed.

By operation V ₂ alone or in combination with operation V ₁ , the user can enable all three categories of control, such as data acquisition and operation of 3D measurement device 100 or display 130. The user can switch between the individual states or can adjust the values step by step or continuously.

In one embodiment, operation V ₂ can be used (alone or in combination with operation V ₁ ) to control the functionality of the 3D measurement device 100. These control functions include, but are not limited to, selecting a measurement method (eg, between photogrammetry, pattern detection and fringe projection combined), acquisition characteristics (eg, projector 121 output). , Exposure time, object distance, pattern or point density) and switching on and off additional measurement characteristics (eg, 3D recognition or flashing of contours).

In certain embodiments, in order to control the visualization of the display 130 or an external computing device, it is possible to use an operation V ₂ (alone or in combination with operation V _1). These visualization functions include, but are not limited to, selecting a user interface (eg, button placement, selecting between 2D and 3D view being scanned), selecting a representation (eg, dot Size, field of view, type of projection such as orthographic or perspective, or map view), selecting a guidance method (eg, object or camera center, whether zooming is allowed), 3D measurement device 100, Use as a mouse with 6 degrees of freedom (x direction, y direction, z direction, yaw angle, roll angle, and pitch angle) to guide through a common coordinate system with a 3D point cloud 3DP, and display Interact with the representation on 130 and the further user interface in the manner of a mouse (e.g. Including moving the contents of the display 130 is shifted aside).

In certain embodiments, in order to control the post-processing function of the display 130 or an external computing device, it is possible to use an operation V ₂ (alone or in combination with operation V _1). These post-processing functions include, but are not limited to, setting a mark (marking a point of interest by surrounding it, etc.), a point of view or a point of rotation Defining a location for guiding through a common coordinate system having a 3D point cloud 3DP (eg, by an action with a change in pitch angle corresponding to a mouse key click), And defining (approximate) gravity direction (z direction).

  In some embodiments, the direction of gravity can be defined at the beginning of the overlay process by a defined movement of the 3D measurement device 100. This defined movement is performed by the user moving the device 100 in a vertical upward and downward movement over a predetermined distance, for example 20 cm. In other embodiments, the direction of gravity can be determined from a set of statistics for all movements during the overlay process. A plane can be obtained by averaging the coordinates of the position taken by the device 100 along the path of movement through space during the recording process. The plane obtained by this averaging is assumed to be positioned horizontally in space, which means that the direction of gravity is perpendicular to this. As a result, the use of inclinometer 119 to determine the direction of gravity can be avoided.

Referring now to FIG. 18, the third group of movements includes actions or movements as indicated by arrow V ₃ . Motion V _3, it should be appreciated that a path through the space. This group of actions can be evaluated by determining the coordinates of the position of the 3D measurement device 100, such as during a measurement process. Using the evaluation of the path of motion V _3, when determining the type of scene, and appropriate, it is possible to provide various representations or operating functions. In one embodiment, the path characteristics of motion V ₃ and the alignment characteristics of 3D measurement device 100 are evaluated and compared to a predetermined motion. Well this evaluation is based on statistical, in some embodiments, appreciated that this comparison, which of the predetermined motion, or has most likely corresponds to the path of motion V ₃ is determined I want to be. In one embodiment, function selected from the evaluation of the motion V ₃ is based on likely than the path of movement V ₃ a threshold. In certain embodiments, the middle of the movement between the two stationary operation, by a change in position of the 3D measuring device 100 in the space, it is possible to distinguish an operation V _2.

In one embodiment, the path of motion V ₃ can also be used to determine the type of scene and to provide various representations or operational possibilities where appropriate. The path of movement around the central location (where the alignment relationship of the 3D measurement device 100 is oriented inward) suggests an image of a single object O (image centered on the object). Similarly, the path of movement that directs the device 100 outwardly (and particularly the longer straight section of this path of movement) is associated with the room image. Therefore, when it is determined that the room is being scanned, an image of a floor plan view (top view) can be inserted into the display 130.

In one embodiment, operation V ₃ can be used (alone or in combination with operations V ₁ and V ₂ ) to control the measurement function of the 3D measurement device 100. These functions include, but are not limited to, selecting a tracking method (as shown for V ₂ ) and tracking or based on, for example, the distance of the captured object to the path of motion V ₃ Including selecting data for superposition.

In one embodiment, operation V ₃ (alone or in combination with operations V ₁ and V ₂ ) can be used to control the visualization function of the 3D measurement device 100. These functions include, but are not limited to, selecting a representation based on measurements, particularly with respect to zoom and orientation, defining the volume of interest with respect to the background, and a floor plan view (top view) as a map on display 130. ), Defining an empty wall within the volume of interest (eg, when no floor plan is present), and selecting a user interface.

In one embodiment, operation V ₃ can be used (alone or in combination with operations V ₁ and V ₂ ) to control the post-processing function of the 3D measurement device 100. These functions include, but are not limited to, detecting the volume of interest, selecting between features of the volume of interest, and choosing between various post-processing options (eg, , Filling gaps on the object surface within the detected volume of interest (video meshing, gap filling) and supporting and joining interpolation, motion Detecting the position of the floor from the z-coordinate of the path of V ₃ (eg defining that these z-coordinates are located on the horizontal plane on average and limiting guidance to this plane), movement detecting a change in the bed from the z coordinate of the path of V _3, to separate the bed when a change in the bed is detected, and, (e.g. 3D measurement devices Includes to determine a) the absolute position of the floor when including advanced instrument.

It should be appreciated that the operations may be combined to assist the user in controlling the functionality of the 3D measurement device 100. For example, certain user interface elements (eg, buttons, sliders, text boxes) may appear on the display 130 by evaluating the speed of movement V ₁ . Then these user interface elements is activated by movement of the operating V _2.

  In one embodiment, corresponding rules can be defined that correlate operations with control functions, such as the operations and control functions described herein. Corresponding rules may be any suitable means for correlating data, such as a look-up table or a database. In this exemplary embodiment, the corresponding rules are stored in memory and can be obtained by a control and evaluation device to determine motion based on, for example, speed, acceleration, motion path, motion direction, or motion type. It is.

  Although the invention has been described in detail in connection with only a limited number of embodiments, it will be readily understood that the invention is not limited to such disclosed embodiments. Rather, the invention can be modified to incorporate any number of alterations, modifications, substitutions or equivalent arrangements not heretofore described, but which are commensurate with the spirit and scope of the invention. In addition, while various embodiments of the invention have been described, it is to be understood that aspects of the invention may include only some of the described embodiments. Accordingly, the invention is not to be seen as limited by the foregoing description, but is only limited by the scope of the appended claims.

Claims (23)

A three-dimensional (3D) measurement device having a first camera, a second camera, and a projector, further comprising a control and evaluation device having a memory and operably coupled to the at least one camera and the projector Providing a 3D measurement device comprising:
Providing a corresponding rule between each of the plurality of commands and each of the plurality of actions, wherein each action from the plurality of actions includes movement along a path by the 3D measurement device; Providing the correspondence rule is based at least in part on movement along a first path, and wherein the correspondence rule is stored in the memory;
Projecting a pattern of light on an object with the projector; recording a first set of images of the light pattern with the first camera and the second camera at a first time;
Recording a second set of images of the light pattern with the first camera and the second camera at a second time;
Creating a 3D scan of the object based at least in part on the first image and the second image;
Moving the 3D measurement device from a first position to a second position along a second path;
Determining an action based at least in part on the movement from the first position to the second position;
Determining a first control function based at least in part on the actions and the corresponding rules;
Performing the first control function on the 3D measuring device;
A method for optically scanning and measuring an environment, comprising:   The method of claim 1, wherein the step of determining the action is further based at least in part on the first image and the second image.   The method of claim 2, wherein the step of determining the action is further based on the 3D scan. The method of claim 1, comprising:
The 3D measurement device further comprises an inclinometer;
The method of determining the action is further based at least in part on a signal from the inclinometer.   5. A method according to claim 4, wherein the inclinometer is an accelerometer.   The method of claim 1, wherein the speed of the 3D measurement device is determined when the 3D measurement device moves from the first position to the second position along the second path. And further comprising:   7. The method of claim 6, wherein the step of determining the movement is further based at least in part on the speed.   8. The method of claim 7, wherein an acceleration of the 3D measurement device is determined when the 3D measurement device moves from the first position to the second position along the second path. And further comprising:   9. The method of claim 8, wherein the step of determining the motion is further based at least in part on the acceleration.   The method of claim 1, wherein the 3D measurement device further comprises a display, and the first control function changes a scale of a third image displayed on the display. .   11. The method according to claim 10, wherein the scale of the third image is increased when the second position is further from the user than the first position.   12. The method according to claim 11, wherein the scale of the third image is reduced when the second position is closer than the first position.   11. The method of claim 10, further comprising measuring the acceleration of the 3D measurement device along the second path, wherein the magnitude of the change in the scale of the third image is: A method characterized in that it is based at least in part on the acceleration.   12. The method of claim 11, wherein the step of performing the first control function includes selecting an element on a user interface displayed on the display.   The method of claim 1, further comprising stationary the 3D measurement device and recording a third set of images with the first camera and the second camera. A method wherein 3 sets are recorded with a low dynamic range.   The method of claim 15, further comprising generating an image having a high dynamic range based on the third set of images.   2. The method of claim 1, wherein the second path includes movement along a radius of a user's arm length.   The method according to claim 1, wherein a measurement method or a measurement characteristic is selected by the execution of the first control function.   The method according to claim 1, wherein the execution of the first control function changes a user interface of the 3D measurement device.   2. The method according to claim 1, wherein a target volume is defined by the execution of the first control function.   21. The method of claim 20, wherein the execution of the first control function further comprises marking a point of interest on a display operably coupled to the 3D measurement device. And how to.   The method of claim 1, wherein the step of determining the action includes determining in the corresponding rule the likelihood that the second path corresponds to the first control function. Feature method. The method of claim 1, comprising:
Identifying an object in the first set of images;
Identifying the object in the second set of images;
Aligning the 3D measurement device to a central location based at least in part on the position of the object.

Images