JP2015137897A - Distance measuring device and distance measuring method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology for accurately measuring a three-dimensional distance to an object even in the place where the surface shape of the object is largely changed.SOLUTION: A distance measuring device estimates a projection position of a pattern on a model representing the three-dimensional shape of a target object when the model is arranged in an approximate position attitude, determines a coordinate position on an image corresponding to the projection position and searches for a pattern from the vicinity of the determined coordinate position, and performs calculation processing based on triangulation by using the correspondence between the pattern searched from the image and a pattern projected by a projection unit so as to calculate the distance from an imaging unit to the surface of the target object.

Description

  The present invention relates to a distance measurement technique.

  Various methods have been proposed as a three-dimensional shape measurement method, but an active type using a combination of a projection device and a photographing device is widely used for high-precision measurement with high robustness. As an active feature, even when there is little surface texture information to be measured, shape measurement can be performed using the projected pattern as a clue. Representative examples of the active type include a method based on the position of the projection pattern, a method based on phase information change, and a method based on the defocus information of the projection pattern.

  Among these, a light cutting method in which a plurality of slit patterns is projected in measurement corresponding to a dynamic scene is known. In general, the light cutting method that projects a plurality of slit patterns projects a slit pattern having the same shape. For this reason, unless the measurement distance range is limited, it is difficult to uniquely associate the imaged slit with the projected slit. Therefore, in the method described in Patent Document 1, a plurality of slits encoded by periodic changes in the width of the slits are projected onto the measurement target. Then, a code is extracted from the imaged slit, compared with the code of the projected slit, and based on the comparison result, the imaged slit and the projected slit are associated, and based on the associated result 3D measurement is performed by the principle of triangulation.

  There has been proposed a method for estimating the position and orientation of a measurement target object using the distance measurement point group obtained as described above and a known shape model. When the shape model of the measurement target object surface is known and multiple distances from the imaging device to the measurement target object surface are obtained as point clouds, the difference in distance from the point group to the nearest point is reduced. A method for estimating the position and orientation is disclosed in Non-Patent Document 1. This method is widely used as an ICP (Iterative Closest Point) method as a method for distance measurement and shape model alignment. Similarly, the method described in Non-Patent Document 2 calculates the distance between the measurement point group and the surface of the shape model, and the measurement target is set to minimize the error between the shape model of the measurement target object and the measurement point group. The position and orientation of the object can be estimated.

Special table 2009-517634

P.J.Besl and N.D. Mckay. “A method forregistration of 3-D shapes,” IEEE Trans. Pattern Analysis and Machine Intelligence, 14 (2), pp.239-256, 1992. Y. Chen and G. Medioni, “Object modeling byregistration of multiple range images,” Robotics and Automation, 1991. Proceedings., 1991 IEEE International Conference on (1991), vol.3, pp. 2724-2729, 1991.

  In the above three-dimensional shape measurement method, when the surface shape of the measurement target object has changed greatly, there is a possibility that the slit pattern cannot be dealt with and the three-dimensional shape measurement may fail.

  The present invention has been made in view of such problems, and provides a technique for accurately measuring a three-dimensional distance to an object even at a location where the surface shape of the object has changed greatly.

  According to one aspect of the present invention, a distance measuring device connected to a projection unit that projects a single pattern having a plurality of slits, and an imaging unit that captures an image of a target object on which the pattern is projected. The projection position of the pattern on the model is estimated when the approximate position and orientation of the target object and the model representing the three-dimensional shape of the target object are arranged in the approximate position and orientation. An estimation means for obtaining a coordinate position on the image corresponding to the projection position, a search means for searching for the pattern from the vicinity of the obtained coordinate position, a pattern searched for from the image, and the projection unit And a calculation means for calculating a distance from the imaging unit to the surface of the target object by performing a calculation process based on triangulation using a correspondence relationship with the pattern to be projected. To.

  With the configuration of the present invention, it is possible to accurately measure the three-dimensional distance to an object even at a location where the surface shape of the object has changed greatly.

The block diagram which shows the function structural example of a system. The figure which shows an example of a pattern image. 10 is a flowchart of processing performed to estimate the position and orientation of the target object 4. The figure explaining the process which the projection position calculating part 35 performs. The figure explaining the calculation principle of the shape information of the target object 4. FIG. The flowchart which shows an example of the process which the position and orientation calculating part 37 performs. The figure which shows an example of a pattern image. The block diagram which shows the function structural example of a system. 10 is a flowchart of processing performed to estimate the position and orientation of the target object 4. The figure which shows an example of a code sequence. The figure explaining a code string. 10 is a flowchart of processing performed to estimate the position and orientation of the target object 4.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
The three-dimensional measurement apparatus (distance measurement apparatus) according to the first embodiment estimates the projection position of a pattern (single pattern having a plurality of slits) on the model from the three-dimensional shape model and its approximate position and orientation. The coordinate position on the captured image corresponding to the projection position is obtained, the slit pattern is associated, and the distance is calculated based on the principle of triangulation. As a result, the measurement object that cannot be associated with the pattern can be associated with each other by using the information about the three-dimensional model and the approximate position and orientation, and three-dimensional measurement can be performed.

  First, the configuration of the system according to the present embodiment will be described with reference to the block diagram of FIG. As shown in FIG. 1, the system according to the present embodiment shoots a projection unit 1 that projects a pattern (light) based on a pattern image onto a target object (measurement target) 4 and the target object 4 on which the pattern is projected. The imaging unit 2 and the three-dimensional measuring device 3 that calculates the position and orientation of the target object 4 are provided.

  First, the projection unit 1 will be described. The projection unit 1 includes a light source 11, an illumination optical system 12, a display element 13, and a projection optical system 14. The light source 11 is various light emitting elements such as a halogen lamp and an LED. The illumination optical system 12 is an optical system having a function of guiding light emitted from the light source 11 to the display element 13. The display element 13 has a function of spatially controlling the transmittance or the reflectance when the light guided from the illumination optical system 12 is further guided to the projection optical system 14. The projection optical system 14 is an optical system configured to form an image of light guided from the display element 13 at a specific position of the target object 4.

  Next, the photographing unit 2 will be described. The photographing unit 2 includes a photographing lens 21 and a photographing element 22. The photographing lens 21 is an optical system configured to form an image of a specific position of the target object 4 on the photographing element 22. The imaging element 22 is various photoelectric conversion elements such as a CMOS sensor and a CCD sensor. Further, the position and orientation of the photographing unit 2 are adjusted so that the vertical direction in the image coordinate system on the photographing element 22 and the straight line formed by the two optical axis center positions of the projecting unit 1 and the photographing unit 2 are parallel. ing.

  Next, the three-dimensional measuring device 3 will be described.

  The shape model holding unit 31 acquires and holds a model (three-dimensional shape model) representing the three-dimensional shape of the target object 4. For example, the shape model holding unit 31 downloads and holds a three-dimensional shape model stored in an external server device from the server device. Further, the shape model holding unit 31 may acquire and hold a three-dimensional shape model input by a user operating an interface (not shown) such as a keyboard or a mouse. In addition, the shape model holding unit 31 displays a 3D shape model candidate on a display screen (not shown), and acquires a candidate selected by the user from the displayed candidates from a memory in the apparatus or an external apparatus and holds the candidate. You may make it. Further, component recognition for selecting a three-dimensional shape model may be used.

  As described above, the acquisition method of the 3D shape model by the shape model holding unit 31 is not limited to a specific acquisition method, and any acquisition method can be adopted as long as the apparatus can acquire the 3D shape model when necessary. It doesn't matter.

  The 3D shape model can be represented by any model that can calculate the 3D coordinate position of the surface of the target object 4 from a viewpoint having an arbitrary 3D coordinate position. It is not limited to.

  For example, the following model can be used as a model that can calculate the three-dimensional coordinate position of the surface of the three-dimensional shape model from a viewpoint having an arbitrary three-dimensional coordinate position. Of course, the three-dimensional shape model applicable to the present embodiment is not limited to the following model.

・ Patch model composed of vertices with 3D coordinate positions and planes composed of line segments connecting vertices ・ Function expressions describing the surface shape of objects and curved surface functions combining the coefficients A model that holds the value of the area occupied by the object among the voxels in the space. The approximate position and orientation holding unit 32 is a reference coordinate system (one specified point in the measurement space) in the space (measurement space) in which the target object 4 exists. Is the origin, and the approximate position and orientation of the target object 4 in the coordinate system in which the three axes orthogonal to each other at the origin are the x-axis, y-axis, and z-axis are held. The range in which the target object 4 exists in the measurement space can be determined by setting the position and orientation as a range for parts and the like that are supplied while maintaining physical contact by the parts feeder. It is also possible to estimate the position and orientation within the range of the result of collation with an image registered in advance by another component recognition device. Here, it is desirable that the “approximate position and orientation” (approximate position and orientation) is an expression with six degrees of freedom that determines the position and orientation in a three-dimensional space (measurement space).

  Various methods are conceivable as a method of inputting the approximate position and orientation to the apparatus, and the method is not limited to a specific method. For example, it is only necessary to input numerical values for the position and orientation parameters, and signals may be input by connecting to other sensors. Further, when the projection unit 1 and the imaging unit 2 are mounted on the tip of the robot arm, an offset between the position and orientation of the hand and the mounting position by forward kinematics using the encoder values of each joint held by the robot arm. It is also possible to use the position and orientation obtained by multiplying. Here, the approximate position and orientation of the target object 4 in the current three-dimensional space may be acquired. If the constraint between the target object 4 and the measurement space has already been defined, it can be converted into a representation of the position and orientation in the reference coordinate system using this.

  The projection pattern image generation unit 33 generates a pattern (pattern image) to be projected onto the target object 4. Since the present embodiment is intended for three-dimensional measurement in a dynamic scene, an image having a plurality of slit patterns as shown in FIG. 2 is generated. Then, the projection pattern image generation unit 33 outputs the generated pattern image to the projection unit 1 via a general display interface such as DVI. The projection unit 1 projects a pattern based on the pattern image output from the projection pattern image generation unit 33 onto the target object 4.

  The captured image acquisition unit 34 captures the digital image signal sampled and quantized by the capturing unit 2, and captures an image in which each pixel is represented by a luminance value (density value) from the captured digital image signal. Get as. The captured image acquisition unit 34 controls the operation of the imaging unit 2 (imaging timing, etc.) via a general-purpose communication interface such as RS232C or IEEE488.

  When the three-dimensional shape model held in the shape model holding unit 31 is arranged in the approximate position and orientation held by the approximate position and orientation holding unit 32, the projection position calculation unit 35 generates a pattern on the three-dimensional shape model. Estimate the projection position. Then, the projection position calculation unit 35 obtains a position on the photographing surface of the photographing unit 2 corresponding to the estimated position (a position projected on the photographing surface of the photographing unit 2) as a projected image coordinate position. More specifically, the projection position calculation unit 35 calculates a coordinate position on the pattern image generated by the projection pattern image generation unit 33 and a position (projected image coordinate position) projected on the captured image by the imaging unit 2. The relationship is calculated using the three-dimensional shape model and the approximate position and orientation. The operation of the projection position calculator 35 will be described in detail below.

  The distance calculation unit 36 searches for a pattern from the vicinity (neighborhood) of the “projection image coordinate position estimated by the projection position calculation unit 35” in the captured image of the target object 4 by the imaging unit 2, and the searched pattern and The pattern image is associated with the pattern on the pattern image. Then, the distance calculation unit 36 uses the result of this association, and based on the principle of triangulation, distance information indicating the distance between the photographing unit 2 and each point on the surface of the target object 4, that is, the target object 4 The shape information of is calculated.

  Note that the projection position calculation unit 35 can use only a pattern having a projection-projection relationship with the projection of the three-dimensional shape model, so that it does not include the result of distance measurement for other objects and background regions. it can.

  Next, processing performed to estimate the position and orientation of the target object 4 will be described with reference to FIG. 3 showing a flowchart of the processing.

  In step S <b> 301, the shape model holding unit 31 acquires and holds a three-dimensional shape model that is a model representing the three-dimensional shape of the target object 4. In step S302, the approximate position and orientation holding unit 32 acquires and holds the approximate position and orientation of the target object 4 in the reference coordinate system.

  In step S <b> 303, the projection pattern image generation unit 33 generates a pattern (pattern image) to be projected onto the target object 4 and outputs it to the projection unit 1. In step S <b> 304, the projection unit 1 projects a pattern based on the pattern image output from the projection pattern image generation unit 33 onto the target object 4.

  In step S305, the photographed image acquisition unit 34 controls the photographing unit 2 to photograph the target object 4 on which the pattern is projected. Then, the captured image acquisition unit 34 captures the digital image signal from the capturing unit 2 and generates a captured image from the captured digital image signal. Of course, the captured object includes the target object 4 on which the pattern is projected.

  In step S <b> 306, when the three-dimensional shape model held by the shape model holding unit 31 is arranged in the approximate position and orientation held by the approximate position and orientation holding unit 32, the projection position calculation unit 35 performs the three-dimensional shape model. Estimate the projected position of the pattern above. Then, the projection position calculation unit 35 obtains a position on the photographing surface of the photographing unit 2 corresponding to the estimated projection position as a projected image coordinate position.

  In step S307, the distance calculation unit 36 searches for a corresponding pattern from the vicinity (near) of the projected image coordinate position in the captured image, and associates the searched pattern with the pattern on the pattern image. . Then, the distance calculation unit 36 uses the result of this association, and based on the principle of triangulation, distance information indicating the distance between the photographing unit 2 and each point on the surface of the target object 4, that is, the target object 4 The shape information of is calculated.

  Next, processing performed by the projection position calculation unit 35 will be described with reference to FIG. First, assuming that the projection illumination system of the projection unit 1 projects a pattern from an ideal point light source, a virtual normalized projection image plane with a focal length f = 1 is set, and this is called a pattern projection image plane. . Typically, when the projection pattern image generation unit 33 writes a pattern on the pattern projection image plane, the projection unit 1 projects the pattern onto the target object 4 by a device such as actual illumination or a lens. In reality, the projected image may be distorted due to the effects of projection lens distortion and the inability to use an ideal point light source. However, optical correction or pattern image correction can be performed in advance. The influence can be reduced.

The coordinate position m of the pattern on the pattern projection image plane is assumed to be m = [u, v, 1] ^T. The coordinate system of the projection center with respect to the reference coordinate system of the measurement space is defined as a projected coordinate system, and the coordinate system is expressed using six degrees of freedom, ie, a rotation component and three translational degrees. In other words, the offset of the position component and the offset of the posture component of the projection coordinate system with respect to the reference coordinate system of the measurement space are each expressed by 3 degrees of freedom, and expressed using a total of 6 degrees of freedom.

As the expression of the rotation component R, any expression method such as Euler angle, roll / pitch / yaw, rotation axis rotation angle, and four-dimensional number may be used. Here, a 3 × 3 rotation matrix R obtained from the rotation representation is used. Translation component _{t x, t y,} 3 is the component 3 × 1 matrix _{t = [t x, t y} , t z] of _{t z} using ^T.

The pattern projected from the projection unit 1 is projected onto a pattern projection position M = [X, Y, Z] ^T in the reference coordinate system. Assuming that the pattern projection position M exists on a straight line passing through the pattern coordinate position m on the pattern projection image plane from the light source, the position on the light beam (straight line) (position that can be the pattern projection position M) is expressed by the following equation. Can express.

Here, λ is a positive value serving as a scale of the linear direction vector. L = [L _x , L _y , L _z ] ^T is a summary of the components of a straight direction vector for simplicity.

Next, the position of the intersection between the straight line of formula (1) and the 3D shape model surface is obtained, and this is set as the projection position M on the 3D shape model surface. Here, it is determined whether or not each of the N triangular patches constituting the three-dimensional shape model held in the shape model holding unit 31 intersects with the straight line of Expression (1). Let the coordinates of the j-th vertex of the i-th triangular patch of the three-dimensional shape model be P _ij = [px _ij , py _ij , pz _ij ] ^T. Then, using the following equation (2), the coordinates P _ij are converted into reference coordinates in the measurement space using the rotation component (rotation) Rm and the translation component (translation) tm of the approximate position and orientation held by the approximate position and orientation holding unit 32. Convert to system coordinates P′ij.

The plane direction (normal vector) of the triangular patch is assumed to be held in the shape model holding unit 31 as data of a three-dimensional shape model. The normal vector n ′ _i of the i-th triangular patch in the measurement space is obtained by multiplying the normal vector n _i of the triangular patch held in the shape model holding unit 31 by the rotation Rm.

Since the line segment connecting the vertex P ′ _ij and the intersection point Pc _{i of the} straight line and the normal vector n ′ _{i of the} triangle patch including the vertex P ′ _ij are orthogonal to each other, Equation (3) holds.

In this formula 3, it is a formula about vertex P ' _i0 . Substituting equation (1) into equation (3) gives the following equation (4).

  Further, formula (4) can be summarized with respect to λ to formula (5).

When this is determined as the coordinates of the intersection Pc _i into Equation (1), the following equation (6).

As a check of whether or not the intersection point Pc _i exists inside the triangle patch, the outer product of each vertex of the triangle patch and the intersection point Pc _i is calculated, and when each of the triangle patches has the same direction as the normal vector n ′ _i , the triangle patch There will be a crossing point inside. Specifically, when all of the following three expressions are positive, the intersection exists inside the triangular patch.

  Note that it is necessary to obtain an intersection with a triangular patch closer to the projection coordinate system origin. That is, among the triangular patches having intersection points, a triangular patch having an intersection point closest to the projected coordinate system origin under the condition of λ> 0 is specified, and the intersection point of the specified triangular patch is determined as the surface of the three-dimensional shape model. The estimated position (projection position M) of the pattern above may be used. Specifically, a calculation process according to the following equation (8) may be performed.

  The intersection point with the triangular patch obtained by Expression (8) becomes the final estimated position (projection position M) of the pattern on the surface of the three-dimensional shape model. In other words, when it is assumed that a pattern at a certain position on the pattern image has been projected onto the three-dimensional shape model arranged in the approximate position and orientation, the projection position is estimated as the estimated position M.

On the other hand, the photographing unit 2 photographs the target object 4 on which the pattern is projected by the projection unit 1 as described above. The imaging system of the imaging unit 2 is considered as a camera and a model of central projection. Let f ′ be the focal length to the captured image plane (projected image plane). Furthermore, it is assumed that the projection position M of the pattern in the reference coordinate system is projected (captured) on the captured image. Here, m ′ = [uc ′, vc ′, 1] ^T is a position (projected image coordinate position on the photographed image) m ′ in the photographed image where the projection position M of the pattern in the reference coordinate system is shown. . If the center of the photographing lens 21 of the photographing unit 2 is expressed by six degrees of freedom of rotation and translation in the reference coordinate system, and expressed as a matrix of R ′ and t ′, respectively, from the relationship of central projection, the following equation (9) is obtained. Become.

Here, s is a scalar value of the scale element. Here, it is assumed that the coordinate system of the lens of the projection unit 1 and the coordinate system of the photographing unit 2 are fixed, and the local coordinate conversion (position and orientation relationship) is obtained in advance by a calibration procedure. A specific calibration procedure can be obtained by taking a plurality of calibration indices and using a plurality of calibration index coordinate values by the least square method, or by another method. When the local coordinate transformation matrix is represented by R _L (rotation matrix) and t _L (parallel progression), the following equation (10) is established.

  By substituting “pattern projection position M on the surface of the three-dimensional shape model” obtained by Expression (8) and Expression (10) into Expression (9), Expression (11) is obtained. Then, from this equation (11), a position (projected image coordinate position on the photographed image) m ′ in the photographed image where the projection position M of the pattern in the reference coordinate system is captured is obtained.

  When the projected image coordinate position m ′ on the captured image matches the position and orientation of the target object 4 with the approximate position and orientation held by the approximate position and orientation holding unit 32, the projected image coordinate position m ′ is the position of the pattern in the captured image. Match. However, if the position and orientation of the target object 4 and the approximate position and orientation held by the approximate position and orientation holding unit 32 are slightly different, the vicinity of the projected image coordinate position m ′ calculated by the projection position calculation unit 35 An image of the projection pattern is shown on the photographed image. Therefore, the projection position calculation unit 35 instructs the distance calculation unit 36 on the range of the collation area. The distance calculation unit 36 searches for a corresponding pattern within the range of the designated collation region (that is, a region near the calculated projected image coordinate position m ′), the searched pattern, the pattern on the pattern image, Is associated.

  Then, the distance calculation unit 36 uses the result of this association, and based on the principle of triangulation, distance information indicating the distance between the photographing unit 2 and each point on the surface of the target object 4, that is, the target object 4 The shape principle of this calculation is described with reference to FIG.

The optical center of the imaging unit 2 is C _c , the captured image plane is I _c , the focal length is fc, and the coordinates of the image center cc are (u _co , v _co ). At this time, the internal matrix A _c of the imaging section 2 may be described as the following equation (12).

Further, it is assumed that the optical center of the projection unit 1 is C _p , the projected image plane is I _p , the focal length is fp, and the coordinates of the image center cp are (u _po , v _po ). At this time, the internal matrix A _p of projection unit 1 is described by the following equation (13).

Internal matrix A _p of projection unit 1 and the internal matrix A _c of the imaging section 2 can be calculated by using the calibration method of the internal parameter is a known technique.

External parameters representing the relative positional relationship between the coordinate system (X, Y, Z) of the imaging unit 2 and the coordinate system (X _p , Y _p , Z _p ) of the projection unit 1 are a rotation matrix R and a parallel progression T (Corresponding to R _L and t _L in FIG. 4 respectively). The rotation matrix R is a 3 × 3 matrix, and the parallel progression T is a 3 × 1 matrix. R and T can be calculated by using a known external parameter calibration method.

The coordinates of the point M in the space with the camera coordinate system of the photographing unit 2 as the origin are defined as (X, Y, Z). The coordinates of the projection the point _{m c} a point M on the photographic image plane _{I c} of the imaging section 2 and _{(u c, v} c). The corresponding relationship is described by the following formula (14).

s is a scalar. Further, the coordinates of the _{m p} points projected on the point M the projection image plane _{I p} of the projection unit 1 _{(u p, v} p). The corresponding relationship is described by the following equation (15).

  s' is a scalar. When the above formulas (14) and (15) are expanded, the following four simultaneous equations represented by the following formula (16) can be obtained.

Point _{m c} of the pixel coordinates _{(u c, v} c) a point _{m p} pixel coordinates _{(u p, v} p) is obtained by using a pattern projection method such as spatial encoding method. Since C _ij (i = 1 to 3, j = 1 to 3) and P _ij (i = 1 to 3, j = 1 to 3) are calculated from the internal matrix and the external parameters, they are obtained in advance by calibration. It has been. In the equation (14), only the coordinate value (X, Y, Z) of the point M is an unknown, and can be obtained by solving simultaneous equations. Incidentally, it determined for the coordinate value of a point is unknown M is three (X, Y, Z), coordinate values of the projection unit 1 (u _{p, v p)} by obtaining one of, the point M Coordinate values can be calculated. The above is the principle of distance measurement based on the triangulation method.

  As described above, the position of the slit pattern on the photographed image is estimated from the three-dimensional shape model and its approximate position and orientation, and the projected slit pattern is associated between the photographing unit 2 and the projection unit 1 to perform three-dimensional measurement. Points can be acquired.

  Each of the three-dimensional measuring device 3, the photographing unit 2, and the projecting unit 1 may be a separate device or may be integrated. In the case of a separate device, for example, a commercially available liquid crystal projector can be used for the projection unit 1, and a commercially available camera and lens can be used for the photographing unit 2. The three-dimensional measuring device only needs to have a mechanism for outputting a signal usable by each device and an input / output portion for receiving the signal. By doing so, the device can be expanded and used in accordance with the user's application and measurement range.

[Second Embodiment]
In the three-dimensional measuring apparatus described in the first embodiment, a plurality of slit patterns are used for distance calculation. Therefore, the initial value of the approximate position and orientation is required for associating the slit pattern.

  In the present embodiment, a plurality of slit patterns and a code pattern that uniquely associates them are created and projected on the target object by periodically changing the width of the slit. For example, the approximate position and orientation are roughly obtained by projecting a code pattern as shown in FIG. For the slit pattern that could not be associated, the position of the slit pattern on the captured image is obtained from the three-dimensional shape model and the roughly estimated position and orientation, and the projected slit pattern is associated between the imaging unit and the projection unit. I do. The purpose is to increase the number of three-dimensional measurement points by the above repeated processing.

  A configuration example of the system according to the present embodiment will be described with reference to the block diagram of FIG. In FIG. 8, the same functional units as those shown in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted.

  The configuration shown in FIG. 8 has a configuration obtained by adding a position and orientation calculation unit 37, a projection code sequence generation unit 81, and an imaging code sequence identification unit 82 to the configuration shown in FIG.

  A method by which the projection code string generation unit 81 generates a code string will be described. As shown in FIG. 10, the code string is a two-dimensional array in which codes are arranged in the u and v directions, and is encoded using the rules described below. In the present embodiment, two de Bruijn sequences are selected one by one in the u direction and one in the v direction, and in two vertical and horizontal directions.

The de Bruijn sequence is a numerical sequence with a period l, and only one pattern appears in the period when m consecutive elements are viewed. The selection of the de Bruijn sequence is such that the number of symbols that can be used as a pattern image is k, and the size of the rectangle surrounding the sampling shape is m × (n + 1), and the de Bruijn sequence in the u direction has l _u = k ^m the, v directions respectively chosen for an _l v = ^{k n.} In the present embodiment, since the number of symbols = 3, k = 3. Also, since the partial code string is all surrounded by a 3 × 3 rectangle, m = 3 and n = 2. The de Bruijn sequence of k = 3 and m = 3 (sequence length 3 ³ = 27) used in the u direction of the present embodiment is a sequence shown below.

Similarly, a de Bruijn sequence of k = 3 and n = 2 (sequence length 3 ² = 9) used in the v direction of the present embodiment is shown below.

Next, a method for generating a code string using the above two de Bruijn sequences will be described. The generated code string f _ij uses the u-direction de Bruijn sequence as it is as the code string as it is in the u-direction i = 1st row as in the equation (19).

  The second and subsequent lines in the u direction are expressed by the following equation (20).

  A code string is a result of adding the de Bruijn sequence in the u direction to the previous row. Here, the addition result is represented by a k-ary number represented by 1 to k, and the carry is ignored. As described above, a code string is obtained by adding all the de Bruijn sequences in the u direction one by one, thereby generating a two-dimensional code string in each of the u and v directions as shown in FIG. I can do it.

  As shown in FIG. 11, the code string generated by the above method is a formula (21) in which a certain row and a certain column are fixed in an area surrounded by a rectangle at an arbitrary size of m × n in the code string. ), There is only one partial code sequence of m and n lengths sampled in the order as shown in FIG. i and j indicate the coordinates of the code sampling.

  However, Expression (21) is premised on the order when the sampling shape as shown in FIG. 11 is a cross shape. Depending on the length to be sampled, the sampling shape may be changed and sampling may be performed in an arbitrary order. In the present embodiment, the De Bruijn sequence is used, but the code sequence may be a pseudo-random sequence in which an arbitrarily selected partial code sequence is uniquely determined with respect to other partial code sequences, such as an M sequence.

  The projection pattern image generation unit 33 generates a pattern image obtained by replacing each code of the code sequence received from the projection code sequence generation unit 81 with a pattern corresponding to the code. For example, the region divided by the epipolar line and the measurement pattern as shown in FIG. 7A is divided into three regions, and each region is assigned to a ternary code. For example, the symbol “1” is associated with the left, the symbol “2” is associated with the center, and the symbol “3” is associated with the right.

  When the code of the uppermost line indicated by the code string is “2” and “1” in order from the left end as shown in FIG. 7B, the left end of the uppermost line is shown as shown on the right side of FIG. The code patterns are assigned so as to correspond to the code “2” and the code “1” in order. In this manner, by assigning code patterns corresponding to the respective codes in the same manner for the second and subsequent lines, an image can be generated according to the code string.

  In this embodiment, as shown in FIG. 7, the projection pattern image is composed of line patterns having two types of directions, that is, a measurement pattern and a code pattern, but a line pattern having two or more types of directions. You may comprise. In this case, it is only necessary to perform a filter process using a Sobel filter corresponding to each directionality and extract only line patterns having the same directionality. Thereby, the distortion of the subject can be measured in more detail.

  The shape of the code pattern is not limited to a square, and may be a circle or a triangle. Further, instead of associating with a code by assigning a region, a pattern shape or direction itself may be associated.

  The photographic code string identification unit 82 extracts a code pattern projected on the target object 4 from the photographic image captured by the photographic image acquisition unit 34, and based on the arrangement relationship of the regions divided by the epipolar line and the measurement pattern. The code of the corresponding code pattern is acquired as a photographing code string.

  The distance calculation unit 36 according to the present embodiment associates the measurement patterns of the pattern image and the captured image with each other from the partial code sequence indicating the characteristic that is uniquely determined in the code sequence. Further, using this association result, distance information indicating the distance between the imaging unit 2 and the target object 4 on which the pattern image is projected, that is, the shape of the target object 4 is calculated by the principle of triangulation.

  The position and orientation calculation unit 37 minimizes the difference between the distance obtained by the distance calculation unit 36 and the distance from the surface of the three-dimensional shape model arranged in the approximate position and orientation to the photographing unit 2. Update posture repeatedly (optimization calculation). The position / orientation calculation unit 37 sets the position / orientation after the repeated update as the position / orientation of the target object 4.

  This position / orientation estimation process (optimization calculation) involves non-linear calculation with respect to the attitude, so basically, iterative error minimization is applied. Specifically, using the ICP method disclosed in Non-Patent Document 1 or the method disclosed in Non-Patent Document 2, the position and orientation of the 3D shape model is obtained from the 3D shape model and the measurement distance. Can do. At that time, as the initial value, the approximate position and orientation held by the approximate position and orientation holding unit 32 can be used.

  Further, the position / orientation calculation unit 37 determines whether or not to continue the optimization calculation from the residual generated as a result of the repetition calculation. The threshold value for this determination may be a threshold value set by the user or a threshold value set using the relationship between the observation area in the work space and the distance from the imaging device. Furthermore, the predetermined value of the three-dimensional measuring device is held, and it may be read out as a threshold value.

  The threshold value set by any method is compared with the value of the residual, but the residual after the calculation may not actually become small. Therefore, in the present embodiment, the number of repetitions is held, the trend for each residual repetition is calculated, and the position / orientation calculation is performed when the number of repetitions exceeds the predetermined number or when the trend tends to increase. The repetition of the optimization calculation by the unit 37 is stopped. This makes it possible to determine whether or not the position / orientation optimization calculation has succeeded.

  Next, processing performed for estimating the position and orientation of the target object 4 will be described with reference to FIG. 9 showing a flowchart of the processing.

  Since the processing in each of step S901 and step S902 is the same as the processing in each of step S301 and step S302, description relating to these steps will be omitted. It should be noted that various initialization processes such as initializing the variable k used by the position / orientation calculation unit 37 to 0, for example, are performed in a state in which preparations in advance in steps S901 and S902 are completed. The process is started.

  In step S903, the projection code string generation unit 81 generates a code string for associating the measurement pattern in each of the pattern image and the captured image.

  In step S904, the projection pattern image generation unit 33 generates a pattern image including a plurality of slit patterns and a code pattern that uniquely associates each slit according to the code string generated in step S903. Then, the projection pattern image generation unit 33 outputs the generated pattern image to the projection unit 1. In step S905, the projection unit 1 projects a pattern based on the pattern image output from the projection pattern image generation unit 33 onto the target object 4.

  In step S906, the photographed image acquisition unit 34 controls the photographing unit 2 to photograph the target object 4 on which the pattern is projected. Then, the captured image acquisition unit 34 captures the digital image signal from the capturing unit 2 and generates a captured image from the captured digital image signal. Of course, the captured object includes the target object 4 on which the pattern is projected.

  In step S907, the imaging code string identification unit 82 extracts a code pattern from the captured image acquired in step S906, and detects the code string as an imaging code string from the extracted code pattern.

  In step S908, the projection position calculation unit 35 arranges the three-dimensional shape model held in the shape model holding unit 31 in the approximate position and orientation held by the approximate position and orientation holding unit 32. Estimate the projected position of the pattern above. Then, the projection position calculation unit 35 obtains a position on the photographing surface of the photographing unit 2 corresponding to the estimated projection position as a projected image coordinate position.

  In the calculation processing in step S909, distance information indicating the distance between the photographing unit 2 and each point on the surface of the target object 4, that is, calculation processing for calculating the shape information of the target object 4 is performed. The calculation method is different between the first calculation process and the second and subsequent calculation processes.

  In the first step S909, the distance calculation unit 36 associates the imaging code sequence detected in step S907 with the code sequence represented by the code pattern written in the pattern image. Then, the distance calculation unit 36 uses the result of this association, and based on the principle of triangulation, distance information indicating the distance between the photographing unit 2 and each point on the surface of the target object 4, that is, the target object 4 The shape information of is calculated.

  In the second and subsequent steps S909, the distance calculation unit 36 searches for the corresponding measurement pattern from the vicinity (neighboring) of the projected image coordinate position obtained in step S908, and the searched measurement pattern and the pattern image on the pattern image. The measurement pattern is associated. That is, a corresponding measurement pattern is searched from the vicinity (near) of the projected image coordinate position obtained based on the previous approximate position and orientation, and the searched measurement pattern is associated with the measurement pattern on the pattern image. I do. Then, the distance calculation unit 36 uses the result of this association, and based on the principle of triangulation, distance information indicating the distance between the photographing unit 2 and each point on the surface of the target object 4, that is, the target object 4 The shape information of is calculated.

  In step S910, the position / orientation calculation unit 37 minimizes the difference between the distance obtained by the distance calculation unit 36 and the distance from the surface of the three-dimensional shape model arranged in the approximate position / orientation to the photographing unit 2. The position and orientation of the target object 4 is estimated (derived).

  In step S911, the position / orientation calculation unit 37 updates the approximate position and orientation held by the approximate position and orientation holding unit 32 with the position and orientation estimated in step S910. In step S912, the position / orientation calculation unit 37 increments the value of the variable k by 1, thereby counting one repetition of the position / orientation estimation process in step S910.

  In step S913, the position / orientation calculation unit 37 determines whether the value of the variable k is equal to or greater than a threshold value (the number of repetitions is equal to or greater than the specified number). As a result of the determination, if the value of the variable k is equal to or greater than the threshold value, the processing according to the flowchart of FIG. 9 ends. On the other hand, if the value of the variable k is less than the threshold, the process proceeds to step S908.

  Next, an example of processing performed by the position / orientation calculation unit 37 will be described with reference to the flowchart of FIG.

  In step S601, the position / orientation calculation unit 37 initializes a variable k used in the position / orientation measurement process to 0, and initializes the value of the moving average value to 0. In step S <b> 602, the position / orientation calculation unit 37 acquires the distance obtained by the distance calculation unit 36.

  In step S603, the position / orientation calculation unit 37 minimizes the difference between the distance obtained by the distance calculation unit 36 and the distance from the surface of the three-dimensional shape model arranged in the approximate position / orientation to the photographing unit 2. The position and orientation of the target object 4 is estimated. Then, the position / orientation calculation unit 37 updates the approximate position and orientation held by the approximate position and orientation holding unit 32 with the estimated position and orientation.

  In step S604, the position / orientation calculation unit 37 acquires the updated position / orientation and the residual obtained by the calculation process in step S603. In step S605, the position / orientation calculation unit 37 determines whether the residual is equal to or less than a specified value. As a result of the determination, if the residual is equal to or less than the specified value, the process proceeds to step S606. In step S606, the position / orientation calculation unit 37 sets a value indicating that the estimation of the position / orientation of the target object 4 has succeeded (successful optimization) to the flag, and ends the present process.

  On the other hand, if the residual is greater than the threshold, the process proceeds to step S607. In step S607, the position / orientation calculation unit 37 calculates a moving average value of the residual in order to calculate a trend of whether the residual is decreasing or increasing. The number of moving average frames n may be specified by the user, or an experimentally obtained value may be set.

  In step S608, if the calculated moving average value tends to be larger than the previously calculated moving average value, the position / orientation calculating unit 37 determines that the error minimization calculation of the optimization calculation is divergent. The process proceeds to step S609. In step S609, the position / orientation calculation unit 37 sets a value indicating that the estimation of the position / orientation of the target object 4 has failed (optimization failure) as a flag, and ends the present process.

  On the other hand, if the calculated moving average value tends to be smaller than the previously calculated moving average value, the process proceeds to step S610. In step S610, the position / orientation calculation unit 37 increments the value of the variable k by one.

  In step S611, the position / orientation calculation unit 37 determines whether or not the value of the variable k is equal to or greater than a threshold value (the number of repetitions is equal to or greater than a specified number). As a result of the determination, if the value of the variable k is greater than or equal to the threshold value, the process proceeds to step S612. In step S612, the position / orientation calculation unit 37 sets a value indicating that the number of repetitions is equal to or greater than the specified number as a flag, and ends the present process. On the other hand, when the value of the variable k is less than the threshold, the process returns to step S603 and the subsequent processes are repeated.

  As described above, according to the present embodiment, the position of the measurement pattern on the photographed image is compared with the three-dimensional shape model and the first time at the location where the association between the photographing unit 2 and the projection unit 1 cannot be performed in the first process. Can be obtained from the approximate position and orientation estimated in step (1). Then, it is possible to acquire a three-dimensional measurement point by associating the projected slit pattern between the imaging unit 2 and the projection unit 1.

[Third Embodiment]
In the second embodiment, the accuracy of three-dimensional measurement is improved by performing measurement processing without changing the pattern. However, the code pattern may interfere with the slit pattern by an optical system such as blur, and the accuracy may decrease. Therefore, in this embodiment, in the first shooting, a pattern including a code pattern as shown in FIG. 7 is projected to associate the slit pattern. In the second and subsequent photographing, for example, the code pattern as shown in FIG. 2 is changed and projected into a pattern that does not interfere with the slit pattern, and the slit pattern is associated with the approximate position and orientation based on the photographed image.

  The configuration of the system according to this embodiment is the same as that of the second embodiment (FIG. 8). Next, processing performed for estimating the position and orientation of the target object 4 will be described with reference to FIG. 12 showing a flowchart of the processing.

  Since the processing in each of steps S1201 to S1207 is the same as the processing in each of steps S901 to S907, description of these steps is omitted.

  In the calculation processing in step S1208, distance information indicating the distance between the imaging unit 2 and each point on the surface of the target object 4, that is, calculation processing for calculating the shape information of the target object 4 is performed. The calculation method is different between the first calculation process and the second and subsequent calculation processes.

  Here, the projection pattern image generation unit 33 sets a pattern image including a code pattern as shown in FIG. 7 as an initial value of the pattern image to be used. However, in the first step S1208, the distance calculation unit 36 associates the captured code string detected from the pattern image in step S1207 with the code string represented by the code pattern described in the pattern image. I do. Then, the distance calculation unit 36 uses the result of this association, and based on the principle of triangulation, distance information indicating the distance between the photographing unit 2 and each point on the surface of the target object 4, that is, the target object 4 The shape information of is calculated.

  In addition, when the projection pattern image generation unit 33 finishes the processing of step S1204 to step S1212 once, a pattern image in which the code pattern does not interfere with the slit pattern as shown in FIG. It sets in step S1214. However, in the second and subsequent steps S1208, the distance calculation unit 36 searches for the corresponding pattern from the vicinity (neighborhood) of the projected image coordinate position obtained in step S1213, and the searched pattern and the pattern image Correspondence with the pattern is performed. That is, a corresponding measurement pattern is searched from the vicinity (near) of the projected image coordinate position obtained based on the previous approximate position and orientation, and the searched measurement pattern is associated with the measurement pattern on the pattern image. I do. Then, the distance calculation unit 36 uses the result of this association, and based on the principle of triangulation, distance information indicating the distance between the photographing unit 2 and each point on the surface of the target object 4, that is, the target object 4 The shape information of is calculated.

  In step S1209, the position / orientation calculation unit 37 minimizes the difference between the distance obtained by the distance calculation unit 36 and the distance from the surface of the three-dimensional shape model arranged in the approximate position / orientation to the photographing unit 2. The position and orientation of the target object 4 is estimated.

  In step S1210, the position / orientation calculation unit 37 updates the approximate position and orientation held by the approximate position and orientation holding unit 32 with the position and orientation estimated in step S1209. In step S1211, the position / orientation calculation unit 37 increments the value of the variable k by 1, thereby counting one repetition of the position / orientation estimation process in step S1209.

  In step S <b> 1212, the position / orientation calculation unit 37 determines whether or not the value of the variable k is equal to or greater than a threshold value (the number of repetitions is equal to or greater than a specified number). If the value of the variable k is equal to or greater than the threshold value as a result of this determination, the processing according to the flowchart of FIG. 12 ends. On the other hand, if the value of the variable k is less than the threshold, the process proceeds to step S1213.

  In step S <b> 1213, the projection position calculation unit 35 arranges the three-dimensional shape model held in the shape model holding unit 31 in the approximate position and orientation held by the approximate position and orientation holding unit 32. Estimate the projected position of the pattern above. Then, the projection position calculation unit 35 obtains a position on the photographing surface of the photographing unit 2 corresponding to the estimated projection position as a projected image coordinate position.

  In step S1214, the projection pattern image generation unit 33 switches the pattern image output to the projection unit 1 as described above. That is, the projection pattern image generation unit 33 sets a pattern image including a code pattern as shown in FIG. 7 as an initial value of the pattern image to be used. In step S1214, the projection pattern image generation unit 33 sets a pattern image in which the code pattern does not interfere with the slit pattern as shown in FIG. 2 as a pattern to be used.

  By doing in this way, in the process after the 2nd time, the accurate three-dimensional measurement point which is not influenced by a code pattern can be acquired. In the present embodiment, only the plurality of slit patterns are changed from the second time onward, but the shape / size of the code pattern is changed or the interval between the slit patterns is changed so as not to interfere with the plurality of slit patterns. I do not care. In addition, since the second slit pattern does not include a code, the measurement line width may be increased, the measurement line interval may be narrowed, and the number of measurement lines may be increased as compared to the first measurement pattern. .

[Fourth Embodiment]
1 and 8 may be configured by hardware, but each function unit excluding the shape model holding unit 31 and the approximate position / posture holding unit 32 may be software (computer). Program).

  For example, an apparatus having a memory functioning as the shape model holding unit 31 and the approximate position and orientation holding unit 32, a memory for storing the software, and a control unit for executing the software is included in the three-dimensional measurement device 3. Can be applied. In this case, when the control unit (for example, CPU) executes the software, the control unit realizes the operation of the three-dimensional measurement apparatus 3 described in the above embodiment.

(Other examples)
The present invention estimates a projection position of a pattern on the model from a three-dimensional shape model and its approximate position and orientation, obtains a coordinate position on a captured image corresponding to the projection position, associates a slit pattern, and Although distance calculation is performed based on the principle of surveying, there is no need for a three-dimensional shape model. As a first process, a code pattern is projected onto a measurement object to perform three-dimensional measurement, and a rough three-dimensional model is generated from the three-dimensional measurement points. As a second process, regarding the slit pattern that could not be matched, the slit pattern position on the captured image is obtained from the three-dimensional shape model that generated the position and orientation, and the projected slit pattern is captured-projected. Correlation between parts may be performed. Since the slit pattern projected in the second time does not arrange a code, the width of the measurement lines may be increased, the distance between the measurement lines may be reduced, and the number of measurement lines may be increased as compared with the first measurement pattern. .

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

<Effect of Example>
According to the first embodiment, the position of the slit pattern on the photographed image is estimated from the three-dimensional shape model and its approximate position and orientation, and the projected slit pattern is associated between the photographing unit 2 and the projecting unit 1. 3D measurement points can be acquired.

  According to the second embodiment, the position of the measurement pattern on the photographed image is estimated with the three-dimensional shape model and the first time at the location where the association between the photographing unit 2 and the projection unit 1 cannot be performed in the first process. It can be obtained from the position and orientation. Then, it is possible to acquire a three-dimensional measurement point by associating the projected slit pattern between the imaging unit 2 and the projection unit 1.

  According to the third embodiment, in the second and subsequent processes, it is possible to acquire an accurate three-dimensional measurement point that is not affected by the code pattern by switching the pattern to be projected.

Claims (8)

A distance measuring device connected to a projection unit that projects a single pattern having a plurality of slits, and an imaging unit that captures an image of a target object on which the pattern is projected,
Means for acquiring a rough position and orientation of the target object;
Estimation means for estimating a projection position of a pattern on the model when the model representing the three-dimensional shape of the target object is arranged in the approximate position and orientation;
Search means for finding a coordinate position on the image corresponding to the projection position, and searching for the pattern from the vicinity of the obtained coordinate position;
The distance from the imaging unit to the surface of the target object is calculated by performing a calculation process based on triangulation using the correspondence between the pattern searched from the image and the pattern projected by the projection unit. Calculation means;
A distance measuring device comprising:   The distance measuring apparatus according to claim 1, wherein the estimation unit obtains, as the projection position, an intersection position between a straight line passing through a pattern projected by the projection unit from a light source of the projection unit and the model. .   The search unit obtains, as the coordinate position, a position obtained by projecting the projection position on the image using information representing a position and orientation relationship between the photographing unit and the projection unit. The distance measuring device described in 1. In the first calculation process, the calculation means uses the correspondence relationship between the code string represented by the pattern in the image and the code string represented by the pattern projected by the projection unit, from the photographing unit to the target. Calculate the distance to the surface of the object,
In the second and subsequent calculation processes, the calculation means uses the approximate position and orientation updated last time, the pattern searched from the image by the estimation means and the search means, and the pattern projected by the projection unit. The distance from the said imaging | photography part to the surface of the said target object is calculated by performing the calculation process based on triangulation using the correspondence of these, The Claim 1 thru | or 3 characterized by the above-mentioned. Distance measuring device. The calculating means includes
In the first calculation process, a pattern including a pattern representing a code string is used,
The distance measurement apparatus according to claim 4, wherein a pattern that does not include a pattern representing a code string is used in the second and subsequent calculation processes. Furthermore,
The position and orientation of the target object are derived by updating the approximate position and orientation based on the distance and the distance from the surface of the model arranged in the approximate position and orientation to the imaging unit. 6. The distance measuring device according to claim 1, further comprising a derivation unit. A distance measuring method performed by a distance measuring device connected to a projecting unit that projects a single pattern having a plurality of slits, and an imaging unit that captures an image of a target object on which the pattern is projected,
The acquisition unit of the distance measuring device acquires a rough position and orientation of the target object;
When the estimation unit of the distance measuring device arranges a model representing the three-dimensional shape of the target object in the approximate position and orientation, an estimation step of estimating a projection position of a pattern on the model;
A search step in which the search means of the distance measuring device obtains a coordinate position on the image corresponding to the projection position, and searches for the pattern from the vicinity of the obtained coordinate position;
The calculation unit of the distance measuring device performs a calculation process based on triangulation using a correspondence relationship between a pattern searched from the image and a pattern projected by the projection unit, so that the target from the imaging unit A calculation process for calculating the distance to the surface of the object;
A distance measuring method comprising:   The computer program for functioning a computer as each means of the distance measuring device of any one of Claims 1 thru | or 6.

Images