JP2010112729A - Method of creating three-dimensional model, and object recognition device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To create a precise three-dimensional model even by a method for performing a three-dimensional measurement by optionally changing a position relationship between an actual model and a camera and aligning three-dimensional information restored by each measurement for integration. <P>SOLUTION: A polygonal mark M having a shape in which a direction can be uniquely specified is attached to a predetermined area of an actual model WM of a work to be three-dimensionally recognized. A process of changing the orientation of the actual model WM such that a state in which the mark M is contained in the view of respective cameras 11, 12, 13 is maintained, and then performing the three-dimensional measurement is executed over a plurality of times. The predetermined number of, or two or more pieces of three-dimensional information is selected from the three-dimensional information restored by each measurement, these pieces of three-dimensional information are aligned and integrated, the information corresponding to the mark M is deleted or invalidated from the integrated three-dimensional information, and the three-dimensional information after such a process is set as a three-dimensional model. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  In order to perform processing for recognizing the position and orientation of an object using three-dimensional information restored by stereo measurement using a plurality of cameras for an object having a predetermined shape, The present invention relates to a method for creating a three-dimensional model. Furthermore, the present invention relates to an apparatus for performing object recognition processing using the model after creating and registering the three-dimensional model.

In order to recognize the position and orientation of an object by three-dimensional recognition processing using a stereo camera, it is necessary to create a three-dimensional model including three-dimensional information of various surfaces of the object to be recognized.
The following Patent Document 1 discloses an invention that addresses the above-described problems. In Patent Document 1, stereo measurement from a plurality of directions is performed on a real model of a recognition target object, and the three-dimensional information restored by each measurement is aligned and integrated to obtain a geometric model of the entire object. Is described.

  Patent Document 1 describes that three-dimensional information of a contour line of a real model is restored from an edge represented in a stereo image by a technique called “segment-based stereo”. According to Patent Document 1, “segment-based stereo” is a method of dividing edges in a stereo image into units called “segments” and performing stereo correspondence search on a segment basis to restore the three-dimensional information of the contour line. Refers to processing.

  Furthermore, Patent Document 1 describes a method of performing coordinate conversion based on a rotation angle by rotating a real model to a predetermined angle using a rotation table with respect to alignment of three-dimensional information. In addition, as a method of aligning 3D information from unknown observation directions, a plurality of alignment candidates are obtained by collating the model being created with the newly restored 3D contour, It also describes specifying the alignment when the degree of coincidence is the highest.

  Further, the following Non-Patent Document 1 describes in detail a technique for associating a two-dimensional segment between images and restoring three-dimensional information. Non-Patent Document 2 discloses a method for recognizing the position and orientation of an object by collating restored three-dimensional information with a pre-registered three-dimensional model.

Japanese Patent No. 2961264 (see paragraphs 0013, 0017, 0028 to 0036) "Correspondence evaluation based on connectivity in segment-based stereo" IPSJ Journal Vol. 40, no. 8, pages 3219-3229, issued in August 1999 "Recognition of three-dimensional objects by stereo vision" IEICE Transactions D-II, vol. J80-D-II, no. 5, 1105-1112 pages, published in May 1997

  The invention described in Patent Document 1 creates a three-dimensional model that integrates the three-dimensional features of an object viewed from various directions, and thus recognizes an object whose position and orientation are not fixed, such as a part that is piled up. A three-dimensional model suitable for the above can be created. However, when a three-dimensional model is created using the “method for aligning three-dimensional information from an unknown observation direction” in Patent Document 1, the accuracy of the three-dimensional model is ensured for the following reason. It seems difficult.

  Specifically, the “method for aligning three-dimensional information from an unknown observation direction” in Patent Document 1 is a method between two stereo measurements performed by arbitrarily changing the positional relationship between a real model and a stereo camera. Then, the degree of coincidence is calculated by associating the three-dimensional information restored by each measurement in a plurality of ways, and alignment is performed using the correspondence when the degree of coincidence becomes the highest. Therefore, if the correspondence between the three-dimensional information cannot be performed with high accuracy, the alignment accuracy cannot be ensured, and the final three-dimensional model also has poor accuracy.

First, when a chamfered portion or a curved surface is a measurement target, there may be an error in correspondence due to the cause as shown in FIG.
FIG. 17 shows an example in which stereo measurement is performed on the corner portion of the recognition object Sb from two directions, f1 and f2. This figure is a cross section of the recognition object Sb, and R1 and R2 in the figure correspond to the boundary position between the flat surface and the surface having a large curvature.
In this example, the measurement from the direction f1 restores the three-dimensional edge at the position R1 in the figure, while the measurement from the direction f2 restores the three-dimensional edge at the position R2. Is done. Accordingly, when associating the three-dimensional information restored by measurement from each direction, the edge corresponding to R1 and the edge corresponding to R2 are erroneously associated with each other, and the alignment accuracy is deteriorated accordingly. There is a fear.

FIG. 18 shows an example in which an error has occurred in associating a part including a repetitive structure. Such an incorrect association may occur when the edge of the repetitive structure is unclear and the restoration accuracy of the three-dimensional information deteriorates.
Also, when an object having a symmetric part is used as a recognition target, a state in which the front and back of the symmetric part are associated with each other may be recognized as correct.

  Many industrial products and molded products include chamfered parts and curved surfaces, and many repetitive structures and symmetrical parts are also recognized. Therefore, when creating a three-dimensional model related to these objects, there is a high possibility that incorrect positioning will be performed due to the above-described causes, and it becomes difficult to ensure the accuracy of the three-dimensional model.

  Therefore, in order to ensure the accuracy of the three-dimensional model in the invention described in Patent Document 1, the actual model is rotated at a predetermined angle each time using the method described in paragraph 0028 of this document, that is, a rotating table. Therefore, it is necessary to adopt a method of performing image conversion and performing coordinate conversion based on this known rotation angle. However, this method requires equipment such as a rotary table, which is expensive. Further, since the rotation angle for each time must be accurately determined, the burden on the user increases.

  This invention pays attention to this problem, performs a three-dimensional measurement by arbitrarily changing the positional relationship between the real model and the camera, and aligns and integrates the three-dimensional information restored by each measurement. It is an object to make it possible to create a highly accurate three-dimensional model even when it is adopted.

  In the method of creating a three-dimensional model according to the present invention, a polygonal mark having no rotational symmetry is attached to a predetermined position on the surface of the real model of the recognition target object, and this mark is the field of view of all cameras constituting the stereo camera. If the state included in is maintained, the positional relationship between each camera and the real model is changed each time, and a plurality of times of imaging is executed. Further, edge features are detected from a stereo image generated by imaging each time, and the three-dimensional information representing the features of the real model and the outline of the mark is restored by obtaining the three-dimensional coordinates of the detected features. Then, two or more three-dimensional information corresponding to all or a part of a plurality of three-dimensional information restored with each imaging is set as an integration target, and each of the three-dimensional information based on the correspondence relationship between the integration target three-dimensional information. The three-dimensional information to be integrated is aligned and then integrated. Further, the information corresponding to the mark is deleted or invalidated from the integrated three-dimensional information, and the three-dimensional information after this processing is determined as a three-dimensional model.

  In the above description, the three-dimensional information restored by the three-dimensional measurement is an aggregate of a plurality of three-dimensional feature data corresponding to individual contour lines of the real model and the mark. “Alignment” of the three-dimensional information refers to a process of converting the coordinates of a plurality of three-dimensional information restored by stereo measurement from different directions so that there is no deviation between them.

  “Polygon without rotational symmetry” is a planar figure having three or more vertices, and when it is rotated in the range of 360 degrees, it does not overlap with the figure before rotation at any rotation angle. Means things. In addition, polygons having a shape in which there is a significant difference between the degree of coincidence when the same shape figure is associated with no rotational deviation and the degree of coincidence due to other associations. desirable.

  The mark attached to the real model is formed in a sheet shape by, for example, paper, cloth, resin, etc., and is attached to the surface of the real model. In this case, if an adhesive layer is formed on the back surface of the sheet constituting the mark, the mark can be easily attached as a so-called “seal”. If the real model is made of metal, it is possible to create a mark by a magnetic sheet and attach it to the surface of the real model by magnetic force.

  In addition, it is desirable to attach a dark color mark to the light-colored real model and attach a light-colored mark to the dark-colored real model. In this way, since an image with a strong contrast between the mark and the background can be generated, the edge of the mark can be detected with high accuracy.

  Since the imaging process performed by the above method is always performed in a state where the same mark is included in the field of view of all the cameras, the three-dimensional information of the edge of the mark can be restored together with the contour line of the real model. . Further, even in the alignment process when integrating the three-dimensional information, the edge of the mark can be the target of alignment.

In the alignment of the three-dimensional information, the correspondence relationship when the degree of coincidence between the three-dimensional information is the highest is specified, and the positional deviation and the angular deviation between the three-dimensional information are eliminated by coordinate conversion based on the correspondence relationship. . Here, since the shape of the mark is a polygon having no rotational symmetry, the posture of the mark represented by the restored three-dimensional information is unique for each measurement direction. Therefore, when associating the three-dimensional information of the mark, any side or vertex is always shifted unless correctly aligned.
Therefore, if the shape of the mark is determined so that there is a large shift in the side or vertex of the mark when it is not correctly aligned, the mark portion when the alignment is not correct even in the alignment of 3D information The degree of coincidence can be greatly reduced.

  According to the above, the degree of coincidence of the entire three-dimensional information greatly varies depending on whether or not the three-dimensional information of the mark is correctly aligned. In other words, the degree of coincidence when each part including the mark is correctly associated is sufficiently large, whereas when the part is not correctly associated, the overall degree of coincidence of the mark portion is greatly reduced. The degree of match is lowered. Therefore, even if the three-dimensional information of the real model includes features that may be erroneously associated with each other, it is possible to prevent the alignment based on the correspondence from being fixed, A good three-dimensional model can be created.

In the above method, it is desirable to attach a mark to a flat portion on the surface of the real model.
When a mark is attached to a curved surface, depending on the curvature of the surface, three-dimensional information that is significantly different from the original one is restored. However, for the collation based on the geometric features of the mark described below, This is because the original three-dimensional information of the mark needs to be accurately restored.

  Next, the mark may be formed integrally with a background portion having a color different from that of the mark, and the integrated object may be attached to the surface of the real model. In this way, it is possible to prevent an error in correspondence by covering a portion where a feature that is likely to cause an error in matching at the time of alignment, such as a repetitive structure, is covered with an integrated object.

  In the method according to a preferred embodiment, three-dimensional feature data representing the geometric feature of the mark is registered in advance. Prior to the process of integrating the three-dimensional information, the three-dimensional information restored by the stereo image generated by each imaging is collated with the three-dimensional feature data of the mark, so that the geometric feature of the mark is improved. It is determined whether or not it has been restored, and the three-dimensional information determined that this geometric feature has not been satisfactorily restored is excluded from the targets of integration processing.

  When 3D measurement is performed by arbitrarily setting the positional relationship between the real model and the camera each time, unless the 3D shape of the real model is known, the accuracy of the information is determined from the restored 3D information. It is difficult to confirm. However, in the case of the present invention, the dimension and shape can be known in the process of designing and creating the mark, and the information representing the mark is included in the measured three-dimensional information. The accuracy of the three-dimensional information can be determined using the known geometric features of the mark. Therefore, three-dimensional information with poor accuracy can be excluded from integration targets based on the determination result, and the accuracy of the three-dimensional model can be further improved.

  In another preferable aspect, in the three-dimensional measurement, an edge is detected from an image generated by each camera, the detected edge is divided into a plurality of two-dimensional segments, and each two-dimensional segment is associated between the images. Then, a plurality of three-dimensional coordinates are calculated for each pair of corresponding segments, a step of setting a three-dimensional segment of a straight line or a curve based on the distribution state of the calculated three-dimensional coordinates, and an intersection of each three-dimensional segment Executing the step of setting the feature point. Further, in the step of setting the feature point, the edge of the straight three-dimensional segment having an edge that does not intersect with the other segment is extended to a predetermined length, and this extended portion is the other three-dimensional segment or its extension. When intersecting parts or intersecting at a distance within a predetermined value, feature points are set between the intersecting points or intersecting segments. Further, in the alignment process before the integration of the three-dimensional information, one of the three-dimensional information to be integrated is used as a reference, and the other three-dimensional information is compared with the reference three-dimensional information with each feature point and three-dimensional segment as the reference. Transform coordinates so that they match.

  In the above aspect, each edge is divided into segment units, and correspondence and three-dimensional coordinate calculation are performed. Then, each intersection of the finally formed three-dimensional segment is set as a feature point, and when the alignment is performed, the state where the feature point and the three-dimensional segment most closely match the reference three-dimensional information is set to an appropriate position. Set to the combined state. However, depending on the positional relationship between the mark and the camera, part of the mark may become unclear due to shading, etc., and as a result, it may not be possible to restore the three-dimensional information of some vertices of the mark and the accuracy of alignment may be reduced. There is. In view of this point, in the above aspect, the edge that does not intersect with the other of the straight three-dimensional segment is extended to a predetermined length, and the extension portion intersects with another three-dimensional segment or the extension portion, or a predetermined value. In the case of intersecting within a distance, feature points are set between the intersecting points or intersecting segments, so that vertices that could not be restored in the mark can be restored by this method. Therefore, it is possible to secure the alignment accuracy by the three-dimensional segment of the mark and create a highly accurate three-dimensional model.

  In yet another preferred embodiment, the first and second marks by polygons having no rotational symmetry are attached to different locations on the surface of the real model, and for each mark, the mark is in the field of view of all cameras. Positions of each camera and the real model in the plurality of times of imaging so that a plurality of states are established and a state where both the first and second marks are included in the field of view of each camera is established at least once. Adjust the relationship. Further, the three-dimensional information restored from the stereo image when only the first mark is included in the field of view of each camera, and the stereo image when only the second mark is captured while included in the field of view of each camera. The three-dimensional information is aligned with the three-dimensional information restored from the stereo image when the first and second marks are included in the field of view of each camera and the three-dimensional information is integrated.

  In order to create a three-dimensional model showing the entire configuration of the recognition target object, it is necessary to set many aspects of the positional relationship between the real model and the camera so that each surface of the real model becomes an imaging target. In this case, it is difficult to set a state in which a mark is always included in the field of view of all cameras by attaching only one mark.

  In this regard, in the above aspect, the first and second marks are attached to the real model, and it is only necessary to measure at least one of these marks, so that the positional relationship between the real model and the camera can be changed. Can be prevented from being limited. In addition, three-dimensional information when only the first mark is the measurement target and three-dimensional information when only the second mark is the measurement target, and the three-dimensional information when each mark is the measurement target. Therefore, the alignment accuracy can be ensured. Note that the number of second marks is not limited to one, and a plurality of second marks may be attached to different surfaces.

  An object recognition apparatus to which the above method is applied includes an image input unit that inputs a stereo image of a recognition target generated by a stereo camera, detects edge features from the input stereo image, and detects the detected features. 3D measurement means for restoring 3D information by performing 3D measurement, and recognition processing means for recognizing the position and orientation of the recognition object by comparing the restored 3D information with a previously registered 3D model And 3D model registration means for creating and registering a 3D model used by the recognition processing means. Further, the three-dimensional model registration means includes the following means: mark registration means, measurement control means, determination means, three-dimensional information integration means, and model confirmation means.

  The mark registration means is for registering three-dimensional feature data representing a geometric feature of a polygonal mark having no rotational symmetry. The measurement control means inputs a stereo image generated by each imaging on the premise that the actual model with the above mark attached to the surface is imaged a plurality of times with the positional relationship with each camera being changed each time. Is received from the unit and processed by the three-dimensional measuring means. The determination unit compares the three-dimensional information restored from the stereo image by the process of the measurement control unit with the three-dimensional feature data registered in the mark registration unit, so that the geometric feature of the mark is well restored. It is determined whether or not.

  The three-dimensional information integration means targets two or more three-dimensional information determined that the geometric feature of the mark is well restored among a plurality of pieces of three-dimensional information restored in response to a plurality of times of imaging. Then, these three-dimensional information is integrated after being aligned based on the corresponding relationship. The model determining unit deletes or invalidates information corresponding to the mark in the integrated three-dimensional information, and determines the three-dimensional information after the processing as a registration target three-dimensional model.

  In the above configuration, in the processing of the determination means, the processing for determining whether or not the three-dimensional information of the mark has been successfully restored includes the three-dimensional information and the mark in addition to the mode automatically executed by the apparatus. It is possible to include a mode in which the result of collation with the three-dimensional feature data is displayed and the input of the determination result by the user is received.

  According to the above apparatus, every time a user attaches a polygonal mark having no rotational symmetry to an appropriate position of the real model, and the state in which the same mark is commonly included in the field of view of each camera is maintained. By adjusting the positional relationship between the real model and the camera, and taking multiple shots, the three-dimensional information restored from the stereo image obtained by each imaging is selected to create a three-dimensional model with high accuracy. It becomes possible to do. Therefore, after creating and registering a highly accurate three-dimensional model, it is possible to accurately recognize the position and orientation of the object.

  According to the present invention, a polygonal mark having no rotational symmetry is attached to the surface of the real model of the object, and imaging is performed a plurality of times while maintaining the state that the mark is included in the field of view of each camera. By performing the above, it becomes possible to improve the accuracy of the alignment of the three-dimensional information. Therefore, even if the positional relationship between the real model and the camera is arbitrarily changed, it is possible to ensure the accuracy of alignment of the three-dimensional information and create a highly accurate three-dimensional model. In addition, since it is not necessary to strictly adjust the positional relationship, it is possible to reduce the user's effort required to create the three-dimensional model, and to improve convenience.

(1) Device Configuration FIG. 1 shows an example of a picking system to which a three-dimensional recognition process is applied.
This picking system is for performing the work of taking out the workpieces W housed in the housing box 6 one by one and transporting them to a predetermined position in the factory. A robot control device 3 that controls the operation of the robot 4 is included. Further, this picking system is provided with a stereo camera 1 and an object recognition device 2 in order to recognize the position and orientation of the workpiece W to be processed.

  The stereo camera 1 is composed of three cameras 11, 12, and 13 whose positional relationship is fixed. In the object recognition device 2, information representing the positional relationship of the cameras 11, 12, 13, the direction of the optical axis, and the three-dimensional model of the workpiece W to be recognized are registered. After the input stereo image is processed to restore the three-dimensional information of the contour line of the workpiece W, the restored three-dimensional information is collated with the three-dimensional model to recognize the position and orientation of the workpiece W. Information indicating the recognition result is output from the object recognition device 2 to the robot control device 3, and is used for processing for controlling the operation of the arm 40 of the robot 4 in the robot control device 3.

FIG. 2 shows a hardware configuration of the object recognition apparatus 2.
In this apparatus, in addition to the image input units 21, 22, and 23 corresponding to the cameras 11, 12, and 13, a CPU 24, a memory 25, an input unit 26, a display unit 27, a communication interface 28, and the like are provided. The input unit 26 is a keyboard and a mouse, and the display unit 27 is a liquid crystal monitor. The input unit 26 and the display unit 27 are used for applications in which a user confirms an imaging state of a real model, or performs a selection operation or a setting operation when creating a three-dimensional model described below. The communication interface 28 is used for communication with the robot control device 4.

The memory 25 includes a large-capacity memory such as a ROM, a RAM, and a hard disk.
Based on the program stored in the memory 25, the CPU 24 executes a series of processes related to the three-dimensional measurement and the recognition of the workpiece W, and the recognition result (specifically, the three-dimensional coordinates representing the position of the workpiece W, and the three-dimensional The rotation angle with respect to the model is output from the communication interface 28.

  The memory 25 also stores a program for creating a three-dimensional model. Prior to the recognition process, the CPU 24 creates a three-dimensional model of the work W using the images of the real models input from the cameras 11 to 13 based on this program, and registers the three-dimensional model in the memory 25.

  Further, although not shown in FIG. 2, the object recognition apparatus 2 of this embodiment is provided with a circuit for outputting a drive signal to each of the cameras 11, 12, and 13, and the CPU 24 receives each camera via this circuit. A function for controlling the imaging operations 11 to 13 is set.

(2) Three-dimensional measurement processing In the object recognition device 2 described above, after detecting an edge from a stereo image, the three-dimensional information is restored for each unit called “segment” in the same manner as the invention described in Patent Document 1. Furthermore, collation with the three-dimensional model is performed on a segment basis. FIG. 3 shows a schematic procedure of processing executed for recognizing one workpiece by this method. Hereinafter, the three-dimensional recognition process in this embodiment will be described with reference to the step codes in FIG.

  First, stereo imaging is performed by each of the cameras 11 to 13, and an edge extraction filter is applied to each generated image to detect an edge in the image (ST1, 2). Next, the detected edge is thinned (set to 1 pixel width data), and the thinned edge is divided into straight or curved segments based on the connection points and branch points (ST3, 4). The segment extracted from the edge on the two-dimensional image is hereinafter referred to as “two-dimensional segment”.

  Next, a process of associating two-dimensional segments that have a correspondence relationship between images is executed (ST5). In this association, one two of the three images is used as a reference, and each two-dimensional segment of the reference image is focused in turn, and the two-dimensional corresponding to each of the two-dimensional segments focused on from the other two images. Identify the segment. That is, a two-dimensional segment satisfying the epipolar condition and matching with a neighboring segment is detected from each image with respect to the focused two-dimensional segment (refer to Non-Patent Document 1 or the like for details). I want.)

Next, for each combination of the two-dimensional segments associated by the above-described process, a process for restoring the three-dimensional information from the correspondence is executed (ST6).
Briefly, for each combination of two-dimensional segments associated with each other, the three-dimensional coordinates of the pixels having a correspondence relationship between the segments are calculated. Further, the calculated distribution state of the three-dimensional coordinates is collated with a model of a straight line and an arc to determine whether the set of these three-dimensional coordinates corresponds to a straight line / curve. With this process, for each combination of two-dimensional segments, a three-dimensional segment of a straight line or a curve corresponding to the combination is specified.

  Further, in ST6, for each three-dimensional segment, the segment is sampled at a predetermined interval, and information in which the segment type (straight line or curve) is associated with the three-dimensional coordinates of each sampling point is created. . As a result, even for a three-dimensional segment in which the number of three-dimensional coordinates calculated from the two-dimensional segment is small, it is possible to obtain more three-dimensional coordinates from the beginning if the sampling interval is made fine.

  The set of three-dimensional segments restored by the above processing corresponds to the three-dimensional information of the workpiece to be recognized. In the next stage, a three-dimensional model is set at a predetermined reference position in the three-dimensional coordinate system, and the positional deviation amount of the workpiece with respect to the three-dimensional model is verified by comparing this three-dimensional model with the restored three-dimensional information. Then, the rotation angle is recognized (ST7).

  In the matching process in ST7, the intersection of each three-dimensional segment is used as a feature point, and each feature point is associated with the brute force formula, and the degree of coincidence of each three-dimensional segment when the association is performed is calculated. The correspondence when the maximum is reached is specified as the correct correspondence.

  ST7 will be described in a little more detail. In this embodiment, each of the feature points of the three-dimensional information to be collated is sequentially associated, and for each association, the shift amount and the rotation angle necessary to move the feature point on the 3D model side to the corresponding point are calculated. To do. All of these are calculated for each of the X, Y, and Z axes. Next, all the coordinates included in the three-dimensional model are converted based on the calculated shift amount and rotation angle, and the degree of coincidence between the converted three-dimensional model and the three-dimensional information to be collated is calculated.

  By executing the above processing for all feature points on the three-dimensional model side, the feature points are associated with the brute force formula, and the degree of coincidence can be obtained for each association. Thereafter, the shift amount and the rotation angle used for the coordinate conversion when the highest degree of coincidence is finally obtained are recognized as the positional deviation amount and the rotation angle of the recognition target workpiece with respect to the three-dimensional model.

  Thereafter, the recognized positional deviation amount and rotation angle are output to the robot controller 3 (ST8), and the process is terminated.

(3) Three-dimensional model creation process 3-1) Principle description In order to execute the above-described three-dimensional recognition process, it is necessary to register a three-dimensional model of the workpiece W in the memory 25 in advance. In the object recognition apparatus of this embodiment, a three-dimensional model is created by a method in which a real model of a workpiece W is measured in stereo from various directions and three-dimensional information restored from each measurement result is integrated. Hereinafter, this process will be described in detail.
In the following drawings, an actual model of the workpiece W is indicated by a symbol WM, and in the specification, the actual model WM is referred to as a “work model WM”.

  In this embodiment, as shown in FIG. 4A, the position and the optical axis direction of the stereo camera 1 are fixed, and the stereo measurement is performed a plurality of times while changing the posture of the work model WM with respect to the stereo camera 1. .

  In the figure, a three-dimensional coordinate system based on the X, Y, and Z axes is a coordinate system for calculating three-dimensional coordinates (hereinafter referred to as “measurement coordinate system”), and is uniquely determined for the stereo camera 1. It is done. On the other hand, the three-dimensional coordinate system based on the X1, Y1, and Z1 axes is a coordinate system uniquely defined for the work model WM (hereinafter referred to as “work coordinate system”). In FIG. 4A, the X1, Y1, and Z1 axes are parallel to the X, Y, and Z axes, respectively. However, when the posture of the work model WM changes, the direction of each axis in the work coordinate system also changes. It changes according to.

  Although the posture of the work model WM can be freely determined by the user, in this embodiment, a specific surface along the X1Z1 plane is always in contact with the work support surface (XZ plane in this example), and is orthogonal to the XZ plane. It is assumed that the work model WM is rotated once with respect to the Y1 axis, and imaging is performed a plurality of times during that time. Since the work model WM is rotated by a method of changing the orientation of the work model WM with the user's hand without using the rotation table, not only the rotation angle but also the position of the work model WM with respect to the measurement coordinate system is determined. Every time, the position is shifted.

FIG. 4B shows a change in the positional relationship of the stereo camera 1 with respect to the work model WM with reference to the work coordinate system. In the figure, rectangles with numbers 1 to 12 indicate the location of the stereo camera 1 at the time of imaging, and the numbers in the rectangles indicate the order of imaging.
Any stereo image generated by imaging at these positions is subject to three-dimensional measurement. Therefore, hereinafter, the position of the stereo camera represented by each rectangle is referred to as a “measurement point”. When individually referring to each measurement point, a number representing the order of measurement is quoted, such as “measurement point [1]” and “measurement point [2]”.

  In this embodiment, the same processing as ST1 to ST6 in FIG. 3 is executed for each of the 12 measurement points to restore the three-dimensional information. Further, at each measurement point excluding the measurement point [1], the three-dimensional information restored at the measurement point is collated with the three-dimensional information of the previous measurement point, so that the position relative to the previous three-dimensional information is obtained. Recognize deviation and rotation angle. These recognition processes are all performed by the same method as that performed in ST7 of FIG.

  For the last measurement point [12], in addition to the displacement amount and rotation angle for the three-dimensional information of the measurement point [11], the displacement amount and rotation angle for the three-dimensional information of the measurement point [1] are also obtained. . Alternatively, the measurement point [12] is transferred to the measurement point [1] again, the measurement is re-executed, and the positional deviation amount and the rotation angle with respect to the three-dimensional information of the measurement point [12] of the restored three-dimensional information are obtained. May be.

  Furthermore, in this embodiment, the accuracy of the three-dimensional information restored for each measurement point is confirmed, two or more three-dimensional information confirmed to be good is selected, and the selected information is aligned and integrated. Like to do. Details of these processes will be described later.

  Next, in this embodiment, stereo measurement is performed by attaching a triangular mark M to an appropriate position on the surface of the work model WM. This mark M is configured as a seal and is affixed to a flat surface of the work model WM (in this embodiment, the center of the upper surface). Each measurement point is determined on condition that the mark M is included in the visual field of all the cameras 11 to 13. Therefore, the three-dimensional information including the three-dimensional segment representing each side of the mark M is restored at any measurement point.

  FIG. 5 shows three of the three-dimensional information (indicated by a, b, and c in the figure) among the three-dimensional information restored at the 12 measurement points for the work model WM with the mark M. A processing procedure for creating a dimensional model is schematically shown. Note that the three-dimensional information a in the figure is restored at the measurement point [12] in FIG. The three-dimensional information b is restored at the measurement point [3], and the three-dimensional information c is restored at the measurement point [6]. In addition, the information m of the mark M is included in any three-dimensional information (however, the substantial content of the information m differs depending on the measurement point. The same applies to information m1 and m2 described later).

  In this example, on the basis of the three-dimensional information a, the other three-dimensional information b and c are coordinate-transformed so as to eliminate the positional deviation and the angular deviation with respect to the reference three-dimensional information a. Then, by integrating the converted three-dimensional information b ′ and c ′ with the reference three-dimensional information a, three-dimensional information d including the characteristics of each information is created. Further, by deleting the information m corresponding to the triangular mark M from the integrated three-dimensional information d, three-dimensional information e representing only the three-dimensional shape of the work model WM is created. Set to model.

  The coordinate conversion process of the above three-dimensional information is performed using the positional deviation amount and the rotation angle previously obtained for each measurement point. For example, in order to align the 3D information b restored at the measurement point [3] with the 3D information a restored at the measurement point [12], the measurement points [3] and [2], the measurement point [2], [1] Coordinate conversion is performed using the positional deviation amount and the rotation angle obtained for each combination of the measurement points [1] and [12]. Similarly, the three-dimensional information c restored at the measurement point [6] is calculated in the range from the measurement point [6] to the measurement point [12] (in this case, either clockwise or counterclockwise may be used). By performing coordinate conversion using the amount of misalignment and the rotation angle, alignment with the three-dimensional information a is performed.

  As described above, when obtaining the positional deviation amount and the rotation angle between the measurement points separated by a distance, the measurement points between them are combined for each adjacent one, and the three-dimensional information for each set of measurement points. The reason why the amount of misalignment and the rotation angle are calculated is that collation between pieces of information including as many common features as possible can ensure the accuracy of collation. However, if the accuracy of the collation process can be sufficiently secured by the function of the mark M described below, the three-dimensional information b and c are directly collated with the three-dimensional information a to calculate parameters necessary for the coordinate conversion process. May be.

Next, the reason for attaching the triangular mark M to the work model WM will be described.
The triangle represented by the mark M used in this embodiment is set so that the ratio of each side becomes 6: 7: 8 as shown in FIG. According to this ratio, the vertices A, B, and C of the mark M can be easily specified. For example, the direction of the mark M can be uniquely specified by the direction of the perpendicular from the vertex A to the side BC. it can. In addition, as shown in FIG. 7, the alignment accuracy can be ensured even when aligning with the triangle A′B′C ′ having the same shape.

  In FIG. 7 (1), a result of selecting one vertex from each of the triangle ABC and the congruent triangle A′B′C ′ and aligning the triangles based on the selected vertex. Indicates. As shown in this figure, triangles do not completely overlap unless vertices in the correct correspondence are combined.

  Regarding the correspondence of the above triangles, the inventors actually calculated the degree of coincidence of the entire triangle for each vertex combination. FIG. 7 (2) is a graph showing the results. As shown in this graph, when the degree of coincidence when the vertices are correctly associated with each other is 1, the degree of coincidence due to other correspondences is about 0.4 at the maximum. As a result, it was found that the correctness of the correspondence between triangles can be correctly determined.

  As described above, according to the triangle having the shape shown in FIG. 6, there is a large difference in the degree of coincidence between the case where the vertices of the triangle are correctly associated and the case where the correspondence is incorrect. Even if each side of the triangle is associated as a three-dimensional segment, the same effect should be obtained. Therefore, when the entire three-dimensional information including the three-dimensional segment of the mark M is collated, each triangle The degree of coincidence regarding the three-dimensional segment of the side greatly affects the degree of coincidence of the whole.

  That is, even if there is an element in the original three-dimensional segment of the work model WM that causes erroneous alignment, if the correspondence is performed incorrectly, the matching degree of the triangular three-dimensional segment is By greatly decreasing, the degree of coincidence of the entire three-dimensional information is lowered. On the other hand, when each three-dimensional segment including a triangular three-dimensional segment is correctly aligned, the degree of coincidence increases in any three-dimensional segment, and the degree of coincidence of the entire three-dimensional information also becomes a high value. Therefore, the degree of coincidence when correctly associated can be made larger than in other cases, and the alignment accuracy can be ensured.

  Note that the shape of the triangle indicated by the mark M is not limited to the example of FIG. 6, and the ratio of the sides is not limited as long as the above-described effect can be obtained. Further, the mark is not limited to a triangle, and a polygonal mark that has four or more vertices and whose direction can be uniquely specified may be created.

  Further, when the work model WM is opaque and the mark M cannot be seen through from the side of the surface facing the sticking surface of the mark M, the mark M may be constituted by a line symmetrical figure such as an isosceles triangle. Good. In other words, it is a figure that does not have rotational symmetry, so that there is a significant difference between the degree of coincidence when matching the same figure without positional deviation and angular deviation and the degree of coincidence in other associations. It can be considered that the designed figure can be used for the purpose of ensuring the alignment accuracy of the entire three-dimensional information.

Next, the reason for applying the mark M to the flat surface will be described.
When the mark M is pasted on the curved surface, each side of the triangle is more likely to be recognized as a three-dimensional segment of the curve, and the shape of the mark M to be restored varies depending on the curvature of the curved surface. Therefore, when the mark M is attached to the curved surface, even if the three-dimensional information can be correctly restored, the restored three-dimensional information is likely to represent a shape different from the original triangular shape. Therefore, it is difficult to stably perform the verification process of the three-dimensional information by collation with the model triangle described below.

  Next, in order to restore the three-dimensional information of the mark M with high accuracy, it is necessary to ensure that the edge of each side of the mark M is detected. From this point of view, in this embodiment, the surface of the mark M is colored in a substantially uniform color without drawing a pattern or the like, and as shown in FIG. M is prepared. In this way, if the color of the mark M is changed in accordance with the color of the workpiece that is the background, the contrast between the mark M and the background portion can be increased and the edge detection accuracy can be increased.

  Furthermore, as shown in FIG. 9, when there is a pattern pn that causes an error in correspondence on the surface of the work model WM, a seal S in which the background BK and the mark M with uniform colors are integrated is provided. It is good to make and stick this sticker S so that pattern pn may be hidden.

3-2) Processing for ensuring accuracy of three-dimensional model Furthermore, in this embodiment, the following processing is executed to ensure the accuracy of the three-dimensional model.

[Feature point completion processing]
As shown in FIGS. 8 and 9, even if the mark M taking into account the color and pattern of the work model WM is attached, a part of the mark M becomes unclear due to the influence of shading or the like depending on the imaging direction. In some cases, the three-dimensional segment of the mark M cannot be sufficiently restored. In particular, if the coordinates of vertices serving as a reference for collation of three-dimensional information are missing, it may be difficult to improve the alignment accuracy by the triangle.

  In view of the above problem, in this embodiment, a three-dimensional image corresponding to a lost vertex is obtained by performing processing as shown in FIG. 10 (1) on a straight line segment among the restored three-dimensional segments. The coordinates are complemented. In FIG. 10, each three-dimensional segment is shown in an enlarged manner in order to clarify the processing content.

  In the example of FIG. 10 (1), an end edge that is not connected is extended in a range up to a predetermined upper limit value for a straight line segment that has an end edge that is not connected to another segment. When the extended portion (for example, the extended portion of the segment L1 in the figure) intersects with another three-dimensional segment L2 or a portion obtained by similarly extending the segment L2, the intersection P is used as a feature point. The three-dimensional coordinates are calculated.

  Further, considering that there is a possibility that segments to be connected do not intersect due to errors in measurement parameters, etc., feature points are identified by a method as shown in FIG. . In this example, when two straight line segments L1 and L2 (both including an extended portion) intersect, the distance D between the segments L1 and L2 at the intersection is calculated. If the distance D is within a predetermined threshold, the three-dimensional coordinates are calculated using the midpoint Q between the corresponding segments as a feature point. The upper limit value of the extension width of the straight segment and the threshold value to be compared with the distance D between the segments are determined in advance based on the size of the mark M and an assumed measurement error.

  10 (1) and 10 (2), even if a part of the triangle corresponding to the mark M disappears from the image, the coordinates of the disappeared vertex of the triangle can be complemented. Thus, the effect of the degree of matching of the triangles on the degree of matching of the entire three-dimensional information is guaranteed. In addition to the vertices of a triangle, these coordinates can be complemented even when the main feature points are not restored due to noise, measurement errors, etc., so that accuracy is ensured when matching three-dimensional information. be able to.

[Verification of 3D information]
As described above, in this embodiment, the work model WM is measured from various directions, but the three-dimensional information cannot be accurately restored at any measurement point. Depending on the orientation of the work model WM, the accuracy of the three-dimensional information may deteriorate due to a large reflection of measurement parameter errors or the generation of an image that includes many misrecognized features.

  In this embodiment, since the user determines the position of the work model WM in each stereo measurement, measurement may be performed in a state where the mark M is not included in the field of view of each camera due to a user's mistake. In the three-dimensional information that does not include the information of the mark M, there is a possibility that the alignment accuracy may deteriorate, so it is necessary to exclude it from the integration target in advance.

  From this point, in this embodiment, it is determined whether or not the three-dimensional information of the mark M has been successfully restored for each measurement point, and the three-dimensional information determined to be good is used for the integration process. Yes.

  Specifically, in the determination process, as shown in FIG. 11A, when information close to an actual triangle is restored, the entire three-dimensional information including this information is considered to be a good integration target. . On the other hand, when the reproducibility of the triangle is poor, such as when the side is curved as shown in FIG. 11B or when the vertex cannot be correctly restored as shown in FIG. It is judged that the whole is not good and is excluded from integration.

  Furthermore, in this embodiment, when it is determined that the three-dimensional information is not good, the three-dimensional information is discarded, and measures such as performing another measurement or changing a measurement point are taken. Further, if good 3D information cannot be obtained even if measurement is repeated a predetermined number of times, the initial setting processing such as position adjustment of the stereo camera 1 and re-execution of 3D calibration is performed again. Such processing ensures the accuracy of the three-dimensional information used for the integration, and prevents the positional deviation amount and the rotation angle necessary for the coordinate conversion during the integration from being calculated by the inaccurate three-dimensional information. Thus, the accuracy of the three-dimensional model can be ensured.

  Specifically, the above determination process is performed using a combination of three-dimensional model data representing the geometric information of the mark M, specifically, data representing a preferred three-dimensional segment of three sides of the triangle ( This model data is hereinafter referred to as “model triangle”). That is, the three-dimensional information of each measurement point is collated with the model triangle to obtain the degree of coincidence with the model triangle. This matching is also performed in the same manner as in the case of matching with the three-dimensional model, that is, when each feature point of the model triangle is associated with each feature point in the three-dimensional information with a brute force formula and the degree of coincidence with the model triangle is the highest. This is done by a method of determining the correspondence of.

  After the above collation, the calculated degree of coincidence is compared with a predetermined threshold, and if the degree of coincidence exceeds the threshold, it is determined that the three-dimensional information is good, and the degree of coincidence is the threshold. If the following, the corresponding three-dimensional information is deleted.

  Next, in this embodiment, not all the three-dimensional information determined to be good are integrated, but the three-dimensional information to be integrated is further selected from among them. This selection is made based on the user's judgment when all the measurement processes are completed.

FIG. 12 shows an example of a screen for selecting three-dimensional information used for integration.
On this screen, a window 30 for imaging and displaying three-dimensional information, two selection buttons 31 and 32 of “add to model” and “not add” are provided. In the window 30, an outline based on the three-dimensional information of the selection candidate is displayed, and an outline M0 of the model triangle is displayed at a location determined to correspond to the model triangle based on the previous matching result. ing. Further, in this window 30, the degree of coincidence of the three-dimensional information with respect to the model triangle is also displayed.

  When the user confirms the above display and determines that the three-dimensional information being displayed should be added to the three-dimensional model, the user operates the “add to model” button 31. Thereby, the three-dimensional information being displayed is selected as an integration target. On the other hand, when the “do not add” button 32 is operated, the three-dimensional information being displayed is excluded from integration targets. However, as will be described later, this three-dimensional information may be additionally registered in the three-dimensional model for the purpose of improving the accuracy of the integrated three-dimensional information.

  In the above description, the accuracy of restoration of the three-dimensional information of the mark is automatically determined by comparing the degree of coincidence with the model triangle with a threshold value. Instead, a screen similar to FIG. 12 is presented. Then, a determination input by the user may be accepted as to whether or not the three-dimensional information of the mark is good. Further, this determination input and selection as to whether or not to add to the model may be received on the same screen.

3-3) Procedure for Creation / Registration of 3D Model FIG. 13 is a flowchart summarizing a procedure for processing from stereo measurement to registration of a 3D model. The procedure for creating the three-dimensional model of this embodiment will be described below along the flow of this flowchart.

  In this embodiment, since the positioning of the work model WM at the time of measurement is left to the discretion of the user, preview images from the respective cameras 11 to 13 are displayed on the display unit 27 during the measurement process. When the user confirms whether or not the work model WM and the mark M are correctly captured by the cameras 11 to 13 on this display screen and performs a measurement instruction operation, the step of “stereo measurement” in FIG. ST101) is executed. In step ST101, specifically, processing from stereo imaging to restoration of three-dimensional information (similar to ST1 to ST6 in FIG. 3) is executed.

  Next, the restored three-dimensional information is collated with a model triangle to determine the accuracy of the three-dimensional information (ST102). As described above, the determination here can be performed using an automatic determination based on the degree of coincidence or a method of accepting a user's determination on the display screen of the matching result.

  If it is determined that the three-dimensional information is not good (ST103 is “NO”), the restored three-dimensional information is deleted, and an error message or the like is displayed on the display unit 27. In response to this, the user repeats the measurement instruction, for example, by changing the position of the work model WM.

  When it is determined that the restored three-dimensional information is good (“YES” in ST103), the positional deviation and the rotation angle with respect to the three-dimensional information one stage before this three-dimensional information are further recognized (ST105). . Further, for the rotation angle, the rotation angle for the three-dimensional information restored first is calculated by a method of adding the recognized value every time (ST106). However, when the three-dimensional information is restored first (ST104 is “YES”), the above steps of ST105 and ST106 are skipped.

  Next, in this embodiment, when the rotation angle with respect to the first three-dimensional information exceeds 360 degrees, it is determined that the work model WM has made one rotation with respect to the stereo camera (ST107). Further, the processes of ST101 to ST107 are repeated until it is determined that the work model WM has made one rotation.

  If it is determined that the work model WM has made one rotation with respect to the first three-dimensional information, the loop of ST101 to ST107 is terminated, and the screen shown in FIG. 12 is displayed to accept selection of the integration target three-dimensional information ( ST108). Then, using one of the selected three-dimensional information as a reference, the other three-dimensional information is aligned with the reference three-dimensional information (ST109), and each three-dimensional information after the alignment is integrated (ST110).

  Note that the process of selecting the integration target three-dimensional information may be automatically performed without depending on the user. For example, based on the rotation angle recognized in ST106 accompanying each measurement, the three-dimensional information restored by the measurement when the work model WM is rotated by a predetermined angle or more can be selected as the integration target.

  FIG. 14 schematically illustrates an example in which four measurement points 3, 6, 9, and 12 are selected from the above-described twelve measurement points and integrated. As in this example, a well-balanced three-dimensional model can be created by selecting the measurement direction to be integrated in consideration of not biasing to a specific range from the entire range surrounding the work model WM. .

  When the integration process ends, information corresponding to the mark M is deleted from the integrated three-dimensional information (ST111). This is because the mark M is not attached to the workpiece W to be actually recognized, and if the information of the mark M is included in the three-dimensional model, the collation accuracy may be lowered. However, here, the information of the mark M may be invalidated so as not to be a collation target without being deleted.

  Next, as illustrated in FIG. 14, even if the three-dimensional information to be integrated is selected so as not to be biased toward measurement by a specific range, depending on the shape of the work model WM, There are cases where recognition by integrated three-dimensional information becomes impossible even if the posture is slightly changed. In view of this point, in this embodiment, after temporarily registering the three-dimensional information from which the information of the mark M has been deleted as a three-dimensional model (ST112), the accuracy of the three-dimensional model is further improved by the following processing. Yes.

  Specifically, three-dimensional information that has not been selected as an integration target by the temporarily registered three-dimensional model is collated in order to calculate a positional deviation amount and a rotation angle with respect to the three-dimensional model. Then, it is determined whether or not correct recognition has been performed by comparing with a value derived from the positional deviation amount and the rotation angle obtained for each measurement point (ST113, 114). Here, when it is determined that the three-dimensional information to be collated has not been correctly recognized (ST114 is “NO”), the three-dimensional information used for the collation is aligned with the three-dimensional model to obtain a three-dimensional model. Add (ST115). On the other hand, when the three-dimensional information to be collated can be correctly recognized (“YES” in ST114), the process proceeds to the next without adding the three-dimensional information to the three-dimensional model. Note that the above alignment is performed by coordinate conversion based on the amount of displacement and the rotation angle obtained for each measurement point.

  In this way, the three-dimensional information excluded from the integration is sequentially targeted, it is determined whether the three-dimensional information can be correctly recognized by the integrated information, and the three-dimensional information that has been determined to have an error in recognition is determined as 3 Add to a dimensional model. As a result of such processing, 3D information that could not be recognized by the model is added to the temporarily registered 3D model, so that the same erroneous recognition does not occur in the subsequent 3D model. Accuracy can be improved. On the other hand, since the correctly recognized 3D information is not added to the 3D model, it is possible to prevent the data amount of the 3D model from increasing.

  Thereafter, in order to further improve the accuracy of the three-dimensional model, a process of removing noise is executed (ST117). For example, among the three-dimensional segments constituting the three-dimensional model, those having a length equal to or smaller than a predetermined threshold and those having a three-dimensional coordinate sample point equal to or smaller than a predetermined number are deleted. Further, a shadow corresponding to a shadow is detected from the distribution state of the three-dimensional coordinates on the XZ plane, and this is deleted. In addition, it is checked whether or not there is an extension part of the straight line segment that exceeds the set upper limit value. If there is an extension part that exceeds the upper limit value, the extension part is deleted together with the set intersection point.

  In this way, by deleting short 3D segments, 3D segments with a small number of sample points, 3D segments representing shadows, 3D segments with incorrect extension, etc., incorrect matching is performed in the matching process. Can be prevented. Further, since the capacity of the three-dimensional model is reduced, the time for the collation process can be shortened.

  When the noise removal is finished, the three-dimensional model after the noise removal is fully registered (ST118), and the process is finished.

  According to the above processing, the three-dimensional information obtained immediately before is obtained by collating the three-dimensional information including the information of the triangular mark M every hour while changing the relationship between the work model WM and the stereo camera 1 little by little. Since the positional deviation amount and the rotation angle with respect to are accurately obtained, it is possible to ensure the accuracy of alignment during the integration process. Therefore, a highly accurate three-dimensional model can be created from the three-dimensional information restored by measurement from various directions.

  Further, according to the above-described embodiment, it is necessary to perform stereo imaging in a state where the same mark M is included in the field of view of each camera 11 to 13, but the alignment accuracy can be ensured by the effect of the mark M. Further, since it is not necessary to strictly adjust the rotation angle of the work model WM, facilities such as a rotary table are not required. Further, since the accurate three-dimensional model can be created by moving the work model WM by an appropriate angle, the burden on the user can be reduced.

3-4) Embodiment using a plurality of marks M In the above-described embodiment, a specific surface along the X1Z1 plane of the workpiece model WM is always in contact with the XZ plane of the measurement coordinate system so that the Y1 axis of the workpiece coordinate system By rotating the workpiece model WM, a three-dimensional model representing the overall shape of the contour line of the workpiece W was created. The WM may be rotated. In this case, it is difficult to cause the cameras 11 to 13 to always capture the mark M, but this problem can be solved by attaching a plurality of marks M to the work model WM as in the embodiment described below. can be solved.

  FIG. 15 shows an example of measurement processing when a three-dimensional model is created using a work model WM with two marks M1 and M2. As shown in FIG. 15 (1), one mark M1 is attached at a position that becomes the upper surface when the work model WM is arranged so that the Y1 axis of the work coordinate system is parallel to the Y axis, and the other mark M2 is attached to the side surface of the work model WM. Each of the marks M1 and M2 is a triangle in which the ratio of each side is set as shown in FIG. 6, but the size is appropriately changed according to the size of the respective sticking surface. .

  In this embodiment, as in the example shown in FIG. 4, the measurement is performed a plurality of times while rotating the work model WM with respect to the Y1 axis (FIG. 15 (1)). When a series of measurements is completed, the rotation axis is changed from the Y1 axis to the X1 axis, and the work model WM is rotated 90 degrees as shown in FIG. Further, as shown in FIG. 15 (3), the work model WM is rotated again by 90 degrees to perform measurement.

  When measurement is performed by rotating the work model WM with respect to the Y1 axis, the mark M1 is always included in the field of view of each camera 11 to 13, and measurement is performed by rotating the work model WM with respect to the X1 axis. When performing, the mark M2 is always included in the visual field of each camera 11-13. Further, at least when shifting from the rotation about the Y1 axis to the rotation about the X1 axis, measurement is performed by setting a state in which the two marks M1 and M2 are both included in the fields of view of the cameras 11-13.

FIG. 16 shows an example of integrating the three-dimensional information restored by the above measurement process.
In the figure, a, b, and c are three-dimensional information restored while rotating the work model WM about the Y1 axis, as in the example of FIG. Among these three-dimensional information, the three-dimensional information a includes information m1 and m2 of two marks M1 and M2. The other three-dimensional information b and c includes the information m1 of the mark M1, but does not include the information of the mark M2.

  Next, g in the figure is three-dimensional information restored by measurement when the work model WM is arranged in the state of FIG. Further, h is three-dimensional information restored by measurement when the work model WM is arranged in the state of FIG. The three-dimensional information g and h includes information m2 corresponding to the mark M2, but does not include information corresponding to the mark M1.

  In this embodiment, the same integration process as in the example of FIG. 5 is performed for the three-dimensional information a, b, and c. That is, with the three-dimensional information a as a reference, coordinate conversion is performed so that the three-dimensional information b and c are aligned with the reference three-dimensional information a, and then the converted three-dimensional information b ′ and c ′ and the reference three-dimensional Information d is created by integrating information a. Since the three-dimensional information a, b, c includes information m1 corresponding to the mark M1, the accuracy of the alignment process can be ensured by this information m1.

  Next, in this embodiment, coordinate conversion for aligning with the three-dimensional information a is also performed on the three-dimensional information g and h restored by the method of performing measurement by rotating the work model WM with respect to the X1 axis. . Then, information i is created by integrating the converted three-dimensional information g ′, h ′ and the three-dimensional information a. Since the three-dimensional information a, g, h includes information m2 corresponding to the mark M2, respectively, the accuracy of the alignment process can be ensured by this information m2.

  Thereafter, by integrating the integrated information d created by the three-dimensional information a, b, c and the integrated information i created by the three-dimensional information a, g, h, the contour of the entire surface of the work model WM is obtained. Three-dimensional information j representing the shape and the contour shape of the marks M1 and M2 is created. Further, by deleting the information m1 and m2 corresponding to the marks M1 and M2 from the three-dimensional information j, three-dimensional information k representing only the entire contour shape of the work model WM is created, and this is determined as the three-dimensional model. To do.

  In the above processing, since the integrated information d and i are both created by alignment based on the three-dimensional information a, theoretically, there is no positional deviation or rotation between them. Can be easily integrated. Further, even if there is a slight deviation between the information d and i due to the influence of a measurement error or the like, it is possible to perform accurate alignment with the information m1 and m2 of the marks M1 and M2 included in both of them. it can.

  Even when the work model WM is rotated with respect to the X1 axis, every time the work model WM is rotated by 90 degrees, an axis orthogonal to the XZ plane under the state (in the case of FIG. 15 (2), the Z1 axis, FIG. In the case of 3), by rotating the work model WM relative to the Y1 axis), the posture of the work model WM may be changed in a plurality of ways, and measurement may be performed each time the change is made. Also in this case, a predetermined number (two or more) of a plurality of three-dimensional information restored by each measurement can be selected as an integration target. Further, the work model WM may be arranged so that the surface with the mark M2 is the upper surface, and the posture may be changed variously in the same manner for measurement.

  Also in this embodiment, the three-dimensional model k created by the integration process is collated with the three-dimensional information that has not been adopted for the integration process, and it is determined whether correct recognition is performed. Can perform processing for newly adding the three-dimensional information to be collated to the three-dimensional model.

  Furthermore, the number of marks attached to the work model WM is not limited to one or two, and three or more marks may be attached. In this case as well, if any one mark is used as a reference and measurement is performed by setting at least one of the other marks in the field of view of each camera 11 to 13 together with the reference mark, two or more The three-dimensional information to be integrated can be aligned through the three-dimensional information including the mark information. In addition, the alignment accuracy can be ensured.

It is a figure which shows schematic structure of the picking system to which a three-dimensional recognition process is applied. It is a block diagram of an object recognition apparatus. It is a flowchart which shows the procedure of a three-dimensional recognition process. It is a figure which shows the measuring method of a work model. It is a figure which shows the general | schematic procedure of the creation process of a three-dimensional model. It is a figure showing the characteristic of the triangle which a mark represents. It is a figure showing a plurality of correspondence states of each vertex of a triangle, and a graph showing a degree of coincidence. It is a figure which shows the kind of mark. It is a figure which shows the mark for the workpiece | work with a repeating structure. It is a figure which shows the complementation processing method of a feature point. It is a figure which shows the example of the three-dimensional information of the mark used for selection of the integration target three-dimensional information. It is a figure which shows the example of the screen for selecting the three-dimensional information used for integration. It is a flowchart which shows the procedure regarding preparation and registration of a three-dimensional model. It is a figure which shows the example of selection of the three-dimensional information of integration object. It is a figure which shows the example of the measuring method by the work model which attached two marks. It is a figure which shows the general | schematic procedure in the case of producing a three-dimensional model using the three-dimensional information decompress | restored by the measuring method of FIG. It is a figure explaining the cause which an error produces in matching of three-dimensional information. It is a figure explaining the other cause which an error produces in matching of three-dimensional information.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Stereo camera 2 Object recognition apparatus 11, 12, 13 Camera 24 CPU
25 memory W work WM work model M mark m, m1, m2 mark three-dimensional information a, b, c, g, h restored three-dimensional information d, j integrated three-dimensional information e, k three-dimensional model

Claims (7)

In order to recognize the position and orientation of an object having a predetermined shape from the three-dimensional information restored by three-dimensional measurement using a stereo camera, the three-dimensional model of the object is obtained using the result of the three-dimensional measurement on the real model of the object. A method of creating,
A polygonal mark having no rotational symmetry is attached to a predetermined position on the surface of the real model, and the condition that the mark is included in the field of view of all cameras constituting the stereo camera is maintained. Change the positional relationship between the camera and the real model every time and execute multiple imaging,
The feature of the edge is detected from the stereo image generated by each imaging, and the three-dimensional information representing the feature of the real model and the outline of the mark is restored by obtaining the three-dimensional coordinates of the detected feature.
Two or more three-dimensional information corresponding to all or a part of the plurality of three-dimensional information restored by the plurality of times of imaging is set as an integration target, and each of the three-dimensional information based on the correspondence relationship between the integration target three-dimensional information. After aligning the 3D information to be integrated,
Deleting or invalidating information corresponding to the mark from the integrated three-dimensional information, and determining the three-dimensional information after the processing as the three-dimensional model;
A method for creating a three-dimensional model. The method of claim 1, wherein
A method for creating a three-dimensional model in which the mark is attached to a flat portion of the surface of the real model. The method of claim 1, wherein
A method for creating a three-dimensional model, wherein the mark is formed integrally with a background portion having a color different from that of the mark, and the integrated object is attached to the surface of the real model. In the method as described in any one of Claims 1-3,
Register in advance three-dimensional feature data representing the geometric features of the mark,
Prior to the process of integrating the three-dimensional information, the three-dimensional information restored from the stereo image generated by each imaging is collated with the three-dimensional feature data of the mark, so that the mark has a good geometric feature. A method for creating a three-dimensional model, wherein it is determined whether or not restoration has been performed, and the three-dimensional information that has been determined that the geometric feature has not been successfully restored is excluded from the target of integration processing. In the method as described in any one of Claims 1-3,
In the three-dimensional measurement, an edge is detected from an image generated by each camera, the detected edge is divided into a plurality of two-dimensional segments, and each two-dimensional segment is associated with each other between the images. A step of calculating a plurality of three-dimensional coordinates for each segment set, setting a three-dimensional segment of a straight line or a curve based on the distribution state of the calculated three-dimensional coordinates, and setting an intersection of each three-dimensional segment as a feature point Step and execute
In the step of setting the feature point, the edge of a straight three-dimensional segment having an edge that does not intersect with another segment is extended to a predetermined length, and the extended portion is another three-dimensional segment or an extension thereof. When intersecting parts or intersecting at a distance within a predetermined value, set feature points based on the intersection or intersection position,
In the alignment process before the integration of the three-dimensional information, one of the three-dimensional information to be integrated is used as a reference, and the other three-dimensional information is the most consistent with the reference three-dimensional information in each feature point and three-dimensional segment. A method for creating a three-dimensional model in which coordinates are transformed so that a state is obtained. The method of claim 1, wherein
Attach the first and second marks with polygons that do not have rotational symmetry to different locations on the surface of the real model, and for each mark, there are multiple states in which the mark is included in the field of view of all cameras. And adjusting the positional relationship between each camera and the real model in the plurality of times of imaging so that the first and second marks are both included in the field of view of each camera at least once.
3D information restored from a stereo image when only the first mark is included in the field of view of each camera and 3D information restored from a stereo image when only the second mark is included in the field of view of each camera The three-dimensional information is integrated after aligning the three-dimensional information with the three-dimensional information restored from the stereo image when the first and second marks are included in the field of view of each camera. How to create a dimensional model. Image input means for inputting a stereo image of a recognition object generated by a stereo camera, and edge features are detected from the input stereo image, and three-dimensional measurement is performed on the detected features to restore three-dimensional information. A three-dimensional measuring means for recognizing, a recognition processing means for recognizing the position and orientation of the recognition object by collating the restored three-dimensional information with a pre-registered three-dimensional model, and the three-dimensional used by the recognition processing means A device comprising a three-dimensional model registration means for creating and registering a model,
The three-dimensional model registration means includes:
Mark registration means for registering three-dimensional feature data representing a geometric feature of a polygonal mark having no rotational symmetry;
The stereo image generated by each imaging is received from the image input unit on the premise that the real model with the polygonal mark attached to the surface is imaged multiple times with the positional relationship to each camera being changed each time. Measurement control means to be processed by the three-dimensional measurement means,
Whether or not the geometric feature of the mark has been successfully restored by comparing the 3D information restored from the stereo image by the processing of the measurement control means with the 3D feature data registered in the mark registration means Determination means for determining whether or not
Among the plurality of three-dimensional information restored by the plurality of times of imaging, two or more three-dimensional information determined that the geometric feature of the mark is well restored is targeted. 3D information integration means for integrating after aligning the dimensional information based on the corresponding relationship;
Model deciding means for erasing or invalidating information corresponding to the mark in the three-dimensional information after integration, and confirming the three-dimensional information after processing as a three-dimensional model to be registered;
An object recognition device comprising:

Images