JP2010113398A - Method of creating three-dimensional model and object recognizing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To create a three-dimensional model of satisfactory accuracy by reducing capacity as much as possible. <P>SOLUTION: Three-dimensional measurement by a stereo camera is performed from a plurality of directions to an actual model WM of a work to be three-dimensionally recognized. Then a temporary three-dimensional model is created by aligning and integrating a predetermined number of pieces of three-dimensional information restored by each measurement (three-dimensional information of each measurement point of [12], [3], [6], and [9]). Further recognition processing by the temporary three-dimensional model is performed to the three-dimensional information which is not integrated and after carrying out alignment of the three-dimensional information (information on a measurement point [10]) of which recognition result is determined as not correct, to the temporary three-dimensional model. Finally, registration of the three-dimensional model to which all pieces of the three-dimensional information which have not been correctly recognized are added is formally carried out. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  In order to perform processing for recognizing the position and orientation of an object using three-dimensional information restored by stereo measurement using a plurality of cameras for an object having a predetermined shape, The present invention relates to a method for creating a three-dimensional model. Furthermore, the present invention relates to an apparatus for performing object recognition processing using the model after creating and registering the three-dimensional model.

In order to recognize the position and orientation of an object by three-dimensional recognition processing using a stereo camera, it is necessary to create a three-dimensional model including three-dimensional information of various surfaces of the object to be recognized.
The following Patent Document 1 discloses an invention that addresses the above-described problems. In Patent Document 1, stereo measurement from a plurality of directions is performed on a real model of a recognition target object, and the three-dimensional information restored by each measurement is aligned and integrated to obtain a geometric model of the entire object. Is described.

  Patent Document 1 describes that three-dimensional information of a contour line of a real model is restored from an edge represented in a stereo image by a technique called “segment-based stereo”. According to Patent Document 1, “segment-based stereo” is a method of dividing edges in a stereo image into units called “segments” and performing stereo correspondence search on a segment basis to restore the three-dimensional information of the contour line. Refers to processing.

  Furthermore, Patent Document 1 describes a method of performing coordinate conversion based on a rotation angle by rotating a real model to a predetermined angle using a rotation table with respect to alignment of three-dimensional information. In addition, as a method of aligning 3D information from unknown observation directions, a plurality of alignment candidates are obtained by collating the model being created with the newly restored 3D contour, It is described that the alignment is specified when the degree of coincidence is the highest.

  Further, the following Non-Patent Document 1 describes in detail a technique for associating a two-dimensional segment between images and restoring three-dimensional information. Non-Patent Document 2 discloses a method for recognizing the position and orientation of an object by collating restored three-dimensional information with a pre-registered three-dimensional model.

Japanese Patent No. 2961264 (see paragraphs 0013, 0017, 0028 to 0036) "Correspondence evaluation based on connectivity in segment-based stereo" IPSJ Journal Vol. 40, no. 8, pages 3219-3229, issued in August 1999 "Recognition of three-dimensional objects by stereo vision" IEICE Transactions D-II, vol. J80-D-II, no. 5, 1105-1112 pages, published in May 1997

  In order to create a highly accurate three-dimensional model using the invention described in Patent Document 1, it seems better to perform three-dimensional measurement on the real model from many directions and integrate the measurement results. It is. However, in such a method, the capacity of the three-dimensional model increases, and the processing time when actually used in the recognition processing becomes long, so that there is a possibility that the model is not suitable for practical use.

  Further, Patent Document 1 does not describe anything about whether or not an actual recognition processing object is correctly recognized by the created three-dimensional model. In this regard, it is generally considered that a trial recognition process is performed using the created 3D model, and if the recognition accuracy is poor, the 3D model is recreated. This method is inefficient and may place a heavy burden on the user.

  This invention pays attention to said problem, and makes it a subject to make a capacity | capacitance as small as possible and to produce a highly accurate three-dimensional model, and to enable this three-dimensional model to be created in a short time.

  The method of creating a three-dimensional model according to the present invention aligns a plurality of three-dimensional information restored by performing three-dimensional measurement using a stereo camera from different directions on a real model of a recognition object. Are integrated to create a three-dimensional model of the recognition object.

  In this method, a temporary three-dimensional model is created using three-dimensional information restored by at least one three-dimensional measurement on the real model, and the three-dimensional model used to create the temporary three-dimensional model for the real model. For the three-dimensional measurement performed with the direction different from the time when the information is restored as the measurement direction, the three-dimensional information restored by the measurement is recognized by a temporary three-dimensional model, and the recognition result is determined. Furthermore, the three-dimensional information that is the recognition target when it is determined that the result of the recognition process using the provisional three-dimensional model is incorrect, or the angular deviation with respect to the measurement direction when the three-dimensional information of the recognition target is restored is a predetermined value. The three-dimensional information restored by measurement from the direction within the three-dimensional model and the result of the recognition process using the three-dimensional model determined to be correct is subjected to coordinate conversion so as to match the temporary three-dimensional model, and the converted three-dimensional information The dimensional information is added to the provisional three-dimensional model, and the three-dimensional model at the time when the addition process is completed is determined as a model used for recognition of the recognition object.

  In the above, the 3D information restored by the 3D measurement is an aggregate of a plurality of 3D feature data. “Alignment” of the three-dimensional information refers to a process of converting the coordinates of a plurality of three-dimensional information restored by stereo measurement from different directions based on the positional deviation and angular deviation between them. The measurement direction with respect to the real model is determined by the relative positional relationship between the real model and the stereo camera.

  According to the above method, the temporary three-dimensional model out of the three-dimensional information obtained by measuring the real model from a different direction from that when the three-dimensional information used to create the temporary three-dimensional model is restored. Then, it is possible to add three-dimensional information that could not be correctly recognized to the three-dimensional model. Therefore, the added three-dimensional model can correctly recognize the three-dimensional information measured from the misrecognized direction, and the accuracy of the three-dimensional model can be improved accordingly. On the other hand, since the three-dimensional information that can be correctly recognized by the provisional three-dimensional model is not added to the three-dimensional model, it is possible to prevent an increase in capacity due to the addition of useless information.

  In this way, by measuring a real model from various measurement directions assumed for an actual recognition target object and adding the erroneously recognized one of the restored three-dimensional information to the three-dimensional model, 3 It is possible to increase the accuracy while suppressing the capacity of the dimensional model.

  However, in order to match the misrecognized three-dimensional information with the three-dimensional model, it is necessary to perform coordinate conversion based on a positional deviation amount and a rotation angle of the three-dimensional information with respect to the three-dimensional model. With respect to this point, for example, when the rotation angle of the real model is controlled by a rotation table, a parameter for coordinate conversion can be determined from each rotation angle. Further, even when the positional relationship between the stereo camera and the real model is arbitrarily changed, it is possible to obtain the coordinate conversion parameters by the calculation described later.

  The determination as to whether or not the result of recognizing the above three-dimensional information by the provisional three-dimensional model is correct, for example, by the same method as in the case of deriving the parameters for coordinate conversion described above, A correct correspondence relationship with the provisional three-dimensional model is obtained, and this relationship is compared with the recognition result of the provisional three-dimensional model. Alternatively, a recognition result based on the temporary three-dimensional model may be displayed, the user may be determined whether it is correct or not, and a determination input may be accepted.

  In the above method, not only the three-dimensional information itself that has not been correctly recognized, but also the angular deviation with respect to the measurement direction when the three-dimensional information is restored is within a predetermined value, and is correctly recognized by the three-dimensional model. In some cases, three-dimensional information is added. This is due to the following reasons.

  If the three-dimensional measurement is executed by changing the measurement direction every time at an appropriate interval, a considerable amount of common features can be included between the three-dimensional information restored by two consecutive measurements. Therefore, even if one of these three-dimensional information cannot be correctly recognized, and the other can be correctly recognized, the latter information can be correctly recognized by adding the latter three-dimensional information to the three-dimensional model. There is a possibility.

Hereinafter, four preferred embodiments of the above method will be described.
First, in the first aspect, a different measurement direction is set each time for a real model, and a plurality of three-dimensional measurements are performed, and the three-dimensional information restored by the second and subsequent three-dimensional measurements is one step ahead. The corresponding relationship between the three-dimensional information is recognized by collating with the three-dimensional information restored by the measurement. Further, two or more three-dimensional information corresponding to a part of a plurality of three-dimensional information restored by a plurality of three-dimensional measurements is selected, and the other one is selected based on one of the selected three-dimensional information. Based on the correspondence relationship between the three-dimensional information recognized for each measurement between the measurement corresponding to the three-dimensional information and the measurement corresponding to the reference three-dimensional information. Are converted so as to be consistent with each other, and the converted three-dimensional information is integrated with reference three-dimensional information to create a provisional three-dimensional model.

  Next, three-dimensional information that has not been integrated into the temporary three-dimensional model among a plurality of three-dimensional information is sequentially processed, and recognition processing using the temporary three-dimensional model is executed to determine whether the recognition result is correct or not. In addition, when it is determined that the result of the recognition process is not correct, the three-dimensional information that is the object of the recognition process is measured between the measurement corresponding to the three-dimensional information and the measurement corresponding to the reference three-dimensional information. Based on the recognized correspondence between the three-dimensional information restored for each measurement, coordinate conversion is performed to match the reference three-dimensional information, and the converted three-dimensional information is added to the temporary three-dimensional model. . When the processing for all the three-dimensional information that has not been integrated into the three-dimensional model is completed, the three-dimensional model at that time is determined.

  According to the above aspect, after performing three-dimensional measurement from various directions assumed as measurement directions for the real model, by selecting three-dimensional information corresponding to each measurement direction at appropriate intervals, The three-dimensional model is created. Furthermore, the process of recognizing the three-dimensional information that has not been integrated into the provisional three-dimensional model by the three-dimensional model is sequentially performed, and the three-dimensional information for which a correct recognition result could not be obtained is added to the three-dimensional model. . The three-dimensional model created here can correctly recognize the measurement from any of the set measurement directions in addition to the direction corresponding to the selected three-dimensional information and the added three-dimensional information. It has accuracy. Even if the number of executions of the three-dimensional measurement increases, only the three-dimensional information necessary for ensuring the recognition accuracy is integrated, so that the capacity of the three-dimensional model can be prevented from increasing.

  Further, in the above aspect, the three-dimensional information restored by the second and subsequent three-dimensional measurements is collated with the three-dimensional information restored by the previous measurement, and the positional relationship between the three-dimensional information is recognized. Then, using these recognition results, coordinate conversion processing is performed when a temporary 3D model is created or when new 3D information is added to the 3D model. There is no need to strictly adjust the positional relationship. Also, if the hourly measurement direction is not changed significantly from the previous measurement direction, it is possible to restore a considerable amount of information restored in the previous measurement in each measurement. Thus, the recognition accuracy of the correspondence between the three-dimensional information can be ensured.

  In the second aspect, after creating a temporary three-dimensional model using a plurality of three-dimensional information restored by a plurality of three-dimensional measurements performed by setting different measurement directions each time for the real model, On the other hand, a process of executing the three-dimensional measurement by changing the measurement direction every time so as to be different from any measurement direction set at the time of creating the temporary three-dimensional model is executed a plurality of times. Then, after performing recognition processing with a temporary 3D model for the 3D information restored by the first 3D measurement of the 3D measurement, the restoration is performed by the measurement every second and subsequent measurements. The first step of recognizing the correspondence between the three-dimensional information by collating the three-dimensional information that has been restored with the three-dimensional information restored one step ago, and the three-dimensional information restored by the measurement, A recognition process using a temporary three-dimensional model is executed, and a second step of determining whether the recognition result is correct or not is executed. When it is determined that the result of the recognition process for the three-dimensional information restored by the second and subsequent measurements is not correct, the recognition result for the three-dimensional information restored by the first three-dimensional measurement, and the current time from the second time Based on the recognition result of the first step for each three-dimensional measurement performed up to now, the coordinate conversion is performed so that the three-dimensional information determined as the result of the recognition process is inconsistent with the temporary three-dimensional model. The later three-dimensional information is added to the temporary three-dimensional model. When all of the three-dimensional measurements and the processes associated therewith are completed, the three-dimensional model at that time is determined.

  According to the second aspect, first, a real model is measured from a representative direction among various directions that are actually assumed, and after creating a provisional three-dimensional model based on the measurement results, other assumptions are made. When the measurement from the direction is performed sequentially, among the three-dimensional information restored by the measurement, the information that could not be correctly recognized by the temporary three-dimensional model is the result of the recognition processing executed so far. Coordinates are converted based on this and added to the temporary three-dimensional model. Also, in the first three-dimensional measurement after creating the temporary three-dimensional model, the three-dimensional information that can be correctly recognized by the temporary three-dimensional model is restored. If a different measurement direction is not set for each measurement, it is possible to restore a number of features restored by the previous measurement in the second and subsequent measurements. One step accuracy can be ensured. Therefore, it is possible to perform the coordinate conversion process on the three-dimensional information to be added with high accuracy, and to ensure the accuracy of the three-dimensional model.

  In a 3rd aspect, the process which changes the measurement direction with respect to a real model each time and performs three-dimensional measurement is performed in multiple times. In addition, the 3D information restored by the first 3D measurement is set as a temporary 3D model, and the 3D information restored by the 3D measurement is restored one step before in the subsequent 3D measurement. First step of recognizing the correspondence between the three-dimensional information by collating with the three-dimensional information, and recognition processing using a temporary three-dimensional model for the three-dimensional information restored by the three-dimensional measurement And the second step of determining whether the recognition result is correct or not is executed. When it is determined in the second step that the result of the recognition process using the three-dimensional model is not correct, the three-dimensional information that is the object of the recognition process is determined for each three-dimensional measurement from the second time to the current time. Based on the recognition result of the first step executed in this way, coordinate conversion is performed so as to match the temporary three-dimensional model, and the converted three-dimensional information is added to the temporary three-dimensional model. When all of the three-dimensional measurements and the processes associated therewith are completed, the three-dimensional model at that time is determined.

  According to the third aspect, after the three-dimensional information restored by the first measurement out of a plurality of three-dimensional measurements is set as a temporary three-dimensional model, it is correctly recognized every time measurement is performed thereafter. The coordinate information is converted so that the three-dimensional information that has not been matched with the provisional three-dimensional model is added to the provisional three-dimensional model. In addition, for the second and subsequent 3D measurements, if the measurement direction is not set to be extremely different from that of the previous stage, the information restored by the previous stage of measurement is equivalent to any measurement. It is possible to ensure the accuracy of the first step by restoring several times. Therefore, it is possible to accurately perform the coordinate conversion process on the three-dimensional information to be added, and the three-dimensional accuracy with which the target can be correctly recognized for the measurement from all measurement directions set for the real model. A model can be created.

  Also in the second and third aspects, the three-dimensional information to be added is added to the temporary three-dimensional model after coordinate conversion based on the result of recognition processing performed until the information is restored. It is not necessary to strictly adjust the positional relationship between the real model and the stereo camera using a rotary table or the like.

  In the fourth aspect, the three-dimensional information restored by the first three-dimensional measurement is assumed to be temporary 3 on condition that the process of rotating the real model at an arbitrary angle and executing the three-dimensional measurement is executed a plurality of times. Each time a three-dimensional measurement is performed, a recognition process using a temporary three-dimensional model is performed on the three-dimensional information restored by the measurement, and the recognition result is determined to be correct or not. To do. When it is determined that the result of the recognition process using the temporary 3D model is not correct, the 3D measurement is performed by rotating the real model so that the measurement direction with respect to the real model approaches the measurement direction one step before; The step of executing a recognition process using a temporary three-dimensional model on the three-dimensional information restored by the three-dimensional measurement and determining whether the recognition result is correct or not until the recognition result is determined to be correct. The three-dimensional information corresponding to the recognition result determined to be correct is subjected to coordinate conversion based on the recognition result, and the converted three-dimensional information is added to the temporary three-dimensional model. The real model is rotated in a direction to return to the state when it is determined to be incorrect. Then, when the three-dimensional measurement when the positional relationship between the real model and the stereo camera is in a predetermined state and the processing associated therewith are completed, the three-dimensional model at that time is determined.

  Also in the fourth aspect, the three-dimensional information restored by the first measurement is set as a temporary three-dimensional model, and thereafter, when the correct recognition cannot be performed by the temporary three-dimensional model, the three-dimensional model is changed. Additional processing is performed. However, in this aspect, not by the 3D information itself that could not be recognized, but by adding to the 3D model 3D information that includes features common to the 3D information and that is correctly recognized by the 3D model. The object is to make the three-dimensional information that could not be recognized become recognizable. Therefore, by repeating the rotation of the real model and the three-dimensional measurement while appropriately performing the addition process of the three-dimensional information, the object can be correctly recognized for the measurement from all the set measurement directions. It becomes possible to create a dimensional model.

  Further, in this aspect, the real model can be rotated each time without considering the rotation angle, and the three-dimensional coordinate added to the three-dimensional model is coordinate-converted based on the result recognized from the three-dimensional model. Therefore, the accuracy of matching the added information with the three-dimensional model is also ensured. In addition, unlike the other aspects, since the verification is not performed every time the three-dimensional measurement is performed, the processing can be simplified.

  Next, an object recognition apparatus according to the present invention includes a three-dimensional measurement unit that restores three-dimensional information of a recognition object by three-dimensional measurement using a stereo camera, a three-dimensional model in which the restored three-dimensional information is registered in advance. The recognition processing means for recognizing the position and orientation of the object to be recognized by collation, and a plurality of three-dimensional information restored by performing three-dimensional measurement from different directions on the real model of the recognition object And a three-dimensional model registration unit for registering the three-dimensional model after the three-dimensional model used by the recognition processing unit is created.

  Furthermore, in the present invention, during each measurement, a user creates a three-dimensional model suitable for recognition processing while executing a plurality of measurements by arbitrarily changing the positional relationship between the real model and the stereo camera. The following first configuration and second configuration are provided as an object recognition device having the function of:

  In the object recognition apparatus according to the first configuration, the following correspondence relationship recognition means, provisional model creation means, recognition result confirmation means, model update means, and model confirmation means are provided in the three-dimensional model registration means.

  Correspondence recognition means 3 3D information restored by the second and subsequent measurements out of a plurality of 3D information restored by the processing of the 3D measurement means for the real model 3 The correspondence between the three-dimensional information is recognized by collating with the dimension information. The temporary model creation means selects two or more three-dimensional information corresponding to a part of a plurality of three-dimensional information restored by a plurality of three-dimensional measurements, and uses one of the selected three-dimensional information as a reference As for other three-dimensional information, the correspondence between the three-dimensional information recognized by the correspondence recognition means for each measurement between the measurement corresponding to the three-dimensional information and the measurement corresponding to the reference three-dimensional information. Based on the above, coordinate conversion is performed to match the reference three-dimensional information, and the converted three-dimensional information is matched with the reference three-dimensional information to create a provisional three-dimensional model.

  The recognition result confirmation means recognizes the focused three-dimensional information with the temporary three-dimensional model by paying attention to the information that has not been integrated into the temporary three-dimensional model among the plurality of three-dimensional information, and whether the recognition result is correct or not Determine.

  When the model update means determines that the result of the recognition process is not correct, the model update means corresponds to the measurement and reference three-dimensional information corresponding to the three-dimensional information corresponding to the current three-dimensional information. Based on the correspondence between the three-dimensional information recognized by the correspondence-recognition means for each measurement, the coordinates are converted so as to match the reference three-dimensional information, and the converted three-dimensional information is temporarily stored. The provisional three-dimensional model is updated by adding to the three-dimensional model. The model confirming means confirms, as a registration target, a three-dimensional model to which all of the three-dimensional information determined that the recognition processing result is not correct is added.

  According to the above configuration, if the user arbitrarily changes the positional relationship between the real model and the stereo camera each time three-dimensional measurement is performed, the three-dimensional information selected from the restored three-dimensional information is used. After the provisional three-dimensional model is created, three-dimensional information that is difficult to correctly recognize is identified in the three-dimensional model, and a three-dimensional model to which these pieces of information are added can be created.

  In addition, each time the second and subsequent measurements are performed, the positional relationship between the three-dimensional information restored by the measurement and the previous three-dimensional information is recognized, and the three-dimensional integration target is based on these recognition results. Since the coordinate conversion is performed on the information, if the measurement direction of each time is not greatly different from the previous measurement direction, the accuracy of collation with the previous three-dimensional information can be ensured. Accuracy can be ensured. Therefore, it is possible to create a three-dimensional model with high accuracy and reduced capacity without strictly adjusting the positional relationship between the real model and the stereo camera each time.

  In the object recognition apparatus according to the second configuration, the following three-dimensional model registration means includes the following temporary model setting means, recognition result confirmation means, model update means, and model confirmation means.

  The temporary model setting means initializes the three-dimensional information restored by the measurement into a temporary three-dimensional model in response to the first measurement performed by the three-dimensional measurement means on the real model. The recognition result confirmation unit executes a recognition process using a temporary three-dimensional model for the three-dimensional information restored by the measurement every time the three-dimensional measurement unit performs the second and subsequent measurements on the real model. A determination input for accepting the recognition result is accepted.

  When the recognition result confirmation unit accepts a determination input indicating that the result of the recognition processing executed is incorrect, the model update unit confirms the recognition result among the three-dimensional information restored from the stereo image input after the reception. In the processing by the means, the first information that has received the determination input that the recognition processing result using the temporary three-dimensional model is correct is subjected to coordinate conversion based on the recognition result, and the converted three-dimensional information is converted into the temporary three-dimensional information. Update the 3D model by adding to the model.

  When all of the processing of the three-dimensional measurement for the real model and the recognition result confirmation unit and the model update unit are completed, the model determination unit determines the three-dimensional model at that time as a registration target.

  According to the above configuration, after the temporary three-dimensional model is initially set according to the first measurement, the user rotates the real model at an arbitrary angle each time after the measurement, so that the respective directions are different from each other. A process of recognizing the three-dimensional information restored by the measurement using the provisional three-dimensional model is performed, and a determination result by the user is input as to whether the recognition result is correct. Here, if the user determines that the three-dimensional information has not been correctly recognized, the determination result is input, and the measurement is performed by rotating the real model so that the measurement direction approaches the previous measurement direction. Is executed a predetermined number of times, it is possible to obtain three-dimensional information that is correctly recognized by the three-dimensional model when the measurement state is returned to at least one stage before. Therefore, by adding the three-dimensional information at this time to the three-dimensional model, the possibility that the three-dimensional information that could not be correctly recognized can be recognized is increased.

  After the 3D model is updated by the above addition, if the user returns the rotation direction of the real model to the original direction, after that, the measurement direction in which the previous 3D information could not be correctly recognized is reset. Then, it can be confirmed whether or not correct recognition is possible by the updated three-dimensional model, and after confirming that recognition is possible, the process can proceed to the next measurement.

  Therefore, according to the apparatus having the second configuration, the user can easily determine a state that is difficult to recognize with the current three-dimensional model while performing measurement while setting the posture of the real model with respect to the stereo camera in various states. Thus, measures can be taken to enable the recognition. Therefore, it is possible to create a three-dimensional model with high accuracy and reduced capacity while changing the positional relationship between the real model and the stereo camera in accordance with a change in posture that can be assumed by the recognition target.

  According to this invention, the object can be recognized from all the measurement directions set by these measurements by executing the process of performing the three-dimensional measurement by changing the measurement direction with respect to the real model every time. In addition, a three-dimensional model that does not include useless information can be created. In addition, since the above three-dimensional model can be created by performing a sufficient number of measurements based on the actually assumed positional relationship between the stereo camera and the recognition object, the efficiency required for creating the three-dimensional model Can be greatly improved.

(1) Device Configuration FIG. 1 shows an example of a picking system to which a three-dimensional recognition process is applied.
This picking system is for performing the work of taking out the workpieces W housed in the housing box 6 one by one and transporting them to a predetermined position in the factory. A robot control device 3 that controls the operation of the robot 4 is included. Further, this picking system is provided with a stereo camera 1 and an object recognition device 2 in order to recognize the position and orientation of the workpiece W to be processed.

  The stereo camera 1 is composed of three cameras 11, 12, and 13 whose positional relationship is fixed. In the object recognition device 2, information representing the positional relationship of the cameras 11, 12, 13, the direction of the optical axis, and the three-dimensional model of the workpiece W to be recognized are registered. After the input stereo image is processed to restore the three-dimensional information of the contour line of the workpiece W, the restored three-dimensional information is collated with the three-dimensional model to recognize the position and orientation of the workpiece W. Information indicating the recognition result is output from the object recognition device 2 to the robot control device 3, and is used for processing for controlling the operation of the arm 40 of the robot 4 in the robot control device 3.

FIG. 2 shows a hardware configuration of the object recognition apparatus 2.
In this apparatus, in addition to the image input units 21, 22, and 23 corresponding to the cameras 11, 12, and 13, a CPU 24, a memory 25, an input unit 26, a display unit 27, a communication interface 28, and the like are provided. The input unit 26 is a keyboard and a mouse, and the display unit 27 is a liquid crystal monitor. The input unit 26 and the display unit 27 are used for applications in which a user confirms an imaging state of a real model, or performs a selection operation or a setting operation when creating a three-dimensional model described below. The communication interface 28 is used for communication with the robot control device 4.

The memory 25 includes a large-capacity memory such as a ROM, a RAM, and a hard disk.
Based on the program stored in the memory 25, the CPU 24 executes a series of processes related to the three-dimensional measurement and the recognition of the workpiece W, and the recognition result (specifically, the three-dimensional coordinates representing the position of the workpiece W, and the three-dimensional The rotation angle with respect to the model is output from the communication interface 28.

  The memory 25 also stores a program for creating a three-dimensional model. Prior to the recognition process, the CPU 24 creates a three-dimensional model of the work W using the images of the real models input from the cameras 11 to 13 based on this program, and registers the three-dimensional model in the memory 25.

  Further, although not shown in FIG. 2, the object recognition apparatus 2 of this embodiment is provided with a circuit for outputting a drive signal to each of the cameras 11, 12, and 13, and the CPU 24 receives each camera via this circuit. A function for controlling the imaging operations 11 to 13 is set. In this embodiment, when an operation for instructing the start of measurement is performed by the input unit 26, each of the cameras 11 to 13 is caused to capture an image.

(2) Three-dimensional measurement processing In the object recognition device 2 described above, after detecting an edge from a stereo image, the three-dimensional information is restored for each unit called “segment” in the same manner as the invention described in Patent Document 1. Furthermore, collation with the three-dimensional model is performed on a segment basis. FIG. 3 shows a schematic procedure of processing executed for recognizing one workpiece by this method. Hereinafter, the three-dimensional recognition process in this embodiment will be described with reference to the step codes in FIG.

  First, stereo imaging is performed by each of the cameras 11 to 13, and an edge extraction filter is applied to each generated image to detect an edge in the image (ST1, 2). Next, the detected edge is thinned (set to 1 pixel width data), and the thinned edge is divided into straight or curved segments based on the connection points and branch points (ST3, 4). The segment extracted from the edge on the two-dimensional image is hereinafter referred to as “two-dimensional segment”.

  Next, a process of associating two-dimensional segments that have a correspondence relationship between images is executed (ST5). In this association, one two of the three images is used as a reference, and each two-dimensional segment of the reference image is focused in turn, and the two-dimensional corresponding to each of the two-dimensional segments focused on from the other two images. Identify the segment. That is, a two-dimensional segment satisfying the epipolar condition and matching with a neighboring segment is detected from each image with respect to the focused two-dimensional segment (refer to Non-Patent Document 1 or the like for details). I want.)

Next, for each combination of the two-dimensional segments associated by the above-described process, a process for restoring the three-dimensional information from the correspondence is executed (ST6).
Briefly, for each combination of two-dimensional segments associated with each other, the three-dimensional coordinates of the pixels having a correspondence relationship between the segments are calculated. Further, the calculated distribution state of the three-dimensional coordinates is collated with a model of a straight line and an arc to determine whether the set of these three-dimensional coordinates corresponds to a straight line / curve. With this process, for each combination of two-dimensional segments, a three-dimensional segment of a straight line or a curve corresponding to the combination is specified.

  Further, in ST6, for each three-dimensional segment, the segment is sampled at a predetermined interval, and information in which the segment type (straight line or curve) is associated with the three-dimensional coordinates of each sampling point is created. . As a result, even for a three-dimensional segment in which the number of three-dimensional coordinates calculated from the two-dimensional segment is small, it is possible to obtain more three-dimensional coordinates from the beginning if the sampling interval is made fine.

  The set of three-dimensional segments restored by the above processing corresponds to the three-dimensional information of the workpiece to be recognized. In the next stage, a three-dimensional model is set at a predetermined reference position in the three-dimensional coordinate system, and the amount of workpiece displacement relative to the three-dimensional model is determined by collating the three-dimensional information restored by the three-dimensional model. The rotation angle is recognized (ST7).

  In the matching process in ST7, the intersection of each three-dimensional segment is used as a feature point, and each feature point is associated with the brute force formula, and the degree of coincidence of each three-dimensional segment when the association is performed is calculated. The correspondence when the maximum is reached is specified as the correct correspondence.

  ST7 will be described in a little more detail. In this embodiment, a specific feature point in the 3D model is sequentially associated with each feature point of the 3D information to be collated, and the feature point on the 3D model side is moved to the corresponding point for each association. The shift amount and rotation angle necessary for the calculation are calculated. All of these are calculated for each of the X, Y, and Z axes. Next, all the coordinates included in the three-dimensional model are converted based on the calculated shift amount and rotation angle, and the degree of coincidence between the converted three-dimensional model and the three-dimensional information to be collated is calculated.

  By executing the above processing for all feature points on the three-dimensional model side, the feature points are associated with the brute force formula, and the degree of coincidence can be obtained for each association. Thereafter, the shift amount and the rotation angle used for the coordinate conversion when the highest degree of coincidence is finally obtained are recognized as the positional deviation amount and the rotation angle of the recognition target workpiece with respect to the three-dimensional model.

  Thereafter, the recognized positional deviation amount and rotation angle are output to the robot controller 3 (ST8), and the process is terminated.

(3) Three-dimensional model creation process 3-1) Principle description In order to execute the above-described three-dimensional recognition process, it is necessary to register a three-dimensional model of the workpiece W in the memory 25 in advance. In the object recognition apparatus 2 of this embodiment, a three-dimensional model is created by a method in which a real model of the work W is measured in stereo from various directions and three-dimensional information restored from each measurement result is integrated. Hereinafter, this process will be described in detail.
In the following drawings, an actual model of the workpiece W is indicated by a symbol WM, and in the specification, the actual model WM is referred to as a “work model WM”.

  In this embodiment, as shown in FIG. 4A, the position and the optical axis direction of the stereo camera 1 are fixed, and the stereo measurement is performed a plurality of times while changing the posture of the work model WM with respect to the stereo camera 1. .

  In the figure, a three-dimensional coordinate system based on the X, Y, and Z axes is a coordinate system for calculating three-dimensional coordinates (hereinafter referred to as “measurement coordinate system”), and is uniquely determined for the stereo camera 1. It is done. On the other hand, the three-dimensional coordinate system based on the X1, Y1, and Z1 axes is a coordinate system uniquely defined for the work model WM (hereinafter referred to as “work coordinate system”). In FIG. 4A, the X1, Y1, and Z1 axes are parallel to the X, Y, and Z axes, respectively. However, when the posture of the work model WM changes, the direction of each axis in the work coordinate system also changes. It changes according to.

  Although the posture of the work model WM can be freely determined by the user, in this embodiment, a specific surface along the X1Z1 plane is always in contact with the work support surface (XZ plane in this example), and is orthogonal to the XZ plane. It is assumed that the work model WM is rotated once with respect to the Y1 axis, and imaging is performed a plurality of times during that time. Since the work model WM is rotated by a method of changing the orientation of the work model WM with the user's hand without using the rotation table, not only the rotation angle but also the position of the work model WM with respect to the measurement coordinate system is determined. Every time, the position is shifted.

FIG. 4B shows a change in the positional relationship of the stereo camera 1 with respect to the work model WM with reference to the work coordinate system. In the figure, rectangles with numbers 1 to 12 indicate the location of the stereo camera 1 at the time of imaging, and the numbers in the rectangles indicate the order of imaging.
Any stereo image generated by imaging at these positions is subject to three-dimensional measurement. Therefore, hereinafter, the position of the stereo camera represented by each rectangle is referred to as a “measurement point”. When individually referring to each measurement point, a number representing the order of measurement is quoted, such as “measurement point [1]” and “measurement point [2]”.

  In this embodiment, the same processing as ST1 to ST6 in FIG. 3 is executed for each of the 12 measurement points to restore the three-dimensional information. Further, at each measurement point excluding the measurement point [1], the three-dimensional information restored at the measurement point is collated with the three-dimensional information of the previous measurement point, so that the position relative to the previous three-dimensional information is obtained. Recognize deviation and rotation angle. These recognition processes are all performed by the same method as that performed in ST7 of FIG.

  For the last measurement point [12], in addition to the displacement amount and rotation angle for the three-dimensional information of the measurement point [11], the displacement amount and rotation angle for the three-dimensional information of the measurement point [1] are also obtained. . However, the measurement point [12] is transferred to the measurement point [1] again, the measurement is re-executed, and the positional deviation amount and the rotation angle of the restored three-dimensional information with respect to the three-dimensional information of the measurement point [12] are obtained. May be.

  Furthermore, in this embodiment, the accuracy of the three-dimensional information restored for each measurement point is confirmed, two or more three-dimensional information confirmed to be good is selected, and the selected information is aligned and integrated. Like to do. Details of these processes will be described later.

  Next, in this embodiment, stereo measurement is performed by attaching a triangular mark M to an appropriate position on the surface of the work model WM. This mark M is configured as a seal and is affixed to a flat surface of the work model WM (in this embodiment, the center of the upper surface). Each measurement point is determined on condition that the mark M is included in the visual field of all the cameras 11 to 13. Therefore, the three-dimensional information including the three-dimensional segment representing each side of the mark M is restored at any measurement point.

  FIG. 5 shows three of the three-dimensional information (indicated by a, b, and c in the figure) among the three-dimensional information restored at the 12 measurement points for the work model WM with the mark M. A processing procedure for creating a dimensional model is schematically shown. Note that the three-dimensional information a in the figure is restored at the measurement point [12] in FIG. The three-dimensional information b is restored at the measurement point [3], and the three-dimensional information c is restored at the measurement point [6]. In addition, the information m of the mark M is included in any three-dimensional information (however, the substantial content of the information m varies depending on the measurement point).

  In this example, on the basis of the three-dimensional information a, the other three-dimensional information b and c are coordinate-transformed so as to eliminate the positional deviation and the angular deviation with respect to the reference three-dimensional information a. Then, by integrating the converted three-dimensional information b ′ and c ′ with the reference three-dimensional information a, three-dimensional information d including the characteristics of each information is created. Further, by deleting the information m corresponding to the triangular mark M from the integrated three-dimensional information d, three-dimensional information e representing only the three-dimensional shape of the work model WM is created. Set to model.

  The coordinate conversion process of the above three-dimensional information is performed using the positional deviation amount and the rotation angle previously obtained for each measurement point. For example, in order to align the 3D information b restored at the measurement point [3] with the 3D information a restored at the measurement point [12], the measurement points [3] and [2], the measurement point [2], [1] Coordinate conversion is performed using the positional deviation amount and the rotation angle obtained for each combination of the measurement points [1] and [12]. Similarly, the three-dimensional information c restored at the measurement point [6] is calculated in the range from the measurement point [6] to the measurement point [12] (in this case, either clockwise or counterclockwise may be used). By performing coordinate conversion using the amount of misalignment and the rotation angle, alignment with the three-dimensional information a is performed.

  As described above, when obtaining the positional deviation amount and the rotation angle between the measurement points separated from each other, the measurement points between them are combined for each adjacent one, and the three-dimensional information for each set of measurement points. The reason why the positional deviation amount and the rotation angle are calculated is to ensure the accuracy of collation by collating information including as many common features as possible. In addition, even when information is added to the three-dimensional model, which will be described later, coordinate conversion processing is performed using the amount of positional deviation and the rotation angle of the three-dimensional information between measurement points.

Next, the reason for attaching the triangular mark M to the work model WM will be described.
As described above, when three-dimensional information is collated in this embodiment, feature points in each information are associated with a brute force formula to obtain a degree of coincidence, and the correspondence with the highest degree of coincidence is obtained. Confirm as correct. However, depending on the shape of the recognition target, the correspondence that maximizes the degree of matching may not always be correct.

  For example, when a measurement target is an edge of a curved surface or a chamfered part, or when a part having a repetitive structure is a measurement target, an incorrect correspondence as shown in FIGS. There is a possibility that the degree of coincidence is maximized and the incorrect association is determined.

FIG. 6 shows an example in which stereo measurement is performed on the corner portion of the recognition object Sb from two directions, f1 and f2. This figure is a cross section of the recognition object Sb, and R1 and R2 in the figure correspond to the boundary position between the flat surface and the surface having a large curvature.
In this example, the measurement from the direction f1 restores the three-dimensional edge at the position R1 in the figure, while the measurement from the direction f2 restores the three-dimensional edge at the position R2. Is done. Accordingly, when associating the three-dimensional information restored by measurement from each direction, the edge corresponding to R1 and the edge corresponding to R2 are erroneously associated with each other, and the alignment accuracy is deteriorated accordingly. There is a fear.

  FIG. 7 shows an example in which an error occurs in associating a part including a repetitive structure. Such an incorrect association may occur when the edge of the repetitive structure is unclear and the restoration accuracy of the three-dimensional information deteriorates.

  In addition, even when a symmetric part is included in the measurement object, there is a possibility that a state in which the front and back of the symmetric part are associated with each other by mistake is recognized as correct.

  In this embodiment, the method of changing the measurement direction with respect to the work model WM at an arbitrary angle in each stereo measurement is taken. Therefore, in order to integrate the three-dimensional information, the three-dimensional information is collated, It is necessary to obtain a positional deviation amount and an angular deviation amount (rotation angle) between them and perform coordinate conversion based on them. Therefore, if the above-described erroneous association is performed in the comparison between the three-dimensional information, the three-dimensional information to be integrated cannot be correctly aligned, and the accuracy of the three-dimensional model cannot be ensured. The triangular mark M is attached to prevent such an error.

  The triangle represented by the mark M used in this embodiment is set so that the ratio of each side is 6: 7: 8, as shown in FIG. According to this ratio, the vertices A, B, and C of the mark M can be easily specified. For example, the direction of the mark M can be uniquely specified by the direction of the perpendicular from the vertex A to the side BC. it can. Further, as shown in FIG. 9, the alignment accuracy can be ensured when aligning with the triangle A′B′C ′ having the same shape.

  In FIG. 9 (1), one vertex is selected from each of the triangle ABC by the above ratio and the congruent triangle A′B′C ′, and the respective triangles are aligned based on the selected vertex. Indicates. As shown in this figure, triangles do not completely overlap unless vertices in the correct correspondence are combined.

  Regarding the correspondence of the above triangles, the inventors actually calculated the degree of coincidence of the entire triangle for each vertex combination. FIG. 9 (2) is a graph showing the results. As shown in this graph, when the degree of coincidence when the vertices are correctly associated with each other is 1, the degree of coincidence due to other correspondences is about 0.4 at the maximum. As a result, it was found that the correctness of the correspondence between triangles can be correctly determined.

  As described above, according to the triangle having the shape shown in FIG. 8, there is a large difference in the degree of coincidence between the case where the vertices of the triangle are correctly associated and the case where the correspondence is incorrect. Even if each side of the triangle is associated as a three-dimensional segment, the same effect should be obtained. Therefore, when the entire three-dimensional information including the three-dimensional segment of the mark M is collated, each triangle The degree of coincidence regarding the three-dimensional segment of the side greatly affects the degree of coincidence of the whole.

  That is, even if there is an element in the original three-dimensional segment of the work model WM that causes erroneous alignment, if the correspondence is performed incorrectly, the matching degree of the triangular three-dimensional segment is By greatly decreasing, the degree of coincidence of the entire three-dimensional information is lowered. On the other hand, when each three-dimensional segment including a triangular three-dimensional segment is correctly aligned, the degree of coincidence increases in any three-dimensional segment, and the degree of coincidence of the entire three-dimensional information also becomes a high value. Therefore, the degree of coincidence when correctly associated can be made larger than in other cases, and the alignment accuracy can be ensured.

  The shape of the triangle indicated by the mark M is not limited to the example of FIG. 8, and the above-described effects can be obtained, and the ratio of the sides is not limited as long as the lengths of the sides are different. Further, the mark is not limited to a triangle, and a polygonal mark that has four or more vertices and whose direction can be uniquely specified may be created.

Next, the reason for applying the mark M to the flat surface will be described.
When the mark M is pasted on the curved surface, each side of the triangle is more likely to be recognized as a three-dimensional segment of the curve, and the shape of the mark M to be restored varies depending on the curvature of the curved surface. Therefore, when the mark M is attached to the curved surface, even if the three-dimensional information can be correctly restored, the restored three-dimensional information is likely to represent a shape different from the original triangular shape. Therefore, it is difficult to stably perform the verification process of the three-dimensional information by collation with the model triangle described below.

  In this embodiment, the mark M is also used for the purpose of confirming the accuracy of the restored three-dimensional information. In this embodiment, the workpiece is measured from various directions, but the three-dimensional information cannot be accurately restored at any measurement point. Depending on the orientation of the workpiece, the accuracy of the three-dimensional information may deteriorate due to a large reflection of measurement parameter errors or the generation of an image containing many features that are easily misrecognized.

  In this embodiment, since the user determines the position of the work model WM in each stereo measurement, measurement may be performed in a state where the mark M is not included in the field of view of each camera due to a user's mistake. In the three-dimensional information that does not include the information of the mark M, there is a possibility that the alignment accuracy may deteriorate, so it is necessary to exclude it from the integration target in advance.

  From this point, in this embodiment, it is determined whether or not the three-dimensional information of the mark M has been successfully restored for each measurement point, and the three-dimensional information determined to be good is used for the integration process. Yes.

  Specifically, in the determination process, as shown in FIG. 10 (a), when information close to an actual triangle is restored, the entire three-dimensional information including this information is considered to be a good integration target. . On the other hand, when the reproducibility of the triangle is poor, such as when the side is curved as shown in FIG. 10B or when the vertex cannot be correctly restored as shown in FIG. It is judged that the whole is not good and is excluded from integration.

  Furthermore, in this embodiment, when it is determined that the three-dimensional information is not good, the three-dimensional information is discarded, and measures such as performing another measurement or changing a measurement point are taken. Further, if good 3D information cannot be obtained even if measurement is repeated a predetermined number of times, the initial setting processing such as position adjustment of the stereo camera 1 and re-execution of 3D calibration is performed again. Such processing ensures the accuracy of the three-dimensional information used for the integration, and prevents the positional deviation amount and the rotation angle necessary for the coordinate conversion during the integration from being calculated by the inaccurate three-dimensional information. Thus, the accuracy of the three-dimensional model can be ensured.

  Specifically, the process for determining the accuracy of the three-dimensional information is a combination of three-dimensional model data representing the geometric information of the mark M, specifically, a combination of data representing the preferred three-dimensional segments of the three sides of the triangle. (Hereinafter, this model data is referred to as “model triangle”). That is, the three-dimensional information of each measurement point is collated with the model triangle to obtain the degree of coincidence with the model triangle. This matching is also performed in the same manner as in the case of matching with the three-dimensional model, that is, when each feature point of the model triangle is associated with each feature point in the three-dimensional information with a brute force formula and the degree of coincidence with the model triangle is the highest. This is done by a method of determining the correspondence of.

  After the above collation, the calculated degree of coincidence is compared with a predetermined threshold, and if the degree of coincidence exceeds the threshold, it is determined that the three-dimensional information is good, and the degree of coincidence is the threshold. If the following, the corresponding three-dimensional information is deleted.

  In this way, by attaching the triangular mark M to the work model WM, it is possible to confirm that the three-dimensional information restored by the hourly measurement is good and to ensure the accuracy of the matching process between the three-dimensional information. Thus, the accuracy of coordinate conversion processing for alignment can be improved. Therefore, highly accurate three-dimensional information can be created by integrating three-dimensional information that has undergone these processes.

3-2) Specific example of 3D model creation procedure In order to create a 3D model that can be used for recognition processing from various measurement directions by the method shown in FIG. 5, 3D information of as many measurement points as possible is used. It seems necessary to select and integrate. However, when the number of three-dimensional information to be integrated increases, the capacity of the three-dimensional model increases, and there is a possibility that a recognition process using the three-dimensional model may be hindered.

  In view of this point, in this embodiment, after a part (two or more) of a plurality of three-dimensional information restored for each measurement point is integrated to create a temporary three-dimensional model, the temporary three-dimensional model is created. The method of adding information that is insufficient to ensure the accuracy while suppressing the capacity of the three-dimensional model.

  FIG. 11 shows a specific example of the model creation process. Hereinafter, with reference to this figure, in this example, it was restored by measurement at the four measurement points [12] [3] [6] [9] among the above-mentioned 12 measurement points. Three-dimensional information is selected, and these are aligned and integrated to create a temporary three-dimensional model. Also in this case, as in the example of FIG. 5, the coordinate conversion is performed on the basis of the three-dimensional information of the measurement point [12] so that the other three three-dimensional information matches the reference three-dimensional information. Each subsequent three-dimensional information and reference three-dimensional information are integrated, and information obtained by deleting information corresponding to the mark M from the integrated three-dimensional information is defined as a temporary three-dimensional model.

  Next, three-dimensional information that has not been used for the integration process, that is, three-dimensional information of each measurement point of [1] [2] [4] [5] [7] [8] [10] [11] is targeted. Then, recognition processing using a temporary three-dimensional model is executed. Then, it is determined whether or not the recognized positional deviation amount and rotation angle are correct, and the three-dimensional information determined to be incorrect is aligned and added to the three-dimensional model. FIG. 11 shows an example in which the three-dimensional information is aligned and added to the three-dimensional model, assuming that the recognition process for the three-dimensional information of the measurement point [10] has not been performed correctly.

  When recognizing three-dimensional information that has not been used for the integration process, the mark information is deleted from the three-dimensional information based on the result of matching using the previous model triangle. Since the mark M is not attached to the actual workpiece W to be recognized, the accuracy of the actual recognition processing can be checked by doing so. Also, when adding three-dimensional information to the temporary three-dimensional model, the mark M information is not added.

  Next, when creating the provisional three-dimensional model, the process of aligning the three-dimensional information of the measurement points [3] [6] [9] with the reference three-dimensional information as described above. In addition, the positional deviation amount and the rotation angle with respect to the reference three-dimensional information (information of the measurement point “12”) are specified from the positional deviation amount and the rotation angle obtained for each measurement point, and coordinate conversion processing using these is performed.

Whether the result of recognition processing using the provisional 3D model is correct or not, and coordinate conversion for aligning the 3D information to be added to the 3D model are also processed for the 3D model in the same manner as described above. This is possible by specifying the positional deviation amount and the rotation angle of the information (specifically, the information corresponding to the reference three-dimensional information in the three-dimensional model).
The three-dimensional information of the measurement point [10] will be described as an example. The displacement amount recognized between the measurement points [10] and [11] and between the measurement points [11] and “12”, and By accumulating the rotation angles, it is possible to specify the amount of positional deviation and the rotation angle of this three-dimensional information with respect to the provisional three-dimensional model. Here, the specified positional deviation amount and rotation angle are compared with the positional deviation amount and rotation angle obtained by direct comparison with the three-dimensional model, and if there is a difference exceeding the allowable value between them, the three-dimensional model is used. It is determined that the result of the recognition process is not correct. In addition, using the positional deviation amount and the rotation angle specified from the collation result for each measurement point, the three-dimensional information of the measurement point [10] is coordinate-converted to perform alignment with the three-dimensional model.

  In this embodiment, the measurement direction of each time is arbitrarily set, but according to each measurement point shown in the figure, since a direction extremely different from the immediately previous measurement direction is not set, any measurement point, There is a high possibility that the three-dimensional information including many features common to the three-dimensional information obtained by the immediately previous measurement is restored.

  Accordingly, since each measurement point can accurately determine the positional deviation amount and the rotation angle with respect to the three-dimensional information of the previous measurement point, based on the accurate positional deviation amount and the rotational angle, the provisional 3 It is possible to easily determine whether or not the recognition result by the dimensional model is correct. Similarly, for the three-dimensional information in which the positional deviation amount and the rotation angle could not be correctly recognized by the direct comparison with the temporary three-dimensional model, the measurement corresponding to the three-dimensional information and the reference three-dimensional information in the three-dimensional model are similarly applied. In order to specify the positional deviation amount and the rotation angle with respect to the three-dimensional model based on the collation result related to each measurement performed between the measurement and the measurement corresponding to, and to align with the temporary three-dimensional model using these as parameters The coordinate conversion can be performed with high accuracy.

  Next, FIG. 12 is a flowchart summarizing a processing procedure from performing stereo measurement to registering a three-dimensional model. The procedure for creating the three-dimensional model of this embodiment will be described below along the flow of this flowchart.

  In this embodiment, since the positioning of the work model WM at the time of measurement is left to the discretion of the user, preview images from the respective cameras 11 to 13 are displayed on the display unit 27 during the measurement process. When the user confirms whether or not the work model WM and the mark M are correctly captured by the cameras 11 to 13 on this display screen and performs a measurement instruction operation, the step of “stereo measurement” in FIG. ST101) is executed. In step ST101, specifically, processing from stereo imaging to restoration of three-dimensional information (similar to ST1 to ST6 in FIG. 3) is executed.

  Next, the restored three-dimensional information is collated with a model triangle to determine the accuracy of the three-dimensional information (ST102). The determination here may be automatic determination based on the degree of coincidence, or may be performed by a method of displaying a collation result and accepting a user's determination on the display screen.

  If it is determined that the three-dimensional information is not good (ST103 is “NO”), the restored three-dimensional information is deleted, and an error message or the like is displayed on the display unit 27. In response to this, the user repeats the measurement instruction, for example, by changing the position of the work model WM.

  When it is determined that the restored three-dimensional information is good (“YES” in ST103), the positional deviation amount and the rotation angle with respect to the three-dimensional information one stage before this three-dimensional information are further recognized (ST105). ). Further, for the rotation angle, the rotation angle for the three-dimensional information restored first is calculated by a method of adding the recognized value every time (ST106). However, when the three-dimensional information is restored first (ST104 is “YES”), the above steps of ST105 and ST106 are skipped.

  Next, in this embodiment, when the rotation angle with respect to the first three-dimensional information exceeds 360 degrees, it is determined that the work model WM has made one rotation with respect to the stereo camera (ST107). Further, the processes of ST101 to ST107 are repeated until it is determined that the work has made one rotation.

  When it is determined that the work model WM has made one rotation with respect to the first three-dimensional information, the loop of ST101 to 107 is terminated, and an image representing each three-dimensional information is displayed. Is accepted (ST108). Then, using one of the selected three-dimensional information as a reference, the other three-dimensional information is aligned with the reference three-dimensional information (ST109), and each three-dimensional information after the alignment is integrated (ST110).

  Note that the process of selecting the integration target three-dimensional information may be automatically performed without depending on the user. For example, based on the rotation angle recognized in ST106 accompanying each measurement, the three-dimensional information restored by the measurement when the work model WM is rotated by a predetermined angle or more can be selected as the integration target.

  When the integration process ends, information corresponding to the mark M is deleted from the integrated three-dimensional information (ST111), and the three-dimensional information after deletion is provisionally registered as a three-dimensional model (ST112). Instead of deleting the information of the mark M, it may be invalidated so as not to be a collation target.

  Thereafter, the three-dimensional information that has not been selected as the integration target by the temporarily registered three-dimensional model is collated in order, and the positional deviation amount and the rotation angle with respect to the three-dimensional model are calculated. It is determined whether or not correct recognition has been performed by comparing with a value derived from a positional deviation amount and a rotation angle obtained for each measurement point (ST113, 114). Here, when it is determined that the three-dimensional information to be collated has not been correctly recognized (ST114 is “NO”), the three-dimensional information used for the collation is aligned with the three-dimensional model to obtain a three-dimensional model. Add (ST115). On the other hand, when the three-dimensional information to be collated can be correctly recognized (“YES” in ST114), the process proceeds to the next without adding the three-dimensional information to the three-dimensional model.

  In this way, the three-dimensional information excluded from the integration is sequentially targeted, it is determined whether the three-dimensional information can be correctly recognized by the integrated information, and the three-dimensional information that has been determined to have an error in recognition is determined as 3 Add to a dimensional model. As a result of such processing, 3D information that could not be recognized by the model is added to the temporarily registered 3D model, so that the same erroneous recognition does not occur in the subsequent 3D model. Accuracy can be improved. On the other hand, since the correctly recognized 3D information is not added to the 3D model, the capacity of the 3D model can be prevented from increasing.

  Thereafter, in order to further improve the accuracy of the three-dimensional model, a process of removing noise is executed (ST117). For example, among the three-dimensional segments constituting the three-dimensional model, those having a length equal to or smaller than a predetermined threshold and those having a three-dimensional coordinate sample point equal to or smaller than a predetermined number are deleted. Further, a shadow corresponding to a shadow is detected from the distribution state of the three-dimensional coordinates on the XZ plane, and this is deleted.

  In this way, by deleting a short three-dimensional segment, a three-dimensional segment with a small number of sample points, a three-dimensional segment representing a shadow, and the like, it is possible to prevent erroneous matching in the matching process. Further, since the capacity of the three-dimensional model is reduced, the time for the collation process can be shortened.

  When the noise removal is finished, the three-dimensional model after the noise removal is fully registered (ST118), and the process is finished.

  As described above, in this embodiment, the measurement direction for the work model WM is changed little by little, and the three-dimensional information including the information of the triangular mark M is collated each time, thereby the three-dimensional information obtained immediately before is checked. Since the displacement amount and the rotation angle are obtained with high accuracy, it is possible to ensure the accuracy of alignment between the integrated three-dimensional information. Also, even if the user tries to increase the accuracy of the 3D model and gives a stereo measurement instruction by changing the measurement direction very finely every time, as long as the 3D information restored by the measurement is not erroneously recognized, 3 Dimensional model information does not increase. Therefore, it is possible to create a three-dimensional model that can cope with all the measurement directions set by the user and has a reduced capacity.

  Further, according to the above-described embodiment, it is necessary to perform stereo imaging with the same mark M included in the field of view of each of the cameras 11 to 13, but it is necessary to strictly adjust the rotation angle of the work model WM. There is no need for equipment such as a rotary table. Therefore, since the accurate three-dimensional model can be created by moving the work model WM by an appropriate angle, the burden on the user can be reduced.

  When the three-dimensional information of the work model WM does not include many elements that cause an error in association, the work model WM without the mark M may be used to execute the processing of FIG. The accuracy of the three-dimensional model can be ensured.

  Next, in the above embodiment, whether or not the recognition process by the three-dimensional model is correct is automatically determined. However, the present invention is not limited to this, and a confirmation screen as shown in FIG. Good.

  This screen is provided with a window 30 for imaging and displaying three-dimensional information, and two selection buttons 31 and 32 for “add model” and “next”. In the window 30, the contour line representing the three-dimensional information restored by the measurement and the contour line representing the three-dimensional model are displayed in different colors (in the figure, shown in place of the line thickness). . It should be noted that those corresponding to the mark M are deleted from the contour lines representing the restored three-dimensional information.

  A contour line (thick line) representing a three-dimensional model represents an image obtained by coordinate-transforming a temporary three-dimensional model at the present time according to a positional deviation amount and a rotation angle recognized by collation with three-dimensional information. It is. Further, in this window 30, the degree of coincidence of the three-dimensional information with respect to the three-dimensional model is also displayed.

  The user determines whether or not the displayed three-dimensional information is correctly recognized from the display of the relationship between the contour lines and the degree of coincidence. If the user determines that the three-dimensional information being displayed is not correctly recognized, The button 31 is operated. When this operation is performed, coordinate conversion for aligning the displayed three-dimensional information with the temporary three-dimensional model is performed, and processing for adding the converted three-dimensional information to the temporary three-dimensional model is performed. .

  On the other hand, when it is determined that the displayed three-dimensional information is correctly recognized, the user operates the “next” button 32. When this operation is performed, the update process of the 3D model is not performed, and the next confirmation screen of 3D information is displayed.

  As described above, according to the method in which the user visually determines whether or not the recognition result based on the temporary three-dimensional model is correct, even if the calculated degree of coincidence is slightly bad, a recognition result that does not hinder the control of the robot 4 is obtained. If so, it is possible to take a flexible measure such as determining that correct recognition is being performed. Therefore, the capacity of the three-dimensional model can be further reduced.

  In addition, the above confirmation screen is not limited to the purpose of confirming the three-dimensional information added to the three-dimensional model, but is a process of comparing the three-dimensional information restored by the stereo measurement performed each time with the three-dimensional information of the previous step ( It can also be used for the purpose of confirming the accuracy of ST105) in FIG. For example, each time the second and subsequent stereo measurements are performed and the three-dimensional information restored by the measurement is collated with the previous three-dimensional information, a confirmation screen indicating the collation result is displayed and the collation result is displayed to the user. The judgment input regarding the suitability of the is performed. Then, when the determination result indicating that the recognition result of the positional deviation amount and the rotation angle with respect to the previous three-dimensional information is correct is input, the process proceeds to the next stereo measurement, and the determination result indicating that the recognition result is incorrect is input. If it is, the measurement result is discarded. In addition, when the user determines that the recognition result is not correct, after inputting the determination result, the user changes the measurement direction to be closer to the previous measurement direction and performs stereo measurement again. In this way, the accuracy of the correspondence between the three-dimensional information restored by the stereo measurement each time is ensured, so that the accuracy of the process of aligning the three-dimensional information to be added to the temporary three-dimensional model is also increased. be able to.

3-3) Other methods related to creation of 3D model In the above embodiment, the 3D model is created after executing all stereo measurements. However, the present invention is not limited to this. It is also possible to proceed with the creation of a dimensional model.

  This will be specifically described below. Also in this case, the measurement is executed by rotating the work model WM at an arbitrary angle with respect to the Y1 axis every time, but the three-dimensional information restored by the first measurement is initially set as a temporary three-dimensional model. Further, in the subsequent measurement, the positional deviation amount and the rotation angle with respect to the three-dimensional information one step before the three-dimensional information restored by the measurement are obtained, and recognition processing by a temporary three-dimensional model is executed. Determine whether it is correct. If it is determined that the recognition processing result is not correct, coordinate conversion is performed so that the three-dimensional information being processed matches the temporary three-dimensional model, and the converted three-dimensional information is added to the temporary model. To do.

  Even in this method, the 3D information that could not be correctly recognized is subjected to coordinate conversion processing based on the positional deviation amount and the rotation angle with respect to the previous 3D information obtained by each measurement, and the converted 3D information. Is added to the temporary three-dimensional model. Therefore, by adjusting the variation width in the measurement direction so that a considerable amount of information included in the previous three-dimensional information is restored in each measurement, the three-dimensional information corresponding to the previous step is adjusted. If the accuracy of collation is ensured, the 3D information to be added can be accurately aligned with the temporary 3D model. In addition, the recognition result may be automatically determined using the positional deviation amount and the rotation angle obtained for each measurement as in the previous embodiment, or using a confirmation screen having the same configuration as in FIG. Thus, a determination input from the user may be received.

  Moreover, it is also possible to take a method in which the above-described embodiment and the embodiment shown in 3-2) above are compromised. For example, first, an appropriate number of stereo measurements are executed by changing the hourly measurement direction, and a temporary three-dimensional model is created from the three-dimensional information restored by these. Thereafter, the process of setting the measurement direction not corresponding to the temporary three-dimensional model and performing the stereo measurement is repeated a plurality of times, and the restored three-dimensional information is recognized by the temporary three-dimensional model for each measurement. To do. Then, the three-dimensional information determined as the result of the recognition process is added to the provisional three-dimensional model after being registered, and the three-dimensional model when all measurements are completed is fully registered.

Even in the case of taking the above method, each time the measurement is performed, the three-dimensional information restored by the measurement is compared with the three-dimensional information restored one step before, and the positional deviation amount and the rotation angle of both are calculated. Is used to perform coordinate conversion processing on the three-dimensional information that could not be recognized correctly, and add the converted three-dimensional information to the three-dimensional model.
However, in this method, in the first measurement after the provisional three-dimensional model is created, it is necessary to restore three-dimensional information that can be correctly recognized by the provisional three-dimensional model (for example, the provisional three-dimensional model). The measurement direction is set within a predetermined angle range from the measurement direction corresponding to any of the plurality of three-dimensional information integrated into the result, or the result of the recognition process for the first measurement after the provisional three-dimensional model is created After the user confirms whether or not is correct, proceed with the subsequent processing.) If the correspondence between the three-dimensional information restored by the first measurement and the temporary three-dimensional model cannot be specified, the correspondence between the three-dimensional information restored by the subsequent measurement and the temporary three-dimensional model is specified. Because it is impossible. In the subsequent measurement, it is necessary to consider the fluctuation range of the measurement direction so as to ensure the accuracy of collation with respect to the three-dimensional information of the previous stage.

Subsequently, an embodiment in which the arithmetic processing is simplified will be described.
In this embodiment as well, after the three-dimensional information restored by the first three-dimensional measurement is initially set in the temporary three-dimensional model, information is added to the temporary three-dimensional model in parallel with the progress of the subsequent measurement. Is. However, in this embodiment, the work model WM without the mark M is used, and the processing for obtaining the positional deviation amount and the rotation angle by comparing with the three-dimensional information one step before every measurement point is not performed. For this reason, in this embodiment, instead of automatically determining whether or not the recognition result based on the temporary three-dimensional model is correct, the user visually determines the recognition result and accepts the input of the determination result.

  The confirmation / determination of the recognition result is performed on a confirmation screen similar to that shown in FIG. Specifically, when the user determines that the displayed three-dimensional information is not correctly recognized and operates the “add model” button 31, the display screen performs an update process of the three-dimensional model described later. Change to the guidance screen. When the user proceeds with the measurement in accordance with the instruction on this screen, the three-dimensional model is updated to a new information added.

  On the other hand, when it is determined that the displayed three-dimensional information is correctly recognized, the user operates the “next” button 32. When this operation is performed, the three-dimensional model update process is not performed, and the process proceeds to the next measurement.

  Further, the confirmation screen in this case is also provided with a selection button for instructing re-measurement when it is determined that the restored three-dimensional information is not good. When this button is operated, the restored three-dimensional information is discarded and measurement is performed again.

  FIG. 14 is a flowchart when a three-dimensional model is created and registered by the above method. Hereinafter, a method for creating a three-dimensional model according to this embodiment will be described in detail with reference to step codes in this flowchart.

  Also in the process according to this flowchart, it is assumed that the process of rotating the work model WM at an arbitrary angle with respect to the Y1 axis and performing the three-dimensional measurement is executed until the work model WM rotates almost once. In addition, the rotation direction of the work model WM is assumed to be constant except when the update process of the three-dimensional model is executed (hereinafter, this rotation direction is referred to as “positive rotation direction”).

  First, the first stereo measurement is performed (ST201). Step ST201 also executes processing corresponding to ST1 to ST6 in FIG. Next, the three-dimensional information restored by the first measurement is provisionally registered as a three-dimensional model (ST202).

  Thereafter, the work model WM is rotated by an arbitrary angle, and the next stereo measurement is executed (ST203). Then, the three-dimensional information restored by the stereo measurement is collated with a provisional three-dimensional model, and the recognition result by the collation is displayed (ST204, 205).

The user confirms whether or not the three-dimensional information is correctly restored and whether the recognition by the three-dimensional model is correctly performed on the display screen. If the user determines that any processing is correctly performed and inputs the determination result, it is determined whether the work model WM has made one rotation from the recognized rotation angle (ST208). If it is determined that one rotation has not been made (ST208 is “YES”), the process returns to ST203.
In this case, the next stereo measurement is executed by further rotating the work model WM in the positive rotation direction.

  On the other hand, when the user determines that the three-dimensional information has not been correctly restored and inputs the determination result (ST207 is “YES”), the process returns to ST203 without confirming whether or not the rotation has been performed once. In this case, the three-dimensional information restored by the immediately preceding stereo measurement is discarded and the next stereo measurement is started.

  Next, when the user determines that the recognition result of the three-dimensional information is not correct and inputs the determination result (ST206 is “YES”), the process proceeds to stereo measurement for model update (ST211). At this time, the display on the display unit 27 is changed to the above-described guidance screen, and the work model WM is rotated with the rotation direction reversed (in the negative direction) so as to approach the previous measurement state. Is instructed to do so.

  When stereo measurement for model update is performed, the three-dimensional information restored by this measurement is collated with the three-dimensional model, and the recognition result by the collation is displayed (ST212, 213). Also in the display screen in this case, the recognized relationship between the three-dimensional information and the three-dimensional model is displayed as in the example of FIG. As the selection buttons, a button for selecting to add the displayed three-dimensional information to the three-dimensional model and a button for selecting not to add are provided.

  Here, when the user selects “do not add” (ST214 is “NO”), stereo measurement for model update (ST211) is executed again, and accordingly, steps ST212 and 213 are executed. In the stereo measurement in this case, the work model WM is not limited to be rotated in the negative direction but may be rotated in the positive direction (for example, the work model WM in the previous measurement in the negative direction). For example, if the rotation is too large.)

  When the user selects to add the displayed three-dimensional information to the three-dimensional model (ST214 is “YES”), the three-dimensional information restored by the stereo measurement for model update is updated with the three-dimensional model. Coordinate conversion is performed based on the positional deviation amount and the rotation angle recognized by the collation (ST215). Then, the converted three-dimensional information is added to the three-dimensional model (ST216).

  When the three-dimensional model is updated by this process, a process of notifying the user that the update has been performed is executed (ST217). In the screen at this stage, in addition to the update of the three-dimensional model, the rotation direction of the work model WM is returned to the positive direction, and it is urged to set again the vicinity of the measurement direction when the correct recognition is not performed first. Guidance is displayed.

Thereafter, the process returns to the stereo measurement of ST203, and ST204 and 205 are further performed in accordance with this measurement process.
If the user changes the posture of the work WM every time according to the displayed instructions, the 3D information related to the features included in the 3D information when the previous recognition was mistaken is added to the 3D model, and the recognition is incorrect. Since the recognition accuracy for the measured direction is improved, it is highly possible that a correct recognition result is obtained at this stage. Even if it is determined again that the recognition result is incorrect, the recognition accuracy can be further improved by rotating the work model WM again in the negative direction and updating the model.

  Similarly, the process of performing stereo measurement by rotating the work model WM at an arbitrary angle is repeated, and for each measurement, the restored three-dimensional information is collated with the three-dimensional model, and whether the recognition result by the collation is correct or not Is determined by the user. When the user determines that the result of the recognition process is not correct, the measurement is performed with the measurement direction close to the direction in which the previous correct recognition was performed, and it was determined that the correct recognition was performed. The coordinates of the three-dimensional information are converted based on the recognition result and added to the three-dimensional model.

If it is determined from the recognition result determined to be correct that the work model WM has made one rotation (ST208 is “YES”), a series of processing relating to measurement is terminated. Also, the user is notified that the measurement has been completed by a method such as displaying a predetermined message on the display unit 27.
Thereafter, processing for removing noise from the three-dimensional model (same processing as ST117 in FIG. 12) is executed (ST209), and the three-dimensional model after noise removal is fully registered (ST210).

  According to the above method, if the user rotates the work model WM according to a correct procedure and performs a selection operation, an appropriate three-dimensional model can be created before the work model WM is rotated. In addition, it is not necessary to obtain the positional deviation amount and the rotation angle every time the second and subsequent measurements are performed, so that the calculation can be simplified.

3-4) Other Examples Regarding Method for Creating 3D Model In the above, the embodiment in which the user creates the 3D model by the method in which the measurement direction with respect to the work model WM is changed at an arbitrary angle has been described. Even when the work model WM is regularly rotated by using the above, it is possible to create a three-dimensional model with high accuracy and reduced capacity by applying the above examples.

  When stereo measurement is performed by changing the measurement direction by a certain angle using a rotary table or the like, the restored three-dimensional information can be obtained without taking a method of collating with the previous three-dimensional information every time measurement is performed. The positional relationship between them can be easily specified. Therefore, it is possible to simplify the calculation for creating the three-dimensional model.

Next, in the processing shown in FIGS. 12 and 14, the stereo measurement is continued while the work model WM makes one rotation with respect to one axis (Y1 axis) of the work coordinate system. When the assumed range of change in orientation is smaller than 360 degrees, the measurement may be terminated when the work model WM rotates within the assumed range.
Further, instead of automatically determining the end time of measurement based on the rotation angle, the recognized rotation angle may be displayed to allow the user to determine the end time.

  Further, the rotation axis of the work model WM is not limited to one axis, and the work model WM may be rotated about a plurality of axes as shown in FIG.

  FIG. 15 shows an example of measurement processing when a three-dimensional model is created using a work model WM with two marks M1 and M2. As shown in FIG. 15 (1), one mark M1 is attached at a position that becomes the upper surface when the work model WM is arranged so that the Y1 axis of the work coordinate system is parallel to the Y axis, and the other mark M2 is attached to the side surface of the work model WM.

  In this embodiment, as in the example shown in FIG. 4, the measurement is performed a plurality of times while rotating the work model WM with respect to the Y1 axis (FIG. 15 (1)). When a series of measurements is completed, the rotation axis is changed from the Y1 axis to the X1 axis, and the work model WM is rotated 90 degrees as shown in FIG. Further, as shown in FIG. 15 (3), the work model WM is rotated again by 90 degrees to perform measurement.

  When measurement is performed by rotating the work model WM with respect to the Y1 axis, the mark M1 is always included in the field of view of each camera 11 to 13, and measurement is performed by rotating the work model WM with respect to the X1 axis. When performing, the mark M2 is always included in the visual field of each camera 11-13. Further, at least when shifting from the rotation about the Y1 axis to the rotation about the X1 axis, measurement is performed by setting a state in which the two marks M1 and M2 are both included in the fields of view of the cameras 11-13.

  Even when the work model WM is rotated with respect to the X1 axis, every time the work model WM is rotated by 90 degrees, an axis orthogonal to the XZ plane under the state (in the case of FIG. 15 (2), the Z1 axis, FIG. In the case of 3), by rotating the work model WM relative to the Y1 axis), the posture of the work model WM may be changed in a plurality of ways, and measurement may be performed each time the change is made.

  As described above, even when the variation of the measurement direction with respect to the work model WM is increased, the processing according to the flowchart of FIG. 12 can be executed. In this case, while the workpiece model WM is rotated with respect to the Y1 axis, the accuracy of collation can be ensured by the presence of the mark M1, and the workpiece model WM is rotated with respect to the X1 axis. While the measurement is being performed, the accuracy of the collation can be ensured by the presence of the mark M2. Therefore, the accuracy of the created three-dimensional model is also ensured.

  Further, when the orientation of the work model WM with respect to the stereo camera 1 is changed more freely without managing the direction of the axis rotation of the work model WM as in the example of FIG. 15, the processing according to the flowchart of FIG. By adding the necessary three-dimensional information to the temporary three-dimensional model while the user confirms the measurement result every time, it is possible to create a three-dimensional model with high accuracy and reduced capacity.

It is a figure which shows schematic structure of the picking system to which a three-dimensional recognition process is applied. It is a block diagram of an object recognition apparatus. It is a flowchart which shows the procedure of a three-dimensional recognition process. It is a figure which shows the measuring method of a work model. It is a figure which shows the general | schematic procedure of the creation process of a three-dimensional model. It is a figure explaining the cause which an error produces in matching of three-dimensional information. It is a figure explaining the other cause which an error produces in matching of three-dimensional information. It is a figure showing the characteristic of the triangle which a mark represents. It is a figure showing a plurality of correspondence states of each vertex of a triangle, and a graph showing a degree of coincidence. It is a figure which shows the example of the three-dimensional information of the mark used for selection of the integration target three-dimensional information. It is a figure which shows typically the specific method of the preparation process of a three-dimensional model. It is a flowchart which shows the procedure regarding preparation and registration of a three-dimensional model. It is a figure which shows the example of the confirmation screen of the recognition result by a three-dimensional model. It is a flowchart which shows the other example of the procedure regarding preparation and registration of a three-dimensional model. It is a figure which shows the example of the measuring method by the work model which attached two marks.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Stereo camera 2 Object recognition apparatus 11, 12, 13 Camera 24 CPU
25 Memory W Work WM Work Model

Claims (7)

By integrating a plurality of three-dimensional information restored by performing three-dimensional measurement using a stereo camera from different directions with respect to the real model of the recognition object, the position of the recognition object is integrated. A method for creating a three-dimensional model,
Creating a temporary 3D model using 3D information restored by at least one 3D measurement of the real model;
With respect to the three-dimensional measurement performed in a direction different from the direction when the three-dimensional information used to create the temporary three-dimensional model is restored for the real model, the three-dimensional information restored by the measurement is Recognize with a temporary 3D model to determine whether the recognition result is correct,
The three-dimensional information that is a recognition target when it is determined that the result of the recognition process by the temporary three-dimensional model is incorrect, or the angle deviation with respect to the measurement direction when the three-dimensional information of the recognition target is restored is within a predetermined value. The three-dimensional information restored by the measurement from the direction and the result of the recognition process by the three-dimensional model determined to be correct is subjected to coordinate conversion so as to match the temporary three-dimensional model, and the converted three-dimensional information Adding the dimensional information to the provisional three-dimensional model, and confirming the three-dimensional model at the time of completion of the addition process as a model used for recognition of the recognition object;
A method for creating a three-dimensional model. The method of claim 1, wherein
For the actual model, a different measurement direction is set each time and a plurality of three-dimensional measurements are performed, and the three-dimensional information restored by the second and subsequent three-dimensional measurements is restored by the previous measurement. Recognize the correspondence between the three-dimensional information by collating with the three-dimensional information,
Two or more three-dimensional information corresponding to a part of the plurality of three-dimensional information restored in accordance with the plurality of three-dimensional measurements is selected, and one of the selected three-dimensional information is used as a reference. The three-dimensional information is converted into the reference three-dimensional information based on the correspondence relationship between the three-dimensional information recognized for each measurement between the measurement corresponding to the three-dimensional information and the measurement corresponding to the reference three-dimensional information. The coordinates are transformed so as to match, and the provisional three-dimensional model is created by integrating the converted three-dimensional information into the reference three-dimensional information,
Among the plurality of three-dimensional information, three-dimensional information that has not been integrated into the temporary three-dimensional model is sequentially processed, and recognition processing using the temporary three-dimensional model is executed to determine whether the recognition result is correct or not. When it is determined that the result of the recognition process is not correct, the three-dimensional information that is the object of the recognition process is determined between the measurement corresponding to the three-dimensional information and the measurement corresponding to the reference three-dimensional information. Based on the recognized correspondence between the three-dimensional information restored for each measurement, coordinate conversion is performed to match the reference three-dimensional information, and the converted three-dimensional information is added to the temporary three-dimensional model,
A method for creating a three-dimensional model, in which, when processing for all three-dimensional information that has not been integrated into the three-dimensional model is completed, the three-dimensional model at that time is determined. The method of claim 1, wherein
After creating a temporary three-dimensional model using a plurality of three-dimensional information restored by a plurality of three-dimensional measurements performed by setting different measurement directions each time for the real model, A process of executing the 3D measurement by changing the measurement direction every time so as to be different from any of the measurement directions set at the time of creating the temporary 3D model is executed a plurality of times.
After the recognition process by the temporary 3D model is performed on the 3D information restored by the first 3D measurement of the plurality of 3D measurements, the measurement is performed each time after the second measurement. A first step of recognizing the correspondence between the three-dimensional information by comparing the restored three-dimensional information with the three-dimensional information restored one step ago, and targeting the three-dimensional information restored by the measurement And a second step of executing a recognition process using the provisional three-dimensional model and determining whether the recognition result is correct or not, and a result of the recognition process for the three-dimensional information restored by the second and subsequent measurements. Is determined to be incorrect, the recognition result for the three-dimensional information restored by the first three-dimensional measurement and the first step for each three-dimensional measurement performed from the second time to the present time Based on the recognition result, coordinate conversion is performed so that the three-dimensional information determined to be incorrect is matched with the temporary three-dimensional model, and the converted three-dimensional information is added to the temporary three-dimensional model. And
A method for creating a three-dimensional model, in which, when the plurality of three-dimensional measurements and the processes associated therewith are all completed, a three-dimensional model at that time is determined. The method of claim 1, wherein
A process of performing three-dimensional measurement by changing the measurement direction with respect to the real model each time is executed a plurality of times,
The three-dimensional information restored by the first three-dimensional measurement is set as a temporary three-dimensional model, and in the subsequent three-dimensional measurement, the three-dimensional information restored by the three-dimensional measurement is restored one step before. A first step of recognizing the correspondence between the three-dimensional information by collating with the three-dimensional information, and a recognition process using the temporary three-dimensional model for the three-dimensional information restored by the three-dimensional measurement. And executing a second step of determining whether the recognition result is correct or not,
When it is determined in the second step that the result of the recognition process using the three-dimensional model is not correct, the three-dimensional information that is the object of the recognition process is determined for each three-dimensional measurement from the second time to the current time. Based on the recognition result of the first step executed in the above, coordinate conversion is performed so as to match the temporary 3D model, and the converted 3D information is added to the temporary 3D model,
A method for creating a three-dimensional model, in which, when the plurality of three-dimensional measurements and all the processes associated therewith are completed, a three-dimensional model at that time is determined. The method of claim 1, wherein
The three-dimensional information restored by the first three-dimensional measurement is set as a temporary three-dimensional model on condition that the real model is rotated at an arbitrary angle and the process of executing the three-dimensional measurement is executed a plurality of times. Then, each time a subsequent three-dimensional measurement is performed, a recognition process using the provisional three-dimensional model is performed on the three-dimensional information restored by the measurement to determine whether the recognition result is correct or not,
When it is determined that the result of the recognition process by the temporary 3D model is not correct, the 3D measurement is performed by rotating the real model so that the measurement direction with respect to the real model approaches the measurement direction of the previous stage; Performing a recognition process using the temporary three-dimensional model on the three-dimensional information restored by the three-dimensional measurement and determining whether the recognition result is correct or not until the recognition result is determined to be correct. The three-dimensional information corresponding to the recognition result determined to be correct is subjected to coordinate conversion based on the recognition result, and the converted three-dimensional information is added to the temporary three-dimensional model. Rotate the real model in a direction to return to the state when it is determined that
A method for creating a three-dimensional model, in which, when the three-dimensional measurement when the positional relationship between the real model and the stereo camera is in a predetermined state and all the processes associated therewith are completed, the three-dimensional model at that time is determined. 3D measurement means for restoring 3D information of a recognition object by 3D measurement using a stereo camera, and the position of the object to be recognized by collating the restored 3D information with a previously registered 3D model A recognition processing means for recognizing the posture, and a plurality of three-dimensional information restored by performing three-dimensional measurement from different directions on the real model of the recognition target object, by combining the positions and integrating the recognition processing A three-dimensional model registration means for registering the three-dimensional model after creating the three-dimensional model used by the means,
The three-dimensional model registration means includes:
Of the plurality of three-dimensional information restored by the processing of the three-dimensional measuring means for the real model, the three-dimensional information restored by the second and subsequent measurements is collated with the three-dimensional information restored one stage before. A correspondence recognition means for recognizing the correspondence between the three-dimensional information,
Two or more three-dimensional information corresponding to a part of the plurality of three-dimensional information restored by the plurality of three-dimensional measurements is selected, and the other three are selected based on one of the selected three-dimensional information. Based on the correspondence between the three-dimensional information recognized by the correspondence recognition means for each measurement between the measurement corresponding to the three-dimensional information and the measurement corresponding to the reference three-dimensional information. A temporary model creation means for creating the provisional three-dimensional model by converting the coordinates so as to match the three-dimensional information and integrating the converted three-dimensional information with the reference three-dimensional information;
Recognizing the three-dimensional information among the plurality of three-dimensional information that are not integrated into the temporary three-dimensional model in order, recognizing the focused three-dimensional information by the temporary three-dimensional model, and determining whether the recognition result is correct or not A result confirmation means;
When the recognition result confirmation unit determines that the result of the recognition process is not correct, the three-dimensional information under attention at that time is measured corresponding to the three-dimensional information and the measurement corresponding to the reference three-dimensional information. The coordinate conversion is performed to match the reference three-dimensional information on the basis of the correspondence between the three-dimensional information recognized by the correspondence recognition means for each measurement between the measurement and the converted three-dimensional information. Model updating means for updating the provisional three-dimensional model by adding to the three-dimensional model;
Model confirmation means for confirming, as a registration target, a three-dimensional model to which all of the three-dimensional information determined that the result of the recognition process is incorrect is added;
An object recognition device comprising: 3D measurement means for restoring 3D information of a recognition object by 3D measurement using a stereo camera, and the position of the object to be recognized by collating the restored 3D information with a previously registered 3D model The recognition processing means for recognizing the posture and a plurality of three-dimensional information restored by performing three-dimensional measurement from different directions with respect to the real model of the recognition target object, by combining the positions and integrating the recognition An apparatus comprising a three-dimensional model registration means for registering the three-dimensional model after the three-dimensional model used by the processing means is created,
The three-dimensional model registration means includes:
Temporary model setting means for initially setting the three-dimensional information restored by the measurement to a temporary three-dimensional model in response to the first measurement performed by the three-dimensional measurement means on the real model;
Each time the three-dimensional measuring unit performs the second and subsequent measurements on the real model, a recognition process using the temporary three-dimensional model is performed on the three-dimensional information restored by the measurement, and the recognition result Recognition result confirmation means for accepting a determination input for correctness;
When the recognition result confirming unit accepts a determination input indicating that the result of the executed recognition process is not correct, the processing by the recognition result confirming unit among the three-dimensional information restored from the stereo image input after the acceptance The first information that has received a determination input indicating that the result of the recognition process using the temporary three-dimensional model is correct is subjected to coordinate conversion based on the recognition result, and the converted three-dimensional information is converted into the temporary three-dimensional model. A model updating means for updating the three-dimensional model by adding to
A model confirmation unit for confirming a three-dimensional model at that time as a registration target when the processing of the three-dimensional measurement for the real model and the recognition result confirmation unit and the model update unit associated therewith are completed,
An object recognition device comprising:

Images