JP2014029664A - Image comparison range generation method, positional orientation detection method, image comparison range generation device, positional orientation detection device, robot, robot system, image comparison range generation program and positional orientation detection program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve detection accuracy of positions or orientation of an object.SOLUTION: An image comparison range generation method includes: feature acquisition procedure where a feature acquisition section 331 acquires feature points of a model represented in a template image and magnitude of features of the feature points for each template image with each of a plurality of images that are images of the model representing contours of an object, and each have the model different from one another in at least any of orientation and positions as the template image; and image comparison range acquisition procedure where an image comparison range acquisition section 334 acquires an image range used for comparing a photographed image of the object and the template image on the basis of the feature points of the model and the magnitude of the features of the feature points acquired by the feature acquisition section 331 for each of the template images as an image comparison range.

Description

  The present invention relates to a comparative image range generation method, a position and orientation detection method, a comparative image range generation device, a position and orientation detection device, a robot, a robot system, a comparative image range generation program, and a position and orientation detection program.

  One of the methods for performing three-dimensional object recognition is a model-based matching method. In this method, a three-dimensional object model (hereinafter also referred to as a CAD model) of an object whose position and orientation are to be detected is stored in advance in a computer, and the CAD model is compared with an image obtained by capturing the object. Then, the object is searched from the image, and the three-dimensional position and orientation of the object are detected.

  For example, Patent Document 1 discloses a model data creation method for recognizing a position and orientation of an object by matching a set of feature data of the object and feature data obtained from an input image. Several types of feature data are disclosed. In Patent Document 1, coordinate conversion of these feature data is performed, and after all feature data is matched to the base coordinates, necessary portions are cut out from each feature data, and all the cut out feature data are superimposed. Creating a base data for a model is disclosed. Further, Patent Document 1 discloses a model data creation method based on a model-based matching method in which data obtained by averaging similar neighboring feature data is registered as model data.

JP-A-8-96133

  In a general model-based matching method, since the CAD model does not completely match the actual object, the desired matching accuracy cannot be achieved. In Patent Document 1, a necessary portion is cut out from each feature data. However, depending on how to cut out the necessary portion, a desired matching accuracy cannot be achieved. As described above, there has been a problem that the position or orientation of the target cannot be detected with high accuracy unless the desired matching accuracy can be achieved.

  Therefore, the present invention has been made in view of the above problems, and a comparison image range generation method, a position / orientation detection method, a comparison image range generation device, which can improve the detection accuracy of the position or orientation of an object, It is an object to provide a position and orientation detection device, a robot, a robot system, a comparative image range generation program, and a position and orientation detection program.

(1) The present invention has been made in view of the above circumstances. In one aspect of the present invention, the feature acquisition unit is an image of a model that represents the contour of an object, and the position or orientation of the model in the image. A feature acquisition procedure for acquiring, as template images, a plurality of images at least one of which is different from each other, and for each template image, the feature points of the model represented in the template images and the feature sizes of the feature points; A comparison image range acquisition unit, for each of the template images, based on the feature points of the model acquired by the feature acquisition unit and the feature size of the feature points; A comparison image range acquisition procedure for acquiring an image range used for comparison with a template image as a comparison image range. A comparison image range generation process for.
According to this configuration, since the comparison image range is acquired based on the feature point of the model represented in the template image and the feature size of the feature point, the larger the feature point feature, the more included in the comparison image range. The ratio of the area around the feature point can be increased. Thereby, regarding the comparison image range, either the position or the posture is estimated by comparing between the template image and the captured image obtained by photographing the object, so that the position or posture estimation accuracy is improved. be able to.

(2) In the comparative image range generation method described above, according to one aspect of the present invention, a low image area is obtained in which the processing accuracy of the object is lower than a predetermined accuracy for each template image. In the comparison image range acquisition procedure, the comparison image range acquisition unit further includes a feature point of the model acquired by the feature acquisition unit and the feature point of the template image in the comparison image range acquisition procedure. The comparison image range is acquired based on the size of the feature and the image region acquired by the low processing accuracy region acquisition unit.
According to this configuration, it is possible to exclude information of a portion with low processing accuracy from the feature points of the model and the object itself. As a result, regarding the comparison image range, either the position or the posture is estimated by comparing between the template image and the photographed image obtained by photographing the object, so that the object has a variation in processing accuracy. Even so, the estimation accuracy of the position or orientation can be improved.

(3) According to one aspect of the present invention, each of the plurality of images in which the acquisition unit is an image of a model representing the contour of an object and at least one of the position or orientation of the model in the image is different from each other. As a template image, for each template image, an image of the object captured and the template image acquired based on the feature points of the model represented in the template image and the feature size of the feature points An acquisition procedure for acquiring a comparison image range that is an image range used in the comparison, a captured image acquisition unit, a captured image acquisition procedure for acquiring an image in which the object is captured, and a position and orientation estimation unit, For each of the plurality of template images acquired by the acquisition unit, the comparison image range corresponding to the template image acquired by the acquisition unit is included in the comparison image range. And a position and orientation estimation procedure for estimating at least one of the position and orientation of the object by comparing with the image acquired by the captured image acquisition unit. Is the method.
According to this configuration, since the comparison image range is acquired based on the feature point of the model represented in the template image and the feature size of the feature point, the larger the feature point feature, the more included in the comparison image range. The ratio of the area around the feature point can be increased. Thereby, regarding the comparison image range, either the position or the posture is estimated by comparing between the template image and the captured image obtained by photographing the object, so that the position or posture estimation accuracy is improved. be able to.

(4) According to one aspect of the present invention, each of the plurality of images in which the feature acquisition unit is an image of a model representing the contour of the object, and at least one of the position or orientation of the model in the image is different from each other For each template image, the feature acquisition procedure for acquiring the feature point of the model represented in the template image and the feature size of the feature point, and the comparison image range acquisition unit Based on the feature points of the model acquired by the feature acquisition unit and the size of the features of the feature points, the image range used in the comparison between the image obtained by photographing the object and the template image is determined. A comparative image range acquisition procedure to be acquired as a comparative image range, and a captured image acquisition unit that captures an image in which the object is captured. The image acquisition procedure and the position / orientation estimation unit are acquired by the captured image acquisition unit for each of the plurality of template images with respect to the comparative image range corresponding to the template image acquired by the comparative image range acquisition unit. A position and orientation estimation procedure for estimating at least one of the position and orientation of the object by comparing with an image.
According to this configuration, since the comparison image range is acquired based on the feature point of the model represented in the template image and the feature size of the feature point, the larger the feature point feature, the more included in the comparison image range. The ratio of the area around the feature point can be increased. Thereby, regarding the comparison image range, either the position or the posture is estimated by comparing between the template image and the captured image obtained by photographing the object, so that the position or posture estimation accuracy is improved. be able to.

  (5) According to one aspect of the present invention, each of a plurality of images that are images of a model representing the contour of an object and at least one of the positions or orientations of the model in the image is a template image. For each template image, a feature acquisition unit that acquires a feature point of the model represented in the template image and a feature size of the feature point, and a model acquired by the feature acquisition unit for each template image A comparison image range acquisition unit that acquires, as a comparison image range, an image range that is used in comparison between the image of the object and the template image based on the feature point of the feature point and the feature size of the feature point; And a comparative image range generation device.

  (6) According to one aspect of the present invention, each of a plurality of images that is an image of a model that represents an outline of an object and in which at least one of the position or orientation of the model in the image is different from each other is used as a template image. For each template image, used for comparison between the template image and the image obtained by photographing the object, which is obtained based on the feature points of the model represented in the template image and the feature size of the feature points For each of the plurality of template images acquired by the acquisition unit that acquires a comparison image range that is an image range to be acquired, a captured image acquisition unit that acquires an image in which the object is captured, and the acquisition unit The comparison image range corresponding to the template image acquired by the unit is compared with the image acquired by the captured image acquisition unit. By a position and posture detecting device characterized by comprising a position and orientation estimation unit for estimating at least one, the one of the position or orientation of the object.

  (7) One aspect of the present invention is an image of a model representing the contour of an object, and each of a plurality of images in which at least one of the position or orientation of the model in the image is different from each other is used as a template image. For each template image, a feature acquisition unit that acquires a feature point of the model represented in the template image and a feature size of the feature point, and a model acquired by the feature acquisition unit for each template image A comparison image range acquisition unit that acquires, as a comparison image range, an image range that is used in comparison between the image of the object and the template image based on the feature point of the feature point and the feature size of the feature point; A robot characterized by comprising:

  (8) According to one aspect of the present invention, each of a plurality of images representing an outline of a model of an object and having at least one of the position or orientation of the model in the image as a template image, For each template image, used for comparison between the template image and the image obtained by photographing the object, which is obtained based on the feature points of the model represented in the template image and the feature size of the feature points For each of the plurality of template images acquired by the acquisition unit that acquires a comparison image range that is an image range to be acquired, a captured image acquisition unit that acquires an image in which the object is captured, and the acquisition unit The comparison image range corresponding to the template image acquired by the unit is compared with the image acquired by the captured image acquisition unit. By a robot, characterized in that it comprises a position and orientation estimation unit that estimates at least one of the position or orientation of the object.

  (9) One aspect of the present invention is an image of a model representing the contour of an object, and a plurality of images in which at least one of the position or orientation of the model in the image is different from each other as template images, For each template image, a feature acquisition unit that acquires a feature point of the model represented in the template image and a feature size of the feature point, and a model acquired by the feature acquisition unit for each template image A comparison image range acquisition unit that acquires, as a comparison image range, an image range that is used in comparison between the image of the object and the template image based on the feature point of the feature point and the feature size of the feature point; A robot system comprising:

  (10) According to one aspect of the present invention, each of a plurality of images representing an outline of a model of a target object and at least one of the positions or orientations of the models in the image is a template image. For each template image, used for comparison between the template image and the image obtained by photographing the object, which is obtained based on the feature points of the model represented in the template image and the feature size of the feature points For each of the plurality of template images acquired by the acquisition unit that acquires a comparison image range that is an image range to be acquired, a captured image acquisition unit that acquires an image in which the object is captured, and the acquisition unit An image acquired by the captured image acquisition unit with respect to the comparative image range corresponding to the template image acquired by the unit; By compare a robotic system, characterized by comprising a position and orientation estimation unit that estimates at least one of the position or orientation of the object.

  (11) In one embodiment of the present invention, each of a plurality of images of a model representing an outline of an object and having at least one of the position and orientation of the model in the image is templated in a computer As each image, for each template image, a feature acquisition step for acquiring a feature point of the model represented in the template image and a feature size of the feature point, and for each of the template images, acquired by the feature acquisition step. A comparison image range that acquires, as a comparison image range, an image range that is used for comparison between the image of the object and the template image based on the feature points of the model and the feature size of the feature points A comparison image range generation program for executing the acquisition step.

  (12) In one embodiment of the present invention, a plurality of images, each of which is an image of a model representing the contour of an object and in which at least one of the position or orientation of the model in the image is different, is templated As an image, for each template image, an image of the object photographed on the basis of the feature point of the model represented in the template image and the feature size of the feature point, and the template image An acquisition step for acquiring a comparison image range that is an image range used in comparison, a captured image acquisition step for acquiring an image in which the object is captured, and each of the plurality of template images acquired by the acquisition step The comparison image corresponding to the template image acquired by the acquisition step A position and orientation detection program for executing a position and orientation estimation step for estimating at least one of the position and orientation of the object by comparing with the image obtained by the captured image acquisition step. is there.

  According to the present invention, it is possible to improve the detection accuracy of the position or orientation of an object.

It is a schematic block diagram which shows the structure of the robot system in 1st Embodiment. It is an example of the perspective view of the robot main body in 1st Embodiment. This is a comparison between a captured image obtained by imaging a sheet metal part and a CG image of a CAD model representing the outline of the sheet metal part. It is a schematic block diagram which shows the structure of the hardware of the position and orientation detection apparatus in 1st Embodiment. It is a schematic block diagram which shows the logical structure of the control part in 1st Embodiment. It is a figure for demonstrating the outline | summary of the process of a feature point extraction part. It is a figure for demonstrating the outline | summary of the process of a characteristic area extraction part. It is a figure for demonstrating the outline | summary of the process of a characteristic area image generation part. It is a figure for demonstrating the outline | summary of a process of a high precision area | region image generation part. It is a figure for demonstrating the outline | summary of the process of an duplication range extraction part. It is a figure for demonstrating the outline | summary of the process of a comparative image generation part. It is a flowchart which shows an example of the flow of the process which the position and orientation detection apparatus of 1st Embodiment performs in advance before a position and orientation detection. 5 is a flowchart illustrating an example of a flow of position and orientation detection processing in which the position and orientation detection apparatus of the first embodiment actually detects the position or orientation of an object. It is a schematic block diagram which shows the structure of the robot system in 2nd Embodiment. It is a schematic block diagram which shows the structure of the comparison image range production | generation apparatus in 2nd Embodiment. It is a schematic block diagram which shows the logical structure of the control part of the comparative image range production | generation apparatus in 2nd Embodiment. It is a schematic block diagram which shows the structure of the position and orientation detection apparatus in 2nd Embodiment. It is a schematic block diagram which shows the logical structure of the control part of the position and orientation detection apparatus in 2nd Embodiment. It is an example of the outline external view of the robot system in case the number of the arms of a robot main body is two.

<First Embodiment>
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of the robot system 1 according to the first embodiment. As shown in FIG. 1, the robot system 1 includes a robot body 10, an imaging device 20, a position / orientation detection device 30, and a robot control device 50.

Based on the control of the robot control device 50, the imaging device 20 captures an image of a target object whose position or orientation is to be detected. Then, the imaging device 20 outputs captured image data indicating a captured image obtained by imaging to the position and orientation detection device 30 and the robot control device 50.
The position / orientation detection apparatus 30 includes a comparative image range generation unit 40. The position / orientation detection device 30 detects at least one of the position and orientation of the object based on the captured image data input from the imaging device 20. Then, the position / orientation detection device 30 outputs position / orientation information including at least one of the detected positions and orientations to the robot control device 50.

  The robot control device 50 controls the robot body 10 based on the captured image data input from the imaging device 20 and the position and orientation information input from the position and orientation detection device 30. Specifically, for example, in the robot control device 50, the grip unit of the robot body 10 grips an object based on one or both of the position and orientation included in the position and orientation information input from the position and orientation detection device 30. One or both of the possible gripping positions and the gripping angles at which the gripping part of the robot body 10 can grip the object are calculated. Then, for example, the robot control device 50 controls the gripping part of the robot body 10 to move to the calculated gripping position while referring to the captured image data, and the gripping angle calculated by the gripping part of the robot body 10 is calculated. Control the angle.

When the position and angle of the gripping portion of the robot body 10 reach the calculated gripping position and gripping angle, for example, the robot control device 50 causes the 10 gripping portions of the robot body to grip the object. Control.
For example, the robot body 10 grips the object under the control of the robot control device 50, and conveys the object to a predetermined target position while holding the object.

  FIG. 2 is an example of a perspective view of the robot main body 10 according to the first embodiment. In the figure, a robot body 10, an imaging device 20, a position / orientation detection device 30, a robot control device 50, and an object 200 are installed in the robot system 1. Further, the robot body 10 and the robot control device 50 are connected via the first signal cable 11. The robot body 10 and the robot control device 50 exchange signals with each other via the first signal cable. In addition, the position / orientation detection device 30 and the robot control device 50 are connected via the second signal cable 12. The position / orientation detection device 30 outputs information to the robot control device 50 via the second signal cable 12. In the figure, the signal cable between the imaging device 20 and the position / orientation detection device 30 and the signal cable between the imaging device 20 and the robot control device 50 are omitted. This figure is a diagram immediately before the robot body 10 starts the operation of grasping the object 200 from now on.

In the example of the figure, the object 200 is positioned in advance so that the center of the object 200 is positioned on the optical axis of the imaging device 20. As shown in FIG. 2, in the robot system 1 of the present embodiment, since the imaging device 20 is arranged independently of the robot body 10, the degree of freedom of the imaging angle with respect to the object 200 is increased. Thereby, the position and orientation detection device 30 can detect the position and orientation of the target object 200 using the captured images taken at various imaging angles, so that the position and orientation detection accuracy is improved.
In the figure, the illustration of the position / orientation detection device 30 and the robot control device 50 is omitted. Further, the scales of parts, structures, etc. in the figure are different from actual ones in order to make the figure clear.

  The robot body 10 includes a support base 10a fixed to the ground, an arm part 10b connected to the support base 10a so as to be capable of turning and bending, and a hand part connected to the arm part 10b so as to be rotatable and swingable. 10c and the holding part 10d attached to the hand part 10c.

The robot body 10 is, for example, a 6-axis vertical articulated robot, and has a 6-axis degree of freedom by a coordinated operation of the support base 10a, the arm unit 10b, and the hand unit 10c. The position and orientation can be changed freely. The robot body 10 moves one or a combination of the arm unit 10b, the hand unit 10c, and the gripping unit 10d under the control of the robot control device 50.
The degree of freedom of the robot body 10 is not limited to six axes, and may have any number of axes. For example, the robot body 10 may have seven axes of freedom. Moreover, you may install the support stand 10a in the place fixed with respect to the grounds, such as a wall and a ceiling.

  FIG. 3 is a comparison between a captured image obtained by capturing a sheet metal part and a CG (Computer Graphics) image of a CAD model representing the contour of the sheet metal part. The image F31 is an example of a captured image obtained by capturing a sheet metal part. As shown in the image F31, the sheet metal part is formed by stamping or bending a metal. The image F32 is a CG (Computer Graphics) image of the CAD model of the image F31. The apparatus using the conventional model-based matching method obtains the position and orientation by comparing the captured image shown in the image F31 with the CG image shown in the image F32. Here, if the object shown in the image F31 and the CAD model shown in the image F32 are not completely the same shape, an accurate position or posture cannot be obtained. However, in actuality, it does not completely match the CAD model due to variations in the processing accuracy of the object.

  In contrast, the position / orientation detection apparatus 30 according to the first embodiment extracts a feature region corresponding to the feature size and a high processing accuracy region whose processing accuracy is higher than a predetermined reference. The size of the feature is a number that represents the degree of the feature. The larger the feature, the larger the feature. The feature size is, for example, a luminance value difference between adjacent pixels. Then, as an example, the position and orientation detection device 30 compares only the image information of the overlapping region where the feature region and the high processing accuracy region overlap when comparing the CG image of the CAD model and the captured image. Improve the detection accuracy of the position or orientation of

FIG. 4 is a schematic block diagram illustrating a hardware configuration of the position / orientation detection apparatus 30 according to the first embodiment. In the figure, the position / orientation detection apparatus 30 includes an input unit 31, a storage unit 32, a control unit 33, and a display unit 34.
The input unit 31 receives an input from a user who uses the position and orientation detection device 30. Specifically, for example, the input unit 31 receives a region (hereinafter referred to as a low processing accuracy region) whose processing accuracy is lower than a predetermined reference in a three-dimensional object model (CAD model) of the target object. Here, the low machining accuracy region is, in other words, a region where the tolerance is higher than a predetermined threshold value. For example, when the object is a sheet metal part, the low processing accuracy region is an image region that has been bent or drawn. This is based on the fact that the normal dimension tolerance of bending and drawing is larger than the normal dimension tolerance of punching in metal pressing. The input unit 31 outputs low machining accuracy region information indicating the received low machining accuracy region to the control unit 33.

The storage unit 32 stores template image information indicating a plurality of template images (for example, CG images) generated based on the CAD model. Here, a plurality of images each representing an image of a model representing the contour of an object and having at least one of the position and orientation of the model in the image are set as template images.
Further, the storage unit 32 stores coordinate data constituting a three-dimensional model of the object and a low machining accuracy area flag indicating whether or not the coordinates are a low machining accuracy area. Here, as an example, the low machining accuracy region flag is 1 when the tolerance exceeds a predetermined reference and 0 when the tolerance is equal to or less than the predetermined reference. However, before the input unit 31 receives a low machining accuracy region from the user, all the low machining accuracy region flags are assumed to be zero.

Subsequently, the processing of the control unit 33 is performed in advance before actually estimating the position or orientation of the object, and the processing for actually estimating the position or orientation of the object (hereinafter, position and orientation estimation). This will be described separately.
First, pre-processing of the control unit 33 will be described. In the preprocessing, the control unit 33 causes the display unit 34 to display a three-dimensional model of the object. Then, the control unit 33 causes the display unit 34 to display that the user is instructed to input a low machining accuracy region. This prompts the user to enter a low machining accuracy region. In addition, the control unit 33 receives the low machining accuracy region information input from the input unit 31 and reflects the received low machining accuracy region information in the storage unit 32. Details of this processing will be described later.

  And the control part 33 acquires the image range used by the comparison with the image in which the target object was image | photographed, and this template image as a comparison image range. At that time, when comparing the CG image of the CAD model and the captured image obtained by capturing the target object, the control unit 33 compares the low processing accuracy region such as a region subjected to bending or drawing processing into the comparative image range. To eliminate. And the control part 33 improves the precision of a matching with a template image by comparing only an image area | region with high processing precision. For example, when the object is a sheet metal part, the control unit 33 performs the bending or drawing processing when the low processing accuracy region information received from the input unit 31 is an image region subjected to bending or drawing processing. The image region is excluded from the region to be compared when the CG image of the CAD model and the captured image obtained by capturing the object are compared.

  Next, the position / orientation estimation process of the control unit 33 will be described. The control unit 33 uses the captured image data input from the imaging device 20 to detect at least one of the position and orientation of the target object. Details of this processing will be described later. Then, the control unit 33 outputs position / orientation information including at least one of the detected positions and orientations to the robot control device 50.

  FIG. 5 is a schematic block diagram showing a logical configuration of the control unit 33 in the first embodiment. The control unit 33 includes a comparison image range generation unit 40, a captured image acquisition unit 338, a comparison image generation unit 339, and a position / orientation estimation unit 340. The comparison image range generation unit 40 includes a feature acquisition unit 331, a feature region extraction unit 332, a low processing accuracy region acquisition unit 333, and a comparison image range acquisition unit 334. In the figure, a pre-processing path that is executed in advance before a captured image is input is indicated by a solid arrow, and a position / orientation estimation processing path that is executed after a captured image is input is indicated by a dashed arrow. Yes.

First, processing performed by each unit of the control unit 33 in the preliminary processing will be described.
The feature acquisition unit 331 reads one template image information among a plurality of template image information stored in the storage unit 32 as target template image information to be processed. Then, the feature acquisition unit 331 uses each of a plurality of images that are images of models representing the contour of the object and at least one of the positions or orientations of the models in the images as template images. For the template image, the feature point of the model represented in the template image and the feature size of the feature point are acquired. Specifically, for example, the feature acquisition unit 331 extracts feature points using SURF (Speeded Up Robust Features). At that time, the feature acquisition unit 331 displays the feature size (one of the SURF parameters) indicating the scale (one of the SURF parameters) at which the detected feature point is detected, and the feature point The size of the feature. The feature acquisition unit 331 outputs data indicating the position of the extracted feature point and the feature size of the feature point to the feature region extraction unit 332. In addition, the feature acquisition unit 331 outputs the target template image information to the high-accuracy region image generation unit 336.

  When the feature acquisition unit 331 uses, for example, SURF, the threshold value of the extracted feature point is changed by increasing the value of the HessianThreshold that is the threshold value of the feature size when the feature point is extracted. Also good. In this case, for example, when it is desired to extract only a large feature portion, the feature acquisition unit 331 may set the HessianThreshold to a large value. For example, when the number of feature points to be extracted is determined, the feature acquisition unit 331 may adjust the HessianThreshold until the number of feature points reaches the target number.

  For each template image, the feature region extraction unit 332 sets a feature image region that is a feature of the template image at a position corresponding to the position of the feature point extracted by the feature acquisition unit 331 according to the size of the feature. Extract with a certain size. Specifically, for example, the feature region extraction unit 332 extracts, from the data input from the feature acquisition unit 331, a circular region having a radius corresponding to the feature size of the feature point from the feature point position as a center. The feature area of the feature point is used. For example, the feature region extraction unit 332 repeats this process for the feature points. Then, the feature region extraction unit 332 sets, for example, a union of all feature regions as a feature image region. The feature region extraction unit 332 outputs feature image region information indicating the extracted feature image region to a feature region image generation unit 335 described later of the comparison image range acquisition unit 334.

  Note that the feature region extraction unit 332 may use a pixel region (for example, a three-pixel radius region) that is determined in advance with the position of the feature point as the center.

  The low processing accuracy area acquisition unit 333 reads the template image information from the storage unit 32 and causes the display unit 34 to display the template image indicated by the read template image information. In addition, the low processing accuracy region acquisition unit 333 acquires, for each template image, an image region of a portion where the processing accuracy of the object is lower than a predetermined accuracy (hereinafter referred to as a low processing accuracy region). Specifically, for example, the low machining accuracy region acquisition unit 333 acquires the low machining accuracy region received by the input unit 31. For example, the low machining accuracy region acquisition unit 333 sets the low machining accuracy region flag associated with the coordinate data included in the low machining accuracy region to 1 in the storage unit 32.

  Information relating to machining accuracy is stored in advance in the storage unit 32, and the low machining accuracy region acquisition unit 333 reads information relating to the machining accuracy from the storage unit 32, and from the read information relating to machining accuracy, the low machining accuracy region acquisition unit 333 may specify the low machining accuracy region by itself.

The comparison image range acquisition unit 334, for each template image, the model feature point acquired by the feature acquisition unit 331, the size of the feature of the feature point, and the image region acquired by the low processing accuracy region acquisition unit 333 To obtain a comparison image range. Here, the comparison image range is a range of pixel positions to be compared when comparing the template image and the captured image.
Since there may be no low processing accuracy area, the comparison image range acquisition unit 334 may generate a comparison image range for each template image based on at least the feature image area extracted by the feature area extraction unit 332.

Here, the comparison image range acquisition unit 334 includes a feature region image generation unit 335, a high-precision region image generation unit 336, and an overlap range extraction unit 337.
The feature region image generation unit 335 generates a feature region explicit image indicating the feature image region indicated by the feature image region information input from the feature region extraction unit 332. Specifically, for example, the feature region image generation unit 335 generates a feature region explicit image in which the brightness value of the feature image region is 1 and the other brightness values are 0. Then, the feature region image generation unit 335 outputs the feature region explicit image information indicating the feature region explicit image to the overlapping range extraction unit 337.

The high accuracy region image generation unit 336 reads a set of coordinate data and a low processing accuracy region flag that constitute the three-dimensional model stored in the storage unit 32. Then, the high-accuracy region image generation unit 336 generates a high-accuracy region explicit image obtained by extracting a portion with high processing accuracy for each template image indicated by the target template image information input from the feature acquisition unit 331. Specifically, for example, for each template image, the high-accuracy region image generation unit 336 increases, in the template image, an area in which the low processing accuracy area flag is 0 among the coordinates constituting the three-dimensional model of the target object. Extract as a precision region. Then, for example, a high-accuracy region explicit image in which the luminance value of the high-precision region is 1 and the other luminance values are 0 is generated. Thereby, even if it is a template image which changed the viewpoint which sees the three-dimensional model of a target object, the high precision area | region image generation part 336 can extract a high precision area | region according to the template image which changed the viewpoint. . The high-accuracy region image generation unit 336 can generate a high-accuracy region mask according to the extracted high-accuracy region.
The high accuracy area image generation unit 336 outputs high accuracy area explicit image information indicating the generated high accuracy area explicit image to the overlapping range extraction unit 337.

  The overlapping range extraction unit 337 extracts a comparison image range based on the feature region explicit image information input from the feature region image generation unit 335 and the high accuracy region explicit image information input from the high accuracy region image generation unit 336. . Specifically, the overlapping range extraction unit 337 extracts the overlapping range between the feature image region and the high accuracy region as a comparison image range. For example, a set of pixels whose characteristic region explicit image has a luminance value of 1 and high-precision region explicit image has a luminance value of 1 is extracted as a comparison image range.

  Then, for example, the overlapping range extraction unit 337 generates a comparative image range explicit image having an image region indicating the comparative image range in the image from the extracted comparative image range. For example, the overlapping range extraction unit 337 generates a comparative image range explicit image in which the luminance value of the extracted comparative image range is 1 and the luminance value of other regions is 0. The overlapping range extraction unit 337 generates comparison image range explicit image information for each of the generated plurality of comparison image range explicit images. As a result, a plurality of comparative image range explicit image information can be obtained as a whole. The overlapping range extraction unit 337 outputs a plurality of comparison image range explicit image information to the comparison image generation unit 339. In addition, the overlapping range extraction unit 337 causes the storage unit 32 to store a plurality of comparison image range explicit image information.

  The comparison image generation unit 339 performs, for each template image, a process of generating the first image based on the comparison image range generated based on the template image from the template image. Specifically, for example, the comparison image generation unit 339 extracts an image region in the comparison image range indicated by the comparison image range explicit image generated for each template image from each of the plurality of template images in advance. Then, the comparison image generation unit 339 generates, for example, an image including only the image area and background of the extracted comparison image range as the first image for each template image. Thereby, the comparison image generation unit 339 generates the same number of first images as the template image. Then, the comparison image generation unit 339 causes the storage unit 32 to store first image information indicating the plurality of generated first images. The comparison image generation unit 339 also includes first image information, position information of a three-dimensional model of the object corresponding to the first image information, and a three-dimensional model of the object corresponding to the first image information. Is stored in the storage unit 32 in association with the posture information indicating the posture.

Next, a process performed by each unit of the control unit 33 in the position / orientation estimation process will be described.
The captured image acquisition unit 338 acquires a captured image that is an image obtained by capturing an object. Then, the captured image acquisition unit 338 supplies captured image information indicating the acquired captured image to the comparison image generation unit 339.

  The comparison image generation unit 339 performs processing for generating the second image from the captured image based on one comparison image range selected from the plurality of comparison image ranges for the comparison image range. Further, for example, when the captured image information is input from the captured image acquisition unit 338, the comparative image generation unit 339 reads all the comparative image range explicit image information from the storage unit 32. Then, the comparison image generation unit 339 extracts the image area of the comparison image range indicated by the one comparison image range explicit image information selected from all the comparison image range explicit image information. Then, the comparison image generation unit 339 generates, for example, an image including only the image area and background of this comparison image range as the second image. The comparison image generation unit 339 performs the process of generating the second image for the comparison image range explicit image information. Thereby, the comparison image generation unit 339 generates the same number of second images as the comparison image range explicit image information. Then, the comparative image generation unit 339 outputs the second image information indicating the generated second image to the position / orientation estimation unit 340 by the number of the second images.

The position / orientation estimation unit 340 compares the first image and a plurality of sets of second images generated using the same comparison image range as the comparison image range used in the first image. Estimate at least one of the position and orientation of
Specifically, the position / orientation estimation unit 340 performs template matching between the first image and the second image generated using the same comparison image range as the comparison image range used in the first image. I do. For example, the position / orientation estimation unit 340 calculates the corresponding pixel between the first image and the second image generated using the same comparison image range as the comparison image range used in the first image. The sum of absolute differences is calculated by adding the absolute values of the pixel value differences over all the pixels. The position / orientation estimation unit 340 includes, for example, all sets of the first image and the second image generated using the same comparison image range as the comparison image range used in the first image, A first image having the smallest difference absolute value sum is extracted. Then, the position / orientation estimation unit 340 reads position information and orientation information associated with the first image information indicating the extracted first image from the storage unit 32, for example. Then, the position / orientation estimation unit 340 outputs position / orientation information including the read position information and orientation information to the robot controller 50, for example.

  FIG. 6 is a diagram for explaining an outline of processing of the feature acquisition unit 331. In the figure, an image G61 which is an example of a template image including a three-dimensional model of an object, and a conceptual diagram G62 after features are extracted from the image G61 by processing by the feature acquisition unit 331 are shown. In the conceptual diagram G62, a plurality of circles having a radius corresponding to the size of the feature centered on the feature point are shown on the three-dimensional model of the object. In this manner, the feature acquisition unit 331 extracts feature points and the feature sizes of the feature points from the template image.

  FIG. 7 is a diagram for explaining the outline of the process of the feature region extraction unit 332. In the same figure, the conceptual diagram G62 shown in FIG. 6 and the conceptual diagram G63 after the feature region is extracted from the conceptual diagram G62 by the processing by the feature region extraction unit 332 are shown. In the conceptual diagram G63, a feature region that is a union of regions having the plurality of circles shown in the conceptual diagram G62 as outer edges is shown. Thus, the feature region extraction unit 332 extracts a feature region from the feature points and the feature sizes of the feature points.

  FIG. 8 is a diagram for explaining the outline of the process of the feature region image generation unit 335. In the figure, a conceptual diagram G63 shown in FIG. 7 and a feature region explicit image G64 generated from the conceptual diagram G63 by processing by the feature region image generation unit 335 are shown. In the feature region explicit image G64, the feature region shown in the conceptual diagram G63 is white (for example, when the luminance is represented by 1 bit, it corresponds to a luminance value of 1), and the other regions are black (for example, the luminance is When represented by 1 bit, it corresponds to a luminance value of 0). Thus, the feature region image generation unit 335 generates a feature region explicit image from the feature region.

  FIG. 9 is a diagram for explaining the outline of the processing of the high-accuracy region image generation unit 336. In the drawing, a conceptual diagram G65 showing a high-precision region, and a high-precision region explicit image G66 generated from the conceptual diagram G65 by the high-precision region image generating unit 336 are shown. In the high-precision region explicit image G66, the high-precision region shown in the conceptual diagram G65 is white (for example, when the luminance is represented by 1 bit, it corresponds to a luminance value of 1), and the other regions are black (for example, When the luminance is expressed by 1 bit, it corresponds to a luminance value of 0). As described above, the high accuracy area image generation unit 336 generates a high accuracy area explicit image from the high accuracy area.

  FIG. 10 is a diagram for explaining the outline of the processing of the overlapping range extraction unit 337. In the figure, the feature region explicit image G64 shown in FIG. 8, the high-precision region explicit image G66 shown in FIG. 9, and the overlapping range extraction unit 337 from the feature region explicit image G64 and the high-precision region explicit image G66. The comparison image range explicit image G67 generated by the above is shown. Here, in the feature region explicit image G64, the specific region is shown as a white region. In the high-precision area explicit image G66, the high-precision area is shown as a white area. In the comparison image range explicit image G67, a comparison image range which is a product set of the specific region and the high accuracy region is indicated by a white region. As described above, the high-precision area image generation unit 336 generates a comparative image range explicit image from the feature area explicit image and the high-precision area explicit image.

  FIG. 11 is a diagram for explaining the outline of the processing of the comparison image generation unit 339. In the figure, the template image G61 shown in FIG. 6 and the comparison image generation unit 339 from the template image G61 to the comparison image range (comparison image range explicit image) shown in the comparison image range explicit image G67 shown in FIG. A first image G68 generated by extracting the white area in G67 is shown. As described above, the comparison image generation unit 339 indicates, from each of a plurality of template images in which at least one of the position and orientation of the target object model is different from each other, the comparison image range explicit image associated with the template image in a one-to-one manner. A first image from which the comparison image range is extracted is generated. Similarly, the comparison image generation unit 339 generates a second image obtained by extracting a comparison image range indicated by one comparison image range explicit image from the captured image. Then, the comparison image generation unit 339 generates the second image by the number of comparison image range explicit images, that is, the number of template images, by performing this process for all the comparison image range explicit images. Thereby, the comparison image generation unit 339 generates the same number of first images and second images.

  FIG. 12 is a flowchart illustrating an example of a flow of processing executed in advance by the position / orientation detection apparatus 30 according to the first embodiment before the position / orientation detection. First, the position / orientation detection apparatus 30 acquires a low machining accuracy region (step S101). Next, the feature acquisition unit 331 reads the template image information from the storage unit 32 as target template image information to be processed (step S102). Next, the feature region extraction unit 332 calculates the position of the feature point and the feature size of the feature point in the model of the object included in the template image (step S103). Next, the feature region extraction unit 332 extracts a feature image region with a size corresponding to the feature size calculated by the feature region extraction unit 332 (step S104).

  Next, the feature region image generation unit 335 generates a feature region explicit image indicating the feature image region using the extracted feature image region (step S105). Next, the high-accuracy region image generation unit 336 generates a high-accuracy region explicit image indicating a high-processing accuracy region in the target template image with reference to the coordinate data stored in the storage unit 32 and the low-processing accuracy region flag. (Step S106).

  Next, the overlapping range extraction unit 337 generates a comparative image range explicit image based on the feature region explicit image and the high-precision region explicit image, and stores the generated comparative image range explicit image in the storage unit 32 (Step S <b> 3). S107). Next, the comparison image generation unit 339 generates a first image based on the comparison image range generated based on the template image from the template image (step S108). Next, the comparison image generation unit 339 determines whether or not the first image has been generated from all the template images (step S109). When the first image has not been generated from all the template images (NO in step S109), the comparison image generating unit 339 reads the next template image (step S110) and returns to the process of step S103. On the other hand, when the first image is generated from all the template images (YES in step S109), the comparative image generation unit 339 ends the process. Above, the process of this flowchart is complete | finished.

  FIG. 13 is a flowchart illustrating an example of a flow of position and orientation detection processing in which the position and orientation detection apparatus 30 according to the first embodiment actually detects the position or orientation of an object. First, the comparison image generation unit 339 reads the first comparison image range explicit image (step S201). Next, the comparative image generation unit 339 generates a second image from the captured image using the read first comparative image range explicit image (step S202). Next, the comparison image generation unit 339 determines whether or not the second image has been generated using all the comparison image range explicit images (step S203). When the second image is not generated using all the comparison image range explicit images (NO in step S203), the comparison image generation unit 339 reads the next comparison image range explicit image from the storage unit 32 (step S204). The process proceeds to step S205. In step S205, the comparative image generation unit 339 generates a second image from the captured image using the next comparative image range explicit image, and the process returns to step S203.

  In step S203, when the second image is generated using all the comparative image range explicit images (YES in step S203), the position / orientation estimation unit 340 selects the first image based on the first comparative image range explicit image. Read (step S206). Next, the position / orientation estimation unit 340 includes pixel values of corresponding pixels between the first image based on the first comparative image range explicit image and the second image based on the first comparative image range explicit image. The sum of absolute differences of pixel values is calculated by adding the absolute values of the differences over all the pixels (step S207). Next, the position / orientation estimation unit 340 determines whether or not the sum of absolute differences has been calculated for all combinations (step S208).

  If the sum of absolute differences has not been calculated for all combinations (NO in step S208), the position / orientation estimation unit 340 reads the first image based on the next comparative image range explicit image (step S209), and in step S210 Proceed to processing. In step S210, the position / orientation estimation unit 340 determines the pixel value of the corresponding pixel between the first image based on the next comparison image range explicit image and the second image based on the same next comparison image range explicit image. By adding the absolute values of the differences over all pixels, the sum of absolute differences of the pixel values is calculated, and the process returns to step S208.

  In step S208, when the sum of absolute differences is calculated for all combinations (YES in step S208), the position / orientation estimation unit 340 uses the first difference absolute value sum of pixel values for the position and orientation of the object. Position information and posture information associated with first image information indicating an image are extracted as position information and posture information of the object, respectively (step S211). Above, the process of this flowchart is complete | finished.

Conventionally, when the object is a sheet metal part, there is a variation in processing accuracy of sheet metal processing in the actual object, and there is a problem that matching accuracy differs for each object with respect to the CAD model.
In contrast, in the position / orientation detection apparatus 30 according to the first embodiment, the low processing accuracy region acquisition unit 333 acquires, for each template image, an image region of a portion where the processing accuracy of the target is lower than a predetermined accuracy. To do.

  Further, in the position / orientation detection apparatus 30, the feature acquisition unit 331 is an image of a model representing the contour of the object, and each of a plurality of images in which at least one of the position or orientation of the model in the image is different from each other. For each template image, the feature point of the model represented in the template image and the feature size of the feature point are acquired. In the position / orientation detection apparatus 30, the comparative image range acquisition unit 334 acquires, for each template image, the model feature point acquired by the feature acquisition unit, the feature size of the feature point, and the low processing accuracy region acquisition. The comparison image range is acquired based on the image area acquired by the unit.

  According to this configuration, the position / orientation detection device 30 increases the ratio of the area around the feature point included in the comparison image range as the feature of the feature point is larger, and processes the feature portion of the model and the object itself. Information with low accuracy can be removed. As a result, regarding the comparison image range, either the position or the posture is estimated by comparing between the template image and the photographed image obtained by photographing the object, so that the object has a variation in processing accuracy. Even so, since the matching is performed using only the information of the portion with high processing accuracy among the characteristic portions of the model and the object itself, the accuracy of matching can be improved. As a result, the position / orientation detection apparatus 30 can improve the accuracy of matching between the template image and the captured image, and can therefore improve the estimation accuracy of the position or orientation.

  Note that the position and orientation detection device 30 may determine that the position and orientation cannot be obtained when the minimum value of the sum of absolute differences of pixel values is larger than a predetermined threshold value. Then, the position / orientation detection apparatus 30 moves the position of the mask area in the comparison image range explicit image, for example, in a predetermined direction. Then, the position / orientation detection apparatus 30 may execute the process of the flowchart of FIG. 13 again using the moved mask region.

Further, the position / orientation detection device 30 determines that the position and orientation cannot be obtained when the minimum value of the sum of absolute differences of pixel values is larger than a predetermined threshold value, and then the robot control device 50 You may make the robot main body 10 change the position and attitude | position of a target object. In that case, the position / orientation detection apparatus 30 causes the imaging apparatus 20 to image the target object. Then, using the captured image obtained by this imaging, the position / orientation detection apparatus 30 may execute the process of the flowchart of FIG. 13 again.
Further, the low processing accuracy area may include an area where the variation of the individual object is large. Accordingly, the position / orientation detection apparatus 30 can exclude an area where the variation of the individual object is large from the comparison image range.

  Further, the position / orientation detection device 30 may generate a comparison image range for each template image based at least on the extracted feature image region. According to this configuration, since the ratio of the area around the feature point included in the comparison image range increases as the feature point feature increases, the position / orientation detection apparatus 30 captures the template image and the target object. Compared with the image, a region with a large feature is compared with emphasis, and a region with a small feature is compared without emphasis. Thereby, since the position and orientation detection device 30 can improve the accuracy of matching between the template image and the captured image, it is possible to improve the detection accuracy of the position or orientation of the object.

  In the first embodiment, the control unit 33 performs pre-processing in advance. However, the present invention is not limited to this, and the pre-processing may be performed every time before the position and orientation estimation processing.

<Second Embodiment>
Next, the second embodiment will be described. In the first embodiment, the position / orientation detection apparatus 30 performs both the pre-processing and the position / orientation estimation process. On the other hand, in the second embodiment, the comparative image range generation device 41 performs pre-processing, and the position / orientation detection device 30b performs position / orientation estimation processing. Thereby, separate devices perform the above two processes.

  FIG. 14 is a schematic block diagram showing the configuration of the robot system 1b according to the second embodiment. Elements common to those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof is omitted. The configuration of the robot system 1b in FIG. 14 is a configuration in which a comparative image range generation device 41 is added to the configuration of the robot system 1 in FIG. 1 and the position / orientation detection device 30 is changed to a position / orientation detection device 30b. ing.

  FIG. 15 is a schematic block diagram showing the configuration of the comparative image range generation device 41 in the second embodiment. The comparative image range generation device 41 includes an input unit 31, a storage unit 32, a display unit 34, a control unit 43, and a communication unit 45. In addition, the same code | symbol is attached | subjected to the element which is common in FIG. 4, and the specific description is abbreviate | omitted.

  For example, the control unit 43 reads the template image information from the storage unit 32 and generates a comparative image range explicit image based on the template image indicated by the read template image information. For example, in response to the comparison image range explicit image request received from the position / orientation detection device 30 via the communication unit 45, the control unit 43 transmits the comparison image range explicit image information indicating the generated comparison image range explicit image to the communication unit 45. To the position and orientation detection device 30b. Here, the comparison image range explicit image request is information for requesting a comparison image range explicit image.

For example, in response to a first image request received from the position / orientation detection device 30 via the communication unit 45, the control unit 43 reads the first image information from the storage unit 32 and communicates the read first image information. The unit 45 transmits to the position / orientation detection apparatus 30b. Here, the first image request is information for requesting the first image information.
For example, when the control unit 43 receives the first image information from the position / orientation detection device 30 via the communication unit 45, the control unit 43 displays the position information and the posture associated with the first image information from the storage unit 32. Read the attitude information shown. For example, the control unit 43 causes the communication unit 45 to transmit position and orientation information including at least one of position information and orientation information to the position and orientation detection device 30b.

The communication unit 45 receives the comparison image range explicit image request or the first image request transmitted from the position / orientation detection apparatus 30, and outputs the received comparison image range explicit image request or first image request to the control unit 43. The communication unit 45 transmits the comparison image range explicit image information output by the control unit 43 in response to the comparison image range explicit image request to the position and orientation detection device 30b. Further, the communication unit 45 transmits the first image information output from the control unit 43 in response to the first image request to the position and orientation detection device 30b.
Further, the communication unit 45 receives the first image information transmitted from the position / orientation detection device 30 and outputs the received first image information to the control unit 43. The communication unit 45 transmits the position / orientation information input from the control unit 43 to the position / orientation detection device 30b.

  FIG. 16 is a schematic block diagram illustrating a logical configuration of the control unit 43 of the comparative image range generation device 41 in the second embodiment. In the control unit 43, elements common to those in FIG. A feature acquisition unit 331, a feature region extraction unit 332, a low processing accuracy region acquisition unit 333, a comparative image range acquisition unit 334, and a communication control unit 431 are provided.

  For example, each time the communication control unit 431 receives a comparison image range explicit image request, the communication control unit 431 sequentially reads the comparison image range explicit image information indicating one comparison image range explicit image, and the read comparison image range explicit image information is read out. It transmits to the communication part 45. For example, when N comparison image range explicit images (N is a positive integer) are stored in the storage unit 32, the communication control unit 431 receives the comparison image range explicit image request N times in total, In total, N pieces of comparison image range explicit image information are transmitted to the communication unit 45.

  In addition, for example, each time the first image request is received, the communication control unit 431 sequentially reads first image information indicating one first image and transmits the read first image information to the communication unit 45. . For example, when N first images are stored in the storage unit 32, the communication control unit 431 receives the first image request N times in total, and receives the N first image information in total. It transmits to the communication part 45.

FIG. 17 is a schematic block diagram illustrating a configuration of the position and orientation detection device 30b according to the second embodiment. The position / orientation detection apparatus 30b includes a storage unit 32b, a control unit 33b, and a communication unit 35.
The storage unit 32b stores a program that is read and executed by the control unit 33b.
The communication unit 35 communicates with the comparative image range generation device 41 according to the control of the control unit 33b.

  The control unit 33 b controls the communication unit 35 to generate a second image every time a comparative image range explicit image is acquired from the comparative image range generation device 41. In addition, the control unit 33 b controls the communication unit 35 to acquire a plurality of first images from the comparative image range generation device 41. Then, the control unit 33b compares the first image and a plurality of sets of second images generated using the same comparison image range as the comparison image range used in the first image. Estimate at least one of the position and orientation of The control unit 33b outputs position and orientation information including at least one of position information indicating the estimated position of the target object and posture information indicating the attitude to the robot control device 50.

FIG. 18 is a schematic block diagram illustrating a logical configuration of the control unit 33b of the position / orientation detection apparatus 30b according to the second embodiment. The control unit 33b includes a comparative image generation unit 339b, a position / orientation estimation unit 340b, and an acquisition unit 341.
The acquisition unit 341 uses, as template images, a plurality of images that are images of models representing the contour of an object and at least one of the positions or orientations of the models in the images is different from each other. An image range that is obtained based on the feature point of the model represented in the template image and the size of the feature of the feature point, and is used in the comparison between the image obtained by photographing the object and the template image. A certain comparison image range is acquired.

  Specifically, for example, a comparison image range explicit image request is transmitted to the comparison image range generation device 41 via the communication unit 35. Then, the comparison image generation unit 339b acquires the comparison image range explicit image information received by the communication unit 35 in response to the comparison image range explicit image request from the communication unit 35. The acquisition unit 341 outputs the acquired comparison image range information to the comparison image generation unit 339b.

  The comparison image generation unit 339b has the same function as the comparison image generation unit 339 in the first embodiment, but differs in the following points. Each time the comparison image generation unit 339b acquires the comparison image range explicit image information from the acquisition unit 341, the comparison image generation unit 339b generates a second image from the captured image based on the comparison image range explicit image. The comparison image generation unit 339b repeats this series of processes for all the comparison image range explicit images generated for a plurality of template images of a certain target object.

  The position / orientation estimation unit 340b has the same function as the position / orientation estimation unit 340 according to the first embodiment, but differs in the following points. The position / orientation estimation unit 340 b transmits a first image request for requesting a first image to the comparative image range generation device 41 via the communication unit 35. Then, the position / orientation estimation unit 340b acquires the first image information received by the communication unit 35 from the communication unit 35 in response to the first image request. Then, the position / orientation estimation unit 340b, for example, calculates a pixel value between the first image and the second image generated using the same comparison image range as the comparison image range used in the first image. The sum of absolute differences is calculated. The position / orientation estimation unit 340b repeats this series of processes for all the first images generated for a plurality of template images of a certain target object.

  Then, the position / orientation estimation unit 340b extracts the first image having the smallest sum of absolute differences of pixel values. Then, the position / orientation estimation unit 340 b transmits first image information indicating the first image extracted via the communication unit 35 to the comparative image range generation device 41. Accordingly, the comparative image range generation device 41 returns position and orientation information including at least one of the position information and the posture information associated with the first image information to the position and orientation detection device 30b. The position / orientation estimation unit 340b acquires the position / orientation information received by the communication unit 35, and transmits the acquired position / orientation information to the robot controller 50.

  As described above, in the robot system 1b according to the second embodiment, the comparison image range generation device 41 performs the comparison image range explicit image generation processing, and the position / orientation detection device 30b performs the position and orientation detection processing of the target object. Thereby, in the robot system 1b according to the second embodiment, in addition to the effects of the first embodiment, the position / orientation detection device 30b has components deleted from the position / orientation detection device 30 according to the first embodiment. Therefore, it can be made smaller than the position and orientation detection device 30 in the first embodiment.

In each embodiment, the robot body 10 is a robot having one arm as an example. However, the present invention is not limited to this, and the robot body 10 may have a plurality of arms.
FIG. 19 is an example of a schematic external view of a robot system when the robot body 60 has two arms. The robot 2 in the figure includes a robot body 60, an imaging device 61, a gripping unit 62, a position / orientation detection device 30, a robot control device 50, a first cable 63, and a second cable 64. Has been. Since the position / orientation detection device 30 and the robot control device 50 have the same configurations as those of the above-described embodiments, detailed descriptions of the position / orientation detection device 30 and the robot control device 50 are omitted.

  Specifically, the robot main body 60 includes a main body 60a that is movably installed on the ground, a neck portion 60b that is pivotably coupled to the main body 60a, a head portion 60c that is fixed to the neck portion 60b, The first arm part 60d connected to the head part 60c so as to be able to bend and extend, the second arm part 60e connected to the head part 60c so as to be able to turn and bend and extend, and the robot body with respect to the installation surface of the robot body 60 60 includes a transfer section 60f attached to the main body 60a so as to be movable.

  A grip part 62 is attached to the hand that is the open end of the first arm part 60d. An imaging device 61 is attached to the hand that is the open end of the second arm portion 60e. The conveyance unit 60f supports the robot body 60 so that the robot body 60 can move in a certain direction or in a freely movable direction with respect to the installation surface of the robot body 60. The conveyance unit 60f is realized by four sets of wheels, four sets of casters, a pair of endless tracks, and the like.

  The robot body 60 is, for example, a vertical articulated robot (double arm robot) having two arms. The robot main body 60 takes in the robot control command supplied from the robot control device 50, and changes the position and orientation of the imaging device 61 and the gripper 62 in the three-dimensional space as desired by drive control based on the robot control command. Further, the claw portion of the grip portion 62 is opened and closed.

The imaging device 61 captures a subject, acquires a captured image that is a still image or a moving image, and supplies the captured image to the robot control device 50. The imaging device 61 is realized by, for example, a digital camera device or a digital video camera device.
The holding part 62 includes a claw part that can hold an object. In FIG. 19, the gripping portion 62 is schematically shown to show its function.

The first cable 63 supplies a robot control command output from the robot control device 50 to the robot main body 60, and supplies a robot control response output from the robot main body 60 to the robot control device 50. The robot control command is a control command for driving and controlling each movable part of the robot body 60. The robot control response includes, for example, a response to a robot control command that the robot body 60 notifies the robot control device 50 of. The first cable 63 is, for example, a cable such as a serial communication line or a computer network.
The second cable 64 supplies the position and orientation information output from the position and orientation detection device 30 to the robot control device 50. The second cable 64 is, for example, a cable such as a serial communication line or a computer network.

The robot control device 50 controls the operations of the neck 60b, the head 60c, and the second arm 60e of the robot body 60, and changes the position and posture of the imaging device 61. Specifically, the robot control device 50 causes the imaging device 61 to be installed at a position where the imaging direction is substantially vertically downward and substantially vertically upward with respect to an assembly component (not shown).
Further, the robot control device 50 takes in a photographed image of the assembly part supplied from the imaging device 61, and the neck 60b, the head 60c, and the first arm of the robot main body 60 based on the photographed image. The operation with the unit 60d is controlled, and the object 200 is gripped as in the first embodiment.

As described above, when the robot system 1 according to the first embodiment is configured to include the robot body 60 according to the modified example instead of the robot body 10, the position and orientation detection device 30 is provided as in the first embodiment. The same effect as the first embodiment is obtained.
When the robot system 1b according to the second embodiment is configured to include the robot body 60 according to the modified example instead of the robot body 10, the position / orientation detection device 30b and the comparison image range generation are performed as in the second embodiment. The apparatus 41 is provided, and there exists an effect similar to 2nd Embodiment.
The robot control device 50 may be housed in a cabinet of the main body 60a of the robot main body 60, for example.

In the second embodiment, the robot control device 50 may include at least one of the position and orientation detection device 30b and the comparative image range generation device 41.
In the first embodiment, the robot control device 50 may include the position / orientation detection device 30.
In addition, a system including a plurality of devices may process each process of the position / orientation detection device (30 or 30b) or the comparative image range generation device 41 of each embodiment in a distributed manner by the plurality of devices.
In addition, a program for executing each process of the position / orientation detection device (30 or 30b) or the comparative image range generation device 41 of each embodiment is recorded on a computer-readable recording medium and recorded on the recording medium. The above-described various processes related to the position / orientation detection device (30 or 30b) or the comparative image range generation device 41 may be performed by causing the computer system to read and execute the program.

  Here, the “computer system” may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if the WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

  Further, the “computer-readable recording medium” refers to a volatile memory (for example, DRAM (Dynamic) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Random Access Memory)) that holds a program for a certain period of time is also included. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  As mentioned above, although embodiment of this invention was explained in full detail with reference to drawings, the concrete structure is not restricted to this embodiment, The design etc. of the range which does not deviate from the summary of this invention are included.

DESCRIPTION OF SYMBOLS 1, 1b Robot system 2 Robot 10, 60 Robot main body 11 1st signal cable 12 2nd signal cable 20, 61 Imaging device 30, 30b Position and orientation detection device 31 Input part 32, 32b Storage part 33, 33b Control part 34 Display unit 35 Communication unit 40 Comparison image range generation unit 41 Comparison image range generation device 43 Control unit 45 Communication unit 50 Robot control device 62 Grasping unit 63 First cable 64 Second cable 331 Feature acquisition unit
332 Feature region extraction unit 333 Low processing accuracy region acquisition unit 334 Comparative image range acquisition unit 335 Feature region image generation unit 336 High accuracy region image generation unit 337 Overlapping range extraction unit 338 Captured image acquisition unit
339, 339b Comparison image generation unit 340, 340b Position and orientation estimation unit 341 Acquisition unit 431 Communication control unit

Claims (12)

For each template image, the feature acquisition unit uses each of a plurality of images, each of which is an image of a model representing the contour of an object, and at least one of the positions or orientations of the model in the image is different from each other. A feature acquisition procedure for acquiring the feature points of the model represented in the template image and the feature size of the feature points;
The comparison image range acquisition unit, for each of the template images, based on the feature point of the model acquired by the feature acquisition unit and the size of the feature of the feature point, and the template A comparative image range acquisition procedure for acquiring an image range used for comparison with an image as a comparative image range;
A comparative image range generation method characterized by comprising: The low processing accuracy region acquisition unit further includes a low processing accuracy region acquisition procedure for acquiring a part of the image region where the processing accuracy of the object is lower than a predetermined accuracy for each template image,
In the comparative image range acquisition procedure, the comparative image range acquisition unit, for each of the template images, the model feature point acquired by the feature acquisition unit, the feature size of the feature point, and the low processing accuracy The comparative image range generation method according to claim 1, wherein the comparative image range is acquired based on the image region acquired by the region acquisition unit. The acquisition unit is an image of a model representing the contour of an object, and a plurality of images in which at least one of the position or orientation of the model in the image is different from each other as template images, and for each template image, This is an image range used for comparison between the template image and the image obtained by photographing the object, which is acquired based on the feature point of the model represented in the template image and the feature size of the feature point. An acquisition procedure for acquiring a comparison image range;
A captured image acquisition unit for acquiring an image in which the object is captured;
For each of the plurality of template images acquired by the acquisition unit, the position / orientation estimation unit, with respect to the comparison image range corresponding to the template image acquired by the acquisition unit, the image acquired by the captured image acquisition unit A position and orientation estimation procedure for estimating at least one of the position and orientation of the object by comparing;
A position and orientation detection method characterized by comprising: For each template image, the feature acquisition unit uses each of a plurality of images, each of which is an image of a model representing the contour of an object, and at least one of the positions or orientations of the model in the image is different from each other. A feature acquisition procedure for acquiring the feature points of the model represented in the template image and the feature size of the feature points;
The comparison image range acquisition unit, for each of the template images, based on the feature point of the model acquired by the feature acquisition unit and the size of the feature of the feature point, and the template A comparative image range acquisition procedure for acquiring an image range used for comparison with an image as a comparative image range;
A captured image acquisition unit for acquiring an image in which the object is captured;
The position / orientation estimation unit compares, for each of the plurality of template images, the image acquired by the captured image acquisition unit with respect to the comparison image range corresponding to the template image acquired by the comparison image range acquisition unit. A position and orientation estimation procedure for estimating at least one of the position and orientation of the object;
A position and orientation detection method characterized by comprising: A model image representing the contour of an object, and each of the plurality of images in which at least one of the position or orientation of the model in the image is different from each other is used as a template image. A feature acquisition unit that acquires the feature points of the model represented and the feature size of the feature points;
For each of the template images, based on the feature points of the model acquired by the feature acquisition unit and the size of the features of the feature points, the template images are used for comparison with the template image. A comparative image range acquisition unit that acquires the image range as a comparative image range;
A comparison image range generation apparatus comprising: A plurality of images representing the outline of the model of the object and at least one of the positions or postures of the model in the image is used as a template image, and each template image is defined as the template image. A comparison image range that is an image range that is acquired based on the feature point of the model represented and the size of the feature point of the model and is used for comparison between the image of the object captured and the template image. An acquisition unit to acquire;
A captured image acquisition unit that acquires an image of the object captured;
For each of the plurality of template images acquired by the acquisition unit, the comparison image range corresponding to the template image acquired by the acquisition unit is compared with the image acquired by the captured image acquisition unit, A position and orientation estimation unit that estimates at least one of the position and orientation of the object;
A position and orientation detection device comprising: A model image representing the contour of an object, and each of the plurality of images in which at least one of the position or orientation of the model in the image is different from each other is used as a template image. A feature acquisition unit that acquires the feature points of the model represented and the feature size of the feature points;
For each of the template images, based on the feature points of the model acquired by the feature acquisition unit and the size of the features of the feature points, the template images are used for comparison with the template image. A comparative image range acquisition unit that acquires the image range as a comparative image range;
A robot characterized by comprising: A plurality of images representing the outline of the model of the object and at least one of the positions or postures of the model in the image is used as a template image, and each template image is defined as the template image. A comparison image range that is an image range that is acquired based on the feature point of the model represented and the size of the feature point of the model and is used for comparison between the image of the object captured and the template image. An acquisition unit to acquire;
A captured image acquisition unit that acquires an image of the object captured;
For each of the plurality of template images acquired by the acquisition unit, the comparison image range corresponding to the template image acquired by the acquisition unit is compared with the image acquired by the captured image acquisition unit, A position and orientation estimation unit that estimates at least one of the position and orientation of the object;
A robot characterized by comprising: A model image representing the contour of an object, and each of the plurality of images in which at least one of the position or orientation of the model in the image is different from each other is used as a template image. A feature acquisition unit that acquires the feature points of the model represented and the feature size of the feature points;
For each of the template images, based on the feature points of the model acquired by the feature acquisition unit and the size of the features of the feature points, the template images are used for comparison with the template image. A comparative image range acquisition unit that acquires the image range as a comparative image range;
A robot system comprising: A plurality of images representing the outline of the model of the object and at least one of the positions or postures of the model in the image is used as a template image, and each template image is defined as the template image. A comparison image range that is an image range that is acquired based on the feature point of the model represented and the size of the feature point of the model and is used for comparison between the image of the object captured and the template image. An acquisition unit to acquire;
A captured image acquisition unit that acquires an image of the object captured;
For each of the plurality of template images acquired by the acquisition unit, the comparison image range corresponding to the template image acquired by the acquisition unit is compared with the image acquired by the captured image acquisition unit, A position and orientation estimation unit that estimates at least one of the position and orientation of the object;
A robot system comprising: On the computer,
A model image representing the contour of an object, and each of the plurality of images in which at least one of the position or orientation of the model in the image is different from each other is used as a template image. A feature acquisition step of acquiring feature points of the model represented and the feature size of the feature points;
For each of the template images, based on the feature points of the model acquired by the feature acquisition step and the feature size of the feature points, the template images are used for comparison with the template image. A comparative image range acquisition step for acquiring the image range as a comparative image range;
The comparison image range generation program for executing On the computer,
A model image representing the contour of an object, and each of the plurality of images in which at least one of the position or orientation of the model in the image is different from each other is used as a template image. A comparison image range that is an image range that is acquired based on the feature point of the model represented and the size of the feature point of the model and is used for comparison between the image of the object captured and the template image. An acquisition step to acquire;
A captured image acquisition step of acquiring an image in which the object is captured;
By comparing each of the plurality of template images acquired in the acquisition step with the image acquired in the captured image acquisition step with respect to the comparison image range corresponding to the template image acquired in the acquisition step. A position and orientation estimation step for estimating at least one of the position and orientation of the object;
Position and orientation detection program for executing