JP2021071420A - Information processing apparatus, information processing method, program, system, manufacturing method for product, and measuring apparatus and measuring method - Google Patents

Abstract

To provide an information processing apparatus for efficiently specifying an attitude of an object.SOLUTION: An information processing apparatus includes: a first calculation unit for calculating a first posture of a target object based on first measurement data obtained by measuring the target object by a sensor; and a determination unit for determining whether or not the measurement data concerning a specific part for specifying the posture of the target object is sufficiently obtained for specifying the posture based on the first measurement data; and a second calculation unit which calculates a second posture of the target object on the basis of second measurement data obtained by measuring a specific part of the target object by changing a relative position of the target object and the sensor in which the first posture is calculated when it is determined that measurement data concerning the specific part is not sufficiently obtained, and which does not calculate the second posture based on the second measurement data when it is determined that the measurement data concerning the specific part is sufficiently obtained for specifying a posture by the determination unit.SELECTED DRAWING: Figure 7

Description

The present invention relates to an information processing device, an information processing method, a program, a system, a method for manufacturing an article, a measuring device, and a measuring method.

In recent years, production lines such as factories have developed a technology that identifies the three-dimensional position and posture (positional posture) of industrial parts stacked separately in a storage box by a vision system, grips them with a hand attached to a robot arm, and supplies them to the next process. It is used in. Many industrial parts have large symmetrical structures with parts (specific parts) such as protrusions, notches, and grooves that can be used as clues for specifying the posture and phase. When the state of stacking such parts is measured by an image sensor such as a camera, there are cases where the posture is uniquely determined and cases where the posture is not determined depending on the observation condition of a specific part. If a specific part can be sufficiently observed in the measurement data, the posture can be uniquely specified, but if the specific part is hidden from the observation viewpoint and cannot be observed, the posture cannot be uniquely determined.

In a vision system and a robot-based supply device for loosely stacked parts, there is a technique for identifying a position or orientation by performing two-step measurement with a sensor. In this technology, after recognizing the local part of the part from the distance image that measures the stacking state, the part is once picked out and rolled on the temporary stand, and then the two-dimensional image of the part of the temporary stand is measured. Identify the final position and orientation. (Patent Document 1)

Japanese Patent No. 5837065

In the method described in Patent Document 1, in the first stage measurement, only the local shape for gripping by the hand is detected, and it is not known which part of the object (work) the detected shape corresponds to. Therefore, it is not possible to uniquely identify the posture of the object when the first stage measurement is completed. Therefore, it is necessary to always perform the second stage measurement regardless of the observation status of the specific part described above.

Therefore, an object of the present invention is, for example, to provide an information processing device that efficiently identifies the posture of an object.

In order to achieve that object, the present invention is an information processing device that calculates the posture of an object, and the first object is based on the first measurement data obtained by measuring the object with a sensor. Based on the first calculation unit that calculates the posture and the first measurement data, it is determined whether or not sufficient measurement data for a specific part of the object for specifying the posture has been obtained to specify the posture. When the judgment unit determines that the measurement data for the specific part is not sufficiently obtained to specify the posture, the relative position between the object and the sensor for which the first posture is calculated is changed. The second posture of the object is calculated based on the second measurement data of the object obtained by measuring the specific part of the object, and it is determined that the measurement data for the specific part is sufficiently obtained to specify the posture. It is characterized by including a second calculation unit that does not calculate the second posture based on the second measurement data when the unit determines.

According to the present invention, for example, it is possible to provide an information processing device that efficiently identifies the posture of an object.

It is a figure which shows the structure of the information processing apparatus which concerns on Example 1. FIG. It is a figure which shows the shape of the object to handle in Example 1. FIG. It is a figure which shows the parts supply system using Example 1. FIG. It is a figure which shows the case where the posture is indistinguishable in the photographed image. It is a figure which shows the processing flow in Example 1. FIG. This is an example of a GUI that specifies a specific part in the first embodiment. It is a figure which shows the arrangement method of an object and a 2nd sensor. It is an example of an object to which Example 1 can be applied. It is an example of an object to which Example 1 can be applied. It is a figure which shows the shape of the object to handle in Example 2. FIG. It is a figure explaining the setting method of the object arrangement with respect to the 2nd sensor. It is a figure which showed the example of the control system which includes the gripping apparatus equipped with the information processing apparatus.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings with reference to examples. In each figure, the same member or element is given the same reference number, and duplicate explanations are omitted or simplified.

[Example 1]
In this embodiment, the observation is performed by so-called self-shielding, in which a specific part (characteristic part), which is a predetermined part of the object (work), is behind oneself or cannot be seen in the direction opposite to the viewpoint. An example of a case where the posture of the object cannot be uniquely specified will be described below.

FIG. 1 shows the configuration of the information processing device 1 according to the present embodiment. The information processing device 1 includes a first calculation unit 10, a determination unit 20, and a second calculation unit 30. Hereinafter, each part constituting the information processing apparatus 1 will be described.

The first calculation unit 10 acquires measurement data of a large number of objects stacked separately in the storage box using a sensor, detects one individual from the measurement data, and calculates its posture (positional posture). In this embodiment, the measurement data acquired by the first calculation unit 10 is the first measurement data, and the posture calculated by the first measurement data is the first posture (first position posture). Further, the sensor used in the first calculation unit 10 is referred to as the first sensor. Then, in the first posture, the posture of the object may not be uniquely specified depending on the observation status of the specific part. In this embodiment, the measurement when acquiring the first measurement data is defined as the first measurement.

Based on the first measurement data calculated by the first calculation unit 10, the determination unit 20 has sufficiently obtained measurement data regarding a specific part which is a part for uniquely specifying the posture of the object. Whether or not (whether or not it can be uniquely specified) is determined. That is, the second calculation unit 30, which will be described later, determines (verifies) whether or not a process for specifying the posture of the object is newly required.

The second calculation unit 30 determines that processing for specifying the posture of the object is necessary because the measurement data for the specific part is not sufficiently obtained by the determination unit 20 to specify the posture. The object is picked (grasped) by the hand based on the first posture. The hand is attached to the robot arm. At this time, the second calculation unit 30 generates a control signal for gripping the object. After that, the relative position is changed so that the specific part of the object can be observed, that is, the arrangement relationship between the sensor and the object is adjusted so that the posture of the specific part can be easily specified. At this time, the determination unit 20 generates a control signal for changing the relative position between the sensor and the object. Then, a new measurement data of the object obtained by measuring a specific part of the object with a sensor is acquired, the posture of the object is specified, and the final posture (positional posture) is calculated. In this embodiment, the measurement when acquiring the second measurement data is referred to as the second measurement.

When the second calculation unit 30 determines that the determination unit 20 has sufficiently obtained the measurement data for the specific part to specify the posture, the second calculation unit 30 newly acquires the measurement data of the object as described above. Therefore, the posture of the object is not specified and the final posture is not calculated. At this time, the determination unit 20 generates a control signal for gripping the object, and generates a control signal for moving the object to supply to the next process.

In this embodiment, the measurement data acquired by the second calculation unit 30 is the second measurement data, and the calculated posture is the second posture (second position posture). Further, the sensor used in the second calculation unit is referred to as a second sensor. Further, the second calculation unit 30 may acquire the measurement data of the object obtained by measuring the specific portion of the object without performing the operation of gripping the object. In that case, the arrangement relationship of the second sensor with respect to the object is adjusted, that is, only the relative position between the second sensor and the object is changed, and the specific part of the object is measured. Then, the posture of the object may be specified from the measured data, and the final posture may be calculated.

Next, the shape of the object handled in this embodiment is shown in FIG. In FIG. 2, the solid line portion is located on the front side when observed from the front side of the paper surface, and the observable contour and the dotted line portion are located on the back side, showing contours that are not originally observed. Further, it is assumed that a unique coordinate system is set in advance for the object of this embodiment. This coordinate system is called the object coordinate system. This object has, for example, a rectangular groove as a specific portion on one surface of a rectangular parallelepiped. The groove on the rectangle indicates a shaded area in FIG. Therefore, if this groove can be sufficiently observed in the first measurement data acquired in the loosely stacked state, the posture can be uniquely specified and the posture can be acquired.

FIG. 3 shows an example of the parts supply system related to this embodiment. A hand attached to the robot arm by acquiring the first measurement data with the first sensor installed above the objects stacked in the containment box, calculating the posture of one individual from the first measurement data, and attaching it to the robot arm. Picking (grasping) is performed with the robot hand (grasping part) and supplied to the next process. The robot hand also functions as a gripping means. However, when the specific part of the object cannot be observed with the first measurement data acquired by the first sensor (the measurement data for the specific part is not sufficiently obtained) and the posture of the object cannot be uniquely specified. Occurs. In this embodiment, the first measurement data is an image taken by the first sensor.

This will be described in more detail with reference to FIG. FIGS. 4A to 4H are examples of images (first measurement data) obtained by photographing an object in eight different postures. Here, for the sake of simplification of the explanation, only one object is arranged instead of being stacked separately. Similar to FIG. 2, the solid line portion is a contour that can be observed in the photographed image, and the dotted line portion is located on the back side of the component, and shows a contour that is not originally observed. In addition, in order to clearly show the distinction between postures, the object coordinate system is also shown. Here, in FIGS. 4 (A), (B), (C), and (D), since the groove which is a specific portion can be observed, the posture can be specified from the captured image. On the other hand, in FIGS. 4 (E), (F), (G), and (H), since the specific part is located on the back side and cannot be observed on the captured image, the appearances are completely matched and distinguished from each other. I can't get it. In this case, it is known that the calculated posture is one of these four, but since the posture cannot be specified by the specific part, the object cannot be correctly supplied to the next process.

Therefore, it is determined whether or not the posture of the object can be uniquely specified with respect to the calculation result using the captured image of the first sensor. If the calculation result is a posture that allows sufficient observation of a specific part, it is judged that the posture can be specified without ambiguity in the calculation result, and after picking by the robot, it is supplied to the next process as it is. .. On the other hand, if the specific part cannot be observed sufficiently, it is judged that the posture cannot be specified and the object is picked once. After that, the robot arm is controlled so that the specific part can be observed by the second sensor installed near the first sensor, the second measurement data of the object is newly acquired, and the posture is specified. It is preferable that the first sensor and the second sensor are installed at different angles. By installing at different angles, when the posture of the object cannot be uniquely specified by the first sensor, the posture of the object can be uniquely specified by the second sensor more reliably.

In FIG. 5, in the present embodiment, the information processing device 1 is used as a measuring device in combination with a sensor for measuring the object, the posture of the objects in a loosely stacked state is specified, and the robot moves to the next process. It is a flowchart which shows the processing procedure which supplies. Hereinafter, each step will be described.

First, in step S100, the determination unit 20 acquires a three-dimensional shape model of the object. As the three-dimensional shape model, for example, a polygon model in which an object shape is approximately represented by a plurality of triangular faces (polygons) can be used. Each polygon is composed of three-dimensional coordinates of points on the surface of an object and connection information of those points. Polygons are generally composed of triangles, but may be quadrangles or pentagons. In addition, as the three-dimensional shape model, a model in which the shape is expressed by a set of divided parameter curved surfaces, which is called a B-Rep (Boundary-Representation) expression like CAD data, may be used. In addition, any model that can express the three-dimensional shape of the object may be used. In this embodiment, the three-dimensional shape model acquired in step S100 is used for determining (verifying) the observation status of a specific part.

Next, in step S110, the determination unit 20 extracts a specific portion from the acquired three-dimensional shape model. Here, it is assumed that the three-dimensional shape model of the object is displayed on the screen and the specific part is extracted by the user specifying it with the GUI. FIG. 6 shows an example of GUI (Graphical User Interface). As shown in FIG. 6, the user changes the three-dimensional shape model displayed on the screen to a posture in which a specific part is observed, and then specifies the specific part by clicking the specific part with the mouse cursor. The determination unit 20 extracts the surface corresponding to the clicked position as a specific portion. As a result, the specific portion as shown in the shaded area in FIG. 2 is extracted. Various other methods can be used to extract a specific site. For example, the observation surface included in the two-dimensional region specified by dragging the mouse may be extracted as a specific part. Alternatively, a three-dimensional region such as a bounding box or a sphere composed of a rectangular frame line that includes a specific portion may be designated by numerical input, and the portion included in the designated region may be extracted as the specific portion. Then, the specific part of the three-dimensional shape model extracted here is recorded in a storage unit (not shown). When extracting the information of the specific portion in step S110, the determination unit 20 also functions as an extraction means. The processes of steps S100 to S110 may be performed online, but it is desirable to perform the processes offline in advance.

Next, in step S120, the first calculation unit 10 first acquires the first measurement data with the first sensor. In this embodiment, a distance image and a shade image are acquired as the first measurement data by using an image sensor capable of simultaneously capturing a distance image and a shade image as the first sensor. It is desirable to install the first sensor in a work space where the entire storage box can be measured at one time in consideration of the measurement range and the size of the storage box. As shown in FIG. 3, a unique coordinate system is set in advance in the first sensor. In this embodiment, this coordinate system will be referred to as a first sensor coordinate system.

The first calculation unit 10 detects one individual from the objects to be stacked separately using the first measurement data, and determines the relative posture (relative position posture) between the sensor coordinate system and the object coordinate system. Calculated as posture (positional posture). Specifically, first, by voting using the pre-learned classification tree for both acquired images (distance image, shading image), one individual is detected from the piled up and the rough posture is calculated. After that, the feature amount (geometric feature) is extracted from both the acquired 3D shape model and the acquired image, and the feature amount on the extracted 3D shape model and the feature amount extracted from both acquired images are fitted. The first posture is calculated by correcting the position and posture so as to do so. The feature amount (geometric feature) is, for example, an edge, a contour, a point, or the like extracted from an image or a three-dimensional shape model. Not limited to this, the user may set characteristic information in advance.

The measurement data to be acquired and the calculation method of the object are not limited to this, and various methods can be combined. For example, a camera capable of acquiring only a grayscale image may be used as the first sensor. Alternatively, a sensor capable of acquiring only a distance image may be used. Sensors that can acquire distance images include those that measure the distance by triangulation by capturing the reflected light of the laser light or slit light that irradiates the target with a camera, and the Time-of-light method that uses the flight time of the light. Is available. Alternatively, one that calculates the distance by triangulation from the image taken by the stereo camera can be used. Further, as for the calculation method of the object, for example, in the detection of one individual, pattern matching with images observed from a large number of postures may be performed instead of voting using a classification tree. Alternatively, a correspondence relationship with a high degree of similarity is specified between the feature amount extracted in advance from the three-dimensional shape model of the object and the feature amount detected from the first measurement data, and the first posture is calculated directly from the feature amount. May be good. In addition, any method other than that described here may be used as long as it is possible to find an individual to be gripped from being piled up in bulk and calculate the first posture.

Next, in step S130, the determination unit 20 determines whether or not the posture needs to be specified by the second calculation unit 30. That is, based on the first measurement data which is the posture calculated by the first calculation unit 10, the measurement data regarding the specific part which is the part for uniquely specifying the posture of the object is sufficiently obtained. In that case, it is determined that sufficient results have been obtained. Then, when the measurement data for the specific part is not sufficiently obtained, it is determined that the measurement data is not sufficiently obtained. Further, it is determined that sufficient measurement data is not obtained even when the posture of the object cannot be specified because the measurement data for the specific part is not possessed. In this embodiment, a virtual image is generated from the virtual measurement data observed by the virtual camera based on the first posture of the three-dimensional shape model of the object, and a specific part in the virtual image is extracted. After that, the size of the specific part is calculated and used as a criterion. A coordinate system similar to that of the first sensor is set in the virtual camera. Further, the internal parameters of the virtual camera may be arbitrarily set by the user, but it is desirable to match the internal parameters with the first sensor. The determination unit 20 arranges the virtual camera and the three-dimensional shape model of the object in the first posture, and renders the three-dimensional shape model of the object on the observation image plane of the virtual camera.

At this time, a hidden surface process is performed so as not to draw a surface that is shaded from the viewpoint and is originally invisible on the virtual image, and a specific part of the object can be identified. For the hidden surface processing, for example, a depth buffer method or a BSP method (Binary Space Partitioning) can be used. In addition, other methods may be used in consideration of the processing time and the nature of the three-dimensional shape model to be used. In addition, a unique brightness value is assigned to the specific part and drawing is performed so that the area corresponding to the specific part of the object can be specified on the virtual image.

Next, the determination unit 20 counts the number of pixels (number of pixels) having the brightness value assigned to the specific portion on the generated virtual image. When this number of pixels is defined as the size of a specific portion of the object and the number of pixels is equal to or greater than a predetermined threshold value, the determination unit 20 calculates the second posture based on the second measurement data in step S140 described later. Not performed. That is, it is determined that the measurement data relating to the specific part is sufficiently obtained (observed) to specify the posture, and the posture can be specified, and the second measurement is not performed. Alternatively, the second measurement data is not calculated. Alternatively, for example, even when the second measurement data is acquired by using another means or the like, since it is determined that the posture can be specified, the second measurement data based on the second measurement data is determined. Posture is not calculated. Then, the robot hand picks the object based on the first posture and supplies it to the next process. On the other hand, when the number of pixels is less than a predetermined value, the measurement data relating to the specific portion is not sufficiently obtained to specify the posture. That is, it is determined that the posture cannot be specified in the first posture, and the process proceeds to step S140. In consideration of the performance of the second calculation unit 30, it is preferable to specify the minimum number of pixels required to specify the posture, for example, about 50 as the threshold value. In this embodiment, the number of pixels in the region corresponding to the specific region is used as it is as the size of the specific region. However, it may be determined by another index according to the method used for the second calculation unit 30. For example, if the feature amounts (edges, contours, points) extracted from the image are used for the calculation of the second posture, the determination may be made based on the number of feature amounts that can be extracted from the region corresponding to the specific portion.

Next, in step S140, the second calculation unit 30 picks the object with the robot hand based on the first posture. After that, the robot arm is controlled to change the relative position in order to adjust the arrangement relationship between the sensor and the object so that the posture of the specific part of the object can be easily specified. This process will be described with reference to FIG. It is assumed that the posture of the object shown by the solid line is calculated using the first measurement data acquired by the first sensor, and the first posture is obtained. Further, it is assumed that in step S130, it is determined that the measurement data relating to the specific portion is not sufficiently obtained to specify the posture in the first posture, and the posture cannot be specified. At this time, there is a high possibility that a specific part will be observed by re-observing the object from a direction facing the first sensor or from a different angle. Therefore, the surface located on the first sensor side is the front surface of the object, the opposite side is the back surface, the back surface faces the second sensor, and the object is located in the center of the measurement range of the second sensor. Control the robot arm.

When arranging the object with respect to the second sensor, it is important that the surface of the object to be measured by the second sensor is not shielded by the robot hand. Therefore, when picking the objects from the loosely stacked state, the objects are arranged in consideration of the relative positional relationship of the objects with respect to the second sensor thereafter. It is good to decide the picking method such as the hand to be used or the part to be picked. For example, when the object is placed by the above method without changing the relative positional relationship between the robot hand and the object when picking, the posture at the time of picking is set so that the shielded area by the hand is as small as possible. To do. In addition, although the opening / closing type multi-finger hand is used in FIGS. 3 and 7, when the processing is performed by the second calculation unit 30, for example, the grip is performed by pressing a magnet or a suction pad. It may be used by switching to a hand, or a combination of these may be used. Further, a hand having a large number of joints and gripping the object in a form of being wound around the object while being flexibly driven may be used. These hand switching described above may be used when processing is performed by the first calculation unit 10, and may be switched to an openable and closable multi-finger hand when processing is performed by the second recognition calculation unit.

In the present embodiment, when the posture cannot be specified in the first posture, it means that the specific portion does not exist on the surface of the object which is the surface located on the first sensor side. Therefore, if the hand is pressed against the surface for picking, the risk of hiding a specific part due to the contact of the hand at the time of picking can be eliminated.

Next, in step S150, the second calculation unit 30 grips the object with the robot hand based on the first posture. Further, the robot is controlled to change the relative position between the object and the second sensor from the position where the first measurement data is acquired (at the time of the first measurement) so that the object fits within the measurement range of the second sensor. After that, the second measurement data of the object is acquired by the second sensor. Then, the second posture of the object is calculated based on the acquired second measurement data. In the example shown in FIG. 7, the second measurement data is one of the four types of images by changing the relative position and arranging the objects in order to adjust the arrangement relationship of the objects in step S140. That is, since a specific portion is observed in each of these four types of images, the posture can be uniquely specified by recalculating the posture by the second calculation unit 30. In this embodiment, the second calculation unit 30 uses the same configuration as that of the first calculation unit 10, and detailed description thereof will be omitted. However, as in step S120, the measurement data to be acquired and the method of recognizing the object are not limited to this, and various methods can be used. For example, in step S140, the relative position of the object in the second sensor can be changed and arranged so as to be limited to a few patterns. In that case, it is possible to simplify the process of calculating the posture of the object by using pattern matching with the observed image in those postures.

In addition, any method other than that described here may be used as long as the posture of the object can be uniquely specified from the specific part included in the second measurement data. Based on the second posture calculated here, the robot hand supplies the object to the next process.

As described above, in the first embodiment, it is determined that the specific part of the object is hidden from the viewpoint and cannot be observed, and the posture cannot be uniquely specified. Described how to identify. As a result, the posture of the object can be efficiently specified by performing the process of determining the necessity of the measurement data in the second stage and verifying the measurement method from the recognition result of the objects in the loosely stacked state.

In addition, there may be a case where a specific part cannot be observed in any of the first measurement data and the second measurement data. For example, it is assumed that an object exists in a stack in a posture in which the surface A of FIG. 2 faces the first sensor. In this case, since the specific portion is shaded from the viewpoint in the first measurement data and the posture cannot be specified, the measurement data is reacquired by the second sensor. However, by the method shown in step S140, even if the object is arranged so that the back side of the surface A, that is, the surface B faces, the specific part cannot be observed and the posture cannot be specified. Similarly, for an object having a posture in which the surface C faces the first sensor, the second sensor observes the surface D, and the posture cannot be specified. In consideration of such a case, the second calculation unit 30 calculates the second posture in step S150, and then again performs the same processing as in step S130 based on the second posture to specify the posture. The determination unit 20 may determine whether or not it has been completed. As a result, if it is determined that the posture cannot be specified, the second calculation unit 30 updates the posture of the object with respect to the second sensor, acquires new measurement data, and obtains the second posture. Is recalculated. This process is repeated until the posture can be uniquely identified.

Here, when updating the posture of the object, it is desirable to be able to observe the object in a posture significantly different from that of the first sensor and the second sensor. For example, according to the method of step S140, the first sensor and the second sensor are observing the object from two opposite directions, respectively. Therefore, it is preferable to change the posture of the object so that the object can be observed from a direction orthogonal to the straight line connecting the observation viewpoints of the first sensor and the second sensor with respect to the object. In this way, by sequentially changing the posture of the object so that the overlapping region between the surface already observed with respect to the object and the surface observed by the updated posture is reduced, the observation probability of the specific part is reduced. Can be enhanced. This makes it possible to suppress the number of acquisitions of measurement data and efficiently identify the posture. Further, as a method of arranging the object with respect to the second sensor, a method other than the above may be adopted.

In this embodiment, an example in which the first sensor and the second sensor are fixedly installed in the work space has been described, but the method of installing the sensors is not particularly limited to this. For example, the first sensor may be attached to the robot arm used for picking an object together with the hand. Further, another robot arm may be prepared separately from the robot arm to which the hand is attached, and the second sensor may be attached to the other robot arm. In this case, by using two robot arms, not only the object picked by the hand but also the position and orientation of the second sensor can be controlled. Therefore, when the relative positions are changed so that a specific part can be observed and both are arranged, it is possible to make it more difficult to be restricted by the movable range of the robot arm. In addition, one sensor in which the first sensor and the second sensor are stacked separately and installed above may be combined. In this case, the number of sensors used can be reduced and the work space can be made compact.

In addition to the objects mentioned here, examples of objects to which the same method can be applied are shown in FIGS. 8 and 9. 8 (A) (B) and 9 (A) (B) both schematically represent the first measurement data for one object existing in the bulk stacking. The solid line part shows the observable contour, and the dotted line part shows the contour that is not originally observed. At this time, since the specific portion can be observed in both FIGS. 8 (A) and 9 (A), the posture can be uniquely specified. On the other hand, in both FIGS. 8 (B) and 9 (B), the specific portion is shaded and cannot be observed. By using the method described in this embodiment for such an object, it is possible to determine the necessity of the measurement data in the second stage and efficiently identify the posture of the object.

[Example 2]
In the first embodiment, it is determined whether or not the posture can be uniquely specified by calculating the size at which the specific part of the object is observed in the virtual image by using the result calculated from the measurement data of the piled-up state. did. On the other hand, it may be determined whether or not the posture can be specified by whether or not there is another posture that cannot be distinguished from the result recognized in the loosely stacked state. Since the configuration of the information processing device 1 of this embodiment is the same as that of the first embodiment, the description thereof will be omitted. The processing procedure of this embodiment does not require extraction of a specific part. Therefore, since step S110 in the first embodiment is unnecessary, the description thereof will be omitted. Instead, the process of step S130 shown in the first embodiment is changed as follows. Further, since steps S120, S140, and S150 are the same processes as in the first embodiment, the description thereof will be omitted.

In step S130, the determination unit 20 generates a virtual image from the virtual measurement data obtained by observing the three-dimensional shape model of the object from the first posture with the virtual camera, and generates this virtual image as a reference image. Further, the determination unit 20 arranges the virtual camera in a reference posture (reference position posture) which is a posture different from the first posture with respect to the three-dimensional shape model of the object. The determination unit 20 generates a virtual image as a reference image from the virtual measurement data when the three-dimensional shape model of the object is observed from the reference posture. Then, as a result of comparing the reference image and the reference image, if they match, it means that the first posture is indistinguishable from the reference posture. Therefore, in this embodiment, the difference amount between the reference image and the reference image is used as a criterion for determining whether or not the posture can be specified in the first posture. The determination unit 20 calculates the amount of difference from the reference image for various reference postures, and if even one item whose difference amount is less than a predetermined threshold value is found, sufficient virtual measurement data is obtained. It is determined that the posture has not been uniquely specified and the posture cannot be uniquely specified, and the process proceeds to step S140. On the other hand, if no one is found, it is determined that sufficient virtual measurement data has been obtained and the posture has been identified, the object is grasped by the robot hand based on the first posture, and the object is supplied to the next process.

Here, for the difference amount, for example, the absolute value sum of the differences using the density values of both images can be used. The depth value of each pixel acquired from the depth buffer may be used instead of the density value. Alternatively, the absolute difference value of each pixel may be calculated, and the number of pixels whose absolute difference value is equal to or greater than a certain threshold value may be used as the difference amount. Further, instead of the difference amount, the normalized cross-correlation between the reference image and the difference image may be calculated as the degree of similarity and used. In this case, if even one reference posture having a similarity equal to or higher than a predetermined threshold value is found, it is determined that the first posture cannot uniquely identify the posture, and conversely, if no one is found, the posture is specified. Judge that it was completed. In addition to this, of course, any other index may be used as long as the degree of agreement between the reference image and the reference image can be verified.

The reference posture information may be used as the object placement method for the second sensor in step S140. In this case, as in the first embodiment, the specific part is extracted from the three-dimensional shape model of the object. For example, assuming that the first posture is FIG. 4 (E), for example, the three postures of FIGS. 4 (F), (G), and (H) are found as indistinguishable reference postures. Suppose. Here, the relative relationship between the first posture and the remaining three reference postures is recorded as a similar posture. Here, the relative posture corresponding to FIG. 4 (F) is referred to as the similar posture 1, the relative posture corresponding to (G) is referred to as the similar posture 2, and the relative posture corresponding to FIG. 4 (H) is referred to as the similar posture 3. From these similar postures 1 to 3 and the information on the specific part of the object, the relative position between the second sensor and the object is changed to determine the arrangement relationship.

FIG. 11 is a diagram illustrating a method of setting an object arrangement with respect to the second sensor. The method of setting the object arrangement with respect to the second sensor will be described with reference to FIG. In FIG. 11, various observation viewpoints surrounding the object are set as candidate viewpoints. Generates a virtual image when observing an object from the set candidate viewpoint. This image is taken as an image of the similar posture 0, and the size of the specific portion observed in the similar image 0 is calculated. For this, the method described in the first embodiment may be used. Further, using the same method, a virtual image is generated when the object is observed by performing posture conversion based on the recorded similar posture from the same viewpoint. This image is used as an image of similar postures 1 to 3, and the size of a specific portion in each similar image is calculated.

Then, the sum of the sizes of the specific parts observed for each candidate viewpoint is calculated. It can be said that the larger this value is, the larger the specific part looks when observed from the candidate viewpoint, and it is easier to distinguish similar postures. Therefore, processing is performed for each of the set candidate viewpoints, and the viewpoint having the largest value is set as a method of arranging the second sensor and the object. Alternatively, for each candidate viewpoint, the maximum value of the size of a specific portion may be calculated from the generated images of similar postures, and these may be compared to set the viewpoint having the largest value. In this case, in the example of FIG. 11, when observing from the candidate viewpoint 2, the specific part cannot be observed in any of the images, whereas when observing from the candidate viewpoint 1, the specific part can be observed from the front, so that the candidate viewpoint cannot be observed. 1 will be set.

As described above, in the second embodiment, a method of determining whether or not the posture can be specified by whether or not there is another posture that cannot be distinguished from the result recognized in the loosely stacked state has been described. As a result, the posture of the object can be efficiently specified by performing the process of determining the necessity of the measurement data in the second stage and verifying the measurement method. Then, by using this embodiment, it is possible to determine whether or not the posture of the object can be specified without extracting a specific part from the three-dimensional shape model.

[Example 3]
In Example 1, the size of the specific portion observed in the first posture was determined (verified) by a virtual image subjected to the hidden surface treatment. However, if the specific part and the non-specific part (parts other than the specific part) have different colors on the actual object and the two can be distinguished by the color on the measurement data, that information. May be used. Since the configuration of the information processing device 1 of this embodiment is the same as that of the first embodiment, the description thereof will be omitted. In this case, as in the second embodiment, step S110 is unnecessary, so the description thereof will be omitted. Instead, a camera capable of acquiring a color image is used as the first sensor, and only the processing in step S130 is changed as follows. Since steps S120, S140, and S150 are the same processes as in the first embodiment, the description thereof will be omitted.

In step S130, the determination unit 20 arranges the virtual camera and the three-dimensional shape model of the object in the first posture, and renders the three-dimensional shape model of the object on the observation image plane of the virtual camera. .. The purpose of generating the virtual image in this embodiment is to specify the projection region of the object recognized in step S120 on the first measurement data. Therefore, it is not necessary to perform the hidden surface treatment, and it is not necessary to perform the treatment of assigning a unique luminance value to a specific portion. The determination unit 20 identifies the projection region of the object in the rendered image, and further refers to the data in the same region from the first measurement data. If a pixel having a brightness value close to the color of a specific part exists in the region referred to here, it is considered that the pixel can measure the specific part.

Therefore, a luminance range that can be regarded as a color peculiar to a specific part of the object is set in advance, and that is used as measurement data related to the specific part characteristic. Then, in the referenced area, pixels having a color within the set luminance value range are detected, and the number is counted. With this number of pixels as the size of the specific part of the object, when the number of pixels is equal to or greater than a predetermined threshold value, it is determined that sufficient measurement data regarding the specific part feature can be obtained and the posture can be specified. At this time, the robot hand picks the object based on the first posture and supplies it to the next process. On the other hand, if the number of pixels is less than a predetermined value, it is determined that sufficient measurement data regarding the specific part has not been obtained, that is, the posture cannot be specified in the first posture, and the process proceeds to step S140. ..

As described above, in Example 3, a method of determining whether or not the posture can be specified by the color when the specific part and the part other than the specific part have different colors on the actual object has been described. From this, the posture of the object can be efficiently specified by performing the process of determining the necessity of the measurement data in the second stage and verifying the measurement method. By using this embodiment, it is possible to determine whether or not the posture of the object can be specified as in the second embodiment without extracting a specific part from the three-dimensional shape model.

[Example 4]
In this embodiment, a case will be described in which the resolution of the sensor that measures a specific part of the object is insufficient and the specific part cannot be observed sufficiently large, and the posture cannot be uniquely specified. FIG. 10 shows an example of the shape of the object to be handled in this embodiment.

As shown in the example shown in FIG. 10 (A), this object has two small holes as specific parts so as to penetrate the main plane. The two small holes indicate the shaded areas in FIG. 10 (A). This specific part serves as a clue to specify the front and back sides of the main plane and the in-plane rotation posture. When the loosely stacked state of the objects is measured by the first sensor installed above the storage box, the objects in the lower layer are observed to be relatively smaller than those in the upper layer of the storage box. Therefore, as shown in FIG. 10B, there is a possibility that the specific part cannot be observed and the posture cannot be uniquely specified. In this embodiment, a method of identifying such a case and specifying the posture by measuring with the second sensor will be described.

Since the configuration of the information processing device 1 of this embodiment is the same as that of the first embodiment, the description thereof will be omitted. Next, the processing flow in this embodiment will be described. Since the basic processing flow is the same as that of the first embodiment, only step S130 and step S140 having different processes will be described here, and the description of step S110, step S120, and step S150 will be omitted.

In step S130, the determination unit 20 arranges the virtual camera and the three-dimensional shape model of the object in the first posture, and places a portion corresponding to the specific part of the object on the observation image plane of the virtual camera. Render. Here, the difference from the first embodiment is that only a specific part is rendered, not the entire three-dimensional shape model of the object. In this way, it is determined (verified) how large the specific part is observed, assuming that it is observed without being shielded at all. Since only a specific part is rendered, it is not necessary to perform the hidden surface processing. If the distance between the sensor and the object is small in the first posture, the area where the specific part is rendered on the virtual image becomes large. Conversely, the larger the distance, the smaller the area.

Therefore, the pixel in which the specific part is rendered is specified and the number is counted. The specific part may be specified, for example, by rendering so that the background and the specific part can be distinguished, or by distinguishing the background and the specific part from the depth value of each image. Of course, other methods may be used. Using the number of pixels of the specific part counted here, if the number of pixels is equal to or greater than a predetermined threshold value, it is determined that sufficient measurement data regarding the specific part feature can be obtained and the posture can be specified. In this case, the robot hand supplies the robot to the next process based on the first posture. On the other hand, when the number of pixels is less than a predetermined value, sufficient measurement data regarding a specific portion is not obtained. That is, it is determined that the posture cannot be specified in the first posture, and the process proceeds to step S140. Similar to the first embodiment, the threshold value is preferably specified as the minimum number of pixels required for specifying the posture, for example, about 50 in consideration of the performance of the second calculation unit 30.

In step S140, the second calculation unit 30 picks the object with the robot hand based on the first posture, and then controls the robot arm so that the specific part can be observed sufficiently large by the second sensor. , Move the picked object. In this embodiment, the surface observed by the first sensor faces the second sensor as it is, and the distance to the object with respect to the second sensor is appropriately set and arranged. Normally, if an object is observed by halving the distance, the observed area will be quadrupled. Therefore, based on the first posture, the distance between the first sensor and the object is calculated, and based on this distance, the number of pixels of the specific part calculated in step S130, and the number of pixels required for posture identification. Then, the distance between the second sensor and the object is calculated.

For the sake of simplicity, it is assumed that the second sensor and the first sensor have the same resolution in this embodiment. For example, it is assumed that the number of pixels of the specific portion is 20 in step S130. Further, it is assumed that a specific part needs to be observed with a size of about 50 pixels in order to specify the posture from the measurement data acquired by the second sensor. At this time, assuming that the distance between the first sensor and the object is D, the distance D'between the second sensor and the object is expressed by the following equation (1).
D'= D × √ (50/20) ≒ D / 1.6 (1)
By arranging the object closer than the distance D'calculated here, it can be expected that the second sensor will acquire the measurement data of the image feature with the size required for the posture identification. When the resolutions of the first sensor and the second measurement sensor are different, D'may be calculated in consideration of the resolutions of both. As in the case of the first embodiment, it is preferable to pick the object by setting the picking position / posture so as not to generate the shielding by the hand as much as possible when the above-mentioned object is arranged.

As described above, in the fourth embodiment, the case where the resolution of the sensor is insufficient and the specific part cannot be observed sufficiently large, the measurement data for the specific part cannot be sufficiently obtained, and the posture cannot be uniquely specified is determined, and the measurement data is measured again. Was obtained, and the method of calculating the posture of the object was described. As a result, the posture of the object can be efficiently specified by performing the process of determining the necessity of the measurement data in the second stage and verifying the measurement method. The method described in this example can also be used in combination with the methods described in Examples 1 to 3.

[Example 5]
In this embodiment, it is determined whether or not the posture can be uniquely specified based on the reliability of the data in the area corresponding to the specific part in the measurement data acquired by the first sensor, and the measurement data is measured again if necessary. Will be described to describe the method of calculating the posture of the object.

Even if the posture calculated in the loosely stacked state is the result of uniquely identifying the posture, the calculated result is doubtful if the measurement data corresponding to the specific part has a click or a defect. Therefore, in this embodiment, when the reliability of the measurement data corresponding to the specific part is extremely low, the measurement data is acquired again and the posture is recalculated. Since the configuration of the information processing device 1 of this embodiment is the same as that of the first embodiment, the description thereof will be omitted.

Next, the processing flow in this embodiment will be described. Since the basic processing flow is the same as that of the first embodiment, only step S130 and step S140 having different processing will be described here, and the description of step S110, step S120, and step S150 will be omitted.

The determination unit 20 arranges the virtual camera and the three-dimensional shape model of the object in the first posture, and renders the three-dimensional shape model of the object on the observation image plane of the virtual camera. At this time, a hidden surface process is performed so as not to draw a surface that is shaded from the viewpoint and is originally invisible on the virtual image, and a specific part of the object can be identified. The processing up to this point is the same as in the first embodiment. Next, the determination unit 20 extracts pixels corresponding to a specific portion on the virtual image, and refers to data corresponding to the same position as the extracted pixels from the first measurement data. The data referred to here corresponds to a specific part in the first measurement data.

Therefore, it is determined whether or not the reliability of the measurement data of the corresponding pixel is high. Here, if there is a gap in the shading image or a defect in the distance image in the first measurement data, it is determined that the data of the pixel has low reliability, and if not, it is determined that the reliability is high. Then, the number of pixels determined to have low reliability is counted, and when the number of pixels is less than a predetermined threshold value, sufficient measurement data regarding a specific portion is obtained, and the first calculation result is correct. That is, it is determined that the posture can be specified, and the robot hand supplies the robot hand to the next process based on the first posture. On the other hand, when the number of pixels with low reliability is equal to or higher than a predetermined threshold value, sufficient measurement data for a specific part is not obtained, and the first posture is suspicious, that is, the posture cannot be specified. The determination is made, and the process proceeds to step S140. Alternatively, if the data with low reliability is equal to or more than a predetermined ratio among all the data corresponding to the specific portion, it may be determined that the posture cannot be specified. Further, in the first measurement data, when the data of each pixel has individual reliability, the value may be used for determination.

In addition, any index or determination method may be used as long as it can be determined whether or not the measurement data is sufficiently reliable for identifying the posture. In addition, the area corresponding to the specific part is specified from the first measurement data, and the degree of agreement with the three-dimensional shape model of the object arranged based on the first posture is calculated for the measurement data in the area. However, if the degree of coincidence is low, it may be determined that the posture cannot be specified correctly.

The second calculation unit 30 picks the object with the robot hand based on the first posture, then controls the robot arm, and is picked so that the specific part can be appropriately observed by the second sensor. Move the object. In the first measurement data, the influence of reflection by the storage box or surrounding objects can be considered as a factor that causes the data corresponding to the specific part to be chipped or missing. In this case, even if the observation posture of the object is not changed, the posture can be specified by correctly measuring the data of a specific part by re-measuring only one object so as not to be affected by surrounding objects. is there. Therefore, in this embodiment, the relative relationship between the sensor and the object with respect to the first sensor is arranged so as to be the relative relationship between the second sensor and the object as it is. Alternatively, the first sensor and the second sensor may slightly change the inclination of the observation surface of the object with respect to the optical axis, rotate the object around the optical axis, and the like. In addition, in the second measurement data, any arrangement method may be used as long as it is possible to reduce defects and scratches in the region corresponding to the specific portion and improve the quality. As in the case of the first embodiment, when the above-mentioned object is arranged, it is preferable to set the picking posture so as not to generate the shielding by the hand as much as possible to pick the object.

As described above, in the fifth embodiment, it is determined whether or not the posture can be uniquely specified based on the reliability of the data in the area corresponding to the specific part in the measurement data acquired by the second sensor, and if necessary, again. The method of acquiring the measurement data and calculating the posture of the object was described. As a result, the posture of the object can be efficiently specified by performing the process of determining the necessity of the measurement data in the second stage and verifying the measurement method. The method described in this example can also be used in combination with the methods described in Examples 1 to 4.

[Examples relating to the article manufacturing method]
The information processing device 1 described above can be used in a state of being supported by a certain support member. In this embodiment, as an example, a control system attached to and used in the robot arm 400 (grip device) as shown in FIG. 12 will be described. Although not shown in FIG. 12, the information processing device 1 is configured as a measuring device using the first sensor and the second sensor as shown in FIG. 3, and the object W placed on the support base T. The pattern light is projected onto the image to be imaged, and an image is acquired. Then, the control unit of the information processing device 1 or the arm control unit 310 that has acquired the image data output from the control unit of the information processing device 1 determines the position and posture of the object W. The arm control unit 310 controls the robot arm 400 by sending a drive command to the robot arm 400 based on the position and posture information (measurement result). The robot arm 400 holds the object W with a robot hand or the like (grip portion) at the tip, and moves the object W in translation or rotation. Further, by assembling (assembling) the object W to other parts by the robot arm 400, it is possible to manufacture an article composed of a plurality of parts, for example, an electronic circuit board or a machine. In addition, an article can be manufactured by processing (processing) the moved object W. The arm control unit 310 includes an arithmetic unit such as a CPU as a computer and a storage device such as a memory for storing a computer program. A control unit that controls the robot may be provided outside the arm control unit 310. Further, the measurement data measured by the information processing apparatus and the obtained image may be displayed on a display unit 320 such as a display. Further, the object W may be gripped and moved in order to align the object W with respect to other parts. The object W has the same structure as the object of FIG. 2 and the like, and the robot arm 400 has the same configuration as the robot arm of FIG. The measuring device may be configured to include the robot arm 400.

[Other Examples]
Although the present invention has been described in detail based on the preferred examples thereof, the present invention is not limited to the above examples, and various modifications can be made based on the gist of the present invention. It is not excluded from the scope of the invention.

Further, a computer program that realizes a part or all of the control in this embodiment may be supplied to an information processing apparatus or the like via a network or various storage media. Then, the computer (or CPU, MPU, etc.) in the information processing device or the measuring device may read and execute the program. In that case, the program and the storage medium that stores the program constitute the present invention. As the recording medium, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory card, a ROM, or the like can be used.

Information processing device 1
First calculation unit 10
Judgment unit 20
Second calculation unit 30

Claims (17)

An information processing device that calculates the posture of an object.
A first calculation unit that calculates the first posture of the object based on the first measurement data obtained by measuring the object with a sensor, and
Based on the first measurement data, a determination unit for determining whether or not measurement data for a specific part of the object for specifying a posture has been sufficiently obtained for specifying a posture, and a determination unit.
When the determination unit determines that the measurement data for the specific portion is not sufficiently obtained to specify the posture, the relative position between the object and the sensor from which the first posture is calculated is changed. The second posture of the object is calculated based on the second measurement data obtained by measuring the specific portion of the object, and the measurement data regarding the specific portion is sufficiently obtained to specify the posture. An information processing device including a second calculation unit that does not calculate the second posture based on the second measurement data when the determination unit determines that the data has been obtained. The first aspect of the present invention is characterized in that, when the number of pixels corresponding to the specific portion is equal to or greater than a predetermined threshold value, the determination unit determines that the measurement data is sufficiently obtained to specify the posture. The information processing device described. The determination unit compares the virtual measurement data of the object in the first posture with the virtual measurement data of the object in a reference posture different from the first posture, so that the measurement data is in the posture. The information processing apparatus according to claim 1, wherein it is determined whether or not sufficient data is obtained to specify. Based on the brightness value corresponding to the color of the specific portion, the determination unit has sufficiently obtained the measurement data to specify the posture when the brightness value corresponding to the color is equal to or more than a predetermined threshold value. The information processing apparatus according to claim 1, wherein the information processing apparatus is determined to be. The determination unit determines that the measurement data is sufficiently obtained to specify the posture when the reliability is equal to or higher than a predetermined threshold value, using the number of pixels of the measurement data in the specific portion as the reliability. The information processing apparatus according to claim 1, wherein the information processing apparatus is used. The determination unit counts the number of pixels of the specific portion of the virtual measurement data based on the result of rendering the specific portion of the virtual measurement data of the object in the first posture, and then the number of the pixels is calculated. The information processing apparatus according to claim 1, wherein it is determined that the measurement data is sufficiently obtained when the value is equal to or higher than a predetermined threshold value. In the determination unit, when the posture of the object is not specified because the first measurement data does not have the measurement data for the specific part, the measurement data for the specific part specifies the posture. The information processing apparatus according to any one of claims 1 to 6, wherein it is determined that the data is not sufficiently obtained. When the determination unit determines that the measurement data for the specific portion is not sufficiently obtained to specify the posture, the determination unit determines the relative position between the object and the sensor calculated as the first posture. The information processing apparatus according to any one of claims 1 to 7, wherein a control signal for changing is generated. The sensors are a first sensor for acquiring the first measurement data of the object and a second sensor for acquiring the second measurement data for calculating the posture of the object. The information processing apparatus according to any one of claims 1 to 8. The information processing apparatus according to claim 9, wherein the first sensor and the second sensor are installed at different angles with respect to the object. An information processing method that calculates the posture of an object.
A first calculation step of calculating the first posture of the object based on the first measurement data obtained by measuring the object with a sensor, and
Based on the first measurement data, a determination step of determining whether or not measurement data relating to a specific part of the object for specifying a posture has been sufficiently obtained to specify the posture, and a determination step.
When it is determined in the determination step that the measurement data for the specific portion is not sufficiently obtained to specify the posture, the relative position between the object and the sensor from which the first posture is calculated is changed. Then, the second posture of the object is calculated based on the second measurement data obtained by measuring the specific portion of the object, and the measurement data regarding the specific portion is sufficient to specify the posture. An information processing method comprising a second calculation step in which the calculation of the second posture based on the second measurement data is not performed when it is determined in the determination step that the data has been obtained. A program for causing a computer to function as each part of the information processing apparatus according to any one of claims 1 to 10. The information processing device according to any one of claims 1 to 10.
A system characterized by having a robot that grips and moves the object based on the posture of the object calculated by the information processing device. It has a gripping means for gripping the object, and the gripping means acquires the posture of the specific portion of the object with respect to the sensor after gripping the object based on the first posture. The system according to claim 13, wherein the relative position is changed for ease of use. A calculation step of calculating the posture of the object by the information processing apparatus according to any one of claims 1 to 10.
A method for manufacturing an article, which comprises a step of manufacturing a predetermined article by grasping the object and performing an assembly process of the object based on the calculated posture of the object. A measuring device that measures the posture of an object
A sensor that measures the object and
A first calculation unit that calculates the first posture of the object based on the first measurement data obtained by the first measurement of measuring the object by the sensor.
Based on the first measurement data, a determination unit for determining whether or not measurement data for a specific part of the object for specifying a posture has been sufficiently obtained for specifying a posture, and a determination unit.
When the determination unit determines that the measurement data for the specific portion is not sufficiently obtained to specify the posture, the relative position between the object and the sensor for which the first posture is calculated is changed. It also has a second calculation unit that calculates the second posture of the object based on the second measurement data obtained by the second measurement that measures the specific portion of the object.
A measuring device characterized in that the second measurement is not performed when the determination unit determines that the measurement data relating to the specific portion has been sufficiently obtained to specify the posture. It is a measurement method that measures the posture of an object.
The first calculation step of calculating the first posture of the object based on the first measurement data obtained by the first measurement of measuring the object by the sensor, and
Based on the first measurement data, a determination step of determining whether or not measurement data relating to a specific part of the object for specifying a posture has been sufficiently obtained to specify the posture, and a determination step.
When it is determined in the determination step that the measurement data relating to the specific portion is not sufficiently obtained to specify the posture, the relative position between the object and the sensor from which the first posture is calculated is determined. A second measurement is performed to measure the specific portion of the object by changing from the relative position at the time of the first measurement, and the object is based on the second measurement data obtained by the second measurement. It has a second calculation step of calculating the second posture, and has.
A measurement method characterized in that the second measurement is not performed when it is determined in the determination step that the measurement data relating to the specific portion is sufficiently obtained to specify the posture.

Images