JP7561590B2 - Information processing device, picking device, and program - Google Patents

Description

The present invention relates to an information processing device , a picking device , and a program .

Patent document 1 shows a system that recognizes the orientation of an object from an image of the object and measurement information of the contact position on the object.

JP 2017-136677 A

It is difficult to recognize the orientation of transparent or black objects, objects with mirror-like surfaces, or objects covered with cloth or the like from images alone. The system in Patent Document 1 makes it possible to recognize the position of objects wrapped in plastic bags or cushioning material by also using measurement information obtained by contacting the object. However, in a device that estimates the orientation of an object by contact, there is room for improvement in the way the object is contacted in order to perform efficient estimation.

The present invention aims to provide an information processing device and a picking device that can efficiently estimate the orientation of an object by using detection information obtained by contact.

The information processing device according to the present invention comprises:
An information processing device that estimates a posture of an object,
A storage unit that stores object information indicating a shape and a size of the object;
a force sensor for acquiring detection information of a contact point by contact;
A movement control unit that moves the force sensor;
an estimation unit that estimates a posture of the object based on the detection information obtained by the force sensor contacting the object multiple times and the object information;
Equipped with
The estimation unit estimates a plurality of candidates for a pose of the object before the plurality of contacts are performed,
The movement control unit moves the force sensor so that, during the multiple contacts, the force sensor comes into contact with different surfaces of the object, and further so that the force sensor comes into contact with a surface included in the multiple candidates having a higher degree of positional uncertainty than other surfaces included in the multiple candidates .
The program according to the present invention comprises:
An information processing device including a storage unit that stores object information indicating the shape and size of an object,
A means for moving the force sensor;
a means for acquiring detection information of a contact point from the force sensor;
a means for estimating a posture of the object based on the detection information obtained by the force sensor contacting the object multiple times and the object information;
A program for executing
the estimating means estimates a plurality of candidates for the pose of the object before the plurality of contacts are performed;
The moving means moves the force sensor so that, during the multiple contacts, the force sensor comes into contact with different surfaces of the object, and further so that the force sensor comes into contact with a surface included in the multiple candidates having a higher degree of positional uncertainty than other surfaces included in the multiple candidates.

The picking device according to the present invention comprises:
The above information processing device;
a take-out mechanism that takes out the object using the estimation result of the estimation unit;
Equipped with.

According to the present invention, it is possible to provide an information processing device , a picking device , and a program capable of efficiently estimating the orientation of an object by utilizing contact.

1 is a block diagram showing a picking device according to an embodiment of the present invention. 10 is a flowchart illustrating an example of a procedure of a picking process executed by a control unit. 1A to 1E are explanatory diagrams illustrating a first example of a process for estimating the orientation of an object, the diagram illustrating a first step (A) to a fourth step (D) (E). 11A to 11F are explanatory diagrams illustrating a first step (A) to a sixth step (F) of a second example of a process for estimating the orientation of an object.

Below, an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a picking device according to an embodiment of the present invention. The picking device 1 according to this embodiment is a device that estimates the posture of an object whose shape and size are known, and removes the object from a storage location by gripping or the like. The picking device 1 includes an imaging unit 21 that captures an image of the object, a force sensor 22 that can obtain detection information of a contact point by contacting the object, a driving device 23 that moves the force sensor 22, a robot arm 24 as a removal mechanism that grips and transports the object, and a control unit 10 that controls these. The picking device 1 corresponds to an example of an information processing device according to the present invention.

The image capturing unit 21 has an image sensor and an imaging lens, and captures an image as a digital signal. It is sufficient for the image capturing unit 21 to capture an image that allows the rough position of an object to be grasped. Alternatively, the image capturing unit 21 may be a depth sensor that can obtain depth information of each point on the image in addition to a two-dimensional image.

The force sensor 22 acquires detection information of the contact point when the contactor comes into contact with any surface. The detection information includes information on the position of the contact point and information indicating the direction of the reaction force applied to the contactor from the contact point, i.e., information on the orientation of the surface that includes the contact point (the surface normal). The force sensor 22 stops when it receives a very small reaction force, making it possible to acquire detection information of the contact point without moving the object in contact.

The driving device 23 is a three-dimensional driving device capable of moving the force sensor 22 (specifically, its contact) along any path. The driving device 23 that moves the force sensor 22 may also serve as the driving unit of the robot arm 24.

The control unit 10 is a computer equipped with a CPU (Central Processing Unit), a RAM (Random access memory) in which the CPU expands data, a storage device that stores a control program executed by the CPU, and an interface 16 that exchanges signals with the imaging unit 21, the force sensor 22, the drive device 23, and the robot arm 24. In the control unit 10, a plurality of functional modules are realized by the CPU executing the control program. The plurality of functional modules include an imaging control unit 12 that controls the imaging unit 21, an object information storage unit 11 that stores object information indicating the shape and size of an object, a movement control unit 13 that controls the drive device 23 to move the force sensor 22, an estimation unit 14 that estimates the posture of the object, and a robot control unit 15 that controls the drive of the robot arm 24.

The object information includes, for example, position data (relative position data) of multiple points uniformly distributed on the surface of the object. In addition to the position data of each point, the object information may also include normal data for each point. The normal data indicates the direction of a normal that is perpendicular to the surface on which each point is included and passes through that point. Note that the normal data for each point does not need to be given in advance, and may be calculated by the control unit 10 from the position data of the multiple points.

The estimation unit 14 estimates the approximate position of the object from the image captured by the imaging unit 21. The estimation based on the image may be a relatively low-precision estimation to the extent that the force sensor 22 can contact any part of the object. The specific method is not limited, but for example, the estimation unit 14 can estimate the center point of the object or the point where the object is located with a large tolerance by performing image recognition processing from the image.

The estimation unit 14 further estimates the posture of the object based on the detection information of the contact point obtained by the force sensor 22 contacting the object once or multiple times. Here, the posture of the object is a concept that includes the position and orientation of the object, but the estimation of the posture according to the present invention may be an estimation of only the orientation. When the number of contacts by the force sensor 22 is small and the posture of the object cannot be narrowed down to one, the estimation unit 14 estimates multiple candidate postures based on the detection information of the contact point. Then, as the number of contacts by the force sensor 22 increases, the candidate postures are narrowed down, and the estimation unit 14 can finally estimate one posture.

The movement control unit 13 determines a contact path along which the force sensor 22 comes into contact with an object, and moves the force sensor 22 along the determined contact path, so that the estimation unit 14 can efficiently estimate the posture of the object. The method of determining the contact path will be described in detail later.

The robot control unit 15 drives the robot arm 24 so that the robot arm 24 grasps and transports the object in the posture estimated by the estimation unit 14 based on the information on the posture of the object.

<Posture Estimation Method>
Next, a specific example of a method for estimating the orientation of an object by the estimation unit 14 will be described. Note that the method for estimating the orientation of an object by the estimation unit 14 is not limited to the following example. Here, a method for the estimation unit 14 to estimate the orientation of an object or multiple orientation candidates using the cost function G(r, t) of equation (1) will be described.

The robot coordinate system is a coordinate system based on the base of the robot arm 24, and refers to a coordinate system in which the origin and three axis directions of the coordinate system are fixed when the base of the robot arm 24 does not move. The object coordinate system is a coordinate system based on an object, and refers to a coordinate system in which the origin and three axis directions of the coordinate system rotate and translate relative to a reference coordinate system (e.g., the Earth's coordinate system) when the position and orientation of the object change. The rotation r and translation t are amounts that represent the position and orientation of an object in the robot coordinate system. For example, the difference between the origin of the robot coordinate system and the origin of the object coordinate system may be represented by the translation t, and the difference in the rotational direction between the three axes of the robot coordinate system and the three axes of the object coordinate system may be represented by the rotation r.

The above adjustment coefficient is a coefficient that adjusts the degree to which the first and second terms in the brackets on the right side of formula (1) affect the cost function. The first term is an element that reduces the cost function if a point m close to the contact point is included in the point group M that indicates the surface of the object, and increases the cost function as the distance between the point m closest to the contact point in the point group M and the contact point increases. The above second term is an element that reduces the cost function if a point m in the point group M has the same normal direction as the surface orientation of the contact point, and increases the cost function as the normal at point m that has the closest normal to the surface orientation of the contact point differs from the surface orientation of the contact point. The absolute value symbol represents the length of the vector, and "〈a, b〉" represents the inner product of vector a and vector b. "min [ ]" indicates the minimum value of the values in brackets at each point m. Therefore, the value in the summation symbol Σ approaches zero if the point cloud M contains a point close to the contact point and the normal direction of that point is close to the direction of the surface of the contact point; if no such point is contained, the value moves away from zero in the positive direction.

The estimation unit 14 searches for arguments (translation r, rotation t) that result in a small value near zero for the cost function G(r, t), and estimates the searched rotation and translation {r, t} as the object's pose or a candidate for the object's pose.

<Estimation procedure>
The estimation unit 14 first sets an initial region for searching the posture of the object based on the image obtained from the imaging unit 21 and the object information indicating the size and shape of the object. The initial region is set to a size with a margin so that the possibility of the object going out of the region is almost zero. For example, the estimation unit 14 obtains a circumscribing sphere of the object whose center is located at the center point based on the center point of the object obtained by image recognition of the image and the object information, increases the diameter of the circumscribing sphere by an error amount corresponding to the estimation accuracy of the image recognition, and sets the inside of the circumscribing sphere as the initial region.

Next, the estimation unit 14 sets all possible postures as the position and orientation of the object included in the initial region as candidates for the initial posture. Each posture is represented by a combination of rotation r and translation t. Therefore, the initial posture candidates are a set of rotation and translation combinations {r, t}. However, if a very small difference in translation or rotation is considered as a different posture, the number of candidates for the initial posture will be very large. Therefore, the estimation unit 14 appropriately sets the minimum difference Δt in translation and the minimum difference Δr in rotation, and creates a set I ₀ ={(r, t) _i , i=1, 2, ...} representing multiple candidates for the initial posture, with postures whose translation or rotation difference is equal to or greater than the minimum difference Δt, Δr being considered as different postures.

In addition, when there is an element that restricts the posture of the object, such as a table on which the object is placed, the estimation unit 14 may perform a process of excluding posture candidates that cannot occur due to the above restriction from the initial posture candidates (set I ₀ ). By reducing the number of candidates through this process, the calculation load of the subsequent estimation process can be reduced, and it becomes possible to narrow down the candidates to one with less contact. An example in which this exclusion is not performed is shown below.

When the force sensor 22 makes a first contact, detection information indicating the position of the contact point and the direction of the reaction force (the orientation of the surface including the contact point) is sent to the estimation unit 14. The estimation unit 14 applies these pieces of information to calculate a cost function G(r, t) for all elements of the set I ₀ = {(r, t) _i , i = 1, 2, ...}. Then, the estimation unit 14 extracts elements of the set I ₀ for which the cost function G(r, t) is equal to or less than a threshold close to zero as orientation candidates. Then, if the orientation candidates include only a series of elements that fall within a certain width corresponding to the threshold, the element that minimizes the cost function G from among these series of elements is extracted as the final estimation result of the orientation of the object. A series of elements means multiple elements that are consecutive with a minimum translation difference Δt or a minimum rotation difference Δr. On the other hand, if the orientation candidates are not only a series of elements, this means that multiple orientation candidates remain, so the estimation unit 14 continues the estimation process and waits for the next contact of the force sensor 22.

When the force sensor 22 makes the next contact, the estimation unit 14 performs the same process as the first contact (replacing the initial candidates with the orientation candidates narrowed down by the first contact) to further narrow down the candidates for the orientation of the object. Then, by repeating this process, the estimation unit 14 can obtain the single orientation of the object as the final estimation result.

<Conditions for contact route>
During the estimation process by the estimation unit 14, the movement control unit 13 calculates a path, which is a position and a direction from which the force sensor 22 (specifically, its contact) is moved to contact the object before the force sensor 22 contacts the object, as a contact path. The contact path is determined based on the following first to third contact conditions.

The first contact condition is a condition that, in the case of multiple contacts, the surface that is in contact with the previous contact is a different surface. Different surfaces of an object refer to each surface that is separated by a ridge or V-groove portion included in the object. If the object does not have a curved surface, each flat surface is a different surface. The contact path according to the first contact condition may be determined as a path that proceeds in a direction that intersects (e.g., perpendicular to) the normal of the contact point detected in the previous contact.

Alternatively, the contact path according to the first contact condition may be determined as a path that proceeds from a direction intersecting a previously uncontacted surface toward a location where the previously uncontacted surface is likely to be located, based on the candidate object postures currently estimated by the estimation unit 14.

The second contact condition is a condition that, when an object has an asymmetrical portion, there is a high probability of contacting the asymmetrical portion. An asymmetrical portion of an object means, for example, a portion that has a low degree of point symmetry, axial symmetry, or plane symmetry with respect to the center point, central axis, or central plane of the object. For example, in the case of a bolt, the head side edge and the shaft end portion correspond to the asymmetrical portion. Also, in the case of a pin member in which the diameter of a specified portion excluding the center in the longitudinal direction is reduced, the specified portion corresponds to the asymmetrical portion. Such asymmetrical portions may be calculated by the movement control unit 13 from the object information described above, or may be included in the object information in advance.

When contact with the object has not yet been made, the contact path according to the second contact condition can be found from the posture estimated by the estimation unit 14 based on the image captured by the imaging unit 21 and the information on the asymmetric surface described above.

When one or more contacts have been made, the contact path according to the second contact condition can be determined by calculating a location where an asymmetric part is likely to be located based on multiple candidate postures estimated at the current stage, and determining the path to that location.

The third contact condition is that, at the stage where one or more contacts have been made, contact is made with a surface of the object with a high degree of positional uncertainty among the multiple pose candidates obtained by estimation by the estimation unit 14.

The surface with a high degree of positional uncertainty may be obtained, for example, as follows, but is not limited to this. That is, first, the movement control unit 13 calculates the distance between the center point of the object estimated from the video and each point included in the point cloud M of the multiple posture candidates, and selects n representative points with long distances and dispersion. Then, the movement control unit 13 sets a cylinder of the same shape and volume with the normal of each of the selected n representative points as a central axis as a local region. Furthermore, the movement control unit 13 extracts all points included in each local region from all point clouds M included in the multiple posture candidates, and calculates the average value and variance value of the positions of these points in the axial direction (direction along the central axis of the cylinder). Then, the movement control unit 13 determines a local region with a large variance value from the n local regions as a place where the surface position has a high degree of uncertainty, and determines the surface included in the local region as a surface with a high degree of positional uncertainty.

In the above example, the n local regions are set based on the n representative points that are far from the center point of the object estimated from the image, but the method of setting the n local regions is not limited to the above example. For example, the n local regions may be set from n representative points selected randomly or n representative points that are distributed isotropically. In the above example, a cylindrical region is set as the local region, and the variance of the axial position of each point included in the local region is calculated, but the local region may have any shape, such as a spherical region, and the variance is not limited to the axial position variance, but may be from the center position of multiple points included in the local region, for example, and the direction of the variance is not limited. In the above example, the variance value of the points included in each local region is calculated, but various methods can be used to calculate the distance between two points farthest from each other in the axial direction in each local region, and the area with a large distance between the two points may be determined as the area including a surface with a high degree of uncertainty. Furthermore, in the above example, multiple local regions were set, and values indicating uncertainty were compared between the local regions to determine regions containing surfaces with a high degree of uncertainty. However, any method may be used as long as it can determine regions containing surfaces with a high degree of uncertainty, such as determining regions with a high degree of scattering of all point clouds M of multiple pose candidates as regions containing surfaces with a high degree of uncertainty without setting local regions.

The direction of the contact path according to the third contact condition may be calculated as a path that moves toward a location that includes a surface with a high degree of uncertainty determined as described above, in a direction that intersects with the surface included in that location. The direction may be the normal direction of a representative point included in the local area with a high degree of uncertainty, or may be a direction calculated from the normal directions of all points included in the local area (such as an average direction).

<How to determine contact route>
There may be many different contact paths that satisfy all of the first to third contact conditions. Alternatively, a contact path that satisfies one or two of the first to third contact conditions but does not satisfy the other contact conditions may be obtained. Therefore, the movement control unit 13 appropriately selects one contact path from the contact paths obtained based on the first to third contact conditions, and determines the selected contact path as the contact path along which the force sensor 22 will be moved next.

The method of selecting one contact route may be, for example, a method of assigning a priority to each contact route and selecting the contact route with the highest priority. The priority may be a different value depending on the first to third contact conditions, or may be a value obtained by multiplying this value by the possibility (probability) of meeting each contact condition. Alternatively, the method of selecting one contact route may be any method, such as random selection.

In the above example, the movement control unit 13 determines the contact path based on all of the first to third contact conditions, but the movement control unit 13 may omit one or two contact conditions and determine the contact path from the remaining two or one contact condition.

<Picking process>
Fig. 2 is a flowchart showing an example of the procedure of the picking process executed by the control unit 10. Picking means taking out an object. The picking process in Fig. 2 is started in a state where the object to be picked is included within the angle of view of the imaging unit 21.

When the picking process is started, first, the imaging control unit 12 acquires an image via the imaging unit 21 and sends the acquired image to the estimation unit 14 (step S1). The estimation unit 14 performs image recognition processing on the image to estimate the center position of the object, and sets the aforementioned initial region R0 (see FIG. 3(A)) that contains the object with a margin based on the center position and the object information (step S2).

Once the initial region R0 is set, the control unit 10 proceeds to a loop process (steps S3 to S13) in which candidates for the object's posture are narrowed down by contacting the object. That is, the movement control unit 13 determines whether or not it is the first contact (step S3), and if it is the first contact, calculates a contact path that matches the second contact condition (contact that is likely to contact an asymmetric part), and performs a process of assigning a priority to the contact path (step S6). The contact path is calculated as a path that proceeds from a direction intersecting the surface located at the part, based on the candidate object posture estimated at that time and the calculation result of the part having the asymmetric property of the object calculated from the object information. The priority assigned to the calculated contact path may be a value preset in the second contact condition to distinguish the priority of the first to third contact conditions. Alternatively, the priority may be a value obtained by multiplying a value preset in the second contact condition by a value (e.g., a probability) indicating the possibility that the contact path matches the contact condition.

On the other hand, if the determination process in step S3 determines that the contact is the second or subsequent contact, the movement control unit 13 calculates a contact path that matches the first contact condition (contact with a surface different from the surface that was previously contacted) based on the candidate posture of the object estimated at that time (step S4). In addition, the movement control unit 13 performs a process of assigning a priority to the contact path (step S4). If there are j posture candidates, the contact path is calculated as a path that extracts a surface that is estimated not to have been contacted in the previous contact in each candidate, and proceeds toward one or more points in the surface in the normal direction associated with the point. The direction of the contact path may be calculated as a direction that intersects with the normal of the surface that was previously contacted. The priority assigned to the contact path may be a value previously set for the first contact condition in order to distinguish the degree of wiredness of the first to third contact conditions. Alternatively, the priority may be a value obtained by multiplying a value preset for the first contact condition by a value (e.g., a probability) indicating the possibility that the contact path meets the contact condition.

Then, the movement control unit 13 calculates a contact path that matches the third contact condition (contact with a surface with a high degree of positional uncertainty) based on the candidate posture of the object estimated at that time, and further performs a process of assigning a priority to the contact path (step S5). The method of determining the contact path is as described above. The priority may be a value that is preset for the third contact condition in order to distinguish the priorities of the first to third contact conditions. Alternatively, the priority may be a value obtained by multiplying a value that is preset for the third contact condition by a value (e.g., a probability) indicating the possibility that the path matches the contact condition. After the calculation in step S5 is completed, the movement control unit 13 performs the process of calculating the contact path and assigning a priority in step S6 described above.

Then, the movement control unit 13 determines whether a contact route has been determined (step S7), and if YES, extracts the calculated contact route with the highest priority (step S8). During this extraction, the movement control unit 13 determines whether the calculated multiple contact routes can be aggregated as a route that simultaneously satisfies multiple contact conditions, and if so, may add up the priorities of the multiple contact routes and treat them as a single contact route to which the priority of the combined result has been added.

On the other hand, if the result of the determination in step S7 is NO, the movement control unit 13 calculates an arbitrary contact path that can contact the object (step S9). For example, the movement control unit 13 calculates a contact path from an arbitrary point on the spherical surface of the initial region R0 (FIG. 3(A)) toward the center.

Once the contact path has been determined, the movement control unit 13 moves the force sensor 22 along the contact path to bring the force sensor 22 (its contact) into contact with the object (step S10). The contact is weak and does not move the object. The estimation unit 14 then obtains information on the position of the contact point and the orientation of the surface from the force sensor 22 (step S11) and executes an estimation process for the posture of the object (step S12). The estimation process is performed using the cost function G(r, t) as described above.

When an estimation result of the object's posture is obtained, the estimation unit 14 determines whether the postures have been narrowed down to one (step S13), and if NO, returns the process to step S3. On the other hand, if the postures have been narrowed down to one, the control unit 10 exits the posture estimation loop process (steps S3 to S13) and proceeds to the next process.

After exiting the loop process, the robot control unit 15 calculates the position and orientation of the object, and the operation procedure for picking up the object with the robot arm 24, based on the estimated object posture (parameters {r, t} indicating rotation and translation) (step S14). Then, the robot control unit 15 drives the robot arm 24 according to the calculated operation procedure and picks up the object (step S15). Then, one picking process is completed.

<First example of posture estimation process>
Next, a specific example of the estimation process of the posture of an object executed during the above-mentioned picking process will be described. Fig. 3 is an explanatory diagram of the first step (A) to the fourth step (D) and (E) explaining a first example of the estimation process. In the first example, the object Q1 whose posture is to be estimated is a nut placed on a table.

3A shows an initial stage where the force sensor 22 has not made contact with the object Q1 even once. At this stage, in step S2, the estimation unit 14 sets the initial region R0 based on the image, and in step S6, the movement control unit 13 determines the surface of the outer surface of the object Q1 that has asymmetrical properties based on the object Q1 information, and calculates the contact path h1. In the case of a nut, the degree of rotational symmetry around the central axis of the screw hole is below a threshold, and the outer peripheral surface of the nut is calculated to be the surface that has asymmetrical properties. On the other hand, in the first example, the estimation accuracy of the posture based on the image is low, so that the path to contact the asymmetrical surface cannot be calculated, and the movement control unit 13 calculates an arbitrary contact path h1 in step S9. The arbitrarily selected contact path h1 may be a path from a point on the outer peripheral surface of the spherical initial region R0 to the center of the initial region R0.

Figure 3 (B) shows the stage where the posture of object Q1 is estimated based on the first contact after the force sensor 22 moves along the contact path h1 in steps S10 to S12. The position of contact point p1 and the orientation (normal k1) of the surface s1 including contact point p1 are detected by this contact, and the estimation unit 14 narrows down the candidates of translation t and rotation r that make the cost function G(r, t) near zero from this detection information. In Figure 3 (B), the range including the narrowed down posture candidates is shown by a two-dot chain line. The two-dot chain line range includes all postures in which contact point p1 corresponds to a point on any surface of object Q1 and the surface including this point is perpendicular to the normal k1. The estimation unit 14 finds many postures that occupy this range as candidates. Note that candidates in which object Q1 protrudes from the initial region R0 may be excluded from the candidates of the posture of object Q1, but Figure 3 shows an example in which they are not excluded.

Figure 3 (C) shows the stage where the contact path h2 of the second contact is calculated in steps S4 to S8. At this stage, the movement control unit 13 calculates the contact path according to the first contact condition, the contact path according to the second contact condition, and the contact path according to the third contact condition based on the candidate posture of the object Q1 obtained by the previous contact (shown by the two-dot chain line), and extracts the contact path h2 with the highest priority among them. Since the contact path h2 is a path that proceeds in a direction that intersects with the normal line of the first contact surface s1, it is highly likely to contact a surface different from the first contact surface s1, and further, the possibility that the contact surface is the outer peripheral surface of a nut having asymmetrical properties is higher than, for example, another contact path c1. Furthermore, in the candidate posture of the object Q1, the surfaces that may intersect with the contact path h2 are distributed in a wide range X1, while the surfaces that may intersect with another contact path c1 are distributed in a narrow range X2. Therefore, the contact path h2 is calculated as a path that is highly likely to contact a surface with a high degree of positional uncertainty.

Figures 3(D) and (E) show the stage where the posture of object Q1 is estimated based on the second contact after the force sensor 22 moves along the contact path h2 in steps S10 to S12. The second contact detects the position of contact point p2 and the orientation (normal k2) of surface s2 including contact point p2, and the estimation unit 14 narrows down candidates for translation t and rotation r that make the cost function G(r, t) close to zero from the detection information obtained in the first and second contacts. As shown by the two-dot chain lines in the plan view of Figure 3(E), the second contact narrows down multiple posture candidates (shown by the two-dot chain lines in Figure 3(E)) in which contact points p1 and p2 are included in the two surfaces s1 and s2 of object Q1 and the nut shifts in a direction parallel to both surfaces s1 and s2.

In this way, as the number of contacts increases, the number of candidates for the posture of object Q1 is narrowed down, and the estimation unit 14 can ultimately estimate a single posture of object Q1.

<Second Example of Posture Estimation Processing>
4 is an explanatory diagram of the first step (A) to the sixth step (F) illustrating a second example of the object posture estimation process. In the second example, an object Q2 whose posture is to be estimated is a bolt.

In FIG. 4(A), in the initial stage before the force sensor 22 has made contact with the object Q2 even once, the estimation unit 14 sets the initial region R0 based on the image (step S2), and the movement control unit 13 determines the asymmetric surface of the outer surface of the object Q2 based on the object Q2 information, and calculates the contact path h11 to comply with the second contact condition (step S6). In the case of a bolt, the degree of symmetry of the plane symmetry based on the central plane that divides the bolt into two in the axial direction is below the threshold, and the side surface s21 of the bolt head and the side surface s11a of the bolt shank end are determined as the surfaces that have asymmetricity. On the other hand, in the example of FIG. 4(A), the initial estimation accuracy of the posture of the object Q2 based on the image is low, so the contact path h11 is calculated to be toward the side surface at the center of the shank that is outside the asymmetric surface.

In FIG. 4B, the force sensor 22 moves along the contact path h11 (step S10), and the first contact is detected (step S11), and the posture of the object Q2 is estimated based on the detection (step S12). The contact detects the contact point p11 and the orientation (normal k11) of the surface s11 including the contact point p11, and the estimation unit 14 calculates the cost function G(r, t) from the detection information to search for multiple posture candidates of the object Q2. Here, the narrowed-down posture candidates include all postures in which the surface s11 and the contact point p11 are on the head side surface, head top surface, shaft end surface, or shaft side surface of the bolt, as shown by the two-dot chain line in FIG. 4B. It should be noted that candidates in which the object Q2 protrudes from the initial region R0 may be excluded from the posture candidates of the object Q2, but FIG. 4 shows an example in which they are not excluded.

In FIG. 4(C), the movement control unit 13 calculates a high-priority contact path h12 based on the first to third contact conditions (steps S4 to S8). The contact path h12 is a path that achieves contact with a surface different from the surface s11 where the first contact occurred, and achieves contact with a surface with a high degree of positional uncertainty. For example, among the candidates for the posture of the object Q2 (shown by the two-dot chain line), the surfaces that may intersect with the contact path h12 are dispersed over a distance greater than the length of the bolt shaft, while the surfaces that may intersect with another contact path c12 are dispersed by the width of the bolt head, with the former having a larger variance and a higher degree of uncertainty.

Then, as shown in FIG. 4(D), the second contact is made (step S10), the contact point p12 and the orientation (normal k12) of the surface s12 including the contact point p12 are detected (step S11), and based on these detection results, the estimation unit 14 narrows down the candidates for the posture of the object Q2 further (step S12). As shown by the two-dot chain line in FIG. 4(D), the second contact narrows down the candidates to two postures in which the head and shank end of the bolt are reversed. Note that if the first contact is made on the head side surface s21 or shank end side surface s11a of the bolt, which has asymmetry, the estimation unit 14 can narrow down the posture of the object Q2 to one at the stage of the second contact.

In FIG. 4(E), the movement control unit 13 calculates a contact path h13 that is likely to realize the first to third contact conditions (steps S4 to S8). In the candidates for the posture at this point, the head side surface s21 or the shank end side surface s11a of the bolt are surfaces with a high degree of positional uncertainty, and the head side surface s21 and the shank end side surface s11a of the bolt are surfaces at a location where they have asymmetry. Therefore, the contact path h13 is a path that complies with the second contact condition and the third contact condition. Furthermore, the contact path h13 is a path that may contact a surface (bolt head side surface s21) different from the surface s11 that was contacted the first time and the surface s12 that was contacted the second time, and is also a path that complies with the first contact condition. Contact path h13 actually contacts the shaft end side surface s11a, which is the same surface as the surface s11 that made contact the first time, but for the orientation candidates estimated at this stage, there is about a 50% chance of contacting the bolt head side surface s21.

Then, as shown in FIG. 4(F), a third contact is made (step S10), and the orientation of contact point p13 and surface s11a including contact point p13 are detected (step S11), and from these detection results, the number of candidates for the orientation of object Q2 is narrowed down to one (step S12).

As described above, according to the picking device 1 of this embodiment, the estimation unit 14 estimates the posture of an object based on the detection information obtained by the force sensor 22 contacting the object multiple times and the object information. Therefore, it is possible to estimate the posture of an object even if it is difficult to estimate the posture of an object from video alone, such as an object covered with cloth, a transparent or black object, or an object with a mirror surface.

Furthermore, according to the picking device 1 of this embodiment, the movement control unit 13 calculates the contact path of the force sensor 22 so that the multiple contacts of the force sensor 22 with the object are made on different surfaces of the object. Therefore, the estimation unit 14 can narrow down the candidates for the object's posture to a smaller number of candidates using detection information of fewer contacts, thereby achieving highly efficient posture estimation.

Furthermore, according to the picking device 1 of this embodiment, information on the position of the contact point and information on the orientation of the surface including the contact point are acquired as detection information when the force sensor 22 makes contact. In addition, the movement control unit 13 calculates a contact path that moves the force sensor 22 in a direction that intersects with the normal to the surface of the contact point that was contacted prior to the nth contact, at the time of the nth contact. By using this method of determining the direction of the contact path, the movement control unit 13 can calculate the contact path of the force sensor 22 so that multiple contacts of the force sensor 22 with an object are made on different surfaces of the object.

Furthermore, according to the picking device 1 of this embodiment, the movement control unit 13 calculates a contact path (a contact path according to the third contact condition) that is likely to contact a surface with a high degree of positional uncertainty from among multiple candidates for the object's posture that have been narrowed down at that time. Therefore, based on the detection information obtained by the movement of the force sensor 22 along that contact path, the estimation unit 14 can narrow down the candidates for the object's posture to fewer candidates. Therefore, the estimation unit 14 can narrow down the object's posture to one using detection information from few contacts, and can achieve highly efficient posture estimation.

Furthermore, according to the picking device 1 of this embodiment, the movement control unit 13 calculates a contact path (a contact path according to the second contact condition) so as to contact a surface of an asymmetric part of the object. Therefore, the orientation of the object can be efficiently identified based on the detection information of the asymmetric surface of the object obtained by moving the force sensor 22 along the contact path. Therefore, the estimation unit 14 can narrow down the orientation of the object to one using detection information of a small number of contacts, and can realize highly efficient orientation estimation.

Furthermore, the picking device 1 of this embodiment is equipped with a photographing unit 21 that captures an image of the object, and the estimation unit 14 estimates the rough position of the object based on the image captured by the photographing unit 21, making it possible to make the first contact. That is, the estimation unit 14 estimates the position at which the object can be contacted based on the image. Therefore, even when the object's position is unknown over a wide range, it is possible to estimate the position at which to contact the object from the image, and it is possible to estimate the object's posture by contacting it. Therefore, it is possible to pick up the object even when the object's position is unknown over a wide range.

The above describes the embodiments of the present invention. However, the present invention is not limited to the above embodiments. For example, in the above embodiment, a method for estimating the posture of an object using information on a point group M uniformly distributed on the outer surface of the object and a cost function G(r, t) was described. However, the present invention is not limited to the above method, and any method may be adopted as long as it can estimate the posture of an object from information detected by contact. Also, as described above, in the above embodiment, an example of performing posture estimation processing without considering the presence of an element that restricts the posture of the object, such as a table on which the object is placed, was shown. However, if there is an element that restricts the posture of the object, for example, a process of excluding candidates for postures that are incompatible with the element may be added.

In the above embodiment, an example was shown in which the object information storage unit 11 (a storage unit that stores object information indicating the shape and size of an object) is located within the device. However, the object information storage unit 11 may be provided in a server computer connected via a communication network, and object information may be sent to the device via communication. In the above embodiment, an example was shown in which the present invention is applied to a picking device 1 that picks objects from shelves, etc., but the information processing device of the present invention does not need to involve control and operation to pick up objects, and can be applied to various devices that estimate the posture of an object. Other details shown in the embodiment can be changed as appropriate without departing from the spirit of the invention.

1. Picking device (information processing device)
REFERENCE SIGNS LIST 10 Control unit 11 Object information storage unit 12 Photography control unit 13 Movement control unit 14 Estimation unit 15 Robot control unit 16 Interface 21 Photography unit 22 Force sensor 23 Driving device 24 Robot arm (removal mechanism)
Q1, Q2 Object R0 Initial region h1, h2, h11-h13 Contact path p1, p2, p11-p13 Contact points s11a, s21 Surfaces with asymmetry

Claims (5)

An information processing device that estimates a posture of an object,
A storage unit that stores object information indicating a shape and a size of the object;
a force sensor for acquiring detection information of a contact point by contact;
A movement control unit that moves the force sensor;
an estimation unit that estimates a posture of the object based on the detection information obtained by the force sensor contacting the object multiple times and the object information;
Equipped with
The estimation unit estimates a plurality of candidates for a pose of the object before the plurality of contacts are performed,
the movement control unit moves the force sensor so that, during the multiple contacts, the force sensor comes into contact with different faces of the object, and further so that the force sensor comes into contact with a face included in the multiple candidates having a higher degree of positional uncertainty than other faces included in the multiple candidates .
Information processing device. the detection information includes information on a position of the contact point and a normal of a surface including the contact point;
the movement control unit moves the force sensor in a direction intersecting a normal to a plane including a contact point that was contacted before the nth contact, when an nth contact occurs among the multiple contacts.
2. The information processing device according to claim 1. A photographing unit is provided for capturing an image of an object,
The estimation unit estimates a contactable position of the object based on the image.
3. The information processing device according to claim 1 or 2 . An information processing device according to any one of claims 1 to 3 ;
a take-out mechanism that takes out the object using the estimation result of the estimation unit;
A picking device comprising: An information processing device including a storage unit that stores object information indicating the shape and size of an object,
A means for moving the force sensor;
a means for acquiring detection information of a contact point from the force sensor;
a means for estimating a posture of the object based on the detection information obtained by the force sensor contacting the object multiple times and the object information;
A program for executing
the estimating means estimates a plurality of candidates for the pose of the object before the plurality of contacts are performed;
the moving means moves the force sensor so that the force sensor comes into contact with different faces of the object during the multiple contacts, and further so that the force sensor comes into contact with a face included in the multiple candidates having a higher degree of positional uncertainty than other faces included in the multiple candidates.
Program.

Images