JP2016099257A - Information processing device and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To measure the position attitude of a target object with high accuracy by taking the influence of a shade (occlusion) generated by a lighting device or the like into consideration.SOLUTION: An information processing device includes: sensor means for acquiring two-dimensional information or three-dimensional information about a target object in a first position attitude; calculation means for respectively calculating hidden areas where the target object is shielded when the target object is looked at from a plurality of view points on the basis of the two-dimensional information or the three-dimensional information acquired by the sensor means in the first position attitude; and determination means for determining a second position attitude different from the first position attitude as a position attitude of the sensor means which can acquire three-dimensional information for measuring the position attitude of the target object on the basis of the hidden areas calculated by the calculation means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus and an information processing method for obtaining a three-dimensional position and orientation of a target object.

  Along with the development of robot technology in recent years, robots are instead performing complicated tasks that have been performed by humans, such as assembly of industrial products. Such a robot performs assembly by picking parts with an end effector such as a hand. Conventionally, parts for picking are supplied by using a device called a parts feeder for supplying parts one by one, or by stacking parts in various positions on a pallet (box). It has been broken.

When using the parts feeder, since the position and orientation of each part is supplied in a predetermined state, picking by the robot is relatively easy. However, the cost for preparing the parts feeder device is excessive. Also, different parts feeders must be prepared according to the shape of the parts. On the other hand, when supplying parts in piles, it is only necessary to place the parts on a pallet, so that an increase in cost can be avoided. Furthermore, in response to the recent trend of low-volume, high-mix production, attention has been focused on the supply of piles that can quickly respond to various parts.
Under these trends, picking work systems are being developed by combining two-dimensional cameras and three-dimensional measuring devices with component recognition.

  As a recognition device for recognizing the position and orientation of a target object, there is a technique described in Patent Document 1. Here, first, the entire plurality of target objects are photographed by a camera installed above the pallet, and a two-dimensional position on the photographed image is obtained. Next, a straight line from the camera to one target object is obtained, and the robot is moved so that the sensor mounted on the hand of the robot is on the straight line. Then, the position and orientation of the target object are measured by the sensor. In this way, the target object can be measured with high accuracy and high speed by combining the camera that shoots the entire image and the sensor that can detect the position and orientation with a narrow measurement range but with high accuracy. ing.

Japanese Patent No. 3556589

In the recognition apparatus described in Patent Document 1, three-dimensional measurement is always performed by a sensor mounted on the hand portion of the robot on a straight line connecting the camera and the target object installed above the pallet. Therefore, depending on the direction in which the target object is facing, it is not always easy to measure. For example, when measuring illumination light is applied to a target object, a shadow generated by the illumination light is reflected on the camera depending on the direction in which the target object is directed. In addition to the above-mentioned shadow, there may occur a state in which an obstacle in the foreground hides a target object behind and prevents it from being seen. When such occlusion occurs, it becomes a hindrance when performing image processing, and thus it becomes difficult to measure the position and orientation.
Therefore, an object of the present invention is to measure the position and orientation of a target object with high accuracy in consideration of the influence of the occlusion even in a situation where occlusion may occur.

  In order to solve the above-described problem, an aspect of the information processing apparatus according to the present invention includes a sensor unit that acquires two-dimensional information or three-dimensional information about a target object in a first position and orientation, and the first position and orientation. Calculating means for calculating a hidden area where the target object is shielded when the target object is viewed from a plurality of viewpoints based on the two-dimensional information or the three-dimensional information acquired by the sensor means; A second position and orientation different from the first position and orientation as the position and orientation of the sensor means capable of acquiring three-dimensional information for measuring the position and orientation of the target object based on the hidden area calculated by the calculating means Determining means for determining.

  According to the present invention, it is possible to measure the position and orientation of a target object with high accuracy in consideration of the influence of the occlusion even in a situation where occlusion may occur.

It is a lineblock diagram showing an example of a robot system provided with an information processor of a first embodiment. It is a figure which shows the structural example of a sensor part. It is an example of the pattern light projected from a projector. It is a figure explaining a three-dimensional shape model (vertex). It is a figure explaining a three-dimensional shape model (vertex). It is a figure explaining a three-dimensional shape model (side). It is a figure explaining a three-dimensional shape model (side). It is a figure explaining a three-dimensional shape model (surface). It is a figure explaining a three-dimensional shape model (surface). It is a figure which shows an example of the generation | occurrence | production state of occlusion. It is a flowchart which shows an example of the three-dimensional position measurement process procedure performed with an image process part. It is a flowchart which shows an example of the 2nd position and orientation determination processing procedure. It is a figure explaining the setting method of a candidate position and orientation. It is a figure explaining the setting method of a candidate position and orientation. It is a figure which shows the imaging | photography state in a 1st measurement position. It is a figure which shows the imaging | photography state in a 2nd measurement position. It is a figure which shows the holding state of the target object in 2nd embodiment. It is a figure which shows the hidden area | region in the characteristic part of a target object. It is a block diagram which shows an example of the robot system provided with the information processing apparatus in 3rd embodiment. It is a block diagram which shows an example of the robot system provided with the information processing apparatus in 4th embodiment. It is a figure which shows the hardware structural example of information processing apparatus.

DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings.
The embodiment described below is an example as means for realizing the present invention, and should be appropriately modified or changed depending on the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment.

(First embodiment)
FIG. 1 is a diagram illustrating a configuration example of a robot system 100 including an information processing apparatus according to the first embodiment.
The robot system 100 includes a robot 10 and an information processing unit 20.
The robot 10 is, for example, an articulated robot, and a sensor unit 11 is attached to the tip portion (hand portion) of the arm so that the posture can be changed. The sensor unit 11 captures an object near the tip of the robot 10 and outputs image data to the information processing unit 20.

  The robot 10 also includes a position and orientation change mechanism 12 that can change the position and orientation of the sensor unit 11, that is, the position of the hand portion of the robot 10 and the orientation of the sensor unit 11. The position and orientation change mechanism 12 changes the position of the hand portion of the robot 10 and the orientation of the sensor unit 11 by changing the angle of each joint of the robot 10. Here, the position / orientation changing mechanism 12 may be driven by an electric motor, or may be driven by an actuator that operates with fluid pressure such as hydraulic pressure or air pressure. The position / orientation changing mechanism 12 is driven in response to a control command from a robot controller 220 (described later) included in the information processing unit 20.

Further, an end effector 13 is attached to the hand portion of the robot 10. The end effector 13 is a tool for realizing work according to the type of work of the robot 10, and is, for example, a robot hand. As the robot hand, a hand that has a chuck mechanism that can be driven by a motor and can grip an object, a hand that uses a suction pad that sucks an object by air pressure, and the like can be used. The end effector 13 is detachably attached to the arm and can be exchanged according to the type of workpiece.
Note that the calibration work of the position and orientation of the sensor unit 11, the position and trajectory of the robot 10 and the hand 13, and the relative position and orientation of the arm of the robot 10 and the sensor unit 11 is performed in advance by a predetermined method. .

The robot 10 is not limited to an articulated robot, and may be a movable machine capable of numerical control (NC).
The information processing apparatus 20 inputs a two-dimensional image of the measurement object 30 photographed by the sensor unit 11, executes predetermined image processing based on the image data, and moves the position / orientation changing mechanism 12 and the hand 13. Drive control.
The measurement target object 30 is obtained by stacking a plurality of target objects 31 on a pallet 32 in various postures. The target object 31 is, for example, a part that constitutes an industrial product. The material of the target object 31 may be any material such as plastic, metal, or vinyl.

The pallet 32 is a box that accommodates the target object 31, and there are no particular restrictions on the material and shape. However, from the viewpoint of ease of production, for example, the material is preferably plastic or paper, and the shape is preferably a cube or a rectangular parallelepiped. Further, the size of the pallet 32 is not limited, but it is preferable that the size of the pallet 32 be within a range where the sensor unit 11 can shoot.
The information processing apparatus 20 measures the three-dimensional position and orientation of the target object 31 based on the two-dimensional image photographed by the sensor unit 11 and drives and controls the robot 10 based on the measurement result. Picking operation of the target object 31 is performed.

Next, the configuration of the sensor unit 11 will be specifically described.
As shown in FIG. 2, the sensor unit 11 includes a projector (projection unit) 111 and a camera (imaging unit) 112, and uses three-dimensional information (distance images and three-dimensional points) about the target object 31 using the principle of triangulation. Group data).
The projector 111 projects pattern light onto the target object 31, and the camera 112 captures a two-dimensional image of the target object 31 on which the pattern light is projected. The sensor unit 11 outputs image data captured by the camera 112 to the information processing unit 20. As the pattern light, it is possible to use a plurality of striped patterns, a plurality of line patterns, and the like having different widths in the spatial encoding method.

FIG. 3 is an example of pattern light projected from the projector 111.
Pattern lights 201 to 203 shown in FIG. 3 are pattern lights having a stripe pattern in which bright and dark slits are alternately arranged. Here, a stripe pattern using a 3-bit gray code is shown, but it is not limited to 3 bits. When performing Nbit spatial encoding, the number of slit light patterns is N.
Here, the light pattern 202 is stripe pattern light having a spatial resolution twice that of the light pattern 201. Similarly, the light pattern 203 is a stripe pattern light whose spatial resolution is twice that of the light pattern 202. The information processing unit 20 recognizes the target object 131 by dividing the imaging range into eight parts based on the image data input from the sensor unit 11.
The pattern light may be a two-dimensional pattern or a pattern such as a random dot. Moreover, the sensor part 11 may be comprised from the diffraction grating, illumination, and the camera. In this case, pattern light may be projected onto the target object using the diffraction grating and illumination, and the pattern may be captured by the camera.

  The projector 111 and the camera 112 are arranged such that three-dimensional measurement by an active stereo method or the like is possible. That is, in order to enable three-dimensional measurement by the active stereo method or the like, it is necessary to take a parallax in a direction perpendicular to the direction of the measurement line of the pattern light for measurement. Therefore, the position of the projector 111 and the camera 112 always shifts in the line of sight due to the influence of parallax. When the target object 31 enters the area where the field of view of the camera 112 and the projection range of the projector 111 overlap, three-dimensional measurement is possible.

Next, the configuration of the information processing unit 20 will be specifically described.
The information processing unit 20 is mounted on, for example, a personal computer (PC), and includes an image processing unit 210 and a robot controller unit 210 as shown in FIG.
The image processing unit 210 acquires the three-dimensional information of the target object 31 acquired by the sensor unit 11 having the first position and orientation, and measures the position and orientation of the target object 31 based on the three-dimensional information. Next, the image processing unit 210 determines the second position and orientation of the sensor unit 11 based on the position and orientation information of the target object 31, and the robot controller unit 220 moves the sensor unit 11 to the second position and orientation. . The image processing unit 210 acquires the three-dimensional information of the target object 31 acquired by the sensor unit 11 having the second position and orientation, and measures the position and orientation of the target object 31 based on the three-dimensional information. Thereafter, the second position and orientation may be reset as the first position and orientation until the predetermined condition is satisfied, and the above processing may be repeated.

  Thus, in the robot system 100, the three-dimensional position / orientation measurement of the target object 31 is performed a plurality of times. At this time, the image processing unit 210 sets the second position / posture of the sensor unit 11 to a position / posture at which the three-dimensional position / posture measurement of the target object 31 can be efficiently performed. Thereby, every time the number of times of acquiring the three-dimensional information is repeated, a detailed three-dimensional position and orientation measurement result for the target object 31 is obtained.

Hereinafter, each part which comprises the image process part 210 and the robot controller part 210 is demonstrated in detail.
The image processing unit 210 includes a sensor information acquisition unit 211, a model information holding unit 212, a three-dimensional position / orientation measurement unit 213, an occlusion estimation unit 214, and a sensor position / orientation determination unit 215. The robot controller unit 210 includes a robot work instruction unit 221 and a robot control unit 222.
The sensor information acquisition unit 211 acquires the two-dimensional image output from the sensor unit 11 and outputs the acquired two-dimensional image to the three-dimensional position / orientation measurement unit 213.
The model information holding unit 212 holds model information used for measuring the position and orientation of the target object 31 by a three-dimensional position and orientation measurement unit 213 described later. The model information includes a three-dimensional geometric model based on three-dimensional CAD, a reference image model obtained by viewing an actual target object 31 or a three-dimensional geometric model imitating the target object 31 from a plurality of predetermined viewpoints, and the like. .

The three-dimensional geometric model is a CAD model itself that can be handled by three-dimensional CAD software, or a three-dimensional CAD model converted into a plurality of polygon elements used in the computer graphics field. In the present embodiment, a case where a polygon element is used will be described.
As shown in FIGS. 4 to 9, the three-dimensional geometric model includes components such as points, lines, and surfaces. Here, FIGS. 4 to 9 show the same three-dimensional geometric model. 4 and 5 show the vertices of the 3D geometric model, FIGS. 6 and 7 show the lines that form the sides of the 3D geometric model, and FIGS. 8 and 9 show the faces of the 3D geometric model. . As shown in FIG. 9, the three-dimensional geometric model includes normal data of surfaces constituting the three-dimensional geometric model.

  The reference image model is data composed of a plurality of two-dimensional images. The reference image model based on the photographed image is created from images obtained by photographing with a camera from various directions around the target object 31. To shoot, you can assemble a tower and place multiple cameras, or you can shoot with a camera by hand, or you can shoot while moving the robot with a camera attached to the robot. . As described above, the photographing may be performed by various methods. However, the relative position and orientation between the camera and the target object 31 at the time of photographing are obtained and stored together with the photographed image. The relative position and orientation can be obtained from the shape of the tower when a plurality of cameras are arranged in the tower. Further, when a person has a camera by hand, it can be obtained by attaching a position and orientation sensor to the camera. Furthermore, when shooting with a camera attached to the robot, it can be obtained using the control information of the robot.

  The reference image model based on the three-dimensional geometric model imitating the target object 31 sets a geodesic dome (Geodesic Dome) whose vertices are at equal distances from the center of the CAD model, and the center of the CAD model from the vertex of the geodesic dome. Use the image when looking at the camera. The geodesic dome has multiple vertices, and each adjacent vertex has the same distance. A certain vertex is used as a reference position, and from which direction the image is viewed based on a relative relationship with another vertex is stored together with the image. Here, an image when the center is observed from each vertex of the geodesic dome is used as a reference image model. Note that the reference image model may be a luminance image or a distance image.

If the target object 31 is known in advance as one type, only that type of model information is stored. On the other hand, when dealing with a plurality of types of target objects 31, a plurality of model information is stored and switched at the time of use.
The three-dimensional position / orientation measurement unit 213 uses the data obtained by analyzing the two-dimensional image output from the sensor information acquisition unit 211 and the model information held by the model information holding unit 212 to determine the position and posture of the target object 31. Ask for. At this time, the three-dimensional position / orientation measurement unit 213 determines the position / orientation information of the plurality of target objects 31 and the position of one target object 31 (measurement target object 311) to be gripped by the robot 10 among the plurality of target objects 31. Attitude information is calculated. Then, the three-dimensional position / orientation measurement unit 213 outputs the obtained position / orientation information to the occlusion estimation unit 213.

When a 3D geometric model is used as the model information, the surface point cloud data extracted from the 3D geometric model and the distance point cloud extracted from the 2D pattern image output from the sensor information acquisition unit 211 By performing the association, the position and orientation of the target object 31 are obtained.
In order to obtain the distance point group from the pattern image, a technique such as a spatial encoding method or a light cutting method may be used. In this embodiment, in order to obtain the position and orientation of the target object 31 using the obtained distance point group and model information, for example, the ICP (Iterative Closest Point) method is used in this embodiment, and the position and orientation of the target object 31 are obtained by iterative calculation. Correct it repeatedly. Note that the method for obtaining the position and orientation of the target object 31 is not limited to the ICP method.

When a reference image model is used as model information, a reference image that most closely matches is obtained by template matching using the reference image model as a template, and the target is determined based on position and orientation information with the target object 31 associated with the reference image. The position and orientation of the object 31 are obtained.
The occlusion estimation unit 214 uses the position / orientation information of the plurality of target objects 31 obtained by the three-dimensional position / orientation measurement unit 213, and the position of the sensor unit 11 for photographing the measurement target object 311 in the pallet 32 in more detail. The posture (second position / posture) is determined. At this time, a plurality of candidate position / postures are created, and the occurrence state of occlusion when photographing at each candidate position / posture is simulated. Based on the simulation result, the optimum second position and orientation are determined.

  Here, the occlusion means that when the distance measurement method using, for example, the stereo method is performed, the measurement target object is captured by the projector 111 based on the relationship between the parallax between the camera 112 and the projector 111 and the arrangement of the measurement target object. It means a state where it cannot be done. This is the case when the shadow of the pattern projection light projected from the projector 111 is applied to the measurement target object. In addition, an area that cannot be captured by the projector 111 due to the occurrence of occlusion (an area where the measurement target object is shielded) is referred to as a hidden area (occlusion area).

FIG. 10 is a diagram illustrating an example of an occurrence state of occlusion.
A case will be described in which the sensor unit 11 is arranged so that the projector 111 and the camera 112 have the position and orientation shown in FIG. 10, and the pattern light projected from the projector 111 toward the measurement target object 311 is captured by the camera 112. In this case, the pattern light projected from the projector 111 becomes a shadow by the obstacle 312 disposed in the vicinity of the measurement target object 311 and is not irradiated to the measurement target object 311. Therefore, the camera 112 cannot capture the pattern projection light of the projector 111. A shadow area 301 generated at this time is a hidden area.
Here, the obstacle 312 is, for example, the target object 31 other than the measurement target object 311. In FIG. 10, the obstacle 312 has the same shape as the measurement target object 311, but is arranged in a vertically standing state.

The amount of the hidden area 301 is observed as an occlusion amount 302. Since FIG. 10 is a plan view, the length of the straight line of the occlusion amount 302 corresponds to the amount of the hidden area 301. However, in practice, the occlusion amount 302 is observed as an area by a rendering method using a three-dimensional model, and thus the occlusion amount 302 is calculated from the total area amount.
Occlusion occurs depending on the position and orientation of the target object 31 and the position and orientation of the sensor unit 11. Therefore, by using the existing computer graphics technology, it is possible to predict the state of the captured image when the sensor unit 11 takes an arbitrary position and orientation.
Therefore, the occlusion estimation unit 214 performs rendering using the position and orientation information of the plurality of target objects 31, the position and orientation information of the pallet 32, the information of the light source (projector 111), and the position and orientation information of the sensor unit 11. As the rendering technique, a ray tracing method or the like can be used.

The occlusion estimation unit 214 calculates a hidden area where the measurement target object 311 is shielded when the measurement target object 311 is viewed from a plurality of viewpoints. That is, the occlusion estimation unit 214 sets a plurality of position and orientation candidates of the sensor unit 11 and estimates the occlusion amount at each position and orientation.
The sensor position / posture determination unit 215 determines the second position / posture of the sensor unit 11 based on the occlusion amount at each candidate position / posture estimated by the occlusion estimation unit 214. In the present embodiment, the candidate position / posture with the smallest occlusion amount is determined as the second position / posture. Then, the sensor position / orientation determination unit 215 outputs the determined position / orientation information of the sensor unit 11 to the robot work instruction unit 221 of the robot controller unit 220.

The robot work instruction unit 221 sets information (work instruction information) for instructing work on the robot 10 based on the measurement results of the three-dimensional position / orientation measurement unit 213 and the sensor position / orientation determination unit 215. For example, the robot work instruction unit 221 sets work instruction information for moving the sensor unit 11 to a predetermined position and work instruction information for setting the sensor unit 11 to a predetermined posture. In addition, the robot work instruction unit 221 sets work instruction information for moving the hand 13 to a position where the target object 31 can be gripped or sucked and executing the gripping or sucking work. Needless to say, the operation of the robot 10 is not limited to movement, gripping, and suction, and includes other operations such as an appearance inspection of the target object 31.
The robot control unit 222 receives the work instruction information set by the robot work instruction unit 221 and controls the robot 10.

Next, an information processing procedure executed by the information processing unit 20 will be described in detail.
FIG. 11 is a flowchart illustrating an information processing procedure executed by the information processing unit 20.
First, in step S1, the information processing unit 20 controls the driving of the robot 10 by the robot control unit 222, and moves the sensor unit 11 to a preset first position and orientation. Here, the first position / posture is a position / posture in which the stacked target objects 31 can be photographed from above the pallet 32 and the entire pallet 32 can be photographed. It is assumed that the relative positional relationship between the position and orientation of the sensor unit 11 and the coordinate system of the robot 10 is obtained in advance by performing a calibration operation.

Next, in step S2, the information processing unit 20 acquires, by the sensor information acquisition unit 211, a two-dimensional image in which the camera 112 captures the target object 31 at the first position and orientation, and proceeds to step S3.
In step S3, the information processing unit 20 measures the three-dimensional position and orientation of the target object 31 with the three-dimensional position and orientation measurement unit 213 based on the two-dimensional image acquired in step S2. Then, the information processing unit 20 selects one of the plurality of target objects 31 as a gripping candidate for picking work. The criterion to be selected can be determined from an estimated value of an error when fitting a CAD model of a part.

Hereinafter, the fitting algorithm will be described.
In order to obtain the position and orientation of the target object 31, the model information held by the model information holding unit 212 is referred to, and the image of the target object 31 and the model information are matched.
When a 3D geometric model is used as model information, as described above, the correspondence between the point group on the surface of the 3D geometric model and the distance image point group obtained from the pattern image output from the sensor information acquisition unit 211 The position and orientation of the target object are obtained by performing the pasting. In order to obtain the distance point group from the pattern image, a technique such as a spatial encoding method or a light cutting method may be used. Here, the ICP method is used in order to associate the point group of the three-dimensional geometric model with the distance image point group.

First, let P be the surface point group of the three-dimensional geometric model.
P = {p _1, p _2, ... _, P _Np } (1)
Also, let the distance image point group be A.
A = {a _1, a _2, ... _{, A Na} } (2)
The surface point group P of the three-dimensional geometric model is converted and aligned with the distance point group A. When a point of point group A that is closest to each point P _i of point group P is b _i εA, an error function of the following equation (3) can be defined. Here, R and t are a posture parameter and a movement vector, respectively.

R and t for reducing the error function E are obtained, and correction is performed by the following equation (4).
P ← RP + t (4)
The convergence judgment of P is performed. If it has converged, the process ends. If not, the correction calculation is repeated. In the convergence determination, it is determined that P has converged when P hardly changes. The position and orientation can be calculated by repeating until convergence.
When a reference image model is used as model information, a reference image that most closely matches is obtained by template matching using the reference image model as a template, and the target object is based on position and orientation information with the target object associated with the reference image. Find the position and orientation of When the distance value of the reference image is G (i, j), the distance value of the distance image obtained from the pattern image is D (i, j), and the reference image is an image of m × n pixels, the matching degree U is as follows: (5) It can obtain | require by Formula.

As described above, the position and orientation of the plurality of target objects 31 and the position and orientation of the target object 31 that is a picking candidate are obtained.
Next, in step S4, the information processing unit 20 determines whether or not the fitting error for the target object 31 that is a picking candidate is equal to or less than a predetermined value set in advance. Here, it is determined whether or not the degree of matching U obtained by the above equation (5) is equal to or less than a threshold value.
If the degree of coincidence U exceeds the threshold, the amount of fitting error is large, so it is determined that re-photographing for three-dimensional measurement of the target object 31 is necessary, and the process proceeds to step S5. On the other hand, if the degree of coincidence U is less than or equal to the threshold, it is determined that the fitting has succeeded without any problem and it is not necessary to re-photograph the target object 31, and the process proceeds to step S12 described later.
In step S5, the information processing unit 20 is the second position that is the position of the sensor unit 11 (camera 112) for photographing the target object 31 in more detail based on the three-dimensional measurement result at the first position and orientation. Determine the measurement position.

  For example, a position obtained by rotating the sensor unit 11 by a predetermined angle from the first position and orientation around the target object 31 in a plane including the optical axis of the camera 112 and the optical axis of the projector 111 is set as the second measurement position. And For example, a position where overexposure does not occur in the target object 31 in the image captured by the camera 112 may be determined as the second measurement position. Further, the bisection direction of the angle formed by the optical axis direction of the camera 112 and the optical axis direction of the projector 111 and the normal direction of the surface of the target object 31 are shifted by a predetermined angle or more (for example, 10 degrees or more). The position may be determined as the second measurement position. Furthermore, the position and orientation of the sensor unit 11 suitable for measurement in advance may be registered in the three-dimensional geometric model or the reference image model, and this may be determined as the second measurement position.

Next, in step S <b> 6, the information processing unit 20 determines the second position and orientation, which is the orientation of the sensor unit 11 at the second measurement position, by performing the second position and orientation determination process shown in FIG. 12. Here, the second position / posture is a posture in which the total amount of occlusion is the minimum at the second measurement position. Hereinafter, the second position and orientation determination process will be specifically described.
As shown in FIG. 12, first, in step S61, the information processing unit 20 acquires the position and orientation information of the plurality of target objects 31 obtained in step S2 and the position and orientation information of the target objects 31 that are picking candidates. . In addition, the information processing unit 20 acquires model information held by the model information holding unit 212. Further, the information processing unit 20 acquires position and orientation information with respect to the pallet 32 of the sensor unit 11 and calibration information of the sensor unit 11.

  Next, in step S62, the information processing unit 20 creates a plurality of posture candidates that are candidates for the second position and posture. Here, as shown in FIG. 13, for example, a direction passing through the center of the lens of the camera 112 and orthogonal to the lens surface is defined as a shooting gaze direction Z, and the projector 111 is centered on the camera 112 with the shooting gaze direction Z as an axis. The position obtained by rotating is used as a posture candidate. More specifically, as shown in FIG. 14, positions obtained by rotating the projector 111 about the camera 112 by 2π / n [rad] are set as posture candidates. Here, n is an arbitrary value determined from the processing amount and accuracy. Thus, in step S62, n posture candidates are created.

Next, in step S63, the information processing unit 20 calculates the respective occlusion amounts for the n posture candidates created in step S62, using the above-described occlusion amount evaluation method.
Next, in step S64, the information processing unit 20 compares the occlusion amounts of the respective posture candidates, and selects the posture candidate having the smallest occlusion amount as the second position / posture.
Returning to FIG. 11, in step S7, the information processing unit 20 controls the driving of the robot 10 by the robot control unit 222, and moves the sensor unit 11 to the second position and orientation determined in step S6.

Next, in step S8, the information processing unit 20 acquires, by the sensor information acquisition unit 211, a two-dimensional image obtained by the camera 112 capturing the target object 31 at the second position and orientation, and proceeds to step S9.
In step S9, the information processing unit 20 determines the three-dimensional position / orientation of the target object 31 in the three-dimensional position / orientation measurement unit 213 based on the two-dimensional image acquired in step S8 by the same process as in step S3 described above. measure. Since the second position and orientation is an arrangement that considers the arrangement state of the surrounding obstacles, in this step S9, three-dimensional measurement with a small occlusion amount is possible. Therefore, the position and orientation of the picking candidate target object 31 can be obtained with relatively high accuracy.

Next, in step S10, the information processing unit 20 determines whether or not to repeat the three-dimensional position and orientation measurement. Here, for example, based on the measurement result of step S9, it is determined whether or not the fitting error for the target object 31 that is a picking candidate is equal to or smaller than a predetermined value. That is, it is determined whether or not the degree of coincidence U obtained by the equation (5) is equal to or less than a threshold value. When the fitting error exceeds a certain value, it is difficult to pick the target object 31, and it is necessary to repeat the three-dimensional position / orientation measurement of the target object 31 (repeated measurement is required). Determination is made, and the process proceeds to step S11.
In step S11, the information processing section 20 resets the second position and orientation determined in step S6 as the first position and orientation, and returns to step S5. That is, based on the measurement result at the second position and orientation determined in step S6, a new second position and orientation is determined, and three-dimensional position and orientation measurement is performed with the new second position and orientation.

On the other hand, if it is determined in step S10 that the fitting error is equal to or less than a certain value, it is determined that the target object 31 can be picked, and the process proceeds to step S12.
In step S10, the case where it is determined whether or not the fitting error is equal to or less than a certain value has been described as the end determination of the repeated measurement. For example, the occlusion amount (the size of the hidden area) in the second position and orientation is described. ) Is less than or equal to the threshold, or when the position and orientation change amount (movement amount of the sensor unit 11), which is the difference between the first position and orientation, is less than or equal to the threshold, You may judge.

  In step S <b> 12, the information processing unit 20 controls the driving of the robot 10 by the robot control unit 222 and instructs the target object 31 as a picking candidate. That is, when the robot 10 has the hand 13 that holds the target object 31, the robot control unit 222 instructs the robot 10 to hold the target object 31. When the robot 10 has the hand 13 that sucks the target object 31, the robot control unit 222 instructs the robot 10 to suck the target object 31.

Next, in step S <b> 13, the information processing unit 20 determines whether or not to finish the work by the robot 10. Here, for example, it is determined whether or not there is an end instruction from the operator. If there is no end instruction, the process returns to step S1, and if there is an end instruction, the process ends.
Note that the processing shown in FIG. 11 can be forcibly terminated by, for example, pressing an emergency stop button (not shown) by the operator.
In FIG. 1, the sensor unit 11 corresponds to an example of a sensor unit, and the position / orientation change mechanism 12 corresponds to an example of a moving unit. The three-dimensional position / orientation measurement unit 213 corresponds to an example of a measurement unit, the occlusion estimation unit 214 corresponds to an example of a calculation unit, and the sensor position / orientation determination unit 215 corresponds to an example of a determination unit. Further, the robot controller unit 220 corresponds to an example of a control unit. In FIG. 11, step S10 corresponds to an example of a repeating unit.

  As described above, in this embodiment, the three-dimensional position and orientation measurement of the target object 31 is performed a plurality of times using the sensor unit 11 including the projector (projection unit) 111 and the camera (imaging unit) 112. In this case, first, as shown in FIG. 15, three-dimensional information related to a plurality of target objects 31 is acquired at the first position and orientation P1, and a target object (measurement target object) that becomes a picking candidate among the plurality of target objects 31. 3D position and orientation measurement of 311 and its surrounding obstacle 312 is performed. Then, based on the three-dimensional position and orientation measurement result, the second measurement position is determined. The second measurement position is a position obtained by rotating the sensor unit 11 by a predetermined angle around a target object 311 within a plane including the optical axis of the projector 111 and the optical axis of the camera 112. For example, as shown in FIG. Assume that the position is determined.

However, in the state illustrated in FIG. 10, when the target object 311 is observed from the projector 111, the target object 311 is hidden behind the obstacle 312. That is, the pattern projection light of the projector 111 is shielded by the obstacle 312 and is not irradiated to the target object 311. Thus, occlusion occurs in the position and orientation shown in FIG. 10, and the camera 112 cannot capture the pattern projection light of the projector 111.
As described above, when unexpected occlusion occurs due to the pattern illumination light of the projector 111 due to the relative posture relationship between the sensor unit 11 and the target object 311, three-dimensional position and posture measurement is impossible in that region.

Therefore, in the present embodiment, a hidden region where the target object 311 is shielded when the target object 311 is viewed by the projector 111 from a plurality of viewpoints is calculated. Then, based on the calculated size (occlusion amount) of the hidden area, the second position / posture of the sensor unit 11 suitable for three-dimensional measurement is determined, and the three-dimensional shape of the target object 311 is determined based on the second position / posture. Make measurements.
Thus, since the second position and orientation of the sensor unit 11 is determined in consideration of the hidden area with respect to the target object 311, appropriate three-dimensional information related to the target object 311 is acquired by the sensor unit 11 having the second position and orientation. can do. Therefore, it is possible to suppress a decrease in measurement accuracy of the position and orientation (three-dimensional shape) of the target object 311 due to the occurrence of occlusion, and appropriate three-dimensional measurement is possible.

  At this time, the second position and orientation may be determined by rotating the projector 111 around the camera 112 with the line-of-sight direction of the camera 112 as the axis at the second measurement position. Specifically, the second position and orientation can be determined by estimating the occlusion amount for each of n posture candidates obtained by rotating the projector 111 by 2π / n [rad] around the camera 112. As a result, the second position / posture candidate can be created relatively easily, and the occurrence state of the occlusion can be easily simulated.

As shown in FIG. 16, even if the coordinates (position) of the camera 112 are the same as those in FIG. 10, the hidden area 301 generated by the obstacle 312 becomes the target object 311 if the arrangement (posture) of the projector 111 is appropriate. It won't take on. Therefore, in the position and orientation shown in FIG. 16, all the pattern projection light of the projector 111 projected onto the target object 311 can be observed by the camera 112 without being affected by occlusion.
Therefore, an evaluation process for searching for a position and orientation that minimizes the occlusion amount of each orientation candidate is performed, and an orientation that minimizes the occlusion amount is determined as the second position and orientation that is the optimum value of the position and orientation of the sensor unit 11. Is preferred. Then, the 3D information of the target object 311 is acquired with the second position and orientation, and the 3D position and orientation is measured.
As described above, the second position and orientation of the sensor unit 11 is determined so that the occlusion amount 302 of the hidden area 301 generated by the geometric arrangement of the obstacle 312 and the sensor unit 11 is minimized. Three-dimensional position and orientation measurement can be performed with high accuracy.

Further, it is preferable that this three-dimensional position / orientation measurement is repeatedly executed until a prescribed condition is satisfied. At this time, when the occlusion amount is equal to or smaller than the threshold value, or the difference between the first position and posture (the movement amount of the sensor unit 11) is equal to or smaller than the threshold value, the three-dimensional position and posture measurement is performed. End the iteration. Thereby, every time the number of times of acquiring the three-dimensional information is repeated, a detailed three-dimensional position and orientation measurement result for the target object can be obtained.
Since the operation (for example, gripping or suctioning) of the robot 10 on the target object 311 is controlled based on the position and orientation of the target object 311 measured with high accuracy as described above, the robot 10 can prevent the above-described operation failure. Suppressing and realizing efficient work.

(Second embodiment)
Next, a second embodiment of the present invention will be described.
In the second embodiment, the second position and orientation are determined by specifying the optimum position and orientation in consideration of minute occlusion due to the shape in the target object 31.
FIG. 17 is a diagram illustrating the shape of the target object 31 in the second embodiment. As shown in FIG. 17, the target object 31 includes a grip portion 31 a that is gripped by the hand 13 of the robot 10. The grip portion 31 a is a characteristic part of the target object 31.

If the target object 31 has a shape having a grip portion 31a as shown in FIG. 17, there is a possibility that occlusion occurs around the connection portion of the grip portion 31a. An example is shown in FIG.
When the position and orientation of the sensor unit 11 is in the state shown in FIG. 18 and the target object 311 that is a picking candidate is photographed by the camera 112, the occlusion caused by the grasping unit 311a is not detected although the occlusion due to the obstacle 312 does not occur. Occurs on 311. As described above, when the occlusion occurs around the grip portion 311a, the location specifying accuracy when the robot 10 grips is lowered.

Therefore, in this embodiment, the position and orientation determination method of the sensor unit 11 that minimizes the hidden region due to the obstacle 312 performed in the first embodiment described above is applied to the hidden region 301 around the grip unit 311a that is a characteristic part. Apply. That is, the position and orientation at which the size (occlusion amount) of the hidden area 301 related to the grip portion 311a that is the characteristic portion is minimized is determined as the second position and orientation.
In addition, the position and orientation optimization method of the sensor unit 11 for minimizing the occlusion amount of the hidden area due to the obstacle 312 shown in the first embodiment, and the hidden area around the grip part 311a shown in the second embodiment The method for optimizing the position and orientation of the sensor unit 11 for minimizing the occlusion amount may be performed sequentially or simultaneously if possible.

As described above, the optimum position and orientation with respect to the minute occlusion caused by the shape of the target object 31 can be specified. Therefore, it is possible to appropriately determine the second position and orientation of the sensor unit 11 and improve the photographing accuracy. Thus, if the amount of minute occlusion caused by the shape in the target object 31 is reduced, it becomes possible to reduce the amount of occlusion around the grip portion 311a, which is a characteristic part, and specify the location when the robot 10 performs gripping. Increased accuracy.
In addition, a local shape such as the grip portion 311a often has a feature amount in image processing (SIFT method, Harris method, Canny method, etc.). Therefore, a reduction in the amount of occlusion around the location where these image feature amounts increase can be expected to improve accuracy in processing such as model fitting.

(Modification of the second embodiment)
In the second embodiment, examples of gripping and sucking are given as work by the robot 10, but when gripping, a location where the target object is gripped (N spaces for an N-finger hand) The position and orientation may be obtained with particularly high accuracy. In the case of performing suction, the position and orientation of a portion (stable surface) that sucks the target object may be obtained particularly accurately. In this case, the method described in the second embodiment is used by using the characteristic parts existing around the gripping part and the suctioning part.

(Third embodiment)
Next, a third embodiment of the present invention will be described.
In the third embodiment, in the first and second embodiments described above, the one sensor unit 11 is moved to the first position and orientation and the three-dimensional information of the target object 31. In contrast to this, two sensor units are installed.
FIG. 19 is a diagram illustrating a configuration example of the robot system 100 according to the third embodiment.
The robot system 100 in this embodiment has the same configuration as the robot system 100 of FIG. 1 except that the sensor unit 14 is added to the robot system 100 shown in FIG. Therefore, here, the description will focus on the different parts.

Similar to the sensor unit 11, the sensor unit 14 includes a projector and a camera, acquires three-dimensional information about the target object 31, and outputs the acquired three-dimensional information to the sensor information acquisition unit 211 of the image processing unit 210. The sensor unit 14 is arranged in the first position and orientation in the first and second embodiments described above.
The image processing unit 20 measures the three-dimensional position and orientation of the target object 31 based on the two-dimensional image acquired by the sensor unit 14 in the first position and orientation. Then, the image processing unit 210 calculates the occlusion amount based on the three-dimensional position / orientation measurement result, and determines the optimum position / orientation (second position / orientation) of the sensor 11 using the result.

As described above, the sensor unit that acquires the three-dimensional information with the first position and orientation is different from the sensor unit that acquires the three-dimensional information with the second position and orientation. By performing three-dimensional position and orientation measurement of the target object 31 using the two sensor units 14 and 11, three-dimensional information can be efficiently acquired.
The sensor unit 14 may be a sensor unit that includes only a camera and acquires two-dimensional information as long as the distance can be calculated and estimated. That is, the information regarding the target object 31 acquired at the first position and orientation may be two-dimensional information. Also in this case, the position and orientation of the target object 31 can be obtained by using the data obtained by analyzing the two-dimensional image acquired by the sensor unit 14 and the model information held by the model information holding unit 212.

At this time, when a 3D geometric model is used as model information, the position and orientation of the target object 31 are obtained by associating a line that is a side of the 3D geometric model with the edge component extracted from the dimensional image. You can ask for. For example, the approximate value of the position and orientation of the target object 31 is repeatedly corrected by iterative calculation so that a three-dimensional geometric model is applied to the two-dimensional image.
When a reference image model is used as model information, a reference image that most closely matches is obtained by template matching using the reference image model as a template, and the target object is based on the position and orientation information of the target object 31 associated with the reference image. What is necessary is just to obtain the position and orientation of 31.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described.
In the fourth embodiment, an obstacle causing occlusion to a target object is specified, and a hidden area is reduced.
FIG. 20 is a diagram illustrating a configuration example of the robot system 100 according to the fourth embodiment.
The robot system 100 according to this embodiment has a configuration in which an obstacle recognition unit 216 is added to the robot system 100 shown in FIG.

The obstacle recognition unit 216 determines whether or not to change the picking candidate based on the occlusion amount of the n posture candidates estimated by the occlusion estimation unit 214. Specifically, it is determined that the picking candidate is changed when the amount of occlusion for n postures is equal to or greater than a preset threshold value. In this case, the target object 31 that is the cause (obstacle) of the occurrence of occlusion is specified based on the position and orientation information of the plurality of target objects 31, and the specific target object 31 is set as a new picking candidate.
Therefore, after that, three-dimensional position and orientation measurement of a new picking candidate (obstacle) is performed, and the obstacle is picked.

As described above, in the present embodiment, the obstacle that causes the occlusion is specified, and the process of removing the obstacle is performed before picking up the target object that was initially selected as the picking candidate. Thereby, the hidden area in the target object initially selected as the picking candidate can be reduced, and the three-dimensional position and orientation of the target object can be accurately measured.
Here, the case where the obstacle causing the occlusion to the target object is removed has been described, but the obstacle may be moved to a position where the hidden area related to the target object becomes small. The obstacle recognition unit 216 in FIG. 20 corresponds to an example of an identification unit.

(Hardware configuration of information processing unit 20)
FIG. 21 is an example of a hardware configuration of the information processing unit 20.
The information processing unit 20 includes a CPU 21, a ROM 22, a RAM 23, an external memory 24, an input unit 25, a display unit 26, a communication I / F 27, and a system bus 28.
The CPU 21 controls the operation of the information processing unit 20 in an integrated manner, and controls each component (22 to 27) via the system bus 28.
The ROM 22 is a non-volatile memory that stores a control program necessary for the CPU 21 to execute processing. The program may be stored in the external memory 24 or a removable storage medium (not shown).

The RAM 23 functions as a main memory and work area for the CPU 21. That is, the CPU 21 implements various functional operations by loading a program or the like necessary from the ROM 22 into the RAM 23 when executing the processing, and executing the program or the like.
The external memory 24 stores, for example, various data and various information necessary for the CPU 21 to perform processing using a program. Further, the external memory 24 stores, for example, various data and various information obtained by the CPU 21 performing processing using a program or the like. The external memory 14 includes the model information holding unit 212 described above.

The input unit 25 includes, for example, a keyboard, a mouse, and the like, and an operator can give an instruction to the information processing unit 20 via the input unit 25.
The display unit 26 includes a monitor such as a liquid crystal display (LCD).
The communication I / F 27 is an interface for communicating with an external device.
The system bus 28 connects the CPU 21, ROM 22, RAM 23, external memory 24, input unit 25, display unit 26, and communication I / F 27 so that they can communicate with each other.
The function of each unit of the information processing unit 20 in the first to fourth embodiments described above is realized by the CPU 21 executing a program stored in the ROM 22 or the external memory 24.

(Modification)
In each of the above embodiments, the case where the sensor unit 11 is a sensor having the projector (projection unit) 111 and the camera (imaging unit) 112 has been described. However, the three-dimensional relationship regarding the target object 31 using the principle of triangulation is described. Any configuration may be used as long as the information can be acquired. For example, the sensor unit may include a plurality of cameras (imaging units) in which at least part of the field of view overlaps. Also in this case, it is possible to perform appropriate three-dimensional measurement by determining the position and orientation of the sensor unit in consideration of the occlusion that occurs due to the parallax of a plurality of cameras.

In each of the above embodiments, the case where the robot 10 is used as the means for changing the position and orientation of the sensor unit 11 has been described. However, the position and orientation change means is not limited to the robot 10. For example, the sensor unit 11 may be mounted on a mechanism unit that combines a linear motion stage and a rotary stage, and the position and orientation may be changed by controlling the stage. In this manner, position and orientation changing means may be provided separately from the robot 10 that performs work on the target object.
Further, in each of the embodiments described above, the case where the image captured by the sensor unit 11 is processed by the image processing unit 210 has been described. However, the image processing mechanism is provided in the sensor unit 11 and the result of image processing is output. It may be.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  DESCRIPTION OF SYMBOLS 10 ... Robot, 11 ... Sensor part, 111 ... Projector, 112 ... Camera, 12 ... Position and orientation change mechanism, 13 ... Hand, 20 ... Information processing part, 210 ... Image processing part, 211 ... Sensor information acquisition part, 212 ... Model Information holding unit, 213 ... three-dimensional position / orientation measurement unit, 214 ... occlusion estimation unit, 215 ... sensor position / orientation determination unit, 216 ... obstacle recognition unit, 220 ... robot controller unit, 221 ... robot work instruction unit, 222 ... robot Control unit, 30 ... object to be measured, 31 ... target object, 32 ... pallet

Claims (16)

Sensor means for acquiring two-dimensional information or three-dimensional information about the target object in the first position and orientation;
Based on two-dimensional information or three-dimensional information acquired by the sensor means at the first position and orientation, a hidden area where the target object is shielded when the target object is viewed from a plurality of viewpoints is calculated. Calculating means for
A second position different from the first position and orientation as the position and orientation of the sensor means capable of acquiring three-dimensional information for measuring the position and orientation of the target object based on the hidden area calculated by the calculation means An information processing apparatus comprising: a determination unit that determines a posture. The determining means includes
The information processing apparatus according to claim 1, wherein the second position and orientation of the sensor unit is determined based on the position information of the viewpoint that minimizes the hidden area calculated by the calculation unit. The determining means includes
3. The second position and orientation of the sensor means is determined based on the position information of the viewpoint at which the hidden area relating to the characteristic part of the target object calculated by the calculation means is minimized. The information processing apparatus described in 1. Moving means for moving the sensor means to the second position and orientation determined by the determining means;
Using the three-dimensional information acquired by the sensor means having the second position and orientation for the calculation of the hidden area by the calculation means, a process for re-determining the second position and orientation by the determination means The information processing apparatus according to claim 1, further comprising: a repeating unit that repeats until the condition is satisfied.   The information processing apparatus according to claim 4, wherein the predetermined condition is that a size of the hidden area calculated by the calculation unit is equal to or less than a threshold value.   The information processing apparatus according to claim 4, wherein the predetermined condition is that a moving amount of the sensor unit by the moving unit is equal to or less than a threshold value. Based on the two-dimensional information or three-dimensional information acquired by the sensor means, a specifying means for specifying an obstacle causing a hidden area with respect to the target object;
A second moving means for moving the obstacle specified by the specifying means to a position where the size of the hidden area is reduced when the size of the hidden area calculated by the calculating means exceeds a threshold; The information processing apparatus according to claim 1, further comprising: an information processing apparatus according to claim 1. The sensor means includes
Projection means for projecting a pattern;
Imaging means for imaging the pattern projected by the projection means from different viewpoints,
The information processing apparatus according to claim 1, wherein three-dimensional information related to the target object is acquired based on a triangulation principle. The sensor means includes
Comprising a plurality of imaging means overlapping at least part of the field of view;
The information processing apparatus according to claim 1, wherein three-dimensional information related to the target object is acquired based on a triangulation principle.   The sensor means for acquiring two-dimensional information or three-dimensional information with the first position and orientation is the same sensor means as the sensor means for acquiring three-dimensional information with the second position and orientation. The information processing apparatus according to claim 1.   The sensor means for acquiring two-dimensional information or three-dimensional information with the first position and orientation is a sensor means different from the sensor means for acquiring three-dimensional information with the second position and orientation. The information processing apparatus according to claim 1.   The information processing apparatus according to claim 1, wherein the sensor unit is attached to a robot that performs work on the target object. Measuring means for measuring the position and orientation of the target object based on the three-dimensional information acquired by the sensor means having the second position and orientation;
The control unit according to any one of claims 1 to 12, further comprising: a control unit that controls an operation of the robot on the target object based on the position and orientation of the target object measured by the measurement unit. The information processing apparatus described. Acquiring two-dimensional information or three-dimensional information about the target object by the sensor means at the first position and orientation;
Based on two-dimensional information or three-dimensional information acquired by the sensor means at the first position and orientation, a hidden area where the target object is shielded when the target object is viewed from a plurality of viewpoints is calculated. And steps to
Based on the calculated hidden area, a second position / posture different from the first position / posture is determined as the position / posture of the sensor means capable of acquiring three-dimensional information for measuring the position / posture of the target object. And an information processing method comprising: steps. A robot that performs work on the target object;
A robot system comprising: the information processing apparatus according to any one of claims 1 to 13. The program for functioning a computer as each means of the information processing apparatus of any one of the said Claims 1-13.

Images