JP6752615B2 - Information processing device, information processing method, robot control device and robot system - Google Patents

Description

The present invention relates to an information processing device, an information processing method, a robot control device, and a robot system.

Conventionally, as a technique for a robot to grasp a randomly piled object one by one, a robot device for grasping a position and a posture of the object object by using a visual sensor is known. In Patent Document 1, an object having the same shape as the target object is imaged from a plurality of directions to create a teaching model, and the teaching model is matched with the image data obtained by capturing the piled target objects. It is disclosed to recognize the position and orientation of a target object.

Japanese Patent No. 3300682

In the measurement using the visual sensor, only the information on the surface position of the imaged object can be acquired. Therefore, in the technique described in Patent Document 1, when the target object is wrapped with a plastic bag or a cushioning material, the target object in the image data does not match the teaching model. Therefore, in this case, the position and posture of the target object, which is the contents wrapped in the plastic bag or the cushioning material, cannot be properly recognized, and the robot cannot grasp the target object.
Therefore, an object of the present invention is to appropriately recognize the position and orientation of the target object even when the target object cannot be accurately recognized only by the surface position information.

In order to solve the above problems, one aspect of the information processing device according to the present invention is an information processing device that recognizes an object composed of a target object and a peripheral object, and is an acquisition means for acquiring surface position information of the object. A determination means for determining a measurement position for actually measuring the object based on the surface position information acquired by the acquisition means, and a contact when the object is contacted at the measurement position determined by the determination means. A measuring means for measuring position information, a creating means for creating map information indicating spatial position information of the object based on the contact position information measured by the measuring means, and a creating means created by the creating means. The measuring means includes a recognizing means for recognizing the position and orientation of the object based on the map information, and the measuring means includes a distance acquiring means for acquiring the distance to the approaching object and comes into contact with the object. based on the distance obtained by the distance obtaining means when being, it measures the contact position information.

According to the present invention, even when the target object cannot be accurately recognized only by the surface position information, the position and orientation of the target object can be appropriately recognized.

It is a figure which shows the configuration example of the robot system of 1st Embodiment. It is a figure which shows the hardware configuration example of an information processing apparatus. It is a functional block diagram of the information processing apparatus in 1st Embodiment. It is a flowchart for demonstrating the operation in 1st Embodiment. It is a figure for demonstrating the operation in 1st Embodiment. It is a figure which shows the configuration example of the robot system of the 2nd Embodiment. It is a functional block diagram of the information processing apparatus in the 2nd Embodiment. It is a flowchart for demonstrating operation in 2nd Embodiment. It is a figure which shows the configuration example of the robot system of the 3rd Embodiment.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings.
The embodiments described below are examples of means for realizing the present invention, and should be appropriately modified or modified depending on the configuration of the device to which the present invention is applied and various conditions. It is not limited to the embodiment of.
(First Embodiment)
In the first embodiment, the three-dimensional position information of the object measured based on the image information of the object (the target object and the peripheral objects around it) obtained by imaging with the imaging device, and the measurement information by the force sensor. To create map information of an object by integrating with. Then, the position and orientation of the target object are recognized by using the created map information. In the present embodiment, the target object is an object to be operated by the robot, and includes, for example, a component such as a toner cartridge wrapped in a plastic bag or a cushioning material. Further, the measurement by the force sensor means that the contact position by the robot is measured by the position detection sensor when the force sensor detects that the robot is in contact with an object. Further, the map information is spatial position information of the target object and surrounding objects.

FIG. 1 is a diagram showing a configuration example of a robot system 100 including an information processing device 20 according to the present embodiment.
The robot system 100 includes a robot 10 and an information processing device 20. The robot 10 is, for example, an articulated robot, and includes a manipulator such as a robot arm and an end effector 11 such as a robot hand. Further, the robot 10 is provided with a position / posture changing mechanism capable of changing the position / posture of the end effector 11 by changing the angle of each joint of the manipulator. The position / orientation changing mechanism may be driven by an electric motor or an actuator operated by a fluid pressure such as hydraulic pressure or pneumatic pressure. This position / orientation changing mechanism is driven according to the operation instruction information output from the information processing apparatus 20.

Further, the end effector 11 is a tool for realizing an operation according to the type of the target object of the robot 10, and is a hand having a chuck mechanism that can drive a motor and capable of gripping the target object, or a hand that can grip the target object by air pressure. A hand using a suction pad that sucks can be used. The end effector 11 is detachably attached to the robot arm and can be replaced according to the type of work. Further, the robot 10 is not limited to the articulated robot, and may be a movable machine capable of numerical control (NC).

The robot 10 executes an action determined by the information processing device 20 and executes an operation of transporting or grasping the target object 41. In the present embodiment, the action is the action of the robot 10 for recognizing or manipulating the target object 41.
The target object 41 is a part such as a toner cartridge that is supposed to be gripped and transported by the robot 10, and may be wrapped in a plastic bag or cushioning material. That is, around the target object 41, there are peripheral objects 42 such as a plastic bag and a cushioning material that wrap the target object 41. The peripheral object 42 is all objects other than the target object 41 located around the target object 41. For example, the peripheral object 42 is placed in a container containing a pile of target objects 41 or in the vicinity of the container. Including objects.

Further, the robot system 100 further includes an image pickup device 31, a light source 32, a force sensor 33, and a position detection sensor 34.
The image pickup device 31 is a visual sensor composed of a camera, a sensor for detecting light, a photodiode, and the like, and acquires image information of the target object 41 and the peripheral object 42. The image pickup apparatus 31 outputs the acquired image information to the information processing apparatus 20. The light source 32 is composed of, for example, a projector, and emits visible light or infrared light from a laser light source to project uniform illumination light or pattern light onto the target object 41 and peripheral objects 42. .. The image pickup apparatus 31 captures images of the target object 41 and the peripheral object 42 to which light is projected by the light source 32. The image pickup device 31 can also take an image of the target object 41 and the peripheral object 42 in a state where the light is not projected by the light source 32. As shown in FIG. 1, the image pickup apparatus 31 and the light source 32 may be fixedly arranged above the image pickup target space, may be mounted on the robot 10, or may be mounted on another work machine. Further, a plurality of image pickup devices 31 may be arranged.

The force sensor 33 is attached to the base of the end effector 11 of the robot 10. The force sensor 33 is composed of a strain gauge and a piezoelectric element, and measures the force (reaction force) applied to the robot 10 when the robot 10 is in contact with the target object 41 or the peripheral object 42, which is a measurement result. The force sensor information is output to the information processing device 20. The force sensor 33 may be a 6-axis force sensor (Force / Torque sensor) or a 3-axis force sensor. Further, the force sensor 33 may be a one-dimensional pressure sensor or a contact sensor that determines the presence or absence of contact.

The position detection sensor 34 detects the position of the robot 10 and outputs the position detection information, which is the detection result, to the information processing device 20. The position detection sensor 34 is configured to include an encoder attached to the robot 10, and can detect the position of the robot 10 by using forward kinematics based on the angle information acquired from the encoder. The position detection sensor 34 can be used together with the force sensor 33 to detect the position of the robot 10 when the robot 10 is in contact with the target object 41 or the peripheral object 42. The method of detecting the position of the robot 10 is not limited to the method using the position detection sensor 34 described above.

The information processing device 20 is a robot control device that controls the operation of the robot 10. The information processing device 20 is composed of, for example, a personal computer (PC). FIG. 2 is an example of the hardware configuration of the information processing device 20. The information processing device 20 includes a CPU 21, a ROM 22, a RAM 23, an external memory 24, an input unit 25, a display unit 26, a communication I / F 27, and a system bus 28.
The CPU 21 comprehensively controls the operation of the information processing apparatus 20, and controls each component (22 to 27) via the system bus 28. The ROM 22 is a non-volatile memory that stores a program required for the CPU 21 to execute a process. The program may be stored in an external memory 24 or a removable storage medium (not shown). The RAM 23 functions as a main memory and a work area of the CPU 21. That is, the CPU 21 loads a program required from the ROM 22 into the RAM 23 when executing the process, and executes the program to realize various functional operations.

The external memory 24 stores, for example, various data and various information necessary for the CPU 21 to perform processing using a program. Further, in the external memory 24, for example, various data and various information obtained by the CPU 21 performing processing using a program are stored. The input unit 25 is composed of, for example, a keyboard or mouse pointing device, and an operator can give an instruction to the information processing device 20 via the input unit 25. The display unit 26 is composed of a monitor such as a liquid crystal display (LCD). The communication I / F 27 is an interface for communicating with an external device. The system bus 28 communicatively connects the CPU 21, ROM 22, RAM 23, external memory 24, input unit 25, display unit 26, and communication I / F 27.
In this way, the information processing device 20 is communicably connected to the robot 10, the image pickup device 31, the light source 32, and the force sensor 33, which are external devices, via the communication I / F 27, and these external devices are connected to each other. Control the operation of.

FIG. 3 is a functional block diagram of the information processing device 20.
The information processing device 20 includes an image acquisition unit 201, a surface position measurement unit 202, a map information creation unit 203, a measurement position determination unit 204, and an action planning unit 205. Further, the information processing device 20 includes a force sense information acquisition unit 206, a force measurement unit 207, a position information acquisition unit 208, a position measurement unit 209, and a gripping position determination unit 210. The surface position measurement unit 202, the map information creation unit 203, the measurement position determination unit 204, the force measurement unit 207, and the position measurement unit 209 constitute a recognition unit 220 that performs processing for recognizing the position and orientation of the target object 41. .. The recognition unit 220 can operate as an object recognition device.
The image acquisition unit 201 acquires image information output from the image pickup apparatus 31. The image acquisition unit 201 converts the acquired image information into image data and outputs it to the surface position measurement unit 202. The image acquisition unit 201 is composed of, for example, a capture board and a memory (RAM). The surface position measuring unit 202 measures the surface position information (surface position information) of the target object 41 and the peripheral object 42 based on the image data acquired from the image acquisition unit 201. The surface position measuring unit 202 outputs the measured surface position information to the map information creating unit 203.

The map information creating unit 203 is based on the surface position information measured by the surface position measuring unit 202, the force information measured by the force measuring unit 207 described later, and the position information measured by the position measuring unit 209 described later. Create map information. The map information creation unit 203 acquires the force information measured by the force measurement unit 207 via the position measurement unit 209. The map information is spatial position information of the target object 41 and the peripheral object 42, and in the present embodiment, it is the position information of the target object 41 and the peripheral object 42 in the three-dimensional space. The map information creation unit 203 first creates map information based on the surface position information, and then updates the map information created based on the surface position information based on the force information and the position information in detail. Create map information.

The surface positions of the target object 41 and the peripheral objects 42 can be obtained from the surface position information measured based on the image data. However, as described above, when the target object 41 is a toner cartridge wrapped in a plastic bag, the surface position obtained from the surface position information is the surface position of the plastic bag, not the surface position of the toner cartridge. As described above, it may not be possible to obtain the accurate position and orientation of the target object 41 from the surface position information measured based on the image data.

Therefore, in the present embodiment, in order to recognize the target object 41 more accurately, the robot 10 directly touches the target object 41 and measures the target object 41. Specifically, the force sensor 33 and the position detection sensor 34 are used to measure the position information (contact position information) of the robot 10 when the robot 10 comes into contact with the target object 41. Then, the map information is updated using the measured position information of the robot 10. That is, when creating the map information, the map information creation unit 203 actually uses the force sensor 33 and the position detection sensor 34 with respect to the surface position of the target object 41 measured by using the imaging device 31. The result of the detailed position of the target object 41 measured by touching is added. In this way, the position and orientation of the target object 41 are measured in more detail. The map information creation unit 203 outputs the created map information to the measurement position determination unit 204 and the grip position determination unit 210.

The measurement position determination unit 204 determines the measurement position for actually measuring the detailed position and posture of the target object 41 based on the map information input from the map information creation unit 203. The map information input to the measurement position determination unit 204 is the map information created by the map information creation unit 203 based on the surface position information. The measurement position determination unit 204 refers to the input map information and determines the position determined to have a high existence probability of the target object 41 as the measurement position. The measurement position determination unit 204 outputs the determined measurement position to the action planning unit 205.

The action planning unit 205 plans the action of the robot 10 based on the measurement position input from the measurement position determination unit 204. Specifically, the action planning unit 205 creates a trajectory of the robot 10 for moving the end effector 11 to the measurement position and a trajectory of the robot 10 for realizing a contact operation with the target object 41 at the measurement position. Then, the operation instruction information is output to the robot 10. Further, the action planning unit 205 can also input the gripping position of the target object 41 from the gripping position determining unit 210 described later. When the gripping position is input, the action planning unit 205 plans the action of the robot 10 based on the input gripping position. Specifically, the action planning unit 205 creates a trajectory of the robot 10 for moving the end effector 11 to the gripping position and a trajectory of the robot 10 for realizing the gripping operation of the target object 41 at the gripping position. , Outputs operation instruction information to the robot 10.

The force sense information acquisition unit 206 acquires the force sense information output from the force sense sensor 33, and outputs the acquired force sense information to the force measurement unit 207. The force sense information acquisition unit 206 is composed of, for example, a memory (RAM). The force measuring unit 207 applies a force (anti-force) applied to the robot 10 when the end effector 11 of the robot 10 is in contact with the target object 41 or the peripheral object 42 based on the force sense information input from the force sense information acquisition unit 206. Force) is measured. The force measuring unit 207 outputs the measured force information to the position measuring unit 209.

The position information acquisition unit 208 acquires the position detection information output from the position detection sensor 34, and outputs the acquired position detection information to the position measurement unit 209. The position information acquisition unit 208 is composed of, for example, a memory (RAM). The position measurement unit 209 contacts the end effector 11 of the robot 10 with the target object 41 or the peripheral object 42 based on the force information input from the force measurement unit 207 and the position detection information input from the position information acquisition unit 208. Measure the position of the contact point when you are. The position measurement unit 209 outputs the measured position information (contact position information) to the map information creation unit 203.

The gripping position determining unit 210 determines the gripping position of the target object 41 based on the map information output from the map information creating unit 203. The map information input to the gripping position determination unit 210 is detailed map information created by the map information creation unit 203 based on the force information and the position information. The gripping position determining unit 210 refers to the input map information and determines the position where the robot 10 can grip the target object 41 as the gripping position. The gripping position determining unit 210 outputs the determined gripping position to the action planning unit 205.
The functions of each part of the information processing apparatus 20 shown in FIG. 3 can be realized by the CPU 21 shown in FIG. 2 executing a program stored in the ROM 22 or the external memory 24.

The operation of the robot system 100 will be described below.
FIG. 4 is a flowchart showing a robot control processing procedure executed by the information processing device 20. The process shown in FIG. 4 is realized by the CPU 21 of the information processing device 20 reading and executing a program stored in the ROM 22 or the external memory 24. However, part or all of the processing of FIG. 4 may be realized by dedicated hardware. The process of FIG. 4 is started, for example, when the operator activates the robot system 100. However, the start timing is not limited to when the robot system 100 is started.

First, in S1, the CPU 21 performs a system initialization process. That is, the CPU 21 loads the program stored in the ROM 22 or the external memory 24 and expands it on the RAM 23 so that it can be executed. In addition, the parameters of each device connected to the information processing device 20 are read and returned to the initial position to make them usable.
Next, in S2, the image processing unit 201 acquires the image information of the target object 41 and the peripheral object 42 output by the image pickup apparatus 31, and shifts to S3. In S3, the surface position measuring unit 202 measures the surface positions of the target object 41 and the peripheral object 42 based on the image information acquired by the image processing unit 201 in S2. Since the image information obtained from the image pickup apparatus 31 is information of a three-dimensional point cloud, the surface position measuring unit 202 recognizes the unevenness of the surfaces of the target object 41 and the peripheral object 42 based on the acquired image information. Can be done. The surface positions of the target object 41 and the peripheral objects 42 may be determined by referring to the component database (component DB) previously stored in the external memory 24 or the like. The component DB holds information on the component type and shape of the target object 41 to be recognized. The shape information of the target object 41 stored in the component DB is, for example, three-dimensional model data such as CAD data and CG data (polygon data). The shape information of the target object stored in the component DB may be composed of a set of two-dimensional images obtained by observing the target object 41 and the peripheral objects 42 from multiple directions.

In S4, the map information creation unit 203 creates map information based on the surface position information measured by the surface position measurement unit 202 in S3. Hereinafter, the method of creating map information will be specifically described. In order to create map information, it is necessary to combine the information obtained from a plurality of different sensors into one spatial map information. However, the image pickup device coordinate system of the image pickup device 31, the force sensor coordinate system of the force sensor 33, and the robot coordinate system of the robot 10 are all different. Therefore, it is necessary to unify these coordinate systems.

First, the world coordinate system Σw is set as a reference coordinate system in the work space. Then, the displacement from the world coordinate system Σw to the robot coordinate system Σr is defined as (RX, RY, RZ). Further, let RM be a 3 × 3 rotation matrix representing the posture of the robot 10.
The displacement from the robot coordinate system Σr to the tip coordinate system Σf of the robot 10 is (FX, FY, FZ). Further, the 3 × 3 rotation matrix representing the posture of the tip of the robot 10 is FM. Further, the displacement from the tip coordinate system Σf to the coordinate system Σe at the tip of the end effector 11 of the robot 10 is (EX, EY, EZ). Further, the 3 × 3 rotation matrix representing the posture of the tip of the end effector 11 is EM. The tip of the end effector 11 is a portion that comes into contact with the target object 41 or the peripheral object 42.

The displacement from the tip coordinate system Σf of the robot 10 to the force sensor coordinate system Σt is (TX, TY, TZ). Further, the rotation matrix of 3 × 3 representing the posture of the force sensor 33 is TM. Further, the displacement from the robot coordinate system Σr to the image pickup device coordinate system Σc is (CX, CY, CZ). Further, the 3 × 3 rotation matrix representing the posture of the image pickup apparatus 31 is CM. Further, the displacement from the image pickup device coordinate system Σc to the target object coordinate system Σv is (VX, VY, VZ). Further, the rotation matrix of 3 × 3 representing the posture of the target object 41 is VM. Here, the displacement of the target object 41 as seen from the world coordinate system Σw is (WX, WY, WZ), and the 3 × 3 rotation matrix representing the posture is WM.
When the robot 10 is in contact with the target object 41, the position of the tip of the end effector 11 attached to the tip of the robot 10 coincides with the position of the target object 41 photographed by the imaging device 31. Therefore, the following equations (1) and (2) hold.

なお、ワールド座標系Σｗから見た、接触動作時にロボット１０にかかる力を４×４の行列ＷＴとおくと、行列ＷＴは以下の式で求まる。この（３）式を用いて、力の方向をワールド座標系Σｗで考慮することができる。

If the force applied to the robot 10 during the contact operation as seen from the world coordinate system Σw is a 4 × 4 matrix WT, the matrix WT can be obtained by the following equation. Using this equation (3), the direction of force can be considered in the world coordinate system Σw.

The map information creation unit 203 creates map information under the above definition. That is, the map information creation unit 203 unifies the coordinate systems of a plurality of different sensors by using the above equations (1) to (3), and when the coordinate systems are unified, the information obtained from each sensor is used. And create map information.
In S4, the map information creation unit 203 first arranges voxels with respect to the work space of the world coordinate system Σw. Here, the region where the voxels exist is a region where the target object 41 or the peripheral object 42 may exist. Further, the size of the voxel is appropriately set according to the accuracy required for the gripping operation of the target object 41.
Next, the map information creation unit 203 uses the surface position information to perform a process of scraping all voxels existing outside the surface positions of the target object 41 and the peripheral object 42. Since the surface position information is three-dimensional point cloud data, and this surface position information can be obtained as information in the image pickup device coordinate system Σc, the surface position of the target object 41 can be determined in the world coordinate system Σw using the above equation (2). Can be converted. If a point cloud exists in the voxels arranged in the world coordinate system Σw, the voxels are to be deleted. The area where the voxels are removed is treated as a part where no object exists. As described above, map information indicating the space in which the target object 41 and the peripheral object 42 roughly exist is created.

In S5, the measurement position determination unit 204 determines the measurement position of the target object 41 based on the map information created in S4. Specifically, the measurement position determination unit 204 selects a portion having a convex surface based on the information on the unevenness of the surface of the target object 41 and the peripheral object 42 with reference to the map information. Then, assuming that the selected portion is a portion having a high existence probability of the target object 41, it is determined as a measurement position for measuring the target object 41 in detail. If the position of the target object 41 can be estimated using the component DB, the measurement position may be determined based on the estimated position. In this way, when the rough position and posture of the target object 41 can be estimated from the component DB, the measurement position for directly contacting and confirming whether or not the estimated position and orientation are correct is determined.

In S6, the action planning unit 205 determines the action plan of the robot 10 for realizing the measurement by the measurement position based on the measurement position determined by the measurement position determination unit 204 in S5. Next, in S7, the action planning unit 205 outputs the action instruction information to the robot 10 based on the action plan determined in S6. As a result, the robot 10 moves the end effector 11 to the measurement position and makes a contact operation with the target object 41 at the measurement position. The operation of the robot 10 at this time is not only the operation of the end effector 11 moving to the surface position measured by the surface position measuring unit 202, but also the operation of acquiring force sense information while moving deeper than the surface position. As shown in FIG. 5 (a), when the target object 41 is wrapped in a peripheral object 42 such as a plastic bag or a cushioning material, the end effector 11 crushes the peripheral object 42 as shown in FIG. 5 (b). Moving.

In S8, the force sense information acquisition unit 206 acquires the force sense information output by the force sense sensor 33. Further, the position information acquisition unit 208 acquires the position detection information output by the position detection sensor 34 in synchronization with the acquisition of the force sense information by the force sense information acquisition unit 206. In S9, the force measuring unit 207 measures the force information indicating the force applied to the robot 10 based on the force sense information acquired by the force sense information acquisition unit 206 in S8. Further, the position measurement unit 209 measures the position information indicating the position of the robot 10 based on the position detection information acquired by the position information acquisition unit 208 in S8.

The force information and the position information may be repeatedly measured while the robot 10 makes a contact operation with the target object 41. In that case, in the flowchart shown in FIG. 4, S7 to S9 are written as a one-way flow, but the processes of S7 to S8, S8 to S9, S9 to S7, and S7 to S9 are repeated at high speed. That is, the robot 10 acquires force sense information / position information while moving, and performs force measurement / position measurement. Further, not only S7 to S9 but also the map information of S10 may be updated. In that case, instead of returning from S9 to S7, the processing flow of S7 to S8, S8 to S9, S9 to S10, and S10 to S7 is repeated at high speed. Here, the contact operation is performed within a range in which the magnitude of the force applied to the robot 10 does not exceed a preset upper limit value. Whether or not the magnitude of the force applied to the robot 10 exceeds the upper limit value is determined based on the force information measured in S9. During the contact operation, the force applied to the robot 10 increases according to the contact force with the target object 41. Therefore, the upper limit value of the above force is set to a force that does not cause a problem such as damage to the target object 41 due to the contact operation. For example, when the target object 41 is a toner cartridge wrapped in a plastic bag, a force that does not crush the plastic bag and break the toner cartridge is set as the upper limit value.

When the robot 10 crushes the plastic bag and comes into contact with the surface of the toner cartridge and the force applied to the robot 10 reaches the upper limit value, the information processing device 20 stops the contact operation of the robot 10. The position of the robot 10 when the contact operation is stopped can be recognized as the actual surface position of the target object 41.
In S10, the map information creation unit 203 updates the map information created in S4 based on the force information and the position information measured in S9, and creates new map information. In S10, the map information creation unit 203 uses the position information acquired in S9 as a result of the robot 10 actually operating at the measurement position. Then, the map information created in S4 is processed to remove the voxels corresponding to the space in which the robot 10 itself has moved.

At that time, the map information creation unit 203 uses the three-dimensional model information of the robot 10 held in advance, and the position (proprietary position) that the robot 10 occupies at each time based on the forward kinematics from the joint angle data. To calculate. Similarly, for the end effector 11 portion, the exclusive position is calculated from the position and posture of the tip of the robot 10 by using the three-dimensional model information of the end effect 11 held in advance. As described above, the exclusive position can be calculated based on the data of each joint angle of the robot 10 which is the position detection information obtained from the position detection sensor 34 installed in the robot 10. At that time, as in the above-mentioned unified processing of the coordinate system, the translational vector of the displacement is obtained from the length between the links, the rotation matrix of 3 × 3 is obtained from the joint angle, and forward kinematics is used. As a result, the three-dimensional position and orientation of the robot 10, that is, the displacement (FX, FY, FZ) from the robot coordinate system Σr to the tip coordinate system Σf of the robot 10 and the rotation matrix FM can be obtained.

The exclusive position obtained by the above is calculated as position information in the world coordinate system Σw by unifying the coordinate system described above. Therefore, the voxel corresponding to the calculated exclusive position can be removed. Even when the robot 10 is in contact with the target object 41 or the peripheral object 42, if the force measured by the force measuring unit 207 is smaller than the upper limit value, the robot 10 continues to move. Therefore, in this case, the map information creation unit 203 determines that the robot 10 has not yet completely contacted the target object 41, and at the same time as the robot 10 continues to move, spatially sweeps and scrapes the voxels. .. When the target object 41 is wrapped with something like a plastic bag, the robot 10 continues to operate by the end effector 11 crushing the plastic bag in the space where only the plastic bag exists. Therefore, the area in the voxel corresponding to the space where only the plastic bag exists will be scraped.

When the robot 10 comes into contact with the target object 41 and the force applied to the robot 10 reaches the upper limit value, the robot 10 stops the operation at that point. At this time, the map information creation unit 203 stops the process of scraping the voxels, determines that there is no object in the space swept up to the contact point, and that there is an object at the contacted position, and ends the map information creation.
In creating the map information, a method in which voxels are not arranged may be used. For example, the surface position information which is the three-dimensional point cloud data may be converted into the surface format data, and the surface positions of the target object 41 and the peripheral objects 42 may be expressed by using polygons. In this case, the surface position information created by the polygons may be used as an initial value, and the surface may be generated based on the three-dimensional position information with respect to the space swept by the robot 10 to create map information. Further, when creating map information in two dimensions, map information may be created on a two-dimensional plane by using the same method as above, using pixels instead of voxels.

In S11, the gripping position determining unit 210 determines whether or not the target object 41 can be gripped by the end effector 11 of the robot 10 based on the result of creating the map information in S10. Then, it is determined that the measurement is completed when the grip is possible, and it is determined that the measurement is continued when the grip is not possible.
Specifically, the position and posture of the target object 41 to be gripped are estimated based on the map information, and the grip is performed based on the estimated position and posture of the target object 41 and the shape of the end effector 11 of the robot 10. Judge whether it is possible or not. When the end effector 11 sandwiches and grips the side surface of the target object 41 from above, the end effector 11 needs to enter the side surface of the target object 41. In that case, first, based on the map information, it is determined whether or not there is a space in which the robot 10 can move on the target object 41. The presence or absence of space is determined by whether or not voxels have been removed from the map information.

If there is space, it is determined that the robot 10 can move above the target object 41. Then, next, whether or not there is a space for the end effector 11 to enter the side surface of the target object 41 is determined by an operation accompanied by contact. Then, when there is a space for the end effector 11 on the side surface of the target object 41, since the end effector 11 can grip the target object 41, it is determined that the measurement is finished and the process proceeds to S12. On the other hand, when it is determined that the end effector 11 cannot grip the target object 41, the process returns to S5 and the measurement is continued. In this way, the gripping position determining unit 210 determines a position having a space around which the end effector 11, which is a gripping portion of the robot 10, can be inserted, as a position where the robot 10 can grip the target object 41.

If it is determined in S11 that the grip is not possible, the process of determining the measurement position in S5 is performed again, but at this time, the result of the process up to this point may be used. For example, in S11, the robot is operating to determine whether it can be gripped, but when determining the measurement position, the next measurement position is obtained from another measurement position. Specifically, it is assumed that there are a plurality of candidate positions for determining the measurement position this time. At this time, as described in the explanation of S5, the portion where the existence probability of the target object 41 is high is determined as the measurement position by utilizing the uneven state of the surface. When this existence probability is used as an evaluation function, the position where this evaluation function shows the maximum value is determined as the measurement position in this measurement, but in the next measurement, the evaluation function will be the next largest value after the maximum value. The indicated position is determined as the measurement position. These results may be included in the created map information. In this way, not only the same processing is simply repeated, but also the result of the current processing may be updated to map information at any time so that it can be used in the next processing. By doing so, the robot 10 does not move to a position where it cannot be grasped many times. In addition, the robot 10 can move in descending order of gripability. Therefore, even if the target object 41 fails to be gripped in the first process, the probability that the target object 41 can be gripped in the second and subsequent processes increases.

In S11, FIG. 4 describes "Is it possible to grip?", But this is not only a determination of whether or not the robot 10 can be gripped, but also a determination of whether or not the robot 10 can move to a position where the gripping operation can be executed. Also includes. When the robot 10 moves, it may not be possible to move to the position where the gripping operation is scheduled to be performed due to an unexpected obstacle or the like. In that case, it is determined that the grip is not possible, and the process of S5 is performed.

In S12, the gripping position determining unit 210 determines the gripping position of the target object 41 based on the map information created in S10. The gripping position determining unit 210 determines the gripping position of the target object 41 from the positions determined to be grippable in S11.
In S13, the action planning unit 205 determines the action plan of the robot 10 for realizing the gripping operation of the target object 41 by the gripping position determined in S12. When determining the action plan, the action planning unit 205 refers to the operation database (operation DB) previously stored in the external memory 24 or the like based on the gripping position. The operation DB holds information on the part type and the work procedure. The action plan of the robot can be determined by using a general route search algorithm. As the route search algorithm, for example, a sampling-based search method such as RRT (Rapidly-exploring Random Tree) or PRM (Probabilistic RoadMap) can be used. At that time, using the map information, three-dimensional position information such as where the target object 41 is and where the obstacle is is input. By using such a method, the action planning unit 205 determines the action plan of the robot 10 that can grasp the target object 41 while avoiding obstacles. However, if similar results can be obtained, another method may be used.

Next, in S14, the action planning unit 205 outputs the action instruction information to the robot 10 based on the action plan determined in S13. As a result, the robot 10 moves the end effector 11 to the gripping position and performs a gripping operation on the target object 41 at the gripping position. The target object 41 gripped by the robot 10 is transported to a predetermined place according to the work procedure accumulated in the operation DB.
In S15, the CPU 21 determines whether or not the next target object 41 to be recognized exists based on the image information acquired by the image acquisition unit 201. Then, the CPU 21 determines that the recognition process is completed when the target object 41 does not exist, ends the process shown in FIG. 4, and determines that the recognition process is continued when the target object 41 exists, and determines that the recognition process is continued in S2. Return to.

In this way, the information processing device 20 determines the measurement position for measuring the position and orientation of the target object 41 in more detail based on the surface position information measured based on the image information obtained from the image pickup device 31. , The robot 10 is moved to the determined measurement position. Then, the robot 10 is actually brought into contact with the target object 41 at the measurement position, and the target object 41 is based on the force information and the position detection information measured by the force sensor 33 and the position detection sensor 34 during the contact operation. Measure the detailed position of. The position and orientation of the target object 41 are recognized by creating map information. As a result, even the target object 41 wrapped in a plastic bag or cushioning material, whose position and posture cannot be accurately recognized only by the imaging device 31, can be appropriately recognized for the gripping operation. Therefore, the number of cases in which the target object 41 can be gripped increases, and work efficiency can be improved.

In the present embodiment, the contact operation with respect to the target object 41 is performed once, but when the target object 41 cannot be recognized by one contact operation, the target object 41 is touched a plurality of times to repeat the measurement to create map information. May be good. Thereby, the accuracy of recognition can be improved.
As described above, in the present embodiment, the surface position measuring unit 202 measures (acquires) the surface position information of the objects (target object 41 and peripheral objects 42), and the measurement position determining unit 204 is based on the surface position information. The measurement position for actually measuring the target object 41 is determined. Further, the force measuring unit 207 and the position measuring unit 209 measure the contact position information when the target object 41 is contacted at the measuring position. Then, the map information creation unit 203 recognizes the position and orientation of the target object 41 based on the surface position information and the contact position information.

In this way, since the contact position information is measured by actually touching the target object 41, the detailed position of the target object 41 can be confirmed. As a result, the position and posture of the target object 41, which is the contents wrapped in the plastic bag or the cushioning material, can be appropriately recognized. That is, even if the actual position and posture of the target object 41 cannot be recognized only from the surface position information, the actual position and posture of the target object 41 can be recognized. Therefore, the number of cases in which the target object 41 can be gripped by the robot 10 increases, and work efficiency can be improved.

Further, the map information creation unit 203 creates map information indicating the spatial position information of the target object 41 based on the surface position information and the contact position information, and based on the created map information, the target object 41 Recognize position and orientation. In this way, since the map information is created in consideration of the position information confirmed by actually touching the target object 41, the position and posture of the target object 41 can be appropriately recognized.
Further, when creating the map information, the map information creation unit 203 first creates the map information based on the surface position information, and then based on the contact position information measured at the measurement position, based on the surface position information. Update the created map information. In this way, the map information is created by integrating the surface position measurement of the target object 41 by the image pickup device 31 and the position measurement results by the force sensor 33 and the position detection sensor 34. As a result, it is possible to appropriately supplement the insufficient information in the rough map information created based on the surface position information and create the final map information. Therefore, even if the position and posture of the target object 41 cannot be obtained only from the information acquired from the image pickup device 31, the shape of the target object 41 can be appropriately recognized and operated by the robot 10.

Further, the measurement position determination unit 204 detects a position where the surface has a convex shape based on the surface position information, and determines the detected position as the measurement position. Therefore, it is possible to select and actually measure the portion of the target object 41 having a high existence probability from the mixture of the target object 41 and the peripheral object 42.
Further, the surface position measuring unit 202 acquires the surface position information based on the image information obtained by imaging the target object 41 and the peripheral object 42. Therefore, the surface position information can be easily acquired.

Further, the force measuring unit 207 measures (acquires) the reaction force when it comes into contact with the target object 41, and the position measuring unit 209 measures (acquires) the position of the contact point with the target object 41. Then, the position measuring unit 209 measures the contact position information by measuring the position of the contact point when the reaction force measured by the force measuring unit 207 reaches a predetermined upper limit value. Therefore, when the target object 41 is wrapped with a soft member such as a plastic bag, the member surrounding the target object 41 can be crushed and directly contacted with the target object 41 to measure the contact position information. .. At that time, if the force that does not destroy the target object 41 that is the contents by crushing the plastic bag or the cushioning material is set as the upper limit value, the position and posture of the target object 41 can be accurately recognized.

The gripping position determining unit 210 determines the gripping position where the robot 10 can grip the target object 41 based on the position and posture of the target object 41 recognized from the map information created by the map information creating unit 203. In this way, the gripping position determining unit 210 determines the gripping position based on the map information created in consideration of the position information confirmed by actually touching the target object 41. Therefore, the gripping position can be determined with high accuracy, and the gripping operation by the robot 10 can be successful with high probability. Further, at this time, since the gripping position determining unit 210 determines the position having a space around which the gripping portion of the robot 10 can be inserted as the gripping position, the gripping position can be appropriately determined.

(Second embodiment)
Next, a second embodiment of the present invention will be described.
In the second embodiment, a gripping force measuring sensor that measures the gripping force during the gripping operation is further added to the first embodiment described above, and the gripping force information measured by the gripping force measuring sensor is used. It is configured to create map information in more detail. Specifically, first, as in the first embodiment described above, the surface position information of the target object 41 and the peripheral object 42 is measured by using the image pickup apparatus 31. Next, as in the first embodiment described above, the measurement position of the target object 41 is determined based on the surface position information, and the contact operation with respect to the target object 41 is performed at the measurement position. Then, during the contact operation, the force sensor 33 and the position detection sensor 34 are used to measure the position information when the robot 10 is in contact with the target object 41, and the target object 41 is gripped based on the measured position information. Determine the position. In the present embodiment, the gripping width information of the target object 41 is measured by using the gripping force measurement sensor and the position detection sensor 34 during the gripping operation at the gripping position. Then, by integrating the measurement results of the surface position information, the position information, and the gripping width information to create map information, the shape for gripping the target object 41 is recognized.

In the first embodiment, the target object 41 is recognized from the information before the robot 10 grips the target object 41. Therefore, when the target target object 41 is mixed in a plurality of types of objects having different sizes, it is not always possible to grasp the target target object 41. Therefore, in the second embodiment, the gripping width measured during the gripping operation is collated with the component DB, and the size (width) of the target target object 41 is referred to, so that the object gripped by the robot 10 by the gripping operation is obtained. It is determined whether or not the target object 41 is the target object. In addition, map information is created including the gripping operation.

FIG. 6 is a configuration example of the robot system 100 in the second embodiment. In FIG. 6, the parts having the same configuration as that of the first embodiment described above are designated by the same reference numerals as those in FIG. 1, and the parts having different configurations will be mainly described below.
The robot system 100 in this embodiment is different from FIG. 1 in that the gripping force measurement sensor 35 is attached to the tip of the end effector 11 of the robot 10. The gripping force measuring sensor 35 is composed of a strain gauge and a piezoelectric element, measures the gripping force when the robot 10 is gripping the target object 41, and transmits the gripping force sensory information as the measurement result to the information processing device 20'. Output. The gripping force measurement sensor 35 may be a 6-axis force sensor or a 3-axis force sensor. Further, the gripping force measurement sensor 35 may be a one-dimensional pressure sensor or a contact sensor that determines the presence or absence of contact.

FIG. 7 is a functional block diagram of the information processing device 20'in this embodiment. The information processing device 20'shown in FIG. 7 is different from the information processing device 20 shown in FIG. 2 in that a gripping force sensory information acquisition unit 211 and a gripping force measuring unit 212 are added.
The gripping force sense information acquisition unit 211 acquires the gripping force sense information output from the gripping force measurement sensor 35, and outputs the acquired gripping force sense information to the gripping force measurement unit 212. The gripping force sense information acquisition unit 211 is composed of, for example, a memory (RAM). The gripping force measuring unit 212 measures the force applied to the robot 10 when the end effector 11 of the robot 10 grips an object based on the gripping force sense information input from the gripping force sense information acquisition unit 211. The gripping force measuring unit 212 outputs the measured gripping force information to the position information measuring unit 209. A recognition unit that performs processing to recognize the position and orientation of the target object 41 by the surface position measurement unit 202, map information creation unit 203, measurement position determination unit 204, force measurement unit 207, position measurement unit 209, and grip force measurement unit 212. It constitutes 220'.

FIG. 8 is a flowchart showing a robot control processing procedure executed by the information processing device 20'. The process shown in FIG. 8 is the same as that of FIG. 4 except that the processes of S21 to S26 are added between the processes of S14 and S15 of FIG. Therefore, the different parts of the processing will be mainly described below.
In S21, the gripping force sense information acquisition unit 211 acquires the gripping force sense information output by the gripping force measurement sensor 35. Further, the position information acquisition unit 208 acquires the position detection information output by the position detection sensor 34 in synchronization with the acquisition of the grip force sense information by the gripping force sense information acquisition unit 211. Next, in S22, the gripping force measuring unit 212 measures the gripping force information when the robot 10 grips the target object 41 based on the gripping force sense information acquired by the gripping force sense information acquisition unit 211 in S21. Further, the position measurement unit 209 measures the position of the tip of the end effector 11 when the robot 10 grips the target object 41 based on the position detection information acquired by the position information acquisition unit 208 in S21. That is, the position measuring unit 209 measures the gripping width information of the target object 41 gripped by the robot 10.

The gripping force information and the gripping width information are repeatedly measured while the robot 10 is performing the gripping operation on the target object 41. The gripping operation is performed within a range in which the magnitude of the force applied to the robot 10 when gripping the target object 41 does not exceed a preset upper limit value. Whether or not the magnitude of the force applied to the robot 10 exceeds the upper limit value is determined based on the gripping force information measured in S22. During the gripping operation, the force applied to the robot 10 increases according to the gripping force on the target object 41. Therefore, the upper limit value of the above force is set to such a force that the target object 41 is not damaged or otherwise caused by the gripping operation. For example, when the target object 41 is a toner cartridge wrapped in a plastic bag, a force that does not crush the plastic bag and break the toner cartridge is set as the upper limit value.

When the robot 10 crushes the plastic bag and comes into contact with the surface of the toner cartridge and the force applied to the robot 10 reaches the upper limit value, the information processing device 20'stops the gripping operation of the robot 10. The gripping width of the robot 10 by the end effector 11 at the time when the gripping operation is stopped can be recognized as the actual size of the target object 41.
In S23, the map information creating unit 203 updates the map information created in S10 based on the gripping force information and the gripping width information measured in S22, and creates new map information. In this S23, the map information creation unit 203 uses the gripping width measured in S22 as information indicating the size of the target object 41. In this way, the map information creation unit 203 creates map information in consideration of the gripping width information obtained by actually gripping the target object 41, and recognizes the target object 41.

The map information creation unit 203 first performs unified processing for each coordinate system. The unification of the coordinate systems of the robot 10, the image pickup device 31, and the force sensor 33 is as described in the first embodiment. Therefore, the coordinate system of the gripping force measurement sensor 35 will be described below. In the present embodiment, the displacement from the tip coordinate system Σf of the robot 10 to the gripping force measurement sensor coordinate system Σg is defined as (GX, GY, GZ). Further, a 3 × 3 rotation matrix representing the posture of the gripping force measurement sensor 35 is defined as GM. Here, assuming that the gripping force applied to the robot 10 during the gripping operation as seen from the world coordinate system Σw is a 4 × 4 matrix GT, the matrix GT can be obtained by the following equation (4). Using this equation (4), the direction of the gripping force can be considered in the world coordinate system Σw.

When creating map information, the map information creation unit 203 unifies the coordinate systems of a plurality of different sensors by using the above equation (4) and the above equations (1) to (3). Then, when the coordinate system is unified, the map information creation unit 203 creates map information using the information obtained from each sensor. In this S23, the map information creation unit 203 performs a process of adding the result of actually grasping the target object 41 to the map information created in S10.
In S24, the CPU 21 determines that the object held by the robot 10 is the target object 41 based on the map information created in S23 and the shape information of the target object 41 obtained by referring to the component DB. Determine if it exists. At the time of determination, the possible width of the target object 41 is estimated in consideration of the degree of deformation of the peripheral object 42 surrounding the target object 41. Then, when the gripping width of the target object 41 is within the estimated possible width, the CPU 21 determines that the object gripped by the robot 10 is the target object 41. If it is determined that the object being gripped is the target object 41, it is determined that the operation will be continued, and the process proceeds to S25. On the other hand, when it is determined that the gripped object is not the target object 41, the process returns to S12, the gripping position is determined again, and the creation of map information is repeated.

In S25, the action planning unit 205 determines the action plan of the robot 10 for realizing the operation on the grasped target object 41. When deciding the action plan, the action planning unit 205 refers to the information of the part type and the work procedure stored in the operation DB based on the gripping position.
Next, in S26, the action planning unit 205 outputs the action instruction information to the robot 10 based on the action plan determined in S25. As a result, the robot 10 conveys the target object 41 gripped by the end effector 11 to a predetermined place according to the work procedure accumulated in the operation DB.
As described above, in the present embodiment, when the target object 41 is recognized by the same method as the first embodiment described above and the target object 41 is gripped, the gripping width of the object gripped by the gripping force measurement sensor 35 is used. To measure. Then, based on the measured gripping width, it is determined whether or not the gripped object is the target object 41, and the map information is created in more detail. As a result, even when a plurality of types of target objects 41 having different sizes exist, they can be recognized with high accuracy. Therefore, the number of cases in which different types of target objects 41 can be accurately grasped increases, and the possibility of continuing the work can be expanded.

Specifically, the gripping force measuring unit 212 measures the gripping width information when the target object 41 is gripped at the gripping position, and the map information creating unit 203 further uses the gripping width information to obtain the position of the target object 41 and Recognize posture. Therefore, the shape for gripping the target object 41 can be recognized in more detail. For example, even when the target object 41 is of a plurality of types having different sizes, the position and posture of the target object 41 can be recognized with higher accuracy, and the possibility of continuing the work by the robot 10 can be expanded. Further, since the gripping width when the target object 41 is actually gripped is measured, the size of the target object 41 can be measured with high accuracy. Therefore, it is possible to appropriately determine whether or not the grasped object is the target object 41.

(Third embodiment)
Next, the third embodiment will be described.
In the third embodiment, with respect to the first and second embodiments described above, when the end effector 11 is inserted into a certain area of occlusion, the created map information is used to interfere with the peripheral object 42. It is possible to make an action plan for the robot 10 that has never happened. Here, the processing procedure for controlling the robot is the same as the flowchart shown in FIG. 4, but the processing in S13 is different.
When the end effector 11 is inserted into a region with occlusion, for example, as shown in FIG. 9A, the region where the target object 41 exists is a peripheral object 42 or the like with respect to the region A0 where the imaging device 31 can image. This is the case when it is in a shielded state. In this case, as shown in FIG. 9B, an area A1 that can recognize the target object 41 and a region A2 that cannot recognize the target object 41 even if the imaging device 31 tries to image the target object 41 are generated. In this state, even if the target object 41 can be gripped, when the target object 41 is transported and taken out, it interferes with the peripheral object 42 that cannot be imaged by the imaging device 31, and the target object being gripped. There is a risk of dropping 41.

Therefore, in the present embodiment, the map information created / updated in S4 and S10 is used in the process of S13. The map information created before grasping the target object 41 includes not only the target object 41 but also information about the peripheral object 42. Therefore, in the action plan when the robot 10 grips the target object 41 and then conveys and takes it out, the space in which the robot 10 can move is limited in consideration of the spatial information in which the peripheral object 42 exists.
When acquiring the position information of the peripheral object 42, a proximity sensor that detects the distance to an approaching object may be used instead of the force sensor 33. In this case, at the measurement position, the contact position information can be measured and the map information can be created by directly acquiring the distance information to the target object 41 by the proximity sensor when the object is in contact with the target object 41. When the proximity sensor is installed at a position close to the force sensor 33 shown in FIGS. 1 and 6, the proximity sensor is the distance from the base of the end effector 11 to the position of the contact point of the robot 10 with respect to the target object 41. Information can be obtained. Further, if the proximity sensors are arranged at a plurality of locations, the distance information at the plurality of locations can be measured at one time to create the map information. Therefore, when there is occlusion in the measurement by the imaging device 31, the operation of the robot 10 that does not interfere with the peripheral objects 42 around the target object 41 is planned in a shorter time in the extraction operation after grasping the target object 41. Can be done.

Further, both the force sensor 33 and the proximity sensor may be used for measurement. By using both the force sensor 33 and the proximity sensor, it is possible to create map information by measuring a plurality of locations at once with the proximity sensor while measuring the position with high accuracy by the measurement by the force sensor 33. Can be done. As a result, the target object 41 and the peripheral object 42 can be recognized in a shorter time, and the action of the robot 10 can be appropriately planned. The proximity sensor may be an infrared sensor, or a magnetic type or ultrasonic type sensor may be used.
By doing so, when the target object 41 held by the robot 10 is transported, the possibility that the target object 41 interferes with the peripheral objects 42 during transportation is reduced, so that the target object 41 may fall during transportation. Can be reduced and work efficiency can be improved.

(Modification)
In each of the above embodiments, when the robot 10 fails in the gripping operation, the gripping position determining unit 210 excludes the gripping position in which the gripping operation fails from the gripping position candidates for the next gripping operation, and grips from another place. The position may be determined. By creating the map information in this way, it is possible to perform processing for preventing the same failure in the robot operation. As a result, the working time of the robot 10 can be shortened.

Further, in each of the above embodiments, the material of the target object 41 is recognized from the force sense information when the robot 10 comes into contact with the target object 41, and the recognized information is used to grip the target object 41. Good. For example, when the target object 41 is wrapped in a plastic bag as in each of the above embodiments, when the robot 10 comes into contact with the plastic bag, a slight force is applied to the robot 10. In that case, since the space where the plastic bag exists can be known from the force applied to the robot 10, the target object 41 may be operated (conveyed) by grasping the plastic bag. As a result, even when the target object 41 itself cannot be grasped, the target object 41 can be operated, and the possibility of continuing the work can be improved.

Further, in each of the above embodiments, after the detailed position measurement of the target object 41 using the force sensor 33, the surface position measurement of the target object 41 by the imaging device 31 may be performed again. When the force sensor 33 makes a plurality of measurements, the deviation may be accumulated due to the sensor drift. Therefore, the map information can be created by correcting the above-mentioned deviation by appropriately measuring the surface position by the imaging device 31. As a result, the influence of the deviation of the sensor drift can be suppressed and the map information can be created more accurately, so that the possibility of performing the work accurately can be improved.

Further, in each of the above embodiments, the measurement by the image pickup apparatus 31 may be performed a plurality of times at first, and the surface position information may be measured in more detail. As a result, the number of subsequent measurements by the force sensor 33 and the proximity sensor can be reduced. As a result, the target object 41 can be recognized in a shorter time, and the action of the robot 10 can be planned.
Further, in each of the above embodiments, a case where the target object 41 whose position and posture are recognized is grasped and conveyed by the robot has been described. However, when the purpose is to recognize the position and posture of the target object 41, it is not necessary to perform operations by the robot such as grasping and transporting. The effect of the present invention is that the position and orientation of the target object 41 can be appropriately recognized even when the target object 41 cannot be accurately recognized only by the surface position information. Therefore, information on the position and posture of the recognized target object 41 is presented (displayed) to the operator, and the target object 41 is inspected based on the recognized position and posture of the target object 41. May be good. Here, the inspection of the target object 41 is to confirm the shape of the target object 41 and to confirm whether it is in a desirable position.

Further, the robot may perform an operation without grasping the target object 41 based on the recognized position and posture of the target object 41. Here, the operation without grasping is an operation of contacting the target object 41 without grasping and changing the position of the target object 41. As described above, when the robot performs an operation on the target object 41, the operation may be an operation involving gripping or an operation not accompanied by gripping.
Further, the number of robots for acquiring surface position information and contact position information may be one or a plurality. Further, the robot for acquiring the surface position information and the contact position information and the robot for performing operations such as grasping and transporting the target object 41 may be the same or different.

Further, in each of the above embodiments, the case where the robot 10 is an assembly type robot has been described, but the robot 10 is not limited to the assembly type robot and may be a mobile robot.
Further, in each of the above embodiments, the case where the map information is three-dimensional information has been described, but the map information may be two-dimensional information. Further, the map information may be created based only on two pieces of information, the force information measured by the force measuring unit 207 and the position information measured by the position measuring unit 209.
Further, in each of the above embodiments, the image pickup device 31 and the light source 32 may be a laser range finder device. Further, stereo measurement may be performed using two image pickup devices 31 without using the light source 32, or only one image pickup device 31 may be used without using the light source 32. That is, the configuration may be such that the surface position information for recognizing the surface positions of the target object 41 and the peripheral object 42 can be measured.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

10 ... Robot, 11 ... End effector, 20, 20'... Information processing device, 31 ... Imaging device, 32 ... Light source, 33 ... Force sensor, 34 ... Position detection sensor, 35 ... Gripping force measurement sensor, 41 ... Target object , 42 ... peripheral objects, 100 ... robot system

Claims (16)

An information processing device that recognizes an object consisting of an object and surrounding objects.
An acquisition method for acquiring surface position information of an object,
A determination means for determining a measurement position for actually measuring the object based on the surface position information acquired by the acquisition means.
A measuring means for measuring contact position information when the object is touched at the measuring position determined by the determining means, and a measuring means.
A creating means for creating map information indicating the spatial position information of the object based on the contact position information measured by the measuring means, and
A recognition means for recognizing the position and orientation of the object based on the map information created by the creation means is provided .
The measuring means includes a distance acquiring means for acquiring the distance to the approaching object.
Based on the distance obtained by the distance obtaining means when in contact with the object, the information processing apparatus characterized that you measure the contact position information. The creation means
A first creating means for creating the map information based on the surface position information acquired by the acquiring means, and
The information according to claim 1, further comprising a second creating means for updating the map information created by the first creating means based on the contact position information measured by the measuring means. Processing equipment. The determination means is
The first or second claim is characterized in that a position where the surface of the object has a convex shape is detected based on the surface position information acquired by the acquisition means, and the detected position is determined as the measurement position. The information processing device described. The acquisition means
The information processing apparatus according to any one of claims 1 to 3, wherein the surface position information is acquired based on the image information obtained by capturing an image of the object. The measuring means
A means for obtaining a reaction force when it comes into contact with the object, and a means for obtaining the reaction force .
Further provided with a position acquisition means for acquiring the position of the contact point with the object,
When the reaction force acquired by the reaction force acquisition means reaches a predetermined upper limit value, the position of the contact point acquired by the position acquisition means is measured as the contact position information. The information processing apparatus according to any one of claims 1 to 4. Any of claims 1 to 5 , further comprising a second determining means for determining a gripping position where the robot can grip the object based on the position and posture of the object recognized by the recognition means. The information processing apparatus according to item 1. The second determination means is
The information processing apparatus according to claim 6 , wherein a position having a space in which the grip portion of the robot can be inserted is determined as the grip position. A second measuring means for measuring gripping width information when the object is gripped at the gripping position determined by the second determining means is further provided.
The recognition means
The information processing apparatus according to claim 6 or 7 , wherein the position and orientation of the object are recognized by further using the grip width information measured by the second measuring means. The robot control device according to any one of claims 1 to 8 , further comprising a control means for controlling the operation of a robot that executes work on the object. A robot system according to claim 9 , wherein the robot control device includes at least one of a sensor and the robot. The sensor robot system according to claim 1 0, characterized in that it comprises a visual sensor for obtaining image information by imaging the object. The sensor robot system according to claim 1 0 or 1 1, characterized in that it comprises a force sensor to obtain the reaction force when in contact with the object. The sensor robot system according to any one of claims 1 0 to 1 2, characterized in that it comprises a position detection sensor for acquiring the position and orientation of the robot. The sensor robot system according to any one of claims 1 0 to 1 3, characterized in that it comprises a force sensor to obtain the reaction force when gripping the object. Steps to acquire surface position information of an object consisting of a target object and surrounding objects,
Based on the acquired surface position information, a step of determining a measurement position for actually measuring the object, and
A step of measuring contact position information when the object is touched at the determined measurement position, and
Based on the measured contact position information, a step of creating map information indicating the spatial position information of the object, and
Based on the map information created, viewed contains a recognizing step the position and orientation of the object,
The step of measuring the contact position information includes a step of acquiring the distance to the approaching object.
An information processing method characterized in that the contact position information is measured based on the distance acquired in the step of acquiring the distance when in contact with the object . A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 8 .

Images