JP6541397B2 - INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PROGRAM - Google Patents

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program, and in particular, is suitable for use in operating an object using a robot.

  Heretofore, in robot manipulation of objects, it has been common to take out the objects being aligned from fixed positions. On the other hand, Patent Document 1 discloses a technique for gripping an object using a force sensor and a vision sensor (vision sensor). Specifically, in the technology described in Patent Document 1, the robot arm is moved based on the position information of the object detected by the vision sensor to bring it close to the object, and the force sensor attached to the tip of the robot arm Correct the position information of the object based on the output.

JP, 2012-11531, A

  However, in the method described in Patent Document 1, the robot arm is brought close to the object based on the position information of the object obtained by the detection by the vision sensor, and the position information of the object based on the information output from the force sensor Correct the Therefore, there is a problem that it can not cope with the case where the displacement of the position information of the object is too large. Further, in the method described in Patent Document 1, an error in the position of the object is detected only when the object is gripped by the robot arm. Therefore, even when the robot does not perform a predetermined operation in the manipulation after gripping the target by the robot arm, there is a problem that it can not be detected.

  The present invention has been made in view of the above problems, and it is an object of the present invention to reliably detect that a robot has not performed a predetermined operation when manipulating an object using a robot. Do.

The information processing apparatus of the present invention is an information processing apparatus for performing a process for manipulating the object by a robot,-out movement of the object, and the movement of the surrounding objects are objects in the neighborhood of the object Based on at least one of force and torque generated in the robot when operating the object, and determining whether the robot is operating the object normally It has a determination means to perform.

  According to the present invention, when manipulating an object using a robot, it is possible to reliably detect that the robot is not performing a predetermined operation.

It is a figure which shows the 1st example of a structure of a robot system. It is a figure showing the 1st example of composition of an information processor. It is a flow chart explaining the 1st example of processing of an information processor. It is a flowchart explaining the process of step S1200. It is a flowchart explaining the 1st example of processing of Step S1600. It is a flowchart explaining the process of step S1620. It is a flow chart explaining processing of Step S1640. It is a flow chart explaining processing of Step S1650. It is a figure explaining the 1st example of the contents of unusual judging processing. It is a figure which shows the 2nd example of a structure of a robot system. It is a flowchart explaining the 2nd example of processing of Step S1600. It is a flow chart explaining processing of Step S1620 '. It is a flowchart explaining the process of step S1650 '. It is a figure explaining the 2nd example of the contents of unusual judging processing. It is a figure which shows the 3rd example of a structure of a robot system. It is a flowchart explaining the 3rd example of processing of Step S1600. It is a flowchart explaining the process of step S1602 ''. It is a flowchart explaining the process of step S1650 ''. It is a figure explaining the 3rd example of the contents of unusual judging processing. It is a figure showing the 2nd example of composition of an information processor. It is a flowchart explaining the 2nd example of processing of an information processor. It is a flowchart explaining the process of step S1300 '.

Hereinafter, embodiments will be described with reference to the drawings.
First Embodiment
First, the first embodiment will be described.
In the present embodiment, the force sensor measures the force and torque applied to the end effector which is the grip portion, and the change in the position and posture of the entire work space is measured by the vision sensor. In this way, even if the object interferes with the surrounding objects, it is possible to detect an abnormal operation of the robot which occurs during manipulation of the object. Here, the target is, for example, a component such as a toner cartridge. Also, the peripheral object is an object that is in the vicinity of the object to be manipulated. For example, in stacked objects, an object not to be held, a container in which the object is placed, and the like are examples of peripheral objects. Manipulation is, for example, a series of operations such as gripping, pulling out, and conveyance. Also, the abnormal movement of the robot means that the robot can not manipulate the object as determined.

FIG. 1 is a diagram showing an example of the configuration of a robot system. In the present embodiment, the apparatus shown in FIG. 1 is used to detect an abnormality in the operation of the robot. However, FIG. 1 is an example of a structure of an apparatus, and does not limit the scope of application of this invention.
The robot 101 includes devices such as a manipulator and an end effector such as a robot hand. The robot 101 performs the action determined by the information processing apparatus 104 and operates the object 106. Here, the action is an operation of the robot 101 for operating the object 106. Specifically, the action is, for example, gripping or transporting the object 106 by the robot 101.

  The imaging device (vision sensor) 102 includes, for example, a camera, a sensor (for example, a photodiode) that detects light, and the like. The imaging device 102 acquires image information of the object 106 and the peripheral object 107. The image information acquired by the imaging device 102 is processed by the information processing device 104. The image information may be a moving image or a plurality of still images captured at different timings.

  The light source 103 has, for example, a projector. The light source 103 illuminates the object 106 and the peripheral object 107 with uniform illumination light or pattern light by projecting visible light or projecting infrared light using a laser device. Alternatively, a laser range finder including the imaging device 102 and the light source 103 may be used. In addition, stereo measurement may be performed using two imaging devices 102 without using the light source 103. In addition, as long as information for measuring the positions and orientations of the object 106 and the peripheral object 107 can be obtained, only one imaging device 102 may be used without using the light source 103.

  The information processing apparatus 104 has a computer such as a computer and an auxiliary storage device such as a hard disk. The information processing device 104 is mutually connected to the robot 101, the imaging device 102, the light source 103, and the force sensor 105 via an interface device. The information processing apparatus 104 can communicate with these devices as needed, and controls the operation of these devices. Further, the information processing apparatus 104 holds information such as the three-dimensional shapes of the object 106 and the peripheral object 107, and the work procedure of the robot 101.

  In the present embodiment, the information processing apparatus 104 detects an abnormality in the operation of the robot 101 based on the force sense information input from the force sensor 105 and the image information input from the imaging device 102. The information processing apparatus 104 determines whether or not the object 106 can be gripped based on the information held in the information processing apparatus 104 and the force sense information, and the force on the object 106 and the peripheral object 107. To detect if it is

Further, the information processing apparatus 104 recognizes the positions and orientations of the target object 106 and the peripheral object 107 based on the information held in the information processing apparatus 104 and the image information. Then, based on the recognized result, the information processing apparatus 104 detects whether the object 106 is moving in synchronization with the holding unit and whether the peripheral object 107 has movement.
The information processing apparatus 104 detects an abnormality in the operation of the robot 101 during manipulation by these detections, and when there is an abnormality, prepares a plan of the action of the robot 101, and indicates that the robot 101 operates according to the prepared plan. An instruction is output to the robot 101.

  The force sensor 105 is attached to the root of the end effector of the robot 101. The force sensor 105 of the present embodiment is configured using a strain gauge or a piezoelectric element, and is a six-axis force that measures the force and torque applied to the end effector of the robot 101 when the robot 106 grips the object 106. It is a sensor (Force / Torque sensor). The force sense information acquired by the force sensor 105 is processed by the information processing apparatus 104. When the object 106 is gripped by the robot 101, the object 106 does not contact the root portion of the end effector of the robot 101 (that is, the portion to which the force sensor 105 is attached).

The object 106 is an object to be manipulated. The object 106 is, for example, a toner cartridge, which is assumed to be gripped and transported by the robot 101.
The peripheral object 107 is an object in the periphery of the object 106 to be subjected to manipulation of manipulation. The peripheral object 107 is, for example, an object (object which can be an object) which is not a gripping target in the stacked objects 106, a container in which the object 106 is placed, or the like.

FIG. 2 is a diagram showing an example of a functional configuration of the information processing apparatus 104. As shown in FIG. In FIG. 2, among the functions possessed by the information processing apparatus 104, functions for realizing detection of an abnormality in the operation of the robot 101 during manipulation of the object 106 are shown.
The robot 101 operates in accordance with the operation command from the action planning unit 205.
As described above, the imaging device 102 captures the object 106 and the peripheral object 107. The object 106 and the peripheral object 107 may be illuminated by the light source 103. Image information captured by the imaging device 102 is acquired by the image acquisition unit 201.

The information processing apparatus 104 is a robot control apparatus including an image acquisition unit 201 to an operation position determination unit 210 described later. In the configuration of the information processing apparatus 104, the abnormality detection unit 301 detects an abnormality in the operation of the robot 101.
The force sensor 105 measures the force and torque applied to the end effector of the robot 101 as force sense information during a period in which the object 106 is gripped and transported (that is, the object 106 is being manipulated). The force sense information measured by the force sensor 105 is obtained by the force sense information acquisition unit 207.

The image acquisition unit 201 acquires image information captured by the imaging device 102. This image information is used by the position / posture change measurement unit 202 and the position / posture measurement unit 209. The image acquisition unit 201 is configured using, for example, a capture board or a RAM (memory).
The part database (DB) 203 holds information on the part type and shape of the object 106 and the peripheral object 107 to be manipulated. The position / posture change measurement unit 202 and the position / posture measurement unit 209 refer to information in the component database 203 as appropriate. The information on the shapes of the object 106 and the peripheral object 107 is, for example, model data such as CAD data and CG polygons. Alternatively, information on the shapes of the object 106 and the peripheral object 107 may be configured from a set of two-dimensional images obtained by observing the object 106 and the peripheral object 107 from multiple directions.

  The abnormality detection unit 301 includes a position / posture change measurement unit 202, a force / torque measurement unit 208, and an abnormality determination unit 204. The abnormality detection unit 301 receives as input the image information from the image acquisition unit 201 and the force sense information from the force sense information acquisition unit 207, and determines whether or not there is an abnormality in the operation of the robot 101 during manipulation of the object 106. And outputs the result of the determination to the action planning unit 205.

  The position / posture change measurement unit 202 measures (recognizes) a change in position and a change in posture of the object 106 and the peripheral object 107 using the image information acquired by the image acquisition unit 201 as an input. Specifically, the position / posture change measurement unit 202 detects the object 106 and the peripheral object 107 based on the information on the part type and shape of the object 106 and the peripheral object 107 accumulated in the component database 203 and the image information. Derive the change of position and the change of posture. Then, the position / posture change measurement unit 202 outputs position / posture change information indicating a change in position and a change in posture of the object 106 and the peripheral object 107 to the abnormality determination unit 204.

The force sense information acquisition unit 207 obtains force sense information sent from the force sense sensor 105. This force sense information is used by the force / torque measurement unit 208. The force sense information acquisition unit 207 is configured using, for example, a RAM (memory).
The force / torque measurement unit 208 recognizes the force and torque applied to the end effector of the robot 101 during manipulation of the object 106, using the force sense information acquired by the force sense information acquisition unit 207 as an input. Force / torque information indicating the result of the recognition is output to the abnormality determination unit 204.

  The abnormality determination unit 204 receives the position / posture change information from the position / posture change measurement unit 202 and the force / torque information from the force / torque measurement unit 208, and operates the robot 101 during manipulation of the object 106. It is determined whether or not there is an abnormality in the The result of this determination is output to the action planning unit 205.

  The position / posture measurement unit 209 measures (recognizes) the position and orientation of the object 106 to be grasped among the stacked objects, using the image information acquired by the image acquisition unit 201 as an input. Specifically, the position and orientation measurement unit 209 determines the position and orientation of the object 106 based on the information on the part type and shape of the object 106 accumulated in the parts database 203 and the image information. The position / posture measurement unit 209 outputs position / posture information indicating the positions and orientations of the object 106 and the peripheral object 107 to the operation position determination unit 210. For example, when the shape of the object 106 is a sphere, measurement (recognition) of the posture of the object 106 is unnecessary.

  The operation position determination unit 210 receives the position / posture information from the position / posture measurement unit 209 as input, and determines the grip position and the release position of the object 106 as the operation position of the object 106. The holding position is the position at which the end effector of the robot 101 holds the object 106. Further, the release position is a position at which the object 106 held by the end effector of the robot 101 is released. The operation position determined by the operation position determination unit 210 is output to the action planning unit 205.

  The operation position determination unit 210 is a processing unit that is first executed in the process of operating the object 106 by the robot 101, and the process of the abnormality detection unit 301 is performed after the robot 101 starts to move according to the operation command. When the processing of one designated object 106 is finished and a new object 106 is to be operated, the processing of the position / posture measurement unit 209 and the operation position determination unit 210 is performed again.

The operation database (DB) 206 holds information on part types and work procedures that the action planning unit 205 refers to when planning the action of the robot 101 with respect to the object 106. The action planning unit 205 refers to the information of the operation database 206 as appropriate.
The action planning unit 205 uses the result of the determination from the abnormality determination unit 204 and the operation position from the operation position determination unit 210 as input to plan the action of the robot 101. The action planning unit 205 refers to the information of the part type and the work procedure held in the operation database 206 to plan the action of the robot 101 with respect to the object 106. The planned action is output to the robot 101 as an action command.

FIG. 3 is a flowchart for explaining an example of processing of the information processing apparatus 104 for realizing the robot system. The procedure of the process will be described below with reference to FIG.
In step S1000, initialization of the robot system in the present embodiment is performed. Specifically, when the user performs an operation for activating the robot system, a program for realizing the robot system is loaded to the information processing apparatus 104, expanded on a memory (not shown), and becomes executable. . In addition, the robot 101, the imaging device 102, and the force sensor 105 also become usable by reading the device parameters and returning to the initial position.

Next, in step S1100, the image acquisition unit 201 acquires image information on the object 106 and the peripheral object 107 using the imaging device 102.
Next, in step S1200, the position and orientation measurement unit 209 measures the position and orientation of the object 106 to be gripped. FIG. 4 is a flowchart for explaining an example of the detailed process of step S1200 of FIG. Hereinafter, the procedure of the process from step S1201 to step S1203 corresponding to step S1200 will be described according to the flowchart of FIG.

  In step S <b> 1201, the position / posture measurement unit 209 determines the component type of the object 106 and the peripheral object 107. In order to determine the component types of the object 106 and the peripheral object 107, for example, pattern matching is performed. The position / posture measurement unit 209 performs pattern matching based on the information on the part type of the object 106 stored in the parts database 203 and the image information acquired by the image acquisition unit 201 to obtain the object 106 and the object 106. The component type of the peripheral object 107 is determined. The pattern matching method includes, for example, correlation based matching. The position / posture measurement unit 209 compares each of the target object 106 and the peripheral object 107 included in the image information by comparing a plurality of types of component patterns stored as component type information in the component database 203 with the image information. , Estimate the most similar part pattern. The position / posture measurement unit 209 determines the part type of the object 106 and the peripheral object 107 in this manner. However, the pattern matching may be another process as long as it is a method capable of determining the component type of the object 106 and the peripheral object 107.

  In step S1202, the position / posture measurement unit 209 determines the positions and orientations of the object 106 and the peripheral object 107 based on the part type determined in step S1201. In order to determine the position and orientation of the object 106 and the peripheral object 107, for example, model fitting is performed. For example, the position / posture measurement unit 209 detects an object 106 from image information captured by the imaging device 102 and acquired by the image acquisition unit 201 in a state where the object 106 and the peripheral object 107 are irradiated with uniform illumination light. And the edges of the peripheral object 107 are extracted. Then, the position / posture measurement unit 209 performs fitting processing of the extracted edge of the object 106 and the peripheral object 107 and the edge model based on the information (CAD data etc.) of the shapes of the object 106 and the peripheral object 107. Do. As described above, the information (CAD data etc.) of the shapes of the object 106 and the peripheral object 107 is stored in the parts database 203 for each part type. The position / posture measurement unit 209 determines the positions and orientations of the object 106 and the peripheral object 107 based on the result of the fitting process.

  Further, the position / posture measurement unit 209 may determine the positions and orientations of the object 106 and the peripheral object 107 as follows, for example, without performing as described above. First, on the surface of the object 106 and the peripheral object 107, the position / posture measurement unit 209 acquires image information captured by the imaging device 102 and captured by the image acquisition unit 201 in a state in which multiline pattern light is irradiated. Detect the projected pattern. Then, based on the principle of triangulation, the position / posture measurement unit 209 derives distance point group data indicating the distances to the points on the surface of the object 106 and the peripheral object 107. The position / posture measurement unit 209 performs fitting processing between the derived distance point group data and the point group model using an IPC (Iterative Closest Point) algorithm, and the position and orientation of the object 106 and the peripheral object 107. To determine

  Next, in step S1203, based on the result (the position and posture of the object 106 and the peripheral object 107) determined in step S1202, the position / posture measurement unit 209 detects the missing or missing object 106 and the peripheral object 107. Perform product judgment processing. For example, the loss / out-of-stock determination process is performed as follows. The position / posture measurement unit 209 partially evaluates the matching degree between the edge of the object 106 and the peripheral object 107 extracted from the image information and the edge model in the model fitting in step S1202. Then, the position / posture measurement unit 209 determines that there is a defect / out-of-stock in the portion when these do not clearly match. After the process of step S1203 ends, the process proceeds to step S1300 of FIG.

  In step S1300, the operation position determination unit 210 determines the operation position of the object 106 based on the result (the position and orientation of the object 106 to be grasped) measured in step S1200. In consideration of the shape of the end effector of the robot 101, the gripping position for operating the object 106 to the target position and the target posture is determined. However, if the end effector does not hold the object 106, for example, if the end effector performs an operation using adsorption or the like, then the adsorption position of the object 106 by the end effector is the operation position. .

  Next, in step S1400, the action planning unit 205 determines an action plan of the robot 101 for realizing the operation position determined in step S1300. Specifically, the action planning unit 205 refers to the information on the part type and the work procedure stored in the operation database 206 based on the operation position determined in step S1300. The planned operation is used when performing the operation of the robot 101 in step S1500.

Next, in step S1500, the action planning unit 205 outputs, to the robot 101, an operation command instructing to operate in the action plan determined in step S1400. Thus, the robot 101 operates in accordance with the action plan determined in step S1400.
Next, in step S1600, an abnormality detection process is performed. FIG. 5 is a flowchart for explaining an example of a detailed process of step S1600 in FIG. The procedures of steps S1610 to S1650 corresponding to step S1600 will be described below with reference to the flowchart of FIG.

In step S1610, the force sense information acquisition unit 207 obtains force sense information regarding the force and torque applied to the end effector of the robot 101 gripping the object 106.
Next, in step S1620, the force / torque measurement unit 208 executes a first derivation process based on the force sense information acquired in step S1610. The first derivation process is a process of deriving force / torque information. The force / torque information is information indicating the force and torque applied to the end effector of the robot 101 during manipulation of the object 106. When the object 106 is gripped and lifted, a force corresponding to the mass of the object 106 is applied to the end effector of the robot 101, and the end effector of the robot 101 is applied by the posture for gripping the object 106 The torque changes. The force / torque measurement unit 208 measures these values based on the values detected by the force sensor 105 using a strain gauge or a piezoelectric element to obtain the end effector of the robot 101 during manipulation of the object 106. Measure the force and torque applied to the

FIG. 6 is a flowchart for explaining an example of the detailed process of step S1620 of FIG. The procedures of steps S1621 to S1622 corresponding to step S1620 will be described below with reference to the flowchart of FIG.
In step S1621, the force / torque measurement unit 208 calculates the resultant force in the vertical direction in the coordinate system of the robot 101 based on the force applied to the end effector when the robot 101 is operating the object 106. The force applied to the end effector when the robot 101 operates the object 106 is obtained from the force sense information. When operating the object 106, a force is applied to the robot 101 by the mass of the object 106. The force measured at this time can be obtained from the measurement values of the X, Y, and Z components of the coordinate system of the force sensor 105. The force / torque measurement unit 208 obtains the resultant force in the vertical direction in the coordinate system of the robot 101 based on these components. By doing this, the force applied to the robot 101 in the direction of the gravitational acceleration can be determined, and can be used in determining whether or not the object 106 can be gripped. However, the resultant force in the vertical direction in the coordinate system of the robot 101 may be calculated by another method as long as the mass of the object 106 held by the robot 101 is obtained.

  In step S1622, the force / torque measurement unit 208 calculates an estimated torque based on the torque applied to the end effector when the robot 101 is operating the object 106. The torque applied to the end effector when the robot 101 operates the object 106 is obtained from the force sense information. When operating the object 106, the robot 101 is torqued based on the moment of inertia of the object 106. The torque measured at this time is obtained with respect to the rotational directions α, β, γ of the respective axes of the X, Y, Z components of the coordinate system of the force sensor 105. The force / torque measurement unit 208 calculates, based on the values of these components, a torque corresponding to the posture in which the robot 101 grips the object 106 as an estimated torque. For example, the force / torque measurement unit 208 calculates, as an estimated torque, a torque with respect to an axis where the largest torque is generated from the position and posture of the object 106. By doing this, the maximum torque applied to the robot 101 can be obtained, and can be used for determining whether or not the object 106 can be gripped. However, the estimated torque may be calculated by another method if the torque applied to the robot 101 is obtained. Moreover, it is not necessary to necessarily perform both processes of step S1621 and step S1622, and any one process may be performed. After the process of step S1622, the process proceeds to step S1630 of FIG.

In step S1630, the image acquisition unit 201 acquires the image information captured by the imaging device 102.
Next, in step S1640, the position / posture change measurement unit 202 executes a second derivation process. The second derivation process is a process of deriving changes in the position and posture of the object 106 and the peripheral object 107. FIG. 7 is a flowchart for explaining an example of the detailed process of step S1640 of FIG. Hereinafter, the procedure of the process of step S1641 to step S1643 corresponding to step S1640 will be described according to the flowchart of FIG.

  In step S1641, the position / posture change measurement unit 202 determines the component type of the object 106 and the peripheral object 107. In order to determine the component types of the object 106 and the peripheral object 107, for example, pattern matching is performed. The position / posture change measurement unit 202 performs pattern matching based on the information on the part type of the target object stored in the part database 203 and the image information acquired by the image acquisition unit 201, thereby obtaining the object 106 and the object. The component type of the peripheral object 107 is determined. The pattern matching method includes, for example, correlation based matching. This method can be realized by, for example, the same method as described in the process of step S1201, and thus the detailed description thereof is omitted here. The pattern matching may be another process as long as it is a method capable of determining the component type of the object 106 and the peripheral object 107.

  Next, in step S1642, the position / posture change measurement unit 202 determines the position and orientation of the object 106 and the peripheral object 107 based on the result (the type of the component of the object 106 and the peripheral object 107) determined in step S1641. To determine In order to determine the position and orientation of the object 106 and the peripheral object 107, for example, model fitting is performed. This method can be implemented, for example, by the same method as described in the process of step S1202, and thus the detailed description thereof is omitted here.

  Next, in step S 1643, the position / posture change measurement unit 202 determines the positions of the object 106 and the peripheral object 107 based on the result (the position and orientation of the object 106 and the peripheral object 107) determined in step S 1642. Posture change determination processing is performed. The position / posture change determination process is performed, for example, as follows. The position / posture change measurement unit 202 compares the feature quantities of the distance point group data derived in step S1642 with the distance point group data at the time of the previous sampling, and examines the change to obtain the object 106 and the peripheral objects. A change in the position and posture of 107 is determined. Also, without doing this, the position / posture change measurement unit 202 derives the optical flow of the entire work space including the object 106 and the peripheral object 107, and detects the motion vector of the object 106 and the peripheral object 107. May be However, as long as it is possible to determine changes in the position and orientation of the object 106 and the peripheral object 107, the position / posture change determination process may be another process. After the process of step S1643 is finished, the process proceeds to step S1650 in FIG.

  In step S1650, abnormality determination unit 204 performs abnormality determination based on the force / torque information derived in step S1620 and changes in the position and orientation of object 106 and peripheral object 107 derived in step S1640. . The abnormality determination unit 204 receives a plurality of pieces of information from the force sensor 105 and the imaging device 102 (vision sensor), and outputs a determination result of the degree of abnormality of the operation of the robot 101. FIG. 8 is a flowchart for explaining an example of the detailed process of step S1650 of FIG. The procedures of steps S1651 to S1654 corresponding to step S1650 will be described below with reference to the flowchart of FIG.

  In step S1651, the abnormality determination unit 204 determines which of the ranges set in advance is the force and torque indicated in the force / torque information derived in step S1620. For example, the abnormality determination unit 204 determines whether or not the force and torque indicated in the force / torque information derived in step S1620 are within the range necessary for gripping or transporting the object 106. Further, the abnormality determination unit 204 is a range in which the force and torque indicated in the force / torque information derived in step S 1620 do not damage the object 106 and the peripheral object 107 even if the range necessary for gripping or transporting the object 106 is broken. It is determined whether or not it is inside.

Parameters such as the value which is a range necessary for gripping or transporting, and the value of a range in which the object 106 and the peripheral object 107 are not damaged are set in advance as follows by a designer, for example. First, it is assumed that the mass of the object 106 is M [g] and the variation thereof is ± m [g]. In this case, the mass of the object 106 is set to M ± m [g]. Therefore, assuming that the gravitational acceleration is g [m / s ² ], the value of the force necessary for the operation of the object 106 can be set as (M ± m) × g [N]. Therefore, if the resultant force in the vertical direction in the coordinate system of the robot 101 calculated in the step S1621 is within the range of the setting value, it is considered that an appropriate force is applied to the robot 101. This set value can be determined in advance by the designer. In addition, a value corresponding to the object 106 may be set in advance as the setting value. Furthermore, this setting value may be set by another method as long as a value necessary for the operation of the object 106 can be set. The abnormality determination unit 204 determines which range the force / torque indicated by the force / torque information derived in step S1620 is by using such a set value.

  Next, in step S1652, the abnormality determination unit 204 determines whether the movement of the object 106 is synchronized with the movement of the end effector (grip unit) of the robot 101. The movement of the object 106 is derived based on, for example, the result of the measurement in step S1640 (change in the position and orientation of the object 106). The motion of the end effector (gripping portion) of the robot 101 uses, for example, forward kinematics based on the information on the position and posture of the tip portion of the end effector of the robot 101 and the information on the angles of the joints of the robot 101 It is derived by performing a calculation. The abnormality determination unit 204 compares the information on changes in the position and orientation of the two (the end effector of the robot 101 and the object 106), thereby to move the object 106 and the end effector (gripping portion) of the robot 101. Determine if there is motion or synchronization. By performing this determination, it is possible to determine how accurately the object 106 is operated by the robot 101.

Next, in step S1653, the abnormality determination unit 204 determines the presence or absence of the movement of the surrounding object 107 based on the result of the measurement in step S1640 (change in the position and orientation of the object 106). By performing this determination, it is possible to determine whether or not the peripheral object 107 is affected by the operation of the object 106 by the robot 101.
Next, in step S1654, the abnormality determination unit 204 performs an abnormality determination process based on the determination results of steps S1651 to S1653.

FIG. 9 is a diagram showing an example of the content of the abnormality determination process in the form of a table. Hereinafter, an example of the abnormality determination process will be described with reference to FIG.
First, the abnormality determination unit 204 determines that the force and torque indicated in the force / torque information derived in step S 1620 correspond to “A. Minimum value of gripping force necessary to grip the object” in FIG. It is determined whether to do. In addition, the abnormality determination unit 204 determines whether or not the force and torque indicated in the force / torque information derived in step S1620 correspond to “B. Value or less of the range in which the object is not damaged” in FIG. Do. This determination is performed according to the result of the determination of the force and torque of step S1651 of FIG.

  Next, the abnormality determination unit 204 determines whether or not “a. The movement of the object is synchronized with the gripping unit” and whether or not “b. The movement of the peripheral object is”. . This determination is performed in accordance with the results of the determinations in steps S1652 and S1653 of FIG.

  In FIG. 9, the applicable case is described as "Yes", and the case not applicable is described as "No". For example, "A or more of the minimum value of the gripping force necessary to grip the object", "B or less of the value of the range in which the object is not broken", "a. And “b. There is movement of peripheral objects”, it is classified into Case 2.

  According to such a determination, the state during manipulation is classified into any of the cases 1 to 6. The gripping force is a holding force of the object 106 by the end effector. For example, when the end effector adsorbs the object 106, the gripping force is replaced by the attracting force.

  Hereinafter, the abnormality determination process for each case will be described. Here, the case where the presence or absence of the abnormality of the operation of the robot 101 is determined using the force applied to the end effector of the robot 101 gripping the object 106 will be described as an example. However, the force may be applied in place of torque. Also, both force and torque may be used.

  In Case 1, the measurement value of the force sensor 105 is equal to or less than the value corresponding to the minimum value of the gripping force required to grip the object 106. Here, in the case where the value of the force necessary for operating the object 106 is set to (M ± m) × g [N] in step S1651, the gripping force necessary for gripping the object 106 is It becomes (M−m) × g [N]. If the object 106 can be manipulated, the measurement result of the force sensor 105 should be at least this value or more. Therefore, the abnormality determination unit 204 determines that the gripping force of the robot 101 is higher than the measurement value of the force sensor 105 in order to operate the object 106 accurately. The action planning unit 205 creates a plan of the action of the robot 101 according to the result of the determination. If the gripping force can not be increased beyond the measurement value of the force sensor 105, the abnormality determination unit 204 determines that the operation of the robot 101 is abnormal.

  In Case 2, the measurement value of the force sensor 105 is equal to or more than the value corresponding to the minimum value necessary to grip the object 106, or less than the value in the range not damaging the object 106 and the peripheral object 107. It is. Furthermore, in Case 2, the movement of the object 106 is synchronized with the movement of the end effector (gripping portion) of the robot 101, and the surrounding object 107 has a movement. In this case, although the object 106 can be accurately operated, the object 106 and the peripheral object 107 interfere with each other, and it is considered that the robot 101 has operated up to the peripheral object 107. At this time, it is necessary to realize an operation method of operating only the object 106 without operating the peripheral object 107. For example, the abnormality determination unit 204 compares the directions of force and torque vectors when operating the object 106 with the directions of motion vectors detected from the optical flow of changes in the position and posture of the peripheral object 107. As a result of this comparison, when the directions coincide with each other, it is considered that an external force is applied to the object 106 and the peripheral object 107 in the operation direction. In this case, the abnormality determination unit 204 changes the direction of the external force applied to the object 106, and determines that the influence on the peripheral object 107 is further reduced. The action planning unit 205 creates a plan of the action of the robot 101 according to the result of the determination.

  In Case 3, the measurement value of the force sensor 105 is not less than the value corresponding to the minimum value of the gripping force necessary to grip the object 106, and not more than the value in the range not damaging the object 106 and the peripheral object 107. The situation is Furthermore, in Case 3, the movement of the object 106 is synchronized with the movement of the end effector (gripping portion) of the robot 101, and the surrounding object 107 does not move. In this case, the robot 101 is considered to operate only the object 106. Therefore, the abnormality determination unit 204 determines that the robot 101 is performing a normal operation. Thus, the action planning unit 205 causes the robot 101 to continue the current operation.

  In case 4, the measurement value of the force sensor 105 is not less than a value corresponding to the minimum value of the gripping force necessary to grip the object 106 and not more than a value in a range where the object 106 and the peripheral object 107 are not damaged. The situation is Furthermore, in Case 4, the movement of the object 106 is not synchronized with the movement of the end effector (gripping portion) of the robot 101, and the peripheral object 107 has a movement. In this case, since the peripheral object 107 is moving, the external force applied by the robot 101 affects the work space. Therefore, there is a possibility that the object 106 can be moved by continuing the movement of the robot 101.

  In this case, the determination result of the detection of the abnormality of the operation of the robot 101 differs depending on the degree of the movement of the object 106 and the peripheral object 107. For example, if the object 106 is not moving at all and only the peripheral object 107 is moving, the robot 101 may be operating the peripheral object 107 instead of the object 106. Therefore, the abnormality determination unit 204 determines that the operation of the robot 101 is abnormal. Conversely, when there is a movement of the object 106, the abnormality determination unit 204 determines whether there is an abnormality in the movement of the robot 101 based on the change in the movement of the peripheral object 107. If it is determined that the operation of the robot 101 is not abnormal as a result of this determination, the action planning unit 205 causes the robot 101 to continue the current operation.

  The degree of movement of the peripheral object 107 can be determined, for example, by the degree of change in the position and orientation of the peripheral object 107 determined in step S1643. For example, when the motion vector detected by obtaining the optical flow divides the entire work space into a grid, the abnormality determination unit 204 determines whether the area corresponds to the area of the entire area. The degree of movement of the object 107 can be obtained. The abnormality determination unit 204 determines that there is an abnormality in the movement of the robot 101 when the area in which the motion vector exists (the area with movement) is equal to or more than the area set in advance in the entire work space. However, as long as the degree of movement of the peripheral object 107 can be determined, such a method need not necessarily be used. For example, a method using a background difference may be adopted as a method of deriving the degree of movement of the peripheral object 107. Also, it is necessary to consider the movement of the peripheral object 107. For example, if a change in movement is observed such that the object 106 and the peripheral object 107 move out of the container in which the object 106 and the peripheral object 107 should originally enter from the change in movement of the peripheral object 107, the abnormality determination unit 204 Determines that the operation of the robot 101 should not be continued.

  In Case 5, the measurement value of the force sensor 105 is equal to or greater than a value corresponding to the minimum value necessary to grip the object 106, or less than a value in a range where the object 106 and the peripheral object 107 are not damaged. It is. Furthermore, in Case 5, the movement of the object 106 is not synchronized with the movement of the end effector (gripping portion) of the robot 101, and the peripheral object 107 does not move. In this case, for the situation where the object 106 is buried under the surrounding object 107 and no object can be manipulated even when an external force is applied, or for a container containing the object 106 and the surrounding object 107 A situation where an external force is applied may be considered.

  However, there is a possibility that the object 106 can be moved by continuing the operation of the robot 101. Therefore, the abnormality determination unit 204 does not determine that the operation of the robot 101 is abnormal as long as the gripping force of the robot 101 is equal to or less than the value of the range in which the object 106 is not damaged. In this case, the behavior planning unit 205 causes the robot 101 to continue the current motion. However, when the external force changes rapidly, the abnormality determination unit 204 determines that the external force to be applied needs to be changed. Based on the result of this determination, the action planning unit 205 creates a plan of the action of the robot 101 so as to change the movement direction of the robot 101 or to reduce the movement speed in order to prevent the breakage of the object 106. Do.

  In Case 6, the measurement value of the force sensor 105 is equal to or more than the value corresponding to the minimum value necessary to grip the object 106, and the value in the range not damaging the object 106 and the peripheral object 107 is also exceeded. The situation is In this case, since one or more of the object 106 and the peripheral object 107 are damaged by operating the robot 101, the abnormality determination unit 204 determines whether the movement of the object 106 and the peripheral object 107 is performed. It is determined that the operation of the robot 101 is abnormal.

  The above is an example of the content performed in the abnormality determination process. However, as long as changes in the position and posture of the object 106 and the peripheral object 107 can be obtained, other methods may be adopted as the method. For example, a sensor capable of measuring the position and orientation is attached to each of the object 106 and the peripheral object 107, and changes in the position and orientation of the object 106 and the peripheral object 107 are obtained by reading information from the sensor. You may get it.

  Further, as long as the degree of abnormality of the operation of the robot 101 can be determined, abnormality determination processing other than steps S1651 to S1654 may be performed. For example, as another abnormality determination processing, an evaluation function, using machine learning, to what extent the object 106 and the peripheral object 107 fall from the inside of the container that should originally be contained by manipulation using the robot 101. Set as.

  At this time, how close is the force and torque measured by force sensor 105 in the process of step S1651 to the assumed range, and how much the change in position and posture determined in the process of step S1643 is Information such as is used as an input value. The forces and torques are determined as scalar values.

  Therefore, the degree of abnormality can be determined by comparing the force and torque measured by the force sensor 105 with the threshold set in one dimension. Difference in the degree of abnormality such as whether the force and torque measured by the force sensor 105 are within the range in which the object 106 can be gripped or in the region where interference is present while gripping the object 106 Use By determining the difference in behavior of the evaluation function with respect to these input values, machine learning of the degree of abnormality with respect to the input parameters is realized, and abnormality determination processing is realized.

  In addition, the abnormality determination process in Case 2 may be similar to Case 12 described in the second embodiment described later or Case 22 described in the third embodiment. Similarly, the abnormality determination process in Case 4 may be similar to Case 14 described in the second embodiment described later or Case 24 described in the third embodiment.

After the abnormality determination process (the process of step S <b> 1654) is finished, the process proceeds to step S <b> 1700 in FIG. 3.
In step S1700, the action planning unit 205 determines whether to end the operation of the robot 101 based on the determination result in the abnormality detection process of step S1600. As a result of the determination, if the operation of the robot is not ended (that is, the operation of the robot 101 is continued), the process returns to step S1400 to continue the processing. At that time, if the motion of the robot 101 changes as a result of the determination of the abnormality detection processing, the behavior planning unit 205 replans the behavior of the robot 101 based on the result. For example, the action planning unit 205 changes the transport path of the object 106 or changes the operation speed of the robot 101. Further, in the case of changing the operation position or changing the gripping target, the process proceeds to step S1800, and then returns to step S1100 again to continue the process.

  On the other hand, when the operation of the robot 101 is ended (that is, the operation of the robot 101 is not continued), the process proceeds to step S1800. In step S1800, based on the image information acquired by the image acquisition unit 201, the information processing apparatus 104 detects whether to end the process (whether or not the next object 106 exists). As a result of this determination, if the next object 106 is present and the process is not ended, the process returns to step S1100 to continue the process. On the other hand, when the next object 106 does not exist and the processing is ended, the processing according to the flowchart of FIG. 3 is ended.

  As described above, in the present embodiment, the force and torque applied to the end effector of the robot 101 are calculated based on the force sense information measured by the force sensor 105 when the object 106 is operated by the robot 101. To derive. Further, based on the image information captured by the imaging device 102 when the object 106 is operated by the robot 101, changes in the position and orientation of the object 106 and the peripheral object 107 are derived. Then, based on the force and torque applied to the end effector of the robot 101 and the change in the position and posture of the object 106 and the peripheral object 107, it is determined whether or not the operation of the robot 101 during manipulation is abnormal. Do. Therefore, for example, when the object 106 is manipulated, even when the object 106 and the peripheral object 107 are piled up and interfere with each other, more accurate manipulation can be realized. Therefore, it is possible to more accurately determine the state of interference between the object 106 and other objects, and to reliably detect an abnormality in the operation of the robot 101 based on the result of the determination. Therefore, the situation where the robot 101 can continue the work can be expanded.

Second Embodiment
Next, a second embodiment will be described. In the first embodiment, the case where abnormality in the operation of the robot 101 is detected using the force sensor 105 that measures the force and torque applied to the end effector (gripping portion) of the robot 101 has been described as an example. On the other hand, in the present embodiment, an abnormality in the operation of the robot 101 is detected by using a sensor that measures the gripping force of the end effector of the robot 101. As described above, in the present embodiment and the first embodiment, the target of measurement by the force sensor is mainly different, and a part of the process by the target of measurement by the force sensor is different. Therefore, in the description of the present embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals as those in FIGS. 1 to 9, and the detailed description will be omitted.

  FIG. 10 is a diagram showing an example of the configuration of a robot system. The robot system shown in FIG. 10 is the robot system shown in FIG. 1 except that the force sensor 105 is eliminated and a gripping force measuring sensor 108 is added. FIG. 10 is an example of the configuration of the device, and does not limit the scope of the present invention.

  The gripping force measurement sensor 108 is attached to the root of the gripping portion of the end effector of the robot 101. The gripping force measurement sensor 108 according to the present embodiment is a force sensor that is configured using a strain gauge or a piezoelectric element, and measures the gripping force when gripping or transporting the object 106 gripped by the robot 101. The force sense information acquired by the gripping force measurement sensor 108 is processed by the information processing apparatus 104.

In the present embodiment, the configuration of functional elements of the information processing apparatus 104 for detecting an abnormality in the operation of the robot 101 during manipulation and the relationship between the functional elements are the same as those shown in FIG. However, the force sensor 105 shown in FIG. 2 replaces the gripping force measuring sensor 108.
Further, in the present embodiment, a flowchart showing an example of processing of the information processing apparatus 104 for realizing the robot system is the same as FIG. 3. However, in the present embodiment, the abnormality detection process in step S1600 is as follows.

FIG. 11 is a flowchart illustrating an example of the abnormality detection process (detailed process of step S1600 ′ corresponding to step S1600 of FIG. 3) in the present embodiment. Hereinafter, the procedure of the abnormality detection process will be described according to the flowchart of FIG.
Step S1600 'is an abnormality detection process performed in the present embodiment. Step S1620 'and step S1650' which are different from step S1600 shown in FIG. 5 will be described.

In step S1620 ′, the force / torque measurement unit 208 executes the first derivation process based on the force sense information acquired in step S1610. The first derivation process is a process of deriving the gripping force of the robot 101 as force sense information when the robot 101 grips the object 106.
FIG. 12 is a flowchart for explaining an example of the detailed process of step S1620 ′ of FIG. Hereinafter, the procedure of the processes of steps S1621 'to S1622' corresponding to step S1620 'will be described according to the flowchart of FIG.

  In step S1621 ', the force / torque measurement unit 208 calculates the gripping force of the robot 101 as a force when the robot 101 obtained from the force sense information is operating the object 106. The force with which the end effector of the robot 101 grips the object 106 is calculated in step S1621 '. At this time, the force with which the end effector of the robot 101 grips the object 106 is obtained as a scalar value in the gripping direction. Therefore, the force / torque measurement unit 208 calculates this scalar value. The force by which the end effector of the robot 101 grips the object 106, which is calculated in this manner, is used in the process of step S1622 '. If the end effector of the robot 101 can obtain the force for gripping the object 106, the method of calculating the gripping force of the robot 101 may be another method.

Next, in step S1622 ', the force / torque measurement unit 208 performs gripping force information determination processing based on the gripping force of the robot 101 calculated in step S1621'. Specifically, the force / torque measurement unit 208 determines whether the gripping force of the robot 101 is an appropriate value for gripping the object 106. For example, it is assumed that the mass of the object 106 is M [g] and the variation thereof is ± m [g]. In this case, the mass of the object 106 is set to M ± m [g]. Therefore, assuming that the gravitational acceleration is g [m / s ² ] and the static friction coefficient is μ, the force required to grip the object 106 is (M ± m) × μ × g [N]. Therefore, if the gripping force obtained in step S1621 'is equal to or greater than this set value, it is considered that the force necessary to grip the object 106 is being applied to the robot 101.

This set value can be determined in advance by the designer. In addition, a value corresponding to the object 106 may be set in advance as the setting value. Furthermore, this setting value may be set by another method as long as a value necessary for the operation of the object 106 can be set.
As described above, the process of step S 1620 ′ of FIG. 11 is performed.

  In step S1650 'of FIG. 11, the abnormality determination unit 204 is based on the gripping force of the robot 101 calculated in step S1620' and changes in the position and orientation of the target object 106 and the peripheral object 107 derived in step S1640. To make an abnormality judgment. The abnormality determination unit 204 receives a plurality of pieces of information from the holding force measurement sensor 108 and the imaging device 102 (vision sensor) and outputs a determination result of the degree of abnormality of the operation of the robot 101. FIG. 13 is a flowchart for explaining an example of the detailed process of step S1650 'of FIG. The procedure of steps S1651 ′ and S1654 ′ corresponding to step S1650 ′ and different in processing from step S1650 will be described below with reference to the flowchart of FIG.

  In step S1651 ', the abnormality determination unit 204 determines which range of the preset range the gripping force of the robot 101 calculated in step S1620' is. For example, the abnormality determination unit 204 determines whether the gripping force of the robot 101 calculated in step S1620 ′ is equal to or greater than a value necessary for gripping the object 106. Further, the abnormality determination unit 204 determines whether or not the gripping force of the robot 101 calculated in step S1620 ′ is within a range that does not damage the object 106 and the peripheral object 107.

Parameters, such as a value necessary for this gripping and a value of a range in which the object 106 and the peripheral object 107 are not damaged, are preset by, for example, a designer.
By performing the determination in step S1651 ', there is an advantage that it is possible to classify what kind of state the dynamic state of operating the object 106 is in based on the gripping force of the robot 101 calculated in step S1620'. There is.

In step S1654 ', the abnormality determination unit 204 performs an abnormality determination process based on the determination results of step S1651', step S1652, and step S1653.
FIG. 14 is a diagram showing an example of the content of the abnormality determination process in the form of a table. Hereinafter, an example of the abnormality determination process will be described with reference to FIG.

  First, the abnormality determination unit 204 determines whether or not the gripping force of the robot 101 calculated in step S1620 ′ corresponds to “C. Minimum value of gripping force necessary to grip the object” in FIG. D. It is determined whether or not the target object falls under the value not exceeding the range of damage. This determination is performed according to the result of the determination of the gripping force of the robot 101 in step S1651 '.

  Next, the abnormality determination unit 204 determines whether or not "c. The movement of the target object is synchronized with the grip part" and whether or not the "d. Movement of the peripheral object exists". . This determination is performed according to the results of the determinations in steps S1652 and S1653 of FIG.

Also in FIG. 14, in the same manner as FIG. 9, the corresponding case is described as “Yes”, and the case not corresponding is described as “No”. For example, “C. minimum value of gripping force required to grip the object”, “D. value or less in the range where the object is not broken”, “c. Movement of the object in synchronization with the gripping part And “d. There is movement of peripheral objects”, the case 2 is classified.
By such determination, the state during manipulation is classified into any of the cases 11-16.

Hereinafter, the abnormality determination process for each case will be described.
In the case 11, the measurement value of the gripping force measurement sensor 108 is less than or equal to the minimum value of the gripping force required to grip the object 106. In this case, the object 106 can not be operated accurately with this holding force. Therefore, the abnormality determination unit 204 determines that the gripping force of the robot 101 is higher than the measurement value of the force sensor 105. The action planning unit 205 creates a plan of the action of the robot 101 according to the result of the determination. If the gripping force can not be increased beyond the measurement value of the gripping force measuring sensor 108, the abnormality determining unit 204 determines that the operation of the robot 101 is abnormal.

  In the case 12, the measurement value of the gripping force measurement sensor 108 is a condition not less than the minimum value necessary to grip the object 106 and not more than a value in a range where the object 106 and the peripheral object 107 are not damaged. Furthermore, in the case 12, the movement of the object 106 is synchronized with the movement of the end effector (gripping portion) of the robot 101, and the surrounding object 107 has a movement. In this case, the operation of the robot 101 can not necessarily be determined to be normal.

  For example, although the robot 106 grips the object 106, it is assumed that the object 106 and the peripheral object 107 interfere with each other. Therefore, the abnormality determination unit 204 determines the degree of movement of the peripheral object 107, and based on the result, determines whether or not there is an abnormality in the operation of the robot 101. Specifically, the abnormality determination unit 204 obtains the degree of movement of the peripheral object 107 by using information on how much the change in the position and orientation of the peripheral object 107 determined in step S1643 is.

  For example, when the motion vector detected by obtaining the optical flow divides the entire work space into a grid, the abnormality determination unit 204 determines whether the area corresponds to the area of the entire area. The degree of movement of the object 107 can be obtained. The abnormality determination unit 204 determines that there is an abnormality in the movement of the robot 101 when the area in which the motion vector exists (the area with movement) is equal to or more than the area set in advance in the entire work space. However, as long as the degree of movement of the peripheral object 107 can be determined, such a method need not necessarily be used. For example, a method using a background difference may be adopted as a method of deriving the degree of movement of the peripheral object 107. Also, it is necessary to consider the movement of the peripheral object 107. For example, if a change in movement is observed such that the object 106 and the peripheral object 107 move out of the container in which the object 106 and the peripheral object 107 should originally enter from the change in movement of the peripheral object 107, the abnormality determination unit 204 Determines that the operation of the robot 101 should not be continued.

  In the case 13, the measurement value of the gripping force measuring sensor 108 is equal to or more than the minimum value of the gripping force necessary to grip the object 106 and not more than the value in the range not damaging the object 106 and the peripheral object 107. It is. Furthermore, in the case 13, the movement of the object 106 is synchronized with the movement of the end effector (gripping portion) of the robot 101, and the surrounding object 107 does not move. In this case, the robot 101 is considered to operate only the object 106. Therefore, the abnormality determination unit 204 determines that the robot 101 is performing a normal operation. Thus, the action planning unit 205 causes the robot 101 to continue the current operation.

  In the case 14, the measurement value of the gripping force measuring sensor 108 is not less than the minimum value of the gripping force necessary to grip the object 106 and not more than the value in the range where the object 106 and the peripheral object 107 are not damaged. It is. Furthermore, in the case 14, the movement of the object 106 is not synchronized with the movement of the end effector (gripping portion) of the robot 101, and the peripheral object 107 has a movement. In this case, it is considered that slippage occurs in the portion holding the object 106. Therefore, the abnormality determination unit 204 determines that the gripping force of the robot 101 is increased to a limit value that does not damage the object 106 and the peripheral object 107 and the process is continued. The action planning unit 205 creates a plan of the action of the robot 101 according to the result of the determination. In addition, when the gripping force of the robot 101 can not be increased to a limit value that does not damage the object 106 and the peripheral object 107, the abnormality determining unit 204 determines that there is an abnormality in the action of the robot 101.

  In the case 15, the measurement value of the gripping force measuring sensor 108 is not less than the minimum value of the gripping force necessary to grip the object 106, and not more than the value in the range not damaging the object 106 and the peripheral object 107. It is. Furthermore, in the case 15, the movement of the object 106 is not synchronized with the movement of the end effector (gripping portion) of the robot 101, and the peripheral object 107 does not move. In this case, as in the case 14, it is considered that slippage occurs in the portion holding the object 106.

  However, unlike the case 14, since there is no movement of the peripheral object 107, the object 106 interferes with the non-moving portion of the peripheral object 107 or interferes so that the peripheral object 107 can not be moved. It is possible. As a non-moving part of the peripheral object 107, for example, a container itself containing the object 106 and the peripheral object 107 can be considered. Therefore, in the case 15 as well as the case 14, the abnormality determination unit 204 determines that the gripping force of the robot 101 is increased to a limit value that does not damage the object 106 and the peripheral object 107 and the process is continued. However, in order to more accurately recognize changes in the object 106 and the peripheral object 107, in addition to this determination, the abnormality determination unit 204 determines that the operation speed of the robot 101 is to be reduced. The action planning unit 205 creates a plan of the action of the robot 101 according to the result of the determination. In addition, when the gripping force of the robot 101 can not be increased to a limit value that does not damage the object 106 and the peripheral object 107, the abnormality determining unit 204 determines that there is an abnormality in the action of the robot 101.

  In the case 16, the measurement value of the gripping force measurement sensor 108 is equal to or more than the minimum value necessary to grip the object 106, and also exceeds the value in the range not damaging the object 106 and the surrounding object 107. It is the situation that. In this case, since one or more of the object 106 and the peripheral object 107 are damaged by operating the robot 101, the abnormality determination unit 204 determines whether the movement of the object 106 and the peripheral object 107 is performed. It is determined that the operation of the robot 101 is abnormal.

  The above is an example of the content performed in the abnormality determination process in this embodiment. However, methods other than the above may be adopted as the method of the abnormality determination processing. That is, it can be determined whether the gripping force of the robot 101 is equal to or more than the value necessary for gripping the object 106 and whether it is equal to or less than the value in the range not damaging the object 106 and the peripheral object 107. Other methods may be adopted. Further, any other method may be employed as long as the change in the position and posture of the object 106 and the peripheral object 107 can be observed.

  Further, the abnormality determination process in the case 12 may be the case 2 described in the first embodiment or the case 22 described in the third embodiment to be described later. Similarly, the abnormality determination process in the case 14 may be similar to the case 4 described in the first embodiment or the case 24 described in the third embodiment to be described later.

  As described above, in the present embodiment, the gripping force of the robot 101 in operation of the object 106 is measured by the gripping force measurement sensor 108. Therefore, it is possible to accurately determine whether or not the operation of the robot 101 during manipulation is abnormal without measuring the force and torque applied to the end effector of the robot 101, as described in the first embodiment. The same effect as the effect is obtained.

Third Embodiment
Next, a third embodiment will be described. In the first embodiment, the case where abnormality in the operation of the robot 101 is detected using the force sensor 105 that measures the force and torque applied to the end effector (gripping portion) of the robot 101 has been described as an example. Also, in the second embodiment, an example is shown in which an abnormality in the operation of the robot 101 is detected using a gripping force measurement sensor 108 (force sensor) that measures the gripping force of the end effector (grip portion) of the robot 101. I mentioned to you. On the other hand, in the present embodiment, among the end effector (gripping portion) of the robot 101, the force and torque applied to the contact portion with the object 106 at the time of gripping and transporting the object 106 are directly measured. The force sensor is used to detect an abnormality in the operation of the robot 101. Thus, in the present embodiment and the first and second embodiments, the target of measurement by the force sensor is mainly different, and part of the processing by the target of measurement by the force sensor is different. Therefore, in the description of the present embodiment, the same parts as those in the first and second embodiments are denoted by the same reference numerals as those in FIGS. 1 to 14, and detailed description thereof will be omitted.

  FIG. 15 is a diagram showing an example of the configuration of a robot system. The robot system shown in FIG. 15 is the robot system shown in FIG. 1 except that the force sensor 105 is eliminated and the grip portion force sensor 109 is added. FIG. 15 is an example of a device configuration, and does not limit the scope of the present invention.

  The gripping portion force sensor 109 is attached to a gripping contact portion which is a portion of the gripping portion of the end effector of the robot 101 that contacts the object (target object 106) to be gripped. The gripping unit force sensor 109 according to the present embodiment is configured using a strain gauge or a piezoelectric element, and is a force sensor that measures the force and torque applied to the gripping contact unit when gripping the object 106 with the robot 101. It is. The force sense information acquired by the grip portion force sense sensor 109 is processed by the information processing apparatus 104.

In the present embodiment, the configuration of functional elements of the information processing apparatus 104 for detecting an abnormality in the operation of the robot 101 during manipulation and the relationship between the functional elements are the same as those shown in FIG. However, the force sensor 105 shown in FIG.
Further, in the present embodiment, a flowchart showing an example of processing of the information processing apparatus 104 for realizing the robot system is the same as FIG. 3. However, in the present embodiment, the abnormality detection process in step S1600 is as follows.

FIG. 16 is a flowchart illustrating an example of the abnormality detection process (detailed process of step S1600 ′ ′ corresponding to step S1600 of FIG. 3) in the present embodiment. The procedure of the abnormality detection process will be described below with reference to the flowchart of FIG.
Step S1600 '' is an abnormality detection process performed in the present embodiment. Step S1620 '' and step S1650 '' different from step S1600 shown in FIG. 5 will be described.

In step S1620 ′ ′, the force / torque measurement unit 208 executes the first derivation process based on the force sense information acquired in step S1610. The first derivation process is a process of measuring the force / torque information of the gripping contact portion. The force / torque information of the gripping contact portion is information indicating the force and torque applied to the gripping contact portion of the end effector of the robot 101 while the object 106 is being manipulated.
FIG. 17 is a flowchart for explaining an example of the detailed process of step S1602 ′ ′ of FIG. Hereinafter, the process from step S1621 ′ ′ to step S1622 ′ ′ corresponding to step S1620 ′ ′ will be described according to the flowchart of FIG.

  In step S1621 ′ ′, the force / torque measurement unit 208 determines the vertical direction of the gripping contact portion in the coordinate system of the robot 101 based on the force applied to the gripping contact portion when the robot 101 is operating the object 106. Calculate the strength. The force applied to the gripping contact portion when the robot 101 is operating the object 106 is obtained from the force sense information. When operating the object 106, a force is applied to the robot 101 by the mass of the object 106. The force measured at this time can be obtained from the measurement values of the X, Y, and Z components of the coordinate system of the grip portion force sensor 109. The force / torque measurement unit 208 obtains the resultant force in the vertical direction of the gripping contact portion in the coordinate system of the robot 101 based on these components. By doing this, the force applied to the robot 101 in the direction of the gravitational acceleration can be determined, and can be used in determining whether or not the object 106 can be gripped. However, the resultant force in the vertical direction of the gripping contact portion in the coordinate system of the robot 101 may be calculated by another method as long as the mass of the object 106 gripped by the robot 101 is obtained.

  Next, in step S1622 ′ ′, the force / torque measurement unit 208 calculates an estimated torque in the grip contact unit based on the torque applied to the grip contact unit when the robot 101 is operating the object 106. The torque applied to the grip contact unit when the robot 101 is operating the object 106 is obtained from the force sense information. When operating the object 106, the robot 101 is torqued by the moment of inertia of the object 106. The torque measured at this time is obtained with respect to the rotational directions α, β and γ of the respective axes of the X, Y and Z components of the coordinate system of the grip portion force sensor 109. The force / torque measurement unit 208 calculates, based on the values of these components, a torque according to the posture in which the robot 101 grips the object 106 as an estimated torque in the grip contact unit. When a force sensor is attached to the gripping contact portion, the torque can be obtained with respect to the XY plane when the robot 101 grips the object 106 in the Z-axis direction. The direction of the torque to be measured is arbitrarily determined in accordance with the posture in which the robot 101 grips the object 106 with respect to the XY plane. By doing this, the torque applied to the gripping contact portion can be determined, and can be used in determining whether or not the object 106 can be gripped. However, the estimated torque at the gripping contact may be calculated by another method if the torque applied to the gripping contact is obtained. Moreover, it is not necessary to necessarily perform both processes of step S1621 '' and step S1622 '', and any one process may be performed. After the process of step S1622 ′ ′ is finished, the process proceeds to step S1630 of FIG.

  In step S1650 ′ ′, the abnormality determination unit 204 changes the force / torque information of the grip contact unit derived in step S1620 ′ ′, and changes in the position and orientation of the object 106 and the peripheral object 107 derived in step S1640. Based on the above, the abnormality determination is performed. The abnormality determination unit 204 receives a plurality of pieces of information from the holding unit force sensor 109 and the imaging device 102 (vision sensor) as an input, and outputs a determination result of the degree of abnormality of the operation of the robot 101. FIG. 18 is a flow chart for explaining an example of the detailed process of step S1650 '' of FIG. The procedure of steps S1651 ′ ′ and S1654 ′ ′ corresponding to step S1650 ′ ′ and different in processing from step S1650 will be described below according to the flowchart of FIG.

  In step S1651 ′ ′, the abnormality determination unit 204 determines which of the ranges set in advance is the force and torque indicated in the force / torque information of the gripping contact portion derived in step S1620 ′ ′. judge. For example, the abnormality determination unit 204 determines whether or not the force and torque indicated in the force / torque information derived in step S1620 are within the range necessary for gripping or transporting the object 106. In addition, the abnormality determination unit 204 determines whether the force and torque indicated in the force / torque information of the grip contact unit derived in step S1620 ′ ′ are within a range not damaging the object 106 and the peripheral object 107. judge.

Parameters such as the value which is a range necessary for gripping or transporting, and the value of a range in which the object 106 and the peripheral object 107 are not damaged are set in advance as follows by a designer, for example. First, it is assumed that the mass of the object 106 is M [g] and the variation thereof is ± m [g]. In this case, the mass of the object 106 is set to M ± m [g]. Therefore, assuming that the gravitational acceleration is g [m / s ² ] and the static friction coefficient is μ, the force required to grip the object 106 can be set as (M ± m) × μ × g [N]. Therefore, if the resultant force in the vertical direction of the gripping contact portion in the coordinate system of the robot 101 calculated in step S1621 ′ ′ is equal to or larger than the set value, a force necessary for gripping the object 106 is applied to the robot 101. It is thought that

This set value can be determined in advance by the designer. In addition, a value corresponding to the object 106 may be set in advance as the setting value. Furthermore, this setting value may be set by another method as long as a value necessary for the operation of the object 106 can be set.
By performing the determination in step S1651 ′ ′, it is possible to classify what kind of state the dynamic state of operating the object 106 is in, based on the gripping force of the robot 101 calculated in step S1620 ′ ′. There is a merit that.

  In step S1654 ′ ′, the abnormality determination unit 204 performs abnormality determination processing based on the determination results in step S1651 ′ ′, step S1652, and step S1653. FIG. 19 is a diagram showing an example of the content of the abnormality determination process in the form of a table. Hereinafter, an example of the abnormality determination process will be described with reference to FIG.

  First, the abnormality determination unit 204 determines that the force and torque shown in the force / torque information of the gripping contact portion derived in step S 1620 ′ ′ are “E. It is determined whether or not the condition is the minimum value or more. In addition, the abnormality determination unit 204 determines that the force and torque indicated in the force / torque information of the gripping contact portion derived in step S1620 ′ ′ are “F. or less of the range where the object is not damaged” in FIG. It is determined whether to do. This determination is performed according to the result of the determination of the force and torque of the gripping contact portion in step S1651 ′ ′ of FIG.

  Next, the abnormality determination unit 204 determines whether or not “e. The movement of the object is synchronized with the gripping unit” and whether or not “f. The movement of the peripheral object is present”. . This determination is made in accordance with the results of the determinations in step S1652 "and step S1653" in FIG.

In FIG. 19 as well as FIG. 9, the corresponding case is described as “Yes”, and the case not corresponding is described as “No”. For example, "E or more of the minimum value of the gripping force necessary to grip the object", "F or less of the range where the object is not broken", "e. The movement of the object is synchronized with the gripping portion And “f. There is movement of peripheral objects”, it is classified into case 22.
By such a determination, the state during manipulation is classified into any of the cases 21 to 26.

  Hereinafter, the abnormality determination process for each case will be described. Here, the case where the presence or absence of an abnormality in the operation of the robot 101 is determined using the force applied to the gripping contact portion of the end effector of the robot 101 gripping the object 106 will be described as an example. However, the force may be applied in place of torque. Also, both force and torque may be used.

  In the case 21, the measurement value of the gripping unit force sensor 109 is equal to or less than the value corresponding to the minimum value of the gripping force required to grip the object 106. In this case, the object 106 can not be operated accurately with this holding force. Therefore, the abnormality determination unit 204 determines that the gripping force of the robot 101 is higher than the measurement value of the gripping unit force sensor 109. The action planning unit 205 changes the action plan of the robot 101 in accordance with the result of the determination. If the gripping force can not be increased beyond the measurement value of the gripping unit force sensor 109, the abnormality determining unit 204 determines that the operation of the robot 101 is abnormal.

  In the case 22, the measurement value of the grip force sensor 109 is not less than the value corresponding to the minimum value of the gripping force necessary to grip the object 106, and not more than the value in the range not damaging the object 106 and the peripheral object 107. It is a situation that is. Furthermore, in the case 22, the movement of the object 106 is synchronized with the movement of the end effector (gripping portion) of the robot 101, and the surrounding object 107 has a movement. In this case, the operation of the robot 101 can not necessarily be determined to be normal.

  For example, although the robot 106 grips the object 106, it is assumed that the object 106 and the peripheral object 107 interfere with each other. Therefore, the abnormality determination unit 204 determines the degree of movement of the peripheral object 107, and based on the result, determines whether or not there is an abnormality in the operation of the robot 101. Specifically, the abnormality determination unit 204 obtains the degree of movement of the peripheral object 107 by using information on how much the change in the position and orientation of the peripheral object 107 determined in step S1643 is.

  For example, when the motion vector detected by obtaining the optical flow divides the entire work space into a grid, the abnormality determination unit 204 determines whether the area corresponds to the area of the entire area. The degree of movement of the object 107 can be obtained. The abnormality determination unit 204 determines that there is an abnormality in the movement of the robot 101 when the area in which the motion vector exists (the area with movement) is equal to or more than the area set in advance in the entire work space. However, as long as the degree of movement of the peripheral object 107 can be determined, such a method need not necessarily be used. For example, a method using a background difference may be adopted as a method of deriving the degree of movement of the peripheral object 107. Also, it is necessary to consider the movement of the peripheral object 107. For example, if a change in movement is observed such that the object 106 and the peripheral object 107 move out of the container in which the object 106 and the peripheral object 107 should originally enter from the change in movement of the peripheral object 107, the abnormality determination unit 204 Determines that the operation of the robot 101 should not be continued.

  When a torque is used as the measurement value of the grip force sensor 109, the measurement value may be used to estimate the state of manipulation. For example, when the robot 101 grips the vicinity of the center of gravity of the object 106, almost no torque in the rotational direction is generated with respect to the plane gripping the object 106 of the end effector. On the other hand, when there is interference between the object 106 and the peripheral object 107, torque is generated. It is estimated that the external force in the rotational direction in which the torque is generated is applied to the object 106. Therefore, the abnormality determination unit 204 determines an approximate interference state of the peripheral object 107 with the object 106, and determines that the robot 101 is operated in a direction to soften the interference state. The action planning unit 205 creates a plan of the action of the robot 101 according to the result of the determination.

  In the case 23, the measurement value of the grip force sensor 109 is not less than the value corresponding to the minimum value of the gripping force necessary to grip the object 106, and not more than the value in the range not damaging the object 106 and the peripheral object 107. It is a situation that is. Furthermore, in the case 23, the movement of the object 106 is synchronized with the movement of the end effector (gripping portion) of the robot 101, and the surrounding object 107 does not move. In this case, the robot 101 is considered to operate only the object 106. Therefore, the abnormality determination unit 204 determines that the robot 101 is performing a normal operation. Thus, the action planning unit 205 causes the robot 101 to continue the current operation.

  In the case 24, the measurement value of the grip force sensor 109 is not less than the value corresponding to the minimum value of the gripping force necessary to grip the object 106, and not more than the value in the range not damaging the object 106 and the peripheral object 107. It is a situation that is. Furthermore, in the case 24, the movement of the object 106 is not synchronized with the movement of the end effector (gripping portion) of the robot 101, and the peripheral object 107 has a movement. In this case, it is considered that slippage occurs in the portion holding the object 106. Therefore, the abnormality determination unit 204 determines that the gripping force of the robot 101 is increased to a limit value that does not damage the object 106 and the peripheral object 107 and the process is continued. The action planning unit 205 changes the action plan of the robot 101 in accordance with the result of the determination. In addition, when the gripping force of the robot 101 can not be increased to a limit value that does not damage the object 106 and the peripheral object 107, the abnormality determining unit 204 determines that there is an abnormality in the action of the robot 101.

  Further, in order to estimate the interference state between the object 106 and the peripheral object 107, as in the case described in the case 22, if the grip force sensor 109 measures torque, the abnormality determination unit 204 Can calculate an approximate interference state from the measured value. Alternatively, the abnormality determination unit 204 directly derives the slip condition of the object 106 by deriving the force in the direction of the plane in which the end effector grips the object 106 from the measurement value of the gripping unit force sensor 109. It may be estimated. When the direction of the slip of the object 106 can be detected, the abnormality determination unit 204 changes the direction of the external force applied by the robot 101 to reduce the slip of the object 106, and determines that the manipulation is to be continued.

  In the case 25, the measurement value of the gripping unit force sensor 109 is not less than the minimum value of the gripping force necessary to grip the object 106 and not more than the value in the range not damaging the object 106 and the peripheral object 107. It is a situation. Furthermore, in the case 25, the movement of the object 106 is not synchronized with the movement of the end effector (gripping portion) of the robot 101, and the surrounding object 107 does not move. In this case, as in the case 24, it is considered that slippage occurs in the portion holding the object 106.

  However, unlike the case 24, since there is no movement of the peripheral object 107, there is a possibility that the object 106 interferes with the non-moving portion of the peripheral object 107. Therefore, the abnormality determination unit 204 continues the process by increasing the gripping force of the robot 101 within a range not damaging the object 106 and the peripheral object 107 based on the direction of the force and the direction of the torque of the gripping contact portion at this time. It will be determined. At this time, as described in the case 24, the abnormality determination unit 204 may determine that the operation speed of the robot 101 is to be reduced. In addition, when the gripping force of the robot 101 can not be increased to a limit value that does not damage the object 106 and the peripheral object 107, the abnormality determining unit 204 determines that there is an abnormality in the action of the robot 101.

  In the case 26, the measurement value of the gripping unit force sensor 109 is equal to or more than the minimum value necessary to grip the object 106, and also exceeds the value in the range not damaging the object 106 and the peripheral object 107. The situation is Since the operation of the robot 101 damages one or more of the object 106 and the peripheral object 107, the abnormality determination unit 204 makes it possible to operate the robot 101 regardless of the movement of the object 106 and the peripheral object 107. It is determined that there is an abnormality in the operation.

  The abnormality determination process in the case 22 may be similar to the case 2 described in the first embodiment or the case 12 described in the second embodiment. Similarly, the abnormality determination process in Case 4 may be similar to Case 4 described in the first embodiment or Case 14 described in the second embodiment.

  The above is an example of the content performed in the abnormality determination process in this embodiment. However, methods other than the above may be adopted as the method of the abnormality determination processing. That is, it can be determined whether the gripping force of the robot 101 is equal to or more than the value necessary for gripping the object 106 and whether it is equal to or less than the value in the range not damaging the object 106 and the peripheral object 107. Other methods may be adopted. Further, any other method may be employed as long as the change in the position and posture of the object 106 and the peripheral object 107 can be observed.

  As described above, it is also possible to accurately determine whether or not the operation of the robot 101 during manipulation is abnormal by measuring the force sense information (force and torque) of the gripping contact portion gripping the object 106. can do. That is, the same effect as the effect described in the first embodiment can be obtained also in the present embodiment.

Fourth Embodiment
Next, a fourth embodiment will be described. In the first to third embodiments, the case where the rigidity of the object 106 is not considered is described as an example. On the other hand, in the present embodiment, by determining the operation position in consideration of the rigidity of the object 106, it is possible to detect the abnormality of the operation of the robot 101 more quickly and accurately. Thus, the present embodiment and the first to third embodiments are mainly different in that the rigidity of the object 106 is considered. Therefore, in the description of the present embodiment, the same parts as those in the first to third embodiments are denoted by the same reference numerals as those in FIGS.

  FIG. 20 is a diagram showing an example of a functional configuration of the information processing apparatus 104 ′. The information processing apparatus 104 ′ shown in FIG. 20 is an information processing apparatus 104 shown in FIG. 2 in which the operation position determination unit 210 is eliminated and a flexible object operation position determination unit 210 ′ is added. The flexible material operation position determination unit 210 'which is a part different from FIG. 2 will be described.

  The flexible object operation position determination unit 210 ′ determines the grip position and the release position of the object 106 as the operation position of the object 106, using the position and orientation information from the position and orientation measurement unit 209 as an input. The flexible object operation position determination unit 210 'takes into consideration the rigidity of each part of the object 106, and the operation position is set so that the robot 101 can grip the more rigid or less deformable part of the object 106. decide. By doing this, it is possible to quickly and accurately determine the haptic information. The operation position determined by the flexible object operation position determination unit 210 ′ is output to the action planning unit 205.

FIG. 21 is a flowchart for explaining an example of processing of the information processing apparatus 104 ′ for realizing the robot system. Hereinafter, the procedure of the process will be described according to the flowchart of FIG. 21 for parts different from the flowchart shown in FIG.
In step S1300 ′, the flexible material manipulation position determination unit 210 determines the manipulation position of the target object 106 based on the result (the position and posture of the target object 106 to be gripped) measured in step S1200.

  In step S1300 of FIG. 3, the operation position determination unit 210 determines the grip position for operating the target 106 to the target position and posture, in consideration of the shape of the end effector of the robot 101. On the other hand, in step S 1300 ′, the flexible material manipulation position determination unit 210 ′ further determines the gripping position in consideration of the rigidity of the object 106. FIG. 22 is a flowchart for explaining an example of the detailed process of step S1300 ′ of FIG. Hereinafter, the procedure of the process from step S1301 ′ to step S1303 ′ corresponding to step S1300 ′ will be described according to the flowchart of FIG.

In step S1301 ′, the flexible material manipulation position determination unit 210 ′ selects a grippable location. The flexible object operation position determination unit 210 ′ can grip any position of the object 106 based on the information of the position and orientation of the object 106 obtained from the position / posture measurement unit 209, and the target operation Decide if you can. Then, the flexible material operation position determination unit 210 ′ selects one or more locations of the object 106 that can be gripped by the robot 101 as grippable locations.
As described above, the component database 203 stores information (CAD data etc.) of the shapes of the object 106 and the peripheral object 107. The flexible object operation position determination unit 210 ′ refers to information on the shape of the object 106 in addition to the information on the position and orientation of the object 106 obtained from the position / posture measurement unit 209 when selecting a grippable location. can do.

  Next, in step S1302 ′, the flexible material manipulation position determination unit 210 ′ estimates the rigidity of the object 106 and the deformation that occurs in the object 106 when the robot 106 grips the object 106. Specifically, the flexible material manipulation position determination unit 210 ′ estimates the degree of rigidity and deformation of one or more grippable places selected in step S1301 ′. For example, in the case where the rigidity of the object 106 is not uniform, it is preferable that the robot 101 grip a portion with high rigidity than a portion with low rigidity. This is because the force sense information at the time of operating the object 106 is easily stabilized, and the stroke of the end effector (gripping portion) is also shortened, so that the abnormal operation of the robot 101 can be detected more quickly and accurately. For the same reason, in the case where the object 106 is a deformable object such as a cable, it is preferable to grip the shrunk and gathered portion rather than the portion which is bulging in a ring.

  In order to perform the process of step S1302 ', for example, information indicating the rigidity of each position of the object 106 can be stored in the component database 203. When the flexible object operation position determination unit 210 ′ estimates the rigidity and the degree of deformation of the grippable portion, the information of the position and orientation of the object 106, the information of the shape of the object 106, and the portions of the object 106 It is possible to refer to the information indicating the rigidity of each position.

  In step S1303 ', the flexible material manipulation position determination unit 210' determines the manipulation position. The flexible object operation position determination unit 210 ′ is higher in rigidity and more in deformation based on the combination of the rigidity and the degree of deformation estimated in step S1302 ′ from among the grippable portions selected in step S1301 ′. The small part is determined as the operation position.

  As described above, in the present embodiment, the operation position of the robot 101 is determined in consideration of the rigidity and the degree of deformation of the object 106. Therefore, force sense information can be accurately acquired in a shorter time, and the tact time of work can be shortened. Therefore, it is possible to accurately detect an abnormality in the operation of the robot 101 during manipulation of the object 106 in a short time.

  The embodiments described above are merely examples of implementation for carrying out the present invention, and the technical scope of the present invention should not be interpreted in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical concept or the main features thereof.

(Other embodiments)
The present invention is also realized by performing the following processing. That is, first, software (computer program) for realizing the functions of the above embodiments is supplied to a system or apparatus via a network or various storage media. Then, a computer (or a CPU or MPU or the like) of the system or apparatus reads out and executes the computer program.

  101: robot, 104: information processing device, 105: force sensor, 106: object

Claims (7)

An information processing apparatus that performs processing for operating an object with a robot,
-Out movement of the object, and, based on the motion of the surrounding objects are objects in the neighborhood of the object still and, at least one of the force and torque generated in the robot when working with the object An information processing apparatus comprising: determination means for determining whether or not the robot normally operates the object based on one . First deriving means for deriving at least one of a force and a torque generated in the robot when operating the object based on a measurement value of a sensor provided in the robot;
The apparatus further includes second deriving means for deriving the movement of the object and the movement of a peripheral object which is an object around the object based on an image including the object and the peripheral object. The information processing apparatus according to claim 1, characterized in that   The first lead-out means is at least one of a force and a torque applied to a portion of the robot holding the object and not in contact with the object, and a holding force of the object in the robot 4. The method according to claim 2, wherein at least one of a force and a torque applied to a portion of the robot holding the object and in contact with the object is derived. The information processing apparatus according to claim 1.   4. The apparatus according to claim 1, further comprising determining means for determining the position of the object held by the robot based on the rigidity of each part of the object and the state of deformation of the object. The information processing apparatus according to any one of the above.   The content of the operation of the robot is determined based on the result of the determination by the determination unit, and the apparatus further includes an output unit that outputs an operation command to operate the robot with the determined content. The information processing apparatus according to any one of 4. An information processing method for performing processing for operating an object by a robot, comprising:
-Out movement of the object, and, based on the motion of the surrounding objects are objects in the neighborhood of the object still and, at least one of the force and torque generated in the robot when working with the object An information processing method comprising the step of determining whether the robot normally operates the object based on one .   A program which causes a computer to function as each means of the information processing apparatus according to any one of claims 1 to 5.