JP2015131367A - Robot, control device, robot system and control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot, a control device, a robot system and a control method which reduce Jacobian operation loads in the control of a robot using an image.SOLUTION: A robot 100 comprises a control unit 110 which calculates a captured image feature quantity that is an image feature quantity of a captured image, and controls the robot 100 on the basis of the captured image feature quantity, a target feature quantity, and the Jacobian, and a storage unit 120 which stores the positional posture of a work object or an end effector 319 of the robot 100 in association with the Jacobian.

Description

  The present invention relates to a robot, a control device, a robot system, a control method, and the like.

  A technique for performing control using a picked-up image (camera image) in various operations by an industrial robot is known. As a method of aligning using a goal image, there is Patent Document 1. Japanese Patent Application Laid-Open No. 2004-151820 discloses a method of remembering a goal image at the time of success and moving the robot directly to the position by position control when the same goal image is obtained during a new operation.

  In addition, there is a technique called visual servo as a technique for performing precise positioning using captured images. Visual servo is a technique that brings a target position closer to a target position by controlling (servoing) the difference between the goal image and the current image so as to eliminate the difference. For example, Patent Document 2 discloses a technique for controlling a robot using visual servo.

JP 2003-231078 A JP 2009-83094 A

  Patent Document 1 performs position control using a position memorized in a previous robot operation. Therefore, it is difficult to perform an accurate operation when a shift occurs in the work environment, such as when the supply position of the work object is different.

  On the other hand, in the visual servo of Patent Document 2, since feedback is performed using information of sequentially captured images, it is possible to control the robot with high accuracy even if a shift occurs in the work environment. However, in the visual servo, a Jacobian (Jacobi matrix) is often used, but in the method of Patent Document 2, the Jacobian is calculated for each visual servo. For this reason, the calculation load required for executing the visual servo becomes large, and there may be an influence such as a low-speed operation of the robot.

  One embodiment of the present invention obtains a captured image feature amount that is an image feature amount of a captured image, and controls a robot based on the captured image feature amount, a target feature amount, and a Jacobian, The present invention relates to a robot that includes a storage unit that stores an object or the position and orientation of an end effector of the robot in association with the Jacobian.

  In one embodiment of the present invention, a Jacobian used for robot control is stored in association with a position and orientation. Therefore, it is possible to use the Jacobian stored according to the situation, so that it becomes possible to reduce the processing load and speed up the robot operation.

  In the aspect of the invention, the control unit reads the Jacobian stored in the storage unit, and controls the robot based on the Jacobian, the captured image feature amount, and the target feature amount. May be performed.

  This makes it possible to reuse the stored Jacobian.

  In one aspect of the present invention, the control unit determines whether or not the Jacobian corresponding to the position and orientation of the work object or the end effector at a processing timing is stored in the storage unit. , When the Jacobian is stored, an operation parameter of the robot is obtained based on difference information between the captured image feature quantity and the target feature quantity, and the Jacobian, and according to the operation parameter, the robot Control may be performed.

  Thereby, when a desired Jacobian is stored, the processing load can be reduced by using the Jacobian.

  Moreover, in one aspect of the present invention, the control unit is configured such that the position when the Jacobian corresponding to the position and orientation of the work object or the end effector at the processing timing is not stored in the storage unit. The Jacobian corresponding to the posture may be obtained, and the storage unit may store the Jacobian in association with the position and posture of the work object or the end effector at the processing timing.

  As a result, when the desired Jacobian is not stored, it is possible to newly calculate the Jacobian and store the Jacobian that is the operation result.

  In the aspect of the invention, the control unit may perform the processing when the Jacobian corresponding to the position and orientation of the work object or the end effector at the processing timing is not stored in the storage unit. From the position and orientation at the timing, it is determined whether the Jacobian corresponding to the position and orientation within a predetermined range whose spatial distance in the position and orientation coordinate space is within a given distance range is stored in the storage unit, and the predetermined When the Jacobian corresponding to the position / posture within the range is stored in the storage unit, the Jacobian corresponding to the position / posture within the predetermined range is obtained as the Jacobian corresponding to the position / posture at the processing timing. The storage unit stores the Jacobian as the work object or the engine at the processing timing. It may be stored in association with the position and orientation of the effector.

  Thereby, even when the desired Jacobian is not stored, the processing load can be reduced by using the Jacobian within the predetermined distance range.

  In the aspect of the invention, the control unit may include the Jacobian corresponding to the position and orientation at the processing timing when the Jacobian corresponding to the position and orientation within the predetermined range is not stored in the storage unit. The storage unit may store the Jacobian in association with the position and orientation of the work object or the end effector at the processing timing.

  Thereby, when the Jacobian that falls within the predetermined distance range is not stored, it is possible to newly calculate the Jacobian and to store the Jacobian that is the calculation result.

  In one aspect of the present invention, the control unit is configured to obtain a difference value between the captured image feature amount obtained by controlling the robot using the Jacobian and a provisional target feature amount estimated from the Jacobian. Is larger than a given threshold, the Jacobian is corrected, and the storage unit may store the corrected Jacobian in association with the position and orientation of the work object or the end effector. .

  This makes it possible to perform Jacobian correction processing and the like.

  In one aspect of the present invention, the storage unit stores time stamp information indicating timing at which processing using the Jacobian is performed in association with the Jacobian, and the control unit stores the Jacobian. When the used robot is controlled, the time stamp information associated with the Jacobian may be updated.

  This makes it possible to store the Jacobian in association with the time stamp.

  In the aspect of the invention, the control unit may determine the Jacobian to be deleted from the storage unit based on the time stamp information.

  Thereby, it becomes possible to determine the Jacobian to be deleted using the time stamp.

  In the aspect of the invention, the storage unit may store counter information indicating the number of times the robot is controlled using the Jacobian in association with the Jacobian.

  The Jacobian can be stored in association with the counter information.

  In one aspect of the present invention, the control unit is configured to obtain a difference value between the captured image feature amount obtained by controlling the robot using the Jacobian and a provisional target feature amount estimated from the Jacobian. When is less than or equal to a given threshold, the counter information associated with the Jacobian may be increased.

  This makes it possible to update the counter information based on a given condition.

  In the aspect of the invention, the control unit may adjust the value of the operation parameter obtained by the processing using the Jacobian based on the counter information associated with the Jacobian.

  This makes it possible to adjust the value of the operation parameter based on the counter information.

  According to another aspect of the present invention, a control unit that obtains a captured image feature amount that is an image feature amount of a captured image and controls a robot based on the captured image feature amount, a target feature amount, and a Jacobian. And a storage unit that stores the position and orientation of the work object or the end effector of the robot and the Jacobian in association with each other.

  In another aspect of the present invention, a Jacobian used for controlling the robot is stored in association with the position and orientation. Therefore, it is possible to use the Jacobian stored according to the situation, so that it becomes possible to reduce the processing load and speed up the robot operation.

  In another aspect of the present invention, a robot, a captured image feature amount that is an image feature amount of the captured image is obtained, and the control of the robot is performed based on the captured image feature amount, the target feature amount, and the Jacobian. The robot system includes a control unit that performs the above and a storage unit that associates and stores the work object or the position and orientation of the end effector of the robot and the Jacobian.

  In another aspect of the present invention, a Jacobian used for controlling the robot is stored in association with the position and orientation. Therefore, it is possible to use the Jacobian stored according to the situation, so that it becomes possible to reduce the processing load and speed up the robot operation.

  According to another aspect of the present invention, a captured image feature amount that is an image feature amount of a captured image is obtained, and the robot is controlled based on the captured image feature amount, the target feature amount, and the Jacobian. And a position and orientation of the work target or the end effector of the robot and the Jacobian in association with each other and storing the same.

  In another aspect of the present invention, a Jacobian used for controlling the robot is stored in association with the position and orientation. Therefore, it is possible to use the Jacobian stored according to the situation, so that it becomes possible to reduce the processing load and speed up the robot operation.

  As described above, according to some aspects of the present invention, it is possible to provide a robot, a control device, a robot system, a control method, and the like that reduce a Jacobian computation load in controlling a robot using an image.

The system structural example of the robot which concerns on this embodiment. Explanatory drawing of the relative relationship between a work target object and an end effector. 1 is a configuration example of a control device and a robot system according to the present embodiment. The example of the structure of the robot which concerns on this embodiment. The other example of the structure of the robot which concerns on this embodiment. The example which implement | achieves the control apparatus etc. which concern on this embodiment with a server system. Explanatory drawing of the mutual relationship of an image feature-value, a position orientation, and a joint angle. Explanatory drawing of visual servo control. An example of a local region set in the position and orientation space. 10A and 10B are diagrams showing the relationship between the joint angle and the image feature amount. The flowchart explaining the process of this embodiment. The figure explaining the specific process of this embodiment. The figure explaining the correction process of Jacobian. The example of the data structure memorize | stored in a memory | storage part. The other flowchart explaining the process of this embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. First, the method of this embodiment will be described. As described above, a technique for controlling a robot using a captured image is widely known. However, when the actual control is position control as in Patent Document 1, it is necessary to use a very rigid robot or to perform highly accurate calibration in order to increase the accuracy of the hand position and the like. is there. Also, the working environment of the robot must be kept constant. For example, the position where the work object (work) is supplied, the position where work is performed on the work object, the position where the work object after work is placed (material removal), etc. should also be highly accurate. Without it, the robot cannot be accurately controlled.

  In particular, the applicant assumes a robot that can operate flexibly according to various environmental changes. For example, a robot that recognizes the surrounding environment using various sensors such as an imaging unit (imaging sensor) and determines and corrects the operation content using the recognition result is conceivable. In such a robot, for example, when the user places a work object in the material supply area, an appropriate robot operation can be performed without deciding the position and orientation strictly. In other words, in the robot assumed by the present applicant, from the viewpoint of reducing the burden on the user, the work object or the like is not strictly positioned. It is difficult to control robots with high accuracy by using position control.

  As a robot control using a captured image, a visual servo as shown in Patent Document 2 is known. In the visual servo, image information is continuously acquired, and a result of comparison processing between information acquired from the image information and target information is fed back. Specifically, the robot is controlled in such a direction that the difference between the information acquired from the latest image information and the target information is reduced. More specifically, the amount of change in the joint angle that approaches the target is obtained, and control for driving the joint according to the amount of change is performed.

  Since visual servo feeds back the image processing results for the captured image, the current position / posture deviates from the target position / posture in the same way that humans can fine-tune how to move their arms and hands while viewing the work situation. Even so, the deviation can be recognized and corrected. Therefore, it is possible to control the robot with high accuracy.

In visual servoing, Jacobian is often used in processing. For example, the feature vector representing the target position or the current position is [F], and the joint angle vector of the robot is [θ]. In this case, the Jacobian [J] associates the feature vector difference [ΔF] and the joint angle vector difference [Δθ] with [ΔF] = [J] [Δθ]. In robot control using feedback processing, it is necessary to calculate [Δθ] as a control parameter when [ΔF] is obtained from a target value, an actual measurement value, and the like. Therefore, an inverse Jacobian that is an inverse matrix of Jacobian is used. Will be used. As will be described later with reference to FIG. 10B, in this embodiment, the relationship between [F] and [θ] is changed to two relationships between [F] and [P], and [P] and [θ]. By thinking separately, it is assumed that the Jacobian is considered as being divided into two groups of J _θP and J _PF , but here, in order to simplify the explanation, [F] and [θ] as shown in FIG. Think about the relationship.

  Here, [F] and [θ] are inherently non-linear relationships, and Jacobian J limits [F] and [θ] to a small change width based on a given value, whereby [ΔF] And [Δθ] are expressed as a linear relationship. That is, even if the Jacobian Ji can be used as an accurate approximation in the vicinity of a certain [θi], the approximation by the Jacobian Ji does not become accurate at [θj], which is somewhat different from [θi]. . Therefore, in the visual servo, the Jacobian used for processing must be changed every time the position and orientation of the arm or the like changes.

  In Patent Document 2, a necessary Jacobian is necessary for performing a series of robot operations (for example, operations including acquisition of a work object from a supply area, processing of the work object, and retreat of the work object to a material removal area). Is required every time. For this reason, the load for calculating the Jacobian is always large regardless of the number of past operations, the details of the operations, and the like, and the influence of slowing the movement speed of the robot can be considered.

  Therefore, the applicant stores the Jacobian calculated in the past in association with the position and orientation of the work object or the position and orientation of the hand of the robot, and reuses them as necessary to reduce the load of the Jacobian calculation. Propose robots. Specifically, as illustrated in FIG. 1, the robot 100 according to the present embodiment obtains a captured image feature amount that is an image feature amount of a captured image, and obtains a captured image feature amount, a target feature amount, and a Jacobian. Based on this, the control unit 110 that controls the robot, and the storage unit 120 that stores the work object or the position and orientation of the end effector 319 of the robot 100 and the Jacobian in association with each other are included.

  Here, the position and orientation can be expressed by, for example, the position of xyz in three axes and the rotation angle uvw around each axis. The position and orientation of the work object can be represented by the position of the reference position set for the work object in the three axes xyz and the rotation angle around each axis with respect to the reference position of the work object. The position and orientation of the end effector 319 are the same, and can be represented by the position of the reference position set for the end effector 319 in the three axes xyz and the rotation angle around each axis with respect to the reference posture of the end effector 319.

  If the gripping position / posture of the work object by the end effector 319 is specified, the position / posture of the work object and the position / posture of the end effector 319 can be converted to each other. For example, if the work object is OB and the gripping position / posture of the work object OB by the end effector 319 is known as shown in FIG. 2, the position / posture of the work object OB and the position / posture of the end effector 319 are determined. Can uniquely determine the other if one is determined. Therefore, in this specification, when the term “position / posture” is used without particular notice, either the position / posture of the work object or the position / posture of the end effector 319 may be indicated.

  The target feature amount is a feature amount representing a target state (goal state). The target feature amount may be, for example, an image feature amount acquired by image processing on an image in the target state (target image). However, the target image need not be used because the feature amount in the target state may be determined. For example, if it is known that the coordinate value on the image of the work object is (xg, yb) in the target state, the (xg, yb) may be used as the target feature amount as it is. It is not necessary to use the target image in the setting.

As will be described later with reference to FIGS. 10A and 10B, when the relationship between [F] and [θ] is captured with the position and orientation [P] of the work object or the end effector, the Jacobian is obtained. it can be divided into _J θP and _{J PF.} Among, J _{[theta] P} is because it can be determined from the structure of the robot analytically, the load in the Jacobian operation is high is J _PF. That is, when the position and orientation becomes [Pk], the processing load for _obtaining the Jacobian J _PkF corresponding to the [Pk] is large. _Therefore , if the J _PkF is obtained once, the J _PkF is _set to [ Pk] is stored in association with it. By doing so, when the position and orientation become [Pk ′] sufficiently close to [Pk] in the subsequent work, it is not necessary to calculate the corresponding Jacobian J _Pk′F from _scratch , and the stored J _PkF may be used.

  That is, in this embodiment, the control unit 110 of the robot 100 reads the Jacobian stored in the storage unit 120, and controls the robot 100 based on the read Jacobian, the captured image feature amount, and the target feature amount (specifically, Specifically, the arm 310 and the end effector 319 are controlled).

  In this way, since the Jacobian calculated in the past can be reused, the load of the Jacobian calculation can be reduced, and the operation of the robot 100 can be speeded up. Even when the method of the present embodiment is used, it is necessary to obtain the Jacobian necessary for the processing if the Jacobian at the position and orientation close to the position and orientation at the processing timing is not stored. Therefore, when the number of robot operations is small and the Jacobian is not sufficiently accumulated, the robot operation may take time. However, by repeating the robot operation and accumulating Jacobians, the effect of reducing the processing load increases, and the robot operation can be speeded up.

  Hereinafter, system configuration examples of the robot 100 and the control device 200 according to the present embodiment will be described, and then details of the processing of the present embodiment will be described.

2. System Configuration Example As shown in FIG. 1, the robot 100 according to the present embodiment includes a control unit 110, a storage unit 120, and a robot mechanism 300. The robot mechanism 300 includes, for example, an arm 310 and an end effector 319. . However, the robot 100 is not limited to the configuration of FIG. 1, and various modifications such as omitting some of these components or adding other components are possible. Further, the points that can be variously modified are the same in the configuration example of FIG.

  The storage unit 120 serves as a work area for the control unit 110 and the like, and its function can be realized by a memory such as a RAM, an HDD (hard disk drive), or the like. The storage unit 120 stores the Jacobian in association with the position and orientation as described above. Further, as will be described later with reference to FIG. 14 and the like, as additional information, a time stamp, the number of counters, a gain amount, and the like may be stored in association with the Jacobian and the position and orientation.

  Further, as shown in FIG. 3, the method of the present embodiment obtains a captured image feature amount that is an image feature amount of a captured image, and based on the captured image feature amount, the target feature amount, and the Jacobian, The present invention can be applied to a control unit 200 that includes a control unit 210 that performs the above-described control, and a storage unit 220 that stores the work object or the position and orientation of the end effector 319 of the robot 100 in association with the Jacobian. In this case, the control unit 210 of the control device 200 appropriately reuses the Jacobian stored in the storage unit 220 to operate the robot 100 (for example, the driving amount of each joint included in the arm 310). And the arm 310 of the robot 100 is controlled using the operation parameter.

  Further, the method of the present embodiment obtains a captured image feature amount that is an image feature amount of the captured image with the robot 100, and controls the robot 100 based on the captured image feature amount, the target feature amount, and the Jacobian. The present invention can be applied to a robot system including a control unit 210 to perform, and a storage unit 220 that stores a work object or a position and orientation of an end effector 319 of the robot 100 and a Jacobian in association with each other. The robot system here may be a system corresponding to the whole of FIG. 3, and may include the robot 100 and a control device 200 that controls the robot 100. In addition, here, the control unit is the control unit 210 included in the control device 200 and the storage unit 220 is the storage unit 220 included in the control device 200. However, the present invention is not limited to this. For example, the control unit of the robot system may be the control unit 110 included in the robot 100, and the storage unit of the robot system may be the storage unit 120 included in the robot 100. Alternatively, the functions of the control unit and the storage unit may be realized by distributed processing of the robot 100 and the control device 200. Alternatively, when the robot system includes another device (not shown) (for example, a server system) not shown in FIG. 3, the control unit and the storage unit may be included in the other device.

  The robot here may be a robot including a device 400 such as a PC and a robot body 500 as shown in FIG. 4, the apparatus 400 includes the control unit 110 shown in FIG. 1 or the like, or the apparatus 400 corresponds to the control apparatus 200 shown in FIG. 3. The robot body 500 includes an arm 310 and an end effector 319.

  Note that the configuration example of the robot 100 according to the present embodiment is not limited to FIG. For example, as illustrated in FIG. 5, the robot 100 may include a robot body 500 and a base unit unit 550. The robot according to the present embodiment may be a double-arm robot as shown in FIG. 5, and includes a first arm 310 and a second arm 320 in addition to portions corresponding to the head and the torso. In FIG. 5, the first arm 310 includes joints 311 and 313 and frames 315 and 317 provided between the joints, and the second arm 320 is similar, but is not limited thereto. In addition, although the example of the double-arm robot which has two arms was shown in FIG. 5, the robot of this embodiment may have three or more arms.

  The base unit 550 is provided below the robot body 500 and supports the robot body 500. In the example of FIG. 5, the base unit 550 is provided with wheels and the like so that the entire robot can move. However, the base unit 550 may be configured to be fixed to a floor surface or the like without having wheels or the like. In the robot system of FIG. 5, the robot main body 500 and the device 400 are integrally configured by storing the device 400 shown in FIG. 4 in the base unit 550.

  Alternatively, the controller 110 may be provided by a substrate (more specifically, an IC or the like provided on the substrate) built in the robot without providing a specific control device like the device 400 such as a PC. A robot that realizes and executes robot control can be considered.

  Further, as shown in FIG. 6, the functions of the control device 200 and the control unit 110 of the robot (hereinafter referred to as the control device) are a server that is connected to the robot through a network 600 including at least one of wired and wireless. 700 may be realized.

  Alternatively, in the present embodiment, a part of the processing of the control device or the like of the present invention may be performed by the control device or the like on the server 700 side. In this case, the processing is realized by distributed processing with a control device or the like provided on the robot side.

  In this case, the control device or the like on the server 700 side performs a process assigned to the control device or the like of the server 700 among the processes in the control device or the like of the present invention. On the other hand, a control device or the like provided in the robot performs a process assigned to the control device or the like of the robot among the processes of the control device or the like of the present invention.

  For example, the control device or the like of the present invention performs the first to Mth (M is an integer) processes, the first process is realized by the subprocess 1a and the subprocess 1b, and the second process is the subprocess. Consider a case where each of the first to Mth processes can be divided into a plurality of sub-processes, as realized by 2a and sub-process 2b. In this case, the control device or the like on the server 700 side performs sub-processing 1a, sub-processing 2a,... Sub-processing Ma, and the control device or the like provided on the robot side performs sub-processing 1b, sub-processing 2b,. Distributed processing such as performing processing Mb is conceivable. At this time, the control device or the like according to the present embodiment, that is, the control device or the like that executes the first to Mth processes may be a control device or the like that executes the sub-process 1a to the sub-process Ma, A control device or the like that executes sub-processing 1b to sub-processing Mb may be used, or a control device or the like that executes all of sub-processing 1a to sub-processing Ma and sub-processing 1b to sub-processing Mb. Furthermore, the control device or the like according to the present embodiment is a control device or the like that executes at least one sub-process for each of the first to M-th processes.

  As a result, for example, the server 700 having a higher processing capability than the terminal device on the robot side (for example, the control device 200 in FIG. 3) can perform processing with a high processing load. Furthermore, the server 700 can collectively control the operations of the robots, and for example, it is easy to cause a plurality of robots to perform coordinated operations.

  In recent years, the production of a small number of various types of parts has been increasing. And when changing the kind of components to manufacture, it is necessary to change the operation | movement which a robot performs. With the configuration as shown in FIG. 6, it is possible to change the operations performed by the robot in a batch without the server 700 performing the teaching operation for each of the plurality of robots again. Furthermore, compared with the case where one control device 200 or the like is provided for each robot, it is possible to significantly reduce the trouble of performing software update of the control device 200 or the like.

3. Details of Processing Details of processing according to the present embodiment will be described. First, robot control (visual servo) using a Jacobian that is generally used as a premise will be described. Next, Jacobian storage and reuse, which is a method of the present embodiment, will be described with reference to the flowchart of FIG. Further, after describing the Jacobian correction process corresponding to S105 in FIG. 11, additional information stored in association with the Jacobian will be described. Finally, a modification of Jacobian reuse will be described.

3.1 Control of Robot Using Jacobian Before explaining the method according to this embodiment, general visual servo control will be described. When the dimension number of the image feature quantity used for the visual servo is n (n is an integer), the image feature quantity F is expressed by an image feature quantity vector F = [F1, F2,..., Fn] ^T. The For each element of F, for example, a coordinate value or the like in an image of a feature point (control point) may be used. In this case, the target feature amount Fg is similarly expressed as Fg = [Fg1, Fg2,..., Fgn] ^T.

The joint angle is also expressed as a joint angle vector having a number of dimensions corresponding to the number of joints included in the robot 100 (arm 310 in a narrow sense). For example, if the arm 310 is a six-degree-of-freedom arm having six joints, the joint angle vector θ is expressed as θ = [θ1, θ2,..., Θ6] ^T.

  In the visual servo, when the current image feature amount F is acquired, the difference between the image feature amount F and the target feature amount Fg is fed back to the movement of the robot. Specifically, the robot is operated in a direction to reduce the difference between the image feature amount F and the target feature amount Fg. For this purpose, it is necessary to know the relationship that the image feature amount F changes if the joint angle θ is moved. In general, this relationship is non-linear. For example, when F1 = g (θ1, θ2, θ3, θ4, θ5, θ6), the function g is a non-linear function.

  Therefore, a technique using Jacobian J in the visual servo is widely known. Even if the two spaces are in a non-linear relationship, a minute amount of change in each space can be expressed in a linear relationship. Jacobian J associates the minute change amounts.

Specifically, when the position / posture P of the hand of the robot 100 is P = [x, y, z, u, v, w] ^T , the Jacobian between the change amount of the joint angle and the change amount of the position / posture J _{[theta] P} is expressed by the following equation (1), the Jacobian J _PF between the change amount and the image feature amount variation of the position and orientation is expressed by the following equation (2).

Then, by using J _θP and J _PF , the relationship between Δθ, ΔP, and ΔF can be expressed as the following expressions (3) and (4). J _θP is generally called a robot Jacobian, and can be calculated analytically if there is mechanism information such as the link length and rotation axis of the robot 100. Meanwhile, J _PF is to can be inferred from such pre-image feature amount when the position and orientation was small change in the amount of the hand of the robot 100 changes, it is also proposed a technique for any time estimate J _PF during operation Yes.
ΔP = J _θP Δθ (3)
ΔF = J _PF ΔP (4)

Furthermore, by using the above equations (3) and (4), the relationship between the image feature amount change amount ΔF and the joint angle change amount Δθ can be expressed as the following equation (5).
ΔF = J _θF Δθ (5)

Here, J _θF = J _θP J _PF and represents the Jacobian between the change amount of the joint angle and the change amount of the image feature amount. J _θF may be expressed as an image Jacobian. FIG. 7 illustrates the relationships of the above equations (3) to (5).

Based on the above, the control unit 110 may determine the joint angle drive amount (joint angle change amount) Δθ by using the difference between F and Fg as ΔF. In this way, it is possible to obtain the amount of change in the joint angle for bringing the image feature amount F close to Fg. Specifically, for determining the Δθ from [Delta] F, from left to both sides of the above equation (5) may be multiplied by the inverse matrix J _.theta.F ^-1 of J _.theta.F but further the control gain in consideration of lambda, the following formula ( In step 6), a target change amount Δθg of the joint angle is obtained.
Δθg = −λJ _θF ⁻¹ (F−Fg) (6)

In the above equation (6), it is _{assumed that} an inverse matrix J _θF ⁻¹ of J _θF can be obtained. However, when J _θF ⁻¹ cannot be obtained, a generalized inverse matrix (pseudo inverse matrix) J of J _θF is obtained. _θF ^# may be used.

By using the above equation (6), a new Δθg is obtained each time a new image is acquired. Therefore, it is possible to perform control to approximate the goal state (state where the image feature amount is Fg) while updating the target joint angle using the acquired image. This flow is illustrated in FIG. If the image feature amount F _m−1 is obtained from the _m− 1th image (m is an integer), Δθ _gm−1 can be obtained by setting F = F _{m−1 in} the above equation (6). Then, the robot 100 may be controlled between the m ₋ 1th image and the mth image as the next image with the calculated Δθ _gm−1 as a target. When the m-th image is acquired, an image feature amount Fm is obtained from the m-th image, and Δθ _gm that is a new target is calculated by the above equation (6). The calculated Δθ _gm is used for control between the m-th image and the (m + 1) -th image. Hereinafter, this process may be continued until the image converges (until the image feature amount is sufficiently close to Fg).

3.2 Use of Jacobian _{Stored in Storage Unit In} the description of the visual servo described above, J _θF is used as the Jacobian. In the present embodiment, as shown in the above equations (3) and (4) to, think to separate the Jacobian to _J θP and _{J PF.}

Originally, the nonlinearity between θ and F is very strong. Therefore, the _region where J _θF is an accurate approximation becomes narrow, and even if J _θF is stored in association with the joint angle θ, it becomes difficult to reuse J _θF . Specifically, even Jacobian J _ShitaFk has been stored in association with the (vector represented by .theta.k for example seven joint angle here) given joint angle .theta.k, reuse the Jacobian J _ShitaFk What can be done is limited to a range in which the joint angle is very close to θk. Therefore, the frequency with which the stored Jacobian is reused decreases. In addition, in order to skip computation from one Jacobian at a given joint angle, a very large number of Jacobians are stored (for example, when a space related to a joint angle is divided into a lattice as shown in FIG. There is a risk of squeezing the memory capacity.

Here, the relationship between θ and F shown in FIG. 10A is obtained by interposing the position and orientation P, and as shown in FIG. Can be separated into the very weak relationship between P and F. Then, if that stored in association with J _PF and the position and orientation P, since weak nonlinearity of P and F, easily it can be reused possible to widen the area where J _PF is a good approximation is there. In other words, in the present embodiment, a _JPF that can be easily reused is stored in association with P, and when an operation parameter (for example, Δθ) necessary for actual robot control is obtained from ΔF, as necessary. J _θP is obtained analytically and used.

First, in this embodiment, the space related to the position and orientation P is divided into local regions. For example, the space may be divided into a lattice as shown in FIG. The size of this local region is set so that one Jacobian can be approximated accurately if it is a position and orientation within the local region. In this way, when a Jacobian is obtained at one position and orientation in a given local region, the Jacobian is reused as an accurate approximation in other positions and orientations within the given local region. It becomes possible to do. As described above, by using the J _PF, it is possible to increase the local region size as compared with the case of using the J _.theta.F.

  Although FIG. 9 shows a lattice-shaped region set in a three-dimensional space, the position and orientation P is six dimensions (x, y, z, u, v, w), so the local region is also six dimensions. It is set for the space.

  FIG. 11 shows a flowchart for explaining the processing of this embodiment. When this process is started, target information is first acquired (S101). Here, the target information is, for example, a target feature amount or a position and orientation corresponding to the target feature amount. Then, it is determined whether or not a Jacobian corresponding to the current position and orientation is stored in the storage unit 120 (S102). Specifically, it is determined whether a Jacobian is stored for the local region including the current position and orientation. That is, the current position / posture and the position / posture associated with the stored Jacobian need not completely match, and there may be an error corresponding to the local region size. This is because, as described above, it is assumed that the local region is set so that an error caused by using the Jacobian does not become a problem.

If the corresponding Jacobian is stored, the Jacobian is read from the storage unit 120 (S103), and visual servoing is executed. In this case, in the present embodiment, this means that J _PF is read in order to determine the actual operating parameters (e.g. [Delta] [theta]), is that the robot Jacobian J _{[theta] P} is required as described above. J _{[theta] P} is may be analytically determined in each case for example, from J _{[theta] P} can be calculated if the structure of the robot is known, for various theta, in advance J _{[theta] P} (before robot operation) You may ask for it. In any case, the Jacobian calculation load in this embodiment is a problem of the Jacobian load related to F that cannot be obtained analytically, and the calculation load of _JθP does not become a large problem.

  When the Jacobian is acquired, visual servo is performed, and it is determined whether the robot has approached the goal as a result (S104). When approaching the goal (Yes in S104), it is determined that the Jacobian used for the process is appropriate, and the counter update process (S106) and the time stamp update process (S107) are performed. On the other hand, when it is not approaching the goal (No in S104), the Jacobian correction process is performed assuming that the Jacobian used for the process is not appropriate (S105). Details of the correction processing in S105 will be described later. After the process of S105, a time stamp update process is performed (S107).

  The counter number and the time stamp are information stored in the storage unit 120 in association with the Jacobian. Details of the counter number and the time stamp will be described later. Further, since the processing of S108 and S109 uses the number of counters, this processing will also be described later. In any case, the steps from S103 to S109 are cases in which the Jacobian corresponding to the current position and orientation is stored in the storage unit 120. In this case, it is not necessary to calculate the Jacobian from scratch, so that the processing load is reduced. Is possible.

  In summary, the control unit 110 according to the present embodiment determines whether or not the Jacobian corresponding to the position or orientation of the work object or the end effector 319 at the processing timing is stored in the storage unit 120 and responds accordingly. When the Jacobian is stored, the operation parameter (for example, Δθ) of the robot 100 is obtained based on the difference information (ΔF) between the captured image feature quantity and the target feature quantity and the Jacobian read from the storage unit 120. The robot 100 is controlled according to the operation parameters.

  On the other hand, when the Jacobian corresponding to the current position and orientation is not stored in the storage unit 120 (No in S102), a new Jacobian is generated (calculated) (S110). For example, the change in the image feature amount F when (x, y, z, u, v, w) is slightly changed independently from the current position and orientation may be obtained. In addition, since various methods for calculating the Jacobian using the actually measured values are known, detailed description thereof is omitted.

  When the Jacobian is calculated in S110, it is determined whether or not there is an empty storage area (entry) related to the Jacobian in the storage unit 120 (S111). If there is a vacancy (Yes in S111), the Jacobian is stored in the vacant area in association with the current position and orientation, the time stamp corresponding to the Jacobian is set to the current time, and the counter is initialized. The value is reset (S113). The number of counters and the time stamp here are the same as those used in S106 and the details will be described later.

  On the other hand, if there is no vacancy in the entry (No in S111), one of the entries is deleted to create a vacancy (S112), and the Jacobian obtained in S110 for the vacant area is replaced with the position / posture and counter. The number is stored in association with the number (S113). Details of the process of S112 will be described later.

  In summary, the control unit 110 according to the present embodiment sets the position / posture of the processing timing when the Jacobian corresponding to the position / posture of the work object or the end effector 319 at the processing timing is not stored in the storage unit 120. The corresponding Jacobian is obtained, and the storage unit 120 stores the obtained Jacobian in association with the work object or the position and orientation of the end effector 319 at the processing timing.

  By performing such processing, when there is already a Jacobian corresponding to the current position and orientation, the Jacobian is reused as in S103 to S109, and when there is no corresponding Jacobian as in S110 to S113. A new Jacobian can be calculated. Therefore, it is possible to reduce the load related to the Jacobian calculation as compared with the conventional method disclosed in Patent Document 2 while appropriately performing robot control (visual servo) using the Jacobian.

  The process flow of FIG. 11 will be described using a specific example. In the following, in order to simplify the explanation, as shown in FIG. 12, the position and orientation is set as a two-dimensional space, and the planar grid area is set as a local area. However, as described above, the processing is expanded to a six-dimensional space or the like. Think about it.

  As shown in FIG. 12, when local regions a1 to f6 are set, it is assumed that robot control is performed to move to a5 starting from f1. At this time, it is assumed that Ja2, Jb2, Jd3, Je2, Je3, Jf1, and Jf2 are stored in the storage unit 120 as Jacobians that have been calculated in the past robot control and have not been deleted until now. Here, Jxy (x is any one of a to f and y is any one of 1 to 5) represents a Jacobian corresponding to the local region xy.

In this case, since the position and orientation are included in f1 after the start, the corresponding Jacobian Jf1 is stored in the storage unit 120. Therefore, processing corresponding to S103 to S109 is performed, and an operation parameter is obtained using Jf1 (and corresponding robot Jacobian J _θP ), and the robot 100 is controlled based on the operation parameter.

  After that, as shown by the curve in FIG. 12, when the position / posture is included in the local region f2, it is determined whether the Jacobian corresponding to f2 is stored. Also in this case, since Jf2 is stored, the processing corresponds to S103 to S109.

  As a result of controlling the robot in this way, the work object or the end effector 319 moves in the order of f1 → f2 → e2 → d3 according to the curved path of FIG. 12, during which the corresponding Jacobian is stored. Therefore, a new Jacobian operation is not necessary. However, in the example of FIG. 12, as a result of performing control using Jd3, the position and orientation become the position and orientation included in c3. In this case, the corresponding Jc3 is not stored in the storage unit 120.

  In such a situation, processing corresponding to S110 to S113 in FIG. 11 is performed. Specifically, a Jacobian corresponding to the current position and orientation is newly calculated and stored as Jc3 in association with the position and orientation. . At this time, if the process of S112 is performed, Jc3 is stored after deleting any of the existing Jacobians.

  In the example of FIG. 12, the position and orientation is moved to the local region b3 by the control using the obtained Jc3, and the corresponding Jacobian is not stored in this case, so Jb3 is newly calculated. Hereinafter, the same applies to a3, a4, and a5, and the work target or the end effector 319 is moved to a5 by controlling the robot while newly calculating the Jacobian.

  When the robot operation shown in FIG. 12 is completed and the next robot operation is performed, the Jacobian at the end of the robot operation in FIG. 12 becomes the target of reuse. For example, if the deletion process in S112 has not been performed, in addition to Ja2, Jb2, Jd3, Je2, Je3, Jf1, Jf2 stored before the operation of FIG. 12, Jc3, Jb3, Ja3, Ja4 Ja5 can also be reused in subsequent robot operations. In other words, as long as the storage capacity permits, the accumulation of Jacobian tends to progress as the robot operation is repeated. Therefore, the possibility of new calculation of Jacobian is lower in later operations, and high-speed robot operations and the like are possible.

  In the above description, whether or not the Jacobian corresponding to the current position and orientation is stored in the storage unit 120 is determined using the local region in the six-dimensional space (x, y, z, u, v, w). It was. In other words, each Jacobian is stored as an index (x, y, z, u, v, w), and the presence / absence of a Jacobian corresponding to the current position and orientation is searched using the index. be able to. However, it is not necessary to use all six dimensions of the position and orientation as the index, and only the three dimensions (x, y, z) may be used. In this case, it is only necessary to divide the xyz three-dimensional space into local regions and determine whether or not the Jacobian corresponding to the local region corresponding to the current position is stored.

  This can be realized, for example, by performing a conversion process between an orthogonal coordinate system set in a three-dimensional space in which a local region is set and a posture of a robot end effector 319 (robot hand coordinate system). Specifically, when a Jacobian corresponding to a given position / posture (x, y, z, u, v, w) is calculated, the posture portion (u, v, w) of the position / posture is represented by xyz. It is memorized after performing a conversion process to obtain a posture corresponding to a given orthogonal coordinate system defined in the space. The posture corresponding to the calculated Jacobian is usually not constant, and various postures are conceivable. This corresponds to a process of unifying this into a given reference posture. In this case, since it is not necessary to consider the posture in the memory of the Jacobian, the index associated with the Jacobian may be three, xyz, and the local region is set in the three-dimensional space as shown in FIG. It corresponds to that.

  When the stored Jacobian is read and used, conversion processing is performed using the relationship between the robot posture at that time and the reference posture. The Jacobian with the three xyz as an index can be used with high accuracy under the condition that the posture of the end effector 319 is the reference posture, and cannot be used in any posture. However, since the robot itself can know the posture of the end effector 319, mutual conversion between the posture and the reference posture is easy. In other words, the Jacobian stored using xyz as an index is searched using the current xyz, and when the corresponding Jacobian is found, it is not used as it is, but the reference posture is set as the current posture. You may use it after adding conversion processing. In this way, the position and orientation (index) stored in association with the Jacobian is not (x, y, z, u, v, w), but three (x, y, z). Therefore, it is possible to easily search for a desired Jacobian.

  Note that even if only three indexes (x, y, z) are used, the processing is only performed by unifying the posture to the given reference posture, and information on the posture is completely lost. It is not. That is, even in this case, there is no change in that information regarding the posture is associated with each Jacobian. Therefore, the description “store the position and orientation of the work target or the robot end effector and the Jacobian in association with each other” in the present specification is (x, y, z, u, v, w) as the Jacobian index. ), And a case using three (x, y, z).

3.3 Jacobian Correction Processing Next, the Jacobian correction processing corresponding to S105 in FIG. 11 will be described. As shown in S105, the control unit 110 performs a correction process for correcting the Jacobian when the visual servo is not successful, and the storage unit 120 converts the Jacobian after the correction process into a work object or an end effector 319. It is stored in association with the position and orientation of. Here, whether or not the visual servo has succeeded is determined by a difference value between the captured image feature amount obtained by controlling the robot 100 using the Jacobian and the provisional target feature amount estimated from the Jacobian. What is necessary is just to determine whether it is larger than a threshold value, and when a difference value is below a threshold value, it determines with visual servo having succeeded.

  In order to facilitate the explanation, the following will first consider the one-dimensional case as shown in FIG. Since x in FIG. 13 actually corresponds to P or θ, it becomes a 6-dimensional or 7-dimensional variable. Since F also actually corresponds to the image feature amount, the order is generally larger than the first order.

As shown in FIG. 13, at a given timing n, the value of the variable is _{x n,} an image feature amount is assumed to be _{F (x n) = F n} . Furthermore, Jacobian at x _n is the J _n, and J _n are corresponded to the straight line shown in A1 of FIG. 13. Since the Jacobian corresponds to the coefficient of the first-order differential term, J _n corresponds to the slope of the straight line A1 in the example of FIG.

In this case, if the target value of the image feature amount at the next timing n + 1 is F _{n + 1} , the variable x _n after the change can be obtained by changing the variable x _n by Δx by using the Jacobian J _n corresponding to A1. It is possible to obtain x _{n + 1} (or Δx) at which the image feature amount is expected to be the target value F _{n + 1 at n + 1} (= x _n + Δx). Here, F _{n + 1} corresponds to the provisional target feature amount described above.

However, since the Jacobian J _n includes an error, when the variable is changed to x _{n + 1} , the image feature amount does not become the expected F _{n + 1,} but becomes F ^* _{n + 1} = F (x _{n + 1} ). . In this case, the Jacobian J _n that was to be used in x _n is considered to be more appropriate for the A1 rather A2. This is because, as can be seen from FIG. 13, when the variable is changed from _xn to _{xn + 1} , A2 changes the change in the image feature amount F accurately (at least so as to match the actual measurement value).

Therefore, as the Jacobian correction processing, A1 using the variables x _n and x _{n + 1} (Δx) actually used, the actually acquired feature amounts F _n and F ^* _{n + 1} , and the expected feature amount F _{n + 1} are used. the _{J n} corresponding to processing may be performed to change _{J n + 1} corresponding to A2. Specifically, since the corrected Jacobian J _{n + 1} has an inclination of the straight line A2, the following expression (7) or the following expression (8) obtained by modifying the following expression (7) may be used.

When used in actual processing, as described above, F may be expanded to the number of dimensions of the image feature amount, and X may be expanded to the number of dimensions of P or θ. Specifically, the above equation (7) may be expanded to the following equation (9), or the above equation (8) may be expanded to the following equation (10).

  The Jacobian correction process described here is merely an example, and other methods may be used. Specifically, nonlinear optimization methods such as Broyden's method can be widely applied.

3.4 Additional Information Next, additional information stored in association with the Jacobian will be described. As illustrated in S106 of FIG. 11, the storage unit 120 stores counter information indicating the number of times the robot 100 is controlled using the Jacobian in association with the Jacobian.

  In addition, since the process of S106 is performed when Yes in S104 (when the visual servo is successful) and is not performed when No in S104, the corresponding Jacobian is processed by the corresponding Jacobian. It can be considered that this is the number of times that the process has been used successfully (for example, the process has appropriately approached the target state).

That is, the control unit 110 according to the present embodiment has a difference value between a captured image feature amount obtained by controlling the robot 100 using a Jacobian and a provisional target feature amount estimated from the Jacobian equal to or less than a given threshold value. If it is, an update process for increasing the counter information associated with the Jacobian is performed. Here, the provisional target feature amount corresponds to F _{n + 1} in FIG. 13, and is a feature amount that the captured image feature amount is expected to be the value when there is no error in the Jacobian. Note that, as shown in the above equation (6), since a gain amount λ of 1 or less is generally used for calculation, the provisional target feature amount is different from the target feature amount corresponding to the goal state.

  That is, a large number of counters associated with a Jacobian indicates that appropriate processing has been performed many times by the Jacobian. Therefore, in the present embodiment, the control unit 110 may adjust the value of the operation parameter obtained by the processing using the Jacobian based on the counter information associated with the Jacobian. Specifically, it is conceivable to increase the value of the gain amount λ shown in the above equation (6). By increasing the gain amount, the distance for moving the robot 100 at a time in the processing using the corresponding Jacobian increases. Normally, when the movement distance is increased, an error for ideal movement is increased. However, for a Jacobian estimated to have a large number of counters and high accuracy, the error can be suppressed to a small value. Further, if the gain amount is increased and the movement amount is increased accordingly, the robot 100 can be brought into a target state at a high speed.

  This process is shown in S108 and S109 in FIG. 11. In S108, the counter number is compared with a given threshold value. If the counter number is larger than the threshold value, the gain amount is updated (S109). Specifically, an update process for increasing the gain amount may be performed. In S109, the gain amount is increased and the counter number is reset. In this way, it is possible to prevent the gain amount associated with a given Jacobian from increasing continuously. Further, since the error increases as the gain amount is increased, it is possible to determine again whether or not the Jacobian is highly accurate even in such a situation.

  Further, as shown in S107, a time stamp may be used as additional information. Specifically, the storage unit 120 stores time stamp information indicating the timing at which processing using the Jacobian is performed in association with the Jacobian, and the control unit 110 controls the robot 100 using the Jacobian. If so, processing is performed to update the time stamp information associated with the Jacobian. More specifically, in S107, the time stamp information is overwritten using the processing timing using the Jacobian (for example, the calculation timing in S103 or the timing at which the processing in S107 is performed). The processing to be performed may be performed. In this way, for each Jacobian stored in the storage unit 120, it is possible to specify the timing at which the Jacobian was last used for processing.

  In that case, as shown in S112, when the storage capacity is limited, it is possible to determine the Jacobian to be deleted based on the time stamp. Specifically, the control unit 110 performs processing for determining a Jacobian to be deleted from the storage unit 120 based on the time stamp information. More specifically, among the stored Jacobians, the Jacobian with the oldest time stamp may be set as the deletion target.

  In the flowchart of FIG. 11 and the above description, the counter number is used to set the gain amount, and the time stamp is used to determine the deletion target. However, the present invention is not limited to this. The number of counters can be used to determine the deletion target, the degree of gain variation can be controlled according to the time stamp, and the counter number and time stamp can also be used in processing related to other index values. .

  FIG. 14 shows an example of the data structure stored in the storage unit 120 of this embodiment. The storage unit 120 stores the Jacobian in association with the position and orientation, the counter number, the time stamp, and the gain amount. The Jacobian, for example, associates the amount of change in position and orientation with the amount of change in image feature, and is a matrix corresponding to the number of dimensions of the position and orientation and the number of dimensions of the image feature. The position and orientation are expressed by, for example, six dimensions (x, y, z, u, v, w). However, in the example using a given reference posture as an index as described above, the posture information can be unified to the given reference posture (ur, vr, wr), so only xyz is stored in the data structure. It is good. The number of counters can be expressed using, for example, a given integer. The time stamp may be information expressed using the date and time, or may be a relative time (elapsed time) based on a given timing. The gain amount is, for example, a number of 1 or less, and is information used for processing in the above equation (6). However, the structure of the data stored in the storage unit 120 is not limited to that shown in FIG. 14, and a part thereof may be omitted (for example, at least one of the counter number, time stamp, and gain amount is omitted).

3.5 Modified Example Finally, a modified example of the present embodiment will be described. In the above description, the local area corresponding to the position and orientation at the processing timing is specified, and when the Jacobian corresponding to the local area is stored, the Jacobian is reused. That is, in consideration of errors, the position and orientation range in which the Jacobian can be reused is limited to a narrow range, and if it is outside the range, the Jacobian is newly calculated.

  However, the method of the present embodiment is not limited to this, and the Jacobian of the local region in the vicinity (for example, adjacent) may be used even if the corresponding local region Jacobian is not stored. Specifically, when the Jacobian corresponding to the position or orientation of the work object or the end effector 319 at the processing timing is not stored in the storage unit 120, the control unit 110 calculates the position and orientation coordinates from the position and orientation at the processing timing. It is determined whether a Jacobian corresponding to a position / posture within a predetermined range in which the spatial distance in the space is within a given distance range is stored in the storage unit 120. When the Jacobian corresponding to the position / posture within the predetermined range is stored in the storage unit 120, the Jacobian corresponding to the position / posture within the predetermined range is obtained as the Jacobian corresponding to the position / posture at the processing timing, and the storage unit 120 stores the Jacobian in association with the position and orientation of the work object or the end effector at the processing timing. In the example of FIG. 12, when the Jacobian Jc3 corresponding to the local region c3 is not stored, the Jacobian is not newly calculated, but Jd3 or Jb2 is used as the Jacobian corresponding to c3.

  In this way, since the distance range in which the Jacobian can be reused can be widened, the frequency of newly calculating the Jacobian can be reduced, and the processing load can be reduced and the robot operation speeded up. Here, the spatial distance in the position / posture coordinate space is, for example, a spatial distance in a six-dimensional space of (x, y, z, u, v, w). Various distances such as Euclidean distance and Mahalanobis distance can be used as the spatial distance.

  However, if the distance range in which the Jacobian is reused becomes wider, the difference between the position and orientation at which the Jacobian should be used and the position and orientation at the processing timing will increase, and the accuracy of processing using the Jacobian will decrease (target feature value) You may not be able to get close to it properly). Therefore, as described above with reference to FIG. 13 and the like, it is desirable not only to use the nearby Jacobian but also to perform a correction process that reduces the error.

  In this case, when the Jacobian corresponding to the position and orientation within the predetermined range is not stored in the storage unit 120, the control unit 110 obtains (newly calculates) the Jacobian corresponding to the position and orientation at the processing timing, and stores the storage unit. 120 stores the Jacobian in association with the work object or the position and orientation of the end effector 319 at the processing timing.

  FIG. 15 is a flowchart for explaining the processing of the above modification. S201 to S209 in FIG. 15 are the same as S101 to S109 in FIG. If the Jacobian corresponding to the local region including the position and orientation at the processing timing is stored, the Jacobian may be used. On the other hand, when the Jacobian corresponding to the local region including the position and orientation of the processing timing is not stored, instead of immediately calculating a new Jacobian, the Jacobian that falls within the given distance range is searched as described above. (S210). If a Jacobian is found here, processing is performed using the Jacobian (S211), and it is determined whether the visual servo is successful (S212). If successful, the Jacobian may be stored as it is, and if not successful, the correction processing may be performed as in S205 (S213) and stored. The storage of the Jacobian is the same as in the case of performing a new calculation, and as shown in S215 to S217, the same processing as S111 to S113 may be performed.

  Further, in this modification, when No in S202 and No in S210, that is, when neither the Jacobian corresponding to the local area at the processing timing nor the nearby Jacobian is stored, the Jacobian is newly calculated ( S214). The processing after S214 may be performed from S215 to S217.

  In the above description, when a new Jacobian is calculated, a value with a certain degree of validity is obtained, such as using an actual measurement value when each value of position and orientation is minutely changed independently. However, as described above with reference to FIG. 13, the Jacobian can be corrected sequentially. That is, a randomly generated value may be used as the initial value of the Jacobian. In this case, immediately after random generation, it is not possible to define in which direction the robot moves (by which direction the captured image feature value changes) by processing using the Jacobian, so the robot can be brought closer to the target state. difficult. However, if the process using the Jacobian is repeated and the correction process is performed, the Jacobian becomes highly accurate.

  In other words, if it is assumed that the Jacobian is sequentially corrected, the Jacobian calculation in S110 or S214 may use only a randomly generated value. In this case, the processing load required for the Jacobian calculation is extremely high. Can be light. For example, as described above, if the method is to change the position and orientation slightly, it is necessary to execute a robot operation when obtaining one Jacobian, but if it is generated randomly, the Jacobian can be generated without operating the robot. is there.

  Note that the control device 200 or the like according to the present embodiment may realize part or most of the processing by a program. In this case, the control device 200 of this embodiment is realized by a processor such as a CPU executing a program. Specifically, a program stored in a non-temporary information storage medium is read, and a processor such as a CPU executes the read program. Here, the information storage medium (computer-readable medium) stores programs, data, and the like, and functions as an optical disk (DVD, CD, etc.), HDD (hard disk drive), or memory (card type). It can be realized by memory, ROM, etc. A processor such as a CPU performs various processes according to the present embodiment based on a program (data) stored in the information storage medium. That is, in the information storage medium, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit) Is memorized.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. Further, the configuration and operation of the robot, the control device, and the like are not limited to those described in the present embodiment, and various modifications can be made.

100 robot, 110 control unit, 120 storage unit, 200 control device,
210 control unit, 220 storage unit, 300 robot mechanism, 310 arm,
311,313 joint, 315,317 frame, 319 end effector,
320 arm, 400 device, 500 robot body, 550 base unit,
600 networks, 700 servers

Claims (15)

A control unit that obtains a captured image feature amount that is an image feature amount of the captured image, and that controls the robot based on the captured image feature amount, the target feature amount, and the Jacobian;
A storage unit for storing a work object or a position and orientation of an end effector of the robot in association with the Jacobian;
A robot characterized by including: In claim 1,
The controller is
A robot that reads the Jacobian stored in the storage unit and controls the robot based on the Jacobian, the captured image feature quantity, and the target feature quantity. In claim 2,
The controller is
It is determined whether or not the Jacobian corresponding to the position and orientation of the work object or the end effector at the processing timing is stored in the storage unit, and the captured image is stored when the Jacobian is stored. A robot characterized in that an operation parameter of the robot is obtained based on difference information between a feature value and the target feature value and the Jacobian, and the robot is controlled according to the operation parameter. In claim 3,
The controller is
When the Jacobian corresponding to the position and orientation of the work object or the end effector at the processing timing is not stored in the storage unit, the Jacobian corresponding to the position and orientation is obtained.
The storage unit
The robot storing the Jacobian in association with the position and orientation of the work object or the end effector at the processing timing. In claim 3,
The controller is
When the Jacobian corresponding to the position and orientation of the work object or the end effector at the processing timing is not stored in the storage unit,
From the position and orientation at the processing timing, it is determined whether the Jacobian corresponding to the position and orientation within a predetermined range in which the spatial distance in the position and orientation coordinate space is within a given distance range is stored in the storage unit,
When the Jacobian corresponding to the position / posture within the predetermined range is stored in the storage unit, the Jacobian corresponding to the position / posture within the predetermined range is changed to the Jacobian corresponding to the position / posture at the processing timing. As sought
The storage unit
The robot storing the Jacobian in association with the position and orientation of the work object or the end effector at the processing timing. In claim 5,
The controller is
When the Jacobian corresponding to the position and orientation within the predetermined range is not stored in the storage unit, the Jacobian corresponding to the position and orientation at the processing timing is obtained,
The storage unit
The robot storing the Jacobian in association with the position and orientation of the work object or the end effector at the processing timing. In any one of Claims 1 thru | or 6.
The controller is
When the difference value between the captured image feature amount obtained by controlling the robot using the Jacobian and the provisional target feature amount estimated from the Jacobian is larger than a given threshold, the Jacobian is determined. correct,
The storage unit
A robot characterized in that the corrected Jacobian is stored in association with the position and orientation of the work object or the end effector. In any one of Claims 1 thru | or 7,
The storage unit
Time stamp information indicating the timing at which the processing using the Jacobian is performed is stored in association with the Jacobian,
The controller is
A robot that updates the time stamp information associated with the Jacobian when the robot is controlled using the Jacobian. In claim 8,
The controller is
The robot that determines the Jacobian to be deleted from the storage unit based on the time stamp information. In any one of Claims 1 thru | or 9,
The storage unit
Counter information representing the number of times the robot is controlled using the Jacobian is stored in association with the Jacobian. In claim 10,
The controller is
When the difference value between the captured image feature amount obtained by controlling the robot using the Jacobian and the provisional target feature amount estimated from the Jacobian is equal to or less than a given threshold, the Jacobian A robot characterized by increasing the counter information associated therewith. In claim 10 or 11,
The controller is
A robot characterized in that an operation parameter value obtained by processing using the Jacobian is adjusted based on the counter information associated with the Jacobian. A control unit that obtains a captured image feature amount that is an image feature amount of the captured image, and that controls the robot based on the captured image feature amount, the target feature amount, and the Jacobian;
A storage unit for storing a work object or a position and orientation of an end effector of the robot in association with the Jacobian;
The control apparatus characterized by including. With robots,
A control unit that obtains a captured image feature amount that is an image feature amount of the captured image and controls the robot based on the captured image feature amount, the target feature amount, and the Jacobian;
A storage unit for storing a work object or a position and orientation of an end effector of the robot in association with the Jacobian;
A robot system characterized by including: Obtaining a captured image feature amount that is an image feature amount of the captured image, and controlling the robot based on the captured image feature amount, the target feature amount, and the Jacobian;
Storing the work object or the position and orientation of the end effector of the robot in association with the Jacobian;
The control method characterized by including.

Images