JP2016013610A - Robot, and control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot which can calculate a Jacobian matrix without moving an imaging unit or an object.SOLUTION: A robot acquires a first image of an object disposed at a second position captured by an imaging unit disposed at a first position, and operates using a plurality of second images corresponding to the images obtained by capturing the object from a plurality of positions different from the first position.

Description

  The present invention relates to a robot and a control method.

  A technique for causing a robot to perform a predetermined operation based on a captured image captured by an imaging unit has been researched and developed. In particular, in automating assembly work in a factory, it is indispensable to estimate the position and orientation of a work target from a captured image captured by an imaging unit.

  In relation to this, a robot apparatus that estimates the position and orientation of a work target using an ESM (Efficient Second-order Minimization Method) method is known (see Patent Document 1).

JP2012-185752A

  However, in order for a conventional robot apparatus to estimate the position and orientation of a three-dimensional object, it is necessary to derive a Jacobian matrix of the three-dimensional object. In order to derive the Jacobian matrix, it is generally necessary to move the imaging unit or the object minutely. Therefore, in a situation where the imaging unit or the object cannot be moved minutely, the Jacobian matrix cannot be derived, and the position and orientation of the three-dimensional object cannot be estimated using the ESM method.

  Accordingly, the present invention has been made in view of the above-described problems of the prior art, and provides a robot that can perform an operation without moving an imaging unit or an object, and a control method.

One embodiment of the present invention acquires a first image of an object placed at a second position taken by an imaging unit placed at a first position, and picks up the object from a plurality of positions different from the first position. This is a robot that operates using a plurality of second images corresponding to the obtained images.
With this configuration, the robot acquires the first image of the object arranged at the second position captured by the imaging unit arranged at the first position, and images the object from a plurality of positions different from the first position. It operates using a plurality of second images corresponding to. Thereby, the robot can perform work without moving the imaging unit or the object.

In another aspect of the present invention, the robot may be configured to operate based on a Jacobian matrix calculated using the plurality of second images.
With this configuration, the robot operates based on the Jacobian matrix calculated using the plurality of second images. As a result, the robot can perform work using the calculated Jacobian matrix without moving the imaging unit or the object.

According to another aspect of the present invention, in the robot, the Jacobian matrix calculates a step size based on a difference image between the reference model of the object and the first image, and is based on the calculated step size. The configuration calculated in the above may be used.
With this configuration, the robot calculates a step size based on a difference image between the reference model of the object and the first image, and calculates a Jacobian matrix based on the calculated step size. Accordingly, the robot can calculate the Jacobian matrix based on an appropriate step size corresponding to the degree of progress of optimization.

In another aspect of the present invention, the robot may have a configuration in which the plurality of second images are generated by performing a minute rotation on the object included in the first image.
With this configuration. The robot generates a plurality of second images by performing micro rotation on the object included in the first image. Thereby, the robot can obtain a plurality of images obtained when the imaging unit 10 is moved without moving the imaging unit 10.

According to another aspect of the present invention, in the robot, the plurality of second images are generated by performing minute translation on the object included in the first image after performing the minute rotation. May be used.
With this configuration, the robot performs a minute rotation on the object included in the first image and then performs a minute translation on the object included in the first image, thereby generating a plurality of the second images. Thereby, the robot can obtain a plurality of images that are images with small errors and that would be obtained when the imaging unit is moved.

According to another aspect of the present invention, a first image of an object arranged at a second position captured by an imaging unit arranged at a first position is acquired, and the object is a plurality of different images from the first position. In this control method, the robot is operated using a plurality of second images corresponding to images picked up from a position.
With this configuration, the control method acquires the first image of the object arranged at the second position picked up by the image pickup unit arranged at the first position, and picks up the object from a plurality of positions different from the first position. The robot is operated using a plurality of second images corresponding to the images. Thereby, the control method can perform work without moving the imaging unit or the object.

  As described above, the robot and the control method acquire the first image of the object arranged at the second position captured by the imaging unit arranged at the first position, and move the object from a plurality of positions different from the first position. It operates using a plurality of second images corresponding to the captured images. Thereby, the robot and the control method can perform work without moving the imaging unit or the object.

It is a lineblock diagram showing an example of robot system 1 concerning this embodiment. 2 is a diagram illustrating an example of a hardware configuration of a control device 30. FIG. 3 is a diagram illustrating an example of a functional configuration of a control device 30. FIG. 3 is a flowchart illustrating an example of a flow of processing in which a control device 30 causes a robot 20 to perform a predetermined operation. It is a figure which illustrates three different-direction images among the six different-direction images generated by the image generation unit and a captured image before micro rotation by homography conversion is performed.

<Embodiment>
Hereinafter, this embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a configuration diagram illustrating an example of a robot system 1 according to the present embodiment. The robot system 1 includes an imaging unit 10, a robot 20, and a control device 30.

  The robot system 1 images the object O placed on the table TB by the imaging unit 10, and calculates the position and orientation of the object O based on the captured image. The table TB is a table on which the object O is placed. The object O is an object related to work performed by the robot 20, and is, for example, industrial parts such as screws, bolts, gears, jigs, containers for storing industrial parts, etc., but is not limited thereto. Other objects may be used as long as they are objects related to work performed by the robot 20. In FIG. 1, the object O is represented as a rectangular parallelepiped object. Hereinafter, for convenience of explanation, the position and orientation of the object O will be referred to as the object position and orientation. The position on the table TB where the object O is installed is an example of the second position.

  When calculating the object position and orientation, the robot system 1 causes the robot 20 to grip the object O based on the calculated object position and orientation. Then, the robot system 1 causes the robot 20 to place the object O at a predetermined placement position X (not shown). Hereinafter, for convenience of explanation, a series of operations of the robot system 1 from imaging of the object O by the imaging unit 10 to arranging the object O at the arrangement position X will be referred to as predetermined work.

  The imaging unit 10 is a camera including, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like, which is an imaging element that converts collected light into an electrical signal. Moreover, although the imaging part 10 is a monocular camera comprised by one camera, for example, you may be comprised by two or more cameras.

  The imaging unit 10 is connected to the control device 30 via a cable so as to be communicable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB (Universal Serial Bus), for example. The imaging unit 10 and the control device 30 may be configured to be connected by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  A three-dimensional coordinate system Σ with the optical center as the origin is set in the imaging unit 10. The Z axis of the three-dimensional coordinate system Σ is set to coincide with the optical axis direction of the imaging unit 10. Further, the X axis and the Y axis of the three-dimensional coordinate system Σ are set to be orthogonal to each other and orthogonal to the Z axis. The three-dimensional coordinate system Σ represents coordinates on a captured image (that is, an image sensor) captured by the imaging unit 10. The imaging unit 10 is installed, for example, so that the optical axis (that is, the Z axis) is orthogonal to a virtual plane set in advance on the object O.

  This virtual plane is a plane including a point associated with the object O, for example, a plane including a corner of the object O and the like. In addition, the imaging unit 10 is installed at a position where a range including the object O installed on the table TB (hereinafter referred to as an imaging range) can be imaged. The position where the imaging unit 10 capable of imaging the imaging range is installed is an example of a first position. In the robot system 1, the imaging unit 10 may be configured to be installed on the arm of the robot 20 (for example, between the manipulator MNP and the gripping unit HND).

  The robot 20 is, for example, a single-arm robot including a gripping unit HND, a manipulator MNP, and a plurality of actuators (not shown). The single-arm robot refers to a robot having one arm composed of a gripping unit HND and a manipulator MNP. Further, the robot system 1 may be a scalar robot (horizontal articulated robot), a double-arm robot, or the like, instead of a configuration including a single-arm robot. A scalar robot is a robot in which the manipulator moves only in the horizontal direction and only the slide shaft at the tip of the manipulator moves up and down. Further, the double-armed robot refers to a robot having two arms each constituted by a gripper HND and a manipulator MNP.

The arm of the robot 20 is a 6-axis vertical articulated type, and can operate with 6 degrees of freedom by an operation in which the support base, the manipulator MNP, and the gripper HND are linked by an actuator. The arm of the robot 20 may operate with 5 degrees of freedom (5 axes) or less, or may operate with 7 degrees of freedom (7 axes) or more. Below, the operation | movement of the robot 20 performed with the arm provided with the holding part HND and the manipulator MNP is demonstrated.
The grip part HND of the robot 20 includes a claw part that can grip an object. The gripping part HND is an example of a hand.

The robot 20 is communicably connected to the control device 30 by a cable, for example. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The robot 20 and the control device 30 may be connected by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark). Further, in the robot system 1, the robot 20 is configured to be connected to the control device 30 installed outside the robot 20 as shown in FIG. The structure built in 20 may be sufficient.
The robot 20 acquires a control signal from the control device 30, grips the object O based on the acquired control signal, and places the gripped object O at the placement position X.

  The control device 30 causes the imaging unit 10 to capture the above-described imaging range as a still image or a moving image, and grips the object O to the robot 20 by visual servoing based on the captured image (the above-described still image or moving image). The gripped object O is placed at the placement position X. More specifically, the control device 30 calculates the object position and orientation based on the captured image captured by the imaging unit 10 and the distance d illustrated in FIG.

  Here, the distance d indicates a distance between the origin of the three-dimensional coordinate system Σ set in the imaging unit 10 and a virtual plane set in advance on the object O described above. It is assumed that information indicating the distance d is registered in the control device 30 in advance. The control device 30 calculates the Jacobian matrix of the object O based on the captured image captured by the imaging unit 10, the distance d registered in advance, and the reference model of the object O registered in advance. Note that the distance d may be approximately the distance between the imaging unit 10 and the table TB, or may be a distance shorter or longer than the distance between the imaging unit 10 and the table TB by a predetermined distance.

  The reference model is three-dimensional model data obtained by three-dimensional modeling of the object O or the shape, color, pattern, etc. of the object O. The reference model is a comparison target with the object O imaged by the imaging unit 10, and is represented by, for example, three-dimensional computer graphics (CG). This three-dimensional CG is represented as a set of polygon data, for example.

  The control device 30 calculates the object position and orientation based on the calculated Jacobian matrix of the object O. The control device 30 controls the robot 20 to hold the object O by the holding unit HND based on the calculated object position and orientation. Then, the control device 30 controls the robot 20 so that the object O grasped by the grasping unit HND is arranged at the arrangement position X.

  Next, the hardware configuration of the control device 30 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration of the control device 30. The control device 30 includes, for example, a CPU (Central Processing Unit) 31, a storage unit 32, an input reception unit 33, and a communication unit 34, and communicates with the control device 30 via the communication unit 34. These components are connected to each other via a bus Bus so that they can communicate with each other. The CPU 31 executes various programs stored in the storage unit 32.

  The storage unit 32 includes, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a ROM (Read-Only Memory), a RAM (Random Access Memory), and the like. Various information, images, programs, information indicating the distance d described above, information indicating the reference model described above, and the like are stored. Note that the storage unit 32 may be an external storage device connected by a digital input / output port such as a USB instead of the one built in the control device 30.

The input receiving unit 33 is, for example, a keyboard, a mouse, a touch pad, or other input device. Note that the input receiving unit 33 may be configured as a touch panel by functioning as a display unit.
The communication unit 34 includes, for example, a digital input / output port such as USB, an Ethernet (registered trademark) port, and the like.

  Next, the functional configuration of the control device 30 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a functional configuration of the control device 30. The control device 30 includes a storage unit 32, an image acquisition unit 35, and a control unit 36. Part or all of the functional units included in the control unit 36 is realized by the CPU 31 executing various programs stored in the storage unit 32, for example. Some or all of these functional units may be hardware functional units such as LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit).

The image acquisition unit 35 acquires a captured image from the imaging unit 10. The image acquisition unit 35 outputs the acquired captured image to the control unit 36.
The control unit 36 controls the entire control device 30. The control unit 36 includes an imaging control unit 37, an image generation unit 38, a Jacobian matrix calculation unit 39, and a robot control unit 40.
The imaging control unit 37 causes the imaging unit 10 to image a range including the object O.

  The image generation unit 38 reads information indicating the distance d from the storage unit 32. Further, the image generation unit 38 reads information indicating the reference model from the storage unit 32. Further, the image generation unit 38 acquires a captured image from the image acquisition unit 35. The image generation unit 38 captures the object O from different directions based on the captured image acquired from the image acquisition unit 35, the distance d read from the storage unit 32, and the reference model read from the storage unit 32. A captured image (hereinafter, referred to as a different direction image) obtained in such a case is generated by image processing. Hereinafter, these processes performed by the image generation unit 38 will be collectively referred to as an image generation process. The captured image acquired from the image acquisition unit 35 is an example of a first image. In addition, the different direction image generated by the image generation unit 38 is an example of a second image.

  The Jacobian matrix calculation unit 39 reads information indicating a reference model from the storage unit 32, and calculates a Jacobian matrix for calculating the object position and orientation by the ESM method based on the read reference model. The Jacobian matrix in the present invention refers to a product obtained by differentiating an image feature vector with a rigid body motion parameter (the image feature vector may be a simple array of pixel values). Hereinafter, for convenience of explanation, this Jacobian matrix will be referred to as a reference Jacobian matrix.

  The Jacobian matrix is a matrix including parameters necessary for a nonlinear minimization method used for estimating an object position and orientation, for example. Nonlinear minimization methods include, for example, the ESM method, the Levenberg-Marquardt method, the Newton method, the Gauss-Newton method, and the like. Of these nonlinear minimization methods, the ESM method has a smaller number of iterative calculations than other methods, and can be suitably used when it is desired to estimate the object position and orientation at a higher speed. Therefore, in the following description, it is assumed that the ESM method is adopted as the nonlinear minimization method and the calculated Jacobian matrix is a parameter used for the ESM method.

  Further, the Jacobian matrix calculation unit 39 relates to the object O based on the read reference model, the captured image acquired by the image acquisition unit 35, and the plurality of different direction images generated by the image generation unit 38. Calculate the Jacobian matrix. Hereinafter, for convenience of explanation, this Jacobian matrix will be referred to as a calculated Jacobian matrix. The Jacobian matrix calculation unit 39 is an example of a calculation unit.

  The robot control unit 40 calculates the object position and orientation by the ESM method based on the reference Jacobian matrix calculated by the Jacobian matrix calculation unit 39 and the calculated Jacobian matrix calculated by the Jacobian matrix calculation unit 39. Further, the robot control unit 40 controls the robot 20 so that the object O is gripped by the gripping unit HND based on the calculated object position and orientation. After the robot 20 grips the object O by the gripping unit HND, the robot control unit 40 controls the robot 20 so as to place the object O at the placement position X.

  Hereinafter, with reference to FIG. 4, a process in which the control device 30 causes the robot 20 to perform a predetermined operation will be described. FIG. 4 is a flowchart illustrating an example of a process flow in which the control device 30 causes the robot 20 to perform a predetermined operation. First, the imaging control unit 37 controls the imaging unit 10 so as to image a range including the object O (step S100).

  Next, the image acquisition unit 35 acquires the captured image captured by the imaging unit 10 from the imaging unit 10 (step S110). Next, the image generation unit 38 performs image generation processing on the captured image acquired by the image acquisition unit 35 (step S120). Here, the image generation process will be described. The image generation unit 38 reads information indicating the distance d from the storage unit 32. The image generation unit 38 performs homography conversion on the captured image acquired by the image acquisition unit 35 based on the read distance d.

  The homography conversion is a conversion in which the first plane is projected onto a second plane different from the first plane by using projective conversion. By this homography conversion, the image generation unit 38 captures a captured image (the first plane described above) in which a range including the object O is captured by the imaging unit 10 from a certain direction, and the object O from the other direction by the imaging unit 10. Approximately converted to a captured image (the above-described second plane) acquired when the range to be captured is captured.

  In other words, the image generation unit 38 has acquired, from the image acquisition unit 35, a captured image that would be acquired from the image acquisition unit 35 when the imaging unit 10 captured a range including the object O from another direction. Approximately generated by performing homography conversion on the captured image. The image generation unit 38 performs this homography conversion on the object O included in the captured image. Thereby, the image generation part 38 can suppress the calculation amount accompanying this homography conversion compared with the case where homography conversion is performed with respect to the whole captured image.

  When the imaging unit 10 is virtually micro-rotated and micro-translated around any one of the X, Y, and Z axes of the three-dimensional coordinate system Σ shown in FIG. This is the direction in which the optical axis of the imaging unit 10 faces. Hereinafter, a captured image in which a range including the object O is captured by the imaging unit 10 from another direction will be referred to as a different direction image. The positions of the plurality of virtual imaging units 10 when the imaging unit 10 is virtually micro-rotated and micro-translated are examples of a plurality of positions different from the first position.

  There are two directions in which the imaging unit 10 is slightly rotated around each of the X axis, the Y axis, and the Z axis of the three-dimensional coordinate system Σ, a positive direction and a negative direction. Therefore, the image generation unit 38 rotates the object O included in the captured image slightly in the positive direction and the negative direction around the X axis, the Y axis, and the Z axis of the three-dimensional coordinate system Σ. To generate six different-direction images. Note that the image generation unit 38 reads the minute rotation amount relating to each of the X axis, the Y axis, and the Z axis stored in the storage unit 32 in advance, and the object O reflected in the captured image based on the read minute rotation amount. Slightly rotate.

  Here, with reference to FIG. 5, the process in which the image generation part 38 produces | generates a different direction image is demonstrated. FIG. 5 shows images in the positive direction around the X axis, Y axis, and Z axis of the three-dimensional coordinate system Σ described above by homography conversion among the six different-direction images generated by the image generation unit 38. Examples of three different-direction images that can be approximately used as images picked up when the unit 10 is slightly rotated and a captured image acquired from the image acquisition unit 35 before the minute rotation by the homography conversion is illustrated. It is a figure to do. Note that the processing related to the generation of three different-direction images that can be approximately used as an image captured when the imaging unit 10 is slightly rotated in the negative direction around each of the X, Y, and Z axes is as follows. Since it is the same as the process related to the generation of the three different-direction images shown in FIG.

  An image P shown in FIG. 5A is an example of a captured image acquired from the image acquisition unit 35 before the homography conversion is performed. The image P includes (images) the object O. An image PX shown in FIG. 5B is an example of a different direction image that can be approximately used as an image captured when the imaging unit 10 is slightly rotated in the positive direction around the X axis. In the image PX, the object O after the minute rotation by the homography conversion is performed on the object O included in the image P is shown. A dotted line PO on the image PX indicates the contour of the object O (that is, the contour of the object O included in the image P) before the minute rotation by the homography conversion is performed.

  As shown in FIG. 5B, since the image PX is an image before the minute translation by the homography conversion is performed, the object O is deviated from the dotted line PO. This shift occurs because the position where the optical axis of the imaging unit 10 intersects the above-described virtual plane is shifted before and after the micro rotation due to the micro rotation due to the homography conversion. Further, the magnitude Δdy of this deviation is expressed by the following equation in accordance with the distance d between the imaging unit 10 and the above-described virtual plane when the minute rotation is performed by the minute angle Δθ by homography conversion. It can be calculated by (1).

Δdy = d × tan (Δθ) ≈dΔθ (1)

  Here, the rightmost side of the above equation (1) indicates an approximation due to Δθ being a minute angle. Therefore, the image generation unit 38 can be approximately used as an image captured when the imaging unit 10 is slightly rotated by a small angle Δθ around the X axis of the three-dimensional coordinate system Σ by homography conversion. When generating the different direction image, the object O included in the image P is rotated slightly by homography conversion to generate the image PX, and then the image PX is changed by Δdy in the Y-axis direction based on the above equation (1). The contained object O is slightly translated in the direction opposite to the direction in which the optical axis of the imaging unit 10 is moved when the imaging unit 10 is slightly rotated by a minute angle Δθ. Thereby, the image generation unit 38 slightly rotates the imaging unit 10 by a small angle Δθ in the positive direction around the X axis of the three-dimensional coordinate system Σ among the different direction images necessary for calculating the Jacobian matrix. A different direction image that can be used as an image that is sometimes captured is generated.

  An image PY shown in FIG. 5B is an example of a different direction image that can be used approximately as an image captured when the imaging unit 10 is slightly rotated in the positive direction around the Y axis. The image PY shows the object O after the object O included in the image P has been finely rotated by homography conversion. A dotted line PO on the image PY indicates the contour of the object O before the homography conversion is performed.

  As in the case of the image PX, the image generation unit 38 is an image captured when the imaging unit 10 is slightly rotated by a small angle Δφ in the positive direction around the Y axis of the three-dimensional coordinate system Σ by homography conversion. When generating an approximately usable different-direction image, an object P included in the image P is micro-rotated by homography conversion to generate an image PY, and then the X-axis direction based on the following equation (2) The object O included in the image PX by Δdx is slightly translated in the direction opposite to the direction in which the optical axis of the imaging unit 10 moved when the imaging unit 10 is slightly rotated by the minute angle Δφ.

Δdx = d × tan (Δφ) ≈dΔφ (2)

  Here, the rightmost side of the above equation (1) indicates an approximation due to Δθ being a minute angle. Thereby, the image generation unit 38 slightly rotates the imaging unit 10 by a small angle Δφ in the positive direction around the Y axis of the three-dimensional coordinate system Σ among the different direction images necessary for calculating the Jacobian matrix. A different direction image that can be used as an image that is sometimes captured is generated.

  An image PZ shown in FIG. 5B is an example of a different direction image that can be used approximately as an image captured when the imaging unit 10 is slightly rotated in the positive direction around the Z axis. The image PZ shows the object O after a minute rotation by homography conversion is performed on the object O included in the image P. A dotted line PO on the image PZ indicates the contour of the object O before the homography conversion is performed.

  When the imaging unit 10 is rotated around the Z axis, the imaging unit 10 is rotated around the optical axis of the imaging unit 10, and thus the position where the optical axis of the imaging unit 10 and the above-described virtual plane intersect. However, it does not shift due to the influence of minute rotation by homography conversion. Accordingly, the image generation unit 38 differs from the image PX or the image PY among the different-direction images necessary for calculating the Jacobian matrix around the Z-axis of the three-dimensional coordinate system Σ without performing a minute translation. A different direction image that can be approximately used as an image captured when the imaging unit 10 is slightly rotated by a minute angle Δφ in the positive direction is generated.

  Next, the Jacobian matrix calculation unit 39 calculates a reference Jacobian matrix and a calculated Jacobian matrix (step S130). More specifically, the Jacobian matrix calculation unit 39 reads the reference model from the storage unit 32. When the reference model is read by the image generation unit 38, the Jacobian matrix calculation unit 39 may perform processing described below using the reference model. When the reference Jacobian matrix is not calculated, the Jacobian matrix calculation unit 39 calculates a reference Jacobian matrix based on the reference model.

  Here, the derivation of the reference Jacobian matrix will be described. The Jacobian matrix calculation unit 39 reads a reference model and derives a Jacobian matrix when the read reference model exists at an arbitrary position. The Jacobian matrix calculation unit 39 performs all the processing for calculating the reference Jacobian matrix on the CG. Therefore, it is not necessary to perform the homography conversion that is performed when the calculated Jacobian matrix is derived. An image obtained when the imaging unit is moved minutely is generated. At this time, since a shift due to a minute rotation change also occurs on the CG, the Jacobian matrix calculation unit 39 performs the process of performing the minute translation (paragraphs [0046] to [0054]) performed when obtaining the calculated Jacobian matrix. When deriving the Jacobian matrix, it is necessary to perform it on the CG.

  After calculating the reference Jacobian matrix, the Jacobian matrix calculation unit 39 outputs and stores information indicating the reference Jacobian matrix in the storage unit 32. The Jacobian matrix calculation unit 39 reads the reference Jacobian matrix from the storage unit 32 when the reference Jacobian matrix has already been calculated. Further, the Jacobian matrix calculation unit 39 calculates a calculation Jacobian matrix based on the captured image acquired from the image acquisition unit 35 and the six different-direction images generated by the image generation unit 38.

  The Jacobian matrix calculation unit 39 performs numerical differentiation used for the calculation of the Jacobian matrix based on a pre-registered step size to calculate a three-dimensional calculation Jacobian matrix. However, when the distance between the reference model and the captured image acquired from the image acquisition unit 35 is large, the Jacobian matrix calculation unit 39 sets the numerical differentiation step size Δ used for calculating the Jacobian matrix to the following equation (3). The configuration may be calculated based on the above.

Δ = u (x) × (1.0 + α × (E / E ₀ ) × Δ ₀ ) (3)

Here, α indicates a design constant, for example, 200. E indicates the number of non-zero pixels in the difference image between the reference model and the captured image acquired from the image acquisition unit 35. E ₀ indicates the total number of pixels included in the contour of the object O in the captured image acquired from the image acquisition unit 35. u (x) is a step function, for example, a function that takes one of 1, 2, 4, and 8 depending on the value of the argument x. For example, u (x) is 8 when x> 8, 4 when 4 <x ≦ 8, 2 when 2 <x ≦ 4, and 1 when x is any other value. Δ ₀ indicates a basic step size set in the design stage. Incidentally, α, u (x), each delta _0, and are stored in advance in the storage section 32 as a parameter information.

  When calculating the step size Δ based on the above equation (3), the Jacobian matrix calculation unit 39 reads the reference model from the storage unit 32. The image generation unit 38 generates a difference image between the reference model read from the storage unit 32 and the captured image acquired from the image acquisition unit 35. The Jacobian matrix calculation unit 39 reads the parameter information from the storage unit 32, and based on the read parameter information, the generated difference image, and the captured image acquired from the image acquisition unit 35, the above formula The step size Δ is calculated using (3).

  The Jacobian matrix calculation unit 39 calculates a calculated Jacobian matrix using the calculated step size Δ. Thereby, the Jacobian matrix calculation unit 39 can increase the step size Δ as compared with the case where the step size Δ is not calculated based on the above equation (3). As a result, the Jacobian matrix calculation unit 39 can suppress the amount of calculation required to converge the distance between the object O reflected in the captured image acquired from the image acquisition unit 35 and the reference model.

  Next, the robot control unit 40 calculates the object position and orientation based on the reference Jacobian matrix calculated in Step S130 and the calculated Jacobian matrix. Then, the robot control unit 40 controls the robot 20 so as to grip the object O by the gripping unit HND based on the calculated object position and orientation (step S140). Next, the robot controller 40 controls the robot 20 so that the object O gripped by the gripper HND is placed at the placement position X described above (step S150).

  As described above, the robot system 1 according to the present embodiment acquires a captured image that is a captured image and includes the object O installed on the table TB, and the object O is a position where the imaging unit 10 is installed. A plurality of different direction images corresponding to images taken from a plurality of different positions are generated, and the robot 20 is operated using the generated plurality of different direction images. Accordingly, the robot system 1 can cause the robot 20 to perform an operation without moving the imaging unit 10 or the object O.

  Further, the robot system 1 operates the robot 20 based on the Jacobian matrix calculated using a plurality of different direction images. As a result, the robot system 1 can cause the robot 20 to perform work using the Jacobian matrix calculated without moving the imaging unit 10 or the object O.

  The robot system 1 calculates a step size based on a difference image between the reference model of the object O and the captured image acquired from the image acquisition unit 35, and calculates a calculated Jacobian matrix based on the calculated step size. To do. Thereby, the robot system 1 can suppress the calculation amount for calculating the calculated Jacobian matrix.

  Further, the robot system 1 generates a plurality of different-direction images by performing minute rotation on the object O included in the captured image acquired from the image acquisition unit 35. Thereby, the robot system 1 can obtain a plurality of images that would be obtained when the imaging unit 10 is moved without moving the imaging unit 10.

  In addition, the robot system 1 performs a minute rotation on the object O included in the captured image acquired from the image acquisition unit 35 and then performs a minute translation on the object O included in the captured image, so that a plurality of different directions are obtained. Generate an image. Thereby, the robot system 1 can obtain a plurality of images that are images with small errors and that would be obtained when the imaging unit 10 is moved as a plurality of different-direction images.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and changes, substitutions, deletions, and the like are possible without departing from the gist of the present invention. May be.

  Further, a program for realizing the function of an arbitrary component in the apparatus described above (for example, the control apparatus 30 of the robot system 1) is recorded on a computer-readable recording medium, and the program is read into the computer system. May be executed. Here, the “computer system” includes hardware such as an OS (Operating System) and peripheral devices. “Computer-readable recording medium” means a portable disk such as a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a CD (Compact Disk) -ROM, or a hard disk built in a computer system. Refers to the device. Further, the “computer-readable recording medium” means a volatile memory (RAM: Random Access) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Memory that holds a program for a certain period of time, such as Memory).

In addition, the above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
Further, the above program may be for realizing a part of the functions described above. Further, the program may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

DESCRIPTION OF SYMBOLS 1 Robot system, 10 Imaging part, 20 Robot, 30 Control apparatus, 31 CPU, 32 Storage part, 33 Input reception part, 34 Communication part, 35 Image acquisition part, 36 Control part, 37 Imaging control part, 38 Image generation part, 39 Jacobian matrix calculation unit, 40 Robot control unit

Claims (6)

Obtaining a first image of an object placed at a second position imaged by an imaging unit placed at a first position;
Operating using a plurality of second images corresponding to images obtained by imaging the object from a plurality of positions different from the first position;
robot. The robot according to claim 1,
Operating based on a Jacobian matrix calculated using a plurality of the second images;
robot. The robot according to claim 2,
The Jacobian matrix calculates a step size based on a difference image between the reference model of the object and the first image, and is calculated based on the calculated step size.
robot. The robot according to claim 1 or 2,
The plurality of second images are generated by performing a minute rotation on the object included in the first image.
robot. The robot according to claim 4,
The plurality of second images are generated by performing minute translation on the object included in the first image after performing the minute rotation.
robot. Obtaining a first image of an object placed at a second position imaged by an imaging unit placed at a first position;
Operating the robot using a plurality of second images corresponding to images obtained by imaging the object from a plurality of positions different from the first position;
Control method.

Images