JP2015100868A - Robot system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot system enabled to control a robot by shortening the time period which is necessary for estimating the position and the attitude of an object in a three-dimensional space.SOLUTION: A robot system comprises: a robot; an imaging part for imaging an object; and a control part for operating the robot, wherein the imaging part includes a plurality of imaging elements including a first imaging element and a second imaging element. The control part operates the robot such that the object is imaged in a first direction by the first imaging element, and such that the object is imaged by the second imaging element in a second direction different from the first direction.

Description

  The present invention relates to a robot system.

  Currently, when work is performed by an industrial robot, the position of an object to be handled by the industrial robot needs to be determined in advance by means such as using a jig. On the other hand, it is an operation that cannot fix the position of an object in advance, and there are many operations that use an industrial robot. The work in which the position of the object cannot be fixed is, for example, assembly of objects or labeling on the object. In order for an industrial robot to perform these operations, the industrial robot needs to recognize the position and orientation of an object in a three-dimensional space accurately and at high speed. In recent years, research and development of a system for recognizing the position and orientation of an object in a three-dimensional space by using an image captured by a camera has been actively conducted because of such necessity.

  In relation to this, an image including an index arranged in the space is captured, the distribution range of the index is derived based on the index included in the captured image, and an algorithm according to the size of the derived distribution range Based on the above, there is known a position and orientation measurement apparatus that estimates the position and orientation of an object in a three-dimensional space (see Patent Document 1).

JP 2006-242943 A

  However, in a conventional position / orientation measurement apparatus or the like, as an example of a nonlinear optimization technique for measuring a position and orientation in a three-dimensional space, an example using Newton's method or the like that requires many iterative calculations is given. However, in the method using the Newton method, the position and orientation of the object in the three-dimensional space may not be estimated in a short time.

  Accordingly, the present invention has been made in view of the above-described problems of the prior art, and provides a robot system capable of controlling a robot while reducing the time required for estimating the position and orientation of an object in a three-dimensional space. .

[1] The present invention has been made to solve the above-described problem, and includes a robot, an imaging unit that images an object, and a control unit that operates the robot. A plurality of image sensors including one image sensor and a second image sensor; and the control unit images the object from a first direction by the first image sensor, and the first image by the second image sensor. The robot system operates the robot by imaging the object from a second direction different from the direction.

  With this configuration, the robot system includes an imaging unit including a plurality of imaging elements including a first imaging element and a second imaging element, and images the object from the first direction by the first imaging element, and the second imaging Since the robot is operated by imaging the object from the second direction different from the first direction by the element, the time required for estimating the position and orientation of the object in the three-dimensional space can be reduced to control the robot. it can.

[2] The present invention is the robot system according to [1], in which the control unit calculates a three-dimensional Jacobian matrix of the object based on a plurality of captured images captured by the imaging unit. And comparing the calculated three-dimensional Jacobian matrix with a reference three-dimensional Jacobian matrix as a reference to estimate the position and posture of the object, and based on the estimated position and posture, the robot A robot system to be operated.

  With this configuration, in the robot system, the control unit calculates the three-dimensional Jacobian matrix of the object based on the plurality of captured images captured by the imaging unit, the calculated three-dimensional Jacobian matrix, and the reference third order as a reference In order to estimate the position and orientation of the object by comparing with the original Jacobian matrix, and to operate the robot based on the estimated position and orientation, for example, estimation using an ESM (Efficient Second-order Minimization Method) method This method estimates the position and orientation of an object using a method, and controls the robot by shortening the time required to estimate the position and orientation of the object compared to the estimation method using a nonlinear minimization method such as the conventional Newton method. Can do.

[3] The present invention is the robot system according to [1] or [2], wherein the imaging unit further includes a half mirror and a plurality of mirrors, and the half mirror and the plurality of mirrors Thus, the robot system collects light from the object incident from the first direction and the second direction on the first image sensor and the second image sensor.

  With this configuration, in the robot system, the imaging unit collects light from an object incident from the first direction and the second direction on the first imaging element and the second imaging element by the half mirror and the plurality of mirrors. A plurality of images captured from different directions can be acquired without changing the position of the imaging unit, and as a result, the time required for estimating the position and orientation of the object can be shortened. In addition, no blurring occurs in the captured image, and the estimation accuracy can be improved.

[4] The present invention is the robot system according to [3], in which the imaging unit branches light from the object incident from the first direction and the second direction by the half mirror. By doing so, the robot system collects light from the object incident from the first direction and the second direction on the first image sensor and the second image sensor.

  With this configuration, in the robot system, the imaging unit branches light from an object incident from the first direction and the second direction by the half mirror, so that the object from the object incident from the first direction and the second direction is split. Since light is collected on the first image sensor and the second image sensor, an object can be imaged simultaneously from different directions by a single imaging unit, and the position and orientation of the object can be estimated in a shorter time.

[5] The present invention is the robot system according to [3], in which the imaging unit further includes a plurality of lenses arranged on a plane parallel to the plane of the imaging element and having different focal points. A robot system that captures an image including information in a depth direction obtained by the plurality of lenses.

  With this configuration, in the robot system, the imaging unit further includes a plurality of lenses having different focal points arranged on a plane parallel to the plane of the imaging element, and includes information in the depth direction obtained by the plurality of lenses. Since an image is captured, an optical system composed of a half mirror and a plurality of mirrors is not required, and the position and orientation of an object can be estimated in a shorter time.

It is a figure which shows an example of the utilization condition of the robot system 1 in 1st Embodiment. 2 is a diagram illustrating an example of a hardware configuration of an object position / orientation estimation apparatus 30. FIG. 3 is a diagram illustrating an example of a functional configuration of an object position / orientation estimation apparatus 30. FIG. 3 is a diagram illustrating an example of a dedicated optical system opt included in the imaging unit 10. FIG. It is a figure explaining an example of the condition where the images pic1-pic4 of the object imaged from a different direction are imaged by the imaging part 10 provided with the exclusive optical system opt shown in FIG. 5 is a flowchart illustrating an example of an operation flow of the object position / orientation estimation apparatus 30. It is a figure which shows an example of the utilization condition of the robot system 1 in 2nd Embodiment.

<First Embodiment>
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram illustrating an example of a usage status of the robot system 1 according to the first embodiment. The robot system 1 in the first embodiment includes, for example, an imaging unit 10, an object position / posture estimation device 30, and a robot body 50. The imaging unit 10 is a camera including an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), which is an imaging element that converts the collected light into an electrical signal. The imaging unit 10 captures still images and moving images. Hereinafter, in order to simplify the description, the imaging unit 10 captures a still image.

  The imaging unit 10 is communicably connected to the object position / orientation estimation apparatus 30 via a cable Cbl. Wired communication via the cable Cbl is performed according to standards such as HDMI (High-Definition Multimedia Interface (registered trademark)) and USB (Universal Serial Bus). The imaging unit 10 and the object position / orientation estimation apparatus 30 may be connected by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark). The imaging unit 10 captures the target object bx placed on the table tbl from different directions, and outputs the captured image to the object position / orientation estimation apparatus 30 via the cable Cbl. Details of the method of imaging from different directions will be described later. In addition, the image imaged from each direction may be a plurality of images corresponding to each direction (for example, a first direction and a second direction different from the first direction), or a plurality of images corresponding to different directions. May be one image picked up collectively by one image sensor. A plurality of images may be combined and handled as one image. The “plurality of captured images” in the claims may be any of these. Hereinafter, in order to simplify the description, it is assumed that images taken from each direction are a plurality of images corresponding to the respective directions.

  The object position / orientation estimation apparatus 30 is connected to the robot body 50 via a cable Eth so as to be communicable. Wired communication via the cable Eth is performed according to a communication standard such as Ethernet (registered trademark), for example. The object position / orientation estimation apparatus 30 and the robot body 50 may be connected by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark). In FIG. 1, the object position / orientation estimation apparatus 30 is installed outside the robot body 50, but may be mounted inside the robot body 50. The object position / orientation estimation apparatus 30 controls the position and orientation of the grip part hnd attached to the robot body 50 based on the captured image acquired from the image capturing unit 10, and the target object bx is determined in advance by the grip part hnd. Move to destination.

  The robot main body 50 is, for example, a 6-axis vertical articulated robot that is movably provided with a gripping portion hnd, and performs a motion with 6 degrees of freedom by a coordinated operation of the support base, the arm portion, and the gripping portion hnd. Can do. The grip portion hnd includes a claw portion that can grip or clamp an object. The robot body 50 may operate with 5 degrees of freedom or less, or may operate with 7 degrees of freedom or more.

  Next, the hardware configuration of the object position / orientation estimation apparatus 30 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration of the object position / posture estimation apparatus 30. The object position / orientation estimation apparatus 30 includes, for example, a CPU (Central Processing Unit) 31, a storage unit 32, an input reception unit 33, and a communication unit 34, and the other imaging unit 10 and robot via the communication unit 34. It communicates with the main body 50. The CPU 31 executes various programs stored in the storage unit 32. The storage unit 32 includes HDD (Hard Disk Drive), SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read-Only Memory), ROM (Read-Only Memory, etc.). Information and programs to be processed by the posture estimation device 30 are stored. The storage unit 32 may be an external storage device in place of the storage unit 32 built in the object position / posture estimation device 30.

  The input receiving unit 33 is a keyboard, mouse, touch pad, or other input device. The input receiving unit 33 may function as a display unit, and may be configured as a touch panel. The communication unit 34 includes an Ethernet (registered trademark) port as well as a digital input / output port such as HDMI or USB.

  Next, the functional configuration of the object position / orientation estimation apparatus 30 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a functional configuration of the object position / posture estimation apparatus 30. The object position / orientation estimation apparatus 30 includes, for example, a storage unit 32, an input reception unit 33, an image acquisition unit 35, and a control unit 36. The image acquisition unit 35 acquires an image captured by the imaging unit 10 from different directions of the target object bx, and stores the acquired image in an image storage unit 321 described later. Among these, the image acquisition unit 35, the Jacobian matrix calculation unit 37, and the position / orientation estimation unit 41 are software function units that function when the CPU 31 executes a program stored in the storage unit 32, for example. Some or all of these may be hardware function units such as LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit).

The storage unit 32 includes, for example, an image storage unit 321 and a model storage unit 322. The image storage unit 321 stores the image acquired by the image acquisition unit 35. The model storage unit 322 stores in advance three-dimensional model data (hereinafter referred to as a reference model) obtained by modeling the target object bx or the shape, color, pattern, and the like of the target object bx. The reference model is a comparison target with the target object bx imaged by the imaging unit 10, for example,
It is expressed by three-dimensional computer graphics (Computer Graphics; CG). This three-dimensional CG is represented as a set of polygon data, for example.

  The control unit 36 includes, for example, a Jacobian matrix calculation unit 37, a position / orientation estimation unit 41, and a drive control unit 43. The Jacobian matrix calculation unit 37 calculates a three-dimensional Jacobian matrix of the target object bx based on the image stored in the image storage unit 321 (the image obtained by capturing the target object bx from different directions). The three-dimensional Jacobian matrix is a matrix including parameters necessary for a nonlinear minimization technique used for estimating the position and orientation of the target object bx, for example. Examples of the nonlinear minimization method include an ESM (Efficient Second-Order Minimization Method) method, a Levenberg-Marquardt method, a Newton method, a Gauss-Newton method, and the like. Among these nonlinear minimization methods, the ESM method has a smaller number of iterative calculations than other methods, and can be suitably used when it is desired to estimate the position and orientation of the target object bx at a higher speed. Therefore, the following description will be made assuming that the ESM method is adopted as the nonlinear minimization method and the calculated three-dimensional Jacobian matrix is a parameter used for the ESM method.

  Further, the Jacobian matrix calculation unit 37 acquires the position and orientation (the virtual position and orientation of the reference model) after the target object bx is moved by the robot body 50 from the input reception unit 33 as input from the user. . The Jacobian matrix calculation unit 37 includes, for example, a temporary storage area and stores input from the user. The Jacobian matrix calculation unit 37 then uses the reference model stored in the model storage unit 322 and the input from the user as a reference for estimating the position and orientation of the target object bx by the ESM method. A matrix (an example of a reference three-dimensional Jacobian matrix in the claims) is calculated. Further, the Jacobian matrix calculation unit 37 outputs the three-dimensional Jacobian matrix calculated from the image and the reference three-dimensional Jacobian matrix calculated from the reference model to the position / orientation estimation unit 41.

  The position / orientation estimation unit 41 uses the ESM method to determine the position and orientation of the target object bx in the camera coordinate system based on the three-dimensional Jacobian matrix calculated from the acquired image and the reference three-dimensional Jacobian matrix calculated from the reference model. Is estimated. Details of the position and orientation estimation by the ESM method will be described later. The camera coordinate system is a coordinate system provided in the virtual space by the position / orientation estimation unit 41, and is a coordinate system used to represent a position in an image captured by the imaging unit 10, for example. The position / orientation estimation unit 41 outputs information representing the estimated position and orientation of the target object bx in the camera coordinate system to the drive control unit 43.

  The drive control unit 43 calculates the position and orientation of the target object bx in the robot coordinate system based on the acquired information representing the position and orientation of the target object bx in the camera coordinate system. The robot coordinate system is a coordinate system that is used to represent the position of the grip portion when the robot body 50 operates. The drive control unit 43 performs a calibration process between a camera coordinate system applied to the position / orientation estimation unit 41 and a robot coordinate system applied to the robot body 50.

  Based on the calculated position and orientation of the target object bx in the robot coordinate system, the drive control unit 43 calculates the position and orientation of the grip portion hnd included in the robot body 50. And the drive control part 43 controls operation | movement of each movable part of the robot main body 50 based on this calculation result.

  Next, an example of a method for imaging the target object bx from different directions will be described with reference to FIGS. 4 and 5. FIG. 4 is a diagram illustrating an example of a dedicated optical system opt included in the imaging unit 10. The reflected light from the target object bx enters the image sensors ccd1 and ccd2 via the optical system opt. The image sensor ccd1 is an example of a first image sensor. The image sensor ccd2 is an example of a second image sensor. The reflected light from the target object bx first enters the half mirror hmr. The half mirror hmr transmits about 50% of the incident light and reflects the remaining 50% in a direction orthogonal to the transmitted light. In FIG. 4, the solid line arrow represents the optical path of the incident light ll1, and the alternate long and short dash line arrow represents the optical path of the incident light ll2. The direction in which the incident light ll1 enters the half mirror hmr is an example of a first direction. The direction in which the incident light ll2 enters the half mirror hmr is an example of a second direction. Incident light ll1 reflected by the half mirror hmr is reflected by the mirrors mr1 and mr2, and reaches the image sensor ccd1. On the other hand, the incident light ll2 transmitted through the half mirror hmr is reflected by the mirrors mr3 and mr4 and reaches the image sensor ccd2. At this time, due to the inclination of the reflecting surface of mr3, the image of the target object bx imaged by each of the incident light ll1 and the incident light ll2 is not simply inverted, but the target object by a minute change amount. The image looks like an image taken after moving bx. The mirrors mr1 to mr4 are examples of a plurality of mirrors.

  Here, the imaging system configured by the mirrors mr1 and mr2 and the imaging device ccd1 and the imaging system configured by the mirrors mr3 and mr4 and the imaging device ccd2 are configured so that the Jacobian matrix calculation unit 37 determines the minute change amount. Based on this, a three-dimensional Jacobian matrix can be calculated. Specifically, each element of the three-dimensional Jacobian matrix indicates the luminance of a pixel in the image representing the target object bx with respect to the minute change in the changed direction when the direction in which the target object bx is imaged is slightly changed. It is a value representing the rate of change when. Instead of moving the target object bx itself, the Jacobian matrix calculating unit 37 compares the images of the target object bx captured from different directions and calculates the difference, thereby calculating the change rate, and based on the change rate To determine each element of the three-dimensional Jacobian matrix.

  In FIG. 4, images pic <b> 1 and pic <b> 2 are images obtained by the incident lights ll <b> 1 and ll <b> 2 incident on the image sensors ccd <b> 1 and ccd <b> 2, respectively. Here, in FIG. 4, an arrow ar is drawn on the target object bx in order to clearly show that the image pic <b> 1 and the image pic <b> 2 are inverted when passing through the dedicated optical system opt. The imaging unit 10 includes two dedicated optical systems opt, one that inverts the target object bx with respect to the left-right direction and one that inverts the target object bx with respect to the up-down direction. Information necessary for the Jacobian matrix calculation unit 37 to calculate the three-dimensional Jacobian matrix can be provided as an image.

  FIG. 5 is a diagram for explaining an example of a situation in which the images pic1 to pic4 of the object captured from different directions are captured by the imaging unit 10 including the optical system opt illustrated in FIG. The reflected light from the target object bx enters the dedicated optical system opt as incident light 111-ll4. Incident light ll1 to ll4 incident on the dedicated optical system opt passes through the respective optical paths and is incident on the image sensor snsr via the lens lns. The image sensor snsr is obtained by arranging four image sensors in a grid pattern. Images pic1 to pic4 are images captured by the respective image sensors.

Next, the flow of operations of the object position / posture estimation apparatus 30 will be described with reference to FIG. FIG. 6 is a flowchart for explaining an example of the operation flow of the object position / posture estimation apparatus 30.
(Step S <b> 100) First, the input receiving unit 33 receives the input from the user, and outputs information indicating the virtual position and orientation of the reference model to the Jacobian matrix calculating unit 37 for setting.
(Step S120) Next, the model storage unit 322 stores the reference model via the input reception unit 33.

(Step S140) Next, the Jacobian matrix calculation unit 37 calculates a three-dimensional Jacobian matrix of the reference model. The Jacobian matrix calculation unit 37 determines each element of the three-dimensional Jacobian matrix of the reference model based on a simulation of moving the reference model virtually by a minute amount from the position and orientation of the reference model set in step S100. . Note that the processing of steps S100 to S140 need not be performed immediately before the processing of step S160 and subsequent steps, and a three-dimensional Jacobian matrix of the reference model may be stored in the storage unit 32 in advance.
(Step S160) Next, the image acquisition unit 35 acquires an image of the target object bx imaged from different directions from the imaging unit 10, and stores the acquired image in the image storage unit 321.

(Step S180) Next, the Jacobian matrix calculation unit 37 compares images from each direction by the above-described method based on the image of the target object bx captured from different directions stored in the image storage unit 321. Thus, each element of the three-dimensional Jacobian matrix of the target object bx is determined.
(Step S200) Next, the Jacobian matrix calculation unit 37 determines whether or not all elements of the three-dimensional Jacobian matrix have been determined in Step S180. When it is determined by the Jacobian matrix calculation unit 37 that all elements of the three-dimensional Jacobian matrix have not been determined (No at Step S200), the image acquisition unit 35 returns to Step S160, and again the target object from the imaging unit 10 The image of bx is acquired. When the Jacobian matrix calculation unit 37 determines that all the elements of the three-dimensional Jacobian matrix have been determined (step S200-Yes), the Jacobian matrix calculation unit 37 calculates the three-dimensional Jacobian matrix of the calculated reference model and the target object. The bx three-dimensional Jacobian matrix is output to the position / orientation estimation unit 41. Then, the position / orientation estimation unit 41 uses the ESM method based on the acquired 3D Jacobian matrix of the reference model and the 3D Jacobian matrix of the target object bx to detect the position and orientation of the target object bx in the camera coordinate system. Is estimated (step S220).

Here, a method for estimating the position and orientation of the target object bx in the camera coordinate system using the ESM method will be briefly described. The estimation method using the ESM method is, for example, a luminance map representing an image of the target object bx (hereinafter referred to as a real image) and an image using a reference model, and the target object bx is moved to a destination. A moving amount (moving amount of six degrees of freedom, that is, translation in three axes and a rotation angle around three axes) that minimizes the difference between the brightness map of the image (hereinafter referred to as a template image) in the case of It is a method of deriving. If the function representing the luminance map of the actual image (for example, a vector having a value representing the luminance of each pixel of the image as an element) is S and the function representing the luminance map of the template image is S ^* , the movement amount Δz (bold (Body) can be represented by the following formulas (1) to (3). Note that (bold) means that the character before the parenthesis is a vector or matrix expressed in bold.

  Here, J (bold body) is a three-dimensional Jacobian matrix of the target object bx, and J (bold body) * is a three-dimensional Jacobian matrix of the reference model. The position and orientation of the target object bx at the destination are known in advance because they are input in advance by the user. Therefore, the position / orientation estimation unit 41 obtains the movement amount Δz (bold body) from the above equation (1), and based on the obtained movement amount Δz and the position and orientation of the target object bx at the movement destination, The position and orientation of the target object bx can be estimated. The position / orientation estimation unit 41 outputs the estimated position and orientation of the target object bx to the drive control unit 43.

(Step S240) Next, the drive control unit 43 converts the coordinates of the acquired position and orientation of the target object bx from the coordinates of the camera coordinate system to the coordinates of the robot coordinate system, and based on the converted coordinates, the robot The target object bx is moved to the destination by the grip portion of the main body 50.

  As described above, the robot system 1 according to the first embodiment includes the imaging unit 10 including a plurality of imaging elements including the imaging element ccd1 and the imaging element ccd2, and enters the half mirror hmr by the imaging element ccd1. In order to operate the robot main body 50 by imaging the target object bx from the imaging direction and imaging the target object bx from the direction in which the incident light ll2 enters the half mirror hmr by the imaging element ccd2, the three-dimensional space of the target object bx It is possible to control the robot main body 50 by shortening the time required for estimating the position and orientation at.

  In the robot system 1, the control unit 36 calculates a three-dimensional Jacobian matrix of the target object bx based on a plurality of captured images captured by the imaging unit 10, and calculates the calculated three-dimensional Jacobian matrix, a reference, In order to estimate the position and orientation of the target object bx and to operate the robot body 50 based on the estimated position and orientation, for example, an estimation method using the ESM method The position and orientation of the target object bx are estimated by the above, and the time required for estimating the position and orientation of the target object bx is shortened compared to the estimation method using the nonlinear minimization method such as the conventional Newton method. 50 can be controlled.

  Also, in the robot system 1, the imaging unit 10 is incident from the direction in which the incident light ll1 is incident on the half mirror hmr and the direction in which the incident light ll2 is incident on the half mirror hmr by the half mirror hmr and the mirrors mr1 to mr4. Since the light from the target object bx to be collected is collected in the image sensor ccd1 and the image sensor ccd2, it is possible to acquire a plurality of images captured from different directions without changing the position of the image pickup unit 10, and as a result The time required for estimating the position and orientation of the object bx can be shortened. In addition, no blurring occurs in the captured image, and the estimation accuracy can be improved.

  Further, in the robot system 1, the imaging unit 10 receives light from the target object bx that is incident from the direction in which the incident light ll1 is incident on the half mirror hmr and the direction in which the incident light ll2 is incident on the half mirror hmr. To collect the light from the target object bx incident from the direction in which the incident light ll1 is incident on the half mirror hmr and the direction in which the incident light ll2 is incident on the half mirror hmr on the image sensor ccd1 and the image sensor ccd2. The target object bx can be imaged simultaneously from different directions by one imaging unit 10, and the position and orientation of the target object bx can be estimated in a shorter time.

<Second Embodiment>
Hereinafter, the second embodiment will be described. About a structure, FIG.2, 3 is used and it attaches | subjects and demonstrates the same code | symbol with respect to the same function part. FIG. 7 is a diagram illustrating an example of a usage status of the robot system 1 according to the second embodiment. In the robot system 1 according to the second embodiment, the imaging unit 10 is installed in the vicinity of the grip unit of the robot body 50 and moves together with the grip unit of the robot body 50. The imaging unit 10 of the second embodiment replaces with the dedicated optical system opt provided in the first embodiment, and the target object bx while the gripping part of the robot body 50 moves by a minute change amount. Are imaged from different directions.

  As described above, in the robot system 1 according to the second embodiment, the imaging unit 10 moves together with the grip unit of the robot body 50 and images the target object bx from different directions, so that imaging is possible from different directions. A two-dimensional image of the object imaged by the unit 10 is acquired, a three-dimensional Jacobian matrix of the target object bx is calculated based on the acquired image, the calculated three-dimensional Jacobian matrix, the three-dimensional Jacobian matrix of the reference model, In order to estimate the position and orientation of the target object bx by the estimation method using the ESM method, the time required to estimate the position and orientation of the target object bx is reduced by a nonlinear minimization method such as the conventional Newton method. The robot main body 50 can be controlled in a shorter manner than the estimation method using.

<Third Embodiment>
Hereinafter, a third embodiment will be described. About a structure, FIG.2, 3 is used and it attaches | subjects and demonstrates the same code | symbol with respect to the same function part. In the robot system 1 according to the third embodiment, the imaging unit 10 is a light field camera, and does not include the dedicated optical system opt provided in the first embodiment, as in the second embodiment. In the light field camera, microlenses having different focal points are arranged on a plane parallel to the surface of the image sensor in front of the image sensor, and an image including information in the depth direction obtained by this configuration is used. Thus, it is a camera that can perform stereo imaging with a single unit. Therefore, the Jacobian matrix calculation unit 37 in the third embodiment calculates the three-dimensional Jacobian matrix of the target object bx based on the three-dimensional image captured in stereo by the imaging unit 10 that is a light field camera.

  As described above, in the robot system 1 according to the third embodiment, the imaging unit 10 includes the light field camera as the imaging unit 10, and the Jacobian is based on the three-dimensional image obtained by the imaging unit 10 performing stereo imaging. Since the matrix calculation unit 37 calculates the three-dimensional Jacobian matrix of the target object bx, the same effect as that of the first embodiment can be obtained.

  In the robot system 1, the imaging unit 10 further includes a plurality of lenses having different focal points arranged on a plane parallel to the plane of the imaging element, and includes information in the depth direction obtained by the plurality of lenses. Since an image is picked up, an optical system composed of the half mirror hmr and the plurality of mirrors mr1 to mr4 is not required, and the position and orientation of the object can be estimated in a shorter time.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and changes, substitutions, deletions, and the like are possible without departing from the gist of the present invention. May be.

DESCRIPTION OF SYMBOLS 1 Robot system, 10 Imaging part, 30 Object position and orientation estimation apparatus, 31 CPU, 32 Storage part, 33 Input reception part, 34 Communication part, 35 Image acquisition part, 37 Jacobian matrix calculation part, 41 Position and orientation estimation part, 43 Drive Control unit, 50 robot body, 321 image storage unit, 322 model storage unit

Claims (5)

With robots,
An imaging unit for imaging an object;
A control unit for operating the robot;
Including
The imaging unit includes a plurality of imaging elements including a first imaging element and a second imaging element,
The control unit images the object from a first direction by the first image sensor, and images the object from a second direction different from the first direction by the second image sensor, so that the robot Make the work,
Robot system. The robot system according to claim 1,
The control unit calculates a three-dimensional Jacobian matrix of the object based on a plurality of captured images captured by the imaging unit, the calculated three-dimensional Jacobian matrix, a reference three-dimensional Jacobian matrix serving as a reference, And estimating the position and orientation of the object, and operating the robot based on the estimated position and orientation.
Robot system. The robot system according to claim 1 or 2,
The imaging unit further includes a half mirror and a plurality of mirrors, and the first mirror captures light from the object incident from the first direction and the second direction by the half mirror and the plurality of mirrors. Collecting the device and the second imaging device;
Robot system. The robot system according to claim 3, wherein
The imaging unit causes the light from the object incident from the first direction and the second direction to be branched by the half mirror, so that the object from the object incident from the first direction and the second direction is separated. Collecting light on the first image sensor and the second image sensor;
Robot system. The robot system according to claim 3, wherein
The imaging unit further includes a plurality of lenses having different focal points arranged on a plane parallel to the plane of the imaging element, and captures an image including information in the depth direction obtained by the plurality of lenses.
Robot system.

Images