JP2020015102A - Control system, control method, and program - Google Patents

Abstract

To provide a control system which can change states of a plurality of objects in cooperation with each other.SOLUTION: A control device performs first processing which acquires an actual image imaged by an imaging device, second processing which selects a target frame from a CG moving image, and third processing which controls each of first to N-th robots on the basis of the actual image and the target frame. In the third processing, the control device calculates a control amount of a j-th robot so as to make the state of a j-th object on the actual image close to the state of the j-th object on the target frame on the basis of change information corresponding to the j-th object, and controls the j-th robot according to the calculated control amount.SELECTED DRAWING: Figure 1

Description

  The present technology relates to a control system, a control method, and a program for controlling a robot.

  "Koichi Hashimoto," Atsushi Murata, and three others, "Development of high-precision positioning method using visual servoing", SEI Technical Review, July 2009, No. 175, p. 98-102 "(Non-Patent Document 1) Discloses a technique for using a visual servo to guide a robot arm to a target position. In the system disclosed in Non-Patent Document 1, an image of a connector is taken by a camera attached to a robot arm, and a terminal supported by a robot hand is inserted into the connector based on the image of the camera.

Atsushi Murata and 3 others, "Development of high-precision positioning method using visual servo", SEI Technical Review, July 2009, No. 175, p. 98-102 Adachi, "Basics of Model Predictive Control", Journal of the Robotics Society of Japan, July 2014, Volume 32, Issue 6, p. 9-12

  The technology described in Non-Patent Document 1 is based on the premise that the relative position between the terminal supported by the robot hand and the camera is constant. Therefore, if the relative positional relationship between the terminal supported by the robot hand and the camera is deviated, the connection between the state of the terminal and the state of the connector may be lost, and the terminal may not be inserted into the connector.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a control system, a control method, and a program in which the states of a plurality of objects can be changed in a coordinated manner.

  According to an example of the present disclosure, the control system includes a first to an N-th robot, an imaging device for imaging the first to the N-th object, and a control device for controlling the first to the N-th robot. A control device. N is an integer of 2 or more. The i-th robot changes the state of the i-th object. i is an integer of 1 to N-1. The Nth robot changes one state of the Nth object and the imaging device. The other of the N-th object and the imaging device is installed at a fixed position. The control device acquires change information for each of the first to Nth objects. The change information corresponding to the j-th object indicates the relationship between the control amount of the j-th robot and the change amount of the state of the j-th object on the image of the imaging device. j is an integer of 1 to N. The control device is configured to perform a first process of acquiring a real image captured by the imaging device, a second process of selecting a target frame from CG moving images indicating the first to Nth object models, a real image and a target frame. And a third process for controlling each of the first to Nth robots based on the above. In the third process, the control device brings the state of the j-th object on the real image closer to the state of the model of the j-th object on the target frame based on the change information corresponding to the j-th object. The control amount of the j-th robot is calculated and the j-th robot is controlled in accordance with the calculated control amount.

  According to this disclosure, the state of the target on the real image can be changed to the state of the model of the target on the target frame based on the real image, the target frame, and the change information. As a result, the states of the first to Nth objects change in a coordinated manner according to the CG moving image.

  In the above disclosure, the CG moving image is created based on the design data of each of the first to Nth objects. In the CG moving image, the first object and the at least one object are joined from a frame indicating a state in which the first object and at least one of the second to Nth objects are separated from each other. This includes up to the frame indicating the status. In the third process, the control device adjusts the control amount of the first robot by an adjustment amount according to a manufacturing error of the first object and at least one object with respect to the design data, and adjusts the control amount of the first robot in accordance with the adjusted control amount. Control one robot.

  According to this disclosure, even if a manufacturing error occurs in the first object and at least one object, the first object and at least one object can be joined.

  In the above disclosure, the control system includes a force sensor that detects a force that one of the first object and the at least one object receives from the other. The control device searches for a state of the first object where the force detected by the force sensor is minimal, and determines an adjustment amount based on the searched state of the first object.

  When a manufacturing error occurs in the first object and at least one object, when the first object and the second object are joined according to the manufacturing error, the first object and the second object are connected to each other. Unintended resistance force (including frictional force) is generated between at least one object and the detection value of the force sensor increases. According to this disclosure, the adjustment amount is determined based on the state of the first target at which the detection value of the force sensor is minimal. Therefore, the state of the first object can be changed in a state in which the relative displacement between the first object and at least one object caused by the manufacturing error is reduced.

  In the above disclosure, the control device is connected to the input device and acquires the adjustment amount from the input device. Thereby, the control device can easily acquire the adjustment amount.

  In the above disclosure, the CG moving image includes a first object and at least one object from a frame indicating a state where the first object and at least one of the second to Nth objects are separated from each other. Includes the frame up to the state where the object is joined. The control system includes a force sensor that detects a force that one of the first object and the at least one object receives from the other. The control device performs a first mode in which the first to third processes are repeated. In the third processing in the first mode, the control device adjusts the control amount of the first robot and searches for an adjustment amount at which the force detected by the force sensor becomes a minimum. The control device acquires, as the reference moving image, a moving image captured by the imaging device when the control amount of the first robot is adjusted by the searched adjustment amount. The control device performs a second mode in which the first process, the fourth process of selecting a target frame from the reference moving image instead of the CG moving image, and the third process are repeated.

  According to this disclosure, the reference moving image is an image captured when the force detected by the force sensor becomes a minimum. Therefore, even if a manufacturing error occurs in the first object and at least one object by performing the processing according to the second mode by the control device, the first object and the at least one object can be compared with each other. Can be joined.

  According to an example of the present disclosure, the control method controls the first to Nth robots using an imaging device for imaging the first to Nth objects. N is an integer of 2 or more. The i-th robot changes the state of the i-th object. i is an integer of 1 to N-1. The Nth robot changes one state of the Nth object and the imaging device. The other of the N-th object and the imaging device is installed at a fixed position. The control method includes a first step of acquiring change information for each of the first to Nth robots. The change information corresponding to the j-th robot indicates a relationship between the control amount of the j-th robot and the change amount of the state of the j-th object on the image of the imaging device. j is an integer of 1 to N. The control method includes a second step of acquiring a real image captured by the imaging device, a third step of selecting a target frame from CG moving images indicating the first to Nth object models, a real image and a target frame. A fourth step of controlling each of the first to Nth robots based on the above. The fourth step calculates a control amount for bringing the state of the j-th object on the real image closer to the state of the j-th object on the target frame based on the change information corresponding to the j-th robot. , And controlling the j-th robot according to the calculated control amount.

  According to an example of the present disclosure, a program is a program for causing a computer to execute the above control method. According to these disclosures, the states of a plurality of objects also change in a coordinated manner.

  According to the present invention, the states of a plurality of objects change in a coordinated manner.

It is a schematic diagram which shows the outline | summary of the control system which concerns on embodiment. FIG. 4 is a diagram illustrating an example of a real image captured by an imaging device and a first CG moving image. FIG. 3 is a diagram illustrating an example of a real image and a second CG moving image captured by an imaging device 22. FIG. 2 is a schematic diagram illustrating a hardware configuration of a control device included in the control system according to the embodiment. FIG. 2 is a block diagram illustrating a functional configuration of a control device according to the embodiment. FIG. 6 is a diagram illustrating an example of a method for creating a template. FIG. 3 is a block diagram illustrating a functional configuration of a first control unit and a second control unit according to the embodiment. FIG. 5 is a diagram illustrating a method of generating a first change information set in a first control unit. FIG. 4 is a diagram illustrating a method of calculating a control amount by a calculation unit of a first control unit. It is a flowchart which shows an example of the flow of a process which controls a target robot so that the state of a target object may be changed along a 1st CG moving image and a 2nd CG moving image. 11 is a flowchart showing the flow of processing of a subroutine of step S4 shown in FIG. FIG. 3 is a diagram illustrating a relationship between a closest frame and a target frame. 11 is a flowchart showing the flow of the processing of a subroutine of step S6 shown in FIG. It is a figure showing the design data of a female connector, and the dimensions of the female connector manufactured according to the design data. FIG. 9 is a schematic diagram illustrating an outline of a control system according to a first modification. FIG. 9 is a block diagram illustrating a functional configuration of a control device according to a first modification. FIG. 9 is a block diagram illustrating a functional configuration of a first control unit and a second control unit according to a first modification. 15 is a flowchart illustrating an example of a flow of an offset amount determination process in Modification Example 1. FIG. 13 is a block diagram illustrating a functional configuration of a control device according to a third modification. FIG. 14 is a block diagram illustrating a functional configuration of a first control unit and a second control unit according to a third modification. FIG. 14 is a schematic diagram illustrating an object of a control system according to Modification 4. 15 is a schematic diagram illustrating an object of a control system according to Modification Example 5. FIG. 15 is a schematic diagram illustrating an outline of a control system according to Modification Example 6. FIG. FIG. 13 is a block diagram illustrating a functional configuration of a control device according to a modification 6. 15 is a flowchart illustrating a flow of processing of a first control unit and a second control unit of a modification 6. FIG. 15 is a schematic diagram illustrating another configuration example of the control system according to Modification 6.

  Hereinafter, embodiments according to the present invention will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are the same. Therefore, detailed description thereof will not be repeated.

<A. Application example>
First, an example of a scene to which the present invention is applied will be described with reference to FIG. FIG. 1 is a schematic diagram illustrating an outline of a control system according to an embodiment.

  The control system 1 according to the present embodiment connects the male connector 2a to the female connector 2b by inserting the male connector 2a into the female connector 2b in, for example, a production line of an industrial product.

  As shown in FIG. 1, the control system 1 includes imaging devices 21 and 22, robots 30a and 30b, robot controllers 40a and 40b, and a control device 50.

  The imaging devices 21 and 22 generate an image data (hereinafter, simply referred to as an “image”) by capturing an image of a subject existing in an imaging visual field. The imaging devices 21 and 22 are installed at fixed positions different from the robots 30a and 30b. The imaging devices 21 and 22 capture images of the male connector 2a and the female connector 2b as subjects from different fixed positions. The imaging devices 21 and 22 perform imaging according to a predetermined imaging cycle, and output a real image obtained by the imaging to the control device 50.

  The robot 30a is a mechanism for changing the state (here, position and posture) of the male connector 2a, and is, for example, a vertical articulated robot. The robot 30a has a hand 31a that supports (holds here) the male connector 2a at its tip, and changes the position and posture of the hand 31a with six degrees of freedom. In other words, the robot 30a changes the position and the posture of the male connector 2a held by the hand 31a with six degrees of freedom. The six degrees of freedom include translational degrees of freedom in the X, Y, and Z directions, and rotational degrees of freedom in the pitch, yaw, and roll directions. However, the number of degrees of freedom of the hand 31a is not limited to six, and may be three to five or seven or more.

  The robot 30a has a plurality of servomotors, and drives and changes the position and posture of the male connector 2a. An encoder is provided for each of the plurality of servomotors, and the position of the servomotor is measured.

  The robot 30b is a mechanism for changing the state (here, position and orientation) of the female connector 2b, and is, for example, an XYθ stage. The robot 30b has a stage 31b supporting (put on) the female connector 2b, and changes the position and the posture of the stage 31b with three degrees of freedom. That is, the robot 30b changes the position and posture of the female connector 2b mounted on the stage 31b with three degrees of freedom. The three degrees of freedom include a translational degree of freedom in the X direction and the Y direction and a rotational degree of freedom in a rotational direction (θ direction) about an axis orthogonal to the XY plane. However, the number of degrees of freedom of the robot 30b is not limited to three, and may be four or more.

  The robot 30b has a plurality of servomotors, and changes the position and posture of the female connector 2b by driving the servomotors. An encoder is provided for each of the plurality of servomotors, and the position of the servomotor is measured.

  The robot controller 40a controls the operation of the robot 30a according to a control command received from the control device 50. The robot controller 40a receives from the control device 50 control commands for translational degrees of freedom in the X, Y, and Z directions and rotational degrees of freedom in the pitch, yaw, and roll directions. These X direction, Y direction, Z direction, pitch direction, yaw direction, and roll direction are indicated by the coordinate system of the robot 30a. The robot controller 40a performs feedback control on the robot 30a so that the translation amounts of the hand 31a in the X, Y, and Z directions approach control commands for the degrees of freedom of translation in the X, Y, and Z directions, respectively. Do. The robot controller 40a performs feedback control on the robot 30a so that the rotational movement amounts of the hand 31a in the pitch direction, the yaw direction, and the roll direction approach the control commands for the rotational degrees of freedom in the pitch, yaw, and roll directions, respectively. Do.

  The robot controller 40b controls the operation of the robot 30b according to a control command received from the control device 50. The robot controller 40b receives from the control device 50 control commands for the degrees of freedom of translation and rotation in the X and Y directions. These X direction, Y direction, and rotation direction are indicated by the coordinate system of the robot 30b. The robot controller 40b performs feedback control on the robot 30b such that the translation amounts of the stage 31b in the X and Y directions approach control commands for the degrees of freedom of translation in the X and Y directions, respectively. The robot controller 40b performs feedback control on the robot 30b such that the rotational movement amount of the stage 31b approaches the control command of the rotational degree of freedom.

  The control device 50 controls the robots 30a and 30b via the robot controllers 40a and 40b, respectively.

  The control device 50 stores a first CG moving image and a second CG moving image that indicate a sample of the male connector 2a and the female connector 2b. The first CG moving image and the second CG moving image are created by simulating a model of the male connector 2a, the female connector 2b, the robots 30a and 30b, and the imaging devices 21 and 22 in a virtual space. The first CG moving image and the second CG moving image show how the model of the male connector 2a and the model of the female connector 2b are connected to each other in the virtual space while changing the position and orientation. The models of the male connector 2a and the female connector 2b are created based on the design data of the male connector 2a and the female connector 2b.

  The first CG moving image is created by setting the position of the model of the imaging device 21 in the virtual space as a viewpoint. The second CG moving image is created by setting the position of the model of the imaging device 22 in the virtual space as a viewpoint.

  Each of the first CG moving image and the second CG moving image includes a plurality of frames (hereinafter, referred to as M (M is an integer of 2 or more)) arranged in time series. The k-th (k is an integer from 1 to M) frame of the first CG moving image and the k-th frame of the second CG moving image are obtained by simultaneously using the model of the male connector 2a and the female connector 2b in a certain state in the virtual space These are images when viewed from different directions.

  The control device 50 acquires change information indicating a relationship between a control amount of the robot 30a and a change amount of a state of the male connector 2a on an image obtained by imaging by the imaging devices 21 and 22. The change information is generated by simulating a model of the male connector 2a, the robot 30a, and the imaging devices 21 and 22 in a virtual space.

  Similarly, the control device 50 acquires change information indicating the relationship between the control amount of the robot 30b and the change amount of the state of the female connector 2b on the images obtained by the imaging devices 21 and 22. The change information is generated by simulating a model of the female connector 2b, the robot 30b, and the imaging devices 21 and 22 in a virtual space.

  Control device 50 performs the following first to third processing. The control device 50 repeatedly executes a series of processes including the first to third processes. The first process is a process of acquiring actual images captured by the imaging devices 21 and 22.

  The second process is a process of selecting a target frame from each of the first CG moving image and the second CG moving image. When selecting the k-th frame from the first CG moving image as the target frame, the control device 50 also selects the k-th frame from the second reference moving image as the target frame.

  The third process is a process of controlling each of the robots 30a and 30b based on the real image and the target frame. The control device 50 calculates and calculates the control amount of the robot 30a for bringing the state of the male connector 2a on the actual image closer to the state of the male connector 2a on the target frame based on the change information corresponding to the robot 30a. The robot 30a is controlled according to the control amount. The control device 50 generates a control command indicating a control amount of the robot 30a, and outputs the generated control command to the robot controller 40a. Further, the control device 50 calculates a control amount of the robot 30b for bringing the state of the female connector 2b on the actual image closer to the state of the female connector 2b on the target frame based on the change information corresponding to the robot 30b, The robot 30b is controlled according to the calculated control amount. The control device 50 generates a control command indicating a control amount of the robot 30b, and outputs the generated control command to the robot controller 40b.

  FIG. 2 is a diagram illustrating an example of a real image captured by the imaging device 21 and a first CG moving image. FIG. 3 is a diagram illustrating an example of a real image captured by the image capturing device 22 and a second CG moving image. FIG. 2 shows real images 90 a to 93 a captured by the imaging device 21 and frames 70 a to 73 a of the first CG moving image. FIG. 3 shows real images 90b to 93b captured by the imaging device 22 and frames 70b to 73b of the second CG moving image.

  The real images 90a and 90b are images captured at the same time. The real images 91a and 91b are images captured at the same time after the real images 90a and 90b. The real images 92a and 92b are images captured at the same time after the real images 91a and 91b. The real images 93a and 93b are images captured at the same time after the real images 92a and 92b.

  Each of the frames 70a and 70b is the first frame in the corresponding CG moving image. Each of the frames 71a and 71b is an s-th (s is an integer of 2 or more) frame in the corresponding CG moving image. Each of the frames 72a and 72b is a t-th (t is an integer greater than s) frame in the corresponding CG moving image. Each of the frames 73a and 73b is a u-th (u is an integer greater than t) frame in the corresponding CG moving image.

  The first CG moving image and the second CG moving image include a model of the hand 31a (hereinafter, referred to as “hand model”) 31a ′ and a model of the stage 31b (hereinafter, referred to as “stage model”) 31b ′. Further, the first CG moving image and the second CG moving image include a model of male connector 2a (hereinafter, referred to as “male connector model”) 2a ′ and a model of female connector 2b (hereinafter, referred to as “female connector model”) 2b ′. .

  The first CG moving image and the second CG moving image show how the female connector model 2b 'on the stage model 31b' moves to a desired state. Further, the first CG moving image and the second CG moving image show that the male connector model 2a 'gripped by the hand model 31a' moves downward from above the female connector model 2b 'and connects to the female connector model 2b'. Is shown.

  The control device 50 acquires real images 90a and 90b including the female connector 2b placed on the stage 31b and the male connector 2a held by the hand 31a from the imaging devices 21 and 22, respectively.

  The control device 50 selects, from the first CG moving image and the second CG moving image, the frames 71a and 71b when the female connector model 2b 'has moved to a desired position and posture, respectively, as target frames.

  Based on the change information corresponding to the female connector 2b, the control device 50 brings the state of the female connector 2b on the actual images 90a, 90b closer to the state of the female connector model 2b 'on the frames 71a, 71b, respectively. Is calculated. Then, the control device 50 outputs a control command indicating the calculated control amount to the robot controller 40b. The robot controller 40b controls the robot 30b according to a control command. Thereby, as shown in the real images 91a and 91b, the position and the posture of the female connector 2b change to a desired position and posture (the position and the posture of the female connector model 2b 'shown by the frames 71a and 71b).

  Further, in parallel with the control of the robot 30b, the control device 50 controls the hand 31a to bring the state of the male connector 2a on the real images 90a and 90b closer to the state of the male connector model 2a 'on the frames 71a and 71b. Calculate the amount. Then, the control device 50 outputs a control command indicating the calculated control amount to the robot controller 40a. The robot controller 40a controls the robot 30a according to a control command. Thereby, as shown in the actual images 91a and 91b, the state of the male connector 2a changes to the position and posture above the female connector 2b (the positions and postures indicated by the frames 71a and 71b).

  Next, the control device 50 selects the frames 72a and 72b when the male connector model 2a 'has moved to a position immediately above the female connector model 2b' as a target frame.

  The control device 50 controls the hands 31a for bringing the state of the male connector 2a on the real images 91a and 91b closer to the state of the male connector model 2a 'on the frames 72a and 72b based on the change information corresponding to the male connector 2a. Is calculated. Then, the control device 50 outputs a control command indicating the calculated control amount to the robot controller 40a. The robot controller 40a controls the robot 30a according to a control command. Thereby, as shown in the real images 92a and 92b, the position and the posture of the male connector 2a are the position and the posture immediately above the female connector 2b (the position and the posture of the male connector model 2a 'indicated by the frames 72a and 72b). Changes to

  Next, the control device 50 selects the frames 73a and 73b when the connection between the male connector model 2a 'and the female connector model 2b' is completed, as target frames.

  The control device 50 controls the hand 31a for bringing the state of the male connector 2a on the real images 92a and 92b closer to the state of the male connector model 2a 'on the frames 73a and 73b based on the change information corresponding to the male connector 2a. Calculate the control amount. Then, the control device 50 outputs a control command indicating the calculated control amount to the robot controller 40a. The robot controller 40a controls the robot 30a according to a control command. As a result, as shown in the real images 93a and 93b, the male connector 2a moves to the position and posture at which the connection to the female connector 2b is completed (the position and posture of the male connector model 2a 'indicated by the frames 73a and 73b). .

  As described above, the control device 50 can change the states of the male connector 2a and the female connector 2b on the actual image to the states of the male connector model 2a 'and the female connector model 2b' on the target frame, respectively. Thereby, the states of the male connector 2a and the female connector 2b change in a coordinated manner according to the first CG moving image and the second CG moving image.

    The control device 50 can control the robots 30a and 30b without using calibration data for associating the coordinate systems of the imaging devices 21 and 22 with the robots 30a and 30b. Therefore, the operator does not need to perform the calibration in advance. Therefore, it is possible to reduce the labor required for changing the states of the male connector 2a and the female connector 2b to desired states by the robots 30a and 30b.

  When there is calibration data, the positions and postures of the male connector 2a and the female connector 2b in the real space are specified from the images taken by the imaging devices 21 and 22, and the robots 30a and 30b are controlled based on the positions and postures. There is a way to do it. However, there is a possibility that the accuracy of the calibration data is reduced in accordance with the aging of the robots 30a and 30b, and the male connector 2a and the female connector 2b cannot be connected well. Furthermore, there is a possibility that the male connector 2a and the female connector 2b cannot be properly connected due to a positional shift or individual difference between the male connector 2a and the female connector 2b. Even in such a case, by using this application example, the male connector 2a and the female connector 2b can be connected as in the CG moving image.

<B. Specific example>
Next, a specific example of the control system according to the present embodiment will be described.

<B-1. Hardware configuration of control device>
FIG. 4 is a schematic diagram illustrating a hardware configuration of a control device included in the control system according to the embodiment. As shown in FIG. 4, the control device 50 has a structure according to a computer architecture, and executes various programs as described below by executing a program installed in advance by a processor.

  More specifically, the control device 50 includes a processor 510 such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit), a RAM (Random Access Memory) 512, a display controller 514, a system controller 516, It includes an I / O (Input Output) controller 518, a hard disk 520, a camera interface 522, an input interface 524, a robot controller interface 526, a communication interface 528, and a memory card interface 530. These units are connected to each other so as to enable data communication with the system controller 516 as the center.

  The processor 510 exchanges programs (codes) and the like with the system controller 516, and executes the programs (codes) and the like in a predetermined order, thereby realizing the intended arithmetic processing.

  The system controller 516 is connected to the processor 510, the RAM 512, the display controller 514, and the I / O controller 518 via buses, respectively, exchanges data with each unit, and performs processing of the entire control device 50. Govern.

  The RAM 512 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory), and includes programs read from the hard disk 520, images (image data) acquired by the imaging devices 21 and 22, The processing result for the image, work data, and the like are stored.

  The display controller 514 is connected to the display unit 532, and outputs a signal for displaying various information to the display unit 532 according to an internal command from the system controller 516.

  The I / O controller 518 controls data exchange between a recording medium connected to the control device 50 and an external device. More specifically, the I / O controller 518 is connected to the hard disk 520, the camera interface 522, the input interface 524, the robot controller interface 526, the communication interface 528, and the memory card interface 530.

  The hard disk 520 is typically a nonvolatile magnetic storage device, and stores various information in addition to the control program 550 executed by the processor 510. The control program 550 installed on the hard disk 520 is distributed while being stored in a memory card 536 or the like. Instead of the hard disk 520, a semiconductor storage device such as a flash memory or an optical storage device such as a DVD-RAM (Digital Versatile Disk Random Access Memory) may be adopted.

  The camera interface 522 corresponds to an input unit that receives image data from the imaging devices 21 and 22, and mediates data transmission between the processor 510 and the imaging devices 21 and 22. The camera interface 522 includes image buffers 522a and 522b for temporarily storing image data from the imaging devices 21 and 22, respectively. Although a single image buffer that can be shared may be provided for a plurality of imaging devices, it is preferable to independently arrange a plurality of image buffers in association with each imaging device in order to speed up processing.

  The input interface 524 mediates data transmission between the processor 510 and an input device 534 such as a keyboard, a mouse, a touch panel, and a dedicated console.

  The robot controller interface 526 mediates data transmission between the processor 510 and the robot controllers 40a, 40b.

  The communication interface 528 mediates data transmission between the processor 510 and another personal computer or server device (not shown). The communication interface 528 is typically made of Ethernet (registered trademark), USB (Universal Serial Bus), or the like.

  The memory card interface 530 mediates data transmission between the processor 510 and the memory card 536 as a recording medium. The memory card 536 circulates in a state where a control program 550 executed by the control device 50 and the like are stored, and the memory card interface 530 reads the control program 550 from the memory card 536. The memory card 536 is a general-purpose semiconductor storage device such as SD (Secure Digital), a magnetic recording medium such as a flexible disk (Flexible Disk), an optical recording medium such as a CD-ROM (Compact Disk-Read Only Memory), and the like. Consists of Alternatively, a program downloaded from a distribution server or the like may be installed in the control device 50 via the communication interface 528.

  When using a computer having a structure according to the general-purpose computer architecture as described above, an OS for providing basic functions of the computer in addition to an application for providing functions according to the present embodiment (Operating System) may be installed. In this case, the control program according to the present embodiment executes processing by calling necessary modules in a predetermined order and / or timing among program modules provided as a part of the OS. Is also good.

  Further, the control program according to the present embodiment may be provided by being incorporated in a part of another program. Even in such a case, the program itself does not include a module included in another program to be combined as described above, and the process is executed in cooperation with the other program. That is, the control program according to the present embodiment may be a form incorporated in such another program.

  Alternatively, part or all of the functions provided by executing the control program may be implemented as a dedicated hardware circuit.

<B-2. Functional configuration of control device>
FIG. 5 is a block diagram illustrating a functional configuration of the control device according to the embodiment. As shown in FIG. 5, the control device 50 includes a moving image storage unit 51, a design data storage unit 52, a CG moving image generation unit 53, a teaching range selection unit 54, an image processing unit 55, and a target frame selection unit. 56, a first control unit 57a, and a second control unit 57b. The moving image storage unit 51 and the design data storage unit 52 include a hard disk 520 and a RAM 512 shown in FIG. The CG moving image generation unit 53, the teaching range selection unit 54, and the image processing unit 55 are realized by the processor 510 illustrated in FIG.

<B-2-1. Movie storage section>
The moving image storage unit 51 stores a first CG moving image and a second CG moving image. The first CG moving image and the second CG moving image are generated by the CG moving image generating unit 53.

<B-2-2. Design Data Storage>
The design data storage unit 52 stores design data of the male connector 2a, the female connector 2b, the robots 30a and 30b, and the imaging devices 21 and 22. The design data is, for example, 3D-CAD (Computer-Aided Design) data.

  Further, the design data storage unit 52 also stores arrangement data indicating installation positions of the robots 30a and 30b and the imaging devices 21 and 22 in the real space. The design data storage unit 52 also stores the directions and the angles of view of the imaging devices 21 and 22.

<B-2-3. CG video generator>
The CG moving image generating unit 53 generates a first CG moving image and a second CG moving image. The CG moving image generation unit 53 may generate the first CG moving image and the second CG moving image using a known CG technique.

  Specifically, the CG moving image generation unit 53 constructs a virtual space in which the models of the robots 30a and 30b and the imaging devices 21 and 22 are arranged according to the arrangement data stored in the design data storage unit 52. A hand model 31a 'and a stage model 31b' are arranged in the virtual space (see FIGS. 2 and 3). The virtual space is a space simulating the real space, and is represented by the same coordinate system as the real space. The CG moving image generating unit 53 causes the display unit 532 to display the constructed virtual space.

  The CG moving image generation unit 53 arranges the female connector model 2b 'in the initial posture at the initial position on the stage model 31b' in the virtual space. The female connector model 2b 'is generated based on 3D-CAD data of the female connector 2b. The initial position and the initial posture are specified by the operator using the input device 534. The operator sets the average position and posture of the female connector 2b as the initial position and initial posture when the female connector 2b conveyed from the upstream device is placed on the stage 31b in the real space. You can specify each one.

  Further, the CG moving image generating unit 53 arranges the male connector model 2a 'held by the hand model 31a' in the initial position and the initial posture in the virtual space. The male connector model 2a 'is generated based on 3D-CAD data of the male connector 2a. The initial position and the initial posture are specified by the operator using the input device 534. The operator specifies the average position and posture of the male connector 2a as the initial position and the initial posture when the hand 31a grips the male connector 2a conveyed from the upstream device in the real space. I just need.

  The CG moving image generation unit 53 generates, as a first CG moving image, a CG moving image having the viewpoint of the model of the imaging device 21 when each model is moved in the virtual space according to the moving program. Similarly, the CG moving image generating unit 53 generates, as a second CG moving image, a CG moving image having the viewpoint of the model of the imaging device 22 when each model is moved in the virtual space according to the moving program. The CG moving image is generated based on the orientation and the angle of view of the imaging devices 21 and 22 stored in the design data storage unit 52.

  The moving program moves the female connector model 2b ′ from the initial position and initial posture to a desired position and posture in the virtual space, and then moves the male connector model 2a ′ from above toward the female connector model 2b ′. This is a program for connecting to the female connector model 2b. The moving program is created according to the moving paths of the female connector model 2b 'and the male connector model 2a' specified by the operator.

<B-2-4. Teaching range selector>
The teaching range selection unit 54 selects, for each object (here, the male connector 2a and the female connector 2b), a teaching range serving as a sample of the object from the first CG moving image and the second CG moving image.

  The teaching range selection unit 54 displays a screen prompting the user to select a teaching range on the display unit 532. The operator confirms each frame of the first CG moving image and the second CG moving image, and operates the input device 534 to determine the first frame and the last frame of a series of frames in which the model of the target object performs a desired operation. specify. The teaching range selection unit 54 selects the designated first frame to the last frame as the teaching range.

  For example, with respect to the first CG moving image shown in FIG. 2, the teaching range selection unit 54 sets the female connector 2b from the first frame 70a to a frame after the s-th frame (a frame where a part of the female connector 2b starts to be cut off). Is selected as the teaching range. The teaching range selection unit 54 selects a range from the first frame 70a (a frame where the entire male connector 2a starts to appear) to the u-th frame 73a as the teaching range of the male connector 2a.

  Similarly, for the second CG moving image shown in FIG. 3, the teaching range selection unit 54 sets the female connector from the first frame 70b to the frame after the s-th frame (the frame where a part of the female connector 2b starts to be cut off). 2b is selected as the teaching range. The teaching range selection unit 54 selects from the first frame 70b (the frame where the entire male connector 2a starts to appear) to the u-th frame 73b as the teaching range of the male connector 2a.

<B-2-5. Image processing section>
The image processing unit 55 performs image processing on the target image, and detects the target from the target image using template matching. In the basic processing of template matching, a template which is data representing the image feature of the target object is prepared in advance, and the degree of matching of the image feature between the target image and the template is evaluated. This is a process of detecting the position and the posture of the camera.

  The target images on which the image processing unit 55 performs the image processing are the frames of the first CG moving image, the frames of the second CG moving image, and the real images captured by the imaging devices 21 and 22.

  The image processing unit 55 creates a template for each object (the male connector 2a and the female connector 2b) as advance preparation.

  FIG. 6 is a diagram illustrating an example of a template creation method. As shown in FIG. 6, the image processing unit 55 generates a 2D model image of the female connector 2b when viewed from various directions, based on the design data (3D-CAD data) of the female connector 2b. The image processing unit 55 extracts a plurality of feature points of the female connector 2b and their feature amounts from each 2D model image of the female connector 2b. The image processing unit 55 creates the coordinates of each of the plurality of feature points on the image and the feature amounts as a template of the female connector 2b.

  Similarly, the image processing unit 55 generates a 2D model image of the male connector 2a when viewed from various directions, based on the design data (3D-CAD data) of the male connector 2a. The image processing unit 55 extracts a plurality of feature points of the male connector 2a and their feature amounts from each 2D model image of the male connector 2a. The image processing unit 55 creates the coordinates of each of the plurality of feature points on the image and the feature amounts as a template of the male connector 2a.

  A feature point is a point characterized by a corner, a contour, or the like included in an image, and is, for example, an edge point. The feature amount is, for example, luminance, luminance gradient direction, quantization gradient direction, HoG (Histogram of Oriented Gradients), HAAR-like, SIFT (Scale-Invariant Feature Transform), and the like. The luminance gradient direction represents a direction (angle) of a luminance gradient in a local region centered on a feature point as a continuous value, and the quantization gradient direction is a local region centered on a feature point. The direction of the luminance gradient is represented by discrete values (for example, eight directions are held by 1-byte information of 0 to 7).

  The image processing unit 55 extracts a plurality of feature points and their feature amounts from the target image (the frame of the first CG moving image, the frame of the second CG moving image, and the real images captured by the imaging devices 21 and 22). The image processing unit 55 detects the target in the target image by comparing the extracted feature points and feature amounts with the template of the target.

  The image processing unit 55 outputs, for each target object (male connector 2a and female connector 2b), coordinates on the image of each feature point of the target object extracted from the target image.

<B-2-6. Target frame selection section>
The target frame selection unit 56 selects a target frame from the first CG moving image and the second CG moving image. However, when the k-th frame of the first CG moving image is selected as the target frame, the target frame selecting unit 56 selects the k-th frame of the second CG moving image as the target frame. A specific example of the target frame selection method will be described later.

<B-2-7. First Control Unit and Second Control Unit>
The first control unit 57a controls the robot 30a via the robot controller 40a and changes the state of the male connector 2a.

  The second control unit 57b controls the robot 30b via the robot controller 40b and changes the state of the female connector 2b.

  FIG. 7 is a block diagram illustrating a functional configuration of the first control unit and the second control unit according to the embodiment. As shown in FIG. 7, each of the first control unit 57a and the second control unit 57b includes a change information generation unit 58, a change information storage unit 59, a calculation unit 60, a command unit 61, and an end determination unit. 62. The change information storage unit 59 includes the hard disk 520 and the RAM 512 shown in FIG. The change information generation unit 58, the calculation unit 60, the command unit 61, and the end determination unit 62 are realized by the processor 510 illustrated in FIG.

<B-2-8. Change information generator>
The change information generating unit 58 is configured to generate, for each of the plurality of degrees of freedom, a first control indicating a relationship between a unit control amount of the target robot and a change amount of the state of the target object on the real image captured by the imaging device 21. Generate change information. The change information generation unit 58 stores a first change information set 591 including a plurality of first change information generated for a plurality of degrees of freedom in the change information storage unit 59.

  Further, the change information generating unit 58 indicates, for each of the plurality of degrees of freedom, the relationship between the unit control amount of the target robot and the change amount of the state of the target on the real image captured by the imaging device 22. The second change information is generated. The change information generation unit 58 stores, in the change information storage unit 59, a second change information set 592 including a plurality of pieces of second change information generated for a plurality of degrees of freedom.

  The target object is the male connector 2a in the first control unit 57a and the female connector 2b in the second control unit 57b. The target robot is the robot 30a in the first control unit 57a and the robot 30b in the second control unit 57b. The plurality of degrees of freedom are six degrees of freedom in the first control unit 57a and three degrees of freedom in the second control unit 57b.

  In the present embodiment, the first change information and the second change information indicate the amount of change in the state of the target on the image when the target robot is controlled by the unit control amount. Specifically, the first change information and the second change information include an object on the image before controlling the target robot by the unit control amount, and an object on the image after controlling the target robot by the unit control amount. Here is a mapping that converts to

  The amount of change in the state of the target on the image when the target robot is controlled by the unit control amount depends on the state of the target in the real space. Therefore, the change information generating unit 58 generates the first change information set 591 for each frame in the teaching range of the first CG moving image. Further, the change information generating unit 58 generates a second change information set 592 for each frame in the teaching range of the second CG moving image.

  The process of generating and storing the first change information set 591 and the second change information set 592 by the change information generating unit 58 is executed as advance preparation.

  With reference to FIG. 8, a method of generating the first change information set 591 in the first control unit 57a will be described. Note that the method of generating the second change information set 592 in the first control unit 57a and the method of generating the first change information set 591 and the second change information set 592 in the second control unit 57b are the same. The description of the generation method is omitted.

  FIG. 8 is a diagram illustrating a method of generating the first change information set 591 in the first control unit. FIG. 8A shows the k-th frame 84 of the first CG moving image. The state (position and posture) in the virtual space of the male connector model 2a 'corresponding to the k-th frame 84 is set as a reference state.

  FIG. 8B shows a CG image 85a with the model of the imaging device 21 as a viewpoint when the male connector model 2a ′ is translated from the reference state in the virtual space by a unit control amount to the degree of translational freedom in the Y direction. Is shown. FIG. 8C shows a CG image 85b with the model of the imaging device 21 as a viewpoint when the male connector model 2a 'is translated from the reference state in the virtual space by a unit control amount to the degree of translational freedom in the X direction. Is shown. FIG. 8D shows a CG image 85c with the model of the imaging device 21 as a viewpoint when the male connector model 2a ′ is translated from the reference state in the virtual space by a unit control amount to the translational freedom in the Z direction. Is shown. FIG. 8E shows a CG image 85d with the model of the imaging device 21 as a viewpoint when the male connector model 2a 'is rotated from the reference state in the virtual space by a unit control amount to the rotational degree of freedom in the pitch direction. Is shown. FIG. 8F shows a CG image 85e with the model of the imaging device 21 as a viewpoint when the male connector model 2a 'is rotated from the reference state in the virtual space by a unit control amount to the rotational degree of freedom in the yaw direction. Is shown. FIG. 8G shows a CG image 85f with the model of the imaging device 21 as a viewpoint when the male connector model 2a 'is rotated from the reference state in the virtual space by a unit control amount to the rotational degree of freedom in the roll direction. Is shown. The change information generating unit 58 generates CG images 85a to 85f by simulating the position and orientation of the male connector model 2a 'in the virtual space.

  The change information generation unit 58 acquires, from the image processing unit 55, the coordinates on the image of each feature point of the male connector model 2a 'extracted from each of the frame 84 and the CG images 85a to 85f.

  The change information generation unit 58 outputs information indicating a mapping that converts the coordinates of the feature points 4a ′ to 4g ′ extracted from the frame 84 into the coordinates of the feature points 5a ′ to 5g ′ extracted from the CG image 85a, respectively. It is generated as first change information corresponding to the degree of freedom of translation in the Y direction.

  Similarly, the change information generation unit 58 converts the information indicating the mapping that converts the coordinates of the feature points 4a ′ to 4g ′ into the coordinates of the feature points 5a ′ to 5g ′ extracted from the CG image 85b by using the translation in the X direction. It is generated as first change information corresponding to the degree. The change information generation unit 58 converts information indicating a mapping that converts the coordinates of the feature points 4a ′ to 4g ′ into the coordinates of the feature points 5a ′ to 5g ′ extracted from the image 85c, in accordance with the translation degree in the Z direction. Generated as first change information. The change information generation unit 58 converts information indicating a mapping that transforms the coordinates of the feature points 4a ′ to 4g ′ into the coordinates of the feature points 5a ′ to 5g ′ extracted from the image 85d according to the rotational degree of freedom in the pitch direction. Generated as first change information. The change information generation unit 58 converts information indicating a mapping that converts the coordinates of the feature points 4a ′ to 4g ′ into the coordinates of the feature points 5a ′ to 5g ′ extracted from the image 85e, in accordance with the degree of rotation in the yaw direction. Generated as first change information. The change information generation unit 58 converts the information indicating the mapping that converts the coordinates of the feature points 4a ′ to 4g ′ into the coordinates of the feature points 5a ′ to 5g ′ extracted from the image 85f according to the rotational degree of freedom in the roll direction. Generated as first change information. In this way, the change information generating unit 58 generates the first change information set 591 corresponding to the k-th frame 84 of the first CG moving image.

  The change information generating unit 58 generates a first change information set 591 corresponding to the remaining frames in the teaching range of the first CG moving image by the same method.

<B1-2-9. Calculation part>
The calculation unit 60 has a plurality of degrees of freedom for bringing the state of the target on the real image captured by the imaging devices 21 and 22 closer to the state of the model of the target on target frames of the first CG moving image and the second CG moving image, respectively. Is calculated.

  The calculation unit 60 obtains the first change information set 591 and the second change information set 592 corresponding to the target frame from the change information storage unit 59, and stores the acquired first change information set 591 and second change information set 592 in the set. The control amount is calculated based on the control amount. As described above, the first change information and the second change information include an object on the image before controlling the target robot by the unit control amount and an object on the image after controlling the target robot by the unit control amount. Here is a mapping that converts to Therefore, the calculation unit 60 calculates the coordinates of the feature points of the object extracted from the real image on the image and the coordinates of the feature points of the model of the object extracted from the target frame from the image processing unit 55. get. The calculation unit 60 calculates a control amount of each of a plurality of degrees of freedom for mapping an object on the real image to a model of the object on the target frame based on the change information.

  With reference to FIG. 9, a method of calculating the control amount by the calculating unit 60 of the first control unit 57a will be described. In addition, the calculation method of the control amount by the calculation unit 60 of the second control unit 57b is the same method, and thus the description is omitted.

  FIG. 9 is a diagram illustrating a method of calculating a control amount by the calculation unit of the first control unit. The calculation unit 60 acquires, from the image processing unit 55, the coordinates on the image of the feature points 4a 'to 4g' of the male connector model 2a 'extracted from the target frame of the first CG moving image. Further, the calculation unit 60 acquires, from the image processing unit 55, the coordinates on the image of the characteristic points 4a to 4g of the male connector 2a extracted from the real image obtained by the imaging of the imaging device 21. The number of feature points is not limited to seven.

  The calculation unit 60 calculates the difference vectors 6a to 6g of each feature point. The difference vectors 6a to 6g are vectors starting from the feature points 4a to 4g and ending at the feature points 4a 'to 4g', respectively. The calculation unit 60 calculates the average x component Δx1 and y component Δy1 of the difference vectors 6a to 6g. The x component and the y component are indicated by the coordinate system of the image.

  In a similar manner, the calculation unit 60 calculates the average x component of the difference vector between the feature point extracted from the real image obtained by the imaging of the imaging device 22 and the feature point extracted from the target frame of the second reference moving image. Calculate Δx2 and y component Δy2.

  First, the calculation unit 60 calculates the control amounts of the three translational degrees of freedom such that the average of the difference vectors of the plurality of feature points is eliminated. Specifically, the calculation unit 60 uses the Δx1, Δy1, Δx2, and Δy2, the first change information set 591 and the second change information set 592 to perform translation free movement in the X, Y, and Z directions of the hand 31a. The control amount of each degree is calculated.

  When the hand 31a translates in any of the degrees of freedom of translation in the X direction, the Y direction, and the Z direction, the male connector 2a translates in a certain direction on an image obtained by imaging by the imaging devices 21 and 22. Therefore, the first change information corresponding to the translation degree of freedom in the first change information set 591 indicates a mapping that converts an arbitrary point on the image into a point translated in a certain direction. Similarly, the second change information corresponding to the degree of freedom of translation in the second change information set 592 indicates a mapping that converts an arbitrary point on the image into a point translated in a certain direction.

  Here, the first change information corresponding to the translational degree of freedom in the X direction of the first change information set 591 corresponding to the target frame indicates a mapping that converts a point (x, y) into a point (x + dX1_1, y + dY1_1). And The first change information corresponding to the degree of freedom of translation in the Y direction indicates a mapping that converts an arbitrary point (x, y) on the image into a point (x + dX1_2, y + dY1_2). It is assumed that the first change information corresponding to the degree of freedom of translation in the Z direction indicates a mapping that converts an arbitrary point (x, y) on the image into a point (x + dX1_3, y + dY1_3).

  Further, the second change information corresponding to the degree of freedom of translation in the X direction of the second change information set 592 corresponding to the target frame converts an arbitrary point (x, y) on the image into a point (x + dX2_1, y + dY2_1). Are shown. The second change information corresponding to the degree of freedom of translation in the Y direction indicates a mapping that converts an arbitrary point (x, y) on the image into a point (x + dX2_2, y + dY2_2). The second change information corresponding to the translation degree of freedom in the Z direction indicates a mapping that converts an arbitrary point (x, y) on the image into a point (x + dX2_3, y + dY2_3).

  At this time, the calculation unit 60 calculates coefficients a1, a2, and a3 by solving the following four linear equations (1) to (4).

a1 × dX1_1 + a2 × dX1_2 + a3 × dX1_3 = Δx1 (1)
a1 × dY1_1 + a2 × dY1_2 + a3 × dY1_3 = Δy1 (2)
a1 × dX2_1 + a2 × dX2_2 + a3 × dX2_3 = Δx2 (3)
a1 × dY2_1 + a2 × dY2_2 + a3 × dY2_3 = Δy2 (4).

  In the present embodiment, the first change information and the second change information indicate the amount of change in the state of the target on the image when the target robot is controlled by the unit control amount. Therefore, the calculation unit 60 determines that the control amount of the translational freedom in the X direction is a1 times the unit control amount, the control amount of the translational freedom in the Y direction is a2 times the unit control amount, and the control amount of the translational freedom in the Z direction. Is a3 times the unit control amount. The control amounts of these translation degrees of freedom are control amounts for bringing the state of the male connector 2a on the real image closer to the state of the male connector 2a on the target frame by the average of the difference vectors of the plurality of feature points.

  Next, the calculation unit 60 calculates a control amount of each of the three rotational degrees of freedom. The calculation unit 60 subtracts the average x component (Δx1 or Δx2) and the y component (Δy1 or Δy2) from the difference vector of each feature point. The calculating unit 60 calculates a control amount of three rotational degrees of freedom in which the residual of the difference vector of each feature point is closest to 0, using a search algorithm such as a hill-climbing method, for example.

  Specifically, the calculation unit 60 starts a search algorithm with a solution in which the control amounts of the rotational degrees of freedom in the pitch direction, the yaw direction, and the roll direction are 0 as the current solution. The calculation unit 60 simulates a change in the residual of the difference vector of each feature point when the robot 30a is controlled according to each of a plurality of solutions near the current solution. The calculation unit 60 replaces the neighboring solution with the current solution when there is a neighboring solution whose residual of the difference vector of each feature point is closer to 0 than the current solution based on the simulation result. The calculation unit 60 searches for a solution in which the residual of the difference vector becomes an extreme value by repeating this process.

<B-2-10. Command part>
The command unit 61 generates a control command for moving the target robot by the control amount calculated by the calculation unit 60, and outputs the generated control command to the target robot controller. The target robot controller is the robot controller 40a in the first control unit 57a and the robot controller 40b in the second control unit 57b.

<B-2-11. End determination section>
The end determination unit 62 calculates a deviation between the state of the object on the real image and the state of the model of the object on the last frame of the teaching range, and when the calculated deviation is less than a predetermined threshold, It is determined that the control of the target robot ends. When determining that the control of the target robot is to be ended, the end determination unit 62 outputs an end notification.

  The deviation is, for example, an average of distances between corresponding feature points extracted from the real image and the last frame.

  The threshold is set according to the accuracy required for the state of the target. The threshold value is the threshold value Tha in the first control unit 57a and the threshold value Thb in the second control unit 57b. The threshold value Tha and the threshold value Thb may be the same or different.

<C. Operation example>
<C-1. Robot control>
With reference to FIG. 10, a description will be given of a processing flow for controlling the target robot so as to change the state of the target object according to the first CG moving image and the second CG moving image. FIG. 10 is a flowchart illustrating an example of a process flow for controlling the target robot so as to change the state of the target object along the first CG moving image and the second CG moving image.

  First, in step S1, the control device 50 determines whether or not an end notification has been output from the end determination units 62 of all the control units (the first control unit 57a and the second control unit 57b). If the end notification has been output from the end determination units 62 of all the control units (YES in step S1), the process ends.

  When the end notification has not been output from the end determination units 62 of all the control units (NO in step S1), in step S2, the control device 50 acquires the actual images captured by the imaging devices 21 and 22.

  In step S3, the image processing unit 55 detects all the objects (male connector 2a and female connector 2b) from the real image by template matching, and extracts the coordinates of the feature points of each object. Further, the image processing unit 55 detects all target models (male connector model 2a 'and female connector model 2b') from the target frame by template matching, and extracts the coordinates of the feature points of each model.

  In step S4, the target frame selection unit 56 selects a target frame from the first CG moving image and the second CG moving image.

  In step S5, the target frame selection unit 56 specifies a target including the target frame within the teaching range and controls the specified target (at least one of the first control unit 57a and the second control unit 57b). And outputs a control instruction to.

  In step S6, the control unit (at least one of the first control unit 57a and the second control unit 57b) that has received the control instruction controls the target robot. After step S6, the process returns to step S1. If NO in step S1, a series of processing in steps S2 to S6 is repeated for each imaging cycle. If NO in step S1, step S2 of acquiring the next actual image may be started while steps S2 to S6 control the target robot in step S6. Thus, the target robot is continuously controlled according to the latest actual image without stopping the operation of the target robot. As a result, the state of the object can be changed quickly.

<C-2. Select target frame>
With reference to FIG. 11 and FIG. 12, a flow of a target frame selecting process performed by the target frame selecting unit 56 will be described. FIG. 11 is a flowchart showing the flow of the processing of the subroutine of step S4 shown in FIG. FIG. 12 is a diagram illustrating the relationship between the closest frame and the target frame.

  First, in step S11, the target frame selection unit 56 acquires from the image processing unit 55 the coordinates of the feature points of the models of all the objects extracted from each frame of the first CG moving image and the second CG moving image.

  In step S12, the target frame selection unit 56 obtains, from the image processing unit 55, the coordinates of the characteristic points of all the objects extracted from the real images captured by the imaging devices 21 and 22.

  In step S13, the target frame selection unit 56 determines whether the first target frame selection has been completed.

  If the first target frame selection has been completed (YES in step S13), in step S14, the target frame selecting unit 56 determines that the previous target frame is the last frame in the teaching range corresponding to any one of the objects. It is determined whether or not there is.

  If the previous target frame is the last frame (YES in step S14), the process proceeds to step S15. In step S15, the target frame selection unit 56 calculates the deviation between the state of the target on the real image and the state of the model of the target on the final frame for the target corresponding to the teaching range to which the final frame belongs. , The deviation is less than the threshold. The deviation is, for example, an average of distances between corresponding feature points in the real image and the final frame.

  If the deviation is equal to or larger than the threshold (NO in step S15), in step S16, the target frame selecting unit 56 selects the same frame as the previous target frame as the target frame. After step S16, the process ends.

  If the deviation is less than the threshold (YES in step S15), it is determined that the state of the object has reached the state of the model of the object on the final frame, and the process proceeds to step S17. If the first target frame selection has not been completed (NO in step S13), and if the previous target frame is not the last frame (NO in step S14), the process proceeds to step S17.

  In step S17, the target frame selecting unit 56 calculates the deviation between the state of all the objects on the real image and the state of the model of all the objects on each frame, and sets the frame with the minimum deviation as the closest frame. Identify. The deviation is, for example, an average of distances between corresponding feature points in the actual image and each frame.

  Specifically, the target frame selection unit 56 determines the state of all the objects on the real image captured by the imaging device 21 and the state of the model of all the objects on the k-th frame of the first CG moving image. A first deviation is calculated. Further, the target frame selection unit 56 is configured to determine the second state of the state of all the objects on the real image captured by the imaging device 22 and the state of the model of all the objects on the k-th frame of the second reference moving image. Calculate the deviation. The target frame selection unit 56 calculates the average of the first deviation and the second deviation as the deviation corresponding to the k-th frame.

  In step S18, the target frame selection unit 56 determines whether or not there is a last frame in any of the teaching ranges up to a predetermined number of frames after the closest frame.

  If there is a final frame (YES in step S18), in step S19, the target frame selecting unit 56 selects the final frame as a target frame. If there are a plurality of final frames up to a predetermined number of frames after the closest frame, the target frame selecting unit 56 selects the last frame having the smallest frame number among the plurality of final frames.

  If there is no last frame (NO in step S18), in step S20, the target frame selection unit 56 selects a frame that is a predetermined number later than the closest frame as the target frame. After step S20, the target frame selection process ends.

<C-3. Processing of control unit>
The processing of the subroutine of step S6 in FIG. 10 will be described with reference to FIG. FIG. 13 is a flowchart showing the flow of the processing of the subroutine of step S6 shown in FIG.

  First, in step S21, the end determination unit 62 determines whether the target frame is the last frame in the teaching range.

  If the target frame is the final frame (YES in step S21), in step S22, the end determination unit 62 determines that the deviation between the state of the target on the real image and the state of the model of the target on the final frame is less than the threshold. It is determined whether or not.

  When the deviation is smaller than the threshold (YES in step S22), in step S23, the termination determination unit 62 outputs a termination notification. After step S23, the process ends.

  If the target frame is not the last frame (NO in step S21) and if the deviation is equal to or larger than the threshold (NO in step S22), the process proceeds to step S24.

  In step S24, the calculation unit 60 converts the state of the target on the real image into the state of the model of the target on the target frame based on the first change information set 591 and the second change information set 592 corresponding to the target frame. Is calculated for each of the plurality of degrees of freedom to approach.

  In step S25, the command unit 61 generates a control command indicating the calculated control amount and outputs the control command to the target robot controller. After step S25, the process ends.

  The first control unit 57a and the second control unit 57b each control the target robot according to the flow shown in FIG. When the target frame is included in the teaching range of the male connector 2a and the female connector 2b, the robot 30a is controlled such that the state of the male connector 2a on the actual image approaches the state of the male connector model 2a 'on the target frame. Further, the robot 30b is controlled such that the state of the female connector 2b on the actual image approaches the state of the female connector model 2b 'on the target frame. As a result, the states of the male connector 2a and the female connector 2b change in a coordinated manner according to the target frame.

<C-4. Action / Effect>
As described above, the control system 1 includes the robots 30a and 30b, the imaging devices 21 and 22, and the control device 50 that controls the robots 30a and 30b. The robots 30a and 30b support the male connector 2a and the female connector 2b, and change the states of the male connector 2a and the female connector 2b, respectively. The imaging devices 21 and 21 are installed at fixed positions different from the robots 30a and 30b, and image the male connector 2a and the female connector 2b supported by the robots 30a and 30b, respectively.

  The control device 50 stores a first CG moving image and a second CG moving image indicating models of the male connector 2a and the female connector 2b.

  Control device 50 acquires change information for each of male connector 2a and female connector 2b. The change information corresponding to the male connector 2a indicates the relationship between the control amount of the robot 30a and the change amount of the state of the male connector 2a on the images of the imaging devices 21 and 22. The change information corresponding to the female connector 2b indicates the relationship between the control amount of the robot 30b and the change amount of the state of the female connector 2b on the images of the imaging devices 21 and 22.

  The control device 50 is configured to perform a first process of acquiring a real image captured by the imaging devices 21 and 22, a second process of selecting a target frame from the first CG video and the second CG video, and a process based on the real image and the target frame. And a third process for controlling each of the robots 30a and 30b. The control device 50 calculates a control amount of the robot 30a for bringing the state of the male connector 2a on the real image closer to the state of the male connector model 2a 'on the target frame, based on the change information corresponding to the male connector 2a. The robot 30a is controlled according to the calculated control amount. The control device 50 calculates a control amount of the robot 30b for bringing the state of the female connector 2b on the actual image closer to the state of the female connector model 2b 'on the target frame based on the change information corresponding to the female connector 2b. The robot 30b is controlled according to the calculated control amount.

  Thereby, the states of the male connector 2a and the female connector 2b on the actual image can be changed to the states of the male connector model 2a 'and the female connector model 2b' on the target frame, respectively. That is, the states of the male connector 2a and the female connector 2b change in conjunction with each other according to the target frame.

  The control device 50 repeatedly executes a series of processes including the first to third processes, and starts the first process of the next series of processes while performing the third process. Thereby, the robots 30a and 30b are continuously controlled according to the latest actual image without stopping the operations of the robots 30a and 30b. As a result, the states of the male connector 2a and the female connector 2b can be changed quickly.

<D. Modification>
<D-1. Modification 1>
FIG. 14 is a diagram showing the design data of the female connector and the dimensions of the female connector manufactured according to the design data. In general, depending on manufacturing conditions, dimensions of a manufactured product may deviate from design data, and a manufacturing error may occur. In the example shown in FIG. 14, a manufacturing error (systematic error) in which the position of the insertion hole 7 tends to shift in a direction away from the notch 8 is shown.

  In the above description, the robot 30b is controlled such that the state of the female connector 2b on the actual image approaches the state of the female connector model 2b 'on the target frame. The target frame is selected from the first GC moving image and the second CG moving image generated based on the design data. Therefore, when a manufacturing error as shown in FIG. 14 occurs, when the states of the male connector 2a and the female connector 2b are changed according to the first CG moving image and the second CG moving image, the pins (not shown) of the male connector 2a are changed. There is a possibility that the connector cannot be inserted into the insertion hole 7 of the female connector 2b.

  In the control system according to the first modification, the control amount calculated by the calculation unit 60 is adjusted by an offset amount according to a manufacturing error. The command unit 61 generates a control command for moving the target robot by the adjusted control amount, and outputs the generated control command to the target robot controller. Thus, even if a manufacturing error occurs, the pins of the male connector 2a can be inserted into the insertion holes 7 of the female connector 2b.

  FIG. 15 is a schematic diagram illustrating an outline of a control system according to the first modification. As shown in FIG. 15, the control system 1A according to the modification 1 further includes a force sensor 32 and a control device 50A instead of the control device 50, as compared with the control system 1 shown in FIG. Differs in that

  Force sensor 32 detects a force that one of male connector 2a and female connector 2b receives from the other. In the example shown in FIG. 15, the force sensor 32 is attached between the arm of the robot 30a and the hand 31a so as to detect the force that the male connector 2a receives from the female connector 2b.

  The control device 50A has a hardware configuration as shown in FIG. Therefore, detailed description of the hardware configuration of the control device 50A is omitted.

  FIG. 16 is a block diagram illustrating a functional configuration of a control device according to the first modification. As shown in FIG. 16, the control device 50A is different from the control device 50 shown in FIG. 5 in that a first control unit 157a and a second control unit 157b are used instead of the first control unit 57a and the second control unit 57b. And an offset amount determination unit 63 is further provided.

  The offset amount determination unit 63 determines an offset amount (also referred to as an “adjustment amount”) for adjusting the control amount calculated by the calculation unit 60 of at least one of the first control unit 157a and the second control unit 157b. In the first modification, the offset amount determination unit 63 determines an offset amount for adjusting the control amount calculated by the calculation unit 60 of the second control unit 57b, and outputs the determined offset amount to the second control unit 157b. I do.

  The offset amount determination unit 63 searches for the state of the female connector 2b at which the force detected by the force sensor 32 is minimal, and determines the offset amount based on the searched state of the female connector 2b. When the force detected by the force sensor 32 is minimal, the resistance when the male connector 2a and the female connector 2b are joined is minimal. At this time, the relative displacement between the pins of the male connector 2a and the insertion holes 7 of the female connector 2b is also minimized.

  The offset amount determining unit 63 determines each offset amount of the three degrees of freedom in the robot 30b to be controlled by the second control unit 57b. Specifically, the offset amount determination unit 63 includes an offset amount ΔX of the translational degree of freedom of the stage 31b in the X direction, an offset amount ΔY of the degree of translational freedom of the stage 31b in the Y direction, and an offset amount of the rotational degree of freedom of the stage 31b. Determine the quantity Δθ.

  FIG. 17 is a block diagram illustrating a functional configuration of the first control unit and the second control unit according to the first modification. As shown in FIG. 17, each of first control unit 157a and second control unit 157b is different from first control unit 57a and second control unit 57b shown in FIG. The difference is that a unit 161 is provided.

  When receiving the offset amount from the offset amount determination unit 63, the command unit 161 adjusts the control amount by adding the offset amount to the control amount calculated by the calculation unit 60. The command unit 161 generates a control command for moving the robot 30b by the adjusted control amount, and outputs the generated control command to the robot controller 40b. When the command unit 161 does not receive the offset amount from the offset amount determination unit 63, similarly to the command unit 61, the command unit 161 generates a control command for moving the robot 30b by the control amount calculated by the calculation unit 60. Then, the generated control command is output to the robot controller 40b.

  With reference to FIG. 18, a description will be given of a flow of an offset amount determination process in the first modification. FIG. 18 is a flowchart illustrating an example of the flow of an offset amount determination process in the first modification.

  The offset amount determination process shown in FIG. 18 is performed at a timing at which a manufacturing error of at least one of the male connector 2a and the female connector 2b tends to fluctuate (for example, a timing at which at least one lot of the male connector 2a and the female connector 2b is changed). It is performed every time.

  First, in step S31, the offset amount determination unit 63 sets (ΔX, ΔY, Δθ) = (0, 0, 0) as the current solution of the offset amount, and sets the set current solution to the second control unit 157b. Output to

  In step S32, control device 50A repeatedly performs a series of steps S1 to S6 shown in FIG. 10 until YES is obtained in step S1. At this time, robot control according to (ΔX, ΔY, Δθ) = (0, 0, 0) is performed. That is, in step S25 of FIG. 13 showing the subroutine of step S6, the command unit 161 of the first control unit 157a and the second control unit 157b performs control for moving the robot 30b by the control amount calculated by the calculation unit 60. A command is generated, and the generated control command is output to the robot controller 40b.

  In step S33, the offset amount determination unit 63 acquires the detection value of the force sensor 32 during the processing in step S32. The detection value is, for example, an average value or a maximum value of the force detected by the force sensor 32 during the process of step S32.

  In step S34, the offset amount determining unit 63 selects one of the neighboring solutions of the current solution, and outputs the selected neighboring solution to the second control unit 157b. When (ΔX, ΔY, Δθ) = (X0, Y0, θ0) is the current solution, (ΔX, ΔY, Δθ) = (X0 + a1, Y0, θ0), (X0−a1, Y0, θ0), ( X0, Y0 + a1, θ0), (X0, Y0-a1, θ0), (X0, Y0, θ0 + a2), and (X0, Y0, θ0-a2) are taken as the neighborhood solutions. a1 indicates a unit control amount of the degree of freedom of translation of the stage 31b. a2 indicates a unit control amount of the degree of freedom of rotation of the stage 31b.

  In step S35, control device 50A repeatedly performs a series of steps S1 to S6 shown in FIG. 10 until YES is obtained in step S1. At this time, the robot is controlled according to the neighborhood solution selected in step S34. That is, the command unit 161 of the second control unit 157b adjusts the control amount by adding the offset amount indicated by the neighborhood solution to the control amount calculated by the calculation unit 60. The command unit 161 of the first control unit 157a does not adjust the control amount calculated by the calculation unit 60 because it has not received the offset amount.

  In step S36, the offset amount determination unit 63 acquires the detection value of the force sensor 32 during the processing in step S35.

  In step S37, the offset amount determination unit 63 determines whether there is an unselected neighboring solution. If there is an unselected neighborhood solution (YES in step S37), the offset amount determination process returns to step S34. As a result, the detection value of the force sensor 32 can be obtained even for an unselected neighborhood solution.

  If there is no unselected neighborhood solution (NO in step S37), in step S38, the offset amount determination unit 63 determines whether the detection value corresponding to the current solution is minimal. Specifically, when the detected value corresponding to the current solution is smaller than all the detected values (detected values corresponding to all of the neighboring solutions) acquired in step S36, the offset amount determination unit 63 sets the current It is determined that the detection value corresponding to the solution is minimal.

  If the detection value corresponding to the current solution is not the minimum (NO in step S38), in step S39, the offset amount determination unit 63 sets the nearby solution having the minimum detection value as the current solution. After step S39, the process of determining the offset amount returns to step S34.

  If the detection value corresponding to the current solution is minimal (YES in step S38), in step S40, the offset amount determination unit 63 maintains the current solution of the offset amount and sets the current solution to the second control unit. 157b. After step S40, the offset amount determination processing ends.

  Note that the offset amount determination unit 63 may execute steps S31 to S40 for each of a plurality of different pairs of the male connector 2a and the female connector 2b to obtain an offset amount, and calculate an average value thereof. .

  According to the flowchart shown in FIG. 18, when the state of the female connector 2b is matched with the state of the female connector model 2b 'shown in the first CG moving image and the second CG moving image in steps S32 and S33, A detection value is obtained. After that, by repeating steps S34 to S39, the state of the female connector 2b at which the detection value of the force sensor 32 is minimized is searched for. Then, the amount of change from the state of the female connector model 2b 'to the searched state of the female connector 2b is maintained as the final offset amount.

  When the male connector 2a and the female connector 2b are manufactured according to the design data, by adjusting the state of the female connector 2b to the state of the female connector model 2b 'shown in the first CG moving image and the second CG moving image, the force sensor 32 Is minimum.

  However, when a manufacturing error occurs in at least one of the male connector 2a and the female connector 2b, the relative position between the pin of the male connector 2a and the insertion hole 7 of the female connector 2b may be shifted according to the manufacturing error. As a result, an unintended resistance force (including frictional force) is generated between the male connector 2a and the female connector 2b, and the detection value of the force sensor 32 increases.

  As described above, the offset amount maintained in step S40 is the amount of change from the state of the female connector model 2b 'to the state of the female connector 2b at which the detection value of the force sensor 32 is minimal. The change amount corresponds to a relative displacement amount between the pin of the male connector 2a and the insertion hole 7 of the female connector 2b. The displacement is an amount corresponding to a manufacturing error of at least one of the male connector 2a and the female connector 2b. That is, the offset amount at which the detection value of the force sensor 32 becomes a minimum is an amount corresponding to a manufacturing error of at least one of the male connector 2a and the female connector 2b, and the pin of the male connector 2a is inserted into the insertion hole 7 of the female connector 2b. Shows the relative positional shift amount.

  According to the first modification, even if a manufacturing error occurs, the pins of the male connector 2a can be inserted into the insertion holes 7 of the female connector 2b.

  In the above description, the offset amount determination unit 63 determines the offset amount for adjusting the control amount calculated by the calculation unit 60 of the second control unit 57b. An offset amount for adjusting the control amount calculated by the above may be determined.

<D-2. Modification 2>
In the first modification, the offset amount determination unit 63 may acquire the offset amount from the input device 534 (see FIG. 4). The operator may measure a systematic error with respect to the design data of the male connector 2a and the female connector 2b, and input an offset amount corresponding to the systematic error to the input device 534.

  In addition, the operator adjusts the control amount calculated by the calculation unit 60 of the first control unit 57a and the offset amount for adjusting the control amount calculated by the calculation unit 60 of the second control unit 57b. At least one of the offset amounts is input to the input device 534.

  For example, the operator inputs an offset amount for adjusting the control amount calculated by the calculation unit 60 of the first control unit 57a based on the systematic error of the male connector 2a. Further, the operator inputs an offset amount for adjusting the control amount calculated by the calculator 60 of the second controller 57b based on the systematic error of the female connector 2b.

  Alternatively, the operator obtains the relative displacement between the pin of the male connector 2a and the insertion hole 7 of the female connector 2b from the systematic error of the male connector 2a and the female connector 2b. The operator may input an offset amount for adjusting the control amount calculated by the calculation unit 60 of the first control unit 57a or the second control unit 57b according to the position shift amount.

<D-3. Modification 3>
The control system according to the third modification differs from the control system 1A according to the first modification in that a control device 50B having a functional configuration as shown in FIG. 19 is provided instead of the control device 50A. The control device 50B according to the third modification has a hardware configuration as shown in FIG. Therefore, a detailed description of the hardware configuration of the control device 50B will be omitted.

  FIG. 19 is a block diagram illustrating a functional configuration of a control device according to the third modification. As shown in FIG. 19, the control device 50B is different from the control device 50A shown in FIG. 16 in that the teaching range selecting unit 54, the image processing unit 55, the target frame selecting unit 56, the first control unit 157a, and the second The difference is that a teaching range selection unit 254, an image processing unit 255, a target frame selection unit 256, a first control unit 257a, and a second control unit 257b are provided instead of the control unit 157b, and a reference moving image generation unit 64 is further provided.

  The control device 50B controls the robot 30a, 30b using the first CG moving image and the second CG moving image, and uses the first reference moving image and the second reference moving image instead of the first CG moving image and the second CG moving image. And a second mode for controlling 30a and 30b.

  In the first mode, the control device 50B performs the processing shown in the flowchart of FIG. 18 and obtains the solution of the offset amount at which the detection value of the force sensor 32 is minimized.

  The reference moving image generation unit 64 stores an image set including a plurality of images acquired for each imaging cycle from the imaging device 21 in each of steps S33 and S35 shown in FIG. The reference moving image generation unit 64 reads an image set corresponding to the solution of the offset amount at which the detection value of the force sensor 32 is minimized, and sets a first reference moving image having each of a plurality of images included in the read image set as a frame. Generate

  Similarly, the reference moving image generation unit 64 stores an image set including a plurality of images acquired for each imaging cycle from the imaging device 22 in each of step S33 and step S35. The reference moving image generation unit 64 reads an image set corresponding to the solution of the offset amount at which the detection value of the force sensor 32 is minimal, and sets a second reference moving image having each of a plurality of images included in the read image set as a frame. Generate

  The reference moving image generation unit 64 stores the generated first reference moving image and the second reference moving image in the moving image storage unit 51. The k-th frame of the first reference moving image (k is an integer from 1 to Q) and the k-th frame of the second reference moving image are images captured by the imaging devices 21 and 22 in the same imaging cycle. is there.

  When the first reference moving image and the second reference moving image are stored in the moving image storage unit 51, the control device 50B controls the robots 30a and 30b according to the second mode.

  The teaching range selection unit 254 performs the teaching from the first reference moving image and the second reference moving image for each object (the male connector 2a and the female connector 2b) in addition to the processing of the teaching range selection unit 54 illustrated in FIGS. Select a range. The teaching range selection unit 254 may select the teaching range for each object from the first reference moving image and the second reference moving image by the same method as that for selecting the teaching range from the first CG moving image and the second CG moving image.

  The image processing unit 255 performs image processing on the target image and detects the target from the target image using template matching, similarly to the image processing unit 55 illustrated in FIGS. 5 and 16. However, in addition to the frames of the first CG moving image, the frames of the second CG moving image, and the real images captured by the imaging devices 21 and 22, the target image on which the image processing unit 255 performs the image processing includes a frame of the first reference moving image and a frame of the first reference moving image. A frame of the second reference moving image is included.

  The target frame selection unit 256 selects a target frame from each of the first CG moving image and the second CG moving image in the first mode, similarly to the target frame selection unit 56 illustrated in FIGS. 5 and 16.

  The target frame selection unit 256 selects a target frame from each of the first reference moving image and the second reference moving image stored in the moving image storage unit 51 in the second mode. The target frame selection unit 256 may select a target frame from each of the first reference moving image and the second reference moving image according to the flow illustrated in FIG. However, the target frame selection unit 256 acquires, from the image processing unit 255, the coordinates of the feature points of all the objects extracted from each frame of the first reference moving image and the second reference moving image in step S11 of FIG.

  FIG. 20 is a block diagram illustrating a functional configuration of the first control unit and the second control unit according to the third modification. As shown in FIG. 20, each of the first control unit 257a and the second control unit 157b is different from the first control unit 157a and the second control unit 157b shown in FIG. The difference is that a change information generating unit 258, a calculating unit 260, and a command unit 261 are provided instead of the unit 60 and the command unit 161.

  The change information generation unit 258 generates the first change information set 591 for each frame in the teaching range of the first CG moving image, and generates each of the teaching ranges of the second CG moving image in the same manner as the change information generation unit 58 illustrated in FIG. A second change information set 592 is generated for the frame.

  Further, the change information generation unit 258 generates a third change information set 593 for each frame in the teaching range of the first reference moving image, and generates a fourth change information set for each frame in the teaching range of the second reference moving image. 594 is generated.

  The change information generation unit 258 acquires from the image processing unit 255 the coordinates of the feature points of the target object (the male connector 2a or the female connector 2b) extracted from each frame of the first reference moving image and the second reference moving image. The change information generation unit 258 acquires from the image processing unit 255 the coordinates of the feature points of the object models (the male connector model 2a 'and the female connector model 2b') extracted from each frame of the first CG moving image and the second CG moving image. I do.

  For the i-th frame of the first reference moving image, the change information generating unit 258 sets corresponding feature points extracted from the i-th frame and the j-th (j is an integer of 1 to M) frame of the first CG moving image. (For example, average distance). The change information generation unit 258 specifies j at which the deviation is minimum. The change information generation unit 258 sets the third change information set 593 corresponding to the i-th frame of the first reference moving image to the same content as the first change information set 591 corresponding to the specified j-th frame in the first CG moving image. And

  Similarly, the change information generating unit 258 determines, for the i-th frame of the second reference moving image, the corresponding extracted from the i-th frame and the j-th frame (j is an integer of 1 to M) of the second CG moving image. The deviation between the feature points (for example, the average of the distances) is calculated. The change information generation unit 258 specifies j at which the deviation is minimum. The change information generation unit 258 sets the fourth change information set 594 corresponding to the i-th frame of the second reference moving image to the same content as the second change information set 592 corresponding to the specified j-th frame in the second CG moving image. And

  The calculation unit 260 calculates the control amount of each degree of freedom of the target robot by performing the same processing as the calculation unit 60 illustrated in FIG. In the first mode, the calculating unit 260 reads the first change information set 591 and the second change information set 592 corresponding to the target frame from the change information storage unit 59, and reads the read first change information set 591 and second change information A control amount is calculated based on the set 592. The calculation unit 260 reads the third change information set 593 and the fourth change information set 594 corresponding to the target frame from the change information storage unit 59 in the second mode, and reads the read third change information set 593 and fourth read information A control amount is calculated based on the set 594.

  The command section 261 outputs a control command to the target robot controller by performing the same processing as the command section 161 shown in FIG. 17 in the first mode. That is, when receiving the offset amount from the offset amount determining unit 63, the command unit 261 adjusts the control amount using the offset amount and generates a control command to move the target robot by the adjusted control amount.

  The command section 261 outputs a control command to the target robot controller by performing the same processing as the command section 61 shown in FIG. 7 in the second mode. That is, the command unit 261 generates a control command to move the target robot by the control amount received from the calculation unit 260.

  According to the third modification, in the first mode, the control device 50B adjusts the control amount calculated by the calculation unit 260 by the offset amount, and searches for the offset amount at which the force detected by the force sensor 32 is minimal. I do. The control device 50B acquires images captured by the imaging devices 21 and 22 as the first and second reference moving images when the control amount is adjusted by the searched offset amount.

  Thereafter, control device 50B performs a process according to the second mode. That is, control device 50B performs a process of selecting a target frame from the first reference moving image and the second reference moving image instead of the first CG moving image and the second CG moving image.

  The first reference moving image and the second reference moving image are images captured when the force detected by the force sensor 32 is minimal. Therefore, according to the third modification, the control device 50B performs the processing according to the second mode so that the pins of the male connector 2a are inserted into the insertion holes 7 of the female connector 2b even if a manufacturing error occurs. Can be.

<D-4. Modification 4>
In the above description, the control system 1 connects (assembles) the male connector 2a and the female connector 2b. However, the control system 1 may assemble another two objects.

  FIG. 21 is a schematic diagram illustrating an object of the control system according to the fourth modification. The control system according to Modification 4 screws a bolt 9a into a bolt hole 13 formed in a plate-like component 9b in an industrial product production line or the like.

  The bolt 9a is gripped by the hand 31a (see FIG. 1) of the robot 30a. The plate-shaped component 9b is placed on the stage 31b of the robot 30b.

<D-5. Modification 5>
The number of robots provided in the control system is not limited to two, and may be three or more. When the states of three or more objects are changed in a coordinated manner according to the CG moving image, the control system may include three or more robots for changing the states of the three or more objects, respectively. . In this case, the CG moving image indicates a change in the state of the model of three or more objects.

  FIG. 22 is a schematic diagram illustrating an object of the control system according to the fifth modification. The control system according to Modification Example 5 joins cylindrical members 10e and 10f with screws 10c and 10d in an industrial product production line or the like.

  Screw holes 11a and 11b are formed in the cylindrical member 10e, and screw holes 12a and 12b are formed in the cylindrical member 10f. With the screw hole 11a and the screw hole 12a overlapping and the screw hole 11b and the screw hole 12b overlapping, the screw 10c is inserted into the screw holes 11a and 12a, and the screw 10d is inserted into the screw holes 11b and 12b.

  The screw 10c is gripped by the hand of the first robot. The screw 10d is gripped by the hand of the second robot. The cylindrical member 10e is gripped by the hand of the third robot. The cylindrical member 10f is mounted on the stage of the fourth robot.

  The imaging devices 21 and 22 are arranged at positions where the screw holes 11a and 12a and the screw 10c can be imaged. However, the imaging devices 21 and 22 cannot image the screw holes 11b and 12b and the screw 10d. Therefore, the control system according to Modification 5 further includes imaging devices 23 and 24. The imaging devices 23 and 24 are arranged at positions where the screw holes 11b and 12b and the screw 10d can be imaged.

  The control device of Modification 5 stores four CG moving images corresponding to the imaging devices 21 to 24, respectively.

  The control device according to Modification 5 may include first to fourth control units respectively corresponding to the first to fourth robots. Each of the first to fourth control units has the same configuration as in FIG. The change information generating units 58 of the first to fourth control units may generate four change information sets respectively corresponding to the four CG moving images. Then, the calculation unit 60 may calculate the control amount of each degree of freedom of the target robot based on the four change information sets. Thus, the states of the four objects (cylindrical members 10e and 10f and screws 10c and 10d) can be changed in a coordinated manner according to the CG moving image.

<D-6. Modification 6>
With reference to FIG. 23, a control system according to Modification 6 will be described. FIG. 23 is a schematic diagram illustrating an outline of a control system according to the sixth modification. In the sixth modification, one of the plurality of objects is set at a fixed position, and the state of the imaging devices 21 and 22 is changed by the robot.

  The control system 1C according to Modification 6 sequentially processes the processing target portions 15 and 16 of the large member 14e using the processing tools 14c and 14d in a production line of an industrial product or the like. The large member 14j is, for example, a housing of a large device, an automobile body, or the like. The processing tools 14h and 14i are, for example, a drill, an electric file, or the like.

  The control system 1C is different from the control system 1 shown in FIG. 1 in that instead of the robots 30a, 30b, the robot controllers 40a, 40b, and the control device 50, the robots 30c, 30d, 30e, the robot controllers 40c, 40d, 40e and The difference is that a control device 50C is provided.

  The robot 30c is a mechanism for changing the state (here, position and posture) of the processing tool 14c, and is, for example, a vertical articulated robot. The robot 30c has a hand 31c at its tip for supporting the processing tool 14c, and changes the position and posture of the hand 31c with a plurality of degrees of freedom. Further, the robot 30c includes a pedestal 33c that is movable along the rail 34 in the direction of the arrow AR.

  The robot 30d is a mechanism for changing the state (here, position and posture) of the processing tool 14d, and is, for example, a vertical articulated robot. The robot 30d has a hand 31d that supports the processing tool 14d at its tip, and changes the position and posture of the hand 31d with a plurality of degrees of freedom. Further, the robot 30d includes a pedestal 33d movable along the rail 34 in the direction of the arrow AR.

  The robot 30e is a mechanism for changing the state (here, position and orientation) of the imaging devices 21 and 22, and is, for example, a vertical articulated robot. The robot 30e has a hand 31e that supports the imaging devices 21 and 22 at the tip, and changes the position and orientation of the hand 31e with a plurality of degrees of freedom. Further, the robot 30e includes a pedestal 33e movable along the rail 34 in the direction of the arrow AR.

  In FIG. 23, the pedestals 33c, 33d, and 33e move along the common rail. However, a rail is provided for each of the pedestals 33c, 33d, and 33e, and each of the pedestals 33c, 33d, and 33e may move along the corresponding rail.

  The robot controllers 40c, 40d, and 40e perform operation control of the robots 30c, 30d, and 30e, respectively, according to control commands received from the control device 50C. The robot controllers 40c, 40d, and 40e change the states of the hands 31c, 31d, and 31e, respectively, and move the pedestals 33c, 33d, and 33e, respectively, according to a control command from the control device 50B.

  Control device 50C has a hardware configuration as shown in FIG. 4, as in the first embodiment. Therefore, a detailed description of the hardware configuration of the control device 50C is omitted.

  FIG. 24 is a block diagram illustrating a functional configuration of a control device according to the sixth modification. As shown in FIG. 24, the control device 50C is different from the control device 50 shown in FIG. 5 in that a first control unit 57c and a second control unit 57d are used instead of the first control unit 57a and the second control unit 57b. And a third control unit 57e.

  However, the moving image storage unit 51 stores a first CG moving image and a second CG moving image showing a sample of the processing tools 14c and 14d and the large member 14e.

  The first CG moving image and the second CG moving image sequentially indicate, for example, the following first to third scenes. The first scene is a scene in which the models of the processing tools 14c and 14d process the model of the processing target portion 15 when the model of the processing target portion 15 is at a fixed position on the image. The second scene is a scene in which the model of the large member 14e moves on the image, and the model of the processing target portion 16 of the large member 14e moves to a fixed position on the image. The third scene is a scene in which the models of the processing tools 14c and 14d process the model of the processing target portion 16 when the model of the processing target portion 16 is at a fixed position on the image.

  The teaching range selection unit 54 selects a teaching range for each of the processing tool 14c, the processing tool 14d, and the large member 14e.

  The image processing unit 55 detects the processing tool 14c, the processing tool 14d, and the large member 14e from the target image. Since the size of the large member 14e is large, the field of view of the imaging devices 21 and 22 includes only a part of the large member 14e. Therefore, the image processing unit 55 detects a pattern formed on the surface of the large member 14e.

  The first control unit 57c controls the robot 30c via the robot controller 40c to change the state of the processing tool 14c.

  The second control unit 57d controls the robot 30d via the robot controller 40d to change the state of the processing tool 14c.

  The third control unit 57e controls the robot 30e via the robot controller 40e to change the states of the imaging devices 21 and 22.

  Each of the first control unit 57c, the second control unit 57d, and the third control unit 57e includes, like the first control unit 57a and the second control unit 57b, a change information generation unit 57, a change information storage unit 59, It includes a calculation unit 60, a command unit 61, and an end determination unit 62 (see FIG. 7).

  However, the target object, the target robot, and the target robot controller in each unit are different from those in the above-described embodiment. The target object is the processing tool 14c in the first control unit 57c, the processing tool 14d in the second control unit 57d, and the large member 14e in the third control unit 57e. The target robot is the robot 30c in the first control unit 57c, the robot 30d in the second control unit 57d, and the robot 30e in the third control unit 57e. The target robot controller is the robot controller 40e in the first controller 57c, the robot controller 40d in the second controller 57d, and the robot controller 40e in the third controller 57e.

  When the robot 30e is not operating, the change information generation unit 58 of the first control unit 57c and the second control unit 57d determines the unit control amount of the target robot and the target object on the real image captured by the imaging device 21. The first change information indicating the relationship with the change amount of the state is generated. Further, the change information generation unit 58 of the first control unit 57c and the second control unit 57d determines the unit control amount of the target robot and the actual control The second change information indicating the relationship with the amount of change in the state of the object is generated.

  Specifically, the change information generation unit 58 of the first control unit 57c performs the first change information and the second change information in the state of the imaging devices 21 and 22 where the processing target portion 15 of the large member 14e is at a fixed position on the image. Generate change information. The change information generation unit 58 of the second control unit 57d generates the first change information and the second change information in the state of the imaging devices 21 and 22 where the processing target portion 16 of the large member 14e is at a fixed position on the image. .

  The state of the large member 14e is not changed by the robot 30e. However, since the state of the imaging devices 21 and 22 is changed by the robot 30e, the state of the large member 14e on the real images of the imaging devices 21 and 22 is changed. Therefore, the change information generation unit 58 of the third control unit 57e performs the first change indicating the relationship between the unit control amount of the robot 30e and the change amount of the state of the large member 14e on the real image captured by the imaging device 21. Generate information. The change information generation unit 58 of the third control unit 57e performs the second change indicating the relationship between the unit control amount of the robot 30e and the change amount of the state of the large member 14e on the real image captured by the imaging device 22. Generate information.

  The processing contents of the calculation unit 60, the command unit 61, and the end determination unit 62 in the third control unit 57e are the same as those of the first control unit 57a and the second control unit 57b according to the above embodiment.

On the other hand, the calculation unit 60 in the first control unit 57c and the second control unit 57d determines that the state of the target on the real image approaches the state of the target on the target frame only when the following start conditions are satisfied. First, the control amount of the target robot is calculated.
Start condition: The deviation between the state of the large member 14e on the actual image and the state of the large member 14e on the target frame is less than the threshold The.

  The processing contents of the command unit 61 and the termination determination unit 62 in the first control unit 57c and the second control unit 57d are the same as those of the first control unit 57a and the second control unit 57b according to the above embodiment.

  The control device 50C according to Modification 6 controls the target robot such that the state of the target object changes according to the first CG moving image and the second CG moving image in accordance with the flowchart in FIG. 10 as in the above embodiment.

  Further, in the sixth modification, the third control unit 57e performs the process of the subroutine of step S6 shown in FIG. 10 according to the flowchart shown in FIG. 13 similarly to the above embodiment. However, the first control unit 57c and the second control unit 57d perform the processing of the subroutine of step S6 shown in FIG.

  FIG. 25 is a flowchart illustrating a flow of processing of the first control unit and the second control unit according to the sixth modification. As shown in FIG. 25, the flow of processing of the first control unit 57c and the second control unit 57d of Modification 6 is different from the flow chart shown in FIG. 13 in that step S90 is provided. Therefore, only step S90 will be described.

  In step S90, it is determined whether the deviation between the state of the large member 14e on the actual image and the state of the large member 14e on the target frame is less than a threshold Thj. If the deviation is equal to or larger than the threshold Thj (NO in step S90), the process ends. If the deviation is less than the threshold Thj (YES in step S90), steps S21 to S25 are performed.

  As described above, when the deviation between the state of the large member 14e on the actual image and the state of the large member 14e on the target frame exceeds the threshold Thj, the control device 50C controls only the robot 30e. The control device 50C controls each of the robots 30c to 30e when the deviation between the state of the large member 14e on the actual image and the state of the large member 14e on the target frame is less than the threshold Thj.

  For example, when the first CG moving image and the second CG moving image sequentially indicate the first to third scenes as described above, the robots 30c to 30e are controlled as follows. When the target frame is selected from the first scene, the robots 30c and 30d move until the imaging devices 21 and 22 move so that the processing target portion 15 of the large member 14e is at a fixed position on the image. It will be in a stopped state. Then, after the processing target portion 15 of the large member 14e is at the fixed position on the image, the first control unit 57c sets the target so that the state of the target on the real image approaches the state of the target on the target frame. Control the robot. As a result, the processing tools 14c and 14d process the processing target portion 15. Also during this time, the third control unit 57e controls the robot 30e such that the state of the large member 14e on the actual image approaches the state of the large member 14e on the target frame. However, in the first scene, since the state of the large member 14e is constant, the robot 30e hardly operates, and the states of the imaging devices 21 and 22 are substantially constant.

  When the target frame is selected from the second scene, the robot 30e is controlled so that the state of the large member 14e on the real image changes, and the imaging devices 21 and 22 move. The robots 30c and 30d are in a stopped state until the imaging devices 21 and 22 move so that the processing target portion 16 of the large member 14e is at a fixed position on the image.

  When the processing target portion 16 of the large member 14e becomes a fixed position on the image and a target frame is selected from the third scene, the first control unit 57c changes the state of the target on the real image to the target frame. The target robot is controlled so as to approach the state of the target object above. As a result, the processing tools 14c and 14d process the processing target portion 16. Also during this time, the third control unit 57e controls the robot 30e such that the state of the large member 14e on the actual image approaches the state of the large member 14e on the target frame. However, in the third scene, since the state of the large member 14e is constant, the robot 30e hardly operates, and the states of the imaging devices 21 and 22 are substantially constant.

  FIG. 26 is a schematic diagram illustrating another configuration example of the control system according to the sixth modification. As shown in FIG. 26, the robots 30c and 30d may be installed on a pedestal 33e included in the robot 30e. In this case, the robots 30c and 30d move integrally with the robot 30e.

<D-7. Modification 7>
The calculation unit 60 performs a known model predictive control (“Adachi,“ Basics of Model Predictive Control ”, Journal of the Robotics Society of Japan, July 2014, Vol. 32, No. 6, p. 9-12) (Non-Patent Document By performing 2)), control amounts of a plurality of degrees of freedom may be calculated.

  Specifically, the target frame selection unit 56 selects a plurality of frames included in the predicted horizon period in the teaching range as target frames. The calculation unit 60 performs control during the control horizon period so as to minimize the deviation between the state of the object on the target frame and the state of the object on the images captured by the imaging devices 21 and 22 during the predicted horizon period. Calculate the amount.

<D-8. Other Modifications>
In the above description, the male connector 2a and the female connector 2b are connected by moving the male connector 2a toward the female connector 2b. However, the male connector 2a may be placed on the stage 31b, and the female connector 2b may be gripped by the hand 31a. In this case, the male connector 2a and the female connector 2b are connected by moving the female connector 2b toward the male connector 2a.

  In the above description, the moving image storage unit 51 of each of the control devices 50, 50A, 50B, and 50C stores the CG moving image. However, external devices of the control devices 50, 50A, 50B, and 50C may store CG moving images.

  The moving image storage unit 51 may store, instead of or in addition to the CG moving image, the coordinates and the feature amounts of the feature points of each object extracted from each frame of the CG moving image. Thus, the processing of the image processing units 55 and 255 for each frame of the CG moving image can be omitted.

<E. Appendix>
As described above, the present embodiment and the modifications include the following disclosure.

(Configuration 1)
First to N-th robots (30a to 30e);
Imaging devices (21 to 24) for imaging the first to Nth objects (2a, 2b, 9a, 9b, 10c to 10f, 14c to 14e);
A control system (1, 1A, 1C) including a control device (50, 50A, 50B, 50C) for controlling the first to Nth robots (30a to 30e),
N is an integer of 2 or more;
The i-th robot (30a to 30e) changes the state of the i-th object,
i is an integer of 1 to N-1,
The N-th robot (30a to 30e) changes one state of the N-th object and the imaging device (21 to 24),
The other of the N-th object and the imaging devices (21 to 24) is installed at a fixed position,
The control device (50, 50A, 50B, 50C) acquires change information for each of the first to Nth objects,
The change information corresponding to the j-th object indicates a relationship between a control amount of the j-th robot and a change amount of a state of the j-th object on an image of the imaging device,
j is an integer of 1 to N;
The control device (50, 50A, 50B, 50C)
A first process of acquiring a real image captured by the imaging device (21 to 24);
A second process of selecting a target frame from a CG moving image showing the models of the first to Nth objects;
Performing a third process for controlling each of the first to Nth robots (30a to 30e) based on the actual image and the target frame;
In the third processing, the control device (50, 50A, 50B, 50C) changes a state of the j-th object on the real image based on the change information corresponding to the j-th object. The control amount of the j-th robot (30a to 30e) for approaching the state of the model of the j-th object on the target frame is calculated, and the j-th robot (30a to 30e) is calculated according to the calculated control amount. ), A control system (1, 1A, 1C).

(Configuration 2)
The CG moving image is created based on the design data of each of the first to Nth objects,
The CG moving image is obtained from a frame indicating a state in which the first object and at least one of the second to Nth objects are separated from each other. And up to the frame showing the joined state,
The control device (50A) may include, in the third processing, the first robot (30a to 30e) by an adjustment amount corresponding to a manufacturing error of the first object and at least one object with respect to the design data. The control system (1A) according to Configuration 1, wherein the control amount is adjusted and the first robot (30a to 30e) is controlled according to the adjusted control amount.

(Configuration 3)
The control system (1A) includes a force sensor (32) for detecting a force that one of the first object and the at least one object receives from the other,
The control device (50A, 50B) searches for the state of the first object where the force detected by the force sensor is minimal, and based on the searched state of the first object, The control system (1A) according to Configuration 2, wherein the control system determines the adjustment amount.

(Configuration 4)
The control system (1A) according to configuration 2, wherein the control device (50A, 50B) is connected to an input device (534) and acquires the adjustment amount from the input device (534).

(Configuration 5)
The CG moving image may include a first object and the at least one object from a frame indicating a state in which the first object and at least one of the second to Nth objects are separated from each other. And up to the frame showing the joined state,
The control system (50B) includes a force sensor (32) for detecting a force that one of the first object and the at least one object receives from the other,
The control device (50B)
Performing a first mode in which the third processing is repeated from the first processing;
In the third process in the first mode, the control amount of the first robot is adjusted to search for an adjustment amount at which the force detected by the force sensor is minimal,
When the control amount of the first robot is adjusted by the searched adjustment amount, a moving image captured by the imaging device is acquired as a reference moving image,
The control system according to Configuration 1, wherein a second mode in which the first processing, the fourth processing for selecting the target frame from the reference moving image instead of the CG moving image, and the third processing are performed is performed.

(Configuration 6)
First to Nth robots (30a) using an imaging device (21 to 24) for imaging the first to Nth objects (2a, 2b, 9a, 9b, 10c to 10f, 14c to 14e). To 30e), wherein:
N is an integer of 2 or more;
The i-th robot (30a to 30e) changes the state of the i-th object,
i is an integer of 1 to N-1,
The N-th robot (30a to 30e) changes one state of the N-th object and the imaging device,
The other of the N-th object and the imaging devices (21 to 24) is installed at a fixed position,
The imaging devices (21 to 24) are installed at fixed positions different from the first to Nth robots (30a to 30e),
The control method includes:
A first step of acquiring change information for each of the first to Nth robots (30a to 30e);
The change information corresponding to the j-th robot (30a to 30e) includes a control amount of the j-th robot (30a to 30e), a change amount of a state of the j-th object on an image of the imaging device, and Shows the relationship
j is an integer of 1 to N;
The control method includes:
A second step of acquiring a real image captured by the imaging device (21 to 24);
A third step of selecting a target frame from a CG video showing the models of the first to Nth objects;
A fourth step of controlling each of the first to Nth robots (30a to 30e) based on the actual image and the target frame,
The fourth step includes, based on the change information corresponding to the j-th robot (30a to 30e), changing the state of the j-th object on the real image to the j-th object on the target frame. A control method for calculating a control amount for approaching a state of an object, and controlling the j-th robot (30a to 30e) according to the calculated control amount.

(Configuration 7)
First to Nth robots (30a) using an imaging device (21 to 24) for imaging the first to Nth objects (2a, 2b, 9a, 9b, 10c to 10f, 14c to 14e). A program (550) for causing a computer to execute a control method for controlling the control method of (e) to (e).
N is an integer of 2 or more;
The i-th robot (30a to 30e) changes the state of the i-th object,
i is an integer of 1 to N-1,
The N-th robot (30a to 30e) changes one state of the N-th object and the imaging device,
The other of the N-th object and the imaging devices (21 to 24) is installed at a fixed position,
The imaging devices (21 to 24) are installed at fixed positions different from the first to Nth robots (30a to 30e),
The control method includes:
A first step of acquiring change information for each of the first to Nth robots (30a to 30e);
The change information corresponding to the j-th robot (30a to 30e) includes a control amount of the j-th robot (30a to 30e), a change amount of a state of the j-th object on an image of the imaging device, and Shows the relationship
j is an integer of 1 to N;
The control method includes:
A second step of acquiring a real image captured by the imaging device (21 to 24);
A third step of selecting a target frame from a CG video showing the models of the first to Nth objects;
A fourth step of controlling each of the first to Nth robots (30a to 30e) based on the actual image and the target frame,
The fourth step includes, based on the change information corresponding to the j-th robot (30a to 30e), changing the state of the j-th object on the real image to the j-th object on the target frame. A program (550) including a step of calculating a control amount for approaching an object state and controlling the j-th robot (30a to 30e) according to the calculated control amount.

  Although the embodiment of the present invention has been described, the embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  1, 1A, 1C control system, 2a male connector, 2a 'male connector model, 2b female connector, 2b' female connector model, 7 insertion hole, 8 notch, 9a bolt, 9b plate-like component, 10c, 10d screw, 10e, 10f cylindrical member, 11a, 11b, 12a, 12b screw hole, 13 bolt hole, 14c, 14d processing tool, 14e large member, 15, 16 part to be processed, 21-24 imaging device, 30a-30e robot, 31a, 31c to 31e hand, 31a 'hand model, 31b stage, 31b' stage model, 32 force sensor, 40a to 40e robot controller, 50, 50A, 50B, 50C control device, 51 moving image storage unit, 52 design data storage unit, 53 CG moving image generator, 54, 254 Teaching range selection Selection unit, 55, 255 image processing unit, 56, 256 target frame selection unit, 57a, 57c, 157a, 257a first control unit, 57b, 57d, 157b, 257b second control unit, 57e third control unit, 58, 258 change information generation unit, 59 change information storage unit, 60, 260 calculation unit, 61, 161, 261 command unit, 62 end determination unit, 63 offset amount determination unit, 64 reference moving image generation unit, 510 processor, 512 RAM, 514 Display controller, 516 system controller, 518 I / O controller, 520 hard disk, 522 camera interface, 522a, 522b image buffer, 524 input interface, 526 robot controller interface, 528 communication interface, 530 memory Over de interface, 532 display unit, 534 input unit, 536 memory card, 550 control program, 591 a first change information set 592 second change information set 593 third variation information set 594 fourth change information set.

Claims (7)

First to N-th robots,
An imaging device for imaging the first to Nth objects;
A control device for controlling the first to Nth robots,
N is an integer of 2 or more;
The i-th robot changes the state of the i-th object,
i is an integer of 1 to N-1,
The N-th robot changes one state of the N-th object and the imaging device,
The other of the N-th object and the imaging device is installed at a fixed position,
The control device acquires change information for each of the first to Nth objects,
The change information corresponding to the j-th object indicates a relationship between a control amount of the j-th robot and a change amount of a state of the j-th object on an image of the imaging device,
j is an integer of 1 to N;
The control device includes:
A first process of acquiring a real image captured by the imaging device;
A second process of selecting a target frame from a CG moving image showing the models of the first to Nth objects;
Performing a third process for controlling each of the first to Nth robots based on the actual image and the target frame;
The control device, in the third processing, based on the change information corresponding to the j-th object, changes the state of the j-th object on the real image to the j-th object on the target frame. A control system for calculating a control amount of the j-th robot for approaching a state of a model of an object, and controlling the j-th robot according to the calculated control amount. The CG moving image is created based on the design data of each of the first to Nth objects,
The CG moving image may include a first object and the at least one object from a frame indicating a state in which the first object and at least one of the second to Nth objects are separated from each other. And up to the frame showing the joined state,
The control device adjusts the control amount of the first robot by an adjustment amount according to a manufacturing error of the first object and the at least one object with respect to the design data in the third processing. The control system according to claim 1, wherein the first robot is controlled in accordance with a later control amount. The control system includes a force sensor that detects a force that one of the first object and the at least one object receives from the other,
The control device searches for the state of the first object where the force detected by the force sensor is minimal, and determines the adjustment amount based on the searched state of the first object. The control system according to claim 2.   The control system according to claim 2, wherein the control device is connected to an input device, and acquires the adjustment amount from the input device. The CG moving image may include a first object and the at least one object from a frame indicating a state in which the first object and at least one of the second to Nth objects are separated from each other. And up to the frame showing the joined state,
The control system includes a force sensor that detects a force that one of the first object and the at least one object receives from the other,
The control device includes:
Performing a first mode in which the third processing is repeated from the first processing;
In the third process in the first mode, the control amount of the first robot is adjusted to search for an adjustment amount at which the force detected by the force sensor is minimal,
When the control amount of the first robot is adjusted by the searched adjustment amount, a moving image captured by the imaging device is acquired as a reference moving image,
The control system according to claim 1, wherein a second mode in which the first processing, the fourth processing for selecting the target frame from the reference moving image instead of the CG moving image, and the third processing are repeated is performed. A control method for controlling first to Nth robots using an imaging device for imaging first to Nth objects,
N is an integer of 2 or more;
The i-th robot changes the state of the i-th object,
i is an integer of 1 to N-1,
The N-th robot changes one state of the N-th object and the imaging device,
The other of the N-th object and the imaging device is installed at a fixed position,
The control method includes:
A first step of acquiring change information for each of the first to Nth robots,
The change information corresponding to the j-th robot indicates a relationship between a control amount of the j-th robot and a change amount of a state of the j-th object on an image of the imaging device,
j is an integer of 1 to N;
The control method includes:
A second step of acquiring a real image captured by the imaging device;
A third step of selecting a target frame from a CG video showing the models of the first to Nth objects;
A fourth step of controlling each of the first to Nth robots based on the real image and the target frame,
The fourth step brings the state of the j-th object on the real image closer to the state of the j-th object on the target frame based on the change information corresponding to the j-th robot. And calculating the control amount for controlling the j-th robot in accordance with the calculated control amount. A program for causing a computer to execute a control method for controlling the first to Nth robots using an imaging device for imaging the first to Nth objects,
N is an integer of 2 or more;
The i-th robot changes the state of the i-th object,
i is an integer of 1 to N-1,
The N-th robot changes one state of the N-th object and the imaging device,
The other of the N-th object and the imaging device is installed at a fixed position,
The control method includes:
A first step of acquiring change information for each of the first to Nth robots,
The change information corresponding to the j-th robot indicates a relationship between a control amount of the j-th robot and a change amount of a state of the j-th object on an image of the imaging device,
j is an integer of 1 to N;
The control method includes:
A second step of acquiring a real image captured by the imaging device;
A third step of selecting a target frame from a CG video showing the models of the first to Nth objects;
A fourth step of controlling each of the first to Nth robots based on the real image and the target frame,
The fourth step brings the state of the j-th object on the real image closer to the state of the j-th object on the target frame based on the change information corresponding to the j-th robot. And calculating a control amount for controlling the j-th robot according to the calculated control amount.

Images