JP2020023024A - Control system, analyzer, and control method - Google Patents

Abstract

To provide a control system where an operator easily determines whether or not control of a robot using a visual servo is performed normally.SOLUTION: A control system includes: a robot for changing states of an object; an imaging device for imaging the object; a control part for controlling the robot so that a change of the object state in an actual image captured by the imaging device approximates a change of the object state on a reference moving image created in advance; and an analysis part for displaying onto a screen of a display device both a first waveform showing a temporal change of a feature amount extracted from the actual image and a second waveform showing a temporal change of a feature amount extracted from a frame of the reference moving image.SELECTED DRAWING: Figure 12

Description

  The present invention relates to a control system, an analysis device, and a control method for controlling a robot.

  "Atsushi Murata and three others," Development of high-precision positioning method using visual servo ", SEI Technical Review, July 2009, No. 175, p. 98-102 (Non-Patent Document 1) U.S. Pat. No. 6,064,056 discloses a technique for guiding a robot arm to a target position. The system disclosed in Non-Patent Document 1 captures an image of a connector by a camera attached to a robot arm, and controls the robot so that the image of the camera approaches a target image. Thereby, the terminal supported by the robot hand is inserted into the connector.

Atsushi Murata and 3 others, "Development of high-precision positioning method using visual servo", SEI Technical Review, July 2009, No. 175, p. 98-102

  4M changes can occur at the production site. A 4M change is a change in a man, a machine, a method, or a material. When the 4M change occurs, the robot cannot be normally controlled using the visual servo, and the state of the target object may not change as expected. For example, in the system described in Non-Patent Document 1, the terminal cannot be inserted into the connector.

  By visually checking the operation of the robot, the operator can determine whether or not the control of the robot using the visual servo is normally performed. However, when the operation speed of the robot is high, or when the distance to the robot is long due to environmental conditions such as a safety fence, it is difficult for the operator to visually check the operation of the robot. In this case, it is difficult for the operator to determine whether or not the control of the robot using the visual servo is normally performed.

  The present invention has been made in view of the above problems, and a purpose thereof is to provide a control system, an analysis device, and a control system that allow an operator to easily determine whether or not control of a robot using a visual servo is normally performed. The purpose is to provide a control method.

  According to an example of the present disclosure, a control system includes a robot for changing a state of a target, an imaging device for capturing an image of the target, a control unit, and an analysis unit. The control unit controls the robot such that a change in the state of the target on the real image captured by the imaging device approaches a change in the state of the target on the previously created reference moving image. The analysis unit displays, on the screen of the display device, both the first waveform indicating the time change of the feature extracted from the real image and the second waveform indicating the time change of the feature extracted from the frame of the reference moving image. Display.

  According to this disclosure, the operator can easily determine whether the control of the robot using the visual servo is normally performed by checking the first waveform and the second waveform displayed on the screen. Become. For example, when the deviation between the first waveform and the second waveform is large, the operator can determine that some abnormality has occurred in the control of the robot using the visual servo.

  In the above disclosure, the analysis unit adjusts the time axes of the first waveform and the second waveform. According to this disclosure, the time lag between the first waveform and the second waveform is corrected. As a result, it becomes easier to determine whether the control of the robot using the visual servo is normally performed.

  In the above disclosure, the analysis unit calculates, for each time on the time axis, a parameter value indicating a difference between the feature amounts of the first waveform and the second waveform, and a third value indicating a parameter value and a time change of the parameter value. At least one of the waveforms is displayed on the screen. According to this disclosure, the operator can easily check which time zone during the control of the robot the deviation between the first waveform and the second waveform is large by checking the parameter value or the third waveform. You can figure out.

  In the above disclosure, the analysis unit calculates, for each time on the time axis, a parameter value indicating a difference between the feature amounts of the first waveform and the second waveform. The analysis unit selects a time at which the parameter value becomes the maximum or a time at which the parameter value exceeds the threshold value as the designated time based on the calculated parameter value. The analysis unit causes the real image corresponding to the designated time and the frame at the designated time in the reference moving image to be simultaneously displayed on the screen.

  According to this disclosure, an operator can check an image when the difference between the feature amounts of the first waveform and the second waveform is large, and it is easy to grasp the cause of the abnormality of the robot.

  In the above disclosure, the analysis unit receives a specified time on the time axis, and simultaneously displays an actual image corresponding to the specified time and a frame at the specified time in the reference moving image on the screen. According to this disclosure, the operator designates the time when the difference between the feature amounts of the first waveform and the second waveform is large as the designated time, so that the difference between the feature amounts of the first waveform and the second waveform is specified. Is large, the cause of the robot abnormality can be easily grasped.

  In the above disclosure, the analysis unit simultaneously reproduces, on the screen, the video of a part of the period including the designated time in both the real moving image including the real images arranged in time series and the reference moving image.

  According to this disclosure, the operator can more easily understand the cause of the abnormality of the control of the robot using the visual servo.

  In the above disclosure, the analysis unit may include a fourth waveform indicating a temporal change in the movement amount of the robot from the reference position when the real image is captured, and a movement amount when the state of the target object changes according to the reference moving image. Is displayed on the screen.

  According to this disclosure, the operator can determine whether there is an abnormality in the robot by checking the deviation between the fourth waveform and the fifth waveform. As a result, the operator can easily understand the cause of the large deviation between the first waveform and the second waveform.

  In the above disclosure, the robot changes the state of the object with a plurality of degrees of freedom. The analysis unit receives an instruction to select one of the degrees of freedom. The movement amount is the amount that the robot has moved from the reference position along the direction of one degree of freedom.

  According to this disclosure, it is easy for an operator to determine which of the degrees of freedom of the robot has an abnormality.

  According to an example of the present disclosure, an analysis device used in the above control system includes an analysis unit.

  According to an example of the present disclosure, a control method controls a robot that changes a state of an object using an imaging device for imaging the object. The control method includes a first step and a second step. The first step is a step of controlling the robot such that a change in the state of the target on the real image captured by the imaging device approaches a change in the state of the target on the previously created reference moving image. . In the second step, the display device displays both a first waveform indicating a temporal change of a feature extracted from a frame of the real image and a second waveform indicating a temporal change of a feature extracted from a frame of the reference moving image. This is a step of displaying on the screen.

  According to these disclosures, the operator can easily determine whether or not the control of the robot using the visual servo is normally performed.

  According to the present invention, it is easy for an operator to determine whether or not control of a robot using a visual servo is normally performed.

It is a schematic diagram which shows the outline | summary of the control system which concerns on embodiment. It is a figure showing an outline of an analysis screen in an embodiment. FIG. 2 is a schematic diagram illustrating a hardware configuration of a control device included in the control system according to the embodiment. FIG. 2 is a block diagram illustrating a functional configuration of a control device according to the embodiment. It is a figure showing an example of a standard animation (the 1st standard animation or the 2nd standard animation). FIG. 5 is a diagram illustrating an example of a first table stored in a reference moving image storage unit according to the embodiment. FIG. 9 is a diagram illustrating an example of a third table stored in the actual moving image storage unit according to the embodiment. FIG. 6 is a diagram illustrating an example of a method for creating a first template. FIG. 4 is a block diagram illustrating a functional configuration of first to fourth control units. It is a figure explaining the generation method of the 1st change information set. FIG. 4 is a diagram illustrating a method of calculating a control amount by a calculation unit. FIG. 9 is a diagram illustrating a specific example of an analysis screen according to the embodiment. FIG. 3 is a diagram illustrating an actual waveform and a reference waveform before adjusting the time axis, and an actual waveform and a reference waveform after adjusting the time axis. FIG. 4 is a diagram illustrating an example of a feature amount. FIG. 9 is a diagram illustrating another example of the feature amount. 9 is a flowchart illustrating an example of a flow of a change information generation process performed by a change information generation unit. 17 is a flowchart illustrating a flow of processing of a subroutine of step S2 illustrated in FIG. 16. It is a flowchart which shows an example of the flow of the processing of the soldering control using a visual servo. 19 is a flowchart showing the flow of processing of a subroutine of step S44 shown in FIG. FIG. 3 is a diagram illustrating a relationship between a closest frame and a target frame. 19 is a flowchart showing the flow of processing of a subroutine of step S46 shown in FIG. 9 is a flowchart illustrating a flow of a process performed by an analysis unit. FIG. 14 is a diagram illustrating an example of an analysis screen according to a first modification. FIG. 14 is a diagram illustrating an example of a fifth table stored in a reference moving image storage unit according to Modification 1. FIG. 13 is a diagram illustrating an example of a sixth table stored in an actual moving image storage unit according to Modification 1. FIG. 14 is a diagram illustrating an example of an analysis screen according to a second modification. FIG. 14 is a diagram illustrating an example of an analysis screen according to a third modification. FIG. 17 is a diagram illustrating an example of a seventh table stored in an actual moving image storage unit according to Modification 3. FIG. 14 is a schematic diagram illustrating an object of a control system according to Modification 4.

  Hereinafter, embodiments according to the present invention will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are the same. Therefore, detailed description thereof will not be repeated.

<A. Application example>
First, an example of a scene to which the present invention is applied will be described with reference to FIG. FIG. 1 is a schematic diagram illustrating an outline of a control system according to an embodiment.

  The control system 1 according to the present embodiment is, for example, a control for soldering an electric wire 2c to a pad 2d of a substrate 3 using a soldering iron 2a and a solder feeder 2b in an industrial product production line or the like (hereinafter, referred to as a control system) , "Soldering control"). The control system 1 performs soldering control every time a new board 3 is transported, and solders the electric wires 2c to the pads 2d of the board 3.

  As shown in FIG. 1, the control system 1 includes imaging devices 21 and 22, robots 30a to 30d, robot controllers 40a to 40d, a control device 50, and a display device 60.

  The imaging devices 21 and 22 capture an image of a subject present in an imaging field of view and generate image data (hereinafter, referred to as “real images”). The imaging devices 21 and 22 are set at fixed positions different from those of the robots 30a to 30d. The imaging devices 21 and 22 are installed at different locations, and capture images of the soldering iron 2a, the solder feeder 2b, the electric wire 2c, and the pad 2d as subjects from different directions. The imaging devices 21 and 22 perform imaging according to a predetermined imaging cycle, and output a real image obtained by the imaging to the control device 50.

  The robot 30a is a mechanism for changing the state (here, position and posture) of the soldering iron 2a, and is, for example, a vertical articulated robot. The robot 30a has a hand 31a at its tip for supporting (holding) the soldering iron 2a, and changes the position and posture of the soldering iron 2a held by the hand 31a with six degrees of freedom. The six degrees of freedom include translational degrees of freedom in the X, Y, and Z directions, and rotational degrees of freedom in the pitch, yaw, and roll directions.

  The robot 30b is a mechanism for changing the state (here, position and orientation) of the solder feeder 2b, and is, for example, a vertical articulated robot. The robot 30b has a hand 31b at its tip for supporting (gripping) the solder feeder 2b, and changes the position and posture of the solder feeder 2b gripped by the hand 31b with six degrees of freedom.

  The robot 30c is a mechanism for changing the state (here, position and posture) of the electric wire 2c, and is, for example, a vertical articulated robot. The robot 30c has a hand 31c at its tip for supporting (gripping) the electric wire 2c, and changes the position and posture of the electric wire 2c held by the hand 31c with six degrees of freedom.

  The robot 30d is a mechanism for changing the state (the position and the posture here) of the pad 2d of the substrate 3, and is, for example, an XYθ stage. The robot 30d has a stage 31d for supporting (putting) the substrate 3 here, and changes the position and posture of the pad 2d of the substrate 3 placed on the stage 31d with three degrees of freedom. The three degrees of freedom include a translational degree of freedom in the X and Y directions and a rotational degree of freedom in a direction (θ direction) about an axis orthogonal to the XY plane.

  Each of the robots 30a to 30d has a plurality of servomotors, and drives the servomotors to change the state of the supported object. An encoder is provided corresponding to each of the plurality of servomotors. The encoder generates a pulse signal corresponding to a driving amount (rotation amount) of the corresponding servomotor. The pulse signal generated by the encoder is output to the corresponding robot controller and also to the control device 50.

  The robot controllers 40a to 40d are provided corresponding to the robots 30a to 30d, respectively. The robot controllers 40a to 40d control the operations of the robots 30a to 30d, respectively, according to control commands received from the control device 50.

  Each of the robot controllers 40a to 40c receives, from the control device 50, a control command for each of the translational degrees of freedom in the X, Y, and Z directions and the rotational degrees of freedom in the pitch, yaw, and roll directions. These X direction, Y direction, Z direction, pitch direction, yaw direction, and roll direction are indicated by the corresponding robot coordinate system.

  Each of the robot controllers 40a to 40c controls the corresponding robot such that the translation amount of the corresponding hand in the X, Y, and Z directions approaches the control command of the degree of freedom of translation in the X, Y, and Z directions. Feedback control. The translational movement amounts of the corresponding hand in the X, Y, and Z directions can be obtained by counting the number of pulses included in a pulse signal from an encoder provided in the corresponding robot.

  Each of the robot controllers 40a to 40c controls the corresponding robot such that the amount of rotational movement of the corresponding hand in the pitch, yaw, and roll directions approaches the control command of the rotational degrees of freedom in the pitch, yaw, and roll directions. Feedback control. The amount of rotational movement of the corresponding hand in the pitch, yaw, and roll directions is obtained by counting the number of pulses included in a pulse signal from an encoder provided in the corresponding robot.

  The robot controller 40 d receives from the control device 50 control commands for the degrees of freedom of translation and rotation in the X and Y directions. These X direction, Y direction and rotation direction are indicated by the coordinate system of the robot 30d.

  The robot controller 40d performs feedback control on the robot 30d such that the translation amounts of the stage 31d in the X and Y directions approach control commands for the degrees of freedom of translation in the X and Y directions, respectively. The translation amounts of the stage 31d in the X and Y directions can be obtained by counting the number of pulses included in a pulse signal from an encoder provided in the robot 30d.

  The robot controller 40d performs feedback control on the robot 30d such that the rotational movement amount of the stage 31d approaches the control command of the rotational degree of freedom. The rotational movement amount of the stage 31d is obtained by counting the number of pulses included in a pulse signal from an encoder provided in the robot 30d.

  The control device 50 controls the robots 30a to 30d via the robot controllers 40a to 40d, respectively.

  The control device 50 stores a first reference moving image and a second reference moving image showing samples of the soldering iron 2a, the solder feeder 2b, the electric wire 2c, and the pad 2d. The first reference moving image is a moving image when viewed from the position of the imaging device 21. The second reference moving image is a moving image when viewed from the position of the imaging device 22.

  The control device 50 controls the robots 30a to 30d using a visual servo as soldering control. The control device 50 controls the change of the state of the soldering iron 2a on the real image captured by the imaging devices 21 and 22 so as to approach the change of the state of the soldering iron 2a on the first and second reference moving images, respectively. , The robot 30a. Similarly, the control device 50 controls the change in the state of the solder feeder 2b on the actual image captured by the imaging devices 21 and 22 to approach the change in the state of the solder feeder 2b on the first and second reference moving images, respectively. Next, the robot 30b is controlled. The control device 50 controls the robot 30c so that the change in the state of the electric wire 2c on the real image captured by the imaging devices 21 and 22 approaches the change in the state of the electric wire 2c on the first and second reference moving images, respectively. Control. The control device 50 controls the robot 30d such that the change in the state of the pad 2d on the real image captured by the imaging devices 21 and 22 approaches the change in the state of the pad 2d on the first and second reference moving images, respectively. Control. Thereby, the states of the soldering iron 2a, the solder feeder 2b, the electric wire 2c, and the pad 2d change in accordance with the first reference moving image and the second reference moving image.

  Further, the control device 50 is an analysis device that displays, on the display device 60, an analysis screen for determining whether the control of the robots 30a to 30d using the visual servo has been normally performed after the soldering control is completed. Also works.

  FIG. 2 is a diagram illustrating an outline of an analysis screen according to the embodiment. The control device 50 includes a real waveform 621 indicating a temporal change of a feature value extracted from an actual image captured by the imaging device 21 and a reference waveform 622 indicating a temporal change of a feature value extracted from a frame of the first reference moving image. Are displayed on the analysis screen 61. Alternatively, the control device 50 may include a real waveform 621 indicating a temporal change of a feature value extracted from an actual image captured by the image capturing device 22 and a reference indicating a temporal change of a feature value extracted from a frame of the second reference moving image. Both the waveform 622 and the waveform 622 are displayed on the analysis screen 61. The actual waveform 621 and the reference waveform 622 are displayed in regions 601 and 602 of the analysis screen 61, respectively.

  The feature amount is, for example, coordinates (pixel coordinates) on the image of the object (the soldering iron 2a, the solder feeder 2b, the electric wire 2c, or the pad 2d).

  When the control of the robots 30a to 30d using the visual servo is normally performed, the change in the state of the target on the real image captured by the imaging devices 21 and 22 is caused by the change in the state of the first and second reference moving images. It follows each change in the state of the object. Therefore, the actual waveform 621 approximates the reference waveform 622. Therefore, the operator checks the actual waveform 621 and the reference waveform 622 displayed on the analysis screen 61 to determine whether or not the control of the robots 30a to 30d using the visual servo is normally performed. It will be easier. For example, when the deviation between the actual waveform 621 and the reference waveform 622 is large, the operator can determine that some abnormality has occurred in the control of the robots 30a to 30d using the visual servo.

<B. Specific example>
Next, a specific example of the control system according to the embodiment will be described.

<B-1. Hardware configuration of control device>
FIG. 3 is a schematic diagram illustrating a hardware configuration of a control device included in the control system according to the embodiment. As shown in FIG. 3, the control device 50 has a structure according to a computer architecture, and executes various programs described later by executing a program installed in advance by a processor.

  More specifically, the control device 50 includes a processor 510 such as a central processing unit (CPU) or a micro-processing unit (MPU), a random access memory (RAM) 512, a display controller 514, a system controller 516, It includes an I / O (Input Output) controller 518, a hard disk 520, a camera interface 522, an input interface 524, a robot controller interface 526, a communication interface 528, and a memory card interface 530. These units are connected to each other so as to enable data communication with the system controller 516 as the center.

  The processor 510 exchanges programs (codes) and the like with the system controller 516, and executes the programs (codes) and the like in a predetermined order, thereby realizing the intended arithmetic processing.

  The system controller 516 is connected to the processor 510, the RAM 512, the display controller 514, and the I / O controller 518 via buses, respectively, exchanges data with each unit, and performs processing of the entire control device 50. Govern.

  The RAM 512 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory), and includes programs read from the hard disk 520, images (image data) acquired by the imaging devices 21 and 22, The processing result for the image, work data, and the like are stored.

  The display controller 514 is connected to the display device 60, and outputs a signal for displaying various information to the display device 60 according to an internal command from the system controller 516.

  The I / O controller 518 controls data exchange between a recording medium connected to the control device 50 and an external device. More specifically, the I / O controller 518 is connected to the hard disk 520, the camera interface 522, the input interface 524, the robot controller interface 526, the communication interface 528, and the memory card interface 530.

  The hard disk 520 is typically a nonvolatile magnetic storage device, and stores various information in addition to the control program 550 executed by the processor 510. The control program 550 installed on the hard disk 520 is distributed while being stored in a memory card 536 or the like. Instead of the hard disk 520, a semiconductor storage device such as a flash memory or an optical storage device such as a DVD-RAM (Digital Versatile Disk Random Access Memory) may be adopted.

  The camera interface 522 corresponds to an input unit that receives image data from the imaging devices 21 and 22, and mediates data transmission between the processor 510 and the imaging devices 21 and 22. The camera interface 522 includes image buffers 522a and 522b for temporarily storing image data from the imaging devices 21 and 22, respectively. Although a single image buffer that can be shared may be provided for a plurality of imaging devices, it is preferable to independently arrange a plurality of image buffers in association with each imaging device in order to speed up processing.

  The input interface 524 mediates data transmission between the processor 510 and an input device 534 such as a keyboard, a mouse, a touch panel, and a dedicated console.

  The robot controller interface 526 mediates data transmission between the processor 510 and the robot controllers 40a, 40b.

  The communication interface 528 mediates data transmission between the processor 510 and another personal computer or server device (not shown). The communication interface 528 is typically made of Ethernet (registered trademark), USB (Universal Serial Bus), or the like.

  The memory card interface 530 mediates data transmission between the processor 510 and a memory card 536 as a recording medium. The memory card 536 circulates in a state where a control program 550 executed by the control device 50 and the like are stored, and the memory card interface 530 reads the control program 550 from the memory card 536. The memory card 536 includes a general-purpose semiconductor storage device such as SD (Secure Digital), a magnetic recording medium such as a flexible disk, an optical recording medium such as a CD-ROM (Compact Disk-Read Only Memory), and the like. Consists of Alternatively, a program downloaded from a distribution server or the like may be installed in the control device 50 via the communication interface 528.

  When using a computer having a structure according to the general-purpose computer architecture as described above, an OS for providing basic functions of the computer in addition to an application for providing functions according to the present embodiment (Operating System) may be installed. In this case, the control program according to the present embodiment executes processing by calling necessary modules in a predetermined order and / or timing among program modules provided as a part of the OS. Is also good.

  Further, the control program according to the present embodiment may be provided by being incorporated in a part of another program. Even in such a case, the program itself does not include a module included in another program to be combined as described above, and the process is executed in cooperation with the other program. That is, the control program according to the present embodiment may be a form incorporated in such another program.

  Alternatively, part or all of the functions provided by executing the control program may be implemented as a dedicated hardware circuit.

<B-2. Functional configuration of control device>
FIG. 4 is a block diagram illustrating a functional configuration of the control device according to the embodiment. As shown in FIG. 4, the control device 50 includes a reference moving image storage unit 51, a real image acquisition unit 56, a real moving image storage unit 57, a robot control unit 58, and an analysis unit 59.

<B-2-1. Reference video storage unit>
The reference moving image storage unit 51 includes the hard disk 520 and the RAM 512 shown in FIG. 3, and stores the first reference moving image and the second reference moving image. The first reference moving image and the second reference moving image show how the electric wire 2c is soldered to the pad 2d using the soldering iron 2a and the solder feeder 2b. The first reference moving image and the second reference moving image are created by capturing an image of the robot 30a to 30d changing the state of the sample of the object (the soldering iron 2a, the solder feeder 2b, the electric wire 2c, and the pad 2d). . Alternatively, the first reference moving image and the second reference moving image may be created by imaging a state in which the state of the sample of the target object is changed by an operator's hand.

  Each of the first reference moving image and the second reference moving image includes a plurality of frames (hereinafter, referred to as M (M is an integer of 2 or more)) arranged in chronological order. The k-th frame (k is an integer from 1 to M) of the first reference moving image and the k-th frame of the second reference moving image are a certain state of the soldering iron 2a, the solder feeder 2b, the electric wire 2c, and the pad. It is an image when 2d is viewed simultaneously from different directions.

  FIG. 5 is a diagram illustrating an example of a reference moving image (a first reference moving image or a second reference moving image). FIG. 5 shows frames 71 to 77 of the reference moving image.

  The frame 71 is a frame when the state of the pad 2d reaches a desired state. The frame 72 is a frame when the electric wire 2c reaches the pad 2d. The frame 73 is a frame when the tip of the soldering iron 2a contacts the pad 2d. The frame 74 is a frame when the tip of the solder feeder 2b contacts the tip of the soldering iron 2a. The frame 75 is a frame when the solder supplied from the solder feeder 2b and melted (hereinafter, referred to as “molten solder 2e”) has a desired size. The frame 76 is a frame when the solder feeder 2b is moved away from the soldering iron 2a. The frame 77 is a frame when the soldering iron 2a is moved away from the pad 2d. By such a series of operations, the electric wire 2c is soldered to the pad 2d.

  As shown in FIG. 4, the reference moving image storage unit 51 stores a first table that associates each frame of the first reference moving image with a feature amount extracted from the frame, and a first table that associates each frame of the second reference moving image with the corresponding frame. And a second table that associates the extracted feature amount with the extracted feature amount. The feature amount is, for example, coordinates (pixel coordinates) on the image of the center of gravity of each object.

  FIG. 6 is a diagram illustrating an example of a first table stored in the reference moving image storage unit according to the embodiment. In the first table of the example shown in FIG. 6, for each frame, a frame number, an image file name of the frame, a time of the frame, and a plurality of types of feature amounts are associated with each other. The second table also has the same data structure as the first table shown in FIG.

  The feature amounts included in the first table and the second table are calculated by an image processing unit 53 described later.

<B-2-2. Actual Image Acquisition Unit>
The real image acquisition unit 56 acquires a real image from the imaging devices 21 and 22 for each imaging cycle. Further, the real image obtaining unit 56 generates a first real moving image in which real images obtained from the imaging device 21 are arranged in a time series during one soldering control, and generates a first real moving image from the imaging device 22 during the soldering control. A second real moving image in which the obtained real images are arranged in time series is generated. The real image acquisition unit 56 stores the generated first real moving image and the second real moving image in the real moving image storage unit 57.

  A part of the real image acquisition unit 56 is configured by the camera interface 522. A part of the actual image acquisition unit 56 is realized by the processor 510 illustrated in FIG.

<B-2-3. Real movie storage unit>
The real moving image storage unit 57 includes the hard disk 520 and the RAM 512 shown in FIG. 3, and stores the first real moving image and the second real moving image for each soldering control. As described above, the first real moving image and the second real moving image are generated by the real image acquiring unit 56.

  Further, the real moving image storage unit 57 includes a third table for associating each of the plurality of real images forming the first real moving image with the feature amounts extracted from the real image, and a plurality of real images forming the second real moving image. A fourth table for associating each of the images with a feature amount extracted from the actual image is stored. The feature amount is the coordinates (pixel coordinates) on the image of the center of gravity of each object.

  FIG. 7 is a diagram illustrating an example of a third table stored in the actual moving image storage unit according to the embodiment. In the third table of the example shown in FIG. 7, the image file name for identifying the real image, the imaging time of the real image, and a plurality of types of feature amounts extracted from the real image are associated with each other. The fourth table also has the same data structure as the third table shown in FIG.

  Note that the feature amounts included in the first and second tables stored in the reference moving image storage unit 51 and the feature amounts included in the third and fourth tables stored in the actual moving image storage unit 57 are the same type. The feature amounts included in the third table and the fourth table are calculated by an image processing unit 53 described later.

  The actual moving image storage unit 57 stores a third table and a fourth table for each soldering control. In one soldering control, the electric wire 2c is soldered to the pad 2d. Therefore, the real moving image storage unit 57 stores the pad identification number for identifying the pad 2d in association with the first real moving image, the second real moving image, the third table, and the fourth table. FIG. 7 shows a third table corresponding to the pad identification number “000001”.

<B-2-4. Robot controller>
The robot control unit 58 controls the robots 30a to 30d using a visual servo. The robot control unit 58 includes a teaching range selection unit 52, an image processing unit 53, a target frame selection unit 54, a first control unit 55a, a second control unit 55b, a third control unit 55c, and a fourth control unit. 55d. The teaching range selection unit 52, the image processing unit 53, and the target frame selection unit 54 are realized by the processor 510 shown in FIG.

<B-2-5. Teaching range selector>
The teaching range selection unit 52 selects, for each target object, a teaching range serving as a sample of the target object from the first reference moving image and the second reference moving image. The object is any of the soldering iron 2a, the solder feeder 2b, the electric wire 2c, and the pad 2d.

  The teaching range selection unit 52 displays a screen prompting the user to select a teaching range on the display device 60. The operator confirms each frame of the first reference moving image and the second reference moving image, and operates the input device 534 (see FIG. 3) to start the first frame of a series of frames in which the object is performing a desired operation. And the last frame. The teaching range selection unit 52 selects a designated frame from the first frame to the last frame as the teaching range.

  For example, for the reference moving image shown in FIG. 5, the teaching range selection unit 52 selects all the frames (that is, the first to Mth frames) as the teaching ranges of the soldering iron 2a and the solder feeder 2b. I do. The teaching range selection unit 52 selects a frame from the first frame to a frame where a part of the electric wire 2c starts to be cut off (a frame slightly before the frame 75) as a teaching range of the electric wire 2c. The teaching range selection unit 52 selects, as the teaching range of the pad 2d, a range from the first frame to a frame (a frame slightly earlier than the frame 72) in which a part of the pad 2d starts to be lost.

<B-2-6. Image processing section>
The image processing unit 53 performs image processing on the target image, and detects an object from the target image using template matching. In the basic processing of template matching, a template which is data representing the image feature of the target object is prepared in advance, and the degree of matching of the image feature between the target image and the template is evaluated. This is a process for detecting the position, posture, shape, and size of the image.

  The target images on which the image processing unit 53 performs the image processing are the frames of the first reference moving image, the frames of the second reference moving image, and the real images captured by the imaging devices 21 and 22.

  The image processing unit 53 creates a first template and a second template for each object as advance preparation. The object is any of the soldering iron 2a, the solder feeder 2b, the electric wire 2c, the pad 2d, and the molten solder 2e. The first template is created based on a frame selected from the first reference moving image. The second template is created based on a frame selected from the second reference moving image.

  FIG. 8 is a diagram illustrating an example of a method for creating the first template. As illustrated in FIG. 8, the image processing unit 53 causes the display device 60 to display a frame selected by the operator from the first reference moving image. The operator may select a frame in which the entire object is shown.

  The image processing unit 53 receives an area designation of the target object on the frame displayed on the display device 60. In the example shown in FIG. 8, the operator operates the input device 534 (see FIG. 3) to change the line 4a surrounding the soldering iron 2a, the line 4b surrounding the solder feeder 2b, and the line 4c surrounding the electric wire 2c. , A line 4d surrounding the pad 2d. The image processing unit 53 specifies an area surrounded by the lines 4a to 4d as image areas of the soldering iron 2a, the solder feeder 2b, the electric wire 2c, and the pad 2d.

  The image processing unit 53 creates data representing the image characteristics of the image area surrounded by the line 4a as the first template of the soldering iron 2a. Similarly, the image processing unit 53 creates data representing the image characteristics of the image area surrounded by the line 4b as the first template of the solder feeder 2b. The image processing unit 53 creates data representing the image characteristics of the image area surrounded by the line 4c as the first template of the electric wire 2c. The image processing unit 53 creates data representing the image characteristics of the image area surrounded by the line 4d as the first template of the pad 2d. The image processing unit 53 creates a first template of the molten solder 2e by the same method.

  In a similar manner, the image processing unit 53 causes the display device 60 to display the frame selected from the second reference moving image, and creates a second template for each object.

  The image processing unit 53 detects the target from the frame of the first reference moving image or the actual image captured by the imaging device 21 by performing template matching using the first template for each target.

  The image processing unit 53 detects the target from the frame of the second reference moving image or the actual image captured by the imaging device 22 by performing template matching using the second template for each target.

  The image processing unit 53 outputs, for each target, coordinates on the image of each feature point of the target detected from the target image.

  Further, the image processing unit 53 extracts a feature amount from the actual image captured by the imaging devices 21 and 22 and each frame of the first reference moving image and the second reference moving image. The feature amount is the coordinates (pixel coordinates) on the image of the center of gravity of each object.

  The first table stored in the reference moving image storage unit 51 includes a feature amount extracted from each frame of the first reference moving image. The second table stored in the reference moving image storage unit 51 includes feature amounts extracted from each frame of the second reference moving image. The third table stored in the real moving image storage unit 57 includes the feature amount extracted from the real image captured by the imaging device 21. The fourth table stored in the actual moving image storage unit 57 includes feature amounts extracted from the actual image captured by the imaging device 22.

<B-2-7. Target frame selection section>
The target frame selection unit 54 selects a target frame from the first reference moving image and the second reference moving image. However, when the k-th frame of the first reference moving image is selected as the target frame, the target frame selecting unit 54 selects the k-th frame of the second reference moving image as the target frame. A specific example of the target frame selection method will be described later.

<B-2-8. First to fourth control units>
The first control unit 55a controls the robot 30a via the robot controller 40a and changes the state of the soldering iron 2a. The second control unit 55b controls the robot 30b via the robot controller 40b to change the state of the solder feeder 2b. The third control unit 55c controls the robot 30c via the robot controller 40c to change the state of the electric wire 2c. The fourth control unit 55d controls the robot 30d via the robot controller 40d to change the state of the pad 2d.

  FIG. 9 is a block diagram illustrating a functional configuration of the first to fourth control units. As shown in FIG. 9, each of the first control unit 55a, the second control unit 55b, the third control unit 55c, and the fourth control unit 55d includes a change information generation unit 551, a change information storage unit 552, A unit 553, a command unit 554, and an end determination unit 555. The change information storage unit 552 includes the hard disk 520 and the RAM 512 shown in FIG. The change information generation unit 551, the calculation unit 553, the command unit 554, and the end determination unit 555 are realized by the processor 510 illustrated in FIG.

<B-2-9. Change information generator>
The change information generation unit 551 performs a first change indicating a relationship between the control amount of the target robot and the change amount of the state of the target object on the real image captured by the imaging device 21 for each of the plurality of degrees of freedom. Generate information. The change information generation unit 551 stores, in the change information storage unit 552, a first change information set 5521 including a plurality of first change information generated for a plurality of degrees of freedom.

  Further, the change information generation unit 551 shows a relationship between the control amount of the target robot and the change amount of the state of the target object on the real image captured by the imaging device 22 for each of the plurality of degrees of freedom. 2 Change information is generated. The change information generation unit 551 stores, in the change information storage unit 552, a second change information set 5522 including a plurality of pieces of second change information generated for a plurality of degrees of freedom.

  The object is the soldering iron 2a in the first control unit 55a, the solder feeder in the second control unit 55b, the electric wire 2c in the third control unit 55c, and the pad 2d in the fourth control unit 55d. The target robot is the robot 30a in the first controller 55a, the robot 30b in the second controller 55b, the robot 30c in the third controller 55c, and the robot 30d in the fourth controller 55d. The plurality of degrees of freedom are six degrees of freedom in the first control unit 55a, the second control unit 55b, and the third control unit 55c, and three degrees of freedom in the fourth control unit 55d.

  In the present embodiment, the first change information and the second change information indicate the amount of change in the state of the target on the image when the target robot is controlled by the unit control amount. Specifically, the first change information and the second change information include an object on the image before controlling the target robot by the unit control amount, and an object on the image after controlling the target robot by the unit control amount. Here is a mapping that converts to

  The amount of change in the state of the target on the image when the target robot is controlled by the unit control amount depends on the state of the target in the real space. Therefore, the change information generating unit 551 generates the first change information set 5521 for each frame in the teaching range of the first reference moving image. Further, the change information generating unit 551 generates a second change information set 5522 for each frame in the teaching range of the second reference moving image.

  The process of generating and storing the first change information set 5521 and the second change information set 5522 by the change information generation unit 551 is executed as advance preparation.

  A method for generating the first change information set 5521 will be described with reference to FIG. Since the method of generating the second change information set 5522 is the same, the description of these methods is omitted.

  FIG. 10 is a diagram illustrating a method of generating the first change information set. FIG. 10A shows the k-th frame 70 of the first reference moving image. The state (here, position and orientation) of the object corresponding to the k-th frame 70 in the real space is set as the reference state. FIG. 10B illustrates an image 80a captured by the imaging device 21 after the target object has been translated from the reference state by the unit control amount in the degree of translational freedom in the Y direction. FIG. 10C illustrates an image 80b captured by the imaging device 21 after the target object has been translated from the reference state by the unit control amount in the degree of translation in the X direction. FIG. 10D illustrates an image 80c captured by the imaging device 21 after the target object has been translated from the reference state by the unit control amount in the degree of translational freedom in the Z direction. FIG. 10E illustrates an image 80d captured by the imaging device 21 after the target object is rotationally moved by the unit control amount to the rotational degree of freedom in the pitch direction from the reference state. FIG. 10F illustrates an image 80e captured by the image capturing apparatus 21 after the target object is rotationally moved from the reference state by the unit control amount to the rotational degree of freedom in the yaw direction. FIG. 10 (g) shows an image 80f captured by the imaging device 21 after rotating the object by the unit control amount to the degree of freedom of rotation in the roll direction from the reference state.

  The change information generation unit 551 acquires, from the image processing unit 53, the coordinates on the image of each feature point of the target object extracted from each of the frame 70 and the images 80a to 80f.

  The change information generation unit 551 outputs information indicating a mapping that converts the coordinates of the feature points 5a ′ to 5g ′ of the object extracted from the frame 70 into the coordinates of the feature points 5a to 5g extracted from the image 80a, respectively. It is generated as first change information corresponding to the degree of freedom of translation in the Y direction.

  Similarly, the change information generation unit 551 converts information indicating a mapping that transforms the coordinates of the feature points 5a ′ to 5g ′ into the coordinates of the feature points 5a to 5g extracted from the image 80b, according to the degree of freedom in translation in the X direction. Is generated as first change information. The change information generating unit 551 converts the information indicating the mapping that converts the coordinates of the feature points 5a ′ to 5g ′ into the coordinates of the feature points 5a to 5g extracted from the image 80c into the first information corresponding to the translation degree in the Z direction. Generated as change information. The change information generating unit 551 converts the information indicating the mapping that converts the coordinates of the feature points 5a ′ to 5g ′ into the coordinates of the feature points 5a to 5g extracted from the image 80d into the first information corresponding to the rotation degree in the pitch direction. Generated as change information. The change information generating unit 551 converts the information indicating the mapping that converts the coordinates of the feature points 5a ′ to 5g ′ into the coordinates of the feature points 5a to 5g extracted from the image 80e into the first information corresponding to the rotational degree of freedom in the yaw direction. Generated as change information. The change information generation unit 551 converts the information indicating the mapping that converts the coordinates of the feature points 5a ′ to 5g ′ into the coordinates of the feature points 5a to 5g extracted from the image 80f into the first information corresponding to the rotational degree of freedom in the roll direction. Generated as change information. In this way, the change information generating unit 551 generates the first change information set 5521 corresponding to the k-th frame 70 of the first reference moving image.

  The change information generating unit 551 generates the first change information set 5521 corresponding to the remaining frames of the teaching range in the first reference moving image by the same method.

<B-2-10. Calculation part>
The calculation unit 553 includes a plurality of degrees of freedom for bringing the state of the target on the real image captured by the imaging devices 21 and 22 closer to the state of the target on target frames of the first reference moving image and the second reference moving image, respectively. Is calculated.

  The calculation unit 553 acquires the first change information set 5521 and the second change information set 5522 corresponding to the target frame from the change information storage unit 552, and stores the acquired first change information set 5521 and second change information set 5522 in the acquired first change information set 5521 and second change information set 5522. The control amount is calculated based on the control amount. As described above, the first change information and the second change information include an object on the image before controlling the target robot by the unit control amount and an object on the image after controlling the target robot by the unit control amount. Here is a mapping that converts to Therefore, the calculating unit 553 acquires from the image processing unit 53 the coordinates of the feature points of the target object extracted from the real image on the image and the coordinates of the feature points of the target object extracted from the target frame on the image. . The calculation unit 553 calculates a control amount of each of a plurality of degrees of freedom for mapping the target on the real image to the target on the target frame based on the first change information and the second change information.

  With reference to FIG. 11, a method of calculating a control amount by the calculation unit 553 of the first control unit 55a to the third control unit 55c will be described. Note that the calculation method of the control amount by the calculation unit 553 of the fourth control unit 55d is the same as that of the fourth control unit 55d, and thus the description is omitted.

  FIG. 11 is a diagram illustrating a method of calculating the control amount by the calculation unit. The calculation unit 553 acquires from the image processing unit 53 the coordinates of the feature points 5a 'to 5g' of the target object extracted from the target frame of the first reference moving image on the image. Further, the calculating unit 553 acquires, from the image processing unit 53, the coordinates on the image of the feature points 5a to 5g of the target object extracted from the real image obtained by the imaging of the imaging device 21. The number of feature points is not limited to seven.

  The calculation unit 553 calculates the first difference vectors 6a to 6g of each feature point. The first difference vectors 6a to 6g are vectors having feature points 5a to 5g as starting points and feature points 5a 'to 5g' as end points, respectively. The calculation unit 553 calculates the average x component Δx1 and y component Δy1 of the first difference vectors 6a to 6g. The x component and the y component are indicated by the coordinate system of the image.

  In a similar manner, the calculation unit 553 calculates the average of the second difference vector between the feature point extracted from the real image obtained by the imaging of the imaging device 22 and the feature point extracted from the target frame of the second reference moving image. The x component Δx2 and the y component Δy2 are calculated.

  First, the calculation unit 553 calculates the three translational degrees of freedom such that the average x component Δx1 and y component Δy1 of the first difference vector and the average x component Δx2 and y component Δy2y of the second difference vector become zero. Is calculated. Specifically, the calculation unit 553 uses Δx1, Δy1, Δx2, and Δy2, the first change information set 5521, and the second change information set 5522 to calculate the X, Y, and Z directions of the target robot hand. The control amount of each translational degree of freedom is calculated.

  When the hand of the target robot translates in any of the degrees of freedom of translation in the X direction, the Y direction, and the Z direction, the target object translates in a certain direction on an image obtained by imaging by the imaging devices 21 and 22. . Therefore, the first change information corresponding to the translation degree of freedom in the first change information set 5521 indicates a mapping for converting an arbitrary point on the image into a point translated in a certain direction. Similarly, the second change information corresponding to the degree of freedom of translation in the second change information set 5522 indicates a mapping that converts an arbitrary point on the image into a point translated in a certain direction.

  Here, the first change information corresponding to the translational degree of freedom in the X direction of the first change information set 5521 corresponding to the target frame indicates a mapping that converts a point (x, y) into a point (x + dX1_1, y + dY1_1). And The first change information corresponding to the degree of freedom of translation in the Y direction indicates a mapping that converts an arbitrary point (x, y) on the image into a point (x + dX1_2, y + dY1_2). It is assumed that the first change information corresponding to the degree of freedom of translation in the Z direction indicates a mapping that converts an arbitrary point (x, y) on the image into a point (x + dX1_3, y + dY1_3).

  Further, the second change information corresponding to the degree of freedom of translation in the X direction of the second change information set 5522 corresponding to the target frame converts any point (x, y) on the image into a point (x + dX2_1, y + dY2_1). Are shown. The second change information corresponding to the degree of freedom of translation in the Y direction indicates a mapping that converts an arbitrary point (x, y) on the image into a point (x + dX2_2, y + dY2_2). The second change information corresponding to the translation degree of freedom in the Z direction indicates a mapping that converts an arbitrary point (x, y) on the image into a point (x + dX2_3, y + dY2_3).

  At this time, the calculation unit 553 calculates coefficients a1, a2, and a3 by solving the following four linear equations (1) to (4).

a1 × dX1_1 + a2 × dX1_2 + a3 × dX1_3 = Δx1 (1)
a1 × dY1_1 + a2 × dY1_2 + a3 × dY1_3 = Δy1 (2)
a1 × dX2_1 + a2 × dX2_2 + a3 × dX2_3 = Δx2 (3)
a1 × dY2_1 + a2 × dY2_2 + a3 × dY2_3 = Δy2 (4).

  The first change information and the second change information indicate the amount of change in the state of the target on the image when the target robot is controlled by the unit control amount. Therefore, the calculation unit 553 calculates the control amount of the translation freedom in the X direction as a1 times the unit control amount, the control amount of the translation freedom in the Y direction as a2 times the unit control amount, and the control amount of the translation freedom in the Z direction. Is a3 times the unit control amount. The control amounts of these translational degrees of freedom are control amounts for bringing the state of the target on the real image closer to the state of the target on the target frame by Δx1, Δy1, Δx2, and Δy2y.

  Next, the calculation unit 553 calculates a control amount of each of the three rotational degrees of freedom. The calculating unit 553 subtracts the average x component Δx1 and y component Δy1 from the first difference vector of each feature point. Similarly, the calculation unit 553 subtracts the above average x component Δx2 and y component Δy2 from the second difference vector of each feature point. The calculation unit 553 calculates a control amount of three rotational degrees of freedom in which the residual of the first difference vector and the second difference vector of each feature point is closest to 0, using a search algorithm such as a hill-climbing method, for example. .

  Specifically, the calculation unit 553 starts a search algorithm with a solution in which the control amounts of the rotational degrees of freedom in the pitch direction, the yaw direction, and the roll direction are 0 as the current solution. The calculation unit 553 simulates a change in the residual of the first difference vector and the second difference vector of each feature point when the target robot is controlled according to each of a plurality of solutions near the current solution. The calculation unit 553 determines from the simulation result that, when there is a nearby solution in which the residual of the first difference vector and the second difference vector of each feature point is closer to 0 than the current solution, the nearby solution is set as the current solution. replace. The calculation unit 553 repeats this process to search for a solution in which the residual of the first difference vector and the second difference vector has an extreme value.

<B-2-11. Command part>
The command unit 554 generates a control command for moving the target robot by the control amount calculated by the calculation unit 553, and outputs the generated control command to the target robot controller. The target robot controllers of the first control unit 55a, the second control unit 55b, the third control unit 55c, and the fourth control unit 55d are the robot controllers 40a to 40d, respectively.

<B-2-12. End determination section>
The end determination unit 555 calculates a deviation between the state of the target on the real image and the state of the target on the last frame of the teaching range, and when the calculated deviation is smaller than a predetermined threshold, the target robot Is determined to end. When determining to end the control of the target robot, the end determination unit 555 outputs an end notification.

  The deviation is, for example, an average of distances between corresponding feature points of the object extracted from the real image and the final frame.

  The threshold is set according to the accuracy required for the state of the target. The threshold is the threshold Th in the first control unit 55a, the threshold Thb in the second control unit 55b, the threshold Thc in the third control unit 55c, and the threshold Thd in the fourth control unit 55d. The thresholds Tha to Thd may be the same or different from each other.

<B-2-13. Analysis section>
When an analysis instruction is input to the input device 534 (see FIG. 3), the analysis unit 59 displays an analysis screen for determining whether or not the control of the robots 30a to 30d using the visual servo has been normally performed. Display on the device 60.

  The analysis unit 59 receives the input of the pad identification number, and reads the first real moving image, the second real moving image, the third table, and the fourth table corresponding to the input pad identification number from the real moving image storage unit 57. The analysis unit 59 displays an analysis screen based on the read first real moving image, the second real moving image, the third table, and the fourth table on the display device 60.

  FIG. 12 is a diagram illustrating a specific example of the analysis screen according to the embodiment. The analysis screen 62 of the example illustrated in FIG. 12 includes areas 601 to 606 and buttons 607 to 612.

  When the analysis instruction is input, the analysis unit 59 displays a first time change of a feature amount extracted from the real image captured by the imaging device 21 based on the third table read from the real moving image storage unit 57. An actual waveform is created as an actual waveform 621. The analysis unit 59 displays the created real waveform 621 in the area 601. Further, the analysis unit 59 creates, as the reference waveform 622, a first reference waveform indicating a temporal change of the feature amount extracted from each frame of the first reference moving image based on the first table stored in the reference moving image storage unit 51. Then, the created reference waveform 622 is displayed in the area 602. At this time, the analysis unit 59 creates an actual waveform 621 and a reference waveform 622 corresponding to one of the plurality of types of feature amounts. In the example shown in FIG. 12, the actual waveform 621 and the reference waveform 622 of the feature amount “pixel position (y coordinate) of the soldering iron” are displayed.

  When an analysis instruction is input, the analysis unit 59 displays a mark 623 specifying a time on the time axis of the real waveform 621 so as to overlap the real waveform 621. The analysis unit 59 displays the value of the feature amount corresponding to the time specified by the mark 623 in an area 625 near the mark 623. Note that the default time designated by the mark 623 is, for example, the leading time of the actual waveform 621.

  When an analysis instruction is input, the analysis unit 59 displays a mark 624 that specifies a time on the time axis of the reference waveform 622 so as to overlap the reference waveform 622. The analysis unit 59 causes the value of the feature amount corresponding to the time designated by the mark 624 to be displayed in an area 626 near the mark 624. Note that the default time designated by the mark 624 is, for example, the leading time of the reference waveform 622.

  When an analysis instruction is input, the analysis unit 59 displays an actual image captured by the imaging device 21 in the area 603. The analysis unit 59 causes the real image corresponding to the time designated by the mark 623 to be displayed in the area 603.

  When the analysis instruction is input, the analysis unit 59 displays the frame of the first reference moving image in the area 604. The analysis unit 59 displays a frame corresponding to the time specified by the mark 624 in the area 604.

  When an analysis instruction is input, the analysis unit 59 displays thumbnails of a plurality of real images captured by the imaging device 21 in the area 605. The thumbnails of the plurality of real images always include the thumbnail 627 of the real image corresponding to the time specified by the mark 623. The analysis unit 59 displays the thumbnail 627 in a display format different from the remaining thumbnails. In the example shown in FIG. 12, only the thumbnail 627 is surrounded by a thick frame.

  When the analysis instruction is input, the analysis unit 59 displays thumbnails of a plurality of frames of the first reference moving image in the area 606. The thumbnails of the plurality of frames always include the thumbnail 628 of the frame corresponding to the time specified by the mark 624. The analysis unit 59 displays the thumbnail 628 in a display format different from that of the remaining thumbnails. In the example shown in FIG. 11, only the thumbnail 628 is surrounded by a thick frame.

  The analysis unit 59 receives an operation of the buttons 607 to 612. When any of buttons 607 to 612 is operated, analysis unit 59 performs the following processing.

(When the button 607 is operated)
The button 607 is a button for switching between the imaging devices 21 and 22. When the button 607 is operated, the analysis unit 59 displays a second time change of the feature amount extracted from the real image captured by the imaging device 22 based on the fourth table read from the real moving image storage unit 57. An actual waveform is created as an actual waveform 621. The analysis unit 59 switches the real waveform 621 displayed in the area 601 to the second real waveform. Further, the analysis unit 59 creates, as the reference waveform 622, a second reference waveform indicating a temporal change of the feature amount extracted from each frame of the second reference moving image based on the second table stored in the reference moving image storage unit 51. I do. The analysis unit 59 switches the reference waveform 622 displayed in the area 602 to the second reference waveform.

  Further, the analysis unit 59 switches the real image displayed in the area 603 to the real image captured by the imaging device 22 and switches the frame displayed in the area 604 to the frame of the second reference moving image.

  Further, the analysis unit 59 switches the thumbnail displayed in the area 605 to the thumbnail of the real image captured by the imaging device 22, and switches the thumbnail displayed in the area 606 to the thumbnail of the frame of the second reference moving image.

  Thereafter, each time the button 607 is operated, the analysis unit 59 switches the real waveform 621 displayed in the area 601 from one of the first real waveform and the second real waveform to the other. Further, each time the button 607 is operated, the analysis unit 59 switches the reference waveform 622 displayed in the area 602 from one of the first reference waveform and the second reference waveform to the other.

  When the first real waveform is displayed in the region 601, the analysis unit 59 displays the real image of the imaging device 21 in the region 603 and displays the thumbnails of the plurality of real images of the imaging device 21 in the region 605.

  When the second real waveform is displayed in the region 601, the analysis unit 59 displays the real image of the imaging device 22 in the region 603 and displays the thumbnails of the plurality of real images of the imaging device 22 in the region 605.

  When the first reference waveform is displayed in the region 602, the analysis unit 59 displays the frame of the first reference moving image in the region 603 and displays the thumbnails of the plurality of frames of the first reference moving image in the region 605.

  When the second reference waveform is displayed in the region 602, the analysis unit 59 displays the frame of the second reference moving image in the region 603 and displays the thumbnails of the plurality of frames of the second reference moving image in the region 605.

  Note that the analysis unit 59 does not change the positions of the marks 623 and 624 regardless of the operation of the button 607.

(When the button 608 is operated)
The button 608 is a button for moving the positions of the marks 623 and 624 by a unit time. By operating the button 608, the analysis unit 59 changes the values displayed in the areas 625 and 626 according to the time specified by the marks 623 and 624, respectively. Furthermore, the analysis unit 59 changes the actual image displayed in the area 603 and the frame displayed in the area 604 according to the time specified by the marks 623 and 624, respectively. Further, the analysis unit 59 changes the thumbnails surrounded by the thick frames in the areas 605 and 606 according to the time specified by the marks 623 and 624, respectively.

  As described above, the analysis unit 59 changes the display contents of the areas 603, 605, and 625 according to the movement of the mark 623. Further, the analysis unit 59 changes the display contents of the areas 604, 606, and 626 according to the movement of the mark 624. In the following description, the display contents of the areas 603, 605, and 625 are changed according to the movement of the mark 623, and the display contents of the areas 604, 606, and 626 are changed according to the movement of the mark 624, unless otherwise specified.

(When the button 609 is operated)
The button 609 is a button for reproducing the real moving image (the first real moving image or the second real moving image) in the region 603 and reproducing the reference moving image (the first reference moving image or the second reference moving image) in the region 604. When the button 609 is operated, the analysis unit 59 reproduces the actual moving image in the region 603 and reproduces the reference moving image in the region 604. At this time, the analysis unit 59 also moves the position of the mark 623 according to the actual image displayed in the area 603. The analysis unit 59 also moves the position of the mark 624 according to the frame displayed in the area 604.

(When the button 610 is operated)
The button 610 is a button for causing the area 603 to reversely play the real moving image (the first real moving image or the second real moving image), and causing the area 604 to reversely play the reference moving image (the first reference moving image or the second reference moving image). . When the button 609 is operated, the analysis unit 59 reversely reproduces the actual moving image in the area 603 and reversely reproduces the reference moving image in the area 604. At this time, the analysis unit 59 also moves the position of the mark 623 according to the actual image displayed in the area 603. The analysis unit 59 also moves the position of the mark 624 according to the frame displayed in the area 604.

(When the button 611 is operated)
The button 611 is a button for adjusting the time axis between the actual waveform 621 and the reference waveform 622. When the button 611 is operated, the analysis unit 59 corrects the actual waveform 621 so that the time axes of the actual waveform 621 and the reference waveform 622 match. Thereby, the time lag between the actual waveform 621 and the reference waveform 622 is corrected. As a result, it becomes easier to determine whether the control of the robot using the visual servo is normally performed.

  FIG. 13 is a diagram showing an actual waveform and a reference waveform before adjusting the time axis, and an actual waveform and a reference waveform after adjusting the time axis. The upper part of FIG. 13 shows the actual waveform 621 and the reference waveform 622 before adjusting the time axis, and the lower part of FIG. 13 shows the actual waveform 621 and the reference waveform 622 after adjusting the time axis.

  As a method of matching the time axes of the actual waveform 621 and the reference waveform 622, for example, there are pattern matching and DP matching.

  Pattern matching is a method of adjusting the time axis so that the distance between two waveforms is minimized. Specifically, the analysis unit 59 calculates the distance between the real waveform 621 and the reference waveform 622 by moving the real waveform 621 in the time axis direction by unit time. The analysis unit 59 moves the actual waveform 621 to a position where the distance becomes minimum. The distance is, for example, the sum of squares of the difference between the feature amounts at each time in the actual waveform 621 and the reference waveform 622.

  DP matching is a method called dynamic programming, in which at least one time axis of two waveforms is expanded or contracted so that the two waveforms best match. Specifically, the analysis unit 59 calculates the distance between the real waveform 621 and the reference waveform 622 by changing the expansion / contraction ratio of the time axis of the real waveform 621 by a unit amount. The analysis unit 59 expands and contracts the actual waveform 621 at the expansion and contraction ratio at which the distance becomes minimum.

  When the button 611 is operated, the analysis unit 59 matches the position of the mark 623 with the position of the mark 624.

  When the buttons 608 to 610 are operated after correcting the actual waveform 621 according to the operation of the button 611, the analysis unit 59 performs a process based on the corrected actual waveform 621.

(When the button 612 is operated)
The button 612 is a button for switching the feature amount. The analysis unit 59 sequentially switches the feature amounts indicated by the actual waveform 621 and the reference waveform 622 each time the button 612 is operated. Alternatively, when the button 612 is operated, the analysis unit 59 may display a list of a plurality of types of feature amounts, and may display the actual waveform 621 and the reference waveform 622 of one of the feature amounts that have received the selection instruction.

  As described above, the plurality of types of feature amounts include coordinates (pixel coordinates) on the image of the soldering iron 2a, the solder feeder 2b, the electric wire 2c, the pad 2d, and the molten solder 2e. However, the plurality of types of feature amounts may include another feature amount extracted from the image.

  FIG. 14 is a diagram illustrating an example of the feature amount. FIG. 15 is a diagram illustrating another example of the feature amount. The feature quantity shown in FIG. 14 is a movement distance D1 from the initial position on the image of the soldering iron 2a. As described above, the plurality of types of feature amounts may include the moving distance from the initial position on the image of the specific target object.

  The feature quantity shown in FIG. 15 is the area Sa on the image of the molten solder 2e. As described above, the plurality of types of feature amounts may include the area of the blob (region) on the image, the fillet diameter, the perimeter, and the like. The image processing unit 53 may extract a blob from the target image by performing image processing such as labeling.

<C. Operation example>
<C-1. Change information generation processing>
With reference to FIG. 16, a flow of a change information generating process performed by the change information generating unit 551 will be described. FIG. 16 is a flowchart illustrating an example of a flow of a change information generation process performed by the change information generation unit. FIG. 16 shows a flow of processing of the change information generation unit 551 of the first control unit 55a. The change information generation unit 551 of the second control unit 55b, the third control unit 55c, and the fourth control unit 55d may generate change information according to a method similar to that in FIG. The process of generating change information is performed as advance preparation.

  First, in step S1, the control device 50 causes the robot 30a to perform a standard operation of moving the soldering iron 2a held by the hand 31a to the initial position. Thereby, the soldering iron 2a is moved within the field of view of the imaging devices 21 and 22.

  In step S2, the change information generation unit 551 controls the robot 30a such that the same images as the first frames in the teaching range of the first reference moving image and the second reference moving image are captured by the imaging devices 21 and 22, respectively. The subroutine of step S2 will be described later.

  In step S3, the change information generator 551 sets k to the frame number of the first frame of the teaching range. In step S4, the change information generator 551 selects one of the six degrees of freedom.

  In step S5, the change information generation unit 551 generates a control command for moving the hand 31a by the unit control amount in the positive direction of the selected degree of freedom, and outputs the control command to the robot controller 40a.

  In step S6, the change information generation unit 551 acquires the latest actual image from the imaging devices 21 and 22 after the hand 31a moves by the unit control amount.

  In step S7, the change information generating unit 551 acquires from the image processing unit 53 the coordinates of the feature points of the soldering iron 2a extracted from the actual image acquired in step S6.

  In step S8, the change information generation unit 551 acquires from the image processing unit 53 the coordinates of the feature points of the soldering iron 2a extracted from the k-th frame of the first reference moving image and the second reference moving image.

  In step S9, the change information generating unit 551 generates the first change information and the first change information corresponding to the k-th frame and the degree of freedom selected in step S4 based on the coordinates obtained in step S8 and the coordinates obtained in step S7. The second change information is generated. That is, the change information generating unit 551 generates a second map showing a mapping for converting the coordinates of the feature points extracted from the real image by the imaging device 21 into the coordinates of the feature points extracted from the k-th frame of the first reference moving image. 1 Generate change information. Further, the change information generating unit 551 displays a mapping for converting the coordinates of the feature points extracted from the real image by the imaging device 22 into the coordinates of the feature points extracted from the k-th frame of the second reference moving image. 2 Change information is generated.

  In step S10, the change information generation unit 551 generates a control command for returning the hand 31a to the original state (the state before the latest step S5), and outputs the control command to the robot controller 40a. Thus, the soldering iron 2a returns to the state before step S5.

  In step S11, the change information generation unit 551 determines whether there is a degree of freedom that has not been selected. If there is an unselected degree of freedom (YES in step S11), the process of generating change information returns to step S4. As a result, steps S4 to S10 are repeated, and the first change information and the second change information are generated for each of the six degrees of freedom.

  If there is no unselected degree of freedom (NO in step S11), in step S12, the change information generation unit 551 determines whether k is the frame number of the last frame in the teaching range.

  If k is not the frame number of the last frame (NO in step S12), in step S13, the change information generating unit 551 generates the same image as the (k + 1) th frame of the first reference moving image and the second reference moving image in the imaging device 21, The robot 30a is controlled so as to be imaged by each of the robots 22.

  Specifically, the change information generation unit 551 changes the state of the soldering iron 2a on the k-th frame by the same method as the processing content of the calculation unit 553 described above. A control amount with six degrees of freedom for approaching the state is calculated. That is, for each of the plurality of feature points extracted from the k-th frame, the change information generation unit 551 sets the difference vector having the feature point as a starting point and the corresponding feature point extracted from the (k + 1) -th frame as an end point. Ask for. The change information generating unit 551 calculates a control amount with six degrees of freedom based on the difference vector for each feature point and the first change information and the second change information corresponding to the k-th frame. The change information generator 551 generates a control command indicating the calculated control amount, and outputs the control command to the robot controller 40a.

  In step S14, the change information generation unit 551 adds 1 to k. After step S14, the process returns to step S4. As a result, also for the (k + 1) th frame, the first change information and the second change information for each of the six degrees of freedom are generated.

  If k is the frame number of the last frame (YES in step S12), the first change information set 5521 and the second change information set 5522 have been generated for all frames, and thus the change information generation processing ends.

  FIG. 17 is a flowchart showing the flow of the processing of the subroutine of step S2 shown in FIG.

  First, in step S21, the change information generating unit 551 acquires from the image processing unit 53 the coordinates of the feature points of the soldering iron 2a extracted from the first frame of the teaching range of the first reference moving image and the second reference moving image.

  In step S22, the change information generation unit 551 acquires the latest actual image from the imaging devices 21 and 22.

  In step S23, the change information generating unit 551 acquires from the image processing unit 53 the coordinates of the feature points of the soldering iron 2a extracted from the actual image acquired in step S22.

  In step S24, the change information generating unit 551 determines whether the deviation between the state of the soldering iron 2a on the actual image acquired in step S22 and the state of the soldering iron 2a on the top frame is less than the threshold Tha. judge. The deviation is, for example, an average of distances between corresponding feature points of the soldering iron 2a extracted from the actual image and the leading frame.

  If the deviation is less than the threshold value Tha (YES in step S24), the change information generation unit 551 determines that the same image as the first frame is captured by the imaging devices 21 and 22, respectively, and ends the process.

  When the deviation is equal to or larger than the threshold Tha (NO in step S24), in step S25, the change information generation unit 551 selects one of the six degrees of freedom. In step S26, the change information generation unit 551 selects one of the positive direction and the negative direction as the control direction.

  In step S27, the change information generation unit 551 generates a control command for moving the hand 31a by the unit control amount in the selected control direction with the selected degree of freedom, and outputs the control command to the robot controller 40a.

  In step S28, the change information generation unit 551 acquires a real image from the imaging devices 21 and 22 after the hand 31a has moved by the unit control amount.

  In step S29, the change information generation unit 551 acquires from the image processing unit 53 the coordinates of the feature points of the soldering iron 2a extracted from the actual image acquired in step S28.

  In step S30, the change information generation unit 551 calculates a deviation between the state of the soldering iron 2a on the actual image acquired in step S28 and the state of the soldering iron 2a on the top frame.

  In step S31, the change information generation unit 551 generates a control command for returning the hand 31a to the original state (the state before the latest step S27), and outputs the control command to the robot controller 40a. Thereby, the soldering iron 2a returns to the state before step S27.

  In step S32, the change information generation unit 551 determines whether there is an unselected control direction. If there is an unselected control direction (YES in step S32), the process returns to step S26.

  If there is no unselected control direction (NO in step S32), in step S33, the change information generation unit 551 determines whether there is an unselected degree of freedom. If there is an unselected degree of freedom (YES in step S33), the process returns to step S25.

  When there is no unselected degree of freedom (NO in step S33), in step S34, the change information generating unit 551 moves the hand 31a by the unit control amount in the degree of freedom corresponding to the minimum deviation and the control direction. The robot 30a is controlled via the controller 40a. After step S34, the process returns to step S22.

<C-2. Robot control>
With reference to FIG. 18, the flow of the soldering control process using the visual servo will be described. FIG. 18 is a flowchart illustrating an example of the flow of soldering control processing using visual servoing.

  First, in step S41, the control device 50 outputs an end notification from the end determination unit 555 of all the control units (the first control unit 55a, the second control unit 55b, the third control unit 55c, and the fourth control unit 55d). Is determined. When the end notification has been output from the end determination units 555 of all the control units (YES in step S41), the process ends.

  When the end notification has not been output from the end determination units 555 of all the control units (NO in step S41), in step S42, the control device 50 acquires the actual images captured by the imaging devices 21 and 22. Step S42 is performed for each imaging cycle.

  In step S43, the image processing unit 53 detects all objects (the soldering iron 2a, the solder feeder 2b, the electric wire 2c, the pad 2d, and the molten solder 2e) from the actual image and the target frame by template matching, and The coordinates of the feature points of the object on the image are extracted.

  In step S44, the target frame selection unit 54 selects a target frame from the first reference moving image and the second reference moving image.

  In step S45, the target frame selection unit 54 specifies an object including the target frame within the teaching range. The target frame selection unit 54 issues a control instruction to a control unit (at least one of the first control unit 55a, the second control unit 55b, the third control unit 55c, and the fourth control unit 55d) that controls the specified target object. Is output.

  In step S46, the control unit that has received the control instruction controls the target robot. After step S46, the process returns to step S41. If NO in step S4, a series of processing in steps S42 to S46 is repeated for each imaging cycle. Further, at this time, if NO in step S41, step S42 of acquiring the next actual image may be started while the target robot is being controlled in step S46. Thus, the target robot is continuously controlled according to the latest actual image without stopping the operation of the target robot. As a result, the state of the object can be changed quickly.

<C-3. Select target frame>
With reference to FIG. 19 and FIG. 20, the flow of the target frame selection processing by the target frame selection unit 54 will be described. FIG. 19 is a flowchart showing the flow of the processing of the subroutine of step S44 shown in FIG. FIG. 20 is a diagram illustrating the relationship between the closest frame and the target frame.

  First, in step S51, the target frame selection unit 54 acquires, from the image processing unit 53, the coordinates of feature points of all objects extracted from each frame of the first reference moving image and the second reference moving image.

  In step S52, the target frame selection unit 54 acquires from the image processing unit 53 the coordinates of the feature points of all the objects extracted from the actual images captured by the imaging devices 21 and 22.

  In step S53, the target frame selection unit 54 determines whether the first target frame selection has been completed.

  If the first target frame selection has been completed (YES in step S53), in step S54, the target frame selecting unit 54 determines that the previous target frame is the last frame in the teaching range corresponding to any one of the objects. It is determined whether or not there is.

  If the previous target frame is the last frame (YES in step S54), the process proceeds to step S55. In step S55, the target frame selecting unit 54 calculates a deviation between the state on the real image and the state on the final frame for the target corresponding to the teaching range to which the final frame belongs, and determines whether the deviation is less than the threshold. Determine whether or not. The deviation is, for example, an average of distances between corresponding feature points of the object in the real image and the final frame.

  If the deviation is equal to or larger than the threshold (NO in step S55), in step S56, the target frame selecting unit 54 selects the same frame as the previous target frame as the target frame. After step S56, the process ends.

  If the deviation is less than the threshold (YES in step S55), it is determined that the state of the target object has reached the state on the last frame, and the process proceeds to step S57. If the first target frame selection has not been completed (NO in step S53), and if the previous target frame is not the last frame (NO in step S54), the process proceeds to step S57.

  In step S57, the target frame selection unit 54 calculates the deviation between the state of all objects on the real image and the state of all objects on each frame, and specifies the frame with the minimum deviation as the closest frame. . The deviation is, for example, an average of distances between corresponding feature points of the object in the real image and each frame.

  Specifically, the target frame selection unit 54 determines the state of all the objects on the real image captured by the imaging device 21 and the state of all the objects on the k-th frame of the first reference moving image. Calculate one deviation. Further, the target frame selection unit 54 calculates a second deviation between the states of all the objects on the real image captured by the imaging device 22 and the states of all the objects on the k-th frame of the second reference moving image. calculate. The target frame selection unit 54 calculates the average of the first deviation and the second deviation as the deviation corresponding to the k-th frame.

  In step S58, the target frame selection unit 54 determines whether or not there is a last frame in any of the teaching ranges up to a predetermined number of frames after the closest frame.

  If there is a final frame (YES in step S58), in step S59, target frame selecting section 54 selects the final frame as a target frame. When there are a plurality of final frames from the closest frame to a frame that is a predetermined number later, the target frame selecting unit 54 selects the last frame having the smallest frame number among the plurality of final frames.

  If there is no last frame (NO in step S58), in step S60, the target frame selection unit 54 selects a frame that is a predetermined number later than the closest frame as the target frame. After step S60, the target frame selection process ends.

<C-4. Processing of control unit>
The processing of the subroutine of step S46 in FIG. 18 will be described with reference to FIG. FIG. 21 is a flowchart showing the flow of the processing of the subroutine of step S46 shown in FIG.

  First, in step S61, the end determination unit 555 determines whether the target frame is the last frame in the teaching range.

  If the target frame is the final frame (YES in step S61), in step S62, the end determination unit 555 determines whether the deviation between the state of the target on the real image and the state of the target on the final frame is less than a threshold. Is determined.

  When the deviation is less than the threshold (YES in step S62), in step S63, the termination determination unit 555 outputs a termination notification. After step S63, the process ends.

  If the target frame is not the last frame (NO in step S61) and if the deviation is equal to or larger than the threshold (NO in step S62), the process proceeds to step S64.

  In step S64, the calculation unit 553 brings the state of the target on the real image closer to the state of the target on the target frame based on the first change information set 5521 and the second change information set 5522 corresponding to the target frame. For each of a plurality of degrees of freedom for the calculation.

  In step S65, the command unit 554 generates a control command indicating the calculated control amount, and outputs the control command to the target robot controller. After step S65, the process ends.

<C-5. Analysis processing>
With reference to FIG. 22, the processing of the analysis unit 59 will be described. FIG. 22 is a flowchart illustrating the flow of the processing of the analysis unit.

  First, in step S71, the analysis unit 59 determines whether an analysis instruction has been input to the input device 534 (see FIG. 3). When the operator wants to check the analysis screen of the soldering control for a certain pad 2d, the operator inputs a pad identification number for identifying the pad 2d and an analysis instruction to the input device 534.

  If the analysis instruction has not been input (NO in step S71), the process returns to step S71.

  When the analysis instruction is input (YES in step S71), in step S72, the analysis unit 59 reads the first reference moving image, the second reference moving image, the first table, and the second table from the reference moving image storage unit 51. Further, the analysis unit 59 reads the first real moving image, the second real moving image, the third table, and the fourth table corresponding to the input pad identification information from the real moving image storage unit 57.

  In step S73, the analysis unit 59 causes the display device 60 to display an analysis screen 62 as shown in FIG. 12 based on the information read in step S72.

  In step S74, the analysis unit 59 determines whether an end instruction has been input. When the end instruction has not been input (NO in step S74), in step S75, the analysis unit 59 determines whether any of the buttons 607 to 612 has been operated.

  When any of the buttons 607 to 612 is operated (YES in step S75), in step S76, the analysis unit 59 shifts the screen according to the operated button. After step S76, the process returns to step S74. If none of the buttons is operated (NO in step S75), the process returns to step S74.

  When the end instruction has been input (YES in step S74), analysis unit 59 ends the display of analysis screen 62.

<C-6. Action / Effect>
As described above, the control system 1 includes the robots 30a to 30d for changing the state of the target, the imaging devices 21 and 22 for imaging the target, and the first control unit 55a to the fourth control unit 55d. And an analysis unit 59. The first control unit 55a to the fourth control unit 55d determine whether a change in the state of an object on a real image captured by the imaging devices 21 and 22 is a change in the state of the object on the first and second reference moving images created in advance. The target robot is controlled so as to approach the state change. The analysis unit 59 analyzes the display device 60 for both the actual waveform 621 indicating the temporal change of the feature extracted from the actual image and the reference waveform 622 indicating the temporal change of the feature extracted from the frame of the reference moving image. It is displayed on the screen 62.

  When the control of the robots 30a to 30d using the visual servo is normally performed, the change in the state of the target on the real image captured by the imaging devices 21 and 22 is caused by the change in the state of the first and second reference moving images. It follows each change in the state of the object. Therefore, the actual waveform 621 approximates the reference waveform 622. Therefore, the worker checks the actual waveform 621 and the reference waveform 622 displayed on the analysis screen 62 to determine whether the control of the robots 30a to 30d using the visual servo is normally performed. It will be easier. For example, when the deviation between the actual waveform 621 and the reference waveform 622 is large, the operator can determine that some abnormality has occurred in the control of the robots 30a to 30d using the visual servo.

<D. Modification>
<D-1. Modification 1>
FIG. 23 is a diagram illustrating an example of the analysis screen according to the first modification. The analysis screen 63 of the example shown in FIG. 23 is different from the analysis screen 62 shown in FIG. 12 in that it includes a button 613 and waveforms 629 and 630 indicating a time change of the movement amount of the robot from the reference position. . A waveform 629 indicates a time change of the movement amount when the real image is captured. A waveform 630 indicates a temporal change of the movement amount when the state of the target object changes according to the reference moving image.

  In order to display the analysis screen 63 as shown in FIG. 23, the reference moving image storage unit 51 further stores a fifth table shown in FIG. The actual moving image storage unit 57 further stores the sixth table shown in FIG.

  FIG. 24 is a diagram illustrating an example of a fifth table stored in the reference moving image storage unit according to the first modification. FIG. 25 is a diagram illustrating an example of a sixth table stored in the actual moving image storage unit according to the first modification.

  As shown in FIG. 24, the reference moving image storage unit 51 stores the times of all the frames (1st to Mth frames) of the first reference moving image and the second reference moving image, the movement amount information for each of the robots 30a to 30d, and Is stored in a fifth table. The movement amount information indicates a movement amount from the reference position and orientation for each of the plurality of degrees of freedom. For example, in the case of the robot 30a, the movement amount information indicates the amount of translation in the X, Y, and Z directions, and the amount of translation in the pitch, yaw, and roll directions.

  The movement amount information stored in the reference moving image storage unit 51 is created when the change information generation unit 551 generates the first change information and the second change information. As described above, the change information generation unit 551 controls the target robot so that the same real image as the frame is obtained for each of the first to Mth frames of the first reference moving image and the second reference moving image. The change information generation unit 551 generates movement amount information for each of the first to Mth frames of the first reference moving image and the second reference moving image based on the control amount of the target robot.

  Alternatively, when the first reference moving image and the second reference moving image are created by changing the state of the sample of the target object by the robots 30a to 30d, each of the first reference moving image and the second reference moving image is generated. Movement amount information may be generated from the operation of the robot.

  As shown in FIG. 25, the actual moving image storage unit 57 stores a sixth table in which the imaging times of the imaging devices 21 and 22 are associated with the movement amount information for each of the robots 30a to 30d. The moving amount information stored in the actual moving image storage unit 57 is generated by the first control unit 55a, the second control unit 55b, the third control unit 55c, and the fourth control unit 55d.

  Each of the first control unit 55a, the second control unit 55b, the third control unit 55c, and the fourth control unit 55d receives pulse signals from a plurality of encoders provided in the target robot. Each control unit generates movement amount information by counting the number of pulses included in the pulse signal.

  The button 613 is a button for switching the amount of movement indicated by the waveforms 629 and 630. When the button 613 shown in FIG. 23 is operated, the analysis unit 59 displays a pop-up screen that prompts the user to input an instruction to select a robot and a degree of freedom. Then, the analysis unit 59 selects the robot and the degree of freedom according to the input selection instruction.

  The analysis unit 59 reads the selected robot and the movement amount of the degree of freedom from the fifth table stored in the actual moving image storage unit 57, and generates a waveform 629 indicating a time change of the movement amount. That is, the movement amount indicated by the waveform 629 is an amount by which the selected robot has moved from the reference position along the direction of the selected degree of freedom. The analysis unit 59 causes the generated waveform 629 to be displayed in the area 601 and the movement amount at the time specified by the mark 623 to be displayed in the area 631. The region 631 is near the intersection of the mark 623 and the waveform 629.

  Further, the analysis unit 59 reads the selected robot and the movement amount of the degree of freedom from the first table stored in the reference moving image storage unit 51, and generates a waveform 630 indicating a time change of the movement amount. The analysis unit 59 causes the generated waveform 630 to be displayed in the area 602 and the movement amount at the time specified by the mark 624 to be displayed in the area 632. The area 632 is near the intersection of the mark 624 and the waveform 630.

  According to the first modification, the operator can determine whether there is an abnormality in the robot by checking the deviation between the waveforms 629 and 630. As a result, the operator can easily understand the cause of the large deviation between the actual waveform 621 and the reference waveform 622.

<D-2. Modification 2>
FIG. 26 is a diagram illustrating an example of an analysis screen according to the second modification. When the button 611 is operated, the analysis unit 59 calculates a parameter value indicating a difference in feature amount between the actual waveform 621 and the reference waveform 622 at each time on the time axis. The analysis unit 59 causes the analysis screen 64 to display a waveform 633 indicating a change in the calculated parameter value, as shown in FIG. Further, the analysis unit 59 causes the parameter value at the time designated by the mark 623 to be displayed in the area 625. The parameter value is, for example, a difference or a ratio of a feature amount between the actual waveform 621 and the reference waveform 622.

  According to the second modification, the operator confirms the parameter value of the waveform 633 or the region 625, so that the deviation between the actual waveform 621 and the reference waveform 622 increases in any time zone during the soldering control. Can be easily grasped. Note that the analysis unit 59 may display only one of the waveform 633 and the parameter value of the area 625.

  Further, as shown in FIG. 26, the analysis screen 64 may include a button 614. The button 614 is a button for searching for a time at which the deviation between the actual waveform 621 and the reference waveform 622 is large.

  When the button 614 is operated, the analysis unit 59 selects the time at which the parameter value becomes maximum as the designated time. Then, the analysis unit 59 moves the marks 623 and 624 to the position at the designated time. As a result, an actual image corresponding to the designated time is displayed in the area 603, and a frame corresponding to the designated time is displayed in the area 604.

  Accordingly, the operator can check the image when the difference between the characteristic amounts of the actual waveform 621 and the reference waveform 622 is large, and it is easy to grasp the cause of the abnormality of the robot.

  Note that the analysis unit 59 may select a time at which the parameter value exceeds the threshold value as the designated time. When there are a plurality of times at which the parameter value exceeds the threshold, the analysis unit 59 selects one of the plurality of times as the designated time.

  When the button 609 is operated, the analysis unit 59 causes the video of a part of the real moving image (the first real moving image or the second real moving image) including the designated time to be reproduced in the area 603, and the reference moving image (the first real moving image or the first real moving image). The video of the period in the reference video or the second reference video) may be reproduced in the area 604. This makes it easier for the operator to grasp the cause of the abnormality in the soldering control.

  Modification 2 can also be applied to Modification 1 described above. That is, the analysis unit 59 may calculate a parameter value indicating a difference in the movement amount between the waveform 629 and the waveform 630 for each time on the time axis. The analysis unit 59 causes the analysis screen 63 to display at least one of the parameter value and a waveform indicating a time change of the parameter value. Alternatively, the analysis unit 59 may select the time at which the parameter value becomes maximum as the designated time.

<D-3. Modification 3>
FIG. 27 is a diagram illustrating an example of an analysis screen according to the third modification. As shown in FIG. 27, the analysis screen 65 includes a button 615 for selecting an object whose movement direction is indicated, and an arrow 634 indicating the movement direction of the selected object.

  In order to display the arrow 634, the real moving image storage unit 57 configures a seventh table in which real images constituting the first real moving image are associated with moving directions of the respective objects on the image, and a second real moving image. And an eighth table in which the actual image to be processed is associated with the moving direction of each object on the image.

  FIG. 28 is a diagram illustrating an example of a seventh table stored in the actual moving image storage unit according to the third modification. In the seventh table of the example shown in FIG. 28, the image file name for identifying the real image is associated with the moving direction information indicating the moving direction of each object. The eighth table also has the same data structure as the seventh table shown in FIG.

  As described above, the calculation unit 553 calculates the first difference vectors 6a to 6g (see FIG. 11) each time a real image is acquired from the imaging device 21. The calculation unit 553 calculates the control amount of the target robot such that the first difference vector becomes smaller. Therefore, the target object moves in the direction of the average of the first difference vector on the real image. Therefore, the calculation unit 553 generates information indicating the average direction of the first difference vector as the movement direction information. The actual moving image storage unit 57 stores the seventh and eighth tables including the moving direction information generated by the calculation unit 553.

  When the button 615 is operated, the analysis unit 59 displays a pop-up screen for prompting an instruction to select an object. The analysis unit 59 reads the moving direction information corresponding to the selected object and the real image at the time specified by the mark 623 from the real moving image storage unit 57. When the actual waveform 621 indicates the feature amount extracted from the actual image captured by the imaging device 21, the analysis unit 59 reads the movement direction information from the seventh table. When the actual waveform 621 indicates the feature amount extracted from the actual image captured by the imaging device 22, the analysis unit 59 reads the movement direction information from the eighth table.

  The analysis unit 59 generates an arrow 634 indicating the moving direction of the target object based on the read moving direction information, and causes the generated arrow 634 to be displayed in the area 603. This makes it easier for the operator to grasp the moving direction of the object.

  Note that the analysis unit 59 may sequentially switch the object whose movement direction is indicated each time the button 615 is operated.

<D-4. Modification 4>
In the above description, the control system 1 performs soldering control. However, the control system 1 may perform control on another object.

  FIG. 29 is a schematic diagram illustrating an object of the control system according to the fourth modification. The control system 1 according to Modification 4 connects the male connector 7a to the female connector 7b by inserting the male connector 7a into the female connector 7b in an industrial product production line or the like.

  The male connector 7a is gripped by the hand 31a of the robot 30a. The female connector 7b is mounted on the stage 31d of the robot 30d.

  The robots 30a to 30d may change the state (here, the size) of the expanding and contracting target object such as a balloon. In this case, the control device 50 acquires change information indicating a change in the size of the target object when the robot is controlled by the unit control amount. Then, the control device controls the robot based on the change information such that the size of the target on the real image approaches the size of the target on the target frame.

  Alternatively, the robots 30a to 30d may change the state (here, the size) of the target object by applying a force to the target object.

<D-5. Other Modifications>
The image processing unit 53 may detect the target from the target image using the 3D-CAD data of the target.

  The first reference moving image and the second reference moving image may be created by CG (Computer Graphics).

  In the above description, the control device 50 includes the robot control unit 58 and the analysis unit 59. That is, the control device 50 operates as a robot control device and also operates as an analysis device. However, the analysis unit 59 may be provided in an analysis device communicably connected to the control device 50.

≪Note≫
As described above, the first to third embodiments and the modifications include the following disclosure.

(Configuration 1)
A robot (30a-30d) for changing the state of the object (2a-2d, 7a, 7b);
An imaging device (21, 22) for imaging the object;
The robots (30a to 30a to 30a to 30d) change the state of the target on a real image captured by the imaging device (21, 22) closer to the change of the state of the target on a previously created reference moving image. A control unit (58) for controlling 30d);
The screen of the display device (60) displays both a first waveform indicating a temporal change of a feature amount extracted from the real image and a second waveform indicating a temporal change of the feature amount extracted from a frame of the reference moving image. A control system (1) including an analysis unit (59) for displaying the information on the display.

(Configuration 2)
The control system (1) according to Configuration 1, wherein the analysis unit (59) adjusts a time axis of the first waveform and the second waveform.

(Configuration 3)
The analysis unit (59) calculates, for each time on the time axis, a parameter value indicating a difference in the feature amount between the first waveform and the second waveform, and calculates a parameter value and a time of the parameter value. The control system (1) according to Configuration 2, wherein at least one of a third waveform indicating a change is displayed on the screen.

(Configuration 4)
The analysis unit (59)
For each time on the time axis, calculate a parameter value indicating a difference between the feature amounts of the first waveform and the second waveform,
Based on the calculated parameter value, the time when the parameter value is the maximum, or the time when the parameter value exceeds a threshold is selected as the designated time,
The control system (1) according to Configuration 2, wherein the real image corresponding to the designated time and the frame at the designated time in the reference moving image are displayed on the screen at the same time.

(Configuration 5)
The analysis unit (59)
Accepting a designated time on the time axis,
The control system (1) according to Configuration 2, wherein the real image corresponding to the designated time and the frame at the designated time in the reference moving image are displayed on the screen at the same time.

(Configuration 6)
The analysis unit (59) simultaneously reproduces, on the screen, video of a part of the period including the designated time in both the real moving image composed of the real images arranged in time series and the reference moving image. The control system (1) according to 4 or 5.

(Configuration 7)
The analysis unit (59) includes a fourth waveform indicating a time change of a movement amount of the robot from a reference position when the real image is captured, and a state in which the state of the object changes according to the reference moving image. The control system (1) according to any one of Configurations 1 to 6, wherein both the fifth waveform indicating the time change of the movement amount and the fifth waveform are displayed on the screen.

(Configuration 8)
The robot (30a to 30d) changes the state of the object with a plurality of degrees of freedom,
The analysis unit (59) receives an instruction to select one of the degrees of freedom,
The control system (1) according to configuration 7, wherein the movement amount is an amount that the robot (30a to 30d) has moved from the reference position along the direction of the one degree of freedom.

(Configuration 9)
An analysis device used in the control system (1) according to any one of the configurations 1 to 8,
An analysis device (50) comprising the analysis unit (59).

(Configuration 10)
A control method for controlling a robot (30a to 30d) that changes a state of an object using an imaging device (21, 22) for imaging the object (2a to 2d, 7a, 7b),
The robots (30a to 30a to 30a to 30d) change the state of the target on a real image captured by the imaging device (21, 22) closer to the change of the state of the target on a previously created reference moving image. Controlling 30d);
A display device (60) for displaying both a first waveform indicating a temporal change of a feature amount extracted from the frame of the real image and a second waveform indicating a temporal change of the feature amount extracted from the frame of the reference moving image; Displaying on the screen of (1).

  Although the embodiment of the present invention has been described, the embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  DESCRIPTION OF SYMBOLS 1 Control system, 2a soldering iron, 2b solder feeder, 2c electric wire, 2d pad, 2e fusion solder, 3 boards, 7a male connector, 7b female connector, 21, 22 imaging device, 30a-30d robot, 31a-31c hand, 31d stage, 40a to 40d robot controller, 50 control device, 51 reference moving image storage unit, 52 teaching range selection unit, 53 image processing unit, 54 target frame selection unit, 55a first control unit, 55b second control unit, 55c 3 control unit, 55d fourth control unit, 56 actual image acquisition unit, 57 actual moving image storage unit, 58 robot control unit, 59 analysis unit, 60 display device, 510 processor, 512 RAM, 514 display controller, 516 system controller, 518 I / O controller, 520 hard disk , 522 camera interface, 522a, 522b image buffer, 524 input interface, 526 robot controller interface, 528 communication interface, 530 memory card interface, 534 input device, 536 memory card, 550 control program, 551 change information generation unit, 552 change Information storage unit, 553 calculation unit, 554 command unit, 555 end determination unit, 5521 first change information set, 5522 second change information set, 601 to 606, 625, 626, 631, 632 area, 607 to 615 button, 621 Actual waveform, 622 reference waveform, 623,624 mark, 627,628 thumbnail, 629,630,633 waveform.

Claims (10)

A robot for changing the state of the object,
An imaging device for imaging the object;
A control unit for controlling the robot so that a change in the state of the target on the real image captured by the imaging device approaches a change in the state of the target on the previously created reference moving image. ,
Displaying both a first waveform indicating a temporal change of a feature extracted from the real image and a second waveform indicating a temporal change of the feature extracted from a frame of the reference moving image on a screen of a display device. A control system, comprising:   The control system according to claim 1, wherein the analysis unit adjusts a time axis of the first waveform and a time axis of the second waveform.   The analysis unit calculates, for each time on the time axis, a parameter value indicating a difference in the feature amount between the first waveform and the second waveform, and indicates a time change of the parameter value and the parameter value. The control system according to claim 2, wherein at least one of the third waveforms is displayed on the screen. The analysis unit,
For each time on the time axis, calculate a parameter value indicating a difference between the feature amounts of the first waveform and the second waveform,
Based on the calculated parameter value, the time when the parameter value is the maximum, or the time when the parameter value exceeds a threshold is selected as the designated time,
The control system according to claim 2, wherein the real image corresponding to the designated time and a frame at the designated time in the reference moving image are simultaneously displayed on the screen. The analysis unit,
Accepting a designated time on the time axis,
The control system according to claim 2, wherein the real image corresponding to the designated time and a frame at the designated time in the reference moving image are simultaneously displayed on the screen.   The analysis unit, in both the real moving image and the reference moving image composed of the real images arranged in time series, simultaneously reproduces a video of a partial period including the designated time on the screen, 6. The control system according to 5.   The analysis unit may further include a fourth waveform indicating a temporal change in a movement amount of the robot from a reference position when the real image is captured, and the movement when the state of the target changes as in the reference moving image. The control system according to any one of claims 1 to 6, wherein both a fifth waveform indicating a temporal change in the amount is displayed on the screen. The robot changes the state of the object with a plurality of degrees of freedom,
The analysis unit receives an instruction to select one of the degrees of freedom, and
The control system according to claim 7, wherein the movement amount is an amount of movement of the robot from the reference position along the direction of the one degree of freedom. An analysis device used for the control system according to any one of claims 1 to 8,
An analysis device comprising the analysis unit. A control method for controlling a robot that changes a state of the object, using an imaging device for imaging the object,
Controlling the robot so that a change in the state of the target on the real image captured by the imaging device approaches a change in the state of the target on the previously created reference moving image,
Both a first waveform indicating a temporal change of a feature amount extracted from the frame of the real image and a second waveform indicating a temporal change of the feature amount extracted from the frame of the reference moving image are displayed on a screen of a display device. And a step of displaying the information.

Images