JP2015024480A - Information processing device, control method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize calibration easily.SOLUTION: Provided is an information processing device 20 including: a storage part 21 which stores three-dimensional information of a reality device 10 that includes an object OBJ, a robot equipped with the object OBJ and a camera imaging at least one part of the object OBJ and the robot; and an operation part 22 which creates a virtual device 23 in which the reality device 10 is reproduced on the basis of the three-dimensional information in the virtual space, synchronizes operation of the virtual device 23 with operation of the reality device 10, creates a first image P1 imaged by a camera 24 of the virtual device 23, and adjusts control parameters A, B of the virtual device 23 on the basis of the difference between the second image P2 imaged by the reality device 10 and the first image P1 corresponding to the second image P2.

Description

  The present invention relates to an information processing apparatus, a control method, and a program.

  In the manufacturing field, various devices such as assembly devices and inspection devices are used. Such an apparatus has a robot that controls the position and orientation of a workpiece (object). Further, in such an apparatus, a camera is installed in the apparatus so that an operator can confirm the position and posture of the work and the robot and visually recognize the appearance of the work. The operator performs an operation for controlling the robot or the like while viewing the captured image captured by the camera. In addition, the captured image captured by the camera may be used for robot calibration.

  For example, a first method for measuring position information of a work from image information of the work taken by a camera has been proposed. In the first method, master data indicating the reference position of the work placed on the work place is used. This master data is data obtained using CAD (Computer Aided Design) information. The apparatus according to the first method calculates a deviation amount between the three-dimensional feature point position of the workpiece obtained from the captured image and the three-dimensional feature point position of the workpiece indicated by the master data, and uses the calculated deviation amount. Correct the trajectory of the robot arm.

  A second method for calculating a rigid body transformation matrix for transforming the camera coordinate system and the robot coordinate system based on feature points extracted from the captured image of the stereo camera has been proposed. The apparatus which concerns on a 2nd method estimates the origin coordinate of the jig coordinate system in a camera coordinate system using the least squares method. Also, a marker is fixed to the robot, a plurality of reference points are set in the three-dimensional space, the robot is operated to position the marker at each reference point, and the reference point and the position where the marker is projected on the camera image A third method for calculating the parameter that minimizes the difference has been proposed.

JP 2009-248214 A JP 2011-011321 A JP 2006-110705 A

Here, consider a system that assists the control and inspection work of a real device by using a virtual device in which a real device on which a work or a robot is installed is reproduced in a virtual space.
The parts and workpieces of the robot installed in the real device can be described by 3D information such as 3D CAD data. Also, the parts and workpieces of the robot installed in the virtual device can be described by three-dimensional information. Therefore, it is possible to synchronously control the virtual device based on the three-dimensional information of the real device.

  However, when a deviation occurs between the real device and the three-dimensional information due to a design error or the like, if the inspection or teaching is performed using the virtual device, the work result is affected by the deviation. Therefore, the inventor of the present invention diligently studied a method for calibrating a virtual device.

  According to one aspect, an object of the present invention is to provide an information processing apparatus, a control method, and a program capable of easily realizing calibration.

  According to one aspect of the present disclosure, a storage unit that stores three-dimensional information of a real device including an object, a robot on which the object is installed, and a camera that images at least a part of the object and the robot; Generating a virtual device that reproduces the real device in the virtual space based on the three-dimensional information, synchronizing the operation of the virtual device with the operation of the real device, and generating a first image captured by the camera of the virtual device; An information processing apparatus is provided that includes an arithmetic unit that adjusts a control parameter of a virtual device based on a difference between a second image captured by a real device and a first image corresponding to the second image.

  According to another aspect of the present disclosure, a memory that stores three-dimensional information of a real device including an object, a robot on which the object is installed, and a camera that images at least a part of the object and the robot The first image captured by the camera of the virtual device by generating a virtual device that reproduces the real device in the virtual space based on the three-dimensional information, synchronizing the operation of the virtual device with the operation of the real device And a control method for adjusting the control parameters of the virtual device based on the difference between the second image captured by the real device and the first image corresponding to the second image is provided.

  According to another aspect of the present disclosure, a memory that stores three-dimensional information of a real device including an object, a robot on which the object is installed, and a camera that images at least a part of the object and the robot A virtual device that reproduces the real device in the virtual space based on the three-dimensional information, synchronizes the operation of the virtual device with the operation of the real device, and is captured by the camera of the virtual device And a program for executing a process of adjusting the control parameter of the virtual device based on the difference between the second image captured by the real device and the first image corresponding to the second image is provided.

  As described above, according to the present invention, calibration can be easily realized.

It is the figure which showed an example of the system which concerns on 1st Embodiment. It is the figure which showed an example of the system which concerns on 2nd Embodiment. It is the figure which showed an example of the real device which concerns on 2nd Embodiment. It is a figure for demonstrating the mechanism of the real apparatus which concerns on 2nd Embodiment. It is the figure which showed an example of the hardware which can implement | achieve the function of the control apparatus which concerns on 2nd Embodiment. It is a figure for demonstrating the function which the control apparatus which concerns on 2nd Embodiment has. It is a figure for demonstrating the function which the information processing apparatus which concerns on 2nd Embodiment has. It is the figure which showed an example of the operation | movement which the system which concerns on 2nd Embodiment performs. It is a figure for demonstrating the extraction process of the test | inspection range which the information processing apparatus which concerns on 2nd Embodiment performs. It is a figure for demonstrating the error factor of the error which arises between the real device and virtual device which concerns on 2nd Embodiment. It is a figure for demonstrating the example of the parameter utilized for the designation method of the feature point which concerns on 2nd Embodiment, and calibration. It is a figure for demonstrating the automatic extraction method of the feature point which concerns on 2nd Embodiment. It is the flowchart which showed the flow of the calibration process which the information processing apparatus which concerns on 2nd Embodiment performs. It is the flowchart which showed the processing flow of the parameter calculation performed in the calibration process which the information processing apparatus which concerns on 2nd Embodiment performs. It is a figure for demonstrating the example of the parameter used for the designation | designated method of the feature point which concerns on the modification of 2nd Embodiment, and calibration. It is the flowchart which showed the process flow of the parameter calculation performed in the calibration process which the information processing apparatus which concerns on the modification of 2nd Embodiment performs.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. In addition, about the element which has the substantially same function in this specification and drawing, duplication description may be abbreviate | omitted by attaching | subjecting the same code | symbol.

<1. First Embodiment>
The first embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of a system according to the first embodiment.

  As shown in FIG. 1, the system according to the first embodiment includes a real device 10 and an information processing device 20. The real device 10 includes an object OBJ, a robot on which the object OBJ is installed, and a camera that images at least a part of the object OBJ and the robot.

The information processing apparatus 20 includes a storage unit 21 and a calculation unit 22.
The function of the storage unit 21 is realized by a volatile storage device such as a RAM (Random Access Memory) or a nonvolatile storage device such as an HDD (Hard Disk Drive) or a flash memory.

  The function of the calculation part 22 is implement | achieved by processors, such as CPU (Central Processing Unit) and DSP (Digital Signal Processor). The function of the calculation unit 22 can also be realized using an electronic circuit other than a processor such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

For example, the calculation unit 22 executes a program stored in the storage unit 21 or another memory.
The storage unit 21 stores three-dimensional information of the real device 10. For example, the storage unit 21 stores three-dimensional CAD data indicating the shape and position of the object OBJ and the robot included in the real device 10. That is, the design information of the real device 10 is stored in the storage unit 21. The calculation unit 22 generates a virtual device 23 that reproduces the real device 10 in the virtual space based on the three-dimensional information stored in the storage unit 21.

  As shown in FIG. 1, the virtual device 23 includes a camera 24 corresponding to the camera of the real device 10 and a robot 25 corresponding to the robot of the real device 10. Further, when the object OBJ is installed in the robot of the real device 10, the object OBJ is also installed in the robot 25 of the virtual device 23. The computing unit 22 synchronizes the operation of the virtual device 23 with the operation of the real device 10. That is, when the state of the real device 10 is changed, the arithmetic unit 22 controls the state of the virtual device 23 so that the state is the same as the state of the real device 10 after the change.

  For example, when the control parameter B for controlling the camera is changed in the real device 10, the control parameter B of the camera 24 is changed to the same value by the calculation unit 22. Similarly, when the control parameter A that controls the robot in the real device 10 is changed, the control parameter A of the robot 25 is changed to the same value by the calculation unit 22. Even when a control parameter for controlling another mechanism included in the real device 10 is changed, the control unit 22 changes the control parameter of the virtual device 23 corresponding to the control parameter.

  The calculation unit 22 generates a first image P <b> 1 captured by the camera 24 of the virtual device 23. Since the environment of the real device 10 is reproduced in the virtual device 23, the first image P1 captured by the camera 24 by simulation is the same image as the second image P2 captured by the real device 10. However, if an installation error, an assembly error, or the like occurs at the stage of manufacturing the real device 10, a deviation occurs between the virtual device 23 generated based on the three-dimensional information that is design information and the real device 10. In this case, a deviation also occurs between the second image P <b> 2 captured by the real device 10 and the first image P <b> 1 captured by the virtual device 23.

  The arithmetic unit 22 adjusts the control parameters A, B,... Of the virtual device 23 based on the difference between the second image P2 captured by the real device 10 and the first image P1 corresponding to the second image P2. Two-dimensional coordinates (hereinafter referred to as two-dimensional coordinates xv) indicating one point on the first image P1 are obtained by using the three-dimensional information, so that the three-dimensional coordinates Q in the virtual space corresponding to the point and the control parameter A , B, ... That is, a model expressing the relationship between the two-dimensional coordinate xv, the three-dimensional coordinate Q, and the control parameters A, B,. The same applies to the real device 10.

  When the feature point is determined on the first image P1, the two-dimensional coordinate xv of the feature point, the control parameters A, B,... Of the virtual device 23 set when the first image P1 is generated, and the two-dimensional coordinate xv are set. A corresponding three-dimensional coordinate Q is obtained. Further, when a feature point on the second image P2 corresponding to a feature point on the first image P1 is determined, a two-dimensional coordinate (hereinafter, two-dimensional coordinate xr) of the feature point is obtained.

  For example, the calculation unit 22 prepares a plurality of sets of the first image P1 and the second image P2, and regarding the feature points, the two-dimensional coordinates xv, xr, the three-dimensional coordinates Q, and the control parameters A, B,. To get. Then, the calculation unit 22 uses the acquired information and the above model to detect errors dA, dB, between the control parameters A, B,... Of the real device 10 and the control parameters A, B,. Estimate ... For example, the calculation unit 22 calculates control parameter errors dA, dB,... Using a least square method or the like. The calculation unit 22 that has obtained the calculation result applies the calculated control parameter errors dA, dB,... To the control parameters A, B,. For example, the calculation unit 22 sets the control parameters of the virtual device 23 as A + dA, B + dB,.

  According to the above method, it is possible to easily adjust the control parameters of the virtual device 23 using only a two-dimensional image without using three-dimensional image information such as a stereo image. Further, according to the above method, calibration can be performed using the captured image used during normal operation on the real device 10 side and the virtual device 23 side. Therefore, it is not necessary to provide additional equipment, and an increase in cost due to the addition of the calibration function can be suppressed.

The first embodiment has been described above.
<2. Second Embodiment>
Next, a second embodiment will be described.

[2-1. system]
First, a system according to the second embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a system according to the second embodiment.

As illustrated in FIG. 2, the system according to the second embodiment includes a real device 100, a control device 200, an information processing device 300, and display devices 250 and 350.
The real device 100 includes a camera 101, an installation table 102, an illumination 103, and a robot 104. The operations of the camera 101, the installation table 102, the illumination 103, and the robot 104 are controlled by the control device 200.

  The camera 101 is an imaging device that captures an area including a work WK installed on the robot 104. The camera 101 includes, for example, an optical system such as a lens, an image sensor such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal-Oxide Semiconductor), and an image processing processor that performs A / D conversion, digital signal processing, and the like. including.

  The installation table 102 is a table for fixing the illumination 103 and the robot 104. The illumination 103 is a light source that is installed on the installation table 102 and illuminates an area including the work WK installed on the robot 104. The robot 104 is a mechanism that holds the workpiece WK, moves the position of the workpiece WK, and changes the posture of the workpiece WK.

  The real device 100 is used, for example, for the state inspection of the surface of the workpiece WK. In this case, the real device 100 reflects an illumination pattern such as a mesh pattern on the surface of the work WK, and images the work WK on which the illumination pattern is reflected. By analyzing the shape of the illumination pattern reflected on the workpiece WK from the captured image thus captured, defects such as scratches and dents on the surface of the workpiece WK can be detected.

  When the work WK has a three-dimensional shape, the illumination pattern may be reflected only in a limited range (inspection visual field) on the surface of the work WK. Therefore, the real device 100 uses the camera 101 to capture an image of the work WK on which the illumination pattern is reflected while changing the position and orientation of the work WK using the robot 104.

  By using a plurality of captured images captured while changing the position and orientation of the workpiece WK, the shape of the illumination pattern can be analyzed over a wide range of the surface of the workpiece WK. Although the position and orientation of the camera 101 may be changed, in the second embodiment, the description is given by taking the case where the camera 101 is fixed and the position and orientation of the workpiece WK are changed as an example.

As described above, if the real device 100 is used, a captured image of the workpiece WK captured from a plurality of inspection fields can be obtained. Note that the operator determines the inspection visual field.
When the position and posture of the workpiece WK are changed, it is difficult for the operator to accurately recognize which range on the surface of the workpiece WK the inspection visual field before the change is. Therefore, the system according to the second embodiment provides a mechanism that allows the operator to easily recognize the inspection visual field. By applying this mechanism, it is possible to suppress the occurrence of inspection omission in a part of the work WK, or the increase in inspection time due to excessive overlap of inspection fields in order to avoid the inspection omission. Can be expected.

  The operation of the real device 100 as described above is controlled by the control device 200. The control device 200 is connected to the information processing device 300 via a communication line. For example, the control device 200 transmits control information of the real device 100 to the information processing device 300 using socket communication.

  The control device 200 is connected to the display device 250. The display device 250 is a display device such as a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), or an ELD (Electro-Luminescence Display). The control device 200 causes the display device 250 to display an image captured by the camera 101. The operator performs an operation while viewing the captured image of the display device 250.

  The information processing device 300 holds the three-dimensional CAD data of the real device 100 and generates a virtual device that reproduces the real device 100 in the virtual space using the three-dimensional CAD data. Further, the information processing apparatus 300 receives control information and the like of the real apparatus 100 and simulates the operation of the virtual apparatus based on the received control information.

  The information processing apparatus 300 calculates the range of the work WK surface corresponding to the inspection visual field. Then, the information processing device 300 transmits information indicating the range (inspection range) of the surface of the workpiece WK corresponding to the calculated inspection visual field to the control device 200.

  The information processing device 300 is connected to the display device 350. The display device 350 is a display device such as a CRT, LCD, PDP, or ELD. The information processing device 300 causes the display device 350 to display a captured image of the work WK virtually captured in the virtual device. The information processing apparatus 300 causes the examination range to be displayed superimposed on the captured image displayed on the display device 350.

  In addition, the information processing apparatus 300 converts the range of the work WK corresponding to the inspection visual field into three-dimensional CAD data and holds it. Information on the examination range is appropriately transmitted to the control device 200 and displayed superimposed on the captured image displayed on the display device 250.

  The system according to the second embodiment has been described above. The inspection method using the reflection of the illumination pattern has been described as an example, but the system according to the second embodiment can be applied to other inspection methods. Hereinafter, hardware, functions, and a processing flow of apparatuses included in the system according to the second embodiment will be described.

[2-2. hardware]
The hardware of the device included in the system according to the second embodiment will be described with reference to FIGS. FIG. 3 is a diagram illustrating an example of a real device according to the second embodiment. FIG. 4 is a diagram for explaining the mechanism of the real device according to the second embodiment. FIG. 5 is a diagram illustrating an example of hardware capable of realizing the function of the control device according to the second embodiment.

(Real equipment)
The hardware of the real device 100 will be described with reference to FIGS. 3 and 4.
As illustrated in FIG. 3, the real device 100 includes a camera 101, an installation table 102, an illumination 103, an X mechanism 111, a Y mechanism 112, a first rotation mechanism 113, a second rotation mechanism 114, and a gripping mechanism 115. The X mechanism 111, the Y mechanism 112, the first rotation mechanism 113, the second rotation mechanism 114, and the gripping mechanism 115 are examples of an installation mechanism.

  The X mechanism 111 is installed on the installation table 102. The Y mechanism 112 is installed on the X mechanism 111. The first rotation mechanism 113 is installed on the Y mechanism 112. The second rotation mechanism 114 is installed on the first rotation mechanism 113. The second rotation mechanism 114 is provided with a gripping mechanism 115. A workpiece WK is installed on the gripping mechanism 115.

  As shown in FIG. 4, the X mechanism 111 is a mechanism that moves as indicated by an arrow a along the X direction. As the X mechanism 111 moves, the Y mechanism 112, the first rotating mechanism 113, the second rotating mechanism 114, the gripping mechanism 115, and the workpiece WK move in the X direction. The Y mechanism 112 is a mechanism that moves as indicated by an arrow b along the Y direction orthogonal to the X direction. As the Y mechanism 112 moves, the first rotating mechanism 113, the second rotating mechanism 114, the gripping mechanism 115, and the workpiece WK move in the Y direction.

  The first rotation mechanism 113 rotates as indicated by an arrow c in the XY plane. As the first rotation mechanism 113 rotates, the second rotation mechanism 114, the gripping mechanism 115, and the workpiece WK rotate in the XY plane. The second rotation mechanism 114 rotates in the direction of tilting the gripping mechanism 115 from the Z direction to the XY plane (rotates as indicated by an arrow d). As the second rotation mechanism 114 rotates, the gripping mechanism 115 and the workpiece WK tilt from the Z direction to the XY plane by the rotation angle.

  The position of the workpiece WK is moved by the operation of the X mechanism 111 and the Y mechanism 112. Further, the posture of the workpiece WK changes by the operation of the first rotation mechanism 113 and the second rotation mechanism 114. Therefore, the inspection visual field can be freely adjusted by controlling the operations of the X mechanism 111, the Y mechanism 112, the first rotating mechanism 113, and the second rotating mechanism 114.

(Control device, information processing device)
Next, the hardware of the control device 200 will be described with reference to FIG.
The functions of the control device 200 can be realized by using, for example, hardware resources of the information processing apparatus illustrated in FIG. That is, the functions of the control device 200 are realized by controlling the hardware shown in FIG. 5 using a computer program.

  As shown in FIG. 5, this hardware mainly includes a CPU 902, a ROM (Read Only Memory) 904, a RAM 906, a host bus 908, and a bridge 910. Further, this hardware includes an external bus 912, an interface 914, an input unit 916, an output unit 918, a storage unit 920, a drive 922, a connection port 924, and a communication unit 926.

  The CPU 902 functions as, for example, an arithmetic processing unit or a control unit, and controls the overall operation of each component or a part thereof based on various programs recorded in the ROM 904, the RAM 906, the storage unit 920, or the removable recording medium 928. .

  The ROM 904 is an example of a storage device that stores a program read by the CPU 902, data used for calculation, and the like. The RAM 906 temporarily or permanently stores, for example, a program read by the CPU 902 and various parameters that change when the program is executed.

  These elements are connected to each other via, for example, a host bus 908 capable of high-speed data transmission. On the other hand, the host bus 908 is connected to an external bus 912 having a relatively low data transmission speed via a bridge 910, for example.

  As the input unit 916, for example, a mouse, a keyboard, a touch panel, a touch pad, a button, a switch, a lever, or the like is used. Furthermore, as the input unit 916, a remote controller capable of transmitting a control signal using infrared rays or other radio waves may be used.

  The output unit 918 is a video output device that outputs an image signal to a display device such as a CRT, LCD, PDP, or ELD. As the output unit 918, an audio output device such as a speaker or headphones, or a printer may be used. That is, the output unit 918 is a device that can output visual or auditory information.

  The storage unit 920 is a device for storing various data. As the storage unit 920, for example, a magnetic storage device such as an HDD is used. Further, as the storage unit 920, a semiconductor storage device such as an SSD (Solid State Drive) or a RAM disk, an optical storage device, a magneto-optical storage device, or the like may be used.

  The drive 922 is a device that reads information recorded on a removable recording medium 928 that is a removable recording medium or writes information on the removable recording medium 928. As the removable recording medium 928, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is used.

  The connection port 924 is a port for connecting an external connection device 930 such as a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal. For example, a printer or the like is used as the external connection device 930.

  The communication unit 926 is a communication device for connecting to the network 932. As the communication unit 926, for example, a communication circuit for wired or wireless LAN (Local Area Network), a communication circuit for WUSB (Wireless USB), a communication circuit or router for optical communication, an ADSL (Asymmetric Digital Subscriber Line) Communication circuits, routers, communication circuits for mobile phone networks, and the like are used. A network 932 connected to the communication unit 926 is a wired or wireless network, and includes, for example, the Internet, a LAN, a broadcast network, a satellite communication line, and the like.

  Note that the functions of the information processing apparatus 300 to be described later can be realized by the hardware illustrated in FIG. Therefore, detailed description of the hardware of the information processing apparatus 300 is omitted.

[2-3. Control device functions]
Next, functions of the control device 200 will be described with reference to FIG. FIG. 6 is a diagram for explaining functions of the control device according to the second embodiment.

  As illustrated in FIG. 6, the control device 200 includes an information management unit 201, a robot control unit 202, an image acquisition unit 203, a light amount adjustment unit 204, a storage unit 205, a work processing unit 206, and a display control unit 207.

  Note that the functions of the information management unit 201, the robot control unit 202, the image acquisition unit 203, the light amount adjustment unit 204, the work processing unit 206, and the display control unit 207 can be realized using the above-described CPU 902 or the like. The function of the storage unit 205 can be realized using the above-described RAM 906, storage unit 920, and the like.

  The information management unit 201 acquires information input by the operator. Information acquired by the information management unit 201 is input to the robot control unit 202, the light amount adjustment unit 204, and the work processing unit 206. The robot control unit 202 controls the robot 104 according to the information input from the information management unit 201. Further, the robot control unit 202 inputs control information of the robot 104 to the information management unit 201. This control information includes control parameters for the mechanism of the robot 104. The information management unit 201 transmits the control information input from the robot control unit 202 to the information processing apparatus 300.

The control information is stored in the storage unit 205.
The image acquisition unit 203 receives a captured image captured by the camera 101 from the real device 100. The captured image received by the image acquisition unit 203 is stored in the storage unit 205. The light amount adjustment unit 204 controls the light amount of the illumination 103 according to the information input from the information management unit 201. Further, the light amount adjustment unit 204 inputs control information of the illumination 103 to the information management unit 201. This control information includes a control parameter indicating the value of the light amount level set for the illumination 103. The information management unit 201 transmits the control information input from the light amount adjustment unit 204 to the information processing apparatus 300.

The control information is stored in the storage unit 205.
In addition, the information management unit 201 receives information on the inspection visual field calculated by the information processing apparatus 300. Information regarding the inspection visual field received by the information management unit 201 is stored in the storage unit 205.

  The light amount adjustment unit 204 adjusts the light amount level of the illumination 103 so that the inspection range is illuminated with a light amount appropriate for the inspection based on information on the inspection visual field. After the light amount adjustment unit 204 adjusts the light amount level of the illumination 103, the work processing unit 206 executes processing related to work such as inspection. For example, the work processing unit 206 analyzes an illumination pattern reflected in the inspection range of the work WK.

  The display control unit 207 causes the display device 250 to display the captured image stored in the storage unit 205. When the storage unit 205 stores information related to the inspection visual field, the display control unit 207 causes the inspection range to be displayed by being superimposed on the captured image displayed on the display device 250. For example, the display control unit 207 superimposes and displays the inspection range that has been completed by the work processing unit 206 in the past on the captured image.

The function of the control device 200 has been described above.
[2-4. Function of information processing apparatus]
Next, functions of the information processing apparatus 300 will be described with reference to FIG. FIG. 7 is a diagram for explaining functions of the information processing apparatus according to the second embodiment.

  As illustrated in FIG. 7, the information processing apparatus 300 includes an image acquisition unit 301, a storage unit 302, a simulator 303, a feature point position setting unit 305, a three-dimensional coordinate calculation unit 306, a difference calculation unit 307, and a parameter calculation unit 308. Have.

  Note that the functions of the image acquisition unit 301, the simulator 303, the feature point position setting unit 305, the three-dimensional coordinate calculation unit 306, the difference calculation unit 307, and the parameter calculation unit 308 can be realized using the above-described CPU 902 or the like. The functions of the storage unit 302 can be realized using the above-described RAM 906, the storage unit 920, and the like.

  The image acquisition unit 301 acquires a captured image captured by the real device 100 via the control device 200. The captured image acquired by the image acquisition unit 301 is input to the feature point position setting unit 305. The storage unit 302 stores the three-dimensional CAD data of the real device 100. The simulator 303 uses the 3D CAD data stored in the storage unit 302 to generate a virtual device 304 that reproduces the real device 100 in the virtual space. The simulator 303 acquires control information of the real device 100 from the control device 200. The simulator 303 controls the virtual device 304 based on the control information acquired from the control device 200.

  For example, the simulator 303 synchronously controls the robot of the virtual device 304 based on the control information of the robot 104 acquired from the control device 200. Further, the simulator 303 controls the amount of illumination light of the virtual device 304 based on the control information of the illumination 103 acquired from the control device 200. Under such control of the virtual device 304, the environment in the real device 100 is reproduced in the virtual device 304.

  The simulator 303 executes a simulation using the virtual device 304 and generates a captured image captured by the camera 101 of the real device 100. For example, the simulator 303 executes an optical simulation to calculate the reflection of illumination on the work, and generates a captured image (captured image of the virtual device 304) reflecting the calculation result. For example, a rendering method such as a ray tracing method, a radiosity method, a photon mapping method, or an environment mapping method is used for calculating the captured image.

  The captured image of the virtual device 304 generated by the simulator 303 is input to the feature point position setting unit 305. Further, the control parameters A, B, and C of the virtual device 304 used when calculating the captured image are input to the three-dimensional coordinate calculation unit 306. The control parameters A, B, and C may be vector quantities.

The control parameter A is an internal parameter of the camera. For example, the control parameter A is a camera optical axis direction, a lens parameter, or the like.
The control parameter B is an internal parameter of the robot. For example, the control parameter B is a parameter that determines the position and posture of each mechanism of the robot.

  The control parameter C is a parameter that defines the relative relationship between the camera and the robot. For example, the control parameter C is the position and posture of the camera, the position and posture of the robot, etc., based on certain absolute coordinates. Here, for the sake of simplicity, the above control parameters are considered, but control parameters can be added or omitted as appropriate according to the embodiment.

  The feature point position setting unit 305 sets corresponding feature points on the captured image of the real device 100 and the captured image of the virtual device 304. For example, the feature point position setting unit 305 sets the points specified by the operator for both captured images as the feature points.

  Note that the feature point position setting unit 305 may perform edge detection processing on both captured images and set edge positions corresponding to each other as feature points. For example, a search-based method for predicting the local direction of the edge from the gradient direction calculated from the first derivative of the pixel value and searching for a portion where the gradient of the predicted direction is locally maximal, or calculation from the target image A zero-crossing method based method using the second-order differential equation can be used.

Information on the feature points set by the feature point position setting unit 305 is input to the three-dimensional coordinate calculation unit 306 and the difference calculation unit 307. The three-dimensional coordinate calculation unit 306 receives the two-dimensional coordinates x _v of the feature points set on the captured image of the virtual device 304. The difference calculation unit 307 receives the two-dimensional coordinate x _v of the feature point set on the captured image of the virtual device 304 and the two-dimensional coordinate x _r of the feature point set on the captured image of the real device 100. The

The three-dimensional coordinate calculation unit 306 calculates a three-dimensional coordinate Q in the virtual space corresponding to the two-dimensional coordinate x _v input from the feature point position setting unit 305. At this time, the three-dimensional coordinate calculation unit 306 stores the control parameters A, B, and C input from the simulator 303, the two-dimensional coordinate x _v input from the feature point position setting unit 305, and the storage unit 302. 3D CAD data is used.

If the virtual device 304 is expressed using a polygon model and rendered by OpenGL (Open Graphics Library; registered trademark), the 2D coordinate x _v can be easily converted to the 3D coordinate Q by using gluUnProject (). Can be converted to Of course, another similar method can be applied.

As described above, the three-dimensional coordinate Q depends on the control parameters B and C. Therefore, in the following description, the three-dimensional coordinate Q may be expressed as Q (B, C). The control parameters A, B, and C and the three-dimensional coordinate Q calculated by the three-dimensional coordinate calculation unit 306 are input to the parameter calculation unit 308. The difference calculation unit 307 calculates a difference dx (dx = x _v −x _r ) between the two two-dimensional coordinates x _v and x _r . The difference dx calculated by the difference calculation unit 307 is input to the parameter calculation unit 308.

  The parameter calculation unit 308 uses the control parameters A, B, C, the three-dimensional coordinate Q input from the three-dimensional coordinate calculation unit 306, and the difference dx input from the difference calculation unit 307 to control the control parameter error dA. , DB, dC are estimated.

The relationship between the two-dimensional coordinate x _v , the control parameters A, B, C, and the three-dimensional coordinate Q in the virtual device 304 is expressed by the model f shown in the following equation (1). In an ideal state where there is no design error, the real device 100 is also expressed by the same model. From the following equation (1), the minute changes dA, dB, and dC of the control parameters A, B, and C can be expressed as the following equation (2) by the minute change dx of the two-dimensional coordinates. Therefore, by substituting the difference dx between the two-dimensional coordinates x _v and x _r into the following equation (2), a relational expression regarding the control parameter errors dA, dB, and dC is obtained.

  The parameter calculation unit 308 substitutes the control parameters A, B, C, the three-dimensional coordinate Q, and the difference dx input from the difference calculation unit 307 to the above equation (2), and controls the error dA, dB of the control parameter. , DC is generated. The parameter calculation unit 308 estimates the control parameter errors dA, dB, and dC using the least square method using the relational expressions related to the control parameter errors dA, dB, and dC generated for the plurality of captured images.

  Control parameter errors dA, dB, and dC estimated by the parameter calculation unit 308 are stored in the storage unit 302. The simulator 303 corrects the control parameters A, B, and C of the virtual device 304 using the control parameter errors dA, dB, and dC stored in the storage unit 302. For example, the simulator 303 generates the virtual device 304 based on the control parameters A + dA, B + dB, C + dC obtained by applying the errors dA, dB, dC to the control parameters A, B, C acquired from the real device 100. Then, the simulation result in the virtual device 304 generated based on the control parameters A + dA, B + dB, C + dC is fed back to the real device 100 side.

The function of the information processing apparatus 300 has been described above.
[2-5. Application example of calibration method]
Next, an application example of the calibration method according to the second embodiment will be described.

(Basic operation of the system)
First, the basic operation of the system according to the second embodiment will be described with reference to FIG. FIG. 8 is a diagram illustrating an example of an operation performed by the system according to the second embodiment.

  As shown in FIG. 8, in the system according to the second embodiment, a virtual device 304 obtained by virtualizing the real device 100 is generated, and the virtual device 304 is controlled in synchronization with the operation of the real device 100 (S1).

In addition, a simulation is executed under the environment of the virtual device 304, and a captured image captured by the camera of the virtual device 304 is acquired (S2).
For example, the range of illumination reflected on the workpiece is calculated by optical simulation, and the range that satisfies the inspection condition is extracted from the illumination range as the inspection range (S3).

  Information on the extracted inspection range is fed back to the real device 100 side, and is used for control for optimizing the light amount level of illumination in the real device 100 and teaching of the inspection range (S4).

(Examination method of inspection range)
Here, the examination range extraction processing executed by the information processing apparatus 300 will be described with reference to FIG. FIG. 9 is a diagram for explaining the examination range extraction processing executed by the information processing apparatus according to the second embodiment.

  The inspection range is extracted based on the shape of the workpiece and the simulation result. When a visual inspection of a workpiece is performed, a range other than the workpiece is not subject to inspection. In addition, since a workpiece surface having a large angle with respect to the imaging surface of the camera may not be inspected with high accuracy, the workpiece surface is set out of the inspection target. Furthermore, when applying a method of inspecting an illumination pattern such as a mesh pattern on the workpiece surface, a range in which the illumination pattern is reflected is an inspection target.

  The range in which the workpiece is shown and the range of the workpiece surface having a large angle with respect to the imaging surface can be calculated using three-dimensional CAD data. These ranges are extracted as ranges based on the workpiece shape (S31).

  On the other hand, the range in which the illumination pattern is reflected can be calculated by using the result of optical simulation and extracting the range where the brightness of the workpiece surface is equal to or higher than the set brightness. This range is extracted as a range based on the reflected luminance (S32).

The final inspection range is extracted by calculating an overlapping range between the range based on the workpiece shape and the range based on the reflection luminance (S33).
The inspection range is extracted as a two-dimensional range on the captured image, but the information processing device 300 converts the points in the two-dimensional range into three-dimensional coordinate points based on the three-dimensional CAD data, and the format of the three-dimensional CAD data. It may be stored as part data expressed in.

(About error factors)
Here, with reference to FIG. 10, an error factor of an error that occurs between the real device 100 and the virtual device 304 will be described. FIG. 10 is a diagram for explaining error factors of errors that occur between the real device and the virtual device according to the second embodiment.

  The deviation between the real device 100 and the virtual device 304 (three-dimensional CAD data) can be caused by various factors. For example, an error between the real device 100 and the virtual device 304 is caused when the camera optical axis direction, lens parameters, and the position and orientation of the camera with respect to the absolute coordinates set on the real device 100 are deviated from the design values for some reason. .

  Also, errors that occur during the manufacture and assembly of the robot, deviations in the initial value of the motor that drives the robot, and deviations in the position and orientation of the robot with respect to the absolute coordinates set on the real device 100 may also occur between the real device 100 and the virtual device 304. Create an error. In addition, the error position of the lighting which influences the simulation result and the molding / installation error of the work are the error factors.

(Calibration process)
A process for correcting the error as described above is a calibration process. Here, the calibration process according to the second embodiment will be described with reference to FIGS. 11 and 12. FIG. 11 is a diagram for explaining an example of parameters used for the feature point designation method and calibration according to the second embodiment. FIG. 12 is a diagram for explaining an automatic feature point extraction method according to the second embodiment.

  In the calibration processing according to the second embodiment, the first feature point designated on the captured image of the virtual device 304 and the second feature point designated on the captured image of the real device 100 are used. The first feature point and the second feature point are located at portions corresponding to each other. For example, the first feature point and the second feature point are set at the corner of the workpiece.

  The designation of the first feature point and the second feature point may be performed manually by an operator, or is automatically executed by the information processing apparatus 300 using edge detection of a captured image as shown in FIG. May be.

As shown in FIG. 11, when the first feature is specified, two-dimensional coordinates x _v on the captured image showing the position of the first feature points are obtained. Further, control parameters A, B, and C in the virtual device 304 when the captured image is acquired are obtained. In the example of FIG. 11, the control parameter A is called a camera parameter, the control parameter B is called a robot parameter, and the control parameter C is called a parameter related to the relative position between the camera and the robot.

Further, a three-dimensional coordinate Q corresponding to the two-dimensional coordinate x _v is obtained from the three-dimensional CAD data and the control parameters A, B, and C. On the other hand, when the second feature point is designated, a two-dimensional coordinate x _r on the captured image indicating the position of the second feature point is obtained. However, errors dA, dB, and dC between the control parameters A, B, and C of the real device 100 and the control parameters A, B, and C of the virtual device 304 are unknown. Therefore, the information processing apparatus 300 estimates the control parameter errors dA, dB, and dC using the model expressed by the above equation (1) (and the above equation (2)) using the least square method. To do. The estimated control parameter errors dA, dB, and dC are set as control parameters of the virtual device 304. That is, the control parameter of the real device 100 is estimated from the difference in the captured images, and the control parameter of the virtual device 304 is adjusted using the estimated control parameter.

  In the above calibration processing, adjustment of control parameters is realized by using a captured image used for system operation, so that no additional calibration equipment is generated. Further, since the error is detected by comparing the two-dimensional information on the captured image, the processing load is lower than the method of detecting the error by comparing the three-dimensional information on the stereo image. . Therefore, the calibration can be easily realized by applying the calibration processing according to the second embodiment.

[2-6. Flow of calibration process]
Next, the flow of calibration processing according to the second embodiment will be described with reference to FIGS. 13 and 14. FIG. 13 is a flowchart showing the flow of calibration processing executed by the information processing apparatus according to the second embodiment. FIG. 14 is a flowchart showing a parameter calculation processing flow executed in the calibration processing executed by the information processing apparatus according to the second embodiment.

(Overall flow: Fig. 13)
First, the overall flow of the calibration process will be described. Note that the calibration process described here is mainly executed by the information processing apparatus 300.

  (S101) The information processing device 300 controls the state of the real device 100 via the control device 200. For example, the information processing device 300 instructs the control device 200 to change the control parameter. Upon receiving the instruction, the control device 200 changes the control parameter and controls the robot 104 and the camera 101 of the real device 100. Note that the value of the control parameter that is changed by the control device 200 during the calibration process may be randomly determined within a set range, or may be sequentially set according to a preset pattern.

  (S102) The information processing device 300 synchronizes the state of the virtual device 304 with the state of the real device 100 by the function of the simulator 303. For example, the simulator 303 controls the virtual device 304 according to the control parameter transmitted to the real device 100 or the control parameter acquired from the real device 100. The control parameters used for the synchronization control of the virtual device 304 are stored in the storage unit 302.

  (S103) The information processing device 300 acquires a captured image of the real device 100 via the control device 200 by the function of the image acquisition unit 301. Note that the captured image of the real device 100 acquired by the image acquisition unit 301 is input to the feature point position setting unit 305.

  (S104) The information processing apparatus 300 acquires a captured image of the virtual apparatus 304 by the function of the simulator 303. For example, the simulator 303 performs an optical simulation under the environment of the virtual device 304, calculates a range in which the illumination is reflected on the work, and generates a captured image reflecting the reflection of the illumination. Note that the captured image of the virtual device 304 is input to the feature point position setting unit 305. The control parameters used by the simulator 303 are input to the three-dimensional coordinate calculation unit 306.

  (S105) The information processing apparatus 300 determines whether the set number of captured images has been acquired. That is, the information processing apparatus 300 determines whether or not the number of sets of the captured image of the real device 100 and the captured image of the virtual device 304 exceeds a set threshold value. Note that the threshold value is set to a number that allows stable estimation of the control parameters of the real device 100 using the least square method described later. If the number of captured image sets exceeds the threshold, the process proceeds to S106. On the other hand, when the number of sets of captured images does not exceed the threshold, the process returns to S101.

  (S106) The information processing device 300 sets the first feature point on the captured image of the virtual device 304 by the function of the feature point position setting unit 305. For example, the feature point position setting unit 305 sets the point designated on the captured image of the virtual device 304 by the operator as the first feature point. Note that the feature point position setting unit 305 may perform edge detection processing on the captured image of the virtual device 304 and set the detected edge position as the first feature point.

  (S107) The information processing device 300 sets a second feature point corresponding to the first feature point on the captured image of the real device 100 by the function of the feature point position setting unit 305. For example, the feature point position setting unit 305 sets the point designated by the operator on the captured image of the real device 100 as the second feature point. The feature point position setting unit 305 executes edge detection processing on the captured image of the real device 100, detects a point corresponding to the first feature point from the detected edge position, and sets it as the second feature point. May be.

  (S108) The information processing apparatus 300 uses the functions of the feature point position setting unit 305, the three-dimensional coordinate calculation unit 306, the difference calculation unit 307, and the parameter calculation unit 308 to control parameters between the real device 100 and the virtual device 304. Calculate the error of. The process of S108 will be described in detail later.

  (S109) The control parameter error calculated in S108 is input to the simulator 303. The simulator 303 applies the input control parameter error to the control parameter of the virtual device 304. Then, the simulator 303 operates the virtual device 304 based on the set control parameter.

  For example, the simulator 303 applies the errors dA, dB, and dC to the control parameters A, B, and C acquired from the real device 100, and generates the virtual device 304 based on the control parameters A + dA, B + dB, and C + dC. Then, the simulator 303 executes simulation under the environment of the generated virtual device 304 and feeds back the simulation result to the real device 100 side. For example, information indicating the inspection range in the virtual device 304 based on the control parameters A + dA, B + dB, and C + dC is provided to the control device 200.

When the process of S109 is completed, the series of processes illustrated in FIG.
(Parameter calculation flow: Fig. 14)
Next, the process of S108 will be further described. The process described here is mainly executed by the feature point position setting unit 305, the three-dimensional coordinate calculation unit 306, the difference calculation unit 307, and the parameter calculation unit 308.

(S111) The feature point position setting unit 305 acquires the set two-dimensional coordinates x _v of the first feature point. The two-dimensional coordinate x _v of the first feature point set by the feature point position setting unit 305 is input to the three-dimensional coordinate calculation unit 306 and the difference calculation unit 307. The two-dimensional coordinate x _v of the first feature point input to the three-dimensional coordinate calculation unit 306 is converted into the three-dimensional coordinate Q in the virtual space and input to the parameter calculation unit 308.

  (S112, S113, S114) The parameter calculation unit 308 acquires the camera parameter A when the captured image of the virtual device 304 is acquired. Also, the parameter calculation unit 308 acquires the robot parameter B when the captured image of the virtual device 304 is acquired. Further, the parameter calculation unit 308 acquires a parameter C related to the relative position between the camera and the robot in the virtual device 304 when the captured image of the virtual device 304 is acquired.

(S115) The feature point position setting unit 305 acquires the two-dimensional coordinate x _r of the set second feature point. The two-dimensional coordinate x _r of the second feature point set by the feature point position setting unit 305 is input to the difference calculation unit 307.

The difference calculation unit 307 to which the two-dimensional coordinate x _v of the first feature point and the two-dimensional coordinate x _r of the second feature point are input calculates a difference dx between the two-dimensional coordinates x _v and x _r . The difference dx calculated by the difference calculation unit 307 is input to the parameter calculation unit 308. Note that the process of calculating the two-dimensional coordinate x _r of the second feature point can be replaced with the process of S111 to S114.

  (S116) The parameter calculation unit 308 calculates the control parameters A, B, C, the three-dimensional coordinate Q, and the difference dx for a set of a plurality of captured images captured while changing the states of the real device 100 and the virtual device 304. prepare. Next, the parameter calculation unit 308 substitutes the control parameters A, B, C, the three-dimensional coordinate Q, and the difference dx in the above equation (2) to generate a relational expression regarding the control parameter errors dA, dB, dC. . Next, the parameter calculation unit 308 calculates control parameter errors dA, dB, and dC by a least square method using a plurality of relational expressions generated for a plurality of sets of captured images. When the process of S116 is completed, the series of processes shown in FIG.

The flow of the calibration process according to the second embodiment has been described above.
[2-7. Modification: When specifying multiple feature points for each captured image]
Next, a modification of the second embodiment will be described with reference to FIGS. 15 and 16. FIG. 15 is a diagram for explaining an example of a feature point designation method and parameters used for calibration according to a modification of the second embodiment. FIG. 16 is a flowchart illustrating a parameter calculation processing flow executed in the calibration processing executed by the information processing apparatus according to the modification of the second embodiment.

(Feature point designation method: Fig. 15)
Until now, the description has been made by taking as an example the case of specifying one feature point for one captured image. However, it is also possible to specify a plurality of feature points for one captured image. For example, as shown in FIG. 15, three first feature points p ₁₁ , p ₁₂ , and p ₁₃ can be designated from the captured image of the virtual device 304. In this case, three second feature points p ₂₁ , p ₂₂ , and p ₂₃ corresponding to the three first feature points p ₁₁ , p ₁₂ , and p ₁₃ , respectively, are designated from the captured image of the real device 100.

As shown in FIG. 15, when the first feature is specified, two-dimensional coordinates x _v on the captured image showing the position of the first feature points are obtained. Further, control parameters A, B, and C in the virtual device 304 when the captured image is acquired are obtained. The control parameter A is a camera parameter, the control parameter B is a robot parameter, and the control parameter C is a parameter related to the relative position between the camera and the robot.

Further, a three-dimensional coordinate Q corresponding to the two-dimensional coordinate x _v is obtained from the three-dimensional CAD data and the control parameters A, B, and C. On the other hand, when the second feature point is designated, a two-dimensional coordinate x _r on the captured image indicating the position of the second feature point is obtained.

  However, the control parameter errors dA, dB, and dC between the real device 100 and the virtual device 304 are unknown. Therefore, the information processing apparatus 300 estimates the control parameter errors dA, dB, and dC using the model expressed by the above equation (1) (and the above equation (2)) using the least square method. To do. The estimated control parameter errors dA, dB, and dC are applied to the control parameters A, B, and C of the virtual device 304. That is, as in the case of designating one feature point for one captured image, the control parameter errors dA, dB, and dC are estimated from the difference in the captured images, and the estimated control parameter errors dA, dB, and dC are estimated. Is used to adjust the control parameters of the virtual device 304.

  In the above calibration processing, adjustment of control parameters is realized by using a captured image used for system operation, so that no additional calibration equipment is generated. Further, since the error is detected by comparing the two-dimensional information on the captured image, the processing load is lower than the method of detecting the error by comparing the three-dimensional information on the stereo image. . Therefore, calibration can be realized easily.

  Furthermore, since a plurality of feature points are designated on one captured image, the control parameter errors dA, dB, and dC can be estimated with a small number of captured images. In other words, by applying this modification, it is possible to further reduce the load and time required for the control and imaging of the real device 100 and the virtual device 304.

(Parameter calculation flow: Fig. 16)
A processing flow of parameter calculation according to this modification will be described. This process corresponds to the process of S108 described above. The processing described here is mainly executed by the feature point position setting unit 305, the three-dimensional coordinate calculation unit 306, the difference calculation unit 307, and the parameter calculation unit 308.

  (S121) The feature point position setting unit 305 reads from the storage unit 302 information on the first feature point that has a low contribution to the calculation of the control parameter errors dA, dB, and dC executed in the past, and reads the read first feature. The point is excluded from the calculation target to be executed this time. Of course, the feature point position setting unit 305 also excludes the second feature point corresponding to the excluded first feature point from the calculation target. The calculation of the contribution rate will be described later.

(S122) The feature point position setting unit 305 acquires the set two-dimensional coordinates x _v of the first feature point. The two-dimensional coordinate x _v of the first feature point set by the feature point position setting unit 305 is input to the three-dimensional coordinate calculation unit 306 and the difference calculation unit 307. The two-dimensional coordinate x _v of the first feature point input to the three-dimensional coordinate calculation unit 306 is converted into the three-dimensional coordinate Q in the virtual space and input to the parameter calculation unit 308.

  (S123, S124, S125) The parameter calculation unit 308 acquires the camera parameter A when the captured image of the virtual device 304 is acquired. Also, the parameter calculation unit 308 acquires the robot parameter B when the captured image of the virtual device 304 is acquired. Further, the parameter calculation unit 308 acquires a parameter C related to the relative position between the camera and the robot in the virtual device 304 when the captured image of the virtual device 304 is acquired.

(S126) the feature point position setting unit 305 acquires the two-dimensional coordinates x _r of the second feature point set. The two-dimensional coordinate x _r of the second feature point set by the feature point position setting unit 305 is input to the difference calculation unit 307.

The difference calculation unit 307 to which the two-dimensional coordinate x _v of the first feature point and the two-dimensional coordinate x _r of the second feature point are input calculates a difference dx between the two-dimensional coordinates x _v and x _r . The difference dx calculated by the difference calculation unit 307 is input to the parameter calculation unit 308. Note that the process of calculating the two-dimensional coordinate x _r of the second feature point can be replaced with the process of S122 to S125.

  (S127) The parameter calculation unit 308 calculates the control parameters A, B, C, the three-dimensional coordinate Q, and the difference dx for a set of a plurality of captured images captured while changing the states of the real device 100 and the virtual device 304. prepare. Next, the parameter calculation unit 308 substitutes the control parameters A, B, C, the three-dimensional coordinate Q, and the difference dx in the above equation (2) to generate a relational expression regarding the control parameter errors dA, dB, dC. . Next, the parameter calculation unit 308 calculates control parameter errors dA, dB, and dC by a least square method using a plurality of relational expressions generated for a plurality of sets of captured images.

  (S128) The parameter calculation unit 308 calculates a stability index indicating the influence of stability when calculating the errors dA, dB, and dC by the least square method. For example, the parameter calculation unit 308 performs singular value decomposition on a matrix (corresponding to the coefficient matrix in the above relational expression) used when calculating the errors dA, dB, and dC by the least square method, and the ratio between the minimum singular value and the maximum singular value. Is the stability index.

The parameter calculation unit 308 notifies the operator of the calculated stability index. For example, the parameter calculation unit 308 displays the stability index on the display device 350.
(S129) The parameter calculation unit 308 calculates a contribution rate for the calculation of the errors dA, dB, and dC for each feature point. This contribution rate is sometimes called a determination coefficient. For example, a value obtained by subtracting a value obtained by dividing the sum of squares of residuals by the sum of squares of deviations from 1 is called a contribution rate. The parameter calculation unit 308 extracts a set of first feature points and second feature points whose contribution rate is lower than the set threshold value, and stores the extracted information of the first feature points and the second feature points in the storage unit 302. . When the process of S129 is completed, the series of processes illustrated in FIG.

The second embodiment has been described above.
In the above description, the case where the type of source used for inspection is assumed to be light (illumination) and optical simulation is executed has been described as an example. However, even if the type of source used for inspection changes, the second embodiment is concerned. The technology is applicable as well.

  For example, when the source is a sound, the technology described above can be applied by installing a sound source and a microphone in a real device and using acoustic simulation. When the source is an electromagnetic field, the above-described technique can be applied by installing an electromagnetic field generation source and an electromagnetic field sensor in a real device and using electromagnetic field analysis simulation.

  As mentioned above, although preferred embodiment was described referring an accompanying drawing, this invention is not limited to the example which concerns. It is obvious for a person skilled in the art that various variations and modifications can be conceived within the scope of the claims, and such variations and modifications are naturally understood by the technical scope of the present invention. It goes without saying that it belongs to a range.

<3. Addendum>
The following additional notes are disclosed with respect to the embodiment described above.
(Additional remark 1) The memory | storage part which memorize | stores the three-dimensional information of the real apparatus containing a target object, the robot in which the said target object is installed, and the camera which images at least one part of the said target object and the said robot,
A virtual device that reproduces the real device in a virtual space based on the three-dimensional information is generated, the operation of the virtual device is synchronized with the operation of the real device, and the first image captured by the camera of the virtual device And a calculation unit that adjusts control parameters of the virtual device based on a difference between the second image captured by the real device and the first image corresponding to the second image;
An information processing apparatus.

(Additional remark 2) The said memory | storage part is model information which shows the relationship between the two-dimensional coordinate in the said 1st image, the three-dimensional coordinate in the said virtual space corresponding to the said two-dimensional coordinate, and the control parameter of the said virtual apparatus. Remember
The computing unit includes two-dimensional coordinates of a first feature point set in the first image and a virtual space corresponding to the first feature point for a plurality of sets of the first image and the second image. 3D coordinates, 2D coordinates of the second feature point in the second image corresponding to the first feature point, and control parameters of the virtual device at the time of imaging the first image are acquired. The information processing device according to claim 1, wherein an error between the control parameter of the real device and the control parameter of the virtual device is calculated based on the model information.

(Additional remark 3) The said calculating part extracts the corner | angular part of the said target object or the said robot using the said three-dimensional information, and sets the two-dimensional coordinate in the said 1st image corresponding to the extracted said corner | angular part to the said 1st. The information processing apparatus according to attachment 2, wherein the information processing apparatus sets the two-dimensional coordinates of one feature point.

(Supplementary Note 4) The information processing apparatus according to Supplementary Note 2 or 3, wherein a plurality of two-dimensional coordinates of the first feature point are set for one first image.

(Additional remark 5) When calculating the said error, the said calculation part calculates the contribution to a calculation result about each of the two-dimensional coordinate of the said 1st feature point, and the said contribution is smaller than the set threshold value The information processing apparatus according to claim 4, wherein information of the first feature point is stored in the storage unit, and the use of the two-dimensional coordinates of the first feature point having a small contribution is excluded when the error is calculated later. .

(Additional remark 6) The computer which has a memory which memorize | stores the three-dimensional information of the real apparatus containing a target object, the robot in which the said target object is installed, and the camera which images at least one part of the said target object and the said robot,
A virtual device that reproduces the real device in a virtual space based on the three-dimensional information is generated, the operation of the virtual device is synchronized with the operation of the real device, and the first image captured by the camera of the virtual device A control method of adjusting the control parameter of the virtual device based on a difference between the second image captured by the real device and the first image corresponding to the second image.

(Supplementary Note 7) The memory stores model information indicating a relationship between two-dimensional coordinates in the first image, three-dimensional coordinates in the virtual space corresponding to the two-dimensional coordinates, and control parameters of the virtual device. Remember,
The processor of the computer, for a plurality of sets of the first image and the second image, two-dimensional coordinates of a first feature point set in the first image and the virtual corresponding to the first feature point Acquire three-dimensional coordinates in space, two-dimensional coordinates of second feature points in the second image corresponding to the first feature points, and control parameters of the virtual device at the time of imaging the first image The control method according to claim 6, wherein an error between the control parameter of the real device and the control parameter of the virtual device is calculated based on the model information.

(Additional remark 8) The said processor extracts the corner | angular part of the said target object or the said robot using the said three-dimensional information, The two-dimensional coordinate in the said 1st image corresponding to the extracted said corner | angular part is said 1st. The control method according to attachment 7, wherein the two-dimensional coordinates of the feature points are set.

(Supplementary note 9) The control method according to supplementary note 7 or 8, wherein a plurality of two-dimensional coordinates of the first feature point are set for one first image.

(Supplementary Note 10) When the processor calculates the error, the processor calculates a contribution to the calculation result for each of the two-dimensional coordinates of the first feature point, and the contribution is smaller than a set threshold value. The control method according to claim 9, wherein information of one feature point is stored in the memory, and when the error is calculated later, the two-dimensional coordinates of the first feature point having a small contribution are not used.

(Additional remark 11) The computer which has a memory which memorize | stores the three-dimensional information of the real apparatus containing the target object, the robot in which the said target object is installed, and the camera which images at least one part of the said target object and the said robot,
A virtual device that reproduces the real device in a virtual space based on the three-dimensional information is generated, the operation of the virtual device is synchronized with the operation of the real device, and the first image captured by the camera of the virtual device And executing a process of adjusting a control parameter of the virtual device based on a difference between the second image captured by the real device and the first image corresponding to the second image.

DESCRIPTION OF SYMBOLS 10 Real apparatus 20 Information processing apparatus 21 Memory | storage part 22 Calculation part 23 Virtual apparatus 24 Camera 25 Robot OBJ Object A, B Control parameter dA, dB Error P1 1st image P2 2nd image

Claims (7)

A storage unit for storing three-dimensional information of a real device including an object, a robot on which the object is installed, and a camera that images at least a part of the object and the robot;
A virtual device that reproduces the real device in a virtual space based on the three-dimensional information is generated, the operation of the virtual device is synchronized with the operation of the real device, and the first image captured by the camera of the virtual device And a calculation unit that adjusts control parameters of the virtual device based on a difference between the second image captured by the real device and the first image corresponding to the second image;
An information processing apparatus. The storage unit stores model information indicating a relationship between two-dimensional coordinates in the first image, three-dimensional coordinates in the virtual space corresponding to the two-dimensional coordinates, and control parameters of the virtual device. And
The computing unit includes two-dimensional coordinates of a first feature point set in the first image and a virtual space corresponding to the first feature point for a plurality of sets of the first image and the second image. 3D coordinates, 2D coordinates of the second feature point in the second image corresponding to the first feature point, and control parameters of the virtual device at the time of imaging the first image are acquired. The information processing apparatus according to claim 1, wherein an error between a control parameter of the real device and a control parameter of the virtual device is calculated based on the model information. The calculation unit extracts a corner of the object or the robot using the three-dimensional information, and calculates a two-dimensional coordinate in the first image corresponding to the extracted corner of the first feature point. The information processing apparatus according to claim 2, wherein the information processing apparatus is set to a two-dimensional coordinate. The information processing apparatus according to claim 2, wherein a plurality of two-dimensional coordinates of the first feature point are set for one first image. The calculation unit calculates a contribution to the calculation result for each of the two-dimensional coordinates of the first feature point when calculating the error, and the first feature point has a smaller contribution than a set threshold. The information processing apparatus according to claim 4, wherein the information is stored in the storage unit, and when the error is calculated later, use of the two-dimensional coordinates of the first feature point having a small contribution is excluded. A computer having a memory for storing three-dimensional information of a real device including an object, a robot on which the object is installed, and a camera that images at least a part of the object and the robot,
A virtual device that reproduces the real device in a virtual space based on the three-dimensional information is generated, the operation of the virtual device is synchronized with the operation of the real device, and the first image captured by the camera of the virtual device A control method of adjusting the control parameter of the virtual device based on a difference between the second image captured by the real device and the first image corresponding to the second image. A computer having a memory for storing three-dimensional information of a real device including an object, a robot on which the object is installed, and a camera that images at least a part of the object and the robot;
A virtual device that reproduces the real device in a virtual space based on the three-dimensional information is generated, the operation of the virtual device is synchronized with the operation of the real device, and the first image captured by the camera of the virtual device And executing a process of adjusting a control parameter of the virtual device based on a difference between the second image captured by the real device and the first image corresponding to the second image.

Images