JP2015232464A - Calibration device, calibration method, and calibration program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a calibration device capable of calibrating design data with respect to parts or the like outside a camera visual field range, a calibration method and a calibration program.SOLUTION: The calibration device includes a first target 70 which is arranged in the visual field range of an imaging apparatus 10, according to the design data and has a mirror surface, a second target 80 which is arranged outside the visual field range of the imaging apparatus, according to the design data and captured by the imaging apparatus through the mirror surface, an estimation part 54 which estimates information relating to the second target on the basis of an image obtained by the imaging apparatus, and a calibration part 62 which calibrates the design data on the basis of the information estimated by the estimation part.

Description

  The present case relates to a calibration apparatus, a calibration method, and a calibration program.

  In recent years, digitization in the manufacturing field has progressed, and design data can be used not only for design and design processes, but also for processes such as prototyping, manufacturing, sales, and services, to improve the efficiency and labor savings of the entire PLM (Product Life cycle Management). The movement to plan is progressing. For example, a virtual device of an inspection device or an assembly device is generated on a PC according to design data, the teaching operation of the real device is performed offline, and the operation of the real device is simulated, thereby improving the operational efficiency of the real device Can do.

  When performing offline teaching or operation simulation of a real device using a virtual device, it is preferable to reduce the difference between the virtual device and the real device. Here, the virtual device can be generated without any difference from the design data, but the real device is not necessarily manufactured to match the design data, and there is an error with the design data. Therefore, after the virtual device is generated according to the design data, it is desirable to perform an operation for correcting the design data on the virtual device side (hereinafter referred to as virtual device calibration) so as to eliminate the difference from the real device. Non-Patent Document 1 discloses a camera calibration method related to an apparatus including a camera.

Z. Zhang.A flexible new technique for camera calibration.IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (11): 1330-1334, 2000.

  However, with the technique of Non-Patent Document 1, calibration cannot be performed for components and the like arranged outside the field of view of the camera.

  In one aspect, an object of the present invention is to provide a calibration apparatus, a calibration method, and a calibration program that can calibrate design data with respect to a component or the like outside the camera field of view.

  In one aspect, the calibration device is arranged in the field of view of the imaging device according to design data, and has a first target having a mirror surface, and is arranged outside the field of vision of the imaging device according to the design data, and the A second target imaged by the imaging device; an estimation unit for estimating information on the second target based on an image obtained by the imaging device; and the design based on the information estimated by the estimation unit. A calibration unit for calibrating the data.

  In one aspect, the calibration method includes: a first target having a mirror surface disposed in a field of view of an imaging device according to design data; and a second target disposed outside the field of view of the imaging device according to the design data. An image is captured using a mirror surface, information on the second target is estimated based on an image obtained by the imaging, and the design data is calibrated based on the estimated information.

  In one aspect, the calibration program includes a first target that is arranged in the field of view of the imaging device according to design data and has a mirror surface, and a second target that is arranged outside the field of vision of the imaging device according to the design data. A process for imaging using a mirror surface, a process for estimating information on the second target based on the image obtained by the imaging, and a process for calibrating the design data based on the estimated information And make the computer execute.

  Design data can be calibrated for parts outside the camera field of view.

It is a figure which illustrates a calibration target. It is a figure illustrated about virtual device calibration concerning a comparative example. It is a figure which illustrates arrangement | positioning of components. 1 is a diagram illustrating an overall configuration of a simulation apparatus according to Embodiment 1. FIG. (A) And (b) is a perspective view which illustrates the 1st target and the 2nd target. It is a figure explaining the principle of the calibration using a simulation apparatus. It is an example of the flowchart explaining the detail of a calibration. It is a perspective view which illustrates other examples of the 1st target and the 2nd target. It is a figure explaining the principle of calibration. It is a figure explaining the principle of calibration. This is another example of the first target. It is a block diagram for demonstrating a hardware configuration.

  Prior to the description of the embodiment, an outline of calibration (comparative example) for a simulation apparatus that generates a virtual apparatus for a real apparatus will be described.

  The real device is a device including a camera. For example, a real device is an assembly device that detects the position / orientation of a component with a CCD camera, a CMOS camera, etc., and assembles the component based on the information, or an inspection / measurement device that inspects / measures a defect of the component, etc. is there. The calibration of the virtual device generated according to the design data of the real device is performed using, for example, a camera and a calibration target. A calibration target is a mark on which a mark whose position and dimensions are known is printed or arranged. For example, as illustrated in FIG. 1, the calibration target 101 has a surface with a mark 102 with a known positional relationship. The virtual device calibration procedure using the camera is shown below.

  FIG. 2 is a diagram illustrating the virtual device calibration according to the comparative example. As illustrated in FIG. 2, the real device includes a camera 103, a stage mechanism 104, and the like. In the virtual device calibration, the calibration target 101 is placed on the stage mechanism 104 so that the surface with the mark 102 faces the camera 103. Next, the mark 102 is imaged by the camera 103 while the stage mechanism 104 changes the posture of the calibration target 101 in plural.

Next, two-dimensional coordinates of the mark 102 in the image 105 obtained by imaging with the camera 103 in a plurality of postures are obtained, and internal parameters (focal length, principal point position, etc.) of the camera 103 are estimated from the obtained coordinates. For example, the technique of nonpatent literature 1 can be used. Next, to estimate the internal parameters of the camera 103, and a two-dimensional coordinate of the mark 102 in a plurality posture, a three-dimensional coordinate ^p _{m n} of the mark 102 relative to the coordinates of the camera 103. Here, “m” is the number of captured postures, and “n” is the number of marks 102 used for calibration.

Next, in the personal computer (PC) 106, the virtual calibration target 107, the virtual camera 108, and the virtual stage mechanism 109 are virtually generated according to the design data (design values, operation specifications, etc.) of the real device. The virtual calibration target 107 is obtained by virtually generating the calibration target 101 according to the design data. The virtual camera 108 is obtained by virtually generating the camera 103 according to the design data. The virtual stage mechanism 109 is a virtual generation of the stage mechanism 104 according to the design data. Next, the virtual camera 108 captures an image of the mark while changing the posture of the virtual calibration target 107 to plural by the virtual stage mechanism 109. Next, determine the two-dimensional coordinates of the mark in the virtual image 110 in the plurality of pose estimates the three-dimensional coordinates P ^m _n of the mark from the obtained coordinates relative to the coordinates of the virtual camera 108.

Next, virtual device calibration is performed. Specifically, the difference between the 3-dimensional coordinates P ^m _n of the mark in the virtual device and the three-dimensional coordinates p ^m _n of the mark in the real device to calibrate the design data so as to reduce. For example, a calibration parameter c that minimizes the following function is obtained by optimization processing and reflected in design data. Note that the calibration parameter c is a parameter group necessary for calibrating the difference between the real device and the virtual device, such as the mechanical system base coordinates and the origin position of each mechanical system.

  However, with this method, calibration cannot be performed for components outside the visual field range of the camera 103. For example, as illustrated in FIG. 3, the illumination component 111 that irradiates the stage mechanism 104 with light is disposed outside the field of view of the camera 103, and thus cannot be a calibration target. Therefore, it is conceivable to install a plurality of calibration cameras. However, there are problems that it is necessary to add a dedicated camera for calibration, there is a case where there is no space for installing a plurality of cameras, and it is necessary to perform calibration between a plurality of cameras. Therefore, in the following embodiments, a calibration apparatus, a calibration method, and a calibration program capable of performing calibration on components outside the camera field of view will be described.

  FIG. 4 is a diagram illustrating the overall configuration of the simulation apparatus 100 according to the first embodiment. As illustrated in FIG. 4, the simulation apparatus 100 includes a camera 10, a stage mechanism 20, an illumination 30, a stage mechanism 40, a PC 50, a PC 60, and the like. The camera 10, the stage mechanisms 20 and 40, and the illumination 30 function as real devices and are created according to design data (design values, operation specifications, etc.).

  The camera 10 and the illumination 30 are devices for performing assembly, inspection, measurement, and the like, for example. The stage mechanisms 20 and 40 are, for example, devices for installing and moving a sample. The stage mechanism 20 is installed within the visual field range of the camera 10. The illumination 30 and the stage mechanism 40 are installed outside the visual field range of the camera 10. In the present embodiment, the illumination 30 can be moved by the stage mechanism 40.

  The PC 50 has a function as a control unit that controls each unit of the real device, and functions as a camera control unit 51, a mechanism control unit 52, an image processing control unit 53, and a mark coordinate estimation unit 54. The PC 60 has a function as a control unit that generates and controls a virtual device, and functions as an optimization calculation unit 61 and a virtual device calibration unit 62. In the simulation apparatus 100, the simulation can be performed by operating the virtual apparatus using the PC 60. On the other hand, when performing virtual device calibration, the simulation device 100 functions as a calibration device.

  When the simulation apparatus 100 is used as a calibration apparatus, the first target 70 is placed on the stage mechanism 20 and the second target 80 is placed on the stage mechanism 40. Since the stage mechanisms 20 and 40 are created according to the design data, the first target 70 is arranged on the stage mechanism 20 according to the design data, and the second target 80 is arranged on the stage mechanism 40 according to the design data. . Both the first target 70 and the second target 80 are calibration targets. Further, the PC 60 generates a virtual device according to the design data. The PC 60 corresponds to the camera 10, the stage mechanisms 20, 40, the illumination 30, the first target 70, and the second target 80, and the virtual camera 10a, the virtual stage mechanisms 20a, 40a, the virtual illumination 30a, the virtual first target 70a, and A virtual second target 80a is generated.

  FIG. 5A is a perspective view illustrating the first target 70. FIG. 5B is a perspective view illustrating the second target 80. As illustrated in FIGS. 5A and 5B, in this embodiment, two types of calibration targets are used. The first target 70 has a configuration in which a plurality of diffusion surface marks 72 are provided on a mirror surface 71. The second target 80 has a configuration in which a plurality of diffusion surface marks 82 are provided on the diffusion surface 81.

  FIG. 6 is a diagram for explaining the principle of calibration using the simulation apparatus 100. As illustrated in FIG. 6, the first target 70 is placed on the stage mechanism 20, and the second target 80 is placed on the stage mechanism 40 via the illumination 30. Further, the stage mechanisms 20 and 40 are controlled so that the second target 80 is reflected by the mirror surface 71 of the first target 70 and can be imaged by the camera 10. Next, the mark 72 and the mark 82 are separated and extracted from the obtained image 90, and the three-dimensional coordinates of the mark 72 are estimated. Further, the three-dimensional coordinates of the virtual image 80b of the mark 82 on the optical axis of the camera 10 are estimated. The position / orientation (normal line) of the first target 70 is calculated from the three-dimensional coordinates of the mark 72. Information on the second target 80 is estimated from the position and orientation of the first target 70 and the three-dimensional coordinates of the virtual image of the mark 82. For example, as information related to the second target 80, the three-dimensional coordinates of the mark 82, the attitude of the second target 80, and the like can be cited. Next, the design data of the real device (design data for generating the virtual device) is calibrated so that the difference between the real device and the virtual device becomes small regarding the information related to the second target 80.

  FIG. 7 is an example of a flowchart for explaining the details of the above calibration. The mechanism control unit 52 controls the stage mechanism 20 and the stage mechanism 40 so that the second target 80 is reflected by the mirror surface 71 of the first target 70 and can be imaged by the camera 10 (step S1). Next, the camera control unit 51 controls the camera 10 so that the first target 70 and the second target 80 are imaged (step S2). Thereby, a real image of the first target and a mirror image of the second target are captured.

  Next, the image processing control unit 53 performs a process of separately separating and extracting the mark 72 of the first target 70 and the mark 82 of the second target 80 from the image obtained in step S2 (step S3). Note that the color of the mark 72 and the color of the mark 82 may be different so that the mark 72 and the mark 82 can be easily separated.

Next, the mark coordinate estimation unit 54 estimates the three-dimensional coordinates of the mark 72 from the extracted two-dimensional coordinates of the mark 72, so that the position and orientation (method) of the first target 70 with respect to the position of the camera 10 is used. (Line vector) is estimated (step S4). Next, the mark coordinate estimation unit 54 estimates a virtual image three-dimensional coordinate q ′ _{n based} on the position of the camera 10 from the extracted two-dimensional coordinates of the mark 82 (step S5). “N” is the number of marks 82. In camera calibration, mark design data (mark size and relative position) is required. However, in this embodiment, since the mirror image of the second target 80 is handled, the right and left are reversed from the design of the regular second target 80. ing. For this reason, in this embodiment, camera calibration is performed by temporarily generating data obtained by horizontally inverting regular design data.

Next, the mark coordinate estimation unit 54 estimates the three-dimensional coordinate q _n of the mark 82 of the second target 80 with reference to the position of the camera 10 from the information obtained in steps S4 and S5 (step S6). . Next, the optimization calculation unit 61 optimizes the calibration parameter c so that the following function including the three-dimensional coordinate q _n and the three-dimensional coordinate Q _n of the mark 82 in the virtual device is minimized. Obtained (step S7). Next, the virtual device calibration unit 62 calibrates the design data using the obtained calibration parameter c (step S8).

  According to this embodiment, since the second target outside the field of view of the camera is imaged through the mirror surface of the first target within the field of view, the design data is calibrated with respect to components outside the field of view of the camera. be able to. In addition, since the position and orientation of the first target are estimated based on the image obtained by imaging, and the three-dimensional coordinates of the second target are estimated from the position and orientation and the virtual image of the second target, the first target can be obtained with high accuracy. Two-dimensional coordinates of two targets can be estimated. Since the mark of the diffusion surface is attached to the mirror surface of the first target, the mark is attached to the diffusion surface of the second target, and these marks in the image are used, the three-dimensional coordinates of the second target are estimated with high accuracy. Can do.

  In the above example, images in a plurality of postures are not acquired, but the postures of the first target 70 and the second target 80 are within a range that can be imaged by both the first target 70 and the second target 80. It may be varied. In this case, the difference between the real device and the virtual device can be further reduced. Moreover, you may use one apparatus as PC50 and PC60.

(Modification)
When the first target 70 and the second target 80 have the same size, the specular image of the second target 80 located far from the camera 10 appears smaller than the first target 70 due to the influence of the lens perspective. Further, if the focal position is adjusted so that the focus is in the vicinity of the first target 70, the second target 80 may become a blurred image. If the image appears small and is further affected by blurring, it is very difficult to detect the two-dimensional coordinates of the mark 82 of the second target 80.

  Therefore, as illustrated in FIG. 8, it is preferable to make the mark 82 larger than the mark 72 by making the second target 80 larger than the first target 70. In this case, as illustrated in FIG. 9, even if the distance between the camera 10 and the virtual image 80b is larger than the distance between the camera 10 and the first target 70, as illustrated in FIG. 82 can be the same size. Alternatively, the mark size may be set so that the mark 82 looks larger than the mark 72 in consideration of the influence of blur. In this case, design data for the first target and the second target are prepared, and the design data to be used is switched as appropriate according to the calibration of each target.

  FIG. 11 is another example of the first target 70. When the diffusing surface mark 72 is attached to the mirror surface 71 of the first target 70, the second target 80 is not reflected on the mark 72. Therefore, as shown in FIG. 11, when the first target 70 is formed so that the half mirror 74 is coupled to the diffusing surface 73 with the mark 72, the first target 70 and the second target 80 are mixed and imaged. Is done. In this case, since both marks are not shielded from each other, the mark position can be detected with high accuracy. However, it is preferable that the colors of the mark 72 and the mark 82 are not similar colors.

  FIG. 12 is a block diagram for explaining the hardware configuration of the camera control unit 51, mechanism control unit 52, image processing control unit 53, mark coordinate estimation unit 54, optimization calculation unit 61, and virtual device calibration unit 62. . Referring to FIG. 12, the camera control unit 51, the mechanism control unit 52, the image processing control unit 53, the mark coordinate estimation unit 54, the optimization calculation unit 61, and the virtual device calibration unit 62 include a CPU 201, a RAM 202, a storage device 203, A display device 204 and the like are provided.

  A CPU (Central Processing Unit) 201 is a central processing unit. The CPU 201 includes one or more cores. A RAM (Random Access Memory) 202 is a volatile memory that temporarily stores programs executed by the CPU 201, data processed by the CPU 201, and the like. The storage device 203 is a nonvolatile storage device. As the storage device 203, for example, a ROM (Read Only Memory), a solid state drive (SSD) such as a flash memory, a hard disk driven by a hard disk drive, or the like can be used. The storage device 203 stores a calibration program. The display device 204 is a liquid crystal display, an electroluminescence panel, or the like, and displays a virtual device, a calibration result, and the like. In the above example, the camera control unit 51, the mechanism control unit 52, the image processing control unit 53, the mark coordinate estimation unit 54, the optimization calculation unit 61, and the virtual device calibration unit 62 are realized by executing a program. Hardware such as a circuit may be used.

  In the above example, the camera 10 is an example of an imaging device. The mark coordinate estimation unit 54 functions as an estimation unit that estimates the three-dimensional coordinates of the second target, and the virtual device configuration unit 62 functions as a calibration unit that calibrates design data based on the three-dimensional coordinates estimated by the estimation unit. Function.

  Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and changes can be made within the scope of the gist of the present invention described in the claims. It can be changed.

DESCRIPTION OF SYMBOLS 10 Camera 20, 40 Stage mechanism 30 Illumination 51 Camera control part 52 Mechanism control part 53 Image processing control part 54 Mark coordinate estimation part 61 Optimization calculation part 62 Virtual apparatus calibration part 70 1st target 71 Mirror surface 72, 82 Mark 80 2nd Target 81 Diffusion surface

Claims (7)

A first target having a mirror surface disposed in the field of view of the imaging device according to the design data;
A second target arranged outside the field of view of the imaging device according to the design data and imaged by the imaging device through the mirror surface;
An estimation unit that estimates information on the second target based on an image obtained by the imaging device;
A calibration device comprising: a calibration unit that calibrates the design data based on information estimated by the estimation unit.   The estimation unit estimates the position and orientation of the first target based on an image obtained by the imaging device, and estimates the position of the second target from the position and a virtual image of the second target. The calibration device according to claim 1. The mirror surface of the first target is marked with a diffusion surface mark.
Any surface of the second target imaged by the imaging device is a diffusion surface, and the diffusion surface is marked.
The estimation unit estimates a position of the mark of the second target based on the mark of the first target and the mark of the second target included in an image obtained by the imaging device. Item 3. The calibration device according to item 1 or 2.   4. The calibration apparatus according to claim 3, wherein the mirror surface of the first target is a surface of a half mirror attached to a diffusion surface with a mark.   The calibration apparatus according to claim 3 or 4, wherein the mark of the second target is larger than the mark of the first target. A first target having a mirror surface arranged in the field of view of the imaging device according to the design data and a second target arranged outside the field of view of the imaging device in accordance with the design data are imaged using the mirror surface,
Based on the image obtained by the imaging, information related to the second target is estimated,
A calibration method, wherein the design data is calibrated based on the estimated information. A process of imaging, using the mirror surface, a first target that is arranged in the field of view of the imaging device according to design data and has a mirror surface, and a second target that is arranged outside the field of vision of the imaging device according to the design data; ,
A process of estimating information on the second target based on an image obtained by the imaging;
A calibration program for causing a computer to execute a process of calibrating the design data based on the estimated information.

Images