JP6410411B2 - Pattern matching apparatus and pattern matching method - Google Patents

Description

  The present invention relates to a pattern matching apparatus and a pattern matching method for performing pattern matching between images based on a template image to be detected and a captured image including the detection target captured by a camera.

  Conventionally, in the field of image processing, a pattern matching method is well known as a method for detecting the position of a workpiece or the like to be detected. Among them, shape pattern matching is widely used because it is robust against illumination changes, object hiding and chipping, and has high robustness. There are various methods for calculating the similarity in pattern matching, such as the sum of squares of luminance difference (SSD) and normalized cross-correlation (NCC). There are many comparisons. In order to detect the template image from the photographed image, the template image needs to be accurately overlapped with the detection part of the photographed image, so the lens position and direction and the distance between the camera and the work must be fixed. .

  However, when using this pattern matching method in production equipment, etc., the position and direction of the lens, the distance between the camera and the work, etc. due to changes over time, camera and lens replacement, collisions of equipment and workers with the camera and work, etc. The shooting parameters may change. Here, shooting parameters mean internal parameters (lens distortion, focal length, pixel pitch / number of pixels, center position of image optical axis, etc.) and external parameters (distance between camera and workpiece, relative position and orientation) related to the shooting camera. To do. If the shooting parameters change between when the template image is shot and when the shot image is shot, the distortion and size of the work in each image will change, and the similarity will be low even if the template image and the shot image are overlaid. , The accuracy of detecting the workpiece is reduced.

  It is also conceivable to regenerate the template image when the shooting parameters at the time of shooting of the shot image change in this way. However, in order to generate a template image that can perform high-precision pattern matching, it is necessary for an operator with knowledge of image processing to spend a long time to generate a template image. It takes time and effort.

  Therefore, conventionally, a pattern matching apparatus using the following technique is known (see Patent Document 1). In this pattern matching apparatus, the resolution, distortion, and the like of an image obtained by photographing a workpiece as a template are corrected, and a template image on which an ideal template is displayed is generated. Moreover, the resolution, distortion, etc. of the image obtained by photographing the workpiece to be detected are corrected to generate a photographed image on which the ideal workpiece is displayed. Then, pattern matching is performed between the corrected images of the template image and the captured image. According to this pattern matching device, even when the shooting parameters at the time of shooting of a shot image change, it is possible to maintain high-precision pattern matching by adjusting the shooting parameters without re-creating the template image as it is.

JP 2000-241120 A

Zhengyou Zhang.A flexible new technique for camera calibration.IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, pp. 1330-1334, 2000.

  However, since the pattern matching apparatus described in Patent Document 1 corrects the resolution of the captured image to be the same as the resolution of the template image, the corrected captured image resolution is the same as the resolution of the corrected template image. It will be fixed to the same.

  As a result, even if you replace the camera that captures the captured image with a low-resolution camera and attempt to speed up the processing by pattern matching between the low-resolution images, the resolution of the captured image is increased to the resolution of the template image. Therefore, it was difficult to increase the speed. Conversely, if you replace the camera that captures the captured image with a high-resolution camera and attempt to improve the detection accuracy by pattern matching between the high-resolution images, the resolution of the captured image will be the resolution of the template image. Since it is lowered, it is difficult to improve the accuracy.

  An object of the present invention is to provide a pattern matching apparatus and a pattern matching method capable of executing pattern matching corresponding to a change in shooting parameters without regenerating a template image when shooting parameters of the shot image change. To do.

The present invention provides a camera capable of capturing a detection target, a template image of the detection target, and a calculation unit that performs pattern matching between images based on the captured image including the detection target captured by the camera; In the pattern matching apparatus, the calculation unit generates a correction template image from the template image using internal parameters and external parameters calculated based on a captured image captured by the camera, and the correction template image A pattern matching apparatus that performs pattern matching with the photographed image .

Further, the present invention provides a pattern matching method for performing pattern matching between images based on a template image to be detected and a photographed image including the detection object photographed by a camera. A correction image generation step of generating a correction template image from the template image using internal parameters and external parameters calculated on the basis of the captured image, and the calculation unit uses the correction template image and the captured image as a pattern. A pattern matching method comprising: a pattern matching process for matching .

  According to the present invention, the calculation unit generates a correction template image from the template image by using correction parameters for correcting the template image so that pattern matching can be performed on the captured image, and the corrected template image and the captured image are patterned. Match. For this reason, when the shooting parameter of the shot image changes, the shot image is not corrected, so that pattern matching corresponding to the change of the shooting parameter can be executed.

  For example, if the camera that captures the captured image is replaced with a low-resolution camera, the template image is corrected to a low-resolution corrected template image by the correction parameter, so the processing speed can be increased by pattern matching between the low-resolution images. You can plan. In addition, when the camera that captures the captured image is replaced with a high-resolution camera, the template image is corrected to a high-resolution corrected template image using the correction parameters, so detection accuracy is improved by pattern matching between the high-resolution images. Can be achieved.

  Further, when the shooting parameter of the shot image changes, the template image is corrected to the correction template image by the correction parameter, so that it is not necessary to regenerate the template image. Thereby, compared with the case where a template image is recreated many times, it is possible to prevent a waste of time and labor of an operator.

It is explanatory drawing which shows schematic structure of the robot apparatus which concerns on 1st Embodiment of this invention. It is explanatory drawing which shows schematic structure of the control apparatus of the robot apparatus which concerns on 1st Embodiment of this invention. It is explanatory drawing which shows schematic structure of the calculating part of the pattern matching apparatus which concerns on 1st Embodiment of this invention. It is explanatory drawing at the time of correct | amending a picked-up image with the pattern matching apparatus which concerns on 1st Embodiment of this invention, (a) (b) (d) shows the positional relationship of a camera and a measurement plane, (c) (B) shows a state where a work part is cut out from the photographing screen. It is explanatory drawing at the time of correct | amending a template image with the pattern matching apparatus which concerns on 1st Embodiment of this invention. It is explanatory drawing at the time of correct | amending a picked-up image with the conventional pattern matching apparatus. It is a flowchart which shows the procedure at the time of performing pattern matching with the pattern matching apparatus which concerns on 1st Embodiment of this invention. It is explanatory drawing which shows schematic structure of the calculating part of the pattern matching apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the procedure at the time of performing pattern matching with the pattern matching apparatus which concerns on 2nd Embodiment of this invention.

  Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.

[First Embodiment]
As shown in FIG. 1, the robot apparatus 1 includes a robot body 2, a camera 4 that is spaced apart from the robot body 2 and that can be photographed from above the workpiece (detection target) W that is moved by the robot body 2, and the robot body 2 and a control device 3 for controlling the camera 4.

  The robot body 2 includes a 6-axis vertical articulated arm (hereinafter referred to as an arm) 21 and a hand 22 as an end effector. In the present embodiment, a 6-axis vertical articulated arm is applied as the arm 21, but the number of axes may be appropriately changed according to the application and purpose. In the present embodiment, the hand 22 is applied as an end effector. However, the present invention is not limited to this, and all tools that can work on the workpiece W can be included.

  The arm 21 includes seven links 61 to 67 and six joints 71 to 76 that connect the links 61 to 67 so as to swing or rotate. As each of the links 61 to 67, one having a fixed length is adopted. However, for example, a link that can be expanded and contracted by a linear actuator may be employed. Each joint 71 to 76 includes a motor that drives each of the joints 71 to 76, an encoder that detects a rotation angle of the motor, a current sensor that detects a current supplied to each motor, and a torque of each joint 71 to 76. And a torque sensor for detecting.

  The hand 22 is attached to and supported by the distal end link 67 of the arm 21, and at least one degree of freedom in position and posture is adjusted by the operation of the arm 21. The hand 22 includes two fingers 23 and a hand main body 24 that supports the distance between the fingers 23 so as to be openable and closable. The hand 22 can grip the workpiece W by a closing operation in which the fingers 23 approach each other.

  As shown in FIG. 2, the control device 3 is configured by a computer and controls the robot body 2 and the camera 4. The computer constituting the control device 3 includes, for example, a CPU 30, a RAM 31 that temporarily stores data, and a ROM 32 that stores a program for controlling each unit. The CPU 30 includes a pattern matching device 5 described later (see FIG. 1), and performs arithmetic processing related to image processing and pattern matching described later. The RAM 31 temporarily stores data being processed by the CPU 30, images captured from the camera interface 33, and the like. The ROM 32 stores a control program and image processing of the robot device 1 performed by the CPU 30, calculation programs and setting parameters related to pattern matching and workpiece position / orientation calculation, imaging parameters of the camera 4, template images for pattern matching, and the like. The data stored in the ROM 32 is held even when the power of the control device 3 is turned off except for writing and erasing from the CPU 30.

  The control device 3 includes a camera interface 33 connected to the camera 4, a monitor interface 35 connected to the monitor 34, an input device interface 37 connected to the mouse 36, and a communication interface 38. . The camera interface 33 captures an image photographed by the camera 4 and transfers it to the RAM 31. The monitor interface 35 displays the state of the robot apparatus 1 and image processing results on the monitor 34. The input device interface 37 receives an operation signal performed by the mouse 36 while the operator looks at the monitor 34 and transmits the operation signal to the CPU 30. The communication interface 38 is connected to the host system of the robot apparatus 1 and communicates information such as the position and orientation of the workpiece W obtained by calculation.

  As shown in FIG. 1, the control device 3 includes a pattern matching device 5 that performs pattern matching on an image based on a template image and a captured image. The pattern matching device 5 includes a calculation unit 50 that generates a correction template image from a template image using correction parameters described later, and performs pattern matching between the correction template image and the captured image. In this embodiment, the pattern matching device 5 is built in the control device 3, and the calculation unit 50 is configured by the CPU 30.

  As shown in FIG. 3, the calculation unit 50 includes a camera calibration unit 51, a parameter management unit 52, a distortion correction unit 53, a template generation unit 54, a template correction unit 55, a pattern matching unit 56, a three-dimensional display. A restoring unit 57.

  The camera calibration unit 51 calculates the shooting parameters 40 and the corrected shooting parameters 41 that are necessary for correction by the distortion correction unit 53. Note that the shooting parameter 40 and the post-correction shooting parameter 41 are correction parameters for correcting the template image 42 so as to allow pattern matching with an actual shot image. The distortion correction unit 53 corrects a captured image captured by the camera 4 using the imaging parameter 40 and the corrected imaging parameter 41 calculated by the camera calibration unit 51. The template generation unit 54 generates a template image 42 for performing pattern matching by the pattern matching unit 56 using an image obtained by photographing the workpiece W to be a template.

  The template correction unit 55 converts the template image 42 using the post-correction shooting parameter 41a at the time of template generation and the post-correction shooting parameter 41b at the time of actual shooting, creates a correction template image 43, and a parameter management unit Save to 52. The pattern matching unit 56 performs pattern matching between the captured image that has been subjected to distortion correction and the correction template image 43, and calculates the position of the workpiece W in the captured image. The three-dimensional restoration unit 57 converts the position on the image into a position on the three-dimensional coordinate using the position of the workpiece W on the image output from the pattern matching unit 56 and the corrected shooting parameter 41, and the three-dimensional Position data is output and sent to the host system as a measurement result.

  Each image generation and pattern matching processing in the calculation unit 50 of the pattern matching device 5 described above will be described with reference to FIGS.

  A camera calibration marker (not shown) placed on the measurement plane 6 is photographed by the camera 4, and the photographed image is taken into the RAM 31 via the camera interface 33, and further, the camera calibration unit 51 and the distortion correction unit 53. Is input. The camera calibration unit 51 calculates the shooting parameter 40 and the corrected shooting parameter 41 from the image shot by the camera 4 and stores them in the parameter management unit 52.

Here, the shooting parameter 40 and the corrected shooting parameter 41 are information on internal parameters and external parameters. As shown in the pinhole camera model of FIG. 4, the internal parameters are lens distortion, focal length fo, pixel pitch c _x , c _y , number of pixels, image optical axis center C, and the like. The image optical axis center C is a point that intersects the optical axis L on the sensor surface 4a, and the influence of lens distortion on the image occurs radially around the image optical axis center C.

  The external parameters are the distance between the measurement plane 6 and the camera 4, the relative position and orientation, and the like. Thereby, the relative positional relationship between the measurement plane 6 and the camera 4 and the relative relationship between the position on the sensor surface 4a (image) of the camera 4 and the position on the measurement plane 6 are expressed. Note that the relative positional relationship between the sensor surface 4a and the measurement plane 6 includes the relative inclination of both, and inclination information (distortion information) for correcting this inclination is also included in the external parameters.

  The camera calibration unit 51 calculates the imaging parameter 40 using the method described in Non-Patent Document 1 based on the input captured image. In this case, several tens of images are captured while changing the posture of the camera calibration marker, and the imaging parameters 40 are calculated using these images. Since the detailed calculation method is described in Non-Patent Document 1, description thereof is omitted here. Note that the calculation method of the imaging parameter 40 is not limited to the method of Non-Patent Document 1, and other existing or new methods can be applied.

  Based on the calculated shooting parameter 40, the camera calibration unit 51 calculates a corrected shooting parameter 41 that has no lens distortion and that makes the relative position and orientation of the measurement plane 6 and the sensor surface 4a of the camera 4 parallel to each other. That is, the post-correction shooting parameter 41 indicates that the optical axis L of the camera 4 is positioned perpendicular to the measurement plane 6 without changing the distance between the image optical axis center O of the camera 4 and the measurement plane 6, and the lens distortion. This is a shooting parameter when the camera 4 without the camera is installed. This calculation process is performed by executing a camera calibration program stored in the ROM 32 by the CPU 30 using a photographed image fetched from the camera 4 to the RAM 31 via the camera interface 33.

  The camera calibration unit 51 registers the calculated shooting parameter 40 and the corrected shooting parameter 41 in the parameter management unit 52 separately at the time of template generation and at the time of actual shooting. The camera calibration unit 51 registers the shooting parameter 40a and the corrected shooting parameter 41a when generating the template, and registers the shooting parameter 40b and the corrected shooting parameter 41b during actual shooting. Each shooting parameter registered in the parameter management unit 52 is recorded in the ROM 32.

  The parameter management unit 52 manages registered data for the shooting parameter 40a and the corrected shooting parameter 41a at the time of template generation, and the shooting parameter 40b and the corrected shooting parameter 41b at the time of actual shooting. Further, the parameter management unit 52 manages the registered data for the pattern matching template image 42 and the correction template image 43. The processing in the parameter management unit 52 is realized by executing the parameter management program in the ROM 32 by the CPU 30, and the registered data is recorded in the ROM 32.

  The distortion correction unit 53 corrects an image shot by the camera 4 using the shooting parameter 40 and the corrected shooting parameter 41 calculated by the camera calibration unit 51. The correction method here will be described with reference to FIG. In the figure, reference numeral 140 indicates the position and orientation of the sensor surface 4a of the camera 4 that is the imaging parameter 40, and reference numeral 141 indicates the position and orientation of the sensor surface 4a of the camera 4 that is the corrected imaging parameter 41.

  The distortion correction unit 53 projects the point P1 of the original image onto the measurement plane 6 using the imaging parameter 40 to obtain the corresponding point P3. Then, the distortion correcting unit 53 obtains the point P2 by projecting the point P3 onto the distortion corrected image by using the post-correction shooting parameter 41. With such a correction method, distortion correction is performed by obtaining corresponding points between images for all pixels of the distortion correction image.

Specifically, the image coordinate of the point P1 of the original image is (u, v), the image coordinate of the virtual point whose lens distortion is corrected is (u ₁ , v ₁ ), and the lens among the internal parameters of the shooting parameter 40 The distortion coefficient is K, and the image optical axis center O is (u ₀ , v ₀ ). From these relationships, Formula 1 is obtained.

Further, a vector obtained by adding A to the end based on the coordinates (u ₁ , v ₁ ), where A is an internal parameter other than lens distortion among the internal parameters of the imaging parameter 40, and [R t] is an external parameter of the imaging parameter 40. Let m = [u ₁ v ₁ 1]. Further, if the three-dimensional coordinate of the point P3 is (x, y, z), and a vector obtained by adding one element at the end is M = [x y z 1], the relationship between the vector m and the vector M is Equation 2 is obtained.

  Similarly, the image coordinate of the point P2 on the coordinate of the distortion corrected image expressed by the post-correction shooting parameter 41 is (u ′, v ′), and a vector obtained by adding one element at the end based on the coordinate is n = [ u ′ v ′ 1]. Further, the internal parameter of the post-correction shooting parameter 41 is A ′, and the external parameter is [R ′ t ′]. As a result, the relationship between the vector M = [x y z 1] and the vector n regarding the three-dimensional coordinates of the point P3 is expressed by Equation 3.

  From Expression 2 and Expression 3, the relationship between the vector m and the vector n is Expression 4.

Note that when the method of Non-Patent Document 1 is used for the camera calibration unit 51, f (u, v, u ₀ , v ₀ , K), the internal parameter A, and the external parameter [R t] are respectively mathematical expressions. 5 to 8 as shown.

但し、ｘ＝（ｕ−ｕ_０）ｃ_ｘ及びｙ＝（ｖ−ｖ_０）ｃ_ｙ

Where x = (u−u ₀ ) c _x and y = (v−v ₀ ) _cy

Here, c _x and c _y are pixel pitches in the u and v directions on the sensor of the camera 4 defined in the shooting parameters.

  In this manner, a corresponding original image pixel is obtained for each pixel of the distortion-corrected image, and the luminance of each pixel of the original image is applied to obtain a distortion-corrected image. When there are a plurality of corresponding pixels, the luminance values of a plurality of corresponding pixels may be obtained by mixing using a method such as bilinear interpolation. As a result of the distortion correction by the distortion correction unit 53, an image as if it was taken from the position and orientation of the camera 4 of the post-correction shooting parameter 41, that is, a position parallel to the measurement plane 6, with a lens without distortion is obtained. Further, the obtained image has a resolution expressed by the post-correction shooting parameter 41.

  The processing in the distortion correction unit 53 is to execute a distortion correction program recorded in the ROM 32 by the CPU 30 using the captured image captured in the RAM 31 and the imaging parameter 40 stored in the ROM 32 by the parameter management unit 52. Done in The distortion-corrected image obtained by the correction is temporarily stored in the RAM 31. The image corrected by the distortion correction unit 53 is input to the template generation unit 54 and the pattern matching unit 56.

  The template generation unit 54 creates a template image 42 for performing pattern matching in the pattern matching unit 56 using an image obtained by photographing the workpiece W. Specifically, the template generation unit 54 displays the distortion correction image temporarily stored in the RAM 31 on the monitor 34 via the monitor interface 35. As shown in FIG. 4 (c), the operator looks at the image 34a on the monitor 34 and uses the mouse 36 as a whole of the image 34b of the work W or a characteristic part of the image 34b of the work W as a cutout part 34c. specify.

  Then, the operation signal input by the operator using the mouse 36 is received by the CPU 30 via the input device interface 37, and the template generation unit 54 generates the template image 42 from the distortion corrected image. The created template image 42 is recorded in the ROM 32 by registering it as the template image 42 in the parameter management unit 52. Further, the position of the template image 42 in the image 34a when the template image 42 is generated is also stored. The template image 42 is not limited to being created with image data, but may be created with information on points constituting an edge (for example, an array of image coordinates, equations representing straight lines or arcs, etc.).

  The template generation processing in the template generation unit 54 is realized by the CPU 30 that executes the model creation program on the ROM 32, the monitor interface 35 and the monitor 34, the input device interface 37, and the mouse 36. In addition, when registering the template image 42, the parameter management unit 52 associates and manages the imaging parameter 40 a and the corrected imaging parameter 41 a at the time of generating the template image 42 and the template image 42. Note that the imaging parameter 40a is changed by performing camera calibration again to regenerate the template image 42, and the association between the imaging parameter 40a and the template image 42 may be lost. In this case, the parameter management unit 52 deletes the template image 42 and the correction template image 43.

  The template correction unit 55 converts the template image 42 using the post-correction shooting parameter 41a at the time of generating the template image 42 and the post-correction shooting parameter 41b at the time of shooting, generates a correction template image 43, and sets the parameter Saved in the management unit 52. The correction method here will be described with reference to FIG. In the figure, reference numeral 140a indicates the position and orientation of the sensor surface 4a of the camera 4 as the imaging parameter 40a, and reference numeral 141a indicates the position and orientation of the sensor surface 4a of the camera 4 as the corrected imaging parameter 41a. Reference numeral 140b indicates the position and orientation of the sensor surface 4a of the camera 4 as the imaging parameter 40b, and reference numeral 141b indicates the position and orientation of the sensor surface 4a of the camera 4 as the corrected imaging parameter 41b.

  The template correcting unit 55 projects the point P5 of the template image 42 onto the measurement plane 6 using the corrected shooting parameter 41a to obtain the corresponding point P4. Then, the template correction unit 55 obtains the corresponding point P6 by projecting the point P4 on the corrected template image 43 by using the post-correction shooting parameter 41b. With such a correction method, template correction is performed by obtaining corresponding points between images for all pixels of the correction template image 43. The coordinate system of the point P5 of the template image 42 uses the image coordinate system at the time of template generation using the position of the template on the generation image at the time of template generation stored in the template image 42 in the template image 42. .

Specifically, the image coordinate of the point P5 of the template image 42 is (u _p5 , v _p5 ), and a vector obtained by adding one element based on the coordinate is m _p5 = [u _p5 v _p5 1]. In addition, the internal parameter of the post-correction shooting parameter 41a is A _p5 and the external parameter is [R _p5 t _p5 ]. Furthermore, the three-dimensional coordinates of the point P4 and _{(x p4, y p4, z} p4), the last vector plus one element to the _M p4 ₌ a _{[x p4 y p4 z p4 1} ] on the basis thereof, a vector The relationship between m _p5 and vector M _p4 is given by Equation 9.

Similarly, the image coordinate of the point P6 on the coordinates of the corrected template image 43 expressed by the post-correction shooting parameter 41b is (u _p6 , v _p6 ), and finally a vector obtained by adding one element based on it is n _p6 = [ _Up6 v _p6 1]. The internal parameter of the post-correction shooting parameter 41b is A _p6 and the external parameter is [R _p6 t _p6 ]. Thus, the relationship between the vector M _p4 = [x _p4 y _p4 z _p4 1] related to the three-dimensional coordinates of the point P4 and the vector n _p6 is expressed by Equation 10.

From Equation 9 and Equation 10, the relationship between the vector m _p5 and the vector n _p6 is Equation 11.

  When the method of Non-Patent Document 1 is used, each coefficient is expressed by Equations 5 to 8 as in the case of the distortion correction unit 53.

  Thus, the template image 42 corresponding to each pixel of the correction template image 43 is obtained, and the luminance of each pixel of the template image 42 is applied to obtain the correction template image 43. When there are a plurality of corresponding pixels, the luminance values of a plurality of corresponding pixels may be obtained by mixing using a method such as bilinear interpolation. Further, when using a template image expressed by an edge composing point group such as shape-based matching, the correction template image 43 is obtained by converting each point constituting the template image using Equation 11.

  The processing in the template correction unit 55 is performed by executing a template correction program recorded in the ROM 32 by the CPU 30. In this template correction program, the template image 42 recorded in the ROM 32, the post-correction shooting parameter 41a at the time of template generation, and the post-correction shooting parameter 41b at the time of actual shooting are read from the parameter management unit 52 and developed in the RAM 31. In the template correction program, the CPU 30 performs the above-described correction calculation, registers the generated correction template image 43 in the parameter management unit 52, and records it in the ROM 32.

  The pattern matching unit 56 performs pattern matching between the distortion-corrected image of the captured image captured by the camera 4 and the corrected template image 43 during actual capturing, and calculates the position of the template in the captured image. As a specific method of pattern matching, an existing or new image coordinate-based matching method such as a normalized correlation method or shape-based matching is used.

  Processing in the pattern matching unit 56 is realized by the CPU 30 executing a pattern matching program in the ROM 32. The pattern matching unit 56 performs matching between the distortion correction image temporarily stored in the RAM 31 and the correction template image 43 read from the ROM 32 via the parameter management unit 52. The pattern matching unit 56 temporarily stores the position of the workpiece W on the coordinates of the distortion corrected image obtained as a matching result in the RAM 31.

  The three-dimensional restoration unit 57 converts the position on the image into a position on the three-dimensional coordinate using the position on the image of the workpiece W output from the pattern matching unit 56 and the post-correction shooting parameter 41b. The original position data is output and sent as a measurement result to the host system.

  The processing of the three-dimensional restoration unit 57 is realized by executing the three-dimensional coordinate conversion program in the ROM 32 by the CPU 30. The three-dimensional restoration unit 57 converts the image coordinates of the pattern matching result temporarily stored in the RAM 31 into three-dimensional coordinates using the post-correction shooting parameters 41b read from the ROM 32 via the parameter management unit 52. The three-dimensional restoration unit 57 temporarily stores the obtained three-dimensional position data in the RAM 31 and then transmits it to the host system via the communication interface 38 in response to a command from the CPU 30.

  Here, correction for executing pattern matching between the template image X and the captured image Y will be described with reference to FIGS. 5 and 6 focusing on only the resolution. 5 and 6, for the sake of explanation, the template image is denoted by reference symbol X, and the photographed image obtained by photographing the workpiece W during actual assembly work is denoted by reference symbol Y.

One pattern matching method, as shown in FIG. 6, the template image X of the resolution as it corrects the captured image Y _{1 (or} Y ₂₎ imaging parameters Q _{1 (or} Q _2), the template image A method for adjusting to the resolution of X can be considered. That is, according to this method, X is matched with Q ₁ · Y ₁ (or Q ₂ · Y ₂ ).

However, in this method, for example, since the resolution of the captured image Y ₁ even if the doubled compared to when generating the template image X, it is necessary to correct the combined resolution template image X, 1/2-fold Matching is performed on the corrected image that has been reduced and discretized. Therefore, accuracy of the matching, becomes equivalent to the generation time of the resolution of the template image X, accuracy of replacement pattern also matching the camera for photographing a photographed image Y ₁ to a high resolution camera is difficult is there.

Further, since in this method, for example, that even when the resolution of the captured image Y ₂ becomes 1/2 times that at the time of generation of the template image X, it is necessary to correct the combined resolution template image X, 2-fold Matching is performed on the corrected image enlarged. Therefore, by replacing the camera for photographing a photographed image Y ₂ in the low resolution camera, high-speed pattern matching is difficult.

Therefore, in the present embodiment, as shown in FIG. 5, the template image X is corrected by the shooting parameter Q ₁ (or Q ₂ ) while maintaining the resolution of the captured image Y ₁ (or Y ₂ ), and the captured image Y ₁ is corrected. The resolution is adjusted to ₁ (or Y ₂ ). That is, according to the method of the present embodiment, X / Q ₁ (or X / Q ₂ ) and Y ₁ (or Y ₂ ) are matched.

Thus, according to the method of this embodiment, for correcting the combined template image X to the resolution captured image Y _1, it is possible to perform matching while maintaining the resolution of the captured image Y _1. Therefore, also accuracy in matching becomes resolution captured image Y _1, doubled compared to the resolution at the time of generation of the template image X, improve the accuracy of detection by the pattern matching between images of high resolution Can do.

Further, according to the method of this embodiment, for correcting the combined template image X to the resolution captured image Y _2, it is possible to perform matching while maintaining the resolution of the captured image Y _2. For this reason, since the area to be searched is 1/4 in area ratio, the detection speed is four times faster than the resolution at the time of generating the template image X, and the processing speed is increased by pattern matching between low resolution images. Can be planned.

  A processing procedure for executing the pattern matching of the workpiece W by the pattern matching device 5 described above will be described with reference to the flowchart shown in FIG.

  First, before generating the template image 42, the camera 4 is calibrated (step S1). Here, a camera calibration marker (not shown) placed on the measurement plane 6 is photographed by the camera 4 and input to the camera calibration unit 51 of the calculation unit 50. Thereby, the shooting parameter 40 a and the corrected shooting parameter 41 a at the time of generating the template image 42 are calculated and registered in the parameter management unit 52.

  Then, the workpiece W to be the template image 42 is set on the measurement plane 6 and photographed by the camera 4, and the photographed image is input to the calculation unit 50 (step S2). The input photographed image is corrected by the distortion correcting unit 53 using the photographing parameter 40a calibrated in step S1 and the post-correction photographing parameter 41a (step S3).

  The corrected photographic image output from the distortion correction unit 53 is input to the template generation unit 54, and a template image 42 is generated. Here, for example, the control device 3 displays the corrected captured image on the monitor 34 (step S4), and causes the operator to select the cutout part 34c on the corrected captured image (step S5).

  And the template production | generation part 54 produces | generates the template image 42 from the correction | amendment picked-up image and the clipping part 34c, and registers it in the parameter management part 52 (step S6). At that time, the parameter management unit 52 associates and registers the template image 42, the shooting parameter 40a, and the corrected shooting parameter 41a.

  Here, after the template image 42 is generated in step S6, it is determined whether or not the shooting parameter 40a has changed due to a change with time, replacement of the camera 4, or a collision with the camera 4 (step S7). . When it is determined that the shooting parameter 40a has not changed, the template image 42 is used as it is, and a pattern matching process (step S10) described later is executed.

  If it is determined that the shooting parameter 40a has changed, the template image 42 is corrected before the pattern matching process. First, the camera calibration unit 51 calculates the shooting parameter 40b and the corrected shooting parameter 41b and registers them in the parameter management unit 52 in the same processing procedure as in step S1 (step S8). Then, the template correction unit 55 generates a corrected template image 43 for the template image 42 using the shooting parameter 40a and the corrected shooting parameter 41a, the shooting parameter 40b, and the corrected shooting parameter 41b (step S9). ). Step S9 corresponds to the corrected image generation process of the present invention. Thereby, a template suitable for the current shooting parameter 40b and the corrected shooting parameter 41b is generated. The template correction unit 55 registers the correction template image 43 in the parameter management unit 52.

  Then, pattern matching processing is executed using the actually photographed image of the workpiece W (step S10, pattern matching process). Here, an image obtained by photographing the workpiece W on the measurement plane 6 by the camera 4 is sent to the distortion correction unit 53, and the distortion correction unit 53 corrects the image using the corrected shooting parameter 41a and the corrected shooting parameter 41b. Thus, a distortion corrected image is generated. Thereafter, the pattern matching unit 56 performs pattern matching between the distortion correction image and the correction template image 43, and calculates the position of the workpiece W in the distortion correction image. The three-dimensional restoration unit 57 calculates three-dimensional position data of the workpiece W from the position calculated by the pattern matching unit 56 and the post-correction shooting parameter 41b.

  As described above, according to the pattern matching device 5 of the present embodiment, the pattern matching is performed after the template image 42 is converted to be the same as the captured image. For this reason, even if the imaging parameter 40a and the corrected imaging parameter 41a change after the generation of the template image 42, it is not necessary to regenerate the template image 42. Thereby, compared with the case where the template image 42 is regenerated many times, it is possible to prevent the waste of the worker's time and labor.

  Further, according to the pattern matching device 5 of the present embodiment, when the shooting parameter of the shot image changes, it is not necessary to match the resolution of the shot image with the template image when correcting the shot image. Pattern matching can be performed.

  For example, when the camera 4 that captures a photographed image is replaced with a high resolution, the template image 42 is corrected to a high-resolution correction template image 43 by the correction parameter, so that high-precision detection is achieved by pattern matching between the high-resolution images. Can be achieved.

In addition, when the camera 4 that captures the captured image is replaced with a low resolution, the template image 42 is corrected to the low-resolution corrected template image 43 by the correction parameter. Therefore, the processing speed is increased by pattern matching between the low-resolution images. Can be achieved. In the example shown in FIG. 6, because a pattern matching from the resolution of the captured image Y ₂ to the resolution at the time of generation of the template image X, the detection accuracy of the apparent compared to the present embodiment is high. However, since it is actually discretized at the resolution of the captured image Y _{2 and} the amount of information is determined, the actual detection accuracy depends on the resolution of the captured image Y ₂ and is equivalent to the detection accuracy of the present embodiment. become.

  In the pattern matching device 5 of the present embodiment, an area for registering the shooting parameters when generating the template image 42 is required as compared with the example shown in FIG. However, since the shooting parameter data includes several tens of values such as focal length, lens distortion, sensor pixel pitch, resolution, and the three-dimensional relative position and orientation of the measurement plane 6 and the camera 4, There will be no increase. For example, the data area necessary for registering the shooting parameters is extremely small compared to the data area required for registering the template image 42 of an image having 200 pixels on a side, that is, an image having 40,000 pixels. For this reason, the influence on the cost etc. of the pattern matching apparatus 5 by this embodiment can be restrained small.

[Second Embodiment]
Next, the calculation unit 250 of the pattern matching device 5 according to the second embodiment of the present invention will be described.

  Compared with the first embodiment, the arithmetic unit 250 of the second embodiment has a different configuration in that the distortion correction unit 53 is omitted as shown in FIG. Since the other configuration is the same as that of the first embodiment, the same reference numerals are given and description thereof is omitted.

  Similarly to the camera calibration unit 51 of the first embodiment, the camera calibration unit 251 calculates the imaging parameter 40 using the technique described in Non-Patent Document 1 based on the input captured image. However, unlike the camera calibration unit 51 of the first embodiment, the camera calibration unit 251 does not calculate the corrected shooting parameter 41. The camera calibration unit 251 registers the calculated shooting parameter 40 in the parameter management unit 252 separately for the template generation and the actual shooting.

  The parameter management unit 252 manages registered data for the shooting parameter 40a at the time of template generation and the shooting parameter 40b at the time of actual shooting, and the template image 242 and the correction template image 243 for pattern matching. The processing in the parameter management unit 52 is realized by executing the parameter management program in the ROM 32 by the CPU 30, and the registered data is recorded in the ROM 32.

  The template correction unit 255 converts the template image 242 using the shooting parameter 40 a at the time of generating the template image 242 and the shooting parameter 40 b at the time of shooting, generates a correction template image 243, and sends it to the parameter management unit 252. save.

  As shown in FIG. 4D, the template correction unit 255 projects the point P7 of the template image 242 with the reference numeral 140a onto the measurement plane 6 using the imaging parameter 40a to obtain the corresponding point P4. Then, the template correction unit 255 obtains a corresponding point P8 by projecting the point P4 onto the correction template image 243 denoted by reference numeral 140b by using the imaging parameter 40b. With such a correction method, template correction is performed by obtaining corresponding points between images for all pixels of the correction template image 243.

Specifically, the image coordinate of the point P7 of the template image 242 is (u _p7 , v _p7 ), and a vector obtained by adding one element at the end based on the coordinate is m _p7 = [u _p7 v _p7 1]. In addition, the internal parameter of the imaging parameter 40a is A _p7 and the external parameter is [R _p7 t _p7 ]. Furthermore, the three-dimensional coordinates of the point P4 and _{(x p4, y p4, z} p4), the last vector plus one element to the _M p4 ₌ a _{[x p4 y p4 z p4 1} ] on the basis thereof, a vector The relationship between m _p7 and vector M _p4 is given by Equation 12.

Similarly, the image coordinates of the point P8 on the coordinates of the correction template image 243 expressed by the imaging parameter 40b is (u _p8 , v _p8 ), and a vector obtained by adding one element at the end based on it is n _p8 = [ u _p8 v _p8 1]. The internal parameter of the shooting parameter 40b is A _p8 , and the external parameter is [R _p8 t _p8 ]. Accordingly, the relationship between the vector M _p4 = [x _p4 y _p4 z _p4 1] related to the three-dimensional coordinates of the point P4 and the vector n _p8 is expressed by Equation 13.

From Expression 12 and Expression 13, the relationship between the vector _mp7 and the vector _NP8 is Expression 14.

  When the method of Non-Patent Document 1 is used, each coefficient is expressed by Equations 5 to 8 as in the case of the distortion correction unit 53 in the first embodiment.

  In this manner, the template image 242 corresponding to each pixel of the correction template image 243 is obtained, and the luminance of each pixel of the template image 242 is applied to obtain the correction template image 243. When there are a plurality of corresponding pixels, the luminance values of a plurality of corresponding pixels may be obtained by mixing using a method such as bilinear interpolation. In addition, when using a template image expressed by an edge constituent point group such as shape-based matching, a corrected template image is obtained by converting each point constituting the template image using Expression 14.

  The processing in the template correction unit 255 is performed by executing a template correction program recorded in the ROM 32 by the CPU 30. In this template correction program, the template image 242 recorded in the ROM 32, the shooting parameter 40a at the time of template generation, and the shooting parameter 40b at the time of actual shooting are read from the parameter management unit 252 and developed in the RAM 31. In the template correction program, the CPU 30 performs the above-described correction calculation, registers the generated correction template image 243 in the parameter management unit 252, and records it in the ROM 32.

  The three-dimensional restoration unit 257 converts the position on the image into the position on the three-dimensional coordinate using the position on the image of the workpiece W output from the pattern matching unit 56 and the imaging parameter 40b, and the three-dimensional position Data is output and sent to the host system as measurement results.

  The processing of the three-dimensional restoration unit 257 is realized by executing the three-dimensional coordinate conversion program in the ROM 32 by the CPU 30. The three-dimensional restoration unit 257 converts the image coordinates of the pattern matching result temporarily stored in the RAM 31 into three-dimensional coordinates using the photographing parameter 40b at the time of photographing read from the ROM 32 via the parameter management unit 252. The three-dimensional restoration unit 257 temporarily stores the obtained three-dimensional position data in the RAM 31 and then transmits it to the host system via the communication interface 38 in response to a command from the CPU 30.

  A processing procedure for executing the pattern matching of the workpiece W by the pattern matching device 5 described above will be described with reference to the flowchart shown in FIG.

  First, before generating the template image 242, the camera 4 is calibrated (step S11). Here, a camera calibration marker (not shown) placed on the measurement plane 6 is photographed by the camera 4 and input to the camera calibration unit 251 of the calculation unit 250. Thereby, the shooting parameter 40 a at the time of generating the template image 242 is calculated and registered in the parameter management unit 252.

  Then, the workpiece W to be the template image 242 is placed on the measurement plane 6 and photographed by the camera 4, and the photographed image is input to the calculation unit 250 (step S12). The captured image is input to the template generation unit 54, and a template image 242 is generated. Here, for example, the control device 3 displays the captured image on the monitor 34 (step S13), and causes the operator to select the cutout part 34c on the captured image (step S14).

  Then, the template generation unit 54 generates a template image 242 from the captured image and cutout unit 34c, and registers it in the parameter management unit 252 (step S15). At this time, the parameter management unit 252 associates and registers the template image 242 and the imaging parameter 40a.

  Here, after the template image 242 is generated in step S15, it is determined whether or not the shooting parameter 40a has changed due to a change with time, replacement of the camera 4, or a collision with the camera 4 (step S16). . If it is determined that the shooting parameter 40a has not changed, the template image 242 is used as it is, and a pattern matching process (step S19) described later is executed.

  If it is determined that the shooting parameter 40a has changed, the template image 242 is corrected before the pattern matching process. First, the camera calibration unit 251 calculates the shooting parameter 40b and registers it in the parameter management unit 252 in the same processing procedure as in step S11 (step S17). Then, the template correction unit 255 generates a correction template image 243 using the shooting parameter 40a and the shooting parameter 40b for the template image 242 (step S18, corrected image generation step). Thus, a template suitable for the current shooting parameter 40b is generated. The template correction unit 255 registers the correction template image 243 in the parameter management unit 252.

  Then, pattern matching processing is executed using the actually photographed image of the workpiece W (step S19, pattern matching process). Here, the pattern matching unit 56 performs pattern matching between the image obtained by photographing the workpiece W on the measurement plane 6 by the camera 4 and the correction template image 243, and calculates the position of the workpiece W in the photographed image. The three-dimensional restoration unit 257 calculates three-dimensional position data of the workpiece W from the position calculated by the pattern matching unit 56 and the imaging parameter 40b.

  As described above, according to the pattern matching device 5 of the present embodiment, the pattern matching is performed after converting the template image 242 to be the same as the captured image, as in the first embodiment. For this reason, even if the imaging parameter 40a changes after the template image 242 is generated, it is not necessary to regenerate the template image 242. Therefore, compared with the case where the template image 242 is regenerated many times, the time of the operator is reduced. The waste of labor can be prevented.

  Further, according to the pattern matching device 5 of the present embodiment, as in the first embodiment, when the shooting parameter of the shot image changes, the shot image is not corrected, so pattern matching corresponding to the change of the shooting parameter is performed. Can be executed.

  Further, according to the pattern matching device 5 of the present embodiment, it is not necessary to correct the distortion of the captured image. For this reason, for example, when the lens distortion of the camera 4 is small and the sensor surface 4a is placed substantially parallel to the measurement plane 6, only the change in the distance between the camera 4 and the workpiece W can be dealt with. It is suitable when no is required. In this case, since distortion correction of the captured image is not necessary, the pattern matching process can be executed at high speed.

  In the first and second embodiments described above, the case where the template images 42 and 242 are generated by photographing the workpiece W with the camera 4 has been described. However, the present invention is not limited to this. For example, the template images 42 and 242 may be generated by projecting a three-dimensional CAD model of the workpiece W into two dimensions. Also in this case, the obtained template images 42 and 242 can be corrected to the corrected template images 43 and 243 and pattern-matched with the captured image.

  In the first and second embodiments described above, the case where the template images 42 and 242 are always corrected in order to perform pattern matching between the template images 42 and 242 and the captured image has been described. I can't. For example, according to the situation, as in the example of FIG. 6, the captured image may be corrected to generate a corrected image so as to match the template images 42 and 242. Thus, for example, when a high-resolution captured image is obtained compared to the template images 42 and 242, and when high matching accuracy is not required, the matching time is reduced by matching the captured image with the low-resolution template images 42 and 242. It can be shortened. That is, it is possible to appropriately select which of the template images 42 and 242 and the captured image is to be corrected according to the conditions required for pattern matching accuracy and speed.

  In the first and second embodiments described above, the case in which the target of pattern matching is the template images 42 and 242 and the captured image has been described. However, the present invention is not limited to this. For example, the present invention can be applied to general pattern matching between two or more images, such as pattern matching between a plurality of images obtained by photographing a plurality of detection targets.

  In the first and second embodiments described above, the case where the pattern matching device 5 is built in the control device 3 of the robot device 1 has been described. However, the present invention is not limited to this. For example, the pattern matching device can be applied to a production facility including a production device such as an NC device or a device arranged on a production line. In this case, for example, the camera of the production facility photographs the workpiece, the pattern matching device determines the position and orientation of the workpiece, authenticity, etc. based on the information obtained by the photographing, and the control device determines whether the production device is based on the result. Control the movement.

  The processing operations of the first and second embodiments described above are specifically executed by the arithmetic units 50 and 250. Therefore, it is achieved by supplying a recording medium recording a software program for realizing the above-described functions to the arithmetic units 50 and 250, and the CPU 30 constituting the arithmetic units 50 and 250 reads and executes the program stored in the recording medium. You may make it do. In this case, the program itself read from the recording medium realizes the functions of the above-described embodiments, and the program itself and the recording medium on which the program is recorded constitute the present invention.

  In the first and second embodiments, the case where the computer-readable recording medium is the ROM 32 and the program is stored in the ROM 32 has been described. However, the present invention is not limited to this. The program may be recorded on any recording medium as long as it is a computer-readable recording medium. For example, an HDD, an external storage device, a recording disk, or the like may be used as a recording medium for supplying the program.

DESCRIPTION OF SYMBOLS 1 ... Robot apparatus, 2 ... Robot main body, 3 ... Control apparatus, 4 ... Camera, 5 ... Pattern matching apparatus, 40a, 41a ... Shooting parameter (correction parameter), 40b, 41b ... After-correction shooting parameter (correction parameter), 42 , 242 ... Template image, 43, 243 ... Correction template image, 50, 250 ... Calculation unit, Q ₁ , Q ₂ ... Shooting parameter (correction parameter), W ... Workpiece (detection target), X ... Shooting image, Y ₁ , Y ₂ ... Template image

Claims (13)

A camera capable of shooting the detection target;
In a pattern matching device comprising: a calculation unit that performs pattern matching between images based on the template image of the detection target and a captured image including the detection target captured by the camera;
The calculation unit generates a correction template image from the template image using internal parameters and external parameters calculated based on a captured image captured by the camera, and performs pattern matching between the corrected template image and the captured image. ,
A pattern matching device. Before Kigai unit parameters include distortion information of the captured image to correct the inclination with respect to the detection target of the camera,
The pattern matching apparatus according to claim 1. The arithmetic unit, when generating the correction template image, adjust the resolution of the corrected template image to the resolution of the shooting image,
The pattern matching apparatus according to claim 1, wherein: The internal parameter and the external parameter are calculated based on a captured image of a camera calibration marker captured by the camera.
The pattern matching apparatus according to claim 1, wherein the pattern matching apparatus is a pattern matching apparatus. A robot body capable of working on the detection target;
5. The pattern matching device according to claim 1, wherein the position and orientation of the detection target are calculated based on a result of executing the pattern matching by the pattern matching device, and the obtained detection is performed. A control device that controls the operation of the robot body based on the position and orientation of the target,
A robot apparatus characterized by that. A production apparatus capable of working on the detection target;
A control device that has the pattern matching device according to any one of claims 1 to 4 and controls an operation of the production device based on a result of executing the pattern matching by the pattern matching device. ,
Production equipment characterized by that. In a pattern matching method for performing pattern matching between images based on a detection target template image and a captured image including the detection target captured by a camera,
A correction image generating step for generating a correction template image from the template image using an internal parameter and an external parameter calculated based on a captured image captured by the camera;
The calculation unit includes a pattern matching step for pattern matching between the correction template image and the captured image.
A pattern matching method characterized by that. Before Kigai unit parameters include distortion information of the captured image to correct the inclination with respect to the detection target of the camera,
The pattern matching method according to claim 7, wherein: In the corrected image generation step, the correction template image is generated in accordance with the resolution of the shooting image,
The pattern matching method according to claim 7, wherein the pattern matching method is a pattern matching method. The internal parameter and the external parameter used in the corrected image generation step are calculated based on a captured image of a camera calibration marker captured by the camera.
The pattern matching method according to claim 7, wherein the pattern matching method is a pattern matching method. The correction image generation step is performed at any one of timing when the camera or lens is changed, timing when the distance between the camera and the work or relative position and orientation is changed, timing when a predetermined time has passed,
The pattern matching method according to claim 7, wherein the pattern matching method is a pattern matching method.   The program for making a computer perform each process of the pattern matching method of any one of Claims 7-11.   A computer-readable recording medium on which the program according to claim 12 is recorded.

Images