JP2020077397A - Image processing method, image processing unit, production system, article manufacturing method, program, and storage medium - Google Patents

Abstract

To generate a model with few false detection independent of background, noises, the shape of a work-piece, etc. while ensuring the speed in an inspection process of images using pattern matching.SOLUTION: Multiple temporary models are generated using a reference image on which an object of pattern matching is recorded. Pattern matching is performed between each of the multiple temporary models and each of multiple evaluation images on which the object of the pattern matching is recorded to obtain a matching score between each of the multiple temporary models and each of the multiple evaluation images. For each of the multiple temporary models, a first matching score group obtained by collecting first matching scores whose similarity is maximum in each of the evaluation images and a second matching score group obtained by collecting second matching scores whose similarity is smaller than that of the first matching scores are acquired. For each of the multiple temporary models, an evaluation value is calculated from the first matching score group and the second matching score group to determine a matching model.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image processing method, an image processing apparatus, a production system, and an article manufacturing method, and more particularly to an image processing method, an image processing apparatus, a production system, and an article manufacturing method that utilize image processing using pattern matching. ..

  In the field of image processing, pattern matching is well known as a method of detecting the position of a detection target object. In order to perform the pattern matching, a reference image obtained by imaging the detection target object in an ideal state is used as the reference information of the detection target object. This reference image is sometimes called a model, a template, or the like. Only a region near the detection target object is cut out from this reference image, and a model used for pattern matching is created.

  In the actual object detection step, a search target image for detecting an object is captured, and the degree of similarity with the model is calculated at each position of the captured image. In this method, the position of the image having the highest degree of similarity is set as the detection position, that is, the position most similar to the model is detected from the search target image.

  A plurality of methods are known for calculating the above-mentioned similarity, and a brightness value difference method (SAD) for calculating a similarity on a luminance basis, a normalized correlation method (NCC), and a similarity on an edge feature basis are used. Shape pattern matching for calculation and the like are known.

  In this type of pattern matching, there may be a problem of erroneous detection, for example, detecting a position other than the detection target object. For example, erroneous detection occurs because the similarity is higher at a position other than the detected object position than at the detected object position. For example, suppose that there is an image as shown in FIG. 1A, and pattern matching is performed to detect the position in the image of the small circle in the circular ring work 100. Reference numerals 101 and 102 in FIG. 1A are images of other works captured in the background of the annular work 100 or the like. In this case, an image of the model 1012 as shown in FIG. 1B may be created. In the model 1012 of FIG. 1B, the model 1012 is created by an image obtained by cutting out only the small circular portion 1011 of the circular workpiece 100 from the template. If such a model 1012 is used, the position (101) of the circle imaged separately from the circular workpiece 100 of FIG. 1A may be erroneously detected. In order to prevent such erroneous detection (erroneous detection), the user has to create a model image so as to include the annular portion of the annular workpiece 100.

  As described above, the frequency of erroneous detection varies depending on what kind of object shape is used as a model, which region is cut out from the template, and the like. Therefore, in the past, when creating a model for pattern matching, trial and error were required, and also delicate settings based on the user's empirical rule were sometimes required, so that work required proficiency. To be done.

  Further, this kind of pattern matching may be executed on an image obtained by photographing a work or a production device in a product production line, and the result may be used for monitoring, production control, or the like. If erroneous detection occurs in such a system, a defective product may occur, or a malfunction due to a malfunction of production equipment may occur. Therefore, there is a demand for a method and system for creating a model that is less likely to cause erroneous detection and can perform reliable pattern matching.

  In view of this point, Patent Document 1 proposes a method of performing pattern matching on a captured image using all of a plurality of different models prepared in advance. In this case, the position where the model overlaps is the detection position among the plurality of detection positions. Then, a method of reducing the probability of false detection by performing pattern matching with a plurality of models is described.

  Further, in Patent Document 2, a plurality of models are created, pattern matching is performed on the evaluation image, and at that time, the degree of similarity (score) at each position of the evaluation image is stored. Then, a score distribution is created for each model, and the variance values of this score distribution are compared to select a model that is difficult to be erroneously detected. In Patent Document 2, for example, a model having the largest variance value of the score distribution is selected as a model that is hard to be erroneously detected, and in the actual object detection step, erroneous detection is performed by executing pattern matching only with the selected model. To avoid.

JP, 2008-165394, A JP 2001-148014 A

  According to the configuration of Patent Document 1 described above, since pattern matching is executed with a plurality of models in the actual image inspection process, the processing time increases by the number of models. Pattern matching is used in various applications, but especially in industrial and FA applications, reduction of production cycle time is emphasized. The configuration of Patent Document 1 may not be suitable for applications requiring such high-speed processing.

  Further, according to the configuration of Patent Document 2, the model is evaluated according to the magnitude of the variance value of the score distribution. However, there is a possibility that it cannot be said that the model in which the variance value of the score distribution is large is hard to be erroneously detected for all the images captured by various production systems. In the technique of Patent Document 2, it is premised that when the possibility of erroneous detection is small, the score becomes high only at the correct position and becomes low at other positions, so that the variance value becomes large. Further, when the possibility of false detection is high, it is assumed that the variance value becomes small due to the overall score distribution becoming high. However, the shape of the score distribution greatly differs depending on the background, noise, the shape of the work, etc. in the actual image captured by the production system, and the evaluation standard using the variance value of the score distribution as in Patent Document 2 is versatile. And may not have generality. For example, in an image captured by an actual production system, it is expected that the variance value may be large in a score distribution in which a plurality of peaks are likely to be erroneously detected.

  An object of the present invention is to make it possible to generate a model that does not depend on the background, noise, the shape of a work, and the like and does not cause a false detection, without impairing the high speed of the image inspection process using pattern matching.

A first aspect of the present invention is a temporary model creating step of creating a plurality of temporary models for pattern matching using a reference image in which an object of pattern matching is recorded, and each of the plurality of temporary models, Pattern matching is performed with each of the plurality of evaluation images in which the pattern matching target is recorded, and a matching score between each of the plurality of temporary models and each of the plurality of evaluation images is obtained. For each of the provisional models, a first matching score group in which first matching scores having the maximum similarity in each evaluation image are aggregated, and a similarity in each evaluation image is higher than the first matching score. A second matching score group in which small second matching scores are collected, and a matching score acquisition step of acquiring
For each of the plurality of temporary models, an evaluation step of calculating an evaluation value from the first matching score group and the second matching score group, and of the plurality of temporary models based on the calculated evaluation value. And a determining step of determining a matching model from the inside.

  A second aspect of the present invention is a temporary model creation unit that creates a plurality of temporary models for pattern matching using a reference image on which a pattern matching object is recorded, and each of the plurality of temporary models. And, performing pattern matching with each of the plurality of evaluation images in which the pattern matching object is recorded, obtain a matching score between each of the plurality of temporary models and each of the plurality of evaluation images, For each of the plurality of provisional models, a first matching score group in which the first matching score having the maximum similarity in each evaluation image is collected, and the first matching score having the similarity in each evaluation image A second matching score group in which a second matching score smaller than the second matching score is acquired, and for each of the plurality of temporary models, the first matching score group and the second matching score. An image processing apparatus, comprising: an evaluation unit that calculates an evaluation value from a score group; and a determination unit that determines a matching model from the plurality of temporary models based on the calculated evaluation value.

  According to the present invention, a model with less false detection is generated without impairing the high speed of the image inspection process, independent of the background, noise, shape of the work, etc., and highly reliable pattern matching is performed using the model. It is possible to carry out image processing using the.

(A) An example of an image in which a plurality of works are photographed. (B) An example of a model used for pattern matching processing. It is explanatory drawing which showed the schematic structure of the production system which concerns on embodiment. It is a block diagram showing the composition of the image processing device concerning an embodiment. It is a flowchart figure which showed the creation process of the pattern matching model which concerns on Embodiment 1 of this invention. It is a flowchart figure which showed the temporary model creation process which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the edge extraction process which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the characteristic image which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed extraction of the connection edge which concerns on Embodiment 1 of this invention. (A) An example of one temporary model composed of connected edges. (B) An example of another temporary model including a combination of connected edges. (C) Another example of the temporary model including another combination of connected edges. It is a flowchart figure which showed the evaluation value calculation process of the temporary model which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the pattern matching process in the image processing which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the score map in the evaluation value calculation which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed an example of the determination process of the vicinity of the position of the maximum detection score in the evaluation value calculation which concerns on Embodiment 1 of this invention. It is a table figure explaining the evaluation value calculation method which concerns on Embodiment 1 of this invention. It is a table figure explaining the evaluation value calculation method which concerns on Embodiment 1 of this invention. It is a flowchart figure which showed the temporary model creation process which concerns on Embodiment 2 of this invention.

  Embodiments for carrying out the present invention will be described below with reference to the accompanying drawings. Note that the configuration shown below is merely an example, and for example, a detailed configuration can be appropriately changed by those skilled in the art without departing from the spirit of the present invention. The numerical values taken in this embodiment are reference numerical values and do not limit the present invention.

<Embodiment 1>
FIG. 2 shows a schematic configuration of the production system according to this embodiment. The production apparatus 200 of FIG. 2 attaches the work W1 (object) to the work W2 (assembly member) to manufacture the article W. The production device 200 includes a robot 401, a camera 500 as an imaging device, and an image processing device 300 configured to communicate with the camera 500 in a wired or wireless manner. The production apparatus 200 also includes a robot control device 400 connected to the robot 401 and the image processing device 300 by a network or a control line.

  The robot 401 has, for example, a vertical articulated robot arm 402 and a robot hand 403 which is an end effector. The robot arm 402 has a configuration in which a plurality of links are pivotally connected by a plurality of joints. The base end (base end link, base portion) of the robot arm 402 is fixed to the upper surface of the base B. A robot hand 403 is attached to the tip (tip link) of the robot arm 402. The robot hand 403 has a plurality of fingers and can hold or release the work (target object) W1.

  The camera 500 is composed of, for example, a digital camera. The image sensor of the camera 500 is configured by an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

  The image processing apparatus 300 controls the image capturing operation of the camera 500 (for example, image capturing timing, shutter speed, etc.) and acquires a captured image from the camera 500. Further, the image processing device 300 performs image processing on the digital image data and outputs the result to the robot control device 400. As this image processing, the image processing apparatus 300 performs pattern matching processing.

  In the production line configured as shown in FIG. 2, pattern matching is performed to detect or measure the actual position or posture of the work W1 or W2 or a part thereof, a specific portion of the robot arm 402, or the like. .. Therefore, for this pattern matching, a model for matching specific parts such as the works W1 and W2 and the robot arm 402 is used. The image processing device 300 outputs the image processing result to the robot control device 400. The format of the output data to the robot control device 400 at this time is arbitrary, but for example, the coordinate value of a specific part of the measurement target in the global coordinate system or the coordinate system of the captured image can be considered. The positional relationship between the arrangement position of the camera 500 and the work space in which the robot arm 402 operates is known. For example, coordinate conversion calculation can be performed between the coordinate system of the captured image and the global coordinate system of the work space of the robot arm 402. I shall. Such coordinate conversion calculation is appropriately performed by the image processing device 300 or the robot control device 400.

  The robot control device 400 controls the operation of the robot 401 based on the result of the image processing by the image processing device 300. For example, in the operation of connecting the works W1 and W2, the robot controller 400 refers to the positional relationship between the two works obtained from the image processing, and the robot arm 402 and the robot are connected so that the two are connected in a predetermined positional relationship. The position and orientation of the hand 403 are controlled.

  FIG. 3 shows an example of the configuration of the image processing apparatus 300 of this embodiment. The image processing apparatus 300 of FIG. 3 performs image processing, specifically, pattern matching processing, and can be configured using an architecture such as a PC (personal computer), for example. The image processing device 300 includes a CPU (Central Processing Unit) 301 as a calculation unit (control device). The image processing apparatus 300 also includes a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, and a HDD (Hard Disk Drive) 304 as storage units. The image processing apparatus 300 also includes a recording disk drive 305 and various interfaces 311 to 315.

  A ROM 302, a RAM 303, an HDD 304, a recording disk drive 305, and interfaces 311 to 315 are connected to the CPU 301 via a bus 310.

  A boot program such as BIOS is stored in the ROM 302. The RAM 303 is a storage device that temporarily stores various data such as the calculation processing result of the CPU 301. A program 321 is stored (recorded) in the HDD 304. Then, the CPU 301 reads out and executes the program 321, thereby executing each step of the image processing method described later. The recording disk drive 305 can read various data, programs, etc. recorded on the recording disk 322. The recording disk 322 may be a semiconductor memory device or the like in addition to various magnetic disks and optical disks. When the recording disk 322 stores the image processing program describing the control procedure of the present invention, the recording disk 322 constitutes the computer-readable recording medium of the present invention.

  The robot control device 400 is connected to the interface 311. The CPU 301 transmits the image processing result to the robot controller 400 via the bus 310 and the interface 311. The robot arm 402 controlled by the robot control device 400 constitutes a production device that executes a production process for the works W1 and W2 using the result of the pattern matching of the image processing device 300. The pattern matching of the image processing apparatus 300 is used, for example, to control the production process by the robot arm 402 that recognizes the relative positions and postures of the works W1 and W2 and assembles them. Alternatively, the pattern matching of the image processing apparatus 300 may be performed, for example, for inspecting each of the works W1 and W2 or an assembly in which they are integrated. In that case, the production process by the robot arm 402, for example, the change of the transfer destination of the work or assembly can be controlled according to the inspection result. The image processing using the pattern matching of the image processing apparatus 300 can be used in various forms for the above-mentioned production processing control in an article production system (line).

  The camera 500 is connected to the interface 312. The CPU 301 transmits a control signal such as a trigger signal to the camera 500 via the bus 310 and the interface 312. The CPU 301 acquires the image signal from the camera 500 via the bus 310 and the interface 312. The acquired image data is stored in a storage device such as the HDD 304 or the external storage device 800 under the control of the CPU 301.

  The interface 313 is connected with an input device 600, such as a mouse or a keyboard, which transmits a command according to an operation performed by an operator to the CPU 301. A display device 700 such as a display for displaying an image is connected to the interface 314. A rewritable nonvolatile memory such as a USB memory or an external storage device 800 such as an external HDD is connected to the interface 315.

  Hereinafter, the flow of model creation by the image processing apparatus 300 and image processing by pattern matching will be described. The CPU 301 of the image processing apparatus 300 reads the program 321 from the HDD 304 and executes the program 321 to execute each step of the following image processing method.

  FIG. 4 shows an outline of the model creating process. In FIG. 4, an operator or an operator who is a user of this system prepares a reference image for creating a model for pattern matching, in order to prepare a reference image, the same lighting conditions and positions as when actually performing production on a production line. Set the workpiece to be detected in. The CPU 301 causes the camera 500 to capture an image of the work, and acquires image data as a reference image of the work from the camera 500 (S100).

  The reference image may be already stored in a storage device such as the HDD 304 or the external storage device 800. In that case, the CPU 301 retrieves the data of the reference image from the storage device such as the HDD 304 or the external storage device 800. Will be acquired. The reference image is taken under the same conditions as the actual shooting in the monitoring process on the production line.

  The CPU 301 displays the reference image on the display device 700, and sets a rectangular area including a portion in which the reference object (work) is reflected or a characteristic portion of the reference object, which is designated by the operator, as a temporary model creation area. (S200). When setting the temporary model creation area, the operator can use a user interface using an input device 600 such as a mouse. The setting of the temporary model creation area is performed by an operation in which the user clicks two points at the upper left and the lower right of the area including the image in which the reference object is reflected in the reference image displayed on the display device 700.

  In step S200, the CPU 301 sets a rectangular area from the positions of the two clicked points (upper left and lower right corner points of the rectangular area) as an area for creating a temporary model. The position and shape of the set temporary model creation area are stored in the RAM 303 or the HDD 304. Although the method of setting in the rectangular area has been described above, a circular area may be set by clicking three points or an arbitrary area connecting the click points may be set. It is not limited to.

  Next, a method of creating a temporary model will be described. FIG. 5 shows a detailed processing example of the temporary model creation process (FIG. 4: step S300). The CPU 301 that executes this temporary model creation step constitutes a functional block as a temporary model creation unit in the image processing apparatus 300, for example. In step S301 of FIG. 5, the CPU 301 performs a process of extracting a connected edge in which edge points are connected in the area set in step S200 in the reference image acquired in step S100. The details of the connection edge extraction processing (step S301) will be described below.

  The CPU 301 calculates the edge strength and the edge direction for each pixel in the reference image area of the connected edge extraction target. The edge strength represents the magnitude of contrast (the steepness of change in luminance), and the higher the contrast of the pixel adjacent to the target pixel, the higher the edge strength. The edge direction represents the direction of contrast, and indicates the direction (angle) in which the edge strength of the target pixel is maximized. To calculate the edge strength, for example, a Sobel filter in the x-axis direction (horizontal axis direction of the image) and the y-axis direction (vertical axis direction of the image) is used.

  FIG. 6 shows the operation of performing the extraction processing of the edge strength and the edge direction on the reference image area Im. In this extraction processing, the CPU 301 first calculates the edge strength Ex in the x-axis direction and the edge strength Ey in the y-axis direction at the target pixel C in the reference image area Im in FIG. This edge strength vector is represented by (Ex, Ey).

  Then, the CPU 301 calculates the edge strength E of the target pixel C as the square root of the sum of squares of the edge strengths Ex and Ey of each axis, for example, as shown in the following expression (1).

  Further, the CPU 301 calculates the edge direction θ by the following formula (2).

  Then, the CPU 301 calculates the edge strength and the edge direction of all the pixels in the reference image area Im by the above method. Further, the CPU 301 extracts pixels having an edge strength equal to or greater than a certain threshold as edge features (edge points), and generates an edge feature image that is a set of a plurality of edge points.

FIG. 7 shows an example of the edge feature image I _T. In FIG. 7, in the edge feature image I _T , pixels showing edge features whose edge strength is equal to or higher than a certain threshold are shown in black as edge points, and pixels whose edge strength is less than the threshold are shown in white as non-edge points. ing.

As a data holding method of the edge feature image I _T , for example, a 2-channel data area is secured in a storage area such as the RAM 303 for each pixel of the image as follows. Then, for example, in the case of the pixel C ₁ extracted as an edge point as shown in FIG. 7, the edge strength is stored in the first channel, and the edge direction information is stored in the second channel. In addition, a value (for example, 0) indicating that the edge feature is not displayed is stored in the pixel C ₂ that is determined to be the non-edge point. A set ΣC of pixels C ₁ indicating these edge points is the edge feature image I _T.

  It should be noted that two images, an edge strength image storing only the edge strength and an edge direction image storing only the edge direction, may be used as one set of feature images. Further, in the present embodiment, the Sobel filter is used for calculating the edge strength, but a commonly known edge extraction filter such as a Canny filter may be used.

  Next, a connected edge in which the edge points are connected is extracted, and labeling is performed for each connected edge. Four adjacent directions of pixels extracted as edge points by the above-described edge extraction processing or eight oblique directions are searched, and if pixels that are edge points are adjacent to each other, they are stored as connected components of the same connected edge. To go. This makes it possible to extract connected edges in which individual connected components are labeled as a unit. If the portion represented by the line in the image 8000 illustrated in FIG. 8 is the connected edge extracted, five connected edges 801 to 805 are extracted.

  Then, in S304A of FIG. 5, a temporary model is created for one combination of connected edges. Further, by repeating the loop of S302 to S304B, temporary models are created for all combinations of connecting edges. That is, for the connected edges extracted in step S301, while changing the combination of connected edges as shown in FIGS. 9A to 9C (S303), a temporary model is created (S304B). For example, the provisional model 901 shown in FIG. 9A is a provisional model regarding a combination method for selecting the connection edge 802 among the five connection edges. Further, the temporary model 902 shown in FIG. 9B is a temporary model regarding a combination method for selecting the connection edge 801 and the connection edge 802. The temporary model 903 shown in FIG. 9C is a temporary model regarding a combination method for selecting the connected edge 801, the connected edge 803, and the connected edge 805.

  For example, when five connected edges are extracted as shown in FIG. 8, a temporary model for pattern matching is created for all the combination methods of the five connected edges and stored in the HDD 304 or the like. It is determined in step S302 whether or not the provisional models have been created for all the combinations of connected edges. If the provisional models have been created for all the combinations of connected edges, this process ends.

  When five connected edges are extracted, all five connected edges can be selected as a temporary model, or only one connected edge can be selected as a temporary model. In the case of five connected edges, the number of combined methods of the connected edges can be calculated by the following formula (3), for example, and there are 31 ways. That is, when five connected edges are extracted, 31 types of temporary models are created.

  Next, an evaluation value calculation step (FIG. 4: S400) of calculating an evaluation value for each of the plurality of temporary models created as described above is performed. In this evaluation value calculation step (S400), as shown in FIG. 10, one of a plurality of temporary models is selected, and the process of calculating the evaluation value of the temporary model is sequentially executed for all the temporary models. By doing. The CPU 301 that executes this evaluation value calculation step configures a functional block as an evaluation unit in the image processing apparatus 300, for example.

  In step S401 of FIG. 10, one temporary model is selected from the plurality of created temporary models. Pattern matching is executed using the selected temporary model and one of the plurality of evaluation images prepared in advance. The evaluation image is an image obtained by actually photographing a detection target such as a work, and an image photographed by the same system as the production apparatus 200 at the time of experiment or an actual image stored during production is used.

That is, as a matching score acquisition step, the CPU 301 reads one image from the plurality of evaluation images stored in the HDD 304 and executes pattern matching using one selected temporary model (S402). For example, the pattern matching method is as follows. First, the evaluation image is subjected to edge extraction to create an evaluation edge image Is including the edges existing in the evaluation image. The method of the edge extraction processing may be the same as the edge extraction processing for the reference image described above. As shown in FIG. 11, an extraction window having the same size as the currently selected temporary model is placed in the evaluation edge image I _S, and an image of a size corresponding to the temporary model is displayed from the evaluation edge image I _S. Extract I _I. The extraction window having the same size as the temporary model sequentially extracts the images I _I while shifting the position in pixel units. A matching score R indicating the degree of similarity between the image I _I extracted in this way and the temporary model is obtained by the following equation (4). In the following, for simplification, the matching score representing the similarity may be simply referred to as “score”.

  That is, the local score of one edge point is calculated by obtaining the cosine value of the edge direction difference between one edge pixel of the temporary model and one edge pixel of the extracted evaluation edge image. By performing this processing for all edge pixels, it is possible to calculate the local scores of all edge points and further calculate the sum thereof. By dividing the total sum of the local scores by the edge score of the tentative model, it is possible to obtain a value that is normalized in the range of -1 to 1 as the final score.

  The position (i, j) in the image of Expression 4 above is the position (i, j) in FIG. 11, that is, in the evaluation edge image obtained by extracting the image of the size corresponding to the temporary model from the evaluation edge image. Corresponds to the position of. The extraction window is moved within the entire range of parallel movement, and the matching score R is calculated. The score is represented by -1 to 1, with 1 indicating the highest similarity and -1 indicating the lowest similarity.

  The method of calculating the score for evaluation of the temporary model is arbitrary, and a score calculation method different from the above may be used. For example, the matching score may be calculated from the x-direction edge strength and the y-direction edge strength of the model and the evaluation edge image as in the following expression (5).

  When the score calculation at all the search positions in the evaluation edge image is completed, the pattern matching using the temporary model ends. If score calculation has not been completed at all search positions, the extraction window is advanced to the next search position (i, j) and calculation of matching scores is continued. In the present embodiment, the pattern matching method for detecting only the parallel movement of the model has been described for simplification, but the pattern matching for detecting the rotation and scale change of the model can be applied by the same method.

  When a matching score equal to or higher than a predetermined threshold is calculated when executing the above-described pattern matching, the matching score R and the position (i, j) are stored in the RAM 303. Thereby, as shown in FIG. 12, a score map for a certain evaluation image is generated. If the predetermined score threshold value is set to -1, a score map including scores of all positions in the evaluation image will be generated.

  Further, the CPU 301 uses the maximum detection score (first detection score) from the score map temporarily stored in the RAM 303 and the second position apart from the first position where the maximum detection score is obtained. The obtained second detection score is extracted (FIG. 10: S403). These extracted scores are stored in a storage unit such as the RAM 303. The method of extracting the two scores will be described in detail below.

  The maximum detection score (first detection score) is the maximum value of the matching score R shown in the score map, such as the peak of 1201 in FIG. That is, the maximum detection score is the first matching score having the first-ranked degree of similarity obtained in the evaluation image.

Further, the second matching score obtained at a position in the evaluation image different from the spatial position in the evaluation image from which the first matching score was obtained is acquired. The second matching score is, for example, as shown in FIG. 12, a detection score at the apex of the mountain 1202 having the second highest height different from the mountain 1201 including the apex at which the maximum detection score is detected. It can be said that the maximum detection score is the first largest local maximum value, but the second matching score is the second largest local maximum value. When only one maximum value appears in the score map, the minimum value in the score map is used as the second matching score.
If the accuracy error of matching, the error due to noise, the error due to individual difference, etc. are large, the matching score at the same position may shift and appear at a plurality of places. In such a case, it is preferable to set the maximum matching score as the second matching score at a position separated by a threshold value (predetermined distance) or more from the first position of the mountain 1201 where the maximum detection score is detected. .. The threshold value of the distance between the first position and the second position may be set in advance by calculating a value based on an error in matching accuracy, an error due to noise, an error due to individual difference, or the like. It may be set arbitrarily. For example, a position within 10 pixels in the X and Y directions of the image from the position of the mountain 1201 where the maximum detection score is detected can be defined as a range near the maximum detection score. In this case, the maximum score within a spatially separated range exceeding 10 pixels may be extracted and used as the second matching score.

  The predetermined distance does not necessarily have to be a position separated by a predetermined number of pixels or more with the position of the maximum detection score as the center. For example, as shown in FIG. 13, a temporary model is arranged at the position 1301 of the maximum detection score and the position 1302 of the second place, and it corresponds to the range excluding the neighborhood due to the overlapping mode of the image ranges of the two temporary models. You may decide whether to do it. The degree of overlap (overlap ratio) of the two temporary models is calculated as N / M from, for example, the area M of the temporary models and the area N of the overlapping portions of the two temporary models. If the degree of overlap (overlap range) is less than or equal to a predetermined overlap range (for example, less than or equal to a predetermined threshold value), it is determined that the position of the second detection score is a predetermined distance or more away from the position of the maximum detection score. When the predetermined threshold of the degree of overlap is set to 0.5, if the temporary model placed at the position of the maximum detection score and the temporary model at other positions overlap by 50% or more, it is regarded as closer than the threshold. If it is smaller than 50%, it is considered that the maximum detection score position is more than a threshold value away, and the score obtained at that position is adopted as the second matching score.

  As described above, the matching score of the first place and the matching score of the second place can be acquired from the one evaluation image and the one provisional model with respect to the matching score corresponding to the similarity of the images. The acquired first matching score and the second matching score obtained at a position away from the position where the first matching score is obtained are stored in the RAM 303. The CPU 301 that executes this matching score acquisition step constitutes a functional block as a matching score acquisition unit in the image processing device 300, for example.

  Then, the CPU 301 executes the above steps S402 and S403 while confirming whether the same processing has been performed on all prepared evaluation images (S404). The first matching score group, which is the set of first-ranked matching scores, and the second matching score group, which is the second-ranked matching score set, are stored in the RAM 303. When the processing of all the evaluation images is confirmed in step S404, the process proceeds to the next step S405. As described above, it is possible to acquire the first matching score group and the second matching score group using a plurality of evaluation images for a certain temporary model.

  When the processing for all the evaluation images for a given temporary model is completed, a table (table data) of the first matching score group and the second matching score group for the temporary model can be obtained. For example, as shown in FIG. 14, a table (table data) of the first and second matching score groups extracted for each evaluation image is obtained on the RAM 303. FIG. 14 shows an example in which there are five evaluation images. 1401 is a number for identifying an evaluation image, 1402 and 1403 are respectively the first matching score (maximum detection score) in the evaluation image and the second at a position separated by a predetermined distance or more from the position of the maximum detection score. It is the value of the matching score of the place.

  In step S405 of FIG. 10, the CPU 301 calculates the evaluation value for the specific temporary model selected in step S401 as follows. For example, the evaluation value of this temporary model is calculated based on the score table (table data) of FIG. 14 created by the processing up to step S404.

  FIG. 15 shows the calculation result of the evaluation value for this one temporary model. In FIG. 15, reference numeral 1501 is a number for identifying the evaluation image, 1502 and 1503 are the maximum detection score in the evaluation image, and the value of the second highest score apart from the position of the maximum detection score (other than the neighborhood). Is. Further, 1504 is the difference between the maximum detection score and the second rank score, and 1505 is the average value, the maximum value, and the minimum value of the maximum detection score and the second rank score of each evaluation image.

  In the present embodiment, for example, as the evaluation value for a given temporary model, the maximum detection score obtained from a plurality of evaluation images and the second highest position apart from the position where the maximum detection score is detected (other than the neighborhood) are used. Use the minimum difference from the score. For example, in the example of the temporary model in which the calculation result of the evaluation value is shown in FIG. 15, the difference (1504) obtained in the third evaluation image is the smallest, and the evaluation value of this temporary model is 0.25. ..

  Note that other methods of calculating the evaluation value for the temporary model can be considered. For example, one of the evaluation value calculation methods is to calculate the average value of the maximum detection scores of each image and the average value of the second scores other than the vicinity of the position where the maximum detection score is detected, and evaluate the difference. It is a method of setting the value. According to this method, in the case of the tentative model in which the calculation result of the evaluation value is shown in FIG. 15, as shown by 1505, the two average values, which is the difference between 0.952 and 0.572, is 0.38. It becomes the evaluation value of the temporary model.

  Another one of the calculation methods of the evaluation value is to calculate the minimum value of the maximum detection score of each image and the maximum value of the second score obtained outside the vicinity of the position where the maximum detection score is obtained, The difference is used as the evaluation value. According to this method, in the case of the temporary model in which the calculation result of the evaluation value is shown in FIG. 15, there is a difference between 0.87 and 0.67 as indicated by 1505, so 0.2 is the value of this temporary model. It becomes an evaluation value. Those skilled in the art may appropriately adopt a method other than the above-mentioned several methods as the method of calculating the evaluation value for the temporary model. For example, using the values in the table of FIG. 15, one evaluation value is calculated for one temporary model as appropriate.

  Further, the CPU 301 executes the processes of steps S401 to S405 of FIG. 10 for all prepared temporary models. When it is determined that the calculation of the evaluation values has been completed for all the temporary models (S406), the temporary model evaluation value calculation processing (FIG. 4: S400) shown in detail in FIG. 10 is completed.

  Finally, the CPU 301 selects the temporary model having the maximum evaluation value from the evaluation values of the temporary models created in step S400, and determines the temporary model to be used as pattern matching during production (FIG. 4 :). S500). This determined model is stored in a storage unit, for example, the HDD 304 so that it can be used at the time of actual pattern matching. The CPU 301 that executes this determination step (FIG. 4: S500) constitutes a functional block as a determination unit in the image processing apparatus 300, for example.

  As described above, the automatic model creating process is completed. The CPU 301 executes the pattern matching between the model created in the above step and the image captured by the camera 500, so that the actual production device 200 operates. The CPU 301 that executes this pattern matching step constitutes a functional block as a pattern matching unit in the image processing apparatus 300, for example.

  As described above, according to the present embodiment, in order to determine a model for pattern matching with few false detections, a plurality of temporary models are created, pattern matching is performed on the evaluation image, and the most of them is executed. A temporary model with a high evaluation value will be adopted as this model (official model). As the evaluation value, for example, a difference between the maximum detection score when the evaluation image is pattern-matched and the second matching score detected outside the vicinity of the position where the maximum detection score is detected is adopted. .. That is, the position where erroneous detection is most likely to occur is a peak position formed at a position different from the peak 1201 having the highest matching score R, such as the peak 1202 in the example of the score map of FIG. .. Therefore, this is adopted as the second matching score other than the neighborhood of the maximum detection score. The larger the difference between the second matching score of the mountain 1202 that is more likely to cause this false detection and the maximum detection score detected by the mountain 1201 that is the correct position for detection, the larger the matching model, the higher the probability of false detection. Is reduced.

  Then, according to the present embodiment, it is possible to automatically select, from a plurality of temporary models, a model in which the difference between the two scores is as large as possible. Further, in the present embodiment, the model evaluation and selection can be performed in advance before the actual production is started on the production line, so that it is selected in advance during monitoring in the actual production process. Only one model can perform pattern matching. That is, in the present embodiment, it is not necessary to perform pattern matching using a large number of matching models in image processing performed for actual production line monitoring. Therefore, the monitoring process based on the image processing can be executed at high speed without increasing the image processing time. Further, even when the shape of the score map is deformed due to the influence of background, noise, work shape, etc. in various production processes, it can be applied for general purpose by using the difference between the two scores described above. Is possible. According to the present embodiment, it is possible to automatically create a model used for pattern matching that can be applied in various production processes and that can prevent false detection without increasing the processing time.

<Embodiment 2>
In the above-described first embodiment, a method of creating a model used for shape (edge) -based pattern matching has been described. The method of the present invention can also be applied to model creation for pattern matching by another method. Hereinafter, in the second embodiment, automatic creation of a model used for brightness-based pattern matching will be described.

The schematic configuration of the production apparatus according to the second exemplary embodiment and the configuration of the image processing apparatus are the same as those shown in FIGS. 2 and 3 of the first exemplary embodiment.
Hereinafter, model creation and pattern matching execution (image processing method) by the image processing apparatus 300 will be described. The CPU 301 of the image processing apparatus 300 reads out the program 321 from the HDD 304 and executes the program 321 to execute each step of the image processing method described below.

  The overall flow of model creation is the same as in FIG. 4 of the first embodiment. In particular, steps S100 and S200 of FIG. 4 are the same as those of the first embodiment, but in the present embodiment, the creation of a plurality of temporary models (FIG. 4: S300) is performed by the process shown in FIG.

  In the creation of the temporary model of FIG. 16, the area set in step S200 in the reference image acquired in step S100 of FIG. 4 is used as the reference area, and the size and shape of the reference area are changed to create a plurality of temporary models. The overall flow of the processing of FIG. 16 is similar to the flow of the processing of FIG. 5 described above. First, in step S309, a partial image is cut out from the reference image for the standard region having a certain size and shape, and the temporary model is created. Created and stored in HDD 304. Then, in step S310, steps S311 and S312 are repeatedly executed while determining whether or not the creation of the temporary model has been completed for each of all the reference regions whose sizes and shapes have been changed.

  In step S311 of FIG. 16, the definition (size and shape) of the reference area of the temporary model is changed. Here, if the reference area is rectangular, the width and height of the area are changed, and if the reference area is circular, the diameter is changed. Further, various shapes such as a rectangle and a circle may be changed with the center of the reference area as the reference position. In step S311, a region in which some one change is added to the reference region as described above is created.

  In step S312, a partial image corresponding to the area created in step S311 is cut out from the reference image and stored in the HDD 304 as a temporary model for pattern matching. In step S310, it is determined whether or not a predetermined number of temporary models have been created by changing the reference area and creation of a temporary model based on the change, and if this step is affirmed, the temporary model creation processing is performed. To finish. When step S310 is denied, it progresses to step S311 and repeats the said process.

  In the following, for example, it is assumed that the reference area is a rectangular area and the width and height of the given reference area are changed by 3 ways, respectively, to create 9 types of temporary models in total.

  In the present embodiment, the overall flow of the evaluation value calculation process of the temporary model is almost the same as in FIG. 10 described above. In step S401 of FIG. 10, one temporary model is selected from the created multiple temporary models. Subsequently, in step S402, pattern matching is performed on the evaluation image prepared in advance as in the first embodiment. This process is different from the first embodiment, and the following brightness-based pattern matching process is performed.

  For example, as shown in FIG. 11, a partial image having a size corresponding to the temporary model is extracted from the evaluation image while shifting the range of the extraction window having the same size as the temporary model in the evaluation image in pixel units. Then, the matching score R between the extracted partial image and the temporary model is calculated by the following equation (6).

  In the above formula (6), the position (i, j) in the image corresponds to the positions in the partial image and the temporary model in which the image of the size corresponding to the temporary model is extracted from the evaluation image. As for the value of this score, 1 indicates the highest similarity and −1 indicates the lowest similarity. When the score calculation at all the search positions in the evaluation image is completed, the pattern matching ends. If score calculation has not been completed for all search positions, (i, j) is advanced to the next search position, and score calculation is advanced.

  When a score equal to or higher than a predetermined score threshold is calculated when executing this pattern matching, the matching score R and the position (i, j) are stored in the RAM 303. As a result, as in the first embodiment, as shown in FIG. 12, a score map of a temporary model for a certain evaluation image is generated. If the predetermined score threshold is set to -1, score maps of all positions in the evaluation image will be generated.

  After the score map is generated as described above, steps S403 to S406 of FIG. 10 can be executed in the same manner as in the first embodiment to calculate the evaluation value for all the temporary models. Further, the model automatic setting (selecting this model from a plurality of temporary models) in step S500 of FIG. 4 can be performed in the same manner as in the first embodiment.

  As described above, when the pattern matching calculation method is different, for example, even when the brightness-based pattern matching is performed as in the present embodiment, if the score value calculation method is appropriately determined, the model is automatically created. Is possible. As a matter of course, when the model created as described above is actually subjected to pattern matching, the same brightness-based pattern matching as that at the time of model creation is executed. In the production line as shown in FIG. 2, the CPU 301 executes the pattern matching between the model created in the above step and the image captured by the camera 500, so that the actual production device 200 operates. ..

  According to the present embodiment, even when the pattern matching is performed on the basis of the brightness, it is possible to automatically create the matching model used for the image processing in the actual manufacturing process by the calculation using the score map. In the above, an example in which the pattern matching used for image processing is shape-based (connected edge) -based as in the first embodiment or luminance-based as in the second embodiment has been shown. .. However, the model generation technique by calculation using the score map of the present invention can be implemented in various pattern matching model creation regardless of the pattern matching processing mode, as long as pattern matching capable of calculating a matching score is performed. ..

  The present invention also provides a process of supplying a program that implements one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It is feasible. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  Reference numeral 1012 ... Model, 200 ... Production equipment, 300 ... Image processing device, 401 ... Robot, 500 ... Camera, 400 ... Robot control device, 301 ... CPU, 302 ... ROM, 303 ... RAM.

Claims (19)

A temporary model creating step of creating a plurality of temporary models for pattern matching using a reference image in which an object of pattern matching is recorded,
Pattern matching is performed between each of the plurality of temporary models and each of the plurality of evaluation images in which the pattern matching target is recorded, and between each of the plurality of temporary models and each of the plurality of evaluation images. A matching score of each of the plurality of temporary models, and a first matching score group in which the first matching score having the maximum similarity in each evaluation image is aggregated, and the similarity in each evaluation image. A second matching score group in which a second matching score smaller than the first matching score is collected, and a matching score acquiring step,
An evaluation step of calculating an evaluation value from the first matching score group and the second matching score group for each of the plurality of temporary models;
A determining step of determining a matching model from the plurality of temporary models based on the calculated evaluation value;
An image processing method comprising:   The image processing method according to claim 1, wherein in the matching score acquisition step, when the score map generated by the pattern matching between the temporary model and the evaluation image has a plurality of maximum values, the matching score is ranked first. Is set as the first matching score, and the maximum value of the second matching score is set as the second matching score.   The image processing method according to claim 1, wherein, in the matching score acquisition step, when the maximum value of the score map generated by the pattern matching between the temporary model and the evaluation image is one, the maximum value is set to the first value. Image matching method, and the minimum value in the score map is used as the second matching score.   The image processing method according to claim 2, wherein a threshold is set for the distance between the second position at which the second matching score is obtained and the first position at which the first matching score is obtained. The image processing method is characterized in that the maximum value of the second matching score in the region distant from the first position by the threshold value or more is the second matching score.   The image processing method according to claim 1, wherein, in the matching score acquisition step, the first matching score is obtained from a range apart from a spatial first position in the evaluation image by a predetermined distance or more. An image processing method in which the obtained maximum matching score is used as the second matching score.   The image processing method according to claim 1, wherein in the matching score acquisition step, when the temporary model is arranged at a spatial first position in the evaluation image from which the first matching score is obtained, An image in which the maximum value of the matching scores obtained by arranging the temporary model at a position such that the overlapping range with the temporary model arranged at the position is less than or equal to a predetermined ratio is the second matching score. Processing method.   The image processing method according to any one of claims 1 to 6, wherein the temporary model creation step is a user who receives an operation of a user who specifies a temporary model creation area for creating the temporary model from the reference image. An image processing method including an interface process.   The image processing method according to claim 1, wherein in the temporary model creating step, a plurality of connected edges in which edge points are connected are extracted from the reference image, and the plurality of extracted connected edges are extracted. An image processing method for creating the plurality of temporary models by combination.   The image processing method according to any one of claims 1 to 7, wherein in the temporary model creating step, the plurality of temporary models are defined by a plurality of regions having different sizes and / or shapes extracted from the reference image. Image processing method to create.   The image processing method according to claim 1, wherein in the evaluation step, the first matching score and the second matching score group are selected from the first matching score group and the second matching score group. An image processing method of calculating a difference between the second matching score and a second matching score, and calculating a maximum value or a minimum value of a difference between the calculated first matching score and the second matching score as the evaluation value. ..   The image processing method according to claim 1, wherein in the evaluation step, a difference between an average value of the first matching score group and an average value of the second matching score group is calculated. An image processing method for calculating the evaluation value.   The image processing method according to any one of claims 1 to 11, further comprising a pattern matching step of performing pattern matching using the matching model determined in the determination step. Each step of the image processing method according to claim 12,
A recognition step of recognizing the position and orientation of the work using the result of the pattern matching step;
A production control step of producing an article using the work, using the information obtained in the recognition step;
A method of manufacturing an article including. Each step of the image processing method according to claim 12,
An inspection step of inspecting the work using the result of the pattern matching step,
A production control step of producing an article using the work according to the inspection result of the inspection step;
A method of manufacturing an article including.   A program that causes a computer to execute the image processing method according to claim 1.   A computer-readable recording medium storing the program according to claim 15. A temporary model creation unit that creates a plurality of temporary models for pattern matching using a reference image in which an object of pattern matching is recorded,
Pattern matching is performed between each of the plurality of temporary models and each of the plurality of evaluation images in which the pattern matching target is recorded, and between each of the plurality of temporary models and each of the plurality of evaluation images. Find the matching score of
For each of the plurality of temporary models,
A first matching score group in which first matching scores having the highest degree of similarity in each evaluation image are collected,
A second matching score group in which a second matching score whose similarity is smaller than the first matching score in each of the evaluation images is collected;
A matching score acquisition unit that acquires
An evaluation unit that calculates an evaluation value from the first matching score group and the second matching score group for each of the plurality of temporary models,
An image processing apparatus, comprising: a determination unit that determines a matching model from the plurality of temporary models based on the calculated evaluation value.   The image processing apparatus according to claim 17, further comprising a pattern matching unit that performs pattern matching using the matching model determined by the determination unit. An image processing apparatus according to claim 18,
Using a result of pattern matching by the pattern matching unit performed on an image of a workpiece, a production device that executes a production process for the workpiece,
Production system with.

Images