JP2022009474A - System and method for detecting lines in vision system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a vision system tool for detecting line features in an acquired image, which efficiently and accurately identifies multiple lines.

SOLUTION: Provided are a system and a method for extracting edge points from an acquired image and detecting lines. Once lines are identified, it is trained in such a manner that a user associates predetermined (e.g., text) labels with the lines. These labels are used to define neural net classifiers. The neural net operates at runtime to identify and score lines in a runtime image detected by using a line detection process. The detected lines are displayed to the user with labels and associated probability score maps based on the neural net results. Unlabeled lines are generally regarded as having low scores, and are identified by an interface as un-flagged or irrelevant.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2022,JPO&INPIT

Description

Related Applications This application claims the benefits of Simultaneously Pending U.S. Patent Application No. 62/24918, "Systems and Methods for Detecting Lines in Vision Systems," filed February 11, 2015. The teachings are incorporated herein by reference.

Description Field The present invention relates to a machine vision system, more specifically a vision system tool for detecting line features in an acquired image.

Machine vision systems (also referred to herein simply as "vision systems") are used for a variety of tasks in manufacturing, logistics and industry. Such tasks include inspection of surfaces and parts, alignment of objects in the assembly process, reading of patterns and ID codes, and other operations in which visual data is acquired and interpreted for use in subsequent processes. You may. A vision system typically uses one or more cameras to capture an image of a scene containing an object or object of interest. The object / object may be stationary or in relative motion. Motion can also be controlled by the information obtained by the vision system, as in the case of manipulating parts with a robot.

A common task of vision systems is to detect and characterize line features in images. Various tools are used to identify and analyze such line features. Typically, these tools rely on the clear contrast differences that occur in the parts of the image. This contrast difference, eg, a caliper tool, is analyzed to determine if individual points in the image with the contrast difference can be aggregated into line-like features. If it is determined that it can be done, the line is identified in the image. In particular, the tool for detecting edge points and the tool for fitting lines to points operate independently of each other. This increases the processing overhead and reduces reliability. If the image contains multiple lines, such tools will have limited ability to pinpoint them. In addition, tools designed to detect single lines in an image may be problematic to use if the image is densely populated with multiple lines of similar orientation and polarity.

Another difficulty is that the lines of the object can sometimes be hidden or obscured in the captured image. Users are uncertain about identifying the detected lines, and mechanisms that can identify such lines individually may include sophisticated rules and scripting, which is the time for vision system setup and training tasks. And increase the cost.

The present invention overcomes the shortcomings of the prior art by providing systems and methods for detecting line features in images that allow multiple lines to be efficiently and accurately identified and characterized. Once the lines are identified, the user can train the system to associate a given (eg, text) label with such lines. These labels (also referred to herein as "tags") can be used to define neural net classifiers. The neural network operates at runtime to identify and score lines in the detected runtime image using the line detection process. The detected lines can be displayed to the user with labels and associated probability score maps based on the neural net results. Unlabeled lines are generally considered low scores and are identified by the interface as unflagged or irrelevant.

In an exemplary embodiment, a system and method for detecting a line in an acquired image is provided. The system and method includes a vision system processor and an interface associated with this vision system processor, which is a separate label for the relevant line identified by the line detection process in the training image of the object. Allows creation. The run-time line detection process identifies the lines in the acquired image, and the neural net process uses a classifier based on the label to determine a probability map of the line features for the label. The run-time result generation process provides labels and probability scores for at least one relevant line. Illustratively, the runtime result generation process provides a probability score for irrelevant lines and / or includes an interface for highlighting the line, providing the probability score associated with the highlighted line. The probability score map may be similar in size to the acquired image. The line detection process has a processor that receives image data of the scene including line features and is equipped with an edge point extractor, which (a) calculates a gradient vector field from the image data and (a) b) the gradient vector field is projected onto a plurality of gradient projection subregions, and (c) multiple edge points are detected in each of the gradient projection subregions based on the projected gradient data. The processor also includes a line finder that produces multiple lines that match the edge points extracted from the image. Illustratively, the linefinder operates a RANSAC-based process to fit an inlier edge point to a new line and iteratively define a line from the outlier edge point relative to a predefined line. include. The projection of the gradient field can be oriented along the direction set according to the expected orientation of one or more line features and / or the grain size can be defined based on the Gauss kernel. The edge point extractor can be configured to detect multiple gradient intensity maximums in each of the gradient projection subregions. The maximum gradient intensity is identified as some of a plurality of edge points, respectively, and is described by a position vector and a gradient vector. The line finder provides a metric based on the distance of at least one edge point from at least one candidate line and the angle difference between the gradient direction of at least one edge point and the normal direction of at least one candidate line. By calculation, it may be configured to determine a match between at least one edge point of the plurality of extracted edge points and at least one candidate line of the plurality of detected lines. Illustratively, image data includes data from multiple images acquired from multiple cameras. This transforms the image into a common coordinate space.

According to an exemplary embodiment, there is provided a system for detecting line features in an image acquired based on one or more cameras. The system and method comprises a vision system processor and an interface associated with the vision system processor, which allows the creation of individual labels for the relevant lines identified by the line detection process in the training image of the object. Become. Positions the lines in the image acquired by the running line detection process and locates the statistical or K-NN classifiers that generate labels for the interface based on the lines positioned by the line detection process.

The present invention will be described below with reference to the drawings.

FIG. 6 is a diagram of an exemplary vision system configuration and vision system processor including an edge detection tool / module and associated label interface processor / module to acquire an image of an object containing a plurality of edge features, according to an exemplary embodiment.

It is a figure which shows the whole system and method for extracting an edge point from the acquired image and detecting a line by an exemplary embodiment.

It is a flowchart of the edge point extraction procedure by the system and method of FIG.

FIG. 3 is a diagram for projecting a field onto an area of an image including edge features, which is part of the edge point extraction procedure of FIG.

It is a figure which shows that the Gauss kernel is applied to the image in order to smooth the image used in the edge point extraction procedure of FIG.

FIG. 3 is a diagram of a field projection that includes the application of a Gaussian kernel to smooth the projection used in the edge point extraction procedure of FIG.

FIG. 3 is an overview of the edge point extraction procedure of FIG. 3, including application of the Gaussian kernel and calculation of absolute and normalized contrast thresholds for edge points.

It is a figure which shows the area of the contrast qualified for the edge point which has a sufficient absolute contrast threshold and the normalized contrast threshold.

FIG. 3 is a flow chart of a line detection procedure based on the edge points detected in FIG. 3 using an exemplary RANSAC according to an exemplary embodiment.

It is a figure which shows the inaccurate alignment and the accurate alignment of an edge point with respect to a dense parallel line feature. It is a figure which shows the inaccurate alignment and the accurate alignment of an edge point with respect to a dense parallel line feature.

It is a figure which shows the accurate alignment and the inaccurate alignment edge point of the edge point with respect to the crossing line feature which can be resolved by the line finder of an exemplary system and method respectively. It is a figure which shows the accurate alignment and the inaccurate alignment edge point of the edge point with respect to the crossing line feature which can be resolved by the line finder of an exemplary system and method respectively.

FIG. 6 shows light to dark polarities, dark to bright polarities, bright to dark or dark to bright polarities, or mixed polarities that can be resolved by linefinders of exemplary systems and methods. FIG. 6 shows light to dark polarities, dark to bright polarities, bright to dark or dark to bright polarities, or mixed polarities that can be resolved by linefinders of exemplary systems and methods. FIG. 6 shows light to dark polarities, dark to bright polarities, bright to dark or dark to bright polarities, or mixed polarities that can be resolved by linefinders of exemplary systems and methods. FIG. 6 shows light to dark polarities, dark to bright polarities, bright to dark or dark to bright polarities, or mixed polarities that can be resolved by linefinders of exemplary systems and methods.

It is a figure which shows the correction of the coverage score for the line detected in consideration of a user-defined mask.

It is a flowchart showing a procedure for training a line finder using an interface including a label / tag that refers to a line of interest in an object image.

FIG. 19 illustrates an exemplary user interface display including a dialog box used to perform the training procedure of FIG. 19 and an image of a training object having a line of interest.

In the flow chart of the run-time procedure for detecting a line using a line finder trained according to Figure 19, including assigning the label to the line of interest and a neural net classifier-based probability score for the detected line. be.

FIG. 6 shows an exemplary user interface display containing an image of an object with lines detected based on the line finder runtime procedure according to FIG. 21, showing labels and probability scores for irrelevant lines.

FIG. 6 shows an exemplary user interface display containing an image of an object with a line detected based on the line finder runtime procedure according to FIG. 21, showing labels and probability scores for the line of interest.

I. System overview

An exemplary vision system configuration 100 that can be used according to an exemplary embodiment is shown in FIG. The system 100 includes at least one vision system camera 110 and one or more additional optional cameras 112 (indicated by virtual lines). Exemplary cameras 110, 112 include an image sensor (or imager) S and related electronics that acquire an image frame and transmit it to the vision system process (processor) 130, where the vision system process (processor) 130 is a stand-alone processor. And / or may be exemplified as a computing device 140. The camera 110 (and 112) may include a suitable lens / optical system 116 focused on the scene containing the object 150 under examination. Cameras 110 (and 112) may include internal and / or external illuminators (not shown) that operate according to the image acquisition process. The computing device 140 may be any acceptable processor-based system capable of storing and manipulating image data according to exemplary embodiments. For example, the computing device 140 may include a PC (illustrated example), a server, a laptop, a tablet, a smartphone or other similar device. The computing device 140 may include a suitable terminal device, such as a bus-based image capture card interconnected with a camera. In an alternative embodiment, the vision processor can be partially or completely contained within the camera body itself and can be networked with other PCs, servers and / or camera-based processors that share and process image data. .. The computing device 140 optionally includes a suitable display 142, which may support a suitable graphical user interface (GUI) that can operate according to the vision system tools and processor 132 provided within the vision system process (processor) 130. can. Note that in various embodiments the display can be omitted and / or provided only for setup and service functions. Vision system tools are part of any acceptable software and / or hardware package approved for use in the inspection of objects, such as those commercially available from Cognex Corporation, Natick, Mass., USA. good. The computing device may also include related user interface (UI) components, including, for example, a keyboard 144 and a mouse 146, as well as a touch screen inside the display 142.

The camera 110 (and 112) captures part or all of the object 150 located inside the scene. Each camera defines an optical axis OA, and the field of view is determined based on the optical system 116, the focal length, and the like around the optical axis OA. The object 150 includes a plurality of edges 152, 154, each arranged in a different direction. For example, the edge of the object may include the edge of the cover glass assembled inside the smartphone body. Illustratively, a camera can capture an entire object or a specific location (eg, a corner where glass intersects the body). The (common) coordinate space can be determined with reference to an object, one camera or another reference point (eg, a movable stage on which the object 150 is mounted). As shown, the coordinate space is represented by the axis 158. These axes exemplify the x-axis, y-axis and z-axis that are orthogonal to each other, and the times Θz centered on the z-axis on the xy plane.

According to an exemplary embodiment, the vision system process 130 collectively interacts with one or more applications / processes (running on a computing device 140) that include a set of vision system tools / processes 132. These tools are a variety of idiomatic and specialized applications used to resolve image data-for example, a variety of calibration tools that can be used to transform acquired image data into a given (eg, common) coordinate system. And affine transformation tools may be included. A tool for converting image grayscale intensity data into a binary image based on a predetermined threshold can also be included. Similarly, tools can be provided to analyze the intensity gradient (contrast) between adjacent image pixels (and subpixels).

The vision system process (processor) 130 includes a line detection process, tool or module 13 that locates a plurality of lines in an image acquired according to an exemplary embodiment. Here, an overview of the line detection procedure 200 according to an exemplary embodiment with reference to FIG. 2 is illustrated. Procedure 200 consists of two main parts. The input image 210 is provided to the processor. As shown, the image contains a pair of intersecting edges 212 and 214. These can represent the corner areas of the object 150 described above. The edge point extractor 220 processes the input image 210 to obtain a set of candidate edge points 230 including edge points 232 and 234 existing along the edges 212 and 214, respectively. Edge points 232, 234 and related data (eg, intensity gradient information described below) are provided to the recursive line finder 240, which performs a series of iterative processes at the selected edge points. The goal of the iterative process is to try to fit other detected edge points to the candidate line features. As a result of the line detection process 240, lines 252 and 254 are detected as shown. These results may be provided for other downstream processes using information 260-eg alignment processes, robotic operations, inspections, ID reads, parts / surface inspections, etc.

II. Line detection process (processor)

Refer to FIG. 3, which describes a procedure for extracting edge points according to one embodiment. One or more images of the scene containing an object or surface with edge features to be detected are acquired (step 310). Images can be extracted by a single camera or multiple cameras. In either case, in step 320 the image pixels can be (optionally) transformed into a new coordinate space and / or a common coordinate space with the appropriate calibration parameters. This step may also include image smoothing as described below. In embodiments, a plurality of cameras capture a discontinuous area of the scene-for example focusing on a corner area of a larger object-in which a common coordinate space can occupy an empty area between the camera fields of view. As described below, a line extending between such fields of view (eg, an object edge connecting two detected corner areas) is an exemplary embodiment system and method. And can be extrapolated by method. In step 330, the edge points required to detect the line are extracted from the image using gradient field projection by an edge point extractor in a suitable coordinate space. The gradient value is first calculated for each pixel to produce an image for the x-gradient component and the y-gradient component. The image is further processed by projecting a gradient field onto a number of caliper-like areas. Unlike conventional caliper tools that project intensity values, by projecting a gradient field according to an embodiment, the orientation of the gradient can be preserved, which facilitates subsequent line detection processes as described below.

In step 340, also referring to the diagram of FIG. 4, the portion of the image (caliper-like area) 400 containing the candidate edge features is a projection of the gradient field (represented by multiple projections 410, 420, 430). It receives and is searched over the (approximately) expected edge orientation in the search direction (arrow SD), and the projection is repeated across the area 400 in the orthogonal projection direction (arrow PD). For each projection (eg, projection 420) the edges appear as maxima in the gradient field 440 associated with the projection. In general, the set of edge points inside the projection associated with the edge show an intensity gradient (vectors 552, 554) perpendicular to the extending direction of the edge. As described below, the user can define the projection direction based on the expected line orientation. Alternatively, this may be provided by a predetermined mechanism or other mechanism-eg, analysis of features in the image.

The gradient projection step described above includes two particle size parameters. Prior to the gradient field calculation, the user can choose to smooth the image using an anisotropic Gaussian kernel. The first particle size determines the size of this Gaussian smoothing kernel. As shown in FIG. 500, a Gaussian kernel of approximately size (eg, large 512, medium 514, small 516) is applied to smooth the image 210. Therefore, the first particle size parameter determines the size of the isotropic Gaussian smoothing kernel prior to the field calculation.

This causes the process to perform a Gauss-weighted projection after the gradient field calculation, rather than a uniform weighting in the conventional caliper tool. The second particle size parameter therefore determines the size of the one-dimensional (1D) Gaussian kernel used during field projection, where the area 600 is the Gaussian smoothing kernel 610, 620, 630 as shown in FIG. Receive. During a typical operation, the user validates all the extracted edges superimposed on the image (using the GUI), and then the number of extracted edges along the line to be detected seems to be sufficient. Adjust the grain size and contrast thresholds up to, while avoiding an excessive number of edges due to background noise in the image. In other words, this step makes it possible to optimize the signal-to-noise ratio for image features. This adjustment can also be performed automatically by the system using default values in various embodiments. Note that the Gauss weighting function is one of a variety of measures for weighting projections, including (eg) uniform weighting.

The overall flow of gradient field extraction and projection is illustrated in Figure 700 of FIG. Two particle size parameters, an isotropic Gaussian kernel 710 and a 1D Gaussian kernel 720, are shown in each half of Figure 700. As shown, each acquired image 210 undergoes smoothing and decimation 730. The resulting image 740 then produces two gradient images 752 and 754 through gradient field calculation 750 as described above. These gradient images are also represented as g _x and _gy , each representing two orthogonal axes in a common coordinate space. Note that in addition to the two gradient images, the intensity image 756 typically undergoes a smoothing, decimation and projection process 760 (using a Gauss-weighted projection 770 based on the 1D Gaussian kernel 720). This is because the processed intensity information is also used to calculate the normalized contrast according to the embodiments described below. The result is a projection profile of the gradient images 772 (g _x ), 774 ( _gy ) and the intensity image 776.

Also with reference to step 350 in step 300 (FIG. 3), qualified edge points are extracted by combining the 1D projection profiles of the x-gradient image and the y-gradient image. This is achieved using a raw contrast calculation 780 and a contrast calculation 790 normalized based on the intensity image. More specifically, local peaks, each with both a raw projected gradient magnitude above the threshold and a normalized projected gradient magnitude, are detected by subsequent lines according to the following exemplary equation: Is considered a candidate edge point for.
(G _x ² + _gy ² ) ^1/2 > T _ABS
(G _x ² + _gy ² ) ^1/2 / I> T _NORM
Here, g _x and gy are the values of the x-gradient projection and the _y -gradient projection at the pixel location, I is the intensity, _{TAB is the absolute contrast threshold for the magnitude of the raw projected gradient, and TNORM is} the intensity normal. It is a normalized contrast threshold for the magnitude of the projected projected gradient.

In particular, a point is considered a candidate edge point only if its absolute contrast and normalized contrast both exceed their respective thresholds. This is shown by the upper right quadrant 810 in the exemplary graph 800 of the contrast threshold T _ABS for the normalized contrast threshold T _NORM . Using double thresholds (absolute and normalized) is generally different from existing strategies that typically use absolute contrast thresholds. The advantage of the dual contrast threshold is evident when, for example, an image contains both dark and light intensity areas, all of which contain edges of interest. It is desirable to set a low contrast threshold to detect edges in dark areas of the image. However, setting such a low contrast can result in the detection of false edges in bright areas of the image. Conversely, it is desirable to set a high contrast threshold to avoid detecting false edges in bright areas of the image. However, with high contrast settings, the system may not be able to properly detect edges in dark areas of the image. By using a second normalized contrast threshold in addition to the traditional absolute contrast threshold, the system can properly detect edges in both dark and bright areas, which is incorrect in the bright areas of the image. You can avoid detecting edges. Therefore, while avoiding false edges while allowing detection of applicable edges, the use of dual contrast thresholds maximizes the speed and robustness of subsequent line detection stages of the overall process. Useful for.

Further referring to step 350 (FIG. 3), once all the edge points have been extracted, they are reproduced and stored in a convenient data structure for subsequent linefinders to operate. Note, for example, the following tuple.
p = (x, y, gx, gy, gm, go, I, gm / I, m, n)
Here, (x, _y ) is the location of the edge point, (g _x , gy) is the value of the _x -gradient projection and the y-gradient projection, respectively, and (g _m , go) is calculated from (g _x , g _y ). The magnitude and orientation of the gradient, I is the intensity at the edge point, g _m / I is the intensity normalized contrast obtained by dividing the gradient magnitude g _m by the intensity I, m is the image index, and n. Is the projected area index. Edge point locations, such as those in standard caliper tools, can be interpolated to improve accuracy.

It should be noted that the edge point extraction process generally operates to project the field in a single direction that substantially matches the expected line angle. Therefore, the tool is most sensitive to edges at this angle, its sensitivity gradually decreases with respect to edges at other angles, and the rate of decrease indirectly determines the length of the field projection. Depends on the settings of. As a result, the process is limited to detecting lines that are "close" to the expected line angle, provided that the angle range is specified by the user. The process is adapted to detect non-orthogonal lines, but in various embodiments any angle across 360 degrees by performing projections in multiple directions, including orthogonal directions (omnidirectional line detection). It is supposed to be generalized so that the line of can be detected.

Referring to step 360 of step 300 (FIG. 3), edge point candidates exceeding the threshold are provided to the line finder according to an exemplary embodiment. For example, the linefinder works recursively and employs (eg) random sample consensus (RANSAC) based technology. The line detection procedure 900 in FIG. 9 is also referred to. In step 910, the user (eg) GUI with the expected maximum number of lines in one image, along with the expected angle, angle error, distance error and (exemplarily) minimum coverage score (generally defined below). Specified by. These parameters are used by the line finder to operate on the next process. Lines for each subregion of the image are detected by recursively running the RANSAC Line Finder, where the edge point outline from one stage becomes the input point for the next stage. Thus, in step 920, step 900 selects a pair of edge points that are a group portion of the edge points identified as extrema in the edge detection process. Step 900 attempts to fit the model line to the selected edge points based on the matching of the gradient values that match the model line (within the selected error range). In step 924, one or more line candidates from step 922 are returned. Each line detection stage returns candidate lines, their inliers and outliers. For the returned line, an inlier edge point with a position and slope that matches the line candidate is calculated (step 926). In step 928, the candidate line with the largest in-line account is identified. The line detection step described above (steps 920-928) ends when the maximum number of RANSAC iterations allowed is reached (decision step 930). The maximum number of iterations inside each line detection stage is automatically calculated using the internally calculated worst outlier ratio and the guarantee level specified by the user. Each line detection step returns the line with the maximum number of camped edge points from all its iterations, subject to user-specified fitting errors, geometric limits and polarities. Each edge point can be assigned to only one line inlier list, and each line is allowed to contain at most one edge point from each projection area. The direction of the slope of the edge point, along with its position, is used to determine if it should be included in the candidate line's inliarist. Specifically, the edge points should have a gradient orientation that matches the angle of the candidate line.

In the determination step 930, more iterations are allowed to determine that the outliers from the best inlier candidates (step 940) are returned for use by the RANSAC process (step 920) in the detection of the line candidates.

At each RANSAC iteration, two edge points belonging to different projection areas are randomly selected and a line is fitted to those two points. The resulting candidate line is further examined only to see if its angle matches the slope angle of both edges of the point pair and if the angle of the line matches the range of uncertainty specified by the user. In general, the edge point gradient direction is usually a right angle, but it is permissible to differ by a user-configured angular error. If the candidate line passes these initial tests, then the number of aligner edge points is evaluated and otherwise a new RANSAC iteration is started. An edge point is considered an aligner of candidate lines only if its slope direction and position match the line, based on the slope angle and distance error specified by the user.

Once the maximum number of RANSAC iterations has been reached (decision step 930), the best line candidate inliar detected is (eg) improved line fitting using least squares regression or other acceptable approximation method. The set of inliar edge points is re-evaluated and these steps are repeated up to N times (eg, 3 or more) until the number of inliar does not increase or decrease any further (step 960). This is a line output as the line detected in step 970.

Decision step 980 determines if more lines should be detected (by searching further (eg) further subregions or other criteria), if so the process loops back to step 920 and sets a new set. It operates at the edge point of (step 982). When the points are exhausted or the maximum number of iteration counts is reached, step 900 returns the set of lines detected in step 990 (in multiple images).

The multi-line finder is adapted to perform final adjustments to existing results when two lines intersect each other within the inspection area. As is commonly shown in FIGS. 10 and 11, for dense parallel lines 1010 and 1020, sometimes incorrect line results (ie, FIG. 10) are obtained due to the statistical accuracy of the RANSAC procedure. May be However, if such an error occurs, the exchange of align point groups (arrow 1120 in group 1110 in FIG. 11) can sometimes locate the correct line with increased coverage score and reduced fitting residue. Point exchange is extremely effective when the image contains dense parallel lines as shown. Conversely, if the images include lines 1210 and 1220 that actually intersect each other as illustrated in FIGS. 12 and 13, the coverage score is reduced after the point exchange (arrow 1230 in group 1240 of FIG. 12). ), The first results obtained prior to the exchange are preserved by the process of successfully detecting the intersection.

It should be noted that the RANSAC procedure is one of the various techniques that allows the line finder to fit points to the line. In an alternative embodiment, candidate points can be selected according to the set of displacements between them, or images can be processed using (eg) exhaustive search techniques. Therefore, references to RANSAC techniques as used herein should be construed to include a wide variety of similar point fitting techniques.

Additional functionality of this system and method can be provided. These include support for mixed polarities, automatic calculation of projected area widths, support for multi-view line detection, and the ability to remove optical distortion with a distortion-free input image. These functions will be described below.

Further referring to the example of FIGS. 14-16, the line detection systems and methods of the exemplary embodiments are generally (respectively) standard light to dark, dark to light and for contrast between detected edges. Supports alternative polarity settings. In addition, the system and method can also support a mixed polarity setting (FIG. 17) in which both light-to-dark and dark-to-light features appear on the same line. The line detection results for all four settings are shown in the following figure. In an exemplary embodiment, the system and method may include a mixed polarity setting that allows the detection of a single line containing edge points of opposite polarity. This is different from the conventional "alternative" polarity setting where all edge points in a line are of either polarity but are limited to one polarity. The mixed polarity setting may be advantageous when used for light-dark grid fringes of (eg) calibration plates, among other applications.

The user can select an improved shift immutability for line detection. In such cases, the edge point extractor uses substantially overlapping projection areas to improve the stability of the results. If these areas do not overlap, the pixel of interest may move out of the projected area when the image shifts, resulting in poor shift invariance in the line detection results. The overlapping projection areas ensure that the pixels of interest are continuously covered by the projection area. If overlapping projection areas are used, incremental calculations can be performed with possible low-level optimizations.

The user can provide a mask that excludes the acquired image and / or a specific portion of the captured surface from the analysis of line features. This is desirable if the surface contains known line features that are of no interest (eg, barcodes, texts analyzed by other mechanisms, and other structures that are not closely related to the task of trying to detect the line). be. Therefore, the edge point extractor can support image masking in which "irrelevant" areas in the image can be masked out and "relational" areas are masked in. When such masking occurs, the coverage score of the exemplary detected line is reweighted according to the number of edge points that fall within the mask.

Referring to the exemplary image area 1800 of FIG. 18, coverage scores in the presence of image masks and the effect of image masking on such coverage scores are shown. The edge point extractor supports image masking that can mask out "irrelevant" areas in an image. As shown, the detected lines 1810 are characterized by relational edge points (based on the "relationship" mask area 1820). Such a relational edge point consists of a relational edgepoint inlier 1830 for line 1810 and an irrelevant edgepoint outlier 1840 for line 1810. The irrelevant edge points 1850 on line 1810 are between the mask's relevant areas 1820 as shown in this example and are not included in the coverage score calculation even if they are on the line as inliers. Potential points 1860 for edge points along line 1810 have also been determined as shown. These potential locations are positioned between known points at predictable intervals based on the intervals between the detected points. Illustratively, the coverage score of the detected lines is reweighted according to the number of edge points that fall into the mask. As a result, the coverage score will be revised as follows.
Coverage score = number of relational edgepoint inliers for the line / (number of relational edgepoint inliers for the line + relational edgepoint outliers for the line + number of potential points of relational edgepoint).

After performing the line detection process according to the systems and methods described herein, the detected lines can be classified in various ways based on the classification criteria specified by the user (eg, by the GUI). The user can choose from internal classification means, such as in-line coverage score, intensity or contrast. The user can also choose from internal classification means, such as signed distances or relative angles. When using the external classification means, the user can specify a reference line segment for calculating the external classification means of the detected line.

As generally mentioned above, this system and method can include multi-viewing angle (MFOV) overloads and can incorporate vectors of images from different fields of view into the process. All images should be in a common client coordinate space based on calibration. As mentioned above, this functionality can be extremely useful in application scenarios where multiple cameras are used to capture a subregion of a single component. Since the edge points retain the gradient information, the line features projected between the gaps in the field of view can still be resolved (when the gradient in both FOVs is comparable to the orientation and alignment of the lines given in each FOV). .. sdrt

In particular, systems and methods do not require distortion removal (ie, do not require the image to be undistorted) (enable distortion-free images, in order to remove non-linear distortion and ensure that the distortion is not significant. do). If the image is not distorted, the system and method can still detect candidate edge points and map point positions and gradient vectors through non-linear transformations.

III. Line labeling training interface and run-time process

Referring again to FIG. 1, the vision system process (processor) 130 further includes a label interface and related process (processor) 136 used during training and runtime as described below. In addition, the vision system process (processor) includes or borders the neural net process (processor) 138, which is the process that borders the label process 136 as described below. (Processor) Receives image data and related classifiers from 137. The label interface process (processor) 136 operates during training to provide a user-specific (usually text / alphanumerical) description (referred to herein as a "label" or "tag") in an image of an object. Associate with the line of interest. These processes (processors) enhance the capabilities of the line detection tool described above by providing the additional ability to automatically label the lines detected in the tool results.

Referring to FIG. 19, a procedure 1900 for training a line detection process as described above is shown. Training procedure 1900 includes a user-provided label (usually in text and / or alphanumerical format) for the line of interest. At step 1910, the user inspects the training image of the object. The training image is usually an actual image of the model object, but may be partially or wholly generated by CAD or other compositing methods. The user identifies a line in the image of interest, such as the edge of the cover glass of a tablet or smartphone. Labels such as "inner housing inner edge", "outer housing inner edge" can be created to describe the line of interest. The user either accesses (from a list of possible labels) or creates a set of terms that define the line, which are stored in a database for use in image training (step 1920).

The user acquires or accesses one or more training images of the object during inspection by the vision system and performs a training process (processor) on them (step 1930). The training process involves the operation of the line detection tool described above. The tool uses custom parameters to automatically detect multiple lines within each training image.

Also with reference to FIG. 20, the user interface display screen 2000 is shown. The display screen 2000 includes a window 2010 showing a training image of the object being inspected by the vision system. The image contains a series of detected lines generally indicated by the highlighted marks 2020, 2022, 2024, 2026, 2028 and 2029. The display highlights a particular line, eg, the "inner housing inner edge" that was clicked or flagged by the user to label it as described below (often in a different color). Marks 2022-2026 relate to lines that are not of particular interest / relevance to the vision system task and are not clicked. Generally, a "line" as defined herein is essentially a linear step edge in an image, to which the line finder returns a mathematical line fit as a result. For element 2030, which also has relevant edges for the task and is labeled by the user during training, two related step edges (see 2028 and 2029) are shown on both sides of element 2030. ..

The user accesses menu 2040, which includes the defined labels 2042, 2044, 2048 and 2048. The user can use the cursor 2050 to click on the desired label or other interface component and then click on the detected line of interest (line 2030) to set the label on a particular line. (Step 1940). Note that not all detected lines need to be labeled, only the relevant lines desired by the user. If one or more related lines are missing from the image, the label associated with that line remains unassigned.

After labeling the line results for the set of training images, the tool is trained and the data is stored in the appropriate database for the object / vision system task (step 1950). When the subsequently trained tool is run on the image, the tool not only detects multiple lines, but also automatically assigns a unique label to each detected line (or the detected line is the user). Do not label if it is not relevant to your application). This eliminates the need for the user to post-process the line results (eg in script code) to determine the identification information for each detected line.

According to (optional) step 1960, the identified (labeled) lines are provided to the neural network tool at runtime for processing and scoring line features of the image. The parameters are provided to a neural network outside the training interface provided to the user and can be pre-programmed, for example, as optimized parameters for retrieving line features in the image. Therefore, the user is only responsible for assigning text labels to the linefinder results during training. The neural network is used for the probability score candidate line features returned by the line detection tool described above. More specifically, once the lines are detected in the training window 2010 during training and the user selects the name of the line they want to label, a neural network classifier is also created for each of these labeled lines. The classifier can adopt the same (or similar) name as the label defined and applied by the user in Training Interface 2000. For example, if the user selects a line and selects the label "Inner Housing Inner Edge", the process creates a classifier with the same name and adds the current image to the classifier along with the line feature vector.

Various off-the-shelf neural network tools can be employed to extract line feature candidates from the input image, with appropriate programming according to customized or conventional techniques. It will be clear that the line detection process described above is an example of various line detection tools and techniques for supplying lines detected from images.

With reference to the run-time procedure 2100 of FIG. 21, a run-time image of the object is obtained and / or provided from the preceding acquisition process (step 2110). This image is passed to the vision system and the associated processor 130, which operates the line detection tool 134 described above based on the trained model including the associated label (step 2120). The line detection tool returns all detected lines with or without labels. Next, in step 2130, the neural network created during training is executed and the probability score map is acquired. A probability score map is a map of whether a pixel corresponds to a trained feature vector in a neural network. This probability score map is the same size as the image. Each line detected next is sampled periodically and the probability score is integrated for each tool from the score map. Then, in step 2140, labels are assigned to the lines based on which line has the maximum probability score for each label. In step 2150, the result of the line detection step is stored with the relevant label and probability score and displayed to the user and / or used for other downstream utilization tasks such as part inspection (pass / fail), robot control, etc. ..

FIG. 22 shows a display of run-time results on an exemplary run-time object (based on the training object of FIG. 20), with relevant lines (step edges) highlighted. Additional relevant lines (eg, 2228) are displayed relative to other lines of interest trained in the training step and surround the line feature 2310. Some irrelevant lines are also highlighted (2220, 2222, 2224). The user clicked on one of the irrelevant lines (highlight 2220), which displays "untagged" in each information window 2230 with a corresponding probability score of 0. Conversely, in FIG. 23, the display 2300, which shows the same results as the display 2200 of FIG. 22, provides an information box 2320 for the relevant line 2210 labeled "inner housing inner edge". This represents a user-labeled line feature from the time of training, with a probability score of 0.635 for such lines, which means that the detected lines are rather correct labeled lines.

Illustratively, the neural net classifier described above receives an image (pixel data) as input along with features that define a line segment. The output of the neural net classifier is an image set, where each pixel in a single image is certain that the corresponding input pixel matches the trained line segment. The number of output images is the same as the number of line segments trained by the classifier. The desired output image trained to reproduce the network can be a binary or grayscale representation of the spatial probability distribution, with high probability narrow ridges corresponding to high gradient edges of lines or other trained patterns. is doing. At runtime, the classifier is. Takes the input image and produces an output image set that highlights the area where the neural net concludes that it may be associated with the trained line segment current label / tag.

Alternatively, the classifier may be statistically trained. The input to the statistically trained classifier is a measured characteristic (eg, distance between adjacent lines, relative angle, etc.) or line that describes the relationship between the current line segment and the adjacent line segment. Measured characteristics of the current line segment (eg, polarity, position, angle, etc.) with calculated characteristics of the image in the vicinity of the segment (1D intensity image projection in contact with the line segment, intensity histogram statistics, etc.). ) May be supplied as a feature vector. Accordingly, in the present specification, the term "classifier" can refer to a neural net classifier or a statistically trained classifier that produces a label. The term can also refer to K-nearest neighbor (K-NN) classifiers and / or processes (processors). If the statistical classifier and / or the K-NN classifier works, the output of the probability score or probability map may be omitted from steps 1900 and 2100 and as part of the interface label / tag display. It may not be provided. However, even such classifiers can publicly improve the labeling process.

IV. Conclusion

It will be clear that the line finder provided according to the system and method and various alternative embodiments / improvements is an effective and robust tool for determining a large number of line features under a variety of conditions. In general, systems and methods when used to detect line features have no particular limitation on the maximum number of lines that should be detected in an image. The detected lines can be labeled and classified so that the estimated correctness can be determined, thereby increasing the versatility and robustness of the line detection process.

The exemplary embodiments of the present invention have been described in detail above. Various modifications and additions can be made without departing from the spirit and scope of the present invention. Each feature of the various embodiments described above may be combined with the features of another described embodiment as long as it is suitable to provide a combination of multiple features in the relevant new embodiment. Further, although a number of distinct embodiments of the devices and methods of the invention have been described above, those described herein merely illustrate the application of the principles of the invention. For example, the terms "process" and / or "processor" as used herein are broadly based on electronic hardware and / or software and various functions and components (alternatively functional "modules" or "elements". Should be construed as including). Further, the illustrated process or processor may be combined with other processes and / or processors or divided into various subprocesses or subprocessors. Such subprocesses and / or subprocessors can be combined in various ways according to the embodiments described herein. Similarly, it is expressly stated that any function, process and / or processor herein can be performed using electronic hardware, software, or a combination of hardware and software consisting of non-temporary computer-readable media of program instructions. It is supposed. In addition, terms used herein to describe various directions and / or orientations, such as "vertical," "horizontal," "top," "bottom," "bottom," "top," and "side." , "Front", "rear", "left", "right" and the like are only used as relative expressions and are based on a fixed coordinate system such as the direction of gravity. It does not represent the absolute orientation. Therefore, this description should be taken as an example only and does not imply limiting the scope of the invention separately. In addition, when the word "substantially" or "approximately" is used with respect to a given measurement, value or feature, it is an amount within the normal operating range to achieve the desired result. However, it includes some inherent inaccuracies and errors within the margin of error allowed by the system (eg 1-5 percent). Therefore, this description is by way of example and does not imply limiting the scope of the invention separately.

Claims

Claims (22)

A system for detecting line features in images acquired based on one or more cameras.
With a vision system processor;
With an interface related to the vision system processor that allows the creation of individual labels for the relevant lines identified by the line detection process in the training image of the object;
With a run-time line detection process that identifies the lines in the captured image;
With a neural net process that uses a classifier based on the label to determine the probability map of the line features for the label;
With a run-time result generation process that provides labels and probability scores for at least one related line;
The above system. The vision system of claim 1, wherein the run-time result generation process provides a probability score for irrelevant lines. The vision system of claim 1, wherein the result generation process includes an interface that highlights a line and provides a probability score associated with the highlighted line. The vision system according to claim 1, wherein the probability score map is similar in size to the acquired image. The vision system of claim 1, wherein the classifier process uses at least one of a neural net classifier and a statistically trained classifier. The line detection process has a processor that receives image data of the scene including line features, and includes an edge point extractor.
The edge point extraction device is
(A) Calculate the gradient vector field from the image data and
(B) The gradient vector field is projected onto a plurality of gradient projection subregions.
(C) A plurality of edge points are detected in each of the gradient projection subregions based on the projected gradient data.
The line finder will generate multiple lines that match the edge points extracted from the image.
The vision system according to claim 1. The line finder includes manipulating a process based on RANSAC to fit inlier edge points to new lines and iteratively define lines from outlier edge points relative to predefined lines. The system according to claim 6. The system of claim 6, wherein the projection of the gradient field is directed along a set direction in response to the expected orientation of one or more features or line features. The system of claim 6, wherein the gradient field projection defines a grain size based on a Gaussian kernel. The edge point extraction device is arranged so as to detect a plurality of gradient intensity maximum values in each of the gradient projection subregions, and the gradient intensity maximum values are each specified as some of the plurality of edge points, and the position vector is used. And the system of claim 6, described by a gradient vector. The line finder is a metric based on the distance of at least one edge point from at least one candidate line and the angle difference between the gradient direction of at least one edge point and the normal direction of at least one candidate line. Arranged to determine a match between at least one edge point of a plurality of extracted edge points and at least one candidate line of a plurality of detected lines by calculating. Item 6. The system according to Item 6. A system for detecting line features in images acquired based on one or more cameras.
With a vision system processor;
With an interface related to the vision system processor that allows the creation of individual labels for the relevant lines identified by the line detection process in the training image of the object;
With a run-time line detection process that identifies the lines in the captured image;
With a statistical classifier that produces labels for interfaces based on the lines identified by the line detection process;
The above system. A system for detecting line features in images acquired based on one or more cameras.
With a vision system processor;
With an interface related to the vision system processor that allows the creation of individual labels for the relevant lines identified by the line detection process in the training image of the object;
With a run-time line detection process that identifies the lines in the captured image;
With a K-NN classifier that produces labels for interfaces based on the lines identified by the line detection process;
The above system. In a method for detecting line features in an image acquired based on one or more cameras.
With steps to provide an interface related to the vision system processor that allows the creation of individual labels for the relevant lines identified by the line detection process in the training image of the object;
With the steps of identifying the detected lines in the captured image by the run-time line detection process;
With the step of generating a label for at least one related detected line with a classifier;
How to have. The classifier comprises at least one neural net classifier, and the method further comprises the at least one neural net classifier based on the label to determine a probability map of the line features detected for the label. 14. The method of claim 14, comprising a step to use and a step to generate a probability score for irrelevant lines. 15. The method of claim 15, wherein the generated step highlights the detected lines in an interface and provides a probability score associated with the highlighted lines. 14. The method of claim 14, wherein the classifier is at least one of a neural net classifier, a statistically trained classifier, and a K-NN classifier. The line detection process has a processor that receives image data of the scene including line features, and includes an edge point extractor.
The edge point extraction device is
(A) Calculate the gradient vector field from the image data.
(B) The gradient vector field is projected onto a plurality of gradient projection subregions.
(C) Detect multiple edge points in each gradient projection subregion based on the projected gradient data and calculate multiple lines matching the edge points extracted from the image.
The method according to claim 14. The calculation step involves manipulating a RANSAC-based process to fit an inlier edge point to a new line, iteratively defining a line from an outlier edge point to a predefined line. 18. The method of claim 18. 18. The method of claim 18, wherein the projection of the gradient field is oriented along a direction set according to the expected orientation of one or more line features. The edge point extractor detects a plurality of gradient intensity maximums in each of the gradient projection subregions, each of which is identified as some of the plurality of edge points and is described by a position vector and a gradient vector. The method according to claim 18. The line detection process is based on the distance of at least one edge point from at least one candidate line and the angular difference between the gradient direction of at least one edge point and the normal direction of at least one candidate line. 18. Claim 18, wherein the calculation of the metric determines a match between at least one edge point of the plurality of extracted edge points and at least one candidate line of the plurality of detected lines. the method of.

Images