KR20240058827A - System and method for finding and classifying lines in an image with a vision system - Google Patents

Abstract

The present invention provides a system and method for finding line features in an image that allows multiple lines to be efficiently and accurately identified and characterized. When lines are identified, the user can train the system to associate predetermined (e.g. text) labels with these lines. These labels can be used to define a neural network classifier. The neural network operates at runtime, identifying and scoring lines in the runtime image found using a line-finding process. The found line may be displayed to the user along with labels based on the neural network results and an associated probability score map. Unlabeled lines are generally considered to have a low score and are not flagged by the interface or identified as irrelevant.

Description

A system and method for finding and classifying lines in an image using a vision system {SYSTEM AND METHOD FOR FINDING AND CLASSIFYING LINES IN AN IMAGE WITH A VISION SYSTEM}

This application is based on pending U.S. Patent Application No. 15/338,445, filed on October 31, 2016 (title: SYSTEM AND METHOD FOR FINDING LINES IN AN IMAGE WITH A VISION SYSTEM), which is a continuation-in-part of U.S. Provisional Application No. 62/249,918, filed on November 2, 2015 (title of the invention: System for finding lines in an image using a vision system) and SYSTEM AND METHOD FOR FINDING LINES IN AN IMAGE WITH A VISION SYSTEM), and the technical ideas of each of the above applications are incorporated herein by reference.

The present invention relates to machine vision systems, and more specifically to vision system tools for finding line features in acquired images.

Machine vision systems (also simply referred to herein as “vision systems”) are used for a variety of tasks in manufacturing, logistics and industry. These tasks may include inspecting surfaces and parts, positioning objects during assembly, reading patterns and ID codes, and other tasks where visual data is acquired and interpreted for use in further processes. Vision systems typically use one or more cameras to acquire images of a scene containing an object or object of interest. Objects/objects can be stationary or have relative motion. Movement can also be controlled by information derived from a vision system, such as in the case of manipulation of parts by a robot.

A common task for a vision system is finding and characterizing line features in an image. A variety of tools are used to identify and analyze these line features. Typically, these tools rely on sharp contrast differences in parts of the image. For example, using a caliper tool to determine whether individual points in an image with contrast differences can be assembled into a line-like feature, these contrast differences is analyzed. If so, a line is identified in the image. In particular, the tool for finding edge points and the tool for matching lines to points operate independently of each other. This increases processing overhead and reduces reliability. If the image contains multiple lines, these tools may be limited in their ability to accurately identify them. Additionally, when an image contains multiple closely spaced lines with similar orientation and polarity, conventional line-finding tools designed to find a single line in an image can be problematic to use.

Another problem is that the lines of an object can often be occluded or unclear in the acquired image. The user cannot be certain about the identity of the line found, and the mechanisms for making this identification may involve writing elaborate rules and scripts, which saves time for vision system setup and training tasks. and costs are added.

The present invention overcomes the shortcomings of the prior art by providing a system and method for finding line features in an image that allows multiple lines to be efficiently and accurately identified and characterized. When lines are identified, the user can train the system to associate predetermined (e.g., text) labels for these lines. These labels (also referred to herein as “tags”) can be used to define neural network classifiers. The neural network operates at runtime, identifying and scoring lines in the runtime image found using a line-finding process. The found lines may be displayed to the user along with labels and an associated probability score map based on the neural network results. Unlabeled lines are generally considered to have a low score and are not flagged by the interface or are identified as irrelevant.

In an example embodiment, a system and method are provided for finding line features in an image acquired based on one or more cameras. The system and method include a vision system processor; An interface associated with the vision system processor, the interface configured to retrieve relevant lines located by a line-finding process in a training image of an object. Allows creation of individual labels. A runtime line-finding process locates lines in the acquired image, and a neural net process generates, based on the labels, a probability map for the line feature with respect to the labels. One or more classifiers are used to determine the probability map. A runtime result-generation process provides labels and probability scores for at least one of the relevant lines. Illustratively, the runtime result-generation process provides probability scores for non-relevant lines, or the runtime result-generation process highlights lines and the highlighted line. Contains an interface that provides probability scores associated with . The probability map may be similar in size to the acquired image. The line-finding process may include a processor receiving image data of a scene including line features, the processor may include an edge point extractor, the edge point extractor comprising: (a ) Compute a gradient vector field from the image data, (b) project the gradient vector field over a plurality of gradient projection sub-regions, and (c) project the gradient vector field. A plurality of edge points are found in each of the gradient projection sub-regions based on the gradient data. Additionally, the processor includes a line-finder, which generates a plurality of lines matching the edge points extracted from the image. Illustratively, the line-finder includes iteratively defining lines from outlier edge points for previously defined lines and inlier edge points for new lines. A RANSAC-based process is run to fit the points. The gradient field projection may be oriented along a set direction in response to the expected orientation of one or more of the line features and/or the gradient field projection may be oriented based on a Gaussian kernel. (granularity) can be defined. The edge point extractor may be configured to find a plurality of gradient magnitude maxima in each of the gradient projection sub-regions. The gradient magnitude maxima are each identified as edge points of some of the plurality of edge points, described by a position vector and a gradient vector. In addition, the line-finder includes the distance of at least one edge point from at least one candidate line, the gradient direction of the at least one edge point, and the normal direction of the at least one candidate line. Consistency between the at least one edge point of the extracted plurality of edge points and the at least one candidate line of the found plurality of lines by calculating a metric based on the angle difference between them ( It can be configured to determine consistency.

In an example embodiment, a system is provided for finding line features in an image acquired based on one or more cameras. The system and method include a vision system processor; An interface associated with the vision system processor, the interface allowing generation of individual labels for relevant lines located by a line-finding process in a training image of the object. A runtime line-finding process identifies the positions of lines in the acquired image, and a statistical classifier or K-NN classifier is based on the lines located by the line-finding process. thus generating the labels for the interface.

The following description of the present invention refers to the accompanying drawings as follows.
1 illustrates an example vision system arrangement and vision system processor including an edge-finding tool/module and associated label interface processor/module for acquiring an image of an object including a plurality of edge features, according to an example embodiment. It is a drawing.
2 is a diagram illustrating an overview of a system and method for edge-point extraction and line-finding from an acquired image, according to an example embodiment.
FIG. 3 is a flowchart of an edge point extraction procedure according to the system and method of FIG. 2.
Figure 4 is a diagram of a field projection onto a region of the image containing edge features that is part of the edge point extraction procedure of Figure 3;
Figure 5 is a diagram illustrating applying a Gaussian kernel to an image to smooth it, for use in the edge point extraction procedure of Figure 3;
Figure 6 is a diagram of a field projection including applying a Gaussian kernel to smooth the projection, for use in the edge point extraction procedure of Figure 3.
Figure 7 is a diagram showing a graphical overview of the edge point extraction procedure of Figure 3, including application of a Gaussian kernel and calculation of absolute and normalized contrast thresholds for edge points.
Figure 8 is a graph showing the area of contrast defined for edge points with sufficient absolute and normalized contrast thresholds.
FIG. 9 is a flow diagram for a line-finding procedure based on edge points found in FIG. 3 using an example RANSAC process in accordance with an example embodiment.
10 and 11 are diagrams showing misplacement and correct placement of edge points associated with each of closely spaced parallel line features.
12 and 13 are diagrams illustrating correct and incorrect placement of edge points associated with each of the intersecting line features, which may be resolved according to the line-finder of the example system and method.
14-17 illustrate “light-to-dark polarity”, “dark-to-light-polarity”, “either light-to-dark or dark”, which can be separated according to the line-finder of the example system and method. This is a diagram showing examples of lines showing “-to-light polarity” or “mixed polarity”, respectively.
Figure 18 is a diagram showing the change in coverage score for a found line, considering a user-defined mask.
FIG. 19 is a flow diagram illustrating a procedure for training a line-finder tool using an interface that includes labels/tags representing lines of interest in an object image.
FIG. 20 shows an example user interface display including an image of a training object with lines of interest and dialog boxes used to perform the training procedure of FIG. 19. This is a drawing.
21 shows FIG. 19 , which involves assigning labels to lines of interest and assigning (e.g., neural) net classifier-based probability scores to found lines. This is a flowchart of the runtime procedure for finding lines using a trained line-finder.
FIG. 22 illustrates an example user interface display including an image of an object with lines found based on the line-finder runtime procedure according to FIG. 21 and illustrating labels and probability scores for non-relevant lines. am.
FIG. 23 is a diagram illustrating an example user interface display including an image of an object with lines found based on the line-finder runtime procedure according to FIG. 21 and illustrating labels and probability scores for lines of interest.

I. System overview

An example vision system configuration 100 that may be used in accordance with example embodiments is shown in FIG. 1 . System 100 includes at least one vision system camera 110 and may include one or more additional optional cameras 112 (shown in dashed lines). Exemplary camera(s) 110, 112 may be configured to capture and transmit image frames to a vision system processor 130, which may be instantiated as an independent processor and/or computing device 140. It includes a sensor (or imager) (S) and associated electronics. Cameras 110 (and 112) include appropriate lens/optics 116 that are focused on a scene containing object 150 under inspection. Cameras 110 and 112 may include internal and/or external illuminators (not shown) that operate according to the image acquisition process. Computing device 140 may be any acceptable processor-based system capable of storing and manipulating image data, according to example embodiments. For example, computing device 140 may include a PC (as shown), a server, a laptop, a tablet, a smartphone, or other similar device. Computing device 140 may include suitable peripherals, such as a bus-based image capture card that interconnects to a camera. In other embodiments, the vision processor may be partially or fully embedded within the camera body itself and may be networked with other PCs, servers, and/or camera-based processors to share and process image data. The computing device 140 includes a vision system tool provided to the vision system processor 130 and a suitable display 142 capable of supporting an appropriate graphical user interface (GUI) capable of operating according to the processor 132. ) is optionally included. It should be noted that displays may be omitted in various embodiments and/or provided only for setup and service functions. Vision system tools may be part of any acceptable software and/or hardware package that can be used to inspect objects, such as those commercially available from Cognex Corporation of Natick, MA. there is. Additionally, the computing device may include associated user interface (UI) components, including, for example, a keyboard 144 and mouse 146 as well as a touch screen within display 142 .

Cameras 110 (and 112) image some or all of an object 150 located within the scene. Each camera defines an optical axis (OA) around which a field of view is established based on optics configuration 116, focal length, etc. Object 150 includes a plurality of edges 152, 154, and 156, each disposed in a different direction. For example, object edges may include edges of a cover glass mounted on the smartphone body. By way of example, the camera may image the entire object or a specific location (eg, corners where glass meets the body). A (common) coordinate space may be established with respect to one of the objects, cameras or other reference points (e.g. a moving stage on which the object 150 is supported). As shown, the coordinate space is represented by axis 158. These axes exemplarily define the x, y and z axes orthogonal in the xy plane and a rotation _z about the z axis.

According to an example embodiment, vision system process 130 interoperates with one or more applications/processes (executing on computing device 140) that collectively comprise a set of vision system tools/processes 132 ( interoperates). These tools may include a variety of conventional specialized applications used to resolve image data, for example various correction tools and affine transform tools, as well as predetermined (e.g. common ) can be used to transform the acquired image data into a coordinate system. A tool may also be included to convert image grayscale intensity data to a binary image based on a predetermined threshold. Likewise, tools may be provided to analyze gradients in intensity (contrast) between adjacent image pixels (and subpixels).

Vision system process (processor) 130 includes a line-finding process, tool, or module 134 that locates a plurality of lines in the acquired image, according to an example embodiment. Accordingly, reference is made to Figure 2, which graphically depicts an overview of a line-finding procedure 200 in accordance with an exemplary embodiment. Procedure 200 consists of two main parts. Input image 210 is provided to the processor. As shown, the image includes a pair of intersecting edges 212 and 214. These may represent corner areas of the object 150 described above. Edge point extractor 220 processes input image 210 to obtain a set of candidate edge points 230, including edge points 232 and 234 along edges 212 and 214, respectively. Edge points 232, 234 and their associated data (e.g., intensity gradient information described below) are provided to a recursive line-finder 240, which The line-finder 240 performs a series of iterative processes on the selected edge point. The goal of the iterative process is to attempt to match other found edge points to candidate line features. The line-finding process 240 results in found lines 252 and 254 as shown. These results may be provided to other downstream processes 260 that use the information, such as alignment processes, robotic manipulation, inspection, ID reading, part/surface inspection, etc.

II. Line-Find Process (Processor)

Referring to FIG. 3, FIG. 3 explains a procedure for extracting edge points according to an embodiment. One or more images are acquired of a scene containing an object or surface having edge features to be found (step 310). Images may be extracted by a single camera or multiple cameras. In either case, the image pixels may (optionally) be transformed at step 320 into a new and/or common coordinate space by appropriate calibration parameters. This step may also include smoothing the image as described below. In certain embodiments, when multiple cameras are imaging a discrete region of a scene (e.g., focusing on a corner region of a larger object), a common coordinate space is defined as an empty region between the camera fields of view. ) can be accounted for. As described below, lines extending between these fields of view (e.g., an object edge connecting two located corner regions) may be extrapolated by the systems and methods of example embodiments. The edge points needed to find the lines are extracted from the image(s) in the appropriate coordinate space by an edge point extractor using gradient field projection at step 330. Gradient values are first calculated for each pixel, creating two images for the x and y gradient components. The image is further processed by projecting the gradient field onto many caliper-like regions. Unlike conventional caliper tools that project intensity values, by projecting a gradient field according to an embodiment, the gradient orientation can be preserved, which can be applied to subsequent lines as explained below. - Facilitates the finding process.

Also, referring to the diagram of FIG. 4 at step 340, the portion of the image containing candidate edge features (caliper-like region 400) is projected to region 400 in an orthogonal projection direction (arrow PD). It receives a gradient field projection (represented by a plurality of projections 410, 420, 430) that is searched over the (approximately) expected orientation of the edge in the search direction (arrow SD), with the projections being repeated across (subjected). Each projection (e.g., projection 420) edge appears to be a local maximum in the gradient field 440 associated with the projection. In general, a set of edge points within a projection associated with an edge will exhibit an intensity gradient (vectors 552, 554) perpendicular to the direction of the edge's extension. As explained below, the user can define the projection direction based on the expected line orientation. Alternatively, this may be provided by default or by other mechanisms, such as analysis of features in the image, for example.

Two granularity parameters are included in the gradient projection step described above. Prior to gradient field calculation, the user can choose to smooth the image using an isotropic Gaussian kernel. The first granularity determines the size of this Gaussian smoothing kernel. As shown in diagram 500 of FIG. 5, application of a Gaussian kernel of appropriate size (e.g., large 512, medium 514, small 516) causes image 210 to be smoothed. . Therefore, the first granularity parameter prior to field calculation determines the size of the isotropic Gaussian smoothing kernel.

After gradient field calculation, a Gaussian-weighted projection is performed by the process rather than the uniform weights in conventional caliper tools. Accordingly, the second granularity parameter determines the size of the one-dimensional (1D) Gaussian kernel used during the field projection shown in FIG. 6 where region 600 receives Gaussian smoothing kernels 610, 620, 630. During the typical operation, the user verifies (using the GUI) all extracted edges that overlap on the image and finds them, avoiding an excessive number of edges due to background noise in the image. The granularity and contrast thresholds are adjusted until the number of edges extracted along the lines to be drawn appears satisfactory. In other words, this step ensures that the signal-to-noise ratio is optimized for the image characteristics. This adjustment may be performed automatically by the system using default values in various embodiments. It should be noted that using a Gaussian weighting function is one approach among a variety of approaches to weighting projections containing (for example) uniform weights.

In diagram 700 of Figure 7 the overall flow for gradient field extraction and projection is graphically depicted. The two granularity parameters, the isotropic Gaussian kernel 710 and the 1D Gaussian kernel 720, are each shown in half for the entire diagram 700. As shown, each acquired image 210 is subjected to smoothing and decimation 730. As described above, the resulting image 740 is subjected to a gradient field calculation 750, producing two gradient images 752 and 754. Additionally, these gradient images are expressed as g _x and g _{y ,} which represent two orthogonal axes in a common coordinate space. Since the processed intensity information is also used to calculate normalized contrast according to an embodiment as described below, in addition to the two gradient images the intensity image 756 is also typically (1D Note that it is subjected to a smoothing, decimation and projection process (760) using a Gaussian-weighted projection (770) based on the Gaussian kernel (720) of . Projection profiles for gradient image 772 (g _x ), gradient image 774 (g _y ), and intensity image 776 are the results.

Also, referring to step 350 of procedure 300 (in FIG. 3), qualified edge points are extracted by combining the 1D projection profiles of the x & y gradient images. This is achieved using a raw contrast calculation 780 and a normalized contrast calculation 790 based on the intensity image. More specifically, any local peaks with both raw projected gradient magnitudes and normalized projected gradient magnitudes exceeding the respective thresholds are defined by the example equation: are considered candidate edge points for follow-up according to .

( g _x ² + g _y ² ) ^1/2 > T _ABS

( g _x ² + g _y ² ) ^1/2 / I > T _NORM

_where g ​ _​ ​ _{​ ​} is the normalized contrast threshold for the intensity-normalized projected gradient magnitudes.

In particular, only one point is considered a candidate edge point when both its absolute and normalized contrast exceed the respective threshold. This is shown in the upper right quadrant 810 of an example graph 800 of normalized contrast threshold (T _NORM ) versus absolute contrast threshold (T _ABS ). The use of dual (absolute and normalized) thresholds is generally different from existing approaches that typically use absolute contrast thresholds. The benefits of dual contrast thresholding are evident when the image contains both dark and bright intensity regions, illustratively containing edges of interest. To detect edges in dark areas of an image, it is desirable to set a low contrast threshold. However, these low contrast settings can cause false edges to be detected in bright parts of the image. Conversely, it is desirable to set a high contrast threshold to avoid detecting false edges in bright areas of the image. However, for high contrast settings, the system may not be able to properly detect edges in dark areas of the image. By using a second normalized contrast threshold in addition to the traditional absolute contrast threshold, the system can properly detect edges in both dark and light areas and avoid detecting false edges in bright areas of the image. Therefore, by being able to detect relevant edges while avoiding spurious edges, using a dual contrast threshold provides for maximizing the speed and robustness of the subsequent line-finding step in the overall process. .

Also, with further reference to procedure step 350 (FIG. 3), once all edge points have been extracted, these edge points are represented and stored in a data structure that is convenient for subsequent line-finders to operate in. For example, it could be a tuple like this:

p = ( x,y,gx,gy,gm,go,I,gm/I,m,n )

where (x,y) is the position of the edge point, ( g _x , g _y ) are the values for each x - gradient and y _- gradient projections, and ( g _m , g _o ) _are ), I is the intensity of the edge point location, and g _m /I is the intensity-normalized contrast obtained by dividing the gradient size ( g _m ) by the intensity ( I ). contrast), m is the image index, and n is the projection area index. As with standard caliper tools, the positions of edge points can be interpolated to improve accuracy.

It should be noted that the edge-point extraction process generally operates to effect a field projection in a single direction that substantially matches the expected line angle. Therefore, the tool is most sensitive to edges of these angles, and its sensitivity drops off progressively for edges of other angles, where the rate of drop depends on the granularity setting, which indirectly determines the field projection length. As a result, the process is limited to finding lines with an angle “near” the expected line angle, according to the angle range specified by the user. Although the above process has been adapted to finding non-orthogonal lines, in some embodiments it is possible to find lines at arbitrary angles across 360 degrees by performing projections in multiple directions, including orthogonal directions (omni-directional lines). find) is considered to be generalizable.

Referring now to step 360 of procedure 300 (in FIG. 3), thresholded edge point candidates are provided to the line-finder in accordance with an example embodiment. As an example, line-finders operate recursively and use (for example) Random Sample Consensus (RANSAC)-based techniques. Reference is also made to the line-finding procedure 900 in Figure 9. At step 910, the user determines the expected angle in the image along with the expected angle, angle tolerance, distance tolerance, and (by way of example) a minimum coverage score (generally defined below) via a GUI. Specifies the maximum number of lines. These parameters are used by the line-finder to run the next process. Lines are found for each subregion of the image by recursively running the RANSAC line-finder, and edge point outliers from one step are input points ( input points). Accordingly, at step 920, procedure 900 selects a pair of edge points that are part of a group of edge points identified as extrema in an edge-finding process. Procedure 900 attempts to fit a model line to selected edge points based on matching gradient values that match the model line (within a selected range of tolerance). At step 924, one or more line candidate(s) from step 922 are returned. Each line-finding step returns a candidate line, its inliers and outliers. The returned line(s) are computed for inlier edge points that have a location and gradient that match the line candidate (step 926). At step 928, the candidate line with the greatest inlier count is identified. The line-finding step described above (steps 920-928) ends when the maximum number of RANSAC iterations allowed is reached (decision step 930). The maximum number of iterations within each line-finding step is automatically calculated using the assurance level specified by the user and the internally calculated worst-case ratio of outliers. Each line-finding step finds the line with the maximum number of captured edge points among all iterations, subject to user-specified fit tolerance, geometric constraints, and polarity. Return. Each edge point can only be assigned to the inlier list of a single line, and each line is allowed to contain at most one edge point from each projection area. The gradient orientation of an edge point along with its location is used to determine whether the edge point should be included in the inlier list of candidate lines. In particular, edge points must have a gradient orientation that matches the angle of the candidate line.

If decision step 930 determines that more iterations are allowed, outliers from the top inlier candidate are returned (step 940) to the RANSAC process (step 920) for use in finding line candidates.

During each RANSAC iteration, two edge points belonging to different projection areas are randomly selected, and a line will be fit to these two points. As a result, the candidate line receives further consideration only if the angle of the candidate line matches the gradient angles of both edges in the point pair and if the angle of the line matches the uncertainty range specified by the user. Typically, the gradient directions of edge points are nominally orthogonal, but are allowed to differ by user-configured angle tolerance. If the candidate line passes these initial tests the number of inlier edge points is evaluated, otherwise a new RANSAC iteration is initiated. An edge point is considered an inlier for a candidate line only if its gradient direction and position match the line based on the gradient angle and distance tolerance specified by the user.

When the RANSAC iteration reaches its maximum (decision step 930), the inliers of the best found line candidates are subjected to improved line fitting (for example) using least squares regression or another acceptable approximation technique. line fit), and repeat this step up to N times (e.g., 3 or more times) until the number of inliers increases or decreases further, and the set of inlier edge points will be reevaluated (step 960 )). This is the line marked as found in step 970.

Decision step 980 determines whether more lines are to be found ((e.g. based on searching another sub-region or other criteria) and, if so, the process continues with step 920. Loops back and operates on a new set of edge points (step 982). When points are exhausted or the maximum iteration count has been reached, procedure 900 returns at step 990 a set of lines (i.e., a plurality) of found lines in the image.

The multi-line-finder is adapted to perform final adjustments to existing results in cases where two line results intersect each other within the inspection area. Generally, as shown in Figures 10 and 11, for closely spaced parallel lines 1010 and 1020, erroneous line results (i.e. Figure 10) are often obtained due to the statistical nature of the RANSAC procedure. It can be. However, when such errors occur, swapping for inlier point groups (arrow 1120 in groups 1110 in Figure 11) often results in increased coverage scores and reduced-fit residuals. You can check the exact position of the line with . Point exchanges can be most effective when the image contains closely spaced parallel lines as shown. Conversely, when the image contains lines 1210 and 1220 that actually intersect each other as shown in FIGS. 12 and 13, after the point exchange (arrow 1230 in group 1240 in FIG. 12) the coverage score is is reduced, the original result obtained prior to the exchange by the process of successfully detecting intersecting lines is maintained.

It should be noted that the RANSAC procedure is one of several techniques that allow a line-finder to fit points into lines. In another embodiment, candidate points may be selected according to a set displacement therebetween, or the image may be processed using (for example) an exhaustive search technique. Accordingly, as used herein, references to RANSAC technology should be broadly interpreted to include several similar point-fitting technologies.

Additional functionality of these systems and methods may be provided. These features include support for mixed-polarity, automatic calculation of the projection area width, support for multi-view line-finding, and ensuring that the input image is pre-warpage free. Including eliminating optical distortion. These functions are explained further below.

With further reference to the example of Figures 14-16, the line-finding system and method of the exemplary embodiment includes standard LightToDark, DarkToLight, and contrast between the found edges. Either polarity settings for (each) are generally supported. Additionally, the system and method have a mixed-polarity setting (mixed-polarity setting) where both Light-To-Dark and Dark-to-Light characteristics are visible on the same line. Figure 17) may also be supported. Line-finding results for all four settings are shown in the following figures. In an exemplary embodiment, the system and method may include a mixed-polarity setup that can find a single line containing edge points of opposite polarity. This is different from the conventional setting of “either” polarity, where all edge points of a single line are of one polarity, but only one polarity. A mixed-polarity setup may be advantageous when used to analyze light and dark checkerboards of a calibration plate (for example), among other applications.

Users can choose improved shift invariance of line-finding. In these cases, the edge point extractor uses substantially overlapped projection areas to improve result stability. When the regions do not overlap, the pixels considered can potentially move from the projection region when the image is shifted, resulting in poor shift invariance in the line-finding results. Overlapping projection areas ensure that the pixels considered are continuously covered by the projection areas. When overlapping projection regions are used, incremental computation is performed to maintain speed with as low-level optimization as possible.

The user can provide masks that omit certain portions of the image and/or the imaged surface obtained from analysis of line features. This may be desirable if the surface contains known line features of no interest (e.g., barcodes that are analyzed by other mechanisms, text, and some other structure not closely related to the task for which the lines were sought). Therefore, the edge point extractor performs image masking, where “don’t care” areas in the image can be masked out and “care” areas are masked in. masking) can be supported. When such masking occurs, the coverage scores of the lines found are exemplarily reweighted according to the number of edge points that fall within the mask.

Referring to the example image region 1800 of FIG. 18, FIG. 18 shows coverage scores when image masks are present, and the effect of image masking on these coverage scores. The edge point extractor supports image masking, where “regions of no interest” in the image can be masked out. As shown, found lines 1810 are characterized by care edge points (based on “care” mask area 1820). These edge points of interest consist of an edge point of interest inlier 1830 into line 1810 and an edge point of interest outlier 1840 into line 1810. Edge points 1850 of no interest in line 1810 exist between regions of interest 1820 of the mask as shown in this example, even if edge points of no interest 1850 exist in the line as inliers. , is not included in the coverage score calculation. Potential locations 1860 for edge points along line 1810 are also determined as shown. These potential locations are located between known points at predictable intervals based on the spacing of the points found. Illustratively, the coverage scores of the found lines are reweighted according to the number of edge points that fall within the mask. Therefore, the coverage score is changed as follows.

Coverage score = number of care edge point inliers to line/(number of care edge point inliers to line + number of care edge point outliers to line) (care edge point outliers to line) + number of care potential locations of edge points)

After executing the line-finding process according to the systems and methods herein, the found lines may be sorted in several ways based on sort criteria specified by the user (e.g., via a GUI). . Users can select from intrinsic sort measures such as inlier coverage score, intensity, or contrast. Users can also select from extrinsic sort measures, such as signed distance or relative angle. When using the extrinsic classification metric, the user can specify a reference line segment from which the extrinsic metric of the found lines will be calculated.

As generally discussed above, these systems and methods may include a Multi-Field-of-View (MFOV) overload in which vectors of images from different fields of view may be passed to the process. Images must all be in a common client coordinate space based on calibration. As mentioned above, this feature can be very useful in application scenarios where multiple cameras are used to capture partial areas of a single part. Because edge points retain gradient information, line features projected between gaps in the field of view remain resolved (when the gradients in both FOVs match the line orientation and placement given in each FOV). You can.

In particular, assuming that the distortion is not significant, in order to remove non-linear distortion, the systems and methods do not require (the image) to remove warpage (i.e., they do not require the image to be unskewed). If the image is not skewed, the system and method can still detect candidate edge points and map point locations and gradient vectors through a nonlinear transform.

III. Line Labeling Training Interface and Runtime Process

Referring back to Figure 1, the vision system process (processor) 130 further includes a label interface and associated process (processor) 136 used for training and runtime as described below. The vision system process (processor) also includes or interfaces with a neural network, statistical classifier, and/or K-nearest neighbor (K-NN classifier) process (processor) 138, which includes: A classifier associated with the image data is received from a process (processor) 137 that interfaces with the label process 136 described in . The label process (processor) 136 allows the user to associate specific descriptions (typically alphabetical/including letters and numbers) with lines of interest in an image of an object (referred to herein as “labels” or “tags”). It operates at training time to do this. This process enhances the functionality of the line-finding tool described above by providing the additional ability to automatically label lines found in the tool results.

Referring to Figure 19, Figure 19 illustrates a procedure 1900 for training the line-finding process described above. The training procedure 1900 includes user-provided labels (typically in the form of text and/or letters and numbers) for lines of interest. At step 1910, the user reviews a training image of the object, which is typically an actual image of the model object, but may be partial or complete and may be generated by CAD or another synthetic approach. . The user identifies lines in the image of interest (eg, the edge of the cover glass of a tablet or smartphone). Labels such as “(Inner Housing Inner Edge)”, “(Outer Housing Inner Edge)”, etc. can be created to describe lines of interest. The user accesses (from a list of possible labels) or creates a set of terms defining lines, which are stored in a database for use in training the image (step 1920).

A user acquires or accesses one or more training images of an object during inspection by the vision system and operates a training process (processor) (step 1930). The training process includes operation of the line-finding tool described above. The tool automatically finds multiple lines in each training image using user-set parameters.

Referring to FIG. 20, a user interface display screen 2000 is shown. The display screen 2000 includes a window 2010 that represents a training image of an object being inspected by the vision system. The image contains a series of found lines, generally marked with highlighted markings (2020, 2022, 2024, 2026, 2028 and 2029). The display highlights (usually in a different color) specific lines 2032 of interest - clicked-upon or flagged by the user to apply a label, illustratively described below. “Inner Housing Inner Edge”. Marks (2022-2026) relate to lines that are not of particular interest/relevance to vision system operations and/or have not been clicked on. In general, a “line” as defined herein is essentially a linear step edge of the image for which the line-finder returns a mathematical line fit. For element 2030 (which may be a task-related edge and may be labeled by the user during training), there are two associated step edges (see 2028 and 2029) of either It is shown on the (either) side.

The user accesses menu 2040 containing defined labels 2042, 2044, 2046, and 2048. The user can click on the desired label using the cursor 2050 or another interface component, and then click on the line of interest found (line 2030) to set the label for that particular line (step 1940). Note that you do not need to label every line you find, but only the relevant lines that you want. If one or more of the associated lines are missing from the image, the label associated with that line remains unassigned.

After labeling the line results for the set of training images, the tool is trained and the data is stored in an appropriate database for object/vision system operation (step 1950). As a result, when a trained tool is run on an image, the tool not only finds multiple lines, but also automatically assigns a unique label to each line it finds (or a label if the line it finds is not relevant to the user's application). may not exist). This saves the user from having to post-process the line results (e.g., in script code) to determine the identity of each found line.

According to (optional) step 1960, the identified (labeled) lines may be provided to a neural network tool for processing and scoring of the line features of the image during runtime. Parameters are provided to the neural network outside of the training interface provided to the user and may be pre-programmed, for example, parameters optimized for searching line features in an image. Therefore, users must assign text labels to line-finder results at training time. A neural network (or other process described generally below) is used to probability-score the candidate line features returned by the line finding tool described above. More specifically, at training time, once the lines are found in the training window (2010), the user selects names for the lines he wants to label, and the neural network classifier uses these labels. It is also created below for each of the created lines. The classifier may use the same (or similar) name as the label defined and applied by the user in the training interface 2000. For example, if the user selects a line and assigns the label "Inner Housing Inner Edge", the process creates a classifier with the same name and adds the current image along with the line feature vector to the classifier. .

A variety of neural network tools, either custom or commercially available, can be used to extract line feature candidates from the input image, using appropriate programming according to those skilled in the art. Additionally, it will be clear that the line finding process described above is an example of various line-finding tools and techniques for delivering lines found in an image.

Referring now to runtime procedure 2100 of Figure 21, a runtime image of an object is acquired and/or provided from a previous acquisition process (step 2110). These images are passed to the vision system and associated processor 130, which operates the line finding tool 134 described above based on the trained model including the associated labels (step 2120). The line-finding tool returns both labeled and unlabeled lines it finds. Next, at step 2130, the neural network created at training time (or other process using the classifier described below) is executed and a probability score map is obtained. The probability score map is a map of whether a pixel corresponds to a feature vector on which a neural network was trained. This probability score map is the same size as the image. Each found line is sampled at regular intervals and a probability score is integrated for each tool from the score map. At step 2140, labels are assigned to lines based on the line with the highest probability score for each label. At step 2150, the results of the line-finding step along with the associated labels and probability scores are stored and displayed for the user or for other downstream utilization tasks such as part inspection (pass/fail), robot control, etc. It is used.

Figure 22 shows a display of runtime results for an example runtime object (based on the training object of Figure 20) with the relevant line (step edge) 2210 highlighted. Additional lines of interest (e.g., lower highlight 2228 around linear feature 2310) are shown and correspond to other lines of interest trained in the training phase. Multiple non-relevant lines are also highlighted (2220, 2222, 2224). The user clicked on one of the non-relevant lines (highlight 2220) indicating “No Tag” in each information window 2230, indicating that the corresponding probability score is 0. Conversely, in FIG. 23 , display 2300 showing the same result as display 2200 of FIG. 22 includes an information box 2320 for the associated line 2210 with the label “Inner Housing Inner Edge.” provides. This indicates that the user-labeled feature and probability score from training time are equal to 0.635, meaning that the line found is more correctly labeled than it would otherwise be.

As an example, the neural network classifier described above receives as input an image (pixel data) along with features defining line segments. The output of a neural network classifier is a set of images in which there is a single confidence whether each pixel in the image matches a line segment for which the corresponding input pixel was trained. The number of output images is equal to the number of line segments learned by the classifier. Desirable output images that the network is trained to reproduce may be a binary or grayscale representation of a spatial probability distribution, high probability narrow ridges corresponding to high gradient edges of lines, or other trained patterns. At runtime, the classifier receives input images and produces a set of output images that highlight regions where the neural network concludes that a trained line segment can be associated with the current label/tag.

Alternatively, the classifier can be trained statistically. The inputs to these statistically trained classifiers are measured properties that describe the relationship between the current line segment and its adjacent line segments (e.g., distance to the nearest line, relative angle, etc.) or nearby line segments. Contains measured properties (e.g. polarity, position, angle, etc.) of the current line segment along with computed properties of the image (e.g., 1D intensity image projections tangential to the line segment, intensity histogram statistics, etc.) Can be provided as a feature vector. Accordingly, the term “classifier” as used herein may refer to a neural network classifier or a statistically trained classifier that generates labels. The term may also refer to a K-Nearest Neighbor (K-NN) classifier and/or process. If statistical classifiers and/or K-NN classifiers are used, the output of probability scores or probability maps may be omitted from procedures 1900 and 2100 and may not be provided as part of the label/tag display in the interface. You can. However, these classifiers continue to benefit improvements in the labeling process.

IV. conclusion

It is clear that the line-finder provided in accordance with the above systems and methods and various alternative embodiments/improvements is an effective and powerful tool for determining multiple line features under various conditions. In general, when used to find line features, the systems and methods have no specific limit on the maximum number of lines found in an image. The lines found can be labeled and classified to determine their possible accuracy, thereby increasing the versatility and power of the line-finding process.

The foregoing is a detailed description of exemplary embodiments of the present invention. Various modifications and additions may be made without departing from the spirit and scope of the present invention. Features of each of the various embodiments described above may be combined with features of other described embodiments as appropriate, so that various feature combinations may be provided in the associated new embodiment. Additionally, while the foregoing describes many distinct embodiments of the apparatus and method of the invention, those described herein are merely illustrative of applications to the principles of the invention. For example, as used herein, the terms “process” and/or “processor” should be broadly interpreted to include various electronic hardware and/or software-based functions and components ( and may alternatively be referred to as functional “modules” or “elements”). Additionally, the depicted process or processor may be combined with other processes and/or processors or may be divided into several sub-processes or processors. These sub-processes and/or sub-processors may be combined in various ways depending on the embodiments herein. Likewise, any function, process and/or processor herein is explicitly stated to be implemented using electronic hardware, software consisting of a non-transitory computer-readable medium of program instructions, or a combination of hardware and software. is considered. Additionally, as used herein, “vertical,” “horizontal,” “above,” “below,” “bottom,” “top,” “side,” “front,” “posterior,” “left,” Various direction and placement terms such as "right" are used only as relative conventions and are not used as absolute directions/positions in a fixed coordinate space, such as the direction of gravity. Additionally, when the terms "substantially" or "approximately" are used for a given measurement, value, or characteristic, the terms refer to the amount that is within the normal range of operation to achieve the desired result, but this does not allow for internal inaccuracies and tolerances in the system. Includes some variation due to errors within the specified tolerance (e.g., 1-5 percent). Accordingly, this description is to be construed as illustrative only and is not meant to limit the scope of the invention.

Claims (22)

A system for finding line features in an image acquired based on one or more cameras, comprising:
vision system processor;
an interface associated with the vision system processor, the interface allowing creation of individual labels for lines located by a line-finding process in a training image of an object;
a runtime line-finding process that determines the location of lines in the acquired image;
a neural network process that uses a classifier associated with the individual label to determine a probability map indicating the probability of a portion of the acquired image corresponding to the individual label; and
The individual label of the line located in the acquired image is determined based on the probability score for the located line determined according to the probability map, and the determined individual label and the probability score are provided together with the located line in the acquired image. runtime result-generating process
A vision system including.
According to paragraph 1,
The runtime result-generation process provides probability scores for non-interest lines,
Vision system.
According to paragraph 1,
The result-generating process is an interface that highlights lines and provides probability scores associated with the highlighted lines.
Including,
Vision system.
According to paragraph 1,
The probability map is similar in size to the acquired image,
Vision system. According to paragraph 1,
The neural network process uses at least one of a neural network classifier and a statistically trained classifier,
Vision system.
According to paragraph 1,
The line-finding process is performed by a processor receiving image data of a scene containing line features.
Including,
The processor includes an edge point extractor; and a line-finder,
The edge point extractor,
(a) calculate a gradient vector field from the image data,
(b) projecting the gradient vector field over a plurality of gradient projection sub-regions,
(c) finding a plurality of edge points in each of the gradient projection sub-regions based on the projected gradient data,
The line-finder is,
Generating a plurality of lines matching the edge points extracted from the image,
Vision system.
According to clause 6,
The line-finder operates a RANSAC-based process to align inlier edge points to new lines, including iteratively defining lines from outlier edge points to previously defined lines.
Vision system.
According to clause 6,
The gradient field projection is oriented along a set direction in response to the expected orientation of one or more of the line features.
Vision system.
According to clause 6,
Gradient field projection defines the granularity based on a Gaussian kernel.
Vision system.
According to clause 6,
the edge point extractor is configured to find a plurality of gradient magnitude maxima in each of the gradient projection sub-regions,
wherein the gradient magnitude maxima are each identified by some edge points of the plurality of edge points, described by a position vector and a gradient vector,
Vision system.
According to clause 6,
The line-finder calculates a metric based on the distance of at least one edge point from at least one candidate line and the angular difference between the gradient direction of the at least one edge point and the normal direction of the at least one candidate line. configured to determine consistency between the at least one edge point among the extracted plurality of edge points and the at least one candidate line among the plurality of lines,
Vision system.
A system for finding line features in an image acquired based on one or more cameras, comprising:
vision system processor;
an interface associated with the vision system processor, the interface allowing creation of individual labels for lines located by a line-finding process in a training image of an object;
a runtime line-finding process that determines the location of lines in the acquired image; and
A statistical classifier that determines a probability map indicating the probability of a portion of the acquired image corresponding to the individual label and generates individual labels for the interface based on lines located by the line-finding process.
Including,
The individual label of a line located in the acquired image is determined based on a probability score for the located line determined according to the probability map,
The system of claim 1, wherein the determined individual label and the probability score are provided together with the located line in the acquired image.
A system for finding line features in an image acquired based on one or more cameras, comprising:
vision system processor;
an interface associated with the vision system processor, the interface allowing creation of individual labels for lines located by a line-finding process in a training image of an object;
a runtime line-finding process that determines the location of lines in the acquired image; and
K- determines a probability map indicating the probability of a portion of the acquired image corresponding to the individual label, and generates individual labels for the interface based on lines located by the line-finding process. NN classifier
Including,
The individual label of a line located in the acquired image is determined based on a probability score for the located line determined according to the probability map,
The system of claim 1, wherein the determined individual label and the probability score are provided together with the located line in the acquired image.
A method for finding line features in an image acquired based on one or more cameras, comprising:
providing an interface associated with a vision system processor that allows creation of individual labels for lines located by a line-finding process in a training image of an object;
Using a runtime line-finding process, confirming the location of lines found in the acquired image;
determining, using a classifier associated with the individual label, a probability map indicating the probability of a portion of the acquired image corresponding to the individual label, and generating individual labels for at least one of the found lines; and
The individual label of the line located in the acquired image is determined based on the probability score for the located line determined according to the probability map, and the determined individual label and the probability score are provided together with the located line in the acquired image. steps to do
How to include .
According to clause 14,
The classifier includes at least one neural network classifier,
The method is:
Based on the individual labels, using the at least one neural network classifier to determine a probability map for line features found associated with the individual labels; and
Generating to provide probability scores for found non-interest lines
How to further include .
According to clause 15,
The generating step is,
highlighting lines found in the interface; and
providing the probability scores associated with the highlighted lines.
How to include .
According to clause 14,
The classifier is at least one of a neural network classifier, a statistically trained classifier, and a K-NN classifier,
method.
According to clause 14,
The line-finding process receives image data of a scene containing line features,
The line-finding process includes an edge point extractor,
The edge point extractor,
(a) calculate a gradient vector field from the image data,
(b) projecting the gradient vector field over a plurality of gradient projection sub-regions,
(c) finding a plurality of edge points in each of the gradient projection sub-regions based on the projected gradient data,
The method is,
calculating a plurality of lines matching the edge points extracted from the image
How to further include .
According to clause 18,
The calculating step is,
operating a RANSAC-based process to fit inlier edge points to new lines, including iteratively defining lines from outlier edge points to previously defined lines,
method.
According to clause 18,
The gradient field projection is oriented along a set direction in response to the expected orientation of one or more of the line features.
method.
According to clause 18,
The edge point extractor finds a plurality of gradient magnitude maxima in each of the gradient projection sub-regions,
wherein the gradient magnitude maxima are each identified as some of the plurality of edge points formed by a position vector and a gradient vector,
method.
According to clause 18,
The line-finding process further includes a line-finder,
The line-finder calculates a metric based on the distance of at least one edge point from at least one candidate line and the angular difference between the gradient direction of the at least one edge point and the normal direction of the at least one candidate line. determining consistency between the at least one edge point among the extracted plurality of edge points and the at least one candidate line among the found plurality of lines,
method.