KR20220152785A - PCB visual inspection apparatus and method using accelerated vision processing and machine learning in heterogeneous platform - Google Patents

Abstract

Proposed are a printed circuit board (PCB) exterior inspection apparatus and method, which are able to easily reuse a software or transplant the same to another platform in a domain requiring a low power and a high performance, such as an embedded system, and provide an accelerated vision processing function optimized for the system. The present invention realizes a PCB exterior inspection system on various heterogeneous embedded platform systems, finds a candidate of defect through a low power accelerated vision processing, and determines a defective PCB by machine learning. To this end, the PCB exterior inspection apparatus of the present invention comprises: an image input unit receiving an input of a PCB image; an accelerated vision detection unit for detecting a defect candidate area by the embedded accelerated vision processing; and a machine learning determination unit determining the presence or absence of a defect by machine learning for the defect candidate area.

Description

PCB visual inspection apparatus and method using accelerated vision processing and machine learning in heterogeneous platform {PCB visual inspection apparatus and method using accelerated vision processing and machine learning in heterogeneous platform}

The technical field of the present invention is related to embedded vision accelerated processing, machine learning, PCB visual inspection, and OpenVX.

In general, visual inspection of a printed circuit board (PCB) includes Automated Optical Inspection (AOI), which automatically inspects mounted components for defects, and Visual Inspection, which is directly inspected by the inspector using a microscope. It is classified as Visual Inspection.

Automatic optical inspection is performed using expensive equipment in the production process, and visual appearance inspection is used for re-examination to determine whether there is a genuine defect for many defect candidates detected through AOI equipment, or small-scale production rather than mass-production It is used for defect inspection of PCBs. In visual appearance inspection, there is a possibility of human error due to worker's proficiency and fatigue, and when inspecting multiple parts, as the number of parts increases, the speed and efficiency decrease. device is required.

Instead of visual inspection, it is recommended to use a relatively inexpensive embedded board to configure a PCB appearance inspection device that increases inspection speed and does not burden installation costs. However, it is not easy to develop a high-efficiency vision application service on an embedded board with limited hardware and software resources, and much effort is required to maximize execution performance and minimize power consumption. In general, the task of improving the performance of embedded vision applications is done in a relatively accessible way that provides acceleration from an algorithmic point of view, and inefficiencies and bottlenecks from an overall system point of view, such as power consumption, bandwidth load, low latency, and communication overhead between processors. The phenomenon always remains a difficult problem to solve.

The present invention proposes a PCB visual inspection device and method that facilitates software reuse and portability to other platforms in a domain requiring low power and high performance, such as an embedded system, and provides a system-optimized vision processing acceleration function. .

The present invention implements a PCB appearance inspection system on various heterogeneous embedded platform systems, finds defect candidates through low-power accelerated vision processing, and determines defective PCBs by machine learning.

Specifically, according to the present invention, it is composed of an image input unit for receiving a printed circuit board image, a vision acceleration detection unit for detecting defect candidates through embedded vision acceleration processing, and a machine learning discrimination unit for discriminating defects through machine learning.

The image input unit is a part that receives a normal reference PCB image and an inspection target PCB image in real time through a camera. The vision acceleration detection unit compares the reference PCB image and the inspection target PCB image by applying various image processing algorithms in the vision acceleration processing in the embedded system and compares the inspection target printed circuit board in real time with possible defect candidate areas (position, coordinates, etc.) ) is detected. The machine learning discriminating unit determines whether or not there is an actual defect in the detected defect candidate area based on pre-learned data using a machine learning inference technique, and in case of a defect, classifies and informs the defect content or type.

In addition, embedded application developers can improve execution performance and reduce power consumption by ensuring that the PCB exterior inspection according to the present invention is performed on the OpenVX standard framework, which can simultaneously consider overall system-level optimization and algorithm-level optimization through hardware and software accelerators. be able to reduce It can be built at low cost using a low-specification embedded platform connected to a camera, and can be configured to be portable to other embedded devices.

The configuration and operation of the present invention introduced above will become more clear through specific embodiments to be described later along with the drawings.

According to the present invention, it can be easily applied to many types of PCB inspections in various fields, thereby increasing usability. In addition, by building a PCB appearance inspection device at a low cost, inspection of single and multiple areas can be performed at once, replacing the visual inspection using an existing microscope to overcome limitations affected by various human resources such as skill level and fatigue. Inspection can be performed through stable vision processing by reducing errors.

It can inspect at high speed from single mounted component inspection to complex inspection of multiple components at once, and can be freely applied to multiple PCB boards. In addition, by detecting defect candidates through low-power accelerated vision processing and determining defects using AI, efficiency and accuracy are increased, while configuration on various embedded systems is possible, minimizing installation costs and reducing operating costs.

The PCB appearance defect inspection function can be basically used for inspections to determine true defects and false defects in the defect candidate group of automatic optical inspection that extracts many defect points, and for PCB inspections produced on a small scale. As PCB is used in various industrial fields such as displays and secondary batteries, it can be used to determine the quality of various PCB products, so the range of expansion can be expanded in various ways.

1 is a configuration diagram of a printed circuit board exterior inspection apparatus according to an embodiment of the present invention.
Figure 2 shows the detailed configuration of the internal module of the PCB appearance inspection device.
3 is a process graph of OpenVX applied to the PCB appearance inspection of the present invention.
FIG. 4 shows an example in which the image of each execution stage of FIG. 3 and defect candidate regions are detected.
5 is an exemplary view of a result of a PCB appearance inspection device.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs. It is provided to fully inform the holder of the scope of the invention, and the present invention is only defined by the scope of the claims. Meanwhile, terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprise" and/or "comprising" means that a stated component, step, operation, and/or element is one or more other components, steps, operations, and/or elements. does not exclude the presence or addition of

1 is a configuration diagram of an apparatus and method for inspecting a printed circuit board appearance according to an embodiment of the present invention.

The configuration of the present invention includes an image input unit 10 that receives a printed circuit board image, a vision acceleration detection unit 20 that detects a defect candidate area by embedded vision acceleration processing, and determines whether or not the defect candidate area is actually defective by machine learning. It consists of a machine learning discrimination unit 30 that does.

The image input unit 10 is a part that receives a normal reference PCB image 6 and an inspection target PCB image 7 through the camera 5 in real time. In order to reduce the distortion of the camera 5 and input an accurate image, one or more cameras and lights may be installed and interlocked.

The vision acceleration detection unit 20 compares the reference PCB image 6 and the inspection target PCB image 7 by applying various image processing algorithms for vision acceleration processing in the embedded system, and the possibility of defects in the printed circuit board to be inspected in real time. Detect defect candidate regions with .

The machine learning determination unit 30 determines whether or not there is an actual defect in the detected defect candidate region using a machine learning inference technique based on pre-learned data, and classifies and outputs the type or content of the defect if it is a defect.

The printed circuit board appearance inspection apparatus having this configuration may be configured to operate on the OpenVX standard framework in order to achieve optimal performance while reducing power consumption cost.

Figure 2 shows the detailed configuration of the internal module of the PCB appearance inspection device. The configuration of each part is an example and can be implemented in various SW methods as needed.

First, the image input unit 10 includes a first image input processing unit 11a that receives in advance a normal reference PCB image 6 to be used as a reference image for comparison during PCB inspection through the camera 5; a second image input processing unit 11b that receives images of the PCB to be inspected 7 from the camera 5 in real time; It includes a first distortion correction unit 12a and a second distortion correction unit 12b for correcting the distortion of the reference PCB image 6 and the distortion of the PCB image 7 to be inspected.

Since parts of a printed circuit board are small and minute in size, ultra-high-definition images are required to increase inspection accuracy, and one or more camera inputs may be required. In addition, in order to increase the comparison accuracy with the reference PCB image (6), it is good if the brightness of lighting, shadow, camera position, etc. are made in the same environment. In addition, since camera input causes lens distortion, correction is required to create an orthographic image, and when images are received from multiple cameras, an image integration process is also required. The image input unit 10 is responsible for these processes.

Subsequently, the vision acceleration detection unit 20 compares the reference PCB image 6 and the inspection target PCB image 7 through vision acceleration processing in the embedded platform to detect potential defect regions in real time. Explain in detail. The PCB image input through the camera from the image input unit 10 is an ultra-high-definition image, and it is more efficient to use traditional accelerated vision processing rather than machine learning methods in order to detect defect candidates in real time. If the high-definition image of the printed circuit board is changed to a small size required for machine learning input, there is a high probability that features that can identify defects will be lost, and when a high-definition image is used as a machine learning input, image processing takes a lot of time. to be. Therefore, the vision acceleration detection unit 20 compares the reference PCB image 6 and the inspection target PCB image 7 through various vision processing algorithms 21 using hardware and software acceleration functions supported on the embedded system, and defects in real time. and a defect candidate region detection unit 25 for detecting candidate regions. In order to detect defect candidate regions, regions such as locations and coordinates of components mounted on the PCB are extracted from preliminary data (22) including PCB CAD files (eg, Gerber files) (23) and matched (24). ) Defect candidate regions are extracted. The defect candidates detected by the defect candidate area detection unit 25 are transmitted to the machine learning determination unit 30 for each part. Here, the vision processing algorithm 21 includes channel extraction through image conversion, feature point matching, image matching, image comparison, region comparison, noise removal, feature enhancement, and the like.

Finally, the machine learning discrimination unit 30 uses the machine learning inference (learning) model 31 pre-learned with the learning data for each area, and the defect candidate area detection unit 25 of the vision acceleration detection unit 20 For defect candidate regions detected by

Specifically, the defect candidate regions detected by the defect candidate region detection unit 25 of the vision acceleration detection unit 20 are transmitted to the machine learning determination unit 30 for each region, and the received defect candidate regions are selected in a size suitable for machine learning inference and It is processed in the candidate region scaling unit 31 that transforms into a shape and input to the machine learning reasoning unit 32. The machine learning reasoning unit 32 can determine not only whether or not there is a defect, but also the contents of the defect, such as whether a part is missing or incorrectly attached.

Each of the detailed functions of the image input unit 10, vision acceleration detection unit 20, and machine learning determination unit 30 described above requires high-level computational power intensively to perform each function, and CPU, memory, power, etc. It requires a lot of computing resources. Since performance and power have a trade-off, how much power consumption can be reduced while achieving high performance is an important factor in an embedded environment. For the development of high-performance, low-power applications in embedded devices, it is necessary to make the most of the parallel acceleration processor installed in the device in terms of hardware, and to separate modules appropriately, select processors, optimize execution paths, and manage memory in terms of software.

The PCB appearance inspection device according to the present invention can be implemented with OpenVX, an acceleration standard for vision processing. The OpenVX standard is a standard framework that improves portability to platforms while providing acceleration for computer vision processing in accordance with the system environment in consideration of heterogeneous embedded environments. If OpenVX, the acceleration standard for computer vision processing, is used, vision processing results can be obtained on various hardware with one application SW development.

It defines the acceleration layer that application developers can use, and separates the acceleration library so that hardware and software companies can implement OpenVX acceleration functions in their own specialized ways. In other words, OpenVX defines an API that vision applications can be implemented and executed on various platforms, and enables developers to implement applications optimized for the platform on various platforms by implementing the inside of the API with software or hardware in a unique way. .

3 is a process graph of OpenVX applied to the PCB appearance inspection of the present invention. The detailed functions of the image input unit 10, the vision acceleration detection unit 20, and the machine learning determination unit 30 are divided into functional units and expressed as OpenVX standard supported kernels, and each of these kernels is represented as a node along with input and output values. The PCB exterior inspection application of the present invention is expressed as a graph made by connecting abstract nodes of each vision processing kernel. The graph goes through a verification process before being executed, checks data connectivity and consistency for each node in the graph, and optimizes overall execution by identifying inefficient bottlenecks that may occur during execution.

Since kernels are defined in terms of functions or algorithms of classical vision processing, they can be implemented and used considering each embedded system acceleration environment. If you want to operate on all platforms, you can use a CPU-based operating kernel implemented in C-language, and if you want a GPGPU acceleration function, you can use an OpenCL-based GPU parallel acceleration kernel. If you want ARM-based CPU acceleration, you can implement and use a kernel that supports NEON acceleration. After supporting algorithm-specific optimization through supported hardware and software accelerators for each device, the bottleneck section is inspected by checking the total and partial interworking time of the application graph. When a bottleneck section occurs, the structure of the application graph can be modified through cause analysis such as data movement and access speed between each node to support overall interworking optimization at the same time.

Referring to FIG. 3, the overall operation flow of the PCB exterior inspection apparatus according to the present invention using low-power vision acceleration processing and machine learning defect determination will be described. The OpenVX application graph of FIG. 3 is an example of implementing a PCB appearance inspection, and kernels of each part may be implemented in various ways according to software algorithms.

In the image input unit 10, PCB images from one or more cameras are generated as a reference PCB image and a target PCB image [1], and distortion is corrected using a remap algorithm [2].

The vision acceleration detection unit 20 converts the input reference PCB image and the PCB image to be inspected from RGB to YUV [3], and extracts Y-data used for defect candidate detection [4]. In order to accurately match the reference PCB image and the inspection target PCB image from the extracted image, image feature points are extracted through the Harris-Corner algorithm [5]. Then, by scaling the image in various ways through the Gaussian Pyramid [6], the feature point movement path between the two images is extracted using the OpticalFlow algorithm [7]. Based on the feature point movement path value, the entire conversion matrix value between the two images is obtained with the Ransac algorithm [8], and then the pixel location of the reference PCB image is changed by applying the WarpAffine algorithm [9]. A difference image representing only the difference between the two images is obtained by matching the converted reference PCB image and the PCB image to be inspected so that each pixel is accurately matched [10]. Irregular noise of the difference image [11] and unnecessary values below a certain value are removed [12], and an algorithm that strengthens or attenuates the characteristics of the pixel values of the difference image [13] or attenuates [14] is applied so that only the defective area remains. After changing the area such as the coordinates of each part of the PCB read from the part location data (22) including the Gerber file to match the size and coordinates of the currently input inspection target PCB image [15], the difference generated in each part area If the image pixel value exceeds a certain limit, it is set as a defect candidate area [16].

In order to utilize machine learning, the machine learning determination unit 30 converts parts extracted as defect candidate regions into machine learning input sizes [17], and then inputs them to an interface for machine learning determination [18]. It is inferred with the pre-learned reasoning model 31 to determine whether there is a defect, and if it is a defect, it classifies and informs what kind of defect it is [19].

FIG. 4 shows an example of an image of each execution stage of FIG. 3 and detected defect candidate regions. A YUV image converted from a reference image, which is an original RGB image, and a target image, a difference image original, a difference image reversal, a binarization image, And an example of a finally detected candidate image (ROI detection image) can be seen.

Fig. 5 shows the type of defect determined as a result of the defect candidate region. In FIG. 5 , results of various types determined for each defect candidate region detected on the PCB to be inspected, that is, normal, misaligned, and missing, are displayed.

In the above, the configuration of the present invention has been described in detail with reference to the accompanying drawings, but this is only an example, and various modifications and changes within the scope of the technical idea of the present invention to those skilled in the art to which the present invention belongs Of course this is possible. Therefore, the scope of protection of the present invention should not be limited to the above-described embodiments, but should be defined by the description of the claims below.

Claims (1)

An image input unit for receiving a printed circuit board image;
A defect candidate detection unit using vision acceleration processing of an embedded system, and
Including a defect determination unit using machine learning,
PCB appearance inspection device and method using the OpenVX standard that utilizes the hardware and software vision processing acceleration function of the embedded system, supports heterogeneous platform portability, and supports optimization of the interworking process of algorithms and systems for low-power vision processing.

Images