TWI808801B - Abnormal inspection apparatus and abnormal inspection method - Google Patents

Abstract

An abnormal inspection apparatus and an abnormal inspection method are provided. The abnormal inspection apparatus includes a carrier, an image capturer and a processor. The processor is configured to drive the image capturer toward the carrier to capture an original image, segment multiple product images corresponding to multiple products placed on the carrier from the original image, and to overlap cut each product image into multiple sub-images, and then input each sub-image to at least one trained detection module to detect whether there is an abnormal area in each sub-image, output abnormal coordinate information when it is determined that there is an abnormal area, and make a merge for the abnormal areas based on all abnormal coordinate information outputted from the detection module.

Description

Abnormality checking device and abnormality checking method

The present invention relates to an automatic mechanism, and in particular to an abnormality checking device and an abnormality checking method.

A memory module is a printed circuit board with a memory volume circuit. Memory modules can be easily installed and replaced in the electronic systems of computers such as personal computers, workstations and servers. The memory module undergoes multiple inspections during the production process, including: welding inspection, appearance inspection, function inspection, etc. The current appearance inspection of memory modules is carried out manually and visually under a magnifying glass. Therefore, there will be differences in the detection capabilities of different inspection personnel and different product inspection conditions, resulting in a gap in personnel conversion.

The invention provides an abnormality inspection device and an abnormality inspection method, which can automatically detect abnormal regions with bad risks.

The anomaly inspection device of the present invention includes: a carrier for placing a plurality of products; a first image capturer for shooting toward the carrier; and a processor coupled to the first image capturer and configured to drive the first image capturer to capture original images to perform an abnormality detection process, including: cutting out a plurality of product images corresponding to the plurality of products from the original image based on product appearance; overlapping and cutting each product image into a plurality of sub-images; inputting the sub-images to at least one trained detection module to detect whether there is an abnormal region in each sub-image , and when it is determined that there is an abnormal area, output the abnormal coordinate information; and combine all the abnormal coordinate information output by the detection module to the abnormal area.

In an embodiment of the present invention, the carrier includes a first loading area and a second loading area. The abnormality inspection device further includes a second image capture device. The first image capture device shoots towards the first loading area. The second image capture device shoots towards the second loading area. The processor is coupled to the second image capturer and configured to drive the second image capturer to capture another original image to execute the anomaly detection process.

In an embodiment of the present invention, the two horizontally adjacent sub-images have an overlapping portion, and the two vertically adjacent sub-images have an overlapping portion.

In one embodiment of the present invention, the resolution of each of the sub-images is M×N, the resolution of the overlapping portion of each sub-image in the horizontal direction with its adjacent sub-image is M/2×N, and the resolution of the overlapping portion of each sub-image in the vertical direction with its adjacent sub-image is M×N/2.

In an embodiment of the present invention, the above-mentioned processor adopts three detection modules, and the detection modules include a solder mask detection module, a connection part detection module and a label detection module. The abnormal detection process performed by the processor includes: inputting the sub-image to the solder mask detection module to detect whether there is an abnormal area on the solder mask area; input the sub-image to the connection portion detection module to detect whether there is an abnormal area on the metal connection portion; and input the sub-image to the label detection module to detect whether there is an abnormal area on the label printing area.

The anomaly inspection method of the present invention uses a processor to perform multiple steps, including: driving the first image capture device to capture the original image to execute the anomaly detection process, including: cutting out a plurality of product images corresponding to a plurality of products from the original image based on the appearance of the product; overlapping and cutting each product image into a plurality of sub-images; inputting the sub-images to at least one trained detection module to detect whether there is an abnormal area in each sub-image, and output abnormal coordinate information when it is determined that there is an abnormal area; to merge.

Based on the above, the present disclosure integrates the use of an image capture device to obtain original images including multiple products, and uses image recognition to find product images corresponding to the appearance of each product in the original images, and then automatically detects abnormal areas in each product image. Accordingly, the appearance of each product can be inspected under the same inspection standard, and abnormal areas with a risk of failure can be quickly detected.

FIG. 1 is a block diagram of an abnormality checking device according to an embodiment of the invention. Please refer to FIG. 1 , the anomaly inspection device 100 includes a processor 110 , an image capture device 120 and a vehicle 130 . The processor 110 is coupled to the image capture unit 120 . The carrier 130 is used to place a plurality of products for the image capture device 120 to photograph.

The processor 110 is, for example, a central processing unit (Central Processing Unit, CPU), a physical processing unit (Physics Processing Unit, PPU), a programmable microprocessor (Microprocessor), an embedded control chip, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuits, ASIC) or other similar devices.

The image capture device 120 is, for example, a video camera or a camera using a charge coupled device (CCD) lens, a complementary metal oxide semiconductor transistors (CMOS) lens, or the like.

In addition, the abnormality checking device 100 also includes a storage element (not shown) for storing at least one program code segment, which will be executed by the processor 110 to implement the abnormality detection process after being installed. The storage element is, for example, any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hard disk or other similar devices or a combination of these devices.

In one embodiment, any electronic device having the processor 110 together with the image capture device 120 and the carrier 130 can be used to implement the abnormality checking device 100 . The electronic device can be implemented by, for example, a personal computer, a notebook computer, a tablet computer, or a smart phone. The image capture device 120 can be built in the electronic device provided with the processor 110 , or can be externally connected to the electronic device provided with the processor 110 . The carrier 130 is set within the shooting range of the image capture device 120 . The processor 110 is configured to drive the image capture unit 120 to capture raw images to execute the anomaly detection process.

FIG. 2 is a schematic diagram of an anomaly inspection device according to another embodiment of the present invention. Referring to FIG. 2 , the abnormality inspection device 200 is an application example of the abnormality inspection device 100 in FIG. 1 . In this embodiment, the abnormality inspection device 200 is realized by a personal computer (including a display and a host computer) paired with two image capture devices 120 a , 120 b and a carrier 130 . The image capture devices 120a, 120b are, for example, disposed above the display to take pictures towards the direction of the carrier 130 . The image capture devices 120a, 120b adopt the same design as the image capture device 120 shown in FIG. 1 . And the processor 110 is set inside the host. Here, the carrier 130 includes a first loading area 130a and a second loading area 130b. The image capture device 120a shoots toward the first loading area 130a, and the image capture device 120b shoots toward the second loading area 130b. The first loading area 130a is used for placing a plurality of products with the first surface facing upward, and the second loading area 130b is used for placing a plurality of products with the second surface facing upward. Accordingly, multiple products can be processed at the same time.

The processor 110 is configured to drive the image capturers 120a, 120b to capture two original images respectively to execute the anomaly detection process. The steps of the anomaly detection process for an original image are illustrated below with an example.

FIG. 3 is a flowchart of an anomaly checking method according to an embodiment of the invention. In this embodiment, the anomaly detection process is described by using the original image captured by the image capture unit 120 a driven by the processor 110 . The anomaly detection process of the original image captured by the processor 110 driven by the image capture unit 120b can also be compared with the following description. In addition, in the following description, FIG. 4A to FIG. 4E are combined for description. 4A-4E are schematic diagrams of image processing in the anomaly detection process according to an embodiment of the present invention.

In step S305, a plurality of product images corresponding to the plurality of products are cut out from the original image based on the appearance of the product. For example, the product to be inspected is a memory module (a printed circuit board) for description. For example, the first loading area 130a and the second loading area 130b can simultaneously place seven memory modules. Seven memory modules with the upper surface facing up are placed in the first loading area 130a, and another seven memory modules with the lower surface facing up are placed in the second loading area 130b. The number of memory modules that can be loaded in the first loading area 130a and the second loading area 130b is only for illustration and is not limited thereto.

The original image 410 shown in FIG. 4A is, for example, captured by the image capture device 120a. It can be seen from FIG. 4A that the image of the 7-chip memory module is included. Here, in order to avoid using the joined picture obtained by joining multiple small-sized pictures (due to overlapping images around the joined picture), the image capture device 120a is set to directly capture all the products placed on the first loading area 130a to obtain an original image 410 . Under the influence of the least mechanical movement of the capture lens, the image capture device 120a directly uses a high-end linear camera to continuously capture full-area high-resolution full-color images of the product.

The processor 110 cuts the original image 410 into product images corresponding to each product based on the appearance of the product. The abnormality inspection device 200 includes a trained product cutting module, which is used to find out the features of product appearance in a large original image 410 , so as to cut out corresponding product images of each product in the original image 410 . Here, the original image 410 can be cut out into a product image 420 as shown in FIG. 4B .

Next, in step S310 , the product image 420 is overlapped and cut into a plurality of sub-images. In order to prevent the product image 420 from cutting the abnormal points too small to be detected by the detection module, an overlap between adjacent sub-images is designed. For example, the product image 420 is cut into multiple sub-images with a resolution of M×N, two horizontally adjacent sub-images have an overlapping portion, and two vertically adjacent sub-images have an overlapping portion. Where M and N can be equal or not.

In one embodiment, the sub-image size can be designed so that the overlapping portion is 1/2, the resolution of each sub-image in the overlapping portion of its adjacent sub-image in the horizontal direction is M/2×N, and the resolution of the overlapping portion of each sub-image in the vertical direction with its adjacent sub-image is M×N/2. As shown in FIG. 4C , the resolution of overlapping portions in the horizontal direction such as the overlapping portion between the sub-image A11 and the sub-image A12 and the overlapping portion between the sub-image A12 and the sub-image A13 is M/2×N. The resolution of the overlapping portion in the vertical direction such as the overlapping portion between the sub-image A11 and the sub-image A21 , the overlapping portion between the sub-image A21 and the sub-image A31 is M×N/2.

After obtaining a plurality of sub-images, in step S315, input the sub-images to the trained detection module to detect whether there is an abnormal area in each sub-image, and output abnormal coordinate information when it is determined that there is an abnormal area. The detection module is used to detect whether there are abnormalities such as scratches and dirt in the sub-image, so as to output abnormal coordinate information when it is determined that there is an abnormal region.

In one embodiment, only one detection module may be pre-trained to detect anomalies. Each sub-image is input to the detection module, so that the detection module finds out whether there is an abnormal region in the sub-image. The detection module adopts the Convolutional Neural Network (CNN) architecture. There are multiple layers in the CNN architecture, and the input sub-images are analyzed layer by layer, and finally the most likely answer is found. The method of training the detection module is: first input a large amount of learning materials with correct answers marked (such as training sample images with various abnormal objects marked), and let the detection module learn how to judge. Every time the judgment result of the detection module does not match the correct answer, this information is fed back to the previous network layer, and the parameters of each layer are adjusted in order to achieve a more accurate judgment next time. In one embodiment, the detection module adopts YOLO algorithm.

After the detection module detects an abnormal object in the input sub-image, it will use a rectangular frame to mark the abnormal object to represent the location of the detected abnormal object in the input image. Output the center coordinates (bx, by) of the rectangular frame and the length bh and width bw of the rectangular frame as abnormal coordinate information. For example, as shown in FIG. 4D , the sub-images 430-1 and 430-2 are two adjacent sub-images with overlapping portions, and abnormal areas 441 and 442 (ie, rectangular frames) with scratches are detected in the sub-images 430-1 and 430-2, respectively. The detection module will output abnormal coordinate information (including the center coordinates of the rectangular frame and the length and width of the rectangular frame) corresponding to the abnormal regions 441 and 442 .

In another embodiment, three detection modules can also be pre-trained according to the appearance characteristics of the memory module, namely, a solder mask detection module, a connection detection module, and a label detection module, which are respectively used for abnormal detection of the solder mask area, the metal connection portion, and the label printing area.

For example, in general, a memory module not only includes a plurality of memory chips, but also includes a solder resist area, a metal connection portion, and a label printing portion. The solder-proof area refers to covering the upper and lower surfaces of the memory module with a solder mask that does not need to be soldered, so as to meet the requirements of moisture-proof, insulation, solder-proof, high-temperature resistance and aesthetics. The metal connecting part is a gold-plated connecting part used to insert the memory module into the slot. It is composed of many golden-yellow conductive contacts. Because the surface is gold-plated and the conductive contacts are arranged like fingers, it is commonly called "connecting finger". The label printing part is, for example, a barcode sticker, which is printed with product serial number, date of manufacture, place of manufacture, barcode and other information.

The solder mask detection module is a module trained in advance for abnormalities on the solder mask area, and is used to detect abnormal areas on the solder mask area. The connection part detection module is a pre-trained module for abnormalities on the metal connection part, and is used to detect abnormal areas on the metal connection part. The label detection module is a module trained in advance for abnormalities on the label printing part, and is used to detect abnormal areas on the label printing part. The anti-solder detection module, the connection part detection module and the label detection module are all realized by using the YOLO algorithm, for example.

After the detection module detects all sub-images, in step S320, the abnormal regions are combined based on all the abnormal coordinate information output by the detection module. For example, the merged result 440 shown in FIG. 4E is obtained by merging the abnormal regions 441 and 442 shown in FIG. 4D . In addition, after detecting the abnormal areas and performing the merging operation, all abnormal areas can be further marked in the product image as shown in FIG. 4B and presented to the user interface. For example, the product image in which the abnormal area is detected can be displayed in the user interface, and the abnormal area can be marked in the product image (the merged result). In addition, the number of combined abnormal regions can be further summed up for user reference.

To sum up, this disclosure integrates the use of an image capture device to obtain original images including multiple products, and uses image recognition to find product images corresponding to the appearance of each product in the original image, and then automatically detects abnormal areas in each product image. Accordingly, the appearance of each product can be inspected under the same inspection standard, and abnormal areas with a risk of failure can be quickly detected.

100, 200: Abnormality checking device 110: Processor 120, 120a, 120b: image capture device 130: Vehicle 130a: the first loading area 130b: the second loading area 410: Original image 420: Product Image 430-1, 430-2, A11, A12, A13, A21, A31: sub image 440: Merged result 441, 442: abnormal area S305～S320: Steps of abnormality checking method

FIG. 1 is a block diagram of an abnormality checking device according to an embodiment of the invention. FIG. 2 is a schematic diagram of an anomaly inspection device according to another embodiment of the present invention. FIG. 3 is a flowchart of an anomaly checking method according to an embodiment of the invention. 4A-4E are schematic diagrams of image processing in the anomaly detection process according to an embodiment of the present invention.

S305~S320: Steps of the abnormality checking method

Claims (10)

An abnormality inspection device includes: a carrier for placing a plurality of products; a first image capturer for shooting toward the carrier; and a processor coupled to the first image capturer and configured to drive the first image capturer to capture an original image to perform an abnormality detection process, including: cutting out a plurality of product images corresponding to the products based on a product appearance from the original image; overlapping and cutting each product image into multiple sub-images, wherein two adjacent sub-images have overlapping parts; At least one detection module is trained to detect whether there is an abnormal area in each of the sub-images, and when it is determined that there is an abnormal area, output an abnormal coordinate information; and based on all the abnormal coordinate information output by the at least one detection module, merge the abnormal areas. The abnormality inspection device as described in claim 1, wherein the carrier includes a first loading area and a second loading area, and the first image capturer shoots toward the first loading area, and the abnormality inspection device further includes: a second image capturer, shoots toward the second loading area; wherein the processor is coupled to the second image capturer, and is configured to drive the second image capturer to capture another original image to execute the abnormality detection process. The anomaly inspection device according to claim 1, wherein two horizontally adjacent sub-images have an overlapping portion, and two vertically adjacent sub-images have an overlapping portion. The anomaly inspection device as described in Claim 1, wherein the resolution of each of the sub-images is M×N, the resolution of the overlapping portion of each of the sub-images in the horizontal direction with its adjacent sub-image is M/2×N, and the resolution of the overlapping portion of each of the sub-images in the vertical direction with its adjacent sub-image is M×N/2. The abnormality inspection device as described in claim 1, wherein the processor uses three detection modules, the detection module includes a solder-proof detection module, a connection portion detection module and a label detection module, the abnormality detection process performed by the processor includes: inputting these sub-images to the solder-proof detection module to detect whether there is an abnormal area on a solder-proof area; inputting these sub-images to the connection portion detection module to detect whether there is an abnormal area on a metal connection portion; and inputting these sub-images to the label detection module to detect Whether there is an abnormal area on a label printing area. An anomaly inspection method, using a processor to execute a plurality of steps, including: driving a first image capture device to capture an original image to execute an anomaly inspection process, including: cutting out a plurality of product images corresponding to a plurality of products based on a product appearance from the original image; Overlapping and cutting each product image into a plurality of sub-images, wherein two adjacent sub-images have overlapping parts; inputting the sub-images to at least one trained detection module to detect whether there is an abnormal area in each of the sub-images, and outputting an abnormal coordinate information when it is determined that there is an abnormal area; and merging the abnormal areas based on all the abnormal coordinate information output by the at least one detection module. The abnormality inspection method described in claim 6 further includes: driving a second image capture device to capture another original image to execute the abnormality detection process, wherein the products are placed on a carrier, the carrier includes a first loading area and a second loading area, the first image capturer shoots towards the first loading area, and the second image capturer shoots towards the second loading area. The anomaly checking method according to claim 6, wherein two horizontally adjacent sub-images have an overlapping portion, and two vertically adjacent sub-images have an overlapping portion. The anomaly checking method as described in claim 6, wherein the resolution of each of the sub-images is M×N, the resolution of the overlapping portion of each of the sub-images in the horizontal direction with its adjacent sub-image is M/2×N, and the resolution of the overlapping portion of each of the sub-images in the vertical direction with its adjacent sub-image is M×N/2. The abnormality inspection method as described in claim 6, wherein the processor uses three detection modules, the detection modules include a solder mask detection module, a connection detection module and a label detection module, The sub-images are input to the solder resist detection module to detect whether there is an abnormal area on a solder resist area; the sub-images are input to the connection portion detection module to detect whether there is an abnormal area on a metal connection portion; and the sub-images are input to the label detection module to detect whether there is an abnormal area on a label printing area.