TWM633152U - Abnormal inspection apparatus - Google Patents

Abstract

An abnormal inspection apparatus is provided. The abnormal inspection apparatus includes a carrier, an image capturer and a processor. The processor is configured to drive the image capturer toward the carrier to capture an original image, segment multiple product images corresponding to multiple products placed on the carrier from the original image, and to overlap cut each product image into multiple sub-images, and then input each sub-image to at least one trained detection module to detect whether there is an abnormal area in each sub-image, output abnormal coordinate information when it is determined that there is an abnormal area, and make a merge for the abnormal areas based on all abnormal coordinate information outputted from the detection module.

Description

Anomaly checking device

The present invention relates to an automation mechanism, and in particular to an anomaly checking device.

A memory module is a printed circuit board with a memory volume circuit. Memory modules can be easily installed and replaced in the electronic systems of computers such as personal computers, workstations and servers. The memory module undergoes multiple inspections during the production process, including: welding inspection, appearance inspection, function inspection, etc. The current appearance inspection of memory modules is carried out manually and visually under a magnifying glass. Therefore, there will be differences in the detection capabilities of different inspection personnel and different product inspection conditions, resulting in a gap in personnel conversion.

This novel creation provides an abnormality inspection device that can automatically detect abnormal areas of bad risk.

The abnormality inspection device created by the present invention includes: a carrier, used to place multiple products; a first image capture device, which shoots toward the carrier; and a processor, coupled to the first image capture device, and through configured to drive the first image capture device to capture the original image to perform the anomaly detection process, including: cutting out a plurality of product images corresponding to the plurality of products from the original image based on the appearance of the product; overlapping each product image Cutting into multiple sub-images; inputting the sub-images to at least one trained detection module to detect whether there is an abnormal area in each sub-image, and outputting abnormal coordinate information when it is determined that there is an abnormal area; and All the abnormal coordinate information output by the above-mentioned detection modules are combined to merge the abnormal regions.

In an embodiment of the new creation, the carrier includes a first loading area and a second loading area. The abnormality inspection device further includes a second image capture device. The first image capture device shoots towards the first loading area. The second image capture device shoots towards the second loading area. The processor is coupled to the second image capturer and configured to drive the second image capturer to capture another original image to execute the anomaly detection process.

In an embodiment of the present invention, the two horizontally adjacent sub-images have an overlapping portion, and the two vertically adjacent sub-images have an overlapping portion.

In an embodiment of the new creation, the resolution of each sub-image above is M×N, and the resolution of the overlapping portion of each sub-image in the horizontal direction with its adjacent sub-image is M/2×N, and each sub-image The resolution of the overlapping portion of the image in the vertical direction with its adjacent sub-images is M×N/2.

In an embodiment of the new creation, the above-mentioned processor adopts three detection modules, and the detection modules include a solder mask detection module, a connection part detection module and a label detection module. The anomaly detection process executed by the processor includes: inputting the sub-image to the solder mask detection module to detect whether there is an abnormal area on the solder mask area; inputting the sub-image to the connection detection module to detect Whether there is an abnormal area; and input the sub-image to the label detection module to detect whether there is an abnormal area on the label printing area.

Based on the above, this disclosure integrates the use of an image capture device to obtain original images including multiple products, and combines image recognition to find out the product images corresponding to the appearance of each product in the original images, and then automatically detects the product images in each product image. abnormal area. Accordingly, the appearance of each product can be inspected under the same inspection standard, and abnormal areas with a risk of failure can be quickly detected.

FIG. 1 is a block diagram of an abnormality checking device according to an embodiment of the present invention. Please refer to FIG. 1 , the anomaly inspection device 100 includes a processor 110 , an image capture device 120 and a vehicle 130 . The processor 110 is coupled to the image capture unit 120 . The carrier 130 is used to place a plurality of products for the image capture device 120 to photograph.

The processor 110 is, for example, a central processing unit (Central Processing Unit, CPU), a physical processing unit (Physics Processing Unit, PPU), a programmable microprocessor (Microprocessor), an embedded control chip, a digital signal processor (Digital Signal Processor) Processor, DSP), Application Specific Integrated Circuits (Application Specific Integrated Circuits, ASIC) or other similar devices.

The image capture device 120 is, for example, a video camera or a camera using a charge coupled device (CCD) lens, a complementary metal oxide semiconductor transistors (CMOS) lens, or the like.

In addition, the abnormality checking device 100 also includes a storage element (not shown) for storing at least one program code segment, which will be executed by the processor 110 to implement the abnormality detection process after being installed. Storage components such as any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hard disk or other similar devices or combinations of these devices.

In one embodiment, any electronic device having the processor 110 together with the image capture device 120 and the carrier 130 can be used to implement the abnormality inspection device 100 . The electronic device can be implemented by, for example, a personal computer, a notebook computer, a tablet computer, or a smart phone. The image capture device 120 can be built in the electronic device provided with the processor 110 , or can be externally connected to the electronic device provided with the processor 110 . The carrier 130 is set within the shooting range of the image capture device 120 . The processor 110 is configured to drive the image capture unit 120 to capture raw images to execute the anomaly detection process.

Fig. 2 is a schematic diagram of an anomaly inspection device according to another embodiment of the present invention. Referring to FIG. 2 , the abnormality inspection device 200 is an application example of the abnormality inspection device 100 in FIG. 1 . In this embodiment, the abnormality inspection device 200 is realized by a personal computer (including a display and a host computer) paired with two image capture devices 120 a , 120 b and a carrier 130 . The image capture devices 120a, 120b are, for example, disposed above the display to take pictures towards the direction of the carrier 130 . The image capture devices 120a, 120b adopt the same design as the image capture device 120 shown in FIG. 1 . And the processor 110 is set inside the host. Here, the carrier 130 includes a first loading area 130a and a second loading area 130b. The image capture device 120a shoots towards the first loading area 130a, and the image capture device 120b shoots towards the second loading area 130b. The first loading area 130a is used to place a plurality of products with the first surface facing upwards, and the second loading area 130b is used to place a plurality of products with the second surface facing upwards. Accordingly, multiple products can be processed at the same time.

The processor 110 is configured to drive the image capturers 120a, 120b to capture two original images respectively to execute the anomaly detection process. The steps of the anomaly detection process for an original image are illustrated below with an example.

Fig. 3 is a flow chart of an abnormality checking method according to an embodiment of the present invention. In this embodiment, the anomaly detection process is described by using the original image captured by the image capture unit 120 a driven by the processor 110 . The anomaly detection process of the original image captured by the processor 110 driven by the image capture unit 120b can also be compared with the following description. In addition, in the following description, FIG. 4A to FIG. 4E are combined for description. 4A-4E are schematic diagrams of image processing in an abnormality detection process according to an embodiment of the present invention.

In step S305, a plurality of product images corresponding to the plurality of products are cut out from the original image based on the appearance of the product. For example, the product to be inspected is a memory module (a printed circuit board) for description. For example, the first loading area 130a and the second loading area 130b can simultaneously place seven memory modules. Seven memory modules with the upper surface facing up are placed in the first loading area 130a, and another seven memory modules with the lower surface facing up are placed in the second loading area 130b. The number of memory modules that can be loaded in the first loading area 130a and the second loading area 130b is only for illustration and is not limited thereto.

The original image 410 shown in FIG. 4A is, for example, captured by the image capture device 120a. It can be seen from FIG. 4A that the image of the 7-chip memory module is included. Here, in order to avoid using the joined picture obtained by joining multiple small-sized pictures (due to overlapping images around the joint picture), the image capture device 120a is set to directly capture the image on the first loading area 130a. All products placed to obtain an original image 410 . Under the influence of the least mechanical movement of the capture lens, the image capture device 120a directly uses a high-end linear camera to capture images of the product in continuous full-area high-resolution full-color.

The processor 110 cuts the original image 410 into product images corresponding to each product based on the appearance of the product. The abnormality inspection device 200 includes a trained product cutting module, which is used to find out the features of the product appearance in a large original image 410, thereby cutting out the corresponding products of each product in the original image 410 image. Here, the original image 410 can be cut out into a product image 420 as shown in FIG. 4B .

Next, in step S310 , the product image 420 is overlapped and cut into a plurality of sub-images. In order to prevent the product image 420 from cutting the abnormal points too small to be detected by the detection module, an overlap between adjacent sub-images is designed. For example, the product image 420 is cut into multiple sub-images with a resolution of M×N, two horizontally adjacent sub-images have an overlapping portion, and two vertically adjacent sub-images have an overlapping portion. Where M and N can be equal or not.

In one embodiment, it can be designed such that the overlapping portion is 1/2 of the sub-image size, and the resolution of the overlapping portion of each sub-image in the horizontal direction with its adjacent sub-image is M/2×N, and each sub-image is at The resolution of the overlapping part of the adjacent sub-images in the vertical direction is M×N/2. As shown in FIG. 4C , the resolution of overlapping portions in the horizontal direction such as the overlapping portion between the sub-image A11 and the sub-image A12 and the overlapping portion between the sub-image A12 and the sub-image A13 is M/2×N. The resolution of the overlapping portion in the vertical direction such as the overlapping portion between the sub-image A11 and the sub-image A21 , the overlapping portion between the sub-image A21 and the sub-image A31 is M×N/2.

After obtaining a plurality of sub-images, in step S315, input the sub-images to the trained detection module to detect whether there is an abnormal area in each sub-image, and output abnormal coordinate information when it is determined that there is an abnormal area. The detection module is used to detect whether there are abnormalities such as scratches and dirt in the sub-image, so as to output abnormal coordinate information when it is determined that there is an abnormal region.

In one embodiment, only one detection module may be pre-trained to detect anomalies. Each sub-image is input to the detection module, so that the detection module finds out whether there is an abnormal region in the sub-image. The detection module adopts the Convolutional Neural Network (CNN) architecture. There are multiple layers in the CNN architecture, and the input sub-images are analyzed layer by layer, and finally the most likely answer is found. The method of training the detection module is: first input a large amount of learning materials with correct answers marked (such as training sample images with various abnormal objects marked), and let the detection module learn how to judge. Every time the judgment result of the detection module does not match the correct answer, this information is fed back to the previous network layer, and the parameters of each layer are adjusted in order to achieve a more accurate judgment next time. In one embodiment, the detection module adopts YOLO algorithm.

After the detection module detects an abnormal object in the input sub-image, it will use a rectangular frame to mark the abnormal object to represent the location of the detected abnormal object in the input image. Output the center coordinates (bx, by) of the rectangular frame and the length bh and width bw of the rectangular frame as abnormal coordinate information. For example, as shown in FIG. 4D, the sub-images 430-1 and 430-2 are two adjacent sub-images with overlapping portions, and abnormal areas with scratches are detected in the sub-images 430-1 and 430-2 respectively. 441, 442 (ie, rectangular boxes). The detection module will output abnormal coordinate information (including the center coordinates of the rectangular frame and the length and width of the rectangular frame) corresponding to the abnormal regions 441 and 442 .

In another embodiment, it is also possible to pre-train three detection modules for the appearance characteristics of the memory module, that is, the solder mask detection module, the connection part detection module and the label detection module, which are respectively used to detect the solder mask Abnormal detection of areas, metal joints and label printing areas.

For example, in general, a memory module not only includes a plurality of memory chips, but also includes a solder resist area, a metal connection portion, and a label printing portion. The solder-proof area refers to covering the upper and lower surfaces of the memory module with a solder mask that does not need to be soldered, so as to meet the requirements of moisture-proof, insulation, solder-proof, high-temperature resistance and aesthetics. The metal connection part is the gold-plated connection part used to insert the memory module into the slot. It is composed of many golden conductive contacts. Because the surface is gold-plated and the conductive contacts are arranged like fingers, it is commonly called "gold finger ( connecting finger)". The label printing part is, for example, a barcode sticker, which is printed with product serial number, date of manufacture, place of manufacture, barcode and other information.

The solder mask detection module is a module trained in advance for abnormalities on the solder mask area, and is used to detect abnormal areas on the solder mask area. The connection part detection module is a pre-trained module for abnormalities on the metal connection part, and is used to detect abnormal areas on the metal connection part. The label detection module is a module trained in advance for abnormalities on the label printing part, and is used to detect abnormal areas on the label printing part. The anti-solder detection module, the connection part detection module and the label detection module are all realized by using the YOLO algorithm, for example.

After the detection module detects all sub-images, in step S320, the abnormal regions are combined based on all the abnormal coordinate information output by the detection module. For example, the merged result 440 shown in FIG. 4E is obtained by merging the abnormal regions 441 and 442 shown in FIG. 4D . In addition, after detecting the abnormal areas and performing the merging operation, all abnormal areas can be further marked in the product image as shown in FIG. 4B and presented to the user interface. For example, the product image in which the abnormal area is detected can be displayed in the user interface, and the abnormal area can be marked in the product image (the merged result). In addition, the number of combined abnormal regions can be further summed up for user reference.

To sum up, this disclosure integrates the use of an image capture device to obtain original images including multiple products, and combines image recognition to find product images corresponding to the appearance of each product in the original image, and then automatically detects each product Unusual areas in the image. Accordingly, the appearance of each product can be inspected under the same inspection standard, and abnormal areas with a risk of failure can be quickly detected.

100, 200: Abnormality checking device 110: Processor 120, 120a, 120b: image capture device 130: Vehicle 130a: the first loading area 130b: the second loading area 410: Original image 420: Product Image 430-1, 430-2, A11, A12, A13, A21, A31: sub image 440: Merged result 441, 442: abnormal area S305～S320: Steps of abnormality checking method

FIG. 1 is a block diagram of an abnormality checking device according to an embodiment of the present invention. Fig. 2 is a schematic diagram of an anomaly inspection device according to another embodiment of the present invention. Fig. 3 is a flow chart of an abnormality checking method according to an embodiment of the present invention. 4A-4E are schematic diagrams of image processing in an abnormality detection process according to an embodiment of the present invention.

100: Abnormality checking device

110: Processor

120: Image grabber

130: Vehicle

Claims (5)

An abnormality checking device, comprising: A carrier for placing multiple products; a first image capture device for shooting towards the vehicle; and A processor, coupled to the first image capture device, configured to drive the first image capture device to capture an original image to perform an anomaly detection process, including: Cutting out a plurality of product images corresponding to the products based on a product appearance from the original image; Overlapping and cutting each of the product images into a plurality of sub-images; Inputting the sub-images into at least one trained detection module to detect whether there is an abnormal area in each of the sub-images, and output an abnormal coordinate information when it is determined that there is an abnormal area; and The abnormal regions are combined based on all abnormal coordinate information output by the at least one detection module. The abnormality inspection device as described in claim 1, wherein the carrier includes a first loading area and a second loading area, the first image capture device shoots toward the first loading area, and the abnormality inspection device Also includes: a second image capture device, shooting towards the second loading area; Wherein the processor is coupled to the second image capture device and is configured to drive the second image capture device to capture another original image to execute the anomaly detection process. The anomaly inspection device according to claim 1, wherein two horizontally adjacent sub-images have an overlapping portion, and two vertically adjacent sub-images have an overlapping portion. The anomaly inspection device as described in claim 1, wherein the resolution of each of the sub-images is M×N, and the resolution of the overlapping part of each of the sub-images in the horizontal direction with its adjacent sub-image is M /2×N, the resolution of each of the sub-images overlapping with its adjacent sub-images in the vertical direction is M×N/2. The abnormality inspection device as described in claim 1, wherein the processor uses three detection modules, the detection modules include a solder mask detection module, a connection detection module and a label detection module, The anomaly detection process performed by the processor includes: Inputting the sub-images to the solder mask detection module to detect whether there is an abnormal area on a solder mask area; inputting the sub-images to the connection detection module to detect whether there is an abnormal area on a metal connection; and These sub-images are input to the label detection module to detect whether there is an abnormal area on a label printing area.

Images