CN112964724B - Multi-target multi-region visual detection method and detection system - Google Patents
Multi-target multi-region visual detection method and detection system Download PDFInfo
- Publication number
- CN112964724B CN112964724B CN202110139585.5A CN202110139585A CN112964724B CN 112964724 B CN112964724 B CN 112964724B CN 202110139585 A CN202110139585 A CN 202110139585A CN 112964724 B CN112964724 B CN 112964724B
- Authority
- CN
- China
- Prior art keywords
- detected
- product
- defect detection
- image
- region
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 154
- 230000000007 visual effect Effects 0.000 title description 19
- 230000007547 defect Effects 0.000 claims abstract description 133
- 238000000034 method Methods 0.000 claims abstract description 37
- 238000012545 processing Methods 0.000 claims abstract description 34
- 238000011179 visual inspection Methods 0.000 claims abstract description 12
- 239000000284 extract Substances 0.000 claims abstract description 6
- 230000008569 process Effects 0.000 claims description 13
- 239000003086 colorant Substances 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 8
- 238000003384 imaging method Methods 0.000 claims description 5
- 238000007689 inspection Methods 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 7
- 238000012937 correction Methods 0.000 description 6
- 230000033001 locomotion Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000000605 extraction Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 239000003292 glue Substances 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 230000007797 corrosion Effects 0.000 description 1
- 238000005260 corrosion Methods 0.000 description 1
- 230000032798 delamination Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 230000003647 oxidation Effects 0.000 description 1
- 238000007254 oxidation reaction Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 229910052710 silicon Inorganic materials 0.000 description 1
- 239000010703 silicon Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/93—Detection standards; Calibrating baseline adjustment, drift correction
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/95—Investigating the presence of flaws or contamination characterised by the material or shape of the object to be examined
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
- G01N2021/8854—Grading and classifying of flaws
- G01N2021/8877—Proximity analysis, local statistics
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
- G01N2021/8854—Grading and classifying of flaws
- G01N2021/888—Marking defects
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
- G01N2021/8887—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges based on image processing techniques
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/95—Investigating the presence of flaws or contamination characterised by the material or shape of the object to be examined
- G01N2021/9511—Optical elements other than lenses, e.g. mirrors
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Biochemistry (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Immunology (AREA)
- Pathology (AREA)
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Signal Processing (AREA)
- Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
Abstract
One or more embodiments of the present disclosure provide a multi-target multi-region visual inspection method and system. The method comprises the following steps: the product bearing device moves the product to be detected to an image acquisition position of the image acquisition device; the image acquisition device scans and images the product to be detected, obtains an image to be detected of the product to be detected and sends the image to the image processing device; the image processing device extracts a plurality of areas of a plurality of products to be detected based on the images to be detected to obtain a plurality of different types of areas to be detected of the products to be detected, performs defect detection based on the different types of areas to be detected of the products to be detected to obtain partial defect detection results of the areas to be detected, generates a complete defect detection result based on the partial defect detection results and sends the complete defect detection result to the upper computer; and the upper computer displays the complete defect detection result. The method and the system of the embodiment of the description can realize defect detection of the product with multiple detection targets.
Description
Technical Field
One or more embodiments of the present disclosure relate to the field of semiconductor inspection technology, and in particular, to a multi-target multi-region visual inspection method and system.
Background
Display fingerprint sensing, also known as screen fingerprint sensing, is an emerging function of smartphones. With the improvement of screen fingerprint sensing technology and the trend of mobile phones to comprehensive screen design, screen fingerprint technology is rapidly developed.
At present, a fingerprint module generally adopts a technical scheme of optical imaging and ultrasonic wave feedback. The fingerprint module belongs to a high-precision complex process, and various defects easily occur in the manufacturing process, so that the fingerprint module needs to be detected.
However, in the prior art, the field type of AOI vision has low popularity at present, and main equipment is strictly customized according to the existing process to match the current model and detection requirements, so that the detection of multiple defects of multiple targets cannot be realized.
Disclosure of Invention
In view of the foregoing, it is an object of one or more embodiments of the present disclosure to provide a multi-target multi-area visual inspection method and system for solving the defect inspection problem of products with multiple inspection targets.
In view of the above objects, one or more embodiments of the present specification provide a multi-target multi-region visual inspection method, including:
the product bearing device moves the product to be detected to an image acquisition position of the image acquisition device;
the image acquisition device scans and images the product to be detected, obtains an image to be detected of the product to be detected and sends the image to the image processing device;
the image processing device extracts a plurality of areas of a plurality of products to be detected based on the images to be detected to obtain a plurality of different types of areas to be detected of the products to be detected, performs defect detection based on the different types of areas to be detected of the products to be detected to obtain partial defect detection results of the areas to be detected, generates a complete defect detection result based on the partial defect detection results and sends the complete defect detection result to an upper computer;
and the upper computer displays the complete defect detection result.
Optionally, the method further comprises: and the image acquisition device simultaneously executes the transmission of the complete image of the product to be detected in the current frame and the shooting of the complete image of the product to be detected in the next frame.
Optionally, the image acquisition device acquires a whole board to-be-detected object as the to-be-detected image, where the whole board to-be-detected object includes one or more to-be-detected products.
Optionally, before the image acquisition device scans and images the product to be detected, the image acquisition device further includes: and obtaining the whole plate mark of the whole plate to-be-detected target.
Optionally, the method further comprises: acquiring an area of interest in the whole board to-be-detected target, wherein the area of interest comprises all the to-be-detected products;
and carrying out row-column division on the region of interest, and determining the product number of each product to be detected based on the position relation between the result of the row-column division and the product to be detected.
Optionally, the method further comprises:
acquiring an array-arranged region of interest in the whole board to-be-detected target, wherein each region of interest comprises one to-be-detected product;
and determining the product number of each product to be detected based on the position relation between the region of interest and the product to be detected.
Optionally, the method further comprises:
the image processing device determines the area mark of the area to be detected of the product to be detected, and processes the partial defect detection result belonging to the same product to be detected based on the product number and the area mark so as to generate a product defect detection result of the product to be detected.
Optionally, the method further comprises:
the image processing device generates the complete defect detection result of the whole board to-be-detected target based on the whole board mark and the product defect detection results of all the to-be-detected targets subordinate to the whole board.
Optionally, performing defect detection based on a plurality of different types of the to-be-detected areas of a plurality of to-be-detected products to obtain partial defect detection results of the to-be-detected areas, including:
acquiring a pre-stored defect detection template based on the whole plate mark and the area mark; the defect detection template comprises a reference image and a mask image of a standard product;
generating a mask image to be detected of the region to be detected based on the region to be detected and the defect detection template;
performing self-adaptive enhancement filtering based on the mask image to be detected to obtain a filtered image to be detected;
and carrying out local dynamic threshold division based on the filtering image to be detected, and obtaining the partial defect detection result of the region to be detected based on the result of the local dynamic threshold division.
One or more embodiments of the present disclosure provide a multi-target multi-region visual inspection system, configured to implement a multi-target multi-region visual inspection method as set forth in any one of the foregoing, including a product bearing device, an image acquisition device, an image processing device, and a host computer; wherein,
the product carrier is configured to: moving the product to be detected to an image acquisition position of an image acquisition device;
the image acquisition device is configured to: scanning and imaging the product to be detected, obtaining a complete image of the product to be detected, and sending the complete image to an image processing device;
the image processing apparatus is configured to: extracting a plurality of areas of a plurality of products to be detected to obtain a plurality of different types of areas to be detected of a plurality of products to be detected, performing defect detection on the plurality of different types of areas to be detected of the plurality of products to be detected respectively to obtain partial defect detection results of the areas to be detected, generating a complete defect detection result based on the partial defect detection results, and sending the complete defect detection result to an upper computer;
the upper computer is configured to: and displaying the complete defect detection result.
From the above, it can be seen that the multi-target multi-region visual detection method and system provided in one or more embodiments of the present disclosure obtain a complete image of a product to be detected through an image acquisition device, perform region division on the complete image of the product to be detected through an image detection device, perform defect detection respectively, aggregate the detected partial defect detection results to generate a complete defect detection result, and display the complete defect detection result through an upper computer, thereby providing a general multi-target detection method; meanwhile, defect detection and motion control are dispersed, so that resource contention and robbery can be avoided when the defect detection and the motion control are in high concurrency and strong calculation force, and the failure rate is reduced; the defect detection is realized by the image detection device, the bottom code of the upper computer is not required to be modified during the change, the visual defect detection part is not influenced, and the expansibility is good.
Drawings
For a clearer description of one or more embodiments of the present description or of the solutions of the prior art, the drawings that are necessary for the description of the embodiments or of the prior art will be briefly described, it being apparent that the drawings in the description below are only one or more embodiments of the present description, from which other drawings can be obtained, without inventive effort, for a person skilled in the art.
FIG. 1 is a flow diagram of a multi-target multi-region visual inspection method according to one or more embodiments of the present disclosure;
FIG. 2 is a schematic diagram of an image acquisition process according to one or more embodiments of the present disclosure;
FIG. 3 is a schematic diagram of multi-target multi-zone detection in accordance with one or more embodiments of the present disclosure;
FIG. 4 is a schematic representation of region of interest extraction according to one or more embodiments of the present disclosure;
FIG. 5 is a schematic representation of another region of interest extraction in accordance with one or more embodiments of the present disclosure;
FIG. 6 is a schematic diagram illustrating defect detection in accordance with one or more embodiments of the present disclosure;
FIG. 7 is a block diagram of a multi-target multi-zone visual inspection system in accordance with one or more embodiments of the present disclosure.
Detailed Description
For the purposes of promoting an understanding of the principles and advantages of the disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the same.
It is noted that unless otherwise defined, technical or scientific terms used in one or more embodiments of the present disclosure should be taken in a general sense as understood by one of ordinary skill in the art to which the present disclosure pertains. The use of the terms "first," "second," and the like in one or more embodiments of the present description does not denote any order, quantity, or importance, but rather the terms "first," "second," and the like are used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that elements or items preceding the word are included in the element or item listed after the word and equivalents thereof, but does not exclude other elements or items. The terms "connected" or "connected," and the like, are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", etc. are used merely to indicate relative positional relationships, which may also be changed when the absolute position of the object to be described is changed.
As described in the background section, the fingerprint module belongs to a high-precision complex process, and includes a circuit area, a golden finger area, a Mark area, a lattice area, an edge and other multi-part structural areas, and various defects are easy to occur in the process of manufacturing. Among them, common defects are bubbles, scratches, leakage, offset, delamination, residual glue, glue overflow, surface dirt, edge chipping, bulges, marks, concave-convex points, saw teeth, notches, oxidation, corrosion and the like. The detection accuracy requirement is high, and defect detection is difficult to carry out.
However, the form difference of the AOI visual equipment provided by different process sections is very large at present, the single-rail type AOI visual equipment has double rails, the round trip type AOI visual equipment has straight-through type AOI visual equipment has completely different logic cooperation of visual schemes, many of the AOI visual equipment models are custom developed, the interfaces are complicated, the interaction difficulty is high, after the AOI visual equipment is sequentially configured, parameters are difficult to modify, when a new product or a new defect occurs, the AOI visual equipment needs to be changed from a code layer again, the work load is easy to downtime, and productivity loss is caused.
The applicant finds that in the process of realizing the present disclosure, if the multi-target multi-region composite scene detection can be uniformly realized, the detection of the fingerprint module can be better realized.
The technical scheme of the present disclosure is further described in detail below through specific examples.
One or more embodiments of the present disclosure provide a multi-target multi-region visual inspection method, as shown in fig. 1, including:
step S101, the product bearing device moves the product to be detected to an image acquisition position of the image acquisition device.
The products to be detected in the step are high-precision, small-vision, multi-target, multi-area and multi-specification detection products applied to industries such as fingerprint modules, display chips, optical communication chips, micro-electromechanical chips (MEMS), general silicon-based chips and the like. The product bearing device is a six-axis robot, and the six-axis robot clamps the product to be detected and moves to the image acquisition position of the image acquisition device under the control of the upper computer.
Step S102, the image acquisition device scans and images the product to be detected, obtains a complete image of the product to be detected and sends the complete image to the image processing device.
In the step, the image acquisition device can be a linear array camera or an area array camera, the linear scanning camera uses an external trigger signal of an I/O card to perform continuous linear scanning imaging, the area array camera performs image acquisition by means of software internal trigger, and the image acquisition under different precision and light fields is realized through parallel acquisition of a plurality of linear array cameras and the area array cameras.
Step S103, the image processing device extracts a plurality of regions of the products to be detected, obtains a plurality of regions to be detected of different types of the products to be detected, performs defect detection based on the regions to be detected of different types of the products to be detected, so as to obtain partial defect detection results of the regions to be detected, generates a complete defect detection result based on the partial defect detection results, and sends the complete defect detection result to the upper computer.
The image processing device is provided with an image detection system and a built-in image detection algorithm, and can detect defects of products to be detected based on the images to be detected. The image processing device may be an industrial personal computer (Industrinl Pesonal Computer, IPC). In this embodiment, as shown in fig. 2, the image acquisition device supports parallel multi-thread detection, and each time the image acquisition device acquires a frame of image to be detected, the image acquisition device transmits the frame of image to be detected to the image processing device, and the image processing device performs detection, that is, the image acquisition device simultaneously performs transmission of the current frame of image to be detected and shooting of the next frame of image to be detected, thereby improving detection efficiency and avoiding mutual competition between resources.
In this step, as shown in fig. 3, after the complete image of the product to be detected is obtained, the image processing device performs region extraction, so as to obtain multiple types of detection regions, such as a line region, a golden finger region, a Mark region, a lattice region, an edge, etc., of each product to be detected, then performs defect detection on each different region, and sums up the detection results and then draws the detection results on a complete image to be detected, so as to facilitate the inspection of detection personnel. Optionally, different defects are assigned different colors for ease of viewing. The complete defect detection result includes detection process data and final detection results, such as original drawings, defect small drawings, defect information and the like.
Step S104, the upper computer displays the complete defect detection result.
In this step, the upper computer displays according to the display requirement preset by the system, so as to be checked by the detection personnel. Meanwhile, the upper computer can provide tracing and associated display of the defect judging result to the image, realize switching preview of the detecting result of the historical products one by one, and display the image, operator executing attribute, judging result, data, state, flow result information and the like which belong to the current product to be detected.
According to the multi-target multi-region visual detection method disclosed by the embodiment of the specification, a complete image of a product to be detected is obtained through an image acquisition device, the complete image of the product to be detected is subjected to region division through an image detection device and then is subjected to defect detection respectively, and the detected partial defect detection results are summarized to generate a complete defect detection result and then are displayed through an upper computer, so that the universal multi-target detection method is provided; meanwhile, defect detection and motion control are dispersed, so that resource contention and robbery can be avoided when the defect detection and the motion control are in high concurrency and strong calculation force, and the failure rate is reduced; the defect detection is realized by the image detection device, the bottom code of the upper computer is not required to be modified during the change, the visual defect detection part is not influenced, and the expansibility is good.
In some embodiments of the present disclosure, the product carrying device may simultaneously carry one or more objects to be detected, that is, the one or more objects to be detected exist in a whole plate shape, and after the product carrying device takes out the whole plate of objects to be detected from the tray and moves the whole plate of objects to be detected to an image collecting position of the image collecting device, the image collecting device scans the whole plate of objects to be detected to obtain an image to be detected. In some optional embodiments, before the image acquisition device performs scanning imaging on the product to be detected, the method further includes: and obtaining the whole plate mark of the whole plate to-be-detected target.
In this embodiment, before the image to be detected is collected, the code is read on the five-bit character string on the product to be detected to obtain the whole board mark of the whole board object to be detected, and the whole board mark is sent to the image collecting device, and the image collecting device can determine the type, model and other characteristics of the product to be detected based on the whole board mark. The whole plate mark is contained in the follow-up process of region extraction and defect detection, so that a multi-camera, multi-state, multi-target, multi-region, multi-parameter, multi-specification and multi-mark complete defect detection chain is realized, and the complete defect detection chain is convenient to finally gather all partial defect detection results belonging to the same whole plate to-be-detected target based on the whole plate mark and provide for detection personnel to check. Optionally, after the whole board mark of the whole board to-be-detected target is obtained, row and column information data of the whole board to-be-detected target is obtained.
Optionally, the movement in the XYZ axis direction is performed based on the position identification point (Mark point) on the whole board to-be-detected target, so as to implement angle correction and position correction of the whole board to-be-detected target, and enable the whole board to-be-detected target to be shot by the image acquisition device. In this embodiment, when the overall pose of the product to be detected is offset, the image acquisition device determines the pose offset of the product to be detected in the image to be detected based on the position identification point (Mark point) of the product to be detected and the pre-stored reference image, and then adjusts the position of the product carrying device based on the pose offset, thereby realizing the angle correction and the position correction of the product carrying device.
After angle correction and position correction are completed, the six-axis robot clamps the whole plate to-be-detected target to pass through an image acquisition area of the image acquisition device at a constant speed, so that an image to be detected is obtained.
In the whole board to-be-detected targets, the number and the positions of the products to be detected may not be fixed, namely, when the positions of the products to be detected in different whole board to-be-detected targets are random and the number of the products to be detected is also random, as shown in fig. 4, the image processing device obtains the region of interest in the whole board to-be-detected targets, then the region of interest ROI is subjected to row and column division, and the product number of each product to be detected is determined based on the position relationship between the result of the row and column division and the products to be detected. In this embodiment, the region of interest includes all the products to be detected, and when the subsequent detection is performed based on the region of interest, the coordinate information of each row and each column in the region of interest can be obtained by dividing the region of interest into rows and columns, and each product to be detected is numbered based on the positional relationship between the coordinate information of each row and each column in the region of interest and the products to be detected.
In other alternative embodiments, when the positions of the products to be detected in the whole board of the objects to be detected are regularly arranged, whether the number of the products to be detected is fixed or not at this time, a plurality of regions of interest which are regularly arranged can be obtained. As shown in fig. 5, the products to be detected are arranged in an array in the whole board of the targets to be detected, at this time, the interested areas of the array arrangement in the whole board of the targets to be detected are obtained, and the product number of each product to be detected is determined based on the position relation between the interested areas and the products to be detected; wherein each region of interest comprises one of the products to be tested. Thus, the position of each product to be detected can be determined based on the position of each region of interest, and each product to be detected is numbered.
Therefore, whether the number of products to be detected in the whole board to-be-detected target fluctuates and changes, whether the position fluctuates and changes, whether the distance fluctuates and changes, and whether even local features change or not, the multi-target multi-region visual detection can extract the region of interest and then detect the defects.
In other optional embodiments of the present disclosure, when the image processing apparatus extracts a plurality of areas of the plurality of products to be detected based on the image to be detected to obtain an area to be detected, an area mark of the area to be detected of the product to be detected is determined at the same time, and the area mark is included in the subsequent defect detection and the detection result, so as to facilitate the subsequent data processing. That is, the image processing device processes the partial defect detection results belonging to the same product to be detected based on the product number and the region mark to generate a product defect detection result of the product to be detected. And meanwhile, the image processing device also generates the complete defect detection result of the whole board to-be-detected target based on the whole board mark and the complete defect detection result of all the product defect detection results belonging to the whole board to-be-detected target.
In the above-described embodiments, a multi-ROI multi-parameter grouping sharing technique may be employed to add multiple regions of interest (ROIs) based on the functionality of the multiple ROI groupings. The added multiple regions of interest can be of different types such as rectangular, circular, polygonal and the like, the multiple different regions of interest are grouped, different groups are marked by adopting different colors so as to distinguish different groups, and input parameters, attribute parameters and tolerance parameters belonging to the same grouped regions of interest are uniformly configured, edited and modified without independently modifying the independent configuration; meanwhile, the parameters of the ungrouped regions of interest can be set independently, so that different types of regions of interest can be identified more conveniently, multi-region detection is realized, and convenience is brought to users for use and viewing.
Optionally, the performing defect detection based on the plurality of different types of the to-be-detected areas of the plurality of to-be-detected products in step S103 to obtain partial defect detection results of the respective to-be-detected areas includes:
step S201, a pre-stored defect detection template is obtained based on the whole plate mark and the area mark; the defect detection template includes a reference image and a mask image of a standard product.
Step S202, generating a mask image to be detected of the region to be detected based on the region to be detected and the defect detection template, extracting edge threshold pixels of brightness change of the edge outline of the target region, filtering out interference objects through area, polarity and the like, smoothing the interference objects into a closed region, and converting the closed region into a mask image after attaching and wrapping the edge of an actual target.
And step S203, performing self-adaptive enhancement filtering based on the mask image to be detected to obtain a filtered image to be detected.
Step S204, carrying out local dynamic threshold division based on the filtering image to be detected, and obtaining the partial defect detection result of the region to be detected based on the result of the local dynamic threshold division.
In this embodiment, first, a product image of a standard sample without defects is obtained as a reference image and a Mask (Mask) image, and the images to be detected in the subsequent detection are compared with the reference image and the Mask image, so as to determine the brightness, the darkness, the gray scale and the color of the difference pixel with the defects.
When detecting, based on the whole plate mark determination of the current region to be detected and the type of the defect detection template used for the region mark determination, the defect detection is carried out by calling the defect detection template corresponding to the region to be detected.
As shown in fig. 6, first, a mask image to be detected is generated from an original image of a region to be detected based on a defect detection template. And then processing the mask image to be detected by adopting a self-adaptive enhancement filtering algorithm, dividing the mask image to be detected based on a dynamic threshold value, judging whether the current area is a defect based on the threshold value dividing result, if so, marking the current area and determining information such as classification, grade and the like of the defect.
It should be noted that the foregoing describes specific embodiments of the present invention. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
According to the multi-target multi-region visual detection method disclosed by the embodiment of the specification, the capability of multi-target simultaneous detection is provided, the tracking precision of 1 pixel is realized after the targets are segmented and extracted and the ROI array is performed, and the fluctuation of the number, the position, the interval and even the local characteristics of the targets are allowed to be changed; through marking the region of the ROI, different definition modes are adopted for realizing defects of different regions, and the defects are drawn to a client through a feature map; the multi-camera, multi-state, multi-target, multi-region, multi-parameter, multi-specification and multi-mark complete defect detection chain of the whole software can be realized when the multi-ROI is used for multi-parameter, and the actual detection scene of a customer site is matched.
Based on the same inventive concept, corresponding to the method of any embodiment, one or more embodiments of the present disclosure further provide a multi-target multi-region visual detection system, configured to implement the multi-target multi-region visual detection method according to any one of the embodiments. As shown in fig. 7, the detection system includes a product bearing device, an image acquisition device, an image processing device and an upper computer; wherein,
the product carrier is configured to: moving the product to be detected to an image acquisition position of an image acquisition device;
the image acquisition device is configured to: scanning and imaging the product to be detected, obtaining a complete image of the product to be detected, and sending the complete image to an image processing device;
the image processing apparatus is configured to: extracting a plurality of areas of a plurality of products to be detected to obtain a plurality of different types of areas to be detected of a plurality of products to be detected, performing defect detection on the plurality of different types of areas to be detected of the plurality of products to be detected respectively to obtain partial defect detection results of the areas to be detected, generating a complete defect detection result based on the partial defect detection results, and sending the complete defect detection result to an upper computer;
the upper computer is configured to: and displaying the complete defect detection result.
The device of the foregoing embodiment is configured to implement the corresponding multi-target multi-area visual detection method of the foregoing embodiment, and has the beneficial effects of the corresponding method embodiment, which is not described herein.
Those of ordinary skill in the art will appreciate that: the discussion of any of the embodiments above is merely exemplary and is not intended to suggest that the scope of the disclosure, including the claims, is limited to these examples; combinations of features of the above embodiments or in different embodiments are also possible within the spirit of the present disclosure, steps may be implemented in any order, and there are many other variations of the different aspects of one or more embodiments described above which are not provided in detail for the sake of brevity.
Additionally, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown within the provided figures, in order to simplify the illustration and discussion, and so as not to obscure one or more embodiments of the present description. Furthermore, the apparatus may be shown in block diagram form in order to avoid obscuring the one or more embodiments of the present description, and also in view of the fact that specifics with respect to implementation of such block diagram apparatus are highly dependent upon the platform within which the one or more embodiments of the present description are to be implemented (i.e., such specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the disclosure, it should be apparent to one skilled in the art that one or more embodiments of the disclosure can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative in nature and not as restrictive.
While the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of those embodiments will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
The present disclosure is intended to embrace all such alternatives, modifications and variances which fall within the broad scope of the appended claims. Any omissions, modifications, equivalents, improvements, and the like, which are within the spirit and principles of the one or more embodiments of the disclosure, are therefore intended to be included within the scope of the disclosure.
Claims (7)
1. A multi-target multi-region visual inspection method, comprising:
the product bearing device moves the product to be detected to an image acquisition position of the image acquisition device; each product to be detected comprises a plurality of different types of areas to be detected;
the image acquisition device scans and images the product to be detected, obtains an image to be detected of the product to be detected and sends the image to the image processing device; the image acquisition device acquires a whole board to-be-detected object as the to-be-detected image, the whole board to-be-detected object comprises one or more to-be-detected products, an interested region in the whole board to-be-detected object is obtained, a plurality of interested regions are added based on the grouping function of the plurality of interested regions, the added plurality of interested regions comprise a plurality of different types, the plurality of different interested regions are grouped, different groups are marked by adopting different colors to distinguish different groups so as to uniformly configure, edit and modify input parameters, attribute parameters and tolerance parameters belonging to the same grouped interested regions, and defects of different to-be-detected regions are defined in different manners;
the image processing device extracts a plurality of areas of a plurality of products to be detected based on the images to be detected to obtain a plurality of different types of areas to be detected of the products to be detected, invokes a defect detection template corresponding to the areas to be detected based on the different types of areas to be detected of the products to be detected to detect defects so as to obtain partial defect detection results of the areas to be detected, generates complete defect detection results of the same product to be detected based on the partial defect detection results of the different areas to be detected of the same product to be detected, and sends the complete defect detection results to an upper computer;
the upper computer displays the complete defect detection result, draws the complete defect detection result on an image to be detected, and distributes different colors for different defects;
the to-be-detected areas of different types comprise a circuit area, a golden finger area, a Mark area, a lattice area and an edge area; assigning different colors to different types of defects in the complete defect detection result;
further comprises: acquiring an interested region in the whole board to-be-detected target, wherein the interested region comprises all the products to be detected, dividing the interested region into rows and columns, and determining the product number of each product to be detected based on the position relation between the result of the row and column division and the products to be detected; or, acquiring the region of interest arranged in an array in the whole board to-be-detected target, wherein each region of interest comprises one to-be-detected product, and determining the product number of each to-be-detected product based on the position relation between the region of interest and the to-be-detected product.
2. The method of detecting according to claim 1, further comprising: and the image acquisition device simultaneously executes the transmission of the complete image of the product to be detected in the current frame and the shooting of the complete image of the product to be detected in the next frame.
3. The inspection method of claim 1, wherein the image acquisition device further comprises, prior to scanning the product to be inspected:
and obtaining the whole plate mark of the whole plate to-be-detected target.
4. The method of detecting according to claim 1, further comprising:
the image processing device determines the area mark of the area to be detected of the product to be detected, and processes the partial defect detection result belonging to the same product to be detected based on the product number and the area mark so as to generate a product defect detection result of the product to be detected.
5. The detection method according to claim 4, wherein the image acquisition device acquires a whole-plate mark of the whole-plate object to be detected; the method further comprises the steps of:
the image processing device generates the complete defect detection result of the whole board to-be-detected target based on the whole board mark and the product defect detection results of all the to-be-detected targets subordinate to the whole board.
6. The detection method according to claim 4, wherein the image acquisition device acquires a whole-plate mark of the whole-plate object to be detected; the defect detection is performed on the basis of a plurality of different types of the to-be-detected areas of a plurality of to-be-detected products to obtain partial defect detection results of the to-be-detected areas, including:
acquiring a pre-stored defect detection template based on the whole plate mark and the area mark; the defect detection template comprises a reference image and a mask image of a standard product;
generating a mask image to be detected of the region to be detected based on the region to be detected and the defect detection template;
performing self-adaptive enhancement filtering based on the mask image to be detected to obtain a filtered image to be detected;
and carrying out local dynamic threshold division based on the filtering image to be detected, and obtaining the partial defect detection result of the region to be detected based on the result of the local dynamic threshold division.
7. A multi-target multi-region visual inspection system, which is used for implementing the multi-target multi-region visual inspection method according to any one of claims 1-6, and comprises a product bearing device, an image acquisition device, an image processing device and an upper computer; wherein,
the product carrier is configured to: moving the product to be detected to an image acquisition position of an image acquisition device; each product to be detected comprises a plurality of different types of areas to be detected;
the image acquisition device is configured to: scanning and imaging the product to be detected, obtaining a complete image of the product to be detected, and sending the complete image to an image processing device; the image acquisition device acquires a whole board to-be-detected object as the to-be-detected image, the whole board to-be-detected object comprises one or more to-be-detected products, an interested region in the whole board to-be-detected object is obtained, a plurality of interested regions are added based on the grouping function of the plurality of interested regions, the added plurality of interested regions comprise a plurality of different types, the plurality of different interested regions are grouped, different groups are marked by adopting different colors to distinguish different groups so as to uniformly configure, edit and modify input parameters, attribute parameters and tolerance parameters belonging to the same grouped interested regions, and defects of different to-be-detected regions are defined in different manners;
the image processing apparatus is configured to: extracting a plurality of areas of a plurality of products to be detected to obtain a plurality of different types of areas to be detected of a plurality of products to be detected, calling defect detection templates corresponding to the areas to be detected based on the different types of areas to be detected of the plurality of products to be detected to detect defects so as to obtain partial defect detection results of the areas to be detected, generating complete defect detection results of the same product to be detected based on the partial defect detection results of the different areas to be detected of the same product to be detected, and sending the complete defect detection results to an upper computer;
the upper computer is configured to: displaying the complete defect detection result, drawing the complete defect detection result on an image to be detected, and distributing different colors for different defects;
the to-be-detected areas of different types comprise a circuit area, a golden finger area, a Mark area, a lattice area and an edge area; assigning different colors to different types of defects in the complete defect detection result;
the image processing apparatus is further configured to: acquiring an interested region in the whole board to-be-detected target, wherein the interested region comprises all the products to be detected, dividing the interested region into rows and columns, and determining the product number of each product to be detected based on the position relation between the result of the row and column division and the products to be detected; or, acquiring the region of interest arranged in an array in the whole board to-be-detected target, wherein each region of interest comprises one to-be-detected product, and determining the product number of each to-be-detected product based on the position relation between the region of interest and the to-be-detected product.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110139585.5A CN112964724B (en) | 2021-02-01 | 2021-02-01 | Multi-target multi-region visual detection method and detection system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110139585.5A CN112964724B (en) | 2021-02-01 | 2021-02-01 | Multi-target multi-region visual detection method and detection system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112964724A CN112964724A (en) | 2021-06-15 |
| CN112964724B true CN112964724B (en) | 2024-02-20 |
Family
ID=76273022
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110139585.5A Active CN112964724B (en) | 2021-02-01 | 2021-02-01 | Multi-target multi-region visual detection method and detection system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112964724B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114155367B (en) * | 2022-02-09 | 2022-05-13 | 北京阿丘科技有限公司 | Method, device and equipment for detecting defects of printed circuit board and storage medium |
| CN116843602B (en) * | 2022-03-25 | 2024-05-14 | 广州镭晨智能装备科技有限公司 | Defect detection method and visual detection equipment |
| CN114965487B (en) * | 2022-06-10 | 2024-06-14 | 招商局重庆交通科研设计院有限公司 | Calibration method and device of automatic monitoring equipment for tunnel typical damage |
| CN115690103A (en) * | 2022-12-30 | 2023-02-03 | 北京阿丘科技有限公司 | Product appearance detection method, device, equipment and storage medium |
| CN116309574B (en) * | 2023-05-19 | 2023-08-18 | 成都数之联科技股份有限公司 | Method, system, equipment and storage medium for detecting panel leakage process defects |
| CN117911411B (en) * | 2024-03-19 | 2024-05-24 | 南京认知物联网研究院有限公司 | Computer vision detection method and device based on parallel detection of picture streams |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103413150A (en) * | 2013-06-28 | 2013-11-27 | 广东电网公司电力科学研究院 | Power line defect diagnosis method based on visible light image |
| CN203535336U (en) * | 2013-11-04 | 2014-04-09 | 北京兆维电子(集团)有限责任公司 | LCD (Liquid Crystal Display) detecting system |
| CN105301007A (en) * | 2015-12-02 | 2016-02-03 | 中国计量学院 | Linear array CCD-based ABS gear ring defect online detection device and method |
| CN105445277A (en) * | 2015-11-12 | 2016-03-30 | 湖北工业大学 | Visual and intelligent detection method for surface quality of FPC (Flexible Printed Circuit) |
| CN105510348A (en) * | 2015-12-31 | 2016-04-20 | 南京协辰电子科技有限公司 | Flaw detection method and device of printed circuit board and detection equipment |
| CN107402221A (en) * | 2017-08-08 | 2017-11-28 | 广东工业大学 | A kind of defects of display panel recognition methods and system based on machine vision |
| CN107966447A (en) * | 2017-11-14 | 2018-04-27 | 浙江大学 | A kind of Surface Flaw Detection method based on convolutional neural networks |
| CN108636820A (en) * | 2018-03-30 | 2018-10-12 | 中国科学院自动化研究所 | The precision component automatic detection of view-based access control model and sorting system and method |
| CN109521022A (en) * | 2019-01-23 | 2019-03-26 | 苏州鼎纳自动化技术有限公司 | Touch screen defect detecting device based on the confocal camera of line |
| CN110441319A (en) * | 2019-09-09 | 2019-11-12 | 凌云光技术集团有限责任公司 | A kind of detection method and device of open defect |
| CN111650205A (en) * | 2020-05-11 | 2020-09-11 | 东风汽车集团有限公司 | Part surface defect detection method and system based on structured light image matching |
| CN111928797A (en) * | 2020-10-12 | 2020-11-13 | 山东海德智汇智能装备有限公司 | 3D high-precision detection system based on laser scanning imaging |
| CN111968082A (en) * | 2020-07-30 | 2020-11-20 | 陕西科技大学 | Product packaging defect detection and identification method based on machine vision |
| CN111999308A (en) * | 2020-08-27 | 2020-11-27 | 银河水滴科技（北京）有限公司 | Defect detection control method, control device and control system |
-
2021
- 2021-02-01 CN CN202110139585.5A patent/CN112964724B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103413150A (en) * | 2013-06-28 | 2013-11-27 | 广东电网公司电力科学研究院 | Power line defect diagnosis method based on visible light image |
| CN203535336U (en) * | 2013-11-04 | 2014-04-09 | 北京兆维电子(集团)有限责任公司 | LCD (Liquid Crystal Display) detecting system |
| CN105445277A (en) * | 2015-11-12 | 2016-03-30 | 湖北工业大学 | Visual and intelligent detection method for surface quality of FPC (Flexible Printed Circuit) |
| CN105301007A (en) * | 2015-12-02 | 2016-02-03 | 中国计量学院 | Linear array CCD-based ABS gear ring defect online detection device and method |
| CN105510348A (en) * | 2015-12-31 | 2016-04-20 | 南京协辰电子科技有限公司 | Flaw detection method and device of printed circuit board and detection equipment |
| CN107402221A (en) * | 2017-08-08 | 2017-11-28 | 广东工业大学 | A kind of defects of display panel recognition methods and system based on machine vision |
| CN107966447A (en) * | 2017-11-14 | 2018-04-27 | 浙江大学 | A kind of Surface Flaw Detection method based on convolutional neural networks |
| CN108636820A (en) * | 2018-03-30 | 2018-10-12 | 中国科学院自动化研究所 | The precision component automatic detection of view-based access control model and sorting system and method |
| CN109521022A (en) * | 2019-01-23 | 2019-03-26 | 苏州鼎纳自动化技术有限公司 | Touch screen defect detecting device based on the confocal camera of line |
| CN110441319A (en) * | 2019-09-09 | 2019-11-12 | 凌云光技术集团有限责任公司 | A kind of detection method and device of open defect |
| CN111650205A (en) * | 2020-05-11 | 2020-09-11 | 东风汽车集团有限公司 | Part surface defect detection method and system based on structured light image matching |
| CN111968082A (en) * | 2020-07-30 | 2020-11-20 | 陕西科技大学 | Product packaging defect detection and identification method based on machine vision |
| CN111999308A (en) * | 2020-08-27 | 2020-11-27 | 银河水滴科技（北京）有限公司 | Defect detection control method, control device and control system |
| CN111928797A (en) * | 2020-10-12 | 2020-11-13 | 山东海德智汇智能装备有限公司 | 3D high-precision detection system based on laser scanning imaging |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112964724A (en) | 2021-06-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112964724B (en) | Multi-target multi-region visual detection method and detection system | |
| CN107992881B (en) | Robot dynamic grabbing method and system | |
| CN107507167B (en) | Cargo tray detection method and system based on point cloud plane contour matching | |
| JP4429298B2 (en) | Object number detection device and object number detection method | |
| CN107014819A (en) | A kind of solar panel surface defects detection system and method | |
| CN110033431B (en) | Non-contact detection device and detection method for detecting corrosion area on surface of steel bridge | |
| CN108090434B (en) | Rapid ore identification method | |
| Hsieh et al. | A kinect-based people-flow counting system | |
| CN110940670B (en) | Machine vision-based flexographic printing label printing first manuscript detection system and implementation method thereof | |
| CN105223208B (en) | A kind of circuit board detecting template and preparation method thereof, circuit board detecting method | |
| CN106355579A (en) | Defect detecting method of cigarette carton surface wrinkles | |
| CN107957246A (en) | Article geometrical size measuring method on conveyer belt based on binocular vision | |
| JP2018036226A (en) | Image processing program, image processing method, and image processing device | |
| CN115775236A (en) | Surface tiny defect visual detection method and system based on multi-scale feature fusion | |
| CN110865077A (en) | Visual inspection system for appearance defects in RFID antenna production | |
| CN114235837A (en) | LED packaging surface defect detection method, device, medium and equipment based on machine vision | |
| CN113807378A (en) | Training data increment method, electronic device and computer readable recording medium | |
| Sun et al. | Cascaded detection method for surface defects of lead frame based on high-resolution detection images | |
| CN117456195A (en) | Abnormal image identification method and system based on depth fusion | |
| CN106558044A (en) | The resolution measuring method of image module | |
| CN101685000B (en) | Computer system and method for image boundary scan | |
| CN114708234A (en) | Method and device for identifying number of detonators on automatic bayonet coding all-in-one machine | |
| CN104966283A (en) | Imaging layered registering method | |
| CN110596118A (en) | Print pattern detection method and print pattern detection device | |
| JP4918615B2 (en) | Object number detection device and object number detection method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |